Difficult debugging problem

Questions & Help Questions
, ,
79

You’re welcome. :slight_smile:

  1. Incoming requests. We did make a request to a server on the local network on each incoming request so both. (As well as cache requests to scylladb.)

  2. We used HTTPoison and Tesla. Both were configured to use hackney. (IIRC hackney handled https connections better, but that may be history more than current status.)

  3. The pools had 256 for max_connections. But we had a pool for each outgoing service. Don’t forget to specify the pool in the options passed to get/post or you’ll get the default pool.

We found that the CPU limit was more from the scheduler being busy than requests. We knew it was time to scale when the message queues started to get behind.

Above I mentioned scylladb. We changed our caching from redis to scylladb. Performance was much better than redis.

One other thing I’ll throw out there is we wrote our phoenix controllers to do most everything in plugs. A controller would look like:

defmodule ServerWeb.FooController do
  use ServerWeb, :controller
  require Logger
  alias Plug.Conn

  plug :put_layout, false
  plug PluOne
  plug PlugTwo
  plug PlugThree
  plug :plug_four
  plug :plug_five
  plug PlugSix
  plug :plug_seven
  plug PlugEight

  def index(%Conn{assigns: %{param: param}} = conn, _params) do
    conn
    |> put_resp_header("cache-control", "no-cache")
    |> render("index.html", param: param)
  end
end

All the business logic happened in plugs. We could test them in isolation develop them in small isolated pieces. Some controllers had as many as 26 plugs.

I hope I am remembering all this correctly. Yieldbot shutdown the middle of December so all of this is from memory.