Our Phoenix production API is experiencing some intermittent very high latency on some requests (1s+, sometimes over 30s!) This seems to happen a few times per day with varying latency amounts. It’s not correlated with extra traffic, and the traffic on this API is low in general (20 requests/min). …

Thanks José! That will do the trick.

Instrumenting the plugs with the expansion was very helpful, thanks again. We can now see that the very high latency appears to be occurring in Plug.Parsers. We saw a few that were extremely slow (8s, 30s) vs the usual ~200us. Plugs before it & after it are very fast, as are other post requests of …

Have you tried to reproduce the problem by directly using the Plug.Parsers with the input that caused the slow parsing? If you have some input that causes serious slow downs then maybe it’s something that can be improved.

also consider large payload, slow client internet connection - how large are the payloads (megabytes?) - and is somebody uploading from a slow connection?

@tcoopman This is unfortunately not straightforward to gather since the logger truncates the base64 image strings. We did base64 decode the image that we eventually upload along with the rest of the payload, and response times were normal (fast). Still trying to gather the initial base64 string from…

@outlog The payload is moderately large (38k), but it’s worth noting we get much larger payloads that the parser rips through. Slow client internet connection - yes, this is entirely possible. It’s not a multipart form/file upload, just a single json payload. Could this still be a factor?

Could you time parsers and if parsing is slow store actual JSON somewhere?

Like @wanton7 is saying, just write the full request down on slow requests

@wanton7 @tcoopman Thanks - we may have more questions on doing that, though this is starting to feel more & more that it’s not about the payload. One new data point - a few minutes after the 30s parse, we saw an 8s parse. These both came from the same client. Many more of the same request type fro…

You can try to reproduce by faking a very slow client. Extremely slow client usually means some networking problem, like large amount of packets dropped. It may have something to do with your hosting company. On the other hand, elixir processes have smaller footprint compare to other stacks so it …

Phoenix API latency > 30s

Questions & Help Questions

Exadra37 January 22, 2021, 10:44pm 25

For anyone wanting a little more details on it: