I’m having a very frustrating issue debugging something, and I’m now resorting to the forum for help. I’m sure people will recognize this question as I’ve exhausted a lot of options trying to solve it. I have an API that currently gets ~100,000 incoming requests/minute (1,600 RPS). For each incomi…

You don’t need to create new images just for debugging, what I suggest is connecting to an existing image and installing the packages for debugging there.

Based on what I’m reading it appears GKE still uses kube-dns kubectl -n kube-system get pods NAME READY STATUS RESTARTS AGE event-exporter-v0.2.3-54f94754f4-q6svn 2/2 Running 0 1d fluentd-gcp-sc…

Ok I got ss -s working. Here’s when no problems: bash-4.4# ss -s Total: 2782 (kernel 88998) TCP: 2683 (estab 511, closed 2167, orphaned 0, synrecv 0, timewait 2094/0), ports 0 Transport Total IP IPv6 * 88998 - - RAW 0 0 0 UDP 0 0 0 TCP…

This is what it looks like if I load it with more traffic than it can handle bash-4.4# netstat -tpan | grep TIME_WAIT | wc -l 21750 and bash-4.4# ss -s Total: 3307 (kernel 90342) TCP: 31988 (estab 993, closed 30990, orphaned 0, synrecv 0, timewait 30874/0), ports 0 Transport Total IP …

I managed to deploy to a plain 64 CPU instance in amazon. I hand installed everything and ran it in foregroud in production mode via ssh. Still same problem, although it got to about 50K total before dying, that is ~5-10% better but I can see when it starts to fail it looks like this: [image]

ok, did you try to test the API you are calling? Is it possible that they are not able to handle the load you are putting on them? I would use some outside tool like ab for example to verify that you are not getting the same timeouts.

I’m sure its not them because of a few things: a) they can handle millions of requests a second (they’re data APIs) b) if it were the API then simply multiplying the number of running apps on a single node wouldnt get wildly improved results

[image] 9mm: fs.aio-max-nr = 65536 afaik this should not be this low for a high concurrency server.. can you echo 1048576 > /proc/sys/fs/aio-max-nr or change in sysctl.conf and report back?

Elixir Programming Language Forum

Difficult debugging problem

Questions & Help Questions

9mm December 27, 2018, 8:35pm 50

Hey hey, i wrote them in the 2nd or 3rd reply if you scroll up