Hi,
Sorry a bit late here. I worked on the adserver at Yieldbot and it had no issues handling 10,000 QPS on 6 instances (or was it 9) using IIRC 8 cpus servers with each instance using about 1G of ram. This should be doable. The servers were CPU bound not ram or network bound.
We were running our servers in Docker containers managed by mesos and singularity so it’s a bit different deployment. One thing we did run into is that the erlang VM thought it had resources for the underlying machine rather than what docker had allocated it. If you are running multiple instances on a single server they may all be creating schedulers for 64 cpus rather than the number you allocated for them. You’ll have way more schedulers than you need. Instead run several smaller servers 8 or 16 cpus and only a single instance of the application on each server. Setup haproxy or similar to handle routing to the instances. We used Baragon to update haproxy config when instances were started. Because we were using mesos we had to make sure only a single instance of a server came up on each slave. Also, new instances have to ramp up their load. If you hammer them all at once the schedulers get overloaded and the instance will crash. IIRC there is a haproxy setting to manage ramping up a new upstream server.
Not sure how Kubernetes works, but in docker/mesos the cpu allocation is a suggestion, but if the process uses too much, mesos kills the process.
Also IIRC don’t use bridge networking use host networking.
Definitely setup pools with hackney. The defaults are not good for this kind of load.
I should mention that Phoenix templates are awesome. We didn’t cache ads. We generated them on the fly. The server rendered html, css and js (along with a lot of checks and other processing) on every request and had a ~12ms response time.
I think that was most of the gotchas we ran into. And I hope I’m remembering everything correctly.
Oh, the only other thing was we had to open two ports. One for the server and one for epmd. Even if epmd wasn’t ever going to connect to another instance. haproxy only routed to the server port.
