It looks OK (as expected) to me 
From my point of view there are so many things that are off with the way they do these tests and they even say themselves that results are not comparable unless you are an “expert”.
From a quick glance:
- They use wrk which has the coordinated omission problem. So basically their test result can only be interpreted for closed system (i.e not something where there is no back-pressure from clients like the internet)
- They report the best case result for each framework. I think worst case is more interesting if I am running something in production
- For latency they use average latency for ranking. It is well known that averages are bad to use in this case. They could have used proper percentiles or if they don’t do any coordinated omission adjustment (which they don’t) they should use the max latency.
- Tests run for way too short. 15 seconds. Max concurrency 256 for anything but plaintext. They say themselves that they didn’t see any difference running test for 60 seconds or 15 seconds. Well I am not surprised:) 60 seconds is way too short as well! Once memory buffers run full, garbage collection kicks in, database connections get stuck you will see other problems. I’ve had servers that looked like they were running OK which ground to a halt and complete stop after 30 minutes. I understand it is hard to run the tests for all these frameworks for such a long time but still.
Are their tests useless? If you are contemplating setting up an http server in a closed system then perhaps no. They might be valid if you interpret the data correctly. If you have the http servers on an open system such as the internet? No then they are not valid.
And they say (Unfair comparisons / Tiamat):
“The tests set what can be considered high-water marks for performance.” That is, any framework is unlikely to perform better than these tests. So at least that is one thing you can takeaway from the results.
If you look through the data tables and latency graphs phoenix actually does pretty well. For latency it has low “max” and standard deviation which is not completely off the walls.
If you look how it performs under the various concurrency settings it performs as expected with little variation. Some of the faster frameworks fluctuated over 100% between the various concurrency settings. That is not what I call “stable”
Finally for these tests erlang VM has a bottleneck which hopefully will be addressed in OTP20 with multi poll-set. Not that it will be the best but I have a feeling it will do much better.
There is a danger with these tests that something like phoenix comes out in a bad light. There are python/php/javascript frameworks that come out on top in the way the present the data. I know from experience that in real life they stand no chance if you need performance out of the box.
I hope people don’t look at this and outright discards any framework because of it, but I think that is the case.
