What I’m saying is that I can’t conclude anything reliably from that benchmark. That’s my main point I’m arguing in this thread.
No measurements are going to be perfectly reliable. The question is whether there is any useful information in the data. It sounds like you believe the answer is no and that if these same tech stacks were used in real-world scenarios, the relevant performance metrics would show little correlation with the TE benchmark results (i.e., you would expect to see essentially a random permutation of the rank ordering shown on the site). Ultimately, I suppose that’s an empirical question.
Once I established that, I didn’t care much about few microseconds here/there.
Sure, but I don’t think the argument for measuring performance is typically for the sake of saving a few microseconds (assuming that’s a small percentage savings for the task at hand). The argument I hear in favor of Phoenix is that it doesn’t require the typical tradeoff between productivity/developer happiness and speed/reliability. So, the claim is being made that its technical performance (both reliability and speed) is meaningfully superior to that of other options, and that technical performance does indeed matter. It’s hard to make such claims convincingly without doing some measurement (and comparison).
Once you have the tech which can handle your load, other things start to matter, such as the support for fault-tolerance, stable latency, and troubleshooting a running system. Erlang/Elixir excel at this, and that matters, because it improves the uptime and availability, and makes the life of developers much easier.
Yes, but how do you know if Elixir exhibits better fault-tolerance, stable latency, and uptime compared with other options? These are technical performance attributes as well. Presumably that would require some measurements of different systems under similar circumstances.
If you want, you could look at any given framework in a set of benchmarks to determine the capacity of that particular framework
That might be true, but one big problem I have with TE…
Yes, there are plenty of criticisms of TE, but I wasn’t referring to TE. I was addressing your more general claim that individual measurements in isolation are useful, whereas measurements of multiple platforms under similar circumstances are not. Given that the latter can be reduced to the former, this doesn’t make sense.
You could get bigger savings by choosing a technology which allows developers to efficiently and confidently manage that kind of load, and to keep the system stable and running as much as possible, and to reduce the load on the support team. Again, raw performance is just a part of the story.
Just substitute “stability” for “raw performance” in all of your arguments against comparative measurements, and it would seem we also have no way of determining that Elixir/Phoenix is generally any more stable or reliable than any other option. Presumably for each new project, we must build a realistic proof of concept in Elixir and several other stacks to see how each perform in that particular unique system (given, of course, that no two systems are ever sufficiently alike to generalize from any previous data or observations).
