I’ve given this question much consideration since my previous reply. Another thing to define is what you mean by ‘Big Data’. 5 years ago, it was all about Hadoop/Map-Reduce on data sets too large for a single machine.
More recently, people are tired of waiting for results and the tedium of actually programming Map-Reduce so you get things like Spark which use functional style to make the programming sane and close-enough-to-real-time processing.
I’m less familiar with Storm, Samza and Flink, but after a quick reading I’d say they all attack stream processing – such as log file analysis or credit fraud detection. Things where a decision is made on dynamic pools of data. I believe the same is possible with Spark, it’s just that these projects have their own specializations.
Just to muddy the waters further, we shouldn’t leave out Machine Learning and Deep Learning from the discussion.
I still have the feeling that Elixir could kill in all these areas. I have messed with Hadoop and Spark and the machinations they go through to manage nodes is unbelievable.
My plan for today is to watch those talks and do some more research on what is missing from Elixir’s ecosystem. Like you, I suspect it’s popularity.
