@ssagaert
Very interesting post.
We are talking in fact to create a possible synergy between Elixir and Python in the same pipe. Leveraging Elixir to access HDFS-like and inter-operate to Python for the map-reduce or machine-learning, and back again to Elixir for the database and Web inter-operability. This can be done by using protobuffer or a common swap space.
Again, one of the weakness of the software you mentioned is of being monolithic and to enforce the use of certain tools (above all Java).
Python can have some overhead problems at performing as the actual tools on the market, so creating a pipeline with the characteristics above can be a great plus for Python users (that are the great majority of data scientists) not using the Apache pipe.
I didn’t say, and the other the same, it’s easy or a few weeks work, but there is quite a demand to “ease” the actual big-data pipe.
2 Likes
