With blockchain technology on the rise and bigdata being where it already is, numerical crunching using native libraries should be a part of Elixir’s toolkit. I want a toolkit so that what I have in mind for the long term becomes feasible without having to invest in other languages.
@josevalim - I am sure you have an architecture in mind to execute this, or you have robust reasons as to why this is the “wrong thing” for Elixir. Either way I would really appreciate your inputs. It looks like the individual parts are available but not the sum of things. Is Numerical Crunching in Elixir missing some effort (development + architecture + supervisory) to fit things together ? Or is it that there are certain aspects of Erlang VM that make this a fool’s errand ?
First list is list of capabilities that exist - straight forward
Second list is list of capabilities that do not exist - From what I read on the forums, Elixir / Erlang lacks the following capabilities in order to be a contender in numerical processing.
Capability Exists:
INPUT/OUTPUT data from disparate non DB sources
- CSV @ GitHub - meh/cesso: CSV handling library for Elixir. · GitHub
- Excel reader @ xlsxir | Hex
- Excel writer @ elixlsx | Hex
Number crunching - arrays @ GitHub - Qqwy/elixir-arrays: Well-structured Arrays with fast random-element-access for Elixir, offering a common interface with multiple implementations with varying performance guarantees that can be switched in your configuration. · GitHub - stable ? fast ?
- tensors @ GitHub - Qqwy/elixir-tensor: The Tensor library adds support for Vectors, Matrixes and higher-dimension Tensors to Elixir. · GitHub - stable ? fast ?
- Pandas-style Dataframe @ GitHub - JordiPolo/dataframe: Package providing functionality similar to Python's Pandas or R's data.frame() · GitHub - abandoned ?
- NumEr @ GitHub - vascokk/NumEr: Numeric Erlang - vector and matrix operations with CUDA. Heavily inspired by Pteracuda - https://github.com/kevsmith/pteracuda · GitHub // vector and matrix operations with CUDA - abandoned ?
Spark-style Distributed processing - Does Elixir already possess this ?
- Does Elixir need an API for Spark ?
- Spark @ GitHub - cjbottaro/tempest: Like Apache Storm, but smaller and for Elixir · GitHub - abandoned ?
- LFE (Lisp Flavoured Erlang Wrapper Library for SciPy, NumPy, etc) @ GitHub - lfeutre/lsci: Scientific Computing on the Erlang VM - An LFE Wrapper Library for SciPy, NumPy, etc. · GitHub - exists for LFE, not Elixir. Is it useful ?
Capability does not exist:
Native NumPy/SciPy
- Possible via Python via ErlPort ?
- Possible via CUDA ? @ Big Data with Elixir
- Possible via BLAS ?
- Matlang for Erlang ?
- Generic Research for natively extending Elixir / Erlang using Rust
** Write Elixir extensions in Rust ? WITH
** Rust support ndarry similar to numpy @ GitHub - alexchandel/array-rs: A Rust library providing a multidimensional array object similar to NumPy's ndarray, with basic manipulation routines. · GitHub
** Rust API for numpy @ GitHub - PyO3/rust-numpy: PyO3-based Rust bindings of the NumPy C-API · GitHub WITH
** Safe Rust bridge for creating Erlang NIF functions - GitHub - rusterlium/rustler: Safe Rust bridge for creating Erlang NIF functions · GitHub
** Rewrite Pandas code in Rust ?
Erlang VM JIT support ? @ Erlang Factory SF Bay Area 2016 - ETA ?
Other references from this forum:
- Number crunching ? @ Using Elixir for Data Science and Machine Learning - #9 by WolfDan
