Due to the fact that Explorer (the elixir library) is based on Polars (the rust library), which is based on a rust implementation of Arrow (a standard to store data in memory), I have tried to re-implement SciEx to be based on Arrow instead of ndarray, in order to make it comaptible with Explorer. It would also make it much easier to store and move data around.
However, I haven’t been able to make it work in a satisfying way… Arrow doesn’t even provide a multidimensional array type you can operate on as a unit. It does provide tensors, but it does not provide vectoried operations on tensors and I can’t think of a good way to implement something like a Fast Fourier Transform algorithm on Arrow datatypes.
If anyone can think of a way to “port” SciEx to Arrrow I’d be very interested.
