Hey! yes, we recently got big news!
Christopher Grainger has been working on Explorer, a dataframe library that currently uses Polars (Rust) which implements Arrow and can export to Parquet. Explorer is now part of Elixir NX libraries, so I think the community will focus on this, which is amazing!. But it is not ready yet.
The other approach that I was thinking to use before I knew about explorer, was DuckDB, which is a single C library / database (like Sqlite but columnar), that can also export to Parquet. But as I don’t have experience with C and Nifs, I was waiting for the ODBC driver which should be ready soon.
If you try any of these approaches please let us know how it went.
Best.
