Explorer.DataFrame - load_ndjson json

Questions & Help Questions
1

Hello,

I’m trying to load a json API message with Explorer.DataFrame.

My api is returning a list like bellow:

resp = [%{"idade" => "8", "nome" => "Mirela"}, %{"idade" => "40", "nome" => "Sergio"}]

When I try to run the following code:

require Explorer.DataFrame, as: DF
DF.load_ndjson!(Jason.encode!(resp), lazy: true)

I got an error:

** (ErlangError) Erlang error: :nif_panicked
    (explorer 0.5.7) Explorer.PolarsBackend.Native.df_load_ndjson("...

But if I change to read the first element it works as well.

require Explorer.DataFrame, as: DF
DF.load_ndjson!(Jason.encode!(List.first(resp)))

How to read the entire data ?

Thank you.

2

The ndjson parser expects JSON objects separated by newlines - for instance, here’s an example from Explorer’s tests:

https://github.com/elixir-nx/explorer/blob/1db7146d83965f6446c4ab52816be4993952724c/test/explorer/data_frame/ndjson_test.exs#L36-L47

That is not what Jason.encode!(resp) will produce - that result will have [ at the beginning and , between the objects. You’d need to encode each item in resp separately.

HOWEVER

In general, re-encoding like that should make you suspect that you’re not using the best API for your situation.

The better choice is Explorer.DataFrame.new, in particular the last example labeled “From row data:”.