Explorer.DataFrame.from_csv problem

@evadne Glad you figured it out :slight_smile:

FWIW I couldn’t get Polars to (which Explorer uses under the hood) to parse the timestamp directly either. So your cast approach is what I’d use too. I will throw a few more pointers out there.

DF.mutate/2 is nice because you don’t need to keep around the original column:

require Explorer.DataFrame, as: DF

path = "./ingestion_test.csv"

path
|> DF.from_csv!(dtypes: %{datetime_ns: :u64})
|> DF.mutate(datetime_ns: cast(datetime_ns, {:naive_datetime, :nanosecond}))
# #Explorer.DataFrame<
#   Polars[1 x 1]
#   datetime_ns naive_datetime[ns] [2025-02-03 09:03:33.286629]
# >

There is also the :lazy option. It will defer the computation so that you never create the intermediate column in the first place:

path
|> DF.from_csv!(dtypes: %{datetime_ns: :u64}, lazy: true)
|> DF.mutate(datetime_ns: cast(datetime_ns, {:naive_datetime, :nanosecond}))
|> DF.collect()
# #Explorer.DataFrame<
#   Polars[1 x 1]
#   datetime_ns naive_datetime[ns] [2025-02-03 09:03:33.286629]
# >

That option is only needed if you need to go real fast for some reason.

Also, we do have support for non-naive datetimes. But that doesn’t seem super relevant for this use case.