Hi all
I have some issues reading nanosecond precision timestamps in CSVs using latest explorer.
Repro as follows:
./script/ingestion_test.csv
datetime_ns
1738573413286629810
./script/ingestion_test.exs
Mix.install([
:explorer
])
{options, _, _} = OptionParser.parse(System.argv(), strict: [path: :string])
{:ok, path} = Keyword.fetch(options, :path)
Explorer.DataFrame.from_csv(Path.relative_to_cwd(path), [
infer_schema_length: 0,
columns: [
"datetime_ns"
],
dtypes: [
{"datetime_ns", {:naive_datetime, :nanosecond}}
]
])
|> IO.inspect()
Outcome:
$ elixir ./script/ingestion_test.exs --path ./script/ingestion_test.csv
{:error,
%RuntimeError{
message: "Polars Error: could not parse `1738573413286629810` as dtype `datetime[ns]` at column 'datetime_ns' (column number 1)\n\nThe current offset in the file is 12 bytes.\n\nYou might want to try:\n- increasing `infer_schema_length` (e.g. `infer_schema_length=10000`),\n- specifying correct dtype with the `dtypes` argument\n- setting `ignore_errors` to `True`,\n- adding `1738573413286629810` to the `null_values` list.\n\nOriginal error: ```could not find a 'date/datetime' pattern for '1738573413286629810'```"
}}
Surely I’m missing something elementary?
https://www.unixtimestamp.com/ agrees the timestamp is good. But it won’t parse.
Many thanks in advance
elixir 1.17.2
erlang 27.0.1
explorer 0.10.1
