Hello, I have found lovely CSV library that I will be using to insert CSV files into postgres. But there are some issues I am wondering about, as I am fresh in elixir. Ok, here is the sample code I will be using to insert the CSV into DB: File.stream!("ignore/customers.csv") |> CSV.decode |> Enu…

If you have a lot of records and don’t need data validation use COPY to pipe the data to postgres. This is very fast. The time between each insert is actually the time it takes to insert the record, So you insert a record and then another. So you probably don’t have to worry about overloading the s…

Thank you, it’s there an example of using copy? Then I think I won’t be able to broadcast progress messages? In general, no more than 25000 record expected in the CSV file

COPY ... TO stdout docs example throws errors · Issue #228 · elixir-ecto/postgrex · GitHub I would definitely use COPY for that volume

Yea, if I need one second for each individual row to insert, it would take hours to complete.. So, as for the link provided, how would I refactor their example below, so that it will be feeded by the CSV.decode in my code above? Postgrex.transaction(pid, fn(conn) → query = Postgrex.prepare!(con…

I recently faced a similar task, so hopefully some of my experiences will come in handy. COPY was my first thought. It’s the most efficient way to import CSV data to postgres, period. It’s also the least flexible way from an Elixir app perspective. Building a custom query, handling separators, err…

Great answer! From the earlier example, I would definitely use Repo.insert_all. Also the parallel processing can be done with Task.async_stream from Elixir v1.4.

Ok, here is my humble try with a CSV file that has 100 entry like: CSV: seq,first,last,email,digit 1,Antonio,Martinez,femeise@bi.gov ,348429376 2,Robert,Butler,amfob@vaim.io ,738848051 Code: NimbleCSV.define(MyParser, separator: ",", escape: "\"") File.stream!("/Users/samir/Downloads/100.csv")…

Task.async_stream returns a lazy stream - it won’t be executed unless you consume it. This can be done with Stream.run if you don’t care for results or any of the functions working on enumerables in an eager way if you do care about return values (most notably from the Enum module).

I do care about results, so I am not using Enum, in-fact, I will be broadcasting results to the client, to implement a progress bar of achievement of the task, as in the top of this thread. If I run the stream as Stream.run will it run all the chunks until the stream is all streamed? or shall I run…

Elixir Programming Language Forum

Inserting CSV into postgres

Questions & Help Questions

sabri February 9, 2017, 10:03am 8

Thanks for detailed answer, I will try to do the best thing I can now