Populating database from CSV file

Questions & Help Questions
,
4

Hi @kylelw23 and welcome!

Apart from getting your code to run, as you already did with @soup’s help, I think there are a couple of things you could improve:

  1. is somebody consuming the return value of column_data ? If yes, then I suggest you keep it a stream in order to avoid storing all the inserted structs in memory when calling Enum.map(). Thus, consider replacing Enum.map() with Stream.map(). Using the latter will allow the result of the mapping step to be lazily evaluated by the consumer, which is more efficient if you’re dealing with a large number of rows. If, on the other hand, no one is consuming the return value of column_data and you are only interested in the side effect of inserting the rows into the DB, then do replace Enum.map() with Enum.each() to make this clear.

  2. In create_or_skip you are hitting the DB twice: once to check if a row with the given ID already exists, and then to insert the row if it doesn’t. Assuming you have a primary key constraint on the id column in your DB schema you could get rid of the first call by calling Repo.insert() with the on_conflict: :nothing option. The only downside is that the campaign struct returned by the insert, in case of a conflict, won’t be the one in the DB but the one you were trying to insert. This may or may not be a problem depending on who (if ever) is consuming this result (the answer to my question at point 1.)

I hope this helps!