Just to be clear: there’s no process communication happening here. Everything happens in the same process, since you’re working with both files in the raw mode (default for file streams). For clarification, see “Processes and raw files” in File doc1.
This is interesting. Initially I also thought there’s no process involved, but I kept that statement from the original article assuming that even if we don’t spawn an extra process (which happens with File.open in its default not-raw config) then there is still some low-level process within Erlang VM itself that’s responsible for serving file operations in a non-blocking fashion.
But then again, even if that would be true, I can’t really be sure if that could even be classified as “process”. It may be a thread instead and use other means for delivering data to our process than what we used to call “sending messages”. That seems to be the case according to the top part of this doc. So ultimately you’re right
It was a leftover from the original article where I’ve clearly mixed up the concepts of interprocess message passing with a tight stream loop that runs on the same process. I’ve fixed it.
Also, I couldn’t sleep over the fact that streamed version was about 3x slower on my machine than the read version (1.8s vs 0.6s). I played with it a bit more and discovered that streaming bytes works much faster than streaming lines.
This is amazing. I’ve tried taking it further, with this and then this, but failed to beat your version. I wonder if it could be made even faster, perhaps by adding some clever concurrency.
Note that I’m splitting input per bytes, so I’m not sure whether this will work correctly with unicode files.
As far as I understood the docs, String.split is UTF-8 aware. Also, you’re using utf8 qualifiers on every pattern match so it looks like it should work with UTF-8 just fine to me. The IO.binwrite doc includes a warning about using it with devices in unicode mode, but that’s not the case here. I’ve added some emoticons to input CSV and they are still there in the output, but I’m not sure that’s an ultimate proof of being UTF-8 compilant ![]()






















