Streaming by chunks

Thanks for your answer! I realize I forgot to say that the input file is a text file whose entries are separated by new lines (JSONL or CSV).

Thus, using IO.binstream/2, reading line by line would result in chunks of 1 line (while what I want is to send, say, 5000 lines at once using streams) while reading chunks would, as far as I understand, result in getting a binary and not a stream, and therefore should be stored in memory before further processing, couldn’t be used with the Tesla API as-is. Plus I’d have to handle cases when a line is cut in the middle.

The more I think about it, the more I’m inclined to think that Streams just don’t work like that and there is not an easy trick to achieve what I want to do :slight_smile: