I want to stream data from a file to JSON lines into an downstream system (ELK). To do that, ideally, I can use Tesla whose put/4 function accepts a stream for the body and write something like this: stream = File.stream!("input_file.ext", [:read]) |> Stream.map(&some_transform_function/1) cli…

Thanks for your answer! I realize I forgot to say that the input file is a text file whose entries are separated by new lines (JSONL or CSV). Thus, using IO.binstream/2, reading line by line would result in chunks of 1 line (while what I want is to send, say, 5000 lines at once using streams) while…

Elixir Programming Language Forum

Streaming by chunks

Questions & Help Questions

mbuhot September 5, 2020, 12:06pm 2

Maybe File.open/2 and IO.binstream/2 would let you stream successive chunks from the file?