I am downloading images through HTTP request with which I am getting a binary image, and writing it to a file such as File.write(image_with_dir, image, [:binary]) |> File.close this whole operation of getting HTTP request and then writing it to disk is done in |> List.flatten() |> Enum.sort() |> …

:waving_hand: Could How to download big files? - #3 by yurgon be helpful maybe?

Actually its about writing them to disk efficiently

The answer handles that part as well.

Open the destination file in raw mode with a big buffer (512KB at least, if not a few megabytes even). This also avoids the file operations going through a single internal process – this happens to all non-raw-opened files.

Okay then I coudnt find in that link :slight_smile:

How to open a file in raw mode? and with a big buffer as well?

You can try File.open(filename, [:append, {:delayed_write, buffer_size, delay}]) and then write to it with IO.binwrite , the post I linked above describes the last part better.

Try with the the standard File.open docs. :face_savoring_food: And yeah, @idi527 is right.

Since I am back at my machine: file = File.open!(path, [:raw, :write, {:delayed_write, 524_288, 2_000}]) Which basically gives you a much faster writing speed, utilising a generous 512KB buffer and 2 seconds of potential delay when the data will eventually make it to the disk (which should be plen…

A file opened in raw mode wouldn’t work with IO.write and IO.binwrite, I think? iex(1)> {:ok, fd} = File.open("some.file", [:raw, :write]) {:ok, {:file_descriptor, :prim_file, %{ handle: #Reference<0.3732712904.755892237.167770>, owner: #PID<0.105.0>, r_ahead_size: 0, r_buffer: #…

Writing 1K images to disk using Task.async_stream

Questions & Help Questions

idi527 May 30, 2020, 6:15am 2

Could How to download big files? - #3 by yurgon be helpful maybe?