I have an app that downloads lots of large files from S3 throughout the day. I’m using ibrowse and streaming the files to disk. This works well, but cpu usage is pretty high when 5+ files are downloading at once. I know elixir isn’t the best choice for computation heavy tasks, but am I running into …

What are you using to download? I use gen_tcp directly and can ingest 100MB/s easily on low end PC, although I buffer the IO a lot. It’s definitely something with your download logic.

I’m using ibrowse with async requests. Here’s what the downloader module looks like. It’s called inside Task.async_stream. defmodule IbrowseDownloader do require Logger alias Downloader.Utils @doc false def download(url, destination, filename, request_headers) do config = %{ ur…

Is the result of IbrowseDownloader.download/4 the entire file contents? And are you sending those back to the process that originated the task before storing it to disk? If so that’s a lot of unnecessary messaging overhead and you’d be better off doing the entire “task” within a the Task. So downloa…

The return result is just an {:ok, file} tuple containing the full path to the downloaded file. The entire body of work is being done inside Task.async_stream. This is the code that initiates the downloads. Task.async_stream(files, fn {url, destination, filename} -> IbrowseDownloader.download(url…

I’ve never used ibrowse before. Can you log how often you are processing response? I think you need better rate control over how often you are writing to disk, the buffer size depends on hardware but in general try 64KB or larger. ibrowse should have something to let you control stream rate because …

I also haven’t used ibrowse but I’ve used httpotion several times with Task.async_stream and have been able to download 200 files sumultaneously for hours at a time (and store them to an NFS volume, all on a small VPS: 256MB of RAM) and when me and a colleague watched it remotely with :observer the …

I found what was causing my issues. One of the async responses I was getting was {:ibrowse_async_response, id, {:error, :req_timedout}} which matched the {:ibrowse_async_response, ^id, chunk} clause and thus called IO.binwrite with {:error, :req_timedout}. Once I added a clause to handle this and re…

Elixir Programming Language Forum

Is Elixir suitable for downloading multiple files simultaneously?

Questions & Help Questions

dimitarvp October 11, 2018, 6:17pm 9

Nice catch! Good job. Glad you solved it.