You can see it ran for over 10 minutes from the graphs. It should be way faster than that. Its not using the CPU enough. The instance I’m using has a 25gbit network link, best available. Using wget I can get a file at about 100mb/s, so 2-4 seconds.
I had trouble getting metrics out of hackney, when I add metrics hackney doesn’t start up when built in a release (works fine otherwise). Easiest way to log "hackney.large_pool.free_count" in an Elixir app? · Issue #567 · benoitc/hackney · GitHub …
I would like to see how many connections are opened during load to determine how the 25gbit is split. Doing some math, the configuration I have only leaves 32 mb/s of bandwidth per worker, so maybe that explains why it takes 10-15 seconds to fetch files. I’m going to give it a try with half the number of workers, that should give me 64-65 mb/s bandwidth per worker. I’ll see if it gives me any better speed improvements.
