Hi All, I am prototyping a system using Broadway to process SQS messages and then using Flow to process each message. Each message being an S3 path of a file that is substantially large, around 200 to 400 MB compressed gzip, expanding to typically around 1 to 2 GB of text data. I’m using a few ver…

You can see it ran for over 10 minutes from the graphs. It should be way faster than that. Its not using the CPU enough. The instance I’m using has a 25gbit network link, best available. Using wget I can get a file at about 100mb/s, so 2-4 seconds. I had trouble getting metrics out of hackney, when…

What are the specs of each instance?

m5.24xlarge, 96 cores 384gb ram, 25 gigabit network link

[image] mgwidmann: Using wget I can get a file at about 100mb/s, so 2-4 seconds. Do you mean 100MB/s? Because at 100mb/s it should take much longer than 2-4 seconds per file. [image] mgwidmann: Doing some math, the configuration I have only leaves 32 mb/s of bandwidth per worker, so mayb…

I’d definitely start by isolating the various parts of the system. Instead of reading messages from SQS, start with a pre-defined list and benchmark how long it takes to work through that. You’re processing each file as far as I can tell 1 per CPU, this is going to have sub optimal download perform…

[image] sribe: Do you mean 100MB/s? Because at 100mb/s it should take much longer than 2-4 seconds per file. That number is what it showed when I tried fetching files via wget, so I assume its megabytes. [image] sribe: Do you mean 32MB/s? Because at 32mb/s, it should take much longer tha…

[image] benwilson512: You’re processing each file as far as I can tell 1 per CPU, this is going to have sub optimal download performance because S3 download speed per path isn’t particularly high. If you want to max that out you want multiple concurrent downloaders per path using something like E…

[image] mgwidmann: Didn’t realize I had it, but here is the network in from cloudwatch. I think you’re pretty much saturating your network connection.

Taking @benwilson512 ’s approach, I figured out one bug, wasn’t in “my” code though. Using SQS send-message instead of send-message-batch meant I wasn’t delivering work fast enough. Once that cleared up, I’m still bottlenecked at 70% CPU using ExAws (going to try soon with presigned links to see if …

Elixir Programming Language Forum

IO Bottlenecked Processing

Questions & Help Questions

sribe June 28, 2019, 6:51pm 2

What kind of throughput are you getting? How does that compare to what you expect?