How I Processed Millions of Database Updates with Go and RabbitMQ

Blogs & Podcasts Blog Posts
,
1

A while back, I had to process millions of database updates in a legacy system that was already hitting its 64 GB RAM limit, so scaling the existing application further wasn’t practical.

I ended up moving the workload out using:

Legacy app → RabbitMQ → Go worker → batched MySQL updates

I wrote a short post about the problem and the approach I used.

Now that I’m learning Elixir/OTP, I’m curious how you would approach the same problem on the BEAM.

1 Like
2

you may also want to try Ergo Framework - it provides you Erlang/OTP patterns but with Golang

1 Like
3

This sounds like a prime case for broadway. I cannot say how it would compare cpu/memory wise, but it can handle the pull from rabbitmq and batch for db update just fine.

3 Likes
4

I would just do the

query =  from(u in Users, where: u.id in ^ids)
Repo.update_all(query, set: [status: :processed])

If the list of ids is around one million of UUIDs, it is still just about 16 megabytes of memory. Sending it over to the database is fine too.

You don’t need RabbitMQ, Go worker or any other solution spanning multiple services to fix the N+1 problem. Even if I have 1GB of UUIDs, I’d just do batching in-service

2 Likes
5

Is it really necessary to introduce Go there? Isn’t it just converting the async job to a background task runner? I haven’t touch PHP for years but I believe PHP should be just fine to do that. What is the hidden challenge that promoted the play of Go here?

1 Like
6

Hi I was not part of their original team so the codebase was not familiar to me and the php on the server was outdated. What I tried was to isloate the problem. Very minimal change to their existing codebase also if my solution didn’t worked we can remove the go worker easily without major changes on the existing system. I know there are better ways thinking now but everyone was in a hurry so priority was to make it running as soon as possible. Thank you for the feedbacks.

1 Like
7

Thanks for the feedback. This was actually something the existing team had already tried, but the database was already under heavy load from other workloads as well. My approach was mainly to isolate the problem and reduce its impact on the existing system.

8

this video might interest you :slight_smile:

Practical Data Orchestration in Elixir - Silvia Zeamer | ElixirConf EU 2026

image
image1200×675 67.6 KB

1 Like
9

You might want to check out Concurrent Data Processing in Elixir: Fast, Resilient Applications with OTP, GenStage, Flow, and Broadway by Svilen Gospodinov

But really, it sounds like all that needs to happen is that this processing happen outside of the HTTP request.

1 Like