Any open source app available that does Job Processing AND Rate Limiting?

I think you got analysis paralysis. :smiley:

Let’s separate your concerns because you seem to be mixing them somewhat.

  • Persistence. Depends how important is the sending of these emails. If they are not mega-important and you don’t care about guaranteeing each one arriving then there’s nothing wrong to just put a few million records in ETS (Erlang’s in-memory DB/cache) and then pull from it in batches and send mails. Redis works fine for this as well. Kafka too but it’s too heavy to setup (unless you’re using Docker, then it’s easy-ish). PostgreSQL is just fine for it as well.

  • Rate limiting. It’s really not as hard to make one yourself as you seem to think. You’d need a combination of :counters.new and one of the functions of the :timer module (both in Erlang but fairly straightforward to use). If not, it’s still better to try and use a library that’s made only for rate limiting so as to reduce confusion.

I could likely write you one for your goals but I am super busy and my schedule is not stable so can’t give you a good estimation as to when.

Yes, absolutely. You haven’t indicated anything in your comments that suggests a super complex scenario. Any of the options you enumerated should serve you just fine.