I’m working on a Phoenix app that supposed to offload jobs (that scrape websites and collect data) on a background worker thread. A quick google search reveals 3 libraries: Exq Verk Tonic All three use Redis at the back but Exq seems to be the most popular one, and I like the fact that it’s in Si…

I think you came around to the answer at the end there. IMHO it takes pretty specific requirements to make me even consider going outside beam for concurrent processing. That said I’ve heard of verk and IIRC it also supports sidekiq format. That situation is really only useful if you have a ruby ap…

[image] What is the best way to do stateful background jobs Questions There’s a lot that can be said here, but I’ll try to keep it brief To start with, we’re gonna set aside brief stuff that can just be handled with Task or synchronously. There are legitimate …

This is a great analysis, but I’ll nitpick on a couple of points. [image] benwilson512: GenServers lose their state when they die. :ets loses its data when it dies. Redis (like a database) lets you offload this problem Persisting the queue to disk directly from Erlang is straightforward usin…

These are great points and I’ve very excited you’ve brought them up. I’m in no way disputing that Erlang can do this, nor even that Erlang can make it easy to do this. My contention however is that for a large class of Elixir users who are relatively new to some of these approaches, there are not ye…

I was mostly suggesting that non-distributed local-node caching is simple to do with Erlang. If you don’t need to distribute the state, then running a background job processing (which was the original problem of this thread) is as easy as starting a process from the request handler :slight_smile: …

Background job processing in Elixir

Questions & Help Questions

benwilson512 October 22, 2016, 7:05pm 3

Background job queues: When to use? When not to use? Which one to use?