I’m building a postgres-backed job queue for Elixir. The way I have it setup, I have a JobDispatcher that polls the DB every N milliseconds for available jobs. It queries the number of jobs that it has available capacity for and then distributes these to worker processes. I can imagine two ways to …

I’ve just released Rihanna as an initial implementation of the concepts discussed here. It uses a very simple single dispatcher with a poll-based approach, which nonetheless has a pretty high throughput of around 1.5k jobs/s due to it’s use of advisory locks. The supervision subtree looks like this…

Congrats! Isn’t Elixir a wonderful language for implementing concurrent job queues. I didn’t see a LICENSE file in the repo, is it MIT?

[image] mbuhot: I didn’t see a LICENSE file in the repo, is it MIT? Yup, I just added it. Both Rihanna and Rihanna UI are released under MIT license.

If something goes wrong in pulling and the genserver crashes you also lose all currently being executed jobs correct?

[image] jordiee: If something goes wrong in pulling and the genserver crashes you also lose all currently being executed jobs correct? Jobs are spawned underneath a Task.Supervisor so they will run to completion regardless of the state of the Dispatcher that spawned them. Since the Dispatcher…

Ah I see. admittedly I did not look at the code because I was on my phone and your diagram above looks like the job tasks are linked to the JobDispatcher. Pretty cool though I will take a look later!

Wonderful! :smiley: [image] samphilipd: [image] jordiee: If something goes wrong in pulling and the genserver crashes you also lose all currently being executed jobs correct? Jobs are spawned underneath a Task.Supervisor so they will run to completion regardless of the state of the Dispa…

@samphilipd Could you highlight the differences between Rihanna and EctoJob other than the obvious dependency on Ecto? Are there huge performance differences? I’m trying to pick between both of them for my first job queue. (EctoJob has the advantage of changing to other databases.. cassandra etc.…

EctoJob is very much tied to Postgres. The main difference in the design is transaction support. From what I can see, Rihanna doesn’t directly support enqueing a job as part of a larger transaction, while ecto_job strongly encourages it. Same for job completion - both frameworks are at-least-once …

Honeydew also recently added Ecto as a backend ( honeydew/README.md at master · koudelka/honeydew · GitHub ), implemented in a DB agnostic way. It will possibly hit scaling issues since it isn’t using the Postgres skip locked mechanism last I checked.

Elixir Programming Language Forum

How to best handle tasks when pulling from a job queue

Questions & Help Questions

LostKobrakai February 22, 2023, 10:17am 33

CubDB barely existed at the time of the post you answered