This definitely seems interesting. Just a few comments:
- It seems like the workers are configured under the
Queapplication and not the user application. This can be an issue if the tasks use the user’s database for example.
Que starts, grabs a persisted job, and tries to run it. If the job talks to the database immediately the user’s Repo process may not yet be up.
-
Unfortunately your
Queuemodule isn’t a queue, it’s a stack. que/lib/que/queue.ex at master · sheharyarn/que · GitHub. You’re placing new jobs on the front of a list, and then pulling jobs off also from the front of a list, treating it as a stack not a queue. -
There seems to be no real isolation between queues. This is problematic for a variety of reasons. From a performance perspective everything is serialized through your Que.Server genserver, which won’t scale as you have more queues or as particular queues become very busy.
It’s also an issue from a fault tolerance perspective. Any issue with one queue will nuke every other queue.
- Isolation between workers is limited. https://github.com/sheharyarn/que/blob/master/lib/que/queue_set.ex#L102 goes through each queue completely synchronously and then just runs que/lib/que/queue.ex at master · sheharyarn/que · GitHub some N jobs in each queue under the main single task supervisor.
Each job is executed in its own process which is good, but by putting them under all the same supervisor with the same settings all you need is 5 job failures in 3 seconds to take down all the other jobs. You can of course just adjust the supervisor settings but even still, different supervisors for different queues would be better.
The root issue is that you do 99% of the stuff with a single genserver. There also appear to be race conditions. For example, two nodes starting at the same time will both query mnesia for incomplete jobs. You do that in a transaction, so one will run after another, but since you don’t update the state in that transaction they’re both still gonna get the same list of incomplete jobs and both try to run them.
I apologize that all of this is so negative. The issue is that while Erlang and Elixir are indeed good fits for the items on that list you have, the reason that they’re good is because Erlang and Elixir offer excellent primitives with which to solve the problem. Those primitives still need to be used correctly, and doing so can actually be pretty hard because the problems themselves are pretty hard.
