Oban - Reliable and Observable Job Processing

[sorentwo]

@benwilson512

Ah I see now, if the node that first tries the job goes down while the job is in progress, other nodes can determine that the job is abandoned by observing that there is a job in the "executing" state that lacks a corresponding lock.

Precisely

@benwilson512

I wonder if this could be achieved by a dedicated table that tracks cluster membership.

I’m sure it could, this is essentially how Redis based tools work. They use two lists (sidekiq, verk) or a list and a hash (kiq) to hold a backup of each job while it executes.

@benwilson512

Each node, when started, inserts a new row into a "nodes" table that contains basically just an id , the node name for debugging, and a alive_at timestamp.

What you’ve described is exactly how sidekiq and kiq handle tracking live nodes. It definitely works in that environment. I eschewed that in favor of advisory locks and pubsub to simply bookkeeping.

@benwilson512

I think advisory locks can be reasonably used by people’s application logic, but in a library it’s tough because it is easy to become incompatible with any other library that also uses them.

This is an excellent point. Based on your points and some of the other discussion in this thread I’m moving to the “namespaced” double int version.

Thanks for all the thought and feedback.