Oban: Tracking job progress / updating state during retries

[jordan0day]

I’m pretty new to Oban, and have what I think is a fairly simple use case, but had one question about how to handle something during the event of a failure/retry scenario:

I Enqueue a job with a list of IDs to process in an external service. These ID’s basically are just DB record ID’s.

• When the job kicks off, load up the records associated with the ID’s from above.
• Then fire off a request to an external service to basically initiate processing in that service (getting a unique “process ID” back).
• For each of the records I’ve loaded up from the DB, send that record (plus the 'unique process ID") to the external service.

Pretty straightforward?

My question is – if a failure occurs somewhere along the line here (either in the initial request to the external service, or while sending the additional messages to the service, I’d like for the job to retry, BUT:

• I want to track that “unique process ID” i retrieved from the external service
• I want to track the list of records that I’ve already successfully sent to the external service.

I assumed I could just update the Oban.Job record (maybe the args, maybe the metadata?) to hold this extra info – and when the worker’s perform/1 gets called again during the retry, I can just check if that extra info is present.
But the Oban docs don’t seem to cover anything around updating an Oban Job record, so I’m wondering if Job’s are really meant to be insert-only? From a user perspective, at least, there’s Oban.insert/3, but no Oban.update/x sort of function.

If I didn’t need to start off the job by retrieving that “unique process ID” from the external service, I could see just implementing a job-per-record (though it feels a little… extravagent? to break it down that much)

Am I thinking about this wrong?

[sorentwo]

@jordan0day

If I didn’t need to start off the job by retrieving that “unique process ID” from the external service, I could see just implementing a job-per-record (though it feels a little… extravagent? to break it down that much)

Am I thinking about this wrong?

You can have a job that spawns other sub-jobs, that’s entirely normal and encouraged. The job is the most granular level of processing available.

@jordan0day

But the Oban docs don’t seem to cover anything around updating an Oban Job record, so I’m wondering if Job’s are really meant to be insert-only? From a user perspective, at least, there’s Oban.insert/3, but no Oban.update/x sort of function.

You’re correct, there’s no top-level mechanism for updating jobs because there are many ways that can go wrong or cause issues. However, you can still use Repo.update(job, params) if you really want to change some fields.

[al2o3cr]

This may be leftover habit from other job queues that used Redis and were harder to update, but consider extracting the variable parts of the data to a separate database record and then pass the job that record’s ID. For concreteness, I’ll call that schema Receipt.

That also comes in handy if you want some parallelism but not job-per-item parallelism; you could instead queue up jobs to perform pagination-style “chunks” of the full list represented by the Receipt.

[jordan0day]

Yeah, I guess if breaking it out into job-per-record is sort of the normal pattern, that’s fine. Just felt like overkill for what I needed to do – but I’m happy to follow established patterns instead of trying to work around the tools in an awkward way.