Tagging @sorentwo as he’s probably the only one to help me :slight_smile: I am using Oban.Pro, with SmartEngine and a queue limited with all: global_limit, rate_limit and local limit. The rate limit is per account_id, which is a parameter in args passed to worker. The usual scenario is that an ac…

[image] hubertlepicki: My guess is, somewhere where Oban queries for the next jobs, there’s a limit on the number of jobs it fetches. This limit is probably significantly smaller than 10,000. This probably results in the “next batch of jobs” being taken in only containing jobs of account_id=1. T…

Okay thanks. What we will do for now, and it may be actually a good enough solution in our case, is we will detect an account that schedules an unusually high amount of jobs. We will then schedule those jobs to a separate queue, where the blocking of concurrent jobs of other accounts doesn’t affect…

You could also space out large blocks of jobs to compensate. For example, if your rate limit only allows N jobs per hour, when you get a large influx of jobs, you can automatically space them out in blocks of N per hour: list_of_args |> Enum.chunk_every(1000) |> Enum.with_index() |> Enum.flat_map(f…

This is a very good idea. I think I may end up doing both things: offloading the jobs to separate queues and scheduling some of them in the future. Basically in your code, a small change: list_of_args |> Enum.chunk_every(1000) |> Enum.with_index() |> Enum.flat_map(fn {chunk, offset} -> opts = if…

[image] sorentwo: @hubertlepicki You’re guess is entirely correct. In fact, that workaround/optimization was put in place precisely because of issues you ran into previously. What I didn’t account for was that the “head of line” would block to that extent. I’m investigating a better solution. …

Correct, this is still an open issue. It isn’t forgotten, but I don’t have an easy fix yet.

Understood, thank you :). Working this around other ways then.

Is there a ticket for this issue or has there been a solve for this? Running into an issue similar to this and also wondering what actually happens to leads that are rate limited

There isn’t an open issue for this and it was “fixed” a while ago with a configuration option. The partition limit can be increased from the default of 5,000 as a compile-time option. For example, to up the limit to 10k: config :oban_pro, Oban.Pro.Engines.Smart, partition_limit: 10_000 Note that s…

Oban.SmartEngine :rate_limit doesn't work as expected

Questions & Help Questions

hubertlepicki March 8, 2023, 3:52pm 6

@sorentwo this is still an open issue, right?