Thanks, that’s nice, but the problem I’m trying to solve is somewhat different than that.
My problem: there’s a huge stream of items (millions of things) that has to be processed, one by one. I want to use something like Task.async_stream with option :max_concurrency set to someting reasonable, that spawns tasks for all elements of the stream but also limits concurrency. This is a well known pattern called pmap. The only problem with this approach is that all tasks run on the same node. So, switching to Task.Supervisor.async_stream instead of Task.async_stream is a bit better, since it has a single supervisor argument. (Now I can start those tasks on any node if there’s a Task.Supervisor on it.) It starts all tasks under that (possibly remote) supervisor. This is better, but still not spawning the processes across the whole cluster. Then comes the above GenServer into the picture. It is basically a fake supervisor. Task.Supervisor.async_stream can talk to this fake supervisor and ask it to start tasks. The fake supervisor then spawns tasks under real Task.Supervisor-s on different nodes, and reports back the pid to async_sream just like a normal supervisor would do.
One little problem with this is that when a task is started, but fails for some reason, there’s no way to know about the failure. All I can guarantee is that the task was successfully started, but who knows if it finished normally or not. Maybe using :transient restart strategy would help in some cases, but it will definitely not help when a whole node goes down with unfinished tasks on it.
Maybe there are some other drawbacks of this solution I am not aware of. I don’t know.
I wanted to ask the community if this is the right direction? Is it possible to ask async_stream to restart failing tasks somehow? Maybe my fake supervisor should keep track of the tasks by monitoring them, and when one fails, it should restart it on another node? Maybe I should use GenFlow instead of this hackie solution?
