In a project there’ll be lots of background tasks run by DynamicSupervisor. Each task will be making network requests to an external service and is expected to run for a few or 10-20 minutes. And a task will need to save a result into a DB. The way a task makes an network request could be in one o…

[image] OrontaMedu: The (2) in this case is more flexible, faster, more properly implemented and generally is preferable – compared to (1). Are you willing to sacrifice all the benefits of elixir for some performance? If yes then go with 2.

[image] OrontaMedu: Question: from the point of view of Elixir, background tasks, ability to run, manage them, get a return result and so on … – could the Elixir library (1) be better still than the non-Elixir one (2)? Yes. With external programs you have to marshall communication in both dir…

Not sure you can have something be faster when it hits the network. The CPU is just waiting for a response. And in your case you don’t even have a Rust library to call in-process, you have an external tool. I’d say stick to Elixir unless the Rust tool is bringing critically important benefits.

You can use Oban for running background jobs in Elixir. Can’t see any reason why this might be a bad choice.

[image] D4no0: Are you willing to sacrifice all the benefits of elixir for some performance? If yes then go with 2. Which benefits? Why sacrifise in this case?

[image] dimitarvp: And in your case you don’t even have a Rust library to call in-process, you have an external tool. What’s "Rust library to call in-process, "? Calling Rust via NIF and Rustler?

The job that the exteranl utility in Rust or Elixir library will do is to send several network requests. As well as returning a result to the calling Elixir code which then will be saved into a DB. There’ll be no computation. Do you claim implemeting it via NIF and Rustler would still be better …

[image] OrontaMedu: Do you claim implemeting it via NIF and Rustler would still be better than calling it as an utility via System.cmd(...) ? I am prone to being preachy and absolutist sometimes so maybe I am not the best person to give an answer… but I am leaning to “yes”. To me calling som…

Nobody cares. But that’s not the reason of the utility being written in Rust, for it – the external utility in Rust – itself depends on yet other libraries which do exist in Rust and don’t in Elixir. Or at least in Elixir some of them aren’t mature, up to date and maintaned.

Long running tasks and external processes: in Elixir or Rust?

Questions & Help Questions

OrontaMedu May 7, 2023, 1:42pm 7

What’s "Rust library to call in-process, "? Calling Rust via NIF and Rustler?