ExCmd - communicate with external programs with back pressure

News Announcing
, , ,
6

I agree I’ll add an option to disable stdin

why do we need to spawn a Task for it to work?

tl;dr if we do not spawn separate process it will cause deadlock.

If anyone interested in this topic,
Stream is hiding synchronization happening between beam processes and external programs under the hood.

Without Task it would look something similar to this

proc_stream = ExCmd.stream!("find", ["/Users/dimi/Downloads/temp", "-name", "*.html"])
Enum.into([], proc_stream)
proc_stream |> Enum.to_list()

this is roughly equivalent to following steps with syscalls

1. create stream struct
2. syscall: fd = open("input.pipe", O_WRONLY)
3. syscall: close(fd)
4. run external program (at some point odu will call exec("cmd"))
5. syscall: fd = open("output.pipe", O_RDONLY)
...

step-2 is blocking call, this will return only after “input.pipe” is opened by the reader, which is the external program. But we start the external program at step-4, hence the deadlock.

This behavior is more visible if one uses low-level API instead of using stream abstraction.

open does have a non-blocking flag O_NONBLOCK. but,

  1. beam does not support passing this flag
  2. behavior is undefined for open with writer mode under POSIX

Interestingly, before OTP-21 allowed opening FIFO. A popular solution to open a FIFO in erlang/elixir was to use :erlang.open_port. :erlang.open_port is blocking call too, but in this case, it blocks the whole vm!

{_, 0} = System.cmd("mkfifo", ["test.pipe"])
spawn(fn -> Port.open('test.pipe', [:eof, :out]) end)
:timer.sleep(500) # force scheduler to execute fifo open
IO.puts "This line is never printed!"
1 Like