ExCmd - communicate with external programs with back pressure

News Announcing
, , ,
10

Options like :stdin_only , :stdout_only and :stdin_and_stdout (the default) would help a lot. Boilerplate should be hidden away behind options and/or convenience functions like stream_stdin! , stream_stdout! and stream! (which expects both as it is right now).

Yes. I’m more inclined towards just adding additional no_stdin, no_stdout options and keep the same stream! interface. Just to avoid adding more functions, which can be confusing. But we can have separate functions if that makes more sense.

As for odu itself, it introduces a needlessly complex external dependency installation that some programmers might not be willing to subject themselves to. I’d suggest you write an in-app small Rust library. I can help you integrate Elixir with Rust – the release candidate of Rustler 0.22 has a much nicer and shorter syntax compared to previous versions and is now a joy to use.

Agree, I want to ditch the odu and have everything in a single library. When I started odu, I was mostly experimenting, and keeping it separate seemed simpler. Rustler looks interesting, I’ll look into it as soon as I get some time :slight_smile:.

Another thing, currently all this ceremony is because beam does not expose file descriptors for stdin and stdout. Definitely there will be some valid reason for that. But if we some how get a NIF/driver interface which let us access stdin/stdout fd, then we can just use :file.open with that fd and we can get rid of whole fifo thingy. This is all hand-waving, there might some issue in actual implementation.

Another thing: would there be a way to stream the spawned command’s output line by line? Currently I have to store a rather huge string in memory and then call String.split(the_whole_command_output, "\n") . Defeats the whole purpose of using Stream really.

We can split as soon as we get the output right? something like

    proc_stream
    |> Stream.transform("", fn data, acc ->
      lines = IO.iodata_to_binary([acc | data]) |> String.split("\n")
      Enum.split(lines, length(lines) - 1)
    end)

˚The size of data chunk we get (data in abve example) depends on the command we are running and when that command flushs its output and when we are issuing read. This size is limited by fifo buffer size which is controlled by OS. Usually this size will be max 65kb (sometimes its less in mac os). So unless user is explicitly collecting output for something there should not be memory leak.

I prefer to avoid adding spliting lines to ex_cmd itself. But if enough people want this, we can add

1 Like