ExCmd - communicate with external programs with back pressure

The main issue which ExCmd tries to solve, which none of these libraries solve is having proper stream with back-pressure. In these libraries progress of the external command is uncontrolled.

Internally they all use the port and some sort middleware program for IO. We can never have back-pressure (or limit) using ports as the process mailbox is unbounded. ExCmd also uses port and a middleware program (odu) but it only uses port for controlling the external program not for IO. For IO it uses named pipe (FIFO) which is demand-driven.

One can just write the output to a file and read that from to “solve” this. But,

  • This involves disk IO for writing and reading which add latency
  • We can not control external program speed
  • we have to cleanup these files properly which might not be trivial
  • and IMHO ergonomics is better with ExCmd approach as the stream is more composable

Apart from these issues,

  1. erlexec: It is more focused on orchestrating and linking the external command than on communicating with it. If erlexec accepts os pid for its functionality maybe we can use erlexec with ExCmd
  2. porcelain: There are few important issues like zombie process and not having the ability to forcefully kill
  3. rambo: similar to porcelain, but rambo does not allow streaming input to stdin so all input must be kept in memory or it has to be passed to the command by writing to a file. It collects output binary in memory by appending it which is not efficient

I think when these libraries were created, they were not trying to solve the issues I mentioned, so I think its not correct to compare them. Please let me if anything is incorrect.

4 Likes