# Help with \`:file\` blocking GenServer process

**URL:** <https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646>\
**Category:** Questions\
**Tags:** linux\
**Created:** [January 24, 2019, 4:32pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646 "2019-01-24T16:32:52Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 4:32pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/1 "2019-01-24T16:32:52Z")

</div>

I have a GenServer that reads/writes to a Linux Pipe or Fifo. I need to do:

```elixir
{:ok, fifo} = :file.open('/path/to/fifo', [:read, :write, :binary])
{:ok, <<protocol_pattern_match>>} = :file.read(fifo, protocol_size)

```

Except `:file.read/2` _blocks the entire calling process_ until it is complete similar to `read()` in C. This is fine because i should be able to just do `task = Task.async(:file, :read, [protocol_size])` and get the result in  
`handle_info/2`. Maybe i misunderstood the docs, but that doesn’t seem to be working for me. I’m expecting to get `handle_info({ref, {:ok, <<protocol_pattern_match>>}, %{task: %{ref: ref}})` but that doesn’t seem to happen.

My other issue is that `Task.shutdown/2` or `:file.close/1` do not seem to be working as expected.  
the docs for `Task.shutdown/2` say when a calling process exits, the task _should_ exit, but even using  
`Task.shutdown(state.task, :brutal_kill)` doesn’t allow me to call `:file.close(state.fifo)` in `terminate/2`. (It just blocks forever)

anyway here’s the entire GenServer implementation:

```elixir
defmodule PipeWorker do
  @moduledoc """
  Proxy for IO operations.
  """
  use GenServer
  require Logger

  def start_link(pipe_name) do
    GenServer.start_link( __MODULE__ , pipe_name)
  end

  def close(pipe) do
    GenServer.stop(pipe, :normal)
  end

  def read(pipe, size) do
    GenServer.call(pipe, {:read, [size]}, :infinity)
  end

  def write(pipe, packet) do
    GenServer.call(pipe, {:write, [packet]}, :infinity)
  end

  def init(pipe_name) do
    with {_, 0} <- System.cmd("mkfifo", [pipe_name]),
         {:ok, pipe} <- :file.open(to_charlist(pipe_name), [:read, :write, :binary]) do
      {:ok, %{pipe_name: pipe_name, pipe: pipe, task: nil, caller: nil}}
    else
      {:error, _} = error -> {:stop, error}
      {_, _num} -> {:stop, {:error, "mkfifo"}}
    end
  end

  def terminate(_, state) do
    Logger.warn("PipeWorker #{state.pipe_name} exit")
    state.task && Task.shutdown(state.task, :brutal_kill)
    Logger.warn("Pipe Task shut down")
    IO.inspect(state.pipe, label: "pipe")
    # :file.close(state.pipe) # blocks indefinitely no matter what. Shell becomes unresponsive.
    # Logger.warn("Pipe closed")
    File.rm!(state.pipe_name)
    Logger.warn("Pipe removed")
  end

  def handle_call({cmd, args}, {pid, _} = _from, state) do
    IO.inspect([state.pipe | args], label: "Pipe task args")
    task = Task.async(:file, cmd, [state.pipe | args])
    IO.inspect(task, label: "Pipe task")
    {:reply, task.ref, %{state | task: task, caller: pid}}
  end

  # This is never called?
  def handle_info({ref, result}, %{task: %{ref: ref}, caller: pid} = state) do
    IO.inspect({ref, result}, label: "Task result")
    send(pid, { __MODULE__ , ref, result})
    {:noreply, %{state | task: nil, caller: nil}}
  end
end

```

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 5:00pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/2 "2019-01-24T17:00:01Z")

</div>

Okay so while investigating this, i discovered something.

This code will hang forever no matter what. It doesn’t matter how much data  
is written to the pipe, the data will never be read by Erlang/Elixir.

```elixir
{:ok, file} = :file.open(path_to_fifo, [:read, :write, :binary])
{:ok, "hello world"} = :file.read(file, 11)
# in a different tab to `File.write(path_to_fifo, "hello world")`

```

This code however works fine. I suspect this is because the `:raw` option is a NIF now. (just a guess?)

```elixir
# notice the `:raw` option.
{:ok, file} = :file.open(path_to_fifo, [:read, :write, :binary, :raw]) 
{:ok, "hello world"} = :file.read(file, 11)
# in a different tab to `File.write(path_to_fifo, "hello world")`

```

The only problem with opening in `:raw` mode is that i can’t use `Task` to call `read/2` because the `Task` is not an owning process. I get the following error:

```elixir
** (ErlangError) Erlang error: :not_on_controlling_process
        :prim_file.get_fd_data/1
        :prim_file.read/2
        (elixir) lib/task/supervised.ex:89: Task.Supervised.do_apply/2
        (elixir) lib/task/supervised.ex:38: Task.Supervised.reply/5
        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

```

I guess this isn’t really an Elixir issue, but an Erlang one but any help would be appreciated.

---

<div class="post-metadata">

**Author:** ![idi527](https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/i/85e7bf/32.png) [@idi527](https://forum.elixirforum.com/u/idi527)\
**Post date:** [January 24, 2019, 6:15pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/3 "2019-01-24T18:15:06Z")

</div>

> [@ConnorRigby](#):
>
> task = Task.async(:file, :read, [protocol\_size])

Doesn’t that need to be used with `Task.await` to receive the result? I also think tasks use `receive` blocks that wouldn’t work in a genserver process.

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 6:17pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/4 "2019-01-24T18:17:07Z")

</div>

From the Task docs:

> [@](#):
>
> ## Compatibility with OTP behaviours
> 
> It is not recommended to `await` a long-running task inside an OTP behaviour such as [`GenServer`](https://hexdocs.pm/elixir/GenServer.html). Instead, you should match on the message coming from a task inside your [`GenServer.handle_info/2`](https://hexdocs.pm/elixir/GenServer.html#c:handle_info/2) callback. For more information on the format of the message, see the documentation for [`async/1`](https://hexdocs.pm/elixir/Task.html#async/1).

---

<div class="post-metadata">

**Author:** ![idi527](https://forum.elixirforum.com/letter_avatar_proxy/v4/letter/i/85e7bf/32.png) [@idi527](https://forum.elixirforum.com/u/idi527)\
**Post date:** [January 24, 2019, 6:19pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/5 "2019-01-24T18:19:15Z")

</div>

Oh, sorry then.

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 6:20pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/6 "2019-01-24T18:20:23Z")

</div>

if i manually change my code to do a `:file.read(fd, 0)` the behaviour works. I recieve the result in `handle_info`, so i at least am fairly certain `Task` is not the problem here.

---

<div class="post-metadata">

**Author:** ![josevalim](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josevalim/32/1787_2.png) [@josevalim](https://forum.elixirforum.com/u/josevalim)\
**Post date:** [January 24, 2019, 6:23pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/7 "2019-01-24T18:23:47Z")

</div>

The solution is to use ports, which will also give asynchrony. See previous discussion here: [Elixir vs Unix named pipe](https://forum.elixirforum.com/t/elixir-vs-unix-named-pipe/1134)

I think it was recently announced that erlang supports pipes, but I cant recall if in port or file. Or I may be completely misremebering it. Anyway, a port should do nowadays.

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 6:30pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/8 "2019-01-24T18:30:25Z")

</div>

Sorry, i should have stated this in the original post, but i have tried this as well and it did not work either.

```elixir
Interactive Elixir (1.7.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> System.cmd("mkfifo", ["/tmp/fifo", "-m", "0666"])          
{"", 0}
iex(2)> port = :erlang.open_port({:spawn, '/tmp/fifo'}, [:binary]) 
#Port<0.7>
iex(3)> sh: /tmp/fifo: Permission denied
                                        sh: line 0: exec: /tmp/fifo: cannot execute: Permission denied

```

and in another terminal, writing to the FIFO just hangs until i send a `ctrl+c`

```elixir
echo "hello world" -e > "/tmp/fifo" Thu 24 Jan 2019 10:28:34 AM PST
^CAn error occurred while redirecting file '/tmp/fifo'
open: Interrupted system call

```

---

<div class="post-metadata">

**Author:** ![josevalim](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josevalim/32/1787_2.png) [@josevalim](https://forum.elixirforum.com/u/josevalim)\
**Post date:** [January 24, 2019, 6:34pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/9 "2019-01-24T18:34:19Z")

</div>

So this worked:

```erl
~/OSS/elixir[jv-basic-releases %]$ mkfifo test.pipe
~/OSS/elixir[jv-basic-releases %]$ erl
Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V10.0 (abort with ^G)
1> Fifo = open_port("test.pipe", [eof]),
1> receive
1> Msg -> io:format("Got ~p", [Msg])
1> end.
Got {#Port<0.5>,{data,"foo\n"}}ok

```

After typing the receive, I ran `echo "foo" > test.pipe` in another terminal. Could it be something related to permissions as the error message says?

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 6:36pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/10 "2019-01-24T18:36:50Z")

</div>

ah ha! i was trying to use `spawn` which was not needed. Will try this method out and report back.

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 6:45pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/11 "2019-01-24T18:45:27Z")

</div>

I can confirm that this _does_ in fact allow reading and writing from a named pipe. However it does add the requirement of buffering now, since you are not telling the port how much data you want to read. Either way thanks for the help!

---

<div class="post-metadata">

**Author:** ![josevalim](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/josevalim/32/1787_2.png) [@josevalim](https://forum.elixirforum.com/u/josevalim)\
**Post date:** [January 24, 2019, 7:03pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/12 "2019-01-24T19:03:19Z")

</div>

Good point. Can that be done at the pipe level? I.e. have some UNIX utility that buffers it for you or similar?

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 7:07pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/13 "2019-01-24T19:07:31Z")

</div>

I’m still investigating solutions. I whipped up a quick NIF that works similarly to `:file.open('test.pipe', [:read, :write, :raw]` but allows for calling `read/2` within a `Task`. Unfortunately this is the best solution so far, however i don’t like putting NIFs into production unless there is _no_ other options.

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 8:33pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/14 "2019-01-24T20:33:54Z")

</div>

For completeness here is my (mostly) final implementation:

```elixir
defmodule PipeWorker do
  @moduledoc """
  Proxy for Pipe IO operations.
  """
  use GenServer
  require Logger

  def start_link(pipe_name) do
    GenServer.start_link( __MODULE__ , [pipe_name])
  end

  def close(pipe) do
    GenServer.stop(pipe, :normal)
  end

  def read(pipe, amnt) do
    GenServer.call(pipe, {:read, amnt})
  end

  def write(pipe, packet) do
    GenServer.call(pipe, {:write, packet}, :infinity)
  end

  def init([pipe_name]) do
    with {_, 0} <- System.cmd("mkfifo", [pipe_name]),
         pipe <- :erlang.open_port(to_charlist(pipe_name), [:eof, :binary]) do
      {:ok, %{pipe_name: pipe_name, pipe: pipe, buffer: <<>>, caller: nil, size: nil}}
    else
      {:error, _} = error -> {:stop, error}
      {_, _num} -> {:stop, {:error, "mkfifo"}}
    end
  end

  def terminate(_, state) do
    Logger.warn("PipeWorker #{state.pipe_name} exit")
    :erlang.port_close(state.pipe)
    File.rm!(state.pipe_name)
  end

  def handle_call({:write, packet}, _from, state) do
    reply = :erlang.port_command(state.pipe, packet)
    {:reply, reply, state}
  end

  def handle_call({:read, amnt}, {_pid, ref} = from, %{caller: nil, size: nil} = state) do
    {:reply, ref, %{state | caller: from, size: amnt, buffer: nil}, {:continue, state.buffer}}
  end

  def handle_info({pipe, {:data, data}}, %{pipe: pipe, buffer: buffer} = state) do
    buffer = buffer <> data
    {:noreply, %{state | buffer: nil}, {:continue, buffer}}
  end

  def handle_continue(buffer, %{size: size} = state) when byte_size(buffer) >= size do
    {pid, ref} = state.caller
    {resp, buffer} = String.split_at(buffer, size)
    send(pid, { __MODULE__ , ref, {:ok, resp}})
    {:noreply, %{state | caller: nil, size: nil, buffer: buffer}}
  end

  def handle_continue(buffer, state) do
    {:noreply, %{state | buffer: buffer}}
  end
end

```

and usage is something like:

```elixir
# Start the Pipe
{:ok, pid} = PipeWorker.start_link("/tmp/test.pipe")
# Start an asynchronous read of 10 bytes
ref = PipeWorker.read(pid, 10)
# This can obviously be done in `handle_info` of a GenServer or other OTP behaviour.
receive do
   {PipeWorker, ^ref, {:ok, <<protocol_pattern_match>> = data} -> handle_data(data) 
end

```

For **EVEN MORE** completeness here’s the NIF implementation i won’t be using: [https://github.com/ConnorRigby/pipe\_fitter/blob/master/c\_src/erl\_pipe\_nif.c](https://github.com/ConnorRigby/pipe_fitter/blob/master/c_src/erl_pipe_nif.c)  
where usage is something like:

```elixir
# Start the pipe
{:ok, pipe} = :erl_pipe_nif.start('/tmp/test.pipe')
# Tell the NIF we want a message as soon as data is ready. 
# This is powered by `enif_select`
:ok = :erl_pipe_nif.poll(pipe)
# This can obviously be done in `handle_info` of a GenServer or other OTP behaviour.
receive do
   {:select, ^pipe, _, _} -> 
      # Read the data. Still blocking here, but as long as it is only called after the `:select`
      # message, NBD. 
      :erl_pipe_nif.read(pipe, 10)
end

```

Anyway Thanks again @josevalim for making me check out the `Port` option again. Hope the information here is enough to answer anyone elses questions in the future.

---

<div class="post-metadata">

**Author:** ![rvirding](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rvirding/32/1409_2.png) [@rvirding](https://forum.elixirforum.com/u/rvirding)\
**Post date:** [January 24, 2019, 10:24pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/15 "2019-01-24T22:24:12Z")

</div>

I have a stupid question. You are reading with:

```
{:ok, <<protocol_pattern_match>>} = :file.read(fifo, protocol_size)

```

Do you write enough bytes into the fifo so this read can complete? I expect yes but it is always best to ask. Start with the simple questions first. 😄

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 24, 2019, 10:27pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/16 "2019-01-24T22:27:36Z")

</div>

Yes, but i understand what you are saying. I understand that `file:read(Fifo, Size)` will block until `Size` amount of bytes are available. This is fine, but in my case i didn’t want to block the `gen_server` process while waiting on those bytes to arrive. In my case they may _never_ arrive for the life of this fifo, meaning i need to be able to cancel, or stop the existing `read` operation.

---

<div class="post-metadata">

**Author:** ![rvirding](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/rvirding/32/1409_2.png) [@rvirding](https://forum.elixirforum.com/u/rvirding)\
**Post date:** [January 24, 2019, 11:02pm UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/17 "2019-01-24T23:02:40Z")

</div>

Well the `file:read` operation blocks by design. Two solutions have already been mentioned: put the read in a separate process which sends you a message when done and which you can kill when you give up on it; use ports which are non-blocking.

---

<div class="post-metadata">

**Author:** ![ConnorRigby](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/connorrigby/32/5964_2.png) [@ConnorRigby](https://forum.elixirforum.com/u/ConnorRigby)\
**Post date:** [January 25, 2019, 12:41am UTC](https://forum.elixirforum.com/t/help-with-file-blocking-genserver-process/19646/18 "2019-01-25T00:41:44Z")

</div>

Right. I chose the port method and it seems to be working well!