# :queue vs genstage state

**URL:** <https://forum.elixirforum.com/t/queue-vs-genstage-state/53555>\
**Category:** Discussions\
**Tags:** queue\
**Created:** [January 31, 2023, 12:09pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555 "2023-01-31T12:09:29Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![fireproofsocks](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/fireproofsocks/32/7669_2.png) [@fireproofsocks](https://forum.elixirforum.com/u/fireproofsocks)\
**Post date:** [January 31, 2023, 12:09pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/1 "2023-01-31T12:09:29Z")

</div>

I’m expanding my understanding of genstage and how to build systems that are more resilient to overload. The `GenStage` docs demonstrate use of the Erlang `:queue` module… however, I am not clear on why that is being demonstrated. The `:queue` module seems to offer some convenience functions for working with lists, but it also has some inconveniences. E.g. it can’t work with a range like `1..5` – enumerables need to be converted into explicit lists, and certain operations like `split` fail if the queue is too small, so its use requires some “defensive coding”. Under the hood, an Erlang `:queue` is just a list, so I would expect the performance characteristics to be pretty much the same, and I would expect both to be equally susceptible to being over-stuffed.

I tried 2 variants of the same `GenStage` producer just to get a bit more familiar with `:queue`, and I think the 2 variants are functionally equivalent (although, the “defensive coding” required to use `:queue` makes me favor the simple list implementation).

Perhaps the conversation should be more about identifying the inputs to your system that could be “3rd rails” and overload it… A simple list (or a `:queue`) is acceptable for buffering demand or cases where the data stored in-memory is expected to remain tangibly small, but not when the input might be huge and unbounded. Anyhow, using `:queue` seems to be a distraction.

Thoughts?

---

<div class="post-metadata">

**Author:** ![LostKobrakai](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lostkobrakai/32/3072_2.png) [@LostKobrakai](https://forum.elixirforum.com/u/LostKobrakai)\
**Post date:** [January 31, 2023, 12:23pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/2 "2023-01-31T12:23:42Z")

</div>

> [@fireproofsocks](#):
>
> Under the hood, an Erlang `:queue` is just a list, so I would expect the performance characteristics to be pretty much the same, and I would expect both to be equally susceptible to being over-stuffed.

That’s not the case. Under the hood `:queue` uses a tuple with two separate lists, which has rather different performance characteristics for the usecase of a fifo queue (hence why `:queue` exists). With just a plain list either your queuing or dequeuing would be expensive due to how linked lists work. Queue shifts that by making both generally inexpensive at the expense of needing to rebalance head and tail ever so often.

But yes sheer size will eventually become a problem no matter how you store items.

---

<div class="post-metadata">

**Author:** ![cmo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/cmo/32/19618_2.png) [@cmo](https://forum.elixirforum.com/u/cmo)\
**Post date:** [January 31, 2023, 1:03pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/3 "2023-01-31T13:03:30Z")

</div>

`:queue` good. Use `:queue` in some GenServer and Broadway (i.e. GenStage) state.

I don’t feel like it required an excess of “defensive coding”. Do you mean checking if you got something out of it or not? You’re not doing that with a list?

---

<div class="post-metadata">

**Author:** ![fireproofsocks](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/fireproofsocks/32/7669_2.png) [@fireproofsocks](https://forum.elixirforum.com/u/fireproofsocks)\
**Post date:** [January 31, 2023, 1:03pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/4 "2023-01-31T13:03:30Z")

</div>

That makes sense. Thank you for the explanation!

---

<div class="post-metadata">

**Author:** ![fireproofsocks](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/fireproofsocks/32/7669_2.png) [@fireproofsocks](https://forum.elixirforum.com/u/fireproofsocks)\
**Post date:** [January 31, 2023, 1:05pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/5 "2023-01-31T13:05:10Z")

</div>

E.g. compare:

```elixir
  def handle_demand(demand, queue) when demand > 0 do
    {events, new_state} =
      case :queue.len(queue) > demand do
        true ->
          {head_queue, tail_queue} = :queue.split(demand, queue)
          {:queue.to_list(head_queue), tail_queue}

        false ->
          {:queue.to_list(queue), :queue.new()}
      end

    {:noreply, events, new_state}
  end

```

to

```elixir
  def handle_demand(demand, state) when demand > 0 do
    {events, new_state} = Enum.split(state, demand)
    {:noreply, events, new_state}
  end

```

---

<div class="post-metadata">

**Author:** ![cmo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/cmo/32/19618_2.png) [@cmo](https://forum.elixirforum.com/u/cmo)\
**Post date:** [January 31, 2023, 1:09pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/6 "2023-01-31T13:09:10Z")

</div>

Is it a good idea to use `len`? You could write yourself a recursive wrapper around `out`

```elixir
defmodule Queue do
  @doc """
  Take `count` items from the front of the queue.
  
  Returns a list of items and the new queue: `{items, queue}`. If there are
  less than `count` items in the queue, all of them will be returned. If the
  queue is empty, an empty list is returned.
  """
  @spec take(count :: pos_integer, :queue.queue()) :: {list, :queue.queue()}

```

---

<div class="post-metadata">

**Author:** ![fireproofsocks](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/fireproofsocks/32/7669_2.png) [@fireproofsocks](https://forum.elixirforum.com/u/fireproofsocks)\
**Post date:** [January 31, 2023, 1:24pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/7 "2023-01-31T13:24:01Z")

</div>

Well, you see what I mean: it requires additional work.

---

<div class="post-metadata">

**Author:** ![LostKobrakai](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lostkobrakai/32/3072_2.png) [@LostKobrakai](https://forum.elixirforum.com/u/LostKobrakai)\
**Post date:** [January 31, 2023, 1:35pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/8 "2023-01-31T13:35:51Z")

</div>

More code for a more appropriate and well working data structure though. You’re comparing apples to oranges here.

---

<div class="post-metadata">

**Author:** ![fireproofsocks](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/fireproofsocks/32/7669_2.png) [@fireproofsocks](https://forum.elixirforum.com/u/fireproofsocks)\
**Post date:** [January 31, 2023, 1:57pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/9 "2023-01-31T13:57:12Z")

</div>

My point was only that with the addition of the relatively unfamiliar (?) `:queue` module, the `GenStage` examples may veer into discussions about data structures instead of focusing on how stages should be organized. I would argue that understanding how to properly organize stages is more important to `GenStage`’s rai·son d’ê·tre, no?

---

<div class="post-metadata">

**Author:** ![LostKobrakai](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lostkobrakai/32/3072_2.png) [@LostKobrakai](https://forum.elixirforum.com/u/LostKobrakai)\
**Post date:** [January 31, 2023, 2:12pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/10 "2023-01-31T14:12:59Z")

</div>

It’s not as black and white. Yes simpler examples are generally a good thing. But using a list in the examples in question is not just making the examples simpler, but also a surefire way for people shooting themselves in the foot, when they copy from the example. Using a list to model a fifo queue is plain not what you want to do if this is meant to hold more than a handful of items.

---

<div class="post-metadata">

**Author:** ![stevensonmt](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/stevensonmt/32/20503_2.png) [@stevensonmt](https://forum.elixirforum.com/u/stevensonmt)\
**Post date:** [January 31, 2023, 2:21pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/11 "2023-01-31T14:21:25Z")

</div>

Thanks for this discussion. I’ve often wondered about the utility of `:queue` myself. It seems easy enough to use lists for this with the understanding that enqueueing would be expensive.

```elixir
#enqueue
q = 1..5 |> Enum.reduce([], fn i, q -> q ++ [i] end)
#dequeue
[hd | new_q] = q

```

I suspect this only matters beyond a certain size, which is where `:queue` becomes important.

---

<div class="post-metadata">

**Author:** ![ChrisYammine](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/chrisyammine/32/19254_2.png) [@ChrisYammine](https://forum.elixirforum.com/u/ChrisYammine)\
**Post date:** [January 31, 2023, 3:01pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/12 "2023-01-31T15:01:16Z")

</div>

No it’s a matter of behaviour – what you’re doing in that code example is popping off the top of a stack

Edit: nevermind, didn’t notice you were concatenating lists in your reduce function body.

---

<div class="post-metadata">

**Author:** ![cmo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/cmo/32/19618_2.png) [@cmo](https://forum.elixirforum.com/u/cmo)\
**Post date:** [January 31, 2023, 8:49pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/13 "2023-01-31T20:49:57Z")

</div>

One should practice good habits at all times.

I don’t think saving a few seconds or minutes is a valid reason to use the wrong data structure. It harms performance, clarity and professional growth.

---

<div class="post-metadata">

**Author:** ![stevensonmt](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/stevensonmt/32/20503_2.png) [@stevensonmt](https://forum.elixirforum.com/u/stevensonmt)\
**Post date:** [January 31, 2023, 9:15pm UTC](https://forum.elixirforum.com/t/queue-vs-genstage-state/53555/14 "2023-01-31T21:15:14Z")

</div>

FTR, I wasn’t trying to argue FOR my example over using `:queue` just stating why from a naive perspective it does seem puzzling.