# \#queues **URL:** https://forum.elixirforum.com/tag/queues/31.md [Latest](https://forum.elixirforum.com/latest.md) · [Categories](https://forum.elixirforum.com/categories.md) · [Tags](https://forum.elixirforum.com/tags.md) --- ## [BullMQ - Elixir port of Redis based message queue and batch processing for NodeJS and Python](https://forum.elixirforum.com/t/bullmq-elixir-port-of-redis-based-message-queue-and-batch-processing-for-nodejs-and-python/73599)
**Author:** [@manast](https://forum.elixirforum.com/u/manast)\ **Replies:** 0\ **Last updated:** [December 9, 2025, 11:52am UTC](https://forum.elixirforum.com/t/bullmq-elixir-port-of-redis-based-message-queue-and-batch-processing-for-nodejs-and-python/73599 "2025-12-09T11:52:11Z")
The popular NodeJS queue library BullMQ is now available for Elixir: Overview — BullMQ v1.3.3 It supports all the same features of the NodeJS version, but it leverages the power of Erlangs gen servers for achieving seaml… --- ## [Fairness in message queue system](https://forum.elixirforum.com/t/fairness-in-message-queue-system/69787)
**Author:** [@benonymus](https://forum.elixirforum.com/u/benonymus)\ **Replies:** 1\ **Last updated:** [March 14, 2025, 7:36pm UTC](https://forum.elixirforum.com/t/fairness-in-message-queue-system/69787 "2025-03-14T19:36:57Z")
Hey there, We are currently using Oban in a multi-tenant application with shared queues. We are running into a problem with noisy-neighbors. When one user can flood the queue and take up all the available resources mak… --- ## [Will GenStage.BroadcastDispatcher send the same event to every consumer when they’re ready for it?](https://forum.elixirforum.com/t/will-genstage-broadcastdispatcher-send-the-same-event-to-every-consumer-when-they-re-ready-for-it/65472)
**Author:** [@KP123](https://forum.elixirforum.com/u/KP123)\ **Replies:** 0\ **Last updated:** [August 13, 2024, 3:21pm UTC](https://forum.elixirforum.com/t/will-genstage-broadcastdispatcher-send-the-same-event-to-every-consumer-when-they-re-ready-for-it/65472 "2024-08-13T15:21:04Z")
I need to process the same piece of data in multiple ways based on user provided calculations. Am I correct in my reading of the docs that a GenStage.BroadcastDispatcher will send the same event to every consumer when t… --- ## [Equivalents of AWS Lambda and AWS SQS in the Elixir Ecosystem?](https://forum.elixirforum.com/t/equivalents-of-aws-lambda-and-aws-sqs-in-the-elixir-ecosystem/64227)
**Author:** [@acrolink](https://forum.elixirforum.com/u/acrolink)\ **Replies:** 1\ **Last updated:** [June 15, 2024, 3:05pm UTC](https://forum.elixirforum.com/t/equivalents-of-aws-lambda-and-aws-sqs-in-the-elixir-ecosystem/64227 "2024-06-15T15:05:15Z")
I am considering using things I am familiar with to do tasks in the background \[basically OCR processing\], namely: AWS Lambda and AWS SQS. My question, what are the equivalent tools and/or setups in the Elixir Ecosystem … --- ## [Local queuing before invoking external URLs](https://forum.elixirforum.com/t/local-queuing-before-invoking-external-urls/63207)
**Author:** [@rump13](https://forum.elixirforum.com/u/rump13)\ **Replies:** 5\ **Last updated:** [April 30, 2024, 3:06am UTC](https://forum.elixirforum.com/t/local-queuing-before-invoking-external-urls/63207 "2024-04-30T03:06:41Z")
Hello. I’m seeking advice on the following scenario. I have a Phoenix app that needs to invoke an external URL to put a message on an external event bus every time a significant event happens (e.g., user-created). To ens… --- ## [Using Process.sleep for rate limiting external requests?](https://forum.elixirforum.com/t/using-process-sleep-for-rate-limiting-external-requests/61537)
**Author:** [@\_jonas](https://forum.elixirforum.com/u/_jonas)\ **Replies:** 24\ **Last updated:** [February 16, 2024, 9:04pm UTC](https://forum.elixirforum.com/t/using-process-sleep-for-rate-limiting-external-requests/61537 "2024-02-16T21:04:16Z")
Hey I’ve got a quick beginner question: I’m making requests to an external API using Finch. The API is rate limited, and when the limit is hit, it returns a 429 response which helpfully comes with a Retry-After header … --- ## [Crawler - easy web crawling / scraping powered by GenStage](https://forum.elixirforum.com/t/crawler-easy-web-crawling-scraping-powered-by-genstage/8238)
**Author:** [@fredwu](https://forum.elixirforum.com/u/fredwu)\ **Replies:** 12\ **Last updated:** [September 28, 2023, 4:02pm UTC](https://forum.elixirforum.com/t/crawler-easy-web-crawling-scraping-powered-by-genstage/8238 "2023-09-28T16:02:38Z")
Hi all, Finally, after a month of hard work (with the occasional “oops it’s 5AM already?!”), I’m happy to announce that Crawler has reached v1.0. It has not been put in production use yet though, so please help test and… --- ## [What is the elixir way of processing long excels/csv files line by line (or any similar large inputs that is itemizable)](https://forum.elixirforum.com/t/what-is-the-elixir-way-of-processing-long-excels-csv-files-line-by-line-or-any-similar-large-inputs-that-is-itemizable/54441)
**Author:** [@artem](https://forum.elixirforum.com/u/artem)\ **Replies:** 10\ **Last updated:** [July 28, 2023, 8:39pm UTC](https://forum.elixirforum.com/t/what-is-the-elixir-way-of-processing-long-excels-csv-files-line-by-line-or-any-similar-large-inputs-that-is-itemizable/54441 "2023-07-28T20:39:58Z")
Hi there I am still learning my elixir ways, doing my first real elixir project. It is going to process excels or CSV files of user feedback, analyze them line by line with AI and provide useful summaries on e.g. whethe… --- ## [Commanded (and other CQRS/ES) \*\*instead\*\* of the long running jobs](https://forum.elixirforum.com/t/commanded-and-other-cqrs-es-instead-of-the-long-running-jobs/56026)
**Author:** [@artem](https://forum.elixirforum.com/u/artem)\ **Replies:** 0\ **Last updated:** [May 22, 2023, 3:13pm UTC](https://forum.elixirforum.com/t/commanded-and-other-cqrs-es-instead-of-the-long-running-jobs/56026 "2023-05-22T15:13:47Z")
Hi all I am trying-learning CQRS via a semi-hobby project on analyzing customer feedback. Basically you upload comments, system runs AI on it extracting sentiments, topics, statements and whatnot. This seems like quite… --- ## [What is the Elixir/OTP way for long queues that in JVM world would live on RabbitMQ or Kafka?](https://forum.elixirforum.com/t/what-is-the-elixir-otp-way-for-long-queues-that-in-jvm-world-would-live-on-rabbitmq-or-kafka/52831)
**Author:** [@artem](https://forum.elixirforum.com/u/artem)\ **Replies:** 8\ **Last updated:** [December 30, 2022, 4:48pm UTC](https://forum.elixirforum.com/t/what-is-the-elixir-otp-way-for-long-queues-that-in-jvm-world-would-live-on-rabbitmq-or-kafka/52831 "2022-12-30T16:48:52Z")
Hi All I am still learning Elixir in semi-hobby mode, so sorry if the question is stupid. Imagine an API that processes survey responses, possibly in computationally expensive way (maybe fashionable AI is being trained … --- ## [What libraries exist for managing / executing period jobs (like cron)](https://forum.elixirforum.com/t/what-libraries-exist-for-managing-executing-period-jobs-like-cron/49454)
**Author:** [@alexslade](https://forum.elixirforum.com/u/alexslade)\ **Replies:** 3\ **Last updated:** [August 10, 2022, 12:08am UTC](https://forum.elixirforum.com/t/what-libraries-exist-for-managing-executing-period-jobs-like-cron/49454 "2022-08-10T00:08:38Z")
I’m looking for an elixir application that will run and execute various jobs at defined intervals, a bit like a cron. Ideally with some kind of UI for reporting success/stats. Any recommendations? I’m struggling to find… --- ## [Pooling and queue libraries in Elixir](https://forum.elixirforum.com/t/pooling-and-queue-libraries-in-elixir/11472)
**Author:** [@sztosz](https://forum.elixirforum.com/u/sztosz)\ **Replies:** 11\ **Last updated:** [February 22, 2022, 2:48pm UTC](https://forum.elixirforum.com/t/pooling-and-queue-libraries-in-elixir/11472 "2022-02-22T14:48:56Z")
Hi, I’ll be definitely needing a pooling in my next small project in work, because I don’t want to kill Elasticsearch cluster. And I was wondering what are the libraries out there to use? One of obvious choices is to sim… --- ## [System design hack: Postgres is a great pub/sub & job server](https://forum.elixirforum.com/t/system-design-hack-postgres-is-a-great-pub-sub-job-server/44656)
**Author:** [@josefrichter](https://forum.elixirforum.com/u/josefrichter)\ **Replies:** 4\ **Last updated:** [December 27, 2021, 7:44am UTC](https://forum.elixirforum.com/t/system-design-hack-postgres-is-a-great-pub-sub-job-server/44656 "2021-12-27T07:44:47Z")
This is very interesting feature of Postgres I didn’t know about. Is anyone using it? --- ## [Pooling third-party API for each app user in the background - ideas?](https://forum.elixirforum.com/t/pooling-third-party-api-for-each-app-user-in-the-background-ideas/43131)
**Author:** [@ostap](https://forum.elixirforum.com/u/ostap)\ **Replies:** 2\ **Last updated:** [October 17, 2021, 8:31am UTC](https://forum.elixirforum.com/t/pooling-third-party-api-for-each-app-user-in-the-background-ideas/43131 "2021-10-17T08:31:17Z")
Hello, Elixir community! Imagine a web app with authentication. When users log in, they can create a new “item”. Items are stored in the database. For each item, my app needs to constantly call a third-party API and che… --- ## [Implementing a queue based on a list zipper?](https://forum.elixirforum.com/t/implementing-a-queue-based-on-a-list-zipper/43061)
**Author:** [@ryanzidago](https://forum.elixirforum.com/u/ryanzidago)\ **Replies:** 2\ **Last updated:** [October 14, 2021, 12:02am UTC](https://forum.elixirforum.com/t/implementing-a-queue-based-on-a-list-zipper/43061 "2021-10-14T00:02:09Z")
Hi all, How to implement a queue in Elixir with O(1) insertion and O(1) deletion using a list zipper? According to this blog post on zippers, one can use a list zipper to have a queue-like behaviour: Zipper lists are… --- ## [What's the difference between Oban and RabbitMQ](https://forum.elixirforum.com/t/whats-the-difference-between-oban-and-rabbitmq/37314)
**Author:** [@onkara](https://forum.elixirforum.com/u/onkara)\ **Replies:** 14\ **Last updated:** [February 11, 2021, 9:03pm UTC](https://forum.elixirforum.com/t/whats-the-difference-between-oban-and-rabbitmq/37314 "2021-02-11T21:03:18Z")
I am trying to wrap my head around the difference between Oban & RabbitMQ i.e. when does it make sense to use Oban vs RabbitMQ? I know RabbitMQ is one extra component in your infrastructure requiring special config, mon… --- ## [Oban perform\_at or perform\_in semantics like in Sidekiq](https://forum.elixirforum.com/t/oban-perform-at-or-perform-in-semantics-like-in-sidekiq/37329)
**Author:** [@onkara](https://forum.elixirforum.com/u/onkara)\ **Replies:** 1\ **Last updated:** [February 5, 2021, 2:00am UTC](https://forum.elixirforum.com/t/oban-perform-at-or-perform-in-semantics-like-in-sidekiq/37329 "2021-02-05T02:00:41Z")
I have a RoR app using Sidekiq which schedules many one time jobs to run on a dynamically computed time i future using perform\_in or perform\_at API of Sidekiq How do you accomplish the same using Oban? Looking at Oban s… --- ## [Yaq - An Enumerable Queue for Elixir](https://forum.elixirforum.com/t/yaq-an-enumerable-queue-for-elixir/31684)
**Author:** [@NullOranje](https://forum.elixirforum.com/u/NullOranje)\ **Replies:** 13\ **Last updated:** [May 21, 2020, 8:30pm UTC](https://forum.elixirforum.com/t/yaq-an-enumerable-queue-for-elixir/31684 "2020-05-21T20:30:13Z")
While working on a work project, I found I needed a more Elixir-like queue versus the Erlang :queue module. So I decided to write one. I liked what I did, so I’m sharing it with others. Yaq (Yet another queue) is a do… --- ## [What abstraction to use for Producer - consumer model with queue implementation](https://forum.elixirforum.com/t/what-abstraction-to-use-for-producer-consumer-model-with-queue-implementation/25139)
**Author:** [@sagar\_k317](https://forum.elixirforum.com/u/sagar_k317)\ **Replies:** 7\ **Last updated:** [September 17, 2019, 2:46pm UTC](https://forum.elixirforum.com/t/what-abstraction-to-use-for-producer-consumer-model-with-queue-implementation/25139 "2019-09-17T14:46:31Z")
Hi! I am trying to build a solution and I need guidance as to what type of abstractions to use. I have a pool of users and operators connected to my Phoenix app via channels. I have entries in my database for each opera… --- ## [Receiving & processing events via incoming Webhooks](https://forum.elixirforum.com/t/receiving-processing-events-via-incoming-webhooks/24575)
**Author:** [@Essstp](https://forum.elixirforum.com/u/Essstp)\ **Replies:** 4\ **Last updated:** [August 11, 2019, 11:16pm UTC](https://forum.elixirforum.com/t/receiving-processing-events-via-incoming-webhooks/24575 "2019-08-11T23:16:54Z")
I’m an Elixir developer (relatively new) from Australia and I am loving it :smile: I’m working on a SaaS startup which is an industry-specific / niche Learning Management System (LMS). Courses will be launched and playe… --- ## [Architecture for updating multiple states in isolation with implicit locking](https://forum.elixirforum.com/t/architecture-for-updating-multiple-states-in-isolation-with-implicit-locking/23979)
**Author:** [@lud](https://forum.elixirforum.com/u/lud)\ **Replies:** 8\ **Last updated:** [July 18, 2019, 5:09pm UTC](https://forum.elixirforum.com/t/architecture-for-updating-multiple-states-in-isolation-with-implicit-locking/23979 "2019-07-18T17:09:17Z")
Sorry for the title, I did not know what to write. Hello, I’m trying to build an architecture for a game. In the game, there are starbases which are basically event processors: they have a state, receive different ev… --- ## [Erlang :queue vs List](https://forum.elixirforum.com/t/erlang-queue-vs-list/20370)
**Author:** [@alvises](https://forum.elixirforum.com/u/alvises)\ **Replies:** 3\ **Last updated:** [February 22, 2019, 11:10pm UTC](https://forum.elixirforum.com/t/erlang-queue-vs-list/20370 "2019-02-22T23:10:42Z")
After benchmarking queues they seem much faster than Lists on appending. \[1,...1000\] ++ \[1001\] O(n) :queue.in(1,q) should be O(1) right? last element List.last(\[1,..,1000\]) again O(n) :queue.get\_r(q) again O(1) a… --- ## [GenServer fail safe queue](https://forum.elixirforum.com/t/genserver-fail-safe-queue/12321)
**Author:** [@quatermain](https://forum.elixirforum.com/u/quatermain)\ **Replies:** 10\ **Last updated:** [February 9, 2018, 4:35pm UTC](https://forum.elixirforum.com/t/genserver-fail-safe-queue/12321 "2018-02-09T16:35:08Z")
hello, I have phoenix application(not important) but I have a couple of GenServers with: use GenServer, restart: :transient, shutdown: 10\_000 and I have questions: How can I prepare code for situation when GenServer… --- ## [OPQ - A simple, in-memory queue with worker pooling and rate limiting](https://forum.elixirforum.com/t/opq-a-simple-in-memory-queue-with-worker-pooling-and-rate-limiting/7783)
**Author:** [@fredwu](https://forum.elixirforum.com/u/fredwu)\ **Replies:** 0\ **Last updated:** [August 14, 2017, 11:59am UTC](https://forum.elixirforum.com/t/opq-a-simple-in-memory-queue-with-worker-pooling-and-rate-limiting/7783 "2017-08-14T11:59:36Z")
It’s been fun looking at GenStage’s examples and trying to understand what they do and how to glue everything together. :slight\_smile: The end result? A lightweight library provides both worker pooling and rate limiting… --- ## [Multiple queues question](https://forum.elixirforum.com/t/multiple-queues-question/5578)
**Author:** [@mithereal](https://forum.elixirforum.com/u/mithereal)\ **Replies:** 0\ **Last updated:** [May 29, 2017, 11:32pm UTC](https://forum.elixirforum.com/t/multiple-queues-question/5578 "2017-05-29T23:32:55Z")
hello i am implementing a queue, to store a list of agents/cities so i can push/pop after i send a message to them ive tried to create a new genserver per city based on a table of cities but cannot get it to function p… --- ## [Que - Elixir Job Processing with Mnesia](https://forum.elixirforum.com/t/que-elixir-job-processing-with-mnesia/4871)
**Author:** [@sheharyarn](https://forum.elixirforum.com/u/sheharyarn)\ **Replies:** 9\ **Last updated:** [May 3, 2017, 2:16pm UTC](https://forum.elixirforum.com/t/que-elixir-job-processing-with-mnesia/4871 "2017-05-03T14:16:13Z")
A few months back I created this thread asking everyone’s opinion on different background job processing libraries available for Elixir and was encouraged to go with a simple GenServer at the time. I eventually needed t… --- ## [Honeydew 1.0.0-rc1: Distributed, Pluggable Job Queue](https://forum.elixirforum.com/t/honeydew-1-0-0-rc1-distributed-pluggable-job-queue/3307)
**Author:** [@koudelka](https://forum.elixirforum.com/u/koudelka)\ **Replies:** 0\ **Last updated:** [January 17, 2017, 12:21am UTC](https://forum.elixirforum.com/t/honeydew-1-0-0-rc1-distributed-pluggable-job-queue/3307 "2017-01-17T00:21:09Z")
Hey folks, I’ve recently rewritten the Honeydew job queue library on top of GenStage. It now supports distribution, cancellation, suspension, and filtration. It also has pluggable: Queues (persist jobs how you like) … --- ## [Jumbo: New job queueing library](https://forum.elixirforum.com/t/jumbo-new-job-queueing-library/3170)
**Author:** [@mspanc](https://forum.elixirforum.com/u/mspanc)\ **Replies:** 18\ **Last updated:** [January 9, 2017, 7:34pm UTC](https://forum.elixirforum.com/t/jumbo-new-job-queueing-library/3170 "2017-01-09T19:34:05Z")
Hello, I’ve just published 1.0.0 of Jumbo - a new job queueing library: GitHub - mspanc/jumbo: Reliable, OTP-style, lightweight job processing queue for the Elixir language · GitHub I was a bit disappointed by lack of … --- ## [How to handle a nil parameter inside an Ecto database query](https://forum.elixirforum.com/t/how-to-handle-a-nil-parameter-inside-an-ecto-database-query/3144)
**Author:** [@acrolink](https://forum.elixirforum.com/u/acrolink)\ **Replies:** 4\ **Last updated:** [January 5, 2017, 5:09pm UTC](https://forum.elixirforum.com/t/how-to-handle-a-nil-parameter-inside-an-ecto-database-query/3144 "2017-01-05T17:09:52Z")
I have created some basic query to load data from the database: page = Policy |\> join(:left, \[p\], c in Client, p.client\_id == c.id) |\> where(\[p, c\], c.user\_id == ^conn.assigns.current\_user.id) |\> where(\[p, c\], p.client\_… --- ## [Pass a list or other type to custom Ecto query](https://forum.elixirforum.com/t/pass-a-list-or-other-type-to-custom-ecto-query/2844)
**Author:** [@boriscy](https://forum.elixirforum.com/u/boriscy)\ **Replies:** 7\ **Last updated:** [December 14, 2016, 11:23am UTC](https://forum.elixirforum.com/t/pass-a-list-or-other-type-to-custom-ecto-query/2844 "2016-12-14T11:23:12Z")
I have a query like this one select o.id::text, o.name, o.geom, o.tags, o.address, o.open from organizations o, jsonb\_array\_elements(tags) as t where t-\>\>'text' in($1) Is there any way I can pass a list (Array) to the … [Next page](https://forum.elixirforum.com/tag/queues/31.md?match_all_tags=true&page=1&tags%5B%5D=queues)