Optimizing Mnesia offline queue fetch for large-scale data — feedback welcome

Questions & Help Questions
1

Hi all,

I’m designing an offline queue system using Mnesia in Elixir, where each eid/channel has millions of messages.

Here’s my current fetch method, which:

  • Uses a queuing_index table to store message IDs and offset for O(1) slicing

  • Fetches messages in batches (default 100)

  • Uses Task.async_stream for parallel batch reads

  • Wraps each batch in :mnesia.activity(:transaction, ...) to avoid :no_transaction errors

My questions:

  1. Does this approach make sense for large-scale queues (millions of messages per eid/channel)?

  2. Are there ways to improve memory efficiency and concurrency without breaking Mnesia transactional safety?

  3. Would you recommend an alternative data structure or method for queuing_index to make slicing/fetching faster?

Thanks in advance for feedback!

def fetch(eid, channel, limit \\ 50, batch_size \\ 100) do
  [{:queuing_index, {^eid, ^channel}, msg_ids, offset}] =
    :mnesia.transaction(fn -> :mnesia.read(:queuing_index, {eid, channel}) end)
    |> case do
      {:atomic, []} -> []
      {:atomic, [record]} -> [record]
      {:aborted, _} -> []
    end

  to_fetch_ids = Enum.slice(msg_ids, offset, limit)
  if to_fetch_ids == [], do: [], else: fetch_batches(eid, to_fetch_ids, batch_size)
end

defp fetch_batches(eid, to_fetch_ids, batch_size) do
  batches = Enum.chunk_every(to_fetch_ids, batch_size)

  batches
  |> Task.async_stream(
    fn batch ->
      :mnesia.activity(:transaction, fn ->
        match_spec =
          for msg_id <- batch do
            {{:queue, {eid, msg_id}, :_, :_}, [], [:"$_"]}
          end

        Enum.flat_map(match_spec, &(:mnesia.select(:queue, [&1])))
      end)
    end,
    max_concurrency: 8,
    timeout: :infinity
  )
  |> Enum.flat_map(fn {:ok, results} -> results end)
end