Need help possibly improve CubDB large query

Questions & Help Questions
,
2

Here’s what I found:

select … returns a lazy stream

Additionally:

The returned lazy stream can be filtered, mapped, and transformed with standard functions in the Stream and Enum modules. The actual database read is deferred to when the stream is iterated or evaluated

And:

If we want to obtain the sum of the first 10 positive numeric values associated to keys from :a to :f, we can do:

sum =
  CubDB.select(db,
    min_key: :a,
    max_key: :f
  )
  |> Stream.map(fn {_key, value} -> value end) # map values
  |> Stream.filter(fn n -> is_number(n) and n > 0 end) # only positive numbers
  |> Stream.take(10) # take only the first 10 entries in the range
  |> Enum.sum() # sum the selected values

Using functions from the Stream module for mapping and filtering ensures that we do not fetch unnecessary items from the database. In the example above, for example, after fetching the first 10 entries satisfying the filter condition, no further entry is fetched from the database.

You might be able to increase the timeout? Or, why are you reading 1_000_000 as a test?

This might work, but why do it in the first place?

Task.async(fn ->
  CubDB.select(ref, min_key: {:tests, 1}, max_key: {:tests, 1_000_000}) 
  |> Enum.count()
end)
|> Task.await(:infinity)

Might be able to process in chunks?

CubDB.select(ref, min_key: {:tests, 1}, max_key: {:tests, 1_000_000})
|> Stream.chunk_every(10_000)
|> Enum.reduce(0, fn chunk, acc -> acc + length(chunk) end)