How to efficiently send data between nodes?

Questions & Help Questions
, ,
1

Hi everyone.

In my system, I have two separated nodes which are connected to each other:

  • Node A have access to a database which has a lot of data.
  • Node B doesn’t have access to that database, so instead it asks node A to query that database and send back that data.

This works fine if the amount of data is small, but if there is a lot of data to be sent, it will basically consume all my RAM during the transfer and crash the node with OOM.

If I wanted to do that in the same node, an obvious solution would be to just use streams, But AFAIK I can’t just send a reference to a stream to another node and expect to be able to consume that stream.

For now, what I’m considering doing is using cursor pagination and just do a bunch of requests to the database until it doesn’t return more that, that way I can chunk it and send a controlled amount per request from node A to B.

I was wondering if there is a better way to do that, how do people normally tackle this kind of problems in the Erlang/Elixir world when they want to send huge amount of data between nodes? Maybe there is a version of Stream that works between nodes that I’m not aware?