I have posted this in StackOverflow as well I have been into a little trouble lately: The memory used by GenServer processes is super high, probably because of large binary leaks. The problem comes from here: we receive large binaries through the GenServer and we pass them to the consumer, which t…

I work on a server that processes a lot of video and audio in real time as well, one of the first things I did is to write my own data pipeline workflow logic, for that I used only OTP. I see you are using gen_stage, after a quick glance I noticed this library buffers data and I will quote the docs : …

Have you tried invoking GC from producer as well as consumers?

Invoking manually the GC in an interval would not be a solution to the problem but a temporary fix for the memory usage. The solution would be fixing the root cause of the problem.

I will take that into account, would it be a huge change removing gen_server? So as you say, OTP would propably process data faster or?

What @xlphs said is that you should do a test with a simple GenServer instead of trying to use GenStage. GenStage try to buffer data to ensure that all stage have always something to do, and in this case, it means the data stays in the buffer. If you stop buffering then it will not stays and the GC …

You may be able to simulate this by simply changing the demand values to 1 to avoid buffering entirely. If this causes the desired change then the memory usage isn’t due to a leak, it’s just due to a buffer configured for more than you want.

Do you mean setting the option of demand_value to 1 for gen stage?

GenStage — gen_stage v1.3.2 This API has changed a little bit since I first used GenStage, but yeah you’d lower your max_demand to 1 on the consumer and producer consumer. The reason is that if you stick with the default it’ll buffer up to MIN_DEMAND which is 5…