EDIT: Ignore the Enum.each comments, I mistook it for map. My bad.
First commit:
- Not sure why you use
Poison.Jasonis the newer library and is faster. - You still have not made your code use string map keys instead of atoms. You only get 1_048_476 atoms (here is the source) and they are never garbage-collected. It’s easy to execute a denial-of-service attack against Elixir code that blindly accepts all atoms thrown at it via external data. I mean this is not actually a major concern in the grand scheme of things (you will probably protect your API behind an authentication wall) but since you are trying to troubleshoot a memory consumption problem I believe it’s worth the effort to modify your code to never use atom JSON keys. I personally also believe your producer/consumer queues hold on to strings for a long time and that’s your problem but hey, why not try everything since you decided to try and tackle the problem?
- Outside of those, I believe the commit mostly covers what @aseigo meant earlier.
Second commit: if you are not interested in return value of Enum.each I suggest the following instead.
Change all these:
Enum.each(data.guilds, fn(guild) -> handle(:GUILD_UPDATE, guild) end)
to these:
Stream.each(data.guilds, fn(guild) -> handle(:GUILD_UPDATE, guild) end)
|> Stream.run
Sure it’s little longer but (1) it does not create a return value and does not pass and modify it between each iteration only for it to eventually be discarded, and (2) communicates the intent of the code better: you are not interested in the results of the loop, you only invoke it for the side effects.
Do this everywhere you do NOT assign anything to the Enum.each calls.
