TL;DR Better way and Efficient methods to build a massive data structure (like a snapshot), or this is job for another language? In-deep In a Redis server, we have segmented objects by parameters. The base object is institute, this have a base json object with parameters like address, phone numbe…

How often does that data change?

Concurrently in the day, in a time period like 9am to 3pm are the high peak of changes (creation, modify, delete, etc), monday to friday. The low concurrency is between 3pm to 9am of the next day and weekend. Image as reference: [foto_no_exif]

You can have a look at this it might be of some help or atleast give you idea over hwo to tackle your problem [image] Interfacing Elixir with Rust to Improve Performance: Discord's Story When the Discord team hit a hard-limit on BEAM's performance dealing with large data structures, the…

In mid-term I proposed a refactor with a new project with a current elixir version (the project is in Elixir 1.6.5) to create a better scalable and updated system. Use Rustler to manage this kind of complex structures is a point in the proposal to the Leader. Yep, a goal for next year is learn Rust. …

Did you find out what part is actually taking time mostly? If building the data is 5 sec but the whole request takes 15-20.. it means something takes 10-15s :wink: Things to check before going further (e.g. rustler) Use iolist not string for redis command (if the library supports) Check Redis li…

The part that take mostl time is build N+1 structure. As I said, this each process, one by one, is where take so much time (the base data is append with more nested attribute in the process). I use Enum.map to process N+1 code of institute. I was think to use reduce in this case. The problem is wh…

[image] codeRaider: The part that take mostl time is build N+1 structure. As I said, this each process, one by one, is where take so much time (the base data is append with more nested attribute in the process). I use Enum.map to process N+1 code of institute. I was think to use reduce in this…

I’m not sure it sounds like Flow may be something to look into for a short term solution. It should help spread the map process out across all your cores.

500 is too small number to notice the slowness hm. Yes, but we want to prepare the API to hold >1000 objects. Check whether you have N+1 query problem. You may need to batch process but should not have N+1 query problem. Are you fetching institutes and all their associated data at once from da…

Oh! This is interesting. I’ll check if this is compatible with our current elixir version, I hope. Thnks!

Build correctly Massive Data Structure

Questions & Help Questions

darnahsan November 10, 2020, 7:06am 4

You can have a look at this it might be of some help or atleast give you idea over hwo to tackle your problem

https://github.com/discord/sorted_set_nif