Good morning fellow alchemists!
I have a question about quite a low level implementation topic regarding copy/concurrency semantics, let me explain the situation briefly first:
- I have some (~10) ETS tables (set/unordered) that contain a few 100k records each, where the primary keys are either straight integers or tuples of integers (mimicking compound keys)
- the values in the tables mostly are HashSet of integers (“IDs”)
- the tables are owned by dedicated GenServer processes, so that only the owning process can update data within its ETS table, but all other processes can read in parallel from it
- the owning processes subscribe to phoenix PubSub events in order to update individual ETS records in realtime as needed
- there is a general “search” function that contains several keys to look up within the ETS tables, each returning a HashSet of IDs, and the final result set is the intersection of all these IDs in the hashsets (meaning IDs that are in all returned lists) + some business logic in between
- this is already a performance optimization on precomputed data in memory, since the database chokes on this calculations (we optimized as much as we could before), therefore “use SQL” is not an option here
- this search function is also a hot code path throughout our whole application and must be as fast as technically possible (few miliseconds, tops)
- it worked quite good so far but now that the amount of records in the tables (and the lenght of the result sets) go up over time, this “algorithm” is becoming noticeably slower
- we don’t want to introduce additional infrastructure components like search engines
so this is the situation right now. the precomputation of data and holding it in ETS tables already brought us a great performance boost, but it seems the “intersection” logic with custom rules is the new bottleneck now.
What we already use is Rustler for a different problem elsewhere in the project, and it basically “solved” another complicated computational problem. But here I think the actual problem is the “copying” of increasing amounts of data (few dozen megabytes) between BEAM processes, plus the computation afterwards.
Now my idea right now is to move the precomputation-code into a Rustler NIF, maybe even store it all in a single Map structure instead of multiple ETS tables, and on request use this map as a parameter + the search parameters to calculate the resulting list of IDs also in the NIF and return it.
The three questions I have right now (couldn’t find details so far):
- Can I pass (and receive!) data back and forth into the NIF via pointer/reference instead of copying it to eliminate any overhead here?
- Can I hold the precomputed data in a Genserver and make it accessible in a concurrent way similar how ETS can be configured for parallel read access? Without too much copying?
- Can I modify parts of the data or replace the whole data container from within Rust to speed up things?
AFAIK a GenServer can not handle true concurrency (it iterates on its single linear inbox), and Task creates new processes that copy memory into it, even ETS copies memory on querying but at least updates are “in-place”.
Thanks in advance!
