tl;dr - some possibly useful background information
It occurs to me that some readers may not be familiar with Labeled Property Graphs in general or Neo4j in particular. More to the point, it may not be obvious why these could be a good fit for storing and accessing system information. So, here’s a (somewhat high level and opinionated) rundown…
A labeled-property graph model is represented by a set of nodes, relationships, properties, and labels. Both nodes of data and their relationships are named and can store properties represented by key–value pairs. Nodes can be labelled to be grouped. The edges representing the relationships have two qualities: they always have a start node and an end node, and are directed; making the graph a directed graph. Relationships can also have properties. This is useful in providing additional metadata and semantics to relationships of the nodes. Direct storage of relationships allows a constant-time traversal.
– Graph_database, in Wikipedia
Careful Reader will notice that this overloads the term node. That is, a Neo4j node can describe an Erlang node, but it could just as easily describe a function, module, process, etc. To reduce confusion, I’ll use the term “entity” below for the Neo4j meaning. However, note that this is not the only linguistic clash between Neo4j and Erlang: other issues include name syntax, etc. So, YMMV…
Stored Information
We’d like to store several kinds of information:
- generic entity definitions (e.g., what is a function?)
- possible relationships (e.g., modules can define functions)
- specific entities (e.g., module
Foo, functionFoo.bar/1) - specific relationships (e.g.,
Foo.bar/2callsFoo.bar/1)
Some of this information is generic; other information is specific to a given code base and/or set of running processes. So, we should gather each type of information when it either becomes available or changes.
Access, Data Structures, and Performance
The Cypher query language isn’t the only way to access Neo4j, but it’s definitely the Golden Path. Cypher syntax looks a bit like a cross between SQL and DOT (graph description) notation, with a bit of pattern matching thrown in. So, a typical query might look something like this:
MATCH (nicole:Actor {name: 'Nicole Kidman'})-[:ACTED_IN]->(movie:Movie)
WHERE movie.year < $yearParameter
RETURN movie
Neo4j has indexes that make it reasonably fast at finding specified entities and relationships, but the real speedup comes when we start traversing chains of relationships. Neo4j keeps connectivity information in memory, so following a relationship only requires dereferencing a pointer. So, it’s real fast at finding related entities, even when multiple relationships are involved.
This means that answering a multi-step “join” like “list all processes running on node name@host that have recently called function Foo.bar/1 via Foo.bar/2” can be expected to happen quite quickly. So, asking a Livebook to generate and display a Mermaid diagram of this sort of information isn’t a crazy idea.
A Possible Use Case
Serendipity for the win… I just stumbled on a possible use case, described by Peer Stritzinger at Code BEAM V EU 21: Jumping gen_servers! A new way of building (…) applications.
In this talk, Peer describes a robotic assembly line which uses a mesh of Erlang nodes to control all of the stages and types of actions involved. In order to migrate processes to the most relevant nodes, he needs a source of information on the entire system.
Because he doesn’t want a single point of failure, a single Neo4j database would not suffice. However, multiple database clones might work just fine. And, even if he wants to keep critical information “in the mesh”, he could use the Neo4j database(s) as a convenient data source.
HTH, Rich
