Journey – simpler scalability, reliability, and postgres-based persistence, with self-computing graphs

Ah, this is a great question, thank you for asking, and thank you for bearing with me as I am figuring out how to describe this!

In my experience, Journey solves the accidental complexity of a lot of web applications. It handles a lot of the plumbing that I keep rebuilding, that’s not part of my app’s actual business logic. Things like:

Persistence and Recovery

  • figuring out how to persist and recover user sessions, across reboots, outages, redeployments, page reloads, etc.,
  • figuring out how to let my users resume their flow a month later.

Orchestration and Reliability

  • figuring out when to compute what,
  • figuring out how to do reliable one-time and recurring scheduling that survives redeployments and outages,
  • figuring out how to do reliable retries when my application redeploys while anthropic has an outage (for example;),
  • figuring out how to do all of these things as my application runs on multiple replicas, and scales up and down.

Observability and Insights

  • figuring out what’s happening with a specific user (“Did Mario get approved?”),
  • figuring out how to get business insights into my application (“what percentage of users got pre-approved, but didn’t claim the offer?”).

Code Simplicity

  • figuring out how to structure my code so it (and my brain) does not turn into mashed potatoes over time, :wink:
  • figuring out how to structure my code so its essence is easy to understand for both LLMs and humans (and fits into our tiny context windows;),
  • figuring out how to manage the complexity of it all – making sure all of the pieces actually work together as i think they do.

I got tired of solving the same problems in every project, so I built Journey – it lets me describe my application as a graph, and then runs executions of the graph, while handling all of those things for me.


Let me make it a bit more specific. I’ll show a basic example to illustrate basic concepts, and then extrapolate to more practical applications.

Let’s say my amazing “hello and goodbye” app says “hello” and “goodbye” to everyone who visits my website.

The graph for this website has three nodes. The value of :name is provided by the visitor. The values of :hello and :goodbye are computed by the functions attached to the nodes.

~/src/journey $ iex -S mix
Erlang/OTP 27 [erts-15.1.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Interactive Elixir (1.18.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> import Journey.Node
iex(2)> graph =
...(2)>   Journey.new_graph("hello and goodbye", "v1.0.0", [
...(2)>     # :name input node.
...(2)>     input(:name),
...(2)>     # :goodbye computation node, waits for :name
...(2)>     compute(:hello, [:name], fn %{name: name} -> {:ok, "Hello, #{name}!"} end),
...(2)>     # ':goodbye' computation node, waits for :hello and :name
...(2)>     compute(:goodbye, [:hello, :name], fn %{name: name, hello: _} -> {:ok, "Goodbye, #{name}!"} end)
...(2)>   ])

This graph now serves as the blueprint for my application.

Whenever a visitor enters their name, the application starts a new execution – a running instance – of the graph.

iex(3)> execution = Journey.start_execution(graph)

and saves the visitor’s name as the value of the :name node:

iex(4)> execution = Journey.set_value(execution, :name, "Mario")

(Journey handles persisting the execution’s values and invoking its functions.)

Now that :name has a value (“Mario”), Journey will call the function attached to :hello node, and its value becomes available (and my UI can now render it to my visitor;):

iex(5)> Journey.get_value(execution, :hello, wait_new: true)
{:ok, "Hello, Mario!"} 

Now that :hello has a value, following the definition of the graph, Journey will call the function attached to :goodbye, and :goodbye’s value will also becomes available:

iex(6)> Journey.get_value(execution, :goodbye, wait_new: true)
{:ok, "Goodbye, Mario!"}

If at any point things got interrupted (our infrastructure crashed, user reloaded their page, user left and came back a year later), as long as we took a note of the execution’s id (e.g. by putting it in the url of the browser),

iex(7)> execution.id
"EXEC3XJVY5ZRX8BHYAYH0E3Y"

we will simply reload it when things are back up

iex(8)> execution = Journey.load("EXEC3XJVY5ZRX8BHYAYH0E3Y")

and continue where we left off

iex(9)> Journey.values(execution)
%{
  name: "Mario",
  last_updated_at: 1756270543,
  execution_id: "EXEC3XJVY5ZRX8BHYAYH0E3Y",
  hello: "Hello, Mario!",
  goodbye: "Goodbye, Mario!"
}

Hopefully, this example made my earlier statement more specific:

it lets me describe my application as a graph, and then runs executions of the graph


To start extrapolating this to a more complex, real-life application, consider this graph, illustrating an example credit card approval process.

Let me call out a few things, around rich dependencies, some useful types of nodes (schedules, mutations), retries, and tools for introspection.

1. rich dependencies

in the “hello and goodbye” application, computations are unblocked as long as the upstream values have been provided. these dependencies could be more expressive

        compute(:congratulate, unblocked_when({:preapproval_decision, &approved?/1}), &Compute.send_congrats/1),
 

and

        compute(
          :send_preapproval_reminder,
          unblocked_when({
            :and,
            [
              {:schedule_request_credit_card_reminder, &provided?/1},
              {:not, {:credit_card_requested, &true?/1}}
            ]
          }),
          &Compute.send_preapproval_reminder/1
        ),

2. mutations

In this example, :ssn_redacted mutate node mutates the value of another node – as soon as :credit_score has been provided, the value of ssn is redacted, to protect the user’s PII.

        mutate(:ssn_redacted, [:credit_score], fn _ -> {:ok, "<redacted>"} end, mutates: :ssn),

(re: “ssn” – apologies for the US-centric example, our Social Security Numbers… long story, let’s just say they are a problem.)

3. schedules (one-time)

schedule_once nodes unblock downstream nodes at the computed time (across reboots, redeployments, restarts, etc ).

(For example, :send_preapproval_reminder can be triggered a week after :congratulate. message was sent, if the customer did not request the card.

        schedule_once(
          :schedule_request_credit_card_reminder,
          [:congratulate],
          &Compute.choose_the_time_to_send_reminder/1
        ),
...
        compute(
          :send_preapproval_reminder,
          unblocked_when({
            :and,
            [
              {:schedule_request_credit_card_reminder, &provided?/1},
              {:not, {:credit_card_requested, &true?/1}}
            ]
          }),
          &Compute.send_preapproval_reminder/1
        ),

4. schedules (recurring)

schedule_recurring nodes let you schedule things to happen recurrently (across reboots, redeployments, restarts, etc )

5. Tools for introspection (individual executions and system-wide)

“What happened with Mario’s application?”

Journey.Tools.summarize_as_text(execution) gives you an instant answer – what data has been provided, what data has been computed, what computations are blocked and why. To see it in action, go to https://demo.gojourney.dev/ and search for “Execution summary” in the “behind the scenes” view.

"Where are my users dropping off?”

You can look through executions to understand your customer’s behavior – you get analytics insights into your business process (“What percentage of your users entered their name but never proceeded to SSN?” “What percentage of users got the :send_preapproval_reminder reminder?” etc.)?

Journey.Insights.FlowAnalytics.to_text provides some initial insights, richer flow analytics is on its way.

To see this in action, go to https://demo.gojourney.dev/ and look for “Analytics (as text)” under “behind the scenes”,

6. Code Structure / Readability

Journey gives the code a neat structure. The workflow is defined in the graph. Most of the business logic can be defined in the “compute” functions attached to the graph’s compute nodes.

LLMs seem to be good at “here is journey documentation. here is my journey graph. explain what this application does. here is the output of summarize(). why did mario not get a reminder?”

7. Computation Retries

The functions attached to computation nodes are executed with retries. So if your :extract_attributes_from_candidates_resume computation failed because anthropic had an outage, it will retry, with a customizable retry policy – even if your own application was redeployed or otherwse restarted.

If the outage lasted all day, and the computation’s retry policy got exhausted, Journey.Tools.retry_computation/2 lets you re-kick the computation the next day.

Here is a tiny example of the code that re-tries a computation after normal retries were exhausted, from https://demo.gojourney.dev/

    updated_execution =
      Journey.Tools.retry_computation(
        socket.assigns.execution_id,
        String.to_existing_atom(node_name)
      )

This retry logic persists across reboots / restarts / etc. Extrapolating OTP terminology, Journey uses DB-based, persisted supervision for running node computations.

A word of warning: while I found this functionality to be extremely useful in my production applications, this current incarnation of this package is relatively new. The upside is that I am actively working on refining its functionality, scalability, documentation, and developer experience, and if Journey is a good fit for your application, I am very interested in making sure it works well for you.

4 Likes