Thanks for taking an interest!
It certainly could be done that way, but there are some drawbacks. The biggest is API ergonomics - Take an operation like get(key). Unlike a Map, the get operations here have side-effects. (The fastest storage servers are located and cached, etc.) If done in a purely functional style, the return from get would need to look like {result, tx}. Pipelining would be extremely clunky for operations like put, breaking the idea that Bedrock is like some really big, durable, transactional map.
There’s an argument for using the process dictionary instead of a GenServer, for similar effect, but spinning up a new process is pretty cheap with the BEAM. I have some ideas that require managing timers and timeouts that pretty much require the use of another process… so, we’ll see what the future holds.
Here’s how it works. The first read gets a read version from the sequencer. All of the subsequent reads for this transaction (and all nested transactions) will use this same version. This one of the ways that Bedrock can fulfill it’s guarantee of repeatable reads and snapshot isolation: All reads for a key at a version will return the same value.
When a transaction is committed, if any reads were performed, the read version is used along with the keys that were read to check to see if any transactions modified those keys in the intervening time – this is part of what the Resolver does. If the keys that were read have been modified, the transaction is aborted and restarted (and it will then pull a higher read_version and so will see the newly modified data).
The sequencer is a cluster wide singleton, but it’s job is extremely simple. FoundationDB, the pattern on which Bedrock is based, uses the notion of “GRV” proxies, read-version proxies that will spread the load. Bedrock may introduce a similar concept that could make use of :ets tables as you describe on each node. For now, calling the Sequencer directly is simple and effective.
There are so many opportunities for optimization in front of us. Currently, the focus is on getting all of the basic operations right, increasing test coverage, getting the documentation right, and working towards getting the system into a place where people might actually dare to use it for real work.
If you see something you want to improve, though, we’re open to a PR! 
The way transactions were being assembled in 0.1 left holes in the “reads” and “writes” that could slip through the Resolver if we tried to add range-reads into the mix. This only worked because 0.1 just didn’t do range operations. Take a look at develop, where 0.2 is coming together, though! I’ve recently (yesterday) overhauled the machinery for building transactions (and the transaction format as well) with an eye toward making range operations like range-reads and clears easier to implement, along with other fun possibilities.
Interestingly, in this system, the read-your-writes all happens entirely within the transaction builder. If in a transaction I read key “a”, and get “apple”… and within that transaction write “aardvark” to “a”, then any subsequent read (or range read!) should return the “aardvark” value for “a”. The rest of the system doesn’t see any of this. When a transaction is committed, passes resolution and is pushed to the logs… any subsequent read using that commit version would see the new value for “a”. If that transaction is rolled back, nothing needs to be done or communicated – we just throw the process away.
This is kind of an interesting thing that FDB does, and I shamelessly copied it!
So they have their famous 5s window for transactions, right? This brings us to a question: How do all of the machines involved know when that window advances?
They all have different clocks, and there could be all kinds of skew… so you can’t use local wall-clock time – synchronizing time is a hassle or requires super-expensive equipment (like Spanner). So clock synchronization is out. How do all the servers in the cluster know when that 5s window has advanced?
Here’s what they (and we) do (and I think it is super cool):
- The sequencer issues versions in microsecond increments. This is monotonic time, and so will always be increasing. You can think of it as uptime for the cluster, because versions only advance when the cluster is in a running state. So, all of these servers don’t need to track their own time, they can just look at the most recent version to come along, and subtract ~5 million, and there’s your window. There’s a catch, though: This only works if transactions are happening. If the system is quiescent, nothing happens, nothing advances, the window doesn’t move.
- Commit proxies will issue an empty transaction if no transactions have been processed within the last 1000ms. This advances the version, and slides the window ever-forward. As a bonus, it serves to “tire kick” the transaction system to ensure that the sequencer/resolvers/logs/ etc. are all reachable and functioning. Any error triggers a recovery.
You get this lovely three-for-one out of this: ensuring the parts are working, advancing the window, synchronization.
indeed. 
Yep. Broken all over the place. It’s a big project, and it’s just me, and it’s hard to keep all of the references up-to-date as things are in flux. As the design settles down, and the number of people contributing (hopefully) grows, I imagine that things like this will be sorted out.
Yep. The storage server that’s in there now, “Basalt,” is a place-holder, an example. I used :dets in it because it’s simple and it’s built into the BEAM. It could easily be replaced with some other k/v store (like rocksdb), but this works to let people try out the system and do so with minimal fuss, even in a livebook.
One thing interesting thing to point out about this model, though: There doesn’t have to be just one kind of storage server in operation at a time. I could easily see using a mix of them for different properties. Maybe a memory-only one that doesn’t guarantee durability, but offers extremely fast reads? Or another that’s good at absorbing writes quickly (built around an LSM, perhaps), or maybe another that packs transactions and sends them off to AWS/GCS for disaster recovery and can answer reads, but maybe slowly. There are a lot of interesting possibilities here!
I considered this, and might reconsider it again at some point. Ultimately, I found that these patching tools (of which there are a few) don’t seem to play well with asynchronous tests, as different tests might need to patch things in different ways, and yet others may want to use the actual implementation.
I haven’t settled a on a good answer for this… and as you note, there are some small drawbacks to this approach. At this time, though, I think the focus needs to be on correctness. There will come a time, though, that those extra few cycles matter and I’m sure the code will change.
I’ll take that into consideration.
I tend to use a lot of pipelining - that’s the whole story. The macros ensure that the functions are defp, and the compiler will inline them. In the compiled code, there’s no difference between this and hard-coding the tuples, but the functions play nicely with pipelining and i don’t need to resort to then(&{:noreply, &1}).
Yay! I’m really quite happy that this is generating some interest. FDB is an awesome system and the patterns and ideas really should be more widely used. It’s just good tech. Keep the questions and feedback coming!
