I mean, I shouldn’t oversell this: it really is not that hard. In the very beginning I had a couple of determinism bugs caused by enumerating maps. After I learned about that I don’t think I ever had a determinism bug again.
Also, if you have a rare determinism bug it may not matter very much. The purpose of the simulator is debugging; deterministic execution does not have to be a correctness condition of the database.
At the end of the day, you actually have to care about these things. Good testing takes effort. Really, anything good takes effort.
Like, I said property-based tests are better than unit tests, and then everyone shows up to defend unit tests. And, look, I am not here to tell you what to do, but if you actually care about correctness and want to get to zero bugs this attitude is just NGMI.
Fundamentally, what set FDB apart is that from the beginning they set a culture of caring deeply about correctness. They went to war with bugs. Flow, the simulator, and the database were all downstream of that, not upstream.
Today I think most of that culture has been erased. They still have the simulator, but the team maintaining the database is several generations removed from those pioneers.
If you want to see who’s really carrying this torch today, it’s Tigerbeetle. In particular, their work on handling storage faults far surpasses FDB, and if FDB’s culture was still alive today they would be working on that instead of replacing their storage engine with RocksDB (lol).
I would like to carry a bit of that torch too.
