Imagine you’re trying to build something like S3. That is, a distributed filesystem or blob store. The purpose of such a system is to map a small identifier (a file name) to a large blob of contiguous bytes.
The way this is done, generally, is to have a system that associates large blobs (or chunks thereof) to opaque ids, and then have a system that associates file names to those ids. This allows you to quickly perform logical updates against the system without moving around huge blobs of data. It’s an indirection mechanism.
Pretty much every object store you can think of follows this pattern, with a metadata service and a blob storage service. If you want to learn more I’d recommend going back to the classic Google Filesystem (GFS) paper, which is a good early example. Another really good one is the classic Haystack paper from FB, which explains pretty much exactly how to store blobs on disk and why the metadata split is important. They talk a lot about storing metadata in-memory but keep in mind this was before SSDs were cheap.
Hobbes, like FDB, is architecturally amenable to being the metadata store. It’s designed to offer fast, small updates to a large dataset with extremely strong consistency guarantees. It is not designed to offer large updates; even a few megabytes in one transaction would be excessive. This is the correct side of the tradeoff to take for OLTP use-cases where you insert or update a few distant rows/indexes at a time.
This “blob tradeoff” goes right down to the hardware. Hobbes is intended for use on SSDs which can efficiently perform random I/O. But for blob storage you would generally want to use spinning disks because they’re cheaper and the I/O pattern is more sequential. This dichotomy is even more dramatic with SMR drives, which physically cannot perform random writes.
So a design kinda naturally falls out of this: use Hobbes to store metadata (filename mappings) and design a simple “blob storage” server that does nothing but write bytes to spinning disks. The blob storage system can dodge basically all hard distributed systems problems (consistency and so on) because Hobbes provides them by default.
I intend to write something like this soon, once Hobbes is ready.
