I've never used CockroachDB, but FoundationDB has been an absolute dream to work with. There are a few downsides to using it:
- There is (currently) no hosted solution you can use, so you have to run it yourself.
- There is no great "layer" that provides a nice document/SQL layer so you kind of have to work with the raw sorted KV abstraction.
- You have to structure your workload to avoid a high rate of conflicts on individual keys.
- It won't scale to extremely high write volume workloads like Cassandra/Scylla will.
That said, in exchange you get:
- Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property. Jepsen refuses to test FDB because FDB's simulation testing is much more rigorous than anything Jepsen does, but AFAIK the most recent releases of Cockroach/Mongo/Cassandra/Fauna/etc have never fully passed a Jepsen test.
- Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput.
- A strongly consistent store you can actually use in anger and that you don't have to baby. We've never managed to actually break a FoundationDB cluster, the worst case scenario we've ever seen is that we slam it with too many writes/reads and it starts to apply backpressure by increasing latency.
- Write/Read throughput scales ~linearly as nodes are added.
- Built in backup/restore from blob storage.
- Strongly consistent disaster recovery. This one is amazing and its hard to go back to other databases that don't support it, but basically you can run 2 FDB clusters in DR mode where one is the leader (accepting writes + reads) and the other is passively replicating from it. You can then fail over from the primary FDB to the secondary *consistently* in such a way that the two clusters coordinate to lock the primary and then unlock the secondary so that your application is guaranteed to not see any inconsistency or data loss with no participation from the application required (except teaching it how to talk to 2 different FDB clusters at once).
Basically its lacking some niceties for building traditional applications on top of it, but it is an amazing primitive for building rock solid distributed systems and will almost never let you down once you've developed some operational experience with it. Its the closest thing to "magic" that I've ever worked with in the distributed storage space.
to the folks asking why you'd use FDB, i'd highlight the main points as:
> Interactive linearizable transactions (highest consistency level possible) over arbitrary key ranges (none of this single partition transactions nonsense) with a guarantee that FDB will basically never violate this property.
the transaction guarantees on FDB are just the best there is in any product you can get for free. not to fault the rest, FDB just went past what ~98% of folks will ever need. but if you're in that last 2%, god speed trying to get anything else to offer you the same semantics.
> A strongly consistent store you can actually use in anger and that you don't have to baby.
i've seen some folks trying to run a cluster without proper monitoring, who were nearly driving the poor thing into the ground, with hosts dying, and having so much pressure put on them they failed out, but it never lost anything. i don't think it ever even became unavailable.
> Automatic range partitioning. So unlike Cassandra you don't have to pick a static partitioning key yourself, FDB just automatically detects and split ranges for you based on storage volume and write throughput
eh, in defense of everything else, FDB is a key value store, and the keys are byte strings, so the situation is a bit easier for them.
Sure, I’m not really knocking Cassandra. I’m just pointing out that Cassandras approach favors raw write throughout over everything else which is great for some use cases, but FDBs approach is much more forgiving and a better fit for many use cases, albeit at the cost of reduced throughout.
