For many workloads it’s on par. Few are slower. Pathological case, working set doesn’t fit in memory allotted to pg but fits in file system page cache. There is not page cache in neon

I find this one a little hard to believe.

Regardless, what are some of the trade offs this implementation makes? (aka the cons)

The cons are mostly:

- Buffer cache evictions of freshly dirtied pages are bad. This is because we request pages with a hint on the latest change was evicted, and with newer and newer changes being evicted from buffers you might start to get limited by the write-through latency to Pageserver instead of only Safekeeper.

- Commit latency can be not great due to cross-AZ communication -- with 3 safekeepers in as many AZs, the second slowest response is the limiting factor

- Write amplification in the whole system is quite high. Plain PG does ~ 2x (WAL + Page), while we have many times that. Admittedly, these writes are spread around several systems, where any one system only really needs to write 1x WAL volume for the data that it is responsible for, but Pageserver is currently configured for something like 4x write amplification due to 4 stages of LSM-tree compaction.

Thanks.

So in layman terms, cache purging is slow (unreliable?) and writing is slower. (which is to be expected)

Maybe another way of begging the question is, what apps would be suitable to build with Neon and what would not?

Suitable:

Anything with a write-based working set that fits in the buffers of the primary instance;

Databases that are unused most of the time;

Databases with a lot of read replicas (potentially with Neon only providing the read replicas, not the write node/hot standby);

Apps that want to run analytics on the data, but don't want to transfer O(datasize) data to a different DB every time, and also don't want to deal with the problems of long-running transactions.

Not very suitable:

OLTP with writeset that doesn't fit in the caches;

Databases that need <1ms commit latency.