That said, their experience probably reflects my own: using SQLite on a personal project for config or something, loving the simplicity, loving the "ditch all that complexity" hype, then trying it on a more demanding project that according to their mental model it should have excelled at (a central job tracking process), watching it completely faceplant under load (20TPS? Surely it just needs an index or something? Maybe a ramdisk? Err.....), panicking when they realize that perf and monitoring and diagnostics and optimized backends were some of those "bells and whistles that nobody needs," deciding "fine I'll do it myself," throwing far too much time and effort at co-debugging python and C, recompiling SQLite with debug symbols, sprinkling printfs, running profilers, reading the source, and tracking down the issue... only to find that it was a bad default which people had been unsuccessfully begging the maintainers to change for years, which was guaranteed to torpedo perf under a wide variety of common workloads, and which the maintainers had been resisting based entirely on inertia and the misguided idea that this setting was easily discoverable. Yikes.
That's how someone picks up rules of thumb like "SQLite is for preferences, not perf, and definitely not prod."
Now, this experience is a decade stale. SQLite bit me in 2012 and the particular bug was fixed in 2015 IIRC. The people begging to fix the bad default finally won. Time passed, SQLite got better, and now, based on these new stories of SQLite doing well under load, I think it's time to lift my "once bit, twice shy" sanctions regarding perf. I had them in place for a reason, though, and if you had YOLO'd when I had YOLO'd, you would have faceplanted when I faceplanted. I fully defend the use of sanctions like this in general. They are the only tool we have against overenthusiastic marketing.
To be fair, the SQLite site is pretty good, it's the fans that tend to take things too far.
Speaking of potentially obsolete sanctions, conferences have been on hold so I might be hanging on to a few more of them past their due date. Does anyone have fresh experiences with these?
------------------
AMD GPUs (2014): Waiting for GPGPU support from Adobe and Blender that was supposed to be operational (Adobe) or getting there (Blender) but never landed.
AMD GPUs (2016): I burned an entire semester trying to port CUDA code to OpenCL only to run into bug after bug after bug. Lockups, black screens (no framebuffers were supposed to be involved but bugs find a way), leaks, stalls which were difficult to attribute, unscruitable errors leading to abandoned threads with pleading users from years past with no reply. I finally cracked when after a long day of "I swear this should work I have no idea how the changes I make correspond to the bugs I see" I ran my ailing OpenCL code on a weaker NVidia card, and it Just Worked. Not only did it just work, it was 10x faster. Even though the card was like 2 generations older, and I was using OpenCL, which NVidia had every reason to sandbag, but they were still winning at. Oh, and the debugging tools actually worked, too. I was persuaded to sell my red cards, eat the ebay tax, eat the nvidia tax, and buy green from then on. Time has passed since 2016, though. ROCm looks good and AMD has money now, so maybe they can afford to pay people to fix bugs. I want to see someone running Blender and a modern ML framework or two before I retire the sanctions, though. Anyone have fresh intuition here?
Numba CUDA (2018): I spent two weeks working through a succession of problems that I eventually resolved in two days by just using CUDA C++. Numba CUDA only worked with year-stale CUDA, it copied float8s over a float4 array despite having access to triply redundant type specs, the mechanism to pass buffers between kernels always forced copy off GPU, and one more that I forget. I made patches for the first three and gave up on the last one before fixing. Has anyone used it recently? Is it in better shape these days?
I was hired by AMD to work on ROCm in 2020. I'm not exactly an unbiased source, but IMO it has improved significantly. There's a number of working frameworks and Blender was recently ported [1]. Just be sure to choose appropriate hardware.
ROCm still has lots of rough edges, but we're slowly filing them down. The documentation is a bit patchy too. Overall, though, it's been a steady progression in the right direction.
For OLAP applications (heavy on reads, pretty much no writes) I find a local SQLite has far superior performance to eith cloud based Postgres or SQL Server instances. YMMV.
That said, their experience probably reflects my own: using SQLite on a personal project for config or something, loving the simplicity, loving the "ditch all that complexity" hype, then trying it on a more demanding project that according to their mental model it should have excelled at (a central job tracking process), watching it completely faceplant under load (20TPS? Surely it just needs an index or something? Maybe a ramdisk? Err.....), panicking when they realize that perf and monitoring and diagnostics and optimized backends were some of those "bells and whistles that nobody needs," deciding "fine I'll do it myself," throwing far too much time and effort at co-debugging python and C, recompiling SQLite with debug symbols, sprinkling printfs, running profilers, reading the source, and tracking down the issue... only to find that it was a bad default which people had been unsuccessfully begging the maintainers to change for years, which was guaranteed to torpedo perf under a wide variety of common workloads, and which the maintainers had been resisting based entirely on inertia and the misguided idea that this setting was easily discoverable. Yikes.
That's how someone picks up rules of thumb like "SQLite is for preferences, not perf, and definitely not prod."
Now, this experience is a decade stale. SQLite bit me in 2012 and the particular bug was fixed in 2015 IIRC. The people begging to fix the bad default finally won. Time passed, SQLite got better, and now, based on these new stories of SQLite doing well under load, I think it's time to lift my "once bit, twice shy" sanctions regarding perf. I had them in place for a reason, though, and if you had YOLO'd when I had YOLO'd, you would have faceplanted when I faceplanted. I fully defend the use of sanctions like this in general. They are the only tool we have against overenthusiastic marketing.
To be fair, the SQLite site is pretty good, it's the fans that tend to take things too far.