Modern operating systems will grant large amounts of RAM to web browsers in general to improve caching. That RAM is easily relinquished back to the OS if another app needs it.
Was the 12GB with another RAM-intensive app running? Or just Firefox by itself.
In addition to the MEM_RESET, MEM_RESET_UNDO mentioned, there's also OfferVirtualMemory[1], ReclaimVirtualMemory[2] to let the system potentially use the memory if it needs it.
Unlike MEM_RESET, this allows you to prioritize the memory you offer, as well as provides some extra safeguards it seems.
That seems to be something different; instead of getting the memory when some other application needs it, the OS receives the memory when the application (browser) doesn't need it any longer, so it can't be used for caching.
This actually looks exactly like the malloc/free use case.
How can't it be used for caching? Say you decode an image, then offer the memory backed by it to the OS. If you need to access the image data again you try to reclaim the memory, if it fails it's a cache miss.
What I was seeing was the same in both cases. Firefox would be using the same amount of RAM either running by itself or alongside Ableton Live (and the performance of Ableton Live would suffer).
is how it should work. If the first step doesnt happen, its not clear if the second one should execute by itself. Id be much more concerned about background CPU usage. Eats battery, and actually slows the computer down.
That all said, its pretty clear that browsers are leaking RAM sometimes. I remember bed bath and beyond using 9 or 12GB of ram the other month.
That's not how it works. Browser might monitor free system memory actively and release memory if the system runs low, but at least Firefox doesn't do that.
There is no mechanism (in Windows, Linux) that allows the OS to notify applications of memory pressure or that allows the application to tell the OS that some allocations can be thrown out if need be. (Windows has a mechanism - MEM_RESET - that only works with paged out allocations)
Edit: Windows 10 (8?) actually expands on MEM_RESET with Offer/ReclaimVirtualMemory, but I kinda doubt browsers use this, because these have similar semantics to MEM_RESET (except they seem to work without paging, which is good). For an application that wants to cache not-well-defined amounts of data itself these might be very useful though, because you can make the data structure simple enough that it works with these.
https://dblohm7.ca/blog/2015/07/28/interesting-win32-apis/ notes that Windows has the required APIs, and that Firefox already supports handling memory pressure notifications on non-Windows platforms. Not sure how up-to-date this information is.
> ...that allows the OS to notify applications of memory pressure or that allows the application to tell the OS that some allocations can be thrown out if need be.
There are mechanisms for both of these in up-to-date Linux - pressure stall information and volatile ranges, respectively.
MADV_FREE didn't work properly until fairly recently, though, because it behaved like Windows' MEM_RESET. AIUI MADV_FREE is not intended for ranges that you actually want to work with (i.e. you want to keep the data, not just the virtual addresses), but rather as an optimization for allocators.
Firefox does have a mechanism for reducing memory pressure, there's a button for it in about:memory. I can't guarantee that this flow actually happens upon low memory states on Windows etc but it wouldn't be hard to at least hack it together. Monitoring available memory (say, on a timer) is not a difficult thing to do on Windows - it would just be difficult to make it completely robust.
It's important to specify your test scenario here and the behavior you expect, as it's a known issue in general that Linux behaves poorly under OOM conditions. It's not necessarily Firefox specific.
If all of my memory is "in use" but half of it is cached file pages, should Firefox compact its heap and evict cached images? What if all of my memory is spoken for but a bunch of it isn't actually committed, due to overcommit?
Those aren't interesting questions tbh. Obviously most of the time the page cache grows to use all memory, that's the whole point of having it (using all memory that is not committed to applications as a disk cache). Overcommit obviously doesn't create memory pressure because it only exhausts address space, not memory (physical memory + swap). The relevant question is how the system behaves if it needs to allocate pages but can't find any clean (just throw'em out) or unused pages. Linux, and to a lesser degree, Windows start to thrash heavily in these scenarios. I've never seen e.g. Firefox unload a tab when this happens, regardless of OS.
Was the 12GB with another RAM-intensive app running? Or just Firefox by itself.