It always amazes me that after all these years, Linux still hasn't fixed this.
In my experience, any program that overloads I/O will make the system grind to a halt on Linux. Any notion of graceful degradation is gone and your system just thrashes for a while.
My theory about this has always been that any I/O related to page faults is starved, which means that every process spends its time slice just trying to swap in its program pages (and evicting other programs from the cache, ensuring that the thrashing will continue).
I've never gotten hard data to prove this, and part of me laments that SSDs are "fast enough" that this may never actually get fixed.
Can anyone who knows more about this comment? It seems like a good rule inside Linux would be never to evict pages that are mapped executable if you can help it.
In its heyday, Solaris was outstanding in terms of being responsive while simultaneously doing large amounts of I/O. (Or at least that's my perhaps clouded recollection, I haven't used Solaris in over 5 years).
> I seriously doubt that any usermode program could overwhelm the OS scheduler like that. What are your use case parameters?
The OS IO scheduler is known to be shitty. That's not an exegeration. You can start swapping because the scheduler does not free unused caches fast enough.
Just after the OS boots, Dropbox needs to index 120 GB of files. Any other program that wants to access the disk takes forever. For Dropbox to finish, for my mail and IDEs to open takes about 10 minutes. Any other program that needs the disk is uselessly slow.
Interesting. How many files do you have? I recorded a trace of dropbox executing on my windows machine (mostly flat folder hierarchy, ~500MiB , ~1000 files) and the file I/O for querying all my data took 71542.070μs (0.07s). I believe dropbox also does some extra things (reading the NTFS journal, its own file cache-journal, updating hashes, etc ) and so the total File I/O cost was around 2944815.431μs (2.9s). Note that the I/O happened sporadically, and the wall clock time is higher as expected (it didn't block the scheduler from scheduling other processes).
I assume since my data was synced and didn't need to be indexed all over again - I got some savings there. Maybe your dropbox configuration data is corrupted and thats why it needs to index it all again.
In my experience, any program that overloads I/O will make the system grind to a halt on Linux. Any notion of graceful degradation is gone and your system just thrashes for a while.
My theory about this has always been that any I/O related to page faults is starved, which means that every process spends its time slice just trying to swap in its program pages (and evicting other programs from the cache, ensuring that the thrashing will continue).
I've never gotten hard data to prove this, and part of me laments that SSDs are "fast enough" that this may never actually get fixed.
Can anyone who knows more about this comment? It seems like a good rule inside Linux would be never to evict pages that are mapped executable if you can help it.
Has anyone experimented with ionice or iotop? http://www.electricmonk.nl/log/2012/07/30/setting-io-priorit...