Over the last few days I had a bit of downtime. I have no monitoring set up for the services, and I didn’t have problems on the previous server. But having migrated to DigitalOcean, it turns out that its default configuration does not provide a swap partition, and that 512mb is too little for Apache2 and MySQL (at least with configurations I had on the previous server).
Even after a bit of reconfiguration to tell both Apache2 and MySQL to back off a bit, every few hours Linux would choose MySQL as the “less important” process to kill, due to lack of memory. And MySQL would not automatically restart (because it’s not set up to do so).
So I’ve done some deleting of old hosted files (freeing up a bit of disk space) and have set up a 1gb swapfile (note that the instructions are intended for 256mb servers, hence creating a 512mb swapfile).
In the process, I learned about sys.vm.swappiness, which is a kernel setting that apparently controls when the kernel will decide to swap things to the swap partition or swap file. By default set to 60, as the article instructs I’ve set it to 0 (which makes sense, as I want swap to be used as rarely as possible). IBM also suggests 0 in KVM-based virtual machines which is also what DigitalOcean uses.
This article from 2007 suggests you may even want to increase it in some cases (when you have a lot of inactive jobs waiting to be run), and force kernel to more aggressively find unused memory pages that it can swap to the disk. This AskUbuntu answer has a good explanation on what swappiness means: a setting of 60 means that at 40% usage kernel will attempt to find inactive pages to swap, and a setting of 10 means that at 90% usage kernel will attempt to find inactive pages to swap. Wikipedia also has an article, which sadly does little to explain the actual mechanisms.
A better solution would be to force MySQL and Apache2 to use appropriate amounts of RAM (and abide by some absolute limits, such as 196mb+196mb). But that’s something I’ll play with on another day… there are more important things to work on right now. 😉
–
via blog.vucica.net