A while back I was asking about how to look at server load issues. I wound up using collectd, which was pretty useful.
I identified a handful of disk I/O spikes - unfortunately it’s hard to match these up against the subjective delays/lag problem. However, I looked into ways to improve I/O handling, and found information about caching and pdflush.
I’m currently experimenting with upping /proc/sys/vm/swappiness (to 90), to encourage more swapping to disk - on the grounds that I do have an active server, and I’d rather use the disk cache more. I think. I’m a little unsure of my logic here, but so far this seems to have improved the situation a bit, so maybe I’m on the right lines.
Have you done anything with hdparm and blockdev?
(is this still on your ldap server? You can reduce IO a lot by reducing logging or messing with syslog settings, and also looking at your DB_CONFIG)
Swappiness is rather misnamed. Increasing it does not promote more swapping, but rather makes the kernel more likely to swap anonymous pages than inactive pages. In order to swap out inactive pages (memory pages with no active references), the kernel has no option but to walk the entire memory space which is quite expensive with large memory sizes. Increasing swappiness tells the kernel to swap out anonymous pages (memory pages with references that aren't linked to files; e.g. process stacks, buffers, etc.) which is a much cheaper operation because the kernel can look at the page table to determine where these pages are in memory.
The kernel will prefer to swap anonymous pages when:
% of memory mapped in page tables + swappiness >= 100
Interresting topic, any new improvement since then?