I have been trying to fix a problem on a server for a while, trying various zfs tunable with varying success.
2 notable problems are, is weak on myisam mysql performance, mostly fixed by disabling zfs prefetch, and it collapses during system backups which are run nightly. These backups involve tarballing entire home directories and databases. I dont want this thread to be a debate on whats more efficient for backups thanks.
Disabling prefetch damages other file access especially the backups, so tuning for one thing damages another, is a big oversight I think to make prefetch a system only variable.
The server is 8.1 with zfsv15, amd64 8 gig of ram (another 4 gig on the way).
Last night I picked up on whats going wrong since every morning there is excess swap been utilised. I started the backups manually, and have graphed the data for memory usage. When the backups started zfs ARC was shrinking and shrinking slowly and 'Inact' usage was growing, cacti reports this as cache, this carried on until the server choked. It choked at around 2.8gig Inact usage (grown from just 700 meg) and zfs at that point had dropped from 3 gig to 1.5gig, at that point was no swap usage. Then there is a sudden spike in processes and then too much strain on ram (cache not releasing fast enough) and so loads of swap used and the server choked going offline for a while until it sorted its ram usage out. When cacti was able to connect again, the cache had been flushed and ram usage was very low (4 gig free unused) so obviously at that point the Inact had been dumped to disk and the server slowly went back to normal.
It would seem from observation the following is happening.
1 - Memory management is not that well handled auto as some say as here, http://forums.freebsd.org/showthread.php?t=9980. I wonder if there is a way to cap Inact usage like zfs ARC can be capped.
2 - ZFS has 2 caches? ARC for read and normal Inact for writes?
3 - Suggestions on how to resolve this, I will provide what info is asked for. As I said please dont remind me tarballing is inefficient, I know this but is how the installed software control panel manages backups.
The same system on UFS did lag during backups but I am talking about a few seconds lag not causing the system to collapse on itself. ZFS is in a 2 drive mirror setup, I was going to add 2 new drives so dual mirror to the root tank but seems it isnt supported on root zfs, likewise a log caching device isnt supported either.
2 notable problems are, is weak on myisam mysql performance, mostly fixed by disabling zfs prefetch, and it collapses during system backups which are run nightly. These backups involve tarballing entire home directories and databases. I dont want this thread to be a debate on whats more efficient for backups thanks.
Disabling prefetch damages other file access especially the backups, so tuning for one thing damages another, is a big oversight I think to make prefetch a system only variable.
The server is 8.1 with zfsv15, amd64 8 gig of ram (another 4 gig on the way).
Last night I picked up on whats going wrong since every morning there is excess swap been utilised. I started the backups manually, and have graphed the data for memory usage. When the backups started zfs ARC was shrinking and shrinking slowly and 'Inact' usage was growing, cacti reports this as cache, this carried on until the server choked. It choked at around 2.8gig Inact usage (grown from just 700 meg) and zfs at that point had dropped from 3 gig to 1.5gig, at that point was no swap usage. Then there is a sudden spike in processes and then too much strain on ram (cache not releasing fast enough) and so loads of swap used and the server choked going offline for a while until it sorted its ram usage out. When cacti was able to connect again, the cache had been flushed and ram usage was very low (4 gig free unused) so obviously at that point the Inact had been dumped to disk and the server slowly went back to normal.
It would seem from observation the following is happening.
1 - Memory management is not that well handled auto as some say as here, http://forums.freebsd.org/showthread.php?t=9980. I wonder if there is a way to cap Inact usage like zfs ARC can be capped.
2 - ZFS has 2 caches? ARC for read and normal Inact for writes?
3 - Suggestions on how to resolve this, I will provide what info is asked for. As I said please dont remind me tarballing is inefficient, I know this but is how the installed software control panel manages backups.
The same system on UFS did lag during backups but I am talking about a few seconds lag not causing the system to collapse on itself. ZFS is in a 2 drive mirror setup, I was going to add 2 new drives so dual mirror to the root tank but seems it isnt supported on root zfs, likewise a log caching device isnt supported either.