@Savagedlight: Nice work. Thanks for your data. I see the exact same patterns as you do.
@Matty: I'm not sure about your vnodes theory. Whether my performance is good, or when my performance is bad, my vnodes seem to be staying around 12,000.
Memory:
I've been watching my memory usage and I have no idea what is consuming memory as 'Active'.
Last night I had around 6500MB 'Active' again, 1500MB Wired, no inact, ~30MB buf, no free, and ~100MB swap used. My performance copying ZFS->ZFS was again slow (<1MB/s). I tried killing rTorrent and no significant amount of memory was reclaimed - maybe 100MB. `ps aux` showed no processes using any significant amount of memory, and I was definitely nowhere near 6500MB usage.
I tried running a perl oneliner to hog a bunch of memory (perl -e '$x="x"x3000000000'), and almost all of the Active memory was IMMEDIATELY marked as Free, and my performance was excellent again.
I'm not sure what in userland could be causing the issue. The only things I've installed are rTorrent, lighttpd, samba, smartmontools, vim, bash, Python, Perl, and SABNZBd. There is nothing that *should* be consuming any serious amount of memory.
Drives:
One thing that came up on FreeBSD-stable is that
WD Green drives have a serious issue with their power management under FreeBSD/Linux. The drive heads park after 8 seconds of inactivity, but BSD/Linux sync to disk less frequently than this, so the drives are continually parking and unparking.
The drives are rated for ~300,000 load cycles; after 2000 power on hours my drives are already over 90,000 load cycles. I used a tool released by WD to change the idle timeout from 8 seconds to 5 minutes (the maximum) and haven't seen my load cycle count increase yet.
Use smartctl to check the Load_Cycle_Count attribute on your drives, and if it is unusually high, get the tool 'wdidle3' and boot to a DOS environment to use it to configure your drive timeouts.
Details:
http://www.silentpcreview.com/Terabyte_Drive_Fix
wdidle3:
http://home.arcor.de/ghostadmin/wdidle3_1_00.zip
Doing this does not seem to have helped with my performance issues, but it ought to help drive longevity.