Hey everyone,
I managed to get one extra SSD to play around with as cache/log device on my home server (3x2 mirror vdevs), and I tried a bunch of different configurations. My understanding of how this all works is pretty much limited to what I know from forums. So I ran a couple of tests and have a few questions, maybe you could help.
All my disks are encrypted, so at first I assumed that I would need to encrypt both ZIL and L2ARC to keep it fully encrypted again. So, for ZIL, I figured, that makes sense.
But what about L2ARC? From my understanding of how this works, L2ARC must be pointed from main memory. So, after a reboot, I would assume that contents of L2ARC can't be retrieved. How would it be possible to retrieve this data, and so, is encrypting this L2ARC necessary?
As it turns out, however, bonnie++ tests showed pretty much same and consistent results whether I used log/cache or not. For example this is with both log/cache on same SSD:
This is just with just L2ARC (unencrypted):
In fact, one thing I noticed is that results seem to be a bit better without (!) L2ARC on SSD (about 10-20MBps better reads) on my mirror vdevs. Why is this? And does this mean that mirror vdevs can only benefit from super fast L2ARC or something? This was unexpected since I figured L2ARC should make this a bit faster, even with a bit higher latency from encryption. Or maybe this just isn't good test.
On ZIL, I was monitoring activity with zpool iostat -v 1 and it always looked like ZIL gets filled up to 200MB or so, to, both geli encrypted or not. Test showed that I get maybe 10MBps extra on sequential writes (if ZIL is unencrypted).
All in all, what would be the best way to test this again and see if this setup even benefits from this SSD? Does it make sense to put both ZIL and L2CARC on this SSD as they are likely to comete for resources? Are there tunable that should be set (noprefetch is already set to 0)? I guess limiting factor to this is geli but I'm unable to test it unencrypted at this time.
Lot's of questions, so whatever you could answer, thanks.
I managed to get one extra SSD to play around with as cache/log device on my home server (3x2 mirror vdevs), and I tried a bunch of different configurations. My understanding of how this all works is pretty much limited to what I know from forums. So I ran a couple of tests and have a few questions, maybe you could help.
All my disks are encrypted, so at first I assumed that I would need to encrypt both ZIL and L2ARC to keep it fully encrypted again. So, for ZIL, I figured, that makes sense.
But what about L2ARC? From my understanding of how this works, L2ARC must be pointed from main memory. So, after a reboot, I would assume that contents of L2ARC can't be retrieved. How would it be possible to retrieve this data, and so, is encrypting this L2ARC necessary?
As it turns out, however, bonnie++ tests showed pretty much same and consistent results whether I used log/cache or not. For example this is with both log/cache on same SSD:
Code:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
zhenbox.home.do 96G 142 99 68457 13 44808 11 366 96 144211 13 180.3 3
Latency 132ms 6085ms 5642ms 117ms 386ms 741ms
Version 1.96 ------Sequential Create------ --------Random Create--------
zhenbox.home.domain -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 27137 97 +++++ +++ 27790 98 21062 97 +++++ +++ 12205 99
Latency 12448us 97us 167us 25335us 82us 237us
1.96,1.96,zhenbox.home.domain,1,1337982754,96G,,142,99,68457,13,44808,11,366,96,144211,13,180.3,3,16
,,,,,27137,97,+++++,+++,27790,98,21062,97,+++++...
This is just with just L2ARC (unencrypted):
Code:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
zhenbox.home.do 96G 129 99 81017 15 51120 13 367 96 166962 15 134.7 2
Latency 264ms 3556ms 3512ms 121ms 382ms 998ms
Version 1.96 ------Sequential Create------ --------Random Create--------
zhenbox.home.domain -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 28051 96 +++++ +++ 27384 98 24952 97 +++++ +++ 27824 98
Latency 9620us 604us 123us 28137us 25us 106us
1.96,1.96,zhenbox.home.domain,1,1337985771,96G,,129,99,81017,15,51120,13,367,96,166962,15,134.7,2,16
,,,,,28051,96,+++++,+++,27384,98,24952,97,+++++...
In fact, one thing I noticed is that results seem to be a bit better without (!) L2ARC on SSD (about 10-20MBps better reads) on my mirror vdevs. Why is this? And does this mean that mirror vdevs can only benefit from super fast L2ARC or something? This was unexpected since I figured L2ARC should make this a bit faster, even with a bit higher latency from encryption. Or maybe this just isn't good test.
On ZIL, I was monitoring activity with zpool iostat -v 1 and it always looked like ZIL gets filled up to 200MB or so, to, both geli encrypted or not. Test showed that I get maybe 10MBps extra on sequential writes (if ZIL is unencrypted).
All in all, what would be the best way to test this again and see if this setup even benefits from this SSD? Does it make sense to put both ZIL and L2CARC on this SSD as they are likely to comete for resources? Are there tunable that should be set (noprefetch is already set to 0)? I guess limiting factor to this is geli but I'm unable to test it unencrypted at this time.
Lot's of questions, so whatever you could answer, thanks.