ZFS L2ARC cache partition using NOP gives errors on 10.1-RELEASE

A

abefar

Greetings!

I just bought an SSD (Crucial MX100 256GB) to use as L2ARC read cache for the RAID-1 system pool in my home server.
According to smartctl (sysutils/smartmontools), the drive has 512 bytes logical and 4096 bytes physical sector sizes, therefore I used a gnop(8) device for attaching it to the zpool as a cache. However, it reports an enormous amount of write errors whenever it is attached through the NOP device. After about a minute it looks like this:

Code: 
# zpool status -v sys
  pool: sys
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
   attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
   using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 0h37m with 0 errors on Sun Dec  7 19:44:05 2014
config:

   NAME  STATE  READ WRITE CKSUM
   sys  ONLINE  0  0  0
    mirror-0  ONLINE  0  0  0
    diskid/DISK-S13PJDWS510284p3  ONLINE  0  0  0
    diskid/DISK-S13PJDWS518340p3  ONLINE  0  0  0
   cache
    diskid/DISK-14340D0D61B1p1.nop  ONLINE  0 21.5K  0

errors: No known data errors
And the error count just keeps increasing. If I add the cache partition to the pool "directly", i.e. without using the .nop device, it works perfectly.

What am I doing wrong? Is gnop(8) even needed for L2ARC? Most of the examples I have seen uses it.

I originally prepared and attached the disk like this (intentionally leaving some free space for the sake of over-provisioning):
Code: 
# gpart create -s GPT diskid/DISK-14340D0D61B1
diskid/DISK-14340D0D61B1 created
# gpart add -t freebsd-zfs -a 4k -s 180G diskid/DISK-14340D0D61B1
diskid/DISK-14340D0D61B1p1 added
# gnop create -S 4096 diskid/DISK-14340D0D61B1p1
# zpool set bootfs="" sys
# zpool add sys cache diskid/DISK-14340D0D61B1p1.nop
# zpool set bootfs=sys/ROOT sys
 
I don't think gnop(8) is needed anymore as gpart(8) now has an alignment (-a) option. That should already take care of aligning the partition on 4 K boundaries.
 
gnop(8) was not used for alignment. It was used to create a device with 4K blocks, which in turn forced ZFS to use that as the device block size. There is a sysctl(8) for the minimum ZFS block size now.
 
I know about the partition alignment issue - my initial reason for going with a partition instead of the entire device was to be able to use -a 4k, so that I could be sure it wouldn't start writing cache data from sector 63 or something like that, and also to leave a slice of the disk unused to ease the work of its wear-leveling system.

I cannot seem to find much information about the data structure of L2ARC however, and I am not very eager to dig into the source code. I was hoping the data is stored on the cache device in chunks matching the block or record size of the overlying ZFS pool, as that would make the block alignment issue non-existent.

But since so many people seems to use gnop(8) I started to think that there might be a reason for it (maybe ZFS writes L2ARC data with the smallest possible block size in order to save space for small files), although it would seem odd for a caching technique developed after the entire Advanced Format fake-sector-size disk era began.
 
gnop(8) is needed on older versions of FreeBSD (pre-10.0, I believe; maybe pre-9.3?). On those systems, you have to fake a sector size to make ZFS set the ashift property correctly for that vdev.

On newer systems, you can set the following sysctl(8) to achieve the same result (minimum 4K blocks):
# sysctl vfs.zfs_min_auto_ashift=12

With that sysctl(8) set, you can just add the L2ARC device directly, no gnop(8) required.
 
Using 4K encrypted drive will cause the same issue as well.
geli init -s 4096 -K /boot/keys/disks.key -P -l 256 -e AES-CBC /dev/gpt/l2arc

But it can be resolved by changing the size to 512 instead of 4096.
geli init -s 512 -K /boot/keys/disks.key -P -l 256 -e AES-CBC /dev/gpt/l2arc
 
You must log in or register to reply here.
Back
Top