ZFS zil/cache creation takes a long time

P

pboehmer

We recently upgraded a 11TB NFS server from 64GB to 192GB RAM and added a 400GB Intel P3700 SSD PCIe card that is recognized by FreeBSD using the nvme and nvd drivers. I partitioned the SSD drive into a 4GB ZIL and 192GB cache (leaving ~176GB unused).

In single user mode, when I issued the command:
zpool add store cache /dev/gpt/cache0
zpool add store log /dev/gpt/log0
It took just over 40 minutes to complete the cache command and another couple of minutes to add the log. At the beginning, I thought that the system had gotten hung up, but I noticed that the LED indicator on the card was showing activity, so I let it go.

Does this seem like normal behavior? If memory serves, creation of the 11TB pool (SAS drives) easily took under a minute.
 
Which version of FreeBSD? Later versions (9.3 at least, maybe earlier) enabled TRIM-on-create for cache devices (actually, it runs TRIM on all SSDs when added to a ZFS pool). There's a sysctl to disable this, and will make the creation of cache devices much speedier:
Code: 
$ sysctl -d vfs.zfs.vdev.trim_on_init
vfs.zfs.vdev.trim_on_init: Enable/disable full vdev trim on initialisation
 
Currently on 9.3-STABLE. Will try this knob during our next maintenance window and see if this makes a difference.
 
You must log in or register to reply here.
Back
Top