Solved [Solved] ZFS Raidz add SSD for cache

E

eduardm

Hi,

I have installed and configured a simple RAIDZ ZFS system on FreeBSD 9.1-RELEASE-p4. My server configuration is:

DELL PowerEdge T110 E3-1270v2
Intel Xeon E3-1270
Memory: 32GB ECC RAM
Harddisk 1: 500GB HDD
Harddisk 2: 500GB HDD
Harddisk 3: 500GB HDD


I have configured my ZFS pool following this:
Code: 
# gpt
gpart create -s gpt ada0
gpart create -s gpt ada1
gpart create -s gpt ada2
gpart add -b 34 -s 94 -t freebsd-boot ada0
gpart add -b 34 -s 94 -t freebsd-boot ada1
gpart add -b 34 -s 94 -t freebsd-boot ada2
gpart add -t freebsd-zfs -l disk0 ada0
gpart add -t freebsd-zfs -l disk1 ada1
gpart add -t freebsd-zfs -l disk2 ada2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2

# create the zroot
zpool create -f zroot raidz /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2
zpool set bootfs=zroot zroot
zpool set listsnapshots=on zroot
zpool set autoreplace=on zroot
zpool set autoexpand=on zroot
zfs set checksum=fletcher4 zroot
zfs set atime=off zroot
zfs set copies=2 zroot
zfs set compression=lzjb zroot
zfs set mountpoint=/mnt zroot
zpool export zroot
zpool import -o cachefile=/var/tmp/zpool.cache zroot

# create the pool
zfs create zroot/tmp
zfs create zroot/usr
zfs create zroot/usr/ports
zfs create zroot/usr/ports/distfiles
zfs create zroot/usr/ports/packages
zfs create zroot/usr/src
zfs create zroot/var
zfs create zroot/var/crash
zfs create zroot/var/db
zfs create zroot/var/db/mysql
zfs create zroot/var/db/pgsql
zfs create zroot/var/db/pkg
zfs create zroot/var/empty
zfs create zroot/var/log
zfs create zroot/var/mail
zfs create zroot/var/run
zfs create zroot/var/tmp
zfs create zroot/var/sonya
zfs create zroot/var/nginx

zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp
zfs create -o compression=lzjb -o setuid=off zroot/usr/ports
zfs set copies=1 zroot/usr/ports
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/distfiles
zfs create -o compression=off -o exec=off -o setuid=off zroot/usr/ports/packages
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src
zfs set copies=1 zroot/usr/src
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/db
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg
zfs create -o exec=off -o setuid=off zroot/var/empty
zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log
zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail
zfs create -o exec=off -o setuid=off zroot/var/run
zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp
zfs create -o compression=off -o exec=off -o setuid=off zroot/var/sonya
zfs set recordsize=16k zroot/var/db/mysql
zfs set recordsize=8k zroot/var/db/pgsql

# swap create
zfs create -V 4G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set checksum=off zroot/swap
zfs set copies=1 zroot/swap

# Create a symlink to /home and fix some permissions.
chmod 1777 /mnt/tmp
cd /mnt ; ln -s usr/home home
chmod 1777 /mnt/var/tmp

# install freebsd
sh
cd /usr/freebsd-dist
export DESTDIR=/mnt
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

# copy zpool cache
cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache

# make sure zfs starts on boot
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom.options=rw' >> /mnt/boot/loader.conf
echo 'vfs.zfs.prefetch_disable=1' >> /mnt/boot/loader.conf
echo 'vfs.zfs.arc_max="16384M"' >> /mnt/boot/loader.conf
echo 'vfs.zfs.arc_meta_limit="128M"' >> /mnt/boot/loader.conf
echo 'aio_load="YES"' >> /mnt/boot/loader.conf

# otherwise bsd will complain
touch /mnt/etc/fstab

# correct mountpoints
zfs set readonly=on zroot/var/empty
zfs umount -af
zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var

I have done some modifications to loader.conf after restart trying to tuning some stuff:
Code: 
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
vfs.root.mountfrom.options=rw
kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256
kern.ipc.semmap=256
kern.dfldsiz=1073741824
kern.maxbcache=64M
vfs.zfs.prefetch_disable=1
vfs.zfs.arc_max="16384M"
vfs.zfs.arc_meta_limit="128M"
accf_data_load="YES"
accf_http_load="YES"
aio_load="YES"
cc_htcp_load="YES"
net.inet.tcp.tcbhashsize=4096
hw.em.rxd=2048
hw.em.txd=2048
hw.em.rx_process_limit="-1"
net.link.ifqmaxlen=512

Everything is working fine although I just want to make some improvements regarding read/write performance because I have to move few clients with heavy traffic CMS websites in one week. I have 2 HDD bays free left in my server chassis.

I just want to add SSD drive for cache and for what i have been googling around I have understood that I could do that on an existing ZFS pool defined. I also read that I could partition a SSD drive for L2ARC and ZIL.
My provider could supply following SSD drives (do not know yet which is better for my scenario):
120GB:

  • OZC Agility 3
    Kingston SSDNow KC300
240GB:
  • Crucial M4 and M5 series
Click to expand...
Now there are the questions.
1. should I use 2 free bays remaining to add 2 SSD drives and do a mirroring cache? or would it suffice to add another 500 HDD drive to existing zroot pool and 1 SSD drive as cache?

2. Would it be better/safer to use both ZIL and L2ARC as different partitions on SSD drives?
i mean is that correct:
Code: 
gpart create -s gpt ssd1
gpart add -s 4G -t freebsd-zfs ssd1    # ZIL partition
gpart add -t freebsd-zfs ssd1             # L2ARC partion

gpart create -s gpt ssd2
gpart add -s 4G -t freebsd-zfs ssd2    # ZIL partition
gpart add -t freebsd-zfs ssd2             # L2ARC partion

# adding ZIL/L2ARC to pool ?
zpool add zroot cache /dev/gpt/ssd1b /dev/gpt/ssd2b log mirror /dev/gpt/ssd1a /dev/gpt/ssd2a
If so, should I delete/empty vfs.zfs.arc_max and vfs.zfs.arc_meta_limit variables from loader.conf?

Thank you very much for your time.

Kind regards,
eduardm
 
Re: ZFS Raidz add SSD for cache

eduardm said:
Now there are the questions.
1. should I use 2 free bays remaining to add 2 SSD drives and do a mirroring cache? or would it suffice to add another 500 HDD drive to existing zroot pool and 1 SSD drive as cache?

2. Would it be better/safer to use both ZIL and L2ARC as different partitions on SSD drives?
Click to expand...
You can't add another HDD to the existing RAIDZ vdev. You can only add a new vdev, consisting of a single HDD, which will then be non fault tolerant! Lose that drive and you lose the pool with all data in it.

A ZIL is usually only beneficial when having many synchronous writes, like when using NFS. Starting with ZFS v19, you don't need to have a mirrored ZIL anymore, but it is still possible if you want to have your ZIL fault tolerant. Using a single SSD as ZIL, ZFS would just disable the ZIL when the SSD fails and no data should be lost. A ZIL usually needs only enough space for 1 transaction group, so 8-16 GB is adequate in most situations. You can then repartition the remainder as L2ARC.
 
Re: ZFS Raidz add SSD for cache

Hi @eduardm!

I would advice against using the same SSD as both SLOG and L2ARC. Det entire meaning of it is after all Separate LOG device. Why is because of the negative performance hit of having both on the same disk, since after flushing the SLOG is almost always fillng the L2ARC, so gets very intense action, that does best in separating as much as possible. Another very big consideration is whether the SSD has built-in power-loss protection. The SSD's we are currently using are Intel DC S3700 200GB, working beautifully with ZFS. This is important to ensure data integrity (power-loss in the middle of a transaction) in ongoing IO's. Not really for ZFS´s sake (it can live through almost anything), but for the ones depending on it; running databases, services, virtual machines, that kind of stuff. Making sure that every little bit of data gets through, even when there´s a power-out.

Last thing is to take extra care into partitioning and how they are added to the pool, which is easier doing from a Live-CD:
Code: 
gpart create -s gpt daX
gpart add -b 2048 -a 4k -s 24G -t freebsd-zfs -l log1 daX
gpart create -s gpt daY
gpart add -b 2048 -a 4k -s 24G -t freebsd-zfs -l log2 daY
gnop create -S 4096 /dev/gpt/log1
zpool add zroot  log mirror /dev/gpt/log1.nop /dev/gpt/log2
zpool export zroot
gnop destroy /dev/gpt/log1.nop
zpool import zroot

This is done to ensure the best performance, aligning the partitions on an even boundary (1 MiB) and forces ZFS to work with 4k IO's. Why so big partitions? Well, there needs to be enough space to recieve the IO intended for it. That is weighted for double 10 Gb/s, makes about 24 GiB. Easiest in terms of future-proofing is to go big from start:) If your throughout demands are smaller you would be fine having only 1 GB for 1 Gb/s, or 2 GB for double 1Gb/s, and so on, but harder to change if your needs change. And to keep it clean from any other partitions also makes the SSD last longer, giving more blocks to relocate when worn out.

It is possible to add those partitions to the pool in a normal running system (not from Live-CD) with the nop-device and just reboot at a more convinient time, it´s fine either way, the nop-device disappeares upon reboot and ZFS chooses to use the underlying, "real" device instead.

/Sebulon
 
Last edited by a moderator:
Re: ZFS Raidz add SSD for cache

Hi @Sebulon and @adri,

Thank you very much for your quick answers. My provider has supplied me today with 2xKingston SSDNow KC300 and put them already into server:
Code: 
# camcontrol devlist -v
scbus0 on ahcich0 bus 0:
<ST500NM0011 PA09>                 at scbus0 target 0 lun 0 (pass0,ada0)
<>                                 at scbus0 target -1 lun -1 ()
scbus1 on ahcich1 bus 0:
<ST500NM0011 PA09>                 at scbus1 target 0 lun 0 (pass1,ada1)
<>                                 at scbus1 target -1 lun -1 ()
scbus2 on ahcich2 bus 0:
<ST500NM0011 PA09>                 at scbus2 target 0 lun 0 (pass2,ada2)
<>                                 at scbus2 target -1 lun -1 ()
scbus3 on ahcich3 bus 0:
<KINGSTON SV300S37A120G 521ABBF0>  at scbus3 target 0 lun 0 (pass3,ada3)
<>                                 at scbus3 target -1 lun -1 ()
scbus4 on ahcich4 bus 0:
<KINGSTON SV300S37A120G 521ABBF0>  at scbus4 target 0 lun 0 (pass4,ada4)
<>                                 at scbus4 target -1 lun -1 ()
scbus5 on ahcich5 bus 0:
<>                                 at scbus5 target -1 lun -1 ()
scbus6 on ctl2cam0 bus 0:
<>                                 at scbus6 target -1 lun -1 ()
scbus-1 on xpt0 bus 0:
<>                                 at scbus-1 target -1 lun -1 (xpt0)

My zfs raidz pool looks like:
Code: 
# zpool status zroot
  pool: zroot
 state: ONLINE
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        zroot          ONLINE       0     0     0
          raidz1-0     ONLINE       0     0     0
            gpt/disk0  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
            gpt/disk2  ONLINE       0     0     0

errors: No known data errors

I have been reading for a few hours in this forum a few topics regarding zfs cache. I also have noticed that my raidz pool is not 4K align :\ :
Code: 
# diskinfo -v ada0
ada0
        512             # sectorsize
        500107862016    # mediasize in bytes (465G)
        976773168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        969021          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        Z1M1DT9X        # Disk ident.
# camcontrol identify ada0 | egrep 'sector size'
sector size           logical 512, physical 512, offset 0

# diskinfo -v ada1
ada1
        512             # sectorsize
        500107862016    # mediasize in bytes (465G)
        976773168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        969021          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        Z1M185ZR        # Disk ident.
# camcontrol identify ada1 | egrep 'sector size'
sector size           logical 512, physical 512, offset 0

# diskinfo -v ada2
ada2
        512             # sectorsize
        500107862016    # mediasize in bytes (465G)
        976773168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        969021          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        Z1M1ASQA        # Disk ident.
# camcontrol identify ada2 | egrep 'sector size'
sector size           logical 512, physical 512, offset 0

However, that is! I guess is too late for that to do something now without remaking the pool and not losing any data. I thought it over and i came up to conclusion that i should use 1 SSD drive for L2ARC (ada3 drive) and 1 SSD drive for SLOG (ada4 drive). Is that the correct setup for gain some improvements on read/write in my situation?

I would like if your are kindly enough to guide me to this process since it is for first time when i am doing this and do not want to have any negative experience (losing current pool or data from it) with current working pool (zroot). Taking @Sebulon code as example would the code below be proper for safely execute into a running system?
Code: 
# create and align SLOG device (use entire hdd space so i left out -s 24G, is that ok?)
gpart create -s gpt ada3
gpart add -b 2048 -a 4k -t freebsd-zfs -l log ada3

# create and align 4k L2ARC device (use entire hdd space so i left out -s 24G, is that ok?)
gpart create -s gpt ada4
gpart add -b 2048 -a 4k -t freebsd-zfs -l cache ada4

# create gnop 4k
gnop create -S 4096 /dev/gpt/log

# add both SLOG/L2ARC device into current pool?
zpool add zroot  log cache /dev/gpt/log.nop /dev/gpt/cache

# exporting new pool
zpool export zroot

# import pool do not distroy gnop yet because i will reboot after import and gnop will disapper by itself
zpool import zroot

# checking new pool
zpool status zroot

# if log and cache appear as new devices and no errors are showing reboot the system
reboot

Also, before reboot, do i need to add additional parameters into loader.conf? I have set already vfs.zfs.l2arc_noprefetch=0 via sysctl (I have a client with a small streaming site which use rtmps protocol and i have read that will make L2ARC to serve out streaming data better).

Thank you very much for your time.

Kind regards,
eduardm
 
Last edited by a moderator:
Re: ZFS Raidz add SSD for cache

Hi @eduardm!

OK, Kingston SSDNow KC300, no power-loss protection on those, according to:
http://www.kingston.com/datasheets/skc300s3_us.pdf

So won´t be able to guarantee 100% data consistency, just so you are aware of that.

# create and align SLOG device (use entire hdd space so i left out -s 24G, is that ok?)
Click to expand...
Well, no, not really. You see, making the partition only as small as it has to be makes the SSD's controller firmware able to use that unused space to reallocate blocks that are worn out. Having a partition that spans across the entire drive leaves nothing to reallocate. Now, SSD's already have blocks to use for this, it´s called default over provisioning, but having more blocks available is never a bad thing.

The 500 GB data drives you started out with are most likely real 512 byte disks, so that´s quite all right leaving as they are.

# add both SLOG/L2ARC device into current pool?
zpool add zroot log cache /dev/gpt/log.nop /dev/gpt/cache
Click to expand...
Yes, but you have the command mixed up. This is how it´s supposed to look like:
zpool add zroot log /dev/gpt/log.nop cache /dev/gpt/cache

# exporting new pool
zpool export zroot
Click to expand...
No, skip export/import, you can´t do that on a running system. Just "add" and then reboot.

Also, before reboot, do i need to add additional parameters into loader.conf?
Click to expand...
Nope, you should be good to go.

/Sebulon
 
Last edited by a moderator:
Re: ZFS Raidz add SSD for cache

Hi @Sebulon!

Thank you very much for your suggestions. I did the following:
Code: 
# create and align SLOG device 
gpart create -s gpt ada3
gpart add -b 2048 -a 4k -s 24G -t freebsd-zfs -l log ada3

# create and align 4k L2ARC device 
gpart create -s gpt ada4
gpart add -b 2048 -a 4k -s 24G -t freebsd-zfs -l cache ada4

# create gnop 4k
gnop create -S 4096 /dev/gpt/log

# add both SLOG/L2ARC device into current pool
zpool add zroot log /dev/gpt/log.nop cache /dev/gpt/cache

# reboot system
reboot

Server has booted up normaly and i can see devices attached to existing pool:
Code: 
[cmd]root@# zpool status zroot[/cmd]
  pool: zroot
 state: ONLINE
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        zroot          ONLINE       0     0     0
          raidz1-0     ONLINE       0     0     0
            gpt/disk0  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
            gpt/disk2  ONLINE       0     0     0
        logs
          ada3p1       ONLINE       0     0     0
        cache
          gpt/cache    ONLINE       0     0     0

errors: No known data errors

Out of curiously i also want to see how much are "eating up" (maybe it is too soon):
Code: 
[cmd]root@# zpool iostat -v[/cmd]
                  capacity     operations    bandwidth
pool           alloc   free   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
zroot           524G   852G     11     42   234K   178K
  raidz1        524G   852G     11     42   234K   174K
    gpt/disk0      -      -      6     14  79.7K  91.4K
    gpt/disk1      -      -      6     14  75.7K  91.4K
    gpt/disk2      -      -      6     14  84.4K  91.4K
logs               -      -      -      -      -      -
  ada3p1        360K  23.9G      0      0    658  3.41K
cache              -      -      -      -      -      -
  gpt/cache     327M  23.7G      0      3    116   280K
-------------  -----  -----  -----  -----  -----  -----

I have noticed that before doing this i had ARC size at 100%, now typing zfs-stats -AL i get:
Code: 
------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Mar  3 01:31:55 2014
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                7
        Recycle Misses:                         0
        Mutex Misses:                           0
        Evict Skips:                            0

ARC Size:                               2.12%   347.68  MiB
        Target Size: (Adaptive)         100.00% 16.00   GiB
        Min Size (Hard Limit):          6.05%   991.36  MiB
        Max Size (High Water):          16:1    16.00   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       49.99%  8.00    GiB
        Frequently Used Cache Size:     50.01%  8.00    GiB

ARC Hash Breakdown:
        Elements Max:                           15.36k
        Elements Current:               100.00% 15.36k
        Collisions:                             766
        Chain Max:                              2
        Chains:                                 208

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        0
        Tried Lock Failures:                    11
        IO In Progress:                         0
        Low Memory Aborts:                      0
        Free on Write:                          0
        Writes While Full:                      3
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           0

L2 ARC Size: (Adaptive)                         317.67  MiB
        Header Size:                    0.00%   0

L2 ARC Breakdown:                               14.10k
        Hit Ratio:                      0.00%   0
        Miss Ratio:                     100.00% 14.10k
        Feeds:                                  1.34k

L2 ARC Buffer:
        Bytes Scanned:                          313.06  GiB
        Buffer Iterations:                      1.34k
        List Iterations:                        85.68k
        NULL List Iterations:                   49

L2 ARC Writes:
        Writes Sent:                    100.00% 462

------------------------------------------------------------------------
I guess it Is normal since I have L2ARC activated and now I have 29G free RAM,at least that shows to me when I type top. Is there anything that I have to tune (as sysctrl param or loader.conf) or to look after?

Many thanks, once again.

Kind regards.
 
Last edited by a moderator:
Re: ZFS Raidz add SSD for cache

Hi,

After 12 hours online my pool and it's cache looks like:
zpool status zroot
Code: 
 pool: zroot
 state: ONLINE
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        zroot          ONLINE       0     0     0
          raidz1-0     ONLINE       0     0     0
            gpt/disk0  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
            gpt/disk2  ONLINE       0     0     0
        logs
          ada3p1       ONLINE       0     0     0
        cache
          gpt/cache    ONLINE       0     0     0

errors: No known data errors
zpool iostat -v
Code: 
                  capacity     operations    bandwidth
pool           alloc   free   read  write   read  write
-------------  -----  -----  -----  -----  -----  -----
zroot           525G   851G      4     42   249K   165K
  raidz1        525G   851G      4     42   249K   159K
    gpt/disk0      -      -      2     13  82.2K  84.3K
    gpt/disk1      -      -      2     13  82.7K  84.3K
    gpt/disk2      -      -      2     13  84.8K  84.3K
logs               -      -      -      -      -      -
  ada3p1        800K  23.9G      0      0     18  5.28K
cache              -      -      -      -      -      -
  gpt/cache    11.9G  12.1G      1      3  12.8K   299K
-------------  -----  -----  -----  -----  -----  -----
zfs-stats -AL
Code: 
------------------------------------------------------------------------
ZFS Subsystem Report                            Mon Mar  3 12:51:13 2014
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                7
        Recycle Misses:                         117.53k
        Mutex Misses:                           2
        Evict Skips:                            41.81k

ARC Size:                               66.08%  10.57   GiB
        Target Size: (Adaptive)         100.00% 16.00   GiB
        Min Size (Hard Limit):          6.05%   991.36  MiB
        Max Size (High Water):          16:1    16.00   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       47.08%  7.53    GiB
        Frequently Used Cache Size:     52.92%  8.47    GiB

ARC Hash Breakdown:
        Elements Max:                           159.29k
        Elements Current:               100.00% 159.29k
        Collisions:                             138.27k
        Chain Max:                              5
        Chains:                                 20.05k

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        1.01m
        Tried Lock Failures:                    4.01k
        IO In Progress:                         0
        Low Memory Aborts:                      0
        Free on Write:                          191
        Writes While Full:                      7
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           0

L2 ARC Size: (Adaptive)                         10.43   GiB
        Header Size:                    0.00%   0

L2 ARC Breakdown:                               252.37k
        Hit Ratio:                      26.62%  67.19k
        Miss Ratio:                     73.38%  185.19k
        Feeds:                                  42.11k

L2 ARC Buffer:
        Bytes Scanned:                          36.29   TiB
        Buffer Iterations:                      42.11k
        List Iterations:                        2.70m
        NULL List Iterations:                   924.08k

L2 ARC Writes:
        Writes Sent:                    100.00% 24.76k

------------------------------------------------------------------------

Other params that might be useful:
sysctl -a | egrep 'arc'
Code: 
device  arcmsr
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.arc_meta_limit: 134217728
vfs.zfs.arc_meta_used: 408825272
vfs.zfs.arc_min: 1039521408
vfs.zfs.arc_max: 17179869184
debug.adaptive_machine_arch: 1
hw.machine_arch: amd64
kstat.zfs.misc.arcstats.hits: 15103524
kstat.zfs.misc.arcstats.misses: 253546
kstat.zfs.misc.arcstats.demand_data_hits: 14573072
kstat.zfs.misc.arcstats.demand_data_misses: 94339
kstat.zfs.misc.arcstats.demand_metadata_hits: 530392
kstat.zfs.misc.arcstats.demand_metadata_misses: 159184
kstat.zfs.misc.arcstats.prefetch_data_hits: 0
kstat.zfs.misc.arcstats.prefetch_data_misses: 0
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 60
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 23
kstat.zfs.misc.arcstats.mru_hits: 2278400
kstat.zfs.misc.arcstats.mru_ghost_hits: 100951
kstat.zfs.misc.arcstats.mfu_hits: 12825079
kstat.zfs.misc.arcstats.mfu_ghost_hits: 33170
kstat.zfs.misc.arcstats.allocated: 353772
kstat.zfs.misc.arcstats.deleted: 7
kstat.zfs.misc.arcstats.stolen: 45697
kstat.zfs.misc.arcstats.recycle_miss: 118263
kstat.zfs.misc.arcstats.mutex_miss: 2
kstat.zfs.misc.arcstats.evict_skip: 41857
kstat.zfs.misc.arcstats.evict_l2_cached: 612597760
kstat.zfs.misc.arcstats.evict_l2_eligible: 351830528
kstat.zfs.misc.arcstats.evict_l2_ineligible: 4096
kstat.zfs.misc.arcstats.hash_elements: 159784
kstat.zfs.misc.arcstats.hash_elements_max: 159784
kstat.zfs.misc.arcstats.hash_collisions: 139169
kstat.zfs.misc.arcstats.hash_chains: 20147
kstat.zfs.misc.arcstats.hash_chain_max: 5
kstat.zfs.misc.arcstats.p: 8086201856
kstat.zfs.misc.arcstats.c: 17179869184
kstat.zfs.misc.arcstats.c_min: 1039521408
kstat.zfs.misc.arcstats.c_max: 17179869184
kstat.zfs.misc.arcstats.size: 11396518840
kstat.zfs.misc.arcstats.hdr_size: 45758376
kstat.zfs.misc.arcstats.data_size: 11114904064
kstat.zfs.misc.arcstats.other_size: 235856400
kstat.zfs.misc.arcstats.l2_hits: 67560
kstat.zfs.misc.arcstats.l2_misses: 185971
kstat.zfs.misc.arcstats.l2_feeds: 42226
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_read_bytes: 554350592
kstat.zfs.misc.arcstats.l2_write_bytes: 12919515136
kstat.zfs.misc.arcstats.l2_writes_sent: 24869
kstat.zfs.misc.arcstats.l2_writes_done: 24869
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 191
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 11245168640
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.arcstats.l2_write_trylock_fail: 4007
kstat.zfs.misc.arcstats.l2_write_passed_headroom: 1009928
kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0
kstat.zfs.misc.arcstats.l2_write_in_l2: 496068106
kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0
kstat.zfs.misc.arcstats.l2_write_not_cacheable: 6742
kstat.zfs.misc.arcstats.l2_write_full: 7
kstat.zfs.misc.arcstats.l2_write_buffer_iter: 42226
kstat.zfs.misc.arcstats.l2_write_pios: 24869
kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 40022047418368
kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 2702446
kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 927196
vmstat
Code: 
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad0 ad1   in   sy   cs us sy id
 0 0 0  25971M    14G   167  36  61   0   217   0   0   0  300 1742 1403  0  0 100
cat /boot/loader.conf
Code: 
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
vfs.root.mountfrom.options=rw
kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256
kern.ipc.semmap=256
kern.dfldsiz=1073741824
kern.maxbcache=64M
vfs.zfs.prefetch_disable=1
vfs.zfs.arc_max="16384M"
vfs.zfs.arc_meta_limit="128M"
accf_data_load="YES"
accf_http_load="YES"
aio_load="YES"

I have noticed that both ARC and L2ARC are growing and available RAM from 29G free, when i have added SLOG/L2ARC, is now 14G and I presume will decrease thenceforth.

Is that a normal situation, I mean this was expected, or is there something wrong? What will happen when L2ARC cache is achiving all 24G allocated?

Thank you very much for your time.

Kind regards
 
Re: ZFS Raidz add SSD for cache

Hi @eduardm!

Sorry, I should have told you to only over provision the log. The cache device is more important to have a bigger partition, for space to actually cache stuff in:) And it´s totally OK if it wears out, it´s nothing important stored in there so it can be lost at any time. And it´s possible to do while running:
Code: 
zpool remove zroot gpt/cache
gpart delete -i 1 ada4
gpart add -b 2048 -a 4k -t freebsd-zfs -l cache ada4
zpool add zroot cache gpt/cache

I have noticed that both ARC and L2ARC are growing and available RAM from 29G free, when i have added SLOG/L2ARC, is now 14G and I presume will decrease thenceforth.
Click to expand...
Yes it will, it might even swap a little and that´s normal as well.

This however, is more confusing, though only cosmetical:
pool: zroot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gpt/disk0 ONLINE 0 0 0
gpt/disk1 ONLINE 0 0 0
gpt/disk2 ONLINE 0 0 0
logs
ada3p1 ONLINE 0 0 0
cache
gpt/cache ONLINE 0 0 0

errors: No known data errors
Click to expand...
I would have expected it to choose the label provider, the one in /dev/gpt called log, but as I said it´s only a cosmetical issue. Though if you want to correct it you need to start it up with a FreeBSD install and choose "Live-CD"
Code: 
zpool import -o cachefile=/tmp/zpool.cache -d /dev/gpt zroot
(Double check that it chose all the disks with label providers)
zpool status
(Looks good?)
mount -t zfs zroot /mnt
cp /tmp/zpool.cache /mnt/boot/zfs/
And then just reboot. That should solve it.

/Sebulon
 
Last edited by a moderator:
Re: ZFS Raidz add SSD for cache

Hi @Sebulon

I have reattached the gpt/cache drive. Your code worked like a charm :e

Regarding gpt/log it does not bother me as long as it works fine. But next time when I have the opportunity to boot my server with FreeBSD DVD install I will take into consideration doing that.

Saying that I consider my topic as solved if one of the moderators/administrators would be kindly enough to change the title.

Thank you very much once again.

Kind regards,

eduardm
 
Last edited by a moderator:
You must log in or register to reply here.
Back
Top