Solved OpenZFS: graphing ARC and L2ARC

Please: can anyone tell, or guess, what's used for the graphs at and under <https://github.com/openzfs/zfs/issues/10508#issuecomment-940713843>?

As far as I can tell, graphs there are ARC-focused, although I see (written) "… 512 GB available in the L2ARC. …".

If not difficult, I'd like to produce graphs with some focus on L2ARC. Context:




From 2013 (the image is no longer available):

You will get a significant performance boost. The below graph is from a very busy web server. I usually get 60-70 % hit ratio. The system was rebooted on the 10th of May. Look how quickly L2ARC catches up.

<http://www.aisecure.net/wp-content/uploads/2013/05/zfs_stats_l2efficiency-week.png>

More recent (2019):

… That machine also has L2ARC, but the hit rate isn't that great. Even after the L2ARC is 'warm', it's only about 25%.
 
I forgot, net-mgmt/netdata/ already installed:
Under efficiencies, for L2 there's a hidden anchor <http://localhost:19999/#chart_zfs_l2hits> that seems to be slightly broken by the page header. In some cases, there's not only overlap, there's also wobble (maybe caused by the overlap).

A manual edition of the URL: <http://localhost:19999/#chart_zfs_l2hits;after=-86400;before=0;theme=slate> then, after scrolling a fraction to avoid the overlap:

1637726580540.png
zfs-mon (not a graph) shortly afterwards:

1637726957314.png
 
Last edited:
 
For now, I'll mark this topic Solved without doing what I originally intended:

… produce graphs with some focus on L2ARC. …

Results over time might be not suitably meaningful, for two reasons:
  1. zpool-iostat(8): unreasonably/impossibly high alloc and free measurements for two cache devices (simple USB thumb drives) · Issue #12779 · openzfs/zfs
  2. occasional sudden chilling, which (for now) I'll attribute to inadequacies involving USB:

1637887237430.png


– <http://localhost:19999/#menu_zfs_submenu_size;after=-410000;before=0>

Code:
% grep -e "<BOOT>" -e suspend /var/log/messages
Nov 22 16:13:41 mowa219-gjp4-8570p-freebsd acpi[50039]: suspend at 20211122 16:13:41
Nov 23 02:24:06 mowa219-gjp4-8570p-freebsd acpi[50562]: suspend at 20211123 02:24:06
Nov 23 07:21:21 mowa219-gjp4-8570p-freebsd acpi[58693]: suspend at 20211123 07:21:21
Nov 23 16:06:44 mowa219-gjp4-8570p-freebsd acpi[88629]: suspend at 20211123 16:06:44
Nov 24 08:31:49 mowa219-gjp4-8570p-freebsd acpi[17820]: suspend at 20211124 08:31:49
Nov 24 16:24:47 mowa219-gjp4-8570p-freebsd acpi[43156]: suspend at 20211124 16:24:47
Nov 25 07:20:43 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Nov 25 07:27:12 mowa219-gjp4-8570p-freebsd kernel: ---<<BOOT>>---
Nov 25 07:44:41 mowa219-gjp4-8570p-freebsd acpi[3921]: suspend at 20211125 07:44:41
Nov 25 16:11:52 mowa219-gjp4-8570p-freebsd acpi[32194]: suspend at 20211125 16:11:52
%
 
… occasional sudden chilling, which (for now) I'll attribute to inadequacies involving USB: …

For reference:

… I think there are 2 ways to expire the cache unless there is a bug: stream of new l2arc writes and deletions of files/snapshots/datasets. …

… your speculation that a small set of files (=1 in this case) get constantly overwritten is exactly what happened here, probably filling up the L2ARC with blocks that soon after get deleted. …

… The L2ARC is implemented as an on-disk ring buffer which is why eventually all of the block were overwritten. This is at the heart of the design …

– <https://github.com/openzfs/zfs/discussions/11209#discussioncomment-232924> under Almost empty L2ARC on raidz2 (bleeding out after reboot).

Ignoring (for a moment) the RAIDZ2 aspect: I suspect that my current case is unrelated. Sudden chills are sometimes noticeable after boot, but there might be a different explanation.
 
Results over time might be not suitably meaningful, for two reasons:

Better recently. Meaningful:

1640254449650.png
  • probably two thumb drives at the time, both for my boot pool
  • 39.4 GiB compressed
  • 110.3 GiB uncompressed
Code:
% date ; uptime
Thu 23 Dec 2021 10:17:10 GMT
10:17a.m.  up 1 day, 23:26, 7 users, load averages: 3.33, 2.86, 2.20
% grep suspend /var/log/messages
Dec 15 17:21:01 mowa219-gjp4-8570p-freebsd acpi[33360]: suspend at 20211215 17:21:01
Dec 16 08:03:30 mowa219-gjp4-8570p-freebsd acpi[78756]: suspend at 20211216 08:03:30
Dec 16 16:00:11 mowa219-gjp4-8570p-freebsd acpi[7415]: suspend at 20211216 16:00:11
Dec 17 07:55:44 mowa219-gjp4-8570p-freebsd acpi[36171]: suspend at 20211217 07:55:44
Dec 17 16:15:21 mowa219-gjp4-8570p-freebsd acpi[59657]: suspend at 20211217 16:15:21
Dec 21 08:07:29 mowa219-gjp4-8570p-freebsd acpi[19452]: suspend at 20211221 08:07:29
Dec 21 13:58:29 mowa219-gjp4-8570p-freebsd acpi[39305]: suspend at 20211221 13:58:29
%

Postscripts

Sunday 2022-04-03:

1649604575153.png
  • three thumb drives
  • 55.0 GiB compressed
  • 124.1 GiB uncompressed.
Sunday 2022-04-10:

1649578007944.png
  • three thumb drives
  • 42.0 GiB compressed
  • 105.5 GiB uncompressed
  • at 07:32 this morning one of three cache devices was rudely disconnected, for a moment – probably trod on, by a cat – no harm done
Code:
% tail -n 48 /var/log/messages
Apr 10 03:55:26 mowa219-gjp4-8570p-freebsd kernel: GEOM_ELI: Device ada0p2.eli created.
Apr 10 03:55:26 mowa219-gjp4-8570p-freebsd kernel: GEOM_ELI: Encryption: AES-XTS 128
Apr 10 03:55:26 mowa219-gjp4-8570p-freebsd kernel: GEOM_ELI:     Crypto: accelerated software
Apr 10 03:55:27 mowa219-gjp4-8570p-freebsd kernel:  blanktime.
Apr 10 03:55:37 mowa219-gjp4-8570p-freebsd dbus-daemon[21140]: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.7" (uid=1002 pid=21550 comm="/usr/local/lib/libexec/org_kde_powerdevil ") interface="org.freedesktop.ConsoleKit.Manager" member="CanSuspendThenHibernate" error name="(unset)" requested_reply="0" destination="org.freedesktop.ConsoleKit" (uid=0 pid=21346 comm="/usr/local/sbin/console-kit-daemon --no-daemon ")
Apr 10 03:55:40 mowa219-gjp4-8570p-freebsd /hp-systray[21637]: hp-systray[21637]: error: option -s not recognized
Apr 10 03:56:48 mowa219-gjp4-8570p-freebsd su[21911]: grahamperrin to root on /dev/pts/3
Apr 10 03:57:00 mowa219-gjp4-8570p-freebsd su[21941]: grahamperrin to root on /dev/pts/6
Apr 10 04:55:48 mowa219-gjp4-8570p-freebsd gnome-keyring-daemon[26041]: couldn't access control socket: /var/run/user/1002/keyring/control: No such file or directory
Apr 10 04:55:51 mowa219-gjp4-8570p-freebsd gnome-keyring-daemon[26041]: couldn't allocate secure memory to keep passwords and or keys from being written to the disk
Apr 10 05:46:30 mowa219-gjp4-8570p-freebsd kernel: pid 22663 (thunderbird), jid 0, uid 1002: exited on signal 11 (core dumped)
Apr 10 07:32:18 mowa219-gjp4-8570p-freebsd kernel: ugen0.6: <Kingston DataTraveler 3.0> at usbus0 (disconnected)
Apr 10 07:32:18 mowa219-gjp4-8570p-freebsd kernel: umass2: at uhub7, port 4, addr 5 (disconnected)
Apr 10 07:32:18 mowa219-gjp4-8570p-freebsd kernel: da2 at umass-sim2 bus 2 scbus5 target 0 lun 0
Apr 10 07:32:18 mowa219-gjp4-8570p-freebsd kernel: da2: <Kingston DataTraveler 3.0 >  s/n E0D55EA1C84FF390A9500FDA detached
Apr 10 07:32:20 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): Periph destroyed
Apr 10 07:32:20 mowa219-gjp4-8570p-freebsd kernel: umass2: detached
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: ugen0.6: <Kingston DataTraveler 3.0> at usbus0
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: umass2 on uhub7
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: umass2: <Kingston DataTraveler 3.0, class 0/0, rev 3.10/0.01, addr 7> on usbus0
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: da2 at umass-sim2 bus 2 scbus5 target 0 lun 0
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: da2: <Kingston DataTraveler 3.0 > Removable Direct Access SPC-4 SCSI device
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: da2: Serial Number E0D55EA1C84FF390A9500FDA
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: da2: 400.000MB/s transfers
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: da2: 29510MB (60437492 512 byte sectors)
Apr 10 07:32:21 mowa219-gjp4-8570p-freebsd kernel: da2: quirks=0x2<NO_6_BYTE>
Apr 10 07:32:24 mowa219-gjp4-8570p-freebsd ZFS[35674]: vdev state changed, pool_guid=1913339710710793892 vdev_guid=7099321644620494833
Apr 10 07:32:24 mowa219-gjp4-8570p-freebsd ZFS[35678]: vdev is removed, pool_guid=1913339710710793892 vdev_guid=7099321644620494833
Apr 10 07:32:26 mowa219-gjp4-8570p-freebsd kernel: ugen0.6: <Kingston DataTraveler 3.0> at usbus0 (disconnected)
Apr 10 07:32:26 mowa219-gjp4-8570p-freebsd kernel: umass2: at uhub7, port 4, addr 7 (disconnected)
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): got CAM status 0x444
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): fatal error, failed to attach to device
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2 at umass-sim2 bus 2 scbus5 target 0 lun 0
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2: <Kingston DataTraveler 3.0 >  s/n E0D55EA1C84FF390A9500FDA detached
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: (da2:umass-sim2:2:0:0): Periph destroyed
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: umass2: detached
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: ugen0.6: <Kingston DataTraveler 3.0> at usbus0
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: umass2 on uhub7
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: umass2: <Kingston DataTraveler 3.0, class 0/0, rev 3.10/0.01, addr 8> on usbus0
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2 at umass-sim2 bus 2 scbus5 target 0 lun 0
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2: <Kingston DataTraveler 3.0 > Removable Direct Access SPC-4 SCSI device
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2: Serial Number E0D55EA1C84FF390A9500FDA
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2: 400.000MB/s transfers
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2: 29510MB (60437492 512 byte sectors)
Apr 10 07:32:27 mowa219-gjp4-8570p-freebsd kernel: da2: quirks=0x2<NO_6_BYTE>
Apr 10 07:32:29 mowa219-gjp4-8570p-freebsd ZFS[35690]: vdev state changed, pool_guid=1913339710710793892 vdev_guid=7099321644620494833
Apr 10 07:32:34 mowa219-gjp4-8570p-freebsd kernel: pid 21587 (bsdisks), jid 0, uid 0: exited on signal 11 (core dumped)
Apr 10 07:32:37 mowa219-gjp4-8570p-freebsd devd[20839]: notify_clients: send() failed; dropping unresponsive client
% zpool status -x
all pools are healthy
% zpool iostat -v
                         capacity     operations     bandwidth
pool                   alloc   free   read  write   read  write
---------------------  -----  -----  -----  -----  -----  -----
Transcend               220G   244G      0      0    145    154
  gpt/Transcend         220G   244G      0      0    145    154
cache                      -      -      -      -      -      -
  gpt/cache-transcend  14.4G  49.8M      0      0    352     38
---------------------  -----  -----  -----  -----  -----  -----
august                  697G   215G     15     33   274K   998K
  ada0p3.eli            697G   215G     15     33   274K   998K
cache                      -      -      -      -      -      -
  gpt/duracell         6.52G  8.90G     26      0   597K  53.0K
  gpt/cache-august     21.1G  7.71G     61      0  1.15M  59.7K
---------------------  -----  -----  -----  -----  -----  -----
% lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:119 932G GPT                                               - -
  ada0p1         0:121 260M efi                                    gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                                 - -
  ada0p2         0:123  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p2.eli     2:60   16G freebsd-swap                                      - SWAP
  ada0p3         0:125 915G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p3.eli     0:131 915G -                                                 - -
  <FREE>         -:-   708K -                                                 - -
cd0              0:153  10M cd9660                    iso9660/ONEPLUS%20DRIVERS -
da0              0:147  15G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da0p1          0:148  15G freebsd-zfs                            gpt/duracell <ZFS>
da1              0:177 466G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da1p1          0:178 466G freebsd-zfs                           gpt/Transcend <ZFS>
da2              2:86   29G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da2p1          2:87   29G freebsd-zfs                        gpt/cache-august <ZFS>
da3              0:199  14G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da3p1          0:200  14G freebsd-zfs                     gpt/cache-transcend <ZFS>
%
 
Last edited:
Back
Top