l2arc degraded

User23 said:
You're right. It is compressing anyway. Even on 9.2 Release.
Click to expand...

There are a lot of changes to /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c between 10.1-RELEASE and 10.1-STABLE. I've checked out STABLE and will try to duplicate the problem there.
 
I have the same problem and I've posted it to the freebsd-stable mailinglist ( https://lists.freebsd.org/pipermail/freebsd-stable/2015-March/081907.html ) I've tried to find out what the problem could be, witnessed a few strange things. But I can't figure it out.

-update-

I forced my l2arc device to ashift 9. My pool is ashift 12. I removed the cache devices, set the vfs.zfs.max_auto_ashift sysctl(8) to 9 and created them again. I ran same tests for several days and it didn't get to the 16.0E free space situation. Test I did before (same tests by writing/reading a lot of data just from and to the pool) it often occurred within hours (with smaller l2arc sizes even few dozen of minutes)
 
Last edited by a moderator:
I had the same issue running 10.1-RELEASE. I removed and recreated the cache device while temporarily setting sysctl vfs.zfs.min_auto_ashift=9 and sysctl vfs.zfs.max_auto_ashift=9 to match the 512kb block size of the Intel SSD. I was unsure of which sysctl to modify, setting both didn't cause a problem. The cache filled and now again shows 16.0E free which seems harmless. L2ARC has given no bad checksums since recreated and is still reported as healthy by zfs-stats after 48 hours.


edit: and now 6 hours later I have to retract that. Bad checksums and degraded L2ARC once again. Frustrating.
 
Well since the last boot it's been 48 hours. The L2ARC filled and has remained healthy according to zfs-stats for over 24 hours. The workload at present is write heavy copying of incompressible files. Last time it was also write heavy but much more compressible data.

zfs-stats -a returns:
Code: 
------------------------------------------------------------------------
ZFS Subsystem Report Mon Mar 30 08:59:30 2015
------------------------------------------------------------------------

System Information:

Kernel Version: 1001000 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64

ZFS Storage pool Version: 5000
ZFS Filesystem Version: 5

FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015 root
8:59AM  up 2 days, 15:13, 1 user, load averages: 0.22, 0.24, 0.24

------------------------------------------------------------------------

System Memory:

0.01% 3.57 MiB Active, 0.22% 71.44 MiB Inact
81.22% 25.27 GiB Wired, 0.01% 3.24 MiB Cache
18.54% 5.77 GiB Free, 0.00% 0 Gap

Real Installed: 32.00 GiB
Real Available: 99.84% 31.95 GiB
Real Managed: 97.39% 31.11 GiB

Logical Total: 32.00 GiB
Logical Used: 81.75% 26.16 GiB
Logical Free: 18.25% 5.84 GiB

Kernel Memory: 333.86 MiB
Data: 92.28% 308.09 MiB
Text: 7.72% 25.78 MiB

Kernel Memory Map: 31.11 GiB
Size: 72.60% 22.59 GiB
Free: 27.40% 8.53 GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
Memory Throttle Count: 0

ARC Misc:
Deleted: 10.29m
Recycle Misses: 336.23k
Mutex Misses: 453
Evict Skips: 33.49m

ARC Size: 76.60% 23.07 GiB
Target Size: (Adaptive) 76.60% 23.07 GiB
Min Size (Hard Limit): 12.50% 3.76 GiB
Max Size (High Water): 8:1 30.11 GiB

ARC Size Breakdown:
Recently Used Cache Size: 98.86% 22.81 GiB
Frequently Used Cache Size: 1.14% 268.56 MiB

ARC Hash Breakdown:
Elements Max: 1.29m
Elements Current: 99.56% 1.28m
Collisions: 3.12m
Chain Max: 6
Chains: 161.42k

------------------------------------------------------------------------

ARC Efficiency: 7.34m
Cache Hit Ratio: 57.06% 4.19m
Cache Miss Ratio: 42.94% 3.15m
Actual Hit Ratio: 47.12% 3.46m

Data Demand Efficiency: 97.99% 621.19k
Data Prefetch Efficiency: 68.95% 109.89k

CACHE HITS BY CACHE LIST:
Anonymously Used: 15.72% 658.13k
Most Recently Used: 37.09% 1.55m
Most Frequently Used: 45.48% 1.90m
Most Recently Used Ghost: 0.87% 36.26k
Most Frequently Used Ghost: 0.84% 35.27k

CACHE HITS BY DATA TYPE:
Demand Data: 14.54% 608.69k
Prefetch Data: 1.81% 75.77k
Demand Metadata: 67.99% 2.85m
Prefetch Metadata: 15.66% 655.72k

CACHE MISSES BY DATA TYPE:
Demand Data: 0.40% 12.50k
Prefetch Data: 1.08% 34.12k
Demand Metadata: 95.73% 3.02m
Prefetch Metadata: 2.79% 87.74k

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
Passed Headroom: 7.19m
Tried Lock Failures: 157.41k
IO In Progress: 200
Low Memory Aborts: 24
Free on Write: 8.74k
Writes While Full: 77.95k
R/W Clashes: 1
Bad Checksums: 0
IO Errors: 0
SPA Mismatch: 199.45m

L2 ARC Size: (Adaptive) 112.05 GiB
Header Size: 0.19% 214.16 MiB

L2 ARC Evicts:
Lock Retries: 110
Upon Reading: 0

L2 ARC Breakdown: 3.15m
Hit Ratio: 0.67% 21.09k
Miss Ratio: 99.33% 3.13m
Feeds: 300.96k

L2 ARC Buffer:
Bytes Scanned: 144.97 TiB
Buffer Iterations: 300.96k
List Iterations: 18.65m
NULL List Iterations: 6.86m

L2 ARC Writes:
Writes Sent: (FAULTED) 160.59k
Done Ratio: 100.00% 160.59k
Error Ratio: 0.00% 0

------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency: 111.63m
Hit Ratio: 95.15% 106.21m
Miss Ratio: 4.85% 5.42m

Colinear: 5.42m
Hit Ratio: 0.01% 583
Miss Ratio: 99.99% 5.42m

Stride: 106.14m
Hit Ratio: 99.99% 106.14m
Miss Ratio: 0.01% 6.74k

DMU Misc:
Reclaim: 5.42m
Successes: 0.08% 4.44k
Failures: 99.92% 5.41m

Streams: 75.21k
+Resets: 2.96% 2.23k
-Resets: 97.04% 72.98k
Bogus: 0

------------------------------------------------------------------------

VDEV Cache Summary: 182.22k
Hit Ratio: 49.20% 89.66k
Miss Ratio: 41.23% 75.14k
Delegations: 9.56% 17.43k

------------------------------------------------------------------------

ZFS Tunables (sysctl):
kern.maxusers                           2380
vm.kmem_size                            33408720896
vm.kmem_size_scale                      1
vm.kmem_size_min                        0
vm.kmem_size_max                        1319413950874
vfs.zfs.arc_max                         32334979072
vfs.zfs.arc_min                         4041872384
vfs.zfs.arc_average_blocksize           8192
vfs.zfs.arc_meta_used                   1291794608
vfs.zfs.arc_meta_limit                  8083744768
vfs.zfs.l2arc_write_max                 8388608
vfs.zfs.l2arc_write_boost               8388608
vfs.zfs.l2arc_headroom                  2
vfs.zfs.l2arc_feed_secs                 1
vfs.zfs.l2arc_feed_min_ms               200
vfs.zfs.l2arc_noprefetch                0
vfs.zfs.l2arc_feed_again                1
vfs.zfs.l2arc_norw                      1
vfs.zfs.anon_size                       19696640
vfs.zfs.anon_metadata_lsize             0
vfs.zfs.anon_data_lsize                 0
vfs.zfs.mru_size                        23915642368
vfs.zfs.mru_metadata_lsize              195094016
vfs.zfs.mru_data_lsize                  23458551808
vfs.zfs.mru_ghost_size                  854230528
vfs.zfs.mru_ghost_metadata_lsize        479495680
vfs.zfs.mru_ghost_data_lsize            374734848
vfs.zfs.mfu_size                        6684672
vfs.zfs.mfu_metadata_lsize              1024
vfs.zfs.mfu_data_lsize                  68608
vfs.zfs.mfu_ghost_size                  14649464320
vfs.zfs.mfu_ghost_metadata_lsize        197552128
vfs.zfs.mfu_ghost_data_lsize            14451912192
vfs.zfs.l2c_only_size                   118107354112
vfs.zfs.dedup.prefetch                  1
vfs.zfs.nopwrite_enabled                1
vfs.zfs.mdcomp_disable                  0
vfs.zfs.dirty_data_max                  3430408601
vfs.zfs.dirty_data_max_max              4294967296
vfs.zfs.dirty_data_max_percent          10
vfs.zfs.dirty_data_sync                 67108864
vfs.zfs.delay_min_dirty_percent         60
vfs.zfs.delay_scale                     500000
vfs.zfs.prefetch_disable                0
vfs.zfs.zfetch.max_streams              8
vfs.zfs.zfetch.min_sec_reap             2
vfs.zfs.zfetch.block_cap                256
vfs.zfs.zfetch.array_rd_sz              1048576
vfs.zfs.top_maxinflight                 32
vfs.zfs.resilver_delay                  2
vfs.zfs.scrub_delay                     4
vfs.zfs.scan_idle                       50
vfs.zfs.scan_min_time_ms                1000
vfs.zfs.free_min_time_ms                1000
vfs.zfs.resilver_min_time_ms            3000
vfs.zfs.no_scrub_io                     0
vfs.zfs.no_scrub_prefetch               0
vfs.zfs.metaslab.gang_bang              131073
vfs.zfs.metaslab.fragmentation_threshold70
vfs.zfs.metaslab.debug_load             0
vfs.zfs.metaslab.debug_unload           0
vfs.zfs.metaslab.df_alloc_threshold     131072
vfs.zfs.metaslab.df_free_pct            4
vfs.zfs.metaslab.min_alloc_size         10485760
vfs.zfs.metaslab.load_pct               50
vfs.zfs.metaslab.unload_delay           8
vfs.zfs.metaslab.preload_limit          3
vfs.zfs.metaslab.preload_enabled        1
vfs.zfs.metaslab.fragmentation_factor_enabled1
vfs.zfs.metaslab.lba_weighting_enabled  1
vfs.zfs.metaslab.bias_enabled           1
vfs.zfs.condense_pct                    200
vfs.zfs.mg_noalloc_threshold            0
vfs.zfs.mg_fragmentation_threshold      85
vfs.zfs.check_hostid                    1
vfs.zfs.spa_load_verify_maxinflight     10000
vfs.zfs.spa_load_verify_metadata        1
vfs.zfs.spa_load_verify_data            1
vfs.zfs.recover                         0
vfs.zfs.deadman_synctime_ms             1000000
vfs.zfs.deadman_checktime_ms            5000
vfs.zfs.deadman_enabled                 1
vfs.zfs.spa_asize_inflation             24
vfs.zfs.txg.timeout                     5
vfs.zfs.vdev.cache.max                  16384
vfs.zfs.vdev.cache.size                 16777216
vfs.zfs.vdev.cache.bshift               16
vfs.zfs.vdev.trim_on_init               1
vfs.zfs.vdev.mirror.rotating_inc        0
vfs.zfs.vdev.mirror.rotating_seek_inc   5
vfs.zfs.vdev.mirror.rotating_seek_offset1048576
vfs.zfs.vdev.mirror.non_rotating_inc    0
vfs.zfs.vdev.mirror.non_rotating_seek_inc1
vfs.zfs.vdev.max_active                 1000
vfs.zfs.vdev.sync_read_min_active       10
vfs.zfs.vdev.sync_read_max_active       10
vfs.zfs.vdev.sync_write_min_active      10
vfs.zfs.vdev.sync_write_max_active      10
vfs.zfs.vdev.async_read_min_active      1
vfs.zfs.vdev.async_read_max_active      3
vfs.zfs.vdev.async_write_min_active     1
vfs.zfs.vdev.async_write_max_active     10
vfs.zfs.vdev.scrub_min_active           1
vfs.zfs.vdev.scrub_max_active           2
vfs.zfs.vdev.trim_min_active            1
vfs.zfs.vdev.trim_max_active            64
vfs.zfs.vdev.aggregation_limit          131072
vfs.zfs.vdev.read_gap_limit             32768
vfs.zfs.vdev.write_gap_limit            4096
vfs.zfs.vdev.bio_flush_disable          0
vfs.zfs.vdev.bio_delete_disable         0
vfs.zfs.vdev.trim_max_bytes             2147483648
vfs.zfs.vdev.trim_max_pending           64
vfs.zfs.max_auto_ashift                 12
vfs.zfs.min_auto_ashift                 12
vfs.zfs.zil_replay_disable              0
vfs.zfs.cache_flush_disable             0
vfs.zfs.zio.use_uma                     1
vfs.zfs.zio.exclude_metadata            0
vfs.zfs.sync_pass_deferred_free         2
vfs.zfs.sync_pass_dont_compress         5
vfs.zfs.sync_pass_rewrite               2
vfs.zfs.snapshot_list_prefetch          0
vfs.zfs.super_owner                     0
vfs.zfs.debug                           0
vfs.zfs.version.ioctl                   4
vfs.zfs.version.acl                     1
vfs.zfs.version.spa                     5000
vfs.zfs.version.zpl                     5
vfs.zfs.vol.mode                        1
vfs.zfs.trim.enabled                    1
vfs.zfs.trim.txg_delay                  32
vfs.zfs.trim.timeout                    30
vfs.zfs.trim.max_interval               1

------------------------------------------------------------------------
I expect the device will fault again but I don't have the knowledge to interpret much of that output. I'm concerned about the
Code: 
Writes Sent: (FAULTED)                160.59k
and the low memory aborts. The system has 32GB of RAM.

I'm also wondering if zfs-stats includes some Oracle specific code that could be reporting garbage on FreeBSD. I don't think zpool status listed the device as degraded last time but I'm waiting to verify that when/if it happens again.
 
I don't think that it is a zfs-stats bug. I am in exactly the same situation and running zpool iostat -v displays also the following:

Code: 
                        capacity     operations    bandwidth
pool                  alloc   free   read  write   read  write
....
cache                     -      -      -      -      -      -
  gpt/cache0           645G  16.0E     10      2   139K   138K
  gpt/cache1           646G  16.0E     12      0   153K      0
Notice the 16.0E value.
 
Code: 
                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
....
cache             -      -      -      -      -      -
  gpt/cache0   225G  16.0E      0     38  3.62K  4.65M

[FONT=verdana]Here a week passed during which the reported size remained close to the actually size of my device (112G). The zpool iostat -v began reporting 16.0E free while the size reported was equal to the actual size. The reported size grew after that very gradually at 113G, then 125G during which time zfs-stats still reported L2ARC as healthy and no checksum errors. After about 6TB of writing to the pool the capacity reported by zpool iostat -v quickly jumped to the present value and zfs-stats now reports a degraded L2ARC along with bad checksums again.[/FONT]

The faulted writes reported by zfs-stats which I mentioned resolved itself and hasn't recurred.

zpool status doesn't list the checksum errors or list the device as faulted.

Code: 
NAME          STATE     READ WRITE CKSUM
nas          ONLINE       0     0     0
raidz1-0     ONLINE       0     0     0
   gpt/hdd0  ONLINE       0     0     0
   gpt/hdd1  ONLINE       0     0     0
   gpt/hdd2  ONLINE       0     0     0
   gpt/hdd3  ONLINE       0     0     0
   gpt/hdd4  ONLINE       0     0     0
logs
mirror-1     ONLINE       0     0     0
   gpt/log0  ONLINE       0     0     0
   gpt/log1  ONLINE       0     0     0
cache
gpt/cache0   ONLINE       0     0     0
 
Same here, it does not list any checksum errors when zfs-stats report errors.

Code: 
NAME                  STATE     READ WRITE CKSUM
storage               ONLINE       0     0     0
  raidz2-0            ONLINE       0     0     0
    multipath/disk1   ONLINE       0     0     0
    multipath/disk2   ONLINE       0     0     0
    multipath/disk25  ONLINE       0     0     0
    multipath/disk4   ONLINE       0     0     0
    multipath/disk5   ONLINE       0     0     0
    multipath/disk6   ONLINE       0     0     0
  raidz2-1            ONLINE       0     0     0
    multipath/disk7   ONLINE       0     0     0
    multipath/disk8   ONLINE       0     0     0
    multipath/disk9   ONLINE       0     0     0
    multipath/disk26  ONLINE       0     0     0
    multipath/disk11  ONLINE       0     0     0
    multipath/disk12  ONLINE       0     0     0
  raidz2-2            ONLINE       0     0     0
    multipath/disk13  ONLINE       0     0     0
    multipath/disk14  ONLINE       0     0     0
    multipath/disk15  ONLINE       0     0     0
    multipath/disk16  ONLINE       0     0     0
    multipath/disk17  ONLINE       0     0     0
    multipath/disk18  ONLINE       0     0     0
  raidz2-3            ONLINE       0     0     0
    multipath/disk19  ONLINE       0     0     0
    multipath/disk20  ONLINE       0     0     0
    multipath/disk21  ONLINE       0     0     0
    multipath/disk22  ONLINE       0     0     0
    multipath/disk23  ONLINE       0     0     0
    multipath/disk24  ONLINE       0     0     0
logs
  mirror-4            ONLINE       0     0     0
    gpt/zil0          ONLINE       0     0     0
    gpt/zil1          ONLINE       0     0     0
cache
  gpt/cache0          ONLINE       0     0     0
  gpt/cache1          ONLINE       0     0     0
spares
  multipath/disk3     AVAIL   
  multipath/disk27    AVAIL   
  multipath/disk28    AVAIL   
  multipath/disk10    AVAIL
But the sizes are reported wrong:
Code: 
cache                     -      -      -      -      -      -
  gpt/cache0           670G  16.0E      9     15   616K  1.95M
  gpt/cache1           671G  16.0E      9     15   622K  1.95M
Their actual capacity is 500GB each.
 
When it happens again I'm going to check the contents of sysctl kstat.zfs.misc.arcstats.l2_cksum_bad and others that may be relevant.
 
I finished copying data to the new server. The workload is no longer write intensive and the L2ARC seems fine now. There may be a problem, but it could be weeks before it happens again here.
 
roper said:
I finished copying data to the new server. The workload is no longer write intensive and the L2ARC seems fine now. There may be a problem, but it could be weeks before it happens again here.
Click to expand...
What is the difference of the new server?
 
gkontos said:
What is the difference of the new server?
Click to expand...

My old NAS is ZFS but has no separate log or cache devices. My new NAS does have redundant log SSD's and a separate cache SSD. It has had this issue cropping up while 9TB of data was being written to it. That is done and now the cache isn't using the entire SSD. The allocated portion of the drive is very slowly increasing but at this rate it will be at least two weeks before it fills.
 
Hm... Sounds like my situation exactly. I have 2X Intel SSDs that are partitioned for the OS, ZIL and CACHE.

Code: 
34  1172123501  ada0  GPT  (559G)
          34           6        - free -  (3.0K)
          40        1024     1  freebsd-boot  (512K)
        1064    33554432     2  freebsd-swap  (16G) ----> SWAP (striped)
    33555496    20971520     3  freebsd-zfs  (10G) ----> OS (mirror)
    54527016    67108864     4  freebsd-zfs  (32G) ----> ZIL (mirror)
  121635880  1048576000     5  freebsd-zfs  (500G) ----> CACHE (striped)
  1170211880     1911655        - free -  (933M)
 
I was informed that there is a relevant patch that solves this. However, we need to try it. The problem is that my server is full production and it is difficult to even reboot it. Is there anyone else with this problem that has a testing machine?

Source: https://reviews.freebsd.org/D2764?download=true

EDIT: The link became broken. I have contacted the developer.
 
I still wait to see the error again on a
FreeBSD 10.1-STABLE #1 r281486: Mon Apr 13

Heavy read, moderate write. Primary and secondarycache = metadata

Code: 
 # uptime
 8:35AM  up 28 days, 21:12, 1 user, load averages: 0.22, 0.45, 0.87

# zpool iostat -v 1

              capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
poolname     11.7T  4.62T    515     74  34.6M  2.15M

  raidz1    4.56T   903G    187     31  12.4M   911K
    da6         -      -    135     10  2.09M   185K
    da8         -      -    130     10  2.08M   184K
    da7         -      -    135     10  2.09M   185K
    da9         -      -    130     10  2.09M   184K
    da10        -      -    135     10  2.09M   185K
    da11        -      -    130     10  2.09M   184K

  raidz1    7.14T  3.73T    327     43  22.1M  1.26M
    da0         -      -    242     12  3.72M   262K
    da1         -      -    234     12  3.70M   261K
    da2         -      -    242     12  3.72M   262K
    da3         -      -    234     12  3.70M   261K
    da4         -      -    242     12  3.72M   262K
    da5         -      -    234     12  3.70M   261K

cache           -      -      -      -      -      -
  ada3      37.1G  37.4G     70      0   301K  25.2K
  ada2      37.2G  37.3G     70      0   301K  25.2K
----------  -----  -----  -----  -----  -----  -----

---

L2 ARC Summary: (HEALTHY)
    Passed Headroom:            64.21m
    Tried Lock Failures:            3.27m
    IO In Progress:                1.76m
    Low Memory Aborts:            771
    Free on Write:                77.82k
    Writes While Full:            3.38k
    R/W Clashes:                17.82k
    Bad Checksums:                0
    IO Errors:                0
    SPA Mismatch:                1.77k

L2 ARC Size: (Adaptive)                235.83    GiB
    Header Size:            1.71%    4.04    GiB

L2 ARC Breakdown:                1.73b
    Hit Ratio:            20.25%    350.00m
    Miss Ratio:            79.75%    1.38b
    Feeds:                    2.47m

L2 ARC Buffer:
    Bytes Scanned:                92.65    TiB
    Buffer Iterations:            2.47m
    List Iterations:            157.64m
    NULL List Iterations:            68.98m

L2 ARC Writes:
    Writes Sent:            100.00%    1.07m

---

kstat.zfs.misc.arcstats.l2_compress_successes: 21766856
kstat.zfs.misc.arcstats.l2_compress_zeros: 0
kstat.zfs.misc.arcstats.l2_compress_failures: 19
 
I applied the patch and have been using it for a week on a secondary server - (replica and backups). I removed the original l2arc device and added an 8G l2arc partition for it for testing. Looks ok to me so far. I am open to suggestions for further testing.

Code: 
# zfs-stats -a | grep -v ^$
------------------------------------------------------------------------
ZFS Subsystem Report  Fri Jun 19 16:00:28 2015
------------------------------------------------------------------------
System Information:
  Kernel Version:  1001518 (osreldate)
  Hardware Platform:  amd64
  Processor Architecture:  amd64
  ZFS Storage pool Version:  5000
  ZFS Filesystem Version:  5
FreeBSD 10.1-STABLE #0 r284285M: Fri Jun 12 08:15:43 EDT 2015 root
4:00PM  up 7 days,  6:02, 1 user, load averages: 0.42, 0.71, 0.69
------------------------------------------------------------------------
System Memory:
  0.01%  13.75  MiB Active,  15.34%  19.12  GiB Inact
  83.62%  104.22  GiB Wired,  0.00%  0 Cache
  1.03%  1.28  GiB Free,  0.00%  4.00  KiB Gap
  Real Installed:  128.00  GiB
  Real Available:  99.95%  127.94  GiB
  Real Managed:  97.41%  124.63  GiB
  Logical Total:  128.00  GiB
  Logical Used:  84.06%  107.60  GiB
  Logical Free:  15.94%  20.40  GiB
Kernel Memory:  1.03  GiB
  Data:  97.33%  1021.76 MiB
  Text:  2.67%  28.07  MiB
Kernel Memory Map:  124.63  GiB
  Size:  76.82%  95.74  GiB
  Free:  23.18%  28.89  GiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
  Memory Throttle Count:  0
ARC Misc:
  Deleted:  196.90m
  Recycle Misses:  80.61m
  Mutex Misses:  33.12k
  Evict Skips:  1.08b
ARC Size:  78.49%  97.04  GiB
  Target Size: (Adaptive)  78.51%  97.06  GiB
  Min Size (Hard Limit):  12.50%  15.45  GiB
  Max Size (High Water):  8:1  123.63  GiB
ARC Size Breakdown:
  Recently Used Cache Size:  93.92%  91.16  GiB
  Frequently Used Cache Size:  6.08%  5.90  GiB
ARC Hash Breakdown:
  Elements Max:  11.31m
  Elements Current:  46.72%  5.28m
  Collisions:  69.55m
  Chain Max:  11
  Chains:  811.88k
------------------------------------------------------------------------
ARC Efficiency:  3.01b
  Cache Hit Ratio:  90.91%  2.74b
  Cache Miss Ratio:  9.09%  273.80m
  Actual Hit Ratio:  60.11%  1.81b
  Data Demand Efficiency:  93.11%  459.72m
  Data Prefetch Efficiency:  1.87%  138.03m
  CACHE HITS BY CACHE LIST:
  Anonymously Used:  33.07%  905.65m
  Most Recently Used:  14.21%  389.22m
  Most Frequently Used:  51.91%  1.42b
  Most Recently Used Ghost:  0.17%  4.72m
  Most Frequently Used Ghost:  0.64%  17.63m
  CACHE HITS BY DATA TYPE:
  Demand Data:  15.63%  428.05m
  Prefetch Data:  0.09%  2.58m
  Demand Metadata:  50.23%  1.38b
  Prefetch Metadata:  34.04%  932.43m
  CACHE MISSES BY DATA TYPE:
  Demand Data:  11.57%  31.67m
  Prefetch Data:  49.47%  135.45m
  Demand Metadata:  29.48%  80.71m
  Prefetch Metadata:  9.49%  25.97m
------------------------------------------------------------------------
L2 ARC Summary: (HEALTHY)
  Passed Headroom:  36.23m
  Tried Lock Failures:  391.87k
  IO In Progress:  101
  Low Memory Aborts:  3.57k
  Free on Write:  4.02m
  Writes While Full:  415.37k
  R/W Clashes:  11.44k
  Bad Checksums:  0
  IO Errors:  0
  SPA Mismatch:  11.72k
L2 ARC Size: (Adaptive)  10.60  GiB
  Header Size:  0.19%  20.17  MiB
L2 ARC Evicts:
  Lock Retries:  2.04k
  Upon Reading:  12
L2 ARC Breakdown:  273.80m
  Hit Ratio:  1.17%  3.19m
  Miss Ratio:  98.83%  270.61m
  Feeds:  869.27k
L2 ARC Buffer:
  Bytes Scanned:  58.90  TiB
  Buffer Iterations:  869.27k
  List Iterations:  46.63m
  NULL List Iterations:  4.11m
L2 ARC Writes:
  Writes Sent:  100.00% 624.93k
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)
DMU Efficiency:  1.79b
  Hit Ratio:  72.45%  1.30b
  Miss Ratio:  27.55%  492.68m
  Colinear:  492.68m
  Hit Ratio:  0.03%  123.77k
  Miss Ratio:  99.97%  492.56m
  Stride:  1.13b
  Hit Ratio:  100.00% 1.13b
  Miss Ratio:  0.00%  32.01k
DMU Misc:
  Reclaim:  492.56m
  Successes:  0.28%  1.38m
  Failures:  99.72%  491.18m
  Streams:  169.08m
  +Resets:  0.01%  18.56k
  -Resets:  99.99%  169.06m
  Bogus:  0
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
  kern.maxusers  8524
  vm.kmem_size  133821857792
  vm.kmem_size_scale  1
  vm.kmem_size_min  0
  vm.kmem_size_max  1319413950874
  vfs.zfs.trim.max_interval  1
  vfs.zfs.trim.timeout  30
  vfs.zfs.trim.txg_delay  32
  vfs.zfs.trim.enabled  1
  vfs.zfs.vol.unmap_enabled  1
  vfs.zfs.vol.mode  1
  vfs.zfs.version.zpl  5
  vfs.zfs.version.spa  5000
  vfs.zfs.version.acl  1
  vfs.zfs.version.ioctl  4
  vfs.zfs.debug  0
  vfs.zfs.super_owner  0
  vfs.zfs.sync_pass_rewrite  2
  vfs.zfs.sync_pass_dont_compress  5
  vfs.zfs.sync_pass_deferred_free  2
  vfs.zfs.zio.exclude_metadata  0
  vfs.zfs.zio.use_uma  1
  vfs.zfs.cache_flush_disable  0
  vfs.zfs.zil_replay_disable  0
  vfs.zfs.min_auto_ashift  9
  vfs.zfs.max_auto_ashift  13
  vfs.zfs.vdev.trim_max_pending  10000
  vfs.zfs.vdev.bio_delete_disable  0
  vfs.zfs.vdev.bio_flush_disable  0
  vfs.zfs.vdev.write_gap_limit  4096
  vfs.zfs.vdev.read_gap_limit  32768
  vfs.zfs.vdev.aggregation_limit  131072
  vfs.zfs.vdev.trim_max_active  64
  vfs.zfs.vdev.trim_min_active  1
  vfs.zfs.vdev.scrub_max_active  2
  vfs.zfs.vdev.scrub_min_active  1
  vfs.zfs.vdev.async_write_max_active  10
  vfs.zfs.vdev.async_write_min_active  1
  vfs.zfs.vdev.async_read_max_active  3
  vfs.zfs.vdev.async_read_min_active  1
  vfs.zfs.vdev.sync_write_max_active  10
  vfs.zfs.vdev.sync_write_min_active  10
  vfs.zfs.vdev.sync_read_max_active  10
  vfs.zfs.vdev.sync_read_min_active  10
  vfs.zfs.vdev.max_active  1000
  vfs.zfs.vdev.async_write_active_max_dirty_percent60
  vfs.zfs.vdev.async_write_active_min_dirty_percent30
  vfs.zfs.vdev.mirror.non_rotating_seek_inc1
  vfs.zfs.vdev.mirror.non_rotating_inc  0
  vfs.zfs.vdev.mirror.rotating_seek_offset1048576
  vfs.zfs.vdev.mirror.rotating_seek_inc  5
  vfs.zfs.vdev.mirror.rotating_inc  0
  vfs.zfs.vdev.trim_on_init  1
  vfs.zfs.vdev.cache.bshift  16
  vfs.zfs.vdev.cache.size  0
  vfs.zfs.vdev.cache.max  16384
  vfs.zfs.vdev.metaslabs_per_vdev  200
  vfs.zfs.txg.timeout  5
  vfs.zfs.space_map_blksz  4096
  vfs.zfs.spa_slop_shift  5
  vfs.zfs.spa_asize_inflation  24
  vfs.zfs.deadman_enabled  1
  vfs.zfs.deadman_checktime_ms  5000
  vfs.zfs.deadman_synctime_ms  1000000
  vfs.zfs.recover  0
  vfs.zfs.spa_load_verify_data  1
  vfs.zfs.spa_load_verify_metadata  1
  vfs.zfs.spa_load_verify_maxinflight  10000
  vfs.zfs.check_hostid  1
  vfs.zfs.mg_fragmentation_threshold  85
  vfs.zfs.mg_noalloc_threshold  0
  vfs.zfs.condense_pct  200
  vfs.zfs.metaslab.bias_enabled  1
  vfs.zfs.metaslab.lba_weighting_enabled  1
  vfs.zfs.metaslab.fragmentation_factor_enabled1
  vfs.zfs.metaslab.preload_enabled  1
  vfs.zfs.metaslab.preload_limit  3
  vfs.zfs.metaslab.unload_delay  8
  vfs.zfs.metaslab.load_pct  50
  vfs.zfs.metaslab.min_alloc_size  33554432
  vfs.zfs.metaslab.df_free_pct  4
  vfs.zfs.metaslab.df_alloc_threshold  131072
  vfs.zfs.metaslab.debug_unload  0
  vfs.zfs.metaslab.debug_load  0
  vfs.zfs.metaslab.fragmentation_threshold70
  vfs.zfs.metaslab.gang_bang  16777217
  vfs.zfs.free_max_blocks  -1
  vfs.zfs.no_scrub_prefetch  0
  vfs.zfs.no_scrub_io  0
  vfs.zfs.resilver_min_time_ms  3000
  vfs.zfs.free_min_time_ms  1000
  vfs.zfs.scan_min_time_ms  1000
  vfs.zfs.scan_idle  50
  vfs.zfs.scrub_delay  4
  vfs.zfs.resilver_delay  2
  vfs.zfs.top_maxinflight  32
  vfs.zfs.zfetch.array_rd_sz  1048576
  vfs.zfs.zfetch.block_cap  256
  vfs.zfs.zfetch.min_sec_reap  2
  vfs.zfs.zfetch.max_streams  8
  vfs.zfs.prefetch_disable  0
  vfs.zfs.delay_scale  500000
  vfs.zfs.delay_min_dirty_percent  60
  vfs.zfs.dirty_data_sync  67108864
  vfs.zfs.dirty_data_max_percent  10
  vfs.zfs.dirty_data_max_max  4294967296
  vfs.zfs.dirty_data_max  4294967296
  vfs.zfs.max_recordsize  1048576
  vfs.zfs.mdcomp_disable  0
  vfs.zfs.nopwrite_enabled  1
  vfs.zfs.dedup.prefetch  1
  vfs.zfs.l2c_only_size  10929157632
  vfs.zfs.mfu_ghost_data_lsize  61428802560
  vfs.zfs.mfu_ghost_metadata_lsize  31655070720
  vfs.zfs.mfu_ghost_size  93083873280
  vfs.zfs.mfu_data_lsize  7424311296
  vfs.zfs.mfu_metadata_lsize  126976
  vfs.zfs.mfu_size  8044302336
  vfs.zfs.mru_ghost_data_lsize  2597035008
  vfs.zfs.mru_ghost_metadata_lsize  5703695360
  vfs.zfs.mru_ghost_size  8300730368
  vfs.zfs.mru_data_lsize  81873646080
  vfs.zfs.mru_metadata_lsize  11627892224
  vfs.zfs.mru_size  94030831616
  vfs.zfs.anon_data_lsize  0
  vfs.zfs.anon_metadata_lsize  0
  vfs.zfs.anon_size  7489536
  vfs.zfs.l2arc_norw  1
  vfs.zfs.l2arc_feed_again  1
  vfs.zfs.l2arc_noprefetch  1
  vfs.zfs.l2arc_feed_min_ms  200
  vfs.zfs.l2arc_feed_secs  1
  vfs.zfs.l2arc_headroom  2
  vfs.zfs.l2arc_write_boost  8388608
  vfs.zfs.l2arc_write_max  8388608
  vfs.zfs.arc_meta_limit  33187028992
  vfs.zfs.arc_free_target  226534
  vfs.zfs.arc_shrink_shift  5
  vfs.zfs.arc_average_blocksize  8192
  vfs.zfs.arc_min  16593514496
  vfs.zfs.arc_max  132748115968
------------------------------------------------------------------------
 
I think I'm affected by this issue, but I'm running FreeBSD 10.2-RELEASE.

Code: 
L2 ARC Summary: (DEGRADED)
  Passed Headroom:  267.76m
  Tried Lock Failures:  333.09k
  IO In Progress:  1.35k
  Low Memory Aborts:  50
  Free on Write:  11.63k
  Writes While Full:  14.12k
  R/W Clashes:  83
  Bad Checksums:  6.06m
  IO Errors:  906.42k
  SPA Mismatch:  4.21b

L2 ARC Size: (Adaptive)  167.86  GiB
  Header Size:  0.19%  324.11  MiB

L2 ARC Evicts:
  Lock Retries:  26
  Upon Reading:  0

L2 ARC Breakdown:  74.65m
  Hit Ratio:  12.38%  9.24m
  Miss Ratio:  87.62%  65.41m
  Feeds:  4.35m

L2 ARC Buffer:
  Bytes Scanned:  279.66  TiB
  Buffer Iterations:  4.35m
  List Iterations:  277.98m
  NULL List Iterations:  1.51m

L2 ARC Writes:
  Writes Sent:  100.00% 95.42k

Apparently my L2ARC size is 167.86 GB, but that's impossible. The partition where this L2ARC is on, is only 30 GB.

I also have quite low hit ratios and high IO Errors and Bad Checksums.
 
The commits did not make it into 10.2-RELEASE, you need to be running 10.2-STABLE from at least about 3 or 4 weeks ago. Good Luck!
 
Thank you!

We're currently testing FreeBSD 10.2-STABLE #0 r291769, will let you know how it goes.

Looking good so far but it's not been very long yet!
 
You must log in or register to reply here.
Back
Top