Software RAID 5 & FreeBSD

One other thought- even though I mentioned that the ZFS array has three 250GB Seagate drives, the models aren't identical in terms of number of sectors. All three models have different model numbers as follows:

WD2500KS-00M 232.9GB
WD2500YD-01N 233.8GB
WD2500YS-01S 233.8GB

The last two of these contain identical sectors but the first doesn't. I had to use the -f parameter to force creation of the ZFS array due to this fact.

Would this be a very good reason why I'm experiencing the crashes? The array wasn't close to being 25% full at the time the copying job caused the array to stop functioning earlier today.

Here's the latest zfs-stats output prior to this morning's crash:

Code: 
aries# zfs-stats -a

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Sep 13 11:57:30 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      i386
        Processor Architecture:                 i386

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE-p3 #0: Tue Jun 12 01:47:53 UTC 2012 root
11:57AM  up 21:21, 2 users, load averages: 3.99, 2.17, 0.93

------------------------------------------------------------------------

System Memory:

        0.56%   20.19   MiB Active,     5.58%   202.19  MiB Inact
        11.97%  433.74  MiB Wired,      0.00%   184.00  KiB Cache
        81.88%  2.90    GiB Free,       0.02%   568.00  KiB Gap

        Real Installed:                         4.00    GiB
        Real Available:                 90.09%  3.60    GiB
        Real Managed:                   98.22%  3.54    GiB

        Logical Total:                          4.00    GiB
        Logical Used:                   22.61%  925.91  MiB
        Logical Free:                   77.39%  3.10    GiB

Kernel Memory:                                  245.58  MiB
        Data:                           93.40%  229.37  MiB
        Text:                           6.60%   16.21   MiB

Kernel Memory Map:                              466.16  MiB
        Size:                           75.98%  354.21  MiB
        Free:                           24.02%  111.95  MiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                46.12k
        Recycle Misses:                         3.31k
        Mutex Misses:                           1
        Evict Skips:                            3.42k

ARC Size:                               91.46%  234.13  MiB
        Target Size: (Adaptive)         91.43%  234.06  MiB
        Min Size (Hard Limit):          12.50%  32.00   MiB
        Max Size (High Water):          8:1     256.00  MiB

ARC Size Breakdown:
        Recently Used Cache Size:       80.88%  189.36  MiB
        Frequently Used Cache Size:     19.12%  44.77   MiB

ARC Hash Breakdown:
        Elements Max:                           9.59k
        Elements Current:               100.00% 9.59k
        Collisions:                             6.94k
        Chain Max:                              4
        Chains:                                 658

------------------------------------------------------------------------

ARC Efficiency:                                 157.52k
        Cache Hit Ratio:                92.50%  145.71k
        Cache Miss Ratio:               7.50%   11.82k
        Actual Hit Ratio:               92.49%  145.70k

        Data Demand Efficiency:         99.73%  8.44k

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           39.62%  57.72k
          Most Frequently Used:         60.38%  87.97k
          Most Recently Used Ghost:     3.54%   5.15k
          Most Frequently Used Ghost:   0.35%   512

        CACHE HITS BY DATA TYPE:
          Demand Data:                  5.78%   8.42k
          Prefetch Data:                0.00%   0
          Demand Metadata:              94.20%  137.26k
          Prefetch Metadata:            0.02%   30

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.19%   23
          Prefetch Data:                0.00%   0
          Demand Metadata:              99.66%  11.78k
          Prefetch Metadata:            0.14%   17

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------


------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            536870912
        vm.kmem_size_scale                      3
        vm.kmem_size_min                        0
        vm.kmem_size_max                        1073741824
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            97386496
        vfs.zfs.mfu_ghost_metadata_lsize        11758080
        vfs.zfs.mfu_ghost_size                  109144576
        vfs.zfs.mfu_data_lsize                  0
        vfs.zfs.mfu_metadata_lsize              0
        vfs.zfs.mfu_size                        7155712
        vfs.zfs.mru_ghost_data_lsize            95168000
        vfs.zfs.mru_ghost_metadata_lsize        10958848
        vfs.zfs.mru_ghost_size                  106126848
        vfs.zfs.mru_data_lsize                  137367552
        vfs.zfs.mru_metadata_lsize              950272
        vfs.zfs.mru_size                        155123712
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       39896064
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  67108864
        vfs.zfs.arc_meta_used                   68315340
        vfs.zfs.arc_min                         33554432
        vfs.zfs.arc_max                         268435456
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            11608596480
        vfs.zfs.write_limit_max                 483691520
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                1
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------

aries#

See anything amiss in there?
 
dougs said:
/etc/sysctl.conf:
Code: 
aries# less /etc/sysctl.conf
# $FreeBSD: release/9.0.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.maxvnodes=400000
vfs.zfs.write_limit_override=268435456

/boot/loader.conf:
Code: 
aries# less /boot/loader.conf
vfs.zfs.prefetch_disable="1"
vm.kmem_size="512M"
vm.kmem_size_max="1G"
vfs.zfs.arc_max="256M"
#vfs.zfs.vdev.cache.size="10M"
zfs_load="YES"
#vfs.zfs.txg.timeout="5"
Click to expand...

First off, comment out the stuff in /etc/sysctl.conf, you want to start with as few changes to settings as possible. Second, change the vm.kmem_size_max in /boot/loader.conf to 512M:

Code: 
vm.kmem_size_max="512M"

If you need more kmem you must recompile the kernel with increased KVA_PAGES. I'm pretty sure the crashes you're now getting are from incorrect vm.kmem_size_max setting.
 
kpa-

I commented out all of the contents of /etc/sysctl.conf and changed vm.kmem_size_max from 1G to 512M. I copied over the same data set (~17.2GB) without incident. Now we wait for tomorrow to see if it crashes.
 
/boot/loader.conf:
Code: 
# ZFS settings
#vm.kmem_size="1G"
vm.kmem_size_max=1G
vm.kmem_size_scale=1
vfs.zfs.arc_max="768M"
vfs.zfs.arc_min="128M"
vfs.zfs.prefetch_disable="1"
vfs.zfs.zil_disable="0"
#vfs.zfs.vdev.max_pending="4"
#vfs.zfs.txg.timeout="7"

$ zpool status
Code: 
pool: newpool
 state: ONLINE
  scan: resilvered 330G in 2h4m with 0 errors on Sat Aug 18 23:29:52 2012
config:

        NAME           STATE     READ WRITE CKSUM
        newpool        ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            gpt/disk3  ONLINE       0     0     0
            gpt/disk1  ONLINE       0     0     0
          mirror-1     ONLINE       0     0     0
            gpt/disk4  ONLINE       0     0     0
            gpt/disk2  ONLINE       0     0     0

errors: No known data errors
$ zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
newpool 1.36T 640G 752G 45% 1.00x ONLINE -[/code]

$ uname -a
Code: 
FreeBSD rogue.ashesofthe.net 9.0-STABLE FreeBSD 9.0-STABLE #0 r236880: Mon Jun 11 05:49:06 PDT 2012     root@rogue.ashesofthe.net:/usr/obj/usr/src-9/sys/ROGUE90  i386

Runs KDE4, Samba, NFS, basically 24/7. Torrents downloaded via ktorrent. Videos played via XBMC on an HTPC over SMB, or via Dragon Player on a laptop over NFS. 2 other laptops access shared folders via SMB. Also has a bridge interface configured (one NIC connected to downstairs router that all clients use for Internet access, one NIC connected to upstairs router that only this box uses for Internet access). The split network allows me to download torrents (which occasionally causes Shaw to throttle all traffic to that public IP) without affecting everyone else's Internet connection (via a separate public IP).

The only time the box is rebooted is when the USB ports act up, and I can no longer access the keyboard/mouse. Something to do with the USB-based KVM (wife's laptop is also connected to it so she can use the big monitor).

RAM usage sits at about 95% all the time. ARC is generally always full. No longer have an L2ARC, as the USB stick I was using was slower than the pool.

Anything else you'd like to know about the setup?
 
This morning I copied over the same large 17.2GB data set and it completed without crashing. I went ahead and deleted data from the array and recopied over. Still no problems. I'm almost ready to declare this issue solved.

However, I have this 4GB of RAM on this machine and I'm debating whether I should compile a new kernel with a higher KVA_PAGES value and thus be able to increase the vfs.kva_size_max value to a higher value like 1GB. This server will also run foswiki/apache on top of samba and that's it. Currently the server shows 2,993MB free memory. Any suggestions/thoughts?

kpa- thanks for the config information. What was the value you assigned to KVA_PAGES in your kernel? Your server has 2GB of RAM, correct? Is there a formula one should follow to properly size the KVA_PAGES option and also the configuration in both /boot/loader.conf and /etc/sysctl.conf (if applicable)?

Are there any feedback in regards to the mention of the fact that not all disks in the ZFS array are sized identically. See this:

Code: 
ada1: <WDC WD2500YD-01NVB1 10.02E01> ATA-7 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 239372MB (490234752 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad5
ada2 at ata3 bus 0 scbus2 target 0 lun 0
ada2: <WDC WD2500KS-00MJB0 02.01C03> ATA-7 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: 238474MB (488395055 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad6
ada3 at ata3 bus 0 scbus2 target 1 lun 0
ada3: <WDC WD2500YS-01SHB0 20.06C03> ATA-7 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 239372MB (490234752 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad7

As you can see /dev/ada2 has slight less sectors than /dev/ada1 and /dev/ada3. Will I run into trouble eventually as more and more data is fed to the ZFS array and approaching the physical hard disk limits?

~Doug
 
You must log in or register to reply here.
Back
Top