ZFS tuning for 4 GB system

A

aupanner

I am running 9.2/amd64 on an old box with 4 GB of RAM. I have mirrored ZFS drives. My system would run fine after a reboot but within a day or so I would experience performance problems: SSH sessions would occasionally hang for several seconds (concurrent sessions would be fine); Samba would hang or take forever to load a directory.

After some trial and error I've found that the problems stem from memory starvation caused by the default ARC settings. The following settings are chosen by the system:

Code: 
real memory  = 4294967296 (4096 MB)
avail memory = 4082089984 (3892 MB)

vm.kmem_size_max: 329853485875 (307 GB)
vm.kmem_size: 4111114240 (3921 MB)
vfs.zfs.arc_min: 379671552 (362 MB)
vfs.zfs.arc_max: 3037372416 (2897 MB)

Making the following changes to /boot/loader.conf has removed the performance issues, but may not be optimal settings.

Code: 
vm.kmem_size="3072M"
vfs.zfs.arc_min="128M"
vfs.zfs.arc_max="2048M"

I'm new to ZFS so I'm still making sense from the output of zfs-stats. It wasn't obvious to me that memory was the issue, but it seems that if your free system memory drops below 5% or the logical free memory drops below 10% you may experience the issues I was seeing.

I'll follow up with the results of further experimentation, but the above settings should help anyone with a 4 GB system.
 
FreeBSD 9.2-RELEASE with 4GB RAM should not need any ZFS tuning. I am also running a very old box for file server with 4GB of RAM.

Make sure you don't use dedup and your hardware is in good shape.

If you want to reduce the ARC you can just set:

Code: 
vfs.zfs.arc_max="2G"

In your /boot/loader.conf. All the other values should be removed.
 
gkontos said:
FreeBSD 9.2-RELEASE with 4GB RAM should not need any ZFS tuning. I am also running a very old box for file server with 4GB of RAM.
Click to expand...

Interesting. Could you post your values for vfs.zfs.arc_max and the memory usage section from zfs-stats -M? Are you running other memory-eaters like Samba or Spamassassin?
 
I am running FreeBSD 10-BETA2 with no ZFS tuning.

The box is acting as:

  • Router/Firewall
  • 1 Site2Site IPSEC VPN
  • DHCPD/DHCPD6 server
  • Internal DNS server
  • Small storage (Root on ZFS with Mirror 2X220GB and another pool Mirror 2X1TB)
  • Samba for 3 users.
  • Apache, PHP5 and mysql-server (local development tests)

Code: 
------------------------------------------------------------------------
ZFS Subsystem Report                            Tue Nov  5 14:55:23 2013
------------------------------------------------------------------------

System Information:

        Kernel Version:                         1000501 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               5000
        ZFS Filesystem Version:                 5

FreeBSD 10.0-BETA2 #1 r257581M: Sun Nov 3 23:14:28 EET 2013 root
 2:55PM  up 1 day, 14:53, 3 users, load averages: 0.30, 0.21, 0.17

------------------------------------------------------------------------

System Memory:

        2.76%   108.97  MiB Active,     7.01%   277.25  MiB Inact
        86.48%  3.34    GiB Wired,      0.11%   4.33    MiB Cache
        3.60%   142.29  MiB Free,       0.04%   1.45    MiB Gap

        Real Installed:                         4.00    GiB
        Real Available:                 99.39%  3.98    GiB
        Real Managed:                   97.11%  3.86    GiB

        Logical Total:                          4.00    GiB
        Logical Used:                   89.65%  3.59    GiB
        Logical Free:                   10.35%  423.88  MiB

Kernel Memory:                                  2.77    GiB
        Data:                           99.13%  2.74    GiB
        Text:                           0.87%   24.74   MiB

Kernel Memory Map:                              3.86    GiB
        Size:                           65.37%  2.52    GiB
        Free:                           34.63%  1.34    GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                17.88k
        Recycle Misses:                         2.27k
        Mutex Misses:                           0
        Evict Skips:                            398

ARC Size:                               100.00% 2.86    GiB
        Target Size: (Adaptive)         100.00% 2.86    GiB
        Min Size (Hard Limit):          12.50%  366.15  MiB
        Max Size (High Water):          8:1     2.86    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       67.72%  1.94    GiB
        Frequently Used Cache Size:     32.28%  945.48  MiB

ARC Hash Breakdown:
        Elements Max:                           207.07k
        Elements Current:               100.00% 207.07k
        Collisions:                             1.04m
        Chain Max:                              15
        Chains:                                 52.51k

------------------------------------------------------------------------

ARC Efficiency:                                 5.06m
        Cache Hit Ratio:                99.46%  5.03m
        Cache Miss Ratio:               0.54%   27.56k
        Actual Hit Ratio:               99.45%  5.03m

        Data Demand Efficiency:         99.86%  3.18m
        Data Prefetch Efficiency:       0.00%   9

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           16.88%  849.40k
          Most Frequently Used:         83.12%  4.18m
          Most Recently Used Ghost:     0.00%   137
          Most Frequently Used Ghost:   0.05%   2.73k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  63.20%  3.18m
          Prefetch Data:                0.00%   0
          Demand Metadata:              36.80%  1.85m
          Prefetch Metadata:            0.00%   29

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  16.23%  4.47k
          Prefetch Data:                0.03%   9
          Demand Metadata:              83.36%  22.97k
          Prefetch Metadata:            0.37%   103

------------------------------------------------------------------------

Part of dmesg:

Code: 
FreeBSD 10.0-BETA2 #1 r257581M: Sun Nov  3 23:14:28 EET 2013
    root@hp.aicom.loc:/usr/obj/usr/src/sys/IPSEC amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Pentium(R) D CPU 3.20GHz (3200.18-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0xf64  Family = 0xf  Model = 0x6  Stepping = 4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe4bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,CNXT-ID,CX16,xTPR,PDCM>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 4294967296 (4096 MB)
avail memory = 4116176896 (3925 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <HP     OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 
@gkontos is correct. 9.2-RELEASE runs ZFS fine on a 4GB system with no tuning. It turns out what I was observing was a network issue, probably a NIC that is doing something screwy under a combination of load and time. I isolated the behavior from two directions:

  1. Samba shared a UFS drive. This had the same issue as the ZFS-shared drive, ergo the problem was not specifically ZFS.
  2. Performed testing from the console. Console displayed none of the issues seen over Samba or sshd.
netstat showed a constantly large Send-Q on em0 and throughput would stutter. netstat -I em0 1 showed bursts of input errors during the stalls.
Code: 
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port 0xb000-0xb03f mem 0xd3020000-0xd303ffff,0xd3000000-0xd301ffff irq 16 at device 6.0 on pci5

I swapped my internal and external NICs, removed all ZFS "tuning", and the system has been happy and stable for two days now. My guess is that em0 was eventually choking under GB loads. It never logged any errors. I'll have to reswap the card and inspect tcpdump output to see if it is dropping ACKs or winnowing the window down to 0 or ???

The default values for this card differ from those indicated in the em(4) manual page.
Code: 
hw.em.eee_setting: 1
hw.em.rx_process_limit: 100
hw.em.enable_msix: 1
hw.em.sbp: 0
hw.em.smart_pwr_down: 0
hw.em.txd: 1024 (man page claims 256)
hw.em.rxd: 1024 (man page claims 256)
hw.em.rx_abs_int_delay: 66 (man page doesn't specify)
hw.em.tx_abs_int_delay: 66 (man page doesn't specify)
hw.em.rx_int_delay: 0
hw.em.tx_int_delay: 66 (man page claims 64)

Code: 
FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013
    root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Dual Core AMD Opteron(tm) Processor 185 (2613.44-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f32  Family = 0xf  Model = 0x23  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow!>
  AMD Features2=0x3<LAHF,CMP>
real memory  = 4294967296 (4096 MB)
avail memory = 4080349184 (3891 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Nvidia AWRDACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 
Last edited by a moderator:
This is the output of # sysctl hw.em on my system. They seem to be identical to yours. I have no such trouble with mine. Therefore, I think your trouble may be unrelated to these settings.
Code: 
sysctl hw.em
hw.em.eee_setting: 1
hw.em.rx_process_limit: 100
hw.em.enable_msix: 1
hw.em.sbp: 0
hw.em.smart_pwr_down: 0
hw.em.txd: 1024
hw.em.rxd: 1024
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.rx_int_delay: 0
hw.em.tx_int_delay: 66

Edit: I use FreeBSD 9.2 AMD64, GENERIC kernel.
 
You must log in or register to reply here.
Back
Top