ZFS slow write

User23 · Jul 8, 2010

Code:

ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <WDC WD3000HLFS-01G6U0 04.04V01> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 286168MB (586072368 512 byte sectors: 16H 63S/T 16383C)

Code:

# zpool status
  pool: home
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        home        ONLINE       0     0     0
          ada0      ONLINE       0     0     0

errors: No known data errors

With sysctl vfs.zfs.txg.write_limit_override=0 i got:

Code:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
                16G    84  99 95612  28 50712  15   239  99 115950  17 177.3  19
Latency               436ms   12484ms    6157ms   45120us    2803ms    1309ms
Version  1.96       ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20065  97 +++++ +++ 13700  98 16218  97 +++++ +++ 14248  98
Latency             16900us     224us     361us   39299us     123us     622us

and with sysctl vfs.zfs.txg.write_limit_override=1048576000 i got:

Code:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
                16G    96  99 92547  28 48933  14   223  92 117699  17 180.9  16
Latency               441ms    3122ms    4999ms    1009ms     550ms    1091ms
Version  1.96       ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 18500  97 +++++ +++ 17332  98 14217  97 +++++ +++ 17178  98
Latency             21881us     153us     216us   43772us      45us      94us

da1 · Feb 11, 2011

This post was very useful for me so I will post my conf:

8.1 amd64 (ZFSonRooT) with 2x 500GB WD RE3 (mirror) and 2x 1.5T WD Caviar Black (mirror).

/boot/loader.conf

Code:

vfs.root.mountfrom="zfs:zroot"
zfs_load="YES"
geom_mirror_load="YES"
linux_load="YES"
sound_load="YES"
snd_ich_load="YES"
#nvidia_load="YES"
accf_http_load="YES"
#vboxdrv_load="YES"
loader_logo="beastie"
coretemp_load="YES"
geom_label_load="YES"
atapicam_load="YES"                     # allows ATAPI devices to be accessed through the SCSI subsystem, cam(4)
ahci_load="YES"                         # Allow S-ATA extra features (NCQ,etc)
if_tap_load="YES"

############# ZFS tunnables
## for AHCI
vfs.zfs.vdev.min_pending=4              #default=4
vfs.zfs.vdev.max_pending=8              #default = 35
## NO AHCI
#vfs.zfs.vdev.min_pending=4             #default=4
#vfs.zfs.vdev.max_pending=8             #default = 35

# Increase vm.kmem_size to allow for ZFS ARC to utilise more memory.
vm.kmem_size="2048M"
vm.kmem_size_max="2048M"
vfs.zfs.arc_max="2048M"

# Disable ZFS prefetching (we will not disable it because we have 6GB of RAM)
# http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html
# vfs.zfs.prefetch_disable="0"

# Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
vfs.zfs.txg.timeout="5"

# Target seconds to sync a txg
vfs.zfs.txg.synctime="1"

bonnie:

Code:

[root@mainserver ~]# bonnie
File './Bonnie.2543', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
          100 251650 96.8 83025 20.1 119967 30.6 225507 99.0 905097 96.1 160539.4 175.5

and bonnie++:

Code:

[root@mainserver ~]# bonn
bonnie   bonnie++
[root@mainserver ~]# bonnie++ -s 16000 -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
mainserver   16000M   104  99 76228  21 48577  13   238  99 172447  20 167.8   3
Latency               115ms    1337ms    2428ms   67607us     596ms     840ms
Version  1.96       ------Sequential Create------ --------Random Create--------
mainserver          -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 11034  95  6396  97  2048  99 20678  89 +++++ +++ 18925  97
Latency             21623us   11981us   15821us   14862us    4989us     102us
1.96,1.96,mainserver,1,1297439902,16000M,,104,99,76228,21,48577,13,238,99,172447,20,167.8,3,16,,,,,11034,95,6396,97,2048,99,20678,89,+++++,+++,18925,97,115ms,1337ms,2428ms,67607us,596ms,840ms,21623us,11981us,15821us,14862us,4989us,102us

Code:

vfs.zfs.txg.write_limit_override: 268435456

ZFS slow write

User23

da1