FreeBSD 8.1-RELEASE NFS writes slowdown with 'sync' mount option

[sidh]

Greetings,

I've recently move a FreeBSD 7.2 NFS/ZFS(mirror) server to FreeBSD 8.1-RELEASE NFS/ZFS(raidz2).

As I reused the rc.conf options and exports file from the 7.2 version, and everything ran fine at startup, i didn't go further in the configuration.

From a Linux client machine, I kept the same mount options :

chimay:/mediatheque  /home/sidh/mediatheque  nfs  tcp,noauto,rsize=32768,wsize=32768,timeo=14,intr,nolock,bg,[B][U]sync[/U][/B]

and when I try to write a 5GB file I get :

dd if=/dev/zero of=/home/sidh/mediatheque/file5GB bs=1k count=5M
[...]
5368709120 bytes (5.4 GB) copied, 2595.15 s, 2.1 MB/s

2.1 MB/s !!!!

If I just unset the 'sync' option from the linux client mount point :

dd if=/dev/zero of=/home/sidh/mediatheque/file5GB bs=1k count=5M
[...]
5368709120 bytes (5.4 GB) copied, 194,995 s, 27.5 MB/s

I had about 40 MB/s when It was NFS/mirror ZFS (with sync activated), so due to raidz2 way of working, I guess decreasing to a little 30 MB/s is not very surprising...

But I know that this sync option preserve from big problems (freezes, and so on) and I would really like to keep it with better NFS writes speed of course (all machines are on a Gigabit Ethernet switch/NIC).

As the Release Note of 8.1 mentionned improvements with NFS, I would like to know if there is something to add on the server/client/both options ?

Regards,

P.S. here are my exports file :

/mediatheque -maproot=sidh -alldirs -network 172.31.255.0 -mask 255.255.255.0

and the rc.conf 's NFS parameters :

rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_client_enable="YES"

 

[phoenix]

If you are going to use NFS with sync, you *NEED* to install an SSD-based separate log device (and you have to create a mirror log device). Otherwise your ZFS performance will go through the floor, as you are forcing it to write out a transaction for every NFS request.

Either install a pair of SSDs as a mirror log device, or disable sync on your NFS mounts.

Search the freebsd-fs and zfs-discuss mailing list archives for several long threads every month on this subject. It's the most common FAQ.

 

[sidh]

Thanks for your advices

As I don't have neither available sata slots nor sata raid/jbod controller, I 'll go without 'sync' mount option :\

By the way I tried the latest EON distro (EON 64-bit x86 Samba ISO image version 0.60.0 based on snv_130), to see if I could have better performance in writes speed.

I didn't zpool upgrade in order to be able to import the pool with FreeBSD 8.1 again. The writes speed with 'sync' mount option is quite the same as FreeBSD 8.1 (14 MB/s). But the write speeds without 'sync' mount options seems to perform better : 45 MB/sec (Vs 30 MB/sec for FreeBSD 8.1). At the time of the test I didn't know about bonnie but I did the test with copying a large file (8GB) and didn't use dd.

I wonder what "server-side" NFS options EON has set-up ...