FreeBSD slow NFS read speed on FreeBSD 9.1

belon_cfy · #1 April 25th, 2012, 16:50

5th Feb 2013
Previously I m managed to solve it by adding a vlan on top of the em interface(no issue on igb interface), but the same solution doesn't work anymore on FreeBSD 9.1.

I have verified it by switching back to FreeBSD 9.0 and it works fine again, the server able to saturate the 1Gbps link. After that I reinstall the same server to FreeBSD 9.1 and the problem came back again, it only able to push below 210Mbps max.

Is there any driver update causing bad performance on FreeBSD 9.1 ?

By the way, below is my test environment on ESXI5 with NFS mount.
- Running a CentOS6.3 64bit guest on ESXI5
- Running dd if=/dev/sda of=/dev/null bs=1M iflag=direct in VM
- iostat showing only 20MB read per sec, and systat -if 1 on FreeBSD 9.1 showing the same result.

Thanks.

-------------------------------------------------------------------------------------
25th Apr 2012

I'm experiencing very slow read speed when connecting to NFS server. The average read speed is only 5-10MB/s while the write speed sustain 50MB/s and 80-100MB/s on ASYNC mode.

I found some people also having the same issue when mounting the NFS with TCP instead of UDP. Understand that mount with UDP will solve the problem but the ESXI does not allow us to do it.

Both of my servers are running FreeBSD 9 Release with the following NFS parameters:

Code:

nfsv4_server_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 128"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"

SCP speed is about 80MB from the server to the client.

Any idea?

belon_cfy · #2 April 26th, 2012, 09:07

I found that the NFS read through igb works much faster, however slow read through the other interfaces such as em or bce. But the scp and nc transfer speed are the same. Anyone know why I'm getting inconsistent NFS read speed through different interfaces?

belon_cfy · #3 April 28th, 2012, 04:45

Seems the problem already existed for more than 5 years, however it's still not resolved yet.

http://lists.freebsd.org/pipermail/f...ry/021928.html

Problem still persists, even changed the network cable and connected directly to different clients. Only TCP NFS read will hit the "bug", others such as SCP, ISCSI are all no issue.

I can't switch to UDP because ESXI doesn't support NFS through UDP.

belon_cfy · #4 April 28th, 2012, 13:03

Speed slightly increases from 10-20MB/s to 30-50MB/s after changing the MTU to 256, but still far more behind the igb interface.

Any idea?

belon_cfy · #5 April 30th, 2012, 03:21

Changing to 'INTEL PRO/1000 PT DUAL PORT PCI-E' for testing, it has been detected as em0 and em1 and the performance is still same as before, systat -ifstat 1 showing only 10-20MB/s on NFS read but other services are able to hit 100MB/s at output by running ping -i 0.0001 -s 8192 xxx.xxx.xxx.xxx.

belon_cfy · #6 May 25th, 2012, 10:16

Problem solved by adding it into a VLAN, both interfaces are able to get a consistent 100MB/s.

belon_cfy · #7 February 5th, 2013, 04:12

5th Feb 2013
Previously I m managed to solve it by adding a vlan on top of the em interface(no issue on igb interface), but the same solution doesn't work anymore on FreeBSD 9.1.

I have verified it by switching back to FreeBSD 9.0 and it works fine again, the server able to saturate the 1Gbps link. After that I reinstall the same server to FreeBSD 9.1 and the problem came back again, it only able to push below 210Mbps max.

Is there any driver update causing bad performance on FreeBSD 9.1 ?

By the way, below is my test environment on ESXI5 with NFS mount.
- Running a CentOS6.3 64bit guest on ESXI5
- Running dd if=/dev/sda of=/dev/null bs=1M iflag=direct in VM
- iostat showing only 20MB read per sec, and systat -if 1 on FreeBSD 9.1 showing the same result.

Thanks.

Sebulon · #8 February 5th, 2013, 07:52

Hey man,

I´m soon to be upgrading a machine to 9.1 that has igb, so I´ll be keeping this in mind. Our problems have been to get Jumbo Frames to work properly with igb and have to use these sysctl's:
/etc/sysctl.conf:

Code:

kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbo9=38400

Otherwise the network wouldn´t go up at all.

/Sebulon

belon_cfy · #9 February 5th, 2013, 09:10

But we are able to get 1Gbps speed on igb interface even without jumbo frame, both of the em and igb with the default mtu size which is 1500.

The em interface are able to saturate the 1Gbps link if vlan is added on FreeBSD 9.0, but the same method doesn't work on FreeBSD 9.1 anymore. I only got 100-200Mbps through the em interface with the exact same cable and port.

By the way, the em interface does not support jumbo frame.

Code:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:25:90:6c:23:0c
        inet6 fe80::225:90ff:fe6c:230c%igb0 prefixlen 64 scopeid 0x1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:25:90:6c:23:0d
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:25:90:6c:23:0c
        inet6 fe80::225:90ff:fe68:99fe%em0 prefixlen 64 scopeid 0x7
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:25:90:68:99:ff
        inet6 fe80::225:90ff:fe68:99ff%em1 prefixlen 64 scopeid 0x8
        inet 10.90.1.246 netmask 0xffffff00 broadcast 10.90.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

Sebulon · #10 February 5th, 2013, 10:17

Quote:

Originally Posted by belon_cfy

By the way, the em interface does not support jumbo frame.

Oh yes, they most certainly do. We have several storage units with em- and igb HW that are configured for Jumbo Frames and are performing magnificantly. Here´s how, ontop of a trunk too:

Code:

ifconfig_em(or igb)0="mtu 9000 up"
ifconfig_em(or igb)1="mtu 9000 up"
cloned_interfaces="lagg0 vlanX"
ifconfig_lagg0="up laggproto lacp laggport em(or igb)0 laggport em(or igb)1"
ifconfig_vlanX="inet XXX.XXX.XXX.XXX netmask 255.255.255.0 vlan X vlandev lagg0 mtu 9000"

This have been confirmed with tcpdump showing transfers exceeding the 1460 that is the maximum for 1500 windows:

Code:

jumboclient# ping -s 8192 jumboserver.foo.com
8200 bytes from jumboserver.foo.com: icmp_req=237 ttl=64 time=0.606 ms
8200 bytes from jumboserver.foo.com: icmp_req=238 ttl=64 time=0.599 ms
...

Code:

jumboserver# tcpdump -i vlanX | grep jumboclient
10:08:07.953778 IP jumboclient.foo.com > jumboserver.foo.com: ICMP echo request, id 11436, seq 75, length 8200
10:08:07.953798 IP jumboserver.foo.com > jumboclient.foo.com: ICMP echo reply, id 11436, seq 75, length 8200

/Sebulon

belon_cfy · #11 February 5th, 2013, 10:58

Quote:

Originally Posted by Sebulon

Oh yes, they most certainly do. We have several storage units with em- and igb HW that are configured for Jumbo Frames and are performing magnificantly. Here´s how, ontop of a trunk too:

Code:

ifconfig_em(or igb)0="mtu 9000 up"
ifconfig_em(or igb)1="mtu 9000 up"
cloned_interfaces="lagg0 vlanX"
ifconfig_lagg0="up laggproto lacp laggport em(or igb)0 laggport em(or igb)1"
ifconfig_vlanX="inet XXX.XXX.XXX.XXX netmask 255.255.255.0 vlan X vlandev lagg0 mtu 9000"

This have been confirmed with tcpdump showing transfers exceeding the 1460 that is the maximum for 1500 windows:

Code:

jumboclient# ping -s 8192 jumboserver.foo.com
8200 bytes from jumboserver.foo.com: icmp_req=237 ttl=64 time=0.606 ms
8200 bytes from jumboserver.foo.com: icmp_req=238 ttl=64 time=0.599 ms
...

Code:

jumboserver# tcpdump -i vlanX | grep jumboclient
10:08:07.953778 IP jumboclient.foo.com > jumboserver.foo.com: ICMP echo request, id 11436, seq 75, length 8200
10:08:07.953798 IP jumboserver.foo.com > jumboclient.foo.com: ICMP echo reply, id 11436, seq 75, length 8200

/Sebulon

Hi Sebulon,
Thanks for your information, I'm able to obtain much higher speed than previous setup with increasing the mtu on em interface and client side.

Just wondering why the em interface perform slower on the default mtu size but the igb does faster?

belon_cfy · #12 February 19th, 2013, 01:37

Thanks Sebulon. Awaiting your confirmation.

Sebulon · #13 February 19th, 2013, 16:22

@belon_cfy

Haven´t gotten around to complete the update on that system just yet. I have however managed to do that on the storage system hosting our oVirt platform. Although the system is equipped with em NIC's, it might be good to have for comparison. So this output is also from a CentOS-6.3 amd64 VM, configured with a 80GB VirtIO HDD:

Code:

# dd if=/dev/vda of=/dev/null bs=1M iflag=direct
81920+0 records in
81920+0 records out
85899345920 bytes (86 GB) copied, 876.063 s, 98.1 MB/s

So no problems there, at least

/Sebulon

belon_cfy · #14 February 20th, 2013, 03:01

Hi Sebulon,
What kind of hypervisor you are using? Is it through NFS with TCP mounted?

We have more than 10 servers with different hardware and all of them are having slowness on read with ESXI4.1-5 + NFS on em interface.

Sebulon · #15 February 20th, 2013, 10:16

Quote:

Originally Posted by belon_cfy

What kind of hypervisor you are using?

Fedora 17 at the moment, planning on upgrading to 18 as soon as we´re done testing the procedure in our test environment.

Quote:

Originally Posted by belon_cfy

Is it through NFS with TCP mounted?

Yup. And Jumbo Frames as well.

Quote:

Originally Posted by belon_cfy

We have more than 10 servers with different hardware and all of them are having slowness on read with ESXI4.1-5 + NFS on em interface.

Our oVirt hosts are a mix of HP DL380 G5 and Sun Fire X4140, whereas the VMWare hosts all are HP DL380 G6. The storage of our oVirt platform is Supermicro-based with em NIC's, while the VMWare storage is a HP DL180 G6 with igb NIC's.

/Sebulon

belon_cfy · #16 February 21st, 2013, 03:14

Quote:

Originally Posted by Sebulon

Fedora 17 at the moment, planning on upgrading to 18 as soon as we´re done testing the procedure in our test environment.

Yup. And Jumbo Frames as well.

Our oVirt hosts are a mix of HP DL380 G5 and Sun Fire X4140, whereas the VMWare hosts all are HP DL380 G6. The storage of our oVirt platform is Supermicro-based with em NIC's, while the VMWare storage is a HP DL180 G6 with igb NIC's.

/Sebulon

How about test it without Jumbo Frames?

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Solved] Slow LAN speed because of pf.conf	Fireball	Firewalls	6	February 9th, 2012 23:17
[Solved] ZFS problems with read speed.	zux	Storage	5	February 6th, 2012 09:39
Gmirror read speed issue	RusDyr	General	5	December 24th, 2011 02:23
[Solved] FreeBSD 8.0: Samba33 read slow and write fast	Thorny	Web & Network Services	18	September 10th, 2010 08:39
slow hdd write speed	da1	General	2	August 30th, 2010 08:59

The Following User Says Thank You to Sebulon For This Useful Post:
belon_cfy (February 5th, 2013)