FreeNAS and slow network gigabit speed

P

paradoxmonkey

Hi,

I recently installed FreeNAS 0.7RC1 (build up on FreeBSD 7.2) on one of my machines. I ran iperf to test the network speed and I never got over 500Mbits/sec. Running Linux I get something between 600-700Mbit/sec (if I recall right, will have to check it again). I'm using a realtek RTL8110S NIC. Also the FTP and Samba speed on FreeNAS is horribly slow, it never reaches above 22MB/sec.
 
i don't know much about freenas but can you test this running iperf to another freebsd machine running freebsd? also, realtek network cards are known for not performing exceptionally well so the question begs which card in in your linux machine? also, how is the network setup? are the freenas machine and linux machine on the same switch?
 
I was testing with a linux distro on the same machine against a Windows PC running iperf. The FreeNas machine and the windows machine are on the same gigabit switch although trying with a direct connection didn't increase the speed.
 
Here it goes:

Code: 
freenas:~# dmesg
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.2-RELEASE-p3 #0: Thu Aug 20 20:10:26 UTC 2009
    root@vmbsd72i386:/usr/obj/freenas/usr/src/sys/FREENAS-i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Celeron(R) CPU 2.66GHz (2663.19-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf34  Stepping = 4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x441d<SSE3,DTES64,MON,DS_CPL,CNXT-ID,xTPR>
real memory  = 266600448 (254 MB)
avail memory = 146407424 (139 MB)
ACPI APIC Table: <A M I  OEMAPIC >
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
wlan: mac acl policy registered
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <A M I OEMRSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, fe00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xec00-0xec07 mem 0xf0000000-0xf7ffffff,0xff280000-0xff2fffff irq 16 at device 2.0 on pci0
agp0: <Intel 82865G (865G GMCH) SVGA controller> on vgapci0
agp0: detected 892k stolen memory
agp0: aperture size is 128M
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0xdc00-0xdc1f irq 16 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0xe000-0xe01f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0xe400-0xe41f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <Intel 82801EB (ICH5) USB controller USB-D> port 0xe800-0xe81f irq 16 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <Intel 82801EB (ICH5) USB controller USB-D> on uhci3
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xff27fc00-0xff27ffff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> port 0xb800-0xb8ff mem 0xff0ffc00-0xff0ffcff irq 20 at device 3.0 on pci1
re0: Chip rev. 0x04000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
re0: Ethernet address: 00:0a:cd:11:a3:15
re0: [FILTER]
rl0: <RealTek 8139 10/100BaseTX> port 0xb400-0xb4ff mem 0xff0ff800-0xff0ff8ff irq 22 at device 5.0 on pci1
miibus1: <MII bus> on rl0
rlphy0: <RealTek internal media interface> PHY 0 on miibus1
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: Ethernet address: 00:19:66:60:0f:bb
rl0: [ITHREAD]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel ICH5 SATA150 controller> port 0xd000-0xd007,0xcc00-0xcc03,0xc800-0xc807,0xc400-0xc403,0xc000-0xc00f irq 18 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pci0: <multimedia, audio> at device 31.5 (no driver attached)
acpi_button0: <Power Button> on acpi0
speaker0: <PC speaker> port 0x61 on acpi0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc9fff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2663193680 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging limited to 5 packets/entry by default
md0: Preloaded image </mfsrooad0: 977MB <SanDisk SDCFB-1024 HDX 4.03> at ata0-master PIO4
t> 98566144 bytes at 0x80ed50b0
GEOM_LABEL: Label for provider md0 is ufsid/4a9431be18a25235.
GEOM_LABEL: Label for provider ad0a is ufsid/4a9431db64cccabb.
ad3: 194481MB <Maxtor 6L200P0 BAH41G10> at ata1-slave PIO4
ad4: 476938MB <WDC WD5000AAKS-00TMA0 12.01C01> at ata2-master SATA150
GEOM_LABEL: Label for provider ad3s1 is ntfs/FENIX.
GEOM_LABEL: Label for provider ad4s1 is ntfs/H2.
Trying to mount root from ufs:/dev/md0
GEOM_LABEL: Label ufsid/4a9431be18a25235 removed.
fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
GEOM_LABEL: Label ntfs/FENIX removed.
GEOM_LABEL: Label ntfs/H2 removed.
GEOM_LABEL: Label for provider md1 is ufsid/4ab310046020fc31.
GEOM_LABEL: Label ufsid/4ab310046020fc31 removed.
re0: link state changed to UP
This module (opensolaris) contains code covered by the
Common Development and Distribution License (CDDL)
see http://opensolaris.org/os/licensing/opensolaris_license/
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS WARNING: Recommended minimum RAM size is 512MB; expect unstable behavior.
ZFS filesystem version 6
ZFS storage pool version 6
GEOM_LABEL: Label ufsid/4a9431db64cccabb removed.
GEOM_LABEL: Label for provider ad0a is ufsid/4a9431db64cccabb.
 
paradoxmonkey said:
Also the FTP and Samba speed on FreeNAS is horribly slow, it never reaches above 22MB/sec.
Click to expand...

You realize that is 220Mbit/sec? That's actually pretty decent speed depending on your underlying IO subsystem and the characteristics of your read/write operations.
 
gordon@ said:
You realize that is 220Mbit/sec? That's actually pretty decent speed depending on your underlying IO subsystem and the characteristics of your read/write operations.
Click to expand...

I don't consider it be that good consdering that a linux distro can push between 30-45Mbit/sec.

The hardware list seems to be on hold for moderation, dunno why...
 
You said 30-45Mbit/sec. Network speeds are usually expressed in Mbit/sec. Note that a lot of applications (typically end-user FTP programs) either use Mbyte/sec or actually confuse the two (expressing Mbit as Mbyte, or the other way around).
 
I'm sorry I thought that I did in fact write Mbyte, with obviously I didn't. My bad, sorry. I have edited the above post to reflect the right unit. The hardware info has also been posted.
 
Sorry again, it seems that you can not edit your posts.

Let me clear out some things.

iperf
----
(machine 1) FreeBSD --> (machine 2) Windows machine = never above 500Megabit/sec

(machine 1) Linux distro --> (machine 2) Windows machine = somewhere between 600-700Megabit/sec.

Samba
----
(machine 1) FreeBSD --> (machine 2) Windows machine = never above 22Megabyte/sec

(machine 1) Linux distro --> (machine 2) Windows machine = somewhere between 35-45Megabyte/sec.


The hardware info has already been posted. As you can see FreeBSD is slower then any random linux distro.
 
Sorry again, it seems that you can not edit your posts.
Click to expand...

you have to have made 10 posts and waited 10 days. then you will be able to edit your posts.


As you can see FreeBSD is slower then any random linux distro
Click to expand...


if i understand correctly FreeNAS is running on the same exact hardware that your Linux distro was running on... and I would again say, install FreeBSD 7.2 and test again. And verify your linux speeds because you did in-fact say;
I get something between 600-700Mbit/sec (if I recall right, will have to check it again).
Click to expand...


Also, can you post the output of
Code: 
ifconfig
and also
Code: 
netstat -i
 
I think it's a good idea to separate network performance from disk performance.

How do you measure the ftp transfer rate? You read/write a file? Could you tell us from which partition/disk? I think this is interesting:

Code: 
ad3: 194481MB <Maxtor 6L200P0 BAH41G10> at ata1-slave PIO4
ad4: 476938MB <WDC WD5000AAKS-00TMA0 12.01C01> at ata2-master SATA150
GEOM_LABEL: Label for provider ad3s1 is ntfs/FENIX.
GEOM_LABEL: Label for provider ad4s1 is ntfs/H2.

ad3 is running in in PIO4 mode, I guess the reason is the flash disk where you boot from (connected to the same ide-channel which forces the other device to PIO4 also). Running PIO4 reduces the performance to 16,6 MByte/s!

I had a similar problem with a CF card directly connected to an ide socket, which was driven in PIO mode instead of a DMA mode. I added "hw.ata.ata_dma_check_80pin=0" to /boot/loader.conf which helped to drive the disk in UDMA100 mode. How is the flash disk connected (IDE<>CF adapter? Any cables? 40Pin/80Pin?).

Could you measure the hard disk performance with dd and post the results? E.g.

Code: 
dd if=/dev/ad3 of=/dev/null bs=10M count=1000
dd if=/dev/ad4 of=/dev/null bs=10M count=1000
...

Also I'm wondering why you are using NTFS as filesystem on your FreeNAS box.

cheers,
honk
 
iperf
----
(machine 1) Linux distro --> (machine 2) Windows machine = 616Megabit/sec.

(machine 1) New clean FreeBSD install (livecd) --> (machine 2) Windows machine = 529Megabit/sec.

New clean FreeBSD install (livecd)
----
Code: 
Fixit#ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_Magic>
ether 00:0a:cd:11:a3:15
inet 6 fe80:20a:cdff:fe11:a315%re0 prefixlen 64 scopeid 0x1
inet 192.168.1.250 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000BaseTX <full-duplex>
status: active
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384



Fixit#netstat -s -p tcp:
tcp:
	1921545 packets sent
		89 data packets (1492 bytes)
		2 data packets (32 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		1244725 ack-only packets (17 delayed)
		0 URG only packets
		0 window probe packets
		676690 window update packets
		39 control packets
	2907903 packets received
		123 acks (for 1507 bytes)
		2 duplicate acks
		0 acks for unsent data
		2907492 packets (3969553569 bytes) received in-sequence
		1 completely duplicate packet (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		19 window update packets
		17 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	19 connection requests
	1 connection accept
	0 bad connection attempts
	0 listen queue overflows
	0 ignored RSTs in the windows
	20 connections established (including accepts)
	25 connections closed (including 1 drop)
		11 connections updated cached RTT on close
		11 connections updated cached RTT variance on close
		3 connections updated cached ssthresh on close
	0 embryonic connections dropped
	122 segments updated rtt (of 128 attempts)
	2 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	0 correct ACK header predictions
	2907488 correct data packet header predictions
	1 syncache entry added
		0 retransmitted
		0 dupsyn
		0 dropped
		1 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	1 cookie sent
	0 cookies received
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	0 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow

Sorry for not posting with "code" tags, I didn't wane delay the post for moderation.
 
honk said:
I think it's a good idea to separate network performance from disk performance.

How do you measure the ftp transfer rate? You read/write a file? Could you tell us from which partition/disk? I think this is interesting:
Click to expand...

I did the ftp and samba test agains the sata harddrive. One can set the dma speed in the webinterface of FreeNAS, so the drive is infact running in UDMA100 but dmesg is unaware of it. I'm using a ide-to-cf adapter.
Code: 
freenas:~# dd if=/dev/ad3 of=/dev/null bs=10M count=1000
1000+0 records in
1000+0 records out
10485760000 bytes transferred in 159.357577 secs (65800197 bytes/sec)

freenas:~# dd if=/dev/ad4 of=/dev/null bs=10M count=1000
1000+0 records in
1000+0 records out
10485760000 bytes transferred in 123.375436 secs (84990662 bytes/sec)


Also I'm wondering why you are using NTFS as filesystem on your FreeNAS box.
Click to expand...

The harddrive were full of data before I put them in my FreeNas and they were formated in NTFS. I just haven't bothered myself with a complete backup and format.
 
Read performance on the raw disk looks good. Could you try to read one of your testfiles (lets say a 100 MByte file) from your NTFS partition with dd (omit the count parameter)?
 
honk said:
Read performance on the raw disk looks good. Could you try to read one of your testfiles (lets say a 100 MByte file) from your NTFS partition with dd (omit the count parameter)?
Click to expand...

Can you give me an example of how to do that? I'm not that very familiar with the dd command.

Wouldn't it be better to use something like this:
Code: 
diskinfo -t /dev/ad3
 
diskinfo measures the raw device, no matter which file system. We already saw that your disks achieve a transfer rate of 65MBytes (ad3) and 85MBytes (ad4), but I'm not sure how fast/slow the NTFS implementation is.

Just read a big file with dd:

Code: 
dd if=/data/blabliblu/bigtestfile of=/dev/null bs=10M
 
Code: 
freenas:/# dd if="/mnt/Drive-A/Episode 16.mov" of=/dev/null bs=10M
18+1 records in
18+1 records out
196083860 bytes transferred in 16.345883 secs (11995917 bytes/sec)

freenas:/# dd if="/mnt/Drive-B/Episode 16.mov" of=/dev/null bs=10M
18+1 records in
18+1 records out
196083860 bytes transferred in 4.468306 secs (43883265 bytes/sec)
 
Could it be that the NTFS implementation is slower in FreeBSD? If so it still doesn't explain why the iperf performance is less then of a linux distro on the same machine.
 
As I see there are two possible problems. The slow network speed and maybe the read/write to an NTFS volume.

Can anyone explain or help me figure out why the network speed on Freebsd 7.2 is so slow.
 
You must log in or register to reply here.
Back
Top