FreeBSD 9.0 slow single network copies

D

diddie17

I have been struggling with really slow network transfers with 8.2 and just upgraded to 9.0 Release in the hope that it would fix things.

Transfer speeds (to a windows machine) across a Gigabit network are around 750Kb/s, and whilst 9.0 has not resolved the problem, I am now noticing that single copies run at about this speed, but if run multiple copy operations the speed ramps up significantly.

The speed tops out at about 21 Mb/s when running 4 or so copy operations, which is still not great, but the FreeBSD machine is running on the troublesome Realtek 8168 network chip and so I'm not expecting it ever to be really quick. but the symptoms I'm seeing sort of suggest some sort of limit per copy process that is constraining the performance.

I'm just wondering if anyone else has seen anything similar (single copies slow, multiple copies much quicker) or could point me towards any new areas to investigate.

In case it helps I'm running AMD64, 4GB RAM, 4 x 2TB disks in a ZFS pool and a separate boot disk. The machine is a dedicated storage server and has no other load.
 
Yes they would be Samba transfers. FTP transfers don't seem to suffer the same sorts of issues. They still top out at around the 21MB/s mark, but certainly I don't need to add extra transfers to get the speed of the initial transfer up.

I installed the Samba package rather than build it. I would have assumed the package came with aio support on, but I'll go back and build it to make sure if you think it might make a difference.

Thanks for the suggestion
 
You should also load the module at boot time in loader.conf

Code: 
aio_load="YES"

Now, regarding you speed does it top up to 21Mbit ?
 
diddie17 said:
I installed the Samba package rather than build it. I would have assumed the package came with aio support on, but I'll go back and build it to make sure if you think it might make a difference.
Click to expand...
It off by default on 3.4.x, 3.5.x and 3.6.x. Packages are built using the default port options.
 
I'll let you know how it goes. I'm having a little trouble building Samba 3.6 with an error around an undefined flag SMB_CTX_FLAG_USE_CCACHE, so I'll let you know how it goes when I've sorted this out.
 
I've now got 3.6 built with aio and an entry for aio is in the loader.conf file.

After boot I run

Code: 
#kldstat -v | grep aio
2    1 0xffffffff813ce000 f420     aio.ko (/boot/kernel/aio.ko)
		1 aio
If I kick off a copy of a 2GB video file the transfer runs at an average of 750Kbs. I've left this running for 10 minutes and it doesn't change.

When I kick off a second copy of a file from the same to and from directories, all of a sudden both copies speed up hugely and they average 22-23Mb/s. This speed is sustained until one of the copies completed at which point the remaining copy operation drops back to the 750Kb/s transfer rate.

The graph below show this in operation. Anyone got any other ideas?

Transfer%2Bspeed.jpg
 
This is hard to troubleshoot because I am not aware of your topology, switches, firewalls, etc.

What I would do first is try a different NIC just to rule out a driver issue. It doesn't have to be Gigabit.

Then, I would place a sniffer between my FreeBSD box and the corresponding switch. That would give me some hints on certain network issues.

I imagine that you have already tried different configurations in your switch.
 
Thanks gkontos, I'll see if I can find another NIC to test the driver issue. I'm aware that FreeBSD doesn't like the Realtek 8168 chips much, so there is every chance it could be some sort of driver issue.

Just on the topology, both machines are connected directly to a GS108T gigabit switch and both are showing connection at 1000MB full duplex. There isn't really anything I could change to make it simpler.

Also given that single FTP transfers run at around the 20 - 23 MB/s speed, it does fell like a Samba or Driver configuration issue.
 
Well, for those who wondering how to fix it. I had the same issues on a recent 8.3 box and Samba 3.5 which is upgraded from 7.4. Under 7.4 everything was fine, but on 8.3 reads were really slow - only 2-3MB/s on 100Mbit network, while writing performance was 8-9Mbit which is acceptable. The problem wasn't connected to hardware, because FTP and HTTP transfers gave 9-10MB/s.

There was some threads which recommended to turn on AIO and play with 'socket options' option - yes, it helped a bit and read speed increased to 3.5-4 MB/s which is still far from perfect.

The solution were simple: turn off ULE scheduler and bring back good old BSD one. This increased read speed to 9.2MB/s with AIO turned off, and to 8.3MB/s with AIO turned on (yes, AIO slow things down, may be this UFS related, but it is not important for now).

Since this could be CPU related problem, here is the excerpt from dmesg:
Code: 
kernel: CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1601.61-MHz 686-class CPU)
kernel: Origin = "GenuineIntel"  Id = 0x106c2  Family = 6  Model = 1c  Stepping = 2
kernel: Features=0xbfe9fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH
kernel: Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
kernel: AMD Features=0x20100000<NX,LM>
kernel: AMD Features2=0x1<LAHF>
kernel: TSC: P-state invariant
kernel: real memory  = 2147483648 (2048 MB)
kernel: avail memory = 2062090240 (1966 MB)
kernel: ACPI APIC Table: <060810 APIC1613>
kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
kernel: FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
 
You must log in or register to reply here.
Back
Top