FreeBSD 8.0: Samba33 read slow and write fast

T

Thorny

Developer
Hello,

i've set up a samba file-server (FreeBSD 8.0-RELEASE, samba 3.3.9). I can write with a speed around 6,5 MByte/s to the shares. It's far away from the maximum, but a good step forward: the old win-server allows speed up to 0,8 MByte/s ;)
The problem is: i can just read with 0,3 MByte/s from the shares.

I've tried many optimizations, but nothing work. I have no idea!

My current config can be found here: http://www.meisterderspiele.de/smb.conf

The system is an AMD Phenom with 4 cores, it has 8 GB RAM and a 2 TB HDD and a Gigabit NIC.

I have also an small server with FreeBSD 7.1 and Samba 3.3.2. There the speed is around 6,5 MByte in reading and writing. The load is around 0.1. ;)

The config is identical. How can i increase the speed? I'm happy about every advice.

Thanks and grettings from germany,
Torsten
 
did you try building it with AIO_SUPPORT

I get REALLY good samba performance with freebsd 8.0 and this option.
after doing that, you'll need to add the proper stuff to smb.conf and enable the aio kernel module.

this is what the global section of my config looks like
Code: 
======================= Global Settings =====================================
[global]

workgroup = HOME
server string = Samba Server
security = user
hosts allow = 192.168.1. 192.168.2. 127. 10.
load printers = yes
printcap name = /etc/printcap
guest account = nobody
log file = /var/log/samba/log.%m
max log size = 50

socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
min receivefile size=16384
use sendfile=true
aio read size = 16384
aio write size = 16384
aio write behind = true
dns proxy = no
 
Hello,

no, i didn't try AIO - i did not know how to integrated it corretly into the kernel. But i wrote a script to make performance tests for varios configurations over the weekend.

Add least i'm ending with changing the line:
Code: 
socket options = TCP_NODELAY
to:
Code: 
socket options = SO_RCVBUF=8192 SO_SNDBUF=8192 TCP_NODELAY IPTOS_LOWDELAY

But i'm still unhappy. With the old config i could read with 0,3 MByte/s - on my Mandriva Linux maschine. I figured out, that the Windows XP Clients and the iMacs could read 2,5 MByte/s from the same share. After changing the config like described, my linux maschine gets 6 MByte/s reading and writing and the Windows clients just get 3 MByte/s reading and 6 MByte/s writing.

This is very strange. Also strange: multiple Windows clients can read from the share with 2,5MByte/s at the same time. I would have believed, that the clients have to share the 2,5 MByte/s, but every client has them.
 
it's easy to get aio working
just add
Code: 
aio_load="YES"
to /boot/loader.conf
compile samba with aio support and add the lines to the config.
 
I haven't had much luck with the socket options, aio on the other hand works great.

I currently have a FreeBSD 8.0-RELEASE amd64 server running Samba 3.3.9 from ports. A group of users are currently transferring data via samba, and a simple netstat shows about ~95MB/sec:
Code: 
> netstat -I em0 -w 1
            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
     85704     0   92897892      90866     0    5087296     0
     87200     0   95474669      92198     0    5164423     0
     88259     0   95926564      93526     0    5237458     0
     87290     0   94937826      92351     0    5168892     0

I have a pretty simple smb.conf, the only tuning I did were these 3 lines to the [global] section:
Code: 
	min receivefile size = 131072
	aio read size = 1
	aio write size = 1
	use sendfile = yes

rc.conf:
Code: 
ifconfig_em0="inet 128.115.132.165  netmask 255.255.255.0 polling tso mtu 9194"

sysctl.conf:
Code: 
kern.ipc.maxsockbuf=16777216
kern.ipc.nmbclusters=32768
kern.ipc.somaxconn=32768
kern.maxfiles=65536
kern.maxfilesperproc=32768
kern.maxvnodes=800000
net.inet.tcp.delayed_ack=0
net.inet.tcp.inflight.enable=0
net.inet.tcp.path_mtu_discovery=0
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=65536
net.inet.tcp.rfc1323=1
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=524288
net.inet.tcp.sendspace=65536
net.inet.udp.maxdgram=57344
net.inet.udp.recvspace=65536
net.local.stream.recvspace=65536
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.mssdflt=1460

I used to think you could only get this type of performance from NFS, not Samba, but I've been incredibly happy with these results.
 
So... after a night of work it's done. I had migrate your configs and run some tests. With AIO and the configuration everything works fine. Now the Win-Clients are even a little faster than the linux clients.

Thanks for your help! :)
 
cool, i'm glad we could help. The thing is, that with sysctls and smb.conf you can play around and tweak it to get the best performance. I get about 100 MB/s myself (but i have zfs with 3 raidz groups each with 4 1tb drives= 12 drives total)

AIO gave me much better performance. I've always had it set to 16 KB but in all honesty most of my files are huge.

I can also say that 8.0, for me, is a major improvement over 7.2


This may be due to ZFS improvements on my end. I was getting 70ish MB/s on 7.2
 
I'm also glad that you could help me :)

If i haven't find a solution till the new year, i would have written my own sharing-software. We have the special situation that there are around 400 GB of files, but the most of them are small (< 100 KB). From the 400 GB just 2 GB are heavily requested a day and the rest maybe every week or less. I've already created a concept for a simplistic software, which make statistic about file-requests and hold the most requested files just in RAM to avoid IO.

Maybe later... :)
 
hehe, yah, i hear ya. I use mine mostly for a network of htpc's

In our house we've done away with cable tv pretty much. I have seedbox which is set up with RSS to get all the tv shows and movies. Then i have rsync automatically sync the seedbox to my home server (which is FreeBSD 8.0 with ZFS) The htpc's are all based on the ionitx motherboards, and samba was the easiest choice (though i might switch to NFS eventual) because xbmc can mount samba shares by default

The only thing i'd really like to see is XBMC and VDPAU on FreeBSD so i could switch THOSE boxes over as well.
 
I am having a similar issue as the OP and the things advised here helped, but not really enough. The initial situation was that local reads and writes saturate the disks with roughly ~75mb/s throughput. Writing to a Samba share over the network, even without AIO, my writes were saturating my disks with 75mb/s. However reads off Samba shares were very slow: only 18mb/s!

At first I tried enabling AIO as per the advice in this thread. I saw a very welcome increase in read speeds, now they are ~48mb/s. I tried the sysctl tunables also described in this thread and they brought the read speeds a tiny bit further, now at ~55mb/s. However, I don't understand, why are local reads and writes saturating the disks easily, but over Samba, only writes can, with reads still lagging behind by a noticable margin, even after all the tuning?
 
Oh my god... Why did noone tell me how much of an enormous performance boost vfs.zfs.prefetch_disable=0 (aka actually enabling prefetch) is. My local reads off the mirror pool jumped from 75mb/s to 96mb/s (ie. they are now nearly 25% faster than reading off an individual disk) and reads off a Samba share skyrocketed to 90mb/s.

By default, FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386 systems and on any amd64 systems with less than 4GB of avaiable memory. My system is amd64 with 4gb ram, but integrated video eats some of that, so the autotuning disabled the prefetch. I had read up on it and a fair amount of people seemed to have performance issues caused by having prefetch enabled and get better results with it turned off, in my case however, it seems that enabling it gave a really solid boost to performance.
 
Jago said:
Oh my god... Why did noone tell me how much of an enormous performance boost vfs.zfs.prefetch_disable=0 (aka actually enabling prefetch) is. My local reads off the mirror pool jumped from 75mb/s to 96mb/s (ie. they are now nearly 25% faster than reading off an individual disk) and reads off a Samba share skyrocketed to 90mb/s.

By default, FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386 systems and on any amd64 systems with less than 4GB of avaiable memory. My system is amd64 with 4gb ram, but integrated video eats some of that, so the autotuning disabled the prefetch. I had read up on it and a fair amount of people seemed to have performance issues caused by having prefetch enabled and get better results with it turned off, in my case however, it seems that enabling it gave a really solid boost to performance.
Click to expand...

this would make sense, because my system had it on by default.
 
Im not having same luck...

On 8.0-RELEASE-p2 From 7.2..

I used too get 77mb on gigabit Lan, no aio or anything fancy, standard ufs2 with samba 3.0.something.

Now i get 15mb tops, i resorted too updating too samba 3.3.10 and enabling aio and all tweaks in this page, now ive improved to 30-39mb.
Over double what i got but no where near 7.2 performance.

Any ideas?
 
thavinci said:
Im not having same luck...

On 8.0-RELEASE-p2 From 7.2..

I used too get 77mb on gigabit Lan, no aio or anything fancy, standard ufs2 with samba 3.0.something.

Now i get 15mb tops, i resorted too updating too samba 3.3.10 and enabling aio and all tweaks in this page, now ive improved to 30-39mb.
Over double what i got but no where near 7.2 performance.

Any ideas?
Click to expand...


The last person i met who had really poor speeds in 8.0 was becaused they have turned the log level up.

Did you do this? If so, remove it entirely.

If not i don't know, that was just a shot in the dark
 
Thanks for attempt but i have switched off logging for the most part in attempts to improve performance, this includes...
Code: 
log file = /dev/null


and

Code: 
syslog = 0

so logging is at its lowest ;>
 
wonslung said:
did you try building it with AIO_SUPPORT

I get REALLY good samba performance with freebsd 8.0 and this option.
after doing that, you'll need to add the proper stuff to smb.conf and enable the aio kernel module.

this is what the global section of my config looks like
Code: 
======================= Global Settings =====================================
[global]

workgroup = HOME
server string = Samba Server
security = user
hosts allow = 192.168.1. 192.168.2. 127. 10.
load printers = yes
printcap name = /etc/printcap
guest account = nobody
log file = /var/log/samba/log.%m
max log size = 50

socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
min receivefile size=16384
use sendfile=true
aio read size = 16384
aio write size = 16384
aio write behind = true
dns proxy = no
Click to expand...

Is it safe to enale aio write behind when using samba 3.4.8 on FreeBSD 8.1 amd64 with ZFS?
The reason for asking is that samba documentation say that it may cause data corruption:
http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html
If Samba has been built with asynchronous I/O support, Samba will not wait until write requests are finished before returning the result to the client for files listed in this parameter. Instead, Samba will immediately return that the write request has been finished successfully, no matter if the operation will succeed or not. This might speed up clients without aio support, but is really dangerous, because data could be lost and files could be damaged.

The syntax is identical to the veto files parameter.

Default: aio write behind =

Example: aio write behind = /*.tmp/
Click to expand...

I run ZFS to protect my data but I also want high speed and performance. However I do no trade the data security for performance. Serching internet for performance tweaks but I also need to know if the tweaks are safe or not. I only get 60 MB/s samba transfer speed now after moving from opensolaris (where I had 90 MB/s using cifs). But since opensolaris was killed I want to use FreeBSD which seems like a good and free operating system. Now I am trying to tweak it in a safe and good way.
 
see, i moved the other way, i went from FreeBSD to OpenSolaris.

If i had to move from OpenSolaris it would have to be to nexenta. There is no way i'd willingly give up CIFS now that i've gotten used to it.
 
You must log in or register to reply here.
Back
Top