Horrible Samba write performance and network dropping

M

mrjayviper

problems:

1. I'm trying to copy several 100+GB worth of files from my MacPro to my newly setup FreeBSD server and the write performance is very slow. It's varying between 21 to 22 Mbytes/sec.

2. During copying my MacPro connection would completely die. And this lines would show up on /var/log/messages

Code: 
Sep  4 09:42:12 myserver kernel: em1: link state changed to DOWN
Sep  4 09:42:15 myserver kernel: em1: link state changed to UP
Sep  4 09:42:15 myserver devd: Executing '/etc/rc.d/dhclient quietstart em1'
Sep  4 09:42:24 myserver kernel: em1: link state changed to DOWN
Sep  4 09:42:26 myserver kernel: em1: link state changed to UP
Sep  4 09:42:26 myserver devd: Executing '/etc/rc.d/dhclient quietstart em1'

I previously was using FreeBSD10.2-stable (custom kernel/buildworld upgrade instead of freebsd-update) and never had these problems. I replaced FreeBSD with Ubuntu 16.04 server (not the desktop version so no GUI) and I'm pretty sure the write performance and network is much more stable as well. I went back to FreeBSD 10.3 yesterday since I realized I prefer FreeBSD over Ubuntu and I'm not willing to test other Linux distributions.

Server (FreeBSD 10.3 updated to recent via freebsd-update):
- AMD A4 3Ghz socket FM2
- 4GB RAM
- dual Intel gigabit NIC PCIexpress (shows up as emX)
- boot USB3 stick (connected to USB2 port though as I get issues during boot-up ig plugged in on built-in USB3 port. The USB3 is provided natively by the AMD chipset.)
- 8 HDDs setup as RAIDZ (5x 2TB and 3x 1TB)

Client (MacPro running OS X El Capitan 10.11.6)
MacPro connected to Samba server via a dedicated Gigabit port using quality (got it from work and they used this on our datacenter) crossover cable.

Configuration:

/boot/loader.conf
Code: 
=> cat /boot/loader.conf
hw.pci.do_power_nodriver=3

/etc/sysctl.conf
Code: 
=> cat /etc/sysctl.conf
# $FreeBSD: releng/10.3/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0

###########################################
security.bsd.see_other_gids=0
security.bsd.see_other_uids=0

/usr/local/etc/smb4.conf
Code: 
=> cat /usr/local/etc/smb4.conf
[global]
load printers = no
workgroup = myserver.local

bind interfaces only = yes
interfaces = em0 em1
hosts allow = 192.168.1.0/24 192.168.2.0/24
host deny = all
security = user

log file = /var/log/samba4/log.%m
max log size = 1000
interfaces = em0 em1
passdb backend = tdbsam

aio read size = 16384
aio write behind = true
aio write size = 16384
dns proxy = no

min receivefile size=16384
socket options=IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536 TCP_NODELAY
use sendfile=true

unix extensions = no
nt acl support = yes
inherit acls = no
map acl inherit = yes
vfs objects = zfsacl

[smb-storage]
path = /Volumes/storage
public = no
writable = yes
printable = no
create mask = 0664
directory mask = 0775

edit1: Reinstalled FreeBSD 10.3 and upgrading sytem via buildkernel/buildworld. I will report back the results once done.
 
Did you try another cable, possibly a known-good one?
I had these connection problems and similar syslog entries also recently and it ended up with discarding a cable that proved defective.
 
Snurg said:
Did you try another cable, possibly a known-good one?
I had these connection problems and similar syslog entries also recently and it ended up with discarding a cable that proved defective.
Click to expand...

It is a good cable. The day before I moved to FreeBSD (i.e. still on Ubuntu 16.04), I copied all the contents of my zpool (around 5TB worth of stuff) to my MacPro.
 
On my home server, which is a low performance Intel Atom system, I got installed net/samba43 as well as net/netatalk3, and my experience is, that samba has a much higher cpu utilization than netatalk, and consequently, I found always higher data transfer rates when copying data to/from the server using afp instead of smb with the Mac clients. This was already the case with samba3 and the situation definitely became worse with samba4.

That said, IMHO, 22 MByte/s = apprx. 220 MBit/s is perhaps not what could be expected on a GB line, however, it is a tad better than 'horrible'.

You set the TCP_NODELAY flag in your file /usr/local/etc/smb4.conf, OK so far, however the default sysctl setting on the Mac is net.inet.tcp.delayed_ack = 2, and this does not match your TCP_NODELAY setting of the smb configuration on your server.

In general, I always found a better network perfomance between my Mac and/or Win clients and/or the FreeBSD server by setting the TCP_NODELAY flag on the respective systems (on Windows this is the default setting). I suggest to add on your Mac OS X system as the user root the file /etc/sysctl.conf having the following line:
Code: 
net.inet.tcp.delayed_ack=0
Then restart the system.

In order that also other network communications between your Mac and the FreeBSD box benefits from and are not spoiled by this, you would need to do the same thing also on your server.

In the case you don't want to change said sysctl setting on the Mac, you would need to remove the TCP_NODELAY flag from the socket options of your smb configuration, since this setting is supposed to match on both sides. If it doesn't match, this may easily explain the experienced 'network dropping'.
 
obsigna said:
On my home server, which is a low performance Intel Atom system, I got installed net/samba43 as well as net/netatalk3, and my experience is, that samba has a much higher cpu utilization than netatalk, and consequently, I found always higher data transfer rates when copying data to/from the server using afp instead of smb with the Mac clients. This was already the case with samba3 and the situation definitely became worse with samba4.

snip....
Click to expand...

Thanks for the tip. I've rebuilding kernel then world. I'll keep in mind your suggestions after I'm installed samba from ports.

That said, I've been using the TCP_NODELAY on FreeBSD 10.2-stable months ago and I wasn't setting anything on my Mac and it was working fine. but perhaps things are different in 10.3 vs 10.2
 
I get 100+ MB/s both ways on Samba with a J1800 Celeron on 10.3-RELEASE & Windows (about 3%/26% CPU/WCPU usage).

If you're on Mac, maybe try NFS instead and see if its a limitation with the protocol.

screen.png
 
I have 2 identical setups running FreeBSD 10.3-stable. The only difference is that the 1st one has more storage capacity than the 2nd one. Both setup experiences the problem when writing to the network shares. I have no problems when copying files out of the share. In fact, I was able to copy 200+GB worth of files overnight and no problems whatsoever.

As a test, I move the 2nd setup to Ubuntu and no problems with copying out/writing to the network share. I suspect the problem could be related to 10.3 as according to the release notes, there's a new network driver for my network cards (Intel em).
 
You must log in or register to reply here.
Back
Top