I'm trying to set up a high speed samba share using 40Gb network cards, and encountered a strange problem. I tested with FreeBSD 13.2 and 14.0-RC2, and Samba 4.16.11. I got the same behavior on both FreeBSD versions.
Here is my setup : a storage with 36 SAS HDD of 6TB, an Epyc 7282 and 128GB of DDR4.
On my pool, compression is off, sync is disabled and I'm using 1M recordsize.
Internal tests using iozone and fio with 5 streams of 100GB give me 4200GB/s read and 3200GB/s write.
If I disable ARC cache (set primarycache to none), I got the exact same performances.
Regarding the network configuration, I'm testing a direct 40Gb link between a Windows 10 system and this server, using a Chelsio T580-LP-CR on both machines.
Doing an iperf test between these two systems give me 35Gb/s with single thread and 40GB/s with 2 parallel threads or more.
All the bandwidth tests were made with very big files copied (between 50 and 300GB per file), and using AJA system test.
Of course, these tests have been run several times with reboot and power down, to be sure it wasn't a one time only (un)lucky test session.
When sharing a ZFS volume from this pool, I got a 2800MB/s write performance, even with a 500GB write.
The problem is with the read performances. If the data are in the ARC, I also got 2800MB/s in read. But as soon as my data are read from the disk, it drops to 1000MB/s.
If I completely disable ARC, write is still at 2800MB/s and read at 1000MB/s.
FYI, here is the content of my smb4.conf file:
In order to be sure of my systems, I did two things :
First I created a ZFS pool using 4x1TB NVMe PCIe 4. Sharing a volume from this pool I got a stable and constant 3000MB/s in read and write, without changing any network or Samba parameters.
Then to be sure my 36 HDD pool was not the problem, I exported it then imported it on Linux and OmniOS to check if I got the same read speed problem. I don't.
With ARC cache disable, on OmniOS 151046, I got stable 1600MB/s in read and write using the kernel SMB server. On Linux with Samba, using latest TrueNas Core as it was easier for testing purposes, I got stable 2600MB/s read and 2200MB/s write.
Does anyone has an idea of what can cause this ? Or if you have any tests in mind I can run to find the reason why I got such a huge difference between internal read performances and through a Samba share.
Thanks in advance for your help
Here is my setup : a storage with 36 SAS HDD of 6TB, an Epyc 7282 and 128GB of DDR4.
On my pool, compression is off, sync is disabled and I'm using 1M recordsize.
Internal tests using iozone and fio with 5 streams of 100GB give me 4200GB/s read and 3200GB/s write.
If I disable ARC cache (set primarycache to none), I got the exact same performances.
Regarding the network configuration, I'm testing a direct 40Gb link between a Windows 10 system and this server, using a Chelsio T580-LP-CR on both machines.
Doing an iperf test between these two systems give me 35Gb/s with single thread and 40GB/s with 2 parallel threads or more.
All the bandwidth tests were made with very big files copied (between 50 and 300GB per file), and using AJA system test.
Of course, these tests have been run several times with reboot and power down, to be sure it wasn't a one time only (un)lucky test session.
When sharing a ZFS volume from this pool, I got a 2800MB/s write performance, even with a 500GB write.
The problem is with the read performances. If the data are in the ARC, I also got 2800MB/s in read. But as soon as my data are read from the disk, it drops to 1000MB/s.
If I completely disable ARC, write is still at 2800MB/s and read at 1000MB/s.
FYI, here is the content of my smb4.conf file:
Code:
[global]
server string = FreeBSD Server - Samba %v
workgroup = TEST
log file = /var/log/samba4/smb.log
max log size = 50
load printers = No
disable netbios = Yes
map to guest = Bad User
security = USER
server role = standalone server
create mask = 0775
directory mask = 0775
invalid users = nobody root
read raw=yes
write raw=yes
strict sync = no
sync always = no
server multi channel support = no
[TestShare]
comment = Files Samba Network Share
path = /pool1/TestShare
valid users = testuser
writable = yes
browsable = yes
inherit acls = Yes
read only = No
vfs objects = zfsacl, streams_xattr, acl_xattr
nfs4: acedup = merge
nfs4: mode = special
nfs4:chown = YesIn order to be sure of my systems, I did two things :
First I created a ZFS pool using 4x1TB NVMe PCIe 4. Sharing a volume from this pool I got a stable and constant 3000MB/s in read and write, without changing any network or Samba parameters.
Then to be sure my 36 HDD pool was not the problem, I exported it then imported it on Linux and OmniOS to check if I got the same read speed problem. I don't.
With ARC cache disable, on OmniOS 151046, I got stable 1600MB/s in read and write using the kernel SMB server. On Linux with Samba, using latest TrueNas Core as it was easier for testing purposes, I got stable 2600MB/s read and 2200MB/s write.
Does anyone has an idea of what can cause this ? Or if you have any tests in mind I can run to find the reason why I got such a huge difference between internal read performances and through a Samba share.
Thanks in advance for your help