ZFS WinXP app -> Samba -> zraid2 very slow. Why?

A

Anatoly_kch

I have "Grass Valley Turbo IDDR" Windows XP based appliance, which is professional HDD video recorder/player. This device records video on internal striped HDD array using MPEG-2 compression. Video stored using some special set of files, not just one *.mpg file per clip. So to get *.mpg clip out of device, I need to use "send to" menu item of supplied software, then point to some folder (windows "mapped network drive" in my case). This exporting process was never fast, about 30-40mbit/s. But recently I upgraded my fileserver from FreeBSD 10.2 to 10.4, and samba from 3.6.25 to 4.7.6, and thing slowed down to 17mbit/s. I tried different things, and found that this happens only when:
1. Writing file using supplied software, not copying some file with windows explorer for example (with explorer I getting ~170 Mbit/s)
2. Share is on ZRAID2. If share is on UFS2 partition on gmirror (same disks) speed is 50Mbit/s
3. playing with "write cache size", "min receivefile size", "socket options" in smb4.conf changes nothing.
I wonder what this chain app->samba can do to ZFS on ZRAID2 at all to slow down it so much (relatively to UFS2?)
Fileserver is in use most time, so it is hard to me to play with samba there, and I have not at the moment other machine with similar ZRAID2 to test. For a while I tested it on my desktop on single ZFS loopdevice (md0), and got no slowdown. It seems to appear on ZRAID2 only. So any suggestions will be useful, thanks.
 
So you have at least 4 disk since you have configured them as RAIDZ2?
There may be different opinions, but for 4 harddrives a RAID-Z1 (more space and faster than RAID-Z2) or striped mirrors (same space but significantly faster than RAID-Z2) are the better choice.
 
I won't say RAIDZ2 is slow in general, instead, it is blazing fast!
# dd if=tmpfs/tmpf of=/usr/home/anatoly/work/tmpf bs=1M
150+0 records in
150+0 records out
157286400 bytes transferred in 0.094759 secs (1659857331 bytes/sec)
Btw, I have 6 drives.
Problem is in that it becomes drastically slow when serving through Samba very specific Windows XP software. Any good advice where to dig is appreciated.
 
Sorry Anatoly_kch I have little to no experience with Samba and Windows, so I can't contribute to your problem.
It was just that 4 drives thing, that came to my mind... with 6 drives, Raid-Z2 of cource is a good choice.
I hope someone comes up soon who can help you further.

Matthias
 
WhIch manufacturers raid and or SAS raid daughter board/expansion card are using? As for the original posters issues with Samba/CIFS and Windows XP which service pack of Xp are you using , are these applications dependent on Xp exclusively?
 
I don't know enough about SMB (and Samba and Windows XP) and how they interact with ZFS to really help, just want to point out two things:

Anatoly_kch said:
I won't say RAIDZ2 is slow in general, instead, it is blazing fast!
# dd if=tmpfs/tmpf of=/usr/home/anatoly/work/tmpf bs=1M
150+0 records in
150+0 records out
157286400 bytes transferred in 0.094759 secs (1659857331 bytes/sec)
Click to expand...
That is a speed of 1660 MByte/second, which is pretty unrealistic for a file system with 6 drives. It works out to about 275 MByte/second per drive. Modern drives have an absolute top speed which is in that neighborhood (about 200-250 MByte/s on the outer edge), but in practice through a file system you can't get the full speed of the disk; 50-75% is considered good. Most likely, your dd command was measuring something else, like caching in RAM (either re-reading from stuff already cached, or writing to cache).

Now a suggestion. To get to good streaming performance (high throughput) through a file system, you need to do large IOs and sequential IOs. Could it be that the process of going through SMB tears IOs up into smaller parts? And that perhaps something in the back-end then forces the IOs to be done to disk too soon, without caching? Perhaps the block size of the IOs coming over the network using SMB is not well matched to the block size of RAID-Z2 and that forces the RAID layer into small writes (read-modify-write cycles) or caching reads inefficiently? Perhaps having multiple IO streams (multiple files) going in parallel defeats merging small sequential IOs into big one? Here is what I would do: Make an inventory of the block sizes of the various participants, and then study the SMB traffic (using wireshark or tcpdump), and see what the IOs really are. And just look at the IOs and get a feel for what they are (how big, with or without sync), and how many streams are active at the same time.
 
ralphbsz said:
your dd command was measuring something else, like caching in RAM
Click to expand...
Yes, I understand this, just to emphasize that Samba neutralizes caching entirely.
ralphbsz said:
with or without sync
Click to expand...
Thanks for that, didnt think about sync
Downgraded Samba to 3.6 and get my speed back... Now my "exporting process" runs at 95 Mbit/s (was 17 with samba 4.x) and Explorer file copying runs at 400 Mbit/s (was 170). Also adopted patch against "Samba cry" from samba 4...

While I feel I should play with dtrace and wireshark, still got no time... Maybe later.
It's really sad that Samba 3.x is deprecated while samba 4.x is not working with Windows XP well...
 
Anatoly_kch said:
It's really sad that Samba 3.x is deprecated while samba 4.x is not working with Windows XP well...
Click to expand...
The end of life for Samba 3 was decided by upstream, and it was kept in FreeBSD for a while past then, IIRC. Then some other changes (like devel/tevent) broke the ability to build / use Samba 3 easily (on recent-ish versions of FreeBSD, at least).

I was one of the Samba 3 holdouts because I run an app that triggers seriously sub-optimal performance in earlier Samba 4.x releases (opening lots of files in quick succession, reading a little bit, then closing them). What I was seeing was 100% of a [modern] CPU core being tied up by Samba. Samba 4.8.2 seems to have fixed this, for both that specific app and for things in general:

samba-throughput.png


Yes, that's megabytes per second.

Somewhere around Samba 4.6, the out-of-the-box config stopped working with Windows XP clients, acting as though incorrect user credentials were given. I worked around this with:
Code: 
# Additions to re-enable use from Windows XP follow
        lanman auth = yes
        ntlm auth = yes
        raw NTLMv2 auth = yes
Users should evaluate for themselves any consequences of continuing to run older, known-insecure versions of the SMB protocols. I'd say the same for XP itself, but I realize you have an expensive appliance that happens to run XP, and an upgrade may not be available for it, or you may have one of those extra-cost XP extended support plans.
 
The theoretical maximum transfer speed for CIFS/SMB on a 1Gbps network is 125MBytes/s, this is a network limit, not a storage bottleneck. So if you're on a gigabit network and are measuring more than that you're doing something wrong and are likely measuring cached data.

I downloaded a nice tool for windows to check the speed properly. I can't remember the name or the link at the moment. I'll recheck this when I get home as I recently used it to measure the speed of two Samba servers I have on my network (I ended up getting around 115-120MByte/s, very close to the max.).
 
SirDice said:
The theoretical maximum transfer speed for CIFS/SMB on a 1Gbps network is 125MBytes/s, this is a network limit, not a storage bottleneck. So if you're on a gigabit network and are measuring more than that you're doing something wrong and are likely measuring cached data.
Click to expand...
It isn't a gigabit network - it is 10GbE. The adapter in the Windows client is an Intel X540:

network-adapter.png
 
You must log in or register to reply here.
Back
Top