ZFS Copying a large amount of data using rsync over NFS vs SSH

degauss4

degauss4

Hello everyone,

I am running a homemade NAS server (consumer hardware with a lot of storage), and I am migrating my data that was all on an external USB disks. I tried out many methods and tests before running a production environment (including installing a base 14.1-RELEASE). Currently I am using TrueNAS Core (the one based on FreeBSD) and hosting two pools
  • /data - 4x1TB SSDs in RAIDz2
  • /media - 2x4TB HDDs in RAID1
In my first experiment I shared one of the pools over NFS, mounted the NFS share on my computer and ran rsync using as source the external hard drive and as destination the local mount. After copying around 20GB everything would slow down to a crawl and have trouble responding. TrueNAS would freeze up and not take keyboard input, my laptop (Ubuntu - KDE Neon) would also slow down and freeze, and the external hard drive would make noises (my guess, parking the head and then continuing later).
  • The network was not an issue, both computers were directly connected and I tried different cables.
  • ZFS ARC was very large (as ZFS likes to use available RAM) but I could sadly never check the amount of Free RAM available. I would wager however that there was always at least 200M+ free RAM.
  • The destination disks were high in usage but not at maximum capacity.
  • CPU was sometimes locked up in 100 sys usage. No idle whatsoever.
  • The TrueNAS server wouldnt respond to ping anymore.
My guess is that something in NFS really starts malfunctioning with large copies. I can only guess what the problem is something starts blocking CPU time (disk reads or memory paging) and then the entire system is unavailable, both the NFS server providing the share and the NFS client, resulting in maybe a "deadlocked" NFS.

After some thinking, I decided to use rsync over SSH directly. No issues whatsoever, I was able to copy over 200GB+ in one go without any freezes. I had thought before that maybe my external hard disk was at fault, but it doesn't have any SMART errors nor did it freeze while reading 200GB+ in syncing over SSH.

I am not asking for support, but I would be interested in knowing if someone ever ran into this sort of problem before. I don't think TrueNAS is at fault since in one of my early tests I installed FreeBSD on my homemade NAS and also ran into the same issue. This was some time ago, so I am sure I only tried syncing over NFS (I didnt try rsync over ssh).

If anyone is interesed I can happily provide more information about my hardware environment!
 
Can you share which kind of USB drives you are copying the data from? What sync mode do you have set for the ZFS volume you are copying to?
 
iRobbery said:
Can you share which kind of USB drives you are copying the data from? What sync mode do you have set for the ZFS volume you are copying to?
Click to expand...

Sure! At the time the TrueNAS WebUI ZFS dataset was set to "Sync: standard", meaning the software should set whether to sync or not. I am not sure what is the default for rsync, but for NFS it should default to a hard sync policy, right?

This is the output of camcontrol inquiry on the external disk and geom disk list. It's just a common WD external 4TB disk. The partition was set to NTFS. After I copy all the data in this disk to the NAS, I will use it as an external backup


pass2: <WD My Passport 2627 4008> Fixed Direct Access SPC-4 SCSI device
pass2: Serial Number [REDACTED]
pass2: 400.000MB/s transfers

Geom name: da0
Providers:
1. Name: da0
Mediasize: 5000947302400 (4.5T)
Sectorsize: 512
Stripesize: 4096
Stripeoffset: 0
Mode: r1w1e1
descr: WD My Passport 2627
lunname: WD My Passport 2627WXD2E11E1K0Y
lunid: WD My Passport 2627WXD2E11E1K0Y
ident: [REDACTED]
rotationrate: 5400
fwsectors: 63
fwheads: 255
 
I've had my share of poor performance using USB disks and copying over the network. Sometimes it even seemed more reliable to copy it to from USB to the host machine and then transfer it over the network, but space not always allows this, might not fit. I have a lacy 2 Tbyte sata (USB-C, though performs same as the WD 5 My passport) and I sometimes copy things over the network but it's like snail mail. Only decent performance is see with USB are ssd based ones. But I never had the destination lock up (freebsd nfs). Do you have any specific NFS settings, which NFS version are you using?

Sync settings _should not_ have any impact as afaik rsync doesn't issue fsync. But still might be worth setting it to disabled just for testing purpose.
 
Oh I agree, I've even had poor performance while writing or copying on the same disk I am using now without copying over the network, so it is surprising to me to see it work flawlessly with rsync over SSH. Between your last comment and now, I was copying 500GB+ and no errors or freezes. I haven't even seen a delay on the copy.

Maybe this problem is that NFS and rsync don't work to well together with a large amount of data. I don't have any specific NFS settings, using the default configuration of TrueNAS. I just mounted the same share I was syncing before and these are the options.

I didn't set any of these, I used sudo mount nas:/mnt/media/stuff /NAS/media/stuff, leaving the configuration to my Ubuntu KDE Neon. I am not too familiar with configuring and optimizing a NFS server, so I cannot help too much here


nas:/mnt/media/stuff on /NAS/media/stuff type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.0.2,mountvers=3,mountport=8
45,mountproto=udp,local_lock=none,addr=172.16.0.2
 
I typically have issues with rsync if its lots of files (from raid arrays already, cant imagine on usb sata). Have you tried using nfs4 too? Just for testing purposes?
 
iRobbery said:
Have you tried using nfs4 too?
Click to expand...
Nope, I haven't tried NFS4 either, I read in Absolute FreeBSD that NFS4 is fundamentally different due to aiming to be more like Microsoft SMB haha.

I will probably just remain syncing using SSH, it worked wonderfully so far
 
Yes privately, freebsd machine with 50T storage, old synology (which is like backup of the backup, because... performance haha, old 1511 5x8T). And professionally well I dont call them NAS, but technically they are, content servers, 'slightly' larger.

[nas]
path = /your/storage/path
readonly = false
comment = NAS Storage

This would already be enough configuration. But I never used truenas :) so maybe even has easy gui options for that.
 
That is quite a lot of storage! Are you using ZFS though? I had understood that if one used ZFS, they should set the prop sharenfs=on instead of configuring an NFS share on it
 
i have WD My Passport 24E2 - just as small back up / video storage. 4TB and 5400rpm as well.
Have really slow transfer rate.
I have 500GB nvme set up with samba for moving files from my laptop and gaimg rig and WS and WD.
All same network. Wifi to wired - took ages, 40GB took around 12-14h and had some issues would not copy files or 10bytes/s. Tried to use wired - never reached more than 30 megabits /s pool of data - mix and small files no more than 5MB max size for a file. Its usually starts quick 20-30 megabits but loads of times drops ... to less than 1. and its like a roller coaster ... very unreliable.
Even when i copy video files ( movies, tv shows ) - very very slow, even when i play directly from disk - having an issues.
I have one more 4TB WD but its dead -- need to take it apart and see some numbers, controllers ... as i think its not up to good. or maybe because its connected trough USB.
But my WD is formatted for usage on Windows as well. Now i dont have Windows so i will try to format as zfs maybe and see hopw it performs.
 
cracauer@ said:
Why do you use NFS when rsync can copy over the network?
Click to expand...
Because I was doing things without thinking :(
GlitchyDot said:
40GB took around 12-14h
Click to expand...
Ooh, that is quite bad performance. Was that from reading from the external disk or writing to it? In my case I could rsync everything fairly quickly, but sometime soon I will format that disk to use ZFS and use it as a backup as well, so I will let you know how the write performance is of around 1.5TB data.
 
degauss4 said:
Ooh, that is quite bad performance. Was that from reading from the external disk or writing to it? In my case I could rsync everything fairly quickly, but sometime soon I will format that disk to use ZFS and use it as a backup as well, so I will let you know how the write performance is of around 1.5TB data.
Click to expand...
Does not matter - reading, writing, copying.
 
You didn't mention CPU/memory specs, but if you have multiple cpus/cores, you could try running multiple rsyncs at the same time.
For example:

Code: 
rsync -a /home/[a-b]* remotehost:/home &
rsync -a /home/[c-d]* remotehost:/home &
....
rsync -a /home/[y-z]* remotehost:/home &

We do something similar, but we start each rsync in detached screen sessions.
 
You must log in or register to reply here.
Back
Top