Solved rsync blocks zfs operations and blocks input devices, unusable

[ralphbsz]

I know I'm an early adopter of FreeBSD

That's funny right there.

BSD has existed for about 40 or 50 years. FreeBSD for about 25 or 30. Quite a few people on this forum have run it for that long. I'm only at about year 15 (although I used other BSD-derived OSes in the 1980s).

Your post here is still completely unhelpful.

Here would be my suggestion if you want help debugging this: First, turn of all GUI; perform the same operations from the CLI. Make sure no other processes are running. While doing these operations, do some monitoring, like some combination of top, vmstat and iostat. Save the output of those. And then post exactly what you were doing, and how many IOs were actually running, and what the memory pressure is. You might be having an IO scheduling problem, or you might have any other problem.

 

[cracauer@]

I still can't figure out what filesystems are involved. Is there a second ZFS? What about the ext4 mentioned on the first place. Is the rsync target an internal SSD or also USB?

 

[loveydovey]

You are all not helpful.

Specific solution: Disable the sync on the dataset

rsync tries to be safe. It often asks the filesystem: "Are you SURE this is written to disk?" (fsync).
When ZFS receives this request, it must write to the ZIL (ZFS Intent Log) immediately.

You can tell ZFS: "Ignore the safety checks. Just cache it in RAM and write it when you can."

The Fix:
Set this on the Target Dataset (where you are copying TO):

zfs set sync=disabled zroot/your/target/dataset

(Warning: If the power cuts out during the copy, you lose the last 5 seconds of data. For a home copy, this is usually acceptable).

All this tells us that IOPS (Input/Output Operations Per Second) and Latency, not bandwidth, are the culprit.

Keeping sync disabled is usually not a problem. You might lose some files within the 5 seconds before abrupt power loss and your browser profile might get corrupted. But definitely not desirable to disable sync on the entire zroot, although will not corrupt your system.

Or does it make sense to disable sync on the entire zroot home folder permanently for desktop users? It's such a small fix with little risks that prevents freezing/locking up to your entire machine during write intensive tasks that are so prevalent for desktop users who would most likely write a lot into their home folders.

[By the way, nothing else worked, I tried so many different things from rtprio to kernel zfs settings, etc etc; only disabling sync works, and it makes sense now too]

 

[bakul]

if you “dd if=usb-device bs=1m of=/dev/null” while doing lots of it on your ssd zfs, do you see a similar slowdown or is it fast?

You can also play with hw.usb.xhci.use_polling and couple other sysctl to see if things improve. You can also play with scheduling parameters such as sysctl kern.sched.steal_thresh…. Websearch for settings for better interactive performance on freebsd. Or ask Gemini!

 

[bakul]

You are all not helpful.

People who argue while asking for help don’t help their cause. Better to acknowledge others who take time to respond & are trying to help you even if you don’t find their suggestions helpful.

bursts of writes to my zroot SSD ranging from 50-85 MB/s or so

You can use zpool iostat 3 for ex to get iostats reported every 3 seconds. What is the raw speed on your ssd? 50-80 MBps seems a bit low. I guess this is limited by your usb hdd?

By the way, nothing else worked, I tried so many different things from rtprio to kernel zfs settings, etc etc; only disabling sync works

Your immediate problem may be solved but I suspect this works around the underlying cause….

 

[ralphbsz]

rsync tries to be safe. It often asks the filesystem: "Are you SURE this is written to disk?" (fsync).

It does not do this by default. It only does this if you use the --fsync option on rsync. Did you use that option? If yes, it seems a bit silly first to ask rsync to write every file synchronously, and then ask ZFS to ignore the sync requests you just ordered rsync to make.

But you also said early on that the target disk (the one with the zroot using ZFS) is an SSD (you wrote SDD). On an SSD, the latency penalty for an fsync request should be just one random write, which should be about a ms; on a spinning HDD that would be about 10ms. The fsync request only happens once per file, so the distribution of file sizes being copied would be helpful to know. If all your files are large (for example 100MB each), the one extra fsync should make very little difference; if all your files are extremely small (say 2KB), the throughput penalty of waiting for that one extra write would reduce your rsync bandwidth.

You also said early on that while the rsync was running (very slowly), your system because unusable because mouse and keyboard were not responsive. Making the file system be effectively faster (because it does not wait for sync writes) should not affect mouse and keyboard at all. That's why I asked for some monitoring data: could it be that mouse and keyboard are causing other processes to perform disk-intensive operations?

Or does it make sense to disable sync on the entire zroot home folder permanently for desktop users?

How valuable is your data to you?

[By the way, nothing else worked, I tried so many different things from rtprio to kernel zfs settings, etc etc; only disabling sync works, and it makes sense now too]

It would help if you shared the measurements you took while you "tried so many things". And I don't agree with the statement that "it makes sense".

 

[atax1a]

It would help if you shared the measurements you took while you "tried so many things". And I don't agree with the statement that "it makes sense".

it would seem to me that OP doesn't actually want help, just wants to complain that freebsd isn't linux

 

[Eric A. Borisch]

gstat -pdoBI5s output (ctrl-c to end) during the slowdown can be useful to see what devices are doing.

If the SSD wasn’t trimmed before installing FreeBSD, it can significant slow down performance on a pre-used drive. Take a look zpool-trim(8).

 

[loveydovey]

If the SSD wasn’t trimmed before installing FreeBSD, it can significant slow down performance on a pre-used drive.

I trimmed it, and have cron set to trim weekly.

f you use the --fsync option on rsync. Did you use that option?

No.

while the rsync was running (very slowly), your system because unusable because mouse and keyboard were not responsive.

Rsync is running at full throttle, not slowly, and yes, input devices would freeze intermittently for a while.

It would help if you shared the measurements you took

I can replicate the problem at any point, and I will, and I will try to get some things that others have suggested a little while later.