UFS tar killing gmirror I/O

P

pboehmer

When I extract a large tar file , like tar -xf /usr/ports/distfiles/firefox-76.0.1.source.tar.gz, the process just grinds the whole system to a crawl. Overall system latency returns to normal after extraction is complete.

I've been fighting this issue for a while, but it never reached a high priority for me until recently. I want to say it's been happening since 11.1, but I do not have any data to back that up. I have FreeBSD 12.1 installed on a pair of Western Digital WD500AAKS drives using a 120Gb UFS gmirror (balanced round-robin), remainder of drives are configured for ZFS (mirrored pool). System is i5-7500 w/32Gb ram, ASRock Z270 mainboard. I have a couple of other systems configured about the same (mix of UFS gmirror and ZFS) and see the same issue.

I realize the above is not a lot of info, just trying to figure out where to start troubleshooting. The only disk tuning I have set in either /boot/loader.conf or /etc/sysctl.conf is just a single ZFS entry limiting ARC usage. Smart is enabled and has not logged any errors. This may be just an issue with cheap drives paired with cheaper onboard SATA controller, but I wouldn't think the system would be in such an unusable state doing a single tar extraction.
 
How big is the tar file (after decompression)? How many files does it contain (same as average file size)? How large is your system memory? Which of the two file systems (UFS on gmirror or ZFS internally mirrored) is it being extracted onto? Which one is it being read from?

My suspicion is not at the hardware level, like cheap drives or controller. The drives you are using are quite old, experienced, and boring. Their throughput should be pretty predictable. Except if they had serious internal problems (like lots of read errors and having to spend lots of time re-vectoring sectors, or suddenly running at really low temperatures and going into verify-after-write mode). I think the likely cause is that you are overwhelming the file system (one of them), but I don't know which one, and why. That's why I'm asking about memory, and Alain is suggesting watching free memory. The other thing I suspect is: If you are simultaneously reading and writing (reading tar = long sequential reads, writing file system: probably small files, lots of metadata) you might be killing the disks, by forcing them to stroke back and forth madly.
 
pboehmer said:
9664M free and 12G inactive. Don't believe it's a memory issue. Something related to I/O.
Click to expand...
Sorry, just saw that after typing. Next suggestion: run iostat on your two drives, and let's look at the number of IOs and byte throughput, separately for reads and writes.
 
Interrupts OK?

What state is the tar process in? I used to hit a suspfs issue on one machine - but think that was hardware RAID related.

Don't think related to what you are seeing but a couple of things to eliminate.
 
Please run systat -v and tell us what %busy shows for these drives during the operation.
pboehmer said:
using a 120Gb UFS gmirror (balanced round-robin), remainder of drives are configured for ZFS (mirrored pool).
Click to expand...
That figures. While zfs can do some optimization, and UFS might do similar, mixing them, just as mixing two pools, on the same device, will certainly invalidate that. So, most likely, your disks are all busy walking. And if the OS is also on these partitions, it will have to enqueue and wait a lot.
That's nothing bad, that's normal with such a layout.
General recommentation is to use complete drives for ZFS. And while I don't care about such recommendations, and did similar things, I have experienced just the same behaviour with them.

Also, when extracting such tar, the ARC is of not much use, because reading is sequential, and writing (lots of files and metadata) has to be done anyway.
Furthermore, ZFS will issue CACHE FLUSH when writing metadata, and that takes a while (watch ms/o in gstat -po) and the drive will not serve anything during that time.
 
To clarify, tar is reading and extracting to same gmirrored UFS partition. Drive walking is expected, just didn't think a single process would (should?) drag the system this much. I guess I am expecting some kind of behind-the-scenes I/O throttling to come into play.

This is a simple workstation, so not service loaded (except OS background processes, housekeeping,etc). The mix of UFS and ZFS is a configuration throwback from the FreeBSD 7.x days (before ZFS was fully implemented) . We configured systems so that the OS was always stored on UFS and data storage was on a different UFS slice that was later migrated to ZFS pools after upgrades to 8.x and on. Over the years, we just stuck with this layout as a precaution if the drives took a dump or there was a configuration issue on boot, it was easier (at the time) to recover from a broken UFS than a ZFS system. Since 11.x, we just about always use a full ZFS system on new system loads.

top shows tar process hitting wdrain and uwait state periodically. Don't think I've ever seen wdrain state before, will have to look into that.
gstat -po shows ms/o at 0 during then entire process, % busy tops at ~130%.

Will look into DTrace.
 
You must log in or register to reply here.
Back
Top