Other Can't unmount md device. Umount becomes a zombie.`

[MG]

In a experimental customized FreeBSD 14.1 system, umount becomes unresponsive on the CLI. This happens at random, about once per 5 times. The device is a USB-backed md-device with a FAT32 partition that's mounted using mount_msdosfs. After a few times mount/umount, it's getting geom write errors: g_vfs_done():md6[WRITE(offset=281918976, length=16384)]error = 5. This is of later concern, but at that point, umount freezes and becomes a zombie process. Here, I can also remove the md device on forehand,using mdconfig -o force -d -u md6 but then the mount keeps on existing and can't be removed.
Without forced deleting the md-device before trying to unmount it, it returns device busy" on everything, also while using the -f option.
Is there any 'checklist' to find out what is still using a device and prevent it from being unmounted? fstat and geom status, as well as the system log don't show anything that could cause it. If a device is lost and can be forgotten, why can't I get rid of anything related to it? A frozen umount process, not doing anything and having no target device, shouldn't the system just be able to drop this, regardless of any write errors that happened? What is blocking it? A kernel subroutine with no solution?

 

[Phishfry]

shouldn't the system just be able to drop this?

Yes the system could. But you invoked the devils spawn.

g_vfs_done():md6[WRITE(offset=281918976, length=16384)]error = 5.

By this I know you are using Xorg and the trashy bsddisks and all that ilk that comes with a meta desktop.

1067  -  I      0:00.03 /usr/local/libexec/gvfsd
1072  -  S      0:00.11 /usr/local/libexec/gvfs-udisks2-volume-monitor (gvfs-ud

There are so many factors here from .Xauth to blah blah blah. Automounting garbage...

You didn't even mention the desktop env used. Obviously a GTK based one.

 

[cy@]

Look for USB device disconnects in dmesg.

 

[Phishfry]

g_vfs_done():md6[

So maybe I am guessing wrong here. Are you using command prompt only?

Maybe the g_ is for geom here. I just assumed gvfsd.

 

[MG]

Yes the system could. But you invoked the devils spawn.


By this I know you are using Xorg and the trashy bsddisks and all that ilk that comes with a meta desktop.

1067  -  I      0:00.03 /usr/local/libexec/gvfsd
1072  -  S      0:00.11 /usr/local/libexec/gvfs-udisks2-volume-monitor (gvfs-ud

There are so many factors here from .Xauth to blah blah blah. Automounting garbage...

You didn't even mention the desktop env used. Obviously a GTK based one.

It's a USB live system, able to run a minimal Xorg/openbox desktop. The md-device is a fat32 partition in a file used for a small amount of permanent storage . How is Xorg related to this? Not doing anything that depends on the graphical output, as far as I know. Just like to see whatever is keeping the md-device busy, even while the physical storage device doesn't exist anymore

 

[Phishfry]

Sorry I guessed wrong.

freebsd-src/sys/geom/geom_vfs.h at main · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src

github.com

 

[MG]

Maybe my solution is dated or too complicated. What I need is a USB filesystem that processes all operations immediately without buffering anything. It must be able to survive a full system crash after which only the data of the operation in progress can be lost.

 

[MG]

Look for USB device disconnects in dmesg.

It seems to re-initialize after the error flood, or that wasn't somehow finished yet before it. I can't see what happened. My kernel message buffer is on 5MB

[190] ...
[191] ...
[192] g_vfs_done():md6[WRITE(offset=249052672, length=16384)]error = 5
[192] g_vfs_done():md6[WRITE(offset=249085440, length=16384)]error = 5
[192] g_vfs_done():md6[WRITE(offset=249118208, length=16384)]error = 5
[213] ugen0.10: <Kingston DataTraveler 3.0> at usbus0
[213] umass0 on uhub0
[213] umass0: <Kingston DataTraveler 3.0, class 0/0, rev 3.20/1.10, addr 12> on usbus0
[213] umass0:  SCSI over Bulk-Only; quirks = 0xc000
[213] umass0:1:0: Attached to scbus1
[213] (probe0:umass-sim0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
[213] (probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
[213] (probe0:umass-sim0:0:0:0): SCSI status: Check Condition
[213] (probe0:umass-sim0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
[213] (probe0:umass-sim0:0:0:0): Error 22, Unretryable error
[213] da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
da0: <Kingston DataTraveler 3.0 0000> Removable Direct Access SPC-4 SCSI device
da0: Serial Number E0D55EA574BB1591694F74F1
da0: 400.000MB/s transfers
da0: 59120MB (121077761 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>

(This happens witn any USB-drive, so it's not the condition)

 

[cy@]

Yes but this can make any filesystem on USB hang.

 

[ralphbsz]

Maybe my solution is dated or too complicated. What I need is a USB filesystem that processes all operations immediately without buffering anything. It must be able to survive a full system crash after which only the data of the operation in progress can be lost.

So you are saying that any write from an application (even if it is 1 byte) must immediately do a read-modify-write cycle on the USB disk, and then immediately update the inode (to reflect the mtime or atime change), with another R-M-W? Sorry, but performance and flash wear out make this impractical.

Can you instead explain what you really want?

 

[MG]

So you are saying that any write from an application (even if it is 1 byte) must immediately do a read-modify-write cycle on the USB disk, and then immediately update the inode (to reflect the mtime or atime change), with another R-M-W? Sorry, but performance and flash wear out make this impractical.

Can you instead explain what you really want?

The data is a few numbers in ASCII, entered with a touch display. It must be instantly written to disk after it's given by the user, so that we don't lose anything at a crash or power loss except what's being typed at that moment.

 

[richardtoohey2]

It must be instantly written to disk after it's given by the user, so that we don't lose anything at a crash or power loss except what's being typed at that moment.

I'm no expert, but USB doesn't sound like it meets any of these criteria?

 

[MG]

I'm no expert, but USB doesn't sound like it meets any of these criteria?

You're bound to the access time plus what's required to write 1 block. That's way less than a second. The USB hub must be fully available.

 

[richardtoohey2]

My experience is that USB is flakier in general (just after Bluetooth), and that's across operating systems and hardware. USB (storage in particular) and words/phrases like "reliable" and "fully available" don't usually go together. And plenty of people posting on these forums about USB storage going to sleep (but don't think that's what you are seeing.)

But I can't add anything valuable other than my poor USB experiences so ? from me.

Sounds like an interesting project, though.

 

[ralphbsz]

The data is a few numbers in ASCII, entered with a touch display. It must be instantly written to disk after it's given by the user, so that we don't lose anything at a crash or power loss except what's being typed at that moment.

Ah, you don't want your whole file system to be instant write, just a file or a few. In that case, I would use a boring normal file system (whether UFS or ZFS is an interesting question, but it doesn't matter much), and the fsync call on that file.

Except for one thing: How are you going to guard against (a) the USB stick failing, (b) the USB stick silently ignoring writes (the firmware on consumer-grade flash and USB devices is somewhere between awful and atrocious), (c) the USB subsystem failing during a write, and (d) the machine crashing during the write, perhaps because USB did something completely awful and unexpected?

If the data really matters to you, may I suggest that you (a) write it in multiple places, (b) make sure these places are not in the same failure domain (so for example one is on the other side of a network), (c) think about a protocol or data format for writing which makes sure that when reading them back you know what the last write was, and reconcile it with other information.

In a nutshell, what you are implementing is a log. Logging is a well-established discipline, and there is ample knowledge about how to do it right.

I'm no expert, but USB doesn't sound like it meets any of these criteria?

Agree. USB, in particular with consumer-grade hardware is about the last thing I would try for data that needs to be reliably written and stored.

 

[MG]

Agree. USB, in particular with consumer-grade hardware is about the last thing I would try for data that needs to be reliably written and stored.

Well, it's a cash register system that I coded 10 years ago. Never had any reliability problems with USB since +/- FreeBSD 9. At that time the stack wasn't really stable but it got much better as well as flash memory.
I'm currently working on a improvement that's supposed to make it possible to detach the boot-volume while it's running by keeping in sync a secondary 'fallback' filesystem, which is a malloc-based md device or ramdrive. This already works without problems for some time, but not including permanent local storage. For compatibility reasons, the data is stored on FAT32. Disadvantage is that you have no symlinks. I need a few nullfs-mounts to solve that.

Another option is to boot everything so that the system's default storage is a ramdrive while permanently syncing with a USB-backed filesystem, but that requires significant changes to the boot procedure that will take more time to implement. It also can cause unexpected problems that are found late. I wish I had a testing team. That would allow a better development strategy.

 

[fmc000]

My experience is that USB is flakier in general (just after Bluetooth), and that's across operating systems and hardware. USB (storage in particular) and words/phrases like "reliable" and "fully available" don't usually go together.

Broacom/Brocade switches and directors use USB mass storage for their firmware images and they are pretty reliable in my experience.

 

[ralphbsz]

Broacom/Brocade switches and directors use USB mass storage for their firmware images and they are pretty reliable in my experience.

Actually, using USB sticks for booting is common in industrial-grade computers. But not using consumer-grade hardware; the way I've seen it done is with carefully controlled firmware (BIOS), extensive testing of the OS (typically Linux), using a dedicated USB port on the motherboard, and most importantly a high quality flash device. I've not heard of anyone writing on these devices, since they are typically difficult to replace (would require opening the server or chassis).

The OP seems to be having a problem somewhere on the USB stack (could be the Kingston device itself, could be upstream), which the OS doesn't handle gracefully and hangs. There is probably an IO request that can not be completed, yet the code a few levels up (like the umount command) needs it to finish. If the OS was perfect and bug-free, this should lead to a timeout and an error being reported up, but things tend to not be perfect. And the USB stack, being very complex, is hard to get perfect.

My only suggestion: Assuming that this has worked in earlier FreeBSD versions, and is a regression introduced by 14.1. If the OP can reproduce this on a normal kernel (for example the -RELEASE version) and not an experimental system, catch a kernel dump when the hang happens, and then open a PR with the dump attached, then perhaps someone could debug and fix this.

 

[MG]

Still trying to find a decent solution. I'm now stacking up 13MB umount zombie processes, which isn't a real big problem since this systen reboots everyday, but still ugly.

Order of what happens:
A file containing a FAT32 filesystem is loaded with mdconfig and a USB-memstick as physical medium with 1 UFS filesystem where the image exists.
I somehow can't unmount this md device anymore, even if it doesn't seem to be used by anything. Umount freezes on it when I try it and remains a zombie process. What I do next is remove the md-device regardless of it being mounted using the mdconfig -o force md6 command. Then it disappears. The trying umount process keeps on existing forever, not doing anything. The md device can be recreated again with the same device number and mountpoint. When trying to unmount, it will leave another umount zombie.

Isn't there a way to delete such processes on own risk by instructing the system process management to abandon any reference to it and deallocate the memory that it used?

 

[covacat]

you can create a ramdisk and mirror it with the file backed md.
then writes will never fail from the file system point of view but if the usb drive is failing you will lose data anyway

 

[MG]

you can create a ramdisk and mirror it with the file backed md.
then writes will never fail from the file system point of view but if the usb drive is failing you will
lose data anyway

Using both. The system constantly checks and updates a malloc-based md device, so a virtual disk made of RAM. This prevents losing everything at once by drive failure. However, the initial boot-device is a USB-stick. It boots perfectly to graphical environment, only permanent storage has problems, specifically detaching it properly with umount.
Writes to a FAT32 md device on a USB (3.0) medium seems to have no probems as long as I immeidiately fsync any created or changed file. (I don't exacly know what it does on FAT32, though. Not doing it can result in files containing garbage). Is it rewriting the file allocation table?

 

[covacat]

the idea is to mirror it with gmirror and mount the gmirror dev
then even if the file backed md writes fail the gmirror will degrade but still complete the write op to the ramdisk
so you can umount without problems

 

[MG]

the idea is to mirror it with gmirror and mount the gmirror dev
then even if the file backed md writes fail the gmirror will degrade but still complete the write op to the ramdisk
so you can umount without problems

I have been trying gmirror and ZFS mirror for this some time ago. I don't remember why I dropped it. They might not do instant writing and use buffered operation, or coming back from a power failure was a problem.