ZFS A power outage left my ZFS file system with inaccessible files and directories.

L

LenR

My power went out while building ports via poudreire (I don't know whether it is related or not, but I figured I'd share it). Upon reboot, several files and directories are in a 'phantom' state. As far as I can tell, the files that were impacted were not being edited at the time or are frequently accessed. These files and directories are missing from the file system and, at the same time, present and cannot be altered. e.g., lswill give me the name of the file or directory in question with an the error No such file or directory. cp or mv returns File exists. To work around the problem, I moved the parent directory with phantom files in it and restored the files in question from a snapshot.

Also, zfs scrub came back clean.

Questions:
What happened?
How can I return the files to their original state or make it so they can be deleted?
I only noticed these phantom files because of the programs that broke, Is there a way I can find other phantom files?
 
cracauer@ said:
Generally there is no escape except what you already did, moving them away. There is no fsck for ZFS.
Click to expand...
Ok, I was hoping that there was something that could be done with zdb. After looking at its manpage, it didn't seem like it was going to be a simple feat.

loveydovey said:
Is this even possible with zfs' sync on? ZFS is designed specifically to avoid this. I had abrupt power offs many times, zero problems.
Click to expand...
I've had many abrupt power cycles over the years, and this is the first time I've encountered something like this.
 
LenR said:
These files and directories are missing from the file system and, at the same time, present and cannot be altered. e.g., lswill give me the name of the file or directory in question with an the error No such file or directory. cp or mv returns File exists.
Click to expand...
This sounds like your ZFS metadata is corrupted, in a rather bad way. Obviously, this should have never had happened.

There might be other explanations, but I can't think of any sensible ones right now. You can get very bizarre effects if file names contain unicode characters (for example, you might have multiple files that "look like" they have the same name, but in reality the names are different, they just render the same way on the screen). To the clueless user trying to access them, it might seem like a file both exists and doesn't exist: you do "ls" and see a file that looks like it is named "a", but "cat a" fails, because the file name is really a character from another language that just happens to look exactly like a. Usually, doing something like "ls -1 > /tmp/foo" and then examining /tmp/foo with hexdump tends to find those problems. But this is very rare.

As already said: Fixing this with zdb is impractical for most people, and probably not worth the effort.

I suspect you have found a bug. In theory, you should report it and allow some developer to access your file system to see what the exact situation is. In practice, it seems unlikely a developer will have the time to do this.

Purely idle curiosity: What version of FreeBSD (and therefore ZFS) were you running? Has your hardware ever shown any symptoms of memory errors (like random spontaneous crashes)? Does your motherboard have ECC? I'm not saying "blame the hardware", but memory errors in a metadata block are a possibility.
 
If one thinks about the way ZFS does writes, they wind up in a TXG, that is roughly queued and flushed to a device at either an interval or when it grows and hits a limit.

That means there is potential for software to think "data is written" but it hasn't reached the device yet. If you lose power at just the correct instant, you can lose data.
Now extend that to metadata and I can imagine you can get inconsistency.
By definition a build machine is doing lots of write operations.

One could setup various things to mitigate this (ZIL, SLOG on separate device, configure for synchronous writes on maybe just metadata, redundancy on the vdevs)

But I think every filesystem has the potential if subjected to sudden loss of power at the exact right instant. Think of just basic gjournal device: writes go to the journal first then the "filesystem". If you lose power when only half the data has been written to the journal, on reboot replay the journal only gets what's in the journal.

Above is just my opinion, based on my understanding of ZFS and may not reflect reality.
 
On a default installation to zroot, everything is synchronous writes (sync=standard), meaning it goes to ZIL unless the application requests otherwise.
 
This has happened here, too. I kept it:

Code: 
/mnt/lib/python3.7/ctypes # ls -l
total 25
drwxr-xr-x  3 root wheel 3 Dec 10  2020 test
/mnt/lib/python3.7/ctypes # ls -l test
ls: __pycache__: No such file or directory
drwxr-xr-x  3 root wheel 3 Dec 10  2020 .
drwxr-xr-x  3 root wheel 3 Dec 10  2020 ..
(notice the "3" in the entry count of the directory)

There is a thread about it: https://forums.freebsd.org/threads/no-longer-fun-with-upgrading-file-offline.77959/#post-486310

Strangely, back then ls could read the directory, but the file had a 'reparse' flag that didn't belong there. Now, after a couple of upgrades, it has changed to not being found at all.
 
You must log in or register to reply here.
Back
Top