Solved Kernel failure left UFS2+J /usr filesystem in inconsistent state, missing '.' and '..'

M

morbit

Hello,

I had unplanned forced reboot while (un)loading various kernel modules, as usual, all files which were read/modified at the time were left with inconsistencies, after numerous fsck -yf runs system more or less works, however, missing '.' and '..' persists.

Disk is old, but has healthy S.M.A.R.T. record.

How to fix that?
Code: 
UNEXPECTED SOFT UPDATE INCONSISTENCY
CANNOT FIX, FIRST ENTRY IN THE DIRECTORY CONTAINS #12248736

UNEXPECTED SOFT UPDATE INCONSISTENCY
MISSING '.' I=6 OWNER=root MODE=40700
SIZE=51200 MTIME=Aug 25 14:30 2016
DIR=?

MISSING '..' I=6 OWNER=root MODE=40700
SIZE=51200 MTIME=Aug 25 14:30 2016
DIR=?

CANNOT FIX, SECOND ENTRY IN THE DIRECTORY CONTAINS #12248737
UNEXPECTED SOFT UPDATE INCONSISTENCY
Code: 
$ dmesg -a
<...>
Trying to mount root from ufs:/dev/ada0s1a [rw]...
Starting file system checks:
/dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ada0s1a: clean, 982629 free (725 frags, 122738 blocks, 0.1% fragmentation)
/dev/ada0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ada0s1e: clean, 5060684 free (12 frags, 632584 blocks, 0.0% fragmentation)
/dev/ada0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ada0s1f: clean, 17534386 free (107378 frags, 2178376 blocks, 0.2% fragmentation)
/dev/ada0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ada0s1d: clean, 2419890 free (14354 frags, 300692 blocks, 0.6% fragmentation)
[b]sysctl: vfs.hirunningspace=4194304 at line 18: Invalid argument[/b]
<...>

Code: 
/usr $ ls -la
total 32894
drwxr-xr-x  16 root  wheel  1024 24 Aug 16:00 .
drwxr-xr-x  22 root  wheel  1536 25 Aug 15:12 ..
drwxrwxr-x  2 root  operator  512  9 May  2009 .snap
-r--------  1 root  wheel  33554432  1 Jan  1970 .sujournal
<...>
 
Thanks for replies.

sysctl error appeared after this incident. System wanted to increase lost+found while fscking , that's all what was changed. It was last rebooted at Aug 23 (FreeBSD 11.0-PRERELEASE #0 r304673: Tue Aug 23).

I had this incident today while Firefox was running so I was expecting to loose Firefox profile, not the whole system (it had huge cache probably so I was hoping that was all that was affected judging from timestamps).

I'm using this system right now to write this and it's still running (appears to run normally, git/svn trees on /usr don't appear corrupted etc), so it can't be all bad. No loose inodes in syslog, though when I've tried to remove /usr/lost+found there was another reboot so I'm cautious (I think due to '.' and '..' problems).

Strange as it sounds, clean reinstall is not an option, though I can recompile from source in place but I do not trust system state right now still.
 
morbit said:
Strange as it sounds, clean reinstall is not an option, though I can recompile from source in place but I do not trust system state right now still.
Click to expand...
If you can't get it to the point where you can get a clean fsck(8) of all filesystems, a clean reinstall (of at least the damaged filesystems; e.g. copy undamaged contents to another fs (properly handling soft and hard links, preserving file metadata, etc), newfs(8), copy back) is the only safe option. Additionally, I would not trust "/dev/ada0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS" for the other filesystems, and force a full check of all of them from single user / rescue mode, if you have not already done so.

After file system structure has been badly damaged (to the point where fsck(8) has trouble dealing with it), nothing less than newfs(8) is guaranteed to fully eliminate trouble. Recompiling will do absolutely nothing to address underlying filesystem damage.

Shorter/alternative version: if fsck(8) does not fix it, there is no safe fix, and continuing to use it is dangerous.
 
I've already forced fsck few times in single user mode. I wonder if it did more damage though. They've all got marked clean in the end, though from my point of view /usr does not have clean record (this confusing '.' and '..' missing message, though they are obviously there). Thanks for reply.
 
morbit said:
I've already forced fsck few times in single user mode. I wonder if it did more damage though. They've all got marked clean in the end, though from my point of view /usr does not have clean record (this confusing '.' and '..' missing message, though they are obviously there). Thanks for reply.
Click to expand...
The filesystems being classed as "clean" for boot+mount purposes is not sufficient for this type of scenario. A "clean fsck(8)" means just that, absolutely no error or warning output from it, regardless of the clean state marker in the superblock or any "clean" messages on boot+mount.

I won't claim it is impossible, just normally very unlikely and rare for fsck(8) to create new damage (assuming there's no hardware faults involved). If it somehow has created new/additional damage, that's probably a symptom of existing underlying damage to the fs structures.
 
I've read about fsck(8) causing damage on the mail lists, though I'm probably reading too much into it. Thanks for response.

On the side note/rant, I've never felt I've benefited from enabling journal. Even if it was somewhat usable and fs marked clean, forced fsck could reveal inconsistencies still. On the other hand, before it I had clean record fsck runs and loose inodes too in syslog, a long time ago. It's not usual for FreeBSD to fail, but when it does, I would like to have a little less fragile native (default) fs. It looks like everybody is using ZFS now though.
 
The purpose of the journalling in SU+J is bit unclear. What it does is just to allow quick recovery of the filesystem metadata at the system start up after a crash that hasn't done too much damage. Other than that it offers just about no other benefits because in a more serious crash you are forced to run fsck(8) manually anyway. UFS isn't that fragile really but it was designed for use where the system admin would diligently run fsck(8) by hand if needed when a serious crash has occured. It's very rare to actually lose data using UFS if the problem isn't on the disk itself.
 
I always run fsck manually. Even when you had bgfsck enabled by default, because to be honest, let's say it was less than usable. I've routinely lost/got corrupted Firefox profile if it was open while system crashed (recently due to juggling sound card kernel modules, but earlier with those usb related too). Similar with other cache heavy programs. YMMV.

Anyway, I've got clean /usr fsck run by manually removing in su mode offending #12248736 and #12248737 files which were to be found under /lost+found, though I agree that clean slate/reinstall/recovery would be best course of action and I'm not surprised it was recommended here (once again, thanks for all responses, I really appreciate all of them, as they helped me to rethink my situation).

Speaking of /lost+found, I might uncover my ignorance here, but did anyone had any luck with it? It always appears to me to be filled with rubbish, and what is lost, is lost.

Still no idea about strange sysctl error, though this might be fall out due to my recent 10-STABLE-11-STABLE switch, which went unnoticed. From the look of it, vfs.hirunningspace=4194304 got merged in /etc/sysctl.conf from $deity knows which version of FreeBSD and 11 doesn't like it. I have no idea why.

Code: 
<...>
vfs.read_max=32
vfs.hirunningspace=4194304
vfs.lorunningspace=2097152
<...>

It was probably tuned a long time ago, for whatever purpose.
 
Glad to hear you managed to push it into a clean fsck. In theory, you should be safe again now, if things seem to be generally behaving themselves. Obviously any individual files which were impacted could be missing or corrupt (generally things which were not "in flight" at the time of the crash should normally be safe in this scenario).

morbit said:
Speaking of /lost+found, I might uncover my ignorance here, but did anyone had any luck with it? It always appears to me to be filled with rubbish, and what is lost, is lost.
Click to expand...

I wouldn't say it's common, but occasionally you might find an undamaged file (or part of a file) in there. A scenario where you might usefully find an undamaged file in there is when fsck has repaired a damaged directory (and that file's directory entry either wasn't there to recover or was damaged to the point that fsck discarded it). In the modern era, with soft updates and/or journaling, it's a bit less common to find a good file in there; the functionality goes back to ancient days where filesystems were less robust.

morbit said:
Still no idea about strange sysctl error, though this might be fallout due to my recent 10-STABLE-11-STABLE switch, which went unnoticed.
Click to expand...

I would guess it's probably coming from /etc/sysctl.conf. I think the best thing to do is just comment it out. That entry looks like some historical attempt at kernel tuning (might have been needed / useful in the past, or not; I can't say). My default recommendation would be to not mess with vfs.hirunningspace without a fairly good reason for it. The historical reason for tuning it may not exist on the newer kernel, and a great deal of stuff is either self-tuning or has reasonable defaults for modern hardware these days. Generally only tune sysctl stuff with a good reason for it, and revisit all tuning on a new major release (with the assumption that many historical settings may no longer be appropriate or necessary). The days of needing to do lots of kernel tuning before getting reasonable performance are generally in the past now, but there are always exceptions to that.

If you can't remember why you configured it, please kick yourself for not adding a comment beside the config. ;)
 
Excellent points all around, including the kicking ;). I (like to) think those sysctls had at some point in time some justification (at some point defaults were definitely lagging), but as it could be a long time ago, I'm going back to them (from what I see, they are a lot higher now anyway). Thanks!
 
morbit said:
Speaking of /lost+found, I might uncover my ignorance here, but did anyone had any luck with it? It always appears to me to be filled with rubbish, and what is lost, is lost.
Click to expand...

The files are information without any defining properties. From a philosophical/information studies perspective, they don't constitute "data," because they lack all the external relationships that give the information meaning. Files in lost+found are like pieces of paper with apparently random stuff written on them stuffed idly into the corner of a file cabinet. From a technical standpoint, the files have no names and the system and applications can't discern what sort of data their information represents, so sometimes they can't readily decode their contents. That's why they appear as rubbish: the filesystem doesn't have any stored information about the files' information.

There are some (tedious) steps you can take to try and determine whether the files are worth restoring or not. (The subtitle of that article pretty much answers that.) I'm glad to say I've only had one occasion where I had to pick through a couple dozen pieces of garbage to find the one that was worth restoring. :p At the very least, if some files made it to lost+found after running fsck and are important to you, you know they're most likely still in perfectly good condition.
 
kpa said:
The purpose of the journalling in SU+J is bit unclear. What it does is just to allow quick recovery of the filesystem metadata at the system start up after a crash that hasn't done too much damage. Other than that it offers just about no other benefits because in a more serious crash you are forced to run fsck(8) manually anyway.
Click to expand...
What it also did for me was to hide the fact there's still damage on the fs after a crash. Without running fsck(8) manually, I would never notice. For my test and experimentation installation of -CURRENT, I resorted to gjournal(8), which proved reliable.
 
morbit said:
I always run fsck manually. Even when you had bgfsck enabled by default, because to be honest, let's say it was less than usable.
Click to expand...
If I remember correctly the background fsck(8) runs with the -p (preen) switch. The background check can't really fix things either as the filesystems have already been mounted at that time.

Code: 
     The fsck utility invokes file system-specific programs to check the
     special devices listed in the fstab(5) file or in the command line for
     consistency.

     It is normally used in the script /etc/rc during automatic reboot.
     Traditionally, fsck is invoked before the file systems are mounted and
     all checks are done to completion at that time.  If background checking
     is available, fsck is invoked twice.  It is first invoked at the
     traditional time, before the file systems are mounted, with the -F flag
     to do checking on all the file systems that cannot do background
     checking.  It is then invoked a second time, after the system has
     completed going multiuser, with the -B flag to do checking on all the
     file systems that can do background checking.  Unlike the foreground
     checking, the background checking is started asynchronously so that other
     system activity can proceed even on the file systems that are being
     checked.
 
I might be biased, but all I remember about background fsck which I've stopped using long time ago, is that the thing was unbearably slow as well as making the system useless from my point of view, thus defeating it's purpose.
 
morbit said:
I might be biased, but all I remember about background fsck which I've stopped using long time ago, is that the thing was unbearably slow as well as making the system useless from my point of view, thus defeating it's purpose.
Click to expand...
Yeah, the "problem" with fsck(8) is that it can take a very long time to scan a large filesystem. Another issue people regularly overlook is the sixth field in fstab.
Code: 
     The sixth field, (fs_passno), is used by the fsck(8) and quotacheck(8)
     programs to determine the order in which file system and quota checks are
     done at reboot time.  The fs_passno field can be any value between 0 and
     `INT_MAX-1'.

     The root file system should be specified with a fs_passno of 1, and other
     file systems should have a fs_passno of 2 or greater.  A file system with
     a fs_passno value of 1 is always checked sequentially and be completed
     before another file system is processed, and it will be processed before
     all file systems with a larger fs_passno.

     For any given value of fs_passno, file systems within a drive will be
     checked sequentially, but file systems on different drives will be
     checked at the same time to utilize parallelism available in the hard-
     ware.  Once all file system checks are complete for the current
     fs_passno, the same process will start over for the next fs_passno.

In general I see additional filesystems all having the same fs_passno, causing them to be checked at the same time. It's usually better to use sequential numbers so the disks are checked consecutively instead of concurrently. Depending on the sizes this can actually be faster than checking them all at the same time.
 
What I've meant, is that the background fsck was a lot slower than running proper full fsck before mounting. It could be just my impression though. I will look into fs_passno, thanks.
 
SirDice said:
If I remember correctly the background fsck(8) runs with the -p (preen) switch. The background check can't really fix things either as the filesystems have already been mounted at that time.
Click to expand...

It runs a basic fsck on a snapshot of the filesystem, and logs any errors that are found to /var/log/messages. Similar to SU+J, the background fsck is meant to at least let an administrator get a mostly intact system up and running, so that it can at least be accessible and somewhat functional while the administrator takes stock of any damage and performs whatever maintenance might be necessary/possible. If no problems are found, then the system is already up and running, and that's that. And because it runs on a snapshot, it can only be used if soft updates are enabled while soft updates journaling is disabled.

Zirias: It's my understanding that gjournal(8) works just fine, but isn't really maintained and its use is not encouraged. I might be mistaken about that, though.
 
You must log in or register to reply here.
Back
Top