8.0 -> 8.1 "panic: root mount failed"

[Caliante]

EDIT: short summary now that the problem is solved so people don't need to read through all this to get a grasp of problem and solution:

Somewhere during the upgrade the script decided that /dev/ad10* now was /dev/ad14*. Hence, after the first part of the upgrade process the machine refused to boot.

Mounting /dev/ad14s1a (previously: /dev/ad10s1a) with the livefs-disk (and from within that the 'Fixit'-program) and editing /mnt/etc/fstab to replace 'ad10' with 'ad14' solved the problem.


Hi all :e

As per:

http://www.freebsd.org/releases/8.1R/announce.html

FreeBSD Update

The freebsd-update(8) utility supports binary upgrades of i386 and amd64 systems running earlier FreeBSD releases. Systems running 7.[0123]-RELEASE, 8.0-RELEASE, 8.1-BETA1, or 8.1-RC[12] can upgrade as follows:

# freebsd-update upgrade -r 8.1-RELEASE

During this process, FreeBSD Update may ask the user to help by merging some configuration files or by confirming that the automatically performed merging was done correctly.

# freebsd-update install

The system must be rebooted with the newly installed kernel before continuing.

# shutdown -r now

After rebooting, freebsd-update needs to be run again to install the new userland components, and the system needs to be rebooted again:

# freebsd-update install

At this point, users of systems being upgraded from FreeBSD 7.3-RELEASE or earlier will be prompted by freebsd-update to rebuild all third-party applications (e.g., ports installed from the ports tree) due to updates in system libraries.

After updating installed third-party applications (and again, only if freebsd-update printed a message indicating that this was necessary), run freebsd-update again so that it can delete the old (no longer used) system libraries:

# freebsd-update install

Finally, reboot into 8.1-RELEASE:

# shutdown -r now

I did the first two:

# freebsd-update upgrade -r 8.1-RELEASE
# freebsd-update install

Yesterday evening. It fetched some 35000 patches (cool ) and after that told me to shut down. I did - both the PC and myself as I went to bed.

This morning I wanted to do the next, for the user land:

# freebsd-update install

So I started up the computer, could briefly see 'FreeBSD 8.1 RELEASE' flashing by, then a lot of normal messages, and then it hangs, as far as I can tell the last I see is something about my all in one card reader. I press ENTER and I get this message:

panic: root mount failed, startup aborted.
cpuid = 0
uptime: 1m11s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds.

It's clear it's looking for something it can not find e) but being the noob I am I have no clue what.

All I found on the net was:

http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00025.html

Where it is said:

You might try to specify this path of root partition

ufs:xbd1s1a

where xbd1s1 stands partition "a" on slice 1 on disk 1. The path to the root partition however depends on how you created your partitions at installation.

But I would need a little bit more help with that:
1. Is this the solution?
2. Where do I need to specify that?
3. How do I find out what slice/partition is what? I can't boot the system.

I would appreciate any suggestions and of course will be in your debt for it :r

Thank you in advance,

 

[Caliante]

It's getting weirder :x

This morning I booted it again. One time only did it not halt at the previous error. It halted at:

vfs.root.mountfrom=ufs:/dev/ad10s1a
vfs.root.mountfrom.options=rw

Manual root filesystem specification:
<fstype>:<device>
eg ufs:/dev/da0s1a
? List valid boot devices

Typing in ? returned me to the previously quoted text.

I tried typing in ufs:/dev/ad10s1a and got:

mountroot > ufs:/dev/ad10s1a
trying to mount root
ROOT MOUNT ERROR:
If you have invalid mount options reboot, and first try the following from the loader prompt: set vfs.root.mountfrom.options=rw, and then 
remove invalid mount options from /etc/fstab

I tried this by booting and 'escaping at the loader prompt', followed by 'boot', but then I simply ran into the same problem of the original error ('panic: root mount failed, startup aborted...automatic reboot').

I also tried booting into single user mode and into safe mode, but these both also led to the same problem.

Finally, I took my install-DVD, booted into sysinstall and had a look at the disk and the partitions. Here is something not right I think. When I look into the disk label editor I see this (apologize for the text formatting, I had to write it on a piece of paper first and now have to type it in manually in this message - sorry, the tabs don't hold in the below):


Disk ad 10, partition name ad10s1

PART      -  MOUNT     -     SIZE     -   NEWFS
ad10s1a   -  none      -     512MB    -     *
ad10s1b   -  SWAP      -     4096MB   -   SWAP
ad10s1d   -  none      -     4334MB   -     *
ad10s1e   -  none      -     512MB    -     *
ad10s1f   -  none      -     456GB    -     *

If I recall correctly, the first time I installed 8.0 and went into this menu I choose (A)auto, and at least there were mount points mentioned. So it seems due to the upgrade to 8.1 all the labels are gone? Is this reason it can't boot? (? boot says /dev/ad10s1a, so it should know enough, no?)

So if this is correct, how do I go on about restoring the disk labels? I can guess ad10s1a is /, but there are three more that currently have 'none' written, and I have no clue what names they should have. Is there any way to find out what there names used to be (taking into account I can not boot into the file system currently = I can not do it since I am stuck with the boot problems; perhaps one of you would know of a way to get the information?)

I really would appreciate some help, because I have no clue what to do next.

Thank you (thank you thank you :e) in advance,

Bye

 

[jb_fvwm2]

boot:/dev/ad10s1a

(or) with ufs after boot: also
Rereading your post you may or may not have tried it already.
In rare instances you may also need additional .ko's (see
release notes or the UPDATING in /src/ (on the web) for
changes since you first installed if the CLI does not work?

 

[Caliante]

boot:/dev/ad10s1a

(or) with ufs after boot: also
Rereading your post you may or may not have tried it already.
In rare instances you may also need additional .ko's (see
release notes or the UPDATING in /src/ (on the web) for
changes since you first installed if the CLI does not work?

Thanks for your reply. I am not sure that I understand it P)

Do you mean I need to do

boot:/dev/ad10s1a

Instead of

ufs:/dev/ad10s1a

 

[Caliante]

I looked up how FBSD does a disk partitioning in the default install.

/
swap
/tmp
/var
/usr

Since I used a default install when installing 8.0 (probably 6 months ago if not longer), my disk layout should probably be the same. So, going from there could I just go into the disk label editor and enter that sequence there (making sure newfs is toggled to 'no' to make sure the disks don't get formatted - this I learned on the fly just a minute ago)?


PART - MOUNT - SIZE - NEWFS
ad10s1a - / - 512MB - *
ad10s1b - swap - 4096MB - SWAP
ad10s1d - /tmp - 4334MB - *
ad10s1e - /var - 512MB - *
ad10s1f - /usr - 456GB - *

Or will I horrible ruin my partitions then beyond the point of repair?

If not, is it sufficient to do just this or do I need to do something else, elsewhere in the system?

Again, thank me for replying to me e:e:e)

 

[kpa]

You seem to have only a 512MB root filesystem, is it possible that the filesystem is now full after the 8.1 upgrade? At least on one of my test installs the default 512MB root filesystem was not enough to hold two different kernels.

 

[jb_fvwm2]

I am not sure of the syntax. I've used it once
but forgotten. Search maybe? But that may not
be the fix per the post above.

 

[wblock@]

I looked up how FBSD does a disk partitioning in the default install.

/
swap
/tmp
/var
/usr

You have /tmp and /var reversed there, I think.

Any time you mess with slices or partitions, you run the risk of damaging existing data.

The odd thing here is that you do manage to boot the kernel, which is on the / filesystem.

What does it do if you boot in single-user mode?

 

[Caliante]

You seem to have only a 512MB root filesystem, is it possible that the filesystem is now full after the 8.1 upgrade? At least on one of my test installs the default 512MB root filesystem was not enough to hold two different kernels.

For me anything is possible e)

But how do I find out:q

Until I have solved the boot problem, I can't not access the hard disk to see what's going on.

 

[Caliante]

I am not sure of the syntax. I've used it once
but forgotten. Search maybe? But that may not
be the fix per the post above.

Then I take it you simply meant:

 ufs:/dev/ad10s1a

Which is what I tried and which didn't work.

 

[Caliante]

You have /tmp and /var reversed there, I think.

I believe anything you say e) But how do I find out?

Any time you mess with slices or partitions, you run the risk of damaging existing data.

That's what I'm scared about, indeed. Of course, I didn't make a backup before starting the upgrade

What will happen when I just add /var /tmp etc to the disk label editor? Suppose it is wrong, will it destroy those partitions or will it simply not boot? Because if the latter then I can safely do some trial and error. If not the latter, then I should be needing another solution to find out which partition is what.

The odd thing here is that you do manage to boot the kernel, which is on the / filesystem.

This what I find surprising as well.

What does it do if you boot in single-user mode?

It doesn't make a difference; both in single user mode and in safe mode the same error occurs as when booting normally.

 

[Beastie]

You have /tmp and /var reversed there, I think.

I believe anything you say e) But how do I find out?

You can simply use a livefs disc (from the FTP, same location as disc1), mount the partitions and check their contents. It is quite straightforward. /tmp will usually hold a few directories filled with pipe and socket files. /var will have a longer hierarchy with typical directories such as cron, db, log, mail, spool, etc.

What will happen when I just add /var /tmp etc to the disk label editor?

Probably a big mess. The installation *may* continue but you will end up with a /tmp partition containing the /var directories and files and vice versa.
If that is the case, I would personally use a livefs disc to back up critical files somewhere safe and redo the installation from scratch, i.e. checking the newfs option at least for these 2 partitions.
Next time, execute the [cmd=""]mount[/cmd] command on a working system and write down the output if you cannot memorize it.

 

[wblock@]

I believe anything you say e) But how do I find out?

You can use file -s to see where it was last mounted. For example:
# file -s /dev/ada0s1e

/dev/ada0s1e: Unix Fast File system [v2] (little-endian) last mounted on /tmp, volume name tmpfs [...]

But to use it, you need a running FreeBSD system. There are FreeBSD fixit CD and memory stick images available for download from the normal sites. If you have an extra hard disk, installing a temporary FreeBSD on it would then let you work on the old disk more easily. I would physically disconnect the old disk while installing the new one, then reconnect it afterward.

What will happen when I just add /var /tmp etc to the disk label editor?

It may be fine, although it may not fix the problem. Worst case, well, imagine the worst that could happen. Also, when you commit changes, the label editor likes to rewrite /etc/fstab. I would feel safer using bsdlabel from a fixit.

If the problem is being caused by an outdated kernel module loaded by the loader, you might be able to get around that. Escape to the loader prompt at the boot menu, then
# unload
# load /boot/kernel/kernel

If that boots into single user, all you have to do is edit /boot/loader.conf to stop loading the bad module.

And this all might be caused by a coincidentally-timed hardware problem. Disk or power supply or RAM going bad, and chose the most confusing moment to do it.

 

[Caliante]

Thank you Beastie and Wblock; I will print your remarks, digest them, try them, and then report back in a couple of days after I have completely reinstalled my system knowing that I am probably going to mess up completely :e�e

No seriously, thanks, I am downloading the live disc now and will try to fix it using your instructions. Thanks again :f

 

[Caliante]

x(x(x(x(x(

 

[Caliante]

You can simply use a livefs disc (from the FTP, same location as disc1), mount the partitions and check their contents.

This is starting to get annoying x(

8.0 worked like a charm, the upgrade messed it up and look at all the time we are wasting (sorry :r)

I downloaded the livefs-iso, booted with it and did this:

mkdir /mnt3
mount /dev/ads10s1a //mnt3

It returns:

mount:/dev/ad10s1a: no such file or directory

when I went into disk label editor from within the fixit-disk it didn't show any partition whatsoever. I rebooted with the 8.0 install DVD, and in the sysinstall there it did find the partitions as I described them above.

FreeBSD turns out to be needing as many reboots and different disks as Windows, I might as well install Windows 7 on this machine x(

(Sorry, I am getting depressed. It's not like I was experimenting with some exotic tricks, I just did a simple upgrade according to the manual, on a non-tweaked FreeBSD 8.0, and on a very recently (with only quality components) built, machine that has been doing 8.0 as well as XP flawlessy for a long time).

 

[Caliante]

You can use file -s to see where it was last mounted. For example:
# file -s /dev/ada0s1e

/dev/ada0s1e: Unix Fast File system [v2] (little-endian) last mounted on /tmp, volume name tmpfs [...]

But to use it, you need a running FreeBSD system. There are FreeBSD fixit CD and memory stick images available for download from the normal sites.

I did as you suggested; I booted the livefs-disk and:

Fixit# file -s /dev/ad10s1a

Which returned:

file: could not find any magic files

Mounting the disk first also didn't solve anything since I can't mount it, as I posted in the previous post.

I will try the unloading of the kernel next.

Thanks again to all of you for trying to help me out. Now first I am going to take some anti-depression pills.

 

[wblock@]

It looks like the disk device numbering changed. So look for the current device number in /dev:
# ls /dev/ad*

If you had a custom kernel under 8.0, that may be the problem.

 

[wblock@]

Something else comes to mind: it's like this system can't see the hard drive controller after loading the kernel. Maybe due to a change in 8.1? Wouldn't think that would affect any IDE controller, but you may have something different.

 

[Caliante]

It looks like the disk device numbering changed. So look for the current device number in /dev:
# ls /dev/ad*

If you had a custom kernel under 8.0, that may be the problem.

You might very well be right

When I did the '?' after the erroneous booting, I now got this:

List of GEOM-managed disk devices:
da3 da2 da 1 da 0
ufsid/659bnn9vn39vn959nv
ntfs/tmp ntfs/work ntfs/data
NEW VOLUME ad14s1f ad14s1e ad14s1d ad14s1b ad14s1a ad10s9 ad10s8 ad10s7 ad10s6 ad10s5 ad14s1 ad14s2 ad10s1 ad14 acd0 ad10

When I booted with Fixit and did the ls, I got:

Fixit# ls /dev/ad*
/dev/ad10
/dev/ad10s1
/dev/ad10s2
/dev/ad10s5
/dev/ad10s6
/dev/ad10s7
/dev/ad10s8
/dev/ad10s9

/dev/ad14
/dev/ad14s1
/dev/ad14s1a
/dev/ad14s1b
/dev/ad14s1d
/dev/ad14s1e
/dev/ad14s1f

The fact that ad14 has a-f behind it makes me think this is now the FreeBSD disk and ad10 is now the Windows disk (PC = dual boot).

If this is the case I still need to find out which slice/partition on disk ad14 goes where, and mounting the partitions didn't work.

Thanks again for your help; much appreciated

 

[Caliante]

Something else comes to mind: it's like this system can't see the hard drive controller after loading the kernel. Maybe due to a change in 8.1? Wouldn't think that would affect any IDE controller, but you may have something different.

Should we look into this (how?) or do we proceed with the other thing (changed disks) first? I have a normal Asus mainboard, btw. Nothing 'exotic'.

 

[Beastie]

mounting the partitions didn't work.

You can try to find out the last mountpoint using wblock's suggestion (file -s).

Also, how are you mounting it? e.g. # mount /dev/ad14s1f /mnt?
What error(s) is it returning?

 

[Caliante]

You can try to find out the last mountpoint using wblock's suggestion (file -s).

Also, how are you mounting it? e.g. # mount /dev/ad14s1f /mnt?
What error(s) is it returning?

Thanks Beastie. In post #16 and #17 in this thread I posted that file -s didn't work, nor did the mounting. Both using a livefs-disk of 8.1 FreeBSD from the FTP-site.

 

[Beastie]

In post #16 and #17 in this thread I posted that file -s didn't work, nor did the mounting.

Yes I know, but that was when you were trying with ad10. Does mounting ad14 return the same errors, i.e. "no such file or directory"? That would be really abnormal since ls shows the slice and partitions do exist.

 

[Caliante]

Yes I know, but that was when you were trying with ad10. Does mounting ad14 return the same errors, i.e. "no such file or directory"? That would be really abnormal since ls shows the slice and partitions do exist.

Stupid me, you are right, I apologize. I will check what happens now. Thanks again