FreeBSD 8.1 update disaster -- what happened?

paulfxh · Jan 14, 2011

I installed FreeBSD 8.1 from CD on a 100 GB partition on a Dell E520 a few weeks ago and it worked fine.
However, two days ago I decided to do a port upgrade and did so with the following commands:

Code:

#portsnap fetch extract update
#pkgdb -F
#portsdb -Uu
#portupgrade -a

Although this took quite a long time (maybe 18 hours), it seemed to go fine with no error messages.
However, after rebooting, I found that two of my apps would no longer launch. These were Gmail-Notify and Midori browser. Trying to launch either from a terminal gave error messages related to library issues. Although I saved the messages, for reasons I'll explain shortly, I was unable to include them in this post.
So, back to the handbook where I read in this section that after the update,

All third party software will now need to be rebuilt and re-installed. This is required as installed software may depend on libraries which have been removed during the upgrade process

OK, that seemed to describe my situation so I issued the following commands:

Code:

# portupgrade -f ruby
# rm /var/db/pkg/pkgdb.db
# portupgrade -f ruby18-bdb
# rm /var/db/pkg/pkgdb.db /usr/ports/INDEX-*.db
# portupgrade -af

Again this took a long time (20 hours) but once again proceeded without errors.
However, on reboot, after getting through the login screen without any sign of abnormality, I was presented with a totally white desktop. Although the mouse pointer was visible and moved, no context menu appeared on right-clicking.
OK, not looking good but maybe all is not lost. So I rebooted into safemode (option 3) but this gave me the following error:

Code:

Fatal trap 9: General protection fault while in kernel mode.

Then I rebooted to single user mode (option 4) and this went fine and I ended up in a tty.
The / directory showed all the usual folders but even though home was shown as one of the folders in /, I could not cd into it.
I could, however, cd into /usr but ls showed absolutely nothing in this folder. Now, that's a serious problem.
Having now run out of things to do I tried a fsck on all of the slices that should be available and got the following results:

Code:

ad4s3a   Clean
ad4s3b   Unknown FS
ad4s3c   No such file or folder
ad4s3d   Clean
ad4s3e   Clean
ad4s3f   Clean

At this point, I'm lost and am ready to do a reinstall. However, without some understanding of what went wrong during the update, I'm likely to run into the very same problem again.

Please advise.
Thanks

wblock@ · Jan 14, 2011

paulfxh said:
I installed FreeBSD 8.1 from CD on a 100 GB partition on a Dell E520 a few weeks ago and it worked fine.
However, two days ago I decided to do a port upgrade and did so with the following commands:

Code:

#portsnap fetch extract update #pkgdb -F #portsdb -Uu #portupgrade -a

Odds are good that you installed the original ports files from your install CD. The ports you installed were very old. Then suddenly you updated to the ports tree to the newest version, and did a portupgrade -a. For major upgrades of lots of ports, it's generally faster to pkg_delete all ports and reinstall them. See the end of portmaster(8) for a semi-automated procedure. (The alternative is to follow all of the steps in /usr/ports/UPDATING that come after the release date of the release you installed. They have to be done in order, too.)

If you have an internet connection, do not install the ports tree or individual ports or packages from the CD. First thing after the install, update your ports tree. Install ports normally after that, paying attention to updates in /usr/ports/UPDATING beforehand.

wblock@ · Jan 14, 2011

paulfxh said:
So, back to the handbook where I read in this section that after the update,

All third party software will now need to be rebuilt and re-installed. This is required as installed software may depend on libraries which have been removed during the upgrade process

Click to expand...

Read that again. It does not apply unless you are changing major versions (FreeBSD 7 to FreeBSD 8). Also, remember that freebsd-update(8) only updates the operating system, nothing to do with ports.

wblock@ · Jan 14, 2011

paulfxh said:
Then I rebooted to single user mode (option 4) and this went fine and I ended up in a tty.
The / directory showed all the usual folders but even though home was shown as one of the folders in /, I could not cd into it.

/home is a link to /usr/home. In single user mode, only / is mounted. If you want the other entries in /etc/fstab to be mounted, you have to mount them yourself.

I could, however, cd into /usr but ls showed absolutely nothing in this folder. Now, that's a serious problem.

/usr is just a mountpoint. It will appear empty until the actual /usr filesystem is mounted, and single-user mode doesn't do that automatically.

Having now run out of things to do I tried a fsck on all of the slices that should be available and got the following results:

ad4s3a Clean
ad4s3b Unknown FS
ad4s3c No such file or folder
ad4s3d Clean
ad4s3e Clean
ad4s3f Clean

That looks fine. "a" is mounted as /, and there is no "c". In a default setup, d is mounted on /var, e is mounted on /tmp, and f is mounted on /usr. This all happens automatically when normally booted. To get back into your system, exit the shell in single-user mode. Press ctrl-D or type exit to exit the shell and the system will boot normally.

phoenix · Jan 14, 2011

paulfxh said:
I installed FreeBSD 8.1 from CD on a 100 GB partition on a Dell E520 a few weeks ago and it worked fine.
However, two days ago I decided to do a port upgrade and did so with the following commands:

Code:

#portsnap fetch extract update

Don't do that. "fetch" will fetch the latest snapshot of the ports tree. "extract" will extract a fresh ports tree, and should be used once, the first time you run portsnap. "update" will update an existing ports tree, using data from the latest snapshot.

The first time you run portsnap is like so: # portsnap fetch extract
Everytime after that is like so: # portsnap fetch update

Code:
Code:

#portupgrade -a

Don't do that. Never blindly run -a, and especially don't run -af. Search the forums for "portupgrade -a" or "portmaster -a" for many, many, many, many long threads on why this is bad.

In fact, don't ever run portupgrade/portmaster without first looking at the output of # pkg_version -vl '<' first to see which apps have updates available, and comparing that list to the entries in /usr/ports/UPDATING.

paulfxh · Jan 14, 2011

Thanks to everybody for the replies and advice.
Seems, I took quite a few wrong turns.
Incidentally, I CAN already boot to the GUI, but the screen is totally white and non-functional apart from the mouse pointer. However, there's nothing to point to and right-click does nothing.
At this point, it seems there's no option other than to re-install FreeBSD 8.1 but this time omitting to install the ports from the CD.

wblock@ · Jan 14, 2011

If reinstalling, you might as well install 8.2-whatever. But you could also just pkg_delete everything (-a), update the ports tree, then install the ports you need.

jtl · Jan 15, 2011

In attempting to update my ports collection after upgrading from FreeBSD 7.2 to 8.1, I've run into very similar problems to paulfxh. At this point I'm ready to attempt a full reinstall of the ports and it was suggested by wblock that following the procedure in portmaster(8) would be appropriate. Since that procedure includes running portmaster with the -a option, I'm hoping to get confirmation that this is indeed a good method to use. Thanks.

wblock@ · Jan 15, 2011

Step 5 uses "-Faf", but that is not the same as "-a". It downloads all the distfiles for later install.

zzatskl · Jan 15, 2011

I'm also thinking of deleting and reinstalling the ports myself for a server in a remote machine room that I connect to using SSH. If I delete using protmaster(8) will I lose my connection to the server? TIA.

wblock@ · Jan 16, 2011

zzatskl said:
I'm also thinking of deleting and reinstalling the ports myself for a server in a remote machine room that I connect to using SSH. If I delete using protmaster(8) will I lose my connection to the server? TIA.

It depends on what ports are installed, how old they are, and probably other things. I'd recommend starting a new thread with an appropriate title (like "Upgrading all ports on remote server" to attract people who've had specific experiences with that.