"device vanished" when upgrading from FreeBSD 12.2 to 13.0

M

molofishy

I could not ssh into my FreeBSD after upgrading (see attached photo -text is displayed indefinitely). I see there is a long thread about this issue already; but is there a quick fix? And if not how can I revert back to 12.2?

qVUW2HsgCrUBdNVjsXxXwtOb8KYZE0UyQHBh8Hovgvb6LgJfPL6eqJnsyfsf9IN9DvEY70maSd0PVcmRkhBZmUTUt1GnmWfps3jGqeMPSCnsJ5ZUv8lHoc6Di1VhtvA1dkugI5J-nGz-WgomGJLFyh80-YYiFMhKUVS_81Z1l62ZjDp0XXoAAPjdpnDBBLLKH6BrbmyVvM3GwFWHyg-uvtzpgOzkBN66QnMu3g3Uwcizw-LTynyCkNBmzHcjwjQ5H_aYNNc8kla9XIE2XtRXUeLlMLRT_0S_aNVepkfBpOrk0P2AqA5db2epMZM1CFIgSYjfV1Cdathuc6xHVR1hHGwAxeH8rdmennNvEfjcHCi5EXqyGU8zoxQoJjRUFxHE3D76aGNnX5fj8c1w1CeJi1wKnnd8-Vp6MIuRrrXHjkobiMXfiHN6LdjQ9w2u5qrjE7tWf9G_QMTsJ-lM0l6Pa7ggfkMp4k3PH2XMMHGWHvmXnr5gi6nnUTUkopzD9vpw8kYXo0h8Kmft1xI3gZmIdFCGDGp3AJQMu8tXvIAIKT61OL9lKLL3jQTlADHdrgLRESaBMGhqmhLUvQ24zvXF85BT0OWzSiXhi_PuQorgEeoutCVMXVdfff2cbIZvh2C6llrPQ9dzKVIlLZuA8jrNr-MzuCwRLNDaUminRgMXHmBcGrZ19JUeGZ48W54bOOd_CIIvbAQr9K5bX7MF5g8gUj8=w1576-h1182-no

Thanks
 
Please retry attaching, or sharing, the photograph.

If remote access (ssh) is not possible, when will you have physical access?

Did you create a ZFS boot environment?
 
Last edited:
The attached photograph showed fine a few hours ago...

Having read the referenced bugs thread, it seems there is a solution (see the last message in the thread). I don't have a machine that hangs sin this manner but I at least verified that replacing 13.0 release /usr/src/sys/dev/usb/usb_hub.c with the version from the stable/13 branch doesn't break things for me. The only difference in this file is some additional error checking so I am not surprised.

molofishy has a few options, none of them ideal but not too bad either.
  1. Revert to 12.2 and wait till 13.1. The issue here is wait (assuming the fix is in 13.1).
  2. Save the current kernel and the source file usb_hub.c in a safe place cd /usr/src/; sudo make kernel. The issue here is that before doing freebsd-update you have to restore the kernel and src file and then, if there are any kernel changes, rebuild the kernel.
  3. Run the latest stable/13 snapshot kernel. Generally newer kernel deal with older userland just fine. The issue here is that freebsd-update won't work well if there are any kernel changes. You will have to find a later snapshot and install it.
Note that there is some danger of foot-shooting with 2. but if you are comfortable building kernels locally, it is something worth trying. I'd definitely recommend using boot environment. Either with bectl (if root on zfs) or beadm(if root on ufs?). 3. is easier than 2. but protection against some future exploit may be an issue for a few days.
 
The picture was there, went away, and is now back - but in case it gets shy again, I've manually copied (don't think any typos!) down the message - basically this repeated forever
Code: 
uhub_reattach_port: giving up port 1 reset - device vanished: change 0xfb status 0x7fb
uhub_reattach_port: giving up port 2 reset - device vanished: change 0xfb status 0x7fb
uhub_reattach_port: giving up port 3 reset - device vanished: change 0xfb status 0x7fb
uhub_reattach_port: giving up port 4 reset - device vanished: change 0xfb status 0x7fb
 
 
bakul said:
The attached photograph showed fine a few hours ago...

Having read the referenced bugs thread, it seems there is a solution (see the last message in the thread). I don't have a machine that hangs sin this manner but I at least verified that replacing 13.0 release /usr/src/sys/dev/usb/usb_hub.c with the version from the stable/13 branch doesn't break things for me. The only difference in this file is some additional error checking so I am not surprised.

molofishy has a few options, none of them ideal but not too bad either.
  1. Revert to 12.2 and wait till 13.1. The issue here is wait (assuming the fix is in 13.1).
  2. Save the current kernel and the source file usb_hub.c in a safe place cd /usr/src/; sudo make kernel. The issue here is that before doing freebsd-update you have to restore the kernel and src file and then, if there are any kernel changes, rebuild the kernel.
  3. Run the latest stable/13 snapshot kernel. Generally newer kernel deal with older userland just fine. The issue here is that freebsd-update won't work well if there are any kernel changes. You will have to find a later snapshot and install it.
Note that there is some danger of foot-shooting with 2. but if you are comfortable building kernels locally, it is something worth trying. I'd definitely recommend using boot environment. Either with bectl (if root on zfs) or beadm(if root on ufs?). 3. is easier than 2. but protection against some future exploit may be an issue for a few days.
Click to expand...

You can always have multiple kernels lying around.
Option 2 is what I would do. Build the kernel with the patches and DO NOT do a freebsd-update at all, or at least until the patch makes it into the next release (13.1?).

Any patches for security issues can easily be patched in and another build of the kernel performed. It just takes a lot longer than a binary upgrade, but if you have a stable machine because of building the kernel yourself, then I know what I would opt for. :)
 
grahamperrin thanks for the correction!

mark_j said:
Build the kernel with the patches and DO NOT do a freebsd-update at all
Click to expand...
This would mean switching source based updates and not everyone would want to do that. And once you are on this path, it is hard to switch back to freebsd-update. Given that 13.0-RELEASE-p3 just came out....
 
bakul said:
grahamperrin thanks for the correction!


This would mean switching source based updates and not everyone would want to do that. And once you are on this path, it is hard to switch back to freebsd-update. Given that 13.0-RELEASE-p3 just came out....
Click to expand...
Not hard. You have just to wait until the patch is included in a release and simply use freebsd-update -r the_release_name upgrade install.

But, it's difficult to follow the security patches. When I was in this case, I just never applied one (because you have to compile the kernel and/or the world and it may be long and resource consuming).
 
It's perfectly practicable to use freebsd-update(8) to binary update 'world' only and build a custom kernel with official and custom patches from source afterwards if necessary. To evaluate necessity inspect the changes made to the tree viewed after (git) updating /usr/src.

To exclude the kernel form binary update set the 'Components' option in freebsd-update.conf(5) accordingly, (exclude source also) i.e.:
Code: 
# Components of the base system which should be kept updated.
#Components src world kernel
Components world

To "switch back" from source based kernel updates, modify /etc/freebsd-update.conf.

And to reduce kernel build time, eventually into a few minutes, depending on the CPU, enable META_MODE.
 
bakul said:
The attached photograph showed fine a few hours ago...

Having read the referenced bugs thread, it seems there is a solution (see the last message in the thread). I don't have a machine that hangs sin this manner but I at least verified that replacing 13.0 release /usr/src/sys/dev/usb/usb_hub.c with the version from the stable/13 branch doesn't break things for me. The only difference in this file is some additional error checking so I am not surprised.

molofishy has a few options, none of them ideal but not too bad either.
  1. Revert to 12.2 and wait till 13.1. The issue here is wait (assuming the fix is in 13.1).
  2. Save the current kernel and the source file usb_hub.c in a safe place cd /usr/src/; sudo make kernel. The issue here is that before doing freebsd-update you have to restore the kernel and src file and then, if there are any kernel changes, rebuild the kernel.
  3. Run the latest stable/13 snapshot kernel. Generally newer kernel deal with older userland just fine. The issue here is that freebsd-update won't work well if there are any kernel changes. You will have to find a later snapshot and install it.
Note that there is some danger of foot-shooting with 2. but if you are comfortable building kernels locally, it is something worth trying. I'd definitely recommend using boot environment. Either with bectl (if root on zfs) or beadm(if root on ufs?). 3. is easier than 2. but protection against some future exploit may be an issue for a few days.
Click to expand...
Thanks for all the support FreeBSD community!
Option 2 sounds too advanced for me (unless you can give be step by step instructions from the start screen), so I will probably try for Option 1: "Revert to 12.2".
How can I revert to 12.2 with freebsd-update if I cannot even get into any command prompt?

The start screen looks like:
Code: 
1. Boot Multi user
2. Boot Single user
3. Escape to loader prompt
4. Reboot
5. Cons: Video

Options:
6. Kernel: default/kernel (1 of 2)
7. Boot Options
1. & 2. result in the infinite "device vanished" issue.
3. provides with me a list of commands, none of which are freebsd-update
5. & 6. toggle between two options on the welcome screen: 5. (video / duel); 6. (kernel / kernel old)
7. gives me 5. Options:

Code: 
1. Back to main menu
2. Load System Defaults
3. Safe Mode
4. Single user
5. Verbose.

Thank you very much and apologies for my rudimentary questions.
 
molofishy said:
Option 2 sounds too advanced for me (unless you can give be step by step instructions from the start screen),
Click to expand...
I could give you a step by step guide but it's not a trivial process. To give you an idea what to expect see list:

- Booting a FreeBSD installer image
- Mounting the root partition of the affected sytem
- chroot(8) into affected system
- Install devel/git if not installed
- Clone the 13.0-RELEASE source code
- Fetch patches 1 and 2 (before booting FreeBSD installer, saved on the installer image or extra USB stick)
- Apply patches
- Build, install new kernel . Depending on the CPU this can take up to 1 hour or more
- Exit chroot(8)
- Reboot into system with patched kernel

And every time a security and/or errata patch is released the source code needs to be updated, the kernel rebuild. That's not a system maintenance which can be called user friendly.

If installing 12.2-RELEASE fresh isn't an option, I suggest you downgrade the affected system. If you have valuable data on the affected system it would be prudent to make a backup before first if not already done. This applies also when installing 12.2 fresh.

To the downgrade process, in short:

- Boot a FreeBSD installer image, 12.2 or 13.0 if file system is UFS2, 13.0 if ZFS
- At the FreeBSD installer 'Welcome' dialog drop to 'Shell'
- Configure the network interface with dhclient(8) <interface >
- Create a temporary mount point, i.e. /tmp/a
- Mount the affected system on that mount point. How to mount depends on the file system (UFS2 or ZFS)
- In this step, optional make a backup of the data on a mounted external disk for example
- chroot(8) into affected systems mount point
- Execute freebsd-update(8)
- After finishing downgrade exit chroot(8), reboot system

If you need more detailed instructions please ask.
 
molofishy said:
Thanks for all the support FreeBSD community!
Option 2 sounds too advanced for me (unless you can give be step by step instructions from the start screen), so I will probably try for Option 1: "Revert to 12.2".
How can I revert to 12.2 with freebsd-update if I cannot even get into any command prompt?

The start screen looks like:
Code: 
1. Boot Multi user
2. Boot Single user
3. Escape to loader prompt
4. Reboot
5. Cons: Video

Options:
6. Kernel: default/kernel (1 of 2)
7. Boot Options
1. & 2. result in the infinite "device vanished" issue.
3. provides with me a list of commands, none of which are freebsd-update
5. & 6. toggle between two options on the welcome screen: 5. (video / duel); 6. (kernel / kernel old)
7. gives me 5. Options:

Code: 
1. Back to main menu
2. Load System Defaults
3. Safe Mode
4. Single user
5. Verbose.

Thank you very much and apologies for my rudimentary questions.
Click to expand...
Sounds like you can’t even boot. The "kernel old" choice will likely boot your 12.2 kernel but old kernel and new userland program combination usually doesn't work all that well....

I think you will have to boot from a usb device or a cd or dvd using an image for 12.2. I haven't done this before but perhaps this post can help you?
forums.FreeBSD.org

Upgrade disaster 11.2-RELEASE -> 12.0-RELEASE

If 12.0-RELEASE memstick image crash then verify it's checksum and try again. If you are sure that the downloaded image is correct and still cannot boot from it then the only option you have left is to revert back to 11.2. You can also can try to update your bios it may help also. Its a good...
forums.FreeBSD.org forums.FreeBSD.org
 
molofishy said:
... from the start screen ...
How can I revert to 12.2 with freebsd-update if I cannot even get into any command prompt?
Click to expand...
If the file system is UFS2, instead booting a installer image to downgrade the affected system from there, the systems previous 12.2 kernel can be used to boot the system to a command prompt. At the boot menu ("start screen") press the 6 key, the displayed kernel menu must show kernel.old):

Code: 
6. Kernel: default/kernel.old (2 of 2)

After system boot and log in try freebsd-update rollback. Check system version with freebsd-version -kru afterwards. freebsd-version(1). If all goes well the output should show
Code: 
12.2-RELEASE-pn
12.2-RELEASE-pn
12.2-RELEASE-pn

(n = patch level)

The best solution would be a fresh 12.2-REALEASE install. But if it's not an option apply one of the methods suggested.

Again, if the data on the system is of any value, and if there are no backups made, it's strongly advised to make a backup before any downgrade operation!
 
grahamperrin said:
Thanks.



molofishy if you did not create a boot environment before things went wrong, now will be a good time to adopt the habit.

In normal circumstances: create, activate then boot an environment before making changes to it.
Click to expand...
I used ZFS only for users home directories; but the operating system was stored on a separate partition in UFS. I cannot ssh at the moment but I have physical access to the machine.
 
T-Daemon said:
If the file system is UFS2, instead booting a installer image to downgrade the affected system from there, the systems previous 12.2 kernel can be used to boot the system to a command prompt. At the boot menu ("start screen") press the 6 key, the displayed kernel menu must show kernel.old):

Code: 
6. Kernel: default/kernel.old (2 of 2)

After system boot and log in try freebsd-update rollback. Check system version with freebsd-version -kru afterwards. freebsd-version(1). If all goes well the output should show
Code: 
12.2-RELEASE-pn
12.2-RELEASE-pn
12.2-RELEASE-pn

(n = patch level)

The best solution would be a fresh 12.2-REALEASE install. But if it's not an option apply one of the methods suggested.

Again, if the data on the system is of any value, and if there are no backups made, it's strongly advised to make a backup before any downgrade operation!
Click to expand...
Booting using the kernel.old option still results in the never-ending "device vanished" text; no command line interface; and no ability to ssh in.
 
molofishy said:
Booting using the kernel.old option still results in the never-ending "device vanished" text; no command line interface; and no ability to ssh in.
Click to expand...
Then you have not many choices left. Or you boot a FreeBSD installer image and try a system downgrade, or install a fresh system.
 
Now when I reboot, the first thing I see is:
"Samsung SSD 960 PRO 2TB
S.M.A.R.T Status Bad, Backup and Replace
Press F1 to Resume..."
I had setup 2 of those 2TB SSDs, with ZFS, with 100% redundancy... surely both didn't fail.. :/

Also, I have booted from a bootable USB with 12.2 on it - the Welcome screen looks a little different, so I know it's booting from the USB... however when I Boot as Multi user (the default),the never-ending "device vanished" text appears once again ;)

Could it be that one of the SSDs is no longer plugged in properly? Hmmm
 
You must log in or register to reply here.
Back
Top