FreeBSD 13.2-RELEASE -> 14.0-RELEASE Upgrade Stuck?

I'd completely skip 13.2 as a ZFS user and just go from 13.1 to 13.3 (or soon 13.4). Though I think it got retroactively patched, 13 in general was getting progressively worse in my experience until 13.2 became nearly unusable with arc_prune calls and other performance bugs; on the flip side, I started reviewing my old performance metrics and management to find various issues and changes.

If you want intermediate steps, patch your current version, then upgrade it to the highest minor version number for your version, then push forward to the next version (normally straight to the highest minor version until documentation tells you not to). It is also wise to manually download a newer freebsd-update from a newer freebsd version before upgrading, which is normally the goal of many steps of "freebsd-update system to this version before further versions."
 
Alternatively, upgrading by building from source also avoids any "freebsd-update to the next major version is taking hours(s, even on an SSD?)", where even on slow magnetic drives it seems to install at a fine pace. Unfortunately that does require a lot of build time, but your system can stay in a ready to use state during that as long as CPU, RAM, and I/O doesn't get too loaded up with build tasks.
 
Alright so I went from 13.1 to 13.3. Took about an hour. So that’s reasonable. Let’s hope going straight to 14.1 is the same….

I literally hammered my FreeBSD 13 and it’s still alive, just wow on its redundancy.

I recently learned about “bectl”, it is your best friend for recovery. Crazy but I opted to upgrade and not fresh reinstall, my OS KDE environmental is highly customized and took 3 years in the making, don’t want all that hard work to disappear and redo again.

IMG_6088 Medium.png


IMG_6089 Medium.png


Also on the side not about the “not a directory” error, I also got that while upgrading from 13.1 to 13.3:

IMG_6087 Medium.png


Alright 50 mins passed by… and…
14.1 :):

IMG_6093 Medium.png


Seems like I have to clean all the mess 13 left over…:

IMG_6091 Large.jpeg


So if anyone curious…

It took 50 mins for 13.1 to upgrade to 13.3 and another 50 from 13.3 to 14.1. About 2 hrs take it or leave it.
I’ve learned a lot, crashed and hammered the kernel, got a basic real understanding now how ZFS File system functions and how to recover and deal with it, able to use bectl create ZFS snapshots to do quick emergency boot recovery points. Completely amazing.

I’m just shocked how fast the recovery process is. PC loads in 3 seconds, in 2 seconds I can choose which snapshot I want, and in 2 seconds FreeBSD can tell me what’s going on with the boot for the chosen snapshot.

It takes only 1 second to create a snapshot using bectl. Creating a snapshot, activating it and changing between snapshots to be the default boot ZFS drive is dead simple.

FreeBSD is inherently irresistible and resilient. I’m completely astound. I had no hopes to be able to upgrade to 14.1 (specially with over thousands of packages installed during my first tryout with FreeBSD over the years)… but I just wanted to test how good FreeBSD’s redundancy is so I gave it some more thought and it’s simply bulletproof. This FreeBSD copy will now be 8 years old (3 years from 13.1 and another 5 years while on 14.x until EOL and 15 comes out)
 
I have been experiencing this on UFS for some time when using
freebsd-update install, but I thought it was related to VM hypervisor bugs. I just ran into it again on a VPS that seems to be RedHat/KVM based, with an IDE(no NCQ) controller and 1 core.

That led me to look into the problem a little more, and discover these reports of similar effects on ZFS. (hours slow instead of hanging)

I am seeing /usr/bin/install go into D-state with a wdrain and never come back. Disk activity and CPU is 0 when this happens. I usually do with a hard reset because its a local instance, but for the VPS reboot -lqn worked.

I think there is some kind of deadlock/race condition in the kernel that is affecting UFS and ZFS differently, as I notice that install -S uses the rename() call.

Hopefully this helps someone narrow down where the problem could be or discover a way to reproduce it.
 
vfs-locking said:
I just ran into it again on a VPS that seems to be RedHat/KVM based, with an IDE(no NCQ) controller and 1 core.
Click to expand...
So there is very likely an extreme disk I/O bottleneck in place, especially if that VM runs on spinning rust.
During upgrade there are thousands of files being changed/replaced, so IOPS are crucial. The CPU going into idle is to be expected as you completely choked the disk and the CPU is always waiting for IO to finish.
 
I've seen the same problem for big, usually fast, machines with zfs on root. Some worse than others, but around 10 % or our servers had this problem, a few with really long install runs, like 16 hours.

This is from 14.0-RELEASE-p10 to 14.1-RELEASE-p4, and it has definitely nothing to do with the file vs directory name clashes mentioned above.

Rather, it stucks on each
Code: 
install -S
for a few seconds. The -S is for

Code: 
     -S      Flush each file to disk after copying.  This has a non-negligible
             impact on performance, but reduces the risk of being left with a
             partial file if the system crashes or loses power shortly after
             install runs.

the servers have little or no load, at least under 1.

Alex_K probably links to the real problem.
Will try
Code: 
sysctl vfs.zfs.dmu_offset_next_sync=0
 
The UFS and ZFS variants definitely seem to be separate bugs. The ZFS slowness could also be related to the arc_prune bug; if it is then the problem should disappear at 13.4+ or 14.1+

There is a possible workaround for the UFS deadlock bug, which is setting kern.maxphys=131072 in /boot/loader.conf
 
It may not be related to this problem but recently during installation I found that geo dns was not selecting the closest mirror and the entire installation hung up. If I select a closest mirror server it complete in a seconds.
 
I am upgrading from 14.0 to 14.2

It take hours. Every commando with 'freebsd-update',
the worst till now is the first after the shutdown.

It is not the first time. This obstructs my work. I am thinking to
go back to OpenBSD.

This command is old, it is not the first time I complain, and
I am not the only that complained. It wonders me, that developers
see no problem and are more concerned on non essential things.
 
I recently upgraded from 14.0 to 14.1. It did not take as terribly long as 13.2 to 14.0, but it still took longer than usual. So there is still something wrong.
 
Then you should definitely clear out /usr/src prior to running freebsd-update. Upgrading sources involves tens of thousands of small patches - on spinning disks with abysmally low IOPS (by todays standards) this will take forever.
If you are bound to using spinning disks for the OS for some reason, then just update the base OS and re-populate /usr/src afterwards either by downloading the tarball or via git. (Of course only if you really *need* the sources...)
 
You must log in or register to reply here.
Back
Top