Solved Escaping from 13.0-RELEASE hell and back to 12.2-RELEASE

NuLL3rr0r

NuLL3rr0r

I had a terrible experience upgrading from 12.2-RELEASE to 13.0-RELEASE in a production environment (such as this and this). Basically, part of my mail server, calendar, contacts, and also my git server is broken. Now, I want to downgrade to 12.2-RELEASE and building from the source. In the single user mode:

Code: 
$ zfs mount -a
$ zfs set readonly=off zroot
$ adjkerntz -i
$ mergemaster -Uip
$ cd /usr/src/
$ make installworld
$ mergemaster -Ui
$ reboot

But it fails in the installworld stage with:

Code: 
make installworld
make[1]: "/usr/obj/usr/src/amd64.amd64/toolchain-metadata.mk" line 1: Using cached toolchain metadata from build at core.babaei.net on Sat Apr 17 14:17:46 UTC 2021
--------------------------------------------------------------
>>> Install check world
--------------------------------------------------------------
mkdir -p /tmp/install.fhTxfKoN
progs=$(for prog in [ awk cap_mkdb cat chflags chmod chown cmp cp  date echo egrep find grep id install   ln make mkdir mtree mv pwd_mkdb  rm sed services_mkdb sh sort strip sysctl test true uname wc zic tzsetup   makewhatis; do  if progpath=`which $prog`; then  echo $progpath;  else  echo "Required tool $prog not found in PATH." >&2;  exit 1;  fi;  done);  libs=$(ldd -f "%o %p\n" -f "%o %p\n" $progs 2>/dev/null | sort -u |  while read line; do  set -- $line;  if [ "$2 $3" != "not found" ]; then  echo $2;  else  echo "Required library $1 not found." >&2;  exit 1;  fi;  done);  cp $libs $progs /tmp/install.fhTxfKoN
Required tool makewhatis not found in PATH.
*** Error code 1

Stop.
make[1]: stopped in /usr/src
*** Error code 1

Stop.
make: stopped in /usr/src

And as simple as that, I am stuck now. I'd appreciate your help.
 
Last edited by a moderator:
In your instruction, you're missing make buildworld before "make installworld".

From what you've done, it may not have caused lasting changes. If it hasn't, using freebsd-update shouldn't be a problem from there. freebsd-update lets you upgrade versions without needing to build the world. Perhaps it will let you downgrade or roll back to 12.2.

edit: see if it can be troubleshot before attempting to downgrade.
 
NuLL3rr0r said:
I had a terrible experience upgrading from 12.2-RELEASE to 13.0-RELEASE in a production environment (such as this and this).
Click to expand...
I don't use this sync tool, but I do use nginx as a reverse proxy and didn't have any problems with 13.0-RELEASE.

I did have some problems with network configuration in general on 13.0 though. Maybe it would make sense to take a look at your network interfaces and first try to debug the problem there.

Downgrading isn't a good option. Next time you do a major upgrade, better use boot environments, so you have a safe and simple rollback plan.
 
I can relate, I had few issues with major upgrades before (it started with the 9 to 10 jump). Can I ask why don't you use booting environments ?
Personally I'd go with the fresh install to 12.2, it'll be most likely faster.
 
sidetone said:
In your instruction, you're missing make buildworld before "make installworld".

From what you've done, it may not have caused lasting changes. If it hasn't, using freebsd-update shouldn't be a problem from there. freebsd-update lets you upgrade versions without needing to build the world. Perhaps it will let you downgrade or roll back to 12.2.

edit: see if it can be troubleshot before attempting to downgrade.
Click to expand...
Sorry, I had to mention, I have already done this:

Code: 
cd /usr/src/ && make clean
cd /usr/obj/ && rm -rf *
cd /usr/src/ && make buildworld -j3 && make buildkernel -j3 KERNCONF=CUSTOM

make installkernel KERNCONF=CUSTOM
reboot

The instructions above were after rebooting and booting into the single-user mode. Thanks for the suggestion, I thought of freebsd-update and it was not installed since I disabled it inside /etc/rc.conf. But, I came up with an idea that actually worked. I'll explain later.
 
Last edited by a moderator:
Zirias said:
I don't use this sync tool, but I do use nginx as a reverse proxy and didn't have any problems with 13.0-RELEASE.

I did have some problems with network configuration in general on 13.0 though. Maybe it would make sense to take a look at your network interfaces and first try to debug the problem there.

Downgrading isn't a good option. Next time you do a major upgrade, better use boot environments, so you have a safe and simple rollback plan.
Click to expand...
Well, 13.0-RELEASE didn't work out for me, I tried the freebsd-update and it seems to work.

I am not familiar with boot environments and have no idea what is it. Thanks for mentioning it, I'll look into it.
 
_martin said:
I can relate, I had few issues with major upgrades before (it started with the 9 to 10 jump). Can I ask why don't you use booting environments ?
Personally I'd go with the fresh install to 12.2, it'll be most likely faster.
Click to expand...
Well, the funny thing is it seems I have hit the brick wall and repeated the same mistake again. I did a google search and it came up with my own thread downgrading 11.0 to 10.3. 😅

As I mentioned earlier I am not familiar with boot environments and have no idea what is it. I'll look into it. Thank you!
 
BostonBSD said:
I found out that pdfjam, from texlive, no longer works because they changed the regex(3) library. From the 13-RELEASE release notes:


Now I'm debating about whether to edit pdfjam myself, file a bug report, etc.
Click to expand...
I guess I also had to test before upgrading. In addition to the same experience from 11.0 to 10.3 release as I mentioned in the previous post, I also experienced some weired behavior from ClamAV on 12.0.
 
Boot Environments are only available if using zfs, no? They allow you to regress to a previous system state [prior to a failed upgrade].
 
A good article on it is by Mike Lucas, who always writes good articles, at

It's simple, and actually, saved me today, in a comedy of errors. First, I upgraded, and forgot to run freebsd-install before booting. So, I got kernel mismatch and it wouldn't boot. Fortunately, I had a working environment, but when I rebooted to get to that one, my CMOS battery died. Also, my wife left for Japan this morning, and looking for a small screwdriver, to pry out the battery, I had no idea where it was.
Anyway, finally got the battery out put in the new one, and as I'm in a contorted position under the desk, my wife calls, from California (we're in NY), because there was a problem with her vaccination papers--a sudden change in what was required from what she'd been told 4 days ago, but she fixed it. Which was fortunate because I'm still contorted under the desk, trying to reattach the monitor cable, since I didn't attach it tightly enough and the reboot gave me a black screen.
Pft, it was funnier in my head, but for awhile there, it seemed like a slapstick sticom---anyway, the point is, I was able to revert to my working boot environment, and all is well, and my wife was able to continue her trip to Japan,where her parents don't want to see her till she's gone through 2 weeks of quarantine. Ah well, first world problems.
 
OK, for anyone who comes across this. I had a custom build. With some options inside /etc/src.conf (for example disabling freebsd-update). First, as usual I did (except I was downgrading from 13.0 to 12.2 instead of upgrade):

Code: 
cd /usr/src/ && make clean
cd /usr/obj/ && rm -rf *
cd /usr/src/ && make buildworld -j3 && make buildkernel -j3 KERNCONF=CUSTOM

make installkernel KERNCONF=CUSTOM
reboot

Then after booting into single-user mode and mounting root and doing mergemaster stuff, I did:

Code: 
$ make installworld

Which was complaining about makewhatis tool. I tried to use freebsd-update and noticed I don't have it. But, I figured out how to install it. And, everything else in case my custom /etc/src.conf could be cause of missing anything else:

Code: 
$ fetch [URL]https://download.freebsd.org/ftp/releases/amd64/amd64/13.0-RELEASE/base.txz[/URL]
$ mkdir ./base/
$ tar xvJf base.txz -C ./base/
$ cp -vr ./base/bin/* /bin/
$ cp -vr ./base/sbin/* /sbin/
$ cp -vr ./base/usr/bin/* /usr/bin/
$ cp -vr ./base/usr/sbin/* /usr/sbin/
$ cp -v /usr/src/usr.sbin/freebsd-update/freebsd-update.conf /etc/

Then, I tried:

Code: 
$ freebsd-update -r 12.2-RELEASE upgrade
freebsd-update: Cannot upgrade from 12.2-RELEASE to itself

Well, it was right I had the kernel from 12.2 but the world was 13.0. So:

Code: 
$ fetch [URL]https://download.freebsd.org/ftp/releases/amd64/amd64/13.0-RELEASE/kernel.txz[/URL]
$ mkdir ./kernel/
$ tar xvJf kernel.txz -C /

And then rebooted, and tried freebsd-update:

Code: 
$ freebsd-update -r 12.2-RELEASE upgrade
SOME LENGTHY OUTPUT

$ /usr/sbin/freebsd-update install
Installing updates...
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

$ reboot

And, after reboot, as suggested I tried it once more:

Code: 
$ /usr/sbin/freebsd-update install
Installing updates...sysctl: unknown oid 'user.localbase'
Scanning //usr/share/certs/blacklisted for certificates...
Scanning //etc/ssl/blacklisted for certificates...
Scanning //usr/share/certs/trusted for certificates...
Skipping blacklisted certificate //usr/share/certs/trusted/GeoTrust_Global_CA.pem (//etc/ssl/blacklisted/2c543cd1.0)
Skipping blacklisted certificate //usr/share/certs/trusted/GeoTrust_Primary_Certification_Authority.pem (//etc/ssl/blacklisted/480720ec.0)
Skipping blacklisted certificate //usr/share/certs/trusted/GeoTrust_Primary_Certification_Authority_-_G3.pem (//etc/ssl/blacklisted/e2799e36.0)
Skipping blacklisted certificate //usr/share/certs/trusted/GeoTrust_Universal_CA.pem (//etc/ssl/blacklisted/ad088e1d.0)
Skipping blacklisted certificate //usr/share/certs/trusted/GeoTrust_Universal_CA_2.pem (//etc/ssl/blacklisted/8867006a.0)
Skipping blacklisted certificate //usr/share/certs/trusted/VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem (//etc/ssl/blacklisted/7d0b38bd.0)
Skipping blacklisted certificate //usr/share/certs/trusted/VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem (//etc/ssl/blacklisted/b204d74a.0)
Skipping blacklisted certificate //usr/share/certs/trusted/thawte_Primary_Root_CA.pem (//etc/ssl/blacklisted/2e4eed3c.0)
Skipping blacklisted certificate //usr/share/certs/trusted/thawte_Primary_Root_CA_-_G2.pem (//etc/ssl/blacklisted/c089bbbd.0)
Skipping blacklisted certificate //usr/share/certs/trusted/thawte_Primary_Root_CA_-_G3.pem (//etc/ssl/blacklisted/ba89ed3b.0)
Scanning //etc/ssl/certs for certificates...
unable to load certificate
34374492160:error:0909006C:PEM routines:get_name:no start line:/usr/src/crypto/openssl/crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
Error: //etc/ssl/certs/dhparam.pem
 done.
$ /usr/sbin/freebsd-update install
No updates are available to install.
Run '/usr/sbin/freebsd-update fetch' first.

Although there were some errors regarding that ssl cert (I generated that), everything seems to be fine. I am going to rebuild 12.2-RELEASE from source again using my old /etc/src.conf. If there were any issues, I will report back.
 
Last edited by a moderator:
OK, this is the first wall I am hitting 🤨

$ cd /usr/src/ && make clean
make: "/usr/src/share/mk/bsd.compiler.mk" line 167: Unable to determine compiler type for CC=/usr/local/bin/ccache cc. Consider setting COMPILER_TYPE.

make: stopped in /usr/src
Click to expand...

I have never seen this before.
 
Building a world with an altered src.conf doesn't work well with freebsd-update. It seems like freebsd-update should take care of everything, and install everything lacking from a src.conf build.

I've fixed some of this before, by going into directories under /usr/src/ and compiled missing pieces. This may not give a standardized base though. Compiling a custom base is good for learning and documenting what's there, but otherwise, it's more trouble than its worth.
 
Building a world with an altered src.conf doesn't work well with freebsd-update
Click to expand...
Well, I've just realized this by paying two days' time for it.

It seems I do not have a working CC and CXX:

Code: 
$ ls -lah /usr/bin/cc
-r-xr-xr-x  6 root  wheel     0B Apr 17 18:58 /usr/bin/cc

$ cc --version
RETURNS NOTHING!

I tried to:

Code: 
$ fetch [URL]https://download.freebsd.org/ftp/releases/amd64/amd64/12.2-RELEASE/base.txz[/URL]
$ reboot

And then in single-user mode:

Code: 
$ zfs mount -a
$ zfs set readonly=off zroot
$ adjkerntz -i

$ tar xvJpf base.txz -C base/
$ cp -rpv base/usr /

But, the file size is still zero for /usr/bin/cc:

Code: 
$ cp -rpv base/usr/bin/cc /usr/bin/cc
cp: base/usr/bin/cc: Function not implemented

$ cp -rpv base/usr/bin/cc /usr/bin/
cp: base/usr/bin/cc: Function not implemented
 
Last edited by a moderator:
At one time you could use LLVM/Clang from pkg to compile base. Then it got more difficult to do, but may have gotten easier after 13.0. The configuration may be difficult, but it's still possible to do.

For instance, setting the directory to the compiler under /usr/local/ rather than under /usr/bin/, maybe using softlinks if the compiler is hard set somewhere in /usr/share/. Hopefully, it won't need that much tinkering any more, and you can set that from make.conf.

Also, when using a custom kernel between upgrades, the settings won't match between them. More of this problem is expected with a custom src.conf for a build world. Then again, with a full upgrade from 12.2 to 13.0, shouldn't freebsd-update have taken care of everything, including what was missing from a custom build world?
 
Thank you for the detailed response. I did not upgrade from 12.2 to 13.0 using freebsd-update. I usually do it by building from the source. From 13.0 to 12.2 I did it using freebsd-update.

I don't understand why I cannot copy cc, cxx, clang, clang++, etc from base.txz file to my /usr/bin/. It's really weird. Maybe they are hard links to somewhere?
 
NuLL3rr0r said:
Thank you for the detailed response. I did not upgrade from 12.2 to 13.0 using freebsd-update. I usually do it by building from the source. From 13.0 to 12.2 I did it using freebsd-update.

I don't understand why I cannot copy cc, cxx, clang, clang++, etc from base.txz file to my /usr/bin/. It's really weird. Maybe they are hard links to somewhere?
Click to expand...

OK, I figured it out. It was a problem with symbolic links I guess.

Code: 
$ cd /
$ tar xvJpf /some/path/base.txz usr/bin
Error exit delayed from previous errors.

$ cc --version
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Target: x86_64-unknown-freebsd12.2
Thread model: posix
InstalledDir: /usr/bin

$ cpp --version
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Target: x86_64-unknown-freebsd12.2
Thread model: posix
InstalledDir: /usr/bin

$ clang --version
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Target: x86_64-unknown-freebsd12.2
Thread model: posix
InstalledDir: /usr/bin

$ clang++ --version
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Target: x86_64-unknown-freebsd12.2
Thread model: posix
InstalledDir: /usr/bin

Despite the error from tar, it seems that now I have a working compiler.
 
One more update, I had some linker errors at the beginning of make buildworld (probably because of missing shared libs from LLVM/Clang). So, I replaced the whole /usr from the archive and it's ongoing now:

Code: 
$ cd /
$ tar xvJpf /some/path/base.txz usr/
Error exit delayed from previous errors.
 
Zirias, I'm too lazy to look for the thread now, but, like some others, I have had a bit of trouble with bectl on occasion, so recommended beadm. The thread is around somewhere--it had to do with bectl, on *occasion* being unable to destroy a no longer used environment. Also, I think Mike Lucas's explanation is very clear. To anyone reading, you can sub beadm with bectl in his article and it should work.

As for my little comedy, I think I did run freebsd-update install before rebooting, and just forgetting I did, jumped to a conclusion when I saw a message about kernel mismatch. I think the mismatch was for the linux and nvidia modules. When I did it again, properly this time, boot was hanging on a kernel mismatch for Linux. So, rebooted into single user mode and set it and Nvidia, which was also giving an error, to not load at boot. Then ran freebsd-update install, then reinstalled all ports and ran freebsd-update install again and all was well.
 
OK, I guess now I can safely mark the thread as resolved. I built everything from source using the following /etc/src.conf:

Code: 
WITHOUT_ASSERT_DEBUG=yes
WITHOUT_BHYVE=yes
WITHOUT_BLUETOOTH=yes
WITHOUT_BSDINSTALL=yes
WITH_CCACHE_BUILD=yes
WITHOUT_CROSS_COMPILER=yes
WITHOUT_DEBUG_FILES=yes
WITHOUT_DICT=yes
WITHOUT_EXAMPLES=yes
WITHOUT_FLOPPY=yes
WITHOUT_FREEBSD_UPDATE=yes
WITHOUT_GAMES=yes
WITHOUT_GPIO=yes
WITHOUT_HTML=yes
WITHOUT_HYPERV=yes
WITHOUT_IPFILTER=yes
WITHOUT_ISCSI=yes
WITHOUT_JAIL=yes
WITHOUT_KDUMP=yes
WITHOUT_KERNEL_SYMBOLS=yes
WITHOUT_KVM=yes
WITHOUT_KVM_SUPPORT=yes
WITHOUT_LEGACY_CONSOLE=yes
WITHOUT_LIB32=yes
WITHOUT_LLVM_TARGET_AARCH64=yes
WITHOUT_LLVM_TARGET_ALL=yes
WITHOUT_LLVM_TARGET_ARM=yes
WITHOUT_LLVM_TARGET_MIPS=yes
WITHOUT_LLVM_TARGET_POWERPC=yes
WITHOUT_LLVM_TARGET_SPARC=yes
WITH_LLVM_TARGET_X86=yes
WITHOUT_MAN=yes
WITHOUT_MAN_UTILS=yes
WITHOUT_PC_SYSINSTALL=yes
WITHOUT_PF=yes
WITHOUT_RADIUS_SUPPORT=yes
WITHOUT_SENDMAIL=yes
WITHOUT_SHAREDOCS=yes
WITHOUT_TESTS=yes
WITHOUT_TESTS_SUPPORT=yes
WITH_USB=yes
WITHOUT_USB_GADGET_EXAMPLES=yes
WITHOUT_VI=yes
WITHOUT_WIRELESS=yes
WITHOUT_WIRELESS_SUPPORT=yes
WITHOUT_WPA_SUPPLICANT_EAPOL=yes

And then, I did the following after the final mergemaster (although there was more manual merging than upgrading) in single-user mode and a reboot:

Code: 
$ tail -n 100 /var/log/messages

And luckily, no errors from the base. So:

Code: 
$ cd /usr/src
$ make check-old
$ make -DBATCH_DELETE_OLD_FILES delete-old
$ make -DBATCH_DELETE_OLD_FILES delete-old-libs
$ cd /usr/src/ && make clean
$ cd /usr/obj/ && rm -rf *
$ pkg-static install -f pkg
$ reboot

And:

Code: 
$ tail -n 100 /var/log/messages

Without any errors. I tested even tested the ldap server, mailserver (partially fucntional), and the webserver and they are functional. The only thing I have to do is rebuild everything using either:

Code: 
$ portupgrade -afkC

Or:

Code: 
$ portmaster -badgyf --no-confirm

For the remaining non-functional packages. Hopefully, it's downhill from here on. That was a hell of a downgrade! Thank you everyone for your help and suggestions.
 
You must log in or register to reply here.
Back
Top