Raspberry Pi 4B - reboot process stops during Upgrade from FreeBSD 14.0-RELEASE-p8 to 14.1-RELEASE-p2 (with mountroot: mmcsd0 error 19). Please help.

blackfoxx

blackfoxx

Hi there :)

My name is Stan from Germany, I'm 45 y.o. and experienced/advanced FreeBSD admin/user since 2019.
(Previously admin/user of several Linux-Distros, until "systemd" occured.)
So I'm familiar with UNIX-like systems...
I'm working with FreeBSD workstations and servers since Version 12.
As a side project, I'm running a private FreeBSD webserver with Apache, sqlite, PHP, calDAV and webDAV.
This system is installed on a "SanDisk Extreme PRO" SDHC-Card, which runs within a "Raspberry Pi 4B (8GB)" at my home.
Since the initial setup in 2019 it runs just fine, without any issues or errors.
The upgrades from Version 12 to 13 and 13 to 14 went through without any problems too.
So I was very happy with my setup for about 5 years... Until last Sunday (2024-06-30).

During my regular maintenance, I decided to upgrade it from 14.0 to 14.1, as the release was already some weeks ago.
But before doing so, I updated it to the latest (14.0-RELEASE-p8) version, followed by pkg update/upgrade,
to ensure the latest versions of all installed packages too.
This pre-process went through without any problems (as usual since 2019!).

Now we're coming to the upgrade process from 14.0-RELEASE-p8 to 14.1-RELEASE-p2,
which resulted in an error (for the first time, in my case)...

Protocol:
Code: 
blackfoxx@BERRYFOXX:~ $ sudo freebsd-update upgrade -r 14.1-RELEASE
Password:
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 14.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic world/base

The following components of FreeBSD do not seem to be installed:
kernel/generic-dbg world/base-dbg

Does this look reasonable (y/n)? y

Fetching metadata signature for 14.1-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... done.
Inspecting system... done.
Fetching files from 14.0-RELEASE for merging... done.
Preparing to download files... done.
Fetching 3271 patches... (...) done.
Applying patches... done.
Fetching 444 files... (...) done.
Attempting to automatically merge changes in files... done.
The following file could not be merged automatically: /etc/ntp/leap-seconds
Press Enter to edit this file in /usr/bin/vi and resolve the conflicts
manually...
(manually merged "leap-seconds" and "sshd_config" files, according to guidelines by the FreeBSD-Forum)
The following files will be removed as part of updating to 14.1-RELEASE-p2:
(tl;dr)
The following files will be added as part of updating to 14.1-RELEASE-p2:
(tl;dr)
The following files will be updated as part of updating to 14.1-RELEASE-p2:
(tl;dr)
To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".

blackfoxx@BERRYFOXX:~ $ sudo /usr/sbin/freebsd-update install
Installing updates...
Kernel updates have been installed. Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

blackfoxx@BERRYFOXX:~ $ sudo reboot

Then the following sh*t happens:
Code: 
> Raspberry Pi 4B (8GB) Firmware boots up successfully...
> FreeBSD 14.1 Kernel + Modules are loading successfully...
> FreeBSD Loader takes over and starts to boot...
>> After a few seconds it stops booting with the message: "mmcsd0: Timeout"
>> followed by:
>>> mountroot: waiting for device /dev/mmcsd0s2a...
>>> Mounting from ufs:/dev/mmcsd0s2a failed with error 19.
>>> ? - List valid disk boot devices
>>>> mountroot> ?
>>>> mountroot: mmcsd0
>>>>> mountroot> ufs:/dev/mmcsd0s2a
>>>>> mountroot: waiting for device /dev/mmcsd0s2a...
>>>>> Mounting from ufs:/dev/mmcsd0s2a failed with error 19.
-END OF STORY- (so far)
- From now on every attempt to (re)boot fails again at this same stage.
- My "SanDisk Extreme PRO" SDHC-Card is still fine, without any (physical) errors.
- I checked it 3 times and in every possible way with my FreeBSD 14.0 Workstation.
- fsck and fsck_ffs detected NO errors or issues > file system is clean!
- I can access the card via my "Transcend" Cardreader and read all files without any issues.
- Another SD-Card, with the original "Raspberry Pi OS", boots up normally.
- If I write my previous made Backup-Image (with the FreeBSD 14.0 Installation) back to the same SD-Card, it boots up normally as it always did before.
- So any Hardware issues (regarding the RPI 4B or the SD-Card) are absolutely excluded.

Filesystem-Layout of mmcsd0:

FS_Server_2024-07-01_18-04-08.png


- I already spent 12+ hours searching the web for "Mounting from ... failed with error 19" issues and tried a dozen "solutions".
- I tried the default versions of "/boot/loader.conf", "/etc/fstab", "/etc/rc.conf" and "/etc/sysctl.conf".
- I tried the latest version of "u-boot.bin"
- I tried the latest Firmware for my RPI model, but with no luck at all...
- I wasn't able to find out what this "error 19" in fact means.

Now I reached the end of my knowledge (and patience) and hope that some (or at least one) of you experts here will be able to help me getting my server up and running again!
(But please avoid suggestions like "You have to install/setup from scratch...", as it will be the absolute last solution I wanna do.)

Thanks in advance!
 
Last edited by a moderator:
blackfoxx said:
please avoid suggestions like "You have to install/setup from scratch..."
Click to expand...
But maybe it would be good to try - on another SD card if you have one - just to make sure that FreeBSD 14.1 will definitely work on this Pi?

So at least you'll know it should definitely work (or not) before you carry on trying to fix the upgrade.
 
This was discussed last month in the mailing lists (see link below). It's likely you have a Raspberry Pi 4B with a revision ending in B0T as opposed to the newer revision ending in C0T. I use a Raspberry Pi 4B as a networking device (since the genet driver performs far better than on Linux) and had the exact same problem when upgrading from 13.2 to 13.3. Then I inserted a new memory card with 14.0 on it and upgraded to 14.1. Same result, so it affects both 13.3 and 14.1. For now I'm back to running 14.0 and it's working fine.

https://lists.freebsd.org/archives/freebsd-arm/2024-May/003924.html

I briefly checked through the bug reports but it seems none of those involved in the linked conversation have submitted one.
 
Thank you Professor_Fate !
This Information helped me (at least) to stop trying to fix here "something" which is out of my influence.
So I'm gonna stay with (fine working) 14.0 too, until a working solution is (hopefully) found...
 
Hi Blackfoxx,

Here is the bug report on the issue but doesn't look like much has been done with it: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280131

From my understanding reading through it this problem only seems to happen on the 2GB RPI4s and possibly the 1GB versions as well. Mine is a 2GB. For now I'll keep it running 14.0 and I'll try upgrading to 14.2 when that comes out. Hopefully it will be fixed in that version. I always keep an offline tar file consisting of of all the config files I edit on a system. That way if the system fails to boot after upgrade I can easily image the card back to 14.0, pkg install my programs, untar my config files, and then be back and running in a matter of minutes.
 
It looks like it is going to be wait and hope (I don't think there will be that many FreeBSD developers able to look at this and fix it? Probably countable on one hand?)

Or try another Pi or two until you find one that works. Or back to Linux for Pi-usage.

I don't think there are going to be any good solutions.

Hopefully I'll be proved wrong, but might be worthwhile coming up with a plan B.
 
Ah, okay. Thanks for your answers so far...
I was wondering a bit, as my RPI 4B model is a 4GB one. But it definitely has this problem too.
My thoughts were similar to Professor_Fate's.
As my webserver doesn't contain any sensible stuff, it doesn't really matter to let it run with 14.0 for another while.
Next attempt to upgrade would be 14.2. If it fails again for the same reasons, I put my RPI 4b into my "currently unusable electronics box" and will continue with an 1 y.o. amd64 Intel NUC (which I can buy from a friend for 50€) and a fresh FreeBSD setup from scratch.
After running the RPI 4b with FreeBSD for about 4 years now, without any issues(!), I wouldn't mind to invest some time and 50 bucks to start over again.
Maybe the amd64 platform is also less "problematic" than the arm64 one...
As I have another 2 Intel amd64 workstations running with FreeBSD, where upgrading (even to new major versions - from 12 to 13 to 14) never resulted in a mess or disaster, my impression is that amd64 is more "robust".
Back then, when I started "experimenting" with the RPI, I had no expectations. It was more kinda fun. And after 1 or 2 years in 24/7 service I already wondered that it still runs and works like a charm. So it felt like a "bonus" to me.
But maybe now it's time to move forward again and try something new...
 
For many years we used a desktop computer (with dual port intel PRO1000) as a multi vlan routing/firewall device. It simply ran PF, Unbound, and ISC-DHCP-Server. After it died I realized it was overkill for what it was doing anyway. Replaced with the Raspberry Pi 4 and the nice thing is the config files needed just a couple little changes. Even a Raspberry Pi for this use case just works with no headaches.

Going back to the first link I provided above, it seems to hint that this problem was not happening in CURRENT so it's possible it's already fixed and will trickle down to 14.2-RELEASE or maybe 15.0-RELEASE.
 
blackfoxx said:
What to do when 14.0 reaches EOL at 2024-09-30?
Click to expand...
Professor_Fate said:
Going back to the first link I provided above, it seems to hint that this problem was not happening in CURRENT so it's possible it's already fixed and will trickle down to 14.2-RELEASE or maybe 15.0-RELEASE.
Click to expand...
If I had an RPI with this issue, I would check out 14.1-STABLE. If it shows the same issue, move on to 15.0-CURRENT.

If 15.0-CURRENT boots ok, build an CURRENT image for production use, excluding debugging features and use it until 15.0-STABLE and ultimately 15.0-RELEASE.
 
Thanks for your hints and suggestions! I will give them a try a.s.a.p.
Of course I wouldn't like to replace an actually working device with an kinda overkill one. That's not my philosophy.
For the moment I just hope that 14.2 or 15.0 will work again like 14.0.
We'll see...
 
richardtoohey2 said:
It looks like it is going to be wait and hope (I don't think there will be that many FreeBSD developers able to look at this and fix it? Probably countable on one hand?)

Or try another Pi or two until you find one that works. Or back to Linux for Pi-usage.

I don't think there are going to be any good solutions.

Hopefully I'll be proved wrong, but might be worthwhile coming up with a plan B.
Click to expand...
Plan B? You could use a high quality S10 USB flash drive (32GB are cheap now, hard to find 8GB and 16GB) and install your FreeBSD image there. Change the EEProm value to 0x0F14 or 0xF214 with RaspiConfig utility to boot from the USB Flash drive or SSD (0x4) before trying to boot from a MicroSD card. Use the UGREEN case with the M.2 NVME stick enclosed inside and hook the USB 3.0 cable to the Blue USB 3.0 ports on the Raspberry Pi 4B.

Has anybody compiled the raspiconfig source code to work directly as a FreeBSD application?

https://ghostbsd-arm64.blogspot.com/2022/09/freebsd-140-compiling-kernel-for.html FreeBSD 14.x compiling kernel on Raspberry Pi.
https://ghostbsd-arm64.blogspot.com/2024/01/hdmi-audio-sound-patches-into-ghostbsd.html
Adding VCHIQ HDMI Audio patches to FreeBSD or to GhostBSD Kernel source code to play audio files on the Television speakers. This is good for watching youtube videos on the HDMI Television. Will these patches get incorporated into the build image for Raspberry Pi 3/4 ??
Where to look at built binary FreeBSD images for the Raspberry Pi 3 and 4 hardware. Will there be a binary build image for the Raspberry Pi 5?

https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.1/ RPI3/4

Will the extra uart serial port hardware on the Raspberry Pi 4B have devices drivers built into the binary image file to interface on Serial Port 2 to a GPS ascii uart interface to make a local network time server? Just asking. The development tree does use 2 uart serial interface /dev/tty0 and /dev/tty1 but not tty2, tty3, tty4, tty5

On the 4 hard do test with a USB flash drive or move up to a USB SSD like the Ugreen case with the M.2 NVME stick 250, 500GB or 1 Terrabyte. Makes a great Poudriere Package builder, with NGINX webserver to serve your freshly built Arm64 Packages.
https://ghostbsd-arm64.blogspot.com/2023/10/poudriere-setup-to-compile-from.html

Wish you well on your use case for your Raspberry Pi 4B, 3B, 400 hardware. It works great as a low cost educational ARM64 (aarch64) Do-It-Yourself (DIY) workstation.
https://ghostbsd-arm64.blogspot.com/2023/12/ghostbsd-arm64-future-road-map-ideas-to.html

Anybody work out PXE booting from a local TFTP server your Raspberry Pi hardware and wrote some documentation steps to follow?
 
Hi there.

Last weekend I invested several hours trying to upgrade my RPI-4B server to 14.2-RELEASE, in hope that this issue (boot process stops with mmc error, during upgrade from 14.0 to 14.1) was fixed.
But unfortunately it wasn't and still isn't.
So I started to dig deeper, as I won't get stuck on 14.0.

My first stop was this link posted by Professor_Fate earlier here:

280131 – MMC is timing out on rpi4

bugs.freebsd.org bugs.freebsd.org

According to their investigation, it's NOT a problem with the RPI itself or any MMC/SD-card or filesystem!
"This isn't a UFS bug, but rather a problem doing I/O to the MMC card."

This problem first occured in 14.1 and 13.3 and still continues. (14.0 and 13.2 were not affected)
So several guys started to experiment with different values within the "config.txt" file.
(which exists in the "MSDOSBOOT" partition of the MMC/SD-card and controls the very basic settings of the RPI before any OS takes in)
"Somehow" the boot-process went through, without any issues, after changing the values of "total_mem=" to specific numbers:

280131 – MMC is timing out on rpi4

bugs.freebsd.org bugs.freebsd.org
"I was able to boot based on:
total_mem=1536
I've tried the total_mem=1536 with:
RPi4B Rev 1.1 "B0T" part that has 4 GiByte
RPi4B Rev 1.4 "B0T" part that has 8 GiByte
RPi4B Rev 1.5 "C0T" part that has 8 GiByte
All booted just fine."

and

280131 – MMC is timing out on rpi4

bugs.freebsd.org bugs.freebsd.org
"#Booted RPi4B just fine:
#total_mem=1536
#total_mem=1792
#total_mem=1856
#total_mem=1857
#
#Failed to boot RPi4B:
#total_mem=1858
#
#Booted RPi4B:
#total_mem=1864
#total_mem=1860
#
#Failed to boot RPi4B:
#total_mem=1982
#total_mem=1920
#total_mem=1888
#total_mem=1872"

According to this, I started to experiment with those numbers too.
As I've got the 4 GB model and don't wanna "loose" 2 GB of RAM, I started with the highest value which was signed as working: 1864.
To my surprise it really worked!
The RPI was booting up without any issue and I was able to finish the upgrade process to 14.2 (at least).
The boot log listed "real memory = 1720 MB".
As this would (may) be enough for my tiny webserver, I noticed it as "Plan B" (better 14.2 with 1720 MB RAM than 14.0 with 4 GB RAM).
For the next attempt I just doubled this "total_mem=1864" to "total_mem=3728" in config.txt.
And voilà: it booted up again without any issues!
The boot log listed "real memory = 3651 MB" this time.
As this IS enough for my webserver, I stopped here to experiment any further.
Now it runs with the recent (14.2) RELEASE version and the loss of a few MB RAM doesn't matter for my setup.

Anyway, I don't understand the connection between this "mmc error" and the value of RAM!
But as long as this "temporary solution" is working, I have no problem at all.
Despite this, I still hope that the underlying problem will be fixed at some point.

Kind regards and merry christmas...
 
blackfoxx said:
I don't understand the connection between this "mmc error" and the value of RAM!
Click to expand...
Could it be some sort of memory mapping issue? If you don't give it a specific amount of RAM, the address space/map is out of whack and it infringes on the address space needed for MMC?

Then when you specifically give it the RAM size it sticks to that part of the address space and that leaves MMC address space.

Or maybe not. 🤭
 
e.g. see https://www.rpi4os.com/part4-miniuart/

It’s because the RPi4 boots into Low Peripheral Mode by default. This maps the peripherals over the last 64mb of RAM, therefore they’re “visible to the ARM at 0x0_FEnn_nnnn”.

So if you do not set the RAM size explicitly, it (FreeBSD) will take that last 64 MB to be RAM space that's available. But actually you need that space for peripherals (like the MMC).

Just an idea, chances are it is utter hokum!
 
Thanks for sharing your thoughts richardtoohey2.
Your explanations seem logical to me, so far.
But, if they are true, there must have been a change in FreeBSD between 14.0 (boots normal, without this "total_mem=" entry in config.txt) and 14.1 (booting stops halfway with mmc-error, without this entry in config.txt), which caused this "bug".
Whatever changed between these versions lies beyond my knowledge. And for digging deeper into that matter I've got no time...
 
Yeah the Raspberry Pi 4b did a great job as a router with PF, ISC DHCP server, and Unbound. I have since abandoned it and moved on to a Protecli device since nobody seems interested in fixing this issue. Thought ARM64 was a tier 1 platform now but maybe not.
 
Professor_Fate said:
Thought ARM64 was a tier 1 platform now but maybe not.
Click to expand...
Seems like NOT.
But in my (and this special) case it doesn't really matter if the RPI runs with or without this entry in config.txt - as long as this (very simple) solution works, I've got no problem.
The trick was to find this solution.
Fixing the reason, which actually causes this problem, should lie in other (competent) people's hands.
Maybe this bug isn't important enough to them or this (temporary) solution is too simple to invest some additional time?!
 
I don't think complaining on a largely-user-populated forum is going to fix anything. There are only so many mostly volunteer developers working on FreeBSD, and a smaller portion working on arm64. The Pi4 is no longer the latest version.

Yes, it would be lovely if there were unlimited resources and everything worked, but that's not true even of Microsoft or Apple or Google products.

As you say, blackfoxx, at least there seems to be a simple workaround, so that's progress. Even if not as nice as someone with the time & skills (& willingness to work for free) to fix this properly.
 
You must log in or register to reply here.
Back
Top