Solved RockPi 4 and NVMe SSD: boot through SPI flash?

dl8dtl

dl8dtl

Developer
I'm currently on the way of replacing my old Intel Atom based firewall by a RockPi 4. The main motivation for selecting the RockPi (besides it is available, unlike the Raspberry Pis) was that it features an M.2 connector so an NVMe SSD can be attached.
I have FreeBSD 13-STABLE running on it, the system can compile itself – but by now, everything still on a Micro SD card. When the system is running, I can see and use (e.g. fsck) the SSD under /dev/nvd0*.

When reading through the RockPi documentation:

Rockpi4/install/NVME - Radxa Wiki

wiki.radxa.com wiki.radxa.com

it seems in order to be able to boot from an NVMe SSD, the bootloader needs to be written to the internal SPI flash. (My RockPi does have an SPI flash mounted, just verified.)

Did anyone here do that already with FreeBSD?

Otherwise, I might follow their guide to do it through a Linux system and USB-OTG, but it would certainly be cool to do it from FreeBSD itself.
 
Writing to SPI is no big deal.
Just dd the image to /dev/flash/spi0 or whatever the addess.
The key is building u-boot for SPI.

I think SleepWalker has work done for this with rockchip. Check out his posts and website.

There is a u-boot port for that board. Maybe build the port and check the work. It may generate an SPI image.
 
Well, there is no /dev/flash/ here.
Building u-boot (including SPI modifications) is described on the Radxa site, so I guess that can be adapted to the RockPi u-boot port.
Thanks for the hint about SleepWalker, I'll have a look.
 
Here is the mods for another platform.
forums.sifive.com

Booting from Flash

I managed to get my Unmatched to boot from flash. It was a matter of enabling the SPI flash drivers in the SPL, and adding an SPI option to spl_boot_device(). Here are the diffs I applied to U-Boot: diff --git a/arch/riscv/dts/hifive-unmatched-a00-u-boot.dtsi...
forums.sifive.com forums.sifive.com

What I don't see is details about TPL=Trusted Program Loader.
That is like secure boot for Arm. I think its a seperate port and you might need to flash TPL and SPL.

Another example of using dd to write to SPI.

244146 – Unable to write to /dev/flash/spi0 on RockPro64

bugs.freebsd.org bugs.freebsd.org

If you have no SPI device showing now you might need to make an DTS overlay to enable it.
 
Another very similar platform showing the build process.:

Asus Tinker Board RK3288​




U-Boot​


  • sudo gmake CROSS_COMPILE=arm-none-eabi- tinker-rk3288_defconfig menuconfig
  • ./tools/mkimage -n rk3288 -T rksd -d ./tpl/u-boot-tpl.bin out
  • cat ./spl/u-boot-spl-dtb.bin >> out
  • sudo dd if=out of=/dev/da0 seek=64
  • dd if=u-boot-dtb.img of=/dev/da0 seek=16384
Click to expand...
 
forum.radxa.com

FreeBSD 13-CURRENT on Rock Pi4

I have got it to boot now, off uSD. I had to enable maskrom mode and remove my nvme disk. I get a (full HD) HDMI display but unfortunately its cropped and doesn’t quite fit so I cannot see the whole of the bottom row of text ie the command line itself and so I can only really use its terminal...
forum.radxa.com forum.radxa.com
 
Thanks for the ideas so far.
I'm tempted to build up the firewall using eMMC as boot device (and the SSD as the main filesystem), and rather obtain a second RockPi for all the experiments. ;-)
 
Yes shorting the SPI is possible if skilled.
I bought two RockPro64 because...

Why they put M.2 slot backwards on these RockPi3/4 boards?
 
Phishfry said:
Yes shorting the SPI is possible if skilled.
I bought two RockPro64 because...

Why they put M.2 slot backwards on these RockPi3/4 boards?
Click to expand...
If they didn't, you would stand no mount a heatsink to the chip, and also insert a card by the same time. They don't need the cooling an Intel or AMD chip needs, but they need some cooling (otherwise you can't clock them high enough).

The existing solution with the M.2 "hat" doesn't look all that bad to me.
 
Note the writeup that SleepWalker gives at the raxda forum

To install this bootloader on an SPI flash just do:
=> load mmc 0 ${kernel_addr_r} u-boot-spi.bin
=> sf probe
=> sf update ${kernel_addr_r} 0 ${filesize}
Click to expand...

So these are commands to issue to the u-boot console.

That is a real good start to see if your SPI is suported. Check the module with u-boot.
Enter u-boot console and start twisting the knobs of the sf probe u-boot commands.
The SPI has to be seen by u-boot for FreeBSD to find it.
 
Phishfry said:
Note the writeup that SleepWalker gives at the raxda forum



So these are commands to issue to the u-boot console.

That is a real good start to see if your SPI is suported. Check the module with u-boot.
Enter u-boot console and start twisting the knobs of the sf probe u-boot commands.
The SPI has to be seen by u-boot for FreeBSD to find it.
Click to expand...
Thanks, I vaguely remember that I've been doing stuff like that a couple of years ago on behalf of a customer (on an STM platform).
Btw., the SPI flash even features a /HOLD pin. When pulled low, that disables the entire flash.
 
Phishfry said:
OK now opinion time.
Intel or Chelsio interfaces for firewall.
Leave the toys at behind the gate.
Click to expand...
I don't consider these platforms toys anymore. Might have been the case some 10 years ago. The big advantage: power consumption. It draws about 1/4 of my current Atom platform, and allows for a completely fan-less design. Given that a jammed fan was one of the failure reasons of my current platform (the PSU fan in that case, which killed the PSU), this is much benefit to me.
 
dl8dtl said:
It draws about 1/4 of my current Atom platform,
Click to expand...
Well check your ethernet speeds.
Gigabit ethernet on Arm means something different than in the x86 world.
They call 300Meg/sec speed 1G Ethernet.
Then throw a packet filter on top... Some basic rules and you have real world hurt.

They are advertising 940MB/sec on these so that looks better than my past experiences and on par with x86.
 
Another solution is to just keep uboot on the sd card and point it to the nvme drive
RK3399 has no issues at all doing gbit line speed however the dwc driver may need to some love in some scenarios.
 
Phishfry said:
Well check your ethernet speeds.
Gigabit ethernet on Arm means something different than in the x86 world.
They call 300Meg/sec speed 1G Ethernet.
Then throw a packet filter on top... Some basic rules and you have real world hurt.

They are advertising 940MB/sec on these so that looks better than my past experiences and on par with x86.
Click to expand...
Remember, this is a firewall, not a superfast server. Internet speed will be the limiting factor anyway.
I just compared ssh speed of the RockPi with my current Intel Atom-based solution using a large ssh copy: while both are far away from reaching full gigabit speed, the RockPi beats the Atom by a factor of 2.5.
 
diizzy said:
Another solution is to just keep uboot on the sd card and point it to the nvme drive
RK3399 has no issues at all doing gbit line speed however the dwc driver may need to some love in some scenarios.
Click to expand...
Yes, while I would have preferred a (small) eMMC then instead of a removable card.
Anyway, the pointer to SleepWalker above was good: I joined his Telegram group, and we could quickly sort it out – basically, using the above magic from inside U-Boot, together with SleepWalker's modified U-Boot port. I'll help him to get his patches into the official tree as my time permits (it adds not only SPI flash support but also USB stick boot support).
So, my system is now starting U-Boot from the SPI flash, and then boots from the NVMe SSD without problems. I reformatted the SSD to ZFS, made good experience with that during the last 10+ years I've been using it at my desktop/server. Oh, and I should add: it went flawlessly, no need to recover from a broken bootloader on the SPI flash. ;)

I'll mark the thread as resolved.
 
dl8dtl said:
I reformatted the SSD to ZFS, made good experience with that during the last 10+ years I've been using it at my desktop/server. Oh, and I should add: it went flawlessly, no need to recover from a broken bootloader on the SPI flash. ;)
Click to expand...
You have some notes you can share about reformatting the SSD to ZFS, particularly the gpart add , gpart modify commands that you used , maybe in a script. You have 10 years experience on a desktop/server. I have experience using on GhostBSD, but not on the actually sequence of commands to install ZFS to root "/" and a second partition home "/home". I realize how to make the partitions:
https://forums.freebsd.org/threads/gpart-cheatsheet-wiping-drives-partitioning-formating.45411/ Gpart Cheat Sheet
gpart destroy -F /dev/da1
gpart create -s GPT /dev/da1
gpart add -i 1 -t fat32 -l ESP_EFI -s 3G /dev/da1
gpart add -i 5 -t freebsd-zfs -l Ghost_root -s 54G /dev/da1
gpart add -i 6 -t freebsd-zfs -l Ghost_home -s 138G /dev/da1
gpart add -i 7 -t freebsd-swap -l Ghost_swap -s 8G /dev/da1

Now some help with zpool create and zfs create commands. I will pause here. Until I look the zpool and zfs command usages. And maybe edit in some more information here.
GhostBSD has its python apps GhostBSD Installer GBI that uses shell script pc-sysintall.sh to perform the low level zpool and zfs stuff.
github.com/ghostbsd/gbi

https://www.reddit.com/r/freebsd/comments/v4b5oj/freebsd_on_zfs_on_raspberry_pi/ found this that has the partitioning
commands and the ZFS create zrpi
https://www.reddit.com/r/freebsd/comments/rzh6xh/comment/hrv68l2/ this is the basis of the script.



https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/bsdinstall/scripts/zfsboot
 
Why are you using so many partitions? The idea behind ZFS is to have a storage pool that can be divided inside ZFS.
I have just two partitions: one FAT32 for EFI, and one freebsd-zfs for the ZFS pool.

Code: 
# gpart list
Geom name: diskid/DISK-Z0HA32PEK2L2
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 488397127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: diskid/DISK-Z0HA32PEK2L2p1
   Mediasize: 52428800 (50M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 16777216
   Mode: r1w1e1
   efimedia: HD(1,GPT,0c7b65ed-a4cf-11ed-9ba6-0cc47ad8b808,0x8000,0x19000)
   rawuuid: 0c7b65ed-a4cf-11ed-9ba6-0cc47ad8b808
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: efi
   length: 52428800
   offset: 16777216
   type: efi
   index: 1
   end: 135167
   start: 32768
2. Name: diskid/DISK-Z0HA32PEK2L2p2
   Mediasize: 249990123520 (233G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 69206016
   Mode: r1w1e1
   efimedia: HD(2,GPT,0c7c0c8c-a4cf-11ed-9ba6-0cc47ad8b808,0x21000,0x1d1a4948)
   rawuuid: 0c7c0c8c-a4cf-11ed-9ba6-0cc47ad8b808
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: ZROOT
   length: 249990123520
   offset: 69206016
   type: freebsd-zfs
   index: 2
   end: 488397127
   start: 135168
Consumers:
1. Name: diskid/DISK-Z0HA32PEK2L2
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r2w2e4

(Side note: it would normally make sense to have two further partitions in front of the EFI one, for both U-Boot stages, rather than having to fiddle with dd offsets when installing U-Boot. However, on the SSD, U-Boot cannot be used anyway.)

ZFS then provides the remaining volumes:

Code: 
# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot   232G  39.7G   192G        -         -     3%    17%  1.00x    ONLINE  -
# zfs list
NAME         USED  AVAIL     REFER  MOUNTPOINT
zroot       47.9G   177G       24K  none
zroot/ROOT  39.1G   177G     39.1G  /
zroot/swap  8.25G   185G     32.7M  -
zroot/tmp   89.5K  2.00G     89.5K  /tmp
zroot/var    488M  9.52G      488M  /var
As you can see, I created the swap area as a ZFS volume as well.

The trickiest part was it to create the ZFS filesystem with a mountpoint of / while initially running from the SD card. On my first attempt, I forgot to specify the altroot attribute, ending up in an empty ZFS volume mounted on top of my SD card root FS. :-(
 
What modifications to uboot are needed compared to mainline?

I also need to look into converting my rockpro64 boards to zfs only, so far they're all based off the official images which uses ufs.
Does anyone have some notes creating a zfs based image?
 
The modified U-Boot is here:

github.com

ports/sysutils/u-boot-rock-pi-4-2020.07 at master · S199pWa1k9r/ports

Unofficial FreeBSD u-boot ports. Contribute to S199pWa1k9r/ports development by creating an account on GitHub.
github.com github.com

If you prefer prebuilt images:


But that's only needed if you want to load U-Boot from SPI flash (for running with just nVME SSD, no SD card or eMMC) or boot from USB mass storage devices. If you want to keep U-Boot on SD card or eMMC, U-Boot does not need any modifications. After all, it only serves to load FreeBSD's loader, which then handles ZFS (just as it does on amd64 PCs).

I haven't looked into what it would take to create a ZFS image. Instead, I used the stock image on SD card for the initial boot, and prepared ZFS on the nVME SSD within that system.
 
You must log in or register to reply here.
Back
Top