poudriere image creation for ARM architecture

DemoNIck · Jan 11, 2024

I want to use poudriere, in order to create a custom pkg repository for use with an embedded ARMv7 SBC I own (Olimex A20-OLinuXino-LIME2, Allwinner A20 dual core Cortex-A7 processor, 1GB DDR3 RAM and 16GB eMMC flash memory) and to also create a custom image for the above mentioned SBC.

In order to do so, I installed ports-mgmt/poudriere-devel and installed and enabled emulators/qemu-user-static on a FreeBSD 14.0-RELEASE system.

 # uname -a

FreeBSD DeathStar 14.0-RELEASE-p2 FreeBSD 14.0-RELEASE-p2 #0 releng/14.0-n265396-06497fbd52e2: Tue Dec 12 01:06:08 EET 2023     root@DeathStar:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

.

I have successfully managed to build some ports for testing.

In brief, these are the steps I followed, after configuring the /usr/local/etc/poudriere.conf configuration file. All commands are issued as root.
# poudriere jail -c -x -j armv7 -m git+https -a arm.armv7 -v releng/14.0 -K GENERIC

 # poudriere jail -l

JAILNAME VERSION                           ARCH      METHOD    TIMESTAMP           PATH

armv7    14.0-RELEASE-p4 1400097 4edf3b807 arm.armv7 git+https 2024-01-10 14:09:19 /var/poudriere/jails/armv7

# poudriere ports -c -m git+https -B main -p HEAD

 # poudriere ports -l

PORTSTREE METHOD    TIMESTAMP           PATH

HEAD      git+https 2024-01-10 13:03:35 /var/poudriere/ports/HEAD

And finally, I successfully build some pkgs listed in /usr/local/etc/poudriere.d/pkglist, by issuing the command
# poudriere bulk -j armv7 -p HEAD -f /usr/local/etc/poudriere.d/pkglist

So far, so good!

poudriere, supports the creation of custom images, through the poudriere-image(8) command, supporting several option through the -t flag, regarding the type of image to be created.
In my case, the image to be created, needs to support the u-boot boot loader and more precisely the sysutils/u-boot-olinuxino-lime2 boot loader.

What type of image should I pass to the -t flag and how should I tell poudriere to use the sysutils/u-boot-olinuxino-lime2 boot loader?

Thank you in advance for your time.

Phishfry · Oct 27, 2024

Did you ever find a solution for this? I am image building now for RockPi4A and I wonder about free space before first partiton.

Worst comes to worse I will delete EFI partition from medium and make offset for u-boot.

gpart delete -i 1 da0
gpart add -t efi -s $M -b 32768 da0 <careful with size not to overwrite into next partition>
newfs_msdos da0
Then mount and repopulate EFI
flash u-boot to free space

Trying with -t firmware as build type option.

Phishfry · Oct 27, 2024

I see an approach here with the -B option of poudriere image

pre-script-ufs.sh

GitHub - jlduran/rpi: Build Raspberry Pi images

Build Raspberry Pi images. Contribute to jlduran/rpi development by creating an account on GitHub.

github.com

Phishfry · Oct 29, 2024

There also is a post-script.sh that would be good .

I could not help but notice that BSDRP Poudriere Image page mentions an Embedded Image Type.

Poudriere image [BSD Router Project]

bsdrp.net

But there is no -t Type Embedded. I also have a silent image fail with -t firmware. This is the nanobsd type build.
It does make an root image with the -t rawfirmware.
So with that you could manually build a EFI partition leaving 32768 for u-boot then flash the rawfirmware image.
The rawfirmware image contains only a root partition.

I am building on RockPi4a and build went pretty fast. I wanted to build poudriere images arm64 on arm64.

My -t firmware build fails with weird message about pmbr missing for mkimg. I look at source and sure enough pmbr is not there.
I am using poudriere-devel.

If nobody has insight I may file issue on github.

Why is -t firmware trying to use PMBR for an EFI build?

Looks like here is my problem:

poudriere/src/share/poudriere/image_firmware.sh at master · freebsd/poudriere

Port/Package build and test system. Contribute to freebsd/poudriere development by creating an account on GitHub.

github.com

mkimg cannot source the file /boot/pmbr.
I am assuming this has been removed from FreeBSD 14.1-RELEASE base? Not in my jail.

Phishfry · Oct 29, 2024

OK so my question still is:

Phishfry said:
Why is -t firmware trying to use PMBR for an EFI build?

To make my case let me quote manual on what poudriere image firmware build is:

firmware A NanoBSD style image with a GPT partitions and a UEFI boot loader.

So all sources say it is UEFI build with GPT partitions. That is what we want.

But script line 135 is copying /boot/gptboot.efi for an installation.

make_esp_file ${espfilename} ${ESP_SIZE} ${WRKDIR}/world/boot/gptboot.efi

That seems wrong. I thought we copy/make ESP from /boot/boot1.efi

Copy the FreeBSD /boot/boot1.efi bootcode file into the efi filesystem.

To me the script image-firmware.sh seems wrong. It is for type Legacy GPT-Boot not EFI-Boot.

The FreeBSD Boot Process - Klara Systems

Understand the FreeBSD boot process, including differences between UEFI and legacy BIOS, GPT and MBR partitioning, and filesystem options like ZFS and UFS.

klarasystems.com

Any thoughts from people with way more experience than me?

Phishfry · Oct 29, 2024

Well after studying the manpage for gptboot.efi I can see why they used this scheme.

gptboot.efi

man.freebsd.org

The key passage:

However, some setups cannot use those mechanisms. When the users cannot rely on host-supplied UEFI variables or they want the contents of the media
alone to decide root, gptboot.efi accomplishes these goals.

So to determine which NanoBSD root partition (ping-pong upgrade) to boot off of you could use the "bootme flags" from gptboot.efi.

Phishfry · Oct 29, 2024

I edited the script at /usr/local/share/poudiere/image-firmware.sh

I noticed my version used loader.efi instead of gptboot.efi.
So devel version has this somewhat fixed.

make_esp_file ${espfilename} ${ESP_SIZE} ${WRKDIR}/world/boot/loader.efi

All I had to do was drop the -b flag for the pmbr. I also axed the freebsd-boot partition.

Code:

        mkimg -s gpt -C ${IMAGESIZE} \
                -p efi:=${espfilename} \
                -p freebsd-ufs/${IMAGENAME}1:=${WRKDIR}/raw.img \
                -p freebsd-ufs/${IMAGENAME}2:=${WRKDIR}/raw.img \
                -p freebsd-ufs/cfg:=${WRKDIR}/cfg.img \
                ${SWAPFIRST} \
                -p freebsd-ufs/data:=${WRKDIR}/data.img \
                ${SWAPLAST} \
                -o "${OUTPUTDIR}/${FINALIMAGE}"

Code:

-rw-r--r--  1 root wheel 4093675008 Oct 27 06:01 rockpi4.img
-rw-r--r--  1 root wheel 2008023040 Oct 27 04:04 rockpi4.raw

Code:

/usr/local/share/poudriere # gpart show md0
=>     34  7995392  md0  GPT  (3.8G)
       34    20480    1  efi  (10M)
    20514  3921920    2  freebsd-ufs  (1.9G)
  3942434  3921920    3  freebsd-ufs  (1.9G)
  7864354    65536    4  freebsd-ufs  (32M)
  7929890    65536    5  freebsd-ufs  (32M)

Notice no free space before first partition. Will need work for u-boot...

Here is the image_firmware script setting for EFI partition size:
ESP_SIZE=10

Phishfry · Oct 29, 2024

Maybe one more edit required. This controls EFI build.
/usr/local/share/poudriere/image.sh

Instead of gpart add this script uses makefs.

makefs

man.freebsd.org

For U-Boot we need to offset the msdos partition for free space. There is no -O option in the script so add one with proper offset..

poudriere/src/share/poudriere/image.sh at master · freebsd/poudriere

Port/Package build and test system. Contribute to freebsd/poudriere development by creating an account on GitHub.

github.com

I have no idea what it uses but "-O 32768" is where I plan on starting in the script mod.

Phishfry · Nov 4, 2024

DemoNIck said:
What type of image should I pass to the -t flag and how should I tell poudriere to use the sysutils/u-boot-olinuxino-lime2 boot loader?

I have some experience now at this and I would recommend a -t usb type install.
Unless you need NanoBSD style dual partition the best image type is USB.
The reason I say that is it builds the disk with a label so it will boot from USB stick, and it should boot from SATA, NVMe or SDCard. This is thanks to the use of disk labels.

The problem with the USB image generated is that it contains no free-space before EFI partition for u-boot to install.
So you must modify the script that Poudriere Image uses to build its usb image type.

I am using poudriere-devel pkg install poudriere-devel.
.
It installs itself into /usr/local/share/poudiere and consists of the scripts used to build the images.
The top level heirachy consists of a script image.sh but the bulk of the work is done in the individual build scripts.

make_usb.sh is the primary builder for -t usb image.
The problem with this script is it only makes a 10 Megabyte partition for the EFI bootloader.
The loader.efi is less than 1 Meg.
This is fine for amd64 but for arm64 you include DTB directory on EFI partition for Arm64.
So you need extra space on EFI/ESP for /dtb. I would say 30-60 megs is a good EFI/ESP size.

OK so enlarge ESP and we also need to make a partition offset for u-boot. The script needs simple editing.
/usr/local/share/poudriere/make_usb.sh
The EFI partition size is set here in Megabytes at default value of "10". Enlarge this to suit your tastes. I use 50.
Line 82:
Original

Code:

make_esp_file ${espfilename} 10 ${WRKDIR}/world/boot/loader.efi

Modified to 50:

Code:

make_esp_file ${espfilename} 50 ${WRKDIR}/world/boot/loader.efi

OK now to add space for u-boot in the image you must add an offset to the EFI Partition. This is a byte value. No size suffix.
Line 104:
Here is original setting. No offset.

Code:

-p efi:=${espfilename} \

Here is the offset in bytes for u-boot. You can use 33554432 for EDK2.

Code:

-p efi:=${espfilename}:16777216 \

That's it. It's up to you to flash u-boot to the free-space after building. You must also add a populated /dtb ditectory to EFI partition.

poudriere image -t usb -s 4G -j rockpi4 -p rockpi4_ports -h rockpi4 -n rockpi4-usb -f rockpi4-pkglist

Code:

cd /poudriere/data/images
mdconfig rockpi4-usb.img

gpart show md0
=>     34  8132574  md0  GPT  (3.9G)
       34    32734       - free -  (16M)
    32768   102400    1  efi  (50M)
   135168  7997440    2  freebsd-ufs  (3.8G)

In the future I will post a post-install script for poudriere image and flashing u-boot..

If you mess your script up its easy enough to restore.
pkg upgrade -y -f poudriere-devel

I am building on arm64 here but I expect no differences on amd64 builds EFI/ESP Size adjust wise. Same knobs.

Phishfry · Nov 9, 2024

Thanks for making my brain stir. I was going to make a single DTB tranfer to EFI for my poudriere image post-install script.

Maybe better approach is copy the whole /dtb directory like the FreeBSD-Arm images do.
Copy the whole device tree that we support. I will need to study that method to see how it is generated.
FreeBSD ESP only uses 3.1 Megabytes with DTB's so 10M EFI partition is OK.

Then on to u-boot copying.
Not sure how to support flashing for different SOC and different schemes/sectors used.

Maybe just make an upload directory for u-boot and DTB for script to find. Easy way out.

Phishfry · Nov 24, 2024

Phishfry said:
Then on to u-boot copying.
Not sure how to support flashing for different SOC and different schemes/sectors used.

This part ended up easier than I imagined for boards supported by a dtb file. U-boot ports.
Use boudiere bulk and just download needed u-boot pkg.
For example sysutils/u-boot-rock64
When building the Rock64 image ports I used pkg binary fetch for speed. I enable it in /usr/local/etc/poudriere.conf.
For the rock64-pkglist file I only have u-boot-rock64.
poudriere bulk -j rock64 -p default -f rock64-pkglist -b latest

I am also messing around with MFS+UFS images. That seems ideal for an eMMC install. I don't want disk writes.

 poudriere image -t usb+mfs -s 4G -j rock64 -h rock64 -n rock64-mfs -c ./overlay/rock64 excluded.files -p default -f rock64-pkglist

I wrote conventional -t USB images to my eMMC on RockPi4 now. I am on to Rock64 now.

 poudriere image -t usb -s 4G -j rock64 -h rock64 -n rock64-usb -c ./overlay/rock64 excluded.files -p default -f rock64-pkglist

To flash I used a microSD with FreeBSD installed and mounted a USB stick with image to flash to eMMC.
Also on USB stick is my u-boot.
So from microSD install I flash my Poudriere built RockPi4 image and then from files I copied to USB stick from /usr/local/share/u-boot-rock64 I flash my u-boot.
All done manual now but I hope to flash u-boot at memdisk stage.
Modify "Create ESP" stage of image.sh is what I have done.
Copy over DTB to ESP with cp -r ${PKGBASE}/world/boot/dtb {ESP}dtb

It will get interesting when it comes to slave U-Boot ports I have built. zirias@ has some good writeups on using the git method but I may start with the more familiar Ports Overlay method of building.

poudriere-bulk(8)

man.freebsd.org

-O overlay
Specify an extra poudriere-ports(8) tree to use as an overlay.
Multiple -O overlay arguments may be specified to stack them.

Post in thread 'private meta-ports and poudriere'

May 29, 2021

For the record: This has worked well. It needs ports-mgmt/poudriere-devel at the moment, but should be included with ports-mgmt/poudriere when version 4.0 comes out.

I created a skeleton overlay ports tree, imported it into poudriere with poudriere ports -c -p overlay -m null -M /my/tree and then ran the bulk run with the appropriate arguments: poudriere bulk -p default -O overlay -j myjail -f myportlist.

Only shortcut I took was I avoided using my own category like local/myport, because I didn't want to deal with overlaying a VALID_CATEGORIES entry...