Installing FreeBSD Alongside Ubuntu and Windows

larkwiot

larkwiot

Hello all,

I have been struggling to get FreeBSD booting after installation. I already have Windows 10 and Ubuntu 18 installed. All three OSes are on their own SSDs. os-prober in Ubuntu doesn't know what FreeBSD is so after extensive searching I found the most current and preferred grub entry to put in /etc/grub.d/40_custom is
Code: 
menuentry "FreeBSD-12.0" {
set root='(hd2,2)' # FreeBSD's UFS partition is recognized as /dev/sdc2 on Ubuntu
kfreebsd /boot/loader
}
However, when I select that boot entry in GRUB, FreeBSD complains about not being able to work out which disk we are booting from and tries disk0, which of course fails. I cannot find any information on why this would be the case, as I was able to install FreeBSD just fine. All my partitions are GPT, so I installed FreeBSD as GPT.

Any input would be appreciated.

Thanks in advance!
 
larkwiot said:
All my partitions are GPT, so I installed FreeBSD as GPT.
Click to expand...
Try set root='(hd2,gpt2)' .

When in GRUB menu, select FreeBSD menu entry, press the E key, edit entry, boot pressing F10. If the entry is good, make it permanent in the Ubuntu grub 40_custom configuration file.

If it's not good, in the Grub menu press the C key, at the grub prompt type ls, look for the partition FreeBSD is installed on, edit FreeBSD menu entry accordingly.
 
T-Daemon said:
Try set root='(hd2,gpt2)' .

When in GRUB menu, select FreeBSD menu entry, press the E key, edit entry, boot pressing F10. If the entry is good, make it permanent in the Ubuntu grub 40_custom configuration file.

If it's not good, in the Grub menu press the C key, at the grub prompt type ls, look for the partition FreeBSD is installed on, edit FreeBSD menu entry accordingly.
Click to expand...

Tried this, still got the same error.

I'm not sure how to find the partition with grub's command line, but this is the output from fdisk:
screenshot_5.png


Looking at this, wouldn't it be hd2,gpt2?

Edit: I figured out what I was doing wrong with ls, not including parentheses. Now I see that the path (hd2,gpt2)/boot/ contains loader.rc and loader.efi. Not sure what's going wrong if I have the correct path.

Edit 2: The fact that trying another partition like (hd2,gpt1) results in "File /boot/loader not found" makes me think the issue is with FreeBSD or how the disk is set up, not with grub.
 
For non-uefi boot there is only two things to have right, and it is straightforward,
1. a correct set root= , kfreebsd in the grub file
2. a correct /etc/fstab , check this file
 
larkwiot said:
Hello all,

I have been struggling to get FreeBSD booting after installation. I already have Windows 10 and Ubuntu 18 installed. All three OSes are on their own SSDs. os-prober in Ubuntu doesn't know what FreeBSD is so after extensive searching I found the most current and preferred grub entry to put in /etc/grub.d/40_custom is
Code: 
menuentry "FreeBSD-12.0" {
set root='(hd2,2)' # FreeBSD's UFS partition is recognized as /dev/sdc2 on Ubuntu
kfreebsd /boot/loader
}
However, when I select that boot entry in GRUB, FreeBSD complains about not being able to work out which disk we are booting from and tries disk0, which of course fails. I cannot find any information on why this would be the case, as I was able to install FreeBSD just fine. All my partitions are GPT, so I installed FreeBSD as GPT.

Any input would be appreciated.

Thanks in advance!
Click to expand...
Why don't you just skip boot loaders altogether? I had a laptop with a similar setup - multiple hard drives, each had its own OS. At the end of the day I figured out it's easier to use the laptop's F11 button which showed a menu from the BIOS that lets you select from which drive to boot.
To summarize - you install every OS on its own drive with its own boot loader (if needed, EFI partition). Then when booting - press F11 (or depends on your particular machine and BIOS) and select the drive manually.

If you do want to use grub, that should be OK. Just create a manual entry in the grub configuration, and do chainloading similar to Windows. Then chainload the FreeBSD boot loader, which should be installed in a ca. 512k partition of type freebsd-boot on the FreeBSD disk.
Example for chainloading: https://www.gnu.org/software/grub/manual/grub/grub.html#Chain_002dloading
An example for loading freebsd kernel can be found here: https://unix.stackexchange.com/questions/109272/add-freebsd-to-grub2-boot-menu
 
larkwiot said:
Tried this, still got the same error.

I'm not sure how to find the partition with grub's command line, but this is the output from fdisk:
View attachment 6990

Looking at this, wouldn't it be hd2,gpt2?

Edit: I figured out what I was doing wrong with ls, not including parentheses. Now I see that the path (hd2,gpt2)/boot/ contains loader.rc and loader.efi. Not sure what's going wrong if I have the correct path.

Edit 2: The fact that trying another partition like (hd2,gpt1) results in "File /boot/loader not found" makes me think the issue is with FreeBSD or how the disk is set up, not with grub.
Click to expand...
You cannot load FreeBSD like this, because it is not a linux kernel.
Take a look at the above examples I have given.
Loading the FreeBSD kernel works differently from the Linux one.
 
Alain De Vos said:
For non-uefi boot there is only two things to have right, and it is straightforward,
1. a correct set root= , kfreebsd in the grub file
2. a correct /etc/fstab , check this file
Click to expand...

Pardon my ignorance, but I don't see how fstab would help because in my understanding that only makes a difference when booted into Linux? The issue I have is before boot of any OS.

roccobaroccoSC said:
Why don't you just skip boot loaders altogether? I had a laptop with a similar setup - multiple hard drives, each had its own OS. At the end of the day I figured out it's easier to use the laptop's F11 button which showed a menu from the BIOS that lets you select from which drive to boot.
To summarize - you install every OS on its own drive with its own boot loader (if needed, EFI partition). Then when booting - press F11 (or depends on your particular machine and BIOS) and select the drive manually.

If you do want to use grub, that should be OK. Just create a manual entry in the grub configuration, and do chainloading similar to Windows. Then chainload the FreeBSD boot loader, which should be installed in a ca. 512k partition of type freebsd-boot on the FreeBSD disk.
Example for chainloading: https://www.gnu.org/software/grub/manual/grub/grub.html#Chain_002dloading
An example for loading freebsd kernel can be found here: https://unix.stackexchange.com/questions/109272/add-freebsd-to-grub2-boot-menu
Click to expand...

I could do that but using grub would be nice from a purely convenience standpoint.

I'll look into chainloading, but when I tried to use it (meaning I copied what people said to put in the grub entry with things like
Code: 
chainloader +1
it still didn't work.

roccobaroccoSC said:
You cannot load FreeBSD like this, because it is not a linux kernel.
Take a look at the above examples I have given.
Loading the FreeBSD kernel works differently from the Linux one.
Click to expand...

This second post confuses me since you say chainloading it should work and the grub entry I was trying to put in is (at a general level) what every doc and person on stackexchange I've seen so far says to do. Maybe it will be cleared up once I investigate the links you gave me.

Edit: Looking at the two links you gave me, I see the stackexchange one is one I've already looked at thoroughly, though maybe I missed something and didn't understand it. The grub manual says FreeBSD is specifically supported for multiboot and therefore should not need chainloading. I still don't know why the config I have should not work.
 
larkwiot said:
This second post confuses me since you say chainloading it should work and the grub entry I was trying to put in is (at a general level) what every doc and person on stackexchange I've seen so far says to do. Maybe it will be cleared up once I investigate the links you gave me.

Edit: Looking at the two links you gave me, I see the stackexchange one is one I've already looked at thoroughly, though maybe I missed something and didn't understand it. The grub manual says FreeBSD is specifically supported for multiboot and therefore should not need chainloading. I still don't know why the config I have should not work.
Click to expand...
You are right. Chainloading is not necessary because grub supports loading FreeBSD. It's just another option (another method).
So when you load FreeBSD kernel directly, in your grub config you need something like "kfreebsd ...." What I meant in my post above was that you cannot load FreeBSD just by changing the number of your hd? and using the "linux" and "initrd" stanzas. If I misunderstood, sorry.

The Chainloading method does something quite simple. Grub simply passes the execution to another boot loader and lets it do the work. In that case you need a fully funcitonal FreeBSD boot loader on that disk and grub will pass the control to the FreeBSD boot loader. It should load nicely. I personally find this setup a bit cleaner, but it's a matter of taste. Both should work.


So, why did chainloading not work for you? There could be different reasons.
Most importantly, make sure all boot loaders load in the same mode. You cannot mix legacy (MBR) and UEFI boots. So if grub is in UEFI mode, make sure FreeBSD boots also in UEFI mode and vice versa.
Also, make sure your boot code is properly installed on the FreeBSD disk. If you want, we could dig into that. Post the contents of your FreeBSD partition table:
gpart show and gpart show -l.
You should have installed the boot code either via the normal setup, or with gpart bootcode ...

For your particular situation I think you can see this wiki entry: https://wiki.freebsd.org/UEFI
 
Based on the output of fdisk(8) in the image in post #3, showing a "EFI System" in partition "/dev/sda" installed, and taking the bootstrapping procedure for uefi(8) in consideration, I would try
Rich (BB code): 
menuentry "FreeBSD-12.0" {
set root='(hd2,gpt2)'
kfreebsd /boot/loader.efi
}
 
roccobaroccoSC said:
You are right. Chainloading is not necessary because grub supports loading FreeBSD. It's just another option (another method).
So when you load FreeBSD kernel directly, in your grub config you need something like "kfreebsd ...." What I meant in my post above was that you cannot load FreeBSD just by changing the number of your hd? and using the "linux" and "initrd" stanzas. If I misunderstood, sorry.
Click to expand...
No, I think it's my lack of knowledge in this area, not you. What/where are the linux and initrd stanzas? Are you meaning that I have to include one of the things I've been seeing people say like insmod ufs to tell GRUB to set up for a FreeBSD environment vs linux?

roccobaroccoSC said:
The Chainloading method does something quite simple. Grub simply passes the execution to another boot loader and lets it do the work. In that case you need a fully funcitonal FreeBSD boot loader on that disk and grub will pass the control to the FreeBSD boot loader. It should load nicely. I personally find this setup a bit cleaner, but it's a matter of taste. Both should work.
Click to expand...
I'm thinking the issue is with how I installed FreeBSD. I put it on a USB stick, installed it to the drive, and then rebooted. I did not drop into a shell on the system or configure anything the setup process did not prompt me to. I will try booting to FreeBSD from my UEFI boot menu to see if it has its own bootloader.

roccobaroccoSC said:
So, why did chainloading not work for you? There could be different reasons.
Most importantly, make sure all boot loaders load in the same mode. You cannot mix legacy (MBR) and UEFI boots. So if grub is in UEFI mode, make sure FreeBSD boots also in UEFI mode and vice versa.
Also, make sure your boot code is properly installed on the FreeBSD disk. If you want, we could dig into that. Post the contents of your FreeBSD partition table:
gpart show and gpart show -l.
You should have installed the boot code either via the normal setup, or with gpart bootcode ...

For your particular situation I think you can see this wiki entry: https://wiki.freebsd.org/UEFI
Click to expand...
I would believe they are all in the same mode, but I will try to check. I'll also assume you mean to use gpart in Linux, though I'll also try it in GRUB if that doesn't work.
Edit: I was able to boot into FreeBSD from my UEFI bootmenu and used gpart there. I don't know what you are looking for about it, but I noticed a few things about its slices compared to Linux and Windows' partitions.
  1. It started at address 40 whereas everything else (linux and windows) started at 34.
  2. Its EFI slice was the first slice, whereas everything else had a 1MB partition before their EFIs.
  3. None of its slices overlapped each other.
  4. When doing gpart show -l all of its slices were marked as "(null)" whereas with Linux and Windows each had a partition "free".
Because I was able to boot into it that way, perhaps chainloading would be the best option.

T-Daemon said:
Based on the output of fdisk(8) in the image in post #3, showing a "EFI System" in partition "/dev/sda" installed, and taking the bootstrapping procedure for uefi(8) in consideration, I would try
Rich (BB code): 
menuentry "FreeBSD-12.0" {
set root='(hd2,gpt2)'
kfreebsd /boot/loader.efi
}
Click to expand...

I believe I tried this at one point but I will try it again just to make sure.
Edit: Just tried it and got error: invalid a.out header

Furthermore, I would like to note that every error I get when I configure the GRUB menu entry in a way I think would be incorrect seems to be a GRUB error, and when I press a key it brings me back to GRUB, whereas the error I currently get seems to be a FreeBSD error because it looks like different output styling and is not recoverable (I have to reboot).
 
Does the FreeBSD disk boot when chosen from the UEFI/BIOS disk boot menu (F11/F12 key)? If not, the attention should directed there.

Assuming it boots, what you could yet try is using supergrubdisk (support for FreeBSD and efi files are listed, scroll down the page). Try booting FreeBSD, if it boots check the FreeBSD menu entry (by pressing the E key) which settings have been used to boot and take the necessary settings over to the /etc/grub.d/40_custom file.
 
T-Daemon said:
Does the FreeBSD disk boot when chosen from the UEFI/BIOS disk boot menu (F11/F12 key)? If not, the attention should directed there.

Assuming it boots, what you could yet try is using supergrubdisk (support for FreeBSD and efi files are listed, scroll down the page). Try booting FreeBSD, if it boots check the FreeBSD menu entry (by pressing the E key) which settings have been used to boot and take the necessary settings over to the /etc/grub.d/40_custom file.
Click to expand...

It boots. Supergrubdisk actually looks cool. I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting. Also, at some point during that, FreeBSD moved from getting detected on hd2 to hd3. I adjusted my grub entry but that did not help, still the same error. Weird that it moved though, since fdisk on Ubuntu still detects its UFS slice on /dev/sdc2. I also tried reinstalling from the memstick img vs the dvd which I had, by force of habit, previously used. No difference. Still able to boot from UEFI and install just fine but GRUB can't get it working.

I'd also like to note that the specific error is an IO error on /boot/lua/loader.lua. I confirmed that that file exists at that path and it does, but I don't know what else to check about it.
 
larkwiot said:
I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting.
Click to expand...
That's not a good sign if supergrubdisk cant't boot FreeBSD. It might be time to think about another boot manager. rEFInd would be a good candidate.
 
T-Daemon said:
That's not a good sign if supergrubdisk cant't boot FreeBSD. It might be time to think about another boot manager. rEFInd would be a good candidate.
Click to expand...

I suppose I will just boot it from UEFI since GRUB is already working well for Ubuntu and Windows and I'd rather not go to the trouble of replacing GRUB. Really disappointing that there doesn't seem to be an answer to the issue though.
 
larkwiot said:
It boots. Supergrubdisk actually looks cool. I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting. Also, at some point during that, FreeBSD moved from getting detected on hd2 to hd3. I adjusted my grub entry but that did not help, still the same error. Weird that it moved though, since fdisk on Ubuntu still detects its UFS slice on /dev/sdc2. I also tried reinstalling from the memstick img vs the dvd which I had, by force of habit, previously used. No difference. Still able to boot from UEFI and install just fine but GRUB can't get it working.

I'd also like to note that the specific error is an IO error on /boot/lua/loader.lua. I confirmed that that file exists at that path and it does, but I don't know what else to check about it.
Click to expand...
I think what might be happening here: In order to load a specific OS and file system (like UFS and FreeBSD) grub needs to load the modules that support it. So you need to configure grub to load the modules for GPT, for UFS and for FreeBSD. If you forgot some of them (for example the last one) this might be the reason why grub cannot recognize your FreeBSD kernel - it lacks the dynamic module to support it.

That's why it's probably the simplest solution to try and get Chainloading to work. You tested it from the UEFI menu and it boots. This means that your FreeBSD disk is installed properly. Now when you make Grub pass the control to it via a Chainload menu entry, it should definitely work in my opinion.
As far as I know, you don't need a specific module in Grub for chainloading, because it is really simple indeed. In the executable code it probably just loads the next boot loader into memory and executes a JMP instruction directly into the other boot loader's code.

Code: 
# /etc/grub.d/40_custom

Take a look here for example: https://wiki.gentoo.org/wiki/GRUB2/Chainloading#Another_bootloader

menuentry "Custom Super Bootloader Example" {
     insmod part_msdos
     insmod chain
     chainloader (hd1,1)+1
}
#

Adapt the HDD numbers to your particular situation. I think HDDs are numbered from 0: hdd0, hdd1, ... but partitions are numbered from 1: 1, 2, 3. That's important to keep in mind!
Make sure you generate the grub configuration file properly after that, via grub-mkconfig. And then after that, open /boot/grub/grub.cfg and make sure your code is in there.

EDIT: You probably need to replace "part_msdos" with "part_gpt" above!!! I think your disk is formatted with GPT and not MBR/legacy).
Scrap the example above, I think with GPT it's like that: https://unix.stackexchange.com/questions/343632/uefi-chainloading-grub-from-grub
Code: 
menuentry "GRUB chainloader" {
 #Load grub on partition 7 of a gpt formatted drive.
 #It will reference its own modules and config.
 chainloader (hd0,gpt7)/path/to/bootloader/on/myOS/core.efi
}
 
larkwiot said:
It boots. Supergrubdisk actually looks cool. I just tried it and it only got white static-like stuff on the top of my screen for all options to boot FreeBSD (it detected multiple) except for the "default bootloader" option, which resulted in the same error I've been getting. Also, at some point during that, FreeBSD moved from getting detected on hd2 to hd3. I adjusted my grub entry but that did not help, still the same error. Weird that it moved though, since fdisk on Ubuntu still detects its UFS slice on /dev/sdc2. I also tried reinstalling from the memstick img vs the dvd which I had, by force of habit, previously used. No difference. Still able to boot from UEFI and install just fine but GRUB can't get it working.

I'd also like to note that the specific error is an IO error on /boot/lua/loader.lua. I confirmed that that file exists at that path and it does, but I don't know what else to check about it.
Click to expand...
See here an example for the first method (kfreebsd): https://unix.stackexchange.com/questions/109272/add-freebsd-to-grub2-boot-menu

Code: 
menuentry "FreeBSD" --class freebsd --class bsd --class os {
insmod ufs2
insmod bsd  # <<======= notice this!!!!
set root=(hd0,1)
kfreebsd /boot/kernel/kernel
kfreebsd_loadenv /boot/device.hints
set kFreeBSD.vfs.root.mountfrom=ufs:/dev/ada0s1a
set kFreeBSD.vfs.root.mountfrom.options=rw
set kFreeBSD.hw.psm.synaptics_support=1
}

See here how they load the "bsd" module.
 
roccobaroccoSC said:
Adapt the HDD numbers to your particular situation. I think HDDs are numbered from 0: hdd0, hdd1, ... but partitions are numbered from 1: 1, 2, 3. That's important to keep in mind!
Make sure you generate the grub configuration file properly after that, via grub-mkconfig. And then after that, open /boot/grub/grub.cfg and make sure your code is in there.

EDIT: You probably need to replace "part_msdos" with "part_gpt" above!!! I think your disk is formatted with GPT and not MBR/legacy).
Scrap the example above, I think with GPT it's like that: https://unix.stackexchange.com/questions/343632/uefi-chainloading-grub-from-grub
Code: 
menuentry "GRUB chainloader" {
#Load grub on partition 7 of a gpt formatted drive.
#It will reference its own modules and config.
chainloader (hd0,gpt7)/path/to/bootloader/on/myOS/core.efi
}
Click to expand...

This worked! At first I tried:
Code: 
menuentry "GRUB chainloader" {
chainloader (hd2,gpt2)/boot/loader.efi # this was on the UFS partition
}

But that did not work. Then I poked around in GRUB looking at the partitions and tried this:
Code: 
menuentry "GRUB chainloader" {
chainloader (hd2,gpt1)/efi/boot/BOOTx64.efi # this was on the EFI partition
}
and it worked!

Thanks for your help!
 
larkwiot said:
This worked! At first I tried:
Code: 
menuentry "GRUB chainloader" {
chainloader (hd2,gpt2)/boot/loader.efi # this was on the UFS partition
}

But that did not work. Then I poked around in GRUB looking at the partitions and tried this:
Code: 
menuentry "GRUB chainloader" {
chainloader (hd2,gpt1)/efi/boot/BOOTx64.efi # this was on the EFI partition
}
and it worked!

Thanks for your help!
Click to expand...
Cool! FreeBSD installed the EFI bootloader as "portable". It is a hard coded path that all BIOS-es should recognize: /efi/boot/BOOTx64.efi
Generally using other paths is also possible, but then one needs to configure the entry in BIOS'es NVRAM via efibootmgr and even then some buggy BIOSes don't recognize the boot loader because they are made by people who think Windows 10 is the only OS that ever existed.
 
Alain De Vos said:
There is something wrong with EFI, does it makes things more secure, I don't think so.
Click to expand...
If Secure boot is off, then - no. It could be made secure in theory, but the problem is that the implementations have quite bad usability. There is a lot written on the topic.
Basically, the hardware vendors deliver hardware with only Microsoft keys preinstalled on the machine. So if you want to install any other OS than Windows, the user has to upload the bootloader's keys to the BIOS manually (and this can be done only from the BIOS interface itself). The process is very technical and not well documented if at all.
So basically any other OS vendor would have to beg Microsoft to sign their bootloder, otherwise Secure boot needs to be turned off. This is the only practical way.

When you turn it off, then UEFI and legacy are equally insecure.

But that's not the only reason to use UEFI. Basically the hardware companies could not fit new BIOS'es functionality in the BIOS chips anymore - too much code, now easy way to update BIOS. So they split the functions and put only the very basic stuff on the chip, the rest went to the EFI parition and so it is a lot more flexible and can be updated easily by the OS. No need to flash BIOS chip every now and then.

In my humble opinion, the idea was good. Implementation could have been done better and more user friendly. Especially the part with installing Keys for other OS-es with Secure boot.
 
SOLVED (just about). IT WORKS. FreeBSD 13-Current on laptop which also boots Win10.64, Mint, Ubuntu, and Kali. "Grub Customizer" used in Linux Mint.

I had to mount the BSD EFI partition in linux Mint (as FAT32), to determine the correct entry to boot:
mount -t msdos /dev/nvme2n1p5 /mnt
for me, the result was 'efi/freebsd/loader.efi'

THEREFORE....
#cat 40_Custom

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.

menuentry "FreeBSD 13-Current on hd2,gpt5" {
chainloader (hd2,gpt5)/efi/freebsd/loader.efi # this was on the EFI partition
}

The ONLY niggle - 4 NVME sticks don't ALWAYS show up in same order (damned if I know why), so now down the rabbit-hole for specifying UUID.

HTH others,
MarkB
 
You must log in or register to reply here.
Back
Top