Solved Installer does not recognize existing EFI system partition

[purefanatic]

I would like to install FreeBSD alongside some other systems on the same GPT-partitioned disk. A 100 MB EFI system partition (ESP) already exists on that disk. When creating the freebsd-ufs partition for the root directory, I get prompted "This partition scheme requires a boot partition for the disk to be bootable. Would you like to make one now?"

If I answer "No" and proceed, the installer will finish without creating a boot loader inside the existing ESP. Hence, FreeBSD will not be bootable.
If I answer "Yes", an additional 200 MB ESP is created automatically which I do not want.

Using the manual partitioning editor, there seems to be no way to make the installer create a boot loader inside an existing ESP. How can I achieve this using other methods?

 

[ShelLuser]

There's a good reason why it doesn't do this automatically; because then it remains to be seen that you can still install your other OS (note: a bit of an assumption on my part, I have no hands on experience with UEFI).

Anyway, a common way to do this is to use dd and apply /boot/boot1.efifat onto your EFI partition. Be warned though that this will overwrite the EFI partition. You might want to make a backup before setting this up.

 

[purefanatic]

When installing a Linux distribution for example, GRUB fits just fine inside an existing EFI system partition, alongside Windows or whatever. It shouldn't be a problem. There surely must be another way than having to install FreeBSD first, as you suggested, and then trying to repair my Windows installation (which might break the FreeBSD loader in turn).

Because it works well with Linux distributions, I just tried installing GRUB for FreeBSD but had no luck. There is no grub-install or update-grub or anything available after pkg install grub2-efi. But I might be doing everything wrong...

 

[ShelLuser]

There surely must be another way than having to install FreeBSD first, as you suggested,

I did not suggest to install FreeBSD first. Please don't read between the lines, there's nothing there.

All I said was that by default FreeBSD provides /boot/boot1.efifat which is used to boot FreeBSD through EFI. I assume this to be some sort of boot menu comparable to GRUB but I simply don't know because I don't have hands on experience with it.

I also said that if you want to achieve what the installer would do you'd use # dd if /boot/boot1.efifat of /dev/<your disk & efi partition here>.

About GRUB: you'll need both sysutils/grub2 and sysutils/grub2-efi. The first supplies the grub-install and other binaries, the latter the extension.

 

[purefanatic]

About GRUB: you'll need both sysutils/grub2 and sysutils/grub2-efi. The first supplies the grub-install and other binaries, the latter the extension.

Ah I thought grub2-efi would depend on grub2. grub2-efi doesn't really make sense on its own, does it? I'll try that tomorrow.

I assume this to be some sort of boot menu comparable to GRUB but I simply don't know because I don't have hands on experience with it.

Well it is probably similar to the MBR boot, but I have yet to see either, so I can't tell ;-)

As for reading between the lines, pardon me. I phrased that wrongly. I only meant to say that installing /boot/boot1.efifat the way you suggested would imply destroying all existing bootloaders and having to reinstall those after FreeBSD is installed. I guess one could destroy and then re-create the EFI system partition during FreeBSD's install procedure and have the same effect.

 

[purefanatic]

I didn't have any success installing GRUB. First I tried pkg install grub2 grub2-efi which apparently succeeded only halfway because of conflicts. In the end, grub2 was installed but grub2-efi was not. Also note that the grub2 package did not provide any grub-install or update-grub executables.

So then I tried installing the grub2 port from /usr/ports/sysutils/grub2/ by typing make install. This didn't work either, make stopped because the indexinfo executable was not found. which indexinfo comes to the same conclusion, there is no such executable. Explicitly trying to install indexinfo from either the port or the package will report that indexinfo is already installed.

Either I must be doing something wrong or those ports are just not working.

 

[ShelLuser]

Did you by any chance mix ports and packages ( pkg install ... vs. make install clean)? Because that could explain a thing or two here. Also: what FreeBSD version are you using?

Thing is, I can't reproduce anything weird when I try to install both packages:

# pkg install -n grub2 grub2-efi
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 26 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        grub2: 2.00_11
        grub2-efi: 2.02_18
        indexinfo: 0.3.1
        gcc6: 6.4.0_7
        mpfr: 4.0.1
        gmp: 6.1.2
        mpc: 1.1.0_1
        gcc-ecj: 4.5

See? Nothing which indicates any conflicts or something.

I hope this can help.

 

[kpa]

Are you sure you have an up to date ports tree? Im pretty sure the indexinfo mess (the one I remember) was fixed years ago and if you have a recent ports tree the indexinfo port should be pulled in automatically.

 

[puretone]

grub2-efi doesn't really make sense on its own, does it? I'll try that tomorrow.

GRUB doesn't make much sense at all in general...

 

[purefanatic]

I have 11.2-RELEASE for x64. I first tried the pkg install vs the make install approaches on two separate trials. Only after that didn't work, I tried mixing up some stuff...

kpa How would I make sure I have up to date ports? I tried both the ports you can get directly from the FreeBSD installer and I tried portsnap fetch followed by portsnap extract. I only got slightly different error messages if I remember correctly.

ShelLuser Have you actually tried installing it with the pkg install approach? Because for me it only notices the conflicts after fetching the packages (and fetching only happens after answering "yes"). Which is kinda dumb if you ask me. Here's my transcript (from the shell you can get in the last FreeBSD install dialogue):

This shell is operating in a chroot in the new system. When finished making configuration changes, type "exit".
# pkg update
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.10.5_1...
Extracting pkg-1.10.5_1: 100%
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B  0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   6.6MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 32191 packages processed
All repositories are up to date.
# pkg upgrade
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (1 candidates): 100%
Processing candidates (1 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
# pkg install grub2 grub2-efi
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   6.6MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 32191 packages processed.
All repositoreis are up to date.
Updating database digests format: 100%
The following 26 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        grub2: 2.00_11
        grub2-efi: 2.02_18
        indexinfo: 0.3.1
        gcc6: 6.4.0_7
        mpfr: 4.0.1
        gmp: 6.1.2
        mpc: 1.1.0_1
        gcc-ecj: 4.5
        binutils: 2.30_3.1
        gettext-runtime: 0.19.8.1_1
        grub2-pcbsd: 2.02q_12
        freetype2: 2.9.1
        python27: 2.7.15
        readline: 7.0.3_1
        libffi: 3.2.1_2
        mtools: 4.0.10_5
        libXau: 1.0.8_3
        xproto: 7.0.31
        libX11: 1.6.5,1
        libxcb: 1.13
        libXdmcp: 1.1.2
        libxml2: 2.9.7
        libpthread-stubs: 0.4
        kbproto: 1.0.7
        libSM: 1.2.2_3,1
        libICE: 1.0.9_1,1

Number of packages to be installed: 26

The process will require 573 MiB more space.
106 MiB to be downloaded.
Proceeed with this action? [y/N]: y

[[ ellipsis to keep my sanity ]]

[26/26] Fetching libICE--1.0.9_1.1.txz: 100%   89 KiB  91.6kB/s    00:01
Checking integrity... done (1 conflicting)
  - grub2-pcbsd-2.02q_12 conflicts with grub2-2.00_11 on /usr/local/bin/grub-editenv
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 23 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        indexinfo: 0.3.1
        gmp: 6.1.2
        xproto: 7.0.31
        mpfr: 4.0.1
        gettext-runtime: 0.19.8.1_1
        libXau: 1.0.8_3
        libXdmcp: 1.1.2
        libxml2: 2.9.7
        libpthread-stubs: 0.4
        mpc: 1.1.0_1
        gcc-ecj: 4.5
        binutils: 2.30_3.1
        libxcb: 1.13
        kbproto: 1.0.7
        libICE: 1.0.9_1,1
        gcc6: 6.4.0_7
        readline: 7.0.3_1
        libffi: 3.2.1_2
        libX11: 1.6.5,1
        libSM: 1.2.2_3,1
        grub2: 2.00_11
        python27: 2.7.15
        mtools: 4.0.10_5

[[ Note that grub2-efi, grub2-pcbsd and freetype2 are missing now ]]

Number of packages to be installed: 23

The process will require 545 MiB more space.

Proceed with this action? [y/N]: y
[1/23] Installing indexinfo-0.3.1...
[1/23] Extracting indexinfo-0.3.1: 100%
[2/23] Installing gmp-6.1.2...
[2/23] Extracting gmp-6.1.2: 100%
/bin/sh: indexinfo: not found
pkg: POST-INSTALL script failed

[[ The previous two lines would also show after mpfg, gettext-runtime, mpc, binutils, gcc6, readline, libffi, grub2 and mtools. ]]
[[ ellipsis to keep my sanity ]]
[[ Then there's some messages, notably these two: ]]

Message from gcc6-6.4.0_7:

To ensure binaries built with this toolchain find appropriate versions
of the necessary run-time libraries, you may want to link using

  -Wl,-rpath=/usr/local/lib/gcc6

For ports leveraging USE_GCC, USES=compiler, or USES=fortran this happens
transparently.
Message from grub2-2.00_11:

#############################################################
To install GRUB on the master boot record of your hard drive
use 'grub-install <drive-to-install>' command.

A typical menu entry in /boot/grub/grub.cfg for FreeBSD:
menuentry "FreeBSD" {
        set root="(hd0,msdos1,bsd1)"
        kfreebsd /boot/loader
}
Or use grub-mkconfig to create the config file.
##############################################################

[[ The grub2 port currently does not have a maintainer bla ]]
[[ Python bla ]]
# which grub-install
# which update-grub
#

 

[purefanatic]

A bit OT (I mean we're OT already): So far, my only FreeBSD experience was having problems installing stuff, therefore can you guys recommend some nice distribution that has better package management? Is Debian GNU/kFreeBSD really dead for good? Is PacBSD maintained? I don't want to be rude, but I must admit I am rather disgusted by the whole package/port juggling :/

 

[kpa]

I would use the built in SVN client to check out a completely new ports tree:

# mv /usr/ports /usr/ports.old
# mkdir /usr/ports
# svnlite co https://svn.freebsd.org/ports/branches/2018Q3 /usr/ports

After that you can keep the ports tree up to date with:

# svnlite up /usr/ports

 

[ShelLuser]

I have 11.2-RELEASE for x64. I first tried the pkg install vs the make install approaches on two separate trials. Only after that didn't work, I tried mixing up some stuff...

Then that's most likely the cause for most of your problems right now.

Mixing ports and packages is a bad idea and can lead to problems. See this post for more info:

https://forums.freebsd.org/threads/guide-about-ports-and-binary-packages.62126/

However, there's more. You should pay much better attention to the messages which these packages give you. As it turns out sysutils/grub2-efi depends on sysutils/grub2-pcbsd. And that last one installs files into the same place as Grub2, because they're basically the same program, yet different.

root@psi:/var/cache/pkg # pkg info -x grub2
grub2-2.00_12
root@psi:/var/cache/pkg # pkg add ./grub2-pcbsd-2.02q_13-44491373fc.txz
[psi.intranet.lan] Installing grub2-pcbsd-2.02q_13...
pkg: grub2-pcbsd-2.02q_13 conflicts with grub2-2.00_12 (installs files into the same place).  Problematic file: /usr/local/bin/grub-editenv

Failed to install the following 1 package(s): ./grub2-pcbsd-2.02q_13-44491373fc.txz

Solution: replace Grub2.

root@psi:/var/cache/pkg # pkg info -x grub2
grub2-pcbsd-2.02q_13
root@psi:/var/cache/pkg # pkg add ./grub2-efi-2.02_19-dde4f91f22.txz
[psi.intranet.lan] Installing grub2-efi-2.02_19...
[psi.intranet.lan] Extracting grub2-efi-2.02_19: 100%
Message from grub2-efi-2.02_19:

===>   NOTICE:

The grub2-efi port currently does not have a maintainer. As a result, it is
more likely to have unresolved issues, not be up-to-date, or even be removed in
the future. To volunteer to maintain this port, please create an issue at:

https://bugs.freebsd.org/bugzilla

More information about port maintainership is available at:

https://www.freebsd.org/doc/en/articles/contributing/ports-contributing.html#maintain-port

Problem solved.

A bit OT (I mean we're OT already): So far, my only FreeBSD experience was having problems installing stuff, therefore can you guys recommend some nice distribution that has better package management?

There is nothing wrong with FreeBSD package management. Any package manager will fail on you if you don't use it properly or don't carefully read and follow up on the error messages it gives you.

 

[purefanatic]

With "two separate trials" I meant install FreeBSD -> try installing packages -> fail. -> wipe and reinstall FreeBSD -> try installing ports -> fail again.

Any package manager will fail on you if you don't use it properly or don't carefully read and follow up on the error messages it gives you.

I now think you are the one reading between the lines or whereever.

Checking integrity... done (1 conflicting)
  - grub2-pcbsd-2.02q_12 conflicts with grub2-2.00_11 on /usr/local/bin/grub-editenv

ShelLuser you can't be serious. This message does NOT give me the information that grub2-pcbsd and grub2 are "basically the same program, yet different". It also does NOT hint me at a solution. The only thing I can infer from this message is that my original request, installing both grub2 and grub2-efi, can not be fulfilled because of said conflict.

The only conflicts I ever encountered when installing Debian packages were related to dependencies that could not be fulfilled. In my opinion, either the GRUB packages/ports are broken because they shouldn't conflict or they should be actual alternatives of one another. Which might be something pkg does not understand. The package description suggests they should be exclusive. I also dislike the fact that I just get presented a new list of packages and I have to find out by reading through the whole package list that this plan will only install ONE of the packages I wanted to have installed, and which of them. pkg does not communicate that at all!

Solution: replace Grub2.

So I should only install sysutils/grub2-efi (and its dependencies, including sysutils/grub2-pcbsd)? Then what would have been the very first thing I tried? I concede defeat.

 

[kpa]

Please take a read of my HOWTO on building of your own binary packages for FreeBSD. I have included some key points about the deficiencies of the FreeBSD ports system in the first section of my HOWTO, they should give you some idea where the real problems are. In summary, the ports-mgmt/pkg package manager itself is pretty sound but the actual ports infrastructure leaves a lot to be desired.

Thread 38859

These deficiencies are the reason why many serious FreeBSD users end up building their own packages because that neatly solves most of the dependency and conflict related problems.

 

[ShelLuser]

ShelLuser you can't be serious. This message does NOT give me the information that grub2-pcbsd and grub2 are "basically the same program, yet different". It also does NOT hint me at a solution.

I never claimed it did. All I said was that they install stuff into the same location because they're the same program. I assumed that you knew that any package has a description which you can read. I guess I was wrong:

root@psi:/ # pkg search grub2
grub2-2.00_11                  Multiboot boot loader
grub2-bhyve-0.40_4             Grub-emu loader for bhyve
grub2-efi-2.02_18              Multiboot EFI boot loader
grub2-pcbsd-2.02q_12           Multiboot boot loader

2 packages with the exact same description. And if you install both together then it complains as shown above. This should honestly be plain out obvious at this point, and if it isn't then I really suggest you re-read on FreeBSD basic knowledge again. The handbook is an excellent source of information for that.

(edit)

And if this isn't enough then you can always use pkg info -f grub2-pcbsd, which would have told you:

Description    :
GNU GRUB is a multiboot boot loader.  It was derived from GRUB, the GRand
Unified Bootloader, which was originally designed and implemented by Erich
Stefan Boleyn.

This port does not install GRUB on the master boot record of your hard drive.
To do this you will need to read the info page that is installed by the port.

This port includes additional patches and fixes making it work properly
with ZFS boot-environments. Users on UFS will probably want to install the
regular sysutils/grub2 port.

Ergo: they're the same program, but packaged differently.

As such the solution is to replace them.

 

[purefanatic]

Despite @Sheluser's obvious trolling attempts I feel like I should share my new knowledge.

You can use the following workaround (e.g. from the post-install prompt) to install FreeBSD's UEFI OS loader into an existing EFI System Partition (ESP). The idea is to extract FreeBSD's loader from the filesystem image that is used when installing to a new ESP (which we do not want).

Mount FreeBSD's ESP image:
mount -o loop /boot/boot1.efifat /mnt
Let's assume the ESP resides in the third partition of the second ATA disk. Mount the preexisting ESP which shall receive the FreeBSD loader:
mkdir /efi
mount /dev/ada1p3 /efi Or from linux: mount /dev/sdb3 /efi
Extract FreeBSD loader from the image to the ESP:
cp -r /mnt/efi/boot/ /efi/EFI/freebsd
You may unmount /mnt and /efi now. Create a new Boot#### variable to register the new loader in EFI NVRAM:
efibootmgr -c -d /dev/ada1 -p 3 -l '\EFI\freebsd\BOOTx64.efi' -L FreeBSD
The last command should also show all boot variables, including the new one. Or you can just type
efibootmgr
Assuming that the new variable is named Boot0002, you may need to activate it first:
efibootmgr -a 0002
You may change the boot order like this:
efibootmgr -o 0002,0000,0001

Note that FreeBSD's ESP image contains a loader like this:
/efi/boot/BOOTx64.efi
But the /efi/boot/ directory specifies a fallback loader. EFI will only resort to the fallback loader if no other loader could be found. As there can only be one fallback loader, it is not wise to install FreeBSD's loader into /efi/boot. Windows for example installs both a fallback loader and a vendor loader, so the /efi/boot directory might be taken already. This is why we properly rename /efi/boot/ to a "vendor directory" according to the UEFI specification. I chose /efi/freebsd above but other names are fine, too. Also remember that FAT is case insensitive.

In my opinion, the above steps (or some equivalent) should just be done by the actual FreeBSD installer. For every existing partition, there should be something like two checkmarks: Use partition, Erase partition. This is how e.g. Debian lets the user choose to create a new ESP or to install to an existing one. The user should have this choice.

Regarding GRUB, the official packages and ports are obviously broken at the moment. I tried every possible combination, it just does not work. The ports do not build and none of the packages contain any of the necessary utilities such as grub-mkconfig, grub-install etc. See my previous posts above for details.

kpa suggested installing ports directly from SVN, but I really don't feel like it. How would those be more usable than the ones that are shipped officially? If the default ports and packages weren't fine, why do they even exist and how do they differ? I also do not understand why so many people seem to take on the burden of building ports for themselves when the official packages are just built from ports with default configuration. But don't bother answering those questions. I guess I could learn more about that matter on my own but I already wasted enough time to conclude that I won't bother for the moment.

 

[tingo]

FWIW, you can just use /boot/boot1.efi as the loader instead of thew whole extracting thing. It might need to be renamed bootx64.efi, not sure.