Other GELI/ZFS

H

holm

\Hi all,

I've just registered to ask this..

I've build a new PC for my personal use, I've installed FreeBSD 13 with the sysinstall on 4 SAS Disks encrypted with geli, so far so good, the system is up and running fine so far. Now I want to add an 2nd ZFS Pool consisting of two 1TB SATA disks in striped configuration with geli additional to be used as kind of a data landfill...distfiles and similar things that I want to have handy but can be replaced if the pool fails..

My Problem is now how I can get the disks added w/o to have to enter a 2nd passphrase at boot time? geli(8) says that all providers listed at init will get the same passphrase key combination..that's what I want, but I cant reinit the existing disks because of their contents that I don't want to loose.

How can I add the 2nd two disks?

Kind regards,
Holm
 
holm said:
My Problem is now how I can get the disks added w/o to have to enter a 2nd passphrase at boot time?
Click to expand...
The mistake I made when doing this was at the geli init stage - only use the -b for drives you want to enter the passphrase for at boot time. (You can use geli configure -B to remove the flag).

I'm not entirely sure what you want to do - just keep an eye on the -b flag.
 
You can grep to find the actual options the installer used for geli init on the original drives:

Code: 
# grep geli /var/log/bsdinstall_log
[...]
DEBUG: zfs_create_boot: geli init -bg -e AES-XTS -J - -l 256 -s 4096 "ada0p3"
[...]

Compare those options to the man page to make sure they look correct. Certainly you'll need to remove -J - to run the geli init command interactively for the new drives. If you use the same options and specify the same passwords, they might just mount at boot like you expect. Use geli status after booting to see if the drives mounted properly. If not, it will likely be a minor configuration issue that someone here will know how to solve.
 
holm said:
... how I can get the disks added w/o to have to enter a 2nd passphrase at boot time? geli(8) says that all providers listed at init will get the same passphrase key combination..that's what I want, but I cant reinit the existing disks because of their contents that I don't want to loose.
Click to expand...
You don't have to reinit the existing disks again.

After creating partition tables and partitions on the 2 disks (you might want to set aliment to 4k, see gpart(8) -a), initialize the 2 disks with the -b option on and the same passphrase the 4 disks were initialized with.

No need to set "-e AES-XTS", geli defaults to it. Set key length (-l) to 256, and sector size (-s) to 4096.

Example: geli init -b -l 256 -s 4096 ada4p1 ada5p1

After attaching the initialized providers, create the pool on the two partitions (make sure to use the virtual devices with the .eli suffix).

When the system comes up, the passphrase is asked once. The -b option the two disks were initialized with allows the providers on the 2 disks to be decrypted without asking the same passphrase a second time.
 
Last edited:
Hmm ..it didn't work as expacted for now, but I don't have the zfs stripe alredy build.
There are two disks like this:
root@trollo:/home/holm # gpart show ada0
=> 40 1953525088 ada0 GPT (932G)
40 1952448512 1 freebsd-zfs (931G)
1952448552 1076576 - free - (526M)

there fore I've entered
geli init -b -l 256 -s 4096 ada4p1 ada0p1 ada1p1

(DEBUG: zfs_create_boot: geli init -bg -e AES-XTS -J - -l 256 -s 4096 "da0p3" was in bsdinstall_log)

I've got asked for the passphrase twice and after reboot I had to enter the passphrase for da0p3.
The kernel boots and later I where asked to enter the (same)passphrase for for ada0p1 after probing for disk devices and after that all .eli devices are created.

I"m trying to build the zfs stripe next and will look how that works..

Regards,
Holm
 
Ok, build the pool zrdata and the behavior is exactly as described above, I get asked twice for the passphrase, wich is really not what I want ...

holm@trollo:~ $ zfs mount
zrpool/ROOT/default /
zrdata /zrdata
zrpool/var/log /var/log
zrpool/tmp /tmp
zrpool/usr/home /usr/home
zrpool/var/crash /var/crash
zrpool/var/tmp /var/tmp
zrpool/var/audit /var/audit
zrpool /zrpool
zrpool/usr/ports /usr/ports
zrpool/usr/src /usr/src
zrpool/var/mail /var/mail
holm@trollo:~ $ zpool status
pool: zrdata
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
zrdata ONLINE 0 0 0
ada0p1.eli ONLINE 0 0 0
ada1p1.eli ONLINE 0 0 0

errors: No known data errors

pool: zrpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
zrpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da0p3.eli ONLINE 0 0 0
da1p3.eli ONLINE 0 0 0
da2p3.eli ONLINE 0 0 0
da3p3.eli ONLINE 0 0 0

errors: No known data errors
holm@trollo:~ $
 
Personally i find a combination of geli and automatic password a bit contradictorial. Meaning sometimes you are better of without geli or without automatic password.
 
ccammack said:
I'm a little concerned that the passwords might accidentally be different or perhaps -g is required for this case.
Click to expand...
I get asked twice for the passphrase during geli init, and I get asked twice when the system boots, don't you think I should have recognized that I'm using accidentally two different passwords?

-g probably enables booting from such disks, what if there is no bootable slice?

Regards,
Holm
 
Alain De Vos said:
Personally i find a combination of geli and automatic password a bit contradictorial. Meaning sometimes you are better of without geli or without automatic password.
Click to expand...
Maybe it's since I'm a german and never learned english in the school or so, that I'm unable to express myself in english, but I think should be pretty clear that no one whises to use geli with an automated passphrase. If you read my first post again, you should be able to understand that I only whish to be asked once for an (identical) passphrase for decrypting two disk pools...
If I had done a geli init over all disks at install time, this would proably already the case. The difference is, that I've added a disk pool later, since the two additional disks where simply not available at install time.

Regards,

Holm


In the meantime I used geli configure -b -g at ada0p1 and ada1p1 to add the -g flag (GELIBOOT) ..it changed nothing.
Adding geom_eli_passphrase_prompt="YES" to /boot/loader.conf displays the (2nd) geli prompt just before the FreeBSD boot menu, not in the middle of kernel messages, this is somewhat useful..but doesn't help to get rid of the problem.

Regards,

Holm
 
Code: 
Geom name: ada0p1.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, AUTORESIZE
KeysAllocated: 233
KeysTotal: 233
Providers:
1. Name: ada0p1.eli
   Mediasize: 999653634048 (931G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: ada0p1
   Mediasize: 999653638144 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1

Geom name: ada1p1.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, AUTORESIZE
KeysAllocated: 233
KeysTotal: 233
Providers:
1. Name: ada1p1.eli
   Mediasize: 999653634048 (931G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: ada1p1
   Mediasize: 999653638144 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
Geom name: da0p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT, AUTORESIZE
KeysAllocated: 210
KeysTotal: 210
Providers:
1. Name: da0p3.eli
   Mediasize: 898036133888 (836G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: da0p3
   Mediasize: 898036137984 (836G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2148532224
   Mode: r1w1e1

Geom name: da1p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT, AUTORESIZE
KeysAllocated: 210
KeysTotal: 210
Providers:
1. Name: da1p3.eli
   Mediasize: 898036133888 (836G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: da1p3
   Mediasize: 898036137984 (836G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2148532224
   Mode: r1w1e1
Geom name: da2p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT, AUTORESIZE
KeysAllocated: 210
KeysTotal: 210
Providers:
1. Name: da2p3.eli
   Mediasize: 898036133888 (836G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: da2p3
   Mediasize: 898036137984 (836G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2148532224
   Mode: r1w1e1

Geom name: da3p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT, AUTORESIZE
KeysAllocated: 210
KeysTotal: 210
Providers:
1. Name: da3p3.eli
   Mediasize: 898036133888 (836G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: da3p3
   Mediasize: 898036137984 (836G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2148532224
   Mode: r1w1e1
Geom name: da0p2.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: accelerated software
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN, AUTORESIZE
KeysAllocated: 1
KeysTotal: 1
Providers:
1. Name: da0p2.eli
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 4096
   Mode: r1w1e0
Consumers:
1. Name: da0p2
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r1w1e1

Geom name: da1p2.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: accelerated software
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN, AUTORESIZE
KeysAllocated: 1
KeysTotal: 1
Providers:
1. Name: da1p2.eli
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 4096
   Mode: r1w1e0
Consumers:
1. Name: da1p2
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r1w1e1

Geom name: da2p2.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: accelerated software
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN, AUTORESIZE
KeysAllocated: 1
KeysTotal: 1
Providers:
1. Name: da2p2.eli
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 4096
   Mode: r1w1e0
Consumers:
1. Name: da2p2
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r1w1e1

Geom name: da3p2.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: accelerated software
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN, AUTORESIZE
KeysAllocated: 1
KeysTotal: 1
Providers:
1. Name: da3p2.eli
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 4096
   Mode: r1w1e0
Consumers:
1. Name: da3p2
   Mediasize: 2147483648 (2.0G)
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r1w1e1

..dont forget to install the root raidz1 pool first, then a system and then add the two disks later...

I've tried to attach a text file to the post but got the information that the post has to wait for approval.
Have no interest on such kind of shit, so now as copy and past...

Regards,
Holm
 
holm said:
..dont forget to install the root raidz1 pool first, then a system and then add the two disks later...
Click to expand...
Exact setup and order you described in post #1. Check out geli status/zpool status/dmesg -a: https://termbin.com/rfnk

geli list wasn't helpful. Since I can't reproduce your systems situation (the passphrase is asked always only once for disk0p3 on my test system), and nothing comes to my mind what the cause could be, I'm out of ideas.

I can offer a workaround to circumvent the annoyance of the second passphrase query.

You could initialize the 2 disks with a key file. The key file would be stored save in the encrypted "zrpool" pool, until it's decrypted.

Example to automate the decryption (from /etc/defaults/rc.conf, "GELI disk encryption configuration" and "Example use"):
Code: 
geli_devices="ada0p1 ada1p1"
geli_ada0p1_flags="-p -k /root/zrdata.key"
geli_ada1p1_flags="-p -k /root/zrdata.key"

You could also try asking on other community channels. Maybe someone more familiar with geli(8) might have a thought what triggers the second passphrase query.
 
Yea, THX T-Daemon, I came over this possibility with the keyfile some 30 minutes before, I've looked at /etc/defaults/rc.conf because of some ps/2 mouse related hassle (Logitech Marble FX wouldn't work together with an Model M on the single one PS/2 connector that the mainboard has, seems I have to get an PS/2 to USB Converter for the Marble) and stumbled over some geli related examples there. For sure this is a good way to go.

Unclear to me is where geli decides for which of the devices the already entered passphrase is to use and for which not. You used da and ad disks like me..and I'm running an almost Generic kernel here.

For completeness my dmesg output:

---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-STABLE #1 stable/13-n248259-b61ed468809: Wed Dec 8 20:03:03 CET 2021
holm@trollo.tsht.lan:/usr/obj/usr/src/amd64.amd64/sys/TROLLO amd64
FreeBSD clang version 12.0.1 (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-0-gfed41342a82f)
VT(vbefb): resolution 1920x1200
CPU: AMD Ryzen 5 3600 6-Core Processor (3600.07-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x870f10 Family=0x17 Model=0x71 Stepping=0
Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x75c237ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX,ADMSKX>
Structured Extended Features=0x219c91a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
Structured Extended Features2=0x400004<UMIP,RDPID>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
AMD Extended Feature Extensions ID EBX=0x108b657<CLZERO,IRPerf,XSaveErPtr,RDPRU,WBNOINVD,IBPB,STIBP,SSBD>
SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
TSC: P-state invariant, performance statistics
real memory = 34359738368 (32768 MB)
avail memory = 33329831936 (31785 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 2 cache groups x 3 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.1> irqs 0-23
ioapic1 <Version 2.1> irqs 24-55
Launching APs: 7 6 1 5 4 11 10 8 2 3 9
Timecounter "TSC-low" frequency 1800034632 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 350
Event timer "HPET2" frequency 14318180 Hz quality 350
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 1.3 on pci0
pci1: <ACPI PCI bus> on pcib1
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xf7aa0000-0xf7aa7fff irq 32 at device 0.0 on pci1
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
ahci0: <AHCI SATA controller> mem 0xf7a80000-0xf7a9ffff irq 33 at device 0.1 on pci1
ahci0: AHCI v1.31 with 8 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
pcib2: <ACPI PCI-PCI bridge> irq 34 at device 0.2 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 32 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> mem 0xf7400000-0xf7401fff irq 32 at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0xe000-0xe0ff mem 0xf7200000-0xf721ffff,0xf7300000-0xf731ffff irq 32 at device 8.0 on pci4
mpt0: MPI Version=1.2.14.0
pcib5: <ACPI PCI-PCI bridge> irq 33 at device 1.0 on pci2
pci5: <ACPI PCI bus> on pcib5
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xd000-0xd0ff mem 0xf7904000-0xf7904fff,0xf7900000-0xf7903fff irq 33 at device 0.0 on pci5
re0: Using 1 MSI-X message
re0: Chip rev. 0x54000000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 00:d8:61:a1:c3:ea
re0: netmap queues/slots: TX 1/256, RX 1/256
pcib6: <ACPI PCI-PCI bridge> irq 32 at device 4.0 on pci2
pci6: <ACPI PCI bus> on pcib6
mpt1: <LSILogic SAS/SATA Adapter> port 0xc000-0xc0ff mem 0xf7610000-0xf7613fff,0xf7600000-0xf760ffff irq 32 at device 0.0 on pci6
mpt1: MPI Version=1.5.18.0
mpt1: Capabilities: ( RAID-0 RAID-1E RAID-1 )
mpt1: 0 Active Volumes (2 Max)
mpt1: 0 Hidden Drive Members (14 Max)
pcib7: <ACPI PCI-PCI bridge> irq 33 at device 5.0 on pci2
pci7: <ACPI PCI bus> on pcib7
isp0: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xb100-0xb1ff mem 0xf7884000-0xf7887fff irq 33 at device 0.0 on pci7
isp0: Mailbox Command (0x8) Timeout (5000000us) (isp_reset:439)
isp0: Mailbox Command 'ABOUT FIRMWARE' failed (TIMEOUT)
isp0: isp_reinit: cannot reset card
isp0: See the ispfw(4) man page on how to load known good firmware at boot time
device_attach: isp0 attach returned 6
isp0: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xb000-0xb0ff mem 0xf7880000-0xf7883fff irq 34 at device 0.1 on pci7
isp0: Mailbox Command (0x8) Timeout (5000000us) (isp_reset:439)
isp0: Mailbox Command 'ABOUT FIRMWARE' failed (TIMEOUT)
isp0: isp_reinit: cannot reset card
isp0: See the ispfw(4) man page on how to load known good firmware at boot time
device_attach: isp0 attach returned 6
pcib8: <ACPI PCI-PCI bridge> irq 34 at device 6.0 on pci2
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> irq 35 at device 7.0 on pci2
pci9: <ACPI PCI bus> on pcib9
uart2: <MosChip MCS9922 PCIe to Peripheral Controller> port 0xa010-0xa017 mem 0xf7703000-0xf7703fff,0xf7702000-0xf7702fff irq 35 at device 0.0 on pci9
uart3: <MosChip MCS9922 PCIe to Peripheral Controller> port 0xa000-0xa007 mem 0xf7701000-0xf7701fff,0xf7700000-0xf7700fff irq 32 at device 0.1 on pci9
pcib10: <ACPI PCI-PCI bridge> at device 3.1 on pci0
pci10: <ACPI PCI bus> on pcib10
vgapci0: <VGA-compatible display> port 0xf000-0xf07f mem 0xf6000000-0xf6ffffff,0xe0000000-0xefffffff,0xf0000000-0xf1ffffff irq 54 at device 0.0 on pci10
vgapci0: Boot video device
hdac0: <NVIDIA (0x0e0a) HDA Controller> mem 0xf7080000-0xf7083fff irq 55 at device 0.1 on pci10
pcib11: <ACPI PCI-PCI bridge> at device 7.1 on pci0
pci11: <ACPI PCI bus> on pcib11
pci11: <unknown> at device 0.0 (no driver attached)
pcib12: <ACPI PCI-PCI bridge> at device 8.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pci12: <unknown> at device 0.0 (no driver attached)
pci12: <encrypt/decrypt> at device 0.1 (no driver attached)
xhci1: <XHCI (generic) USB 3.0 controller> mem 0xf7b00000-0xf7bfffff irq 39 at device 0.3 on pci12
xhci1: 64 bytes context size, 64-bit DMA
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
hdac1: <AMD X570 HDA Controller> mem 0xf7d00000-0xf7d07fff irq 36 at device 0.4 on pci12
pcib13: <ACPI PCI-PCI bridge> at device 8.2 on pci0
pci13: <ACPI PCI bus> on pcib13
ahci1: <AMD KERNCZ AHCI SATA controller> mem 0xf7f00000-0xf7f007ff irq 41 at device 0.0 on pci13
ahci1: AHCI v1.31 with 1 6Gbps ports, Port Multiplier supported with FBS
ahcich8: <AHCI channel> at channel 0 on ahci1
pcib14: <ACPI PCI-PCI bridge> at device 8.3 on pci0
pci14: <ACPI PCI bus> on pcib14
ahci2: <AMD KERNCZ AHCI SATA controller> mem 0xf7e00000-0xf7e007ff irq 45 at device 0.0 on pci14
ahci2: AHCI v1.31 with 1 6Gbps ports, Port Multiplier supported with FBS
ahcich9: <AHCI channel> at channel 0 on ahci2
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
acpi_button0: <Power Button> on acpi0
ppc1: <Parallel port> port 0x378-0x37f irq 5 on acpi0
ppc1: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc1
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROM> at iomem 0xce800-0xcefff pnpid ORM0000 on isa0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
hdacc0: <NVIDIA (0x0040) HDA CODEC> at cad 0 on hdac0
hdaa0: <NVIDIA (0x0040) Audio Function Group> at nid 1 on hdacc0
pcm0: <NVIDIA (0x0040) (HDMI/DP 8ch)> at nid 4 on hdaa0
pcm1: <NVIDIA (0x0040) (HDMI/DP 8ch)> at nid 5 on hdaa0
pcm2: <NVIDIA (0x0040) (HDMI/DP 8ch)> at nid 6 on hdaa0
pcm3: <NVIDIA (0x0040) (HDMI/DP 8ch)> at nid 7 on hdaa0
hdacc1: <Realtek ALC892 HDA CODEC> at cad 0 on hdac1
hdaa1: <Realtek ALC892 Audio Function Group> at nid 1 on hdacc1
pcm4: <Realtek ALC892 (Rear Analog 7.1/2.0)> at nid 20,22,21,23 and 24,26 on hdaa1
pcm5: <Realtek ALC892 (Front Analog)> at nid 27 and 25 on hdaa1
Trying to mount root from zfs:zrpool/ROOT/default []...
Root mount waiting for: usbus0 CAM usbus1
ugen1.1: <0x1022 XHCI root HUB> at usbus1
uhub0 on usbus1
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
ugen0.1: <0x1022 XHCI root HUB> at usbus0
uhub1 on usbus0
uhub1: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 8 ports with 8 removable, self powered
ugen1.2: <vendor 0x0424 product 0x2514> at usbus1
uhub2 on uhub0
uhub2: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 1> on usbus1
uhub2: MTT enabled
Root mount waiting for: usbus0 CAM usbus1
uhub1: 22 ports with 22 removable, self powered
uhub2: 4 ports with 4 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich4 bus 0 scbus2 target 0 lun 0
ada0: <ST1000DM010-2EP102 CC43> ATA8-ACS SATA 3.x device
ada0: Serial Number ZN1HT4HD
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors)
ada0: quirks=0x1<4K>
ada1 at ahcich5 bus 0 scbus3 target 0 lun 0
ada1: <WDC WD10EZEX-00RKKA0 80.00A80> ATA8-ACS SATA 3.x device
ada1: Serial Number WD-WCC1S2514022
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors)
da2 at mpt1 bus 0 scbus5 target 2 lun 0
da2: <NETAPP X423_TAL13900A10 NA01> Fixed Direct Access SCSI-3 device
da2: Serial Number X3S0A0GNFUZ5
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 858483MB (1758174768 512 byte sectors)
da0 at mpt1 bus 0 scbus5 target 0 lun 0
da0: <NETAPP X423_TAL13900A10 NA01> Fixed Direct Access SCSI-3 device
da0: Serial Number X3T0A03UFUZ5
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 858483MB (1758174768 512 byte sectors)
da3 at mpt1 bus 0 scbus5 target 3 lun 0
da3: <NETAPP X423_TAL13900A10 NA01> Fixed Direct Access SCSI-3 device
da3: Serial Number X3T0A05CFUZ5
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 858483MB (1758174768 512 byte sectors)
da1 at mpt1 bus 0 scbus5 target 1 lun 0
da1: <NETAPP X423_TAL13900A10 NA01> Fixed Direct Access SCSI-3 device
da1: Serial Number X3T0A097FUZ5
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 858483MB (1758174768 512 byte sectors)
Enter passphrase for ada0p1: GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 858483MB (1758174768 512 byte sectors)
da0 at mpt1 bus 0 scbus5 target 0 lun 0
da0: <NETAPP X423_TAL13900A10 NA01> Fixed Direct Access SCSI-3 device
da0: Serial Number X3T0A03UFUZ5
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 858483MB (1758174768 512 byte sectors)
da3 at mpt1 bus 0 scbus5 target 3 lun 0
da3: <NETAPP X423_TAL13900A10 NA01> Fixed Direct Access SCSI-3 device
da3: Serial Number X3T0A05CFUZ5
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 858483MB (1758174768 512 byte sectors)
da1 at mpt1 bus 0 scbus5 target 1 lun 0
da1: <NETAPP X423_TAL13900A10 NA01> Fixed Direct Access SCSI-3 device
da1: Serial Number X3T0A097FUZ5
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 858483MB (1758174768 512 byte sectors)
Enter passphrase for ada0p1: GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da0p3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da1p3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da2p3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da3p3.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da0p2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da1p2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da2p2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: accelerated software
GEOM_ELI: Device da3p2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI: Crypto: accelerated software
nvidia0: <Quadro K5000> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 470.86 Tue Oct 26 21:43:42 UTC 2021
amdsmn0: <AMD Family 17h System Management Network> on hostb0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb0
intsmb0: <AMD FCH SMBus Controller> at device 20.0 on pci0
smbus0: <System Management Bus> on intsmb0
isp0: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xb100-0xb1ff mem 0xf7884000-0xf7887fff irq 33 at device 0.0 on pci7
isp1: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xb000-0xb0ff mem 0xf7880000-0xf7883fff irq 34 at device 0.1 on pci7
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: cannot find EC device
acpi_wmi0: Embedded MOF found
ACPI: \134AOD.WQBA: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
driver bug: Unable to set devclass (class: ppc devname: (unknown))
acpi_wmi1: <ACPI-WMI mapping> on acpi0
acpi_wmi1: cannot find EC device
acpi_wmi2: <ACPI-WMI mapping> on acpi0
acpi_wmi2: cannot find EC device
acpi_wmi2: Embedded MOF found
ACPI: \134_SB.WMIC.WQBA: 1 arguments were passed to a non-method ACPI object (Buffer) (20201113/nsarguments-361)
lo0: link state changed to UP
re0: link state changed to DOWN
re0: link state changed to UP
Security policy loaded: MAC/ntpd (mac_ntpd)
ugen0.2: <A4Tech Wireless Battery Free Optical Mouse> at usbus0
ums0 on uhub1
ums0: <A4Tech Wireless Battery Free Optical Mouse, class 0/0, rev 2.00/0.01, addr 1> on usbus0
ums0: 8 buttons and [XYZT] coordinates ID=0
.

Look for "enter passphrase" in there..

root@trollo:/home/holm # geli status
Name Status Components
ada0p1.eli ACTIVE ada0p1
ada1p1.eli ACTIVE ada1p1
da0p3.eli ACTIVE da0p3
da1p3.eli ACTIVE da1p3
da2p3.eli ACTIVE da2p3
da3p3.eli ACTIVE da3p3
da0p2.eli ACTIVE da0p2
da1p2.eli ACTIVE da1p2
da2p2.eli ACTIVE da2p2
da3p2.eli ACTIVE da3p2
root@trollo:/home/holm # zpool status
pool: zrdata
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
zrdata ONLINE 0 0 0
ada0p1.eli ONLINE 0 0 0
ada1p1.eli ONLINE 0 0 0

errors: No known data errors

pool: zrpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
zrpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da0p3.eli ONLINE 0 0 0
da1p3.eli ONLINE 0 0 0
da2p3.eli ONLINE 0 0 0
da3p3.eli ONLINE 0 0 0

errors: No known data errors
root@trollo:/home/holm #

THX again,
Holm
 
i took a look at the source code and here is what i understood (caveat emptor / im not a kernel developer)
you don't want geom_eli_passphrase_prompt in loader.conf
the passphrase cache is wiped clean when rootfs mounts
the "other" geli devices need to have -b or -g set
so if your other geli devs have not been probed by then you are asked for a pass again
try to play with kern.cam.boot_delay ?
 
I've already commented out the geom_eli_passphrase_prompt in loader.conf, therefore I get the 2nd question
in the middle of the kernel boot.

I get 6 disks listed from the loader "...bios drive 0 is ..." after which I get the first Prompt for the passphrase,
After entering it, fours disks of six get decrypted with some iterations.
(sorry for the not exact messages)
While kernel boot I get
Code: 
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich4 bus 0 scbus2 target 0 lun 0
ada0: <ST1000DM010-2EP102 CC43> ATA8-ACS SATA 3.x device
ada0: Serial Number ZN1HT4HD
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors)
ada0: quirks=0x1<4K>
ada1 at ahcich5 bus 0 scbus3 target 0 lun 0
ada1: <WDC WD10EZEX-00RKKA0 80.00A80> ATA8-ACS SATA 3.x device
ada1: Serial Number WD-WCC1S2514022
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors)

..."Root mount waiting for CAM" and next the disks getting recognized.
Don't you think that kern.cam.boot_delay means this delay?

Regards,
Holm

PS: Why are messages from me waiting for approval?
 
password is cached until rootfs is mounted
EVENTHANDLER_DEFINE(mountroot, zero_intake_passcache, NULL, 0);
its not clear to me if the event is fired before mount-root is tried or after it succeeds
but zero_intake_passcache will wipe the cached passphrase
so i speculated that if you can somehow delay mount root until the geli sees/(tastes) the devices it might work
 
I don't think that the root mount is the problem here. For the logic, the loader has to load kernel and modules from the already decrypted disk, therefore the possible boot disks must be already decrypted to this time, bit in my case it don't touch the two ada disks. They are initialized with geli with the flags -g -b, but in my case they don't have a bootable partition. Maybe that's the cause the disks getting ignored at loader stage?
In the moment I don't have any content on that disk that can't be deleted (copies still in the /home filesystem) so maybe I should try to make them bootable disks, at least with a boot partition?

T-Daemon how you have partitioned the ada disks?
Regards,

Holm
 
try the following experiment
stop the loader and set
vfs.root.mountfrom="ufs:/dev/ada0p2"
when mount root fails input the real rootfs
see if ada partitions get geli attached
 
covacat, all of the systems providers are ZFS vdev's. vfs.root.mountfrom=ufs: wouldn't do any good.

holm said:
T-Daemon how you have partitioned the ada disks?
Click to expand...
Code: 
# gpart create -s gpt ada0
# gpart add -t freebsd-zfs -a 4k ada0
Same for ada1.

holm said:
I get 6 disks listed from the loader "...bios drive 0 is ..."
Click to expand...
I can point at what the issue is.

It seems BIOS systems are affected with the second time passphrase query of later geli initialized providers with an identical passphrase used and the -b (boot) flag set.

I'm accustom to create UEFI VM's for testings. I didn't expected it would make a difference and didn't ask on which firmware your system is running.

Your systems issue is reproducible on a BIOS system VM. In case of the test system, the boot process is interrupted after the base system providers and before the swap providers are decrypted, asking for the same passphrase to decrypt the first disks provider of those later added two.

This is a bug. I'm not sure which part of the bootstrapping process handles geli passphrases, but the issue is, the geli passphrase entered at the very beginning isn't exported into the loaders boot environment on a BIOS system.

Comparing the BIOS and UEFI loader environments, the UEFI loader environment has set variable kern.geom.eli.passphrase=<passphrase here in clear text>, BIOS has not.

Setting the variable manually corrects the issue, a second passphrase isn't asked during boot anymore.

To reproduce: After the first passphrase is asked and entered, at the Boot menu, enter "3. Escape to loader prompt", set:
Code: 
show
(shows all the variables, geli passphrase is missing)

set kern.geom.eli.passphrase=<passphrase in clear text>
boot

If the problem is important to you, you should open a bug report. Meanwhile, you could apply following workaround:

/boot/loader.conf
Code: 
kern.geom.eli.passphrase=<passphrase>

If you decide to report the bug set a link to this forums thread and this post.
 
Last edited:
T-Daemon said:
covacat, all of the systems providers are ZFS vdev's. vfs.root.mountfrom=ufs: wouldn't do any good
Click to expand...
i know, i just want to force the kernel to bring the devices up before attempting to mount root (to test the below)
geli cached password is zeroed out when rootfs gets mounted
so if his ada devs come up after root mount his passphrase is lost
 
T-Daemon: I can fully confirm what you've investigated :)
After setting kern.geom.eli.passphrase at the loader prompt I get no additional prompts for that Passphrase.
For now I've entered the pass in /boot/loader.conf, but I think I should send a bug report as you suggested.

THX @ all from germany!

Holm
 
You must log in or register to reply here.
Back
Top