Solved zpool missing after increasing disk size (AWS)

yep, glabel is pretty sweet — here's our setup, the swaps are actually gmirrors :)
Code: 
# glabel status
        Name  Status  Components
    gpt/efi0     N/A  nvd0p1
  gpt/boot/0     N/A  nvd0p2
    gpt/efi1     N/A  nvd1p1
  gpt/boot/1     N/A  nvd1p2
gpt/swap/b/0     N/A  da4p1
  gpt/pool/4     N/A  da4p2
gpt/swap/a/0     N/A  da0p1
  gpt/pool/0     N/A  da0p2
gpt/swap/a/2     N/A  da2p1
  gpt/pool/2     N/A  da2p2
gpt/swap/a/3     N/A  da3p1
  gpt/pool/3     N/A  da3p2
gpt/swap/a/1     N/A  da1p1
  gpt/pool/1     N/A  da1p2
gpt/swap/b/3     N/A  da7p1
  gpt/pool/7     N/A  da7p2
gpt/swap/b/2     N/A  da6p1
  gpt/pool/6     N/A  da6p2
gpt/swap/b/1     N/A  da5p1
  gpt/pool/5     N/A  da5p2
 
dvl@ said:
Overnight, I had recurring dreams/hallucinations of how using device labels (e.g. gtp/zfs0) instead of partition names (e.g. /dev/nda2p1) would have avoided this. It kept going through my head (I was dealing with a fever) most of the night.

Today I found I have done this in the past: https://dan.langille.org/2019/10/15/going-from-partition-to-label-in-zpool-status/

The above solution deals with a mirror. The procedure might not translate well to a single drive zpool (remember this is an AWS host, so it's not a physical drive).
Click to expand...
Yes, so labeling on creating partition for ZFS pool and using it for creating the pool would be important.

This would NOT be storngly wanted for usual clean installation using bsdinstall, but helps to ensure installations without any installer, attaching existing but not the targeted computer to create working drive (i.e., attach via USB converter [/dev/da*]) and attaching to target computer afterwards (i.e., as /dev/nda*).
 
dvl@ said:
Overnight, I had recurring dreams/hallucinations of how using device labels (e.g. gtp/zfs0) instead of partition names (e.g. /dev/nda2p1) would have avoided this. It kept going through my head (I was dealing with a fever) most of the night.

Today I found I have done this in the past: https://dan.langille.org/2019/10/15/going-from-partition-to-label-in-zpool-status/

The above solution deals with a mirror. The procedure might not translate well to a single drive zpool (remember this is an AWS host, so it's not a physical drive).
Click to expand...
For Dan and FreshPorts the problem came from how the swap partition was specified in /etc/fstab. Using labels for the ZFS partitions (e.g. gpt/zfs0 instead of nvd2p1/nda2p1) would not have prevented the problem.

In general, ZFS is capable of automatically discovering the disks and/or partitions belonging to a pool based on the ZFS labels they contain. Those ZFS labels contain GUIDs which protect against disk renumbering.

I made an experiment which everyone can repeat. I installed FreeBSD 14.3 in a VM with three virtual disks. The disks were initially SATA disks with names ada0, ada1 and ada2. I chose root-on-ZFS in the installer and gave it one disk (ada0). This led to having a zpool named zroot based on ada0p3. After booting into the installed system I created a second zpool named mydata which was a mirror based on ada1 and ada2. Then I shut down the VM and changed the type of ada2 from SATA to SCSI. Now the virtual disk was known as da0. The system booted successfully with all zpools and their vdevs being ONLINE. Then I stopped the VM again and changed ada1 to be da1. Again, everything worked perfectly. The last part of the experiment was changing ada0 to be an NVMe device (nda0). The system booted successfully, all zpools and vdevs were ONLINE. The only problem was that the swap device was specified in /etc/fstab as /dev/ada0p2 so the swapon command executed automatically during system startup failed. The VM was running but without any active swap devices.

What would have prevented the problem which affected FreshPorts is simply using a label for the swap device (e.g. /dev/gpt/swap).
 
Here's a followup. Let's label the swap drive:

Code: 
[21:41 aws-1 dan ~] % gpart show
=>       40  524287920  nda0  GPT  (250G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048  524285912     2  freebsd-zfs  (250G)

=>      40  16777136  nda1  GPT  (8.0G)
        40  16777136     1  freebsd-swap  (8.0G)

=>       40  524287920  nda2  GPT  (250G)
         40  524287920     1  freebsd-zfs  (250G)

[21:41 aws-1 dan ~] % sudo gpart modify -i 1 -l swap01 nda1
nda1p1 modified
[21:42 aws-1 dan ~] % gpart show nda1
=>      40  16777136  nda1  GPT  (8.0G)
        40  16777136     1  freebsd-swap  (8.0G)

[21:42 aws-1 dan ~] %

[21:46 aws-1 dan ~] % ls -l /dev/gpt/swap01 
crw-r-----  1 root operator 0x7f 2025.09.09 21:42 /dev/gpt/swap01

Let's try that entry in /etc/fstab :

Code: 
/dev/gpt/swap01 none swap sw         0 0

Can we turn this on?
Code: 
[21:48 aws-1 dan ~] % sudo swapon -a
swapon: adding /dev/gpt/swap01 as swap device
[21:48 aws-1 dan ~] % swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/gpt/swap01   8388568        0  8388568     0%
[21:48 aws-1 dan ~] %

I think this should work.
 
dvl@ said:
Code: 
[21:41 aws-1 dan ~] % sudo gpart modify -i 1 -l swap01 nda1
nda1p1 modified
[21:42 aws-1 dan ~] % gpart show nda1
=>      40  16777136  nda1  GPT  (8.0G)
        40  16777136     1  freebsd-swap  (8.0G)

[21:42 aws-1 dan ~] %

[21:46 aws-1 dan ~] % ls -l /dev/gpt/swap01
crw-r-----  1 root operator 0x7f 2025.09.09 21:42 /dev/gpt/swap01
Click to expand...
btw you can also confirm the gpart modify action by using glabel status as shown in our earlier message
 
atax1a said:
btw you can also confirm the gpart modify action by using glabel status as shown in our earlier message
Click to expand...
Code: 
[22:19 aws-1 dan ~] % glabel status
      Name  Status  Components
gpt/swap01     N/A  nda1p1
 
just recently i had to install system and was surprised it labeled (did it?) swap but didn't use them in fstab. i was like wait who does that. that would allow system to shit, i mean, swap all over the disks

i changed it for those obvious reasons, but, damn... so that gets some fix?

at least some actual fix

unsure how do deal with existing installs but, at least new installs should get at minimal, labels somewhere. but ideally some markings so it will not write all over the place! i'm surprised it went on so long

i mean, there could be other disks in machine. installer is very careful and asks about write a lot. imagine system just swapping later on random device names that were there during install in that order

yeeeah that's bad. let's fix it!
 
Disk Labels are not unique so they are not good solution. The only good solution is to use unique id which is GUID. If you move disks between the computers which have auto label like efiboot0, zfs0, swap0 you will have two disk with the same labels on the same computer.
 
fair point!

but swapping onto wrong swap0 could be less of a problem

only other way is to indeed somehow mark the swap part to be only used in that one machine. unsure how. did gpt had partition id's too? who's going to keep track of this even

so partition would be typed swap and id'ed to specific host. and what if this is mbr?

what if... what was that other concern i had...

anyway, yeah, bad ideas
 
however we disable them due this causing zfs to pick up on random disks by random ids/names

/boot/loader.conf:
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"

?
 
VladiBG said:
Disk Labels are not unique so they are not good solution. The only good solution is to use unique id which is GUID. If you move disks between the computers which have auto label like efiboot0, zfs0, swap0 you will have two disk with the same labels on the same computer.
Click to expand...
I prefer partition labels. Admins can set unique (at least throughout the organization) label and would be easier to remember / understand than GUID.

Another thing I suspect is disk images. Does disk images copied from somewhere (including backups) assured to have really globally unique GUID?

This means, to ensure the GUID is really unique, it must be updated on copy, auto"magic"ally, with all configurations chase it auto"magic"ally.. But it would hurt restores from backups. To avoid it, the automatic update shouldn't updated "only when restored from backups".

This is why I prefer labels which is required to intentionally set at some point and can be documented in the procedures.
 
You must log in or register to reply here.
Back
Top