ZFS How to extend zfs geli encrypted disk? Space not showing

Ok - so I did this and it's resized (using resize without specifying size actually handles it automatically!)
Very very nervous about this last step on my live system ?
sudo gpart resize -i 3 ada0
Password:
ada0p3 resized
[c1utt4r@toaster /usr/src]$ gpart show
=> 40 976773088 ada0 GPT (466G)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 959993864 3 freebsd-zfs (458G)
Click to expand...
However - now I am stuck at the last step (hopefully this has worked out right -but not sure)
Should I be doing (#in from handbook describing it for ufs - but i'm on zfs+geli)
zpool online -e zroot /dev/ada0p2
Click to expand...
OR ( # VladiBG suggested)
zpool online -e ada0p3.eli
Click to expand...
Which one is the correct option? And what's the difference between the two?

Note : I still haven't set zroot autoexpand to "on" - it's still the default "off" - I did set sysctl kern.geom.debugflags=16

zpool list only showing 290G instead of 458G
zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 290G 270G 19.9G - - 69% 93% 1.00x ONLINE -
Click to expand...
 
VladiBG said:
it's zpool online -e zroot ada0p3.eli i corrected my previous post.

zpool-online(8)

www.freebsd.org www.freebsd.org
Click to expand...
Do I need to set autoexpand to "on" ? And this should be the final step? How do I confirm if everything is working fine after this?

UPDATE : I did it without autoexpand "on" - as was suggested earlier here is the output - which was actually no output strangely:
sudo zpool online -e zroot /dev/ada0p3.eli
Password:
Click to expand...
Now the output of zpool list is also same as before :
zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 290G 270G 19.9G - - 69% 93% 1.00x ONLINE -
Click to expand...
Basically no change from 290G to 458G - did I do something wrong? ? - I'm guessing the expected output should have reflected 458G.... or does that happen upon restart? I'm lost here
 
This is the output for "geli list" - seems like the providers is showing the earlier value (290G) while the consumers is showing the resized value (458G) - Is this correct? I suspect not - How do I fix it?
geli list
Geom name: ada0p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT
KeysAllocated: 73
KeysTotal: 73
Providers:
1. Name: ada0p3.eli
Mediasize: 311481593856 (290G)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: ada0p3
Mediasize: 491516858368 (458G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 8590983168
Mode: r1w1e1

Geom name: ada0p2.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: accelerated software
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN, AUTORESIZE
KeysAllocated: 2
KeysTotal: 2
Providers:
1. Name: ada0p2.eli
Mediasize: 8589934592 (8.0G)
Sectorsize: 4096
Mode: r1w1e0
Consumers:
1. Name: ada0p2
Mediasize: 8589934592 (8.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 1048576
Mode: r1w1e1
Click to expand...
 
VladiBG said:
after the resize of ada0p3 did you see a GELI message about new size of ada0p3.eli
Click to expand...
I resized it using gpart - and as I mentioned above the output was this :
sudo gpart resize -i 3 ada0
Password:
ada0p3 resized
Click to expand...
Further it reflected in gpart show :
gpart show
=> 40 976773088 ada0 GPT (466G)
40 1024 1 freebsd-boot (512K)
1064 984 - free - (492K)
2048 16777216 2 freebsd-swap (8.0G)
16779264 959993864 3 freebsd-zfs (458G)
Click to expand...
There was no mention of ada0p3.eli

From my earlier output of "geli list" the p3 partition DOES NOT have the AUTORESIZE flag (ada0p2 does however) - do you think it was mistakenly adviced to not use "geli resize" ?
 
Yes your ada0p3.geli doesn't have autoresize flag so you need to use geli resize.

Edit: in your first post of geli list i misread ada0p2.eli flags where it has autoresize
 
VladiBG said:
Yes your ada0p3.geli doesn't have autoresize flag so you need to use geli resize.

Edit: in your first post of geli list i misread ada0p2.eli flags where it has autoresize
Click to expand...
Thanks for clarifying - makes more sense now to me.

So I tried running geli resize and it complained about the s option missing - Is that a must? If yes - where do I get the exact size old size - under "Mediasize" from geli list output for p3 ?
geli: Option 's' not specified.
Click to expand...
 
Yes it's your mediasize of ada0p3 before resizing (the consumer size) 311481597952

geli resize -s 311481597952 ada0p3
zpool set autoexpand=on zroot
zpool online -e zroot /dev/ada0p3.eli
zpool list


geli list -a
Geom name: ada0p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT
KeysAllocated: 73
KeysTotal: 73
Providers:
1. Name: ada0p3.eli
Mediasize: 311481593856 (290G)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: ada0p3
Mediasize: 311481597952 (290G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 8590983168
Mode: r1w1e1
Click to expand...
 
Ok this is getting weird now - I thought this would be the last step and I would have solved it but it seems like geli won't allow me to do it. Now I'm worried why it won't even let me sudo this ?
sudo geli resize -s 311481593856 /dev/ada0p3.eli
Password:
geli: Cannot open /dev/ada0p3.eli: Operation not permitted.

sudo geli resize -s 311481593856 ada0p3.eli
geli: Cannot open ada0p3.eli: Operation not permitted.
Click to expand...
Should I not be typing the `.eli` extension for the partition?
 
VladiBG said:
You need to enter the mediasize of consumer ada0p3 not the provider ada0p3.eli
Click to expand...
This is confusing : the value of 311481597952 is of provider (from geli output)
Providers:
1. Name: ada0p3.eli
Mediasize: 311481593856 (290G)
Click to expand...
The consumer value is different :
Consumers:
1. Name: ada0p3
Mediasize: 491516858368 (458G)
Click to expand...
The geli man page says s value should be the "oldsize"
resize Inform geli that the provider has been resized. The old metadata block is relocated to the correct position at the
end of the provider and the provider size is updated.
Additional options include:
-s oldsize The size of the provider before it was resized.
Click to expand...
 
Holy crap - tried rebooting. And now it says
"gptzfsboot : No ZFS pools located, can't boot"!
Not even the geli password prompt!
What do I do?
 
After resizing of /dev/ada0p3 the metadata of geli is no longer located on the end of that partition. You need to boot using liveCD and follow my previous post #63 of loading the geli and then resizing it.

By default, if the underlying provider grows, the encrypted
provider will grow automatically too. The
metadata will be moved to the new location.
If automatic expansion if turned off and the
underlying provider changes size, attaching
encrypted provider will no longer be possible
as the metadata will no longer be located in
the last sector. In this case GELI will only
log the previous size of the underlying
provider, so metadata can be found easier, if
resize was done by mistake.
Click to expand...
 
VladiBG said:
After resizing of /dev/ada0p3 the metadata of geli is no longer located on the end of that partition. You need to boot using liveCD and follow my previous post #63 of loading the geli and then resizing it.
Click to expand...
Ok - so I booted into live cd
Did this as you asked
geli load
geli resize -s 311481597952 ada0p3
Click to expand...
And I could boot into my system again! Thank you soo much - specially for getting the size correct of s - I was misunderstanding it until I read your answer later again.

However I did NOT do these steps below (yet) as you asked :
zpool set autoexpand=on zroot
zpool online -e zroot /dev/ada0p3.eli
Click to expand...
However my geli list output for provider and consumer are now matching (458G) ! - I'm guessing I don't need to do the above steps and the space issue is resolved? Still trying to understand how it got resolved though ?
geli list
Geom name: ada0p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT
KeysAllocated: 115
KeysTotal: 115
Providers:
1. Name: ada0p3.eli
Mediasize: 491516854272 (458G)
Sectorsize: 4096
Mode: r1w1e1
Consumers:
1. Name: ada0p3
Mediasize: 491516858368 (458G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 8590983168
Mode: r1w1e1
Click to expand...
 
Hmmm so I think I need to do as you asked and it's not yet resolved , zpool still showing 290G :
zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 290G 270G 20.1G - 166G 69% 93% 1.00x ONLINE -
Click to expand...
zpool get autoexpand
NAME PROPERTY VALUE SOURCE
zroot autoexpand off default
Click to expand...
 
UPDATE : After doing as you asked I think we finally fixed the issue - the zroot size now reflecting 456G!

Thank you VladiBG mer and others who helped me ? You'll are amazing people!
zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 456G 270G 186G - - 43% 59% 1.00x ONLINE -
Click to expand...

Let me know if there is anything more to be done and if there is any step left?
1) Do I switch the kernel value back to original 0 that was changed to 16 as suggested in the handbook?
2) Do I let the autoexpand value remain on?
 
  • Like
Reactions: mer
Awesome to hear that everything worked. Always good to hear feedback.

I don't see any issue with leaving autoexpand on. It would let you start creating mirrors using bigger devices like say 1TB disks. At first you'll only use half of the 1TB, but when you replace all the devices in the mirror with 1TB, it should automatically expand.
At least that's what would happen without GELI; with GELI, maybe turn it off so you can keep the order of operations consistent.
 
Tracker said:
Let me know if there is anything more to be done and if there is any step left?
Click to expand...
You need to deploy a backup strategy.

You don't need to change kern.geom.debugflags it's better to revert it back to 0.
It doesn't metter if you leave autoexpand on or off as you don't change the partition size every now and then. I would keep it on and also enable autoexpand of the geli provider. So next time you won't need to use LiveCD to change it manually as you need to know the original provider size anyway or restore it from the backup metadata file.
 
VladiBG said:
You need to deploy a backup strategy.
Click to expand...
This - definitely.
VladiBG said:
You don't need to change kern.geom.debugflags it's better to revert it back to 0.
Click to expand...
Yes, it reverted back on it's own since it wasn't an loader.conf change

mer said:
I don't see any issue with leaving autoexpand on. It would let you start creating mirrors using bigger devices like say 1TB disks. At first you'll only use half of the 1TB, but when you replace all the devices in the mirror with 1TB, it should automatically expand.
Click to expand...
VladiBG said:
It doesn't metter if you leave autoexpand on or off as you don't change the partition size every now and then. I would keep it on and also enable autoexpand of the geli provider.
Click to expand...
Will try to set autoexpand flag for adaop3 on for geli.

Considering this was my first time trying to expand an encrypted fs - I feel great now. Won't lie - I almost thought the data was lost at one point.
 
You must log in or register to reply here.
Back
Top