Solved Preparing to restore geli devices - clarification on .eli files

[listentoreason]

Nutshell: Do I only need the *.eli files from /var/backups if the drive metadata has become mangled? That is, an undamaged geli-encrypted device shouldn't need its backup .eli file, yes?

I recently managed to bork my system (only gets ~18 process IDs into boot before it segfaults attempting to start the first /bin/sh). As far as I know, my storage pool is ok, and ASX has suggested re-installation from media, which I am preparing to do. My zpool is built on 11 geli-encrypted drives, and I've been perpetually nervous about the encryption becoming a failure point in my ability to preserve/restore my data. In particular, I've treated the /var/backups/<device_id>.eli files with both awe and terror, believing them to be essential for the proper decryption of the devices. I was trying to figure out where the "original versions" were, so I could carefully restore them when I rebuilt the system.

Reading a post by asteriskRoss and then geli(8), I think I understand now that the original location is a dedicated space on the last sector of each device, and the backup files are only needed when something dreadful happens to that last sector. This makes a lot more sense and is a lot less terrifying. I'm still planning on backing up the .eli files (and the rest of /etc/ and /var/ and whatever else I can copy off the boot disk), but am significantly less stressed.

So: Are my new insights valid, or are there other preparatory steps I should take to assure I can bring my pool back up once I have a clean system re-installed?

Thanks,
CAT

 

[asteriskRoss]

Nutshell: Do I only need the *.eli files from /var/backups if the drive metadata has become mangled? That is, an undamaged geli-encrypted device shouldn't need its backup .eli file, yes?

You are correct. Under normal circumstances you will not need the GELI metadata backup files.

I'm still planning on backing up the .eli files (and the rest of /etc/ and /var/ and whatever else I can copy off the boot disk)

If you use key files with your GELI containers, it is absolutely essential that you back these up. Without them you will not be able to decrypt your data!

are there other preparatory steps I should take to assure I can bring my pool back up once I have a clean system re-installed?

You don't need to wait until you have reinstalled your system to access the data in your ZFS pool. If you are able to boot the system to the install media then you can use the Live CD option and attach your GELI containers by hand from the command line (see the attach option in the geli(8) man page). With the GELI containers attached you can then import the ZFS pool (see the import option in the zpool(8) man page). Consider using the -o readonly=on switch with the zpool import command if you only need read access. This might be useful if you have anything critical that you didn't back up before you borked your system.

Going forward, you might want to spend some time thinking about system backups so you can recovery more easily from borkville. The backup section in the FreeBSD handbook is a useful starting point in combination with the snapshots and replication sub-sections in the ZFS section of the FreeBSD handbook.

 

[listentoreason]

Thank you! I need to figure out how to set up mount points for the live system; it looks like I can work with /tmp to make mount points there. That will let me mount the system disk to get various components off of it, and hopefully to mount the zpool to do a final refresh of my backups.

I have a backup system, but it's informal and somewhat rickety; I have two sets of USB drives, one kept at home, the other at my parents' house in another state. The sets cycle when we visit each other, 2-4 times a year. However, I'm not 100% satisfied with the system, as there are "less important" files that don't make it to the backups. The zpool is 20TB, currently using ~6TB, which makes management problematic. I have a somewhat complex rsync / find system that freezes drives when they're full (no more syncing, no more state-to-state cycling) then starts on a new drive where find limits the files-to-sync to those with mtime past the frozen device. The complexity is concerning. Long term plans are to actually build a second, less expensive FreeBSD system whose sole role is to periodically rsync the entire pool (at least to a certain size) and maintain the fortnightly snapshots. USB drives would still fill in the "off site" component.

So if this all ends tragically, I can get 95% of my files back, but it would be a hassle and I'd likely wonder what I was missing.

 

[listentoreason]

Ok, all good as far as I can tell: I've updated the system and brought the pool back online. I booted off the install media, mounted a USB thumb drive, and copied /var, /etc and /home onto it. I also dumped some information on what I had installed (eg find /usr/bin -type f > /tmp/usb/installedBinaries.txt) to help with reinstalling packages. I managed to overlook /usr, not sure what that will cost me - I was worried I overlooked my crontabs, but I found those in /var. The pool scrubbed with no errors, and since in my case "the pool" ~= "the server", I'm counting this as success.

I will try to write up the steps I went through for posterity in another post - In addition to the text files I saved in the process I have several sheets of paper recording what I did, I need to put them into something intelligible. Important note: The default shell, sh, has "history" in the sense of hitting up arrow to get to prior commands. I was unable, however, to find a history equivalent that would dump the full history. That is, I wanted to run history > /tmp/usb/thingsThatIDid.txt, but couldn't. Next time first thing I'm going to do is pkg install bash and then switch over to better capture the steps I took.

Thanks again for the help!

 

[asteriskRoss]

I'm counting this as success.

Great! Sounds like a success to me.

I'm counting this as success.
I was unable, however, to find a history equivalent that would dump the full history.

For next time (hoping there won't be a next time for this particular issue) you might find script(1) useful. Running script /tmp/usb/thingsThatIDid.txt before you started would have recorded your terminal session as you saw it; commands and outputs. It won't give you a neat record but does mean that you don't need to write lengthy separate notes as you go.