I brutally screwed up my system, recommendations for a recovery... 🙏

[freezr]

Hi guys, this time I really screwed my system so badly that I do not have even the strength to cry...
Perhaps because it was late and I was already tired, but in the attempt to clean up /usr/src/ I run the command to delete everything into / instead; now the system doesn't boot and I don't know what I actually screwed up.

To complicate the situation my system is a ZFS MIR on top of GELI...

Possible solutions I thoughts:

1. I chroot with a live FreeBSD (perhaps NomadBSD), and try to activate a previous boot environments, this would be the ideal fix.
2. I mount (if still exists) /usr/home, copy the data I need somewhere else, and I do a fresh install.

Either case I am not very experienced in unlock a Geli disk and mounting a ZFS data set from a live session, here I would need some guidance, thanks...

 

[freezr]

And now the ranting part...

/Start ranting

I am very frustrated because this happened cause at every minor on major upgrade there is this annoying question about the graphics modules and the Nvidia GPU, which is not settle clearly, is not explained in the handbook, anyone has a different opinion or a different method, and I am very pissed off...

I am so frustrated that I am not going to do anything for all the weeks that I need to mentally recovering from this stress, but cause I am really pissed off, and I am thinking:

1. Fixing FreeBSD, get rid off Nvidia driver and using the Nouveau driver, eventually I don't need GPU acceleration since with FreeBSD you can't watch movie, doing 3D stuff, etc...
2. Move to GhostBSD and forget about the Nvidia issue, since there (and it is 1 only person) know how to handle this issue.
3. Say good-bye to FreeBSD, I am tired to deal with the same annoying problem every time...

/Stop ranting

 

[SirDice]

Perhaps because it was late and I was already tired, but in the attempt to clean up /usr/src/ I run the command to delete everything into / instead;

Congratulations and welcome to the club. Everybody makes that mistake at least once. I certainly did, it was with /usr/ports/ but the effect was the same, nuked everything under /usr/ instead of /usr/ports

I chroot with a live FreeBSD (perhaps NomadBSD), and try to activate a previous boot environments, this would be the ideal fix.

Agree, that's probably the best way.

get rid off Nvidia driver and using the Nouveau driver

There is no Nouveau driver on FreeBSD. You might be thinking of the nv(4) Xorg driver.

 

[sko]

Fixing FreeBSD, get rid off Nvidia driver and using the Nouveau driver, eventually I don't need GPU acceleration since with FreeBSD you can't watch movie, doing 3D stuff, etc...

I've been using the nvidia driver for ~10 years now without issues. You just need to rebuild the kernel module for new kernel versions, that's it. This is also true for linux and the derived linux-drm drivers (which requires the additional drm abstraction-layer, which *also* needs to be built for the specific kernel version- so there's nothing gained there. just use the plain, standard nvidia driver...)

Perhaps because it was late and I was already tired, but in the attempt to clean up /usr/src/ I run the command to delete everything into / instead;

IIRC FreeBSD mimics the same behavior as Solaris and if given a rm -rf / it always tries to delete the CWD first, which isn't allowed and hence the command fails and prevents you from shooting yourself in the foot while still remaining POSIX-compliant. Did you issue a rm -rf / or some other command? (hint: on zfs it is much easier and cleaner to just nuke the dataset, especially for things like /usr/src which are easily re-populated from tarball or git)

But after all: such things happen - I once nuked the first part of a laptopt disk (containing LUKS encryption headers) by issueing a dd to the wrong drive...
So just rebuild from backup and call it a day - or in your case even simpler: just boot the previous BE.

 

[freezr]

I ran rm -rf /*, the problem is that I can't get into to boot screen to try to activate a boot environment, my only hope is to chroot and activate a previous snapshot.

Rebuild the new kernel is something that I hope the system can do for me, and it should do it for me, I don't use the ports so I constantly forgot to keep them updated, if I launch an upgrade it would be polite from the upgrade software to upgrade also the ports tree, I don't know... I constantly fall in the same trap...

 

[sko]

Rebuild the new kernel is something that I hope the system can do for me, and it should do it for me

The kernel doesn't need to be rebuilt, just some kernel-specific drivers (i.e. graphics), which are pre-built in the 'FreeBSD-drmkmods' repository.
Beginning with 14.3-RELEASE this is repo is automagically added, so the *only* thing you need to do after the OS-upgrade is to also upgrade your packages.

Also: if /usr/src is populated, freebsd-update upgrade also updates this - which is horribly slow as that upgrade consist of thousands of tiny patches. So it's always faster and easier to just nuke /usr/src (the dataset) before running freebsd-update upgrade and re-populate it afterwards via git. (same goes for /usr/ports).

Unless you are building (parts of) the OS or ports from source, you won't need /usr/src or /usr/ports anyways - so *if* you do need them, you should be aware how to keep them updated manually as you read the relevant sections from the handbook.


I know it is frustrating to recover from a nuked system - but it's especially bad to blame others for one's mistakes. The FreeBSD documentation truly is one-of-its-kind amongst operating systems - especially compared to linux which is an absolute dumpster fire of cruft that has been cobbled together in all regards, and especially when it comes to documentation, which in linux-land is quite often non-existent, outdated or simply wrong.

 

[freezr]

sko

I created the new repository for the kernel module, however the modules there were still built against 14.2; I quad-checked into the handbook and this new way to keep your kernel modules updated it hasn't added yet.

Screwing my system is my fault, but ranting help to calm down though...

 

[johnjohn]

I always hate to see something like this happen but it happens to even the best of us and it is not specific to FreeBSD. You are not alone and the forum is the best place to get help with FreeBSD (better than the Handbook!)

FreeBSD could be programmed to not allow user invoked deletion of root but that would not fall in line with the system being left in full control of the user (and what help would that be to an Admin?). A warning could be implemented but denying such an action is ridiculously wrong.

sko
No system has better documentation than Microsoft. [full stop] Have you ever seen the documentation for updates? updates alone contains better info than the Handbook. Hello? For crying out loud, a Google Search pulls up more helpful info about FreeBSD than that which the Handbook contains. I also get more info about certain topics from OpenBSD and NetBSD documentation. How about this: you copy all of the data from Microsoft about Windows and i will copy-and-paste all of the info from FreeBSD Handbook. Tip: you'll be copying and pasting for 6 months to a year just with Updates (i've done it before.) In depth nonpareil documentation from Microsoft. [full stop]

back to the topic:
FreeBSD is not ready to challenge the Laptop/Desktop market but FreeBSD is - without a doubt - the best operating system on the planet. Nothing compares. I wish that FreeBSD was fully laptop/desktop ready. I believe that FreeBSD could dominate the laptop/desktop market if done correctly. I think that you should keep using FreeBSD.

Also, i have yet to perform a full installation of FreeBSD in a system and spend more than 20 minutes before i'm looking at a login (even Linux Mint, marketed as a lightweight distro, costs an hour of your time). mean time: 12 minutes for me. Hopefully, you backup config files on a separate medium.

 

[ShelLuser]

1. Fixing FreeBSD, get rid off Nvidia driver and using the Nouveau driver, eventually I don't need GPU acceleration since with FreeBSD you can't watch movie, doing 3D stuff, etc...
2. Move to GhostBSD and forget about the Nvidia issue, since there (and it is 1 only person) know how to handle this issue.
3. Say good-bye to FreeBSD, I am tired to deal with the same annoying problem every time...

/Stop ranting

With all due respect (!), but those are all attempts at fighting sympthoms. In other words: none of these options will prevent a repeat. How about adding some self reflection as well: "Ensuring I keep recent backups before doing system upgrades"?

Also, can't do 3D graphics with FreeBSD? I beg to differ

Anyway, sorry to see you got into that mess, seriously. That's just nasty... and yah, been there done that too. Worse yet... it wasn't even a personal system but, errr, something company related? Close before 17:30? I wasn't even in a rush, but did got distracted for a brief second. And of course.... daily backups only run during the evenings.

After calling the higher ups, and getting a good (and deserved) chewing out our team could at least ensure that no personal data would be lost. sko is fully right up there with his comments about Solaris... We "only" had to try and restore the proper system files, fire up backup servers, yaddah, yaddah. 2:30(am) everything was back online and because this was an internal production machine the damage wasn't (too) excessive other than one department not being able to work during most of that evening.

Seriously... that was a bit of a defining moment for me when I started pushing for "soft limits" on root usage, a standard I still enforce on all my servers today (for example: root always having 'another' shell than most commonly used accounts).

Wish you the best of luck!

 

[wismos]

which are pre-built in the 'FreeBSD-drmkmods' repository.

nvidia ones aren't

this is on a 14.3 system, 140300 packages exist for i915,amdgpu and radeonkms drm but not for nvidia-drm

Also: if /usr/src is populated, freebsd-update upgrade also updates this - which is horribly slow as that upgrade consist of thousands of tiny patches. So it's always faster and easier to just nuke /usr/src (the dataset) before running freebsd-update upgrade and re-populate it afterwards via git. (same goes for /usr/ports).

there is a better solution to skip nuking /usr/src, that is switch to the branch that corresponds to the update freebsd-update is fetching, in this case it will be releng/14.3, freebsd-update upgrade won't upgrade this if the files it is pulling matches those in /usr/src

 

[sko]

Maybe they failed to build during the last build cycle? There was some fallout the last few days (e.g. due to new rust version which broke some stuff)
Is there a list of ports that are in the build queue for the '-kmods' repository?

Also: I really don't get why one should use the 'drm-nvidia' packages if there are native FreeBSD-drivers from nvidia available via graphics/nvidia-driver. Just skip the linux intermediate-layer, it most likely won't do any good...


I've been building my own packages for a few years now, so the buildhosts are the first systems that are upgraded and then they rebuild all packages for my other systems. Therefore I did not havet hose minor version inconsistencies for quite a while now.
Before that IIRC I just re-ran the installer script provided by the nvidia-driver which rebuilt the kernel module (maybe I used the tarball from nvidia for that - I really can't remember)

 

[freezr]

wismos holy shit, at least I wasn't crazy when I wrote that I couldn't find the module built against 14.3.

 

[Erichans]

Also: I really don't get why one should use the 'drm-nvidia' packages if there are native FreeBSD-drivers from nvidia available via graphics/nvidia-driver. Just skip the linux intermediate-layer, it most likely won't do any good...

Except that, af far as I understand it, for some applications, notably Wayland, it seems necessary to make use of packages such as graphics/nvidia-drm-kmod that contain a mix of bot the Nvidia fully proprietary drivers and additonal drivers that stem from open source developement, such as drivers based on Linux developed drivers for Nvidia graphics cards.

 

[T-Daemon]

To complicate the situation my system is a ZFS MIR on top of GELI...

Those factors are not relevant.

Possible solutions I thoughts:

1. I chroot with a live FreeBSD (perhaps NomadBSD), and try to activate a previous boot environments, this would be the ideal fix.
2. I mount (if still exists) /usr/home, copy the data I need somewhere else, and I do a fresh install.

Unless there are some recursive snapshots of the zroot/home (zroot/usr/home) datasets, or otherwise on machine / off machine backups, then there won't be any home data to access to. ZFS boot environments doesn't include the home dataset. This is explained in the bectl(8) manual, paragraph "Boot Environment Structures" near the end of the man page.

Either case I am not very experienced in unlock a Geli disk and mounting a ZFS data set from a live session, here I would need some guidance, thanks...

Unlocking a geli provider and mounting (importing) a ZFS pool can be done in a few steps, but this won't get the home data back in your case without a backup or snapshot.

Assuming the ZFS boot environment is intact (not rm -rf /*), boot the installer media, drop to "Live System", enter "root" at the login prompt, execute gpart show, identify which are the "freebsd-zfs" partitions in the mirrored configuration (i.e. ada0p4 and ada1p4)

# geli  attach  ada0p4  ada1p4
# mkdir  /tmp/zfs
# zpool import -R /tmp/zfs  zroot
# zfs mount zroot/ROOT/default

These are instructions to restore a rm -rf /* purged ZFS boot environment in case you don't want to lose installed packages, configuration, etc. I repeat, home data can be recovered only when there are ZFS snapshots of the home dataset taken before.

Assuming that there is at least one additional boot environment besides the default, boot installer media, drop to "Live System", recreate ESP first. The FreeBSD loader.efi in the ESP and all directories/subdirectories were deleted because it was mounted on /boot/efi during the system files deletion .

# mount_msdosfs  /dev/ada0p1  /mnt
# mkdir  -p  /mnt/efi/freebsd
# cp /boot/loader.efi  /mnt/efi/freebsd
# umount  /mnt

On this occasion, optional make a ESP copy on the second disk. Be sure to set correct partition names:
# dd if=/dev/ada0p1  of=/dev/ada1p1 bs=1m

Check file system
# fstyp  /dev/ada0p1
# fstyp  /dev/ada1p1

Attach (unlock) geli providers:
# geli  attach  ada0p4  ada1p4
# zpool import  -N   zroot

Check boot environment snapshot (i.e.)
# zfs list  -t  snap  zroot
zroot/ROOT/default@2025-06-16-10:12:43

Clone snapshot as a new boot environment:
# zfs  clone  zroot/ROOT/default@2025-06-16-10:12:43  zroot/ROOT/rescue

Reboot system into installer media. At the boot menu "Escape to loader prompt", enter:

OK  set currdev=zfs:zroot/ROOT/rescue:    (mind the colon at end of line)
OK  load  /boot/kernel/kernel
OK  load  /boot/kernel/geom_eli.ko
OK  load  /boot/kernel/zfs.ko
OK  boot

Log in system:

# bectl  activate  rescue
# bectl  destroy  default
# bectl  rename  rescue  default

Restore home data from snapshot:

# zfs  list  -t  snap
# zfs rollback zroot/home/<user>@<snap>

The system should be restored now.

This accident should be taken as an example to keep a few backups off machine/site, plus a sound backup strategy. There are several ZFS snapshot utilities in ports to ease the procedure automatically scheduled, periodically.

 

[bob2112]

I'm curious as to how an rm -rf / can prevent choosing an alternate BE environment from the boot menu. The only obvious way is if the UEFI partition were mounted at the time, but that is simple to restore.

 

[T-Daemon]

I'm curious as to how an rm -rf / can prevent choosing an alternate BE environment from the boot menu.

There is no boot menu available anymore. All drawing and function files that make up the boot menu, as well as all other files under /boot (including the kernel) required to boot the system are deleted from the boot environment. The FreeBSD loader wouldn't find anything to work with.

For that matter, there are no base system files either, mostly, only a handful commands and some libraries can survive a /* deletion due to denied permission.

If reinstalling the system is not an option, all the necessary boot parts and base system files can be recreated from a snapshot, provided there is one.

The only obvious way is if the UEFI partition were mounted at the time, but that is simple to restore.

That's exactly the case. In a menu guided installation, the ESP (UEFI partition) is configured by default to be mounted via fstab:

Device              Mountpoint    FStype     Options     Dump       Pass#
/dev/gpt/efiboot0      /boot/efi     msdosfs    rw          2          2

However, for obvious reasons, restoring the ESP will not restore a boot menu to choose an alternate boot environment.

 

[Andriy]

Depending on how the system is booted it may be possible to interrupt the boot process even before loader and manually type in an alternative loader or boot environment.
See gptzfsboot(8) and its references.

 

[Jose]

Except that, af far as I understand it, for some applications, notably Wayland, it seems necessary to make use of packages such as graphics/nvidia-drm-kmod that contain a mix of bot the Nvidia fully proprietary drivers and additonal drivers that stem from open source developement, such as drivers based on Linux developed drivers for Nvidia graphics cards.

It saddens me to see Linux flakiness creeping into Freebsd. The attack vector is graphics, as usual.

 

[bob2112]

There is no boot menu available anymore. All drawing and function files that make up the boot menu, as well as all other files under /boot (including the kernel) required to boot the system are deleted from the boot environment. The FreeBSD loader wouldn't find anything to work with.

For that matter, there are no base system files either, mostly, only a handful commands and some libraries can survive a /* deletion due to denied permission.

If reinstalling the system is not an option, all the necessary boot parts and base system files can be recreated from a snapshot, provided there is one.


That's exactly the case. In a menu guided installation, the ESP (UEFI partition) is configured by default to be mounted via fstab:

Device              Mountpoint    FStype     Options     Dump       Pass#
/dev/gpt/efiboot0      /boot/efi     msdosfs    rw          2          2

However, for obvious reasons, restoring the ESP will not restore a boot menu to choose an alternate boot environment.

As far as I know, /boot is an ordinary directory in the root dataset, so each BE has it's own /boot. Are you saying that if /boot is corrupted on the default BE, the boot process can't fall back to the other versions of /boot?

 

[T-Daemon]

As far as I know, /boot is an ordinary directory in the root dataset, so each BE has it's own /boot.

That is correct.

Are you saying that if /boot is corrupted on the default BE, the boot process can't fall back to the other versions of /boot?

That's exactly the case.

The loader can probe only the active BE. If no kernel can be found there, the boot process stops at the loader, presenting the user a loader prompt.

There is no logic in the loader built in to probe other BEs if no kernel is found in the active BE. In such a case the user must tell the loader explicitly an alternate BE to use.

Example:

Assuming OPs UEFI system, loader.efi(8) would be the kernel loader, and "currdev" the variable to set a different BE (for "currdev" see loader_simp(8)). See [1] for BIOS.

# kenv
...
bootenv_autolist="YES"
bootenvs[0]="zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:14:05"
bootenvs[1]="zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:14:01"
bootenvs[2]="zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:13:56"
bootenvs[3]="zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:13:46"
bootenvs[4]="zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:13:39"
bootenvs[5]="zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:13:33"
bootenvs[6]="zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:03:25"
bootenvs[7]="zfs:zroot/ROOT/default"
bootenvs_count="8"
bootfile="kernel"
...
currdev="zfs:zroot/ROOT/default:"

In this VM setup are 8 BEs, 1 activated default and 7 (simulated) freebsd-update(8) created BEs. The default device to loader the kernel from is set to currdev="zfs:zroot/ROOT/default:".

To override a corrupted BE, the user must specify at the loader prompt a different BE with an intact file system to boot a kernel from, i.e.:

OK  set currdev=zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:13:33:

After setting the BE, the file system can be inspected with "ls". Then, to boot the system, load .../kernel, load .../zfs.ko, boot.

In the OPs system setup, geli(8) encrypted Root-on-ZFS is involved, which makes it impossible to view other BEs from the loader prompt. If the user has no knowledge of the exact BEs name (which is usually the case of freebsd-update created BEs), a installer medias live system must be booted, the Root-on-ZFS geli(8) provider attached, the pool imported, to "zfs list" BE snapshots.

Cloning a BE snapshot with an easier-to-remember name (zroot/ROOT/rescue in the example in post # 14) is for user convenience, instead of making a note or memorizing the exact BE name, which includes year/month/day/hour/minute/second, underscore, hyphens, colons as part of the name.


[1]
On BIOS, gptzfsboot(8) is the loader probing ZFS partition for an kernel. The setting format for another pool is [zfs:pool/filesystem:][/path/to/loader], i.e.:

boot: zfs:zroot/ROOT/rescue:/boot/zfsloader

It looks like, gptzfsboot(8) doesn't understand BE names containing underscore, hyphens, colons (like zfs:zroot/ROOT/14.3-RELEASE_2025-06-29-10:13:33:). An alternate BE should be cloned from a snapshot with a simple name when on a live rescue system.