Solved ZFS: i/o error - all block copies unavailable; Error: memory allocation error: block too big

[takumo]

Dear ZFS and freebsd loader experts,

I just stumbled across a new kind of ZFS: i/o error, which I could not find any hint or cause using google.

During boot I get the following error screen:

which I cannot really understand.

How I got there (on freebsd 12.2):
I just added the follwing line to loader.conf

## Loader Config ##
net.fibs=2

My feeling is that the change in loader.conf has nothing todo with the real error, its merly the fact that i changed loader.conf at all

System configuration and history:

- 64 GB ECC memory (for sure its not a real out of memory error)
- i have 4 hdds in a 2x2 zfs mirror setting (like RAID 10)
- the first mirror set has 3 paritions each, setted up by the freebsd (12.1) installer (boot, temp, zfs(freebsd on zfs)), 2x2TB
- the second mirror set has only 1 big partition each and was added later to increase the size of the pool, 2x4TB

Some naive thoughts:
During the update form 12.1 to 12.2 i fell into the trap that my second mirror which I added later, originally was without partitons but bare hdd. Howeverthe current loader can only handle zfs spans if all parts are on a parition. solved the problem by exporting, reformating and importing the pool. There was some thread explaining this issue which I dont find anymore. Could it be that similar to this issue by changing the a file at the /boot/loader.conf partition some metadata was moved to the second mirror, where it again connot be accessed by the loader?



Things I allready tried:
- Booting from LiveCD and mounting the zfs pool -> works
- reverting the loader.conf to the previous state -> no effect


Has anyone an idea what is the cause of this error and how to solve it (best without reinstalling evrything, export/import of all the data...)

Thank you very much for any help.

 

[takumo]

Additional info:

if i do

more /boot/loader.conf

on the BTX loader console I get zfs io error - all block copies unavailable. However:
- I can read other files in the boot directory
- I can read and modify /boot/loader.conf from a live CD

Even more info

I played arround with zdb and as far as I understand the file really is on vdev1 (the second vdev).

- Is there anything knwon that the bootloader cannot handle a zfs boot dataset spanning multiple vdevs?
- Can i somehow force freebsd to save a file to the first vdev (vdev0)? I tried to save i several times and hoped it will endup by chance at the right vdev, but until yet I was unlucky..

 

[takumo]

Some more observations:
- The BTX loader can read old files (e.g. using more command) from both vdevs.
- /boot/defaults/loader.conf is on vdev0 can be read
- /boot/device.hints is on vdev1 can be read
- If I create new files on the zpool, e.g. copy loader.conf, create simple text files, BTX loader throws an ZFS io error: all block copies unavailable upon reading them.
- all new files go into vdev1
- However I can read and write all of the files using the live CD
- BTX loader recongnizes all disks and the pool (see attachment, lsdev -v) properly

I really start to have no more ideas how to debug this problem. Why can the loader not read new files?

 

[takumo]

After investigating further i realized that new files can be read by the loader if their data is embedded in the zfs block pointer ( zdb -O zroot/ROOT/default boot/loader.conf shows dsize=0). I stripped down my loader.conf to fit into the block pointer and FreeBSD boots now again. WOW

However since I had to strip down my loader.conf I cannot load all modules. Is there something like a INCLUDE statement for the loader.conf file, so I can chain many of them?

And of course the most pressing question is why can't the loader read "new" files which store their data in indirect blocks. I hope someone has an answer...

 

[SirDice]

However since I had to strip down my loader.conf I cannot load all modules. Is there something like a INCLUDE statement for the loader.conf file, so I can chain many of them?

Use kld_list in rc.conf instead.

    kld_list    (str) A list of kernel modules to load right after the local
                 disks are mounted.  Loading modules at this point in the boot
                 process is much faster than doing it via /boot/loader.conf
                 for those modules not necessary for mounting local disk.

See rc.conf(5).

 

[takumo]

Use kld_list in rc.conf instead.

Thank you. Indeed that works like a charm.

 

[takumo]

For the records:
I finally moved the boot pool to a fresh single vdev pool and now everything works again. I strongly assume that there is some problem with the zfsloader, which could not properly access my pool. Its not very satisfying to not find the root cause of the problem, however I can carry on now.