To be or not to be

[DavidMarec]

This Sunday, my FreeBSD box has entered some weird state.

This machine is following 11-stable, and,
- after calling delete-old-libs maybe, I don't really know -
a lot of modules and commands fail on missing libjail.so.1

I checked out this file within /lib and ls confirmed that the file was missing.
But, each attempt to copy it from a backup into this directory using cp fails on File exists.

root@dmarec:/home/david # ls /lib/libjail.so.1
ls: /lib/libjail.so.1: No such file or directory
root@dmarec:/home/david # cp libjail.so.1  /lib/libjail.so.1
cp: /lib/libjail.so.1: File exists

I dropped the file into /usr/lib to make the box works again and tried to re-install the base system with make buildworld and make installworld.
... then, install -C stopped on the same error, but on other files ( e.g. libipsec ).

Another strange behavior, when booting into single user mode, ls /libdisplays libjail.so.1!

Any clue ? It sounds like ZFS gone crazy.
- a scrub of the pool doesn't help -

Same issue occurred if the filesystem was mounted from a liveCD, or backuped up into another zpool.

 

[DavidMarec]

With deeper investigation, I found that this file truly exists in some way:

david:~>find /lib -name "libjail.so.1" -print                        
/lib/libjail.so.1                                                    
david:~>find /lib -name "libjail.so.1" -ls                            
find: /lib/libjail.so.1: No such file or directory

Moreover, It holds an inode:

root@dmarec:~ # ls -di /lib
13 /lib
root@dmarec:~ # zdb -dddd zroot/ 13
Dataset zroot [ZPL], ID 21, cr_txg 1, 114G, 2570014 objects, rootbp DVA[0]=<0:b9ea62a00:200> DVA[1]=<0:1c22507e00:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=3852314L/3852314P fill=2570014 cksum=15178e0d15:6bb5b6fdb08:129fff02e6446:24c77cf42d8159

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
        13    1    16K     4K     2K     4K  100.00  ZFS directory
                                        168   bonus  System attributes
   dnode flags: USED_BYTES USERUSED_ACCOUNTED
   dnode maxblkid: 0
   path   /lib
   uid     0
   gid     0
   atime   Wed Dec 14 19:38:27 2016
   mtime   Mon Nov 28 23:38:51 2016
   ctime   Mon Nov 28 23:38:51 2016
   crtime   Sun Dec  9 08:58:13 2012
   gen   79
   mode   40755
   size   56
   parent   4
   links   4
   pflags   40800000144
   microzap: 4096 bytes, 54 entries

       libalias_skinny.so = 368798 (type: Regular File)
       libavl.so.2 = 10555003 (type: Regular File)
       libalias_irc.so = 368789 (type: Regular File)
       libalias_nbt.so = 368792 (type: Regular File)
       libjail.so.1 = 10552574 (type: Regular File)
       [snip]

So, I tried to get more information:

root@dmarec:~ # zdb -dddd zroot/ 10552574
Dataset zroot [ZPL], ID 21, cr_txg 1, 114G, 2570014 objects, rootbp DVA[0]=<0:b9eb2f000:200> DVA[1]=<0:1c225dd400:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=3852325L/3852325P fill=2570014 cksum=134e72a8fc:641dcf95752:11bd55bd292b4:23f76108682f89

    Object  lvl   iblk   dblk  dsize  lsize   %full  type
zdb: dmu_bonus_hold(10552574) failed, errno 2

Any clue is welcome.