ZFS ZFS inconsistent, can't delete files

Hi,

I have a nanopi R5S running 14-CURRENT and now I am trying to build opnsense for aarch64 on it. Now my issue, I begin to build and it crashes. After that my issue is:

Code:
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO/tools/config/23.7/aarch64/mnt/local/opnsense/R5SJUNTO/ports # rm -rf x11-fonts/urwfonts
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO/tools/config/23.7/aarch64/mnt/local/opnsense/R5SJUNTO/ports # ls x11-fonts/urwfonts
ls: x11-fonts/urwfonts: No such file or directory

I can't delete any files there. So I can't build anything in the remainder of the build folders. See:

Code:
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO # rm -rf src
rm: src/arm64.aarch64/release/dist/base/libexec/ld-elf.so.1: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/libexec: Directory not empty
rm: src/arm64.aarch64/release/dist/base/sbin/init: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/sbin: Directory not empty
rm: src/arm64.aarch64/release/dist/base/var/empty: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/var: Directory not empty
rm: src/arm64.aarch64/release/dist/base/usr/lib/librt.so.1: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/lib: Directory not empty
rm: src/arm64.aarch64/release/dist/base/usr/bin/passwd: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/su: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/crontab: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/opieinfo: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/chpass: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/opiepasswd: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin: Directory not empty
rm: src/arm64.aarch64/release/dist/base/usr: Directory not empty
rm: src/arm64.aarch64/release/dist/base/lib/libthr.so.3: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/lib/libcrypt.so.5: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/lib/libc.so.7: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/lib: Directory not empty
rm: src/arm64.aarch64/release/dist/base: Directory not empty
rm: src/arm64.aarch64/release/dist: Directory not empty
rm: src/arm64.aarch64/release: Directory not empty
rm: src/arm64.aarch64: Directory not empty
rm: src: Directory not empty

I tried to scrub it:

Code:
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO # zpool scrub zroot
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO # zpool status
  pool: zroot
 state: ONLINE
  scan: scrub in progress since Sun Aug 20 22:22:45 2023
        14.9G / 23.7G scanned at 1.66G/s, 0B / 23.7G issued
        0B repaired, 0.00% done, no estimated completion time
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0

errors: No known data errors
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO #

Now I can't figure out how to delete this folder tree to start over. I have two folders that got this error and can see that somewhere in the future I will have run out of space and names to build :)

Anyone has a clue on how to fix this?

Thanks,

none
 
Part 1: You said you deleted a directory called x11-fonts/urwfonts, and all the files in it (first line in the first quote). Then the directory doesn't exist any longer. Seems like it's perfect to me. What is wrong here?

Part 2: You don't seem to have permissions to delete certain files or directories. The error messages seem quite clear. Please check what your logged in user is, who owns these files/directories, and what their permissions are. Note that ZFS also supports certain "immutable" flags (called schg and/or uchg), which can prevent things from being deleted.

Part 3: Your scrub is in process, it has found no problems. I don't know why you think that a scrub would change the fact that you don't have permission to delete certain files.
 
Code:
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO/tools/config/23.7/aarch64/mnt/local/opnsense/R5SJUNTO/ports # rm -rf x11-fonts/urwfonts
root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO/tools/config/23.7/aarch64/mnt/local/opnsense/R5SJUNTO/ports # ls x11-fonts/urwfonts
ls: x11-fonts/urwfonts: No such file or directory
Where is the issue?

I can't delete any files there. So I can't build anything in the remainder of the build folders. See:

Code:
 root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO # rm -rf src
rm: src/arm64.aarch64/release/dist/base/libexec/ld-elf.so.1: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/libexec: Directory not empty
rm: src/arm64.aarch64/release/dist/base/sbin/init: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/sbin: Directory not empty
rm: src/arm64.aarch64/release/dist/base/var/empty: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/var: Directory not empty
rm: src/arm64.aarch64/release/dist/base/usr/lib/librt.so.1: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/lib: Directory not empty
rm: src/arm64.aarch64/release/dist/base/usr/bin/passwd: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/su: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/crontab: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/opieinfo: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/chpass: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin/opiepasswd: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/usr/bin: Directory not empty
rm: src/arm64.aarch64/release/dist/base/usr: Directory not empty
rm: src/arm64.aarch64/release/dist/base/lib/libthr.so.3: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/lib/libcrypt.so.5: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/lib/libc.so.7: Operation not permitted
rm: src/arm64.aarch64/release/dist/base/lib: Directory not empty
rm: src/arm64.aarch64/release/dist/base: Directory not empty
rm: src/arm64.aarch64/release/dist: Directory not empty
rm: src/arm64.aarch64/release: Directory not empty
rm: src/arm64.aarch64: Directory not empty
rm: src: Directory not empty

Usually, /usr/src is a mount point, so, you can't delete it like this. If you want to delete it's content you should issue this command:
# rm -rf /usr/src/* && rm -rf /usr/src/.* as root. Only then you can delete the dataset it self.

I tried to scrub it:
Why?
 
Has nothing to do with ZFS. Those files in /usr/obj/ have have the system immutable flag set.

Code:
           schg, schange, simmutable
                       set the system immutable flag (super-user only)
chflags(1)
 
Part 1: You said you deleted a directory called x11-fonts/urwfonts, and all the files in it (first line in the first quote). Then the directory doesn't exist any longer. Seems like it's perfect to me. What is wrong here?

Part 2: You don't seem to have permissions to delete certain files or directories. The error messages seem quite clear. Please check what your logged in user is, who owns these files/directories, and what their permissions are. Note that ZFS also supports certain "immutable" flags (called schg and/or uchg), which can prevent things from being deleted.

Part 3: Your scrub is in process, it has found no problems. I don't know why you think that a scrub would change the fact that you don't have permission to delete certain files.
Part1: I was on a hurry, forgot to put all information:

root@NanopiR5S-nvme:/usr/obj/mnt/local/opnsense/R5SJUNTO/tools/config/23.7/aarch64/mnt/local/opnsense/R5SJUNTO/ports # ls -ld x11*
ls: x11-fonts/urwfonts: No such file or directory

when I issue an "ls", the folder x11-fonts/urwfonts

BobSlacker said it also.

Part 2: i guess that's the info, I didn't know about. Thanks you and Sir DIce

Part 3: As my box crashed, I thought it was caused by ZFS inconsistency for the crash. That's why I tried a scrub. Afterwards I saw it was ongoing, but finished:

Code:
zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:00:58 with 0 errors on Sun Aug 20 22:23:43 2023
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      nda0p3    ONLINE       0     0     0

errors: No known data errors
 
Back
Top