ZFS ZFS: Corrupt data? Having an issue deleting a directory, causes processes to run away

Hello

I've been wrestling with an issue with my FreeBSD 14.1 workstation, I'm unable to delete a folder. Initially, I thought that I may have been experiencing early signs of a NVME failure, but after cloning the original drive I'm still having issues, so all I can point to at this point is it being an issue at the ZFS level.

Here are the symptoms I'm experiencing due to this problem:

  • Unable to seek (find, ls, etc) into subject directory (currently, /home/user/tmp/Cache_data. originally, /home/user/.cache/chromium/Cache/Cache_data)
  • After some unknown time: Unable to seek into the PARENT directory (currently in, /home/user/tmp)
  • *Something* is kicking off a find process, I'm guessing something that has to do with ZFS since it's listing off all of my mounted filesystems (info below)
  • When this find process kicks off, the process will hang forever and can't be killed no matter what signal I pass to it
  • Find process utilizes practically all of my system resources, making my workstation run like crap
  • Unable to gracefully restart / shutdown my computer, the system will indefinitely hang right after the disks are sync'd. I'm able to gracefully shut down if the find process was never ran.

The directory in question was a folder created by the chromium browser, specifically the directory $HOME/.cache/chromium/Cache/Cache_Data. I managed to be able to at least move this directory outside of .cache since I began to wonder if the symptoms I'm having are due to a different application seeking into .cache. However, it appears to be something else, maybe in /etc/periodic, that is kicking off a job that is attempting to seek into it's new directory. Before this process starts, I seem to have some flexibility in at least observing what is in the directory and it's size, after it starts is when all of the symptoms I have are started, if I attempt to remove the directory, the rm -rf command will hang forever. I noticed that the directory in question also has a size much greater than other directories, even though there is nothing in it. I also don't see anything in the kernel buffer and / or messages log that would indicate that the system is having an issue.

Anyone have any ideas what's going on here, and anyway I can delete this directory? Really seems like a ZFS bug to me.

Code:
$ ls -iltash tmp
total 15671
     2     9 drwxr-xr-x  38 user user   90B Sep 24 16:17 ..
672230     9 drwxr-xr-x   3 user user    6B Sep 24 12:12 .
101914 15641 drwx------   3 user user  108K Sep 12 11:08 Cache_Data

$ top -n
PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
18698 root          1  56    0    30M    17M CPU2     2 771:54 100.27% / /home /zroot /usr/local/bastille /usr/local/poudriere /usr/src /var/mail /usr/local/bastille/cache /usr/local/bastille/jails /usr/local/b

# zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:01:44 with 0 errors on Sun Sep 22 06:28:45 2024
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      nda0p4    ONLINE       0     0     0

errors: No known data errors

# cat /etc/rc.conf
devmatch_blacklist="if_iwm if_iwlwifi"
clear_tmp_enable="YES"
hostname="freebsd"
ifconfig_em0="DHCP"
#wlans_iwlwifi0="wlan0"
#create_args_wlan0="wlanmode sta regdomain FCC country US"
#ifconfig_wlan0="WPA SYNCDHCP"
sshd_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
dbus_enable="YES"
lightdm_enable="YES"
kld_list="i915kms vmm if_bridge nmdm"
libvirtd_enable="YES"
webcamd_enable="YES"
vboxdrv_load="YES"
vboxnet_enable="YES"
xrdp_sesman_enable="YES"
xrdp_enable="YES"
ubuntu_enable="NO"
wifibox_enable="YES"
devmatch_enable="YES"
defaultrouter="10.0.0.1"
ifconfig_wifibox0="inet 10.0.0.2/24"
background_dhclient_wifibox0="YES"
defaultroute_delay="0"
#linux_enable="YES"
#ubuntu_enable="YES"
cupsd_enable="YES"
devfs_system_ruleset="system"
sndiod_enable="YES"
davmail_enable="YES"

# dmesg
---<<BOOT>>---
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE-p4 GENERIC amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(efifb): resolution 1920x1080
CPU: 11th Gen Intel(R) Core(TM) i5-1145G7 @ 2.60GHz (1497.60-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x806c1  Family=0x6  Model=0x8c  Stepping=1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xf3bfa7eb<FSGSBASE,TSCADJ,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,NFPUSG,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,SHA,AVX512BW,AVX512VL>
  Structured Extended Features2=0x18c07fde<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,TME,AVX512VPOPCNTDQ,RDPID,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0xfc100510<FSRM,AVX512VP2INTERSECT,MD_CLEAR,IBT,IBPB,STIBP,L1DFL,ARCH_CAP,CORE_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 32671911936 (31158 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <DELL   Dell Inc>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-119
Launching APs: 1 7 6 5 4 2 3
random: entropy device external interface
kbd1 at kbdmux0
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
smbios0: <System Management BIOS> at iomem 0x4a470000-0x4a47001e
smbios0: Version: 3.2, BCD Revision: 3.2
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
acpi0: <DELL Dell Inc>
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 19200000 Hz quality 950
Event timer "HPET" frequency 19200000 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
acpi_ec0: <Embedded Controller: GPE 0x6e> port 0x930,0x934 on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x3000-0x303f mem 0x6052000000-0x6052ffffff,0x4000000000-0x400fffffff at device 2.0 on pci0
vgapci0: Boot video device
pcib1: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pcib2: <ACPI PCI-PCI bridge> at device 7.1 on pci0
xhci0: <Intel Tiger Lake-LP Thunderbolt 4 USB controller> mem 0x6053180000-0x605318ffff at device 13.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <serial bus, USB> at device 13.2 (no driver attached)
pci0: <simple comms, UART> at device 18.0 (no driver attached)
xhci1: <Intel Tiger Lake-LP USB 3.2 controller> mem 0x6053160000-0x605316ffff at device 20.0 on pci0
xhci1: 32 bytes context size, 64-bit DMA
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <network> at device 20.3 (no driver attached)
pci0: <serial bus> at device 21.0 (no driver attached)
pci0: <serial bus> at device 21.1 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms, UART> at device 22.3 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib3
pci1: <unknown> at device 0.0 (no driver attached)
pcib4: <ACPI PCI-PCI bridge> at device 29.0 on pci0
pci2: <ACPI PCI bus> on pcib4
nvme0: <Generic NVMe Device> mem 0xa2100000-0xa2103fff at device 0.0 on pci2
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
hdac0: <Intel Tiger Lake HDA Controller> mem 0x6053190000-0x6053193fff,0x6053000000-0x60530fffff at device 31.3 on pci0
pci0: <serial bus> at device 31.5 (no driver attached)
em0: <Intel(R) I219-LM TGP(13)> mem 0xa2300000-0xa231ffff at device 31.6 on pci0
em0: EEPROM V0.8-4
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 38:14:28:4f:9c:0f
em0: netmap queues/slots: TX 1/1024, RX 1/1024
acpi_lid0: <Control Method Lid Switch> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
acpi_acad0: <AC Adapter> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
hwpstate_intel4: <Intel Speed Shift> on cpu4
hwpstate_intel5: <Intel Speed Shift> on cpu5
hwpstate_intel6: <Intel Speed Shift> on cpu6
hwpstate_intel7: <Intel Speed Shift> on cpu7
Timecounter "TSC" frequency 1497608702 Hz quality 1000
Timecounters tick every 1.000 msec
ugen1.1: <Intel XHCI root HUB> at usbus1
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus1
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
uhub1 on usbus0
uhub1: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
nvme0: Allocated 508MB host memory buffer
hdacc0: <Realtek ALC236 HDA CODEC> at cad 0 on hdac0
hdaa0: <Realtek ALC236 Audio Function Group> at nid 1 on hdacc0
pcm0: <Realtek ALC236 (Internal Analog)> at nid 20 and 18 on hdaa0
pcm1: <Realtek ALC236 (Front Analog Headphones)> at nid 33 on hdaa0
hdacc1: <Intel Tiger Lake HDA CODEC> at cad 2 on hdac0
hdaa1: <Intel Tiger Lake Audio Function Group> at nid 1 on hdacc1
pcm2: <Intel Tiger Lake (HDMI/DP 8ch)> at nid 4 on hdaa1
nda0 at nvme0 bus 0 scbus0 target 0 lun 1
nda0: <SPCC M.2 PCIe SSD APF1M3R1 WWFG240729003000775>
nda0: Serial Number WWFG240729003000775
nda0: nvme version 1.3
nda0: 488386MB (1000215216 512 byte sectors)
Trying to mount root from zfs:zroot/ROOT/default []...
uhub1: 5 ports with 5 removable, self powered
uhub0: 16 ports with 16 removable, self powered
ugen0.2: <Dell Inc. Dell dock> at usbus0
uhub2 on uhub1
uhub2: <Dell Inc. Dell dock, class 9/0, rev 3.10/1.49, addr 1> on usbus0
uhub2: 4 ports with 3 removable, self powered
Root mount waiting for: usbus0 usbus1
ugen1.2: <Dell Inc. Dell dock> at usbus1
uhub3 on uhub0
uhub3: <Dell Inc. Dell dock, class 9/0, rev 2.10/1.49, addr 1> on usbus1
uhub3: MTT enabled
ugen0.3: <Dell Inc. Dell dock> at usbus0
uhub4 on uhub2
uhub4: <Dell Inc. Dell dock, class 9/0, rev 3.10/1.21, addr 2> on usbus0
uhub3: 5 ports with 4 removable, self powered
ugen1.3: <Dell Inc. Dell dock> at usbus1
uhub5 on uhub3
uhub5: <Dell Inc. Dell dock, class 9/0, rev 2.10/1.21, addr 2> on usbus1
uhub5: MTT enabled
Root mount waiting for: usbus0 usbus1
uhub4: 4 ports with 4 removable, self powered
ugen0.4: <Generic 4-Port USB 3.0 Hub> at usbus0
uhub6 on uhub4
uhub6: <Generic 4-Port USB 3.0 Hub, class 9/0, rev 3.10/1.41, addr 3> on usbus0
uhub5: 6 ports with 6 removable, self powered
uhub6: 2 ports with 2 removable, self powered
Root mount waiting for: usbus0 usbus1
ugen0.5: <Realtek USB 10/100/1000 LAN> at usbus0
ugen1.4: <Generic 4-Port USB 2.0 Hub> at usbus1
uhub7 on uhub5
uhub7: <Generic 4-Port USB 2.0 Hub, class 9/0, rev 2.10/1.41, addr 3> on usbus1
uhub7: MTT enabled
uhub7: 4 ports with 4 removable, self powered
ugen1.5: <Logitech USB Receiver> at usbus1
ukbd0 on uhub7
ukbd0: <Logitech USB Receiver, class 0/0, rev 2.00/39.06, addr 4> on usbus1
kbd2 at ukbd0
Root mount waiting for: usbus1
ugen1.6: <Logitech USB Receiver> at usbus1
ukbd1 on uhub7
ukbd1: <Logitech USB Receiver, class 0/0, rev 2.00/5.01, addr 5> on usbus1
kbd3 at ukbd1
ugen1.7: <Generic USB Audio> at usbus1
ugen1.8: <vendor 0x413c Dell dock> at usbus1
Root mount waiting for: usbus1
ugen1.9: <vendor 0x413c Dell dock> at usbus1
ugen1.10: <vendor 0x046d HD Pro Webcam C920> at usbus1
Root mount waiting for: usbus1
ugen1.11: <Broadcom Corp 58200> at usbus1
ugen1.12: <vendor 0x8087 product 0x0026> at usbus1
drmn0: <drmn> on vgapci0
vgapci0: child drmn0 requested pci_enable_io
vgapci0: child drmn0 requested pci_enable_io
[drm] Unable to create a private tmpfs mount, hugepage support will be disabled(-19).
[drm] Got stolen memory base 0x6c800000, size 0x4000000
lkpi_iic0: <LinuxKPI I2C> on drmn0
iicbus0: <Philips I2C bus> on lkpi_iic0
iic0: <I2C generic I/O> on iicbus0
lkpi_iic1: <LinuxKPI I2C> on drmn0
iicbus1: <Philips I2C bus> on lkpi_iic1
iic1: <I2C generic I/O> on iicbus1
lkpi_iic2: <LinuxKPI I2C> on drmn0
iicbus2: <Philips I2C bus> on lkpi_iic2
iic2: <I2C generic I/O> on iicbus2
lkpi_iic3: <LinuxKPI I2C> on drmn0
iicbus3: <Philips I2C bus> on lkpi_iic3
iic3: <I2C generic I/O> on iicbus3
lkpi_iic4: <LinuxKPI I2C> on drmn0
iicbus4: <Philips I2C bus> on lkpi_iic4
iic4: <I2C generic I/O> on iicbus4
lkpi_iic5: <LinuxKPI I2C> on drmn0
iicbus5: <Philips I2C bus> on lkpi_iic5
iic5: <I2C generic I/O> on iicbus5
lkpi_iic6: <LinuxKPI I2C> on drmn0
iicbus6: <Philips I2C bus> on lkpi_iic6
iic6: <I2C generic I/O> on iicbus6
lkpi_iic7: <LinuxKPI I2C> on drmn0
iicbus7: <Philips I2C bus> on lkpi_iic7
iic7: <I2C generic I/O> on iicbus7
lkpi_iic8: <LinuxKPI I2C> on drmn0
iicbus8: <Philips I2C bus> on lkpi_iic8
iic8: <I2C generic I/O> on iicbus8
drmn0: successfully loaded firmware image 'i915/tgl_dmc_ver2_12.bin'
drmn0: [drm] Finished loading DMC firmware i915/tgl_dmc_ver2_12.bin (v2.12)
sysctl_warn_reuse: can't re-use a leaf (hw.dri.debug)!
lkpi_iic9: <LinuxKPI I2C> on drm1
iicbus9: <Philips I2C bus> on lkpi_iic9
iic9: <I2C generic I/O> on iicbus9
lkpi_iic10: <LinuxKPI I2C> on drm3
iicbus10: <Philips I2C bus> on lkpi_iic10
iic10: <I2C generic I/O> on iicbus10
lkpi_iic11: <LinuxKPI I2C> on drm5
iicbus11: <Philips I2C bus> on lkpi_iic11
iic11: <I2C generic I/O> on iicbus11
[drm] Initialized i915 1.6.0 20201103 for drmn0 on minor 0
VT: Replacing driver "efifb" with new "fb".
fbd0 on drmn0
device_attach: fbd0 attach returned 6
lkpi_iic12: <LinuxKPI I2C> on drmn0
iicbus12: <Philips I2C bus> on lkpi_iic12
iic12: <I2C generic I/O> on iicbus12
fbd0 on drmn0
device_attach: fbd0 attach returned 6
lkpi_iic13: <LinuxKPI I2C> on drmn0
iicbus13: <Philips I2C bus> on lkpi_iic13
iic13: <I2C generic I/O> on iicbus13
start FB_INFO:
type=11 height=1080 width=1920 depth=32
pbase=0x4000000000 vbase=0xfffffe0162e00000
name=drmn0 flags=0x0 stride=7680 bpp=32
end FB_INFO
ig4iic0: <Intel Tiger Lake-LP I2C Controller-4> at device 21.0 on pci0
ig4iic0: Using MSI
iicbus14: <Philips I2C bus (ACPI-hinted)> on ig4iic0
iic14: <I2C generic I/O> on iicbus14
ig4iic1: <Intel Tiger Lake-LP I2C Controller-5> at device 21.1 on pci0
ig4iic1: Using MSI
iicbus15: <Philips I2C bus (ACPI-hinted)> on ig4iic1
iicbus15: <unknown card> at addr 0x2c
iic15: <I2C generic I/O> on iicbus15
ichsmb0: <Intel Tiger Lake SMBus controller> port 0xefa0-0xefbf mem 0x605319c000-0x605319c0ff at device 31.4 on pci0
smbus0: <System Management Bus> on ichsmb0
rtsx0: <2.1g Realtek RTS525A PCIe SD Card Reader> mem 0xa2200000-0xa2200fff at device 0.0 on pci1
rtsx0: No card is detected
acpi_wmi0: <ACPI-WMI mapping> on acpi0
acpi_wmi0: Embedded MOF found
ACPI: \134_SB.WFDE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
acpi_wmi1: <ACPI-WMI mapping> on acpi0
acpi_wmi1: Embedded MOF found
ACPI: \134_SB.WFTE.WQCC: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
acpi_wmi2: <ACPI-WMI mapping> on acpi0
acpi_wmi2: Embedded MOF found
ACPI: \134_SB.AMW0.WQMO: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
acpi_wmi3: <ACPI-WMI mapping> on acpi0
acpi_wmi3: Embedded MOF found
ACPI: \134_SB.AMW2.WQMO: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
acpi_wmi4: <ACPI-WMI mapping> on acpi0
acpi_wmi4: Embedded MOF found
ACPI: \134_SB.AMW5.WQMO: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
acpi_wmi5: <ACPI-WMI mapping> on acpi0
acpi_wmi5: Embedded MOF found
ACPI: \134_SB.AMW4.WQMO: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
acpi_wmi6: <ACPI-WMI mapping> on acpi0
acpi_wmi6: Embedded MOF found
ACPI: \134_SB.DIAG.WQMO: 1 arguments were passed to a non-method ACPI object (Buffer) (20221020/nsarguments-361)
bridge0: Ethernet address: 58:9c:fc:10:1b:50
bridge0: changing name to 'wifibox0'
tap0: Ethernet address: 58:9c:fc:10:ef:46
tap0: promiscuous mode enabled
wifibox0: link state changed to DOWN
ppt0 mem 0x6053194000-0x6053197fff at device 20.3 on pci0
tap0: link state changed to UP
wifibox0: link state changed to UP
KLD vboxdrv.ko: depends on kernel - not available or version mismatch
lo0: link state changed to UP
Cuse v0.1.37 @ /dev/cuse
ure0 on uhub2
ure0: <Realtek USB 10/100/1000 LAN, class 0/0, rev 3.00/31.11, addr 4> on usbus0
ums0 on uhub7
ums0: <Logitech USB Receiver, class 0/0, rev 2.00/39.06, addr 4> on usbus1
ums0: 16 buttons and [XYZT] coordinates ID=2
uhid0 on uhub7
uhid0: <Logitech USB Receiver, class 0/0, rev 2.00/39.06, addr 4> on usbus1
ums1 on uhub7
ums1: <Logitech USB Receiver, class 0/0, rev 2.00/5.01, addr 5> on usbus1
ums1: 16 buttons and [XYZT] coordinates ID=2
uhid1
 on uhub7
uhid1: <Logitech USB Receiver, class 0/0, rev 2.00/5.01, addr 5> on usbus1
miibus0: <MII bus> on ure0
rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
ue0: <USB Ethernet> on ure0
ue0: Ethernet address: d8:d0:90:46:ef:0a
ue0: link state changed to DOWN
uhid2 on uhub5
uhid2: <vendor 0x413c Dell dock, class 0/0, rev 2.01/1.01, addr 7> on usbus1
uhid3 on uhub3
uhid3: <vendor 0x413c Dell dock, class 0/0, rev 2.01/1.01, addr 8> on usbus1
uaudio0 on uhub5
uaudio0: <Generic USB Audio, class 0/0, rev 2.00/0.01, addr 6> on usbus1
uaudio0: Play[0]: 48000 Hz, 2 ch, 24-bit S-LE PCM format, 2x4ms buffer. (selected)
uaudio0: Play[1]: 48000 Hz, 2 ch, 24-bit S-LE PCM format, 2x4ms buffer. (selected)
uaudio0: Record[0]: 48000 Hz, 2 ch, 24-bit S-LE PCM format, 2x4ms buffer. (selected)
uaudio0: No MIDI sequencer.
pcm3 on uaudio0
pcm4 on uaudio0
uaudio0: No HID volume keys found.
uaudio1 on uhub0
uaudio1: <vendor 0x046d HD Pro Webcam C920, class 239/2, rev 2.00/0.11, addr 9> on usbus1
uaudio1: No playback.
uaudio1: Record[0]: 32000 Hz, 2 ch, 16-bit S-LE PCM format, 2x4ms buffer. (selected)
uaudio1: Record[0]: 24000 Hz, 2 ch, 16-bit S-LE PCM format, 2x4ms buffer.
uaudio1: Record[0]: 16000 Hz, 2 ch, 16-bit S-LE PCM format, 2x4ms buffer.
uaudio1: No MIDI sequencer.
pcm5 on uaudio1
uaudio1: No HID volume keys found.
ubt0 on uhub0
ubt0: <vendor 0x8087 product 0x0026, class 224/1, rev 2.01/0.02, addr 11> on usbus1
iichid0: <DELL0A20:00 0488:101A I2C HID device> at addr 0x2c on iicbus15
iichid0: Interrupt setup failed. Fallback to sampling
hidbus0: <HID bus> on iichid0
hms0: <DELL0A20:00 0488:101A Mouse> on hidbus0
hms0: 3 buttons and [XYW] coordinates ID=6
hmt0: <DELL0A20:00 0488:101A TouchPad> on hidbus0
hconf0: <DELL0A20:00 0488:101A Configuration> on hidbus0
hmt0: Multitouch touchpad with 0 external buttons, click-pad
hmt0: 5 contacts with [C] properties. Report range [0:0] - [1436:831]
Security policy loaded: MAC/ntpd (mac_ntpd)
bridge1: Ethernet address: 58:9c:fc:10:ff:f2
bridge1: changing name to 'virbr0'
 
Interesting that the hanging find(1) is consuming resources. "Normally" it should just sit there.

Can you attach truss to it?
 
Interesting that the hanging find(1) is consuming resources. "Normally" it should just sit there.
This is a contradiction, indeed. The other symptoms (unkillable process, system not rebooting) are the typical zfs locking stall (and there is a couple of unresolved issues which can create that, e.g. running "zpool import" with a bunch of spun-down SCSI disks, which aio does not handle correctly), but then these do not use ressources.

I might assume the find process is an independent matter, not related to the zfs locking.
 
cracauer

So, I can attach truss to the process, however it appears to be dead with no output.

Code:
$ sudo truss -p 37477

Process according to ps is in a runnable state (R)
Code:
$ ps ax |grep zroot
37477  -  R     575:18.10 / /zroot /home /usr/src /usr/local/bastille /var/mail /usr/local/poudriere /var/log/bastille /usr/local/bastille/backups /usr/local/poudriere/data /usr/local/bastille/jails /usr/local/
18342  0  S+      0:00.00 grep zroot

Code:
$ top -n
last pid: 22118;  load averages:  1.53,  1.39,  1.27; battery: 100%  up 0+20:02:11    12:39:39
135 processes: 2 running, 133 sleeping
CPU:  0.4% user,  0.0% nice,  6.2% system,  0.1% interrupt, 93.2% idle
Mem: 1388M Active, 6655M Inact, 617M Laundry, 6306M Wired, 81K Buf, 16G Free
ARC: 2640M Total, 583M MFU, 1292M MRU, 94K Anon, 23M Header, 738M Other
     1335M Compressed, 4452M Uncompressed, 3.34:1 Ratio
Swap: 2048M Total, 2048M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
37477 root          1  20    0    30M    17M CPU1     1 578:32 100.00% find
 
I just wrote myself a little find listener and restarted my computer, will report back hopefully with a log of truss when it starts again.
 
I used clonezilla to do a full disk backup, probably could have used zfs send.. but nonetheless I didn't have any issues cloning / restoring the entire disk.
 
cracauer covacat Cath O'Deray

Ok, so the plot thickens. I was able to determine when the process initially started, but the initial find process gets forked off and I didn't use -f to follow the forked process. But I did get the first find process, truss of that is below along with procstat and (attempted) gdb.

find listener

Code:
Waiting for find to start...
Process find with PID 70189 detected. Attaching truss.
Thu Sep 26 03:01:00 EDT 2024
Process find with PID 70189 has terminated or truss detached.


cron log
Code:
Sep 26 03:00:00 freebsd /usr/sbin/cron[6017]: (root) CMD (newsyslog)
Sep 26 03:00:00 freebsd /usr/sbin/cron[6248]: (operator) CMD (/usr/libexec/save-entropy)
Sep 26 03:01:00 freebsd /usr/sbin/cron[33173]: (root) CMD (adjkerntz -a)
Sep 26 03:05:00 freebsd /usr/sbin/cron[75842]: (root) CMD (/usr/libexec/atrun)
Sep 26 03:10:00 freebsd /usr/sbin/cron[77119]: (root) CMD (/usr/libexec/atrun)
Sep 26 03:11:00 freebsd /usr/sbin/cron[77511]: (operator) CMD (/usr/libexec/save-entropy)
Sep 26 03:15:00 freebsd /usr/sbin/cron[79172]: (root) CMD (/usr/libexec/atrun)
Sep 26 03:20:00 freebsd /usr/sbin/cron[80110]: (root) CMD (/usr/libexec/atrun)
Sep 26 03:22:00 freebsd /usr/sbin/cron[80445]: (operator) CMD (/usr/libexec/save-entropy)


truss

Code:
fstatat(AT_FDCWD,"json_parser.hpp",{ mode=-rw-r--r-- ,inode=498417,size=10301,blksize=10752 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"orcus_import_ods.hpp",{ mode=-rw-r--r-- ,inode=498424,size=803,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"json_parser_base.hpp",{ mode=-rw-r--r-- ,inode=498418,size=1088,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"orcus_import_xlsx.hpp",{ mode=-rw-r--r-- ,inode=498425,size=947,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"orcus_xls_xml.hpp",{ mode=-rw-r--r-- ,inode=498428,size=1177,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"tokens.hpp",{ mode=-rw-r--r-- ,inode=498715,size=2022,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"measurement.hpp",{ mode=-rw-r--r-- ,inode=498421,size=1291,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"sax_parser_base.hpp",{ mode=-rw-r--r-- ,inode=498691,size=5849,blksize=6144 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"orcus_csv.hpp",{ mode=-rw-r--r-- ,inode=498422,size=1041,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"detail",{ mode=drwxr-xr-x ,inode=268309,size=4,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"exception.hpp",{ mode=-rw-r--r-- ,inod...skipping...
0)                                                      close(5)                                         = 0 (0x
0)                                                      open("xfce4-wm",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXE
C,01467437114) = 5 (0x5)                                fcntl(5,F_ISUNIONSTACK,0x0)                      = 0 (0x
0)                                                      fstat(5,{ mode=drwxr-xr-x ,inode=197526,size=6,blksize=4
096 }) = 0 (0x0)                                        fchdir(0x5)                                      = 0 (0x
0)                                                      getdirentries(5,"\M^V\^C\^C\0\0\0\0\0\^A\0\0\0\0"...,409
6,{ 0x0 }) = 224 (0xe0)                                 fstatat(AT_FDCWD,"distinfo",{ mode=-rw-r--r-- ,inode=197
528,size=170,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"pkg-descr",{ mode=-rw-r--r-- ,inode=197529,size=159,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0
x0)                                                     fstatat(AT_FDCWD,"Makefile",{ mode=-rw-r--r-- ,inode=197
527,size=1231,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
fstatat(AT_FDCWD,"pkg-plist",{ mode=-rw-r--r-- ,inode=197530,size=39953,blksize=40448 },AT_SYMLINK_NOFOLLOW) = 0
 (0x0)                                                  getdirentries(5,0x5a8d004f1000,4096,{ 0x1dfaf09c }) = 0
(0x0)                                                   close(5)                                         = 0 (0x
0)
open("..",O_RDONLY|O_DIRECTORY|O_CLOEXEC,00)     = 5 (0x
...skipping...
close(5)                                         = 0 (0x0)
fchdir(0x4)                                      = 0 (0x0)
fchdir(0x4)                                      = 0 (0x0)
statfs("/zroot",{ fstypename=zfs,mntonname=/zroot,mntfromname=zroot,fsid=04cddbe5de4816da }) = 0 (0x0)
open("/zroot",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,034566746404) = 5 (0x5)
fcntl(5,F_ISUNIONSTACK,0x0)                      = 0 (0x0)
fstat(5,{ mode=drwxr-xr-x ,inode=34,size=2,blksize=4096 }) = 0 (0x0)
fchdir(0x5)                                      = 0 (0x0)
getdirentries(5,""\0\0\0\0\0\0\0\^A\0\0\0\0\0\0\0"...,4096,{ 0x0 }) = 64 (0x40)
getdirentries(5,0x5a8d004f1000,4096,{ 0x2 })     = 0 (0x0)
close(5)                                         = 0 (0x0)
fchdir(0x4)                                      = 0 (0x0)
fork()                                           = 72146 (0x119d2)
wait4(72146,{ EXITED,val=0 },0x0,0x0)            = 72146 (0x119d2)
exit(0x0)

gdb
Code:
$ sudo gdb -p 74779
GNU gdb (GDB) 14.1 [GDB v14.1 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 74779
bt
^C^C^C^C
^C


procstat
Code:
$ sudo procstat -r 74779
  PID COMM             RESOURCE                          VALUE   
74779 find             user time                    00:00:00.236219
74779 find             system time                  08:06:32.383643
74779 find             maximum RSS                            17796 KB
74779 find             integral shared memory             103809580 KB
74779 find             integral unshared data              14829940 KB
74779 find             integral unshared stack            474558080 KB
74779 find             page reclaims                           4005
74779 find             page faults                                0
74779 find             swaps                                      0
74779 find             block reads                                0
74779 find             block writes                               0
74779 find             messages sent                              0
74779 find             messages received                          0
74779 find             signals received                           0
74779 find             voluntary context switches                16
74779 find             involuntary context switches           53757
 
Here is the htop tree view


Code:
10446 root        20   0 12916  2552 S   0.0  0.0  0:00.19 ├─ /usr/sbin/cron -s
32935 root        21   0 12916  2556 S   0.0  0.0  0:00.00 │  └─ cron: running job
33469 root        40   0 13376  2924 S   0.0  0.0  0:00.00 │     └─ /bin/sh - /usr/sbin/periodic daily
34220 root        40   0 12712  2136 S   0.0  0.0  0:00.00 │        └─ lockf -s -t 0 /var/run/periodic.daily.lock /bin/sh /usr/sbin/periodic LOCKED daily
34580 root        68   0 13376  2908 S   0.0  0.0  0:00.00 │           └─ /bin/sh /usr/sbin/periodic LOCKED daily
36476 root        68   0 13376  2920 S   0.0  0.0  0:00.00 │              ├─ /bin/sh /usr/sbin/periodic LOCKED daily
65291 root        68   0 13376  2916 S   0.0  0.0  0:00.00 │              │  └─ /bin/sh /etc/periodic/daily/450.status-security
65938 root        68   0 13376  2912 S   0.0  0.0  0:00.00 │              │     └─ /bin/sh - /usr/sbin/periodic security
66524 root        68   0 12712  2128 S   0.0  0.0  0:00.00 │              │        └─ lockf -s -t 0 /var/run/periodic.security.lock /bin/sh /usr/sbin/periodic LOCKED security
66649 root        68   0 13376  2916 S   0.0  0.0  0:00.00 │              │           └─ /bin/sh /usr/sbin/periodic LOCKED security
68743 root        20   0 13376  2924 S   0.0  0.0  0:00.00 │              │              ├─ /bin/sh /usr/sbin/periodic LOCKED security
73105 root        36   0 13376  2936 S   0.0  0.0  0:00.00 │              │              │  └─ /bin/sh - /etc/periodic/security/110.neggrpperm
74439 root        37   0 13376  2928 S   0.0  0.0  0:00.00 │              │              │     └─ /bin/sh - /etc/periodic/security/110.neggrpperm
74779 root        20   0 31200 17812 R 100.0  0.1  8h13:51 │              │              │        ├─ / /usr/src /zroot /home /usr/local/bastille /usr/local/poudriere /var/mail /var/log/bastille /usr/local/poudri
 
It's just a shell script, you can try running it manually with "-x" from a root shell and see if it logs anything. In my case I tried and it didn't log anything, though. So I just added a "-print" before the "-type f" and it of course printed every file it was checking at the moment. It should hang on the last accessed file.
 
Here is truss for the manual run of /etc/periodic/security/110.neggrpperm


Code:
49289: getdirentries(5,"\M-fA\n\0\0\0\0\0\^A\0\0\0\0\0\0"...,4096,{ 0x0 }) = 200 (0xc8)
49289: fstatat(AT_FDCWD,"Cache_Data",{ mode=drwx------ ,inode=101914,size=110419,blksize=16384 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
49289: fstatat(AT_FDCWD,"tmp2",{ mode=-rw-r--r-- ,inode=768596,size=2207,blksize=4096 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
49289: fstatat(AT_FDCWD,"old",{ mode=-rw-r--r-- ,inode=672354,size=9411,blksize=9728 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
49289: fstatat(AT_FDCWD,"new",{ mode=-rw-r--r-- ,inode=699468,size=9376,blksize=9728 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
49289: getdirentries(5,0x2ecb260f1000,4096,{ 0x1f284822 }) = 0 (0x0)
49289: close(5)                     = 0 (0x0)
49289: open("Cache_Data",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,017354302024) = 5 (0x5)
49289: fcntl(5,F_ISUNIONSTACK,0x0)         = 0 (0x0)
49289: fstat(5,{ mode=drwx------ ,inode=101914,size=110419,blksize=16384 }) = 0 (0x0)
49289: fchdir(0x5)

... and there it stays. Doesn't explain much of anything outside the fact that Cache_Data is what's causing this problem, and I cannot delete it / seek into it.
 
What about Midnight Commander? In case undeleteable files and dirs, because of symbols in their names, this tool is like magic stick.
 
I know it's sort of an insulting stupid question, but have you checked for any hardware problems? Such as a disk that is intermittently not communicating, or throwing an error here or there, or IOs taking minutes instead of milliseconds? And HBA that keeps getting reset and has to restart all IOs? A look at dmesg or /var/log/messages would show these kinds of problems. It's a long shot; the description of what is going wrong really sounds more like a metadata (=bug?) issue within ZFS.
 
I don't know whether it's a bug, I'm fairly certain that there's no problem if the cron job is given time to complete.

I'm sometimes careless with my timing of work on boot environments.

Reminding myself of occasions when I used lsof to identify the cause of inability to unmount an environment:

Code:
% cd Documents/IT/BSD/FreeBSD/updates,\ upgrades/
% rg --count --sort path lsof .
./1500014-aa base package upgrades.txt:1
./1500014-v base package upgrades.txt:1
./1500015-06-base.txt:1
./1500017-17-ports 1500017-18-kde6.txt:1
./1500018-080-base-ports.txt:1
./1500018-36-base-ports.txt:1
./1500019-026-base-linux-c7.txt:1
./1500019-037-base.txt:4
./1500023-028-base-ports.txt:1
./1500023-044-base.txt:3
./1500023-048-base-ports.txt:1
./1500023-050-ports.txt:2
./1500023-072-base.txt:1
./2024-01-13 package upgrades.txt:3
% less ./1500023-072-base.txt
% less ./1500023-050-ports.txt
% less ./1500023-048-base-ports.txt
% less ./1500023-044-base.txt
% less ./1500023-028-base-ports.txt
%

From the latter (after stopping cron, I used htop to kill processes):

Code:
…
root@mowa219-gjp4-zbook-freebsd:~ # bectl umount 1500023-028-base-ports
cannot unmount '/tmp/up': pool or dataset is busy
specified path is busy
Failed to unmount bootenv 1500023-028-base-ports
root@mowa219-gjp4-zbook-freebsd:~ # lsof /tmp/up
COMMAND   PID USER  FD   TYPE                DEVICE SIZE/OFF    NODE NAME
find    40160 root cwd   VDIR 1125959602,4144094829     4721 4444027 /tmp/up/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG (august/ROOT/1500023-028-base-ports)
find    40160 root   5r  VDIR 1125959602,4144094829     4721 4444027 /tmp/up/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG (august/ROOT/1500023-028-base-ports)
…
root@mowa219-gjp4-zbook-freebsd:~ # htop
root@mowa219-gjp4-zbook-freebsd:~ # service cron stop
Stopping cron.
Waiting for PIDS: 2913.
…
root@mowa219-gjp4-zbook-freebsd:~ # htop
root@mowa219-gjp4-zbook-freebsd:~ # bectl umount 1500023-028-base-ports
root@mowa219-gjp4-zbook-freebsd:~ #
…
 
… hardware problems? …

Somehow, I doubt it.

… created by the chromium browser, specifically the directory $HOME/.cache/chromium/Cache/Cache_Data. …

No such directory when I run Chromium:

Code:
% ps aux | grep chrome:\ -- | wc -l
      10
% file $HOME/.cache/chromium/Cache/Cache_Data
/home/grahamperrin/.cache/chromium/Cache/Cache_Data: cannot open `/home/grahamperrin/.cache/chromium/Cache/Cache_Data' (No such file or directory)
%

I have this:

Code:
% file $HOME/.cache/chromium/Default/Cache/Cache_Data
/home/grahamperrin/.cache/chromium/Default/Cache/Cache_Data: directory
% du -hs $HOME/.cache/chromium/Default/Cache/Cache_Data
313M    /home/grahamperrin/.cache/chromium/Default/Cache/Cache_Data
%
 
I know it's sort of an insulting stupid question, but have you checked for any hardware problems? Such as a disk that is intermittently not communicating, or throwing an error here or there, or IOs taking minutes instead of milliseconds? And HBA that keeps getting reset and has to restart all IOs? A look at dmesg or /var/log/messages would show these kinds of problems. It's a long shot; the description of what is going wrong really sounds more like a metadata (=bug?) issue within ZFS.

I've already changed the hard disk that the OS is installed on to rule out HDD issues. There is nothing in the kernel buffer or messages log that indicates any sort of issue.
No such directory when I run Chromium:

Code:
% ps aux | grep chrome:\ -- | wc -l
      10
% file $HOME/.cache/chromium/Cache/Cache_Data
/home/grahamperrin/.cache/chromium/Cache/Cache_Data: cannot open `/home/grahamperrin/.cache/chromium/Cache/Cache_Data' (No such file or directory)
%

I have this:

Code:
% file $HOME/.cache/chromium/Default/Cache/Cache_Data
/home/grahamperrin/.cache/chromium/Default/Cache/Cache_Data: directory
% du -hs $HOME/.cache/chromium/Default/Cache/Cache_Data
313M    /home/grahamperrin/.cache/chromium/Default/Cache/Cache_Data
%
Yes I apologize, the directory is (was) located in $HOME/.cache/chromium/Default/Cache/Cache_Data.

For the time being, I added in the below into the /etc/periodic/security/110.neggrpperm file so it ignores that directory..but I'd still like to know why the directory is causing so many issues. Is it a bug in chrome? ZFS? I'm really thinking this isn't due to hardware issues. In all the years I've been using Unix/Linux I never once have had this issue before, even when there's clearly hardware issues present.

After this change, I was able to run the job successfully, it really is just this directory causing the problem

Code:
/etc/periodic/security/110.neggrpperm: Ln 60.
n=$(find -sx $MP /dev/null -path /home/user/tmp/Cache_Data -prune \( ! -fstype local \) -prune -o -type f
 
So what do we know? Here's my summary. There is a directory /home/user/tmp/Cache_Data that causes a find job that tries to read the directory to hang, indefinitely. Similarly, user access (from the CLI) is also impossible, although I don't know what the exact symptoms are (hang? crash? program gets an error and exits?). Hardware problems are very unlikely.

How would one debug the root cause? I'm not a ZFS expert (never took Kirk's internals class, never looked at the source code), but I would start with zdb and look at that directory.
 
Back
Top