ZFS zfs: remove a stubborn directory

vext01

vext01

Hi,

Something strange is happening with a directory on one of my ZFS partitions:

Code: 
$ find deleteme/
deleteme/
find: deleteme/: Input/output error
$ cd deleteme/
$ ls
$ cd ..
$ rm -rf deleteme
rm: deleteme: Input/output error
$ find . -name 'deleteme' -type d
./deleteme
find: ./deleteme: Input/output error
$ find . -name 'deleteme' -type d -delete
find: ./deleteme: Input/output error

zpool status -v indeed lists this as a corrupted directory. It's supposed to contain a file too. I have a backup, but I need to be able to delete the corrupted directory first.

I also tried copying the missing file into the directory, but alas, this is also an I/O error.

Any ideas? Thanks

(Got a scrub going now to see if anything else is busted)
 
Last edited by a moderator:
It looks like one of your vDevs has been damaged. Could you tell us what your pool configuration is? It looks like it's in a srtipe.

If you have it configured that way, you would need to use a backup to recover the directory or file in question. On the other hand, you can't delete it because the sector is likely damaged and it's impossible to perform operations on it. What is the model of your drive?

If your vDev is composed of an HDD, example:

diskinfo -v da0
geom disk list

Please show us the identifier, where we can see the model. I'm telling you this because the disk depends on its quality, and you'd have to anticipate that sector error.

On the other hand use the tool smartctl(8) and run a test to see if the disk is indeed damaged. To eliminate that I/O, you'll likely have to recreate the pool, but with a different vdev.
 
Hi,

Thanks for the response. I should have posted a follow up, because I've tried a few things since and somewhat resolved the issue..

I've done the following things:
- ran memtest86 on the machine. All OK.
- done smartctl long tests on the two disks involved in the mirror. All OK.
- made a new zfs dataset, copied everything from the troublesome dataset to the new dataset and restored the corrupted directory from a backup.
- reviewed offsite backups

I don't know what caused this in the first place, but (fingers crossed) all seems well for now. All I need to do is `zfs destroy` the broken dataset -- I've kept it around for now in case we want to do any post-mortem on it.

Here's the info you requested:

Code: 
$ doas zpool status
pool: sea
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 05:39:25 with 1 errors on Sat Mar  8 15:06:44 2025
config:

    NAME        STATE     READ WRITE CKSUM
    sea         ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada2    ONLINE       0     0    28
        ada3    ONLINE       0     0    28

errors: 1 data errors, use '-v' for a list

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:02:10 with 0 errors on Sat Mar  8 08:02:31 2025
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p4  ONLINE       0     0     0
        ada1p4  ONLINE       0     0     0

Code: 
$ doas diskinfo -v ada2
Password:
ada2
    512             # sectorsize
    4000787030016    # mediasize in bytes (3.6T)
    7814037168      # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    7752021         # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    ST4000VN006-3CW104    # Disk descr.
    ZW609S4H        # Disk ident.
    ahcich2         # Attachment
    id1,enc@n3061686369656d30/type@0/slot@3/elmdesc@Slot_02    # Physical path
    No              # TRIM/UNMAP support
    5400            # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Code: 
$ doas diskinfo -v ada3
Password:
ada3
    512             # sectorsize
    4000787030016    # mediasize in bytes (3.6T)
    7814037168      # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    7752021         # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    ST4000VN006-3CW104    # Disk descr.
    ZW608F2X        # Disk ident.
    ahcich3         # Attachment
    id1,enc@n3061686369656d30/type@0/slot@4/elmdesc@Slot_03    # Physical path
    No              # TRIM/UNMAP support
    5400            # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Code: 
$ doas geom disk list
Password:
Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   descr: ST4000VN006-3CW104
   lunid: 5000c500e5b888ed
   ident: ZW609S4H
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada3
Providers:
1. Name: ada3
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   descr: ST4000VN006-3CW104
   lunid: 5000c500e5a35373
   ident: ZW608F2X
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 480103981056 (447G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: CT480BX500SSD1
   lunid: 500a0751e688800e
   ident: 2247E688800E
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r3w3e5
   descr: Samsung SSD 840 EVO 250GB
   lunid: 50025388a05f7ec8
   ident: S1DBNSAF820123L
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Does this all sound OK to you?
 
Looks OK, but you have 28 errors on both devices in a 2-way mirror. There's probably a couple other broken filesystem items.

ZFS has no fsck und sometimes that matters.
 
Everything seems fine; the disks in the pool are of decent quality. On the other hand, if I run smartctl and everything seems correct, and since it's a mirror, it should have recovered those errors since it has redundancy, unless both disks had the same error.

So my next question is, is your RAM ECC?

On the other hand, I'm ruling out running a SCRUB since you have the errors on both Vdevs.

EDIT:

Could you display the pool status in verbose mode to see those failed objects?
 
Ram is not ECC. It's a desktop machine repurposed.

I think the 28 errors are repeated failures for the same file, as only one file is listed corrupted:

Code: 
$ doas zpool status -v
Password:
  pool: sea
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 05:39:25 with 1 errors on Sat Mar  8 15:06:44 2025
config:

    NAME        STATE     READ WRITE CKSUM
    sea         ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada2    ONLINE       0     0    28
        ada3    ONLINE       0     0    28

errors: Permanent errors have been detected in the following files:

        /sea/media/music.corrupted/OGG/various_artists/ungrouped_files/deleteme

Why would the checksum be incorrect on both disks though?
 
The checksum is incorrect on both vdevs, possibly due to a failure in the RAM or SATA controller. These could be two causes. This data corruption has been replicated across your vdevs, making it impossible to recover the file due to a lack of replicas in your pool.

I'm not a ZFS specialist, but it's highly recommended to use ECC RAM since ZFS uses RAM as a cache (ARC). If something goes wrong in the RAM, its bits could be incorrect, resulting in data corruption.

But I don't want to get into a future discussion with a user, for example, like this: "Memory failures aren't common... I've been using ZFS without ECC RAM for years and everything's fine..."

I don't know if a more experienced user could help diagnose these devices, but my experience tells me that memtest86 can't be giving you results every minute or second to determine if there's a failure.

If you have a backup, as you mentioned, try saving everything in that datasheet. Then, replicate the backup again to delete the datasheet with the data corruption and clean up the pool.
 
Thanks.

Yeah, ideally I'd have ECC RAM, but this is just an upcycled desktop machine and I probably don't have the budget to buy hardware at the moment.

In a way it's good that ZFS flagged this, as otherwise I'd have been none the wiser.
 
freejlr said:
The checksum is incorrect on both vdevs, possibly due to a failure in the RAM or SATA controller. These could be two causes
Click to expand...
vext01, in addition, there could be a bad physical connection from your motherboard sata connectors to your hard disks. First (cheap) thing I'd try is carefully check (think cleanlyness and physical feal) & reseat all connectors and check regularly (scrub). After that you could try installing new cables. I'm all for ECC but, the problem might be somewhere else. If you don't have much (expensive) RAM, you also could consider swapping out modules, allthough if there is a bad memory module, stress testing memory should probably reveal this as well. This search for the cause of errors may be somewhat painstaking but with ZFS at least you should get a predictable error warning, and as long as you don't run out of redundancy the problem can be easily temporarily resolved.
 
I don't think it is SATA or cables thereof, as you would have to have 2 failures at the same time. Memory corruption is more likely. But RAM is not the only source of memory corruption. CPUs like to do it, too, although they usually have ECC in the caches.

As always the issue without ECC memory is that if you don't have ECC you can't tell whether you need ECC, as your lack both the correction and the reporting.

Goes back to ZFS not having fsck. It is hard to get rid of such minor corruptions with ZFS, especially if the affected entity is not a plain file.

I'd run memtest86 just to be sure.
 
You must log in or register to reply here.
Back
Top