ZFS unable to overwrite file content in zpool (full of files)

[Alessandro]

hi fellow,

have got unsuspected behavior in Zpool volumes and am looking for a workaround, if any, or at least better understand of my system.

shortly the goal : fill an USB memstick with same-size, plain-text files; then delete the content of the files, leaving untouched the name and location.

root@Amsterdam:~ #  uname -a
FreeBSD Amsterdam 15.0-RELEASE FreeBSD 15.0-RELEASE releng/15.0-n280995-7aedc8de6446 GENERIC amd64

root@Amsterdam:~ #  gpart destroy -F da0
root@Amsterdam:~ #  gpart create -s GPT /dev/da0
root@Amsterdam:~ #  mount /dev/da0 /mnt/test

root@Amsterdam:~ #  gpart destroy -F da1
root@Amsterdam:~ #  zpool create -f -O compression=off -O encryption=on -O keyformat=passphrase ZP_test da1
root@Amsterdam:~ #  zpool status
  pool: ZP_test
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        ZP_test     ONLINE       0     0     0
          da1       ONLINE       0     0     0

errors: No known data errors

said that /tmp holds sufficient files to fillup both memsticks

root@Amsterdam:~ #  cp /tmp/pic* /mnt/test/
root@Amsterdam:~ #  cp /tmp/pic* /ZP_test/

manually deleted last, partially wrote, file : both USB are now full of equally-sized text files

root@Amsterdam:~ #  df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
..............
ZP_test                56G     56G      0B   100%    /ZP_test
/dev/da0               56G     56G   -4.5G   109%    /mnt/test

exec a simple C program that write zeroes (%llu) to whole file(s)

root@Amsterdam:~ #  ./delete /mnt/test
try /mnt/test/pic.0000000 :       zero filled
try /mnt/test/pic.0000001 :       zero filled
try /mnt/test/pic.0000002 :       zero filled
try /mnt/test/pic.0000003 :       zero filled
try /mnt/test/pic.0000004 :       zero filled
try /mnt/test/pic.0000005 :       zero filled
^C


root@Amsterdam:~ #  ./delete /ZP_test
try /ZP_test/pic.0000000 :       zero filled
try /ZP_test/pic.0000001 :       zero filled
try /ZP_test/pic.0000002 :       zero filled
try /ZP_test/pic.0000003 :       zero filled
try /ZP_test/pic.0000004 :       zero filled
try /ZP_test/pic.0000005 :       zero filled
try /ZP_test/pic.0000006 :       zero filled
try /ZP_test/pic.0000007 :       zero filled
^C

while the first run effectively zero-fills the files, the second report the same result, but original data remains in place.

if I instruct program to do multiple SEEK_SET the zfs deletion reports an unexpected error 28 (ENOSPC) that arise a question about how it is possible, while the program opens file in "r+" and does not try to add anything at all.
maybe a misuse of code 28 under some circumstances?

while it is reasonable that system-reported free-space differs, it is unreasonable that overwriting a file in a "full" volume does not.

so the question is : do you know a way to get the desired behavior with zfs ?
in suborder : what is happen for zfs react in that way?

thank you all

______________________________________

fyi the code is

int main(int argc, char **argv) {

    long                size=262144;
    char                fnam[50];
    unsigned int        a,c;
    FILE*               fp;

    for (a=0;a<29000;a++) {
        sprintf (fnam, "/%s/pic.%07u", argv[1],a);
        printf ("try %s :\t",fnam);
        fp = fopen(fnam, "r+");
        if (fp == NULL) {
            printf ("%s unreadable, skipping\n",fnam);
            return 1;
        } else {
            if (fseek(fp, 0L , SEEK_SET)) {
                fprintf(stderr, "%d\n", errno);
                return 4;
            }
            for (c=0;c<size;c++) {
                fprintf(fp,"%llu\n",(long long) 0);
            }
            printf ("zero filled\n");
        }
        fclose (fp);
    }

    printf ("\n");
    return 0;
}

 

[SirDice]

You hopefully understand that ZFS is a Copy-on-write filesystem, while UFS is not?

In traditional file systems, modifying a file overwrites the original data blocks in place. In a copy-on-write (COW) file system, the original blocks remain unchanged. When part of a file is modified, only the affected blocks are written to new locations, and metadata is updated to point to them, preserving the original version until it's no longer needed.

You're never overwriting something on ZFS, it creates new data blocks, which will be really difficult to do if the filesystem is 100% full.

 

[covacat]

i don't know the original file sizes but you are writing 2 bytes in every cycle so your files will have 512k in the end. you don't really zero the file you fill it with
0
0
0
in ascii

 

[hedwards]

You hopefully understand that ZFS is a Copy-on-write filesystem, while UFS is not?



You're never overwriting something on ZFS, it creates new data blocks, which will be really difficult to do if the filesystem is 100% full.

Yep, and that's where reservations and quotas can be rather helpful. But, that's something best to consider putting in ahead of time to keep you under generally 80% for appropriate wiggle room to fix things if need be.

 

[T-Aoki]

Another thing to consider on ZFS is snapshots.

Once a snapshot is created, following writes are "records of what's changed starting from the snapshot". Deletions of files / directories that were existed BEFORE snapshot is created are just records for "this item is deleted and should not be accessed anymore". Not actual deletion. Only deletions for anything after the snapshot was created can be actual deletion.

So the only way to free diskspace for a file is to destroy ALL snapshots after the file is created.

But as snapshots are (basically) per-dataset, you can mount a snapshot that still contains missingly deleted file(s) and copy them temporarily to different dataset before destroying snapshots, and restore after destroys.

 

[mer]

Yep T-Aoki is on target with the explaination. We've had more than a few threads on the forum, usually related to BEs and updating.
Blocks are not freed until all references to them are released.
BEs are technically "clones", a writable layer above a snapshot, when you destroy a BE you need to make sure you destroy the underlying snapshot too. I think beadm does that by default, bectl I think in the beginning did not, so you had to add the "-o" but that may have changed by now.