Solved Turn CoW off zfs?

LibreQuest

LibreQuest

I'm getting a bit frustrated with zfs again. I have 2.8TB of data in trying to move over to a ZFS FreeBSD 14.0 64Bit system and it keeps running out of space. The system has x2 3TB and x1 250GB SATA 3.5" Drives. I assume there should be nearly 5.5TB of useable space but it runs out of space everytime. df -h reports 5.5TB of space on /home.

Am I wrong to believe the data is being copied as written the copy on write function of ZFS and that is why it runs out of storage?

If that is in fact the issue can it be turned off?

If not I will buy more storage.

Thanks in advance!
 
You may want to provide the output of zpool status and zfs list on both the source and destination systems.
df -h on ZFS may not be the most accurate way of reporting space.
 
2x3Tb, but is this a zfs mirror? If so, your zpool capacity will not be more than 3TB (actually it will be less than 3TB, approx. 2.7TB)
So show
zpool status
zfs list
on a both systems (old and new)
 
I'm not at the system at the moment. However the configuration is not mirror it's a raid 0.

The source disks are x4 1tb drives in ext4. They are my backup drives I am transferring over from a Linux backup. I'm moving my backups to FreeBSD. Been meaning to do this for a while.

If I cannot disable a function of the ZFS file system, CoW, I will just expand the available storage.
 
CoW in ZFS is actually a simple concept.
Everything in a filesystem is based on "blocks". A block is a chunk of data.
How does that relate to ZFS/CoW?

ZFS, I have a file, sized K blocks. I take a snapshot that includes that file; the SIZE of the snapshot is almost nothing, but effectively a refcount on the original blocks is incremented.
Now I change that file, maybe changing the contents of block #2 of the file. What happens? CoW means the block, BEFORE the change, moves to the snapshot. So the size of the snapshot increases by the number of changed blocks AND the current file allocation has the new blocks. That means 2 blocks for the price of one; snapshot has the old block added, the current dataset has the new block. That's what CoW does. CoW means "increment refcount on a block unless it is changed. Then copy the old block and apply changes to a new block"

So if your source is 4x 1TB drives and you are doing a "cp -RpP sourcedir destdir" where sourcedir is the ext4 and destdir is a ZFS dataset then all CoW aspects of ZFS do not come into play. But you need to keep in mind how you are doing the copy (rsync, dd, ????) and where the destdir lives. That's why we've asked about zfs list and zpool status on both or at least the destination system.
 
I can give the details when I am with the system.

The method is this

scp -r /backup/dir1 name@0.0.0.0:/destination/dir1

For each drive I am moving a copy over the local network to the FreeBSD machine.

In this case the src is a Linux system and destination is FreeBSD.

The FreeBSD system is a fresh install with x2 3TB and x1 250GB striped raid 0 drives.

Edit: I have installed ncdu, sudo and a user account in the FreeBSD system.
 
mer said:
CoW in ZFS is actually a simple concept.
Everything in a filesystem is based on "blocks". A block is a chunk of data.
How does that relate to ZFS/CoW?

ZFS, I have a file, sized K blocks. I take a snapshot that includes that file; the SIZE of the snapshot is almost nothing, but effectively a refcount on the original blocks is incremented.
Now I change that file, maybe changing the contents of block #2 of the file. What happens? CoW means the block, BEFORE the change, moves to the snapshot. So the size of the snapshot increases by the number of changed blocks AND the current file allocation has the new blocks. That means 2 blocks for the price of one; snapshot has the old block added, the current dataset has the new block. That's what CoW does. CoW means "increment refcount on a block unless it is changed. Then copy the old block and apply changes to a new block"

So if your source is 4x 1TB drives and you are doing a "cp -RpP sourcedir destdir" where sourcedir is the ext4 and destdir is a ZFS dataset then all CoW aspects of ZFS do not come into play. But you need to keep in mind how you are doing the copy (rsync, dd, ????) and where the destdir lives. That's why we've asked about zfs list and zpool status on both or at least the destination system.
Click to expand...
I understand. It sounds perfect for my use case but overkill for a workstation. Should I be using ufs on my desktop and laptop systems?
 
LibreQuest said:
I understand. It sounds perfect for my use case but overkill for a workstation. Should I be using ufs on my desktop and laptop systems?
Click to expand...
I don't know. I personally like ZFS for almost everything. Why? Boot Environments.
Root on ZFS datasets, Boot environments let you do system upgrades the right way.
You have a system running FreeBSD 13.x and you want to upgrade to FreeBSD 14.x. You can do that into a new Boot Environment (BE); if the upgrade doesn't work you simply reboot into the old 13.x system.

UFS: awesome, stable, been around for a long time so has had lots of issues fixed. Doesn't have BEs (but you can emulate with extra partitions and bootme flags) , maybe lighter on system resources.

My opinion:
I use ZFS on everything.
 
I think the CoW discussion is not really relevant here. It's one way to implement a file system, it has performance tradeoffs that are different from file systems that use traditional (extent or block) allocation. But it doesn't change significantly how much space data uses (modulo minor temporary waste and rounding issues).

The problem here is that the source of the copy has somewhere between 2.8 TB and 4 TB (looking at different messages), but it seems to use over 5.5 TB of space on the target of the copy. Things like that can happen due to expanding sparse files, making multiple copies of soft- or hard-linked files, or copying from a file system that supports dedup and/or compression to one that does not, but for common file system workloads, neither of those should be such a massive effect. And given that the source file system is ext4, compression and dedup should not be an issue. I think we just need to wait for more data.

Here's a suggestion: On the source file system, measure both the total disk space usage (using df), and the total logical size of the source files (adding up all their sizes). If those two numbers disagree wildly, that could point towards a potential root cause.
 
ralphbsz said:
expanding sparse files, making multiple copies of soft- or hard-linked files, or copying from a file system that supports dedup and/or compression to one that does not,
Click to expand...
That is a great explanation.
It shows what great complexity ZFS holds.

I personally feel its best to start with zfs on a dedicated storage machine.
Once you become comfortable with the concepts you may chose to adopt it for everything.
Take small steps.
 
LibreQuest said:
Am I wrong to believe the data is being copied as written the copy on write function of ZFS and that is why it runs out of storage?
Click to expand...

Why do you assume your problem is CoW related?

Note that on ZFS CoW is only for *updating* files, not *copying* them. That is, if a file is updated after a snapshot, this file may share data blocks (& pointer blocks) with the previous version. But if you copy file A to file B, the two won't share any blocks (unless you use dedup).
 
LibreQuest said:
Am I wrong to believe the data is being copied as written the copy on write function of ZFS and that is why it runs out of storage?
Click to expand...
Very unlikely.

Three filesystems, containing the same data:

Code: 
$ df /usr/ports/    # ashift 9
Filesystem   1K-blocks   Used   Avail Capacity  Mounted on
bf/usr/ports   2713385 190609 2522776     7%    /usr/ports
$ cksum /usr/ports/UPDATING
1189486180 194062 /usr/ports/UPDATING
$ du /usr/ports/UPDATING 
64      /usr/ports/UPDATING
$ /sbin/zfs get compression bf/usr/ports
bf/usr/ports  compression  zstd            inherited from bf

$ df /usr/ports/    # ashift 12
Filesystem     1K-blocks   Used    Avail Capacity  Mounted on
build/sd/ports  46526640 803828 45722812     2%    /usr/ports
$ cksum /usr/ports/UPDATING 
1189486180 194062 /usr/ports/UPDATING
$ du /usr/ports/UPDATING 
77      /usr/ports/UPDATING
$ zfs get compression build/sd/ports
build/sd/ports  compression  zstd            inherited from build

$ df /usr/ports/   # ashift 12, raidz1 3+1
Filesystem  1K-blocks    Used     Avail Capacity  Mounted on
bm/sd/ports 244099326 1129799 242969527     0%    /usr/ports
$ cksum /usr/ports/UPDATING 
1189486180 194062 /usr/ports/UPDATING
$ du /usr/ports/UPDATING 
82      /usr/ports/UPDATING
$ zfs get compression bm/sd/ports
bm/sd/ports  compression  zstd            inherited from bm
 
Sparse files not being copied "as sparse" and no compression enabled on your ZFS filesystems might be the most likely culprit. Re-duplicating hard linked files is another. rsync can solve both of these problems (the -H and -S options), but simply enabling compression on ZFS can eliminate any guess work for sparse files too. compression=zle will only "sparse-ify" files with long runs of zeros, but the default compression=on is pretty lightweight for almost all workloads and I'd highly recommend it.

ZFS defaults to compression=off because it plays it very conservative with defaults to keep compatibility with traditional file systems (eg, UFS). I'd argue that it's better to just go with compression=on.
 
Is compression off the default? Last time I did a full install, around 12.something, zpool history shows the zroot pool being created with compress=lz4.

Now that is going through the installer, so perhaps it specifically creates the zroot pool with compression on. Rereading the first post, it sounds like the destination may be a pool the OP created by hand, which may not have been created with compression on.

That would be an easy thing to check with zpool history whateverthepoolnameis.
 
mer said:
Is compression off the default? Last time I did a full install, around 12.something, zpool history shows the zroot pool being created with compress=lz4.

Now that is going through the installer, so perhaps it specifically creates the zroot pool with compression on. Rereading the first post, it sounds like the destination may be a pool the OP created by hand, which may not have been created with compression on.

That would be an easy thing to check with zpool history whateverthepoolnameis.
Click to expand...

Yes, compression=off is the default if no compression is specified. The manpage zfsprops(7) doesn't mention the default value in the description of the compression property, but it is mentioned in the description of the compressratio property, and we can see it in the sourcecode:

Code: 
$ fgrep 'ZIO_COMPRESS_DEFAULT' /usr/src/sys/contrib/openzfs/include/sys/zio.h
#define ZIO_COMPRESS_DEFAULT        ZIO_COMPRESS_OFF

It is indeed the installer who spezifies compress=lz4:

Code: 
$ fgrep 'lz4' /usr/libexec/bsdinstall/zfsboot
: ${ZFSBOOT_POOL_CREATE_OPTIONS:=-O compress=lz4 -O atime=off}
 
  • Like
Reactions: mer
As I'm still on 13.2-RELEASE I missed the change: on 14.0-RELEASE compression is on by default. ZIO_COMPRESS_DEFAULT is set to ZIO_COMPRESS_ON in /usr/src/sys/contrib/openzfs/include/sys/zio.h and the manpage zfsprops(7) now also says that on is the default value for compression. Now the compressratio section is out of date, because it still says compression is off by default. The installer still sets compress=lz4 explicitly.
 
  • Like
Reactions: mer
I resolved my issue. I purchased x4 4TB sata drives for an existing system that I have. I set the disks in Raid10 and completed moving my backups from the other disks. I did not run into any issues this time. I think I was just running into lack of storage space on my prior setup.
 
You must log in or register to reply here.
Back
Top