Hey folks,
I've stumbled upon a strange issue on my server concerning a discrepancy in 'used' space between two ZFS volumes after a ZFS send/receive operation. I transferred data from basia-ssd/vm-118-disk-0 to basia-nvme/vm-118-disk-0. Initially, the source volume basia-ssd/vm-118-disk-0 showed 300GB used, but post-transfer, the destination volume basia-nvme/vm-118-disk-0 shows 475GB used. When I send it back to the source, it shows 300GB used again.
Here are the specifics:
zfs list -o space (I removed unrelated entries)
zfs get all basia-ssd/vm-118-disk-0
zfs get all basia-nvme/vm-118-disk-0
zpool get all basia-ssd
zpool get all basia-nvme
I'm puzzled by the discrepancy in the 'used' space following a send/receive operation. Has anyone faced a similar issue or have any insights on what's happening here?
Thanks for any help,
feri
I've stumbled upon a strange issue on my server concerning a discrepancy in 'used' space between two ZFS volumes after a ZFS send/receive operation. I transferred data from basia-ssd/vm-118-disk-0 to basia-nvme/vm-118-disk-0. Initially, the source volume basia-ssd/vm-118-disk-0 showed 300GB used, but post-transfer, the destination volume basia-nvme/vm-118-disk-0 shows 475GB used. When I send it back to the source, it shows 300GB used again.
Here are the specifics:
- Server: Both pools are on the same server.
- Pool Topology:
- basia-ssd comprises 2 mirrored vdevs.
- basia-nvme consists of 5 disks in RAID-Z.
- Both pools have an ashift value of 12.
- Volume Properties: Both volumes have a volsize of 512G, volblocksize of 4K, and compression is off. Although there are snapshots created for the send/receive operations, they show 0B used, indicating they do not hold any additional data. Here are the results of zfs list -t snapshot -r basia-nvme/vm-118-disk-0 basia-ssd/vm-118-disk-0:
Code:
NAME USED AVAIL REFER MOUNTPOINT
basia-nvme/vm-118-disk-0@snapshot 0B - 475G -
basia-ssd/vm-118-disk-0@snapshot 0B - 300G -- ZFS Send/Receive Command:
Code:
zfs send basia-ssd/vm-118-disk-0@snapshot | pv | zfs receive basia-nvme/vm-118-disk-0- Volume Usage: zfs list -o space shows different 'used' space on the two volumes, as well as zfs get all.
zfs list -o space (I removed unrelated entries)
Code:
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
basia-nvme 6.66T 475G 0B 153K 0B 475G
basia-nvme/vm-118-disk-0 6.66T 475G 0B 475G 0B 0B
basia-ssd 457G 3.07T 0B 96K 0B 3.07T
basia-ssd/vm-118-disk-0 457G 300G 0B 300G 0B 0Bzfs get all basia-ssd/vm-118-disk-0
Code:
NAME PROPERTY VALUE SOURCE
basia-ssd/vm-118-disk-0 type volume -
basia-ssd/vm-118-disk-0 creation Tue Oct 17 2:27 2023 -
basia-ssd/vm-118-disk-0 used 300G -
basia-ssd/vm-118-disk-0 available 457G -
basia-ssd/vm-118-disk-0 referenced 300G -
basia-ssd/vm-118-disk-0 compressratio 1.00x -
basia-ssd/vm-118-disk-0 reservation none default
basia-ssd/vm-118-disk-0 volsize 512G local
basia-ssd/vm-118-disk-0 volblocksize 4K -
basia-ssd/vm-118-disk-0 checksum fletcher4 inherited from basia-ssd
basia-ssd/vm-118-disk-0 compression off inherited from basia-ssd
basia-ssd/vm-118-disk-0 readonly off default
basia-ssd/vm-118-disk-0 createtxg 7438363 -
basia-ssd/vm-118-disk-0 copies 1 inherited from basia-ssd
basia-ssd/vm-118-disk-0 refreservation none default
basia-ssd/vm-118-disk-0 guid 16907164831660435540 -
basia-ssd/vm-118-disk-0 primarycache all default
basia-ssd/vm-118-disk-0 secondarycache all default
basia-ssd/vm-118-disk-0 usedbysnapshots 0B -
basia-ssd/vm-118-disk-0 usedbydataset 300G -
basia-ssd/vm-118-disk-0 usedbychildren 0B -
basia-ssd/vm-118-disk-0 usedbyrefreservation 0B -
basia-ssd/vm-118-disk-0 logbias latency default
basia-ssd/vm-118-disk-0 objsetid 70716 -
basia-ssd/vm-118-disk-0 dedup off default
basia-ssd/vm-118-disk-0 mlslabel none default
basia-ssd/vm-118-disk-0 sync disabled inherited from basia-ssd
basia-ssd/vm-118-disk-0 refcompressratio 1.00x -
basia-ssd/vm-118-disk-0 written 0 -
basia-ssd/vm-118-disk-0 logicalused 296G -
basia-ssd/vm-118-disk-0 logicalreferenced 296G -
basia-ssd/vm-118-disk-0 volmode default default
basia-ssd/vm-118-disk-0 snapshot_limit none default
basia-ssd/vm-118-disk-0 snapshot_count none default
basia-ssd/vm-118-disk-0 snapdev hidden default
basia-ssd/vm-118-disk-0 context none default
basia-ssd/vm-118-disk-0 fscontext none default
basia-ssd/vm-118-disk-0 defcontext none default
basia-ssd/vm-118-disk-0 rootcontext none default
basia-ssd/vm-118-disk-0 redundant_metadata all default
basia-ssd/vm-118-disk-0 encryption off default
basia-ssd/vm-118-disk-0 keylocation none default
basia-ssd/vm-118-disk-0 keyformat none default
basia-ssd/vm-118-disk-0 pbkdf2iters 0 default
basia-ssd/vm-118-disk-0 org.freebsd.ioc:active yes inherited from basia-ssdzfs get all basia-nvme/vm-118-disk-0
Code:
NAME PROPERTY VALUE SOURCE
basia-nvme/vm-118-disk-0 type volume -
basia-nvme/vm-118-disk-0 creation Wed Oct 18 8:38 2023 -
basia-nvme/vm-118-disk-0 used 475G -
basia-nvme/vm-118-disk-0 available 6.66T -
basia-nvme/vm-118-disk-0 referenced 475G -
basia-nvme/vm-118-disk-0 compressratio 1.00x -
basia-nvme/vm-118-disk-0 reservation none default
basia-nvme/vm-118-disk-0 volsize 512G local
basia-nvme/vm-118-disk-0 volblocksize 4K -
basia-nvme/vm-118-disk-0 checksum fletcher4 inherited from basia-nvme
basia-nvme/vm-118-disk-0 compression off inherited from basia-nvme
basia-nvme/vm-118-disk-0 readonly off default
basia-nvme/vm-118-disk-0 createtxg 775 -
basia-nvme/vm-118-disk-0 copies 1 inherited from basia-nvme
basia-nvme/vm-118-disk-0 refreservation none default
basia-nvme/vm-118-disk-0 guid 3711294694037928121 -
basia-nvme/vm-118-disk-0 primarycache all default
basia-nvme/vm-118-disk-0 secondarycache all default
basia-nvme/vm-118-disk-0 usedbysnapshots 0B -
basia-nvme/vm-118-disk-0 usedbydataset 475G -
basia-nvme/vm-118-disk-0 usedbychildren 0B -
basia-nvme/vm-118-disk-0 usedbyrefreservation 0B -
basia-nvme/vm-118-disk-0 logbias latency default
basia-nvme/vm-118-disk-0 objsetid 661 -
basia-nvme/vm-118-disk-0 dedup off default
basia-nvme/vm-118-disk-0 mlslabel none default
basia-nvme/vm-118-disk-0 sync disabled inherited from basia-nvme
basia-nvme/vm-118-disk-0 refcompressratio 1.00x -
basia-nvme/vm-118-disk-0 written 0 -
basia-nvme/vm-118-disk-0 logicalused 296G -
basia-nvme/vm-118-disk-0 logicalreferenced 296G -
basia-nvme/vm-118-disk-0 volmode default default
basia-nvme/vm-118-disk-0 snapshot_limit none default
basia-nvme/vm-118-disk-0 snapshot_count none default
basia-nvme/vm-118-disk-0 snapdev hidden default
basia-nvme/vm-118-disk-0 context none default
basia-nvme/vm-118-disk-0 fscontext none default
basia-nvme/vm-118-disk-0 defcontext none default
basia-nvme/vm-118-disk-0 rootcontext none default
basia-nvme/vm-118-disk-0 redundant_metadata all default
basia-nvme/vm-118-disk-0 encryption off default
basia-nvme/vm-118-disk-0 keylocation none default
basia-nvme/vm-118-disk-0 keyformat none default
basia-nvme/vm-118-disk-0 pbkdf2iters 0 defaultzpool get all basia-ssd
Code:
NAME PROPERTY VALUE SOURCE
basia-ssd size 3.62T -
basia-ssd capacity 84% -
basia-ssd altroot /mnt local
basia-ssd health ONLINE -
basia-ssd guid 14698881280998055258 -
basia-ssd version - default
basia-ssd bootfs - default
basia-ssd delegation on default
basia-ssd autoreplace off default
basia-ssd cachefile /data/zfs/zpool.cache local
basia-ssd failmode continue local
basia-ssd listsnapshots off default
basia-ssd autoexpand on local
basia-ssd dedupratio 1.00x -
basia-ssd free 572G -
basia-ssd allocated 3.07T -
basia-ssd readonly off -
basia-ssd ashift 12 local
basia-ssd comment - default
basia-ssd expandsize - -
basia-ssd freeing 0 -
basia-ssd fragmentation 88% -
basia-ssd leaked 0 -
basia-ssd multihost off default
basia-ssd checkpoint - -
basia-ssd load_guid 8069873666185711073 -
basia-ssd autotrim off default
basia-ssd compatibility off default
basia-ssd feature@async_destroy enabled local
basia-ssd feature@empty_bpobj active local
basia-ssd feature@lz4_compress active local
basia-ssd feature@multi_vdev_crash_dump enabled local
basia-ssd feature@spacemap_histogram active local
basia-ssd feature@enabled_txg active local
basia-ssd feature@hole_birth active local
basia-ssd feature@extensible_dataset active local
basia-ssd feature@embedded_data active local
basia-ssd feature@bookmarks enabled local
basia-ssd feature@filesystem_limits enabled local
basia-ssd feature@large_blocks enabled local
basia-ssd feature@large_dnode enabled local
basia-ssd feature@sha512 enabled local
basia-ssd feature@skein enabled local
basia-ssd feature@userobj_accounting active local
basia-ssd feature@encryption enabled local
basia-ssd feature@project_quota active local
basia-ssd feature@device_removal enabled local
basia-ssd feature@obsolete_counts enabled local
basia-ssd feature@zpool_checkpoint enabled local
basia-ssd feature@spacemap_v2 active local
basia-ssd feature@allocation_classes enabled local
basia-ssd feature@resilver_defer enabled local
basia-ssd feature@bookmark_v2 enabled local
basia-ssd feature@redaction_bookmarks enabled local
basia-ssd feature@redacted_datasets enabled local
basia-ssd feature@bookmark_written enabled local
basia-ssd feature@log_spacemap active local
basia-ssd feature@livelist enabled local
basia-ssd feature@device_rebuild enabled local
basia-ssd feature@zstd_compress enabled local
basia-ssd feature@draid enabled localzpool get all basia-nvme
Code:
NAME PROPERTY VALUE SOURCE
basia-nvme size 9.08T -
basia-nvme capacity 6% -
basia-nvme altroot /mnt local
basia-nvme health ONLINE -
basia-nvme guid 6895560553673080764 -
basia-nvme version - default
basia-nvme bootfs - default
basia-nvme delegation on default
basia-nvme autoreplace off default
basia-nvme cachefile /data/zfs/zpool.cache local
basia-nvme failmode continue local
basia-nvme listsnapshots off default
basia-nvme autoexpand on local
basia-nvme dedupratio 1.00x -
basia-nvme free 8.50T -
basia-nvme allocated 595G -
basia-nvme readonly off -
basia-nvme ashift 12 local
basia-nvme comment - default
basia-nvme expandsize - -
basia-nvme freeing 0 -
basia-nvme fragmentation 0% -
basia-nvme leaked 0 -
basia-nvme multihost off default
basia-nvme checkpoint - -
basia-nvme load_guid 18047062458252002911 -
basia-nvme autotrim off default
basia-nvme compatibility off default
basia-nvme feature@async_destroy enabled local
basia-nvme feature@empty_bpobj active local
basia-nvme feature@lz4_compress active local
basia-nvme feature@multi_vdev_crash_dump enabled local
basia-nvme feature@spacemap_histogram active local
basia-nvme feature@enabled_txg active local
basia-nvme feature@hole_birth active local
basia-nvme feature@extensible_dataset active local
basia-nvme feature@embedded_data active local
basia-nvme feature@bookmarks enabled local
basia-nvme feature@filesystem_limits enabled local
basia-nvme feature@large_blocks enabled local
basia-nvme feature@large_dnode enabled local
basia-nvme feature@sha512 enabled local
basia-nvme feature@skein enabled local
basia-nvme feature@userobj_accounting active local
basia-nvme feature@encryption enabled local
basia-nvme feature@project_quota active local
basia-nvme feature@device_removal enabled local
basia-nvme feature@obsolete_counts enabled local
basia-nvme feature@zpool_checkpoint enabled local
basia-nvme feature@spacemap_v2 active local
basia-nvme feature@allocation_classes enabled local
basia-nvme feature@resilver_defer enabled local
basia-nvme feature@bookmark_v2 enabled local
basia-nvme feature@redaction_bookmarks enabled local
basia-nvme feature@redacted_datasets enabled local
basia-nvme feature@bookmark_written enabled local
basia-nvme feature@log_spacemap active local
basia-nvme feature@livelist enabled local
basia-nvme feature@device_rebuild enabled local
basia-nvme feature@zstd_compress enabled local
basia-nvme feature@draid enabled localI'm puzzled by the discrepancy in the 'used' space following a send/receive operation. Has anyone faced a similar issue or have any insights on what's happening here?
Thanks for any help,
feri