ZFS Something weird going on with replication since FreeBSD 13

[Dragony]

Hi.

I know the title is not very helpful, but it is what it is.... weird.... just weird....

I don't know if something bad is going to happen soon, but since I have upgraded to FreeBSD 13, my dataset are showing weird properties:

NAME                           WRITTEN
bigstorage/dragony@2021_09_03     267K
bigstorage/dragony@2021_09_04     849K
bigstorage/dragony@2021_09_05     942K
bigstorage/dragony@2021_09_06     872K
bigstorage/dragony@2021_09_07     314K
bigstorage/dragony@2021_09_08     314K
bigstorage/dragony@2021_09_09     581K
bigstorage/dragony@2021_09_10     581K
bigstorage/dragony@2021_09_11     581K
bigstorage/dragony@2021_09_12     314K
bigstorage/dragony@2021_09_13     314K
bigstorage/dragony@2021_09_14     314K
bigstorage/dragony@2021_09_15     465K
bigstorage/dragony@2021_09_16     314K
bigstorage/dragony@2021_09_17     628K
bigstorage/dragony@2021_09_18     570K
bigstorage/dragony@2021_09_19     314K
bigstorage/dragony@2021_09_20     314K
bigstorage/dragony@2021_09_21     314K
bigstorage/dragony@2021_09_22     314K
bigstorage/dragony@2021_09_23     523K
bigstorage/dragony@2021_09_24     314K
bigstorage/dragony@2021_09_25     430K
[Here I have upgraded to FreeBSD 13]
bigstorage/dragony@2021_09_26        0
bigstorage/dragony@2021_09_28        0
bigstorage/dragony@2021_09_29        0
bigstorage/dragony@2021_09_30        0
[Since then the written property falls down to 0.... atime is off, so it might indeed be that nothing happened on that storage. Still, it is a change of behaviour.]
bigstorage/dragony@2021_10_01     331M

The big thing is the last line. That day I did NOTHING than renamed two directories. No copying took place. It was just a renaming. I thought about some malware or something unexpected, which has modified my files, so I did a diff:

diff -r /bigstorage/dragony /bigstorage/dragony/.zfs/snapshot/2021_09_30

After 2 hours and 3 TB of processed data it showed that nothing was different except that two folders have been renamed.

But the mysterious things don't stop here.

I execute this command: zfs send -vLI 2021_09_30 bigstorage/dragony@2021_10_01 > test

And this is the output:

send from @2021_09_30 to bigstorage/dragony@2021_10_01 estimated size is 114M
total estimated size is 114M
TIME        SENT   SNAPSHOT bigstorage/dragony@2021_10_01
03:00:04   24.7M   bigstorage/dragony@2021_10_01
03:00:05   54.0M   bigstorage/dragony@2021_10_01
03:00:06   82.6M   bigstorage/dragony@2021_10_01
03:00:07    111M   bigstorage/dragony@2021_10_01
03:00:08    141M   bigstorage/dragony@2021_10_01
03:00:09    169M   bigstorage/dragony@2021_10_01
03:00:10    199M   bigstorage/dragony@2021_10_01
03:00:11    231M   bigstorage/dragony@2021_10_01
03:00:12    260M   bigstorage/dragony@2021_10_01
03:00:13    290M   bigstorage/dragony@2021_10_01
03:00:14    320M   bigstorage/dragony@2021_10_01
03:00:15    349M   bigstorage/dragony@2021_10_01
03:00:16    378M   bigstorage/dragony@2021_10_01
03:00:18    410M   bigstorage/dragony@2021_10_01
03:00:19    439M   bigstorage/dragony@2021_10_01
03:00:20    468M   bigstorage/dragony@2021_10_01
03:00:21    497M   bigstorage/dragony@2021_10_01
03:00:22    526M   bigstorage/dragony@2021_10_01
03:00:23    555M   bigstorage/dragony@2021_10_01
03:00:24    584M   bigstorage/dragony@2021_10_01
03:00:25    614M   bigstorage/dragony@2021_10_01
03:00:26    643M   bigstorage/dragony@2021_10_01
03:00:27    673M   bigstorage/dragony@2021_10_01

And indeed:

-rw-r--r--  1 root  wheel  725182176  2 Okt. 03:00 test

I can't help it but think WTF??? Something is seriously wrong here.... Anything else I can do to find out whats going on here?

 

[sidetone]

Could it be a harddrive going bad? Is it secondhand? Has it been used for a long time? Is it making noises?

It could be a process ( ps) that keeps trying to run. See if a process keeps showing up on dmesg. You can also check logs in /var/logs/. It's like a cron that keeps being run every minute. Or a process that can't complete. Sendmail did something like this before. Ntpd did something similar.

 

[Dragony]

It is a new server and is running smoothly. As diff gave no diffs, I don't see any point why a demon or program has to do anything with the observed behaviour? The big question is, where are the diffs? Where are the discrepancies coming from? Why does it report 330 MB written, while expecing 110 MB to transfer via zfs send and actually transferring 720 MB. There are many question marks here.....

Expected behaviour is: written 150 kb, expected transfer around 500 kb, actually transferring 500 kb.