D
Deleted member 67440
Guest
For how long do you plan to retain the snapshots?
For how long do you plan to retain the snapshots?
$ ls -i zp.c
30839 zp.c
$ sudo zdb -vvv newsys/usr/home 30839
Dataset newsys/usr/home [ZPL], ID 2852, cr_txg 628, 4.78G, 66374 objects, [..snip..] cksum=e39e384ca:2610a4d6f3a1:3786e93888ff9d:3a5fb186e4833641
Step 0: know (almost all) the tools
The title of the thread is "Frequent incremental zfs backup problems"
More like a "rsync on steroid", instead of a "tar on steroid"
Storing "copies" on zfs replication, for how long?
I know very well.
How to retain the .zfs? Simply
Well no.
Not verifiable "somewhere else"
As explained this is NOT the "hard" hash of the file
/// merge two CRC32 such that result = crc32(dataB, lengthB, crc32(dataA, lengthA))
uint32_t crc32_combine(uint32_t crcA, uint32_t crcB, size_t lengthB)
{
// degenerated case
if (lengthB == 0)
return crcA;
/// CRC32 => 32 bits
const uint32_t CrcBits = 32;
uint32_t odd [CrcBits]; // odd-power-of-two zeros operator
uint32_t even[CrcBits]; // even-power-of-two zeros operator
// put operator for one zero bit in odd
odd[0] = Polynomial; // CRC-32 polynomial
for (unsigned int i = 1; i < CrcBits; i++)
odd[i] = 1 << (i - 1);
// put operator for two zero bits in even
// same as gf2_matrix_square(even, odd);
for (unsigned int i = 0; i < CrcBits; i++)
{
uint32_t vec = odd[i];
even[i] = 0;
for (int j = 0; vec != 0; j++, vec >>= 1)
if (vec & 1)
even[i] ^= odd[j];
}
// put operator for four zero bits in odd
// same as gf2_matrix_square(odd, even);
for (unsigned int i = 0; i < CrcBits; i++)
{
uint32_t vec = even[i];
odd[i] = 0;
for (int j = 0; vec != 0; j++, vec >>= 1)
if (vec & 1)
odd[i] ^= even[j];
}
// the following loop becomes much shorter if I keep swapping even and odd
uint32_t* a = even;
uint32_t* b = odd;
// apply secondLength zeros to firstCrc32
for (; lengthB > 0; lengthB >>= 1)
{
// same as gf2_matrix_square(a, b);
for (unsigned int i = 0; i < CrcBits; i++)
{
uint32_t vec = b[i];
a[i] = 0;
for (int j = 0; vec != 0; j++, vec >>= 1)
if (vec & 1)
a[i] ^= b[j];
}
// apply zeros operator for this bit
if (lengthB & 1)
{
// same as firstCrc32 = gf2_matrix_times(a, firstCrc32);
uint32_t sum = 0;
for (int i = 0; crcA != 0; i++, crcA >>= 1)
if (crcA & 1)
sum ^= a[i];
crcA = sum;
}
// switch even and odd
uint32_t* t = a; a = b; b = t;
}
// return combined crc
return crcA ^ crcB;
}Or to expand on this a bit, ZFS always writes blocks into empty storage space (on disk), only once everything is 'settled' on disk for that transaction, it updates the metadata for the corresponding file to point to the new storage for that/those blocks, updates checksums on everything all the way up to the uberblock. If the replaced/old blocks (on disk) are not referenced by any snapshots, they are freed.
In fact I do not have to
BLOCKS
If you know a quick way (aka: do not read all the file) to make zfs expose sha256 hashes of files (I stress files) please let me know
Can zfs say "sha256 of the content of file pippo.txt is 49"?
It is the "wet dream" of any system administrator
I don't want to sound rude, but maybe it's better if you study better how a file system works in general, zfs in particular, a "true" hash (sha256 for example), how it is calculated, and how it is not calculated for a file
I'm waiting too
Just kidding
Maybe I will learn something new from this thread
You do not need, until you have to
/var/run that we care about. GCP block storage allow us more storage than we'll ever need to use.Sure, but snapshots are backups, and replicated snapshots are remote backups. Right?
With the storage I have available to me, it's later - much, much later. Later enough that if I ever get to that point, it will be financially feasible to get more storage.
scp remotebackup:/zbackup/.zfs/snapshot-2022-12-02-1351/path/to/pippo.txt ..$ ssh remotebackup "mtree -c -p /zbackup/.zfs/snapshot-2022-12-02-1400/" | mtree -p /.zfs/snapshot-2022-12-02-1400/
Actually, a backup is anything that stores a copy of data "somewhere else". You could use (tape) archives for that. Or, of course, a full replication of your filesystem if you want. All that matters is storing it somewhere else (which could even be a disk normally not mounted in your live system, although some physical distance is better for obvious reasons).
: It depends on the level you're looking at it. On the file level, a local snapshot is a backup (as is storing copies of files with e.g. .bak extension). On the system level, a local snapshot is not a backup, but a snapshot replicated in a different pool is.
Same here, I run zfs and love it. Using it for boot environments and data snapshots. However I'm using also unison (rsync) to copy
No.
No, at all
snapshots eat space (and speed)
speed.
In fact, no
Fragile, complex
Well, I write zpaqfranz and the README too, so I think I can tell
Ahem.. the question is very, very simply
For a backup, of course
Ahem, no
You do not have a pool.
In fact no, too slow.
zpaqfranz does this, too
