ZFS Help moving from manual to automatic backup

[Deleted member 43773]

fcorbelli,​

you're missing the point.

1. I do not do full backups daily with 100G or more.
That's I ment with

you'll run into a situation when your system is doing too much backupping (cost you cpu-time and bandwith),
or even may run into some point, when your system wants to start the new backup, but the former one is not finsihed yet

I may be not the best scriptwriter, but I'm not a complete moron neither.

2. You're efforts to promote zapfranz are acknowledged and admitted to knew about another bu-tool.
But you also have to respect the fact that some people may look for another solution - for whatever reasons.
One of those reasons is, that neither me nor as I understand mefizto wants to have the backuped data being in some kind of databank or special file.
And if I understand your post right zpaqfranz backups to *.zpaq
So you need zpaqfranz to get to your data.
That's exactly what we do not want.
(And I do not want to discuss this point [again], besides we're running heavily off-topic, already)

3.

It is a typical "ancient-time" archiving strategy

Maybe so. But it does the job for me the way I wants it.
tar maybe ancient, some may say mature, thus sophisticated.
It does a good solid and reliable job.
And tar still will be a standard tool available on any (unixoid) system when nobody knew about zapfranz anymore.

I do not tar 100G/d, it's more about 14G, which is done within 20 minutes, which I can live with, because I do not backup

All, all, and all

e.g. I do not backup my ~.wine or /download, or the whole system daily, including source tree, /var, /temp etc... because this is replaced quickly from reliable sources, as already someone mentioned here, and you may run into inconsistency troubles, when falling back is mixed up with an updated system.
I pick what I backup when, where, how and where to.

However,
I never said my script is exemplary.
But I say sometimes there are needs to design the individual solution fitting your needs best yourself.
That's why FreeBSD, because of the Unix-Philosophie, which contains the idea of having modules being assembled (redirected, piped, scripted...)
I do not say "don't use backup tools like backula or zpaqfranz". For sure those are great tools.
All I say is: They don't suit my needs.
And if I understood mefizto right neither his.
So he's looking for another backup solution, based on some kind of sh-script or whatever.

All I was trying to say here was:
By principle there are 3 possible bu-solutions:
1. Use some kind of a given script - but nearly all of those are wrote for one very special purpose only (such as mine) - they will not suit.
2. Use some kind of backup tool - but those try to respect any bu-problem possible. Every one-size-fits-all jack-of-all-trades always contain downsizes, so you compromise.
3. Create your own solution.

Since I am a big fan of Unix-Philosophie I believe anybody working seriously on unix-machines shall be capable to create own solutions - or at least being aware of the possibiltiy.
That's why the basic nature of this system is a powerful shell (several shells) with ([very] powerful) scripting capabilties, coming along with many small, maybe "ancient" but sophisticated, powerful and flexible tools, capable of being interconnected doing very powerful things together, and for if that's not sufficient, being a complete software development platform (editor + compiler) even in the barest installation.

Even if my script stinks,
it does the job for me - I'am satisfied, because I get what I want, not more, not less.
I wasn't promoting it.
Mefizto asked for ideas, that's all.

For me additionally to give me the satisfying solution I didn't found
it gaves me an opportunity to get a bit more deeper into shell-scripting.
One have to start somewhere.
Then develop it further.
And one needs to actually do something useful, if you want to really learn something.

Now I am really out of this topic here.
I have already written much more than useful, and I cannot give mefizto any additional useful help on his challenge anymore.

So, peace out.

 

[mefizto]

Hi Profighost,

thank you for the reply.

But if you promise yourself to get the one or the other idea from it,
or at least being encouraged to start get into scripting yourself,
I'll give it a review and translate comments/explanations from german to english.

I believe that I mentioned that I had written scripts for my use, so I am not too worried about it. This is not to say that I am an expert, but the scripts I had written did what I wanted.

The reason I am reviewing the scripts, and asked for yours, is to understand the overall design issues and - specifically - the corner cases. (This is not saying that I am above shamelessly copying .)

...here you go - but don't expect fancy hacker stuff
It's more ment to encourage that scritping is not that hard.
I for myself like to start with small, simple example pieces than being overwhelmed by the complete and perfect profi source.

First, thank you for providing the script, what a bummer that we are apart time-wise, I could have save you the effort of translation because while Mein Deutsch ist eingerostet, es hätte gereicht (sein?).

Second, yes that is how I approach it. I split the task into the smallest chunks I can think of and then write a function for each of the chunks and test it.

Hi tanis,

thank you very much for the time you put into your reply. I will definitely look at the zxfer(8), if only for the inspiration. And, who knows, maybe it will even fit my needs.

Regarding the zfs-auto-snapshot, I will likely stay away as I have some strange, and perhaps, unnecessary requirements. For example, I would like to have an arbitrary name for the snapshot. The reason is, that from time to time I rearrange my folder structure, and then I panic when I cannot find a file/folder at its customary location. So, if I could name the snapshot pool/dataset@snapshot_date_rearranged, I would know what happened at that point of time.

Hi fcorbelli,

your promotion of zapqfranz is acknowledged, but your insistence that it is a tool that will fit my needs is rather off-putting. Cf. Profighost's reply.

Saying that, I am glad that you mentioned it, if only for the sake of people monitoring the thread that another option is available. As such, I do not mind your further posting, but please be aware that people might not find your posts based on the title of the thread and the first few posts.

Kindest regards,

M

 

[Deleted member 67440]

Sorry but I think there is a misunderstanding

I'm not the author of zpaq
I already wrote it above
I have nothing to gain, it is opensource software
This program (zpaq) has already been present in FreeBSD packages for years and years
Anyone can install or compile it in seconds (beware the port is called paq, not zpaq)

So you can do everything zpaqfranz does with zpaq, if you feel that for some reason it is better to use that software

Why do I suggest using zpaqfranz?
Because zpaq does not "understand" the .zfs folders

But it doesn't matter, everyone can use the old version from 2014

I'm just trying, with some difficulty, to point out that there are extremely more advanced tools than those proposed (tar etc) already in the port tree!

Only, you have to know them, like "hidden gems"
That's all

If anyone use, or don't use, my program, it doesn't change anything
Noone have to pay me anything
I will not get richer or more famous
I don't think I asked for anyone's credit card number (just kidding)

Since FreeBSD (in particular zfs) has some peculiarities that the initial developer did not know, and that he has abandoned development, I have implemented, gradually, the changes to make it suitable for FreeBSD (actually zfs)

Just one question: have you tried, or not, zpaq?
Just one time?

PS I think it is always advisable to specify that I am the author of the program, otherwise it would seem that I want to advertise it for some reason. But it does not matter. Use zpaq (don't throw it on a snapshot folder, aka something with a .zfs inside) and you'll see for yourself
If, on the other hand, you believe you have a perfect knowledge of all the mechanisms suitable for backups, then I don't know why the thread is opened

 

[Deleted member 67440]

you're missing the point.

1. I do not do full backups daily with 100G or more.
That's I ment with

It is just an example, to make calculation easy
It really doesn't matter.
I thought it was clear

I may be not the best scriptwriter, but I'm not a complete moron neither.

2. You're efforts to promote zapfranz are acknowledged and admitted to knew about another bu-tool.

I don't know what a bu-tool is.
If you want to see how zpaq is done and how it works, just do it
pkg install paq

But you also have to respect the fact that some people may look for another solution - for whatever reasons.

Of course I respect that.
But if people know the other solutions.
Knowledge is the key
You know 10 methods, choose the one you like best
Do you know 1 method (just another example)? Perhaps there are other better ones

One of those reasons is, that neither me nor as I understand mefizto wants to have the backuped data being in some kind of databank or special file.

It is not a databank
It is an archive, just like a .tar, or .7z

And if I understand your post right zpaqfranz backups to *.zpaq
So you need zpaqfranz to get to your data.
That's exactly what we do not want.

Ahem, no
You need zpaq to get back your data, just like rar or tar
NOT ZPAQFRANZ
And, BTW, you CAN use zpaqfranz to make rsync-like (better, robocopy-like) backups
Yes, with command r (robocopy)
Did I mention that it does pretty much everything I've been asked to do in years of perfecting? (OK, I'm joking)

Maybe so. But it does the job for me the way I wants it.
tar maybe ancient, some may say mature, thus sophisticated.
It does a good solid and reliable job.
And tar still will be a standard tool available on any (unixoid) system when nobody knew about zapfranz anymore.

What part, exactly, of zpaq is ALREADY PART OF THE PORT TREE is not clear?
What part of (for zpaqfranz)
zpaq fork that stores by default CRC32+XXHASH64 (SHA-1|SHA-2|SHA-3|MD5|XXH3|BLAKE3)
with full backward zpaq 7.15 compatibility is not clear?
When tar will be garbage, any (FreeBSD) user can extract .zpaq archives
As anyone will continue to extract .zip, or .7z
Because, without those software, you cannot get back your precious data from tar.gz (gzip, you know)
As an anecdote: on very old Linux systems tar is unable to extract modern files, because they are compressed with algorithms that did not exist (at release time).
In that case you need to update tar first. I stop here

I do not tar 100G/d, it's more about 14G, which is done within 20 minutes, which I can live with, because I do not backup

OK, I do 14GB backups in about 4 seconds with zpaq/zpaqfranz
Yes, 4 seconds
Without ever pruning data
Never
For 10 years, for example

e.g. I do not backup my ~.wine or /download, or the whole system daily, including source tree, /var, /temp etc...

You can copy whatever you like
On Windows zpaqfranz can backup (almost) a full C: drive in (just about) 2 minutes
But this is Windows

All I say is: They don't suit my needs.

Just one question: have you try zpaq just one time, or not?
How, exactly, are you sure your method fits your needs if you don't know any others?
It's like, to clarify, if you don't know about electric screwdrivers, and you keep using manual ones
If someone tells you "hey, try an electric screwdriver, you'll see how comfortable it is!" AND THIS DOESN'T COST YOU ANYTHING, maybe you could try
At least, I would

So, peace out.

Always
But I must confess that electric screwdrivers exist, and they are much more comfortable than manual ones

 

[Deleted member 67440]

Just to be clear
You use the "evil" zpaqfranz (creating thetest.zpaq archive, just like rar, or tar, or 7z)

root@bsd131:/tmp # zpaqfranz a thetest.zpaq /root
zpaqfranz-bsd v55.9-experimental archiver,  compiled Aug  9 2022
Creating thetest.zpaq at offset 0 + 0
Adding 32.720 (31.95 KB) in 10 files (3 dirs), 4 threads @ 2022-08-11 19:42:29
13 +added, 0 -removed.

0 + (32.720 -> 32.720 -> 13.164) = 13.164 @ 652.10 KB/s

0.049 seconds (00:00:00)  (all OK)

BUT EXTRACT WITH ZPAQ (if you really do not want to use evilzpaq)

root@bsd131:/tmp # zpaq x thetest.zpaq -to ./extracted
zpaq v6.57 journaling archiver, compiled Aug  5 2022
thetest.zpaq: 1 versions, 13 files, 10 fragments, 0.013164 MB
Extracting 0.032720 MB in 10 files with 4 jobs
0:00:00 to go: 0.032720 MB (100.00%)
Extracted 10 of 10 files OK (0 errors) using 1.116 MB x 4 threads
0.007 seconds (all OK)
root@bsd131:/tmp #

zpaq 6.57 is in the port from 2014

root@bsd131:/usr/ports/archivers/paq # ls -l
total 19
-rw-r--r--  1 root  wheel  3983 Jul 20 16:20 Makefile
-rw-r--r--  1 root  wheel  2954 Dec  5  2014 distinfo
drwxr-xr-x  2 root  wheel     6 Aug  5 14:46 files
-rw-r--r--  1 root  wheel   979 Mar 28  2013 pkg-descr
-rw-r--r--  1 root  wheel   141 Aug 18  2014 pkg-plist
root@bsd131:/usr/ports/archivers/paq #

 

[Deleted member 43773]

.

 

[Sivan!]

I have now finished the arduous task of unifying files from a plurality of computers onto a central server and have made a backup to a backup server.

How?

 

[mefizto]

Hi Sivan!,

are you asking how I unified the files or how I backed the server up?

I will be happy to detail both, but I am not sure whether it will help you, and it is rather off-topic. Perhaps via p.m.?

Kindest regards,

M