replicate - Initial and continous ZFS filesystems replication

ZFS back-ups

Hi Sebulon,

I am new to the Z/os world and currently in the environment we have a lot of different ZFS's which are owned by different teams.

I am trying to work on a strategy (script) where I can take twice weekly back-ups of particular ZFS's (filter it based on the High level qualifier) on tape.

Can you provide some inputs regarding the script?

Regards
 
Have you considered hosting the script on Github? That way you could use Git to take care of versioning and other people could contribute or fork the code for their own purposes.
 
Amitra3 said:
Hi Sebulon,

I am new to the Z/os world and currently in the environment we have a lot of different ZFS's which are owned by different teams.
Hi @Amitra3, welcome to UNIX, we have daemons...and cookies:)

Amitra3 said:
I am trying to work on a strategy (script) where I can take twice weekly back-ups of particular ZFS's (filter it based on the High level qualifier) on tape.
"On tape", what do you mean by that, I don't follow. Are you working on your own script or are you trying to figure out if replicate would be a candidate from this job?

Amitra3 said:
Can you provide some inputs regarding the script?

Regards
Of course, I´ll help in any way I can.

/Sebulon
 
Last edited by a moderator:
cbunn said:
Have you considered hosting the script on Github? That way you could use Git to take care of versioning and other people could contribute or fork the code for their own purposes.

Sounds like a great idea, I just don't know how:)

/Sebulon
 
Sebulon said:
Sounds like a great idea, I just don't know how:)

/Sebulon

Ah, well, I find Git to be less than intuitive, but there is help out there. Git for the lazy is a good place to start. Then using Github is just a matter of setting up an account and using your Github URL with your local instance of Git. There are other repositories out there, but Github is probably the most popular and hosting public projects is free.

I'm no expert, but if there's anything I can do to help out, just let me know.
 
Re: replicate - Initial and continous ZFS filesystems replic

Hey all!

Two updates:
* Being able to choose SSH port per job file, in case the receiver is running on a non-standard SSH port.
* Sending with SSH compression activated, for times when bandwidth is very constraint. Makes a huge difference on big transfers over slow lines.

/Sebulon
 
Re: replicate - Initial and continous ZFS filesystems replic

Hello guys,
I'm using this script for a year but lately i have some troubles with it. I'll be glad to check am I the only one with such an issue. On a remote host I did daily/weekly/monthly/yearly snapshots to have a very good point in time recovery options. But after the script finished on a source host all snapshots on the remote host are deleted and all history data is gone.

I'm still debugging the issue but with no success. All comments and ideas will be highly appreciated.

Please help.

Regards: Ivan
 
Re: replicate - Initial and continous ZFS filesystems replic

I found the reason for this behavior. Switch -F used in zfs recv is doing exactly this. Maybe the switch appeared in the new version of the replicate script.

-F Force a rollback of the file system to the most recent snap-
shot before performing the receive operation. If receiving an
incremental replication stream (for example, one generated by
"zfs send -R -Fi -iI"), destroy snapshots and file systems
that do not exist on the sending side
.

Regards: Ivan
 
Re: replicate - Initial and continous ZFS filesystems replic

Hey all!

Never got any feedback from @zag about patch for it, so I´ve made my own:

http://pastebin.com/Xnr29FVz

Untested as of yet, so not sure whether I nailed it on the first shot, but fairly confident. Uploaded new version, updated in top post.

/Sebulon
 
Last edited by a moderator:
Re: replicate - Initial/continous ZFS filesystems replicatio

Was a little nervous about making a change on a friday, bad things usually happens :) But I´ve had a chance to test the changes now and they seem to work, no negative consequences so far:

receiving_system# zfs list -H -r -t snapshot -o name pool2/<sending_system>
Code:
pool2/<sending_system>/root@remote_replicate.base
pool2/<sending_system>/root/usr@remote_replicate.base
pool2/<sending_system>/root/usr/home@remote_replicate.base
pool2/<sending_system>/root/usr/local@remote_replicate.base
pool2/<sending_system>/root/var@remote_replicate.base

receiving_system# zfs snapshot -r pool2/<sending_system>@now
receiving_system# zfs list -H -r -t snapshot -o name pool2/<sending_system>
Code:
pool2/<sending_system>/root@remote_replicate.base
pool2/<sending_system>/root@now
pool2/<sending_system>/root/usr@remote_replicate.base
pool2/<sending_system>/root/usr@now
pool2/<sending_system>/root/usr/home@remote_replicate.base
pool2/<sending_system>/root/usr/home@now
pool2/<sending_system>/root/usr/local@remote_replicate.base
pool2/<sending_system>/root/usr/local@now
pool2/<sending_system>/root/var@remote_replicate.base
pool2/<sending_system>/root/var@now

sending_system# replicate

receiving_system# zfs list -H -r -t snapshot -o name pool2/<sending_system>
Code:
pool2/<sending_system>/root@now
pool2/<sending_system>/root@remote_replicate.base
pool2/<sending_system>/root/usr@now
pool2/<sending_system>/root/usr@remote_replicate.base
pool2/<sending_system>/root/usr/home@now
pool2/<sending_system>/root/usr/home@remote_replicate.base
pool2/<sending_system>/root/usr/local@now
pool2/<sending_system>/root/usr/local@remote_replicate.base
pool2/<sending_system>/root/var@now
pool2/<sending_system>/root/var@remote_replicate.base

So I´m sticking with it. It is now completely possible to have different snapshot scheduling on the receiving system to keep versions further back in time than what you have on the sending system. Thanks to @zag for making this happen!

/Sebulon
 
Last edited by a moderator:
Hello, I was wondering if the script is in any way connected to this one sysutils/zfs-replicate?

Hi priyadarshan,

so sorry for the late reply, I rarely log in and have thought that I would get emails about stuff like this. Apparently not.

No, there is no connection between this project and that port, even though the names are very similar. From what I remember, it seems to be something taken from Solaris a long time ago and not really adapted well to FreeBSD.

/Sebulon
 
Thank you, I will try out your script, very interesting.

Yeah, about that, I noticed that the sendspace link (for the archive) was broken yet again, so I´ve put the archive on a permanent server. If you tried to download it before and failed, please try again.

/Sebulon
 
This hasn't been uploaded to github yet, has it?

I'm looking for a script that eases the backup of my ZFS root pool to an external machine. I haven't had a look at the code yet but from what I read this looks interesting.

I'm not a git geek but I'd know how we could get your script on github. Do you need help with that? Once that is done, people would be able to contribute and we can further think about creating a port.
 
This hasn't been uploaded to github yet, has it?

I'm looking for a script that eases the backup of my ZFS root pool to an external machine. I haven't had a look at the code yet but from what I read this looks interesting.

I'm not a git geek but I'd know how we could get your script on github. Do you need help with that? Once that is done, people would be able to contribute and we can further think about creating a port.

niklaas
I'm a horrible person that have waited so long to respond this, sorry!

OK, so from the posts here I've gathered that there are two desires that keeps popping up from time to time.

First is the wish to have the innards of the script displayed in public up on Github. It's just that I'm too self conscious about it, it looks rather awful to tell you the truth. If I would be writing it today it would look drastically different. Don't get me wrong, I've always had the intention of putting it up somewhere "when I was done"... It's just that I never seem to get there, in my mind there's always more to do before it´s ready:)

There other thing is to make a port of this. This I could probably do quicker, except for the fact that I have job-related stuff that I prioritize higher.

But in all of this, remember that I've never prevented anyone from just taking it and do whatever you want with it. If anyone wishes to get started without me, by all means, do it. And I can choose to jump in later and help, or do my thing in my own time.

So sit tight and wait or start getting busy; your choice:)

/Sebulon
 
Last edited by a moderator:
Back
Top