ZFS Best method for dataset migration to new server

[dbtrade]

Hoping to leverage the varied and extensive experience of forum members for what would seem to be a somewhat common task. Yet not finding consistent suggestions in my search here or with AI help.

I have a 3 disk zfs RAIDZ1 version 28 on an old server I am replacing.
The new server has been bumped to a 4 disk RAIDZ2 configuration built with latest zfs defaults.

The question is the most efficient and safe method to transfer the data from the old to the new while preserving the original folder structure.

The servers are on the same local network, there's also an external single disk backup pool that could be the source, but for some reason that makes me nervous.

I tried to start a zfs send/receive from the new server side but was not clear on how to enable root on the remote machine in a safe manner for the transfer.

Thoughts?

 

[richardtoohey2]

Not technical thoughts, but from experience - assume it's going to go wrong at least once, and you'll be learning each time. If you are aiming for the perfect move first time, that's not likely to happen (but might).

Another option is to set up a smaller test system to practice on, but appreciate with the number of drives and configuration that might be time-consuming & expensive.

How much data are we talking about?

 

[gpw928]

You don't mention if the root is separate from the "tank". Life is much easier if it is built fresh, and separate from the tank.

The best way to get root access on the receiving end it to modify the ssh setup to allow root login, so add the following temporarily to /etc/ssh/sshd_config on the receiving host and restart sshd(8):

PermitRootLogin yes

This opens up attack vectors, so beware of the risks if security matters.

You then create ~root/authorized_keys ~root/.ssh/authorized_keys on the receiving host to contain the public ssh key for the root account of the sending host.

Assuming that your source and target pools are both called "tank" this should be close to what you want to do, as root, on the source host:

TAB=$(echo | tr '\n' '\t')
zfs snapshot -r tank@replica
size=$(zfs send -nP -R tank@replica | grep "^size" | sed -e 's/size[ $TAB]*//')
zfs send -R tank@replica | pv -s $size -ptebarT | ssh root@target zfs receive -Fdu tank

 

[Deleted member 72508]

There are several ways.
As already was asked:

How much data are we talking about?

The method strongly depends on that,
how many slots for drives are available, or of the bandwith of your LAN (I so learned to check my cables; there is no use of having fast hardware, when the cables are CAT-3 ?)
...and how much time you have.

I moved my first NAS' data to the second NAS by simply copying the files via LAN - most comfortable, 'cause less work, and no tinkering with hardware, but ? It was not quite 1T of data, but it took >24...36h or so (can't remember anymore - it took a while.)

When I moved from the second to the 3rd I already had >3T of data.
Because nothing is faster to transfer data as between drives within a computer
I attached an additional large HDD to the free remaining SATA port of the old one (simply a single large freebsd-ufs partition), copied the data to it, attached it to a free SATA port on the new machine, and copied the data again into the new ZFS pool.
Doing this with e.g. a 3T HDD two times, you may move 6T within two turns.
It may be a bit 'shutdown, tinker, boot,...shutdown, fumble, boot...' but it's unlike faster than via LAN.

Depending on the amount and kind of data you may benefit a lot of using compression anyway.

Another idea was to attach the whole pool into the new machine, and copy the data internally.
If not have enough free ports you may add an extra extension card (PCI) into your new server to get additional SATA ports

safe method to transfer

As long as you test the copy was made correctly and don't write anything to your old pool unless you ensured the copy was correct, any method is safe.

If you don't already know by now I recommend rsync instead of cp for doing such copies.
It's pretty reliable about producing exact copys of whole filesystems, and it comes with a lot features such as using compression.

P.S.:
To avoid copying the whole shebang the next time I designed my latest NAS to have independent pools.
The system is installed on 2 extra drives as mirrors each for boot, swap, and / (a single drive may be fully sufficient for that; but I'm a bit paranoid about hardware failures.)
Since the machine is headless, it has no GUI, and not much at all installed, a small drive is fully sufficient for that. 8GB swap + <8GB used for system is what my NAS uses;
so the 500G pool for that is almost empty - enough reserve for /var/log ?.
But I have the system physically separated from my data.

All the data is separate within a raidz2-pool of five disks with one large partition each.

The idea is to grow the pool by adding more disks, exchange them by larger ones - using the benefits of ZFS, or when the day come simply physically move the whole data pool only into the new machine,
but not to copy data between pools anymore.

 

[dbtrade]

quick Interim replies:

How much data are we talking about?

About 6tb

You don't mention if the root is separate from the "tank". Life is much easier if it is built fresh, and separate from the tank.

Yes, root is separate, It's on its own pool in a mirrored configuration.

It's very late here and I will look at these replies more completely later, Thanks so much!

 

[Argentum]

quick Interim replies:

About 6tb

If you have a temporary media (a 6TB drive) available and free drive interfaces in both systems, then, IMHO, probably the safest and easiest way for this one-time operation would be to mount this drive to an old system, then zfs send to a file on this drive. After that mount this drive to a new system and zfs receive from the file created in the first step. This will also leave a backup copy of your data on that drive.

 

[dbtrade]

.........
I attached an additional large HDD to the free remaining SATA port of the old one (simply a single large freebsd-ufs partition), copied the data to it, attached it to a free SATA port on the new machine, and copied the data again into the new ZFS pool.

Yes, I have an 8tb external drive that I could attach, it possibly has an e-sata port, I know the machine I am moving to does. Perhaps this is the most straightforward method for me... I would have to closely look at the folder structure to prevent accidentally nesting things

Another idea was to attach the whole pool into the new machine, and copy the data internally.
If not have enough free ports you may add an extra extension card (PCI) into your new server to get additional SATA ports

If I have the e-sata interface, while it would be slower, it would also be much more straightforward then buying more ports and figuring out the power if I don't have the right cables.

If you don't already know by now I recommend rsync instead of cp for doing such copies.
It's pretty reliable about producing exact copys of whole filesystems, and it comes with a lot features such as using compression.

I should probably learn more about rsync. I have read but never really used it.
I understand it can do a resume if things go south. I cannot do that with the version 28 pool on the old system. I just never had occasion to use rsync.

The idea is to grow the pool by adding more disks, exchange them by larger ones - using the benefits of ZFS, or when the day come simply physically move the whole data pool only into the new machine,
but not to copy data between pools anymore.

This is how I grew the original pool over many years, replacing drives with larger ones when one started to report errors on zpool status.

But the old pool uses a 512B block size as that is what the original drives needed, but the newer ones want a 4096B native size. I don't think I can just convert that. Plus the drives are getting very old now. Probably a dozen years. Time to upgrade it all and break out these drives for long-term and off-site storage or what-not.

The 8 TB backup drive has been an incremental zfs send/receive over USB. And the first one was very long.

So my choice seems to be either learn rsync, which I think being network interfaced will make it very slow, or hook up the backup drive and restore from that.

I really appreciate all the thoughts and am thankful for the thoughtful responses.

I will keep this open until the deed is done. At least not until this weekend.

If you have a temporary media (a 6TB drive) available and free drive interfaces in both systems, then, IMHO, probably the safest and easiest way for this one-time operation would be to mount this drive to an old system, then zfs send to a file on this drive. After that mount this drive to a new system and zfs receive from the file created in the first step. This will also leave a backup copy of your data on that drive.

Yes, you are right and that is essentially what I will probably do.

....
The best way to get root access on the receiving end it to modify the ssh setup to allow root login, so add the following temporarily to /etc/ssh/sshd_config on the receiving host and restart sshd(8):

PermitRootLogin yes

This opens up attack vectors, so beware of the risks if security matters.

You then create ~root/authorized_keys ~root/.ssh/authorized_keys on the receiving host to contain the public ssh key for the root account of the sending host.

Assuming that your source and target pools are both called "tank" this should be close to what you want to do, as root, on the source host:

TAB=$(echo | tr '\n' '\t')
zfs snapshot -r tank@replica
size=$(zfs send -nP -R tank@replica | grep "^size" | sed -e 's/size[ $TAB]*//')
zfs send -R tank@replica | pv -s $size -ptebarT | ssh root@target zfs receive -Fdu tank

Thank you very much for providing the step-by-step. There are a few things in there I haven't seen before.

 

[richardtoohey2]

So my choice seems to be either learn rsync, which I think being network interfaced will make it very slow, or hook up the backup drive and restore from that.

rsync is a gem, but for a one-off copy maybe not so useful. Definitely add it to your toolkit, though!

Any first rsync will be slow, but after that it will only copy changed/new files, so a lot faster. It also picks up where it left off, so if you start a copy and it drops after 1TB (or whatever) you can resume it (because it will recognise what still needs to be copied).

6TB is going to take a while over the network; but a slow USB will also be painful.

Copying to the external drive is probably the easiest and probably fastest (well faster than a slow network.)

But rsync very simple to use - so you could try with a directory or two, get a feel for it.

Then if you are comfortable with it, let it run for everything, 6TB will take at least a day or two. As soon as it finishes, run it again and it will take a few hours to copy any new/changed files (depending on how much is changing.)

This is not so good for databases if you've got anything like MySQL in the mix (but that is most likely to be true whatever method you use.)

I don't know about the ZFS send/recv stuff, so all the above is based on rsync experience not ZFS experience!

 

[gpw928]

Moving 6TB over a Gigabit network is going to take about 14 hours at wire speed. Maybe double that if there's a lot of small files. Maybe less if there's a lot of compressible large files. Allow at least a full day. If you use ssh to move bulk data across a network, choose the fastest cipher available. This applies to both zfs-send and rsync.

A transport disk will have a faster transfer rate (roughly 5 times faster for SATA), but remember that you have to copy the data twice - once to the transport disk, and once to the final destination.

rsync is a great tool, and was certainly the best way to copy data via a network until zfs-send and zfs-receive came along. Both can do the initial bulk send while all your data remain on line, and you can then take an outage, shut down data access, and do a quick final incremental sync of only the deltas, before switching service to the new host. If you have zfs on both hosts, I would use zfs-send and zfs-receive, regardless of the transport mechanism.

Otherwise, I'm a big rsync fan. But I absolutely hate rsync for favouring speed over correctness. If you use rsync, look at the "rsync -SHAXax --delete" options carefully.

The ashift is a parameter of the pool. You can create your target pool with a 4K block size, and use what ever method you want to move the data into it. But that new pool may take up a lot more space to accommodate your original (512 byte block) data.

 

[richardtoohey2]

Maybe double that if there's a lot of small files.

A very good point, I've only ever copied a few TB, but over slower networks and that's been at least 24 hours, and millions and millions of small files have been a lot to do with the rsync performance.

 

[Deleted member 72508]

I should probably learn more about rsync. I have read but never really used it.

Yes. You will be glad if you do.

I protracted to start on rsync for too long, as I figured out when I learned it.
The vast scale of features put me off:'damnd, what a complicated thing...'
After I took the time to actually read its manpage and played a bit on some test directory I learned it's not really that hard to use, but a very valuable must-have in your power tools box.

Best start on rsync may to use the man's examples
with the predefined 'jack-of-all-trades' option -a
on a test directory,
e.g:
rsync -au --delete sourcedir destinationdir

--archive, -a archive mode is -rlptgoD (no -A,-X,-U,-N,-H)
--delete delete extraneous files from dest dirs

is a good way to get and keep a 1:1 copy of a directory.
Without --delete the files you removed in sourcedir after the last sync will stay in destdir.

You may fine tune details for special circumstances later.

I understand it can do a resume if things go south.

It simply transfers the differences of a file, only.
If there is a broken copy, it will 'continue' to do the 'rest'.
But it also means the very first time on an empty directory of course it needs to copy all data at least once,
which of course cost the time it needs.

But I cannot judge if rsync in this case was best choice - what's faster, safer.
E.g. I'm simply amazed that 6T could be transferred via LAN within ~14h.
Hard to believe for me. But I value gpw928 to be more expert than me, and my judgment may be caused from obsolete experiences on old/slow hardware

All I can advise is the old engineering wisdom:
Never try to solve new problems with new solutions.
Solve new problems with known solutions.
And test new solutions on known problems.

 

[richardtoohey2]

E.g. I'm simply amazed that 6T could be transferred via LAN within ~14h.

It was qualified with a “double that if small files” and talked of gigabit speeds.

 

[Deleted member 72508]

It was qualified with a “double that if small files” and talked of gigabit speeds.

Yeah, I know. I read it.
It still is hard for me to believe, which does not prove, not even say anything.
To my experience 6T over LAN would at least last a week - magnitudes slower.
But as I said:
I'm not having the most recent hardware at top speed in perfect configuration.
As I hinted in my first post I rooted out the one or the other CAT-5 cable (and found even one CAT-3) in my set-up a few years ago. Since the speed was fully sufficient for my daily use (several G/day, not T) I never gave it a thought before that.
So my word is no scale on this.
I am amazed.
That's what I said. Without suggesting something, but relativating my own words.

 

[freejlr]

I tried to start a zfs send/receive from the new server side but was not clear on how to enable root on the remote machine in a safe manner for the transfer.

You can create a user on the remote server where you will receive the data stream and give it the necessary permissions:

zfs allow -u receiver_user compression,mountpoint,create,mount,receive pool/backup

Also if you are going to authenticate with root use the public key method and disable the Password authentication method, At least I could have a little more security.

 

[gpw928]

To my experience 6T over LAN would at least last a week - magnitudes slower.

Such estimates always have to be made with caveats. I mentioned large files, a fast ssh cipher, and a Gigabit network -- where overheads for transmission and metadata management would be minimised. Experience tells me I'm right in a first estimate of 14 to 24 hours for that set of circumstances.

Times can blow out a lot if your network is slow or congested, your hosts under-powered or otherwise busy, the files small, the ssh cipher a CPU hog, or ...

My home networks use Gigabit switches, and have no difficulty sustaining one-way data transfers in excess of 100 MiB/sec between physical hosts. iperf(1) [now probably iperf3(1)] provides a good way to see what the raw network can sustain.

However, if you want an estimate to include all the variables, you probably just need to test.

 

[lgrant]

Time to upgrade it all and break out these drives for long-term and off-site storage or what-not.

Just an note about my experience: I have made it a practice to never reuse old drives. When I upgrade to a larger drive, I put the old one in a box, marked "Keep forever", just in case I need it for a backup someday. Occasionally I have needed files off of one of these old drives, and I find that many of them no longer work. (My favorite was the one that started smoking when I turned it on.) Now, part of the problem may be that I live in the tropics; even though we have the A/C going 24/7 in the room where the disks live, the humidity is still a lot more than it was in Chicago. Luckily, I have enough old disks that I can usually find what I want on one of them. But I no longer think that keeping old disks around is a reliable long-term backup method. (I still do it, because I haven't found an alternative. I am still looking for a good write-only output device, but DVDs are too small, and I have some worries about their longevity.)

 

[richardtoohey2]

Occasionally I have needed files off of one of these old drives, and I find that many of them no longer work.

Just what I was reading about in the news today (sorry, going a bit OT!)

Music Industry's 1990s Hard Drives Are Dying - Slashdot

An anonymous reader quotes a report from Ars Technica: One of the things enterprise storage and destruction company Iron Mountain does is handle the archiving of the media industry's vaults. What it has been seeing lately should be a wake-up call: roughly one-fifth of the hard disk drives dating...

hardware.slashdot.org

 

[dbtrade]

Just an note about my experience: I have made it a practice to never reuse old drives. When I upgrade to a larger drive, I put the old one in a box, marked "Keep forever", just in case I need it for a backup someday. Occasionally I have needed files off of one of these old drives, and I find that many of them no longer work. (My favorite was the one that started smoking when I turned it on.) Now, part of the problem may be that I live in the tropics; even though we have the A/C going 24/7 in the room where the disks live, the humidity is still a lot more than it was in Chicago. Luckily, I have enough old disks that I can usually find what I want on one of them. But I no longer think that keeping old disks around is a reliable long-term backup method. (I still do it, because I haven't found an alternative. I am still looking for a good write-only output device, but DVDs are too small, and I have some worries about their longevity.)

I have a separate backup drive that I rotate.

I meant I would use the old drives in some other backup scenario, not as a backup of their current data. I have been lucky in that those drives are more than 10 years old.

 

[dbtrade]

You can create a user on the remote server where you will receive the data stream and give it the necessary permissions:

zfs allow -u receiver_user compression,mountpoint,create,mount,receive pool/backup

Also if you are going to authenticate with root use the public key method and disable the Password authentication method, At least I could have a little more security.

I read the following recently:
"When performing a ZFS send and receive operation over SSH, it is generally better for the local user to be on the sending machine as this minimizes network latency and allows for smoother data transfer, especially when dealing with large datasets"

Which is what I think you are suggesting.
Thanks for the tip on creating the user on the remote receiving machine.