HOWTO: Clone a FreeBSD (or others) system using SSH and Named Pipe

khashmeshab

khashmeshab

(I'M NOT ENGLISH!)

Hello everybody,

Yesterday I decided to move my web server from a German datacenter to US. Then I needed to copy the whole system and then make the changes to the configurations (because of IP change).

First of all, I installed a fresh and simple FreeBSD on the destination system to be able to use the basic commands and tools such as SSH and SCP.

At first, I tried the scp command to copy files as below:
(I couldn't use the dd command, because the hard disk space was changed.)

# scp -C -p -r root@<source_ip>:/ /

Using the above command, I faced a weird problem. The speed of copying a file was very good (around 5-6 MB/s). But there was more than one second delay while switching between files. As you know, the operating system environment consists of so many small files and the 1 second delay between each file may need a couple of days to copy 3 GB of small files.

So I researched a little and remembered the Named Pipe thing exists in modern *NIX-like OSes. So I ran the following commands on the source computer:

# mkfifo /tmp/portal
# tar czpf /tmp/portal /

The first command creates a new Named Pipe. The second command tries to create a compressed archive from the root directory to the created Named Pipe. But since there is no program at the other end of the pipe, it blocks temporarily.

Then I ran these commands on the destination server:

# mkfifo /tmp/portal
# cd /
# tar xzf /tmp/portal &
# bg
# ssh root@<source_ip> 'cat /tmp/portal' > /tmp/portal

The first command creates another Named Pipe but in the destination server. The tar command tries to read the pipe and extract it, but because there is nothing at the other end of pipe, it blocks. The ssh command actually connects the end of two pipes to each other by sending the end of the source pipe to the start of the destination pipe. At this time, the source pipe has an end and the tar command on the source server starts archiving and compressing files. And the destination pipe also has a sender at its other end and the tar command starts extracting the archive. In this case the archive created on the fly and there is no need to store the whole archive as a separate file (which needs space that I didn't have!) and it takes less than half the time. Actually the archiving, moving and extracting operations are executed simultaneously and in parallel.

And the whole copying was completed in less than 30 minutes! I think I created a wormhole (http://en.wikipedia.org/wiki/Wormhole) ;)

[ Edited - see port #8 and #9 below - Mod. ]
 
rsync(8) works really well for this, as it handles all that internally. :) And, if you install security/openssh-portable with the HPN patches enabled, you can use the none cipher to not encrypt the SSH data channel for even better performance (can also use that with the named pipe solution).

Like the description of the journey you took. :) There's always more than one way to do it, and there's not really "the one true / best / correct way". :D
 
phoenix said:
rsync(8) works really well for this, as it handles all that internally. :) And, if you install security/openssh-portable with the HPN patches enabled, you can use the none cipher to not encrypt the SSH data channel for even better performance (can also use that with the named pipe solution).

Like the description of the journey you took. :) There's always more than one way to do it, and there's not really "the one true / best / correct way". :D
Click to expand...

Thank you for you comment. The rsync(8) command is also a good alternative. But using ssh(8) without encryption for a person who lives in Iran is more than dangerous! ;) (I know that data is transmitted between Germany and US, just kidding!)
 
wblock@ said:
The pipe is probably not necessary. ssh(1) can pipe output from the local system to commands on the remote system. Examples of doing that with dump(8) are in Backup Options For FreeBSD.
Click to expand...

What an idiot I am! You are right. The Named Pipe isn't necessary and I could pipe the output of the tar(8) command from the remote system to local. The problem was that Named Pipes are a great and tempting feature and I had not used them yet. So my brain just locked on it and it didn't let me think of any other solution.

By the way, on the destination computer:

Code: 
# cd /
# ssh root@<source_ip> 'tar czp /' | tar xz

No need to run anything on the remote (source) computer.

Thank you very much.
 
I think the commands displayed in the first post are wrong.

Code: 
1) mkfifo /tmp/portal     # creates a named pipe which never gets used.
2) tar czpf /tmp/karsha / # creates a tar archive named karsha.
3) mkfifo /tmp/portal
4) cd /
5) tar xzf /tmp/portal &
6) bg
(3-6) create named pipe "portal" on the other machine.
Then read the pipe in the background.
Code: 
7) ssh root@<source_ip> 'cat /tmp/karsha' > /tmp/karsha

(7) Creates an ssh connection, cats the tar archive called karsha on the "source" system and redirects it to a file called karsha (which would be a tar archive again) on the "destination" system.
End result is: two archives called karsha on src and dst systems.
Two unused named pipes called portal on src and dst systems.
One hanging tar process reading from the portal pipe on the dst system.
 
Sorry bro! It's my mistake and I apologize for it. But I don't have the edit permission to correct the commands. So anybody who reads this, please use "/tmp/portal" instead of "/tmp/karsha" everywhere.

Thank you for you attention and for reporting this.
 
You must log in or register to reply here.
Back
Top