Virtualize Physical FreeBSD Server to Run in VirtualBox

[nx]

How I can virtualize a physical FreeBSD server to run in VirtualBox?

An OS image as .img file can be converted into a VirtualBox .vdi (on Mac and in Ubuntu/linux), using the following command:

VBoxManage convertfromraw -format VDI physical_server.img virtual_server.vdi

How can I dump my server's FreeBSD install to a single .img to my workstation, using SSH from the server to my workstation or from my workstation ssh-ing to my server?

I've previously dumped individual partitions to single .tar.gz files, but I'm not sure how to combine (and convert?) to a single .img file. Is this possible?


Thanks

nx

 

[vermaden]

I have used that recently, but with SLES, the procedure was as follows:
1. Create virtual machine with needed resources
2. Boot that virtual with LiveCD
3. Create and mount needed filesystems on virtual machine (for example under /NEW path)
4. Start sshd daemon on virtual machine and allow root login
5. On the physical server You want to move type: # tar -czf - /* | ssh -c blowfish root@vm "tar -C /NEW -xzvpf -"
6. Edit differences in /etc/fstab on the virtual server
7. Eventually reboot and check what else needs to be tweaked, if needed, boot again with LiveCD and fix needed things

 

[nx]

Thanks Vermaden

Vermaden that's awesome advice!

I'll give this a try in the next couple of days and post back if I get in a muddle. Haha.

Though your ssh command looks perfect (thanks again), the only bit I'll probably have difficulty with is creating the vm's address to be accessible outside VirtualBox (from the server), but I should be able to work this out with a little trial and error.
Advice to do this would probably qualify this thread to have [Solved] stamped on it immediately.

Many thanks again!

nx

 

[wblock@]

It's really only a variation of backing up and restoring a system. dump via SSH from my backup article shows an example. Also see the next section, restore(8) Over A Live System.

 

[nx]

Thanks wblock

Thanks wblock, that had occurred to me too, though only after vermaden advised his method. LOL

I must thank you for your 'Backup Options For FreeBSD'.
It's an excellent guide that I had stumbled on a while back and I wish I had sooner!
I recommend anyone reading this thread to check it out. It's clarity in covering some complex processes is masterly.

For now I'm going to attempt vermaden's advice, but as I already have some servers dumped to individual partition files (most bzip2 - don't ask me why), I wonder if you could answer the 2 questions I asked at the start of this thread:

1.
How can I dump my server's FreeBSD install to a single .img to my workstation, using SSH from the server to my workstation or from my workstation ssh-ing to my server?
Is it possible to dump to a single dump file and then convert this to a .img file?

2.
I've previously dumped individual partitions to single .tar.gz (and other server's partitions to bzip2) files, but I'm not sure how to combine (and convert?) to a single .img file. Is this possible?

If I could do 2., this would make converting dumps to .vdi files much quicker than having to follow vermaden's excellent advice to dump each server again, and be valuable info for anyone else in a similar situation.

Or I guess I could simply follow your excellent guide to restoring the individual dump files I have to a virtual machine created with the LiveCD, which is I guess is what you are recommending. Haha

Thanks,

nx

 

[wblock@]

1.
How can I dump my server's FreeBSD install to a single .img to my workstation, using SSH from the server to my workstation or from my workstation ssh-ing to my server?
Is it possible to dump to a single dump file and then convert this to a .img file?

The dd(8) example in the article shows a way to do this.

2.
I've previously dumped individual partitions to single .tar.gz (and other server's partitions to bzip2) files, but I'm not sure how to combine (and convert?) to a single .img file. Is this possible?

Yes. Create an empty image file, for example with dd(8):
# dd if=/dev/zero of=myimagefile.img bs=1m count=8192

mdconfig(8) lets you mount that file as a device. Set up the "disk" with fdisk(8) and bsdlabel(8) if needed. Restore with restore(8) or dd(8), depending on your source files.

This will go faster if the source and destination files are on separate disks to avoid disk contention. But it'll work even if you only have one disk.

 

[nx]

Thanks wblock [SOLVED]

Wow!

Thanks for solving those questions wblock - your answers will help me and many others.

I have a lot to learn, but with amazing help like this, someday soon I may be able to help others.

These forums and its members like you and Vermaden make me happy I chose FreeBSD for my servers!

Many thanks again,

nx

 

[vermaden]

@nx

Welcome mate

Though your ssh command looks perfect (thanks again), the only bit I'll probably have difficulty with is creating the vm's address to be accessible outside VirtualBox (from the server), but I should be able to work this out with a little trial and error.

You can do that in two steps, on the server that do not 'see' the virtual machine IP, do that:
[cmd=]# tar -czf sys.tar.gz /*[/cmd]

Then on the system that 'sees' IP of virtual machine:
[cmd=]# dd < sys.tar.gz | ssh -c blowfish root@vm "tar -C /NEW -xzvpf -"[/cmd]

I have also recently done that using 'raw' dd(1) instead of tar(1) because the system (SLES/Linux) contained LVM groups and so and I *definitely* did not wanted to mess with that, so I used dd(1) and the only things I have to do after moving the data was to install vmware-tools (it was PHYS -> ESX conversion) and 'move' from eth0 to eth2 (only SLES/Linux knows why), the method looks like that:
[cmd=]# dd < /dev/sda bs=*M | ssh -c blowfish root@vm "dd > /dev/sda bs=8M"[/cmd]

 

[danbi]

What follows in just and advice and comment, not a guide..

As you have seen, there are two possible scenarios, the 'image' and 'filesystem' scenario. If you want to preserve the same disk layout in the virtual machine as it was in the 'real' system, you may use both. The image in a VM is what you will see as 'disk' in the operating system, so the 'proper' way to do it as follows.

Assume you have single disk, named ad0 on the 'real' hardware. You create the image with a command like this:

(boot from LiveCD, USB stick or single user, before mounting any filesystem read-write.

# dd if=/dev/ad0 of=/somepath_with_enough_space/disk.img bs=128k

You may use external, USB attached drive to create the image on the real server.

The other way is to replicate filesystems. For this, I strongly suggest you avoid tar and use dump/restore instead! If you are happy with using tar, you may well use rsync instead, which is way better over the network. One great advantage in using rsync is that you can reduce downtime by doing incremental syncs by first creating your VM, doing the initial rsync which may take long time (at that time, your 'real' system is running), then doing one more sync to catch up with what changed, stop the 'real' system, do a final sync (which should be very quick) and start the VM with it's proper addresses etc.
In either case (of not using dump/restore) it is best to reinstall the OS afterwards, for example by doing an in-place source build -- this way you will be sure that all permissions, links, special files etc. are available in the new system.
By the way, you may enable special flags preservation etc. when building net/rsync from ports, that may help avoid this last step.

What you will chose depends on your goals. If you do not have much free space in the 'real' system, the 'entire disk image' is the best approach, because it wastes less time and produces an exact copy. Otherwise, it is better to duplicate the filesystems.

By the way, if your host system is also FreeBSD, it is probably wiser to use jails for the virtual machines, instead of VirtualBox, especially if you are not trying to emulate all the hardware aspects of the original (disk sizes etc).

 

[nx]

@vermaden

Thanks for your latest post - fortunately (flukely?) I got ssh working from my server to the vm on my workstation, but your information will be useful at sometime in the future no doubt when I can't do this.

So now I'm writing after trying your advice.

First I tried to follow your first post and method, but somehow I stuffed up creating and mounting the new file systems properly. My server and virtual machine have identical fstab:

/dev/ad0s1b   none     swap    sw         0 0
/dev/ad0s1a   /        ufs     rw         1 1
/dev/ad0s1e   /tmp     ufs     rw         2 2
/dev/ad0s1f   /usr     ufs     rw         2 2
/dev/ad0s1d   /var     ufs     rw         2 2
/dev/acd0     /cdrom   cd9660  ro,noauto  0 0

I'll give this another try, if you wouldn't mind detailing:

3. Create and mount needed filesystems on virtual machine (for example under /NEW path) 

and how this relates to (I have edited fstab paths before):

6. Edit differences in /etc/fstab  on the virtual server

At first the top level directories were copied from the server to the vm on my workstation but then I got a lot of 'Failed to create directory' errors reported on the server and the vm reported a tar related error (sorry didn't get it copied to show here), which I guess was the cause. But I was still confused as to how to then use the new filesystem to replace the existing one on the vm, though it appeared from the error that I didn't have the whole server copied to replace the vm's file system with.

So I thought I'd try using raw dd instead of tar as you later suggested with:

# dd < /dev/ad0 bs=8M | ssh -c blowfish root@vm "dd > /dev/ad0 bs=8M"

but although ssh connected - the server asked for workstations password - the server reported:

/dev/ad0: Operation not permitted.

and does not start the dd to the vm.

Is this because of read/write permissions in the vm or on the servers disk, or both?
I'm sure I'm using the correct disk label - ad0 is the disk in both the server and vm.

Thanks again, I hope you can clarify these things for me and I can successfully complete the steps you have graciously outlined.

---------------------------

@danbi,

Thanks for all your advice!

If you could rewrite:

# dd if=/dev/ad0 of=/somepath_with_enough_space/disk.img bs=128k

to be done over ssh from the server to another machine to store the image file, then I could convert the .img file to a .vdi, and run my physical server as a virtual machine.

wblock's Backup Options For FreeBSD page:

http://www.wonkity.com/~wblock/docs/html/backup.html#_tt_dump_tt_via_ssh

looks to have most the info needed to put this together - I'm just too stupid! LOL


Many thanks to everyone helping me!
I hope my questions and the answers/info provided here help other noobs like myself and perhaps those more experienced with FreeBSD who want a guide to virtualize their servers.
Once I have virtualized my servers I feel I will learn more quickly as I'll be less scared to try new things on them compared to experimenting on my physical production servers.
I already have installed FreeBSD as vm's but there's nothing like having your current installation from a physical server as a vm to experiment with!
And it is my crazy hope that in the future I can convert my vms to improved physical servers.

 

[wblock@]

So I thought I'd try using raw dd instead of tar as you later suggested with:

# dd < /dev/ad0 bs=8M | ssh -c blowfish root@vm "dd > /dev/ad0 bs=8M"

but although ssh connected - the server asked for workstations password - the server reported:

/dev/ad0: Operation not permitted.

and does not start the dd to the vm.

There's a safety that prevents overwriting the disk boot block by default. You can disable it when you really do want to write to the first block of the disk:
# sysctl kern.geom.debugflags=16

Last night I overcame my normal turtle-like torpor and wrote up my notes on Disk Setup On FreeBSD, which may be useful. Please note that this is preliminary; the first section on fdisk(8) and bsdlabel(8) is well-tested, but I haven't re-checked the alternate method gpart(8) section yet. Feedback welcome.

 

[nx]

Thanks wblock,

I've used:

# sysctl kern.geom.debugflags=16

when setting up gmirror on other drives, so perhaps it should have occurred to me for using dd in this way.

I'm going to have a quick read of your new page on Disk Setup On FreeBSD first, and then have another try at using dd from the physical server to vm.

I hope you do overcome your torpor more often! You have an excellent style of documentation, which has a clarity that's hard to find elsewhere.
But perhaps this is from taking a turtle-like approach rather than rushing like a hare? If so, keep doing what you are doing.
You have a fan here and I hope others take the time to read your work. It's very helpful stuff!

Many thanks again,

nx

 

[danbi]

The command to do dd over ssh is something like this. On the VM host, do

$ ssh root@server dd if=/dev/ad0 bs=128k > disk.img

Then, on the machine running as host for the VM you will have the file disk.img, which is exact copy of the server's disk. Beware if your server's disk is big

The alternative with rsync is something like this:

1. Create whatever virtual disk you think fits for the VM (it needs to be big enough to hold all your servers data);
2. Boot the VM from LiveCD;
3. Partitition the virtual disk the same way it is on the real server;
4. Install boot blocks to the virtual disk - it won't boot otherwise (note, when you use dd, you are copying the disk sector by sector, so this step is not neccesary)
5. Assuming, you mounted the new filesystem tree under /mnt, run

# rsync -avH --numeric-ids root@server:/ /mnt/

As I mentioned, you may run this command while the server is operational, then run it again (it will finish fast) when it is stopped to transfer any remaninig changes. Beware, that sometimes this won't work well, because of the special immutable flags on some files. Since these are all OS files and are not supposed to change on the server between runs, you may use the --ignore-existing option to rsync to skip those.
But as all other advices assume your server is not operating, I think you may just run it once.
You need to install net/rsync on both server and VM..

The other option is to use dump/restore. It's the most 'canonical' way for this task
To use dump/restore you do steps 1-4, then for each filesystem, run the following in the VM (for example /var)

# cd /mnt/var
# ssh root@server dump 0Lf - /var | restore rf -

restore will compain there is a .snap directory already and for the root directory that the mountpoints for other filesystems exist, but you can safely ignore these warnings. There will be new file restoresymtable in the root of each filesystem that you may safely delete afterwards.

With all my respect, I do not see any value in using tar and related in duplicating UFS filesystems. dump/restore are more than sufficient and preserve all filesystem information. For other filesystem types, I would default to rsync all the time. But tar can do just as well.

In summary:

• dd will copy over every sector from the original disk; your disk layout and size remains the same.
• dump will copy over every sector from the filesystem, that is in use; you may change destination filesystem type, but note dump only works on UFS.
• rsync/tar will copy over every byte of the original files; these work on any file system type both source and destination;

 

[nx]

==================================================

SOLVED - Quick Successful Virtualization of Physical Server to Run in VirtualBox
(Based on Above Posts in this Thread)

==================================================

1. From your workstation that will run the VirtualBox virtual machine, ssh to the physical server and dd the file server_disk.img back to your workstation:
(Many thanks to danbi for the command!)

$ ssh root@server dd if=/dev/ad0 bs=128k > server_disk.img

2. On your workstation convert the server_disk.img file to virtual_disk.vdi with:
(If your .img file is big this will take a few minutes to convert to .vdi.)

VBoxManage convertfromraw -format VDI server_disk.img virtual_disk.vdi

3. Create a virtual machine in VirtualBox to represent your physical server using the virtual_disk.vdi as the hard disk, and boot!


If the virtual machine doesn't boot to the FreeBSD install imaged from your physical server, this may be caused by an option that you have or haven't ticked in the VirtualBox Settings for your VM.

Troubleshooting in VirtualBox Settings
------------------------------------------------
In the System settings, make sure you have ticked the option for hard disk to boot (first), and below this make sure you have ticked the Hardware clock in UTC time option.
I unticked all the Acceleration settings options, though these may work for you.
In the Network settings I chose Bridged Adapter to give the VM access to the net and allow ssh to and from the host OS (OS X), though I probably should have selected NAT and tweaked this somewhere.
To first get the VM to boot properly I've tried to keep my virtual machine settings as simple and as close to my original server settings as possible, and thereafter will experiment with settings to optimize them.


If anyone has any methods, tips, etc from their virtualization of servers for VirtualBox, please add to this thread.

I hope this simple virtualization method works for you!

nx

==================================================

Many thanks to vermaden, danbi, and wblock!


@danbi

That's an awesome explanation of backup/imaging/restoration methods and the differences between them!
I've used dump/restore before but haven't yet had time to get rsync working. Thanks for the reminder. I'm keen to give it another try after reading your post.
Your ssh dd command worked perfectly for me - obvious from my post above the line - and allowed me to solve the first question I started this thread with. Woohoo!
Many thanks again!



@vermaden

I first tried your dd method, unfortunately only with partial success (due to a mistake on my part), by using:

1. # sysctl kern.geom.debugflags=16

2. # dd < /dev/ad0 bs=8M | ssh -c blowfish root@vm "dd > /dev/ad0 bs=8M"

which replicated the server to the virtual hard disk, but gave too many errors on boot for me to fix correctly, mostly likely caused by me creating the virtual hard disk to be too small by 1 or 2 megs.
I would like to revisit this approach again and do it right, and I would like to try the other methods you have described, as I'm sure they will be very handy when I'm not so time constrained in the future.
Thanks for presenting these alternative methods so promptly. I wish I knew more to use them sooner.
(It takes just over 2hours per ssh dd operation, so I'm using the method that works for me for now.)
Many thanks again!



@wblock

Your excellent page explaining Disk Setup on FreeBSD cleared up my confusion about mounting new file systems, and your style of documentation is second to none. Please write more often!
Thank you for contributing the method of creating an empty image file and using this to mount individually dumped partitions. When I'm more confident I will certainly give this a try as I already have individual partition backups of my servers.
And thanks for reminding me to turn the 'safety switch' off to use dd by with sysctl.
Many thanks again!



I hope this thread helps a lot of others. It certainly has helped me.
Again, if others have any info from their virtualization of FreeBSD servers for VirtualBox, please add to this thread.

Thanks,

nx

 

[vermaden]

@nx

Welcome mate.