Xen is virtualization software, which lets you run multiple operating systems on the same physical machine. The goal of this guide is to show you how to create a FreeBSD 8 installation that will run as a guest OS, without the need for a processor that supports hardware-assisted virtualization. It would go beyond the scope of this guide to explain how exactly Xen works or what the differences between PV and HVM are, suffice to say dom0 is the host OS and domU the guest OS. Although FreeBSD fully supports being a Xen domU, it is not yet very well documented and not suited for novice Xen users. Especially the lack of a Xen-aware installer or pre-built Xen-aware images make things more difficult. If you are a Xen novice, I recommend starting first with pre-built Linux images, and then trying a NetBSD domU installation, before proceeding with FreeBSD.
Preparing the host
We will start with a normal Debian installation, we will need some free diskspace, beware that Linux names harddisks differently from FreeBSD, so if you have other OSes on this machine, be very careful you don't partition and format the wrong one. I recommend including the Desktop environment, having a graphical browser and the Synaptic Package Manager will be really useful. Now we need to install Xen itself. Debian mostly automates this process if you use apt-get or aptitude or synaptic. Installing xen-linux-system-2.6.26-2-xen should pull in all other needed dependencies (exact version may have changed by the time you read this guide). The required boot entries should be automatically added to GRUB, so in the next reboot we can start the Xen hypervisor.
Preparing the guest
There are no pre-built kernels and images we can use (but see below for mine), so we need to make our own. I'll assume you already have a FreeBSD installation somewhere, I used one existing inside a VirtualBox on a Windows 7 host. I have only tested with an i386 kernel and image, I am not certain whether amd64 is supported. The files we are going to create will eventually have to be transported to your dom0, so make sure you have some way to get them there. Go to a directory on your FreeBSD system with enough space.
Create a 256 MB file that will become our image:
Create a virtual disk backed by this file:
If this commands yields something other than md0, adjust all the following commands to that.
Create a MBR and single slice on the virtual disk:
Create a BSD-label inside our slice:
Create a UFS2 filesystem with softupdates enabled:
It is probably possible to directly create a filesystem inside the image and skip the MBR and label.
Mount the newly created filesystem:
Make sure you have a fully populated 8.0 /usr/src tree:
We are about to build our world and kernel. By default this will populate /usr/obj, this will do no harm, but it is possible to specify an alternate location:
csh users use setenv
Now it is time to start compiling, if you need multiple attempts to get things working, it is not necessary to do this step again each time (provided you did it correctly the first time):
Our file-backed virtual disk should still be mounted, so now we can install to it:
Now we need to modify a few files to prepare our installation for living in a Xen world.
Add to /mnt/etc/fstab:
Add to /mnt/etc/ttys:
We are almost done, but need to extract our kernel because we will need to put it in your dom0:
Now we can unmount and destroy the virtual disk:
The results of all our labour are now in freebsd.img and freebsd-kernel. Put these somewhere safe and reachable.
Configuring on the dom0
Go back to your dom0 installation, reboot into Xen if you hadn't already. Put the image and kernel we created somewhere on the dom0. They don't have to be in the same directory.
Check whether your Xen is working:
Now create /etc/xen/freebsd and use the following configuration:
Of course change the kernel and disk paths to wherever you put them. Later you can tweak the configuration, but first we must determine if everything is working.
Also edit /etc/xen/xend-config.sxp to enable bridged networking:
Everything should now be in place to start the domU:
The -c option will directly attach to the console output, alternative you can attach seperately:
At this point you should see the regular output of a booting FreeBSD and eventually the login prompt. If you can login, you have succeeded in creating a FreeBSD 8 Xen domU.
How to proceed
Of course all of this was just a small test installation not suited for real use. It's probably possible to increase the size of the image-file and growfs to a larger installation, but file-backed virtual disks aren't very efficient. If you want to make serious use of virtual FreeBSD installations, creating real partitions is the way to go. Let us know if you succesfully set this up, I am especially interesting in knowing whether GPT and ZFS will work in a Xen domU.
Known problems
ifconfig_xn0="DHCP" will not work, run dhclient or configure manually
References
This guide was heavily based on http://wiki.freebsd.org/AdrianChadd/XenImages and http://wiki.freebsd.org/AdrianChadd/XenHackery
Closing words
I started out about a week ago trying to figure out how to run FreeBSD in Xen. At first I was disappointed because I had assumed that since FreeBSD has full domU support, it would be relatively easy to do. Documentation was hard to find and much of it was outdated and being a total novice to Xen did not help either. Luckily I did not give up and soon become more acquainted with Xen terminology, which made it easier to understand Adrian Chadd's notes on the Wiki. I had already decided I did not want a UFS1 image, but a full simulated disk with an MBR, disklabel, UFS2 and softupdates, so I went that route. For simplicity I also decided to go for file-backed disk instead of LVM and to not use pygrub or pvgrub. I haven't yet extensively used and tested my FreeBSD domU, the moment I got it to actually boot (I made many mistakes at first) I started writing this guide.
Attached are the config, kernel and image file I created. This is an old image, you probably don't want to use it. Beware that sshd and dhclient are enabled, an account xenguest with password "password" exists and root is still password-less, so it is not a secure installation, but only meant for testing.
config
kernel (gzipped, 1.8M)
image (gzipped, 83M)
mirror
Preparing the host
We will start with a normal Debian installation, we will need some free diskspace, beware that Linux names harddisks differently from FreeBSD, so if you have other OSes on this machine, be very careful you don't partition and format the wrong one. I recommend including the Desktop environment, having a graphical browser and the Synaptic Package Manager will be really useful. Now we need to install Xen itself. Debian mostly automates this process if you use apt-get or aptitude or synaptic. Installing xen-linux-system-2.6.26-2-xen should pull in all other needed dependencies (exact version may have changed by the time you read this guide). The required boot entries should be automatically added to GRUB, so in the next reboot we can start the Xen hypervisor.
Preparing the guest
There are no pre-built kernels and images we can use (but see below for mine), so we need to make our own. I'll assume you already have a FreeBSD installation somewhere, I used one existing inside a VirtualBox on a Windows 7 host. I have only tested with an i386 kernel and image, I am not certain whether amd64 is supported. The files we are going to create will eventually have to be transported to your dom0, so make sure you have some way to get them there. Go to a directory on your FreeBSD system with enough space.
Create a 256 MB file that will become our image:
# truncate -s 256M freebsd.img
Create a virtual disk backed by this file:
# mdconfig -f freebsd.img
If this commands yields something other than md0, adjust all the following commands to that.
Create a MBR and single slice on the virtual disk:
# fdisk -BI md0
Create a BSD-label inside our slice:
# bsdlabel -wB md0s1
Create a UFS2 filesystem with softupdates enabled:
# newfs -U md0s1a
It is probably possible to directly create a filesystem inside the image and skip the MBR and label.
Mount the newly created filesystem:
# mount /dev/md0s1a /mnt
Make sure you have a fully populated 8.0 /usr/src tree:
# csup -h nearby.mirror.FreeBSD.org -L 2 /usr/share/examples/cvsup/standard-supfile
We are about to build our world and kernel. By default this will populate /usr/obj, this will do no harm, but it is possible to specify an alternate location:
# export MAKEOBJDIRPREFIX=/other/dir
csh users use setenv
Now it is time to start compiling, if you need multiple attempts to get things working, it is not necessary to do this step again each time (provided you did it correctly the first time):
# make buildworld && make buildkernel KERNCONF=XEN
Our file-backed virtual disk should still be mounted, so now we can install to it:
# export DESTDIR=/mnt && make installworld && make installkernel KERNCONF=XEN && cd etc && make distribution
Now we need to modify a few files to prepare our installation for living in a Xen world.
Add to /mnt/etc/fstab:
Code:
/dev/xbd0 / ufs rw 1 1
Code:
xc0 "/usr/libexec/getty Pc" vt100 on secure
We are almost done, but need to extract our kernel because we will need to put it in your dom0:
# cp /mnt/boot/kernel/kernel /some/place/freebsd-kernel
Now we can unmount and destroy the virtual disk:
# umount /mnt
# mdconfig -d -u md0
The results of all our labour are now in freebsd.img and freebsd-kernel. Put these somewhere safe and reachable.
Configuring on the dom0
Go back to your dom0 installation, reboot into Xen if you hadn't already. Put the image and kernel we created somewhere on the dom0. They don't have to be in the same directory.
Check whether your Xen is working:
# xm list
Now create /etc/xen/freebsd and use the following configuration:
Code:
kernel = "/virt/freebsd-8.0p2-i386-xen-domu-kernel"
memory = 512
name = "freebsd"
vif = [ '' ]
disk = [ 'file:/virt/freebsd-8.0p2-i386-xen-domu.img,hda,w' ]
extra = "boot_verbose=1"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"
extra += ",kern.hz=100"
Also edit /etc/xen/xend-config.sxp to enable bridged networking:
Code:
#(network-script network-dummy)
(network-script network-bridge)
Everything should now be in place to start the domU:
# xm create -c freebsd
The -c option will directly attach to the console output, alternative you can attach seperately:
# xm console freebsd
At this point you should see the regular output of a booting FreeBSD and eventually the login prompt. If you can login, you have succeeded in creating a FreeBSD 8 Xen domU.
How to proceed
Of course all of this was just a small test installation not suited for real use. It's probably possible to increase the size of the image-file and growfs to a larger installation, but file-backed virtual disks aren't very efficient. If you want to make serious use of virtual FreeBSD installations, creating real partitions is the way to go. Let us know if you succesfully set this up, I am especially interesting in knowing whether GPT and ZFS will work in a Xen domU.
Known problems
ifconfig_xn0="DHCP" will not work, run dhclient or configure manually
References
This guide was heavily based on http://wiki.freebsd.org/AdrianChadd/XenImages and http://wiki.freebsd.org/AdrianChadd/XenHackery
Closing words
I started out about a week ago trying to figure out how to run FreeBSD in Xen. At first I was disappointed because I had assumed that since FreeBSD has full domU support, it would be relatively easy to do. Documentation was hard to find and much of it was outdated and being a total novice to Xen did not help either. Luckily I did not give up and soon become more acquainted with Xen terminology, which made it easier to understand Adrian Chadd's notes on the Wiki. I had already decided I did not want a UFS1 image, but a full simulated disk with an MBR, disklabel, UFS2 and softupdates, so I went that route. For simplicity I also decided to go for file-backed disk instead of LVM and to not use pygrub or pvgrub. I haven't yet extensively used and tested my FreeBSD domU, the moment I got it to actually boot (I made many mistakes at first) I started writing this guide.
Attached are the config, kernel and image file I created. This is an old image, you probably don't want to use it. Beware that sshd and dhclient are enabled, an account xenguest with password "password" exists and root is still password-less, so it is not a secure installation, but only meant for testing.
config
kernel (gzipped, 1.8M)
image (gzipped, 83M)
mirror