HOWTO: Create a FreeBSD 8 i386 Xen PV domU

Xen is virtualization software, which lets you run multiple operating systems on the same physical machine. The goal of this guide is to show you how to create a FreeBSD 8 installation that will run as a guest OS, without the need for a processor that supports hardware-assisted virtualization. It would go beyond the scope of this guide to explain how exactly Xen works or what the differences between PV and HVM are, suffice to say dom0 is the host OS and domU the guest OS. Although FreeBSD fully supports being a Xen domU, it is not yet very well documented and not suited for novice Xen users. Especially the lack of a Xen-aware installer or pre-built Xen-aware images make things more difficult. If you are a Xen novice, I recommend starting first with pre-built Linux images, and then trying a NetBSD domU installation, before proceeding with FreeBSD.

Preparing the host
We will start with a normal Debian installation, we will need some free diskspace, beware that Linux names harddisks differently from FreeBSD, so if you have other OSes on this machine, be very careful you don't partition and format the wrong one. I recommend including the Desktop environment, having a graphical browser and the Synaptic Package Manager will be really useful. Now we need to install Xen itself. Debian mostly automates this process if you use apt-get or aptitude or synaptic. Installing xen-linux-system-2.6.26-2-xen should pull in all other needed dependencies (exact version may have changed by the time you read this guide). The required boot entries should be automatically added to GRUB, so in the next reboot we can start the Xen hypervisor.

Preparing the guest
There are no pre-built kernels and images we can use (but see below for mine), so we need to make our own. I'll assume you already have a FreeBSD installation somewhere, I used one existing inside a VirtualBox on a Windows 7 host. I have only tested with an i386 kernel and image, I am not certain whether amd64 is supported. The files we are going to create will eventually have to be transported to your dom0, so make sure you have some way to get them there. Go to a directory on your FreeBSD system with enough space.

Create a 256 MB file that will become our image:
# truncate -s 256M freebsd.img

Create a virtual disk backed by this file:
# mdconfig -f freebsd.img
If this commands yields something other than md0, adjust all the following commands to that.

Create a MBR and single slice on the virtual disk:
# fdisk -BI md0

Create a BSD-label inside our slice:
# bsdlabel -wB md0s1

Create a UFS2 filesystem with softupdates enabled:
# newfs -U md0s1a
It is probably possible to directly create a filesystem inside the image and skip the MBR and label.

Mount the newly created filesystem:
# mount /dev/md0s1a /mnt

Make sure you have a fully populated 8.0 /usr/src tree:
# csup -h nearby.mirror.FreeBSD.org -L 2 /usr/share/examples/cvsup/standard-supfile

We are about to build our world and kernel. By default this will populate /usr/obj, this will do no harm, but it is possible to specify an alternate location:
# export MAKEOBJDIRPREFIX=/other/dir
csh users use setenv

Now it is time to start compiling, if you need multiple attempts to get things working, it is not necessary to do this step again each time (provided you did it correctly the first time):
# make buildworld && make buildkernel KERNCONF=XEN

Our file-backed virtual disk should still be mounted, so now we can install to it:
# export DESTDIR=/mnt && make installworld && make installkernel KERNCONF=XEN && cd etc && make distribution

Now we need to modify a few files to prepare our installation for living in a Xen world.
Add to /mnt/etc/fstab:
Code:
/dev/xbd0       /               ufs     rw              1       1
Add to /mnt/etc/ttys:
Code:
xc0     "/usr/libexec/getty Pc"         vt100   on  secure

We are almost done, but need to extract our kernel because we will need to put it in your dom0:
# cp /mnt/boot/kernel/kernel /some/place/freebsd-kernel

Now we can unmount and destroy the virtual disk:
# umount /mnt
# mdconfig -d -u md0

The results of all our labour are now in freebsd.img and freebsd-kernel. Put these somewhere safe and reachable.

Configuring on the dom0
Go back to your dom0 installation, reboot into Xen if you hadn't already. Put the image and kernel we created somewhere on the dom0. They don't have to be in the same directory.

Check whether your Xen is working:
# xm list

Now create /etc/xen/freebsd and use the following configuration:
Code:
kernel = "/virt/freebsd-8.0p2-i386-xen-domu-kernel"
memory = 512
name = "freebsd"
vif = [ '' ]
disk = [ 'file:/virt/freebsd-8.0p2-i386-xen-domu.img,hda,w' ]
extra = "boot_verbose=1"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"
extra += ",kern.hz=100"
Of course change the kernel and disk paths to wherever you put them. Later you can tweak the configuration, but first we must determine if everything is working.

Also edit /etc/xen/xend-config.sxp to enable bridged networking:
Code:
#(network-script network-dummy)
(network-script network-bridge)

Everything should now be in place to start the domU:
# xm create -c freebsd
The -c option will directly attach to the console output, alternative you can attach seperately:
# xm console freebsd

At this point you should see the regular output of a booting FreeBSD and eventually the login prompt. If you can login, you have succeeded in creating a FreeBSD 8 Xen domU.

How to proceed
Of course all of this was just a small test installation not suited for real use. It's probably possible to increase the size of the image-file and growfs to a larger installation, but file-backed virtual disks aren't very efficient. If you want to make serious use of virtual FreeBSD installations, creating real partitions is the way to go. Let us know if you succesfully set this up, I am especially interesting in knowing whether GPT and ZFS will work in a Xen domU.

Known problems
ifconfig_xn0="DHCP" will not work, run dhclient or configure manually

References
This guide was heavily based on http://wiki.freebsd.org/AdrianChadd/XenImages and http://wiki.freebsd.org/AdrianChadd/XenHackery

Closing words
I started out about a week ago trying to figure out how to run FreeBSD in Xen. At first I was disappointed because I had assumed that since FreeBSD has full domU support, it would be relatively easy to do. Documentation was hard to find and much of it was outdated and being a total novice to Xen did not help either. Luckily I did not give up and soon become more acquainted with Xen terminology, which made it easier to understand Adrian Chadd's notes on the Wiki. I had already decided I did not want a UFS1 image, but a full simulated disk with an MBR, disklabel, UFS2 and softupdates, so I went that route. For simplicity I also decided to go for file-backed disk instead of LVM and to not use pygrub or pvgrub. I haven't yet extensively used and tested my FreeBSD domU, the moment I got it to actually boot (I made many mistakes at first) I started writing this guide.

Attached are the config, kernel and image file I created. This is an old image, you probably don't want to use it. Beware that sshd and dhclient are enabled, an account xenguest with password "password" exists and root is still password-less, so it is not a secure installation, but only meant for testing.
config
kernel (gzipped, 1.8M)
image (gzipped, 83M)
mirror
 
Hi Aprogas,

first of all: Thanks for all your effort to create that rar how to and also for uploading significant files.

I tried your how to several times now [ with an i386 version ;) ] but unfortunatley am I always getting a kernel panic when I'm about booting the virtual FreeBSD. No matter what image I used (I even tryed the actual pfsense which is based on 7.2) plus I tryed your kernel and image as well - but the result was always the same.
Code:
xm create -c /FreeBSD-i38.cfg
started the machine and it looked like it would work, but then all of a sudden it broke up again and switched back to the regular console.
Code:
xm list
showed me that the thing broke while booting.
It must be a Kernel issue?! OR the only thing I could think of is that my Xen might be incompatible with the FreeBSD DomU project cause I'm using NetBSD 5.0.2 with the actual ported Xen 3.3.2 - but as far as I know should that 3.3.2 do the job?!

... I even saw the Login once for a millisecond ... but then it broke again ... ;(

Any Ideas?

P.S. I'll post the last boot information as soon as possible.


Regards,

Leander
 
This won't help you alot Leander, but I just tried with a plain Debian install with Xen and the image and kernel provided here, and it's working without any problems. So thats Xen 3.2.1 on Debian 5.0

I'm up to compiling the kernel myself now and creating an image. Thanks for this tutorial!
 
I'd like to post my attempts at creating a FreeBSD 8.1-RC PV domU.

Hardware:
CPU = AMD Athlon II Regor 250

DomU is Debian 5.0.5 (Lenny), linux kernel 2.6.26-2, xen 3.2-1. I installed FreeBSD 8.1-RC on Virtualbox, created a dev environment, pulled down sources, compiled the kernel with KERNCONF=XEN. I also installed FreeBSD to an image by creating a FreeBSD HVM (which works fine btw, using the loader option hw.clflush_disable=1). I then tried to use the XEN kernel as the PV kernel (i.e. kernel='/path/to/xen/kernel'), but I get this kernel panic:

Code:
panic: HYPERVISOR_update_va_mapping(((unsigned long)(va)), (pa | 0x002 | 
0x001 | pgeflag | pmap_cache_bits(mode, 0)), UVMF_INVLPG| UVMF_ALL) < 0: 
/usr/src/sys/i386/xen/pmap.c:1269
cpuid = 0
KDB: enter: panic
[thread pid 0 tid 0 ]
Stopped at      kdb_enter+0x3a: movl    $0,kdb_why

Found very little documentation, other than this "successful" attempt. I say "successful" because simply using the image and kernel provided here also results in a kernel panic, albeit a different one.

Is the FreeBSD PV domU more favorable on Intel HW? I can switch if needed, but would prefer not to.
 
I downloaded your kernel and image files (from a mirror). They worked without issue.

System:

Lenovo R61 ThinkPad
Intel Core2 Duo
Debian Lenny (5.0.4) Stable (2.6.26-2-xen-amd64)
Xen 3.2.1 (AMD64)

Next, I will follow your howto and try to build my own.

Thank you!
 
I've just successfully built a kernel and disk image with RELENG_8_0 . I guess Xen kernel is broken in 8.1
 
Thank you @Aprogas . Excellent guide.

Just for the record if you try to create your own images and kernel from a amd64 release you should simply make the following changes (the kernel configuration file is called XENHVM on amd64) :

Code:
make buildworld && make buildkernel KERNCONF=[B]XENHVM[/B]

Code:
export DESTDIR=/mnt && make installworld && make installkernel KERNCONF=[B]XENHVM[/B] && cd etc && make distribution

Tested with 8.1. The generated amd-64 kernel and the relative image are not suitable for paravitualized guests, but only for full virtualization, so your domu configuration file should be different. I tried to test it with as a full virtualization image - it seems to boot - but i cannot proceed as i need vnc to take output... (i'm lazy to do this now...)
 
I have spent/wasted a lot of time trying to get FreeBSD run as a DomU under NetBSD Dom0. My NetBSD is the amd64 flavour of release 5.0.2. I use Xen 3.3.2 (installed via pkgsrc, NetBSD's answer to FreeBSD's ports).

I tried the i386 images presented here, and they yield immediate kernel panics. I have compiled my own i386 kernels (using KERNCONF=XEN) and they work no better.

I have compiled amd64 images, using KERNCONF=XENHVM and made futile attempts at getting them to boot. In the apparent absence of documentation (at least such documentation that is not obsolete and/or sloppy) I've had problems figuring out how to boot this image. I started with /usr/pkg/share/examples/xen/hvm, but it is not clear how to point to the FreeBSD disk image from there; I made it the disk device. But it is clear that this is not the correct approach, since there is no reason that the system should be able to read FreeBSD's file system.

I have tried various version of FreeBSD 8.x, RELEASEs and RELENGs during this struggle.

My question is: Does anybody here have a working FreeBSD of any sort under a NetBSD dom0, and is willing to explain exactly how to get it to run without triggering bugs?

(My ultimate goal is setting up a large number of DomU on all my test systems, but I have thus far only succeeded with NetBSD i386 and NetBSD amd64... My main development platform is FreeBSD, so it that fails, my test array project fails.)
 
Good news everyone!

Using a combination of these instructions, the NetBSD instructions, some good luck and some help from #bsdcode people, I've managed to get FreeBSD i386 to boot under Linode (which is XEN+PV).

The 8.0 images included above boot on linode reasonably trivially, but upgrading the kernel at all yields immediate panics. Thanks to nox @ #bsdcode, I discovered this morning that the panics many people are seeing under i386 paravirtualized Xen are caused by the new x86bios stuff that wasn't in 8.0-RELEASE... it's only called from the keyboard stuff (nox found that, and vesa which call it, but vesa isn't in the default i386 XEN kernel config). Commenting out atkbdc/atkbd/kbdmux from the default XEN kernel config results in an 8.1-RELEASE-p1 kernel that boots and doesn't seem to blow up at all, though I don't know which device or function is actually the buggy part and I'm not smart enough to track down where the issue is and fix it. :(

Since the process of getting FreeBSD on a linode isn't documented at all, anywhere, I'm working on a set of instructions which I'll post on my site, then mirror here and on the linode wiki. The instructions are still a work in progress, because each time I follow through my own notes, something is missing or goes wrong, so obviously I'm doing stuff from muscle memory and not documenting it, then forgetting it in subsequent tries.

I don't know what version of Xen linode are using, nor what they're using for a host, but this is a current FreeBSD that's working-for-me under Xen-PV which is a start. :D

Edit: Google-snacks:

Code:
panic: HYPERVISOR_update_va_mapping(((unsigned long)(va)), (pa |
0x002 | 0x001 | pgeflag | pmap_cache_bits(mode, 0)), UVMF_INVLPG| 
UVMF_ALL) < 0: /usr/src/sys/i386/xen/pmap.c:1269
cpuid = 0

This particular panic with i386 kernels >8.0-R can be fixed by removing atkbd (I removed atkbdc + kbdmux too, not sure if that's necessary though) from your kernel, then booting from the updated kernel.
 
I now have FreeBSD 6.4 amd64 as well as FreeBSD 6.4 i386 running under a NetBSD Dom0. Unlike 7.x and 8.x installation and operation are smooth.

Code:
kernel = "/usr/pkg/lib/xen/boot/hvmloader"
builder='hvm'
memory = 256
vcpus = 1
name = "freebsd-64"
vif = [ 'mac=00:16:3e:00:01:64, bridge=bridge0, type=ioemu, model=ne2k_pci' ]
device_model = '/usr/pkg/libexec/qemu-dm'
disk = [
    'file:/home/xen/freebsd/64/6.4/disk,ioemu:hda,w',
#    'file:/home/tege/6.4-RELEASE-amd64-disc1.iso,ioemu:hdc:cdrom,r'
]
boot='c'
#boot='d'
pae = 1
usb=1
usbdevice='tablet'

The #-marked lines need to be uncommented for installation.

It seems that using a mac address not starting with 00:16:3e makes NetBSD/Xen confused (sic!). It seems that the default simulated network card is unreliable, therefore the need for the ne2k_pci moniker.

I have made some progress with 8.1 amd64 too, with a custom kernel. It boots, but the installation is not yet robust.
 
fwaggle said:
Good news everyone!

Code:
panic: HYPERVISOR_update_va_mapping(((unsigned long)(va)), (pa |
0x002 | 0x001 | pgeflag | pmap_cache_bits(mode, 0)), UVMF_INVLPG| 
UVMF_ALL) < 0: /usr/src/sys/i386/xen/pmap.c:1269
cpuid = 0

This particular panic with i386 kernels >8.0-R can be fixed by removing atkbd (I removed atkbdc + kbdmux too, not sure if that's necessary though) from your kernel, then booting from the updated kernel.

Thank you! Removing atkbd + atkbdc + kbdmux worked also for me. I also removed INVARIANTS, INVARIANT_SUPPORT, WITNESS and WITNESS_SKIPSPIN to improve performance.

Do you get the following notification about every 3 minutes in your terminal?

Code:
# [XEN] hypervisor wallclock nudged; nudging TOD.

It seems to be generated by the rev.193033. My domu clock it seems to work fine, although I do not know how to stop them (add a comment?).
 
vtypal said:
Thank you! Removing atkbd + atkbdc + kbdmux worked also for me. I also removed INVARIANTS, INVARIANT_SUPPORT, WITNESS and WITNESS_SKIPSPIN to improve performance.

nox via #bsdcode was the one who figured it out. He also learned later that the fix for that problem was in -CURRENT the entire time, but it can be reversed because apparently in -CURRENT that whole section of code is replaced anyway.

He and a developer on the freebsd-xen mailing list asked if I could try boot a -CURRENT kernel with the "fix" reverted, and it exploded on boot in a different place, so I went back to my *kbd*-less kernel and it's been running fine ever since. I was going to post the backtrace from -CURRENT to the freebsd-xen mailing list, but it seems like every time I go to post on a FreeBSD mailing list my posts I stick my foot in my mouth, and laziness eventually won out.

It's spontaneously blown up on me once when I left nload running (nload has garbage in the interface name too, which might be related, but I can't be bothered tracking the issue down), but other than that I have a Mumble server on that box that's been chugging along for a couple weeks now.

Do you get the following notification about every 3 minutes in your terminal?

Code:
# [XEN] hypervisor wallclock nudged; nudging TOD.

It seems to be generated by the rev.193033. My domu clock it seems to work fine, although I do not know how to stop them (add a comment?).

I actually get that closer to every 30 seconds - so annoying I actually commented out the line that generates it in case it blew away something more pertinent in dmesg (in my notes, the notice is generated at sys/i386/xen/clock.c:344). As you said, the clock seems to be working fine, and I don't feel like tracking down the actual cause (I'm not near familiar enough with Xen/FreeBSD). :)

I said I'd mirror my notes on this forum and the Linode wiki, but honestly I keep going back and editing them and putting them in multiple places just seems like way too much effort, so I published them on one of my sites and I'll mirror them when they're a little less turbulent:

http://www.hungryhacker.com/os/freebsd-on-linode/

Hope it helps someone. :)
 
fwaggle said:
Code:
# [XEN] hypervisor wallclock nudged; nudging TOD.

I actually get that closer to every 30 seconds - so annoying I actually commented out the line that generates it in case it blew away something more pertinent in dmesg (in my notes, the notice is generated at sys/i386/xen/clock.c:344). As you said, the clock seems to be working fine, and I don't feel like tracking down the actual cause (I'm not near familiar enough with Xen/FreeBSD). :)

Instead of commenting this line to halt these messages

Code:
# [XEN] hypervisor wallclock nudged; nudging TOD.

I decided to turn on the independent_wallclock in the dom0 side.
For my Centos dom0 I did the following actions (no need to restart):

Code:
# DOMO(CENTOS)
echo '/bin/echo 1 > /proc/sys/xen/independent_wallclock' >> /etc/rc.local
echo 1 > /proc/sys/xen/independent_wallclock

I also turned on the relative mibs on my domU:

Code:
#add the following lines to /etc/sysctl.conf
machdep.disable_rtc_set = 1
machdep.indipendent_wallclock = 1

and voila' the messages stopped appearing in my terminal.

I run ntp to synchronize my domU clock. Perfect :)

This is a greatly working FreeBSD paravirtualized guest (I use the tap:aio driver) in XEN and it consumes less memory than all other linux distros!
 
I've used this manual to build my own freebsd FreeBSD PV domU but every 1-2 days i I run into this kernel panic message.

Any advice on this?

Code:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x8b045029
fault code              = supervisor read, page not present
instruction pointer     = 0x21:0xc0176808
stack pointer           = 0x29:0xd593983c
frame pointer           = 0x29:0xd593983c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 1, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 40786 (sendmail)
[thread pid 40786 tid 100057 ]
Stopped at      strlen+0x8:     cmpb    $0,0(%edx)
 
A new one;

Code:
panic: mtx_lock() of spin mutex (null) @ /usr/src/sys/net/netisr.c:830
cpuid = 0
KDB: enter: panic
[thread pid 15756 tid 100049 ]
Stopped at      kdb_enter+0x3a: movl    $0,kdb_wh
y
 
fwaggle said:
Good news everyone!

Using a combination of these instructions, the NetBSD instructions, some good luck and some help from #bsdcode people, I've managed to get FreeBSD i386 to boot under Linode (which is XEN+PV).

The 8.0 images included above boot on linode reasonably trivially, but upgrading the kernel at all yields immediate panics. Thanks to nox @ #bsdcode, I discovered this morning that the panics many people are seeing under i386 paravirtualized Xen are caused by the new x86bios stuff that wasn't in 8.0-RELEASE... it's only called from the keyboard stuff (nox found that, and vesa which call it, but vesa isn't in the default i386 XEN kernel config). Commenting out atkbdc/atkbd/kbdmux from the default XEN kernel config results in an 8.1-RELEASE-p1 kernel that boots and doesn't seem to blow up at all, though I don't know which device or function is actually the buggy part and I'm not smart enough to track down where the issue is and fix it. :(

Since the process of getting FreeBSD on a linode isn't documented at all, anywhere, I'm working on a set of instructions which I'll post on my site, then mirror here and on the linode wiki. The instructions are still a work in progress, because each time I follow through my own notes, something is missing or goes wrong, so obviously I'm doing stuff from muscle memory and not documenting it, then forgetting it in subsequent tries.

I don't know what version of Xen linode are using, nor what they're using for a host, but this is a current FreeBSD that's working-for-me under Xen-PV which is a start. :D

Edit: Google-snacks:

Code:
panic: HYPERVISOR_update_va_mapping(((unsigned long)(va)), (pa |
0x002 | 0x001 | pgeflag | pmap_cache_bits(mode, 0)), UVMF_INVLPG| 
UVMF_ALL) < 0: /usr/src/sys/i386/xen/pmap.c:1269
cpuid = 0

This particular panic with i386 kernels >8.0-R can be fixed by removing atkbd (I removed atkbdc + kbdmux too, not sure if that's necessary though) from your kernel, then booting from the updated kernel.

Woo that fixed that problem. Now it's crashing at this point:

Code:
[XEN] xen_rtc_attach: attaching Hypervisor RTC clock
Timecounters tick every 10.000 msec
kernel trap 9 with interrupts disabled


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer     = 0x21:0xc0313a61
stack pointer           = 0x29:0xc21ffca0
frame pointer           = 0x29:0xc21ffca8
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 1, pres 1, def32 1, gran 1
processor eflags        = resume, IOPL = 0
current process         = 11 (idle: cpu0)
[thread pid 11 tid 100003 ]
Stopped at      outb_+0x6d1:    hlt
 
Ah and here's the backtrace
Code:
Tracing pid 11 tid 100003 td 0xc2308a00
outb_(1,c21ffcf8,c00fc7ee,1,0,...) at outb_+0x6d1
cpu_idle(1,0,c0355dd0,9dd,c2308a00,...) at cpu_idle+0x12
sched_idletd(0,c21ffd38,c034f58a,344,c2306aa0,...) at sched_idletd+0x23e
fork_exit(c00fc5b0,0,c21ffd38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc21ffd70, ebp = 0 ---
 
and fwaggle, do you mind uploading your kernel somewhere? Also, what hardware are you running on?

PS: Sorry for the triple post and where's the edit button?
 
I do realize this thread is over 1 year old, but FreeBSD 8.2-RC2 works almost flawlessly with Xen. Almost flawlessly being that, if you compiled the kernel on Intel hardware (CPU)...this will not work on AMD hardware (CPU). I don't know if it is the same vice versa, as I don't have an AMD64 FreeBSD box running (yet!).
 
Back
Top