2bd7d
![]() |
|
|
|
|
|||||||
| Howtos & FAQs (Moderated) Would you like to share some of your solutions for certain problems? Tips or tricks? Post here. All new topics are automatically moderated. |
![]() |
|
|
Thread Tools | Display Modes |
|
#1
|
|||
|
|||
|
Xen is virtualization software, which lets you run multiple operating systems on the same physical machine. The goal of this guide is to show you how to create a FreeBSD 8 installation that will run as a guest OS, without the need for a processor that supports hardware-assisted virtualization. It would go beyond the scope of this guide to explain how exactly Xen works or what the differences between PV and HVM are, suffice to say dom0 is the host OS and domU the guest OS. Although FreeBSD fully supports being a Xen domU, it is not yet very well documented and not suited for novice Xen users. Especially the lack of a Xen-aware installer or pre-built Xen-aware images make things more difficult. If you are a Xen novice, I recommend starting first with pre-built Linux images, and then trying a NetBSD domU installation, before proceeding with FreeBSD.
Preparing the host We will start with a normal Debian installation, we will need some free diskspace, beware that Linux names harddisks differently from FreeBSD, so if you have other OSes on this machine, be very careful you don't partition and format the wrong one. I recommend including the Desktop environment, having a graphical browser and the Synaptic Package Manager will be really useful. Now we need to install Xen itself. Debian mostly automates this process if you use apt-get or aptitude or synaptic. Installing xen-linux-system-2.6.26-2-xen should pull in all other needed dependencies (exact version may have changed by the time you read this guide). The required boot entries should be automatically added to GRUB, so in the next reboot we can start the Xen hypervisor. Preparing the guest There are no pre-built kernels and images we can use (but see below for mine), so we need to make our own. I'll assume you already have a FreeBSD installation somewhere, I used one existing inside a VirtualBox on a Windows 7 host. I have only tested with an i386 kernel and image, I am not certain whether amd64 is supported. The files we are going to create will eventually have to be transported to your dom0, so make sure you have some way to get them there. Go to a directory on your FreeBSD system with enough space. Create a 256 MB file that will become our image: # truncate -s 256M freebsd.imgCreate a virtual disk backed by this file: # mdconfig -f freebsd.imgIf this commands yields something other than md0, adjust all the following commands to that. Create a MBR and single slice on the virtual disk: # fdisk -BI md0Create a BSD-label inside our slice: # bsdlabel -wB md0s1Create a UFS2 filesystem with softupdates enabled: # newfs -U md0s1aIt is probably possible to directly create a filesystem inside the image and skip the MBR and label. Mount the newly created filesystem: # mount /dev/md0s1a /mntMake sure you have a fully populated 8.0 /usr/src tree: # csup -h nearby.mirror.FreeBSD.org -L 2 /usr/share/examples/cvsup/standard-supfileWe are about to build our world and kernel. By default this will populate /usr/obj, this will do no harm, but it is possible to specify an alternate location: # export MAKEOBJDIRPREFIX=/other/dircsh users use setenv Now it is time to start compiling, if you need multiple attempts to get things working, it is not necessary to do this step again each time (provided you did it correctly the first time): # make buildworld && make buildkernel KERNCONF=XENOur file-backed virtual disk should still be mounted, so now we can install to it: # export DESTDIR=/mnt && make installworld && make installkernel KERNCONF=XEN && cd etc && make distributionNow we need to modify a few files to prepare our installation for living in a Xen world. Add to /mnt/etc/fstab: Code:
/dev/xbd0 / ufs rw 1 1 Code:
xc0 "/usr/libexec/getty Pc" vt100 on secure # cp /mnt/boot/kernel/kernel /some/place/freebsd-kernelNow we can unmount and destroy the virtual disk: # umount /mnt# mdconfig -d -u md0The results of all our labour are now in freebsd.img and freebsd-kernel. Put these somewhere safe and reachable. Configuring on the dom0 Go back to your dom0 installation, reboot into Xen if you hadn't already. Put the image and kernel we created somewhere on the dom0. They don't have to be in the same directory. Check whether your Xen is working: # xm listNow create /etc/xen/freebsd and use the following configuration: Code:
kernel = "/virt/freebsd-8.0p2-i386-xen-domu-kernel" memory = 512 name = "freebsd" vif = [ '' ] disk = [ 'file:/virt/freebsd-8.0p2-i386-xen-domu.img,hda,w' ] extra = "boot_verbose=1" extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a" extra += ",kern.hz=100" Also edit /etc/xen/xend-config.sxp to enable bridged networking: Code:
#(network-script network-dummy) (network-script network-bridge) # xm create -c freebsdThe -c option will directly attach to the console output, alternative you can attach seperately: # xm console freebsdAt this point you should see the regular output of a booting FreeBSD and eventually the login prompt. If you can login, you have succeeded in creating a FreeBSD 8 Xen domU. How to proceed Of course all of this was just a small test installation not suited for real use. It's probably possible to increase the size of the image-file and growfs to a larger installation, but file-backed virtual disks aren't very efficient. If you want to make serious use of virtual FreeBSD installations, creating real partitions is the way to go. Let us know if you succesfully set this up, I am especially interesting in knowing whether GPT and ZFS will work in a Xen domU. Known problems ifconfig_xn0="DHCP" will not work, run dhclient or configure manually References This guide was heavily based on http://wiki.freebsd.org/AdrianChadd/XenImages and http://wiki.freebsd.org/AdrianChadd/XenHackery Closing words I started out about a week ago trying to figure out how to run FreeBSD in Xen. At first I was disappointed because I had assumed that since FreeBSD has full domU support, it would be relatively easy to do. Documentation was hard to find and much of it was outdated and being a total novice to Xen did not help either. Luckily I did not give up and soon become more acquainted with Xen terminology, which made it easier to understand Adrian Chadd's notes on the Wiki. I had already decided I did not want a UFS1 image, but a full simulated disk with an MBR, disklabel, UFS2 and softupdates, so I went that route. For simplicity I also decided to go for file-backed disk instead of LVM and to not use pygrub or pvgrub. I haven't yet extensively used and tested my FreeBSD domU, the moment I got it to actually boot (I made many mistakes at first) I started writing this guide. Attached are the config, kernel and image file I created. This is an old image, you probably don't want to use it. Beware that sshd and dhclient are enabled, an account xenguest with password "password" exists and root is still password-less, so it is not a secure installation, but only meant for testing. config kernel (gzipped, 1.8M) image (gzipped, 83M) mirror Last edited by Aprogas; February 26th, 2012 at 21:03. Reason: specify that image is old |
| The Following 17 Users Say Thank You to Aprogas For This Useful Post: | ||
bsd4me (July 27th, 2010), erikano (October 23rd, 2011), graudeejs (January 13th, 2010), hbokh (May 19th, 2010), hydra (July 5th, 2010), ian-nai (January 26th, 2011), Jochem (March 8th, 2010), krutik (February 11th, 2010), nervous (February 26th, 2012), nicedream (January 21st, 2010), nickwithers (March 7th, 2010), nl0pvm (November 5th, 2010), rbelk (January 14th, 2010), SaveTheRbtz (June 4th, 2010), t0ly (February 10th, 2010), vermaden (January 13th, 2010), vtypal (October 13th, 2010) | ||
|
#2
|
|||
|
|||
|
Hi Aprogas,
first of all: Thanks for all your effort to create that rar how to and also for uploading significant files. I tried your how to several times now [ with an i386 version ] but unfortunatley am I always getting a kernel panic when I'm about booting the virtual FreeBSD. No matter what image I used (I even tryed the actual pfsense which is based on 7.2) plus I tryed your kernel and image as well - but the result was always the same.Code:
xm create -c /FreeBSD-i38.cfg Code:
xm list It must be a Kernel issue?! OR the only thing I could think of is that my Xen might be incompatible with the FreeBSD DomU project cause I'm using NetBSD 5.0.2 with the actual ported Xen 3.3.2 - but as far as I know should that 3.3.2 do the job?! ... I even saw the Login once for a millisecond ... but then it broke again ... ;( Any Ideas? P.S. I'll post the last boot information as soon as possible. Regards, Leander |
|
#3
|
|||
|
|||
|
This won't help you alot Leander, but I just tried with a plain Debian install with Xen and the image and kernel provided here, and it's working without any problems. So thats Xen 3.2.1 on Debian 5.0
I'm up to compiling the kernel myself now and creating an image. Thanks for this tutorial! |
|
#4
|
|||
|
|||
|
I'd like to post my attempts at creating a FreeBSD 8.1-RC PV domU.
Hardware: CPU = AMD Athlon II Regor 250 DomU is Debian 5.0.5 (Lenny), linux kernel 2.6.26-2, xen 3.2-1. I installed FreeBSD 8.1-RC on Virtualbox, created a dev environment, pulled down sources, compiled the kernel with KERNCONF=XEN. I also installed FreeBSD to an image by creating a FreeBSD HVM (which works fine btw, using the loader option hw.clflush_disable=1). I then tried to use the XEN kernel as the PV kernel (i.e. kernel='/path/to/xen/kernel'), but I get this kernel panic: Code:
panic: HYPERVISOR_update_va_mapping(((unsigned long)(va)), (pa | 0x002 | 0x001 | pgeflag | pmap_cache_bits(mode, 0)), UVMF_INVLPG| UVMF_ALL) < 0: /usr/src/sys/i386/xen/pmap.c:1269 cpuid = 0 KDB: enter: panic [thread pid 0 tid 0 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why Is the FreeBSD PV domU more favorable on Intel HW? I can switch if needed, but would prefer not to. |
|
#5
|
|||
|
|||
|
And here is someone else having the exact same problem on (recent) AMD hardware:
http://lists.freebsd.org/pipermail/f...er/000323.html The kernel panic there is the same, even under FreeBSD-CURRENT (9) and Xen 3.4. Starting to lose hope that FreeBSD PV domU will be supported any time soon, judging by the lack of response to the various kernel panic bug reports
|
|
#6
|
|||
|
|||
|
I downloaded your kernel and image files (from a mirror). They worked without issue.
System: Lenovo R61 ThinkPad Intel Core2 Duo Debian Lenny (5.0.4) Stable (2.6.26-2-xen-amd64) Xen 3.2.1 (AMD64) Next, I will follow your howto and try to build my own. Thank you! |
|
#7
|
|||
|
|||
|
blackOut - just wanted to let you know I'm experience the same problem on Intel hardware ( http://www.freebsd.org/cgi/query-pr.cgi?pr=148262 ) with 8.1
If I combine the disk image I made with the kernel attached to the OP, it does boot fine. I suppose next thing to try is go back to 8.0 |
|
#8
|
|||
|
|||
|
I've just successfully built a kernel and disk image with RELENG_8_0 . I guess Xen kernel is broken in 8.1
|
|
#9
|
|||
|
|||
|
Thank you @Aprogas . Excellent guide.
Just for the record if you try to create your own images and kernel from a amd64 release you should simply make the following changes (the kernel configuration file is called XENHVM on amd64) : Code:
make buildworld && make buildkernel KERNCONF=XENHVM Code:
export DESTDIR=/mnt && make installworld && make installkernel KERNCONF=XENHVM && cd etc && make distribution |
|
#10
|
|||
|
|||
|
I have spent/wasted a lot of time trying to get FreeBSD run as a DomU under NetBSD Dom0. My NetBSD is the amd64 flavour of release 5.0.2. I use Xen 3.3.2 (installed via pkgsrc, NetBSD's answer to FreeBSD's ports).
I tried the i386 images presented here, and they yield immediate kernel panics. I have compiled my own i386 kernels (using KERNCONF=XEN) and they work no better. I have compiled amd64 images, using KERNCONF=XENHVM and made futile attempts at getting them to boot. In the apparent absence of documentation (at least such documentation that is not obsolete and/or sloppy) I've had problems figuring out how to boot this image. I started with /usr/pkg/share/examples/xen/hvm, but it is not clear how to point to the FreeBSD disk image from there; I made it the disk device. But it is clear that this is not the correct approach, since there is no reason that the system should be able to read FreeBSD's file system. I have tried various version of FreeBSD 8.x, RELEASEs and RELENGs during this struggle. My question is: Does anybody here have a working FreeBSD of any sort under a NetBSD dom0, and is willing to explain exactly how to get it to run without triggering bugs? (My ultimate goal is setting up a large number of DomU on all my test systems, but I have thus far only succeeded with NetBSD i386 and NetBSD amd64... My main development platform is FreeBSD, so it that fails, my test array project fails.) Last edited by DutchDaemon; September 21st, 2010 at 23:37. |
|
#11
|
||||
|
||||
|
Good news everyone!
Using a combination of these instructions, the NetBSD instructions, some good luck and some help from #bsdcode people, I've managed to get FreeBSD i386 to boot under Linode (which is XEN+PV). The 8.0 images included above boot on linode reasonably trivially, but upgrading the kernel at all yields immediate panics. Thanks to nox @ #bsdcode, I discovered this morning that the panics many people are seeing under i386 paravirtualized Xen are caused by the new x86bios stuff that wasn't in 8.0-RELEASE... it's only called from the keyboard stuff (nox found that, and vesa which call it, but vesa isn't in the default i386 XEN kernel config). Commenting out atkbdc/atkbd/kbdmux from the default XEN kernel config results in an 8.1-RELEASE-p1 kernel that boots and doesn't seem to blow up at all, though I don't know which device or function is actually the buggy part and I'm not smart enough to track down where the issue is and fix it. ![]() Since the process of getting FreeBSD on a linode isn't documented at all, anywhere, I'm working on a set of instructions which I'll post on my site, then mirror here and on the linode wiki. The instructions are still a work in progress, because each time I follow through my own notes, something is missing or goes wrong, so obviously I'm doing stuff from muscle memory and not documenting it, then forgetting it in subsequent tries. I don't know what version of Xen linode are using, nor what they're using for a host, but this is a current FreeBSD that's working-for-me under Xen-PV which is a start. ![]() Edit: Google-snacks: Code:
panic: HYPERVISOR_update_va_mapping(((unsigned long)(va)), (pa | 0x002 | 0x001 | pgeflag | pmap_cache_bits(mode, 0)), UVMF_INVLPG| UVMF_ALL) < 0: /usr/src/sys/i386/xen/pmap.c:1269 cpuid = 0 Last edited by fwaggle; September 26th, 2010 at 16:42. Reason: formatting |
|
#12
|
|||
|
|||
|
I now have FreeBSD 6.4 amd64 as well as FreeBSD 6.4 i386 running under a NetBSD Dom0. Unlike 7.x and 8.x installation and operation are smooth.
Code:
kernel = "/usr/pkg/lib/xen/boot/hvmloader"
builder='hvm'
memory = 256
vcpus = 1
name = "freebsd-64"
vif = [ 'mac=00:16:3e:00:01:64, bridge=bridge0, type=ioemu, model=ne2k_pci' ]
device_model = '/usr/pkg/libexec/qemu-dm'
disk = [
'file:/home/xen/freebsd/64/6.4/disk,ioemu:hda,w',
# 'file:/home/tege/6.4-RELEASE-amd64-disc1.iso,ioemu:hdc:cdrom,r'
]
boot='c'
#boot='d'
pae = 1
usb=1
usbdevice='tablet'
It seems that using a mac address not starting with 00:16:3e makes NetBSD/Xen confused (sic!). It seems that the default simulated network card is unreliable, therefore the need for the ne2k_pci moniker. I have made some progress with 8.1 amd64 too, with a custom kernel. It boots, but the installation is not yet robust. |
|
#13
|
|||
|
|||
|
Quote:
Do you get the following notification about every 3 minutes in your terminal? Code:
# [XEN] hypervisor wallclock nudged; nudging TOD. Last edited by DutchDaemon; October 7th, 2010 at 22:51. |
|
#14
|
||||
|
||||
|
Quote:
He and a developer on the freebsd-xen mailing list asked if I could try boot a -CURRENT kernel with the "fix" reverted, and it exploded on boot in a different place, so I went back to my *kbd*-less kernel and it's been running fine ever since. I was going to post the backtrace from -CURRENT to the freebsd-xen mailing list, but it seems like every time I go to post on a FreeBSD mailing list my posts I stick my foot in my mouth, and laziness eventually won out. It's spontaneously blown up on me once when I left nload running (nload has garbage in the interface name too, which might be related, but I can't be bothered tracking the issue down), but other than that I have a Mumble server on that box that's been chugging along for a couple weeks now. Quote:
![]() I said I'd mirror my notes on this forum and the Linode wiki, but honestly I keep going back and editing them and putting them in multiple places just seems like way too much effort, so I published them on one of my sites and I'll mirror them when they're a little less turbulent: http://www.hungryhacker.com/os/freebsd-on-linode/ Hope it helps someone.
|
|
#15
|
|||
|
|||
|
Quote:
Code:
# [XEN] hypervisor wallclock nudged; nudging TOD. For my Centos dom0 I did the following actions (no need to restart): Code:
# DOMO(CENTOS) echo '/bin/echo 1 > /proc/sys/xen/independent_wallclock' >> /etc/rc.local echo 1 > /proc/sys/xen/independent_wallclock Code:
#add the following lines to /etc/sysctl.conf machdep.disable_rtc_set = 1 machdep.indipendent_wallclock = 1 I run ntp to synchronize my domU clock. Perfect ![]() This is a greatly working FreeBSD paravirtualized guest (I use the tap:aio driver) in XEN and it consumes less memory than all other linux distros! Last edited by DutchDaemon; October 13th, 2010 at 22:06. Reason: writing style: http://forums.freebsd.org/showthread.php?t=18043 |
|
#16
|
|||
|
|||
|
I've used this manual to build my own freebsd FreeBSD PV domU but every 1-2 days i I run into this kernel panic message.
Any advice on this? Code:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x8b045029
fault code = supervisor read, page not present
instruction pointer = 0x21:0xc0176808
stack pointer = 0x29:0xd593983c
frame pointer = 0x29:0xd593983c
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 1, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 40786 (sendmail)
[thread pid 40786 tid 100057 ]
Stopped at strlen+0x8: cmpb $0,0(%edx)
Last edited by DutchDaemon; November 4th, 2010 at 16:07. |
|
#17
|
|||
|
|||
|
A new one;
Code:
panic: mtx_lock() of spin mutex (null) @ /usr/src/sys/net/netisr.c:830 cpuid = 0 KDB: enter: panic [thread pid 15756 tid 100049 ] Stopped at kdb_enter+0x3a: movl $0,kdb_wh |
|
#18
|
|||
|
|||
|
Quote:
Code:
[XEN] xen_rtc_attach: attaching Hypervisor RTC clock
Timecounters tick every 10.000 msec
kernel trap 9 with interrupts disabled
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x21:0xc0313a61
stack pointer = 0x29:0xc21ffca0
frame pointer = 0x29:0xc21ffca8
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 1, pres 1, def32 1, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (idle: cpu0)
[thread pid 11 tid 100003 ]
Stopped at outb_+0x6d1: hlt
|
|
#19
|
|||
|
|||
|
Ah and here's the backtrace
Code:
Tracing pid 11 tid 100003 td 0xc2308a00 outb_(1,c21ffcf8,c00fc7ee,1,0,...) at outb_+0x6d1 cpu_idle(1,0,c0355dd0,9dd,c2308a00,...) at cpu_idle+0x12 sched_idletd(0,c21ffd38,c034f58a,344,c2306aa0,...) at sched_idletd+0x23e fork_exit(c00fc5b0,0,c21ffd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc21ffd70, ebp = 0 --- |
|
#20
|
|||
|
|||
|
and fwaggle, do you mind uploading your kernel somewhere? Also, what hardware are you running on?
PS: Sorry for the triple post and where's the edit button? |
|
#21
|
||||
|
||||
|
Read your sign-up email. We send it for a reason.
__________________
FreeBSD Forums: Information for New Members | FreeBSD Forums Rules FreeBSD Resources: The FreeBSD Handbook | Manuals | FAQ | Wiki Before you post: How to ask questions the smart way If you must know .. So, what does an Administrator/Moderator do? ---> Do not PM me with FreeBSD questions. I do not work here. <--- |
|
#22
|
|||
|
|||
|
I see, 10 posts required. I guess I'm just used to forums automerging posts
.
|
|
#23
|
|||
|
|||
|
I've fixed this in HEAD, and will MFC some time in the next few weeks.
|
|
#24
|
|||
|
|||
|
I do realize this thread is over 1 year old, but FreeBSD 8.2-RC2 works almost flawlessly with Xen. Almost flawlessly being that, if you compiled the kernel on Intel hardware (CPU)...this will not work on AMD hardware (CPU). I don't know if it is the same vice versa, as I don't have an AMD64 FreeBSD box running (yet!).
|
|
#25
|
|||
|
|||
|
I've created a mirror of the files at http://www.cran.org.uk/~brucec/freebsd/xen/ .
Could I copy the instructions to other formats such as the Wiki and possibly the Handbook? The current instructions have a broken link so I'd like to replace them. |
![]() |
| Tags |
| virtualization, xen |
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| HOWTO: create custom FreeBSD usb bootable flash | graudeejs | Howtos & FAQs (Moderated) | 17 | August 11th, 2010 11:17 |
| Installing FreeBSD 8.0 as Xen DomU? | mr_daemon | Installing & Upgrading | 5 | December 25th, 2009 22:19 |
| freebsd 8 on xen or xenserver 5.5 | jack | Installing & Upgrading | 3 | December 14th, 2009 22:57 |
| HOWTO: create a GELI password/data safe | aragon | Howtos & FAQs (Moderated) | 1 | November 24th, 2009 08:56 |
| Xen Supoprt in FreeBSD | susanth | General | 1 | January 25th, 2009 23:54 |