Bhyve with ZVOL

sydney6 · Jan 15, 2014

Hello everyone,

I`m trying to start a bhyve VM (FreeBSD 9 Guest) with a ZVOL as a disk backend. I’ve downloaded Michael Dexter’s bhyve scripts from bhyve.org, prepared the host (.ko’s, network, ZVOL), changed DEVTYPE= and

Code:

VMDEV=zvol/zroot/vm0

Provisioned the VM. Running sh vm0 start returns the bootloader and exits with:

Code:

Booting...
vmload: vm0 appears to have loaded.
vmboot: zvol/zroot/vm0 does not exist! Exiting.

I’m on 10-RC5.

Anyone more success on running bhyve with ZVOL’s?

SirDice · Jan 15, 2014

Bhyve is still very much a work in progress. And there aren't a lot of developers on these forums. I would suggest contacting the mailinglist; freebsd-virtualization@freebsd.org

sydney6 · Jan 15, 2014

I already did, but I suppose they have/had more important things to do for the last weeks. Anyhow, since this wasn't generally a question about the FreeBSD Base, I didn’t know if I was right anyway.

http://lists.freebsd.org/pipermail/free ... 01806.html

Though, thanks for your answer, Sir. I guess, we’ll be reading more about this awesome addition to FreeBSD in the next weeks.

SirDice · Jan 15, 2014

I sure hope so, I very much like the possibilities of a proper FreeBSD hypervisor

usdmatt · Jan 16, 2014

What does your ZVOL look like in zfs list?

sydney6 · Jan 16, 2014

Code:

root@freebsd10:~ # zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot               5.92G  11.7G   144K  none
zroot/ROOT           777M  11.7G   144K  none
zroot/ROOT/default   777M  11.7G   777M  /
zroot/tmp            176K  11.7G   176K  /tmp
zroot/usr            576K  11.7G   144K  /usr
zroot/usr/home       144K  11.7G   144K  /usr/home
zroot/usr/ports      144K  11.7G   144K  /usr/ports
zroot/usr/src        144K  11.7G   144K  /usr/src
zroot/var           1.26M  11.7G   640K  /var
zroot/var/crash      148K  11.7G   148K  /var/crash
zroot/var/log        208K  11.7G   208K  /var/log
zroot/var/mail       144K  11.7G   144K  /var/mail
zroot/var/tmp        152K  11.7G   152K  /var/tmp
zroot/vm0           5.16G  16.8G    72K  -

The ZVOL device is shown under /dev/zvol/zroot/vm0 and was created via zfs create -V 5G zroot/vm0.

gesperon · Jan 31, 2014

I just downloaded the latest: http://bhyve.org/vmrc/vmrc.tar. After install I set up:

/usr/local/etc/vm.conf:

Code:

...
host_zpool="rpool"           # ZFS pool for VM zvols
...

/usr/local/vm/vm0/vm0.conf:

Code:

...
vm_dev_type="zvol"        # "img" for image, "zvol" or blank for other device
vm_device="vm0" 
...

and had to modify /usr/local/etc/rc.d/vm.

Code:

$  diff vm vm.orig
1011c1011
< 	elif [ ! -e "/dev/$vm_device" && ! -e "/dev/zvol/$host_zpool/$vm_device" ]; then
---
> 	elif [ ! -e "/dev/$vm_device" ]; then

and now:
/usr/local/etc/rc.d/vm provision vm0
/usr/local/etc/rc.d/vm start vm0
works.

EDIT: haven't had the chance to test all the other functionality, but I hope it helps

frankpeng · Feb 14, 2014

I have made bhyve working in FreeBSD-10.0. But I can not make the network working in a gateway server. I can ping from the host to the guest. But I cannot ping from the guest to the host.

Code:

root@frank:~ # ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
^C
--- 192.168.0.1 ping statistics ---
9 packets transmitted, 0 packets received, 100.0% packet loss
root@frank:~ # ifconfig
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 00:a0:98:eb:76:05
        inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

root@frank:~ # netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS         0        0 vtnet0
127.0.0.1          link#2             UH          0        0    lo0
192.168.0.0/24     link#1             U           0        0 vtnet0
192.168.0.4        link#1             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%lo0/32                     ::1                           U           lo0

Code:

root@www:/usr/home/peng # ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 54:04:a6:ef:ac:ef
        inet 209.141.128.208 netmask 0xffffffe0 broadcast 255.255.255.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
re1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 00:21:2f:2f:b3:61
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::221:2fff:fe2f:b361%re1 prefixlen 64 scopeid 0x2
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:ce:08:a0:12:00
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 200000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:96:93:00:00
        inet 209.141.128.208 netmask 0xffffff00 broadcast 209.141.128.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        Opened by PID 1556


PING 192.168.0.4 (192.168.0.4): 56 data bytes
64 bytes from 192.168.0.4: icmp_seq=0 ttl=64 time=0.230 ms
64 bytes from 192.168.0.4: icmp_seq=1 ttl=64 time=0.060 ms
64 bytes from 192.168.0.4: icmp_seq=2 ttl=64 time=0.111 ms
....


root@www:/usr/home/peng # netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            209.141.128.193    UGS         0     8162    re0
127.0.0.1          link#3             UH          0     5309    lo0
192.168.0.0/24     link#2             U           0    15917    re1
192.168.0.1        link#2             UHS         0        0    lo0
209.141.128.0/24   link#5             U           0        0   tap0
209.141.128.192/27 link#1             U           0        0    re0
209.141.128.208    link#1             UHS         1        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif          Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%re1/64                     link#2                        U           re1
fe80::221:2fff:fe2f:b361%re1      link#2                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff01::%re1/32                     fe80::221:2fff:fe2f:b361%re1  U           re1
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%re1/32                     fe80::221:2fff:fe2f:b361%re1  U           re1
ff02::%lo0/32                     ::1                           U           lo0
root@www:/usr/home/peng #

After I ping from the host to guest, the guest now is OK to ping host.
If I do not ping from host to guest, the guest won't be able to ping host.
Please help!

Bhyve with ZVOL

sydney6

SirDice

Administrator

sydney6

SirDice

Administrator

usdmatt

sydney6

gesperon

frankpeng