Bhyve - trying to mount root

IPTRACE · Apr 28, 2017

Sometime I encounter the problem with boot UFS in bhyve VM.
When I start or restart the guest I think in my mind it starts but just It doesn't start completely.

Connection to the console has no effects due to the "black" screen with no console messages.
But when I start bhyve VM with console from scratch I see the following.

Code:

uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
nvme cam probe device init
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/vtbd0p2 [rw]...

..and nothing.
I cannot send kill signal to VM so I have to kill process from the host.
After killing from the host, bhyve VM starts OK.
Both host and guests are FreeBSD 11p10. The problem was on FreeBSD 10 as well.

How can I find out where is the problem?

 bhyvectl --vm=fw.local --destroy

bhyveload -m 1G -c /dev/nmdm100A -d /vm/fw.local.img fw.local

bhyve -c 4 -m 1G -A -H -P -s 0:0,hostbridge -s 1:0,virtio-net,tap100 -s 2:0,virtio-blk,/vm/fw.local.img -s 31,lpc -l com1,/dev/nmdm100A fw.local &

IPTRACE · Apr 29, 2017

I have all VM data when system stopped.
Maybe It can be useful.

Code:

ID  Length      Name
0   2048MB      sysmem
Address     Length      Segment     Offset      Prot  Flags
0           2048MB      sysmem      0           RWX
efer[0]         0x0000000000000d01
cr0[0]          0x000000008005003b
cr3[0]          0x00000000021c9001
cr4[0]          0x00000000000626e0
dr7[0]          0x0000000000000000
rsp[0]          0xfffffe007b7998f0
rip[0]          0xffffffff80facc7e
rax[0]          0x00000000a28f7e75
rbx[0]          0xfffff80003517000
rcx[0]          0x0000000000000000
rdx[0]          0xfffff800fed000f0
rsi[0]          0x0000000000000040
rdi[0]          0xfffff80003517000
rbp[0]          0xfffffe007b7998c0
r8[0]           0xfffffffe00000000
r9[0]           0xfffffe007af710a0
r10[0]          0x0000000000000000
r11[0]          0x0000000016f600b3
r12[0]          0x0000000000000761
r13[0]          0x000002bb4ab17efd
r14[0]          0x0000000000000760
r15[0]          0x0000000000000000
rflags[0]       0x0000000000000046
ds desc[0]      0x0000000000000000/0xffffffff/0x0000c0f3
es desc[0]      0x0000000000000000/0xffffffff/0x0000c0f3
fs desc[0]      0x0000000000000000/0xffffffff/0x0000c0f3
gs desc[0]      0xffffffff81e38000/0xffffffff/0x0000a093
ss desc[0]      0x0000000000000000/0xffffffff/0x0000a093
cs desc[0]      0x0000000000000000/0xffffffff/0x0000a09b
tr desc[0]      0xffffffff81e2af10/0x00002068/0x0000008b
ldtr desc[0]    0x0000000000000000/0xffffffff/0x0001c000
gdtr[0]         0xffffffff81e31750/0x00000067
idtr[0]         0xffffffff81d844c0/0x00000fff
cs[0]           0x0020
ds[0]           0x003b
es[0]           0x003b
fs[0]           0x0013
gs[0]           0x0028
ss[0]           0x0028
tr[0]           0x0048
ldtr[0]         0x0000
cr0_mask[0]             0xffffffff60000020
cr0_shadow[0]           0x0000000080000031
cr4_mask[0]             0xffffffffffe8b800
cr4_shadow[0]           0x0000000000000020
cr3_target_count[0]     0x0000000000000000
cr3_target0[0]          0x0000000000000000
cr3_target1[0]          0x0000000000000000
cr3_target2[0]          0x0000000000000000
cr3_target3[0]          0x0000000000000000
pinbased_ctls[0]        0x00000000000000bf
procbased_ctls[0]       0x00000000f52065f2
procbased_ctls2[0]      0x0000000000001323
gla[0]          0xfffff800fed000f0
gpa[0]          0x00000000fed000f0
entry_interruption_info[0]      0x0000000000000000
tpr_threshold[0]        0x0000000000000000
instruction_error[0]    0x0000000000000000
exit_ctls[0]            0x000000000033effb
entry_ctls[0]           0x00000000000093fb
host_pat[0]             0x0001050600070406
host_cr0[0]             0x000000008005003b
host_cr3[0]             0x00000007addf92c8
host_cr4[0]             0x00000000001726e0
host_rip[0]             0xffffffff821f8570
host_rsp[0]             0xfffffe0028b65700
vmcs_pointer[0] 0xffffffffffffffff
vmcs_exit_interruption_info[0]  0x00000000800000fb
vmcs_exit_interruption_error[0] 0x0000000000000000
vmcs_guest_interruptibility[0]  0x0000000000000000
vmcs_exit_inst_length[0]        0x00000002
vmcs_exit_qualification[0]      0x0000000000000380
x2apic_state[0] 0
eptp[0]         0x00000007adb4c05e
exception_bitmap[0]     0x40000
io_bitmap_a[0]  0
io_bitmap_b[0]  0
tsc_offset[0]   0x0000000000000000
msr_bitmap[0]           0x59bb64000
MSR_TSC             [0]         R-
MSR_EFER            [0]         RW
MSR_STAR            [0]         RW
MSR_LSTAR           [0]         RW
MSR_CSTAR           [0]         RW
MSR_SF_MASK         [0]         RW
MSR_FSBASE          [0]         RW
MSR_GSBASE          [0]         RW
MSR_KGSBASE         [0]         RW
MSR_SYSENTER_CS_MSR [0]         RW
MSR_SYSENTER_ESP_MSR[0]         RW
MSR_SYSENTER_EIP_MSR[0]         RW
vpid[0]         0x01a1
guest_pat[0]            0x0000000000000000
guest_sysenter_cs[0]    0
guest_sysenter_sp[0]    0
guest_sysenter_ip[0]    0
exit_reason[0]  0x1
rtc nvram[000]: 0x11
rtc time 0x590531f3: Sun Apr 30 00:38:11 2017
Capability "hlt_exit" is set on vcpu 0
Capability "mtrap_exit" is not set on vcpu 0
Capability "pause_exit" is set on vcpu 0
Capability "unrestricted_guest" is not set on vcpu 0
Capability "enable_invpcid" is set on vcpu 0
active cpus:     0, 1, 2, 3
suspended cpus:  (none)
pending:        n/a
current:        n/a
vcpu0 stats:
number of times rdmsr was intercepted           4
number of times wrmsr was intercepted           1
number of monitor trap exits                    0
number of times pause was intercepted           451675
vm exits due to interrupt window opening        0
vm exits due to nmi window opening              0
number of times in/out was intercepted          273131
number of times cpuid was intercepted           284
vm exits due to nested page fault               24077
vm exits for instruction emulation              83131
number of vm exits for unknown reason           0
number of times astpending at exit              125
number of times idle requested at exit          5600
number of vm exits handled in userspace         848723
number of times rendezvous pending at exit      0
number of vm exits due to exceptions            0
vcpu total runtime                              8592864109
number of ticks vcpu was idle                   695834
number of NMIs delivered to vcpu                0
number of ExtINTs delivered to vcpu             0
EOI without any in-service interrupt            0
error interrupts generated by vlapic            0
timer interrupts generated by vlapic            10707
corrected machine check interrupts generated by vlapic  0
lvts triggered[0]                               0
lvts triggered[1]                               0
lvts triggered[2]                               0
lvts triggered[3]                               0
lvts triggered[4]                               0
lvts triggered[5]                               0
lvts triggered[6]                               0
ipis sent to vcpu[0]                            0
ipis sent to vcpu[1]                            732
ipis sent to vcpu[2]                            715
ipis sent to vcpu[3]                            759
ipis sent to vcpu[4]                            0
ipis sent to vcpu[5]                            0
ipis sent to vcpu[6]                            0
ipis sent to vcpu[7]                            0
ipis sent to vcpu[8]                            0
ipis sent to vcpu[9]                            0
ipis sent to vcpu[10]                           0
ipis sent to vcpu[11]                           0
ipis sent to vcpu[12]                           0
ipis sent to vcpu[13]                           0
ipis sent to vcpu[14]                           0
ipis sent to vcpu[15]                           0
Resident memory                                 99835904
Wired memory                                    0
Number of vpid invalidations saved              132
Number of vpid invalidations done               3040
vcpu migration across host cpus                 3172
total number of vm exits                        892242
vm exits due to external interrupt              1424
number of times hlt was intercepted             11103
number of times %cr access was intercepted      2

Or maybe It's a FreeBSD problem in VM somehow?!

IPTRACE · May 10, 2017

I was thinking about the problem and I'm confused about disk labelling.
Some of my VM has the same GPTID usidng glabel status .

Is that the reason of my problem?

Bhyve - trying to mount root

IPTRACE

IPTRACE

IPTRACE