Solved Disk gets detached then "Periph destroyed" according to /var/run/dmesg.boot (Azure - Data Disks)

Preetpal · Jun 10, 2017

I'm not sure if this is a configuration issue, a driver issue, or a hardware issue. I am running FreeBSD on Microsoft Azure and am trying to attach multiple "data disks" to a Basic A1 VM (which allows for 2 data disks (in addition to a disk for the OS), has 1 CPU core, and 1.75GB of ram) for use in a ZFS zpool. Basically I attached 2 data disks to the FreeBSD 11 virtual machine and they do not show up in the /dev directory as device nodes (I am assuming that they should appear there (as this is the behaviour I encountered when attaching additional disks when running FreeBSD in VirtualBox and running Ubuntu (Linux) in Azure)). When I check the /var/run/dmesg.boot file, I find that the drives were detected, then detached, and then were recipients of associated "Periph destroyed" messages. I have pasted the /var/run/dmesg.boot file contents in this message. Any help would be appreciated as I am not sure why the additional drives are not appearing as device nodes under the /dev directory.

Code:

Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p9 #0: Tue Apr 11 08:48:40 UTC 2017
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
SRAT: Ignoring memory at addr 150200000
SRAT: Ignoring memory at addr 1000000000
VT(vga): text 80x25
Hyper-V Version: 10.0.14393 [SP0]
  Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0xed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz (2411.65-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2328<BMI1,AVX2,BMI2,ERMS,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1879048192 (1792 MB)
avail memory = 1681596416 (1603 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <VRTUAL MICROSFT>
random: unblocking device.
ioapic0: Changing APIC ID to 0
ioapic0 <Version 1.1> irqs 0-23 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff8101d970, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <VRTUAL MICROSFT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 20000000 vs 1fffffff
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0
vgapci0: Boot video device
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
vmbus0: <Hyper-V Vmbus> irq 5,7 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcbfff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
nvme cam probe device init
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <Msft Virtual CD/ROM 1.0> Removable CD-ROM SPC-3 SCSI device
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from ufs:/dev/label/rootfs [rw]...
hvet0: <Hyper-V event timer> on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
storvsc1: <Hyper-V IDE Storage Interface>(probe0:blkvsc0:0:0:0): storvsc scsi_status = 2
 on vmbus0
da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
hvheartbeat0: <Hyper-V Heartbeat Service> on vmbus0
(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
hvkvp0: <Hyper-V KVP Service>da1 at blkvsc1 bus 0 scbus3 target 1 lun 0
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 40960MB (83886080 512 byte sectors)
 on vmbus0
hvshutdown0: <Hyper-V Shutdown Service> on vmbus0
hvtimesync0: <Hyper-V Time Synch Service> on vmbus0
storvsc2: <Hyper-V SCSI Storage Interface> on vmbus0
storvsc3: <Hyper-V SCSI Storage Interface> on vmbus0
hn0: <Hyper-V Network Interface> on vmbus0
hn0: LRO: entry count 128
(probe0:storvsc3:0:0:0): storvsc scsi_status = 2
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1047552MB (2145386496 512 byte sectors)
(probe0:storvsc3:0:0:1): storvsc scsi_status = 2
da3 at storvsc3 bus 0 scbus5 target 0 lun 1
da3: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1047552MB (2145386496 512 byte sectors)
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> detached
da3 at storvsc3 bus 0 scbus5 target 0 lun 1
da3: <Msft Virtual Disk 1.0> detached
hn0: unknown status 1073872902 received
hn0: unknown status 1073872902 received
hn0: hv send offload request succeeded
hn0: 1 TX ring, 1 RX ring
hn0: Ethernet address: 00:0d:3a:f9:5d:11
hn0: TSO: 65517/31/4096
vmbus0: device scan, probe and attach done
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Timecounter "TSC-low" frequency 1205826506 Hz quality 800
(da2:storvsc3:0:0:0): Periph destroyed
(da3:storvsc3:0:0:1): Periph destroyed
hn0: link state changed to UP
pflog0: promiscuous mode enabled

Preetpal · Jun 11, 2017

I solved this issue by:

Detaching the data disks that had I added to the virtual machine. I then deleted those data disks since they were no longer being used.
Stopping the virtual machine so that it was stopped and de-allocated.
Attaching a set of new data disks to the virtual machine.

I suspect the issue was caused by the fact that I had added the data disks to a live running machine (although this works for Ubuntu on Azure). I am just happy that it works and that I can now move forward

.

As proof, I have pasted the contents of the /var/run/dmesg.boot file below:

Code:

Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p9 #0: Tue Apr 11 08:48:40 UTC 2017
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
VT(vga): text 80x25
Hyper-V Version: 6.3.9600 [SP18]
  Features=0xe7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0x17b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz (2396.21-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2328<BMI1,AVX2,BMI2,ERMS,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1879048192 (1792 MB)
avail memory = 1681580032 (1603 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <VRTUAL MICROSFT>
random: unblocking device.
ioapic0: Changing APIC ID to 0
ioapic0 <Version 1.1> irqs 0-23 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff8101d970, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <VRTUAL MICROSFT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0
vgapci0: Boot video device
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
vmbus0: <Hyper-V Vmbus> irq 5,7 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcbfff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
nvme cam probe device init
Trying to mount root from ufs:/dev/label/rootfs [rw]...
hvet0: <Hyper-V event timer> on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
storvsc0: <Hyper-V IDE Storage Interface> on vmbus0
storvsc1: <Hyper-V IDE Storage Interface> on vmbus0
(probe0:blkvsc0:0:0:0): storvsc scsi_status = 2
da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
hvheartbeat0: <Hyper-V Heartbeat Service> on vmbus0
(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
hvkvp0: <Hyper-V KVP Service> on vmbus0
da1 at blkvsc1 bus 0 scbus3 target 1 lun 0
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 40960MB (83886080 512 byte sectors)
hvshutdown0: <Hyper-V Shutdown Service> on vmbus0
hvtimesync0: <Hyper-V Time Synch Service> on vmbus0
hn0: <Hyper-V Network Interface> on vmbus0
hn0: LRO: entry count 128
hn0: unknown status 1073872902 received
hn0: unknown status 1073872902 received
hn0: hv send offload request succeeded
hn0: 1 TX ring, 1 RX ring
hn0: Ethernet address: 00:0d:3a:f9:5d:11
hn0: TSO: 65517/31/4096
storvsc2: <Hyper-V SCSI Storage Interface> on vmbus0
storvsc3: <Hyper-V SCSI Storage Interface> on vmbus0
vmbus0: device scan, probe and attach done
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Timecounter "TSC-low" frequency 1198105965 Hz quality 800
(probe0:storvsc3:0:0:0): storvsc scsi_status = 2
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1047552MB (2145386496 512 byte sectors)
(probe0:storvsc3:0:0:1): storvsc scsi_status = 2
da3 at storvsc3 bus 0 scbus5 target 0 lun 1
da3: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-3 SCSI device
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1047552MB (2145386496 512 byte sectors)
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
hn0: link state changed to UP
pflog0: promiscuous mode enabled

gnoma · Jun 12, 2017

Hello,

In such cases there's the camcontrol(8) tool which will help you troubleshoot the issue without rebooting and removing the disks.

Preetpal · Jun 12, 2017

gnoma said:
Hello,

In such cases there's the camcontrol(8) tool which will help you troubleshoot the issue without rebooting and removing the disks.

I tried camcontrol devlist and nothing showed up at the time. Do you have any other suggestions?

Even after I rebooted the machine with the original disks, they still never became available.

I will look more into this issue when I have the time.

gnoma · Jun 16, 2017

Hello Preetpal,

You can use camcontrol rescan all or camcontrol reset all and then try devlist again.

Just to be sure if this is disk controller issue, or geom issue. If camcontrol shows no disks attached to the controller, or shows no controller then it may be a driver issue.
But if camcontrol shows/starts showing after rescan/reset the controller and disks than it may be a geom issue.

Just out of curiosity if I had this issue I think it's worth playing around. However if you don't have the time to bother with this issue, or you are unable to reproduce I think it's OK to count it solved and hope it doesn't reappear

Preetpal · Jun 17, 2017

gnoma said:
You can use camcontrol rescan all or camcontrol reset all and then try devlist again.

Just to be sure if this is disk controller issue, or geom issue. If camcontrol shows no disks attached to the controller, or shows no controller then it may be a driver issue.
But if camcontrol shows/starts showing after rescan/reset the controller and disks than it may be a geom issue.

Just out of curiosity if I had this issue I think it's worth playing around. However if you don't have the time to bother with this issue, or you are unable to reproduce I think it's OK to count it solved and hope it doesn't reappear

So curiosity got the best of me, and I looked into the issue again. I recreated the issue and attempted to resolve it with your device. The following makes me suspect there is a driver issue based on your response.

So on a new virtual machine (Basic A1 VM (which allows for 2 data disks (in addition to a disk for the OS), has 1 CPU core, and 1.75GB of ram)), with no disks attached, running ls /dev/ yields:

Code:

acpi            auditpipe       console         cuau0.lock      da0p1           devctl          fd0             hv_appvss_dev   kbd0            label           netmap          pass2           reroot          sysmouse        ttyu1.init      ttyv3           ttyv8           urandom
apm             bpf             consolectl      cuau1           da0p2           devctl2         fido            hv_fsvss_dev    kbd1            log             nfslock         pci             sndstat         ttyu0           ttyu1.lock      ttyv4           ttyv9           usbctl
apmctl          bpf0            ctty            cuau1.init      da0p3           devstat         full            hv_kvp_dev      kbdmux0         mdctl           null            psm0            stderr          ttyu0.init      ttyv0           ttyv5           ttyva           xpt0
atkbd0          bpsm0           cuau0           cuau1.lock      da1             dumpdev         geom.ctl        hv_tsc          klog            mem             pass0           pts             stdin           ttyu0.lock      ttyv1           ttyv6           ttyvb           zero
audit           cd0             cuau0.init      da0             da1s1           fd              gptid           io              kmem            midistat        pass1           random          stdout          ttyu1           ttyv2           ttyv7           ufssuspend

Without rebooting, but after adding two data disks to live running VM, running ls /dev/ yields (the two data disks appear as da2 and da3):

Code:

acpi            bpf             ctty            cuau1.lock      da1s1           dumpdev         gptid           kbd0            log             null            pci             stderr          ttyu0.lock      ttyv2           ttyv8           usbctl
apm             bpf0            cuau0           da0             da2             fd              hv_appvss_dev   kbd1            mdctl           pass0           psm0            stdin           ttyu1           ttyv3           ttyv9           xpt0
apmctl          bpsm0           cuau0.init      da0p1           da3             fd0             hv_fsvss_dev    kbdmux0         mem             pass1           pts             stdout          ttyu1.init      ttyv4           ttyva           zero
atkbd0          cd0             cuau0.lock      da0p2           devctl          fido            hv_kvp_dev      klog            midistat        pass2           random          sysmouse        ttyu1.lock      ttyv5           ttyvb
audit           console         cuau1           da0p3           devctl2         full            hv_tsc          kmem            netmap          pass3           reroot          ttyu0           ttyv0           ttyv6           ufssuspend
auditpipe       consolectl      cuau1.init      da1             devstat         geom.ctl        io              label           nfslock         pass4           sndstat         ttyu0.init      ttyv1           ttyv7           urandom

After rebooting, running ls /dev/ yields (note that da2 and da3 are now missing):

Code:

acpi            auditpipe       console         cuau0.lock      da0p1           devctl          fd0             hv_appvss_dev   kbd0            label           netmap          pass2           reroot          sysmouse        ttyu1.init      ttyv3           ttyv8           urandom
apm             bpf             consolectl      cuau1           da0p2           devctl2         fido            hv_fsvss_dev    kbd1            log             nfslock         pci             sndstat         ttyu0           ttyu1.lock      ttyv4           ttyv9           usbctl
apmctl          bpf0            ctty            cuau1.init      da0p3           devstat         full            hv_kvp_dev      kbdmux0         mdctl           null            psm0            stderr          ttyu0.init      ttyv0           ttyv5           ttyva           xpt0
atkbd0          bpsm0           cuau0           cuau1.lock      da1             dumpdev         geom.ctl        hv_tsc          klog            mem             pass0           pts             stdin           ttyu0.lock      ttyv1           ttyv6           ttyvb           zero
audit           cd0             cuau0.init      da0             da1s1           fd              gptid           io              kmem            midistat        pass1           random          stdout          ttyu1           ttyv2           ttyv7           ufssuspend

And then finally running the commands you suggested yields errors for one command and neither command resolves the issue. I have pasted the command line session below:

Code:

root@disktest:~ # camcontrol rescan all
Re-scan of bus 0 was successful
Re-scan of bus 1 was successful
Re-scan of bus 2 was successful
Re-scan of bus 3 was successful
Re-scan of bus 4 was successful
Re-scan of bus 5 was successful
root@disktest:~ # cam
camcontrol camdd
root@disktest:~ # camcontrol devlist
<Msft Virtual CD/ROM 1.0>          at scbus1 target 0 lun 0 (cd0,pass0)
<Msft Virtual Disk 1.0>            at scbus2 target 0 lun 0 (pass1,da0)
<Msft Virtual Disk 1.0>            at scbus3 target 1 lun 0 (pass2,da1)
root@disktest:~ # ls /dev/
acpi            console         da0p1           fd0             kbd0            netmap          reroot          ttyu1.init      ttyv8
apm             consolectl      da0p2           fido            kbd1            nfslock         sndstat         ttyu1.lock      ttyv9
apmctl          ctty            da0p3           full            kbdmux0         null            stderr          ttyv0           ttyva
atkbd0          cuau0           da1             geom.ctl        klog            pass0           stdin           ttyv1           ttyvb
audit           cuau0.init      da1s1           gptid           kmem            pass1           stdout          ttyv2           ufssuspend
auditpipe       cuau0.lock      devctl          hv_appvss_dev   label           pass2           sysmouse        ttyv3           urandom
bpf             cuau1           devctl2         hv_fsvss_dev    log             pci             ttyu0           ttyv4           usbctl
bpf0            cuau1.init      devstat         hv_kvp_dev      mdctl           psm0            ttyu0.init      ttyv5           xpt0
bpsm0           cuau1.lock      dumpdev         hv_tsc          mem             pts             ttyu0.lock      ttyv6           zero
cd0             da0             fd              io              midistat        random          ttyu1           ttyv7
root@disktest:~ # camcontrol reset all
Reset of bus 0 was successful
Reset of bus 1 was successful
Reset of bus 2 returned error 0x6
Reset of bus 3 returned error 0x6
Reset of bus 4 returned error 0x6
Reset of bus 5 returned error 0x6
root@disktest:~ # camcontrol devlist
<Msft Virtual CD/ROM 1.0>          at scbus1 target 0 lun 0 (cd0,pass0)
<Msft Virtual Disk 1.0>            at scbus2 target 0 lun 0 (pass1,da0)
<Msft Virtual Disk 1.0>            at scbus3 target 1 lun 0 (pass2,da1)
root@disktest:~ # ls
.cshrc          .k5login        .login          .profile
root@disktest:~ # ls /dev/
acpi            auditpipe       console         cuau0.lock      da0p1           devctl          fd0             hv_appvss_dev   kbd0            label           netmap          pass2           reroot          sysmouse        ttyu1.init      ttyv3           ttyv8           urandom
apm             bpf             consolectl      cuau1           da0p2           devctl2         fido            hv_fsvss_dev    kbd1            log             nfslock         pci             sndstat         ttyu0           ttyu1.lock      ttyv4           ttyv9           usbctl
apmctl          bpf0            ctty            cuau1.init      da0p3           devstat         full            hv_kvp_dev      kbdmux0         mdctl           null            psm0            stderr          ttyu0.init      ttyv0           ttyv5           ttyva           xpt0
atkbd0          bpsm0           cuau0           cuau1.lock      da1             dumpdev         geom.ctl        hv_tsc          klog            mem             pass0           pts             stdin           ttyu0.lock      ttyv1           ttyv6           ttyvb           zero
audit           cd0             cuau0.init      da0             da1s1           fd              gptid           io              kmem            midistat        pass1           random          stdout          ttyu1           ttyv2           ttyv7           ufssuspend

Preetpal · Jun 17, 2017

Pasting the contents of the /var/run/dmesg.boot file of that test VM below (note I rebooted the VM twice (da2 and da3 do not appear as devices after rebooting); errors for da2 and da3 can be seen right before second reboot sequence initiation):

Code:

Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-BETA1 #0 r319728: Fri Jun  9 02:21:09 UTC 2017
    root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
SRAT: Ignoring memory at addr 0x150200000
SRAT: Ignoring memory at addr 0x1000000000
VT(vga): text 80x25
Hyper-V Version: 10.0.14393 [SP0]
  Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0xed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz (2400.48-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2328<BMI1,AVX2,BMI2,ERMS,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1879048192 (1792 MB)
avail memory = 1685307392 (1607 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <VRTUAL MICROSFT>
random: unblocking device.
ioapic0: Changing APIC ID to 0
ioapic0 <Version 1.1> irqs 0-23 on motherboard
Timecounter "TSC-low" frequency 1200237810 Hz quality 800
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f5a190, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <VRTUAL MICROSFT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 20000000 vs 1fffffff
vmbus0: <Hyper-V Vmbus> on pcib0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0
vgapci0: Boot video device
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
vmbus_res0: <Hyper-V Vmbus Resource> irq 5,7 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcbfff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
nvme cam probe device init
vmbus0: version 3.0
hvet0: <Hyper-V event timer> on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
storvsc0: <Hyper-V IDE> on vmbus0
storvsc1: <Hyper-V IDE> on vmbus0
hvkbd0: <Hyper-V KBD> on vmbus0
hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
hvkvp0: <Hyper-V KVP> on vmbus0
hvshutdown0: <Hyper-V Shutdown> on vmbus0
hvtimesync0: <Hyper-V Timesync> on vmbus0
hvvss0: <Hyper-V VSS> on vmbus0
hvtimesync0: RTT
storvsc2: <Hyper-V SCSI> on vmbus0
storvsc3: <Hyper-V SCSI> on vmbus0
hn0: <Hyper-V Network Interface> on vmbus0
hn0: Ethernet address: 00:0d:3a:f9:6a:8b
hn0: link state changed to UP
(probe0:blkvsc0:0:0:0): storvsc scsi_status = 2
(probe3:blkvsc1:0:1:0): storvsc scsi_status = 2
da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
da1 at blkvsc1 bus 0 scbus3 target 1 lun 0
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 40960MB (83886080 512 byte sectors)
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <Msft Virtual CD/ROM 1.0> Removable CD-ROM SPC-3 SCSI device
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: 1MB (576 2048 byte sectors)
Trying to mount root from ufs:/dev/label/rootfs [rw]...
(probe0:storvsc3:0:0:1): storvsc scsi_status = 2
da2 at storvsc3 bus 0 scbus5 target 0 lun 1
da2: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1047552MB (2145386496 512 byte sectors)
(probe0:storvsc3:0:0:0): storvsc scsi_status = 2
da3 at storvsc3 bus 0 scbus5 target 0 lun 0
da3: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1047552MB (2145386496 512 byte sectors)
(probe0:storvsc3:0:0:1): storvsc scsi_status = 2
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 done
All buffers synced.
Uptime: 5m40s
Rebooting...
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-BETA1 #0 r319728: Fri Jun  9 02:21:09 UTC 2017
    root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
SRAT: Ignoring memory at addr 0x150200000
SRAT: Ignoring memory at addr 0x1000000000
VT(vga): text 80x25
Hyper-V Version: 10.0.14393 [SP0]
  Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0xed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz (2403.11-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2328<BMI1,AVX2,BMI2,ERMS,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1879048192 (1792 MB)
avail memory = 1685307392 (1607 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <VRTUAL MICROSFT>
random: unblocking device.
ioapic0: Changing APIC ID to 0
ioapic0 <Version 1.1> irqs 0-23 on motherboard
Timecounter "TSC-low" frequency 1201554504 Hz quality 800
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f5a190, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <VRTUAL MICROSFT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 20000000 vs 1fffffff
vmbus0: <Hyper-V Vmbus> on pcib0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0
vgapci0: Boot video device
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
vmbus_res0: <Hyper-V Vmbus Resource> irq 5,7 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcbfff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
nvme cam probe device init
vmbus0: version 3.0
hvet0: <Hyper-V event timer> on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
storvsc0: <Hyper-V IDE> on vmbus0
storvsc1: <Hyper-V IDE> on vmbus0
hvkbd0: <Hyper-V KBD> on vmbus0
hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
hvkvp0: <Hyper-V KVP> on vmbus0
hvshutdown0: <Hyper-V Shutdown> on vmbus0
hvtimesync0: <Hyper-V Timesync> on vmbus0
hvtimesync0: RTT
hvvss0: <Hyper-V VSS> on vmbus0
storvsc2: <Hyper-V SCSI> on vmbus0
storvsc3: <Hyper-V SCSI> on vmbus0
hn0: <Hyper-V Network Interface> on vmbus0
hn0: Ethernet address: 00:0d:3a:f9:6a:8b
hn0: link state changed to UP
(probe0:blkvsc0:0:0:0): storvsc scsi_status = 2
(probe3:blkvsc1:0:1:0): storvsc scsi_status = 2
(probe6:storvsc3:0:0:0): storvsc scsi_status = 2
(probe7:storvsc3:0:0:1): storvsc scsi_status = 2
da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
da1 at blkvsc1 bus 0 scbus3 target 1 lun 0
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 40960MB (83886080 512 byte sectors)
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <Msft Virtual CD/ROM 1.0> Removable CD-ROM SPC-3 SCSI device
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from ufs:/dev/label/rootfs [rw]...
(probe0:blkvsc0:0:0:0): storvsc scsi_status = 2
(probe1:blkvsc1:0:1:0): storvsc scsi_status = 2
(probe0:storvsc3:0:0:0): storvsc scsi_status = 2
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1047552MB (2145386496 512 byte sectors)
(probe0:storvsc3:0:0:1): storvsc scsi_status = 2
da3 at storvsc3 bus 0 scbus5 target 0 lun 1
da3: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1047552MB (2145386496 512 byte sectors)
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> detached
da3 at storvsc3 bus 0 scbus5 target 0 lun 1
da3: <Msft Virtual Disk 1.0> detached
g_access(918): provider da2 has error
g_access(918): provider da2 has error
g_access(918): provider da2 has error
g_access(918): provider da2 has error
(da2:storvsc3:0:0:0): Periph destroyed
(da3:storvsc3:0:0:1): Periph destroyed
(noperiph:blkvsc0:0:-1:ffffffff): bus reset not supported.
(noperiph:blkvsc1:0:-1:ffffffff): bus reset not supported.
(noperiph:storvsc2:0:-1:ffffffff): bus reset not supported.
(noperiph:storvsc3:0:-1:ffffffff): bus reset not supported.
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 1 0 done
All buffers synced.
Uptime: 27m30s
Rebooting...
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-BETA1 #0 r319728: Fri Jun  9 02:21:09 UTC 2017
    root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
SRAT: Ignoring memory at addr 0x150200000
SRAT: Ignoring memory at addr 0x1000000000
VT(vga): text 80x25
Hyper-V Version: 10.0.14393 [SP0]
  Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0xed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz (2410.47-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2328<BMI1,AVX2,BMI2,ERMS,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1879048192 (1792 MB)
avail memory = 1685307392 (1607 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <VRTUAL MICROSFT>
random: unblocking device.
ioapic0: Changing APIC ID to 0
ioapic0 <Version 1.1> irqs 0-23 on motherboard
Timecounter "TSC-low" frequency 1205232823 Hz quality 800
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f5a190, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <VRTUAL MICROSFT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 20000000 vs 1fffffff
vmbus0: <Hyper-V Vmbus> on pcib0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0
vgapci0: Boot video device
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
vmbus_res0: <Hyper-V Vmbus Resource> irq 5,7 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcbfff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
nvme cam probe device init
vmbus0: version 3.0
hvet0: <Hyper-V event timer> on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
storvsc0: <Hyper-V IDE> on vmbus0
storvsc1: <Hyper-V IDE> on vmbus0
hvkbd0: <Hyper-V KBD> on vmbus0
hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
hvkvp0: <Hyper-V KVP> on vmbus0
hvshutdown0: <Hyper-V Shutdown> on vmbus0
hvtimesync0: <Hyper-V Timesync> on vmbus0
hvtimesync0: RTT
hvvss0: <Hyper-V VSS> on vmbus0
storvsc2: <Hyper-V SCSI> on vmbus0
storvsc3: <Hyper-V SCSI> on vmbus0
hn0: <Hyper-V Network Interface> on vmbus0
hn0: Ethernet address: 00:0d:3a:f9:6a:8b
hn0: link state changed to UP
(probe0:blkvsc0:0:0:0): storvsc scsi_status = 2
(probe3:blkvsc1:0:1:0): storvsc scsi_status = 2
(probe6:storvsc3:0:0:0): storvsc scsi_status = 2
(probe7:storvsc3:0:0:1): storvsc scsi_status = 2
da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
da1 at blkvsc1 bus 0 scbus3 target 1 lun 0
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 40960MB (83886080 512 byte sectors)
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <Msft Virtual CD/ROM 1.0> Removable CD-ROM SPC-3 SCSI device
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from ufs:/dev/label/rootfs [rw]...

Preetpal · Jun 17, 2017

One last session on the VM:

Code:

root@disktest:~ # camcontrol devlist
<Msft Virtual Disk 1.0>            at scbus2 target 0 lun 0 (pass0,da0)
<Msft Virtual Disk 1.0>            at scbus3 target 1 lun 0 (pass1,da1)
root@disktest:~ # camcontrol rescan all
Re-scan of bus 0 was successful
Re-scan of bus 1 was successful
Re-scan of bus 2 was successful
Re-scan of bus 3 was successful
Re-scan of bus 4 was successful
Re-scan of bus 5 was successful
root@disktest:~ # camcontrol devlist
<Msft Virtual Disk 1.0>            at scbus2 target 0 lun 0 (pass0,da0)
<Msft Virtual Disk 1.0>            at scbus3 target 1 lun 0 (pass1,da1)
root@disktest:~ # camcontrol reset all
Reset of bus 0 was successful
Reset of bus 1 was successful
Reset of bus 2 returned error 0x6
Reset of bus 3 returned error 0x6
Reset of bus 4 returned error 0x6
Reset of bus 5 returned error 0x6
root@disktest:~ # camcontrol devlist
<Msft Virtual Disk 1.0>            at scbus2 target 0 lun 0 (pass0,da0)
<Msft Virtual Disk 1.0>            at scbus3 target 1 lun 0 (pass1,da1)
root@disktest:~ # cat /var/run/dmesg.boot
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-BETA1 #0 r319728: Fri Jun  9 02:21:09 UTC 2017
    root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
SRAT: Ignoring memory at addr 0x150200000
SRAT: Ignoring memory at addr 0x1000000000
VT(vga): text 80x25
Hyper-V Version: 10.0.14393 [SP0]
  Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0xed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz (2394.96-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2328<BMI1,AVX2,BMI2,ERMS,NFPUSG>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"
real memory  = 1879048192 (1792 MB)
avail memory = 1685307392 (1607 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <VRTUAL MICROSFT>
random: unblocking device.
ioapic0: Changing APIC ID to 0
ioapic0 <Version 1.1> irqs 0-23 on motherboard
Timecounter "TSC-low" frequency 1197477660 Hz quality 800
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
taskqgroup_adjust failed cnt: 1 stride: 1 mp_ncpus: 1 smp_started: 0
Timecounter "Hyper-V-TSC" frequency 10000000 Hz quality 3000
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80f5a190, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <VRTUAL MICROSFT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 20000000 vs 1fffffff
vmbus0: <Hyper-V Vmbus> on pcib0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0
vgapci0: Boot video device
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
vmbus_res0: <Hyper-V Vmbus Resource> irq 5,7 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcbfff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
nvme cam probe device init
vmbus0: version 3.0
hvet0: <Hyper-V event timer> on vmbus0
Event timer "Hyper-V" frequency 10000000 Hz quality 1000
storvsc0: <Hyper-V IDE> on vmbus0
storvsc1: <Hyper-V IDE> on vmbus0
(probe0:blkvsc0:0:0:0): storvsc scsi_status = 2
hvkbd0: <Hyper-V KBD> on vmbus0
hvheartbeat0: <Hyper-V Heartbeat> on vmbus0
(probe2:blkvsc1:0:1:0): storvsc scsi_status = 2
hvkvp0: <Hyper-V KVP> on vmbus0
da0 at blkvsc0 bus 0 scbus2 target 0 lun 0
da0: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 30720MB (62914560 512 byte sectors)
hvshutdown0: <Hyper-V Shutdown> on vmbus0
da1 at blkvsc1 bus 0 scbus3 target 1 lun 0
da1: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 40960MB (83886080 512 byte sectors)
hvtimesync0: <Hyper-V Timesync> on vmbus0
hvtimesync0: RTT
hvvss0: <Hyper-V VSS> on vmbus0
storvsc2: <Hyper-V SCSI> on vmbus0
storvsc3: <Hyper-V SCSI> on vmbus0
hn0: <Hyper-V Network Interface> on vmbus0
(probe0:storvsc3:0:0:0): storvsc scsi_status = 2
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1047552MB (2145386496 512 byte sectors)
(probe0:storvsc3:0:0:1): storvsc scsi_status = 2
da3 at storvsc3 bus 0 scbus5 target 0 lun 1
da3: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1047552MB (2145386496 512 byte sectors)
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> detached
da3 at storvsc3 bus 0 scbus5 target 0 lun 1
da3: <Msft Virtual Disk 1.0> detached
hn0: Ethernet address: 00:0d:3a:f9:6a:8b
hn0: link state changed to UP
Trying to mount root from ufs:/dev/label/rootfs [rw]...
(da2:storvsc3:0:0:0): Periph destroyed
(da3:storvsc3:0:0:1): Periph destroyed

gnoma · Jun 17, 2017

Hello,

Thank you for the detailed output investigation.

Seems to me like a driver issue indeed. However I wonder what is the difference between the OS disks and the disks 2 and 3.

If they are attached to the same controller they would be affected from the driver issue too. This would be very strange if all 4 disks are attached to the same controller, but only 2 of them appears.

I'm not very sure about azure, but I know that vmware offers different type of disk controllers like scsi, sata, sas... etc.
Could you please check if this is also the case in azure? And if this is true you may try different controllers.

Thanks, iddqd, idkfa

Preetpal · Jun 18, 2017

Well one difference between the OS disks and the other 2 disks (data disks) is that I am using them in ZFS ZPool versus the OS disk which is mounted using UFS.

Azure does have different types of disk controllers, like for example they SSD backed data disks. I'll try something with a different controller later when I have time.

gnoma · Jun 19, 2017

aloha,

There shouldn't be a difference between using disks with ZFS and UFS. You can actually not use them at all and they should still be available as raw devices.

I am out of ideas and I would suggest opening a bug report where people smarter than me would take a look

Cheers

reddy · Sep 2, 2017

Just came here to mention that I had the exact same issue (I was even using the same VM size - A1), and the steps you mentioned for the Azure portal worked

Preetpal · Sep 2, 2017

reddy said:
Just came here to mention that I had the exact same issue (I was even using the same VM size - A1), and the steps you mentioned for the Azure portal worked

Hi, you may want to check out the following thread (https://forums.freebsd.org/threads/61335/) since it involves an issue (a solvable) I encountered with my VM immediately afterwards. Basically make sure to scrub your zpool after you create it to possibly avoid some problems. I have been running the same VM for a few months and ZFS has been running fine on Azure.

SamB · Sep 24, 2017

Hi, I know this is solved but just wanted to say this can be recreated using FreeSBD 11.1 as a Generation 2 guest in Hyper-V on Windows Server 2016 or Windows 10 when the disks are attached to additional virtual SCSI controllers.

There are no issues if the disks are all attached to the same controller as hinted by gnoma above.

tarcisiocjr · Jan 5, 2018

Just to inform that the problem persists on Azure. Apparently I had this problem after maintenance reboot to fix the Meltdown and Spectre bug.

I tried to recreate a new virtual machine and reconnect the disk with no luck.

dmesg.log

Code:

da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> Fixed Direct Access SPC-2 SCSI device
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 20480MB (41943040 512 byte sectors)
da2 at storvsc3 bus 0 scbus5 target 0 lun 0
da2: <Msft Virtual Disk 1.0> detached
(da2:storvsc3:0:0:0): Periph destroyed

Code:

$ sudo camcontrol devlist
<Msft Virtual Disk 1.0>            at scbus2 target 0 lun 0 (pass0,da0)
<Msft Virtual Disk 1.0>            at scbus3 target 1 lun 0 (pass1,da1)

gnoma · Jan 5, 2018

Hello,

Again I'm note sure about Azure but probably there is analog of the vmware "events" tab on the VM settings.
It shows the events concerning this virtual machine like power on/off, changes in the VM hardware, migration between hosts/disks, agent inside the VM events... etc

If there is such place could you please check for any events/entries recently after you power on the VM?

Also above I suggested opening a bug but I don't know if such is opened and is it in the FreeBSD Bugzilla, or in the Azure's analog.

Preetpal · Jan 16, 2018

tarcisiocjr said:
Just to inform that the problem persists on Azure. Apparently I had this problem after maintenance reboot to fix the Meltdown and Spectre bug.

Hi,

I never encountered this issue on rebooting the VM just now (although I had it stopped and de-allocated during the maintenance reboot). If you could give more info about your system, somebody might be able to help you.

Preetpal · Jan 16, 2018

SamB said:
Hi, I know this is solved but just wanted to say this can be recreated using FreeSBD 11.1 as a Generation 2 guest in Hyper-V on Windows Server 2016 or Windows 10 when the disks are attached to additional virtual SCSI controllers.

There are no issues if the disks are all attached to the same controller as hinted by gnoma above.

Just curious, was FreeBSD running when you added the disks to the additional virtual SCSI controllers. If not, you may want to try to add the disks when the FreeBSD VM is stopped.

zentex · Jan 18, 2018

tarcisiocjr said:
Just to inform that the problem persists on Azure. Apparently I had this problem after maintenance reboot to fix the Meltdown and Spectre bug.

I tried to recreate a new virtual machine and reconnect the disk with no luck.

da2: <Msft Virtual Disk 1.0> detached
(da2:storvsc3:0:0:0): Periph destroyed[/CODE]
[/CODE]

I have an open ticket about this exact problem that happened right after the maint. Last night at 3am, after no contact for 11 days, one of their "Azure Linux Support Escalation Engineers" called me to tell me it was an "OS problem" and I needed to contact the vendor (FBSD). I know it's not the OS, it was running just fine and I had rebooted the FBSD11.1p4 VM when I updated it and for another reason prior to the maint.

In my t/s steps I spun up a new FBSD VM (from the image MSFT provides in the marketplace) and tried attaching the 2 disks I already had. Same issue. I then tried another new VM and created a new 10GB disk. Same effect.

My VM is unusable. One of the disks is /usr...of all things. I'm pretty sure that I'm going to get nowhere and FBSD's time in Azure is now over. I got tired of waiting and made my own custom 100GB VM and hope I never need more space.

If you have an open ticket with them, I'd love to throw it at my support contact to fuel the fire.

Preetpal · Jan 18, 2018

zentex said:
In my t/s steps I spun up a new FBSD VM (from the image MSFT provides in the marketplace) and tried attaching the 2 disks I already had. Same issue. I then tried another new VM and created a new 10GB disk. Same effect.

My VM is unusable. One of the disks is /usr...of all things. I'm pretty sure that I'm going to get nowhere and FBSD's time in Azure is now over. I got tired of waiting and made my own custom 100GB VM and hope I never need more space.

Did you try adding the disks when the VM was stopped and de-allocated? It never worked for me otherwise.

mrst3v3n · Feb 14, 2018

Worked with Azure to resolve this issue, had to rebuild the kernel following the instructions below.

Download the following kernel release to your freebsd azure vm using git clone:
- # git clone https://github.com/freebsd/freebsd.git -b releng/11.1 or
- # git clone https://github.com/freebsd/freebsd.git -b stable/11 (depending upon what you want )
Next compile :
- # cd /freebsd ( I downloaded to root directory therefore /)
- # make -j8 buildkernel KERNCONF=GENERIC
Install the kernel:
- #make installkernel
Reboot

Next, we need to manually edit the file sys/cam/cam_periph.c and make the following changes:

--- a/sys/cam/cam_periph.c
+++ b/sys/cam/cam_periph.c
@@ -1942,8 +1942,10 @@ cam_periph_error(union ccb *ccb, cam_flags camflags,
* Let peripheral drivers know that this
* device has gone away.
*/
+#if 0
xpt_async(AC_LOST_DEVICE, newpath, NULL);
xpt_free_path(newpath);
+#endif
}
}

Note: The link for the patch : https://github.com/dcui/freebsd/commit/bab9d9449a16a8a33f0beb2392d76a2805e0b27d

After that, rebuild the kernel, but this time with noclean=yes flag:
- # make -j8 buildkernel KERNCONF=GENERIC NOCLEAN=YES
Install the new kernel:
- #make installkernel
Reboot

Preetpal · Feb 14, 2018

mrst3v3n said:
--- a/sys/cam/cam_periph.c
+++ b/sys/cam/cam_periph.c
@@ -1942,8 +1942,10 @@ cam_periph_error(union ccb *ccb, cam_flags camflags,
* Let peripheral drivers know that this
* device has gone away.
*/
+#if 0
xpt_async(AC_LOST_DEVICE, newpath, NULL);
xpt_free_path(newpath);
+#endif
}
}

This is a step in the right direction. Personally, I would probably avoid this patch if I were planning on removing disks at any point in time from a running virtual machine (I could be wrong though).

Dexuan Cui · Mar 13, 2018

I reported a bug here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226583
In the bug, I proposed a fix and I'm working to fully test it.

Preetpal · Mar 14, 2018

Dexuan Cui As mentioned earlier in this thread, I was able to workaround the bug by adding the disks to the VM while it was stopped and deallocated. After using the workaround to get the disks attached to the VM, I ran into some mysterious ZFS logging issues that I suspect may be related to this issue (https://forums.freebsd.org/threads/...anged-every-minute-should-i-be-worried.61335/). I also shared my script (https://forums.freebsd.org/threads/...-minute-should-i-be-worried.61335/post-353237) that I used to set up the ZFS pool with PostgreSQL on Azure (which I still have running on Azure today). You may be able to use that script for your testing purposes. Again, thanks for your work.

Dexuan Cui · Mar 14, 2018

Thanks Preetpal! I'm not sure how your workaround can work, but anyway I'm glad it works for you.

As for me, I tried your workaround (add the disks to the VM while it was stopped and deallocated), and it didn't work for me.

It looks some people (e.g. tarcisiocjr) said the issue persisted for them as well.
I hope my fix in the bug can help them: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226583