bhyve Passing through SAS controller disturbs/interrupts some virtual disks.

willyvmm · Mar 8, 2022

Host:
FreeBSD 13 (latest stable)

loader.conf

Code:

kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"

cryptodev_load="YES"
zfs_load="YES"
vmm_load="YES"

pptdevs="1/0/0 2/0/0"

Code:

01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
02:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

vm.conf

Code:

loader="grub"
cpu="4"
memory="12G"

debug="yes"

network0_type="virtio-net"
network0_switch="public"

disk0_type="virtio-blk"
disk0_name="disk0.img"

disk1_type="virtio-blk"
disk1_name="/dev/ada2p2"
disk1_dev="custom"

disk2_type="virtio-blk"
disk2_name="/dev/ada3p2"
disk2_dev="custom"

grub_run_partition="1"
grub_run_dir="/boot/grub"

uuid="7f12e80b-9591-11ec-a424-30d042f39767"
network0_mac="58:9c:fc:01:fb:f1"

#passthru0="1/0/0"

#passthru1="2/0/0"

bhyve_options="-s 15,virtio-9p,farmd=/mnt/farmd"

Guest:
Debian “bullseye” (latest stable)

The config is quite complicated and is working ... almost.

output of lsblk in guest:

Code:

vda    254:0    0    1G  0 disk
└─vda1 254:1    0 1023M  0 part /
vdb    254:16   0  8,4T  0 disk
└─vdb1 254:17   0  8,4T  0 part
vdc    254:32   0  8,4T  0 disk

Everything is working as expected.
vdb is /dev/ada2p2
vdc is /dev/ada3p2

But, when I enable passthru the SAS controller VDB and VDC stop working correctly. Data is not acessible, trying to create partitions generates errors in both host and guest:
host:

Code:

(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 d0 ff 7f 40 2f 08 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 d0 ff 7f 40 2f 08 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 d0 ff 7f 40 2f 08 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 1 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 d0 ff 7f 40 2f 08 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 0 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 58 e0 09 20 40 20 00 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 78 70 b0 20 40 20 00 00 05 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 d0 ff 7f 40 2f 08 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 58 e0 09 20 40 20 00 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 78 70 b0 20 40 20 00 00 05 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 50 20 20 0c 40 20 00 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 50 20 20 0c 40 20 00 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 1 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 50 20 20 0c 40 20 00 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 1 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 0 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 50 20 20 0c 40 20 00 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 0 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 50 20 20 0c 40 20 00 00 01 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fc ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fe ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 02 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 3 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fc ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fe ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 02 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 2 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fc ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 1 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fe ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 1 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 1 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 02 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 1 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fc ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 0 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fe ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 0 more tries remain
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 0 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 02 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Retrying command, 0 more tries remain
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fc ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 fe ff 40 ff 03 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted
(ada2:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 28 00 00 40 00 04 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted
(ada2:ahcich2:0:0:0): READ_FPDMA_QUEUED. ACB: 60 10 38 02 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Uncorrectable parity/CRC error
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted

and guest:

Code:

[   53.226274] blk_update_request: I/O error, dev vdb, sector 17976786856 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[   53.227572] Buffer I/O error on dev vdb, logical block 2247098357, lost async page write
[   64.658334] blk_update_request: I/O error, dev vdb, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[   64.659591] Buffer I/O error on dev vdb, logical block 0, lost async page write
[   67.983695] blk_update_request: I/O error, dev vdb, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[   67.985216] Buffer I/O error on dev vdb, logical block 0, lost async page write

After that partition is damaged and needs to be recreated.
Changing virtio-blk to ahci-hd changes only reported errors, does not help.

Removing the passthrough SAS controller restores control over the vdisks(partitions). Everything works FINE. (except for damaged partition)
Errors may suggest physical disk errors - Both disks were checked, including a short smart test. No error.

What is interesting, passing through the second card (10g net) does not interact with anything. Everything is working.

So... is it caused by the Host or The Guest?
Both are working stable except the broken vdisk interface.

Geezer · Mar 8, 2022

willyvmm said:
Host:
FreeBSD 13 (latest stable)

Why are you using stable?

willyvmm · Mar 8, 2022

It's 13.0 RELEASE

I wrote wrong, should be latest release. Same for Debian.

In meantime i have blacklisted all drivers related to this card in Debian. No effect.

covacat · Mar 8, 2022

try to reproduce the problem with a freebsd guest

Phishfry · Mar 8, 2022

covacat your analytical skills are very sharp. You are wise beyond your years.
That is an excellent debugging suggestion. I was thinking maybe BARS conflicting.

I want you on my trip to Mars..

willyvmm · Mar 8, 2022

FreeBSD guest with identical config has same problem.

covacat · Mar 8, 2022

PR filing time

sko · Mar 8, 2022

just for clarification and to make absolutely sure: /dev/ada2 and /dev/ada3 are NOT connected to the SAS controller you want to pass through to the VM?

willyvmm · Mar 8, 2022

sko said:
just for clarification and to make absolutely sure: /dev/ada2 and /dev/ada3 are NOT connected to the SAS controller you want to pass through to the VM?

That's correct.