Solved USB 3.0 disks not recognized by Windows if passed through as slots.

ziomario

ziomario

Hello.

I'm going to make another question about the bhyve behavior,because I want to understand if I'm dealing with a bug or with a regular method if working. So,please be patient with me. What I want to achieve is to pass thru two of my NTFS "formatted" disks to a Windows 11 VM,but without pass them thru using the USB controller. I'm using :

Code: 
FreeBSD marietto 13.0-RELEASE FreeBSD 13.0-RELEASE #5 n244809-dff3dead3734: Wed Feb 23 13:16:32 CET
2022     marietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

I've configured bhyve like this :

Code: 
bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,virtio-blk,/dev/da4p2 \
-s 3,virtio-blk,/dev/da2p1 \
-s 8,virtio-net,tap4 \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5904,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \                          
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm4 < /dev/null & sleep 2 && vncviewer 0:4

These are the NTFS disks that I would like to see inside the Windows 11 guest os :
Code: 
-s 2,virtio-blk,/dev/da4p2 \
-s 3,virtio-blk,/dev/da2p1 \


=>         34  19532873661  da4  GPT  (9.1T)
           34        32734    1  ms-reserved  (16M)
32768  19532838912    2  ms-basic-data  (9.1T)
19532871680         2015       - free -  (1.0M)


=>         34  23437705149  da2  GPT  (11T)
           34         2014       - free -  (1.0M)
2048  23437701120    1  ms-basic-data  (11T)
23437703168         2015       - free -  (1.0M)

as u can see I've used the virtio-blk driver,so inside the Windows 11 VM I've installed the lastest version of the virtio drivers. You can see what's the problem giving a look at the images below. They are 0 byte large,so they aren't recognized by Windows 11. Is this a bug or what ? I've also used the ahci-hd driver but I've got the same result. They are USB 3.0 disks.

1.jpg


2.jpg
 

Attachments

  • 2.jpg
    2.jpg
    183 KB · Views: 240
Last edited:
Code: 
Assertion failed: (n >= 2 && n <= BLOCKIF_IOV_MAX + 2), function pci_vtblk_proc, file /usr/src/usr.s
bin/bhyve/pci_virtio_block.c, line 324.
 
I tried with :

Code: 
-s 2,virtio-scsi,/dev/da4p2 \
-s 3,virtio-scsi,/dev/da2p1 \

and I have also installed the virtio-scsi driver,that has been accepted by windows,but the disks aren't recognized :

Screenshot_2022-03-13_11-23-54.jpg


Screenshot_2022-03-13_11-10-16.jpg
 

Attachments

  • Screenshot_2022-03-13_11-23-54.jpg
    Screenshot_2022-03-13_11-23-54.jpg
    255.1 KB · Views: 164
Last edited:
I tried by virtualizing ubuntu with bhyve using the virtio-scsi driver because I wanted to see the error that it produced. Im not able to understood what's the error,but I see that the USB disks aren't recognized.

Code: 
bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,virtio-scsi,/dev/da4p2 \
-s 3,virtio-scsi,/dev/da2p1 \
-s 8,virtio-net,tap2 \
-s 9,virtio-9p,sharename=/ \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5902,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm2 < /dev/null & sleep 2 && vncviewer 0:2

dmesg :

[    0.343597] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver
[    0.343715] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    0.343825] virtio-pci 0000:00:08.0: virtio_pci: leaving for legacy driver
[    0.343935] virtio-pci 0000:00:09.0: virtio_pci: leaving for legacy driver

[    0.909102] ahci 0000:00:01.0: version 3.0
[    0.909348] ahci 0000:00:01.0: SSS flag set, parallel bus scan disabled
[    0.909439] ahci 0000:00:01.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x1 impl SATA mode
[    0.909441] ahci 0000:00:01.0: flags: 64bit ncq sntf ilck stag pm led clo pio slum part sxs apst
[    0.910538] xhci_hcd 0000:00:1e.0: xHCI Host Controller
[    0.910542] xhci_hcd 0000:00:1e.0: new USB bus registered, assigned bus number 1
[    0.911704] scsi host1: ahci
[    0.915330] scsi host2: ahci
[    0.915484] xhci_hcd 0000:00:1e.0: hcc params 0x03401281 hci version 0x100 quirks 0x000000000000b930
[    0.915821] scsi host3: ahci
[    0.917473] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.13
[    0.917475] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.917476] usb usb1: Product: xHCI Host Controller
[    0.917477] usb usb1: Manufacturer: Linux 5.13.0-35-generic xhci-hcd
[    0.917477] usb usb1: SerialNumber: 0000:00:1e.0
[    0.917561] hub 1-0:1.0: USB hub found
[    0.917586] hub 1-0:1.0: 4 ports detected
[    0.917786] xhci_hcd 0000:00:1e.0: xHCI Host Controller
[    0.917788] xhci_hcd 0000:00:1e.0: new USB bus registered, assigned bus number 2
[    0.917790] xhci_hcd 0000:00:1e.0: Host supports USB 3.0 SuperSpeed
[    0.917862] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    0.917874] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.13
[    0.917875] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.917876] usb usb2: Product: xHCI Host Controller
[    0.917877] usb usb2: Manufacturer: Linux 5.13.0-35-generic xhci-hcd
[    0.917877] usb usb2: SerialNumber: 0000:00:1e.0
[    0.917952] hub 2-0:1.0: USB hub found
[    0.917973] hub 2-0:1.0: 4 ports detected
[    0.918181] scsi host0: Virtio SCSI HBA
[    0.918257] blk_queue_max_hw_sectors: set to minimum 8
[    0.918457] virtio_net virtio2 enp0s8: renamed from eth0
[    0.918483] scsi host4: ahci
[    0.918544] scsi host6: ahci
[    0.918597] scsi host7: ahci
[    0.918626] ata1: SATA max UDMA/133 abar m1024@0xc100d000 port 0xc100d100 irq 32
[    0.918628] ata2: DUMMY
[    0.918629] ata3: DUMMY
[    0.918629] ata4: DUMMY
[    0.918630] ata5: DUMMY
[    0.918630] ata6: DUMMY
[    0.918893] scsi host5: Virtio SCSI HBA
[    0.918968] blk_queue_max_hw_sectors: set to minimum 8
[    1.152020] usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd
[    1.174661] usb 2-1: New USB device found, idVendor=fb5d, idProduct=0001, bcdDevice= 0.00
[    1.174674] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    1.174680] usb 2-1: Product: HID Tablet
[    1.174685] usb 2-1: Manufacturer: BHYVE
[    1.174689] usb 2-1: SerialNumber: 01
[    1.182997] hid: raw HID events driver (C) Jiri Kosina
[    1.184223] usbcore: registered new interface driver usbhid
[    1.184225] usbhid: USB HID core driver
[    1.185428] input: BHYVE HID Tablet as /devices/pci0000:00/0000:00:1e.0/usb2/2-1/2-1:1.0/0003:FB5D:0001.0001/input/input4
[    1.185572] hid-generic 0003:FB5D:0001.0001: input,hidraw0: USB HID v10.01 Mouse [BHYVE HID Tablet] on usb-0000:00:1e.0-1/input0
[    1.235331] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    1.236218] ata1.00: ATA-9: BHYVE SATA DISK, 001, max UDMA/133
[    1.236226] ata1.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 32)
[    1.237249] ata1.00: configured for UDMA/133
[    1.237520] scsi 1:0:0:0: Direct-Access     ATA      BHYVE SATA DISK  001  PQ: 0 ANSI: 5
[    1.238283] sd 1:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[    1.238305] sd 1:0:0:0: [sda] Write Protect is off
[    1.238310] sd 1:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    1.238336] sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.238352] sd 1:0:0:0: Attached scsi generic sg0 type 0
[    1.264716]  sda: sda1 sda2 sda3 sda4 sda5 sda6 sda7
[    1.298924] sd 1:0:0:0: [sda] Attached SCSI disk

[    3.996516] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[    3.996519] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 3
[    3.996524] vhci_hcd: created sysfs vhci_hcd.0
[    3.996544] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.13
[    3.996546] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.996547] usb usb3: Product: USB/IP Virtual Host Controller
[    3.996548] usb usb3: Manufacturer: Linux 5.13.0-35-generic vhci_hcd
[    3.996548] usb usb3: SerialNumber: vhci_hcd.0
[    3.996631] hub 3-0:1.0: USB hub found
[    3.996635] hub 3-0:1.0: 8 ports detected
[    3.996713] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[    3.996714] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 4
[    3.996720] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    3.996731] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.13
[    3.996732] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.996733] usb usb4: Product: USB/IP Virtual Host Controller
[    3.996734] usb usb4: Manufacturer: Linux 5.13.0-35-generic vhci_hcd
[    3.996734] usb usb4: SerialNumber: vhci_hcd.0
[    3.996804] hub 4-0:1.0: USB hub found
[    3.996807] hub 4-0:1.0: 8 ports detected



marietto@Z390-AORUS-PRO-DEST:/home/ziomario# fdisk -l

Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: BHYVE SATA DISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Disk identifier: F66E6C53-78C2-4D44-AC4E-9B99FCE0D555


Dispositivo      Start       Fine    Settori   Size Tipo
/dev/sda1         2048    1751039    1748992   854M EFI System
/dev/sda2      1751040 1115258879 1113507840   531G Microsoft basic data
/dev/sda3   1948444672 1949763583    1318912   644M Windows recovery environment
/dev/sda4   1949765632 1951076351    1310720   640M Windows recovery environment
/dev/sda5   1951078400 1952344056    1265657   618M Microsoft basic data
/dev/sda6   1952344064 1953523704    1179641   576M Microsoft basic data
/dev/sda7   1115258880 1948444426  833185547 397.3G Microsoft basic data

Partition table entries are not in disk order.
 
this worked :

Code: 
bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,virtio-blk,/dev/da4p2 \
-s 3,virtio-blk,/dev/da2p1 \
-s 8,virtio-net,tap2 \
-s 9,virtio-9p,sharename=/ \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5902,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm2 < /dev/null & sleep 2 && vncviewer 0:2

the USB disks have been recognized by Ubuntu. They are accesible,but fdisk does not show them.
 
this is the point that I wanted to reach :

Code: 
bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,virtio-blk,/dev/da4p2 \
-s 3,virtio-blk,/dev/da2p1 \
-s 4:0,passthru,2/0/0 \
-s 4:1,passthru,2/0/1 \
-s 4:2,passthru,2/0/2 \
-s 4:3,passthru,2/0/3 \
-s 8,virtio-net,tap2 \
-s 9,virtio-9p,sharename=/ \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5902,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm2 < /dev/null & sleep 2 && vncviewer 0:2

If I try to pass thru the graphic card without the USB controller,but with two USB disks configured as slots,the first physical disk will not be able to boot :

Screenshot_2022-03-13_16-35-42.jpg
 
you export the partitions not the whole disk so fdisk will display crap because the partition table is not visible to the guest os
fdisk will read the 1st sector of the whatever fs you have on them and will try to interpret as a partition table
 
ok. I've exported the whole disks and I've got the same boot failure than before. Like this :

Code: 
bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/dev/nvd0 \
-s 2,virtio-blk,/dev/da4 \
-s 3,virtio-blk,/dev/da2 \
-s 4:0,passthru,2/0/0 \
-s 4:1,passthru,2/0/1 \
-s 4:2,passthru,2/0/2 \
-s 4:3,passthru,2/0/3 \
-s 8,virtio-net,tap2 \
-s 9,virtio-9p,sharename=/ \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5902,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm2 < /dev/null & sleep 2 && vncviewer 0:2
 
and yes,you are right. Using the whole disks in bhyve they are recognized by fdisk. But as I said,what matters a lot for me,is to boot the nvme physical disk passing thru at the same time my graphic card but it does not work. I care so much about this feature,because Ubuntu boots a lot much faster if I use the physical disk instead of the img / raw file. And this is again more valid if I want to boot Windows 11. It gets frozen if I use a raw/img disk. I can't use it. But if I use the nvme disk where I have installed it physically is much,much faster and I can use it.
 
ziomario said:
-s 1,ahci-hd,/dev/nvd0 \
Click to expand...
I would try and pass the whole drive thru.

-s 1:0,passthru, 5/0/0
Use the PCIe address for the NVMe.

That is the quickest disk access I found.
forums.freebsd.org

NVMe passthru on Bhyve findings

I am currently using a SuperMicro X10DRL board for my Bhyve Virtualization machine. OS boots from a 64GB DOM. I use two Samsung 512GB PM953 drives in a M.2 form factor mounted on a SuperMicro AOC-SLG3-2M2. The slot where the card is installed is set to x4x4 in the BIOS for bitfurication. I am...
forums.freebsd.org forums.freebsd.org
 
The scope of this experiment was to not to pass thru the disks using the USB controller,because I did it in the past and it didn't work. It means that the main disk won't boot if you add a passthru device (another disk of a graphic card though the usb controller) ; so this time I tried without using the usb controller. I wanted to see if it was able booting Windows or Linux adding another disk as slot and passing thru the graphic card. It didn't work.
 
Anyway,the nvme parameter is not good for me,since I'm trying to mount in bhyve two disks that are USB,not nvme.
 
ubuntu21-10-nvidia495-vm0.sh :

Code: 
bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \
-s 2,nvme,/dev/nvd0 \
-s 3,passthru,5/0/0 \
-s 4,passthru,1/0/0 \
-s 8,virtio-net,tap0 \
-s 9,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0 < /dev/null & sleep 2 && vncviewer 0:0

on the Ubuntu guest os :

Code: 
mario@marietto-BHYVE:/home/marietto# fdisk -l

Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: bhyve-NVMe                          
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

the nvme disk is recognized,but I don't see it's partitions.
 
done. It does not work :

Code: 
mario@marietto-BHYVE:/home/marietto# fdisk -l


Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors

Disk model: bhyve-NVMe                            

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

I don't see the disk partitions.
 
I'm not able to mount the NTFS partition that's mapped as nvd0p2 under FreeBSD :

Code: 
mario@marietto:/usr/home/marietto # gpart show

=>        34  1953525101  nvd0  GPT  (932G)
          34        2014        - free -  (1.0M)
        2048     1748992     1  efi  (854M)
     1751040   921985024     2  ms-basic-data  (440G)
   923736064   191522816        - free -  (91G)
  1115258880   833185547     7  ms-basic-data  (397G)
  1948444427         245        - free -  (123K)
  1948444672     1318912     3  ms-recovery  (644M)
  1949763584        2048        - free -  (1.0M)
  1949765632     1310720     4  ms-recovery  (640M)
  1951076352        2048        - free -  (1.0M)
  1951078400     1265657     5  ms-basic-data  (618M)
  1952344057           7        - free -  (3.5K)
  1952344064     1179641     6  ms-basic-data  (576M)
  1953523705        1430        - free -  (715K)

For example :

bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \
-s 2,nvme,/dev/nvd0p2 \
-s 3,passthru,5/0/0 \
-s 4,passthru,1/0/0 \
-s 8,virtio-net,tap0 \
-s 9,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0 < /dev/null & sleep 2 && vncviewer 0:0

mario@marietto-BHYVE:/home/marietto# fdisk -l

Disk /dev/nvme0n1: 439,64 GiB, 472056332288 bytes, 921985024 sectors
Disk model: bhyve-NVMe
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


mario@marietto-BHYVE:/home/marietto# ntfs-3g /dev/nvme0n1 /mnt/nvme


ntfs_mst_post_read_fixup_warn: magic: 0x00000000  size: 1024   usa_ofs: 0  usa_count: 0: Argoment not valid
Record 0 has no FILE magic (0x0)
Failed to load $MFT: Error input/output
Failed to mount '/dev/nvme0n1': Error input/output
NTFS is either inconsistent, or there is a hardware fault, or it's a SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows
then reboot into Windows twice. The usage of the /f parameter is very important! If the device is a SoftRAID/FakeRAID then first activate
it and mount a different device under the /dev/mapper/ directory, (e.g. /dev/mapper/nvidia_eahaabcc1). Please see the 'dmraid' documentation for more details.
 
ziomario said:
Disk /dev/nvme0n1:
Click to expand...
Are you sure this is right? I don't know shit about Linux but on freebsd the nvme0ns0 devices are the namespace.
You don't mount that.

That could be completely wrong but thats what you get asking a Ubuntu question on a FreeBSD forum.
 
Phishfry said:
Are you sure this is right? I don't know shit about Linux but on freebsd the nvme0ns0 devices are the namespace.
You don't mount that.

That could be completely wrong but thats what you get asking a Ubuntu question on a FreeBSD forum.
Click to expand...

I don't agree with that. I'm asking an FreeBSD / Bhyve question about how to mount a nvme disk from FreeBSD to Ubuntu as guest os.
 
You must log in or register to reply here.
Back
Top