KVM+FreeBSD+10G NIC: Link is always down.

[legumen]

Hi guys:

I'm working on a FreeBSD virtulizaion project under KVM. Everything works fine, except the 10 G Intel 82599 SFP NIC. I'm sure there are some body else who have encountered this problem before. See:
FreeBSD guest with VTD NIC not passing traffic
VT-d not working for FreeBSD 9.0 guest
But, unfortunately, none of them solved it or gave the clear way to solve it.

Firstly, let's make it clear. This problem could not be reproduced with:
1. Xen hypervisor + FreeBSD guest OS
2. or KVM + Linux guest OS (e.g. Fedora 12)
So, it must be something wrong with KVM or FreeBSD.

Here is my environment:
Host OS:
OpenSUSE 12.2
Linux linux-kj2u 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux
QEMU emulator version 1.1.1 (kvm-1.1.1-1.8.1), Copyright (c) 2003-2008 Fabrice Bellard

Guest OS:
FreeBSD 9.1 amd64

What I did:

1. Make the PCI device of 82599SFP assignable:
   

   0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
           Subsystem: QLogic, Corp. Device 00e4
           Flags: fast devsel, IRQ 25
           Memory at f8fe0000 (64-bit, prefetchable) [size=128K]
           I/O ports at cc00 [size=200]
           Memory at f8fdc000 (64-bit, prefetchable) [size=16K]
           Capabilities: [40] Power Management version 3
           Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
           Capabilities: [70] MSI-X: Enable- Count=64 Masked-
           Capabilities: [a0] Express Endpoint, MSI 00
           Capabilities: [e0] Vital Product Data
           Capabilities: [100] Advanced Error Reporting
           Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00
           Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
           Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
           Kernel driver in use: pci-stub

2. Passthrough it to the guest FreeBSD OS. Here is my VM configuration file
   

   <domain type='kvm'>
     <name>FreeBSD9.1</name>
     <uuid>4d56613e-400c-f37d-2a9a-e0332d81ca63</uuid>
     <memory>2097152</memory>
     <currentMemory>2097152</currentMemory>
     <vcpu>2</vcpu>
     <os>
       <type arch='x86_64' machine='pc'>hvm</type>
       <boot dev='hd'/>
     </os>
     <features>
       <pae/>
       <apic/>
       <acpi/>
     </features>
       <clock offset='utc'>
       </clock>
       <on_poweroff>destroy</on_poweroff>
       <on_reboot>restart</on_reboot>
       <on_crash>destroy</on_crash>
     <devices>
       <emulator>/usr/bin/qemu-kvm</emulator>
         <disk type='file' device='disk'>
         <driver name='qemu' type='raw' cache='default'/>
         <source file='/home/vm/FreeBSD9.1/FreeBSD9.1.raw'/>
         <target dev='hdc'/>
       </disk>
         <interface type='bridge'>
           <source bridge='br0'/>
           <mac address='52:54:00:0d:92:6f'/>
           <model type='rtl8139'/>
         </interface>
       <hostdev mode='subsystem' type='pci' managed='yes'>
         <source>
           <address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
         </source>
       </hostdev>
       <input type='mouse' bus='ps2'/>
       <graphics type='vnc' port='-1' autoport='yes'/>
     
     </devices>
   </domain>

3. The device could be probed and attached by FreeBSD:
   

   ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.8> port 0xc020-0xc03f mem 0xfe000000-0xfe01ffff,0xfe020000-0xfe023fff irq 11 at device 4.0 on pci0
   ix0: Using MSIX interrupts with 2 vectors
   ix0: Ethernet address: 00:0c:bd:05:7d:4a
   ix0: PCI Express Bus: Speed 5.0Gb/s Width x8

   But no interrupt could be found:
   

   interrupt                          total       rate
   irq1: atkbd0                         143          0
   irq4: uart0                          177          1
   irq15: ata1                         1593         10
   cpu0:timer                          7140         46
   Total                               9053         58

   And ix0 keeps to be "no carrier" (even if I config an IP address on it and ping):
   

   ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
   	options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
   	ether 00:0c:bd:05:7d:4a
   	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   	media: Ethernet autoselect
   	status: no carrier

4. In the host side, KVM could see the change made by the guest on the device, but no IRQ:
   

   [cmd=linux-kj2u:~ #]cat /proc/interrupts[/cmd]
               CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23      
    210:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm:0000:0d:00.0
    211:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm:0000:0d:00.0

So, anyone who has the same problem or has a clue to solve it, or has some ideas to do a debug job, please help.

Thanks! �e

 

[legumen]

Hi guys. I got a reply from Jack from freebsd-virtualization. The driver may have some problem, but we can use SRIOV as a substitution. According to my test, using SRIOV and doing the passthrough works fine. So, I will make this post as "solved".

Thanks.