Lot of fun with vtnet(4) and Linux KVM (Proxmox)

After I started using a FreeBSD router over KVM host some random funny things started happening. At first everything looked fine but then begun random connection slowdowns ONLY to some hosts. After some investigation I found out that I see broken packets from these host only. Of course the first thing that I tried was ifconfig vtnet0 -rxcsum, ifconfig vtnet1 -rxcsum ... and it worked ;) . After few days the client started complaining that some mail servers do not receive his mails and I saw that SFTP fails with very strange error on write of a comparatively large file "Permission denied" which made no sense at all but it matched the MTA errors which too didn't make ANY sense ;) . After digging around google I found out similar errors but with physical interfaces and the fix was ifconfig vtnet0 -tso, ifconfig vtnet1 -tso. And of course it worked and fixed the strange error "Permission denied" (that can't be right and I have no idea why that message appears). Now everything is all flowers and roses BUT the manual page of vtnet(4) states:
  If the hypervisor advertises the appreciate features, the vtnet driver
  supports TCP/UDP checksum offload for both transmit and receive, TCP seg-
  mentation offload (TSO), TCP large receive offload...
That leads me to believe that the host advertise those hw accelerations but something is messed up in the whole pipeline of the packets from, to and through the FreeBSD router. And it would be nice if I can use the HW acceleration, but I hope that helps somebody having my problems ;)

FreeBSD 10.2-RELEASE-p14 #0: Wed Mar 16 20:46:12
running on
Linux pve 4.2.8-1-pve #1 SMP Sat Mar 19 10:44:29 CET 2016 x86_64 GNU/Linux
KVM (qemu) hypervisor

02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
  Subsystem: Super Micro Computer Inc Device 1533
  Flags: bus master, fast devsel, latency 0, IRQ 17
  Memory at df100000 (32-bit, non-prefetchable) [size=512K]
  I/O ports at d000
  Memory at df180000 (32-bit, non-prefetchable) [size=16K][/SIZE]
  Capabilities: [40] Power Management version 3
  Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
  Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
  Capabilities: [a0] Express Endpoint, MSI 00
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [140] Device Serial Number 0c-c4-7a-ff-ff-b3-cb-a3
  Capabilities: [1a0] Transaction Processing Hints
  Kernel driver in use: igb

PS: Fixed (so far) the TSO problem by doing ethtool -K vmbr0/1 sg off tx off at the hypervisor ... After reboot - same problems. had to use -tso again and ethtool -K vmbr0/1 sg off tx off didn't help now. I guess ifconfig vtnet0 tso didn't actually enabled the TSO.
Last edited by a moderator:
Turn it off permanently by add the -tso to the ifconfig_ line in rc.conf. It's not a setting that "sticks" once you set it.
Turn it off permanently by add the -tso to the ifconfig_ line in rc.conf. It's not a setting that "sticks" once you set it.
Of course I did that :) I meant that after the reboot of the Linux hypervisor and then setting the ethtool -K vmbr0/1 sg off tx off TSO didn't worked. The first time before the reboot I did this:

ifconfig vtnet0 -tso so it worked then
ethtool -K vmbr0/1 sg off tx off on the hypervisor and
ifconfig vtnet0 tso on the FreeBSD and it was still working but I guess the last command didn't actually turned the TSO on because:
after the reboot of the hypervisor I set ethtool -K vmbr0/1 sg off tx off and expected the vtnet0 to work but it doesn't so i've put -tso in rc.conf