Trouble with quad-port NICs, 32-bit DMA

B

bbzz

I have 3 Quad-port VIA Velocity Gigabit network cards on my i7, 12GB RAM, amd64 platform. The system was running on linux fine (with some limitations such as no 802.1q tags, but that's due to the driver), and I am unable to run this on FreeBSD.

The issue
---------

All cards are identified by FreeBSD. However, as soon as there is at least some traffic ran trough at least one port, the system either freezes (8.2-RELEASE) momentarily, or there is a fatal kernel trap 28 (7.3, 7.4, 8.1 - Release).

Code: 
vge0: <VIA Networking Velocity Gigabit Ethernet> port 0xce00-0xceff mem 0xfd1ff000-0xfd1ff0ff irq 17 at device 8.0 on pci10
miibus1: <MII bus> on vge0
vge0: Ethernet address: 00:0c:42:1a:00:a4
vge0: [ITHREAD]
vge1: <VIA Networking Velocity Gigabit Ethernet> port 0xcc00-0xccff mem 0xfd1fe000-0xfd1fe0ff irq 18 at device 9.0 on pci10
miibus2: <MII bus> on vge1
vge1: Ethernet address: 00:0c:42:1a:00:a5
vge1: [ITHREAD]
vge2: <VIA Networking Velocity Gigabit Ethernet> port 0xca00-0xcaff mem 0xfd1fd000-0xfd1fd0ff irq 19 at device 10.0 on pci10
miibus3: <MII bus> on vge2
vge2: Ethernet address: 00:0c:42:1a:00:a6
vge2: [ITHREAD]
vge3: <VIA Networking Velocity Gigabit Ethernet> port 0xc800-0xc8ff mem 0xfd1fc000-0xfd1fc0ff irq 16 at device 11.0 on pci10
miibus4: <MII bus> on vge3
vge3: Ethernet address: 00:0c:42:1a:00:a7
vge3: [ITHREAD]
vge4: <VIA Networking Velocity Gigabit Ethernet> port 0xbe00-0xbeff mem 0xfd0ff000-0xfd0ff0ff irq 18 at device 8.0 on pci11
miibus5: <MII bus> on vge4
vge4: Ethernet address: 00:0c:42:1a:00:30
vge4: [ITHREAD]
vge5: <VIA Networking Velocity Gigabit Ethernet> port 0xbc00-0xbcff mem 0xfd0fe000-0xfd0fe0ff irq 19 at device 9.0 on pci11
vge5: 4GB boundary crossed, switching to 32bit DMA address mode.
miibus6: <MII bus> on vge5
vge5: Ethernet address: 00:0c:42:1a:00:31
vge5: [ITHREAD]
vge6: <VIA Networking Velocity Gigabit Ethernet> port 0xba00-0xbaff mem 0xfd0fd000-0xfd0fd0ff irq 16 at device 10.0 on pci11
miibus7: <MII bus> on vge6
vge6: Ethernet address: 00:0c:42:1a:00:32
vge6: [ITHREAD]
vge7: <VIA Networking Velocity Gigabit Ethernet> port 0xb800-0xb8ff mem 0xfd0fc000-0xfd0fc0ff irq 17 at device 11.0 on pci11
vge7: 4GB boundary crossed, switching to 32bit DMA address mode.
miibus8: <MII bus> on vge7
vge7: Ethernet address: 00:0c:42:1a:00:33

If you look at messages you'll see 4GB boundary crossed, switching to 32bit DMA address mode after first card.

To test this I took out two cards, left just one, and so far it's been running successfully (as in, not freezing at least so far).

Why this problem with DMA boundary at 4 GB, can it be lifted (I understand what it is, just not sure why is there limit at 4 GB with x64)? Again, not sure this is the problem but so far it seems the best candidate. How do I go about running all 3 cards?

Thanks
 
If I read the source correctly the message is just informational. It's explained a bit more in the source:
Code: 
 * The other issue has to do with the way 64-bit addresses are handled.
 * The DMA descriptors only allow you to specify 48 bits of addressing
 * information. The remaining 16 bits are specified using one of the
 * I/O registers. If you only have a 32-bit system, then this isn't
 * an issue, but if you have a 64-bit system and more than 4GB of
 * memory, you must have to make sure your network data buffers reside
 * in the same 48-bit 'segment.'
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/vge/if_vge.c?rev=1.74

As for the freezes or panics you could send an email to the freebsd-net mailinglist and/or file a PR.
 
Thanks! Yeah, that does explain it. But how do I "make sure your network data buffers reside in the same 48-bit 'segment."?

My server has been working for hours now with one card, so this is what is making system freeze for sure. Seems like a known issue so I won't file this just yet, I want to try make it works. But it doesn't say anything about how to do it..
 
My WAG would be that the driver assign a global pool of memory for use by all NICs using that driver, and if that pool goes over 4 GB, then bad things happen. If you look at the dmesg output, you'll see the memory ranges used by each interface, and that it's only after the 4th interface that it complains about crossing the 4 GB barrier. Using fewer interfaces keeps the pool under the 4 GB barrier.

Someone with more insight into hexadecimal addressing should be able to confirm/refute the above.

Short-term solution: don't use more than 4 vge(4) ports in a single system. Maybe look into non-VIA multi-port cards.

Long-term solution: VIA fixes their hardware and/or driver, and it gets updated in FreeBSD.

Hrm, or, you may be able to do something with /boot/loader.conf hints to specify the starting address for the mem ranges, so that the second card stays within the same 4 GB as the first card??
 
phoenix said:
Hrm, or, you may be able to do something with /boot/loader.conf hints to specify the starting address for the mem ranges, so that the second card stays within the same 4 GB as the first card??
Click to expand...

Makes sense, but I have no clue where to start.

I installed 8.2-RELEASE again, connected two cards, and there was no "32-bit DMA" message (maybe there never was at 8.2, I was mostly working with 7.4). So all excited I tried downloading something, and then it just froze the system again. No messages of any sort, no kernel panic, just complete stop. Had to manually reset. Took one card out, and then it works. So yeah, most definitely only 1 4-ports GE card allowed. Buying new quad-port cards is not an option right now..bummer.

Cards work just fine under linux, but under linux there's no support for additional 802.1q tags. -_-

Thanks again
 
The "4GB boundary crossed, switching to 32bit DMA address mode" message comes from hardware limitation. The controller in question has two severe DMA address space limitation. The first one is TX/RX buffer should be within 48bit address space and the second one is TX/RX ring should reside in 4GB address space. Because you have many vge(4) instances, some vge(4) instance can not meet the second hardware restriction such that driver used the last resort, limiting address space to 32bit. Limiting DMA address space means it will use bounce buffers which in turn will reduce RX/TX performance a lot since it have to copy frames from/to bounce buffers. If your concern is performance on system that has more than 4GB memory, use other controller that support full 64bit DMA addressing. All server class adapters have no such limitation.

The lock up issue you've seen could be related with VLAN hardware tagging. Try disabling VLAN hardware tagging for each vge(4) instance with command "ifconfig vge0 -vlanhwtag" and see whether that makes any difference.
 
Unfortunately, that one card that was running for hours froze system as well at the end.
I will try disabling vlanhwtagging, see if that helps. Could be the reason why tagging doesn't work on linux (can't believe I didn't think of that one).

The server is for personal use, not professional use, and the cards were bought cheap.
They are Mikrotik 44GV, and I heard bad things about them later on. Still, no reason why they should randomly freeze the system, and I don't believe the linux driver is of greater quality.

I'll let you know how it goes.

Thanks!
 
No dice.
No matter what I try to do, the system just freezes. And with 8.2-Release, it doesn't show any message, you can't do any debug, there's nothing to do but hard reset. I know the cards are not the best out there, but why do they work flawlessly on linux? Really would hate to go back.
 
Thanks for testing. Please open a new PR for the issue and let me know the PR number. Make sure to post full dmesg and pciconf -lcbv output to the PR.
 
You must log in or register to reply here.
Back
Top