I have a couple questions about the oce(4) driver. The cards are HP NC550SFP 10 gigabit dual adapters, which are really Emulex OneConnect devices. FreeBSD server is an older IBM x3650 server.
1: I don't see oce anywhere in FreeBSD 9.x. Has it been dropped or is Emulex being slow in getting a driver out?
2: Bridging oce interfaces using ifconfig bridge just doesn't seem to work, right? I can imagine there being issues with the TOE stuff and bridge.
3: Now to the tough one. Since the bridge doesn't work, I've moved onto routing. This is a kind of odd requirement with firewalls and traffic shaping, so it has to be this way. So, three systems:
(RHEL 6 eth0: 192.168.65.2/28) <-> (FreeBSD 8.3 -updated: oce0 192.168.65.1/28, oce2 192.168.65.17/28) <-> (RHEL 6: eth0 192.168.65.18/28)
Essentially, FreeBSD is being a router between two stand alone boxes. Fiber cables directly between each, no switches. The problem is it's very slow. As in 7 KILOBYTES/s between the RHEL boxes. Yes, Kilo. If I transfer a file directly to/from FreeBSD to either RHEL system, I get reasonable data rates (400MB/s). If I connect the RHEL boxes directly to each other (and adjust the subnet masks) they get a very reasonable 7-800MB/s. I've tried various sysctl tuning, but orders of magnitude too slow feels like a bug or something bigger than that. The system routes fine between regular gigabit interfaces (there's 10 in the box, 2 bce-based on-board and 2 4-port em-based PCIe cards). Ping times are sub-1ms.
For fun, I booted the FreeBSD system off a RHEL 6.3 disc into rescue mode. After configuring the interfaces for routing as above, the two client boxes are able to transfer at a reasonable 500MB/s. There's some interaction going on between oce and the rest of routing that I just can't figure out.
Any ideas?
1: I don't see oce anywhere in FreeBSD 9.x. Has it been dropped or is Emulex being slow in getting a driver out?
2: Bridging oce interfaces using ifconfig bridge just doesn't seem to work, right? I can imagine there being issues with the TOE stuff and bridge.
3: Now to the tough one. Since the bridge doesn't work, I've moved onto routing. This is a kind of odd requirement with firewalls and traffic shaping, so it has to be this way. So, three systems:
(RHEL 6 eth0: 192.168.65.2/28) <-> (FreeBSD 8.3 -updated: oce0 192.168.65.1/28, oce2 192.168.65.17/28) <-> (RHEL 6: eth0 192.168.65.18/28)
Essentially, FreeBSD is being a router between two stand alone boxes. Fiber cables directly between each, no switches. The problem is it's very slow. As in 7 KILOBYTES/s between the RHEL boxes. Yes, Kilo. If I transfer a file directly to/from FreeBSD to either RHEL system, I get reasonable data rates (400MB/s). If I connect the RHEL boxes directly to each other (and adjust the subnet masks) they get a very reasonable 7-800MB/s. I've tried various sysctl tuning, but orders of magnitude too slow feels like a bug or something bigger than that. The system routes fine between regular gigabit interfaces (there's 10 in the box, 2 bce-based on-board and 2 4-port em-based PCIe cards). Ping times are sub-1ms.
For fun, I booted the FreeBSD system off a RHEL 6.3 disc into rescue mode. After configuring the interfaces for routing as above, the two client boxes are able to transfer at a reasonable 500MB/s. There's some interaction going on between oce and the rest of routing that I just can't figure out.
Any ideas?
)