I'm trying to migrate my product from linux to freebsd. For this to happen I also need to replace mikrotik as pppoe server with mpd5, so that my product can run in the same box as the pppoe server. I have installed it under pfsense for easier network configuration, but I think my problem is in fact freebsd related, so I'm posting it here.
Decently capable hardware (for this purpose): I7-4500U with two igb cards, in portchannel to arista switch. Main interface is untagged on the lagg interface, pppoe server is running on a tagged vlan. All other mentioned equipment (test servers, internet, etc.) is connected to the same switch at 10Gbps. I have tried a less complex setup - one phisycall port as WAN and another phisycal port as pppoe, without lagg and vlans, but it doesn't seem to make any difference.
The setup is powerful enough and I CAN reach speeds up to 950 mbps on a 1 gbps link with various pppoe clients (tested with several residential routers).
But there is one particular setup that gives me tragic results: when the pppoe client is Windows 10 AND the server is mpd5, the speed is practically shaped to 10 mbps (8-9 in reality).
If the client is NOT windows OR the server is NOT mpd5, then the speed is good.
One of the windows PCs is dual-boot and when I boot it to Ubuntu, it still reaches 950 mbps through the mpd5's pppoe server. The same hardware, different OS, so it is not a hardware issue at the PCs site, however I have tried with an external 2.5Gbps adapter with the same result, and with another Windows 10 PC, still with the same result.
The same setup, but with a router as a pppoe client in front of the PC (between the same pc and mpd5), works with no problems.
I am using radius as AAA with no mpd-limit attributes whatsoever, besides all the tests are done with the same radius user / account, so it should not be radius-related.
I have tried all possible ways to measure the speed: iperf3 on the pfsense box, iperf3 on a linux box behind it (quite capable - it also runs a ookla speedtest server), nearby ookla servers, netflix, you name it. The results are consistent - high speed with all possible combinations of client and server, low speed with windows + mpd5. No matter if the box works as a router to another iperf3 server via wan interface or as a test box, so the tcp stack of the pfsense box is irrelevant. I have tried all possible combinations of hardware offloading and altq, I have also tried turning off firewall - that's why I think it is a more general freebesd problem and not specifically pfsense's.
Fresh install with only one package - iperf3. No other services running except mpd5.
What I can see by tcpdumping the pppoe interface is that windows 10 is not using SACK (selective acknowledgements) in this particular scenario. I first thought it may be something latencly-related, if windows has a small window and is sending single acks, however opening many parallel connections (iperf3 -P 5) should help to reach higher throughput, but it only worsens the situation and the combined speed is even lower. So it is not exactly tcp-window related as it affects all parallel tcp connections.
There are no mtu / mss issues as far as I can tell. No packets are lost, big or small, the highest packet passing without fragmentation is 1452 bytes, which is fine. Windows adapts it's mss / mtu accordingly. I have also tried to make mpd5 accept multilink connections so I can bring the mtu back to 1500 and I have successfully achieved it, but it didn't help - the speed was still shaped to 10 mb even without the mtu issues.
And I really mean "shaped". Initially it bursts a little bit to 40 mbps, then after two seconds it goes down to 9-10 mps and stays there indefinitely.
I have tried to disable and enable compression and or encryption from mpd's side but to no avail. I mean - I can disable or enable it and I see it in the mpd's web interface, but it does not affect my problem.
I have played with some tunables, without any success. I finally left them like this:
net.isr.dispatch="deferred"
net.isr.numthreads="1"
net.isr.bindthreads="1"
net.isr.maxthreads="-1"
net.inet.rss.enabled="1"
net.inet.rss.bits="2"
My assumption is that I am hitting (at least) two problems ta once:
1. for some reason, windows decides not to use SACK when connected to mpd5. I don't see why level 3 functionality should be changed by a layer 2 protocol, but it does - no SACKs are used in this case.
2. something (igb nic, kernel, mpd?) is throttling the ACK packets (in both directions!), may be because there are too many 0-sized packets. I can see them coming in bursts when this problem appears: 6-8 big data packets in one direction, followed by 2-3-4 ack packets in the other direction. ACKs are always a little bit less in number and they always come in groups (coalescence somewhere?).
At this point I've run out of ideas. Do you have any?
Decently capable hardware (for this purpose): I7-4500U with two igb cards, in portchannel to arista switch. Main interface is untagged on the lagg interface, pppoe server is running on a tagged vlan. All other mentioned equipment (test servers, internet, etc.) is connected to the same switch at 10Gbps. I have tried a less complex setup - one phisycall port as WAN and another phisycal port as pppoe, without lagg and vlans, but it doesn't seem to make any difference.
The setup is powerful enough and I CAN reach speeds up to 950 mbps on a 1 gbps link with various pppoe clients (tested with several residential routers).
But there is one particular setup that gives me tragic results: when the pppoe client is Windows 10 AND the server is mpd5, the speed is practically shaped to 10 mbps (8-9 in reality).
If the client is NOT windows OR the server is NOT mpd5, then the speed is good.
One of the windows PCs is dual-boot and when I boot it to Ubuntu, it still reaches 950 mbps through the mpd5's pppoe server. The same hardware, different OS, so it is not a hardware issue at the PCs site, however I have tried with an external 2.5Gbps adapter with the same result, and with another Windows 10 PC, still with the same result.
The same setup, but with a router as a pppoe client in front of the PC (between the same pc and mpd5), works with no problems.
I am using radius as AAA with no mpd-limit attributes whatsoever, besides all the tests are done with the same radius user / account, so it should not be radius-related.
I have tried all possible ways to measure the speed: iperf3 on the pfsense box, iperf3 on a linux box behind it (quite capable - it also runs a ookla speedtest server), nearby ookla servers, netflix, you name it. The results are consistent - high speed with all possible combinations of client and server, low speed with windows + mpd5. No matter if the box works as a router to another iperf3 server via wan interface or as a test box, so the tcp stack of the pfsense box is irrelevant. I have tried all possible combinations of hardware offloading and altq, I have also tried turning off firewall - that's why I think it is a more general freebesd problem and not specifically pfsense's.
Fresh install with only one package - iperf3. No other services running except mpd5.
What I can see by tcpdumping the pppoe interface is that windows 10 is not using SACK (selective acknowledgements) in this particular scenario. I first thought it may be something latencly-related, if windows has a small window and is sending single acks, however opening many parallel connections (iperf3 -P 5) should help to reach higher throughput, but it only worsens the situation and the combined speed is even lower. So it is not exactly tcp-window related as it affects all parallel tcp connections.
There are no mtu / mss issues as far as I can tell. No packets are lost, big or small, the highest packet passing without fragmentation is 1452 bytes, which is fine. Windows adapts it's mss / mtu accordingly. I have also tried to make mpd5 accept multilink connections so I can bring the mtu back to 1500 and I have successfully achieved it, but it didn't help - the speed was still shaped to 10 mb even without the mtu issues.
And I really mean "shaped". Initially it bursts a little bit to 40 mbps, then after two seconds it goes down to 9-10 mps and stays there indefinitely.
I have tried to disable and enable compression and or encryption from mpd's side but to no avail. I mean - I can disable or enable it and I see it in the mpd's web interface, but it does not affect my problem.
I have played with some tunables, without any success. I finally left them like this:
net.isr.dispatch="deferred"
net.isr.numthreads="1"
net.isr.bindthreads="1"
net.isr.maxthreads="-1"
net.inet.rss.enabled="1"
net.inet.rss.bits="2"
My assumption is that I am hitting (at least) two problems ta once:
1. for some reason, windows decides not to use SACK when connected to mpd5. I don't see why level 3 functionality should be changed by a layer 2 protocol, but it does - no SACKs are used in this case.
2. something (igb nic, kernel, mpd?) is throttling the ACK packets (in both directions!), may be because there are too many 0-sized packets. I can see them coming in bursts when this problem appears: 6-8 big data packets in one direction, followed by 2-3-4 ack packets in the other direction. ACKs are always a little bit less in number and they always come in groups (coalescence somewhere?).
At this point I've run out of ideas. Do you have any?