Roundrobin or loadbalance?

B

belon_cfy

Hi
I have a server with 2 network cables plugged into the same unmanaged switch, can I use loadbalance as well as roundrobin?

All of the client machines connected to this server only have a single NIC and plugged through the same switch.

Below is the loadbalance output, seems all the traffic were went through a single interface.
Code: 
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   ||||||||||||||||

      Interface           Traffic               Peak                Total
          vlan9  in     77.555 MB/s         84.287 MB/s          474.400 GB
                 out   398.266 KB/s        520.706 KB/s            2.875 GB

          lagg0  in     77.334 MB/s         87.232 MB/s          474.382 GB
                 out   397.873 KB/s        523.133 KB/s            2.875 GB

            lo0  in      0.000 KB/s          7.299 KB/s            1.011 MB
                 out     0.000 KB/s          7.299 KB/s            1.011 MB

           igb1  in     77.557 MB/s         84.287 MB/s          241.584 GB
                 out   398.266 KB/s        451.433 KB/s            1.480 GB

           igb0  in      1.727 KB/s         79.952 MB/s          232.819 GB
                 out     0.000 KB/s        409.717 KB/s            1.396 GB

            em0  in      0.321 KB/s          3.249 KB/s            1.801 MB
                 out     0.747 KB/s          4.132 KB/s           12.914 MB

However, when I set it to roundrobin, the server will leverage all the network interface.
Code: 
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |||||||||||||||||||||||||

      Interface           Traffic               Peak                Total
          vlan9  in     69.446 MB/s         84.287 MB/s          483.722 GB
                 out   433.124 KB/s        520.706 KB/s            2.923 GB

          lagg0  in    142.256 MB/s        142.256 MB/s          483.736 GB
                 out   892.964 KB/s        892.964 KB/s            2.924 GB

            lo0  in      0.000 KB/s          7.299 KB/s            1.024 MB
                 out     0.000 KB/s          7.299 KB/s            1.024 MB

           igb1  in     35.153 MB/s         84.287 MB/s          250.723 GB
                 out   216.108 KB/s        451.433 KB/s            1.527 GB

           igb0  in     34.270 MB/s         79.952 MB/s          233.002 GB
                 out   217.024 KB/s        409.717 KB/s            1.397 GB

            em0  in      0.116 KB/s          3.832 KB/s            1.830 MB
                 out     0.526 KB/s          4.132 KB/s           12.989 MB

Sound likes the roundrobin does better job on it, but I found the following statement from FreeBSD official website :

This mode distributes outgoing traffic using a round-robin scheduler through all active ports and accepts incoming traffic from any active port. Since this mode violates Ethernet frame ordering, it should be used with caution.

Is roundrobin usable in production environment?
 
It probably depends on a lot of things. At the moment it doesn't seem to be causing any big problems, unless some client machine gets slow throughput or crashes from being out of buffers (the expected symptoms if too many packets are arriving in the wrong order). Most of your traffic is inbound to your server in the scenario you provided, and this means there aren't that many packets sent back-to-back, so not many could possibly be out of order. If the traffic was mostly outbound from the server to your clients then there is a much greater potential for packet ordering problems. And longer network delays (e.g. clients out on the WAN or a VPN) and/or varying packet sizes could potentially worsen the problem (e.g. 3 long frames followed by a short frame could be more likely to cause the shorter frame to arrive early). Of course, it also depends on how gracefully the client machines handle the out-of-order frames: what software do they run?

How well does the scenario you are running represent the day-to-day network behavior of the client machines? Maybe you need to run a scenario that involves more data outbound from the server than inbound to see what happens.

Could you please show us an ifconfig output from your server (just curious)?
 
Hi,
Does it means if running roundrobin on LAN should be fine with the client(ESXI host) directly attached to the same switch?

I'm running FreeBSD 10.1 with roundrobin network and NFS shared to multiple ESXI host.

Below is my ifconfig:
Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
        ether 00:30:48:61:da:82
        inet 10.90.1.254 netmask 0xffffff00 broadcast 10.90.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:30:48:61:da:83
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:91:dd:a8
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:91:dd:a8
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:91:dd:a8
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        laggproto roundrobin lagghash l2,l3,l4
        laggport: igb1 flags=4<ACTIVE>
        laggport: igb0 flags=4<ACTIVE>
vlan9: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether 00:25:90:91:dd:a8
        inet 10.90.11.254 netmask 0xffffff00 broadcast 10.90.11.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 9 parent interface: lagg0
 
belon_cfy said:
Does it means if running roundrobin on LAN should be fine with the client(ESXI host) directly attached to the same switch?
Click to expand...

I think it is more likely to work well than if the clients were distributed over a wider network. I suggest you run some tests to see. Possibly useful testing scenarios:

A single client reading huge files as fast as possible.

A single client writing huge files as fast as possible.

Many clients reading huge files.

A mix of clients reading and writing in patterns as close to your "normal" production scenario as possible.

If all of that works well then it is hard for me to see how a problem will crop up in actual production.
 
My opinion: You should never ever use round-robin when you use TCP. Period. Go with loadbalance. Take the warning about the packet reordering serious. In your setup with round-robin the first frame is sent out via ibg0, the second via igb1, the third via ibg0 and so on. As Uniballer already said, it depends on a lot of factors (cable lenght, your switching environment, the TCP congestion control mechanism choosen...) whether the packets arive in-order at the other side. If they arrive not in order (e.g. TCP segment number 1000 arrives before 999) the other side sends a duplicate ACK to your server. The server assumes a packet loss and the TCP congestion control mechanism might trottle/slow down the connection massively! You might not see this during simple tests, rather later when more traffic is going over the wire (and you forgot about this).

(Every three months or so I have to troubleshoot performance problems in our 5000+ servers datacenter. I've seen too many issues with slow database/tsm-backup/ftp/nfs/put-you-app-here. The server guys always blame the network, the firewall etc. just because round-robin sounds like you can make a 4 Gigabit connection by bundling four 1Gig network cards.)

BTW: We are talking about the transmit or outgoing policy here! The output in your opening post said when using loadbalance 1.480 GB at igb0 and 1.396 GB at igb1 which is nearly the same and no benefit.

Hope that helps!
 
Uniballer said:
I think it is more likely to work well than if the clients were distributed over a wider network. I suggest you run some tests to see. Possibly useful testing scenarios:

A single client reading huge files as fast as possible.

A single client writing huge files as fast as possible.

Many clients reading huge files.

A mix of clients reading and writing in patterns as close to your "normal" production scenario as possible.

If all of that works well then it is hard for me to see how a problem will crop up in actual production.
Click to expand...
Hi.

Tested, poor throughput during large transfer with round robin. So I will stick with loadbalance or failover as well as it.

However, my loadbalance seems is not working as expected, the incoming and outgoing traffic still go through the same NIC to multiple clients.
 
honk said:
My opinion: You should never ever use round-robin when you use TCP. Period. Go with loadbalance. Take the warning about the packet reordering serious. In your setup with round-robin the first frame is sent out via ibg0, the second via igb1, the third via ibg0 and so on. As Uniballer already said, it depends on a lot of factors (cable lenght, your switching environment, the TCP congestion control mechanism choosen...) whether the packets arive in-order at the other side. If they arrive not in order (e.g. TCP segment number 1000 arrives before 999) the other side sends a duplicate ACK to your server. The server assumes a packet loss and the TCP congestion control mechanism might trottle/slow down the connection massively! You might not see this during simple tests, rather later when more traffic is going over the wire (and you forgot about this).

(Every three months or so I have to troubleshoot performance problems in our 5000+ servers datacenter. I've seen too many issues with slow database/tsm-backup/ftp/nfs/put-you-app-here. The server guys always blame the network, the firewall etc. just because round-robin sounds like you can make a 4 Gigabit connection by bundling four 1Gig network cards.)

BTW: We are talking about the transmit or outgoing policy here! The output in your opening post said when using loadbalance 1.480 GB at igb0 and 1.396 GB at igb1 which is nearly the same and no benefit.

Hope that helps!
Click to expand...
Hi,
Thanks for sharing, roundrobin doesn't work as expected performance, but loadbalance as well, however it is still much faster than roundrobin in large transfer.

Are you using loadbalance by bonding 4X1Gbps interface? How is your switch configuration? I'm using unmanaged switch without LACP supported so that it might be the reason why my loadbalance doesn't work as expected.
 
I have a hard time understanding how you expected LAGG to work well with an unmanaged switch. You are probably getting two copies of some or all TCP segments that arrive on igb0 and igb1. That has got to hurt CPU utilization.

EDIT: No, maybe not. Depending on the switch it may simply deliver packets to the port that it last heard from that used the MAC address as a source. Still not likely to balance the traffic in a useful way.

If you only have a few clients and can come up with a few NICs and some cable how about giving each client a dedicated ethernet port on your NFS server and cabling it directly. Or buy one of the newer unmanaged 10GbE switches (e.g. Netgear XS708E) or a gigabit switch with a 10GbE uplink port and throw a 10GbE adapter in the NFS server.

This article about doing 10GbE on the cheap may still be relevant.
 
You must log in or register to reply here.
Back
Top