commvault.com commvault.com
Question

Has anyone noticed their LACP bonds are not balanced across the interfaces for hyperscale.

  • 6 March 2023
  • 2 replies
  • 255 views
D
Rank
Badge +2

We’ve noticed that while using LACP mode 4  on our dell r740xd2 hyerperscales that the interfaces have unbalanced amount of traffic if you do IFCONFIG.  I have p1p2 bonded with p5p2 for the storage network.  P1p1 and p5p1 on the data traffic network.  Notice my rx packets and tx packets are very unbalanced.

p1p1 tx packets are at 410Gib and its partner p5p1 is at 11TiB for example.  

 

Does anyone see the same on their LACP config or solved this issue?  We see the same behavior on a dell 48 port switch and the cisco 9k using Cisco ACI.  Also on hyperscale 1.5 and hyperscale x deployments.

 

p1p1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether bc:97:e1:2c:9b:00  txqueuelen 1000  (Ethernet)
        RX packets 11213468144  bytes 14318195329557 (13.0 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1279302179  bytes 440269032106 (410.0 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p1p2: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether bc:97:e1:2c:9b:01  txqueuelen 1000  (Ethernet)
        RX packets 7873700537  bytes 9617619985802 (8.7 TiB)
        RX errors 1  dropped 0  overruns 0  frame 59
        TX packets 28099434107  bytes 32698396187309 (29.7 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p5p1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether bc:97:e1:2c:9b:00  txqueuelen 1000  (Ethernet)
        RX packets 9995455993  bytes 13642001835485 (12.4 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10186853507  bytes 12253200378182 (11.1 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p5p2: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether bc:97:e1:2c:9b:01  txqueuelen 1000  (Ethernet)
        RX packets 24459306896  bytes 29464733404296 (26.7 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11148  bytes 1381894 (1.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 

2 replies

M
Rank
Badge

consider that there’s also a switch element that determines how traffic through ether-channel member interfaces will distribute.  Depends on the vendor, but the options are typically:

source-destination-mac

source-destination-ip

source-destination-port

some higher-end switches will have additional methods (L4 f.x.)

If the method is src-dst-mac and flows are such that the source mac is always the same then you’ll likely get an imbalance on the receive side.  Often, this is a switch-wide setting, and so all port-channels will be affected by any change - even uplinks from switch to core.  Your network admin may be willing to try different options to see if there’s any impact.

chrisknows
Rank
Userlevel 3
Badge +10

I have never seen a properly balanced LACP link ever. In any scenario, Window, Linux, Unix.

In fact I have never heard someone explain exactly how to force them to be balanced. Typically rather than getting double the bandwidth and redundancy, you just Get redundancy. 

 

I am NOT a networking guy but every time I have ever gone down the rabbit hole with any networking team I have ever worked with I have not gotten a straight answer on what is needed to get this working. 

 

does this help you in any way? 

Nope. It’s just a pet peeve.

The poster formerly known as @christopherlecky

Reply


Terms & ConditionsCookie settings