Vport-Based Load Balancing on a LAG
This chapter describes Vport-based load balancing on a LAG.
Appendix with configuration files
Applicability
The information and MD-CLI configuration in this chapter is based on SR OS Release 22.10.R1.
Vport-based load balancing on a LAG applies to all service router (SR) systems based on FP4 and higher. This functionality is not supported on Virtualized Service Router (VSR) because VSR does not support Vports.
The target audience for this chapter is subscriber management users who are already familiar with subscriber management and EVPN VPWS. In this chapter, the emphasis is on the LAG configuration.
Overview
In enhanced subscriber management (ESM), a Vport is a representation of a downstream access node that hosts subscribers. A Vport is used to control bandwidth toward the access node. When deployed on a link aggregation group (LAG), Vports must be configured on every LAG member port, as shown in the figure Vport concept.
The Vport bandwidth in a LAG can be configured in two ways:
- The configured Vport bandwidth is assigned to every LAG member port.
- The Vport on each link gets only a fraction of the total bandwidth of the access node that it represents.
In both cases, the configured Vport bandwidth corresponds to the full capacity of the access node that it controls (a bandwidth cap or ceiling per access node). The difference is in how this bandwidth is assigned by the system to each LAG member port, which is controlled by the lag access adapt-qos mode link | port-fair command.
When the configured Vport bandwidth is assigned to every LAG member port, oversubscription of the access node must be avoided by directing all traffic associated with a Vport to a single LAG member link. This configuration requires load balancing per Vport and is the focus of this chapter. Per-Vport load balancing is suitable in environments with a large variation in bandwidth consumption between subscribers within a Vport.
When the Vport on each link gets only a fraction of the total bandwidth of the access node, the configured Vport bandwidth is automatically distributed over the LAG member links: the Vport bandwidth per LAG member port is the Vport bandwidth divided by the number of LAG member ports.
In other words, each LAG member port gets an equal share of the configured bandwidth. This scenario calls for load balancing per subscriber, where traffic toward the same access node is distributed over the member links. This scenario is suitable for environments with subscribers with similar bandwidth requirements. Otherwise, a few high-bandwidth-consuming subscribers within a Vport can use the same link, causing the aggregate bandwidth to exceed the Vport cap per LAG member port. This issue can cause unnecessary congestion and packet drops, while the instance of the same Vport on the other links remain underutilized.
In Releases earlier than Release 22.10, the implementation of load balancing per Vport, where the configured Vport bandwidth is assigned to every member, implies the following:
- The subscriber SAPs must be instantiated on all LAG member ports, which has a negative effect on the subscriber scalability that is driven by the finite number of SAPs per forwarding complex.
- All Vports are accounted with equal weight in the hashing algorithm, regardless of the configured bandwidth of the Vports, which could lead to suboptimal load balancing when Vports with different bandwidths are present on the LAG.
Release 22.10 expands load balancing per Vport to include the class and weight per Vport and the support for a single subscriber SAP instance per forwarding complex.
The focus of this chapter is on the class- and weight-based Vport load balancing where a single copy of a SAP is created per forwarding complex for each subscriber.
The figure Example topology shows the example topology used in this chapter.
In this setup:
- The subscribers are instantiated in 1:1 mode of operation (one subscriber per SAP) on a pseudowire (PW) port in the Broadband Network Gateway (BNG) "BNG-1".
- The PW-port is the termination point for the EVPN VPWS, which connects the BNG to an aggregation node "AGG-1" in the access network.
- The PW-port is based on the Forwarding Path Extension (FPE), with the LAG "lag-3" containing two PXC member ports, pxc-1.b and pxc-2.b.
- There are 16 Vports configured on each of the PXC ports in the LAG "lag-3".
- 32 IPv4 subscribers are associated with these Vports. The individual subscriber association with Vports is shown in Table 1.
- The EVPN VPWS connecting the BNG to "AGG-1" is configured over the LAG "lag-1" with two member ports on the BNG (ports 1/1/c1/1 and 1/1/c1/2).
- Traffic is sent in the downstream direction, from traffic generator port "p2" toward the simulated subscribers on traffic generator port "p1".
- The traffic flow and traffic load balancing in the BNG is examined at two
points:
- on the "lag-3" underlaying the PW-port, where traffic is load-balanced per Vport in the ESM context
- on the "lag-1" where traffic is load-balanced in the service context, outside of the ESM
- Bandwidth utilization on the LAG links is examined via show and monitor commands at the BNG.
Traffic load balancing on the LAGs "lag-1" and "lag-3" are independent of each other. Traffic on the LAG "lag-3" is load-balanced in the subscriber context and is Vport-aware, while traffic on the LAG "lag-1" is disjoined from the subscriber context. On the LAG "lag-1", service hashing mechanisms apply; on the LAG "lag-3", subscriber management hashing mechanisms apply.
To demonstrate the two-stage load balancing, first on "lag-3" and then on "lag-1", an example topology with subscribers instantiated on a FPE based PW-port is chosen.
Although the focus of this chapter is class- and weight-based Vport load balancing, using the same traffic patterns, in addition, the following two other types of load balancing are evaluated:
- load balancing based on Vports without class and weights
- load balancing based on subscribers (subscriber IDs that are internally assigned identifiers for each subscriber)
Comparing the three types of load balancing helps to form a better overall picture about load balancing traffic options in a subscriber management context.
Association between subscribers and Vports
Subscribers are associated with Vports during the subscriber authentication phase. In this example, RADIUS is used for authentication.
The table Subscriber association with Vports shows the mapping between subscribers and Vport names, Vport configured rates (bandwidth), classes, weights, and transmission (Tx) rates in frames per second (FPS) of generated traffic (offered traffic).
| Class | Vport | Weight | Vport rate in kb/s | Tx rate FPS | Subscriber |
|---|---|---|---|---|---|
| 1 | vport-1 | 15 | 9 000 000 | 900 | 1 |
| 1 | vport-2 | 5 | 3 000 000 | 300 | 2, 3 |
| 1 | vport-3 | 5 | 3 000 000 | 300 | 4, 5 |
| 1 | vport-4 | 5 | 3 000 000 | 300 | 6, 7 |
| 2 | vport-5 | 15 | 3 000 000 | 300 | 8 |
| 2 | vport-6 | 5 | 1 000 000 | 100 | 9, 10 |
| 2 | vport-7 | 5 | 1 000 000 | 100 | 11, 12 |
| 2 | vport-8 | 5 | 1 000 000 | 100 | 13, 14 |
| 3 | vport-9 | 7 | 1 400 000 | 140 | 15 |
| 3 | vport-10 | 1 | 200 000 | 20 | 16, 17 |
| 3 | vport-11 | 1 | 200 000 | 20 | 18, 19 |
| 3 | vport-12 | 1 | 200 000 | 20 | 20, 21 |
| 3 | vport-13 | 1 | 200 000 | 20 | 22, 23 |
| 3 | vport-14 | 1 | 200 000 | 20 | 24, 25 |
| 3 | vport-15 | 1 | 200 000 | 20 | 26, 27 |
| 3 | vport-16 | 1 | 200 000 | 20 | 28, 29, 30, 31, 32 |
The Vports in SR have names from "vport-1" to "vport-16".
The class represents a bandwidth tier. Vports with similarly configured bandwidth values can be grouped into classes and can be load-balanced independently of Vports in other classes. For example, Vports in class 1 are distributed over the LAG member ports independently of the Vports in classes 2 and 3.
The weight is configured for each Vport. Within each class, the hashing algorithm tries to equalize the sum of weights across the LAG member ports.
The traffic generator sends a traffic stream for each subscriber. For example, the traffic generator sends 900 fps toward subscriber 1 associated with Vport "vport-1", while toward the subscribers 28, 29, 30, 31, and 32 associated with "vport-16", the traffic generator sends an aggregated rate of 20 fps, or 4 fps toward each subscriber. The packet size is uniform (200 bytes) across the subscribers.
The selected traffic rates represent the Vport bandwidth. In other words, the ratio of the traffic rates within each Vport is the same as the ratio between the configured Vport bandwidths, which allows for a more intuitive interpretation of the results.
Because this setup is run on PC-based sims, the selected offered rates are low (tens of fps rather than millions of fps). However, the rates are high enough for the purpose of this chapter.
For brevity, the subscribers in this table are represented by numerical values. In SR, the subscriber numbers are prepended with the string "ipoe -", for example, subscriber 1 in the table represents subscriber "ipoe-1" in the SR.
The table Subscriber association with Vports shows that the configured Vport bandwidth and, consequently, the transmission rates are selected in a very unbalanced way. These unbalanced values are used to allow for a clearer interpretation of the results that are presented later in this chapter.
For example, traffic for "vport-9" is seven times higher than traffic in each of the Vports from 10 to 16. If the eight Vports from 9 to 16 were distributed without regarding their configured Vport bandwidth, then the most likely outcome would be that four Vports would be hashed to one link and four to the other. This outcome would result in a very uneven load balancing with traffic utilization on one link being more than twice the utilization on the other link.
Vport classes and weights ensure that uneven load balancing does not happen and that traffic is more evenly split across the two LAG member ports.
The figure Vport bandwidth distribution is a graphical representation of the Vport bandwidth in the table Subscriber association with Vports, where the size of each rectangle represents the Vport bandwidth configured rate. Vports in different classes are represented in different colors. Vports are divided into smaller areas, where each area represents the bandwidth of subscribers associated with the Vport. For example, the size of Vport 1 (9 Gb/s) is three times the size of each of the Vports 2, 3, 4, and 5 (3 Gb/s), or 45 times the size of any of the Vports from 10 to 16 (200 Mb/s). The SR independently attempts to equalize traffic in class 1, class 2, and class 3.
Configuration
Complete configuration files for the AGG-1 and the BNG-1 node in this topology are provided in the appendix.
This section describes key configuration blocks and show output on the BNG.
The configuration section is split into the three following sections:
- FPE
- Vports
- LAG
FPE
The setup on the BNG requires FPE-based PXCs for the PW-port. An arbitrary choice is made to select a port-based PXC instead of the MAC- or internal-based PXC. The following two PXCs will become part of the PXC-based LAG on which the subscribers are instantiated.
[pr:/configure port-xc]
A:admin@bng-1#
pxc 1 {
admin-state enable
description "pw-port lag 2,3"
port-id 1/1/c4/2
}
pxc 2 {
admin-state enable
description "pw-port lag 2,3"
port-id 1/1/c4/3
}
A:admin@bng-1# /show port-xc
===============================================================================
Port Cross-Connect Information
===============================================================================
PXC Admin Oper Port Description
Id State State Id
-------------------------------------------------------------------------------
1 Up Up 1/1/c4/2 pw-port lag 2,3
2 Up Up 1/1/c4/3 pw-port lag 2,3
-------------------------------------------------------------------------------
No. of PXCs: 2
===============================================================================
The logical PXC ports must be explicitly configured and enabled by the operator:
[pr:/configure]
A:admin@bng-1#
port pxc-1.a {
admin-state enable
}
port pxc-1.b {
admin-state enable
}
port pxc-2.a {
admin-state enable
}
port pxc-2.b {
admin-state enable
}
The FPE configuration that is required for the PW-port is as follows:
[pr:/configure]
A:admin@bng-1#
fwd-path-ext {
sdp-id-range {
start 17500
end 17600
}
fpe 1 {
description "pw-port on a lag"
path {
xc-lag-a "lag-2"
xc-lag-b "lag-3"
}
application {
pw-port-extension {
}
}
}
}
A:admin@bng-1#/show fwd-path-ext fpe 1
===============================================================================
FPE Id: 1
===============================================================================
Description : pw-port on a lag
Multi-Path : Disabled
Path : lag-2, lag-3
Pw Port Extension : Enabled Oper : up
Sub Mgmt Extension : Disabled Oper : N/A
Vxlan : Disabled Oper : down
Segment-Routing V6 : Disabled
If-A Qos Policy : default
If-B Qos Policy : default
===============================================================================
Vports
The egress QoS hierarchy for the subscriber is the following:
- subscriber queues with their own rates and buffering capabilities
- the aggregate rate of the subscriber – the total traffic from all the queues of a subscriber cannot exceed this rate
- the aggregate rate of a group of subscribers connected to the same access node or a PON – this aggregate rate is controlled by a Vport
- the aggregate rate of subscribers connected to the same local port – this aggregate rate is controlled by a port-scheduler-policy
Vports are configured on every member port of "lag-3". The LAG underlies the PW-port on which the subscribers are terminated. The member ports of "lag-3" are pxc-1.b and pxc-2.b.
The Vports are attached to the port scheduler via a port scheduler policy named "lag-3-pxc", which is shown in the following configuration.
Only the configuration of the pxc-1.b is shown. The configuration of port-2.b is identical to that for port pxc-1.b.
Each Vport is configured with an aggregate rate, the class, and the weight as described in the table Subscriber association with Vports.
The mapping between the Vport and the subscriber is based on the
int-dest-id string returned by RADIUS during subscriber
authentication. For example, the RADIUS-returned string for subscriber "ipoe-1"
is carried in the Nokia VSA Alc-Int-Dest-Id-Str with value
"vport-1".
The Vport configuration and port-scheduler policy reference under pxc-1.b (a member port of "lag-3") is as follows:
pr:/configure port pxc-1.b]
A:admin@bng-1#
admin-state enable
description "PXC lag-3, pw-port termination"
ethernet {
access {
egress {
virtual-port "vport-1" {
aggregate-rate {
rate 9000000
}
host-match {
int-dest-id "vport-1" { }
}
lag-per-link-hash {
class 1
weight 15
}
}
virtual-port "vport-2" {
aggregate-rate {
rate 3000000
}
host-match {
int-dest-id "vport-2" { }
}
lag-per-link-hash {
class 1
weight 5
}
}
virtual-port "vport-3" {
aggregate-rate {
rate 3000000
}
host-match {
int-dest-id "vport-3" { }
}
lag-per-link-hash {
class 1
weight 5
}
}
virtual-port "vport-4" {
aggregate-rate {
rate 3000000
}
host-match {
int-dest-id "vport-4" { }
}
lag-per-link-hash {
class 1
weight 5
}
}
virtual-port "vport-5" {
aggregate-rate {
rate 3000000
}
host-match {
int-dest-id "vport-5" { }
}
lag-per-link-hash {
class 2
weight 15
}
}
virtual-port "vport-6" {
aggregate-rate {
rate 1000000
}
host-match {
int-dest-id "vport-6" { }
}
lag-per-link-hash {
class 2
weight 5
}
}
virtual-port "vport-7" {
aggregate-rate {
rate 1000000
}
host-match {
int-dest-id "vport-7" { }
}
lag-per-link-hash {
class 2
weight 5
}
}
virtual-port "vport-8" {
aggregate-rate {
rate 1000000
}
host-match {
int-dest-id "vport-8" { }
}
lag-per-link-hash {
class 2
weight 5
}
}
virtual-port "vport-9" {
aggregate-rate {
rate 1400000
}
host-match {
int-dest-id "vport-9" { }
}
lag-per-link-hash {
class 3
weight 7
}
}
virtual-port "vport-10" {
aggregate-rate {
rate 200000
}
host-match {
int-dest-id "vport-10" { }
}
lag-per-link-hash {
class 3
weight 1
}
}
virtual-port "vport-11" {
aggregate-rate {
rate 200000
}
host-match {
int-dest-id "vport-11" { }
}
lag-per-link-hash {
class 3
weight 1
}
}
virtual-port "vport-12" {
aggregate-rate {
rate 200000
}
host-match {
int-dest-id "vport-12" { }
}
lag-per-link-hash {
class 3
weight 1
}
}
virtual-port "vport-13" {
aggregate-rate {
rate 200000
}
host-match {
int-dest-id "vport-13" { }
}
lag-per-link-hash {
class 3
weight 1
}
}
virtual-port "vport-14" {
aggregate-rate {
rate 200000
}
host-match {
int-dest-id "vport-14" { }
}
lag-per-link-hash {
class 3
weight 1
}
}
virtual-port "vport-15" {
aggregate-rate {
rate 200000
}
host-match {
int-dest-id "vport-15" { }
}
lag-per-link-hash {
class 3
weight 1
}
}
virtual-port "vport-16" {
aggregate-rate {
rate 200000
}
host-match {
int-dest-id "vport-16" { }
}
lag-per-link-hash {
class 3
weight 1
}
}
}
}
egress {
port-scheduler-policy {
policy-name "lag-3-pxc"
}
}
}
The following command shows the Vport instances created under port pxc-1.b:
A:admin@bng-1# show port pxc-1.b vport
===============================================================================
Port pxc-1.b Access Egress vport
===============================================================================
VPort Name : vport-1
Description : (Not Specified)
Port Sched Policy : None
Sched Policy : None
Rate Limit : 9000000
Limit Unused BW : Disabled
Rate Modify : disabled
Modify delta : 0
Monitor Port Sched : Disabled
Lag PLHW class : 1
Lag PLHW weight : 15
Host-Matches
-------------------------------------------------------------------------------
Dest: vport-1
-------------------------------------------------------------------------------
VPort Name : vport-2
Description : (Not Specified)
Port Sched Policy : None
Sched Policy : None
Rate Limit : 3000000
Limit Unused BW : Disabled
Rate Modify : disabled
Modify delta : 0
Monitor Port Sched : Disabled
Lag PLHW class : 1
Lag PLHW weight : 5
Host-Matches
-------------------------------------------------------------------------------
Dest: vport-2
-------------------------------------------------------------------------------
VPort Name : vport-3
Description : (Not Specified)
Port Sched Policy : None
Sched Policy : None
Rate Limit : 3000000
Limit Unused BW : Disabled
Rate Modify : disabled
Modify delta : 0
Monitor Port Sched : Disabled
Lag PLHW class : 1
Lag PLHW weight : 5
Host-Matches
-------------------------------------------------------------------------------
Dest: vport-3
-------------------------------------------------------------------------------
---snip---
The following command shows the association between the Vports and the subscribers for port pxc-1.b:
A:admin@bng-1# show port pxc-1.b vport associations
===============================================================================
Port pxc-1.b Access Egress vport
===============================================================================
-------------------------------------------------------------------------------
VPort "vport-1"
-------------------------------------------------------------------------------
svc-id : 10
sap : pw-1:1.1
subscr: ipoe-1
ip : 10.10.0.16
mac : 00:14:01:00:00:01 pppoe-sid: N/A
-------------------------------------------------------------------------------
VPort "vport-2"
-------------------------------------------------------------------------------
svc-id : 10
sap : pw-1:1.2
subscr: ipoe-2
ip : 10.10.0.21
mac : 00:14:01:00:00:02 pppoe-sid: N/A
svc-id : 10
sap : pw-1:1.3
subscr: ipoe-3
ip : 10.10.0.41
mac : 00:14:01:00:00:03 pppoe-sid: N/A
-------------------------------------------------------------------------------
VPort "vport-3"
-------------------------------------------------------------------------------
svc-id : 10
sap : pw-1:1.4
subscr: ipoe-4
ip : 10.10.0.24
mac : 00:14:01:00:00:04 pppoe-sid: N/A
svc-id : 10
sap : pw-1:1.5
subscr: ipoe-5
ip : 10.10.0.38
mac : 00:14:01:00:00:05 pppoe-sid: N/A
---snip---
The following configuration shows that each subscriber has only one egress queue, which is port-parented. This queue is configured in the egress SAP policy, which is then referenced in the SLA profile assigned to the subscriber during the authentication phase.
Port-schedule policy "lag-3-pxc" is also configured under the QoS hierarchy, using default values in this case (which is why there is nothing explicitly configured under port-scheduler-policy).
[pr:/configure qos]
A:admin@bng-1#
sap-egress "sap-egess-1" {
policy-id 2
queue 1 {
port-parent {
}
}
}
port-scheduler-policy "lag-3-pxc" {
}
The SLA profile for the subscriber must include vport as the port-parent location, otherwise the subscriber would not be associated with the Vport:
[pr:/configure subscriber-mgmt sla-profile "sla-profile-1"]
A:admin@bng-1#
egress {
qos {
sap-egress {
policy-name "sap-egess-1"
port-parent-location vport
}
}
}
There are 32 subscribers instantiated in the BNG:
A:admin@bng-1# show service active-subscribers summary
============================================================================
Active Subscriber table summary
============================================================================
Total Count : 32
============================================================================
The following show command displays the subscriber hierarchy:
A:admin@bng-1# show service active-subscribers hierarchy
=============================================================================
Active Subscribers Hierarchy
=============================================================================
-- ipoe-1
(sub-profile-1)
|
+-- sap:[pw-1:1.1] - sla:sla-profile-1
|
+-- IPOE-session - mac:00:14:01:00:00:01 - svc:10
|
+-- 10.10.0.25 - DHCP
-- ipoe-2
(sub-profile-1)
|
+-- sap:[pw-1:1.2] - sla:sla-profile-1
|
+-- IPOE-session - mac:00:14:01:00:00:02 - svc:10
|
+-- 10.10.0.26 - DHCP
-- ipoe-3
(sub-profile-1)
|
+-- sap:[pw-1:1.3] - sla:sla-profile-1
|
+-- IPOE-session - mac:00:14:01:00:00:03 - svc:10
|
+-- 10.10.0.27 - DHCP
---snip---
-----------------------------------------------------------------------------
Number of active subscribers : 32
Flags: (N) = the host or the managed route is in non-forwarding state
=============================================================================
---snip---
Some of the additional show commands that can be used to display subscriber states are:
show service active-subscribers detail
show service id "vport-hashing" subscriber-hosts detail
show service id "vport-hashing" ipoe session detail
show service id "vport-hashing" dhcp lease-state detail
The following show commands display information related to the QoS hierarchy:
show qos scheduler-hierarchy port pxc-1.b detail
show qos agg-rate port pxc-1.b vport "vport-1" detail
LAG
The two LAGs examined in this chapter are:
- PXC LAG "lag-3", which underlies the PW-port where the subscribers are terminated. This LAG is implicitly applied to the ESM context(directly via FPE, indirectly via PW-port and PW capture SAP) where load balancing is performed based on Vports.
- LAG "lag-1" connects the BNG to the aggregation node "AGG-1". This LAG is applied in the service context. Load balancing on this LAG is performed based on the service context.
This chapter examines load balancing over member links for both LAGs.
PXC LAG "lag-3" (ESM)
The configuration for "lag-3" is the following:
[pr:/configure lag "lag-3"]
A:admin@bng-1#
admin-state enable
description "fpe pw-port pxc lag - termination side"
mode hybrid
max-ports 64
access {
per-fp-ing-queuing true
per-fp-egr-queuing true
per-fp-sap-instance true
adapt-qos {
mode link
}
}
per-link-hash {
weighted {
subscriber-hash-mode vport
}
}
port pxc-1.b {
}
port pxc-2.b {
}
The per-link-hash command enables weighted Vport hashing in the subscriber context.
If classes and weights for Vports are not configured, hashing is still performed per Vport, in which case all Vports are treated equally, regardless of the bandwidth.
With the per-fp-sap-instance command enabled, the system allocates only one SAP instance per subscriber per forwarding complex.
For example, if a LAG has two member ports on the same forwarding complex, then for each subscriber, only one SAP instance is allocated (as opposed to one SAP instance on each member port). The result of this allocation is improved scaling because the number of SAPs per forwarding complex is finite. The scaling improvement can be verified with the following commands.
Before any subscriber is instantiated, the number of subscriber hosts and SLA profile instances is 0. A number of SAPs are already allocated (three in this case), related to non-subscriber interfaces (EVPN VPWS, regular interfaces, and so on).
A:admin@bng-1# tools dump resource-usage card "1" fp 1
=============================================================================
Resource Usage Information for Card Slot #1 FP #1
=============================================================================
Total Allocated Free
-----------------------------------------------------------------------------
Subscriber Hosts - 262143 0 262143
Subscriber SLA Profile Instances | 131071 0 131071
SAP Instances | 98303 3 98300
=============================================================================
After 32 subscribers are instantiated on the PW-port and on "lag-3", only 32 SAP instances are allocated:
The following command shows that 32 subscriber hosts and 35 SAP instances are allocated:
A:admin@bng-1# /tools dump resource-usage card 1 fp 1
=============================================================================
Resource Usage Information for Card Slot #1 FP #1
=============================================================================
Total Allocated Free
-----------------------------------------------------------------------------
Subscriber Hosts - 262143 32 262111
Subscriber SLA Profile Instances | 131071 32 131039
SAP Instances | 98303 35 98268
=============================================================================
The per-fp-ing-queuing and per-fp-egr-queuing commands have similar effect for queues, although the per-fp-egr-queuing true command is the only mode of operation in subscriber management on a LAG.
The access adapt-qos mode link command instructs the system to apply the configured Vport bandwidth to every member port of a LAG. This command is required in per-Vport load balancing.
The alternative mode is port-fair, where the configured Vport bandwidth is divided across the member ports in the LAG (each member port gets a share of the configured bandwidths). Per-Vport load balancing is not recommended in this case due to likely overshoot of the Vport assigned bandwidth to each member port, which would result in unnecessary packet drops.
Examining Vport load balancing effects
For a better understanding of what per-Vport load balancing means, the results for three cases are compared:
- traffic utilization on LAG member links with weighted (for class and weight) per-Vport load balancing
- traffic utilization on LAG member links with Vport load balancing but without classes and weights configured
- traffic utilization on LAG member links with per-subscriber load balancing (Vport load balancing is disabled)
The expectation is that there is a higher degree of load balancing in the first case than in the next two cases. This different degree for load balancing is because of the unequal nature of the generated traffic, where some subscribers send significantly more traffic than others, and the inability of the latter two cases to deal with this lack of balance.
Load balancing per Vport with classes and weights
The configuration file for BNG-1 can be found here.
In this setup, classes and weights configured under Vports are considered in the hashing algorithm. The following command shows how Vports are distributed over the LAG member ports. Although all Vports are instantiated on both member ports of "lag-3", this command shows how the actual traffic for each Vport is distributed over the member links of "lag-3":
A:admin@bng-1# show lag "lag-3" associations per-link-hash vport
=====================================================
VPort Associations
=====================================================
Vport Name Active Link
-----------------------------------------------------
vport-1 pxc-1.b
vport-2 pxc-2.b
vport-3 pxc-2.b
vport-4 pxc-2.b
vport-5 pxc-1.b
vport-6 pxc-2.b
vport-7 pxc-2.b
vport-8 pxc-2.b
vport-9 pxc-2.b
vport-10 pxc-1.b
vport-11 pxc-1.b
vport-12 pxc-1.b
vport-13 pxc-1.b
vport-14 pxc-1.b
vport-15 pxc-1.b
vport-16 pxc-1.b
=====================================================
Number of VPort associations: 16
=====================================================
The following command shows how subscriber traffic is distributed over the member ports, based on each subscriber’s association with a Vport. For example, the subscriber "ipoe-1" is associated with "vport-1". The output of the previous command shows that "vport-1" is hashed to port pxc-1.b and, therefore, the LAG active link (as shown below) for the subscriber "ipoe-1" is pxc-1.b.
A:admin@bng-1# show service id "vport-hashing" subscriber-hosts detail
===============================================================================
Subscriber Host table
===============================================================================
Sap
IP Address
MAC Address PPPoE-SID Origin Fwding State
Subscriber
-------------------------------------------------------------------------------
[pw-1:1.1]
10.10.0.42
00:14:01:00:00:01 N/A DHCP Fwding
ipoe-1
-------------------------------------------------------------------------------
Subscriber-interface : sub-int-1
Group-interface : group-int-1
Sub Profile : sub-profile-1
SLA Profile : sla-profile-1
App Profile : N/A
Egress Q-Group : N/A
Egress Vport : vport-1
LAG Active Link : pxc-1.b
Acct-Session-Id : B496FF00000070639A19D0
Acct-Q-Inst-Session-Id: B496FF00000071639A19D0
Address Origin : Dynamic
OT HTTP Rdr IP-FltrId : N/A
OT HTTP Rdr Status : N/A
OT HTTP Rdr Fltr Src : N/A
HTTP Rdr URL Override : N/A
GTP local break-out : No
DIAMETER session ID Gx: N/A
-------------------------------------------------------------------------------
[pw-1:1.2]
10.10.0.43
00:14:01:00:00:02 N/A DHCP Fwding
ipoe-2
-------------------------------------------------------------------------------
Subscriber-interface : sub-int-1
Group-interface : group-int-1
Sub Profile : sub-profile-1
SLA Profile : sla-profile-1
App Profile : N/A
Egress Q-Group : N/A
Egress Vport : vport-2
LAG Active Link : pxc-2.b
Acct-Session-Id : B496FF00000072639A19D0
Acct-Q-Inst-Session-Id: B496FF00000073639A19D0
Address Origin : Dynamic
OT HTTP Rdr IP-FltrId : N/A
OT HTTP Rdr Status : N/A
OT HTTP Rdr Fltr Src : N/A
HTTP Rdr URL Override : N/A
GTP local break-out : No
DIAMETER session ID Gx: N/A
-------------------------------------------------------------------------------
---snip---
The table Vport distribution over member ports repeats the Vport distribution shown in the table Subscriber association with Vports, but adds a column with the active member link for each Vport:
| Class | Vport | Weight | Vport rate in kb/s | Tx rate FPS | Subscriber | Member link |
|---|---|---|---|---|---|---|
| 1 | vport-1 | 15 | 9 000 000 | 900 | 1 | pxc-1.b |
| 1 | vport-2 | 5 | 3 000 000 | 300 | 2, 3 | pxc-2.b |
| 1 | vport-3 | 5 | 3 000 000 | 300 | 4, 5 | pxc-2.b |
| 1 | vport-4 | 5 | 3 000 000 | 300 | 6, 7 | pxc-2.b |
| 2 | vport-5 | 15 | 3 000 000 | 300 | 8 | pxc-1.b |
| 2 | vport-6 | 5 | 1 000 000 | 100 | 9, 10 | pxc-2.b |
| 2 | vport-7 | 5 | 1 000 000 | 100 | 11, 12 | pxc-2.b |
| 2 | vport-8 | 5 | 1 000 000 | 100 | 13, 14 | pxc-2.b |
| 3 | vport-9 | 7 | 1 400 000 | 140 | 15 | pxc-2.b |
| 3 | vport-10 | 1 | 200 000 | 20 | 16, 17 | pxc-1.b |
| 3 | vport-11 | 1 | 200 000 | 20 | 18, 19 | pxc-1.b |
| 3 | vport-12 | 1 | 200 000 | 20 | 20, 21 | pxc-1.b |
| 3 | vport-13 | 1 | 200 000 | 20 | 22, 23 | pxc-1.b |
| 3 | vport-14 | 1 | 200 000 | 20 | 24, 25 | pxc-1.b |
| 3 | vport-15 | 1 | 200 000 | 20 | 26, 27 | pxc-1.b |
| 3 | vport-16 | 1 | 200 000 | 20 | 28, 29, 30, 31, 32 | pxc-1.b |
To verify the Vport distribution over member ports, the traffic on the member ports of "lag-3" can be monitored.
An efficient way to observe utilization on ports in an SR is to run a monitor port command that periodically collects the port data (forwarded octets and packets) and displays them in various forms (raw counts or as rates). This monitor port command is used on regular ports on "lag-1" in the next section. However, the monitor port command cannot be used on PXC ports.
Because the monitor port command is not supported on PXC ports, a dedicated monitor-lag command is developed via pysros. The output of this command shows the distribution of traffic per LAG member port, per Vport, and per subscriber. Rates are displayed in percentages of the total downstream traffic sent toward all subscribers. During each command execution, the data is, by default, collected three times in 10 second intervals (this frequency is configurable).
This example demonstrates that bandwidth is evenly distributed over the two LAG member ports (approximately 50% on pxc-1.b port and 50% on pxc-2.b port). Subscriber rates and subscriber to Vport mappings are also consistent with the expectations according to the table Vport distribution over member ports .
The output shows that in each class, the high-bandwidth Vports ("vport-1" in class 1, "vport-5" in class 2, and "vport-9" in class 3) are counterbalanced with lower-bandwidth Vports from the same class that are hashed to the opposite link. In this way, load balancing is distributed in 50/50 ratio over the two member ports on the "lag-3": both member ports pxc-1.b and pxc-2.b get approximately 50% of the load.
A:admin@bng-1# monitor-lag -l lag-3 -i 10 -r 3
-----------------------------------------------------------------------
Run 1
-----------------------------------------------------------------------
Object | BW % of Total
-----------------------------------------------------------------------
pxc-1.b------------------------------------49.41
|--------vport-1------------------------------------34.41
| |--------ipoe-1-------------------------------------34.41
|--------vport-2-------------------------------------0.00
|--------vport-3-------------------------------------0.00
|--------vport-4-------------------------------------0.00
|--------vport-5------------------------------------11.47
| |--------ipoe-8-------------------------------------11.47
|--------vport-6-------------------------------------0.00
|--------vport-7-------------------------------------0.00
|--------vport-8-------------------------------------0.00
|--------vport-9-------------------------------------0.00
|--------vport-10------------------------------------0.59
| |--------ipoe-16-------------------------------------0.29
| |--------ipoe-17-------------------------------------0.29
|--------vport-11------------------------------------0.59
| |--------ipoe-19-------------------------------------0.29
| |--------ipoe-18-------------------------------------0.29
|--------vport-12------------------------------------0.59
| |--------ipoe-20-------------------------------------0.29
| |--------ipoe-21-------------------------------------0.29
|--------vport-13------------------------------------0.59
| |--------ipoe-23-------------------------------------0.29
| |--------ipoe-22-------------------------------------0.29
|--------vport-14------------------------------------0.59
| |--------ipoe-25-------------------------------------0.29
| |--------ipoe-24-------------------------------------0.29
|--------vport-15------------------------------------0.59
| |--------ipoe-27-------------------------------------0.29
| |--------ipoe-26-------------------------------------0.29
|--------vport-16------------------------------------0.00
|--------ipoe-32-------------------------------------0.00
|--------ipoe-30-------------------------------------0.00
|--------ipoe-31-------------------------------------0.00
|--------ipoe-29-------------------------------------0.00
|--------ipoe-28-------------------------------------0.00
pxc-2.b------------------------------------50.59
|--------vport-1-------------------------------------0.00
|--------vport-2------------------------------------11.47
| |--------ipoe-3--------------------------------------5.88
| |--------ipoe-2--------------------------------------5.59
|--------vport-3------------------------------------11.47
| |--------ipoe-4--------------------------------------5.88
| |--------ipoe-5--------------------------------------5.59
|--------vport-4------------------------------------11.18
| |--------ipoe-7--------------------------------------5.59
| |--------ipoe-6--------------------------------------5.59
|--------vport-5-------------------------------------0.00
|--------vport-6-------------------------------------3.53
| |--------ipoe-10-------------------------------------1.76
| |--------ipoe-9--------------------------------------1.76
|--------vport-7-------------------------------------3.53
| |--------ipoe-12-------------------------------------1.76
| |--------ipoe-11-------------------------------------1.76
|--------vport-8-------------------------------------4.12
| |--------ipoe-14-------------------------------------2.06
| |--------ipoe-13-------------------------------------2.06
|--------vport-9-------------------------------------5.29
| |--------ipoe-15-------------------------------------5.29
|--------vport-10------------------------------------0.00
|--------vport-11------------------------------------0.00
|--------vport-12------------------------------------0.00
|--------vport-13------------------------------------0.00
|--------vport-14------------------------------------0.00
|--------vport-15------------------------------------0.00
|--------vport-16------------------------------------0.00
-----------------------------------------------------------------------
Run 2
-----------------------------------------------------------------------
Object | BW % of Total
-----------------------------------------------------------------------
pxc-1.b------------------------------------49.99
|--------vport-1------------------------------------33.58
| |--------ipoe-1-------------------------------------33.58
|--------vport-2-------------------------------------0.00
|--------vport-3-------------------------------------0.00
|--------vport-4-------------------------------------0.00
|--------vport-5------------------------------------11.20
| |--------ipoe-8-------------------------------------11.20
|--------vport-6-------------------------------------0.00
|--------vport-7-------------------------------------0.00
|--------vport-8-------------------------------------0.00
|--------vport-9-------------------------------------0.00
|--------vport-10------------------------------------0.74
| |--------ipoe-16-------------------------------------0.37
| |--------ipoe-17-------------------------------------0.37
|--------vport-11------------------------------------0.74
| |--------ipoe-19-------------------------------------0.37
| |--------ipoe-18-------------------------------------0.37
|--------vport-12------------------------------------0.74
| |--------ipoe-20-------------------------------------0.37
| |--------ipoe-21-------------------------------------0.37
|--------vport-13------------------------------------0.74
| |--------ipoe-23-------------------------------------0.37
| |--------ipoe-22-------------------------------------0.37
|--------vport-14------------------------------------0.74
| |--------ipoe-25-------------------------------------0.37
| |--------ipoe-24-------------------------------------0.37
|--------vport-15------------------------------------0.74
| |--------ipoe-27-------------------------------------0.37
| |--------ipoe-26-------------------------------------0.37
|--------vport-16------------------------------------0.75
|--------ipoe-32-------------------------------------0.15
|--------ipoe-30-------------------------------------0.15
|--------ipoe-31-------------------------------------0.15
|--------ipoe-29-------------------------------------0.15
|--------ipoe-28-------------------------------------0.15
pxc-2.b------------------------------------50.01
|--------vport-1-------------------------------------0.00
|--------vport-2------------------------------------11.19
| |--------ipoe-3--------------------------------------5.60
| |--------ipoe-2--------------------------------------5.59
|--------vport-3------------------------------------11.20
| |--------ipoe-4--------------------------------------5.60
| |--------ipoe-5--------------------------------------5.60
|--------vport-4------------------------------------11.20
| |--------ipoe-7--------------------------------------5.60
| |--------ipoe-6--------------------------------------5.59
|--------vport-5-------------------------------------0.00
|--------vport-6-------------------------------------3.73
| |--------ipoe-10-------------------------------------1.87
| |--------ipoe-9--------------------------------------1.87
|--------vport-7-------------------------------------3.73
| |--------ipoe-12-------------------------------------1.87
| |--------ipoe-11-------------------------------------1.87
|--------vport-8-------------------------------------3.73
| |--------ipoe-14-------------------------------------1.86
| |--------ipoe-13-------------------------------------1.86
|--------vport-9-------------------------------------5.22
| |--------ipoe-15-------------------------------------5.22
|--------vport-10------------------------------------0.00
|--------vport-11------------------------------------0.00
|--------vport-12------------------------------------0.00
|--------vport-13------------------------------------0.00
|--------vport-14------------------------------------0.00
|--------vport-15------------------------------------0.00
|--------vport-16------------------------------------0.00
-----------------------------------------------------------------------
Run 3
-----------------------------------------------------------------------
Object | BW % of Total
-----------------------------------------------------------------------
pxc-1.b------------------------------------50.01
|--------vport-1------------------------------------33.58
| |--------ipoe-1-------------------------------------33.58
|--------vport-2-------------------------------------0.00
|--------vport-3-------------------------------------0.00
|--------vport-4-------------------------------------0.00
|--------vport-5------------------------------------11.19
| |--------ipoe-8-------------------------------------11.19
|--------vport-6-------------------------------------0.00
|--------vport-7-------------------------------------0.00
|--------vport-8-------------------------------------0.00
|--------vport-9-------------------------------------0.00
|--------vport-10------------------------------------0.74
| |--------ipoe-16-------------------------------------0.37
| |--------ipoe-17-------------------------------------0.37
|--------vport-11------------------------------------0.74
| |--------ipoe-19-------------------------------------0.37
| |--------ipoe-18-------------------------------------0.37
|--------vport-12------------------------------------0.75
| |--------ipoe-20-------------------------------------0.38
| |--------ipoe-21-------------------------------------0.38
|--------vport-13------------------------------------0.75
| |--------ipoe-23-------------------------------------0.38
| |--------ipoe-22-------------------------------------0.38
|--------vport-14------------------------------------0.75
| |--------ipoe-25-------------------------------------0.38
| |--------ipoe-24-------------------------------------0.38
|--------vport-15------------------------------------0.75
| |--------ipoe-27-------------------------------------0.38
| |--------ipoe-26-------------------------------------0.38
|--------vport-16------------------------------------0.74
|--------ipoe-32-------------------------------------0.15
|--------ipoe-30-------------------------------------0.15
|--------ipoe-31-------------------------------------0.15
|--------ipoe-29-------------------------------------0.15
|--------ipoe-28-------------------------------------0.15
pxc-2.b------------------------------------49.99
|--------vport-1-------------------------------------0.00
|--------vport-2------------------------------------11.19
| |--------ipoe-3--------------------------------------5.60
| |--------ipoe-2--------------------------------------5.60
|--------vport-3------------------------------------11.19
| |--------ipoe-4--------------------------------------5.60
| |--------ipoe-5--------------------------------------5.60
|--------vport-4------------------------------------11.19
| |--------ipoe-7--------------------------------------5.60
| |--------ipoe-6--------------------------------------5.60
|--------vport-5-------------------------------------0.00
|--------vport-6-------------------------------------3.73
| |--------ipoe-10-------------------------------------1.86
| |--------ipoe-9--------------------------------------1.86
|--------vport-7-------------------------------------3.73
| |--------ipoe-12-------------------------------------1.87
| |--------ipoe-11-------------------------------------1.86
|--------vport-8-------------------------------------3.74
| |--------ipoe-14-------------------------------------1.87
| |--------ipoe-13-------------------------------------1.87
|--------vport-9-------------------------------------5.22
| |--------ipoe-15-------------------------------------5.22
|--------vport-10------------------------------------0.00
|--------vport-11------------------------------------0.00
|--------vport-12------------------------------------0.00
|--------vport-13------------------------------------0.00
|--------vport-14------------------------------------0.00
|--------vport-15------------------------------------0.00
|--------vport-16------------------------------------0.00
Another way to verify the Vport distribution over member ports is to run the following command:
show port "pxc-1.b" vport statistics
The output of this command shows the number of forwarded octets and packets. By running this command periodically, and with the help of some arithmetic, it can be deduced how much traffic is forwarded by each port.
Load balancing per Vport without classes and weights
In this example, Vport classes and weights are all set to the same value. This scenario is the same as one where classes and weight are not used at all. Traffic is still load-balanced per Vport but with no regard to the bandwidth of each Vport. In other words, all Vports are treated equally, contrary to the fact that their configured bandwidth is very different.
The configuration file for BNG-1 can be found here .
The same commands are used as in the previous example. The output of the following command shows that the ports are hashed very differently than before, with Vports evenly distributed between the two member ports (eight Vports hashed to pxc-1.b and eight Vports hashed to pxc-2.b).
A:admin@bng-1# show lag "lag-3" associations per-link-hash vport
=====================================================
VPort Associations
=====================================================
Vport Name Active Link
-----------------------------------------------------
vport-1 pxc-1.b
vport-2 pxc-2.b
vport-3 pxc-1.b
vport-4 pxc-2.b
vport-5 pxc-2.b
vport-6 pxc-1.b
vport-7 pxc-1.b
vport-8 pxc-2.b
vport-9 pxc-1.b
vport-10 pxc-2.b
vport-11 pxc-1.b
vport-12 pxc-2.b
vport-13 pxc-1.b
vport-14 pxc-2.b
vport-15 pxc-1.b
vport-16 pxc-2.b
=====================================================
Number of VPort associations: 16
=====================================================
The subscriber association with Vports and member ports is consistent with the output of the preceding command:
A:admin@bng-1# show service id "vport-hashing" subscriber-hosts detail
=============================================================================
Subscriber Host table
=============================================================================
Sap
IP Address
MAC Address PPPoE-SID Origin Fwding State
Subscriber
-----------------------------------------------------------------------------
[pw-1:1.1]
10.10.0.74
00:14:01:00:00:01 N/A DHCP Fwding
ipoe-1
-----------------------------------------------------------------------------
Subscriber-interface : sub-int-1
Group-interface : group-int-1
Sub Profile : sub-profile-1
SLA Profile : sla-profile-1
App Profile : N/A
Egress Q-Group : N/A
Egress Vport : vport-1
LAG Active Link : pxc-1.b
Acct-Session-Id : B496FF000000D0639A24FD
Acct-Q-Inst-Session-Id: B496FF000000D1639A24FD
Address Origin : Dynamic
OT HTTP Rdr IP-FltrId : N/A
OT HTTP Rdr Status : N/A
OT HTTP Rdr Fltr Src : N/A
HTTP Rdr URL Override : N/A
GTP local break-out : No
DIAMETER session ID Gx: N/A
-----------------------------------------------------------------------------
[pw-1:1.2]
10.10.0.75
00:14:01:00:00:02 N/A DHCP Fwding
ipoe-2
-----------------------------------------------------------------------------
Subscriber-interface : sub-int-1
Group-interface : group-int-1
Sub Profile : sub-profile-1
SLA Profile : sla-profile-1
App Profile : N/A
Egress Q-Group : N/A
Egress Vport : vport-2
LAG Active Link : pxc-2.b
Acct-Session-Id : B496FF000000D4639A24FD
Acct-Q-Inst-Session-Id: B496FF000000D5639A24FD
Address Origin : Dynamic
OT HTTP Rdr IP-FltrId : N/A
OT HTTP Rdr Status : N/A
OT HTTP Rdr Fltr Src : N/A
HTTP Rdr URL Override : N/A
GTP local break-out : No
DIAMETER session ID Gx: N/A
-----------------------------------------------------------------------------
---snip---
The table Vport distribution over member ports repeats the Vport distribution over member ports from the table Subscriber association with Vports while adding a column with the active member link for each Vport.
| Class | Vport | Weight | Vport rate in kb/s | Tx rate FPS | Subscriber | Member link |
|---|---|---|---|---|---|---|
| 1 | Vport-1 | 15 | 9 000 000 | 900 | 1 | pxc-1.b |
| 1 | Vport-2 | 5 | 3 000 000 | 300 | 2, 3 | pxc-2.b |
| 1 | Vport-3 | 5 | 3 000 000 | 300 | 4, 5 | pxc-1.b |
| 1 | Vport-4 | 5 | 3 000 000 | 300 | 6, 7 | pxc-2.b |
| 2 | Vport-5 | 15 | 3 000 000 | 300 | 8 | pxc-2.b |
| 2 | Vport-6 | 5 | 1 000 000 | 100 | 9, 10 | pxc-1.b |
| 2 | Vport-7 | 5 | 1 000 000 | 100 | 11, 12 | pxc-1.b |
| 2 | Vport-8 | 5 | 1 000 000 | 100 | 13, 14 | pxc-2.b |
| 3 | Vport-9 | 7 | 1 400 000 | 140 | 15 | pxc-1.b |
| 3 | Vport-10 | 1 | 200 000 | 20 | 16, 17 | pxc-2.b |
| 3 | Vport-11 | 1 | 200 000 | 20 | 18, 19 | pxc-1.b |
| 3 | Vport-12 | 1 | 200 000 | 20 | 20, 21 | pxc-2.b |
| 3 | Vport-13 | 1 | 200 000 | 20 | 22, 23 | pxc-1.b |
| 3 | Vport-14 | 1 | 200 000 | 20 | 24, 25 | pxc-2.b |
| 3 | Vport-15 | 1 | 200 000 | 20 | 26, 27 | pxc-2.b |
| 3 | Vport-16 | 1 | 200 000 | 20 | 28, 29, 30, 31, 32 | pxc-2.b |
The output of the monitor-lag command for "lag-3" shows unequal load balancing: member port pxc-1.b receives approximately 60% of the load and member port pxc-2.b only 40%.
A:admin@bng-1# monitor-lag -l lag-3 -i 10 -r 3
-----------------------------------------------------------------------
Run 1
-----------------------------------------------------------------------
Object | BW % of Total
-----------------------------------------------------------------------
pxc-1.b------------------------------------60.29
|--------vport-1------------------------------------34.49
| |--------ipoe-1-------------------------------------34.49
|--------vport-2-------------------------------------0.00
|--------vport-3------------------------------------11.30
| |--------ipoe-4--------------------------------------5.51
| |--------ipoe-5--------------------------------------5.80
|--------vport-4-------------------------------------0.00
|--------vport-5-------------------------------------0.00
|--------vport-6-------------------------------------4.06
| |--------ipoe-10-------------------------------------2.03
| |--------ipoe-9--------------------------------------2.03
|--------vport-7-------------------------------------3.48
| |--------ipoe-12-------------------------------------1.74
| |--------ipoe-11-------------------------------------1.74
|--------vport-8-------------------------------------0.00
|--------vport-9-------------------------------------5.22
| |--------ipoe-15-------------------------------------5.22
|--------vport-10------------------------------------0.00
|--------vport-11------------------------------------0.58
| |--------ipoe-19-------------------------------------0.29
| |--------ipoe-18-------------------------------------0.29
|--------vport-12------------------------------------0.00
|--------vport-13------------------------------------0.58
| |--------ipoe-23-------------------------------------0.29
| |--------ipoe-22-------------------------------------0.29
|--------vport-14------------------------------------0.00
|--------vport-15------------------------------------0.58
| |--------ipoe-27-------------------------------------0.29
| |--------ipoe-26-------------------------------------0.29
|--------vport-16------------------------------------0.00
pxc-2.b------------------------------------39.71
|--------vport-1-------------------------------------0.00
|--------vport-2------------------------------------11.30
| |--------ipoe-3--------------------------------------5.51
| |--------ipoe-2--------------------------------------5.80
|--------vport-3-------------------------------------0.00
|--------vport-4------------------------------------11.30
| |--------ipoe-7--------------------------------------5.51
| |--------ipoe-6--------------------------------------5.80
|--------vport-5------------------------------------11.30
| |--------ipoe-8-------------------------------------11.30
|--------vport-6-------------------------------------0.00
|--------vport-7-------------------------------------0.00
|--------vport-8-------------------------------------3.77
| |--------ipoe-14-------------------------------------2.03
| |--------ipoe-13-------------------------------------1.74
|--------vport-9-------------------------------------0.00
|--------vport-10------------------------------------0.87
| |--------ipoe-16-------------------------------------0.58
| |--------ipoe-17-------------------------------------0.29
|--------vport-11------------------------------------0.00
|--------vport-12------------------------------------0.58
| |--------ipoe-20-------------------------------------0.29
| |--------ipoe-21-------------------------------------0.29
|--------vport-13------------------------------------0.00
|--------vport-14------------------------------------0.58
| |--------ipoe-25-------------------------------------0.29
| |--------ipoe-24-------------------------------------0.29
|--------vport-15------------------------------------0.00
|--------vport-16------------------------------------0.00
|--------ipoe-32-------------------------------------0.00
|--------ipoe-30-------------------------------------0.00
|--------ipoe-31-------------------------------------0.00
|--------ipoe-29-------------------------------------0.00
|--------ipoe-28-------------------------------------0.00
-----------------------------------------------------------------------
Run 2
-----------------------------------------------------------------------
Object | BW % of Total
-----------------------------------------------------------------------
pxc-1.b------------------------------------59.70
|--------vport-1------------------------------------33.58
| |--------ipoe-1-------------------------------------33.58
|--------vport-2-------------------------------------0.00
|--------vport-3------------------------------------11.20
| |--------ipoe-4--------------------------------------5.60
| |--------ipoe-5--------------------------------------5.60
|--------vport-4-------------------------------------0.00
|--------vport-5-------------------------------------0.00
|--------vport-6-------------------------------------3.73
| |--------ipoe-10-------------------------------------1.87
| |--------ipoe-9--------------------------------------1.87
|--------vport-7-------------------------------------3.73
| |--------ipoe-12-------------------------------------1.87
| |--------ipoe-11-------------------------------------1.87
|--------vport-8-------------------------------------0.00
|--------vport-9-------------------------------------5.23
| |--------ipoe-15-------------------------------------5.23
|--------vport-10------------------------------------0.00
|--------vport-11------------------------------------0.74
| |--------ipoe-19-------------------------------------0.37
| |--------ipoe-18-------------------------------------0.37
|--------vport-12------------------------------------0.00
|--------vport-13------------------------------------0.74
| |--------ipoe-23-------------------------------------0.37
| |--------ipoe-22-------------------------------------0.37
|--------vport-14------------------------------------0.00
|--------vport-15------------------------------------0.74
| |--------ipoe-27-------------------------------------0.37
| |--------ipoe-26-------------------------------------0.37
|--------vport-16------------------------------------0.00
pxc-2.b------------------------------------40.30
|--------vport-1-------------------------------------0.00
|--------vport-2------------------------------------11.19
| |--------ipoe-3--------------------------------------5.60
| |--------ipoe-2--------------------------------------5.59
|--------vport-3-------------------------------------0.00
|--------vport-4------------------------------------11.20
| |--------ipoe-7--------------------------------------5.60
| |--------ipoe-6--------------------------------------5.60
|--------vport-5------------------------------------11.20
| |--------ipoe-8-------------------------------------11.20
|--------vport-6-------------------------------------0.00
|--------vport-7-------------------------------------0.00
|--------vport-8-------------------------------------3.73
| |--------ipoe-14-------------------------------------1.86
| |--------ipoe-13-------------------------------------1.87
|--------vport-9-------------------------------------0.00
|--------vport-10------------------------------------0.74
| |--------ipoe-16-------------------------------------0.37
| |--------ipoe-17-------------------------------------0.37
|--------vport-11------------------------------------0.00
|--------vport-12------------------------------------0.74
| |--------ipoe-20-------------------------------------0.37
| |--------ipoe-21-------------------------------------0.37
|--------vport-13------------------------------------0.00
|--------vport-14------------------------------------0.74
| |--------ipoe-25-------------------------------------0.37
| |--------ipoe-24-------------------------------------0.37
|--------vport-15------------------------------------0.00
|--------vport-16------------------------------------0.75
|--------ipoe-32-------------------------------------0.15
|--------ipoe-30-------------------------------------0.15
|--------ipoe-31-------------------------------------0.15
|--------ipoe-29-------------------------------------0.15
|--------ipoe-28-------------------------------------0.15
-----------------------------------------------------------------------
Run 3
-----------------------------------------------------------------------
Object | BW % of Total
-----------------------------------------------------------------------
pxc-1.b------------------------------------59.70
|--------vport-1------------------------------------33.58
| |--------ipoe-1-------------------------------------33.58
|--------vport-2-------------------------------------0.00
|--------vport-3------------------------------------11.19
| |--------ipoe-4--------------------------------------5.60
| |--------ipoe-5--------------------------------------5.59
|--------vport-4-------------------------------------0.00
|--------vport-5-------------------------------------0.00
|--------vport-6-------------------------------------3.73
| |--------ipoe-10-------------------------------------1.86
| |--------ipoe-9--------------------------------------1.86
|--------vport-7-------------------------------------3.73
| |--------ipoe-12-------------------------------------1.87
| |--------ipoe-11-------------------------------------1.86
|--------vport-8-------------------------------------0.00
|--------vport-9-------------------------------------5.22
| |--------ipoe-15-------------------------------------5.22
|--------vport-10------------------------------------0.00
|--------vport-11------------------------------------0.75
| |--------ipoe-19-------------------------------------0.38
| |--------ipoe-18-------------------------------------0.38
|--------vport-12------------------------------------0.00
|--------vport-13------------------------------------0.75
| |--------ipoe-23-------------------------------------0.38
| |--------ipoe-22-------------------------------------0.38
|--------vport-14------------------------------------0.00
|--------vport-15------------------------------------0.75
| |--------ipoe-27-------------------------------------0.38
| |--------ipoe-26-------------------------------------0.38
|--------vport-16------------------------------------0.00
pxc-2.b------------------------------------40.30
|--------vport-1-------------------------------------0.00
|--------vport-2------------------------------------11.19
| |--------ipoe-3--------------------------------------5.59
| |--------ipoe-2--------------------------------------5.59
|--------vport-3-------------------------------------0.00
|--------vport-4------------------------------------11.20
| |--------ipoe-7--------------------------------------5.60
| |--------ipoe-6--------------------------------------5.60
|--------vport-5------------------------------------11.20
| |--------ipoe-8-------------------------------------11.20
|--------vport-6-------------------------------------0.00
|--------vport-7-------------------------------------0.00
|--------vport-8-------------------------------------3.73
| |--------ipoe-14-------------------------------------1.87
| |--------ipoe-13-------------------------------------1.87
|--------vport-9-------------------------------------0.00
|--------vport-10------------------------------------0.74
| |--------ipoe-16-------------------------------------0.37
| |--------ipoe-17-------------------------------------0.37
|--------vport-11------------------------------------0.00
|--------vport-12------------------------------------0.75
| |--------ipoe-20-------------------------------------0.38
| |--------ipoe-21-------------------------------------0.38
|--------vport-13------------------------------------0.00
|--------vport-14------------------------------------0.75
| |--------ipoe-25-------------------------------------0.38
| |--------ipoe-24-------------------------------------0.38
|--------vport-15------------------------------------0.00
|--------vport-16------------------------------------0.74
|--------ipoe-32-------------------------------------0.15
|--------ipoe-30-------------------------------------0.15
|--------ipoe-31-------------------------------------0.15
|--------ipoe-29-------------------------------------0.15
|--------ipoe-28-------------------------------------0.15
Per-subscriber load balancing
In this example, Vports are not considered in hashing, due to the removal of the subscriber-hash-mode vport command:
per-link-hash {
weighted {
delete subscriber-hash-mode
The configuration file for BNG-1 can be found here .
The "lag-3" configuration is now as follows:
lag "lag-3" {
admin-state enable
description "fpe pw-port pxc lag - termination side"
mode hybrid
max-ports 64
access {
per-fp-ing-queuing false
per-fp-egr-queuing false
per-fp-sap-instance false
adapt-qos {
mode port-fair
}
}
port pxc-1.b {
}
port pxc-2.b {
}
}
Because Vports are not used, a simpler monitor command is used to collect the data that is going through the port scheduler without visibility into subscribers. The output of this command is:
A:admin@bng-1# monitor-scheduler -l lag-3 -i 10 -r 3
-----------------------------------------------------------------------
Run 1 (interval = 10 seconds)
-----------------------------------------------------------------------
Lag Member Port | BW % of Total
-----------------------------------------------------------------------
pxc-1.b | 36.27
pxc-2.b | 63.73
-----------------------------------------------------------------------
Run 2 (interval = 10 seconds)
-----------------------------------------------------------------------
Lag Member Port | BW % of Total
-----------------------------------------------------------------------
pxc-1.b | 36.27
pxc-2.b | 63.73
-----------------------------------------------------------------------
Run 3 (interval = 10 seconds)
-----------------------------------------------------------------------
Lag Member Port | BW % of Total
-----------------------------------------------------------------------
pxc-1.b | 36.27
pxc-2.b | 63.73
The output shows that the per-subscriber hashing also produces unequal load balancing due to the significant difference in rates between the subscribers.
This unequal load balancing can be corrected by configuring classes and weights per subscriber (defined in the sub-profile), but this configuration is beyond the scope of this chapter.
LAG "lag-1"
This section describes the load balancing on the LAG that connects the BNG to the aggregation node "AGG-1". LAG "lag-1" contains regular ports (no PXCs) and is referenced under the L3 interfaces that connect the BNG to the aggregation node.
[pr:/configure router "Base"]
A:admin@bng-1#
autonomous-system 64500
router-id 192.0.2.20
interface "int-1-bng-1-p-1" {
port lag-1:1
ipv6 {
address 2001:db8::501 {
prefix-length 120
}
}
}
Although not explicitly referenced in the Epipe (EVPN VPWS), this LAG is used to reach the network destination in EVPN, which is the BGP neighbor in the following configuration:
[pr:/configure service epipe "access"]
A:admin@bng-1#
admin-state enable
service-id 1
customer "1"
bgp 1 {
}
bgp-evpn {
evi 10
local-attachment-circuit "bng" {
eth-tag 2
}
remote-attachment-circuit "access" {
eth-tag 1
}
mpls 1 {
admin-state enable
send-tunnel-encap {
mpls-over-udp false
}
auto-bind-tunnel {
resolution any
}
route-next-hop {
system-ipv6
}
}
}
[pr:/configure router "Base" bgp]
A:admin@bng-1#
admin-state enable
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
rapid-update {
evpn true
}
group "evpn" {
peer-as 64500
local-address 2001:db8::14
family {
evpn true
}
}
neighbor "2001:db8::5" {
group "evpn"
}
LAG "lag-1" is configured as follows with the member ports shown in the figure Example topology:
[pr:/configure lag "lag-1"]
A:admin@bng-1#
admin-state enable
description "lag to p-1"
mode hybrid
max-ports 64
port 1/1/c1/1 {
}
port 1/1/c1/2 {
}
This section describes two ways of traffic load balancing toward the network destination (in this case, the system IPv6 address, 2001:db8::5, on the aggregation node) in the Epipe (EVPN VPWS).
By default, traffic is load-balanced per source and destination IP address pair. Traffic is sent from the same source IP address on the traffic generator toward each individual subscriber, each subscriber with its own IP address. Because every <source, destination> IP address pair in the setup represents a unique subscriber, one could conclude that this would be equivalent to per-subscriber load balancing. However, the hashing keys here are the source and destination IP address, while in per-subscriber hashing, the hashing key is the subscriber ID (and internal SR number). As a result, the outcome of traffic distribution is different in "lag-1" and "lag-3" in the preceding monitor-scheduler command with load-balancing per subscriber.
Optimally, L4 ports can be added to the source and destination IP address as the hashing input.
This is a very different distribution from the output in the preceding monitor-scheduler command with load balancing per subscriber where the traffic distribution over the two ports was approximately 36/63.
A:admin@bng-1# monitor lag 1 interval 3 repeat 999 rate packets
=============================================================================
Monitor statistics for LAG ID 1
=============================================================================
Port-id Input packets Output packets
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
At time t = 3 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 0 811
1/1/c1/2 0 1864
-----------------------------------------------------------------------------
Totals 0 2675
-----------------------------------------------------------------------------
At time t = 6 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 0 812
1/1/c1/2 0 1868
-----------------------------------------------------------------------------
Totals 0 2680
-----------------------------------------------------------------------------
At time t = 9 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 2 813
1/1/c1/2 1 1869
-----------------------------------------------------------------------------
Totals 3 2682
---snip---
A more even distribution is achieved by enabling additional fields for hashing purposes. In the following example, in addition to the source and destination IP addresses, the hashing algorithm also considers L4 ports, which in this example, are randomly changing in each packet.
[pr:/configure system load-balancing]
A:admin@bng-1#
l4-load-balancing true
The result of such hashing is more even load balancing (approximately 50/50), as follows:
A:admin@bng-1# monitor lag 1 interval 3 repeat 5 rate packets
=============================================================================
Monitor statistics for LAG ID 1
=============================================================================
Port-id Input packets Output packets
-----------------------------------------------------------------------------
At time t = 3 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 1 1354
1/1/c1/2 0 1319
-----------------------------------------------------------------------------
Totals 1 2673
-----------------------------------------------------------------------------
At time t = 6 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 1 1341
1/1/c1/2 1 1339
-----------------------------------------------------------------------------
Totals 2 2680
-----------------------------------------------------------------------------
At time t = 9 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 0 1326
1/1/c1/2 0 1354
-----------------------------------------------------------------------------
Totals 0 2680
-----------------------------------------------------------------------------
At time t = 12 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 0 1326
1/1/c1/2 0 1353
-----------------------------------------------------------------------------
Totals 0 2679
-----------------------------------------------------------------------------
At time t = 15 sec (Mode: Rate)
-----------------------------------------------------------------------------
1/1/c1/1 2 1360
1/1/c1/2 1 1321
-----------------------------------------------------------------------------
Totals 3 2681
Conclusion
This chapter provides fundamentals for understanding and configuring the class- and weight-based Vport traffic load balancing on a LAG in ESM. This is used in deployments where not only the bandwidth service offerings for subscribers vary greatly, but also where there is a significant variation in configured bandwidth between the Vports.
The configuration setup relies on FPE-based PW-ports, with two stage LAGs. The first stage LAG contains PXC based member ports as the subscriber termination points. This stage is the focus of this chapter. The second stage LAG is applied in the EVPN VPWS service context with the regular faceplate ports connecting the BNG to the access nodes. The distinct role of each LAG is explained, and the outcome of the load balancing techniques is each stage is explored.
Traffic utilization on the LAG member links is analyzed and compared between different hashing algorithms (per Vport with classes and weights, per Vport without classes and weighs, and per subscriber).