EVPN unequal ECMP for RT5 IFL and IFF routes
This chapter provides information about EVPN unequal ECMP for IFL and IFF IP prefix routes (RT5) with MPLS, VXLAN, or SRv6 transport.
Topics in this chapter include:
Applicability
The information and configuration in this chapter are based on SR OS Release 24.10.R1.
EVPN unequal ECMP for IFL IP prefix routes with MPLS transport is supported on FP-based platforms in SR OS Release 22.7.R2 and later. EVPN unequal ECMP for IFL IP prefix routes with SRv6 transport is supported on FP4-based platforms in SR OS Release 23.7.R1 and later. EVPN unequal ECMP for IFF IP prefix routes with VXLAN transport is supported on FP-based platforms in SR OS Release 22.7.R2 and later.
Overview
SR OS Release 22.7.R2 and later supports unequal ECMP for EVPN IPv4 and IPv6 prefix routes in the EVPN interface-less (EVPN IFL) and EVPN interface-ful (EVPN IFF) models.
Based on draft-ietf-bess-evpn-unequal-lb, it introduces a new EVPN link bandwidth extended community in the IP prefix routes to indicate a weight that the receiver PE must consider when load balancing traffic to multiple next hops (EVPN or CE).
Several multi-rack Container Network Functions (CNFs) are connected to a few Top of Rack (TOR) switches in an EVPN network. Each CNF advertises the anycast IP prefix in a single EBGP PE-CE session to the connected TOR. IPv4 prefix 198.51.100.1/32 is used as an example. An IPv6 prefix is also possible.
Each TOR re-advertises the IP prefix in an EVPN IP prefix route.
Without unequal ECMP, the Border Leaf (BL) creates an ECMP set for the IP prefix and distributes the traffic flows to it equally across the TORs (when the number of TORs is not larger than ecmp).
With unequal ECMP, each TOR adds a weight encoded in the EVPN IP prefix route. By default, the (dynamic) weight matches the number of CNFs that are attached to that TOR. The BL creates an ECMP set with the weights taken into account for the IP prefix and distributes the traffic flows to it according to the weights across the CNFs (when the number of TORs is not larger than ecmp).
- advertising procedures can be enabled with:
- for EVPN-IFL with MPLS transport: configure service vprn <vprn> bgp-evpn mpls evpn-link-bandwidth advertise
- for EVPN-IFL with SRv6 transport: configure service vprn <vprn> bgp-evpn segment-routing-v6 bgp <bgp instance> evpn-link-bandwidth advertise
- for EVPN-IFF: configure service vpls <vpls> bgp-evpn ip-route-link-bandwidth advertise
- receiving procedures can be enabled with:
- for EVPN-IFL with MPLS transport: configure service vprn <vprn> bgp-evpn mpls evpn-link-bandwidth weighted-ecmp
- for EVPN-IFL with SRv6 transport: configure service vprn <vprn> bgp-evpn segment-routing-v6 bgp <bgp instance> evpn-link-bandwidth weighted-ecmp
- for EVPN-IFF: configure service vpls <vpls> bgp-evpn ip-route-link-bandwidth weighted-ecmp
- for EVPN-IFL and EVPN-IFF IP prefix routes with dynamic or configured weight. EVPN-IFL IP prefix routes (with a default preference of 170) and EBGP PE-CE routes (with a default preference of 170 too) can be combined in the same ECMP set.
- for EVPN-IFL IP prefix routes with dynamic or configured weight and bgp <EBGP group> neighbor <EBGP neighbor> configured with evpn-link-bandwidth add-to-received-bgp <number>. EVPN-IFF IP prefix routes (with a default preference of 169) are by default preferred over EBGP PE-CE routes (with a default preference of 170).
When a BGP route toward a specific destination IP prefix has N ECMP next hops H[i] with signaled weight S[i] each, the normalised weight W[i] for each next hop H[i] is computed using the following algorithm:- Compute the Greatest Common Divisor (GCD) of all S[i] and compute Stotal=(sum of S[i]), for i = 1 to N
- If (Stotal/GCD) is not larger than 64, the normalised weight W[i] for next hop H[i] is S[i]/GCD
- Else, the normalised weight W[i] for next hop H[i] is (1 + (the integer result of (S[i]/Stotal * (64 - N)))
For EVPN-IFL, for unequal ECMP across EVPN next hops and CE next hops:
- service vprn <vprn> bgp group <EBGP group> neighbor <EBGP neighbor> evpn-link-bandwidth add-to-received-bgp <number> and
- service vprn <vprn> bgp eibgp-loadbalance
For EVPN-IFF, for unequal ECMP across EVPN next hops:- service vprn <vprn> bgp group <EBGP group> neighbor <EBGP neighbor> evpn-link-bandwidth add-to-received-bgp <number>
Configuration
- cards, MDAs, ports
- router interfaces
- IBGP in the EVPN network for the EVPN address family
- IS-IS on the router interfaces in the EVPN network (OSPF or OSPF3 router interfaces are also possible)
- LDP in the EVPN network
- base router SRv6 segment routing (for the SRv6 transport example)
Router configuration
# On TOR3:
configure router "Base"
interface "int-TOR3-BL5"
address 192.168.35.1/30
port 1/1/c5/1:10
ipv6
address 2001:db8::168:35:1/126
exit
exit
interface "system"
address 192.0.2.3/32
ipv6
address 2001:db8::2:3/128
exit
exit
autonomous-system 64500
isis 0
level-capability level-2
area-id 49.0001
traffic-engineering
traffic-engineering-options
ipv6
application-link-attributes
exit
exit
advertise-router-capability as
ipv6-routing native
no ipv6-multicast-routing
level 2
wide-metrics-only
exit
interface "system"
no shutdown
exit
interface "int-TOR3-BL5"
interface-type point-to-point
no shutdown
exit
no shutdown
exit
ldp
import-pmsi-routes
exit
tcp-session-parameters
exit
interface-parameters
interface "int-TOR3-BL5" dual-stack
ipv4
no shutdown
exit
ipv6
no shutdown
exit
no shutdown
exit
exit
targeted-session
exit
no shutdown
exit
bgp
enable-peer-tracking
rapid-withdrawal
split-horizon
rapid-update evpn
group "BL"
family evpn
type internal
neighbor 192.0.2.5
exit
exit
no shutdown
exitThe router configuration on TOR4 and TOR1 is similar.
# On BL5:
configure router "Base"
interface "int-BL5-TOR1"
address 192.168.15.2/30
port 1/1/c1/1:10
ipv6
address 2001:db8::168:15:2/126
exit
exit
interface "int-BL5-TOR3"
address 192.168.35.2/30
port 1/1/c3/1:10
ipv6
address 2001:db8::168:35:2/126
exit
exit
interface "int-BL5-TOR4"
address 192.168.45.2/30
port 1/1/c4/1:10
ipv6
address 2001:db8::168:45:2/126
exit
exit
interface "system"
address 192.0.2.5/32
ipv6
address 2001:db8::2:5/128
exit
exit
autonomous-system 64500
isis 0
level-capability level-2
area-id 49.0001
traffic-engineering
traffic-engineering-options
ipv6
application-link-attributes
exit
exit
advertise-router-capability as
ipv6-routing native
no ipv6-multicast-routing
level 2
wide-metrics-only
exit
interface "system"
no shutdown
exit
interface "int-BL5-TOR1"
interface-type point-to-point
no shutdown
exit
interface "int-BL5-TOR3"
interface-type point-to-point
no shutdown
exit
interface "int-BL5-TOR4"
interface-type point-to-point
no shutdown
exit
no shutdown
exit
ldp
import-pmsi-routes
exit
tcp-session-parameters
exit
interface-parameters
interface "int-BL5-TOR1" dual-stack
ipv4
no shutdown
exit
ipv6
no shutdown
exit
no shutdown
exit
interface "int-BL5-TOR3" dual-stack
ipv4
no shutdown
exit
ipv6
no shutdown
exit
no shutdown
exit
interface "int-BL5-TOR4" dual-stack
ipv4
no shutdown
exit
ipv6
no shutdown
exit
no shutdown
exit
exit
targeted-session
exit
no shutdown
exit
bgp
enable-peer-tracking
rapid-withdrawal
split-horizon
rapid-update evpn
group "TOR"
family evpn
type internal
neighbor 192.0.2.1
exit
neighbor 192.0.2.3
exit
neighbor 192.0.2.4
exit
exit
no shutdown
exit# On BL5:
configure card 1 mda 1
xconnect
mac 1 create
loopback 1 create
exit
loopback 2 create
exit
exit
exit
no shutdown
configure port-xc
pxc 1 create
port 1/1/m1/1
no shutdown
exit
pxc 2 create
port 1/1/m1/2
no shutdown
exit
configure
port 1/1/m1/1
no shutdown
exit
port 1/1/m1/2
no shutdown
exit
port pxc-1.a
no shutdown
exit
port pxc-1.b
no shutdown
exit
port pxc-2.a
no shutdown
exit
port pxc-2.b
no shutdown
exit
configure fwd-path-ext
fpe 1 create
path pxc 1
srv6 origination
interface-a
exit
interface-b
exit
exit
exit
fpe 2 create
path pxc 2
srv6 termination
interface-a
exit
interface-b
exit
exit
exit
configure router "Base" mpls-labels
sr-labels start 20001 end 20199
reserved-label-block "SRv6"
start-label 19001 end-label 19999
exit
configure router "Base" segment-routing segment-routing-v6
origination-fpe 1
locator "BL5-loc"
block-length 48
termination-fpe 2
prefix
ip-prefix 2001:db8:aaaa:105::/64
exit
static-function
label-block "SRv6"
exit
no shutdown
exit
base-routing-instance
locator "BL5-loc"
function
end-x-auto-allocate srh-mode usp protection protected
end 1
srh-mode usp
exit
exit
exit
configure router "Base" isis 0 segment-routing-v6
locator "BL5-loc"
level-capability level-2
level 1
exit
level 2
exit
exit
no shutdownThe base SRv6 segment routing configuration for TOR1, TOR3, and TOR4 is similar, with different locators and locator prefixes.
Use cases
EVPN-IFL model
To advertise the 198.51.100.1/32 prefix, export policy "vrf-1-export-1" for prefix-list "prefix-1" with prefix 198.51.100.1/32 is configured on the CNFs:
# On CNF2:
configure router "Base" policy-options
begin
prefix-list "prefix-1"
prefix 198.51.100.1/32 exact
exit
policy-statement "vrf-1-export-1"
entry 10
from
prefix-list "prefix-1"
exit
to
protocol bgp
exit
action accept
exit
exit
exit
commitThe export policy configuration on CNF3, CNF4, CNF5, and CNF1 is identical.
To illustrate SRv6 transport, a similar export policy "vrf-6-export-1" for the same prefix-list is configured on the CNFs.
# On BL5:
configure router "Base" policy-options
begin
community "comm-vrf-1"
members "target:64500:1"
exit
policy-statement "vrf-1-export-1"
entry 10
action accept
as-path-prepend 64503 1
community add "comm-vrf-1"
exit
exit
exit
commitThe as-path-prepend command ensures that the prefixes are advertised as if they originate in an external autonomous system.
To illustrate SRv6 transport, a similar export policy "vrf-6-export-1" is configured on BL5, but then for community "comm-vrf-6" with members "target:64500:6".
IPv4 prefix advertisement from CNFs - MPLS transport
CNF only advertisement illustrates how EVPN-IFL unequal ECMP works when only the CNFs advertise the IPv4 prefix. There are four CNFs (CNF2, CNF3, CNF4, CNF5), two TORs (TOR3, TOR4) and one BL (BL5). TOR3 and TOR4 are connected to BL5. CNF2, CNF3, and CNF4 are connected to TOR3. CNF5 is connected to TOR4. VPRN 1 and VPRN 6 are configured on the CNFs, the TORs, and the BL.
# On CNF2:
configure service vprn 1 name "VPRN 1" customer 1 create
autonomous-system 64501
interface "to-TOR3" create
address 10.13.12.2/30
sap 1/1/c3/1:10 create
exit
exit
interface "subs-1" create
address 198.51.100.1/32
loopback
exit
bgp
rapid-withdrawal
group "PE-CE"
type external
peer-as 64500
neighbor 10.13.12.1
export "vrf-1-export-1"
exit
exit
no shutdown
exit
no shutdownThe configuration on CNF3, CNF4, and CNF5 is similar, with CNF5 associated with TOR4.
- Dynamic PE-CE weights (5 cases)
- Configured maximum weight=73 on TOR3
- Configured weight=23 on TOR3
- Reduced ECMP on TOR3 (2 cases)
- TOR3 weights summary - configuration: summarises, for each case, the evpn-link-bandwidth add-to-received-bgp values that are configured for each CNF connected to TOR3, and the value that TOR3 advertises for each case (obtained from EVPN-IP-PREFIX BGP update messages sent from TOR3)
- TOR3 weights summary - route table and FIB: summarises, for each case, the normalised weights in the route table of TOR3 (obtained from: show router 1 route-table 198.51.100.1/32 extensive) and the normalised weights in the FIB of TOR3 (obtained from: show router 1 fib 1 198.51.100.1/32 extensive)
- BL5 weights summary: summarises, for each case, the weights that BL5 receives from TOR3 and TOR4 (obtained from EVPN-IP-PREFIX BGP update messages received on BL5), the normalised weights in the route table of BL5 (obtained from: show router 1 route-table 198.51.100.1/32 extensive), and the normalised weights in the FIB of BL5 (obtained from: show router 1 fib 1 198.51.100.1/32 extensive)
The detailed configuration, BGP update, and show command output is explicitly included in the further description only for the first two cases.
| No configured PE-CE weights | Equal configured PE-CE weights (GCD>1) | Different configured PE-CE weights (GCD>1) | Different configured PE-CE weights (GCD=1 and Stotal/GCD≤64) | Different configured PE-CE weights (GCD=1 and Stotal/GCD>64) | Configured maximum weight=73 on TOR3 | Configured weight=23 on TOR3 | ecmp=2 on TOR3 | ecmp=1 on TOR3 | ||
|---|---|---|---|---|---|---|---|---|---|---|
| for | Configured PE-CE weights on TOR3 | |||||||||
| CNF2 | S[1] | - (1) | 5 | 4 | 5 | 5 | 5 | 5 | 5 | 109 |
| CNF3 | S[2] | - (1) | 5 | 8 | 17 | 17 | 17 | 17 | 113 | [113] |
| CNF4 | S[3] | - (1) | 5 | 12 | 31 | 59 | 59 | 59 | [23] | [23] |
| Stotal | - (3) | 15 | 24 | 53 | 81 | 81 | 81 | 118 | 109 | |
| from | Advertised weight from TOR3 (evpn-bandwidth in EVPN-IP-PREFIX BGP update message from TOR3) | |||||||||
| TOR3 | 3 | 15 | 24 | 53 | 81 | 73 | 23 | 118 | 109 | |
The values in brackets are used instead of the non-configured values. The values in square brackets are not used.
| No configured PE-CE weights | Equal configured PE-CE weights (GCD>1) | Different configured PE-CE weights (GCD>1) | Different configured PE-CE weights (GCD=1 and Stotal/GCD≤64) | Different configured PE-CE weights (GCD=1 and Stotal/GCD>64) | Configured maximum weight=73 on TOR3 | Configured weight=23 on TOR3 | ecmp=2 on TOR3 | ecmp=1 on TOR3 | ||
|---|---|---|---|---|---|---|---|---|---|---|
| for | Normalised weights on TOR3 (route table) | |||||||||
| CNF2 | W[1] | NA | 1 | 1 | 5 | 4 | 4 | 4 | 3 | 1 |
| CNF3 | W[2] | NA | 1 | 2 | 17 | 13 | 13 | 13 | 60 | - |
| CNF4 | W[3] | NA | 1 | 3 | 31 | 45 | 45 | 45 | - | - |
| sum | NA | 3 | 6 | 53 | 62 | 62 | 62 | 63 | 1 | |
| for | Normalised weights on TOR3 (FIB) | |||||||||
| CNF2 | W[1] | 1 | 1 | 1 | 5 | 4 | 4 | 4 | 3 | 1 |
| CNF3 | W[2] | 1 | 1 | 2 | 17 | 13 | 13 | 13 | 60 | - |
| CNF4 | W[3] | 1 | 1 | 3 | 31 | 45 | 45 | 45 | - | - |
| sum | 3 | 3 | 6 | 53 | 62 | 62 | 62 | 63 | 1 | |
| No configured PE-CE weights | Equal configured PE-CE weights (GCD>1) | Different configured PE-CE weights (GCD>1) | Different configured PE-CE weights (GCD=1 and Stotal/GCD≤64) | Different configured PE-CE weights (GCD=1 and Stotal/GCD>64) | Configured maximum weight=73 on TOR3 | Configured weight=23 on TOR3 | ecmp=2 on TOR3 | ecmp=1 on TOR3 | ||
|---|---|---|---|---|---|---|---|---|---|---|
| from | Received weights on BL5 (evpn-bandwidth in EVPN-IP-PREFIX BGP update message from TOR3 and TOR4) | |||||||||
| TOR3 | S[1] | 3 | 15 | 24 | 53 | 81 | 73 | 23 | 118 | 109 |
| TOR4 | S[2] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Stotal | 4 | 16 | 25 | 54 | 82 | 74 | 24 | 119 | 110 | |
| for | Normalised weights on BL5 (route table) | |||||||||
| TOR3 | W[1] | 3 | 15 | 24 | 53 | 62 | 62 | 23 | 62 | 62 |
| TOR4 | W[2] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| sum | 4 | 16 | 25 | 54 | 63 | 63 | 24 | 63 | 63 | |
| for | Normalised weights on BL5 (FIB) | |||||||||
| TOR3 | W[1] | 3 | 15 | 24 | 53 | 62 | 62 | 23 | 62 | 62 |
| TOR4 | W[2] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| sum | 4 | 16 | 25 | 54 | 63 | 63 | 24 | 63 | 63 | |
Dynamic PE-CE weights
There is one EBGP PE-CE session for each CNF. Each CNF advertises the anycast 198.51.100.1/32 prefix in its specific EBGP PE-CE session to its corresponding TOR. To enable unequal ECMP processing for EVPN-IFL IPv4 prefix routes, the advertising and receiving procedures are configured in the service vprn <vprn> bgp-evpn mpls context on the TORs and on the BL. To indicate with what relative weight the load balancing across the EVPN-IFL IPv4 prefix routes must be executed, the add-to-received-bgp <number> is configured for each EBGP PE-CE session in the service vprn <vprn> bgp group <EBGP group> neighbor <EBGP neighbor> context on the TORs.
No configured PE-CE weights
Initially, add-to-received-bgp is not configured. This is the same as if add-to-received-bgp 1 is configured for each EBGP PE-CE session, so that each EBGP PE-CE session is taken into account equally. Unequal ECMP for EVPN-IFL IPv4 prefix routes then load balances traffic from the BL to the CNFs equally across all CNFs.
VPRN 1 is configured on the TORs as follows:
# On TOR3:
configure service vprn 1 name "VPRN 1" customer 1 create
ecmp 3
autonomous-system 64500
interface "to-CE2" create
address 10.13.12.1/30
sap 1/1/c2/1:10 create
exit
exit
interface "to-CE3" create
address 10.13.13.1/30
sap 1/1/c3/1:10 create
exit
exit
interface "to-CE4" create
address 10.13.14.1/30
sap 1/1/c4/1:10 create
exit
exit
bgp-evpn
mpls
auto-bind-tunnel
resolution any
exit
evi 1
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 192.0.2.3:1
vrf-target target:64500:1
no shutdown
exit
exit
bgp
multi-path
ipv4 10
exit
eibgp-loadbalance
rapid-withdrawal
group "PE-CE"
type external
peer-as 64501
neighbor 10.13.12.2
exit
neighbor 10.13.13.2
exit
neighbor 10.13.14.2
exit
exit
no shutdown
exit
no shutdownThe configuration on TOR4 is similar, with only one CNF.
VPRN 1 is configured on the BL as follows:
# On BL5:
configure service vprn 1 name "VPRN 1" customer 1 create
ecmp 3
autonomous-system 64500
bgp-evpn
mpls
auto-bind-tunnel
resolution any
exit
evi 1
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 192.0.2.5:1
vrf-target target:64500:1
no shutdown
exit
exit
bgp
no shutdown
exit
no shutdownOn TOR3, each next hop along which prefix 198.51.100.1/32 can be reached is assigned S[i]=1. TOR3 has three such next hops, so GCD=1, Stotal=3, S[i]/GCD=1, Stotal/GCD=3 (not larger than 64), W[i]=1.
# On TOR3:
1 2024/12/09 21:29:09.296 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 96
Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 198.51.100.1/32 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 6 AS Path:
Type: 2 Len: 1 < 64501 >
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
evpn-bandwidth:1:3
bgp-tunnel-encap:MPLS
"
2 2024/12/09 21:29:09.296 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 162
Flag: 0x90 Type: 14 Len: 117 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 10.13.12.0/30 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 10.13.13.0/30 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 10.13.14.0/30 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
evpn-bandwidth:1:1
bgp-tunnel-encap:MPLS
"TOR4 has only one such next hop, so GCD=1, Stotal=1, S[1]/GCD=1, Stotal/GCD=1 (not larger than 64), W[1]=1. TOR4 advertises Stotal=1 for prefix 198.51.100.1/32 and 1 for the next hop prefix.
*A:TOR3# show router 1 route-table 198.51.100.1/32 extensive
===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : BGP
Age : 00h06m28s
Preference : 170
Indirect Next-Hop : 10.13.12.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : N/A
Resolving Next-Hop : 10.13.12.2
Interface : to-CE2
Metric : 0
ECMP-Weight : N/A
Indirect Next-Hop : 10.13.13.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : N/A
Resolving Next-Hop : 10.13.13.2
Interface : to-CE3
Metric : 0
ECMP-Weight : N/A
Indirect Next-Hop : 10.13.14.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : N/A
Resolving Next-Hop : 10.13.14.2
Interface : to-CE4
Metric : 0
ECMP-Weight : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:TOR3# show router 1 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : BGP
Installed : Y
Indirect Next-Hop : 10.13.12.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.12.2
Interface : to-CE2 (VPRN 1)
ECMP-Weight : 1
Indirect Next-Hop : 10.13.13.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.13.2
Interface : to-CE3 (VPRN 1)
ECMP-Weight : 1
Indirect Next-Hop : 10.13.14.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.14.2
Interface : to-CE4 (VPRN 1)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================Similarly, TOR4 installs in its VPRN route table and VPRN FIB: the EBGP PE-CE route for prefix 198.51.100.1/32 that TOR4 receives via EBGP from CNF5, with ECMP weight N/A in the VPRN route table and normalised ECMP weight W[1]=1 in the VPRN FIB.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=3 and one from TOR4 with S[2]=1. So GCD=1, Stotal=4, S[1]/GCD=3, S[2]/GCD=1, Stotal/GCD=4 (not larger than 64), W[1]=3, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=3 and W[2]=1:
*A:BL5# show router 1 route-table 198.51.100.1/32 extensive
===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFL
Age : 00h01m20s
Preference : 170
Indirect Next-Hop : 192.0.2.3
Label : 524277
VPN Next-Hop Index : 30
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 3
Resolving Next-Hop : 192.0.2.3 (LDP tunnel)
Metric : 10
ECMP-Weight : N/A
Indirect Next-Hop : 192.0.2.4
Label : 524277
VPN Next-Hop Index : 32
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 192.0.2.4 (LDP tunnel)
Metric : 10
ECMP-Weight : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:BL5# show router 1 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFL
Installed : Y
Indirect Next-Hop : 192.0.2.3
Label : 524277
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 3
Resolving Next-Hop : 192.0.2.3 (LDP tunnel)
ECMP-Weight : 1
Indirect Next-Hop : 192.0.2.4
Label : 524277
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 192.0.2.4 (LDP tunnel)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================*A:BL5# show router bgp routes evpn ip-prefix prefix 198.51.100.1/32
===============================================================================
BGP Router ID:192.0.2.5 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Flag Route Dist. Prefix
Tag Gw Address
NextHop
Label
ESI
-------------------------------------------------------------------------------
u*>i 192.0.2.3:1 198.51.100.1/32
0 00:00:00:00:00:00
192.0.2.3
LABEL 524277
ESI-0
u*>i 192.0.2.4:1 198.51.100.1/32
0 00:00:00:00:00:00
192.0.2.4
LABEL 524277
ESI-0
-------------------------------------------------------------------------------
Routes : 2
===============================================================================The weight that TOR3 signals in the extended community for prefix 198.51.100.1/32 can be verified in the detail of the route entry for TOR3 (route distinguisher: 192.0.2.3:1), as follows:
*A:BL5# show router bgp routes evpn ip-prefix prefix 198.51.100.1/32
rd 192.0.2.3:1 detail
===============================================================================
BGP Router ID:192.0.2.5 AS:64500 Local AS:64500
===============================================================================
---snip---
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Original Attributes
Network : n/a
Nexthop : 192.0.2.3
Path Id : None
From : 192.0.2.3
Res. Nexthop : 192.168.35.1
Local Pref. : 100 Interface Name : int-BL5-TOR3
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
AIGP Metric : None IGP Cost : 10
Connector : None
Community : target:64500:1 evpn-bandwidth:1:3 bgp-tunnel-encap:MPLS
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 192.0.2.3
Origin : IGP
Flags : Used Valid Best
Route Source : Internal
AS-Path : 64501
EVPN type : IP-PREFIX
ESI : ESI-0
Tag : 0
Gateway Address: 00:00:00:00:00:00
Prefix : 198.51.100.1/32
Route Dist. : 192.0.2.3:1
MPLS Label : LABEL 524277
Route Tag : 0
Neighbor-AS : 64501
DB Orig Val : N/A Final Orig Val : N/A
Source Class : 0 Dest Class : 0
Add Paths Send : Default
Last Modified : 00h01m39s
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================Similar for TOR4, with Community : target:64500:1 evpn-bandwidth:1:1 bgp-tunnel-encap:MPLS
- units is a decimal value [0-255] representing the units. For practical purposes only values 0 and 1 are specified. Only value 1 is supported at the moment.
- weight is a decimal value [0-1099511627775] –five octets– encoding either the bandwidth in Mbps (if units=0x00) or the number of BGP paths (if units=0x01). Because only units=0x01 is supported, the weight is always a "Generalised Weight", as described in draft-ietf-bess-evpn-unequal-lb.
# On TOR3:
*A:TOR3>configure>service>vprn>bgp-evpn>mpls>evpn-link-bw# info detail
----------------------------------------------
advertise weight dynamic max-dynamic-weight 128
weighted-ecmp
----------------------------------------------Verifying that the evpn-link-bandwidth is enabled for VPRN 1 can be done as follows:
*A:TOR3# show service id 1 bgp-evpn
===============================================================================
BGP EVPN MPLS Table
===============================================================================
Admin State : Up Oper State : Up
VRF Import : None
VRF Export : None
Route Dist. : 192.0.2.3:1
Oper Route Dist. : 192.0.2.3:1
Oper RD Type : configured
Route Target : target:64500:1
Route Target Import: None
Route Target Export: None
Default Route Tag : None
Domain-Id : None
Dyn Egr Lbl Limit : Disabled
EVI : 1
Advertise : Enabled
Weight : Dynamic
Max Dynamic Weight : 128
Weighted ECMP : Enabled
---snip---
===============================================================================Configured PE-CE weights
Equal configured PE-CE weights (GCD>1)
VPRN 1 is reconfigured on TOR3 as follows:
# On TOR3:
configure service vprn 1
ecmp 3
---snip---
bgp-evpn
mpls
---snip---
evpn-link-bandwidth
advertise
weighted-ecmp
exit
no shutdown
exit
exit
bgp
---snip---
eibgp-loadbalance
rapid-withdrawal
group "PE-CE"
---snip---
neighbor 10.13.12.2
evpn-link-bandwidth
add-to-received-bgp 5
exit
exit
neighbor 10.13.13.2
evpn-link-bandwidth
add-to-received-bgp 5
exit
exit
neighbor 10.13.14.2
evpn-link-bandwidth
add-to-received-bgp 5
exit
exit
exit
no shutdown
exit
no shutdownNothing changes on TOR4.
On TOR3, S[i]=5. So, GCD=5, Stotal=15, S[i]/GCD=1, Stotal/GCD=3 (not larger than 64), W[i]=1.
TOR3 advertises Stotal=15 for prefix 198.51.100.1/32, as follows:
# On TOR3:
1 2024/12/09 21:34:05.568 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 96
Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 198.51.100.1/32 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 6 AS Path:
Type: 2 Len: 1 < 64501 >
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
evpn-bandwidth:1:15
bgp-tunnel-encap:MPLS
"TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, CNF3, and CNF4, with normalised ECMP weights W[i]=1:
*A:TOR3# show router 1 route-table 198.51.100.1/32 extensive
===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : BGP
Age : 00h01m24s
Preference : 170
Indirect Next-Hop : 10.13.12.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.12.2
Interface : to-CE2
Metric : 0
ECMP-Weight : N/A
Indirect Next-Hop : 10.13.13.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.13.2
Interface : to-CE3
Metric : 0
ECMP-Weight : N/A
Indirect Next-Hop : 10.13.14.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.14.2
Interface : to-CE4
Metric : 0
ECMP-Weight : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:TOR3# show router 1 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : BGP
Installed : Y
Indirect Next-Hop : 10.13.12.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.12.2
Interface : to-CE2 (VPRN 1)
ECMP-Weight : 1
Indirect Next-Hop : 10.13.13.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.13.2
Interface : to-CE3 (VPRN 1)
ECMP-Weight : 1
Indirect Next-Hop : 10.13.14.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.13.14.2
Interface : to-CE4 (VPRN 1)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=15 and one from TOR4 with S[2]=1. So GCD=1, Stotal=16, S[1]/GCD=15, S[2]/GCD=1, Stotal/GCD=16 (not larger than 64), W[1]=15, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=15 and W[2]=1:
*A:BL5# show router 1 route-table 198.51.100.1/32 extensive
===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFL
Age : 00h01m27s
Preference : 170
Indirect Next-Hop : 192.0.2.3
Label : 524277
VPN Next-Hop Index : 30
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 15
Resolving Next-Hop : 192.0.2.3 (LDP tunnel)
Metric : 10
ECMP-Weight : N/A
Indirect Next-Hop : 192.0.2.4
Label : 524277
VPN Next-Hop Index : 32
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 192.0.2.4 (LDP tunnel)
Metric : 10
ECMP-Weight : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:BL5# show router 1 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFL
Installed : Y
Indirect Next-Hop : 192.0.2.3
Label : 524277
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 15
Resolving Next-Hop : 192.0.2.3 (LDP tunnel)
ECMP-Weight : 1
Indirect Next-Hop : 192.0.2.4
Label : 524277
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 192.0.2.4 (LDP tunnel)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================Different configured PE-CE weights (GCD>1)
VPRN 1 is reconfigured on TOR3 with: S[1]=4, S[2]=8, S[3]=12. So, GCD=4, Stotal=24, S[1]/GCD=1, S[2]/GCD=2, S[3]/GCD=3, Stotal/GCD=6 (not larger than 64), W[1]=1, W[2]=2, W[3]=3.
TOR3 advertises Stotal=24 for prefix 198.51.100.1/32.
TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, CNF3, and CNF4, with normalised ECMP weights W[1]=1, W[2]=2, W[3]=3.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=24 and one from TOR4 with S[2]=1. So GCD=1, Stotal=25, S[1]/GCD=24, S[2]/GCD=1, Stotal/GCD=25 (not larger than 64), W[1]=24, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=24 and W[2]=1.
Different configured PE-CE weights (GCD=1 and Stotal/GCD≤64)
VPRN 1 is reconfigured on TOR3 with: S[1]=5, S[2]=17, S[3]=31. So, GCD=1, Stotal=53, S[1]/GCD=5, S[2]/GCD=17, S[3]/GCD=31, Stotal/GCD=53 (not larger than 64), W[1]=5, W[2]=17, W[3]=31.
TOR3 advertises Stotal=53 for prefix 198.51.100.1/32.
TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, CNF3, and CNF4, with normalised ECMP weights W[1]=5, W[2]=17, W[3]=31.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=53 and one from TOR4 with S[2]=1. So GCD=1, Stotal=54, S[1]/GCD=53, S[2]/GCD=1, Stotal/GCD=54 (not larger than 64), W[1]=53, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=53 and W[2]=1.
Different configured PE-CE weights (GCD=1 and Stotal/GCD>64)
- W[1]=1+integer(5/81*(64-3))=4
- W[2]=1+integer(17/81*(64-3))=13
- W[3]=1+integer(59/81*(64-3))=45
TOR3 advertises Stotal=81 for prefix 198.51.100.1/32.
TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, CNF3, and CNF4, with normalised ECMP weights W[1]=4, W[2]=13, W[3]=45.
-
W[1]=1+integer(81/82*(64-2))=62
-
W[2]=1+integer(1/82*(64-2))=1
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=62 and W[2]=1.
Configured maximum weight=73 on TOR3
A TOR can limit the maximum dynamic weight to a value below 128.
VPRN 1 is reconfigured on TOR3 as follows:
# On TOR3:
configure service vprn 1 bgp-evpn mpls evpn-link-bandwidth
advertise weight dynamic max-dynamic-weight 73On TOR3, S[1]=5, S[2]=17, S[3]=59. So, GCD=1, Stotal=81, S[1]/GCD=5, S[2]/GCD=17, S[3]/GCD=59, Stotal/GCD=81 (larger than 64). The normalised weights are computed, such that sum(W[i]*GCD) is not larger than 64. W[1]=4, W[2]=13, W[3]=45.
TOR3 advertises min(max-dynamic-weight,Stotal)=73 instead for prefix 198.51.100.1/32.
TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, CNF3, and CNF4, with normalised ECMP weights W[1]=4, W[2]=13, W[3]=45.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=73 and one from TOR4 with S[2]=1. So GCD=1, Stotal=74, S[1]/GCD=73, S[2]/GCD=1, Stotal/GCD=74 (larger than 64). The normalised weights are computed, such that sum(W[i]*GCD) is not larger than 64. W[1]=62, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=62 and W[2]=1.
Verifying that the max-dynamic-weight is reconfigured for VPRN 1 can be done as follows:
*A:TOR3# show service id 1 bgp-evpn
===============================================================================
BGP EVPN MPLS Table
===============================================================================
Admin State : Up Oper State : Up
VRF Import : None
VRF Export : None
Route Dist. : 192.0.2.3:1
Oper Route Dist. : 192.0.2.3:1
Oper RD Type : configured
Route Target : target:64500:1
Route Target Import: None
Route Target Export: None
Default Route Tag : None
Domain-Id : None
Dyn Egr Lbl Limit : Disabled
EVI : 1
Advertise : Enabled
Weight : Dynamic
Max Dynamic Weight : 73
Weighted ECMP : Enabled
---snip---
===============================================================================Configured weight=23 on TOR3
A TOR can overwrite the dynamic advertised weight with a configured weight <weight [1-128]>. Unequal ECMP for EVPN-IFL IPv4 prefix routes then load balances traffic from the BL to the TOR as if the TOR has <weight> CNFs (EBGP PE-CE sessions) connected to it.
# On TOR3:
configure service vprn 1 bgp-evpn mpls evpn-link-bandwidth
advertise weight dynamic max-dynamic-weight 128
advertise weight 23On TOR3, S[1]=5, S[2]=17, S[3]=59. So, GCD=1, Stotal=81, S[1]/GCD=5, S[2]/GCD=17, S[3]/GCD=59, Stotal/GCD=81 (larger than 64). The normalised weights are computed, such that sum(W[i]*GCD) is not larger than 64. W[1]=4, W[2]=13, W[3]=45.
TOR3 advertises weight=23 instead for prefix 198.51.100.1/32 and weight=23 for each next hop prefix, as follows:
# On TOR3:
1 2024/12/09 21:43:00.534 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 96
Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 198.51.100.1/32 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 6 AS Path:
Type: 2 Len: 1 < 64501 >
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
evpn-bandwidth:1:23
bgp-tunnel-encap:MPLS
"
2 2024/12/09 21:43:00.534 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 162
Flag: 0x90 Type: 14 Len: 117 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 10.13.12.0/30 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 10.13.13.0/30 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.3:1, ESI: ESI-0, tag: 0,
ip_prefix: 10.13.14.0/30 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
evpn-bandwidth:1:23
bgp-tunnel-encap:MPLS
"TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, CNF3, and CNF4, with normalised ECMP weights W[1]=4, W[2]=13, W[3]=45.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=23 and one from TOR4 with S[2]=1. So GCD=1, Stotal=24, S[1]/GCD=23, S[2]/GCD=1, Stotal/GCD=24 (not larger than 64), W[1]=23, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=23 and W[2]=1.
*A:TOR3# show service id 1 bgp-evpn
===============================================================================
BGP EVPN MPLS Table
===============================================================================
Admin State : Up Oper State : Up
VRF Import : None
VRF Export : None
Route Dist. : 192.0.2.3:1
Oper Route Dist. : 192.0.2.3:1
Oper RD Type : configured
Route Target : target:64500:1
Route Target Import: None
Route Target Export: None
Default Route Tag : None
Domain-Id : None
Dyn Egr Lbl Limit : Disabled
EVI : 1
Advertise : Enabled
Weight : 23
Max Dynamic Weight : 128
Weighted ECMP : Enabled
---snip---
===============================================================================Reduced ECMP on TOR3
In the previous scenarios, service vprn <vprn> ecmp is configured with a value that is not smaller than the number of CNFs that are connected to the TOR. When service vprn <vprn> ecmp is configured with a value <max-ecmp> that is smaller than the number of CNFs that are connected to the TOR, only <max-ecmp> CNFs are taken into account, with lowest IP first.
ecmp=2 on TOR3
VPRN 1 is reconfigured on TOR3 with ecmp 2 and S[1]=5, S[2]=113, S[3]=23.
- W[1]=1+integer(5/118*(64-2))=3
- W[2]=1+integer(113/118*(64-2))=60
TOR3 advertises Stotal=118 for prefix 198.51.100.1/32.
TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2 and CNF3, with normalised ECMP weights W[1]=3, W[2]=60.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=118 and one from TOR4 with S[2]=1. So GCD=1, Stotal=119, S[1]/GCD=118, S[2]/GCD=1, Stotal/GCD=119 (larger than 64). The normalised weights are computed, such that sum(W[i]*GCD) is not larger than 64. W[1]=62, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=62 and W[2]=1.
ecmp=1 on TOR3
VPRN 1 is reconfigured on TOR3 with ecmp 1 and S[1]=109, S[2]=113, S[3]=23.
S[2] and S[3] are not used in the computation. So, GCD=109, Stotal=109, S[1]/GCD=1, Stotal/GCD=S[1]/GCD=1 (not larger than 64). W[1]=1.
TOR3 advertises Stotal=109 for prefix 198.51.100.1/32.
TOR3 installs in its VPRN route table and VPRN FIB: the EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, with normalised ECMP weight W[1]=1.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=109 and one from TOR4 with S[2]=1. So GCD=1, Stotal=110, S[1]/GCD=109, S[2]/GCD=1, Stotal/GCD=110 (larger than 64). The normalised weights are computed, such that sum(W[i]*GCD) is not larger than 64. W[1]=62, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=62 and W[2]=1.
IPv4 prefix advertisement from CNFs - SRv6 transport
The same scenarios as for MPLS transport apply for SRv6 transport. Only one specific scenario is described here for the same CNF only advertisement: advertising with dynamic weights, using different EBGP PE-CE configured weights S[i] with GCD=1 and Stotal/GCD larger than 128.
There is one EBGP PE-CE session for each CNF. Each CNF advertises the anycast 198.51.100.1/32 prefix in its specific EBGP PE-CE session to its corresponding TOR. To enable unequal ECMP processing for EVPN-IFL IPv4 prefix routes, the advertising and receiving procedures are configured in the service vprn <vprn> bgp-evpn segment-routing-v6 bgp context on the TORs and on the BL. To indicate with what relative weight the load balancing across the EVPN-IFL IPv4 prefix routes must be executed, the add-to-received-bgp <number> is configured for each EBGP PE-CE session in the service vprn <vprn> bgp group <EBGP group> neighbor <EBGP neighbor> context on the TORs.
VPRN 6 is configured on the CNFs, the TORs, and the BL.
VPRN 6 is configured on the CNFs as follows:
# On CNF2:
configure service vprn 6 name "VPRN 6" customer 1 create
autonomous-system 64501
interface "to-TOR3" create
address 10.63.12.2/30
sap 1/1/c3/1:60 create
exit
exit
interface "subs-1" create
address 198.51.100.1/32
loopback
exit
bgp
rapid-withdrawal
group "PE-CE"
type external
multihop 10
local-as 64501
peer-as 64500
neighbor 10.63.12.1
export "vrf-6-export-1"
exit
exit
no shutdown
exit
no shutdownThe configuration on CNF3, CNF4, and CNF5 is similar, with CNF5 associated with TOR4.
VPRN 6 is configured on the TORs as follows:
# On TOR3:
configure service vprn 6 name "VPRN 6" customer 1 create
ecmp 3
autonomous-system 64500
interface "to-CE2" create
address 10.63.12.1/30
sap 1/1/c2/1:60 create
exit
exit
interface "to-CE3" create
address 10.63.13.1/30
sap 1/1/c3/1:60 create
exit
exit
interface "to-CE4" create
address 10.63.14.1/30
sap 1/1/c4/1:60 create
exit
exit
segment-routing-v6 1 create
locator "TOR3-loc"
function
end-dt4
end-dt6
end-dt46
exit
exit
exit
bgp-evpn
segment-routing-v6 bgp 1
evi 6
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 6:3
source-address 10:20:1::3
srv6-instance 1 default-locator "TOR3-loc"
vrf-target target:64500:6
no shutdown
exit
exit
bgp
multi-path
ipv4 10
ipv6 10
exit
eibgp-loadbalance
multihop 10
rapid-withdrawal
group "PE-CE"
type external
peer-as 64501
neighbor 10.63.12.2
evpn-link-bandwidth
add-to-received-bgp 2
exit
exit
neighbor 10.63.13.2
evpn-link-bandwidth
add-to-received-bgp 9
exit
exit
neighbor 10.63.14.2
evpn-link-bandwidth
add-to-received-bgp 120
exit
exit
exit
no shutdown
exit
no shutdownThe configuration on TOR4 is similar, with only one CNF with add-to-received-bgp 1.
VPRN 6 is configured on the BL as follows:
# On BL5:
configure service vprn 6 name "VPRN 6" customer 1 create
ecmp 3
autonomous-system 64500
segment-routing-v6 1 create
locator "BL5-loc"
function
end-dt4
end-dt6
end-dt46
exit
exit
exit
bgp-evpn
segment-routing-v6 bgp 1
evi 6
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 6:5
source-address 10:20:1::5
srv6-instance 1 default-locator "BL5-loc"
vrf-target target:64500:6
no shutdown
exit
exit
bgp
no shutdown
exit
no shutdown- W[1]=1+integer(2/131*(64-3))=1
- W[2]=1+integer(9/131*(64-3))=5
- W[3]=1+integer(120/131*(64-3))=56
# On TOR3:
17 2024/12/09 22:23:22.964 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 128
Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-IP-PREFIX Len: 34 RD: 6:3, ESI: ESI-0, tag: 0,
ip_prefix: 198.51.100.1/32 gw_ip 0.0.0.0 Label: 8084432 (Raw Label: 0x7b5bd0)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 6 AS Path:
Type: 2 Len: 1 < 64501 >
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 16 Extended Community:
target:64500:6
evpn-bandwidth:1:128
Flag: 0xc0 Type: 40 Len: 37 Prefix-SID-attr:
SRv6 Services TLV (37 bytes):-
Type: SRV6 L3 Service TLV (5)
Length: 34 bytes, Reserved: 0x0
SRv6 Service Information Sub-TLV (33 bytes)
Type: 1 Len: 30 Rsvd1: 0x0
SRv6 SID: 2001:db8:aaaa:103::
SID Flags: 0x0 Endpoint Behavior: 0x13 Rsvd2: 0x0
SRv6 SID Sub-Sub-TLV
Type: 1 Len: 6
BL:48 NL:16 FL:20 AL:0 TL:20 TO:64
"
14 2024/12/09 22:22:08.661 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 194
Flag: 0x90 Type: 14 Len: 117 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-IP-PREFIX Len: 34 RD: 6:3, ESI: ESI-0, tag: 0,
ip_prefix: 10.63.12.0/30 gw_ip 0.0.0.0 Label: 8084432 (Raw Label: 0x7b5bd0)
Type: EVPN-IP-PREFIX Len: 34 RD: 6:3, ESI: ESI-0, tag: 0,
ip_prefix: 10.63.13.0/30 gw_ip 0.0.0.0 Label: 8084432 (Raw Label: 0x7b5bd0)
Type: EVPN-IP-PREFIX Len: 34 RD: 6:3, ESI: ESI-0, tag: 0,
ip_prefix: 10.63.14.0/30 gw_ip 0.0.0.0 Label: 8084432 (Raw Label: 0x7b5bd0)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 16 Extended Community:
target:64500:6
evpn-bandwidth:1:1
Flag: 0xc0 Type: 40 Len: 37 Prefix-SID-attr:
SRv6 Services TLV (37 bytes):-
Type: SRV6 L3 Service TLV (5)
Length: 34 bytes, Reserved: 0x0
SRv6 Service Information Sub-TLV (33 bytes)
Type: 1 Len: 30 Rsvd1: 0x0
SRv6 SID: 2001:db8:aaaa:103::
SID Flags: 0x0 Endpoint Behavior: 0x13 Rsvd2: 0x0
SRv6 SID Sub-Sub-TLV
Type: 1 Len: 6
BL:48 NL:16 FL:20 AL:0 TL:20 TO:64
"TOR3 installs in its VPRN route table and VPRN FIB: each EBGP PE-CE route for prefix 198.51.100.1/32 that TOR3 receives via EBGP from CNF2, CNF3, and CNF4, with normalised ECMP weights W[1]=1, W[2]=5, W[3]=56:
*A:TOR3# show router 6 route-table 198.51.100.1/32 extensive
===============================================================================
Route Table (Service: 6)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : BGP
Age : 00h02m07s
Preference : 170
Indirect Next-Hop : 10.63.12.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.63.12.2
Interface : to-CE2
Metric : 0
ECMP-Weight : N/A
Indirect Next-Hop : 10.63.13.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 5
Resolving Next-Hop : 10.63.13.2
Interface : to-CE3
Metric : 0
ECMP-Weight : N/A
Indirect Next-Hop : 10.63.14.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 56
Resolving Next-Hop : 10.63.14.2
Interface : to-CE4
Metric : 0
ECMP-Weight : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:TOR3# show router 6 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 6)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : BGP
Installed : Y
Indirect Next-Hop : 10.63.12.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.63.12.2
Interface : to-CE2 (VPRN 6)
ECMP-Weight : 1
Indirect Next-Hop : 10.63.13.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 5
Resolving Next-Hop : 10.63.13.2
Interface : to-CE3 (VPRN 6)
ECMP-Weight : 1
Indirect Next-Hop : 10.63.14.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 56
Resolving Next-Hop : 10.63.14.2
Interface : to-CE4 (VPRN 6)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================Similarly, TOR4 installs in its VPRN route table and VPRN FIB: the EBGP PE-CE route for prefix 198.51.100.1/32 that TOR4 receives via EBGP from CNF5, with normalised ECMP weight W[1]=1.
BL5 receives via IBGP two EVPN-IFL routes for prefix 198.51.100.1/32: one from TOR3 with S[1]=128 and one from TOR4 with S[2]=1. So GCD=1, Stotal=129, S[1]/GCD=128, S[2]/GCD=1, Stotal/GCD=129 (larger than 64). The normalised weights are computed, such that sum(W[i]*GCD) is not larger than 64. W[1]=62, W[2]=1.
BL5 installs in its VPRN route table and VPRN FIB: the two received EVPN-IFL routes for prefix 198.51.100.1/32, with normalised ECMP weights W[1]=62 and W[2]=1:
*A:BL5# show router 6 route-table 198.51.100.1/32 extensive
===============================================================================
Route Table (Service: 6)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFL
Age : 00h02m10s
Preference : 170
Indirect Next-Hop : 192.0.2.3
SRV6 SID : 2001:db8:aaaa:103:7b5b:d000::
VPN Next-Hop Index : 48
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 62
Resolving Next-Hop : 2001:db8:aaaa:103:7b5b:d000:: (SRV6 tunnel)
Metric : 10
ECMP-Weight : 62
Indirect Next-Hop : 192.0.2.4
SRV6 SID : 2001:db8:aaaa:104:7b5b:d000::
VPN Next-Hop Index : 50
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 2001:db8:aaaa:104:7b5b:d000:: (SRV6 tunnel)
Metric : 10
ECMP-Weight : 1
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:BL5# show router 6 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 6)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFL
Installed : Y
Indirect Next-Hop : 192.0.2.3
SRV6 SID : 2001:db8:aaaa:103:7b5b:d000::
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 62
Resolving Next-Hop : 2001:db8:aaaa:103::/64 (SRV6-ISIS tunnel:524291)
ECMP-Weight : 1
Indirect Next-Hop : 192.0.2.4
SRV6 SID : 2001:db8:aaaa:104:7b5b:d000::
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 2001:db8:aaaa:104::/64 (SRV6-ISIS tunnel:524292)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================The EVPN IP prefix routes that the BL uses to reach the 198.51.100.1/32 prefix can be verified as follows:
*A:BL5# show router bgp routes evpn ip-prefix prefix 198.51.100.1/32
===============================================================================
BGP Router ID:192.0.2.5 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Flag Route Dist. Prefix
Tag Gw Address
NextHop
Label
ESI
-------------------------------------------------------------------------------
u*>i 6:3 198.51.100.1/32
0 00:00:00:00:00:00
192.0.2.3
505277
ESI-0
u*>i 6:4 198.51.100.1/32
0 00:00:00:00:00:00
192.0.2.4
505277
ESI-0
-------------------------------------------------------------------------------
Routes : 2
===============================================================================The weight that TOR3 signals in the extended community for prefix 198.51.100.1/32 can be verified in the detail of the route entry for TOR3 (route distinguisher: 6:3), as follows:
*A:BL5# show router bgp routes evpn ip-prefix prefix 198.51.100.1/32
rd 6:3 detail
===============================================================================
BGP Router ID:192.0.2.5 AS:64500 Local AS:64500
===============================================================================
---snip---
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Original Attributes
Network : n/a
Nexthop : 192.0.2.3
Path Id : None
From : 192.0.2.3
Res. Nexthop : 192.168.35.1
Local Pref. : 100 Interface Name : int-BL5-TOR3
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
AIGP Metric : None IGP Cost : 10
Connector : None
Community : target:64500:6 evpn-bandwidth:1:128
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 192.0.2.3
Origin : IGP
Flags : Used Valid Best
Route Source : Internal
AS-Path : 64501
EVPN type : IP-PREFIX
ESI : ESI-0
Tag : 0
Gateway Address: 00:00:00:00:00:00
Prefix : 198.51.100.1/32
Route Dist. : 6:3
MPLS Label : 505277
Route Tag : 0
Neighbor-AS : 64501
DB Orig Val : N/A Final Orig Val : N/A
Source Class : 0 Dest Class : 0
Add Paths Send : Default
Last Modified : 00h02m41s
SRv6 TLV Type : SRv6 L3 Service TLV (5)
SRv6 SubTLV : SRv6 SID Information (1)
Sid : 2001:db8:aaaa:103::
Full Sid : 2001:db8:aaaa:103:7b5b:d000::
Behavior : End.DT4 (19)
SRv6 SubSubTLV : SRv6 SID Structure (1)
Loc-Block-Len : 48 Loc-Node-Len : 16
Func-Len : 20 Arg-Len : 0
Tpose-Len : 20 Tpose-offset : 64
Modified Attributes
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================IPv4 prefix advertisement from CNFs and BL
The CNF advertises prefix 198.51.100.1/32 in its specific EBGP PE-CE session to its corresponding TOR. To enable unequal ECMP processing for EVPN-IFL IPv4 prefix routes, the advertising and receiving procedures are configured in the service vprn <vprn> bgp-evpn mpls (or service vprn <vprn> bgp-evpn segment-routing-v6 bgp) context on the TOR and on the BL. To indicate with what relative weight the load balancing across the EVPN-IFL IPv4 prefix routes must be executed, the add-to-received-bgp <number> is configured for the EBGP PE-CE session in the service vprn <vprn> bgp group <EBGP group> neighbor <EBGP neighbor> context on the TOR.
VPRN 1 is configured on CNF1, TOR1, and BL5.
# On CNF1:
configure service vprn 1 name "VPRN 1" customer 1 create
autonomous-system 64501
interface "to-TOR1" create
address 10.11.11.2/30
sap 1/1/c1/1:10 create
exit
exit
interface "subs-1" create
address 198.51.100.1/32
loopback
exit
bgp
rapid-withdrawal
group "PE-CE"
type external
peer-as 64500
neighbor 10.11.11.1
export "vrf-1-export-1"
exit
exit
no shutdown
exit
no shutdownVPRN 1 is configured on TOR1 as follows:
# On TOR1:
configure service vprn 1 name "VPRN 1" customer 1 create
ecmp 3
autonomous-system 64500
interface "to-CE1" create
address 10.11.11.1/30
sap 1/1/c1/1:10 create
exit
exit
bgp-evpn
mpls
auto-bind-tunnel
resolution any
exit
evi 1
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 192.0.2.1:1
vrf-target target:64500:1
no shutdown
exit
exit
bgp
multi-path
ipv4 10
exit
rapid-withdrawal
group "PE-CE"
type external
peer-as 64501
neighbor 10.11.11.2
evpn-link-bandwidth
add-to-received-bgp 1
exit
exit
exit
no shutdown
exit
no shutdownVPRN 1 is reconfigured on BL5 as follows:
# On BL5:
configure service vprn 1
ecmp 3
autonomous-system 64500
interface "subs-1" create
address 198.51.100.1/32
loopback
exit
bgp-evpn
mpls
auto-bind-tunnel
resolution any
exit
evi 1
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 192.0.2.5:1
vrf-export "vrf-1-export-1"
vrf-target target:64500:1
no shutdown
exit
exit
bgp
no shutdown
exit
no shutdown- installs in its VPRN route table and FIB: the EBGP PE-CE route for prefix 198.51.100.1/32 that the TOR receives via EBGP from the CNF, with weight 1
- does not install in its VPRN route table and FIB: the EVPN-IFL route for prefix 198.51.100.1/32 that the TOR receives via IBGP from the BL, because EBGP learned routes (PE-CE) have preference over IBGP learned routes (EVPN-IFL).
- installs in its VPRN route table and FIB: the EBGP PE-CE route for prefix 198.51.100.1/32 that the TOR receives via EBGP from the CNF, with weight 1
- additionally installs in its VPRN route table and FIB: the
EVPN-IFL route for prefix 198.51.100.1/32 that the TOR receives
via IBGP from the BL, with weight 1, because EBGP learned routes
(PE-CE) and IBGP learned routes (EVPN-IFL) are then treated
equally, and EVPN-IFL routes have the same preference as EBGP
PE-CE routes.Note: configure router bgp path-selection bgp-ibgp-equal is not needed if service vprn <vprn> bgp eibgp-loadbalance and ecmp>1 are configured under the VRF.VPRN 1 is reconfigured on TOR1 with:
# On TOR1: configure service vprn 1 bgp eibgp-loadbalance*A:TOR1# show router 1 route-table 198.51.100.1/32 extensive =============================================================================== Route Table (Service: 1) =============================================================================== Dest Prefix : 198.51.100.1/32 Protocol : BGP Age : 00h00m11s Preference : 170 Indirect Next-Hop : 10.11.11.2 QoS : Priority=n/c, FC=n/c Source-Class : 0 Dest-Class : 0 ECMP-Weight : 1 Resolving Next-Hop : 10.11.11.2 Interface : to-CE1 Metric : 0 ECMP-Weight : N/A Indirect Next-Hop : 192.0.2.5 Label : 524277 QoS : Priority=n/c, FC=n/c Source-Class : 0 Dest-Class : 0 ECMP-Weight : 1 Resolving Next-Hop : 192.0.2.5 (LDP tunnel) Metric : 10 ECMP-Weight : N/A ------------------------------------------------------------------------------- No. of Destinations: 1 ===============================================================================The EVPN IP prefix routes that the TOR uses to reach the 198.51.100.1/32 prefix can be verified as follows:
*A:TOR1# show router bgp routes evpn ip-prefix =============================================================================== BGP Router ID:192.0.2.1 AS:64500 Local AS:64500 =============================================================================== Legend - Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid l - leaked, x - stale, > - best, b - backup, p - purge Origin codes : i - IGP, e - EGP, ? - incomplete =============================================================================== BGP EVPN IP-Prefix Routes =============================================================================== Flag Route Dist. Prefix Tag Gw Address NextHop Label ESI ------------------------------------------------------------------------------- u*>i 192.0.2.5:1 198.51.100.1/32 0 00:00:00:00:00:00 192.0.2.5 LABEL 524277 ESI-0 ------------------------------------------------------------------------------- Routes : 1 ===============================================================================
EVPN link bandwidth advertisement via policies
# On CNF1:
configure router "Base" policy-options
begin
prefix-list "prefix-5"
prefix 198.51.100.5/32 exact
exit
policy-statement "vrf-1-export-5"
entry 10
from
prefix-list "prefix-5"
exit
to
protocol bgp
exit
action accept
exit
exit
exit
commit# On CNF1:
configure service vprn 1 name "VPRN 1" customer 1 create
autonomous-system 64501
interface "to-TOR1" create
address 10.11.11.2/30
sap 1/1/c1/1:10 create
exit
exit
interface "subs-5" create
address 198.51.100.5/32
loopback
exit
bgp
rapid-withdrawal
group "PE-CE"
type external
peer-as 64500
neighbor 10.11.11.1
export "vrf-1-export-5"
exit
exit
no shutdown
exit
no shutdown# On TOR1:
configure service vprn 1 name "VPRN 1" customer 1 create
ecmp 3
autonomous-system 64500
interface "to-CE1" create
address 10.11.11.1/30
sap 1/1/c1/1:10 create
exit
exit
bgp-evpn
mpls
auto-bind-tunnel
resolution any
exit
evi 1
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 192.0.2.1:1
vrf-target target:64500:1
no shutdown
exit
exit
bgp
multi-path
ipv4 10
exit
eibgp-loadbalance
rapid-withdrawal
group "PE-CE"
type external
peer-as 64501
neighbor 10.11.11.2
evpn-link-bandwidth
add-to-received-bgp 1
exit
exit
exit
no shutdown
exit
no shutdown# On BL5:
configure service vprn 1 name "VPRN 1" customer 1 create
ecmp 3
autonomous-system 64500
bgp-evpn
mpls
auto-bind-tunnel
resolution any
exit
evi 1
evpn-link-bandwidth
advertise
weighted-ecmp
exit
route-distinguisher 192.0.2.5:1
vrf-target target:64500:1
no shutdown
exit
exit
bgp
no shutdown
exit
no shutdown- Type: EVPN (0x06)
- Sub-type: EVPN Link BW (0x10)
- Value Units: encoded as:
- 0x00 weight expressed using default units of Mbps
- 0x01 Generalised Weight (supported in SR OS Release 22 and further)
- Value Weight: when Value Units = 0x01, this field encodes the number of PE-CE multi-paths for a specific prefix that is re-advertised into an RT5.
As an example, 0610:010000000002 represents evpn-bandwidth:1:2. When evpn-link-bandwidth advertise is not configured on the BL, but an "evpn-link-bw" community with members "ext:0610:010000000002" is configured instead in an EBGP export policy for the BL, the BL advertises the IPv4 prefix together with the additional EVPN link bandwidth extended community evpn-bandwidth:1:2 to the TOR.
# On BL5:
configure router "Base" policy-options
begin
community "comm-vrf-1"
members "target:64500:1"
exit
community "evpn-link-bw"
members "ext:0610:010000000002"
exit
policy-statement "vrf-1-export-5"
entry 10
action accept
as-path-prepend 64503 1
community add "evpn-link-bw" "comm-vrf-1"
exit
exit
exit
commitThe as-path-prepend command ensures that the prefixes are advertised as if they originate in an external autonomous system.
# On BL5:
configure service vprn 1
ecmp 3
autonomous-system 64500
interface "subs-5" create
address 198.51.100.5/32
loopback
exit
bgp-evpn
mpls
auto-bind-tunnel
resolution any
exit
evi 1
evpn-link-bandwidth
weighted-ecmp
exit
route-distinguisher 192.0.2.5:1
vrf-export "vrf-1-export-5"
vrf-target target:64500:1
no shutdown
exit
exit
bgp
no shutdown
exit
no shutdownAs for any other transitive extended community, BGP accepts the EVPN link bandwidth extended community from IBGP or EBGP neighbors and re-advertises it to IBGP or EBGP neighbors. As for any other EVPN extended community, the EVPN link bandwidth extended community is not propagated when re-exporting RT5s installed in the VPRN route table into other BGP owners (for example: EVPN-IFF, IFL, VPN-IP, BGP PE-CE).
TOR1 receives a BGP update from BL5 for the EVPN IP prefix route to 198.51.100.5/32, containing the EVPN link bandwidth extended community with the specified weight 2.
# On TOR1:
1 2024/12/09 22:08:20.541 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Received BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 96
Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.5:1, ESI: ESI-0, tag: 0,
ip_prefix: 198.51.100.5/32 gw_ip 0.0.0.0 Label: 8388432 (Raw Label: 0x7fff50)
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 6 AS Path:
Type: 2 Len: 1 < 64503 >
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
evpn-bandwidth:1:2
target:64500:1
bgp-tunnel-encap:MPLS
"- installs in its VPRN route table and FIB the EBGP PE-CE route for prefix 198.51.100.5/32 that it receives via EBGP from CNF1, with weight 1
- additionally installs in its VPRN route table and FIB the EVPN-IFL route for prefix 198.51.100.5/32 that it receives via IBGP from BL5, with weight 2, because service vprn 1 bgp eibgp-loadbalance is configured on TOR1
*A:TOR1# show router 1 route-table 198.51.100.5/32 extensive
===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.5/32
Protocol : BGP
Age : 00h00m11s
Preference : 170
Indirect Next-Hop : 10.11.11.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.11.11.2
Interface : to-CE1
Metric : 0
ECMP-Weight : N/A
Indirect Next-Hop : 192.0.2.5
Label : 524277
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 2
Resolving Next-Hop : 192.0.2.5 (LDP tunnel)
Metric : 10
ECMP-Weight : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:TOR1# show router 1 fib 1 198.51.100.5/32 extensive
===============================================================================
FIB Display (Service: 1)
===============================================================================
Dest Prefix : 198.51.100.5/32
Protocol : BGP
Installed : Y
Indirect Next-Hop : 10.11.11.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.11.11.2
Interface : to-CE1 (VPRN 1)
ECMP-Weight : 1
Indirect Next-Hop : 192.0.2.5
Label : 524277
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 2
Resolving Next-Hop : 192.0.2.5 (LDP tunnel)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================*A:TOR1# show router bgp routes evpn ip-prefix prefix 198.51.100.5/32
===============================================================================
BGP Router ID:192.0.2.1 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Flag Route Dist. Prefix
Tag Gw Address
NextHop
Label
ESI
-------------------------------------------------------------------------------
u*>i 192.0.2.5:1 198.51.100.5/32
0 00:00:00:00:00:00
192.0.2.5
LABEL 524277
ESI-0
-------------------------------------------------------------------------------
Routes : 1
===============================================================================*A:TOR1# show router bgp routes evpn ip-prefix prefix 198.51.100.5/32 detail
===============================================================================
BGP Router ID:192.0.2.1 AS:64500 Local AS:64500
===============================================================================
---snip---
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Original Attributes
Network : n/a
Nexthop : 192.0.2.5
Path Id : None
From : 192.0.2.5
Res. Nexthop : 192.168.15.2
Local Pref. : 100 Interface Name : int-TOR1-BL5
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
AIGP Metric : None IGP Cost : 10
Connector : None
Community : evpn-bandwidth:1:2 target:64500:1 bgp-tunnel-encap:MPLS
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 192.0.2.5
Origin : IGP
Flags : Used Valid Best
Route Source : Internal
AS-Path : 64503
EVPN type : IP-PREFIX
ESI : ESI-0
Tag : 0
Gateway Address: 00:00:00:00:00:00
Prefix : 198.51.100.5/32
Route Dist. : 192.0.2.5:1
MPLS Label : LABEL 524277
Route Tag : 0
Neighbor-AS : 64503
DB Orig Val : N/A Final Orig Val : N/A
Source Class : 0 Dest Class : 0
Add Paths Send : Default
Last Modified : 00h00m45s
Modified Attributes
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================EVPN-IFF model
# On CNF1:
configure router "Base" policy-options
begin
prefix-list "prefix-1"
prefix 198.51.100.1/32 exact
exit
policy-statement "vrf-2-export-1"
entry 10
from
prefix-list "prefix-1"
exit
to
protocol bgp
exit
action accept
exit
exit
exit
commitTo advertise prefix 198.51.100.1/32, the following export policy is configured on BL5:
# On BL5:
configure router "Base" policy-options
begin
community "comm-vrf-2"
members "target:64500:2"
exit
policy-statement "vrf-2-export-1"
entry 10
action accept
as-path-prepend 64503 1
community add "comm-vrf-2"
exit
exit
exit
commitThe as-path-prepend command ensures that the prefixes are advertised as if they originate in an external autonomous system.
# On TOR1 and BL5:
configure service vpls 21 name "VPLS 21" customer 1 create
allow-ip-int-bind
exit
vxlan instance 1 vni 2 create
exit
bgp
exit
bgp-evpn
ip-route-advertisement
ip-route-link-bandwidth
advertise weight dynamic max-dynamic-weight 128
weighted-ecmp
exit
evi 21
vxlan bgp 1 vxlan-instance 1
no shutdown
exit
exit
stp
shutdown
exit
no shutdown# On CNF1:
configure service vprn 2 name "VPRN 2" customer 1 create
autonomous-system 64501
interface "to-TOR1" create
address 10.21.11.2/30
sap 1/1/c1/1:20 create
exit
exit
interface "subs-1" create
address 198.51.100.1/32
loopback
exit
bgp
rapid-withdrawal
group "PE-CE"
type external
peer-as 64500
neighbor 10.21.11.1
export "vrf-2-export-1"
exit
exit
no shutdown
exit
no shutdown# On TOR1:
configure service vprn 2 name "VPRN 2" customer 1 create
ecmp 3
autonomous-system 64500
interface "BD 1" create
vpls "VPLS 21"
evpn-tunnel
exit
exit
interface "to-CE1" create
address 10.21.11.1/30
sap 1/1/c1/1:20 create
exit
exit
bgp
rapid-withdrawal
group "PE-CE"
type external
peer-as 64501
neighbor 10.21.11.2
evpn-link-bandwidth
add-to-received-bgp 7
exit
exit
exit
no shutdown
exit
no shutdown# On BL5:
configure service vprn 2 name "VPRN 2" customer 1 create
autonomous-system 64500
interface "BD 1" create
vpls "VPLS 21"
evpn-tunnel
exit
exit
bgp-evpn
mpls
route-distinguisher 192.0.2.5:2
vrf-target target:64500:2
no shutdown
exit
exit
bgp
no shutdown
exit
no shutdownIPv4 prefix advertisement from CNFs
The CNF advertises prefix 198.51.100.1/32 in its specific EBGP PE-CE session to its corresponding TOR. To enable unequal ECMP processing for EVPN-IFF IPv4 prefix routes, the advertising and receiving procedures are configured in the service vpls <vpls> bgp-evpn context on the TOR and on the BL. To indicate with what relative weight the load balancing across the EVPN-IFF IPv4 prefix routes must be executed, the add-to-received-bgp <number> is configured for the EBGP PE-CE session in the service vprn <vprn> bgp group <EBGP group> neighbor <EBGP neighbor> context on the TOR.
TOR1 installs in its VPRN route table and FIB: the EBGP PE-CE route for prefix 198.51.100.1/32 that the TOR receives via EBGP from the CNF, with ECMP weight 1. The Next hop is the interface to the CE. The preference for the PE-CE route route is 170.
*A:TOR1# show router 2 route-table 198.51.100.1/32
===============================================================================
Route Table (Service: 2)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
198.51.100.1/32 Remote BGP 00h00m45s 170
10.21.11.2 0
-------------------------------------------------------------------------------
No. of Routes: 1
---snip---
===============================================================================*A:TOR1# show router 2 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 2)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : BGP
Installed : Y
Indirect Next-Hop : 10.21.11.2
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
Resolving Next-Hop : 10.21.11.2
Interface : to-CE1 (VPRN 2)
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================*A:BL5# show router 2 route-table
===============================================================================
Route Table (Service: 2)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
10.21.11.0/30 Remote EVPN-IFF 00h01m09s 169
BD 1 (ET-00:01:fe:ff:ff:52) 0
198.51.100.1/32 Remote EVPN-IFF 00h00m48s 169
BD 1 (ET-00:01:fe:ff:ff:52) 0
-------------------------------------------------------------------------------
No. of Routes: 2
---snip---
===============================================================================*A:BL5# show router 2 route-table 198.51.100.1/32 extensive
===============================================================================
Route Table (Service: 2)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFF
Age : 00h00m48s
Preference : 169
Next-Hop : BD 1 (ET-00:01:fe:ff:ff:52)
Interface : BD 1
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
Metric : 0
ECMP-Weight : 1
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================*A:BL5# show router 2 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 2)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFF
Installed : Y
Next-Hop : BD 1 (ET-00:01:fe:ff:ff:52)
Interface : BD 1 (VPRN 2)
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================The weight that TOR1 signals in the extended community for prefix 198.51.100.1/32 can be verified in the detail of the route entry for TOR1 (route distinguisher: 192.0.2.1:21), as follows:
*A:BL5# show router bgp routes evpn ip-prefix prefix 198.51.100.1/32 detail
===============================================================================
BGP Router ID:192.0.2.5 AS:64500 Local AS:64500
===============================================================================
---snip---
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Original Attributes
Network : n/a
Nexthop : 192.0.2.1
Path Id : None
From : 192.0.2.1
Res. Nexthop : 192.168.15.1
Local Pref. : 100 Interface Name : int-BL5-TOR1
Aggregator AS : None Aggregator : None
Atomic Aggr. : Not Atomic MED : None
AIGP Metric : None IGP Cost : 10
Connector : None
Community : target:64500:21 evpn-bandwidth:1:7 mac-nh:00:01:fe:ff:ff:52
bgp-tunnel-encap:VXLAN
Cluster : No Cluster Members
Originator Id : None Peer Router Id : 192.0.2.1
Origin : IGP
Flags : Used Valid Best
Route Source : Internal
AS-Path : No As-Path
EVPN type : IP-PREFIX
ESI : ESI-0
Tag : 0
Gateway Address: 00:01:fe:ff:ff:52
Prefix : 198.51.100.1/32
Route Dist. : 192.0.2.1:21
MPLS Label : VNI 2
Route Tag : 0
Neighbor-AS : n/a
DB Orig Val : N/A Final Orig Val : N/A
Source Class : 0 Dest Class : 0
Add Paths Send : Default
Last Modified : 00h00m48s
Modified Attributes
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================IPv4 prefix advertisement from CNFs and BL
The CNF advertises prefix 198.51.100.1/32 in its specific EBGP PE-CE session to its corresponding TOR. To enable unequal ECMP processing for EVPN-IFF IPv4 prefix routes, the advertising and receiving procedures are configured in the service vpls <vpls> bgp-evpn context on the TOR and on the BL. To indicate with what relative weight the load balancing across the EVPN-IFF IPv4 prefix routes must be executed, the add-to-received-bgp <number> is configured for the EBGP PE-CE session in the service vprn <vprn> bgp group <EBGP group> neighbor <EBGP neighbor> context on the TOR. The BL advertises the 198.51.100.1/32 prefix as if it is received from an EBGP session. This requires adding an external community via an export policy.
VPRN 2 is reconfigured on BL5 as follows:
# On BL5:
configure service vprn 2
autonomous-system 64500
interface "BD 1" create
vpls "VPLS 21"
evpn-tunnel
exit
exit
interface "subs-1" create
address 198.51.100.1/32
loopback
exit
bgp-evpn
mpls
route-distinguisher 192.0.2.5:2
vrf-export "vrf-2-export-1"
vrf-target target:64500:2
no shutdown
exit
exit
bgp
no shutdown
exit
no shutdown- installs in its VPRN route table and FIB: the EVPN-IFF route for prefix 198.51.100.1/32 that it receives via IBGP from the BL, with weight 1, because the EVPN-IFF route has lower priority than the EBGP PE-CE route and always wins. The Next hop is the interface to the VPLS.
*A:TOR1# show router 2 route-table
===============================================================================
Route Table (Service: 2)
===============================================================================
Dest Prefix[Flags] Type Proto Age Pref
Next Hop[Interface Name] Metric
-------------------------------------------------------------------------------
10.21.11.0/30 Local Local 00h02m48s 0
to-CE1 0
198.51.100.1/32 Remote EVPN-IFF 00h00m32s 169
BD 1 (ET-00:05:fe:ff:ff:52) 0
-------------------------------------------------------------------------------
No. of Routes: 2
---snip---
===============================================================================*A:TOR1# show router 2 fib 1 198.51.100.1/32 extensive
===============================================================================
FIB Display (Service: 2)
===============================================================================
Dest Prefix : 198.51.100.1/32
Protocol : EVPN-IFF
Installed : Y
Next-Hop : BD 1 (ET-00:05:fe:ff:ff:52)
Interface : BD 1 (VPRN 2)
QoS : Priority=n/c, FC=n/c
Source-Class : 0
Dest-Class : 0
ECMP-Weight : 1
===============================================================================
Total Entries : 1
===============================================================================The BGP route to prefix 198.51.100.1/32 on CNF1 is not used:
*A:TOR1# show router 2 bgp routes
===============================================================================
BGP Router ID:192.0.2.1 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP IPv4 Routes
===============================================================================
Flag Network LocalPref MED
Nexthop (Router) Path-Id IGP Cost
As-Path Label
-------------------------------------------------------------------------------
*i 198.51.100.1/32 None None
10.21.11.2 None 0
64501 -
-------------------------------------------------------------------------------
Routes : 1
===============================================================================Conclusion
SR OS supports unequal ECMP for EVPN IPv4 and IPv6 prefix routes in the EVPN IFL and EVPN IFF models. Traffic from the BL to the anycast IP prefix destination is load balanced according to configurable weights. EVPN-IFL IP prefix routes and EBGP PE-CE routes are combined in the ECMP set for an IP prefix route. EVPN-IFF IP prefix routes are by default preferred over EBGP PE-CE routes in the ECMP set for an IP prefix route.