EVPN Interconnect Ethernet Segments in Dual EVPN-VXLAN Instance VPLS Services
This chapter provides information about EVPN Interconnect Ethernet Segments in Dual EVPN-VXLAN Instance VPLS Services.
Topics in this chapter include:
Applicability
The information and configuration in this chapter are based on SR OS Release 21.7.R1. EVPN multi-homing on dual VXLAN instance VPLS services is supported on SR OS Release 19.10.R1, and later.
Overview
Some service providers are deploying large Data Centers (DCs) where SR OS routers are used as leaf switches in a VXLAN fabric. In those cases, all-active multi-homing can provide redundancy and maximize the bandwidth utilization.
SR OS supports Interconnect Ethernet Segments (I-ESs) for VXLAN as per RFC 9014. Chapter EVPN Interconnect Ethernet Segments (I-ESs) describes how I-ESs allow Data Center Gateways (DCGWs) with two BGP instances (one for EVPN-MPLS and one for EVPN-VXLAN) to handle redundancy in VXLAN access networks, as supported in SR OS 15.0.R4, and later.
This chapter describes similar scenarios with EVPN-VXLAN in the core network instead of EVPN-MPLS. The following scenarios are supported with I-ES in VXLAN instance 1:
-
dual instance VPLS with two EVPN-VXLAN instances
-
dual instance VPLS with one EVPN-VXLAN instance and one static VXLAN instance
-
dual instance VPLS with one EVPN-VXLAN instance and one EVPN-MPLS instance
The first two of these scenarios are described in this chapter.
CLI
Sample topology shows VPLS 1 with different EVPN-VXLAN instances: VXLAN instance 1 in DC 1 (and DC2) and VXLAN instance 2 in the WAN.
On DCGW PE-2, the following all-active I-ES is configured for VXLAN instance 1 and service id 1:
# on DCGW PE-2:
configure {
service {
system {
bgp-auto-rd-range {
ip-address 192.0.2.2
community-value {
start 1
end 1000
}
}
bgp {
evpn {
ethernet-segment "I-ES-23_1" {
admin-state enable
type virtual
esi 00:23:23:23:23:23:23:00:00:01
multi-homing-mode all-active
df-election {
service-carving-mode manual
manual {
evi 1 {
end 1
}
preference {
value 100
}
}
}
association {
network-interconnect-vxlan 1 {
virtual-ranges {
service-id 1 {
end 1
}
}
}
}
}
}
The following command configures VPLS 1 with dual EVPN-VXLAN instance. VXLAN instance 1 is a member of the I-ES and VXLAN instance 2 is configured with mh-mode network and routes>auto-disc>advertise true:
# on DCGW PE-2:
configure {
service {
vpls "VPLS 1" {
admin-state enable
service-id 1
customer "1"
vxlan {
instance 1 {
vni 11
rx-discard-on-ndf bum
}
instance 2 {
vni 12
}
}
bgp 1 {
route-distinguisher auto-rd
route-target {
export "target:64500:11"
import "target:64500:11"
}
}
bgp 2 {
route-distinguisher auto-rd
route-target {
export "target:64500:12"
import "target:64500:12"
}
}
bgp-evpn {
evi 1
vxlan 1 {
admin-state enable
vxlan-instance 1
default-route-tag 0xb
ecmp 2
routes {
auto-disc {
advertise true
}
}
}
vxlan 2 {
admin-state enable
vxlan-instance 2
default-route-tag 0xc
ecmp 2
mh-mode network
routes {
auto-disc {
advertise true
}
}
}
}
sap 1/2/1:1 {
}
}
By default, the multi-homing mode for EVPN-VXLAN is access, but for VXLAN instance 2, it is modified to mh-mode network. The following error is raised when attempting to configure VXLAN instance 1—as a member of an I-ES—with mh-mode network:
[ex:/configure service vpls "VPLS 1" bgp-evpn vxlan 1]
A:admin@PE-2# mh-mode network
*[ex:/configure service vpls "VPLS 1" bgp-evpn vxlan 1]
A:admin@PE-2# commit
MINOR: MGMT_CORE #4001: configure service vpls "VPLS 1" bgp-evpn vxlan 1 mh-mode - mh-mode network not supported when vxlan instance is member of ethernet-segment - configure service system bgp evpn ethernet-segment "I-ES-23_1" association network-interconnect-vxlan 1 virtual-ranges service-id 1
With mh-mode network configured, it is mandatory to configure routes>auto-disc>advertise true; for mh-mode access, it is optional. When routes>auto-disc>advertise is enabled in an access instance associated to an I-ES, AD per-ES/EVI routes and MAC/IP routes are advertised for the I-ES.
The following AD per-EVI route is sent by DCGW PE-2:
10 2021/09/29 17:53:31.575 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.1
"Peer 1: 192.0.2.1: UPDATE
Peer 1: 192.0.2.1 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 81
Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.2
Type: EVPN-AD Len: 25 RD: 192.0.2.2:1 ESI: 00:23:23:23:23:23:23:00:00:01,
tag: 0 Label: 11 (Raw Label: 0xb) PathId:
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
origin:64500:23
target:64500:11
bgp-tunnel-encap:VXLAN
"
For MAC routes and their ESI value for an access VXLAN instance, the following redistribution considerations apply.
-
With mh-mode access and routes>auto-disc>advertise true configured, MAC routes are redistributed from the instance network to the instance access with the I-ESI if present, regardless of the original ESI.
-
With mh-mode access and routes>auto-disc>advertise false, MAC routes are redistributed with zero ESI, regardless of the original ESI.
The following EVPN-MAC route is sent by DCGW PE-2 with I-ESI 00:23:23:23:23:23:23:00:00:01 of "I-ES-23_1":
20 2021/09/29 17:53:31.577 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.1
"Peer 1: 192.0.2.1: UPDATE
Peer 1: 192.0.2.1 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.2
Type: EVPN-MAC Len: 33 RD: 192.0.2.2:1 ESI: 00:23:23:23:23:23:23:00:00:01,
tag: 0, mac len: 48 mac: 00:ca:fe:ca:fe:06, IP len: 0, IP: NULL, label1: 11
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
origin:64500:23
target:64500:11
bgp-tunnel-encap:VXLAN
"
The following ES route is sent by DCGW PE-2:
29 2021/09/29 17:53:31.661 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 71
Flag: 0x90 Type: 14 Len: 34 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.2
Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.2:0
ESI: 00:23:23:23:23:23:23:00:00:01, IP-Len: 4 Orig-IP-Addr: 192.0.2.2
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 16 Extended Community:
df-election::DF-Type:Preference/DP:0/DF-Preference:100/AC:1
target:23:23:23:23:23:23
"
The following commands are not supported when mh-mode network is configured:
-
proxy-arp/nd
-
assisted replication
-
source-vtep-security
Attempting to enable these unsupported commands while a BGP-EVPN VXLAN instance has mh-mode network triggers error messages, as follows:
-
proxy-arp
*[ex:/configure service vpls "VPLS 1" proxy-arp] A:admin@PE-2# commit MINOR: SVCMGR #12: configure service vpls "VPLS 1" bgp-evpn vxlan 2 mh-mode - Inconsistent Value error - mh-mode network not supported with proxy-arp - configure service vpls "VPLS 1" proxy-arp MINOR: MGMT_CORE #4001: configure service vpls "VPLS 1" - multiple bgp-evpn instances not supported with proxy-arp -
proxy-nd
*[ex:/configure service vpls "VPLS 1" proxy-nd] A:admin@PE-2# commit MINOR: SVCMGR #12: configure service vpls "VPLS 1" bgp-evpn vxlan 2 mh-mode - Inconsistent Value error - mh-mode network not supported with proxy-nd - configure service vpls "VPLS 1" proxy-nd MINOR: MGMT_CORE #4001: configure service vpls "VPLS 1" - multiple bgp-evpn instances not supported with proxy-nd -
assisted replication
[ex:/configure service vpls "VPLS 1" vxlan instance 2 assisted-replication] A:admin@PE-2# replicator *[ex:/configure service vpls "VPLS 1" vxlan instance 2 assisted-replication] A:admin@PE-2# commit MINOR: MGMT_CORE #4001: configure service vpls "VPLS 1" bgp-evpn vxlan 2 vxlan-instance - replicator role on vxlan instance not supported when it is in use by bgp-evpn with mh-mode network - configure service vpls "VPLS 1" vxlan instance 2 assisted-replication replicator -
source-vtep-security
[ex:/configure service vpls "VPLS 1" vxlan instance 2] A:admin@PE-2# source-vtep-security *[ex:/configure service vpls "VPLS 1" vxlan instance 2] A:admin@PE-2# commit MINOR: MGMT_CORE #4001: configure service vpls "VPLS 1" bgp-evpn vxlan 2 vxlan-instance - source-vtep-security on vxlan instance not supported when it is in use by bgp-evpn with mh-mode network - configure service vpls "VPLS 1" vxlan instance 2 source-vtep-security
Local bias
When EVPN-VXLAN is used in the instance network of a dual-instance VPLS service, local bias—as described in RFC 8365—is used for split horizon in all-active I-ESs. In VXLAN, there is no multicast label or multicast BMAC, so BUM traffic is identified by the MAC destination address. The modified forwarding rules for the I-ES-sourced BUM traffic for ingress PE and egress PE are as follows:
-
ingress PE
-
The Non-Designated Forwarder (NDF) must discard BUM traffic, so one of the following two commands must be configured in VXLAN instance 1.
-
send-incl-mcast-ir-on-ndf false
-
rx-discard-on-ndf bum
-
-
BUM frames received on any SAP or I-ES VXLAN binding are flooded to:
-
local non-ES and single-active DF ES SAPs
-
local all-active ES SAPs (DF and NDF)
-
EVPN-VXLAN destinations (BUM frames received on an I-ES VXLAN binding follow split-horizon rules, so they can only be forwarded to EVPN-VXLAN destinations belonging to the other VXLAN instance.)
-
-
-
egress PE
-
Look up source VTEP for BUM frames received on EVPN-VXLAN.
-
If the source VTEP matches a PE with which the local PE shares an ES and a VXLAN service, then the local PE does not forward to the shared local ESs (this includes port, lag, and network-interconnect-VXLAN ESs).
-
The local PE forwards to local ESs that are not shared but only when in DF state.
-
-
EVPN-VXLAN network interconnect VXLAN multi-homing and local bias shows the BUM forwarding with local bias procedures in multi-instance VPLS services.
In the example, GW1 and GW2 are configured with send-incl-mcast-ir-on-ndf false. TOR1 generates BUM traffic that will only reach DF GW1 and is forwarded as follows.
-
Ingress PE GW1 forwards to CE-1 and EVPN-VXLAN destinations GW2, GW3, and GW4.
-
Egress PE GW2 identifies the source VTEP as a PE with which I-ES-1 is shared, so it does not forward the BUM frames to the local I-ES. PE GW2 forwards only to the non-shared ES and local SAPs, in this case, to CE-2.
-
Egress PE GW3 receives the BUM traffic with a source VTEP that does not match any PE with which GW3 shares an ES, so it forwards to all ESs that are DF, in this case, to CE-3.
-
Egress PE GW4 receives the BUM traffic with a source VTEP that does not match any PE with which GW4 shares an ES, so it forwards to all ESs that are DF, in this case, to TOR2 through I-ES-2.
Local bias with static VXLAN on I-ES
When a static VXLAN instance coexists with an EVPN-VXLAN instance in the same VPLS service, traffic blackholes may occur when the static VXLAN instance is associated to an all-active I-ES. This is because, when multi-homing is used with an EVPN-VXLAN network instance, the NDF PE always discards unknown unicast traffic to the static VXLAN instance (this is not the case with EVPN-MPLS if the unknown traffic has a BUM label).
All-active I-ES NDF PE-5 drops unknown unicast traffic shows the packet blackhole for unknown unicast traffic at all-active I-ES NDF PE-5.
In the event that the remote PE-1 has learned the destination MAC address MAC6 via I-ES-45 EVPN destination, but the DCGWs PE-4 and PE-5 do not know MAC6, regular aliasing procedures allow that PE-1 sends unicast traffic with destination MAC6 to the NDF PE-5, which does not know MAC6 and drops all unknown unicast traffic, creating a blackhole for the flow.
When a static VXLAN instance coexists with an EVPN-VXLAN instance in the same VPLS service, Nokia recommends using a single-active I-ES or an anycast solution without I-ES instead of an all-active I-ES.
Configuration
Sample topology shows the sample topology with six SR OS nodes:
The initial configuration includes:
-
Cards, MDAs, and ports
-
Router interfaces
-
IS-IS on all interfaces (level 1 in the DCs; level 2 in the WAN)
BGP is configured for the EVPN address family. PE-1 acts as Route Reflector (RR) in DC 1 and PE-6 as RR in DC 2; no RR is used in the WAN.
The BGP configuration on RR PE-1 in DC 1 is as follows. The BGP configuration on RR PE-6 in DC2 is similar.
# on PE-1:
configure {
router "Base" {
autonomous-system 64500
bgp {
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
family {
ipv4 false
evpn true
}
cluster {
cluster-id 192.0.2.1
}
rapid-update {
evpn true
}
group "DC" {
type internal
}
neighbor "192.0.2.2" {
group "DC"
}
neighbor "192.0.2.3" {
group "DC"
}
On DCGWs PE-2 and PE-3, BGP is configured as follows. The policies are explained in the next section.
# on PE-2, PE-3:
configure {
router "Base" {
autonomous-system 64500
bgp {
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
family {
ipv4 false
evpn true
}
rapid-update {
evpn true
}
group "DC" {
type internal
import {
policy ["drop SOO-DCGW-23"]
}
export {
policy ["export DC routes and add SOO"]
}
}
group "WAN" {
type internal
export {
policy ["export WAN routes only"]
}
}
neighbor "192.0.2.1" {
group "DC"
}
neighbor "192.0.2.4" {
group "WAN"
}
neighbor "192.0.2.5" {
group "WAN"
}
}
On DCGWs PE-4 and PE-5, BGP is configured as follows. The policies are explained in the next section.
# on PE-4, PE-5:
configure {
router "Base" {
autonomous-system 64500
bgp {
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
family {
ipv4 false
evpn true
}
rapid-update {
evpn true
}
group "DC" {
type internal
import {
policy ["drop SOO-DCGW-45"]
}
export {
policy ["export DC routes and add SOO"]
}
}
group "WAN" {
type internal
export {
policy ["export WAN routes only"]
}
}
neighbor "192.0.2.6" {
group "DC"
}
neighbor "192.0.2.2" {
group "WAN"
}
neighbor "192.0.2.3" {
group "WAN"
}
}
The following examples are configured:
All-active multi-homing I-ESs in dual EVPN-VXLAN instance VPLS
All-active multi-homing for I-ESs shows the example topology with the service VPLS 1 on all nodes and two all-active I-ESs:
On PE-1, VPLS 1 is configured as follows. The configuration on PE-6 is similar.
# on PE-1:
configure {
service {
system {
bgp-auto-rd-range {
ip-address 192.0.2.1
community-value {
start 1
end 1000
}
}
}
vpls "VPLS 1" {
admin-state enable
service-id 1
customer "1"
vxlan {
instance 1 {
vni 11
}
}
bgp 1 {
route-distinguisher auto-rd
route-target {
export "target:64500:11"
import "target:64500:11"
}
}
bgp-evpn {
evi 1
vxlan 1 {
admin-state enable
vxlan-instance 1
ecmp 2
}
}
sap 1/2/1:1 {
}
}
On DCGW PE-2, the following all-active multi-homing I-ES is configured for VXLAN instance 1 and service id 1. The configuration on DCGW PE-3 is similar, but the preference value is 150 instead of 100.
# on PE-2:
configure {
service {
system {
bgp-auto-rd-range {
ip-address 192.0.2.2
community-value {
start 1
end 1000
}
}
bgp {
evpn {
ethernet-segment "I-ES-23_1" {
admin-state enable
type virtual
esi 00:23:23:23:23:23:23:00:00:01
multi-homing-mode all-active
df-election {
service-carving-mode manual
manual {
evi 1 {
end 1
}
preference {
value 100 # on PE-3: preference value 150
}
}
}
association {
network-interconnect-vxlan 1 {
virtual-ranges {
service-id 1 {
end 1
}
}
}
}
}
}
}
On DCGWs PE-4 and PE-5, the following I-ES is configured:
# on PE-4, PE-5:
configure {
service {
system {
bgp {
evpn {
ethernet-segment "I-ES-45_1" {
admin-state enable
type virtual
esi 00:45:45:45:45:45:45:00:00:01
multi-homing-mode all-active
association {
network-interconnect-vxlan 1 {
virtual-ranges {
service-id 1 {
end 1
}
}
}
}
}
}
}
On DCGWs PE-2, PE-3, PE-4, and PE-5, VPLS 1 is configured as follows. The rx-discard-on-ndf bum command makes the NDF drop any BUM traffic in VXLAN instance 1. VXLAN instance 2 is configured with mh-mode network and routes>auto-disc>advertise true.
# on PE-2, PE-3, PE-4, PE-5:
configure {
service {
vpls "VPLS 1" {
admin-state enable
service-id 1
customer "1"
vxlan {
instance 1 {
vni 11
rx-discard-on-ndf bum
}
instance 2 {
vni 12
}
}
bgp 1 {
route-distinguisher auto-rd
route-target {
export "target:64500:11"
import "target:64500:11"
}
}
bgp 2 {
route-distinguisher auto-rd
route-target {
export "target:64500:12"
import "target:64500:12"
}
}
bgp-evpn {
evi 1
vxlan 1 {
admin-state enable
vxlan-instance 1
default-route-tag 0xb
ecmp 2
routes {
auto-disc {
advertise true
}
}
}
vxlan 2 {
admin-state enable
vxlan-instance 2
default-route-tag 0xc
ecmp 2
mh-mode network
routes {
auto-disc {
advertise true
}
}
}
}
sap 1/2/1:1 {
}
}
On PE-2 and PE-3, the following policies are configured.
-
The import policy "drop SOO-DCGW-23" in group "DC" is used to drop all VXLAN instance 1 routes between PE-2 and PE-3.
-
The export policy "export WAN routes only" in group "WAN" is applied to avoid sending VXLAN instance 1 routes to the WAN PEs.
-
The export policy "export DC routes and add SOO" in group "DC" is used to tag VXLAN instance 1 routes with community "SOO-23".
# on PE-2, PE-3:
configure {
policy-options {
community "SOO-23" {
member "origin:64500:23" { }
}
policy-statement "drop SOO-DCGW-23" { # import in group "DC"
entry 10 {
from {
family [evpn]
community {
name "SOO-23"
}
}
action {
action-type reject
}
}
default-action {
action-type accept
}
}
policy-statement "export WAN routes only" { # export in group "WAN"
entry 10 {
from {
family [evpn]
tag 11
}
action {
action-type reject
}
}
default-action {
action-type accept
}
}
policy-statement "export DC routes and add SOO" { # export in group "DC"
entry 10 {
from {
family [evpn]
tag 11
}
action {
action-type accept
community {
add ["SOO-23"]
}
}
}
default-action {
action-type accept
}
}
On PE-4 and PE-5, the following policies are configured:
# on PE-4, PE-5:
configure {
policy-options {
community "SOO-45" {
member "origin:64500:45" { }
}
policy-statement "drop SOO-DCGW-45" { # import in group "DC"
entry 10 {
from {
family [evpn]
community {
name "SOO-45"
}
}
action {
action-type reject
}
}
default-action {
action-type accept
}
}
policy-statement "export WAN routes only" { # export in group "WAN"
entry 10 {
from {
family [evpn]
tag 11
}
action {
action-type reject
}
}
default-action {
action-type accept
}
}
policy-statement "export DC routes and add SOO" { # export in group "DC"
entry 10 {
from {
family [evpn]
tag 11
}
action {
action-type accept
community {
add ["SOO-45"]
}
}
}
default-action {
action-type accept
}
}
For VPLS 1, PE-2 is DF and PE-3 is NDF in the I-ES "I-ES-23_1":
[/]
A:admin@PE-2# show service id 1 ethernet-segment
No sap entries
No sdp entries
===============================================================================
VXLAN Ethernet-Segment Information
===============================================================================
VXLAN Instance Eth-Seg Status
-------------------------------------------------------------------------------
1 I-ES-23_1 DF
===============================================================================
[/]
A:admin@PE-3# show service id 1 ethernet-segment
No sap entries
No sdp entries
===============================================================================
VXLAN Ethernet-Segment Information
===============================================================================
VXLAN Instance Eth-Seg Status
-------------------------------------------------------------------------------
1 I-ES-23_1 NDF
===============================================================================
PE-4 is NDF and PE-5 is DF in the I-ES "I-ES-45_1":
[/]
A:admin@PE-4# show service vxlan-instance-using ethernet-segment
===============================================================================
VXLAN Ethernet-Segment Information
===============================================================================
SvcId VXLAN Instance ES Name Status
-------------------------------------------------------------------------------
1 1 I-ES-45_1 NDF
===============================================================================
[/]
A:admin@PE-5# show service vxlan-instance-using ethernet-segment
===============================================================================
VXLAN Ethernet-Segment Information
===============================================================================
SvcId VXLAN Instance ES Name Status
-------------------------------------------------------------------------------
1 1 I-ES-45_1 DF
===============================================================================
On leaf PE-1, the VXLAN destinations in VXLAN instance 1 are the following:
[/]
A:admin@PE-1# show service id 1 vxlan destinations
===============================================================================
Egress VTEP, VNI
===============================================================================
Instance VTEP Address Egress VNI EvpnStatic Num
Mcast Oper State L2 PBR SupBcasDom MACs
-------------------------------------------------------------------------------
1 192.0.2.2 11 evpn 0
BUM Up No No
1 192.0.2.3 11 evpn 0
BUM Up No No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 2
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-VXLAN Ethernet Segment Dest
===============================================================================
Instance Eth SegId Num. Macs Last Change
-------------------------------------------------------------------------------
1 00:23:23:23:23:23:23:00:00:01 5 09/29/2021 17:54:10
-------------------------------------------------------------------------------
Number of entries: 1
-------------------------------------------------------------------------------
===============================================================================
On DCGW PE-2, the VXLAN destinations in VXLAN instances 1 and 2 are the following:
[/]
A:admin@PE-2# show service id 1 vxlan destinations
===============================================================================
Egress VTEP, VNI
===============================================================================
Instance VTEP Address Egress VNI EvpnStatic Num
Mcast Oper State L2 PBR SupBcasDom MACs
-------------------------------------------------------------------------------
1 192.0.2.1 11 evpn 1
BUM Up No No
2 192.0.2.3 12 evpn 1
BUM Up No No
2 192.0.2.4 12 evpn 1
BUM Up No No
2 192.0.2.5 12 evpn 1
BUM Up No No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 4
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-VXLAN Ethernet Segment Dest
===============================================================================
Instance Eth SegId Num. Macs Last Change
-------------------------------------------------------------------------------
2 00:45:45:45:45:45:45:00:00:01 1 09/29/2021 17:54:35
-------------------------------------------------------------------------------
Number of entries: 1
-------------------------------------------------------------------------------
===============================================================================
ECMP 2 is configured, so aliasing is used. PE-1 can reach the I-ES "I-ES-23_1" in VXLAN instance 1 via PE-2 and PE-3:
[/]
A:admin@PE-1# show service id 1 vxlan esi 00:23:23:23:23:23:23:00:00:01
===============================================================================
BGP EVPN-VXLAN Ethernet Segment Dest
===============================================================================
Instance Eth SegId Num. Macs Last Change
-------------------------------------------------------------------------------
1 00:23:23:23:23:23:23:00:00:01 5 09/29/2021 17:54:10
-------------------------------------------------------------------------------
Number of entries: 1
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-VXLAN Dest TEP Info
===============================================================================
Instance TEP Address Egr VNI Last Change
-------------------------------------------------------------------------------
1 192.0.2.2 11 09/29/2021 17:53:32
1 192.0.2.3 11 09/29/2021 17:54:10
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================
In a similar way, PE-4 can reach the I-ES "I-ES-23_1" via PE-2 and PE-3 in VXLAN instance 2:
[/]
A:admin@PE-4# show service id 1 vxlan esi 00:23:23:23:23:23:23:00:00:01
===============================================================================
BGP EVPN-VXLAN Ethernet Segment Dest
===============================================================================
Instance Eth SegId Num. Macs Last Change
-------------------------------------------------------------------------------
2 00:23:23:23:23:23:23:00:00:01 1 09/29/2021 17:54:21
-------------------------------------------------------------------------------
Number of entries: 1
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-VXLAN Dest TEP Info
===============================================================================
Instance TEP Address Egr VNI Last Change
-------------------------------------------------------------------------------
2 192.0.2.2 12 09/29/2021 17:54:21
2 192.0.2.3 12 09/29/2021 17:54:21
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================
The following command on PE-2 shows the ES information for "I-ES-23_1": DF status, DF candidate list, VXLAN instance service range, and so on:
[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment name "I-ES-23_1" all
===============================================================================
Service Ethernet Segment
===============================================================================
Name : I-ES-23_1
Eth Seg Type : Virtual
Admin State : Enabled Oper State : Up
ESI : 00:23:23:23:23:23:23:00:00:01
Oper ESI : 00:23:23:23:23:23:23:00:00:01
Auto-ESI Type : None
AC DF Capability : Include
Multi-homing : allActive Oper Multi-homing : allActive
ES SHG Label : 524287
Source BMAC LSB : None
VXLAN Instance Id : 1
ES Activation Timer : 3 secs (default)
Oper Group : (Not Specified)
Svc Carving : manual Oper Svc Carving : manual
Cfg Range Type : lowest-pref
-------------------------------------------------------------------------------
DF Pref Election Information
-------------------------------------------------------------------------------
Preference Preference Last Admin Change Oper Pref Do No
Mode Value Value Preempt
-------------------------------------------------------------------------------
revertive 100 09/29/2021 17:53:32 100 Disabled
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
EVI Ranges
-------------------------------------------------------------------------------
From To
-------------------------------------------------------------------------------
1 1
-------------------------------------------------------------------------------
ISID Ranges: <none>
===============================================================================
===============================================================================
EVI Information
===============================================================================
EVI SvcId Actv Timer Rem DF
-------------------------------------------------------------------------------
1 1 0 yes
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI DF Address
-------------------------------------------------------------------------------
1 192.0.2.2
1 192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
===============================================================================
---snip---
===============================================================================
Vxlan Instance Service Ranges
===============================================================================
Svc Range Start Svc Range End Last Changed
-------------------------------------------------------------------------------
1 1 09/29/2021 17:53:32
-------------------------------------------------------------------------------
Number of Entries: 1
===============================================================================
When traffic is sent between CE-1 and CE-6, the EVPN-MAC routes are sent with I-ESI. The FDB for VPLS 1 on PE-1 shows I-ESI 00:23:23:23:23:23:23:00:00:01 of "I-ES-23_1" as source identifier for MAC address 00:ca:fe:ca:fe:06 of CE-6:
[/]
A:admin@PE-1# show service id 1 fdb detail
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:ca:fe:ca:fe:01 sap:1/2/1:1 L/0 09/29/21 18:04:32
1 00:ca:fe:ca:fe:06 eES: Evpn 09/29/21 18:04:32
00:23:23:23:23:23:23:00:00:01
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
On PE-2, the FDB for VPLS 1 shows I-ESI 00:45:45:45:45:45:45:00:00:01 of "I-ES-45_1" as source identifier for MAC address 00:ca:fe:ca:fe:06 of CE-6:
[/]
A:admin@PE-2# show service id 1 fdb detail
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:ca:fe:ca:fe:01 vxlan-1: Evpn 09/29/21 18:04:32
192.0.2.1:11
1 00:ca:fe:ca:fe:06 eES: Evpn 09/29/21 18:04:32
00:45:45:45:45:45:45:00:00:01
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
Single-active multi-homing I-ES when static VXLAN coexists with EVPN-VXLAN in the same VPLS
I-ES with EVPN-VXLAN in DC 1 and static VXLAN in DC2 shows the sample topology for VPLS 2 with static VXLAN in DC 2 and the single-active "I-ES-45_2" on PE-4 and PE-5.
The configuration for VPLS 2 on PE-1, PE-2, and PE-3 is similar to the configuration for VPLS 1, so only the configuration on PE-4, PE-5, and PE-6 is shown.
On PE-6, VPLS 2 is configured with static VXLAN using non-anycast VTEP addresses:
# on PE-6:
configure {
service {
vpls "VPLS 2" {
admin-state enable
service-id 2
customer "1"
vxlan {
instance 1 {
vni 21
egress-vtep 192.0.2.4 { }
egress-vtep 192.0.2.5 { }
}
}
sap 1/2/1:2 {
}
}
To avoid blackholes, the I-ES between DCGWs PE-4 and PE-5 must not be all-active.
On PE-4 and PE-5, the single-active I-ES "I-ES-45_2" is configured as follows:
# on PE-4, PE-5:
configure {
service {
system {
bgp {
evpn {
ethernet-segment "I-ES-45_2" {
admin-state enable
type virtual
esi 00:45:45:45:45:45:45:00:00:02
multi-homing-mode single-active
association {
network-interconnect-vxlan 1 {
virtual-ranges {
service-id 2 {
end 2
}
}
}
}
}
}
}
On PE-4 and PE-5, VPLS 2 is configured as follows:
# on PE-4, PE-5:
configure {
service {
vpls "VPLS 2" {
admin-state enable
service-id 2
customer "1"
vxlan {
instance 1 {
vni 21
egress-vtep 192.0.2.6 { }
}
instance 2 {
vni 22
}
}
bgp 2 {
route-distinguisher auto-rd
route-target {
export "target:64500:22"
import "target:64500:22"
}
}
bgp-evpn {
evi 2
vxlan 2 {
admin-state enable
vxlan-instance 2
default-route-tag 0x16 # default route tag 22
ecmp 2
mh-mode network
routes {
auto-disc {
advertise true
}
}
}
}
sap 1/2/1:2 {
}
}
The policies on all DCGWs must be modified with tag 21 for VXLAN instance 1 in VPLS 2, as follows:
# on PE-2, PE-3:
configure {
policy-options {
policy-statement "export WAN routes only" {
entry 20 {
from {
family [evpn]
tag 21
}
action {
action-type reject
}
}
default-action {
action-type accept
}
}
policy-statement "export DC routes and add SOO" {
entry 20 {
from {
family [evpn]
tag 21
}
action {
action-type accept
community {
add ["SOO-23"]
}
}
}
default-action {
action-type accept
}
}
DCGW PE-5 is NDF for "I-ES-45_2":
[/]
A:admin@PE-5# show service id 2 ethernet-segment
No sap entries
No sdp entries
===============================================================================
VXLAN Ethernet-Segment Information
===============================================================================
VXLAN Instance Eth-Seg Status
-------------------------------------------------------------------------------
1 I-ES-45_2 NDF
===============================================================================
On PE-5, the status of VXLAN instance 1 in VPLS 2 is mhStandby, as follows:
[/]
A:admin@PE-5# show service id 2 vxlan
===============================================================================
VPLS VXLAN
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
Vxlan Instance
===============================================================================
VXLAN Instance VNI AR Oper-flags VTEP
security
-------------------------------------------------------------------------------
1 21 none mhStandby disabled
2 22 none none disabled
-------------------------------------------------------------------------------
Number of Entries : 2
-------------------------------------------------------------------------------
===============================================================================
The VXLAN destinations in VPLS 2 on PE-5 are the following:
[/]
A:admin@PE-5# show service id 2 vxlan destinations
===============================================================================
Egress VTEP, VNI
===============================================================================
Instance VTEP Address Egress VNI EvpnStatic Num
Mcast Oper State L2 PBR SupBcasDom MACs
-------------------------------------------------------------------------------
1 192.0.2.6 21 static 0
- Up No No
2 192.0.2.2 22 evpn 0
BUM Up No No
2 192.0.2.3 22 evpn 0
BUM Up No No
2 192.0.2.4 22 evpn 0
BUM Up No No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 4
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-VXLAN Ethernet Segment Dest
===============================================================================
Instance Eth SegId Num. Macs Last Change
-------------------------------------------------------------------------------
2 00:23:23:23:23:23:23:00:00:02 1 09/29/2021 18:14:49
-------------------------------------------------------------------------------
Number of entries: 1
-------------------------------------------------------------------------------
===============================================================================
An anycast solution without I-ES can also be configured when an EVPN-VXLAN coexists with a static VXLAN.
Conclusion
Service providers can use I-ESs for better bandwidth utilization and redundancy in large DCs. EVPN all-active multi-homing I-ESs can be used in dual EVPN-VXLAN instance VPLS services. However, when a static VXLAN instance coexists with EVPN-VXLAN in the same VPLS, a single-active multi-homing I-ES (or an anycast solution without I-ES) is required to avoid blackholes.