PBB-EVPN ISID-based CMAC Flush
This chapter provides information about PBB-EVPN ISID-based CMAC Flush.
Topics in this chapter include:
Applicability
This chapter was initially written for SR OS Release 15.0.R4, but the MD-CLI in the current edition is based on SR OS Release 21.2.R2. PBB-EVPN ISID-based CMAC flush is supported on the following objects in an I-VPLS:
-
SAPs in a BGP multi-homing site (no Ethernet Segment (ES))-supported in SR OS Release 14.0.R4, and later
-
SAPs in ESs or virtual ESs (vESs)-SR OS Release 15.0.R1, and later
-
Spoke-SDPs (that may be part of an ES/vES or not)-SR OS Release 15.0.R4, and later.
Chapter EVPN for PBB over MPLS (PBB-EVPN) is prerequisite reading.
Overview
CMAC flush when SAP in BGP multi-homing site fails shows an example topology with PBB-EVPN where a CMAC flush is triggered after a SAP in a BGP multi-homing site fails.
I-VPLS 1001 is configured in PE-2 and PE-3 with pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true and connected to MTU-1. In the example, the SAP goes operationally down in I-VPLS 1001 on PE-2. To speed up convergence without flushing CMAC addresses in other I-VPLS services, PE-2 sends a BGP-EVPN BMAC route for ISID 1001 with increased sequence number to trigger a MAC-flush for I-VPLS 1001 on the remote PEs. All CMAC addresses in the FDB for other I-VPLS services, such as I-VPLS 1010 in this example, will be preserved. When PE-4 needs to send traffic to one of the flushed CMAC addresses in I-VPLS 1001, it will flood the frames until the CMAC address is learned again (via PE-3).
When SAPs or SDP-bindings-associated with ESs, vESs, or BGP-MH sites-in an I-VPLS service fail, a BGP-EVPN BMAC route (route type 2) can trigger an ISID-based CMAC flush on the remote PEs. For the CMAC addresses to be flushed from the FDB of the I-VPLS, the existing EVPN BMAC routes will be used with the Ethernet tag equal to the ISID. EVPN BMAC route with ISID indication shows the EVPN BMAC route with ISID indication (BMAC/ISID). A BMAC/ISID update may trigger a selective MAC-flush for a specific I-VPLS, whereas a BMAC/0 update (BMAC/ISID route where ISID=0) may trigger a MAC-flush for all I-VPLS services. This procedure is based on draft-snr-bess-pbb-evpn-isid-cmacflush.
By default, ISID-based CMAC flush is disabled: no I-VPLS will send a B-VPLS EVPN flush message and no B-VPLS will accept any I-VPLS EVPN flush messages. The router only installs CMAC entries corresponding to a zero Ethernet tag and ignores non-zero Ethernet tag MAC routes. However, when the B-VPLS is configured to accept BMAC/ISID routes, non-zero Ethernet tag BMAC routes can be processed for CMAC flush. The CMAC flush trigger will be an EVPN BMAC/ISID route with a sequence number that is higher than before. The receiving PE will then flush all CMACs associated with this BMAC address in the I-VPLS.
The first time that a BMAC/ISID route is received, it is added to the database as a baseline. It does not cause a CMAC flush. Only subsequent BMAC/ISID updates with increased sequence number or withdrawals will cause CMAC flush.
The following command shows that B-VPLS 1000 does not accept any I-VPLS EVPN flush messages. This is the default behavior.
[/]
A:admin@PE-2# show service id 1000 bgp-evpn | match "Accept IVPLS Flush"
Accept IVPLS Flush : Disabled
At the receiving node, B-VPLS 1000 will accept BMAC/ISID routes when the following command is configured:
# on PE-2:
configure {
service {
vpls "B-VPLS 1000" {
bgp-evpn {
accept-ivpls-evpn-flush true
By default, I-VPLS 1001 will not send any B-VPLS EVPN flush messages, as follows:
[/]
A:admin@PE-2# show service id 1001 base | match SendBvplsEvpnFlush
SendBvplsEvpnFlush: Disabled
The following configuration allows I-VPLS 1001 to send B-VPLS EVPN flush messages when a SAP or SDP-binding fails:
# on PE-2:
configure {
service {
vpls "I-VPLS 1001" {
pbb {
i-vpls-mac-flush {
bgp-evpn {
send-to-bvpls true
When enabled, the I-VPLS will send a BMAC/ISID route and subsequent updates with a higher sequence number whenever a SAP fails in the I-VPLS on the node. The default setting for a SAP allows a B-VPLS EVPN flush message to be sent (when enabled in the I-VPLS itself):
[/]
A:admin@PE-2# show service id 1001 sap 1/2/1:1001 detail | match SendBvplsEvpnFlush
SendBvplsEvpnFlush : Enabled
When no alternative route via another node is available for specific SAPs (single-homed SAPs), no CMAC flush should be triggered. When no B-VPLS EVPN flush messages need to be sent from PE-4 when SAP 1/2/1:1001 goes down, the configuration is as follows:
# on PE-4:
configure {
service {
vpls "I-VPLS 1001" {
sap 1/2/1:1001 {
i-vpls-mac-flush {
bgp-evpn {
send-to-bvpls false
The router only installs the BMACs received in MAC routes that have Ethernet tag zero. When CMAC flush is enabled, MAC routes with Ethernet tag equal to the ISID (always non-zero) are for CMAC flush, but not for installing the conveyed BMACs.
BMAC/ISID routes have the following characteristics:
-
BMAC/ISID routes are sent with the static bit flag set as for any other BMAC route. The static bit is ignored at reception because this route is never used to install a BMAC in the FDB.
-
BMAC/ISID routes received with non-zero ESI and non-zero Ethernet tag are treated as withdraw by the router at application level. Route Reflectors (RRs) treat such BMAC/ISID routes as valid routes that can be forwarded.
-
BMAC/ISID routes are shown as valid in the show router bgp routes evpn mac commands, as in the following output, even though they are not used to populate the FDB. This shows that BGP is sending the routes to the application layer for CMAC flush processing. The BMAC/0 route should be sent before the BMAC/ISID routes for the same BMAC. Also, when the B-VPLS goes operationally down, the BMAC/0 should be withdrawn before the BMAC/ISID routes.
[/]
A:admin@PE-2# show router bgp routes evpn mac rd 192.0.2.3:1000
===============================================================================
BGP Router ID:192.0.2.2 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
0 Static LABEL 524282
n/a
192.0.2.3
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.3
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
When pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true is configured in an I-VPLS that is associated with a B-VPLS, BGP-EVPN BMAC/ISID updates will be sent when certain events take place in the I-VPLS or B-VPLS. CMAC flush transmission behavior shows the CMAC flush transmission behavior at the egress PE.
|
Local Event |
pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls |
sap>i-vpls-mac-flush>bgp-evpn>send-to-bvpls |
Action |
|---|---|---|---|
|
Reconfigure I-VPLS: enable or disable send-to-bvpls |
true or false |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
|
Associate/disassociate I-VPLS to/from B-VPLS |
true |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
|
I-VPLS oper-up/oper-down |
true |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
|
B-VPLS oper-up/oper-down |
true |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 Note: All BMACs are also advertised/withdrawn. |
|
B-VPLS bgp-evpn mpls enabled/disabled |
true |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
|
B-VPLS operational source BMAC change |
true |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
|
SAP oper-up |
true |
N/A |
No operation |
|
SAP oper-down
|
true |
true |
Send update source BMAC/ISID Seq=Seq+1 |
|
true |
false |
No operation |
CMAC flush reception behavior shows the reception behavior at the ingress PE. For the CMAC flush triggered by a BMAC/ISID update with increased sequence number, the B-VPLS in the receiving PE must be configured with accept-ivpls-evpn-flush true. BMAC/0 refers to a BMAC route where the Ethernet Tag is 0.
|
Received route |
Action |
|---|---|
|
BMAC/0 withdraw |
Flush all CMACs for that BMAC |
|
BMAC/ISID withdraw |
Flush all CMACs for that BMAC and ISID |
|
BMAC/0 update + Seq change |
Flush all CMACs for that BMAC |
|
BMAC/ISID update + Seq change |
Flush all CMACs for that BMAC and ISID |
|
BMAC/0 update + PE (NHop) change |
No CMAC-flush |
|
BMAC/ISID update + PE (NHop) change |
Flush all CMACs for that BMAC and ISID |
BMAC/ISID updates will trigger CMAC flush procedures regardless of the Termination Endpoint (TEP) or Route Distinguisher (RD) with which the update is received. CMAC flush will be processed even if the BMAC-ISID comes from a TEP or RD different from the BMAC/0 route. Even when the sequence number is the same as in the previous BMAC/ISID update, CMAC flush will happen when the TEP is different. When the same BMAC/ISID is received from two PEs, both are accepted and any change in sequence number causes a MAC flush. However, when the same BMAC/ISID route is received from two PEs with the same RD, BGP will select only one, so the router only sees one.
CMAC flush for ES/vES
RFC 7623 (PBB-EVPN) defines the following CMAC Flush notification mechanisms for single-active multi-homing. These notifications do not include the local ISIDs:
-
When ES-BMACs are used and the ES goes operationally down, the ES-BMAC will be withdrawn.
-
When source-BMACs are used and the ES goes operationally down, a BGP-EVPN BMAC/0 is sent with a higher sequence number.
ISID-independent CMAC flush when ES fails shows the following two scenarios for ISID-independent CMAC flush that are supported in SR OS Release 13.0.R4, and later:
-
PBB frames are sent with the source-BMAC. When the ES goes operationally down, a BMAC update is sent with an increased sequence number, triggering a CMAC flush for all CMAC addresses associated with the BMAC address in I-VPLS, regardless of the ISID.
-
PBB frames are sent with the ES-BMAC address. When the ES goes operationally down, a BMAC withdraw message is sent, triggering the remote PEs to flush all CMAC addresses associated to the ES-BMAC address, regardless of the ISID.
In addition to the preceding ISID-independent CMAC flush mechanisms, ISID-based CMAC flush is also supported in I-VPLS services with SAP or spoke-SDPs that are part of an ES or vES. ISID-based CMAC flush is enabled in the I-VPLS with the pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true command. An I-VPLS that is configured with pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true requires one of the following conditions to be met:
-
The SAP or spoke-SDP has i-vpls-mac-flush>bgp-evpn>send-to-bvpls false configured.
-
The SAP or spoke-SDP has i-vpls-mac-flush>bgp-evpn>send-to-bvpls true configured (default) and one of the following conditions is met:
-
The SAP or spoke-SDP is not on an ES.
-
The SAP or spoke-SDP is on an ES or vES with no src-bmac-lsb configured.
-
The B-VPLS has pbb>source-bmac>use-es-bmac-lsb false configured.
-
For ES SAPs with i-vpls-mac-flush>bgp-evpn>send-to-bvpls true in I-VPLS services that have pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true configured, the ISID-based CMAC flush replaces the RFC 7623-based CMAC flush mechanism.
For each ES/vES and B-VPLS, the system will check whether all I-VPLS services in the ES/B-VPLS have ISID-based MAC-flush enabled.
-
If all I-VPLSs have pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true configured:
-
No BMAC/0 updates with increased sequence number will be triggered when the ES/vES goes operationally down.
-
Only BMAC/ISID updates with increased sequence number will be sent when the I-VPLS attachment circuit goes operationally down.
-
-
If at least one I-VPLS has pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls false configured:
-
BMAC/0 updates with increased sequence number will be triggered when the ES/vES goes operationally down.
-
Also, BMAC/ISID updates with increased sequence number will be generated for those I-VPLS services that have pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true configured.
-
The number of CMAC addresses that may be flushed at the remote nodes can be reduced by enabling ISID-based MAC-flush for all the I-VPLS services in the ES/vES.
When attempting to set use-es-bmac-lsb true in B-VPLS 1000 on PE-4 when the SAP/SDP-binding has default settings (and pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true in the I-VPLS), the following error is raised:
[ex:/configure service vpls "B-VPLS 1000" pbb source-bmac]
A:admin@PE-4# use-es-bmac-lsb true
*[ex:/configure service vpls "B-VPLS 1000" pbb source-bmac]
A:admin@PE-4# commit
MINOR: MGMT_CORE #4001: configure service vpls "I-VPLS 1024" spoke-sdp 46:1024 - ethernet-segment ESI-45 using es-bmac and service has send-bvpls-evpn-flush enabled - configure service vpls "I-VPLS 1024" pbb i-vpls-mac-flush bgp-evpn send-to-bvpls
MINOR: MGMT_CORE #4001: configure service vpls "I-VPLS 1001" spoke-sdp 46:1001 - ethernet-segment ESI-45 using es-bmac and service has send-bvpls-evpn-flush enabled - configure service vpls "I-VPLS 1001" pbb i-vpls-mac-flush bgp-evpn send-to-bvpls
However, when the ES is disabled, the B-VPLS can be configured with use-es-bmac-lsb true. When attempting to re-enable the ES afterward, the following error is raised.
[ex:/configure service system bgp evpn ethernet-segment "ESI-45"]
A:admin@PE-4# admin-state enable
*[ex:/configure service system bgp evpn ethernet-segment "ESI-45"]
A:admin@PE-4# commit
MINOR: MGMT_CORE #4001: configure service vpls "I-VPLS 1024" spoke-sdp 46:1024 - ethernet-segment ESI-45 using es-bmac and service has send-bvpls-evpn-flush enabled - configure service vpls "I-VPLS 1024" pbb i-vpls-mac-flush bgp-evpn send-to-bvpls
MINOR: MGMT_CORE #4001: configure service vpls "I-VPLS 1001" spoke-sdp 46:1001 - ethernet-segment ESI-45 using es-bmac and service has send-bvpls-evpn-flush enabled - configure service vpls "I-VPLS 1001" pbb i-vpls-mac-flush bgp-evpn send-to-bvpls
Configuration
Example topology shows the example topology.
The initial configuration includes the following:
-
Cards, MDAs
-
Ports: the ports between the MTUs and the PEs are hybrid or access ports with dot1q encapsulation; the ports between the PEs are network ports with null encapsulation
-
Router interfaces
-
IS-IS on all router interfaces (alternatively, OSPF could be used)
-
LDP on all router interfaces
The following use cases are described in this section:
-
ISID-based CMAC flush for BGP non-EVPN multi-homing (no ES)
-
ISID-based CMAC flush for BGP-EVPN in a single-active ES
ISID-based CMAC flush for BGP multi-homing
Example topology with BGP multi-homing shows the example topology with BGP multi-homing site 1 between PE-2 and PE-3. B-VPLS 1000 is configured on all the core nodes (PEs) and I-VPLS 1001 and I-VPLS 1010 are associated with this B-VPLS in the PEs. On MTU-1, regular VPLSs are configured. For more information about BGP non-EVPN multi-homing, see chapter BGP Multi-Homing for VPLS Networks.
BGP is configured for address family EVPN on all PEs with PE-2 as RR. For BGP multi-homing, address family L2-VPN is enabled between PE-2 and PE-3. The BGP configuration on PE-2 is as follows:
# on PE-2:
configure {
router "Base"
autonomous-system 64500
bgp {
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
peer-ip-tracking true
split-horizon true
rapid-update {
l2-vpn true
evpn true
}
group "internal" {
peer-as 64500
family {
evpn true
}
cluster {
cluster-id 1.1.1.1
}
}
neighbor "192.0.2.3" {
group "internal"
family {
l2-vpn true
evpn true
}
}
neighbor "192.0.2.4" {
group "internal"
family {
evpn true
}
}
}
The BGP configuration on PE-4 is as follows:
# on PE-4:
configure {
router "Base"
autonomous-system 64500
bgp {
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
peer-ip-tracking true
split-horizon true
rapid-update {
evpn true
}
group "internal" {
peer-as 64500
family {
evpn true
}
}
neighbor "192.0.2.2" {
group "internal"
}
}
The configuration of B-VPLS 1000 and I-VPLS 1001 on PE-2 is as follows. ISID-based CMAC flush is disabled by default. BGP multi-homing site "MH-site-1" is configured on PE-2 with SAP 1/1/2:1001 associated with it, whereas SAP 1/2/1:1001 is not associated to the MH site. CE-21 is attached to I-VPLS 1001 with SAP 1/2/1:1001.
# on PE-2:
configure {
service {
system {
bgp-auto-rd-range {
ip-address 192.0.2.2
community-value {
start 1
end 999
}
}
}
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:02
}
}
bgp 1 {
}
bgp-evpn {
evi 1000
mpls 1 {
admin-state enable
auto-bind-tunnel {
resolution any
}
}
}
}
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
bgp 1 {
route-distinguisher auto-rd
route-target {
export "target:64500:1001"
import "target:64500:1001"
}
}
sap 1/1/2:1001 {
}
sap 1/2/1:1001 {
}
bgp-mh-site "MH-site-1" {
admin-state enable
id 1
sap 1/1/2:1001
}
}
vpls "I-VPLS 1010" {
admin-state enable
service-id 1010
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1010
}
}
bgp 1 {
route-distinguisher auto-rd
route-target {
export "target:64500:1010"
import "target:64500:1010"
}
}
sap 1/1/2:1010 {
}
}
I-VPLS 1010 is configured without multi-homing. The configuration of VPLS 1001 on PE-3 is similar, but without I-VPLS 1010.
ISID-based CMAC flush is not enabled yet. The PEs exchange BGP-EVPN MAC routes with Ethernet tag zero. PE-3 has received BMAC/0 routes from PE-2 and PE-4, as follows:
[/]
A:admin@PE-3# show router bgp routes evpn mac
===============================================================================
BGP Router ID:192.0.2.3 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.2:1000 00:00:00:00:00:02 ESI-0
0 Static LABEL 524282
n/a
192.0.2.2
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
0 Static LABEL 524282
n/a
192.0.2.4
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
PE-2 and PE-4 have also received BMAC/0 routes from the other PEs.
ISID-based CMAC flush is enabled in I-VPLS 1001 on PE-2 and PE-3. PE-4 has no multi-homing in I-VPLS 1001, so it should not send any CMAC flush. I-VPLS 1010 has no multi-homing in any PE, so ISID-based MAC-flush should not be enabled in I-VPLS 1010.
# on PE-2, PE-3:
configure {
service {
vpls "I-VPLS 1001" {
pbb {
i-vpls-mac-flush {
bgp-evpn {
send-to-bvpls true
PE-2 and PE-3 will send BMAC/1001 updates with sequence number 0 to the other two PEs. As an example, the following EVPN-MAC route for BMAC 00:00:00:00:00:03 with tag 1001 is sent by PE-3:
22 2021/04/15 08:07:57.818 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:03, IP len: 0, IP: NULL, label1: 8388512
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
PE-4 has received the following BMAC routes from PE-2 and PE-3, with Ethernet tag zero and Ethernet tag 1001. BMAC routes are always static (received with the sticky bit set).
[/]
A:admin@PE-4# show router bgp routes evpn mac
===============================================================================
BGP Router ID:192.0.2.4 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.2:1000 00:00:00:00:00:02 ESI-0
0 Static LABEL 524282
n/a
192.0.2.2
u*>i 192.0.2.2:1000 00:00:00:00:00:02 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.2
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
0 Static LABEL 524282
n/a
192.0.2.3
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.3
-------------------------------------------------------------------------------
Routes : 4
===============================================================================
When a failure occurs on PE-2, PE-3 and PE-4 should accept the BMAC/ISID with increased sequence number; for a failure on PE-3, PE-2 and PE-4 should accept the BMAC/ISID update. Therefore, the B-VPLS on all PEs should accept the CMAC flush message for ISID 1001, and this is configured as follows:
# on PE-2, PE-3, PE-4:
configure {
service {
vpls "B-VPLS 1000" {
bgp-evpn {
accept-ivpls-evpn-flush true
The FDB for VPLS 1001 on PE-4 includes MAC address 00:00:11:11:11:11 with source-identifier 192.0.2.2:524282, so PE-4 will forward traffic toward that MAC address to PE-2.
[/]
A:admin@PE-4# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 b-mpls: L/420 04/15/21 08:03:47
192.0.2.2:524282
ldp:65537
1001 00:00:41:41:41:41 sap:1/2/1:1001 L/0 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
A failure is simulated on SAP 1/1/2:1001 in multi-homing site 1 on PE-2 as follows:
# on PE-2:
configure {
service {
vpls "I-VPLS 1001" {
sap 1/1/2:1001 {
admin-state disable
SAP 1/1/2:1001 has the default i-vpls-mac-flush>bgp-evpn>send-to-bvpls true and I-VPLS 1001 is configured with pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true, so PE-2 will send BMAC/ISID updates for BMAC 00:00:00:00:00:02, ISID 1001, and sequence number 1 to its BGP peers. The following BGP update is sent by PE-2 to PE-4:
# on PE-2:
64 2021/04/15 08:12:55.058 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.2
Type: EVPN-MAC Len: 33 RD: 192.0.2.2:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:02, IP len: 0, IP: NULL, label1: 8388512
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:1/Static
"
This BMAC/ISID with sequence number 1 triggers a CMAC flush in the FDB for VPLS 1001, so the entry for 00:00:11:11:11:11 will be flushed, along with all other MAC addresses associated with BMAC 00:00:00:00:00:02. The FDB on PE-4 does not contain any entries with source-identifier BMAC 00:00:00:00:00:02, as follows:
[/]
A:admin@PE-4# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:41:41:41:41 sap:1/2/1:1001 L/150 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 1
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
When the MAC address 00:00:11:11:11:11 is learned via PE-3, the FDB is as follows:
[/]
A:admin@PE-4# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 b-mpls: L/0 04/15/21 08:15:16
192.0.2.3:524282
ldp:65538
1001 00:00:41:41:41:41 sap:1/2/1:1001 L/0 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The CMAC flush is only applied for VPLS 1001, so the FDB for VPLS 1010 on PE-4 will keep entries learned from PE-2, as follows:
[/]
A:admin@PE-4# show service id 1010 fdb detail
===============================================================================
Forwarding Database, Service 1010
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1010 00:00:13:13:13:13 b-mpls: L/0 04/15/21 08:03:48
192.0.2.2:524282
ldp:65537
1010 00:00:43:43:43:43 sap:1/2/1:1010 L/0 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
ISID-based CMAC flush in single-active ES
CMAC flush only makes sense for single-active multi-homing. Also, CMAC flush only works for single-active multi-homing; not for all-active multi-homing, because ES-BMAC is required in all-active multi-homing. Example topology with single-active ES shows the example topology with a single-active ES "ESI-45" configured in PE-4 and PE-5.
The multi-homing configuration has been removed from PE-2 and PE-3, so no CMAC flush should be sent by PE-2 or PE-3. VPLS 1001 is configured as follows on PE-2 and PE-3:
# on PE-2, PE-3:
configure {
service {
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
bgp 1 {
route-distinguisher auto-rd
route-target {
export "target:64500:1001"
import "target:64500:1001"
}
}
sap 1/2/1:1001 {
}
sap lag-1:1001 {
}
}
SDPs are configured between PE-4 and MTU-6, and between PE-5 and MTU-6. These SDPs are associated with the single-active ES "ESI-45".
The configuration of B-VPLS 1000 on PE-4 is as follows. The B-VPLS configuration on the other PEs is similar, but with a different source BMAC.
# on PE-4:
configure {
service {
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:04
}
}
bgp 1 {
}
bgp-evpn {
accept-ivpls-evpn-flush true
evi 1000
mpls 1 {
admin-state enable
auto-bind-tunnel {
resolution any
}
}
}
}
The service configuration on PE-4 includes an SDP toward PE-6 and a single-active multi-homing ES, as follows:
# on PE-4:
configure {
service {
system {
bgp {
evpn {
ethernet-segment "ESI-45" {
admin-state enable
esi 01:00:00:00:00:45:00:00:00:01
multi-homing-mode single-active
df-election {
es-activation-timer 3
}
association {
sdp 46 {
}
}
pbb {
source-bmac-lsb 45-04
}
}
}
}
}
sdp 46 {
admin-state enable
delivery-type mpls
ldp true
far-end {
ip-address 192.0.2.6
}
}
The configuration on PE-5 is similar. The configuration of B-VPLS 1000 is similar to the one for PE-2, with only a different BMAC. The configuration of I-VPLS 1001 on PE-4 is as follows:
# on PE-4:
configure {
service {
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
i-vpls-mac-flush {
bgp-evpn {
send-to-bvpls true
}
}
}
bgp 1 {
route-distinguisher auto-rd
route-target {
export "target:64500:1001"
import "target:64500:1001"
}
}
spoke-sdp 46:1001 {
}
sap 1/2/1:1001 {
}
}
ISID-based MAC-flush is enabled in B-VPLS 1000 and I-VPLS 1001 on all PEs.
I-VPLS 1024 is also associated with B-VPLS 1000 and contains one object (SAP or spoke-SDP) in each PE. The configuration of I-VPLS 1024 is identical on PE-2 and PE-3, as follows:
# on PE-2, PE-3:
configure {
service {
vpls "I-VPLS 1024" {
admin-state enable
service-id 1024
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1024
}
}
sap lag-1:1024 {
}
}
The configuration of I-VPLS 1024 on PE-4 has pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls true configured and contains a spoke-SDP instead of a SAP, as follows. The configuration on PE-5 is similar, but with a different SDP.
# on PE-4:
configure {
service {
vpls "I-VPLS 1024" {
admin-state enable
service-id 1024
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1024
}
i-vpls-mac-flush {
bgp-evpn {
send-to-bvpls true
}
}
}
spoke-sdp 46:1024 {
}
}
ISID-based MAC-flush is enabled on PE-4 and PE-5 for both I-VPLS 1001 and I-VPLS 1024, and BMAC/ISID updates are sent for ISID 1001 and ISID 1024, as follows:
[/]
A:admin@PE-3# show router bgp routes evpn mac rd 192.0.2.4:1000
===============================================================================
BGP Router ID:192.0.2.3 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
0 Static LABEL 524282
n/a
192.0.2.4
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.4
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
1024 Static LABEL 524282
n/a
192.0.2.4
-------------------------------------------------------------------------------
Routes : 3
===============================================================================
PE-5 is the DF for VPLS 1001 in the single-active ES "ESI-45", but not for VPLS 1024, as follows:
[/]
A:admin@PE-5# show service id 1001 ethernet-segment
No sap entries
===============================================================================
SDP Ethernet-Segment Information
===============================================================================
SDP Eth-Seg Status
-------------------------------------------------------------------------------
56:1001 ESI-45 DF
===============================================================================
No vxlan instance entries
[/]
A:admin@PE-5# show service id 1024 ethernet-segment
No sap entries
===============================================================================
SDP Ethernet-Segment Information
===============================================================================
SDP Eth-Seg Status
-------------------------------------------------------------------------------
56:1024 ESI-45 NDF
===============================================================================
No vxlan instance entries
The following FDB for VPLS 1001 on PE-5 shows that traffic toward CMAC 00:00:11:11:11:11 (CE-11) in VPLS 1001 will be forwarded to PE-3:
[/]
A:admin@PE-5# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 b-mpls: L/0 04/15/21 08:19:47
192.0.2.3:524282
ldp:65539
1001 00:00:41:41:41:41 b-mpls: L/0 04/15/21 08:19:47
192.0.2.4:524282
ldp:65537
1001 00:00:61:61:61:61 sdp:56:1001 L/0 04/15/21 08:19:42
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following FDB for VPLS 1024 on PE-4 shows that traffic toward CMAC 00:00:14:14:14:14 (CE-14) will be forwarded to PE-2:
[/]
A:admin@PE-4# show service id 1024 fdb detail
===============================================================================
Forwarding Database, Service 1024
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1024 00:00:14:14:14:14 b-mpls: L/0 04/15/21 08:19:48
192.0.2.2:524282
ldp:65537
1024 00:00:64:64:64:64 sdp:46:1024 L/0 04/15/21 08:19:48
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following FDB for VPLS 1001 on PE-3 shows that traffic toward CMAC 00:00:61:61:61:61 (CE-61) will be forwarded to PE-5:
[/]
A:admin@PE-3# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 sap:lag-1:1001 L/0 04/15/21 08:19:47
1001 00:00:41:41:41:41 b-mpls: L/0 04/15/21 08:19:47
192.0.2.4:524282
ldp:65538
1001 00:00:61:61:61:61 b-mpls: L/0 04/15/21 08:19:42
192.0.2.5:524282
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following FDB for VPLS 1024 on PE-2 shows that traffic toward CMAC 00:00:64:64:64:64 (CE-64) will be forwarded to PE-4:
[/]
A:admin@PE-2# show service id 1024 fdb detail
===============================================================================
Forwarding Database, Service 1024
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1024 00:00:14:14:14:14 sap:lag-1:1024 L/0 04/15/21 08:19:48
1024 00:00:64:64:64:64 b-mpls: L/0 04/15/21 08:19:48
192.0.2.4:524282
ldp:65538
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
PE-5 is the DF for VPLS 1001 in "ESI-45". A failure is simulated by disabling the SDP toward PE-5 on MTU-6, as follows:
# on MTU-6:
configure {
service {
sdp 65 {
admin-state disable
PE-5 sends the following BMAC/ISID with increased sequence number for ISID 1001 to the RR PE-2:
50 2021/04/15 08:24:35.567 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388496
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:1/Static
"
When PE-3 receives this BMAC/ISID, all MAC routes with next-hop PE-5 are flushed and the FDB will contain the following MAC entries:
[/]
A:admin@PE-3# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 sap:lag-1:1001 L/0 04/15/21 08:19:47
1001 00:00:41:41:41:41 b-mpls: L/0 04/15/21 08:19:47
192.0.2.4:524282
ldp:65538
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
If MAC address 00:00:61:61:61:61 is learned again, the next hop will be PE-4 instead of PE-5.
The configuration is restored as follows:
# on MTU-6:
configure {
service {
sdp 65 {
admin-state enable
No CMAC/ISID update will be sent when the last SAP/SDP-binding in a service goes operationally down. VPLS 1024 only has one SAP/SDP-binding in DF PE-4: spoke-SDP 46:1024. A failure of the spoke-SDP is simulated as follows:
# on MTU-6:
configure {
service {
sdp 64 {
admin-state disable
When the last SAP/SDP-binding is down, the service will be operationally down, as follows:
[/]
A:admin@PE-4# show service id 1024 base | match "Oper State"
Admin State : Up Oper State : Down
PE-4 sends the following withdrawal message instead of a CMAC/ISID:
56 2021/04/15 08:26:10.691 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 61
Flag: 0x90 Type: 15 Len: 57 Multiprotocol Unreachable NLRI:
Address Family EVPN
Type: EVPN-INCL-MCAST Len: 17 RD: 192.0.2.4:1000, tag: 1024,
orig_addr len: 32, orig_addr: 192.0.2.4
Type: EVPN-MAC Len: 33 RD: 192.0.2.4:1000 ESI: ESI-0, tag: 1024, mac len: 48
mac: 00:00:00:00:00:04, IP len: 0, IP: NULL, label1: 0
"
The configuration is restored as follows:
# on MTU-6:
configure {
service {
sdp 64 {
admin-state enable
ISID-based and regular CMAC flush in ES
When ISID-based CMAC flush is not enabled in all I-VPLS services using the ES, a failure in the ES will trigger BMAC/0 updates and BMAC/ISID updates with increased sequence number. An additional I-VPLS is configured on the nodes with pbb>i-vpls-mac-flush>bgp-evpn>send-to-bvpls false (default). The configuration of I-VPLS 1021 on PE-5 is as follows:
# on PE-5:
configure {
service {
vpls "I-VPLS 1021" {
admin-state enable
service-id 1021
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1021
}
}
spoke-sdp 56:1021 {
}
sap 1/2/1:1021 {
}
}
The configuration on PE-4 is similar; PE-2 and PE-3 have SAP lag-1:1021 instead of the spoke-SDP.
On MTU-6, SDP 65 is disabled, which will cause an ES failure on PE-5:
# on MTU-6:
configure {
service {
sdp 65 {
admin-state disable
The following BMAC updates are sent by PE-5:
-
BMAC/0 with increased sequence number, which will trigger a CMAC flush for all entries received from PE-5 for all I-VPLS services (ISID-independent)
-
BMAC/ISID with increased sequence number, which will trigger a CMAC flush for all entries received from PE-5 for VPLS 1001
73 2021/04/15 08:32:57.204 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388496
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:1/Static
"
74 2021/04/15 08:32:57.204 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388496
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:3/Static
"
Conclusion
ISID-based MAC-flush speeds up convergence after a SAP or spoke-SDP failure, triggering a selective CMAC flush on the receiving nodes, which flushes all CMAC entries associated with that ISID and BMAC. The feature can be enabled per I-VPLS and disabled for those SAPs or spoke-SDPs for which no alternative route is available, or for those SAPs that are contained in an all-active Ethernet Segment. The BMAC/ISID update always contains the source-BMAC, not the ES-BMAC. CMAC flush based on ES-BMAC is not performed per ISID.