PBB-EVPN ISID-based CMAC Flush
This chapter provides information about PBB-EVPN ISID-based CMAC Flush.
Topics in this chapter include:
Applicability
This chapter was initially written for SR OS Release 15.0.R4, but the CLI in the current edition is based on SR OS Release 21.2.R2. PBB-EVPN ISID-based CMAC flush is supported on the following objects in an I-VPLS:
SAPs in a BGP multi-homing site (no Ethernet Segment (ES))-supported in SR OS Release 14.0.R4, and later
SAPs in ESs or virtual ESs (vESs)-SR OS Release 15.0.R1, and later
Spoke-SDPs (that may be part of an ES/vES or not)-SR OS Release 15.0.R4, and later.
Chapter EVPN for PBB over MPLS (PBB-EVPN) is prerequisite reading.
Overview
CMAC flush when SAP in BGP multi-homing site fails shows an example topology with PBB-EVPN where a CMAC flush is triggered after a SAP in a BGP multi-homing site fails.
I-VPLS 1001 is configured in PE-2 and PE-3 with send-bvpls-evpn-flush and connected to MTU-1. In the example, the SAP goes operationally down in I-VPLS 1001 on PE-2. To speed up convergence without flushing CMAC addresses in other I-VPLS services, PE-2 sends a BGP-EVPN BMAC route for ISID 1001 with increased sequence number to trigger a MAC-flush for I-VPLS 1001 on the remote PEs. All CMAC addresses in the FDB for other I-VPLS services, such as I-VPLS 1010 in this example, will be preserved. When PE-4 needs to send traffic to one of the flushed CMAC addresses in I-VPLS 1001, it will flood the frames until the CMAC address is learned again (via PE-3).
When SAPs or SDP-bindings-associated with ESs, vESs, or BGP-MH sites-in an I-VPLS service fail, a BGP-EVPN BMAC route (route type 2) can trigger an ISID-based CMAC flush on the remote PEs. For the CMAC addresses to be flushed from the FDB of the I-VPLS, the existing EVPN BMAC routes will be used with the Ethernet tag equal to the ISID. EVPN BMAC route with ISID indication shows the EVPN BMAC route with ISID indication (BMAC/ISID). A BMAC/ISID update may trigger a selective MAC-flush for a specific I-VPLS, whereas a BMAC/0 update (BMAC/ISID route where ISID=0) may trigger a MAC-flush for all I-VPLS services. This procedure is based on draft-snr-bess-pbb-evpn-isid-cmacflush.
By default, ISID-based CMAC flush is disabled: no I-VPLS will send a B-VPLS EVPN flush message and no B-VPLS will accept any I-VPLS EVPN flush messages. The router only installs CMAC entries corresponding to a zero Ethernet tag and ignores non-zero Ethernet tag MAC routes. However, when the B-VPLS is configured to accept BMAC/ISID routes, non-zero Ethernet tag BMAC routes can be processed for CMAC flush. The CMAC flush trigger will be an EVPN BMAC/ISID route with a sequence number that is higher than before. The receiving PE will then flush all CMACs associated with this BMAC address in the I-VPLS.
The first time that a BMAC/ISID route is received, it is added to the database as a baseline. It does not cause a CMAC flush. Only subsequent BMAC/ISID updates with increased sequence number or withdrawals will cause CMAC flush.
The following command shows that B-VPLS 1000 does not accept any I-VPLS EVPN flush messages. This is the default behavior.
*A:PE-2# show service id 1000 bgp-evpn | match "Accept IVPLS Flush"
Accept IVPLS Flush : Disabled
At the receiving node, B-VPLS 1000 will accept BMAC/ISID routes when the following command is configured:
# on PE-2:
configure
service
vpls "B-VPLS 1000"
bgp-evpn
accept-ivpls-evpn-flush
By default, I-VPLS 1001 will not send any B-VPLS EVPN flush messages, as follows:
*A:PE-2# show service id 1001 base | match SendBvplsEvpnFlush
SendBvplsEvpnFlush : Disabled
The following configuration allows I-VPLS 1001 to send B-VPLS EVPN flush messages when a SAP or SDP-binding fails:
# on PE-2:
configure
service
vpls "I-VPLS 1001"
pbb
send-bvpls-evpn-flush
When enabled, the I-VPLS will send a BMAC/ISID route and subsequent updates with a higher sequence number whenever a SAP fails in the I-VPLS on the node. The default setting for a SAP allows a B-VPLS EVPN flush message to be sent (when enabled in the I-VPLS itself):
*A:PE-2# show service id 1001 sap 1/2/1:1001 detail | match SendBvplsEvpnFlush
SendBvplsEvpnFlush : Enabled
When no alternative route via another node is available for specific SAPs (single-homed SAPs), no CMAC flush should be triggered. When no B-VPLS EVPN flush messages need to be sent from PE-4 when SAP 1/2/1:1001 goes down, the configuration is as follows:
# on PE-4:
configure
service
vpls "I-VPLS 1001"
sap 1/2/1:1001
disable-send-bvpls-evpn-flush
The router only installs the BMACs received in MAC routes that have Ethernet tag zero. When CMAC flush is enabled, MAC routes with Ethernet tag equal to the ISID (always non-zero) are for CMAC flush, but not for installing the conveyed BMACs.
BMAC/ISID routes have the following characteristics:
BMAC/ISID routes are sent with the static bit flag set as for any other BMAC route. The static bit is ignored at reception because this route is never used to install a BMAC in the FDB.
BMAC/ISID routes received with non-zero ESI and non-zero Ethernet tag are treated as withdraw by the router at application level. Route Reflectors (RRs) treat such BMAC/ISID routes as valid routes that can be forwarded.
BMAC/ISID routes are shown as valid in the show router bgp routes evpn mac commands, as in the following output, even though they are not used to populate the FDB. This shows that BGP is sending the routes to the application layer for CMAC flush processing. The BMAC/0 route should be sent before the BMAC/ISID routes for the same BMAC. Also, when the B-VPLS goes operationally down, the BMAC/0 should be withdrawn before the BMAC/ISID routes.
*A:PE-2# show router bgp routes evpn mac rd 192.0.2.3:1000
===============================================================================
BGP Router ID:192.0.2.2 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
0 Static LABEL 524282
n/a
192.0.2.3
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.3
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
When send-bvpls-evpn-flush is enabled in an I-VPLS that is associated with a B-VPLS, BGP-EVPN BMAC/ISID updates will be sent when certain events take place in the I-VPLS or B-VPLS. CMAC flush transmission behavior shows the CMAC flush transmission behavior at the egress PE.
Local Event |
Send-bvpls-evpn-flush |
SAP disable-bvpls-evpn-flush |
Action |
|---|---|---|---|
Reconfigure I-VPLS: enable or disable send-bvpls-evpn-flush |
Enable or disable |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
Associate/disassociate I-VPLS to/from B-VPLS |
Enabled |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
I-VPLS oper-up/oper-down |
Enabled |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
B-VPLS oper-up/oper-down |
Enabled |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 Note: All BMACs are also advertised/withdrawn. |
B-VPLS bgp-evpn mpls no shut/shut |
Enabled |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
B-VPLS operational source BMAC change |
Enabled |
N/A |
Send update/withdraw source BMAC/ISID with Seq=0 |
SAP oper-up |
Enabled |
N/A |
No operation |
SAP oper-down |
Enabled |
No disable |
Send update source BMAC/ISID Seq=Seq+1 |
Enabled |
Disable |
No operation |
CMAC flush reception behavior shows the reception behavior at the ingress PE. For the CMAC flush triggered by a BMAC/ISID update with increased sequence number, the B-VPLS in the receiving PE must be configured with accept-ivpls-evpn-flush. BMAC/0 refers to a BMAC route where the Ethernet Tag is 0.
Received Route |
Action |
|---|---|
BMAC/0 withdraw |
Flush all CMACs for that BMAC |
BMAC/ISID withdraw |
Flush all CMACs for that BMAC and ISID |
BMAC/0 update + Seq change |
Flush all CMACs for that BMAC |
BMAC/ISID update + Seq change |
Flush all CMACs for that BMAC and ISID |
BMAC/0 update + PE (NHop) change |
No CMAC-flush |
BMAC/ISID update + PE (NHop) change |
Flush all CMACs for that BMAC and ISID |
BMAC/ISID updates will trigger CMAC flush procedures regardless of the Termination Endpoint (TEP) or Route Distinguisher (RD) with which the update is received. CMAC flush will be processed even if the BMAC-ISID comes from a TEP or RD different from the BMAC/0 route. Even when the sequence number is the same as in the previous BMAC/ISID update, CMAC flush will happen when the TEP is different. When the same BMAC/ISID is received from two PEs, both are accepted and any change in sequence number causes a MAC flush. However, when the same BMAC/ISID route is received from two PEs with the same RD, BGP will select only one, so the router only sees one.
CMAC flush for ES/vES
RFC 7623 (PBB-EVPN) defines the following CMAC Flush notification mechanisms for single-active multi-homing. These notifications do not include the local ISIDs:
When ES-BMACs are used and the ES goes operationally down, the ES-BMAC will be withdrawn.
When source-BMACs are used and the ES goes operationally down, a BGP-EVPN BMAC/0 is sent with a higher sequence number.
ISID-independent CMAC flush when ES fails shows the following two scenarios for ISID-independent CMAC flush that are supported in SR OS Release 13.0.R4, and later:
PBB frames are sent with the source-BMAC. When the ES goes operationally down, a BMAC update is sent with an increased sequence number, triggering a CMAC flush for all CMAC addresses associated with the BMAC address in I-VPLS, regardless of the ISID.
PBB frames are sent with the ES-BMAC address. When the ES goes operationally down, a BMAC withdraw message is sent, triggering the remote PEs to flush all CMAC addresses associated to the ES-BMAC address, regardless of the ISID.
In addition to the preceding ISID-independent CMAC flush mechanisms, ISID-based CMAC flush is also supported in I-VPLS services with SAP or spoke-SDPs that are part of an ES or vES. ISID-based CMAC flush is enabled in the I-VPLS with the send-bvpls-evpn-flush command. An I-VPLS that is configured with send-bvpls-evpn-flush requires one of the following conditions to be met:
The SAP or spoke-SDP has disable-send-bvpls-evpn-flush configured.
The SAP or spoke-SDP has no disable-send-bvpls-evpn-flush configured (default) and one of the following conditions is met:
The SAP or spoke-SDP is not on an ES.
The SAP or spoke-SDP is on an ES or vES with no src-bmac-lsb configured.
The B-VPLS has no use-es-bmac configured.
For ES SAPs with no disable-send-bvpls-evpn-flush in I-VPLS services that have send-bvpls-evpn-flush configured, the ISID-based CMAC flush replaces the RFC 7623-based CMAC flush mechanism.
For each ES/vES and B-VPLS, the system will check whether all I-VPLS services in the ES/B-VPLS have ISID-based MAC-flush enabled.
If all I-VPLSs have send-bvpls-evpn-flush enabled:
No BMAC/0 updates with increased sequence number will be triggered when the ES/vES goes operationally down.
Only BMAC/ISID updates with increased sequence number will be sent when the I-VPLS attachment circuit goes operationally down.
If at least one I-VPLS has no send-bvpls-evpn-flush enabled:
BMAC/0 updates with increased sequence number will be triggered when the ES/vES goes operationally down.
Also, BMAC/ISID updates with increased sequence number will be generated for those I-VPLS services that have send-bvpls-evpn-flush enabled.
The number of CMAC addresses that may be flushed at the remote nodes can be reduced by enabling ISID-based MAC-flush for all the I-VPLS services in the ES/vES.
When attempting to set use-es-bmac in B-VPLS 1000 on PE-4 when the SAP/SDP-binding has default settings (and send-bvpls-evpn-flush is enabled in the I-VPLS), the following error is raised:
*A:PE-4>config>service>vpls>pbb# use-es-bmac
MINOR: SVCMGR #1433 Cannot set use-es-bmac - spoke 46:1001 on ethernet-segment ESI-45 has "no disable-send-bvpls-evpn-flush"
When the ES is disabled, the B-VPLS can be configured with use-es-bmac. When attempting to enable the ES afterward, the following error is raised.
*A:PE-4# configure service system bgp-evpn ethernet-segment "ESI-45" shutdown
*A:PE-4# configure service vpls "B-VPLS 1000" pbb use-es-bmac
*A:PE-4# configure service system bgp-evpn ethernet-segment "ESI-45" no shutdown
MINOR: SVCMGR #8057 Ethernet segment cannot change admin state -
spoke 46:1001 has "no disable-send-bvpls-evpn-flush"
Configuration
Example topology shows the example topology.
The initial configuration includes the following:
Cards, MDAs
Ports: the ports between the MTUs and the PEs are hybrid or access ports with dot1q encapsulation; the ports between the PEs are network ports with null encapsulation
Router interfaces
IS-IS on all router interfaces (alternatively, OSPF could be used)
LDP on all router interfaces
The following use cases are described in this section:
ISID-based CMAC flush for BGP non-EVPN multi-homing (no ES)
ISID-based CMAC flush for BGP-EVPN in a single-active ES
ISID-based CMAC flush for BGP multi-homing
Example topology with BGP multi-homing shows the example topology with BGP multi-homing site 1 between PE-2 and PE-3. B-VPLS 1000 is configured on all the core nodes (PEs) and I-VPLS 1001 and I-VPLS 1010 are associated with this B-VPLS in the PEs. On MTU-1, regular VPLSs are configured. For more information about BGP non-EVPN multi-homing, see chapter BGP Multi-Homing for VPLS Networks.
BGP is configured for address family EVPN on all PEs with PE-2 as RR. For BGP multi-homing, address family L2-VPN is enabled between PE-2 and PE-3. The BGP configuration on PE-2 is as follows:
# on PE-2:
configure
router Base
autonomous-system 64500
bgp
vpn-apply-import
vpn-apply-export
enable-peer-tracking
rapid-withdrawal
split-horizon
rapid-update l2-vpn evpn
group "internal"
cluster 1.1.1.1
peer-as 64500
neighbor 192.0.2.3
family l2-vpn evpn
exit
neighbor 192.0.2.4
family evpn
exit
exit
exit
The BGP configuration on PE-4 is as follows:
# on PE-4:
configure
router Base
autonomous-system 64500
bgp
vpn-apply-import
vpn-apply-export
enable-peer-tracking
rapid-withdrawal
split-horizon
rapid-update evpn
group "internal"
family evpn
peer-as 64500
neighbor 192.0.2.2
exit
exit
exit
The configuration of B-VPLS 1000 and I-VPLS 1001 on PE-2 is as follows. ISID-based CMAC flush is disabled by default. BGP multi-homing site "site 1" is configured on PE-2 with SAP 1/1/2:1001 associated with it, whereas SAP 1/2/1:1001 is not associated to the MH site. CE-21 is attached to I-VPLS 1001 with SAP 1/2/1:1001.
# on PE-2:
configure
service
system
bgp-auto-rd-range 192.0.2.2 comm-val 1 to 999
exit
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:02
exit
bgp
exit
bgp-evpn
evi 1000
mpls bgp 1
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
no shutdown
exit
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
bgp
route-distinguisher auto-rd
route-target export target:64500:1001 import target:64500:1001
exit
stp
shutdown
exit
site "MH-site-1" create
site-id 1
1/1/2:1001
no shutdown
exit
sap 1/1/2:1001 create
no shutdown
exit
sap 1/2/1:1001 create
no shutdown
exit
no shutdown
exit
vpls 1010 name "I-VPLS 1010" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
bgp
route-distinguisher auto-rd
route-target export target:64500:1010 import target:64500:1010
exit
stp
shutdown
exit
sap 1/1/2:1010 create
no shutdown
exit
no shutdown
exit
I-VPLS 1010 is configured without multi-homing. The configuration of VPLS 1001 on PE-3 is similar, but without I-VPLS 1010.
ISID-based CMAC flush is not enabled yet. The PEs exchange BGP-EVPN MAC routes with Ethernet tag zero. PE-3 has received BMAC/0 routes from PE-2 and PE-4, as follows:
*A:PE-3# show router bgp routes evpn mac
===============================================================================
BGP Router ID:192.0.2.3 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.2:1000 00:00:00:00:00:02 ESI-0
0 Static LABEL 524282
n/a
192.0.2.2
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
0 Static LABEL 524282
n/a
192.0.2.4
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
PE-2 and PE-4 have also received BMAC/0 routes from the other PEs.
ISID-based CMAC flush is enabled in I-VPLS 1001 on PE-2 and PE-3. PE-4 has no multi-homing in I-VPLS 1001, so it should not send any CMAC flush. I-VPLS 1010 has no multi-homing in any PE, so ISID-based MAC-flush should not be enabled in I-VPLS 1010.
# on PE-2, PE-3:
configure
service
vpls "I-VPLS 1001"
pbb
send-bvpls-evpn-flush
PE-2 and PE-3 will send BMAC/1001 updates with sequence number 0 to the other two PEs. As an example, the following EVPN-MAC route for BMAC 00:00:00:00:00:03 with tag 1001 is sent by PE-3:
22 2021/04/15 08:07:57.818 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:03, IP len: 0, IP: NULL, label1: 8388512
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
PE-4 has received the following BMAC routes from PE-2 and PE-3, with Ethernet tag zero and Ethernet tag 1001. BMAC routes are always static (received with the sticky bit set).
*A:PE-4# show router bgp routes evpn mac
===============================================================================
BGP Router ID:192.0.2.4 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.2:1000 00:00:00:00:00:02 ESI-0
0 Static LABEL 524282
n/a
192.0.2.2
u*>i 192.0.2.2:1000 00:00:00:00:00:02 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.2
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
0 Static LABEL 524282
n/a
192.0.2.3
u*>i 192.0.2.3:1000 00:00:00:00:00:03 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.3
-------------------------------------------------------------------------------
Routes : 4
===============================================================================
When a failure occurs on PE-2, PE-3, and PE-4 should accept the BMAC/ISID with increased sequence number; for a failure on PE-3, PE-2, and PE-4 should accept the BMAC/ISID update. Therefore, the B-VPLS on all PEs should accept the CMAC flush message for ISID 1001, and this is configured as follows:
# on PE-2, PE-3, PE-4, PE-5:
configure
service
vpls "B-VPLS 1000"
bgp-evpn
accept-ivpls-evpn-flush
The FDB for VPLS 1001 on PE-4 includes MAC address 00:00:11:11:11:11 with source-identifier 192.0.2.2:524282, so PE-4 will forward traffic toward that MAC address to PE-2.
*A:PE-4# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 b-mpls: L/420 04/15/21 08:03:47
192.0.2.2:524282
ldp:65537
1001 00:00:41:41:41:41 sap:1/2/1:1001 L/0 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
A failure is simulated on SAP 1/1/2:1001 in multi-homing site 1 on PE-2 as follows:
# on PE-2:
configure
service
vpls "I-VPLS 1001"
sap 1/1/2:1001
shutdown
SAP 1/1/2:1001 has the default no disable-send-bvpls-evpn-flush and I-VPLS 1001 is configured with send-bvpls-evpn-flush, so PE-2 will send BMAC/ISID updates for BMAC 00:00:00:00:00:02, ISID 1001, and sequence number 1 to its BGP peers. The following BGP update is sent by PE-2 to PE-4:
# on PE-2:
64 2021/04/15 08:12:55.058 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.2
Type: EVPN-MAC Len: 33 RD: 192.0.2.2:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:02, IP len: 0, IP: NULL, label1: 8388512
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:1/Static
"
This BMAC/ISID with sequence number 1 triggers a CMAC flush in the FDB for VPLS 1001, so the entry for 00:00:11:11:11:11 will be flushed, along with all other MAC addresses associated with BMAC 00:00:00:00:00:02. The FDB on PE-4 does not contain any entries with source-identifier BMAC 00:00:00:00:00:02, as follows:
*A:PE-4# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:41:41:41:41 sap:1/2/1:1001 L/150 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 1
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
When the MAC address 00:00:11:11:11:11 is learned via PE-3, the FDB is as follows:
*A:PE-4# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 b-mpls: L/0 04/15/21 08:15:16
192.0.2.3:524282
ldp:65538
1001 00:00:41:41:41:41 sap:1/2/1:1001 L/0 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The CMAC flush is only applied for VPLS 1001, so the FDB for VPLS 1010 on PE-4 will keep entries learned from PE-2, as follows:
*A:PE-4# show service id 1010 fdb detail
===============================================================================
Forwarding Database, Service 1010
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1010 00:00:13:13:13:13 b-mpls: L/0 04/15/21 08:03:48
192.0.2.2:524282
ldp:65537
1010 00:00:43:43:43:43 sap:1/2/1:1010 L/0 04/15/21 08:11:36
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
ISID-based CMAC flush in single-active ES
CMAC flush only makes sense for single-active multi-homing. Also, CMAC flush only works for single-active multi-homing; not for all-active multi-homing, because ES-BMAC is required in all-active multi-homing. Example topology with single-active ES shows the example topology with a single-active ES "ESI-45" configured in PE-4 and PE-5.
The multi-homing configuration has been removed from PE-2 and PE-3, so no CMAC flush should be sent by PE-2 or PE-3. VPLS 1001 is configured as follows on PE-2 and PE-3:
# on PE-2, PE-3:
configure
service
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
bgp
route-distinguisher auto-rd
route-target export target:64500:1001 import target:64500:1001
exit
stp
shutdown
exit
sap 1/2/1:1001 create
no shutdown
exit
sap lag-1:1001 create
no shutdown
exit
no shutdown
exit
SDPs are configured between PE-4 and MTU-6, and between PE-5 and MTU-6. These SDPs are associated with the single-active ES "ESI-45".
The configuration of B-VPLS 1000 on PE-4 is as follows. The B-VPLS configuration on the other PEs is similar, but with a different source BMAC.
# on PE-4:
configure
service
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:04
exit
bgp
exit
bgp-evpn
accept-ivpls-evpn-flush
evi 1000
mpls bgp 1
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
no shutdown
The service configuration on PE-4 includes an SDP toward PE-6 and a single-active multi-homing ES, as follows:
# on PE-4:
configure
service
sdp 46 mpls create
far-end 192.0.2.6
ldp
keep-alive
shutdown
exit
no shutdown
exit
system
bgp-evpn
ethernet-segment "ESI-45" create
esi 01:00:00:00:00:45:00:00:00:01
source-bmac-lsb 45-04 es-bmac-table-size 8
es-activation-timer 3
service-carving
mode auto
exit
multi-homing single-active
sdp 46
no shutdown
exit
exit
exit
The configuration on PE-5 is similar. The configuration of B-VPLS 1000 is similar to the one for PE-2, with only a different BMAC. The configuration of I-VPLS 1001 on PE-4 is as follows:
# on PE-4:
configure
service
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
send-bvpls-evpn-flush
exit
bgp
route-distinguisher auto-rd
route-target export target:64500:1001 import target:64500:1001
exit
stp
shutdown
exit
sap 1/2/1:1001 create
no shutdown
exit
spoke-sdp 46:1001 create
no shutdown
exit
no shutdown
exit
ISID-based MAC-flush is enabled in B-VPLS 1000 and I-VPLS 1001 on all PEs.
I-VPLS 1024 is also associated with B-VPLS 1000 and contains one object (SAP or spoke-SDP) in each PE. The configuration of I-VPLS 1024 is identical on PE-2 and PE-3, as follows:
# on PE-2, PE-3:
configure
service
vpls 1024 name "I-VPLS 1024" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
shutdown
exit
sap lag-1:1024 create
no shutdown
exit
no shutdown
exit
The configuration of I-VPLS 1024 on PE-4 has send-bvpls-evpn-flush enabled and contains a spoke-SDP instead of a SAP, as follows. The configuration on PE-5 is similar, but with a different SDP.
# on PE-4:
configure
service
vpls 1024 name "I-VPLS 1024" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
send-bvpls-evpn-flush
exit
stp
shutdown
exit
spoke-sdp 46:1024 create
no shutdown
exit
no shutdown
exit
ISID-based MAC-flush is enabled on PE-4 and PE-5 for both I-VPLS 1001 and I-VPLS 1024, and BMAC/ISID updates are sent for ISID 1001 and ISID 1024, as follows:
*A:PE-3# show router bgp routes evpn mac rd 192.0.2.4:1000
===============================================================================
BGP Router ID:192.0.2.3 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
0 Static LABEL 524282
n/a
192.0.2.4
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
1001 Static LABEL 524282
n/a
192.0.2.4
u*>i 192.0.2.4:1000 00:00:00:00:00:04 ESI-0
1024 Static LABEL 524282
n/a
192.0.2.4
-------------------------------------------------------------------------------
Routes : 3
===============================================================================
PE-5 is the DF for VPLS 1001 in the single-active ES "ESI-45", but not for VPLS 1024, as follows:
*A:PE-5# show service id 1001 ethernet-segment
No sap entries
===============================================================================
SDP Ethernet-Segment Information
===============================================================================
SDP Eth-Seg Status
-------------------------------------------------------------------------------
56:1001 ESI-45 DF
===============================================================================
No vxlan instance entries
*A:PE-5# show service id 1024 ethernet-segment
No sap entries
===============================================================================
SDP Ethernet-Segment Information
===============================================================================
SDP Eth-Seg Status
-------------------------------------------------------------------------------
56:1024 ESI-45 NDF
===============================================================================
No vxlan instance entries
The following FDB for VPLS 1001 on PE-5 shows that traffic toward CMAC 00:00:11:11:11:11 (CE-11) in VPLS 1001 will be forwarded to PE-3:
*A:PE-5# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 b-mpls: L/0 04/15/21 08:19:47
192.0.2.3:524282
ldp:65539
1001 00:00:41:41:41:41 b-mpls: L/0 04/15/21 08:19:47
192.0.2.4:524282
ldp:65537
1001 00:00:61:61:61:61 sdp:56:1001 L/0 04/15/21 08:19:42
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following FDB for VPLS 1024 on PE-4 shows that traffic toward CMAC 00:00:14:14:14:14 (CE-14) will be forwarded to PE-2:
*A:PE-4# show service id 1024 fdb detail
===============================================================================
Forwarding Database, Service 1024
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1024 00:00:14:14:14:14 b-mpls: L/0 04/15/21 08:19:48
192.0.2.2:524282
ldp:65537
1024 00:00:64:64:64:64 sdp:46:1024 L/0 04/15/21 08:19:48
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following FDB for VPLS 1001 on PE-3 shows that traffic toward CMAC 00:00:61:61:61:61 (CE-61) will be forwarded to PE-5:
*A:PE-3# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 sap:lag-1:1001 L/0 04/15/21 08:19:47
1001 00:00:41:41:41:41 b-mpls: L/0 04/15/21 08:19:47
192.0.2.4:524282
ldp:65538
1001 00:00:61:61:61:61 b-mpls: L/0 04/15/21 08:19:42
192.0.2.5:524282
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following FDB for VPLS 1024 on PE-2 shows that traffic toward CMAC 00:00:64:64:64:64 (CE-64) will be forwarded to PE-4:
*A:PE-2# show service id 1024 fdb detail
===============================================================================
Forwarding Database, Service 1024
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1024 00:00:14:14:14:14 sap:lag-1:1024 L/0 04/15/21 08:19:48
1024 00:00:64:64:64:64 b-mpls: L/0 04/15/21 08:19:48
192.0.2.4:524282
ldp:65538
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
PE-5 is the DF for VPLS 1001 in "ESI-45". A failure is simulated by disabling the SDP toward PE-5 on MTU-6, as follows:
# on MTU-6:
configure
service
sdp 65
shutdown
PE-5 sends the following BMAC/ISID with increased sequence number for ISID 1001 to the RR PE-2:
50 2021/04/15 08:24:35.567 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388496
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:1/Static
"
When PE-3 receives this BMAC/ISID, all MAC routes with next-hop PE-5 are flushed and the FDB will contain the following MAC entries:
*A:PE-3# show service id 1001 fdb detail
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:11:11:11:11 sap:lag-1:1001 L/0 04/15/21 08:19:47
1001 00:00:41:41:41:41 b-mpls: L/0 04/15/21 08:19:47
192.0.2.4:524282
ldp:65538
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
If MAC address 00:00:61:61:61:61 is learned again, the next hop will be PE-4 instead of PE-5.
The configuration is restored as follows:
# on MTU-6:
configure
service
sdp 65
no shutdown
No CMAC/ISID update will be sent when the last SAP/SDP-binding in a service goes operationally down. VPLS 1024 only has one SAP/SDP-binding in DF PE-4: spoke-SDP 46:1024. A failure of the spoke-SDP is simulated as follows:
# on MTU-6:
configure
service
sdp 64
shutdown
When the last SAP/SDP-binding is down, the service will be operationally down, as follows:
*A:PE-4# show service id 1024 base | match "Oper State"
Admin State : Up Oper State : Down
PE-4 sends the following withdrawal message instead of a CMAC/ISID:
56 2021/04/15 08:26:10.691 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 61
Flag: 0x90 Type: 15 Len: 57 Multiprotocol Unreachable NLRI:
Address Family EVPN
Type: EVPN-INCL-MCAST Len: 17 RD: 192.0.2.4:1000, tag: 1024,
orig_addr len: 32, orig_addr: 192.0.2.4
Type: EVPN-MAC Len: 33 RD: 192.0.2.4:1000 ESI: ESI-0, tag: 1024, mac len: 48
mac: 00:00:00:00:00:04, IP len: 0, IP: NULL, label1: 0
"
The configuration is restored as follows:
# on MTU-6:
configure
service
sdp 64
no shutdown
ISID-based and regular CMAC flush in ES
When ISID-based CMAC flush is not enabled in all I-VPLS services using the ES, a failure in the ES will trigger BMAC/0 updates and BMAC/ISID updates with increased sequence number. An additional I-VPLS is configured on the nodes with no send-bvpls-evpn-flush (default). The configuration of I-VPLS 1021 on PE-5 is as follows:
# on PE-5:
configure
service
vpls 1021 name "I-VPLS 1021" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
shutdown
exit
sap 1/2/1:1021 create
no shutdown
exit
spoke-sdp 56:1021 create
no shutdown
exit
no shutdown
exit
The configuration on PE-4 is similar; PE-2 and PE-3 have SAP lag-1:1021 instead of the spoke-SDP.
On MTU-6, SDP 65 is disabled, which will cause an ES failure on PE-5:
# on MTU-6:
configure
service
sdp 65
shutdown
The following BMAC updates are sent by PE-5:
BMAC/0 with increased sequence number, which will trigger a CMAC flush for all entries received from PE-5 for all I-VPLS services (ISID-independent)
BMAC/ISID with increased sequence number, which will trigger a CMAC flush for all entries received from PE-5 for VPLS 1001
73 2021/04/15 08:32:57.204 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388496
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:1/Static
"
74 2021/04/15 08:32:57.204 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 1001, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388496
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:3/Static
"
Conclusion
ISID-based MAC-flush speeds up convergence after a SAP or spoke-SDP failure, triggering a selective CMAC flush on the receiving nodes, which flushes all CMAC entries associated with that ISID and BMAC. The feature can be enabled per I-VPLS and disabled for those SAPs or spoke-SDPs for which no alternative route is available, or for those SAPs that are contained in an all-active Ethernet Segment. The BMAC/ISID update always contains the source-BMAC, not the ES-BMAC. CMAC flush based on ES-BMAC is not performed per ISID.