Black-hole MAC for EVPN Loop Protection
This chapter provides information about Black-hole MAC for EVPN Loop Protection.
Topics in this chapter include:
Applicability
This chapter was initially written based on SR OS Release 15.0.R4, but the CLI in the current edition corresponds to SR OS Release 21.2.R2. Black-hole MAC for EVPN loop protection is supported in SR OS Release 15.0.R1, and later.
Chapters Auto-Learn MAC Protect in EVPN and Conditional Static Black-Hole MAC in EVPN are prerequisite reading.
Overview
Service providers are migrating VPLS networks to EVPN and require the same or better loop protection mechanisms, such as mac-move or auto-learn-mac-protect (ALMP). Chapter Auto-Learn MAC Protect in EVPN describes how traffic is protected in "static" networks, where the CEs do not move to a different port or PE, and MAC addresses are always learned first on the correct SAP/SDP-bindings. However, ALMP does not provide a loop protection solution in EVPN networks that require mobility and ALMP has issues with all-active multi-homing. Since mobility and all-active multi-homing are two of the key advantages of EVPN compared to VPLS, an alternate loop protection mechanism is required. This chapter describes an example for the black-hole based loop protection solution, based on draft-snr-bess-evpn-loop-protect.
Black-hole MAC for EVPN loop protection shows a topology using black-hole MAC for EVPN loop protection.
VPLS 1 with EVI 1 is configured on all PEs. A backdoor link exists between PE-2 and PE-3 (in this case, caused by misconfiguration: additional SAPs are configured in VPLS 1). When CE-20 sends Broadcast, Unknown unicast, or Multicast (BUM) traffic, its source address MAC2 is learned by PE-2, which sends an EVPN-MAC route for MAC2 to its BGP peers. PE-2 floods the frame to its EVPN-MPLS destinations (PE-1 and PE-3) as well as its local SAPs (including the backdoor link to PE-3).
PE-3 receives the EVPN-MAC route from PE-2, but due to the backdoor link, it also learns MAC2 on its local SAP. Following the MAC mobility procedures, PE-3 advertises MAC2 with a higher sequence number to its BGP peers. PE-3 floods the frame to its EVPN-MPLS destinations and to its local SAPs.
The preceding simplified description assumes that PE-3 receives the EVPN-MAC route prior to learning MAC2 from the backdoor link, which may or may not be the case. Regardless of how MAC2 is learned, the MAC duplication procedures are invoked.
PE-2 and PE-3 keep learning and advertising MAC2 until the configured number of MAC moves (num-moves) has been reached. Then, MAC2 is detected as duplicate and will not be advertised again until the retry interval has expired.
If the mac-duplication black-hole-dup-mac option is configured, MAC2 will be added to the FDB as black-hole MAC, so traffic with MAC DA = MAC2 will be discarded. Also, MAC addresses assigned to a black-hole destination are considered as protected, so traffic with MAC SA = MAC2 will not be forwarded due to one of the following reasons:
When the SAPs/SDP-bindings or BGP-EVPN MPLS/VXLAN destinations are configured with restrict-protected-source discard-frame, the frames are discarded before any MAC SA is learned or the MAC DA is looked up.
When the SAP/SDP-binding is configured with restrict-protected-source, an incoming frame with MAC SA = black-hole MAC causes the system to bring down the corresponding SAP/SDP-binding.
Assuming PE-3 detects MAC2 as duplicate and installs it as black-hole MAC, PE-3 will discard the broadcast frames with MAC SA = MAC2, so the loop is broken, whereas the legitimate traffic between CE-10 and CE-20 is allowed (assuming PE-2 does not black-hole MAC2).
Black-hole MAC duplication is enabled with the black-hole-dup-mac keyword in the mac-duplication context, as follows:
*A:PE-3>config>service>vpls>bgp-evpn# mac-duplication ?
- mac-duplication
[no] black-hole-dup* - Enable/disable BGP-EVPN black-hole duplicate MAC traffic
detect - Configure BGP EVPN Mac Duplication Detection
[no] retry - Configure BGP EVPN Mac Duplication Retry
# on PE-3:
configure
service
vpls "VPLS 1"
bgp-evpn
mac-duplication
black-hole-dup-mac
When enabled, the operation is as follows:
Each node that learns a MAC address that has been advertised by a BGP peer will send an EVPN-MAC route for that MAC address with a higher sequence number. When the number of MAC moves exceeds the configured threshold (by default, five MAC moves in three minutes), the MAC address is detected as duplicate and no EVPN-MAC routes will be sent for that MAC address until the retry interval (default nine minutes) has elapsed.
When MAC2 is detected as duplicate, the system will:
Add MAC2 to the duplicate MAC list
Add MAC2 in the FDB as protected MAC associated with a black-hole endpoint (type EvpnD:P and source identifier black-hole)
Incoming frames with MAC DA = MAC2 will be discarded based on a MAC lookup in the FDB.
MAC addresses assigned to a black-hole destination are protected and incoming frames with MAC SA = MAC2 will be discarded or the system will bring down the SAP/SDP-binding, depending on the restrict-protected-src setting on the SAP/SDP/EVPN endpoint.
The following output shows the FDB with black-hole MAC address ca:fe:02:20:20:20 (type EvpnD:P):
*A:PE-3# show service id 1 fdb detail
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 ca:fe:01:10:10:10 mpls: Evpn 04/28/21 09:59:12
192.0.2.1:524284
ldp:65537
1 ca:fe:02:20:20:20 black-hole EvpnD:P 04/28/21 09:59:12
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The duplicate MAC address will be removed from the FDB and the process will be restarted in the following cases:
Retry interval events:
When the retry interval expires.
When the user configures no retry on the service that detected the duplicate MAC address.
MAC relearning events:
When the remote PE withdraws the MAC address (due to aging or clear service fdb). Local attempts to clear a black-hole MAC (via clear service fdb) will fail because the type of the MAC entry is not "learned", but "EvpnD:P".
When configuring a local conditional static MAC address (CStatic:P) prevents the EvpnD:P entry for the same MAC address from being installed in the FDB as black-hole, if the SAP/SDP-binding where the MAC is configured is operationally up.
CPM switchover event
Configuration
Example topology shows the example topology with three PEs and two CEs. A loop will occur when CE-20 sends Broadcast, Unknown unicast, or Multicast (BUM) traffic. Traffic between PE-2 and PE-3 will be sent over the regular router interfaces between the PEs, but also over the backdoor link (SAP 1/1/2:1 in VPLS 1 on PE-2 and SAP 1/1/1:1 in VPLS 1 on PE-3).
The initial configuration includes:
Cards, MDAs, ports
Router interfaces
IS-IS on all router interfaces (alternatively, OSPF can be used)
LDP on all router interfaces
Enable black-hole MAC duplication detection in EVPN
BGP is configured for address family EVPN on all PEs with PE-3 as route reflector. The following is the BGP configuration on PE-3:
# on PE-3:
configure
router Base
autonomous-system 64500
bgp
rapid-withdrawal
split-horizon
rapid-update evpn
group "internal"
family evpn
cluster 192.0.2.3
peer-as 64500
neighbor 192.0.2.1
exit
neighbor 192.0.2.2
exit
exit
exit
VPLS 1 is configured on all PEs with BGP-EVPN and MAC duplication enabled; on PE-2, as follows:
# on PE-2:
configure
service
vpls 1 name "VPLS 1" customer 1 create
bgp
exit
bgp-evpn
evi 1
mac-duplication
detect num-moves 3 window 1
retry 2
black-hole-dup-mac
exit
mpls bgp 1
restrict-protected-src discard-frame
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
sap 1/1/2:1 create # backdoor link to PE-3
exit
sap 1/2/1:1 create # to CE-20
exit
no shutdown
exit
To speed up MAC duplication detection, MAC duplication is detected after three MAC moves (default: five MAC moves). To shorten the retry interval, the time window is reduced to one minute (default: three minutes). When a MAC address has been detected as duplicated, the system removes the duplicate MAC entry after a retry interval of two minutes (default: nine minutes). The retry interval must be at least twice the time window for MAC duplication detection.
On the EVPN-MPLS endpoints, restrict-protected-src discard-frame must be configured. When MAC address ca:fe:02:20:20:20 is detected on PE-3 as a duplicate MAC address that is black-holed, the EVPN-MPLS endpoints on PE-3 should discard all frames with MAC SA ca:fe:02:20:20:20.
The configuration on the other PEs is similar; only the SAPs are different. VPLS 1 on PE-1 has SAP 1/2/1:1 to CE-10, but no SAP to a backdoor link; VPLS 1 on PE-3 has SAP 1/1/1:1 to the backdoor link to PE-2, but no SAP to a CE.
When CE-20 sends BUM traffic, its MAC SA ca:fe:02:20:20:20 is learned by PE-2 and advertised in EVPN-MAC routes. Because of the backdoor link to PE-3, PE-3 also learns MAC SA ca:fe:02:20:20:20 and advertises it to its BGP peers. The MAC-mobility sequence number is increased until the threshold of three MAC moves is reached. The following BGP EVPN-MAC route with sequence number 2 is sent by PE-2 to PE-3:
# on PE-2:
17 2021/04/28 09:59:11.599 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.2
Type: EVPN-MAC Len: 33 RD: 192.0.2.2:1 ESI: ESI-0, tag: 0, mac len: 48
mac: ca:fe:02:20:20:20, IP len: 0, IP: NULL, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
mac-mobility:Seq:2
"
The FDB on PE-2 shows that MAC ca:fe:02:20:20:20 has been learned on the SAP toward CE-20 (but it could also have been learned on the backdoor SAP or even be black-holed), as follows:
*A:PE-2# show service id 1 fdb detail
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 ca:fe:01:10:10:10 mpls: Evpn 04/28/21 09:59:12
192.0.2.1:524284
ldp:65537
1 ca:fe:02:20:20:20 sap:1/2/1:1 L/0 04/28/21 09:59:12
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following FDB on PE-3 shows that MAC ca:fe:02:20:20:20 has been detected as a duplicate and protected MAC (type EvpnD:P) associated with a black-hole endpoint:
*A:PE-3# show service id 1 fdb mac ca:fe:02:20:20:20
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 ca:fe:02:20:20:20 black-hole EvpnD:P 04/28/21 09:59:12
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following BGP-EVPN information for VPLS 1 on PE-3 shows the settings for MAC duplication detection, and the number of and list of detected duplicate MAC addresses:
*A:PE-3# show service id 1 bgp-evpn
===============================================================================
BGP EVPN Table
===============================================================================
MAC Advertisement : Enabled Unknown MAC Route : Disabled
CFM MAC Advertise : Disabled
Creation Origin : manual
MAC Dup Detn Moves : 3 MAC Dup Detn Window: 1
MAC Dup Detn Retry : 2 Number of Dup MACs : 1
MAC Dup Detn BH : Enabled
IP Route Advert : Disabled
Sel Mcast Advert : Disabled
EVI : 1
Ing Rep Inc McastAd: Enabled
Accept IVPLS Flush : Disabled
-------------------------------------------------------------------------------
Detected Duplicate MAC Addresses Time Detected
-------------------------------------------------------------------------------
ca:fe:02:20:20:20 04/28/2021 09:59:12
-------------------------------------------------------------------------------
===============================================================================
---snip---
The following message is logged in log "99" on PE-3 when VPLS 1 has detected duplicate MACs:
# on PE-3:
69 2021/04/28 10:04:40.266 UTC MINOR: SVCMGR #2331 Base
"VPLS Service 1 has MAC(s) detected as duplicates by EVPN mac-duplication detection."
MAC address ca:fe:02:20:20:20 remains in the FDB as duplicate and black-holed until the retry interval expires, as follows:
*A:PE-3>config>service>vpls>bgp-evpn>mac-duplication# retry ?
- no retry
- retry <minutes>
<minutes> : [2..60]
By default, the retry interval is nine minutes, but in this example, it is set to two minutes, which is the minimum value. The retry interval must be at least twice the time window for MAC duplication detection, which is by default three minutes, but reduced to one minute in this example. The following error is raised when attempting to configure a retry interval of two minutes for a detection time window of three minutes:
*A:PE-3>config>service>vpls>bgp-evpn>mac-duplication# retry 2
MINOR: SVCMGR #1003 Inconsistent value - mac-duplication detection retry time should be atleast twice that of detect window
After the retry interval expires, the MAC duplication is released.
Log "99" shows the following message when VPLS 1 no longer has duplicate MAC addresses:
# on PE-3:
70 2021/04/28 10:06:43.398 UTC MINOR: SVCMGR #2332 Base
"VPLS Service 1 no longer has MAC(s) detected as duplicates by EVPN mac-duplication detection."
MAC address ca:fe:02:20:20:20 remains in the FDB with type Evpn instead of EvpnD:P. BGP routes only disappear after a withdraw message has been received, whereas locally learned MAC addresses are flushed.
*A:PE-3# show service id 1 fdb mac ca:fe:02:20:20:20
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 ca:fe:02:20:20:20 mpls: Evpn 04/28/21 10:06:43
192.0.2.2:524284
ldp:65538
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
Clear commands
The following FDB entry on PE-3 of type EvpnD:P cannot be cleared with a normal FDB clear command:
*A:PE-3# show service id 1 fdb mac ca:fe:02:20:20:20
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 ca:fe:02:20:20:20 black-hole EvpnD:P 04/28/21 10:07:52
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The following error is raised when attempting to clear this FDB entry:
*A:PE-3# clear service id 1 fdb mac ca:fe:02:20:20:20
MAJOR: LOG #1202 Cannot perform clear operation - Entry is not of learned type
Log "99" shows the following message:
72 2021/04/28 10:08:17.960 UTC INDETERMINATE: LOGGER #2010 Base Clear SVCMGR
"Clear function clearSvcIdFdbMac has been run with parameters: svc-id="1" mac="ca:fe:02:20:20:20". The completion result is: failure. Additional error text, if any, is: Entry is not of learned type"
The following clear command releases the MAC duplication from the entry in the FDB, but it does not remove the entry from the FDB if it was learned from EVPN. The type is changed from EvpnD:P to Evpn.
*A:PE-3# clear service id 1 evpn mac-dup-detect ca:fe:02:20:20:20
*A:PE-3# show service id 1 fdb mac ca:fe:02:20:20:20
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 ca:fe:02:20:20:20 mpls: Evpn 04/28/21 10:09:50
192.0.2.2:524284
ldp:65538
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
Instead of clearing the MAC duplication state for one specific MAC address, all duplicate MAC addresses can be cleared by the following command:
*A:PE-3# clear service id 1 evpn mac-dup-detect all
When the MAC duplication is released, VPLS 1 no longer has duplicate MAC addresses detected, as follows:
*A:PE-3# show service id 1 bgp-evpn | match "Detected" pre-lines 2 post-lines 5
-------------------------------------------------------------------------------
Detected Duplicate MAC Addresses Time Detected
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
Log "99" shows the following messages related to the clear commands:
76 2021/04/28 10:10:13.078 UTC INDETERMINATE: LOGGER #2010 Base Clear SVCMGR
"Clear function cliClearSvcIdEvpnDupDetMacAll has been run with parameters: svc-id="1". The completion result is: success. Additional error text, if any, is: "
75 2021/04/28 10:09:49.947 UTC INDETERMINATE: LOGGER #2010 Base Clear SVCMGR
"Clear function cliClearSvcIdEvpnDupDetMac has been run with parameters: svc-id="1"mac="ca:fe:02:20:20:20". The completion result is: success. Additional error text, if any, is: "
Restrict Protected Source option
By default, the frames with MAC SA or DA equal to the duplicate MAC address are discarded, but the SAP/SDP-binding where the frame enters the VPLS remains operationally up. With the restrict-protected-src option, the system will bring the SAP/SDP-binding down where the frame with duplicate source MAC enters. The configuration on PE-2 and PE-3 is modified with restrict-protected-src on the SAP to the backdoor link, as follows:
# on PE-2:
configure
service
vpls "VPLS 1"
sap 1/1/2:1
restrict-protected-src
# on PE-3:
configure
service
vpls "VPLS 1"
sap 1/1/1:1
restrict-protected-src
When CE-20 sends BUM traffic, PE-3 detects MAC ca:fe:02:20:20:20 as duplicate. Log "99" shows that a duplicate MAC address has been detected, that protected MAC address ca:fe:02:20:20:20 has been received on SAP 1/1/1:1 in VPLS 1, and that the status of SAP 1/1/1:1 in VPLS 1 is changed to operationally down, with flag RxProtSrcMac indicating that a protected source MAC has been received.
80 2021/04/28 10:11:40.885 UTC MINOR: SVCMGR #2203 Base
"Status of SAP 1/1/1:1 in service 1 (customer 1) changed to admin=up oper=down flags=RxProtSrcMac "
79 2021/04/28 10:11:40.885 UTC MINOR: SVCMGR #2208 Base
"Protected MAC ca:fe:02:20:20:20 received on SAP 1/1/1:1 in service 1. The SAP will be disabled."
78 2021/04/28 10:11:39.886 UTC MINOR: SVCMGR #2331 Base
"VPLS Service 1 has MAC(s) detected as duplicates by EVPN mac-duplication detection."
The following shows that SAP 1/1/1:1 in VPLS 1 on PE-3 is operationally down with flag RxProtSrcMac:
*A:PE-3# show service id 1 sap 1/1/1:1
===============================================================================
Service Access Points(SAP)
===============================================================================
Service Id : 1
SAP : 1/1/1:1 Encap : q-tag
Description : (Not Specified)
Admin State : Up Oper State : Down
Flags : RxProtSrcMac
Multi Svc Site : None
Last Status Change : 04/28/2021 10:11:41
Last Mgmt Change : 04/28/2021 10:11:19
===============================================================================
The only way to re-enable the SAP is to disable and enable the SAP, as follows:
*A:PE-3# configure service vpls 1 sap 1/1/1:1 shutdown
*A:PE-3# configure service vpls 1 sap 1/1/1:1 no shutdown
*A:PE-3# show service id 1 sap
===============================================================================
SAP(Summary), Service 1
===============================================================================
PortId SvcId Ing. Ing. Egr. Egr. Adm Opr
QoS Fltr QoS Fltr
-------------------------------------------------------------------------------
1/1/1:1 1 1 none 1 none Up Up
-------------------------------------------------------------------------------
Number of SAPs : 1
-------------------------------------------------------------------------------
===============================================================================
Black-hole MAC duplication in all-active multi-homing
Example topology with all-active multi-homing shows the example topology with all-active multi-homing.
In this topology, the backdoor link is removed. On PE-1, VPLS 2 is configured without EVPN; on PE-2 and PE-3, VPLS 2 is configured with EVPN-MPLS. LAG 1 is configured on the PEs and Ethernet Segment (ES) ESI-23 is created on PE-2 and PE-3, as follows:
# on PE-2, PE-3:
configure
service
system
bgp-evpn
ethernet-segment "ESI-23" create
esi 01:00:00:00:00:23:00:00:00:01
es-activation-timer 3
service-carving
mode auto
exit
multi-homing all-active
lag 1
no shutdown
exit
The reason why black-hole MAC duplication should be configured instead of ALMP is the following. When ALMP is configured on SAP lag-1:2 on PE-2 and PE-3, MAC address ca:fe:01:12:12:12 of CE-12 is learned and protected on the SAP on both PEs. Traffic sent from CE-12 to CE-22 that is hashed over the direct link between PE-1 and PE-2 will reach its destination. Traffic that is hashed over the link between PE-1 and PE-3 will be forwarded by PE-3 to PE-2, but PE-2 will drop the traffic because it contains a MAC SA that is protected locally, as shown in Traffic dropped when ALMP is configured in all-active multi-homing.
When black-hole MAC duplication is configured instead of ALMP, traffic hashed on the link to PE-3 is forwarded to PE-2 and to CE-22. This is because MAC duplication is ES-aware and the same MAC seen on the same ES in two different PEs will never be detected as duplicate.
The configuration of VPLS 2 in PE-2 is as follows:
# on PE-2:
configure
service
vpls 2 name "VPLS 2" customer 1 create
bgp
exit
bgp-evpn
evi 2
mac-duplication
black-hole-dup-mac
exit
mpls bgp 1
restrict-protected-src discard-frame
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
sap 1/2/1:2 create
no shutdown
exit
sap lag-1:2 create
no shutdown
exit
no shutdown
The configuration of VPLS 2 on PE-3 is similar.
Conclusion
Black-hole MAC for EVPN MAC duplication protects EVPN services against customer-created backdoors or loops, while supporting MAC mobility and all-active multi-homing.