Conditional Static Black-Hole MAC in EVPN
This chapter provides information about Conditional Static Black-Hole MAC in EVPN.
Topics in this chapter include:
Applicability
This chapter was initially written for SR OS Release 14.0.R6, but the CLI in the current edition is based on SR OS Release 21.2.R1. Conditional static black-hole MAC is supported on EVPN services only, including EVPN-VXLAN and EVPN-MPLS, in SR OS Release 14.0.R1, and later.
Overview
A static black-hole MAC address is a local FDB record associated with a black-hole instead of a SAP or SDP-binding. Black-hole MAC addresses offer a scalable way to filter frames in the data plane based on MAC DA or SA, regardless of how the frame is arriving in the system. Black-hole MAC addresses can be configured in EVPN in the following ways:
Static configured black-hole MAC address
Anti-spoof MAC address in proxy Address Resolution Protocol/Neighbor Discovery (proxy-ARP/ND)
MAC-duplication black-hole (supported in SR OS Release 15.0.R1, and later), see chapter Black-hole MAC for EVPN Loop Protection
When a specific MAC address is configured as a static black-hole MAC address, all frames with MAC DA equal to this black-hole MAC address will be dropped. Also, black-hole MAC addresses are treated as protected MAC addresses, which allows filtering on MAC SA; see chapter Auto-Learn MAC Protect in EVPN.
The default behavior on the SAP/SDP-bindings is Restricted Protected Source Discard Frame (RPS-DF). Therefore, all frames with MAC SA equal to the black-hole MAC address will, by default, be dropped on the SAP/SDP-binding where the frames enter the service. Instead of dropping the frames, the entire SAP/SDP-binding can be brought operationally down, if the SAP/SDP-binding is explicitly configured with Restricted Protected Source (RPS) without any parameter. The SAP/SDP-binding can only be brought up manually by disabling (shutdown) and re-enabling (no shutdown) the SAP/SDP-binding. On the EVPN endpoints between PEs, it is possible to configure RPS-DF, not RPS. When configured, the EVPN endpoint will drop frames with MAC SA equal to the black-hole MAC address.
Black-hole MAC addresses can be used as an alternative to MAC filters, which simplifies the deployment of proxy-ARP/ND with anti-spoof MAC addresses. ARP/ND spoofing is a technique whereby an attacker sends fake ARP/ND messages to a broadcast domain. Generally, the aim is to get the routers in the broadcast domain to associate the attacker's MAC address with the IP address of another host, causing any traffic destined to that IP address to be sent to the attacker instead. To prevent this from happening, a proxy-ARP/ND with duplicate IP detection monitors the number of times the MAC changes for an offending IP address. When a certain number of MAC moves are detected in a defined period, the system flags the proxy-ARP entry as duplicate for a defined hold time and an alarm is sent to log 99.
Chapter EVPN for MPLS Tunnels describes the proxy-ARP/ND configuration with the option to define an anti-spoof MAC (AS-MAC) address for EVPN-MPLS networks using MAC filters, including some recommended settings. The AS-MAC address will be advertised with the duplicate IP address in gratuitous ARP (GARP) and ARP replies to all CEs in the EVPN (in the case of proxy-ND, unsolicited Neighbor Advertisement messages are sent instead of GARP messages).
ARP/ND broadcast traffic is a security issue for Internet eXchange Providers (IXPs) and service providers with large Layer 2 domains. In such networks, administrators try to avoid ARP/ND flooding. Proxy-ARP/ND and ARP spoofing shows the proxy-ARP/ND feature where local ARP/ND requests are responded by the system on behalf of the IP interface owners.
EVPN can suppress ARP/ND flooding within an EVPN service if all the attached hosts advertise their presence. Therefore, EVPN is preferred in IXPs to mitigate and even eliminate the ARP/ND flooding issue. The proxy-ARP/ND agent responds to local ARP/ND requests using a proxy-ARP/ND table per service. This table is populated by EVPN entries (MAC-IP pairs), static entries configured in the service, and dynamic entries snooped from ARP/GARP/ND messages sent by the ISP routers. The static entries and snooped dynamic entries are also advertised in EVPN-MAC routes.
As well as the proxy-ARP/ND, SR OS supports an anti-spoofing mechanism that can detect and block an ARP spoofing attack or a misconfigured duplicated IP address. When using MAC filters, the same anti-spoof-mac option must be configured in all the PEs and this filter may be configured on all the PE SAPs/SDP-bindings to discard all the frames with MAC DA equal to the anti-spoof MAC address. This requires a lot of configuration and is prone to configuration errors.
Conditional static black-hole MAC addresses can be configured for the anti-spoof MAC address so that frames with MAC DA equal to the anti-spoof MAC address can be discarded based on a MAC address lookup in the FDB, as opposed to a MAC filter entry. Less configuration is required and this simplifies the deployment of proxy-ARP/ND with AS-MAC. The configuration example in this chapter includes proxy-ARP, but the behavior is similar for proxy-ND.
Configuration
Example topology shows the example topology. Traffic will be sent between the CEs and may be dropped in the PEs if the MAC DA or MAC SA matches a black-hole MAC address. IP address 172.16.0.10/24 is duplicate (CE-10 and CE-11).
The initial configuration on the nodes includes:
Cards, MDAs, ports
Router interfaces
IS-IS between PEs
LDP between PEs
BGP is configured between the PEs for address family EVPN with PE-2 as route reflector (RR). Instead of an RR, a full mesh can also be configured between the PEs. The BGP configuration on PE-2 is as follows:
# on RR PE-2:
configure
router Base
autonomous-system 64500
bgp
rapid-withdrawal
split-horizon
rapid-update evpn
group "internal"
family evpn
cluster 1.1.1.1
peer-as 64500
neighbor 192.0.2.3
exit
neighbor 192.0.2.4
exit
exit
exit
VPLS 1 is configured on all PEs and on MTU-1 (MTU-1's VPLS 1 is connected to PE-3 by a SAP). The VPLS configuration on the PEs includes EVPN-MPLS, as follows:
# on PE-3:
configure
service
vpls 1 name "VPLS 1" customer 1 create
bgp
exit
bgp-evpn
evi 1
mpls bgp 1
ingress-replication-bum-label
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
sap 1/2/1:1 create
no shutdown
exit
sap 1/2/3:1 create
no shutdown
exit
no shutdown
exit
Conditional static black-hole MAC
Conditional static black-hole MAC address is an extension to the conditional static MAC address, but with the black-hole keyword. It is a scalable way to filter MAC DA or SA in the data plane, regardless of how the frame is arriving at the system (SAP/SDP-bindings or EVPN termination endpoints).
When the static black-hole MAC is added to the FDB, all Ethernet frames with MAC DA equal to the black-hole MAC are dropped. Filtering based on the MAC SA is explained in the next section: Conditional static black-hole MAC in combination with restrict protected source.
Conditional static black-hole MAC shows the example setup with conditional static black-hole MAC 00:00:aa:aa:aa:aa.
When no conditional static black-hole MAC is configured, CE-30 can receive and send traffic from and to the other CEs; for instance, from and toward CE-20, as follows:
*A:PE-2# ping router 10 172.16.0.30
PING 172.16.0.30 56 data bytes
64 bytes from 172.16.0.30: icmp_seq=1 ttl=64 time=0.836ms.
64 bytes from 172.16.0.30: icmp_seq=2 ttl=64 time=0.841ms.
---snip---
*A:PE-3# ping router 10 172.16.0.20
PING 172.16.0.20 56 data bytes
64 bytes from 172.16.0.20: icmp_seq=1 ttl=64 time=3.69ms.
64 bytes from 172.16.0.20: icmp_seq=2 ttl=64 time=0.814ms.
---snip---
In this example, CE-20 and CE-30 correspond to VPRN 10 configured on PE-2 and PE-3 (using a hairpin to loop the traffic back to the PE).
Conditional static black-hole MAC 00:00:aa:aa:aa:aa (which corresponds to the MAC address of CE-30) is configured in VPLS 1 on PE-3 as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
static-mac
mac 00:00:aa:aa:aa:aa create black-hole
The black-hole MAC is added as a conditional static (CStatic) MAC that is protected (P), as follows:
*A:PE-3# show service id 1 fdb mac 00:00:aa:aa:aa:aa
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:aa:aa:aa:aa black-hole CStatic: 03/24/21 15:46:33
P
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The source identifier is black-hole and it is applicable to frames that enter the VPLS on this node, regardless of how they enter the VPLS (SAP, SDP-binding, or EVPN endpoint).
The conditional static black-hole MAC is advertised to the BGP peers in a BGP-EVPN MAC route with the sticky/static bit set, as follows:
# on PE-3:
9 2021/03/24 15:46:32.883 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:aa:aa:aa:aa, IP len: 0, IP: NULL, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
The MAC route is added to the FDB on the other PEs as a static (S) and protected (P) MAC; for example, on PE-2, as follows:
*A:PE-2# show service id 1 fdb mac 00:00:aa:aa:aa:aa
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:aa:aa:aa:aa mpls: EvpnS:P 03/24/21 15:46:33
192.0.2.3:524284
ldp:65537
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
When CE-20 sends an ICMP request to CE-30, the MAC DA 00:00:aa:aa:aa:aa is black-holed on PE-3, and no ICMP request succeeds, as follows:
*A:PE-2# ping router 10 172.16.0.30
PING 172.16.0.30 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
Request timed out. icmp_seq=3.
Request timed out. icmp_seq=4.
Request timed out. icmp_seq=5.
---- 172.16.0.30 PING Statistics ----
5 packets transmitted, 0 packets received, 100% packet loss
The port statistics show that the traffic was sent from PE-2 to PE-3, where it entered on port 1/1/3, then got discarded. To verify this, the port statistics are cleared on PE-2 and PE-3, then 1000 ICMP packets are sent from CE-20, as follows:
*A:PE-2# clear port 1/[1..2]/[1..4] statistics
*A:PE-3# clear port 1/[1..2]/[1..4] statistics
*A:PE-2# ping router 10 172.16.0.30 rapid count 1000
---snip---
1000 packets transmitted, 0 packets received, 100% packet loss
The 1000 packets are received at SAP 1/2/1:1 on PE-2, as follows:
*A:PE-2# show port 1/2/1 statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/2/1 1000 106000
0 0
===============================================================================
These packets are forwarded to port 1/1/3 toward PE-3, as follows:
*A:PE-2# show port 1/1/3 statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/3 13 1306
1013 125254
===============================================================================
On the interfaces between the PEs, other packets are sent besides the ICMP requests, such as IS-IS messages; therefore, the number of packets is slightly greater than 1000.
On PE-3, these packets are received on port 1/1/3, as follows:
*A:PE-3# show port 1/1/3 statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/3 1024 126444
24 2351
===============================================================================
The FDB entry for this MAC DA is black-holed and no traffic is received on SAP 1/2/1:1 toward CE-30; therefore, the statistics for port 1/2/1 are empty and nothing is displayed, as follows:
*A:PE-3# show port 1/2/1 statistics
*A:PE-3#
It is possible to configure the black-hole MAC address on a different PE; for example, on PE-4 instead of PE-3. The conditional static black-hole MAC address configuration in VPLS 1 on PE-3 is removed, as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
static-mac
no mac 00:00:aa:aa:aa:aa
The conditional static black-hole MAC is configured on PE-4 instead, as follows:
# on PE-4:
configure
service
vpls "VPLS 1"
static-mac
mac 00:00:aa:aa:aa:aa create black-hole
PE-4 sends EVPN-MAC updates to its peers. PE-2 learns that all traffic with MAC DA 00:00:aa:aa:aa:aa should be redirected to PE-4, as shown in the FDB on PE-2:
*A:PE-2# show service id 1 fdb mac 00:00:aa:aa:aa:aa
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:aa:aa:aa:aa mpls: EvpnS:P 03/24/21 15:51:14
192.0.2.4:524284
ldp:65538
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The port statistics are cleared on all PEs and 1000 ICMP packets are sent from CE-20 to CE-30, as follows:
*A:PE-2# ping router 10 172.16.0.30 rapid count 1000
---snip---
1000 packets transmitted, 0 packets received, 100% packet loss
On PE-2, traffic is not forwarded on the direct link (port 1/1/3) toward PE-3, but redirected to PE-4 (port 1/1/1) instead, as follows:
*A:PE-2# show port 1/1/[1..3] statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/1 15 1464
1014 125360
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/3 15 1433
16 1589
===============================================================================
On PE-4, traffic is received on port 1/1/2, then discarded because the MAC DA equals the static black-hole MAC in the FDB, as follows. No traffic is forwarded to PE-3, where CE-30 is attached.
*A:PE-4# show port 1/[1..2]/[1..4] statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/1 22 2192
22 2192
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/2 1025 126476
24 2351
===============================================================================
The configuration is restored with conditional static black-hole MAC in VPLS 1 on PE-3, not on PE-4, as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
static-mac
mac 00:00:aa:aa:aa:aa create black-hole
# on PE-4:
configure
service
vpls "VPLS 1"
static-mac
no mac 00:00:aa:aa:aa:aa
Conditional static black-hole MAC in combination with restrict protected source
For Ethernet frames with MAC SA equal to the static black-hole MAC, the treatment is the same as for protected MACs (see chapter Auto-Learn MAC Protect in EVPN), but for conditional static black-hole MACs, ALMP need not be enabled on the SAP or SDP-binding:
When a frame is received with MAC SA equal to the black-hole MAC, it is dropped, because RPS-DF is enabled on the SAP or SDP-binding, by default. RPS-DF need not be enabled explicitly. The default is no restrict-protected-src, which operates as RPS-DF. An error message is raised when the following command is entered:
*A:PE-3>config>service>vpls>sap# restrict-protected-src discard-frame MINOR: SVCMGR #7888 Cannot be configured/enabled with EVPNWhen RPS is enabled instead of RPS-DF, the SAP or SDP-binding where the frame was received, with MAC SA equal to the black-hole MAC, is brought operationally down. The SAP or SDP-binding can be brought up manually by disabling (shutdown) and re-enabling (no shutdown) the SAP or SDP-binding. RPS is enabled on SAP 1/2/1:1 as follows:
# on PE-3: configure service vpls "VPLS 1" sap 1/2/1:1 restrict-protected-srcOptionally, RPS-DF can be enabled on the EVPN-MPLS endpoint or EVPN-VXLAN endpoint. When enabled, the EVPN endpoint will discard frames with MAC SA equal to the black-hole MAC. RPS cannot be configured instead of RPS-DF on EVPN endpoints. It is not an option to bring the EVPN endpoint down when a frame is received with MAC SA equal to the static black-hole MAC. The commands to enable RPS-DF on the EVPN-MPLS endpoints and EVPN-VXLAN endpoints are as follows:
*A:PE-3>config>service>vpls>bgp-evpn>mpls# restrict-protected-src ? - no restrict-protected-src - restrict-protected-src discard-frame <discard-frame> : keyword - discard frame and trap on a protected MAC*A:PE-3>config>service>vpls>vxlan$ restrict-protected-src ? - no restrict-protected-src - restrict-protected-src discard-frame
With the default configuration (RPS-DF on SAP/SDP-bindings), the behavior is as follows for conditional static black-hole MAC 00:00:aa:aa:aa:aa configured in VPLS 1 on PE-3. All traffic from CE-30 with MAC SA 00:00:aa:aa:aa:aa is black-holed on SAP 1/2/1:1 on PE-3, because the default behavior on SAP 1/2/1:1 is RPS-DF, and the frame is discarded. The packets are received on port 1/2/1 (SAP 1/2/1:1) and dropped. No packets are forwarded to port 1/1/3 toward PE-2 or any other port.
*A:PE-3# clear port 1/[1..2]/[1..4] statistics
*A:PE-3# ping router 10 172.16.0.20 rapid count 1000
---snip---
1000 packets transmitted, 0 packets received, 100% packet loss
*A:PE-3# show port 1/[1..2]/[1..4] statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/2 17 1732
17 1732
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/3 20 1995
18 1787
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/2/1 1000 106000
0 0
===============================================================================
If the static MAC is configured in VPLS 1 on PE-4 and not on PE-3, PE-3 will still discard the packets with MAC SA 00:00:aa:aa:aa:aa arriving on SAP 1/2/1:1, because it learned from the EVPN-MAC updates that MAC 00:00:aa:aa:aa:aa is a protected MAC on PE-4. Therefore, traffic with this MAC SA is not expected and not allowed on PE-3, as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
static-mac
no mac 00:00:aa:aa:aa:aa
# on PE-3:
configure
service
vpls "VPLS 1"
static-mac
mac 00:00:aa:aa:aa:aa create black-hole
*A:PE-3# show service id 1 fdb mac 00:00:aa:aa:aa:aa
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:aa:aa:aa:aa mpls: EvpnS:P 03/24/21 15:54:54
192.0.2.4:524284
ldp:65538
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
*A:PE-3# ping router 10 172.16.0.20 rapid count 1000
---snip---
1000 packets transmitted, 0 packets received, 100% packet loss
*A:PE-3# show port 1/[1..2]/[1..4] statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/2 22 2192
22 2192
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/3 25 2476
24 2351
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/2/1 1000 106000
0 0
===============================================================================
The configuration is restored as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
static-mac
mac 00:00:aa:aa:aa:aa create black-hole
# on PE-4:
configure
service
vpls "VPLS 1"
static-mac
no mac 00:00:aa:aa:aa:aa
Optionally, RPS-DF can be configured on the EVPN-MPLS endpoints on the PEs, as follows:
# on PE-2, PE-3, PE-4:
configure
service
vpls "VPLS 1"
bgp-evpn
mpls bgp 1
restrict-protected-src discard-frame
When RPS-DF is configured on the EVPN-MPLS endpoints, frames with MAC SA 00:00:aa:aa:aa:aa can be discarded by the EVPN endpoints between the PEs. However, in this example this is not required, because any frame with MAC SA 00:00:aa:aa:aa:aa will be dropped by the local SAP before it can be forwarded to an EVPN endpoint.
It is possible to configure RPS without any parameters on SAP 1/2/1:1 on PE-3, as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
sap 1/2/1:1
restrict-protected-src
When CE-30 sends traffic with MAC SA equal to a protected MAC address (black-hole or not), the entire SAP 1/2/1:1 will be brought operationally down, as follows:
*A:PE-3# ping router 10 172.16.0.20
PING 172.16.0.20 56 data bytes
Request timed out. icmp_seq=1.
Request timed out. icmp_seq=2.
---snip---
---- 172.16.0.20 PING Statistics ----
5 packets transmitted, 0 packets received, 100% packet loss
*A:PE-3# show service id 1 sap
===============================================================================
SAP(Summary), Service 1
===============================================================================
PortId SvcId Ing. Ing. Egr. Egr. Adm Opr
QoS Fltr QoS Fltr
-------------------------------------------------------------------------------
1/2/1:1 1 1 none 1 none Up Down
1/2/3:1 1 1 none 1 none Up Up
-------------------------------------------------------------------------------
Number of SAPs : 2
-------------------------------------------------------------------------------
===============================================================================
The following information for SAP 1/2/1:1 in VPLS 1 shows that this SAP is operationally down because a protected source MAC address was received on this SAP (Flags: RxProtSrcMac), as follows:
*A:PE-3# show service id 1 sap 1/2/1:1
===============================================================================
Service Access Points(SAP)
===============================================================================
Service Id : 1
SAP : 1/2/1:1 Encap : q-tag
Description : (Not Specified)
Admin State : Up Oper State : Down
Flags : RxProtSrcMac
Multi Svc Site : None
Last Status Change : 03/24/2021 15:54:06
Last Mgmt Change : 03/24/2021 15:53:25
===============================================================================
Log 99 shows that a protected MAC was received on SAP 1/2/1:1 and the SAP went operationally down with flag RxProtSrcMac, as follows:
107 2021/03/24 15:54:05.661 UTC MINOR: SVCMGR #2208 Base
"Protected MAC 00:00:aa:aa:aa:aa received on SAP 1/2/1:1 in service 1. The SAP will be disabled."
108 2021/03/24 15:54:05.662 UTC MINOR: SVCMGR #2203 Base
"Status of SAP 1/2/1:1 in service 1 (customer 1) changed to admin=up oper=down flags=RxProtSrcMac "
The SAP can only be brought up manually by disabling and re-enabling the SAP, as follows:
*A:PE-3# configure service vpls "VPLS 1" sap 1/2/1:1 shutdown
*A:PE-3# configure service vpls "VPLS 1" sap 1/2/1:1 no shutdown
*A:PE-3# show service id 1 sap
===============================================================================
SAP(Summary), Service 1
===============================================================================
PortId SvcId Ing. Ing. Egr. Egr. Adm Opr
QoS Fltr QoS Fltr
-------------------------------------------------------------------------------
1/2/1:1 1 1 none 1 none Up Up
1/2/3:1 1 1 none 1 none Up Up
-------------------------------------------------------------------------------
Number of SAPs : 2
-------------------------------------------------------------------------------
===============================================================================
The default behavior of SAP 1/2/1:1 is RPS-DF, which is configured by removing the RPS configuration, as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
sap 1/2/1:1
no restrict-protected-src
The conditional static black-hole MAC configuration is removed as follows:
# on PE-3:
configure
service
vpls "VPLS 1"
static-mac
no mac 00:00:aa:aa:aa:aa
Black-hole MAC in services with proxy-ARP/ND
In this example, only proxy-ARP is shown, not proxy-ND. However, the configuration and procedures for proxy-ND would be equivalent.
First, the implementation of proxy-ARP and AS-MAC is described without static black-hole MAC addresses. MAC filters will be required to drop or redirect traffic, but these are not shown in the example. Configuring MAC filters and applying them on SAP/SDP-bindings is labor-intensive and can be error-prone. Afterward, the implementation with AS-MAC as static black-hole is described.
Services with proxy-ARP and AS-MAC - no static black-hole MAC
IP duplication works when the IP address moves between:
Dynamic (learned on SAP) and EVPN
EVPN and dynamic
Dynamic and dynamic
The following example shows IP address moves from dynamic to dynamic between SAP 1/2/1:1 (to CE-10) and SAP 1/2/1:2 (to CE-11) in VPLS 1 on MTU-1. However, the duplicate IP address could have been in PE-3 and MTU-1 instead (EVPN or dynamic) and still the IP address would have been detected as duplicate.
VPLS 1 with proxy-ARP and AS-MAC shows the example setup with duplicate IP address 172.16.0.10/24 for CE-10 and CE-11. VPLS 1 is configured with proxy-ARP with duplicate IP detection in PE-2 and PE-3 (and possibly also in other PEs). MAC address 00:00:bb:bb:bb:bb is configured as AS-MAC, which will be used when a duplicate IP address has been detected.
For IP duplication detection, the following parameters can be customized so that the system can react to particular conditions in the network. The syntax is as follows:
*A:PE-3>config>service>vpls>proxy-arp$ dup-detect ?
- dup-detect [anti-spoof-mac <mac-address>] window <minutes> num-moves <count>
hold-down <minutes|max>
- dup-detect anti-spoof-mac <mac-address> window <minutes> num-moves <count>
hold-down <minutes|max> [static-black-hole]
<mac-address> : xx-xx-xx-xx-xx-xx or xx:xx:xx:xx:xx:xx (hex chars)
<minutes> : [1..15] minutes - default:3
<count> : [3..10] - default:5
<minutes|max> : [2..60] default=9 | max - permanent hold
<static-black-hole> : keyword
In VPLS 1 on PE-3, a proxy-ARP with duplicate IP detection is configured, including an optional anti-spoof MAC (AS-MAC) 00:00:bb:bb:bb:bb for offending IP addresses, as follows:
#on PE-3:
configure
service
vpls "VPLS 1"
proxy-arp
dup-detect window 3 num-moves 3 hold-down max
anti-spoof-mac 00:00:bb:bb:bb:bb
dynamic-arp-populate
static 172.16.0.20 00:00:02:20:20:20
no shutdown
exit
The proxy-ARP table contains one static entry (for IP 172.16.0.20). In this case, dynamic ARP populate is enabled. Therefore, the proxy-ARP table will be updated with ARP entries for IP 172.16.0.10 and MAC 00:00:01:10:10:10 or MAC 00:00:01:11:11:11 for frames originating from CE-10 or CE-11.
When a duplicate IP is detected for IP 172.16.0.10 (after three changes of MAC for IP 172.16.0.10 in a period of three minutes), the corresponding ARP entry contains the duplicate IP address 172.16.0.10 and the AS-MAC 00:00:bb:bb:bb:bb and its type is duplicate (dup). Therefore, this ARP entry is always active until it is removed. Until now, this configuration does not include a static black-hole MAC, and this option is by default disabled. This configuration for duplicate IP detection can be used in combination with MAC filters. The configuration with static black-hole MAC is shown in the section Services with proxy-ARP and AS-MAC configured as static black-hole MAC.
The configured AS-MAC will be advertised in an EVPN-MAC route with the sticky/static bit set and without any IP address (because there is no IP duplication detected yet), as follows:
# on PE-3:
28 2021/03/24 16:01:30.827 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 124
Flag: 0x90 Type: 14 Len: 79 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 02:17:ff:00:03:3a, IP len: 0, IP: NULL, label1: 8388544
Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:bb:bb:bb:bb, IP len: 0, IP: NULL, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
Without the option static black-hole, the configured AS-MAC is not added to the local FDB, but this MAC address is treated as a local MAC. The FDB on PE-3 does not contain AS-MAC 00:00:bb:bb:bb:bb, as follows:
*A:PE-3# show service id 1 fdb mac 00:00:bb:bb:bb:bb
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================
Debugging is enabled for proxy-ARP for IP address 172.16.0.10 in VPLS 1 on PE-3, as follows:
# on PE-3:
debug
service
id 1
proxy-arp ip 172.16.0.10
When traffic is sent from CE-11 to CE-21, a dynamic ARP entry for IP address 172.16.0.10 and MAC 00:00:01:11:11:11 is added to the proxy-ARP table for VPLS 1 in PE-3, and an EVPN-MAC update is sent to the peer PEs, as follows:
# on PE-3:
35 2021/03/24 16:06:02.919 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:11:11:11 evpn advertise"
36 2021/03/24 16:06:02.919 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 type: Dyn mac: 00:00:01:11:11:11 Added"
37 2021/03/24 16:06:02.919 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 85
Flag: 0x90 Type: 14 Len: 48 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 37 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:01:11:11:11, IP len: 4, IP: 172.16.0.10, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 16 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
"
There is no duplicate IP detected yet.
CE-10 and CE-11 have the same IP address for different MAC addresses. When CE-10 sends traffic to CE-20, the ARP entry for IP 172.16.0.10 changes MAC from 00:00:01:11:11:11 to 00:00:01:10:10:10, and an EVPN-MAC withdraw message is sent, as follows:
# on PE-3:
38 2021/03/24 16:06:07.137 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:11:11:11 evpn withdraw"
39 2021/03/24 16:06:07.137 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 Mac Change: 00:00:01:11:11:11->00:00:01:10:10:10 "
40 2021/03/24 16:06:07.137 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 46
Flag: 0x90 Type: 15 Len: 42 Multiprotocol Unreachable NLRI:
Address Family EVPN
Type: EVPN-MAC Len: 37 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:01:11:11:11, IP len: 4, IP: 172.16.0.10, label1: 0
"
When the MAC changes, the system sends an ARP request for confirmation of the old MAC 00:00:01:11:11:11 for IP 172.16.0.10, as follows:
# on PE-3:
41 2021/03/24 16:06:07.290 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:11:11:11 confirm"
When MAC 00:00:01:11:11:11 is confirmed, the MAC in the ARP entry is changed once again to 00:00:01:11:11:11 and another ARP request is sent asking to confirm MAC 00:00:01:10:10:10 for IP 172.16.0.10, as follows:
# on PE-3:
42 2021/03/24 16:06:11.147 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 Mac Change: 00:00:01:10:10:10->00:00:01:11:11:11 "
43 2021/03/24 16:06:11.290 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:10:10:10 confirm"
When CE-10 confirms MAC 00:00:01:10:10:10 for IP 172.16.0.10, IP duplication is detected for IP address 172.16.0.10 (after three MAC moves in a detection period of three minutes), and the following message is raised in log 99 after a duplicate proxy-ARP entry was detected for IP 172.16.0.10:
# log "99" on PE-3:
136 2021/03/24 16:06:15.358 UTC MINOR: SVCMGR #2346 Base
"A duplicate proxy ARP entry was detected with new MAC 00:00:01:10:10:10 for entry
IP 172.16.0.10 MAC 00:00:01:11:11:11 in service 1"
The following proxy-ARP debug messages show that the ARP entry for IP 172.16.0.10 in the proxy-ARP table changed MAC to the AS-MAC 00:00:bb:bb:bb:bb, and the type from dynamic to duplicate:
# on PE-3:
44 2021/03/24 16:06:15.357 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:bb:bb:bb:bb evpn advertise"
45 2021/03/24 16:06:15.358 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 Mac Change: 00:00:01:11:11:11->00:00:bb:bb:bb:bb Type Change: Dyn->Dup "
46 2021/03/24 16:06:15.358 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 type: Dup Dup Detected"
If a duplicate IP is detected, AS-MAC 00:00:bb:bb:bb:bb is advertised with duplicate IP address 172.16.0.10 in an EVPN-MAC update to the BGP peers with the sticky/static bit set, as follows:
47 2021/03/24 16:06:15.358 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 93
Flag: 0x90 Type: 14 Len: 48 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 37 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:bb:bb:bb:bb, IP len: 4, IP: 172.16.0.10, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
The difference with the first EVPN-MAC update for AS-MAC is the IP address. Immediately after the AS-MAC was configured, it was also advertised to the BGP-EVPN peers, but without any IP address.
The proxy-ARP entry is shown with type duplicate (dup) and active status in the proxy-ARP table for VPLS 1 on PE-3, as follows:
*A:PE-3# show service id 1 proxy-arp detail
-------------------------------------------------------------------------------
Proxy Arp
-------------------------------------------------------------------------------
Admin State : enabled
Dyn Populate : enabled
Age Time : disabled Send Refresh : disabled
Table Size : 250 Total : 2
Static Count : 1 EVPN Count : 0
Dynamic Count : 0 Duplicate Count : 1
Dup Detect
-------------------------------------------------------------------------------
Detect Window : 3 mins Num Moves : 3
Hold down : max
Anti Spoof MAC : 00:00:bb:bb:bb:bb
EVPN
-------------------------------------------------------------------------------
Garp Flood : enabled Req Flood : enabled
Static Black Hole : disabled
EVPN Route Tag : 0
-------------------------------------------------------------------------------
===============================================================================
VPLS Proxy Arp Entries
===============================================================================
IP Address Mac Address Type Status Last Update
-------------------------------------------------------------------------------
172.16.0.10 00:00:bb:bb:bb:bb dup active 03/24/2021 16:06:15
172.16.0.20 00:00:02:20:20:20 stat inActv 03/24/2021 16:01:31
-------------------------------------------------------------------------------
Number of entries : 2
===============================================================================
A duplicate entry is always active, regardless of the AS-MAC. When the entry with the duplicate IP address and the AS-MAC address are installed in the proxy-ARP table as active, every ARP request for the duplicate IP address will be replied by the system. The entry in the proxy-ARP table is treated as active, even if the AS-MAC address is not in the FDB (AS-MAC addresses do not consume FDB space). The AS-MAC address, along with the duplicate IP address, is advertised in EVPN with the sticky/static bit set, as shown earlier. GARP messages with AS-MAC/IP information are flooded locally to make the CEs update their ARP caches to use the AS-MAC address for traffic to the duplicate IP 172.16.0.10, as follows.
# on PE-3:
48 2021/03/24 16:06:15.490 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 type: Dup mac: 00:00:bb:bb:bb:bb Gratuitous Update"
The AS-MAC address will always be "unique" in the system. When the AS-MAC is configured, the system will flush any entry with the same MAC address learned through EVPN or dynamic sources. Conditional static MAC addresses or OAM MAC addresses with the same value as the AS-MAC address are only allowed when they are configured as black-hole, which is not the case yet.
When the duplicate proxy-ARP entry is cleared from the list (hold-down timer expires, or clear command, or replacement of the duplicate entry for a static entry), an ARP request asking who has IP 172.16.0.10 is flooded by the proxy-ARP agent. This ARP refresh triggers an ARP reply from the IP owner, which will be learned in the proxy-ARP table and advertised in EVPN. The system will also send a GARP to local SAP/SDP-bindings. This will correct all host ARP caches in the network. In this example, the duplicate proxy-ARP entry is manually cleared, as follows:
*A:PE-3# clear service id 1 proxy-arp duplicate
Log "99" shows that the clear function has been run and the duplicate proxy-ARP entry 172.16.0.10 is cleared. The system forces a refresh and, if the condition with the duplicate IP address remains, this is detected almost immediately and a message is logged that a duplicate proxy-ARP entry was detected, as follows:
# on PE-3:
137 2021/03/24 16:08:51.964 UTC INDETERMINATE: LOGGER #2010 Base Clear SVCMGR
"Clear function clearSvcIdProxyArpDups has been run with parameters: svc-id="1"
ip-address="". The completion result is: success. Additional error text, if any, is: "
138 2021/03/24 16:08:51.965 UTC MINOR: SVCMGR #2347 Base
"A duplicate proxy ARP entry 172.16.0.10 is cleared in service 1"
139 2021/03/24 16:08:52.193 UTC MINOR: SVCMGR #2346 Base
"A duplicate proxy ARP entry was detected with new MAC 00:00:01:11:11:11 for entry
IP 172.16.0.10 MAC 00:00:01:10:10:10 in service 1"
The following debug messages for proxy-ARP on PE-3 show the process in more detail. Initially, an EVPN-MAC route withdraw message is sent and the proxy-ARP entry is deleted.
# on PE-3:
49 2021/03/24 16:08:51.964 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:bb:bb:bb:bb evpn withdraw"
50 2021/03/24 16:08:51.964 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 type: Dup mac: 00:00:bb:bb:bb:bb Deleted"
The following BGP-EVPN MAC update is sent by PE-3 to indicate that the AS-MAC is withdrawn for IP 172.16.0.10 (multiprotocol unreachable NLRI):
# on PE-3:
52 2021/03/24 16:08:51.965 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 46
Flag: 0x90 Type: 15 Len: 42 Multiprotocol Unreachable NLRI:
Address Family EVPN
Type: EVPN-MAC Len: 37 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:bb:bb:bb:bb, IP len: 4, IP: 172.16.0.10, label1: 0
"
Removing the active duplicate entry from the proxy-ARP table triggers an ARP flooding request asking who has IP 172.16.0.10 in VPLS 1, as follows:
# on PE-3:
51 2021/03/24 16:08:51.964 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 flood request"
The result of the ARP flooding request is that the IP owners reply with their MAC, at the local or a remote PE. In this case, the reply from CE-10 is received first (IP 172.16.0.10 - MAC 00:00:01:10:10:10), a dynamic proxy-ARP entry is added, and the MAC/IP route is advertised, as follows:
# on PE-3:
53 2021/03/24 16:08:51.967 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:10:10:10 evpn advertise"
54 2021/03/24 16:08:51.967 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 type: Dyn mac: 00:00:01:10:10:10 Added"
When CE-11 answers with its MAC 00:00:01:11:11:11, the MAC/IP route is withdrawn for IP 172.16.0.10, and the MAC address in the proxy-ARP entry for IP 172.16.0.10 is changed from MAC 00:00:01:10:10:10 to MAC 00:00:01:11:11:11, as follows:
# on PE-3:
55 2021/03/24 16:08:51.967 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:10:10:10 evpn withdraw"
56 2021/03/24 16:08:51.967 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 Mac Change: 00:00:01:10:10:10->00:00:01:11:11:11 "
Any change of MAC address in a proxy-ARP entry triggers an ARP request asking for confirmation of the old MAC address for IP 172.16.0.10, in this case for MAC 00:00:01:10:10:10, as follows:
# on PE-3:
57 2021/03/24 16:08:52.090 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:10:10:10 confirm"
MAC address 00:00:01:10:10:10 is confirmed for IP address 172.16.0.10; therefore, the MAC address is changed in the proxy-ARP entry from 00:00:01:11:11:11 to 00:00:01:10:10:10, and an ARP confirmation is asked for the old MAC address 00:00:01:11:11:11, as follows:
# on PE-3:
58 2021/03/24 16:08:52.093 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 Mac Change: 00:00:01:11:11:11->00:00:01:10:10:10 "
59 2021/03/24 16:08:52.190 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:01:11:11:11 confirm"
MAC address 00:00:01:11:11:11 is confirmed and, therefore, three MAC moves occurred within three minutes. Duplicate IP 172.16.0.10 is detected and the proxy-ARP entry has the AS-MAC 00:00:bb:bb:bb:bb and type duplicate (Dup), as follows:
# on PE-3:
60 2021/03/24 16:08:52.193 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 mac: 00:00:bb:bb:bb:bb evpn advertise"
61 2021/03/24 16:08:52.193 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 Mac Change: 00:00:01:10:10:10->00:00:bb:bb:bb:bb Type Change: Dyn->Dup "
62 2021/03/24 16:08:52.193 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 type: Dup Dup Detected"
63 2021/03/24 16:08:52.193 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 93
Flag: 0x90 Type: 14 Len: 48 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 37 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:bb:bb:bb:bb, IP len: 4, IP: 172.16.0.10, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
A GARP update is sent for IP 172.16.0.10 and AS-MAC 00:00:bb:bb:bb:bb, as follows:
# on PE-3:
64 2021/03/24 16:08:52.290 UTC MINOR: DEBUG #2001 Base proxy arp
"proxy arp:
svc: 1 ip: 172.16.0.10 type: Dup mac: 00:00:bb:bb:bb:bb Gratuitous Update"
The AS-MAC address is optionally configured and populates all the host ARP caches when a duplicate IP address is detected. All traffic destined to the suspicious IP address 172.16.0.10 will have the AS-MAC address 00:00:bb:bb:bb:bb as MAC DA. The user can configure MAC filters on all SAP/SDP-bindings where the CEs are connected to drop, log, or redirect traffic destined to the AS-MAC. This will block any interception or man-in-the-middle attack (due to ARP-spoofing) in the network.
The AS-MAC address is independently configured on each PE for the same service. When a different AS-MAC address is configured per PE for the same service, the user will need to filter all the AS-MAC addresses in the service at each PE, which increases the complexity of the filters. Nokia recommends using the same AS-MAC address for the same service in all the PES where duplicate detect is active and MAC filters need to be configured. However, this recommendation is suspended when the AS-MAC address is configured as static black-hole MAC address, as described in the following section.
Services with proxy-ARP and AS-MAC configured as static black-hole MAC
With the AS-MAC address configured as static black-hole MAC address, MAC-filters do not need to be configured to discard frames with MAC DA equal to the AS-MAC address. Instead, the user can decide whether to use the same AS-MAC address on all the PEs. This scalability is not limited by the number of filters, but by the number of FDB entries.
The static-black-hole parameter is optional and disabled by default. In the example, the static-black-hole option is not configured yet for the AS-MAC address and the behavior is as follows:
The AS-MAC address is added to the MAC DB as local, but not programmed in the FDB.
The AS-MAC address is advertised in EVPN (initially without an IP address, and with an IP address as soon as the IP is detected as duplicate).
The AS-MAC address cannot be overridden by any other MAC address.
The AS-MAC address value cannot be configured on a static MAC address, because that MAC address is reserved for the proxy-ARP, as follows:
*A:PE-3>conf>serv>vpls>static-mac# mac 00:00:bb:bb:bb:bb create sap 1/2/3:1 monitor fwd-status
MINOR: SVCMGR #7875 Cannot create conditional static mac - Mac reserved by proxy
*A:PE-3>conf>serv>vpls>static-mac# mac 00:00:bb:bb:bb:bb create black-hole
MINOR: SVCMGR #7875 Cannot create conditional static mac - Mac reserved by proxy
When the static-black-hole option is not configured, the AS-MAC address is considered as a local MAC address and cannot be overridden. The MAC address priority is as follows:
Local MAC address (including AS-MAC addresses without static-black-hole, es-bmacs, src-bmacs, OAM, and so on)
Conditional static MAC addresses (including AS-MAC addresses with static-black-hole)
Auto-Learn Protected MAC addresses
EVPN-MAC addresses with sticky/static bit set
Data plane learned MAC addresses (regular learning on SAP/SDP-binding)
EVPN-MAC addresses without sticky/static bit set
To configure an AS-MAC address with static-black-hole option, a static black-hole MAC address needs to be configured first. The following error is raised when no static black-hole MAC has been configured for AS-MAC 00:00:bb:bb:bb:bb:
*A:PE-3>config>service>vpls>proxy-arp# dup-detect window 3 num-moves 5 hold-down max anti-spoof-mac 00:00:bb:bb:bb:bb static-black-hole
MINOR: SVCMGR #8007 Cannot modify proxy arp - black-hole mac not configured on service
In that case, the AS-MAC address needs to be removed from the proxy-ARP configuration, as follows:
# on PE-2, PE-3:
configure
service
vpls "VPLS 1"
proxy-arp
shutdown
dup-detect window 3 num-moves 5 hold-down max
exit
Then, the static black-hole MAC address can be created as follows:
# on PE-2, PE-3:
configure
service
vpls "VPLS 1"
static-mac
mac 00:00:bb:bb:bb:bb create black-hole
After the conditional static black-hole MAC address is configured, duplicate IP address detection cannot be configured with AS-MAC address, unless the static-black-hole option is added, as follows:
*A:PE-3>config>service>vpls>proxy-arp# dup-detect window 3 num-moves 5 hold-down max anti-spoof-mac 00:00:bb:bb:bb:bb
MINOR: SVCMGR #8007 Cannot modify proxy arp - conditional static mac configured on service
When the static black-hole MAC 00:00:bb:bb:bb:bb is configured, the AS-MAC address can only be configured with the static-black-hole option in VPLS 1 on PE-2 and PE-3, as follows:
# on PE-2, PE-3:
configure
service
vpls "VPLS 1"
static-mac
mac 00:00:bb:bb:bb:bb create black-hole
exit
proxy-arp
dup-detect window 3 num-moves 5 hold-down max
anti-spoof-mac 00:00:bb:bb:bb:bb static-black-hole
dynamic-arp-populate
static 172.16.0.20 00:00:02:20:20:20
no shutdown
exit
When the AS-MAC address is configured with the static black-hole option, the AS-MAC will be added not only to the MAC DB, but also to the FDB as CStatic, and associated with a black-hole endpoint, as follows:
*A:PE-3# show service id 1 fdb mac 00:00:bb:bb:bb:bb
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:bb:bb:bb:bb black-hole CStatic: 03/24/21 16:13:59
P
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
Any frame with MAC DA equal to the AS-MAC with static black-hole will be dropped, regardless of the ingress endpoint and without any need for a filter. This mechanism is the only way to filter MAC DAs on EVPN endpoints, because MAC filters cannot be configured on EVPN endpoints.
The AS-MAC with static black-hole will be advertised in EVPN with the sticky/static bit set, as follows:
# on PE-3:
75 2021/03/24 16:13:58.752 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:bb:bb:bb:bb, IP len: 0, IP: NULL, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
When a duplicate IP address is detected, the EVPN-MAC update contains the IP address 172.16.0.10, as follows:
90 2021/03/24 16:15:36.266 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 93
Flag: 0x90 Type: 14 Len: 48 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.3
Type: EVPN-MAC Len: 37 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:bb:bb:bb:bb, IP len: 4, IP: 172.16.0.10, label1: 8388544
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1
bgp-tunnel-encap:MPLS
mac-mobility:Seq:0/Static
"
The local CEs receive a GARP update with the AS-MAC address. The ARP table of CE-30 and CE-31 have an entry for the duplicate IP address 172.16.0.10 with the AS-MAC address 00:00:bb:bb:bb:bb, as follows:
*A:PE-3# show router 10 arp
===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.0.10 00:00:bb:bb:bb:bb 03h43m02s Dyn[I] int-CE-30-PE-3
172.16.0.30 00:00:aa:aa:aa:aa 00h00m00s Oth[I] int-CE-30-PE-3
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================
*A:PE-3# show router 11 arp
===============================================================================
ARP Table (Service: 11)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.0.10 00:00:bb:bb:bb:bb 03h47m43s Dyn[I] int-CE-31-PE-3
172.16.0.31 00:00:03:31:31:31 00h00m00s Oth[I] int-CE-31-PE-3
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================
CE-30 and CE-31 cannot reach CE-10 or CE-11, because the MAC DA will be the AS-MAC address and all traffic to this MAC DA is black-holed instead of forwarded to SAP 1/2/3:1 toward CE-10 or CE-11. When 1000 ICMP packets are sent by CE-30, they arrive in SAP 1/2/1:1 on PE-3 and are then discarded, as follows:
*A:PE-3# clear port 1/[1..2]/[1..4] statistics
*A:PE-3# ping router 10 172.16.0.10 rapid count 1000
PING 172.16.0.10 56 data bytes
---snip---
---- 172.16.0.10 PING Statistics ----
1000 packets transmitted, 0 packets received, 100% packet loss
*A:PE-3# show port 1/[1..2]/[1..4] statistics
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/2 13 1274
13 1274
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/1/3 16 1537
16 1537
===============================================================================
===============================================================================
Port Statistics on Slot 1
===============================================================================
Port Ingress Packets Ingress Octets
Id Egress Packets Egress Octets
-------------------------------------------------------------------------------
1/2/1 1000 106000
0 0
===============================================================================
No packets were forwarded to SAP 1/2/3:1 toward MTU-1; therefore, there are no statistics for port 1/2/3.
Conclusion
Static black-hole MAC addresses can be applied in EVPN for security as a scalable alternative to MAC filters. Static black-hole MAC addresses are programmed in the FDB and all frames with MAC DA equal to the static black-hole MAC address are dropped, regardless of how the frame arrived at the system (SAP/SDP-binding or EVPN endpoint). Also, static black-hole MAC addresses are treated like protected MAC addresses and, in combination with RPS(-DF), filtering on MAC SA is performed in the data plane. Black-hole MAC addresses can be an option for an AS-MAC address in services with proxy-ARP/ND enabled, which simplifies the configuration because MAC filters are not required.