Advertising ARP for FDB Entries Only in EVPN L3 All-Active Multihoming
This chapter provides information about advertising Address Resolution Protocol (ARP) for MAC entries in EVPN L3 all-active multihoming.
Topics in this chapter include:
Applicability
The information and the configuration in this chapter are based on SR OS Release 24.3.R1. Advertising ARP or ND for MAC entries in the FDB only in EVPN L3 all-active multihoming (AA MH) is supported in SR OS Release 23.10.R3 and later.
Overview
Example topology shows an EVPN L3 service with AA MH on PE-2 and PE-3. Multiple CEs are connected to VPLS-1 on PE-1, which is multihomed to PE-2 and PE-3.
The CEs are connected to VPLS-1 on PE-1; an EVPN L3 service with all-active multihoming is configured on PE-2 and PE-3. When CE-11 sends an ARP request to retrieve the MAC address for IP address 172.16.1.12 of CE-12, these ARP requests may be hashed toward the DF or NDF in the AA MH "AA-ES-23". For example, the ARP request is hashed toward the DF PE-3, so the CE-11 MAC address 00:00:5e:00:53:11 is dynamically learned on PE-3. When CE-11 sends another ARP request, the ARP request may be hashed toward NDF PE-2, so the CE-11 MAC address 00:00:5e:00:53:11 is dynamically learned on PE-2 instead of PE-3.
If no previous EVPN MAC/IP or MAC-only route for MAC address 00:00:5e:00:53:11 was advertised with the ESI 01:00:00:00:00:23:00:00:00:01 of AA-ES-23, ARP messages trigger the advertisement of EVPN MAC/IP routes with ESI-0 because, at the time of advertisement, the router has not yet determined the ESI associated with the learned MAC address. As a result, the advertised EVPN MAC/IP routes may be flagged as MAC moves, even though the MAC address remains within the ES SAPs. When this happens, the MAC mobility sequence number is incremented and eventually, the CE-11 MAC address 00:00:5e:00:53:11 may be marked as duplicate, because the MAC address is bouncing between the MH PEs.
This occasional MAC mobility can be prevented by configuring arp-nd-only-with-fdb-advertisement in the VPLS "BD-1" on PE-2 and PE-3. With this configuration, EVPN MAC/IP routes for ARP entries are only advertised when the MAC address is programmed as FDB entry and with ESI 01:00:00:00:00:23:00:00:00:01, so the MAC address is not subject to mobility.
Configuration
The initial configuration on the PEs includes the following:
- Cards, MDAs, ports
- LAG-1 on PE-1, PE-2, PE-3
- Router interfaces between PE-2 and PE-3
- SR-ISIS between PE-2 and PE-3
BGP is configured for the EVPN address family between PE-2 and PE-3, as follows:
# on PE-2:
configure {
router "Base" {
autonomous-system 64500
bgp {
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
peer-ip-tracking true
split-horizon true
rapid-update {
evpn true
}
group "internal" {
peer-as 64500
family {
evpn true
}
}
neighbor "192.0.2.3" { # on PE-3: 192.0.2.2
group "internal"
}
Initial service configuration
On PE-1, VPLS-1 is configured with different SAPs for each connected CE and one SAP using LAG-1 toward the PEs:
# on PE-1:
configure {
service {
vpls "VPLS-1" {
admin-state enable
service-id 1
customer "1"
sap 1/1/c4/1:1 {
description "SAP to CE-12"
}
sap 1/1/c6/1:1 {
description "SAP to CE-13"
}
sap 1/1/c8/1:1 {
description "SAP to CE-14"
}
sap 1/1/c10/1:1 {
description "SAP to CE-11"
}
sap 1/1/c12/1:1 {
description "SAP to CE-15"
}
sap 1/1/c14/1:1 {
description "SAP to CE-16"
}
sap lag-1:1 {
description "SAP to PEs"
}
}
- Ethernet segment "AA-ES-23" associated with LAG 1
- VPLS "BD-1" with SAP using LAG 1
- VPRN-10 with interface "int-BD-1" using VPLS "BD-1".
# on PE-2, PE-3 (identical):
configure {
service {
system {
bgp {
evpn {
ethernet-segment "AA-ES-23" {
admin-state enable
esi 0x01000000002300000001
multi-homing-mode all-active
df-election {
es-activation-timer 3
}
association {
lag "lag-1" {
}
}
}
}
}
}
vpls "BD-1" {
admin-state enable
service-id 1
customer "1"
routed-vpls {
}
bgp 1 {
}
bgp-evpn {
evi 1
mpls 1 {
admin-state enable
auto-bind-tunnel {
resolution any
}
}
}
sap lag-1:1 {
}
}
vprn "VPRN-10" {
admin-state enable
service-id 10
customer "1"
interface "int-BD-1" {
ipv4 {
primary {
address 172.16.1.223
prefix-length 24
}
neighbor-discovery {
learn-unsolicited true
}
vrrp 1 {
backup [172.16.1.223]
owner true
passive true
}
}
vpls "BD-1" {
evpn {
arp {
learn-dynamic false
advertise dynamic {
}
}
}
}
}
}
With ipv4 neighbor-discovery learn-unsolicited true configured in VPRN-10, the ARP application learns new entries based on received ARP messages, such as Gratuitous ARP (GARP), ARP request, or ARP reply. The arp advertise dynamic command enables the advertisement of MAC/IP routes for the dynamic ARP entries.
Normal operation - CE MAC entry in FDB and EVPN MAC routes with ESI
CE-11 is multihomed to the R-VPLS on PE-2 and PE-3. When CE-11 sends an ARP request, it may be hashed to PE-3 and PE-3 learns the MAC address of CE-11 dynamically (L), as follows:
[/]
A:admin@PE-3# show service id "BD-1" fdb mac 00:00:5e:00:53:11
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:5e:00:53:11 sap:lag-1:1 LT/330 11/21/24 14:36:54
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================
With ipv4 neighbor-discovery learn-unsolicited true configured in VPRN-10 on PE-3, the ARP application learns the IP address and MAC address of CE-11 from the ARP request and adds a dynamic entry for CE-11:
[/]
A:admin@PE-3# show router service-name "VPRN-10" arp
===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.1.11 00:00:5e:00:53:11 03h54m58s Dyn[I] int-BD-1
172.16.1.223 00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================
PE-3 advertises an EVPN MAC-only and an EVPN MAC/IP route for MAC address 00:00:5e:00:53:11 with ESI 01:00:00:00:00:23:00:00:00:01 to PE-2:
[/]
A:admin@PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
BGP Router ID:192.0.2.2 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.3:1 00:00:5e:00:53:11 01:00:00:00:00:23:00:00:00:01
0 Seq:0 LABEL 524286
n/a
192.0.2.3
u*>i 192.0.2.3:1 00:00:5e:00:53:11 01:00:00:00:00:23:00:00:00:01
0 Seq:0 LABEL 524286
172.16.1.11
192.0.2.3
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
PE-3 does not receive any EVPN MAC routes for MAC address 00:00:5e:00:53:11 from PE-2, as follows:
[/]
A:admin@PE-3# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
BGP Router ID:192.0.2.3 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
No Matching Entries Found.
===============================================================================
The ARP table on PE-2 shows an EVPN entry for CE-11, which is added upon receiving an EVPN MAC/IP route:
[/]
A:admin@PE-2# show router service-name "VPRN-10" arp
===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.1.11 00:00:5e:00:53:11 00h00m00s Evp[I] int-BD-1
172.16.1.223 00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================
The FDB on PE-2 shows an EVPN entry for MAC address 00:00:5e:00:53:11:
[/]
A:admin@PE-2# show service id "BD-1" fdb mac 00:00:5e:00:53:11
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:5e:00:53:11 sap:lag-1:1 Evpn 11/21/24 14:36:54
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================
In this scenario, the advertised MAC/IP routes have ESI 01:00:00:00:00:23:00:00:00:01. Different ARP requests from CE-11 may get hashed toward the DF or the NDF, but that will not be considered as MAC moves because the MAC address 00:00:5e:00:53:11 stays within the ES SAPs.
MAC move scenario - no CE MAC entry in FDB and EVPN MAC routes with ESI-0
To simulate a situation where no MAC learning takes place, the FDB table size is reduced to 1, as follows:
# on PE-2, PE-3:
configure {
service {
vpls "BD-1"
fdb {
table {
size 1
With the FDB table size reduced to 1, the CE-11 MAC address 00:00:5e:00:53:11 is not programmed in the FDB of PE-3:
[/]
A:admin@PE-3# show service id "BD-1" fdb detail
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:5e:00:01:01 cpm Intf 11/21/24 14:36:50
1 00:02:fe:ff:ff:3e mpls-1: EvpnS:P 11/21/24 14:36:52
192.0.2.2:524286
isis:524290
1 00:03:fe:ff:ff:3e cpm Intf 11/21/24 14:36:50
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================
However, the FDB on PE-2 contains an EVPN entry for the CE-11 MAC address 00:00:5e:00:53:11:
[/]
A:admin@PE-2# show service id "BD-1" fdb detail
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1 00:00:5e:00:01:01 cpm Intf 11/21/24 15:14:01
1 00:00:5e:00:53:11 mpls-1: Evpn 11/21/24 15:14:32
192.0.2.3:524286
isis:524290
1 00:02:fe:ff:ff:3e cpm Intf 11/21/24 14:36:07
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================
Even though PE-3 did not program MAC address 00:00:5e:00:53:11 to the FDB of BD-1, PE-3 advertised the following EVPN MAC/IP route with ESI-0 (instead of ESI 01:00:00:00:00:23:00:00:00:01) to PE-2:
[/]
A:admin@PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
BGP Router ID:192.0.2.2 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.3:1 00:00:5e:00:53:11 ESI-0
0 Seq:0 LABEL 524286
172.16.1.11
192.0.2.3
-------------------------------------------------------------------------------
Routes : 1
===============================================================================
The ARP table on PE-3 contains a dynamic entry after receiving the ARP request from CE-11:
[/]
A:admin@PE-3# show router service-name "VPRN-10" arp
===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.1.11 00:00:5e:00:53:11 03h58m48s Dyn[I] int-BD-1
172.16.1.223 00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================
The ARP table on PE-2 shows an EVPN entry for MAC address 00:00:5e:00:53:11, as follows:
[/]
A:admin@PE-2# show router service-name "VPRN-10" arp
===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.1.11 00:00:5e:00:53:11 00h00m00s Evp[I] int-BD-1
172.16.1.223 00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================
In this scenario, the MAC/IP routes are advertised with ESI-0. Different ARP requests from CE-11 may get hashed toward the DF or the NDF, which could be wrongly considered as MAC moves even though the MAC address stays within the ES SAPs (because the ESI is not taken into account).
Preventing MAC move - EVPN MAC routes for FDB entries only
When the PEs only advertise EVPN MAC routes for MAC addresses that are programmed in the FDB, the EVPN MAC routes are advertised with the correct ESI and there are no incorrect MAC mobility events. On PE-2 and PE-3, BD-1 is configured as follows:
# on PE-2, PE-3:
configure {
service {
vpls "BD-1" {
admin-state enable
service-id 1
customer "1"
fdb {
table {
size 1
}
}
routed-vpls {
}
bgp 1 {
}
bgp-evpn {
evi 1
routes {
mac-ip {
arp-nd-only-with-fdb-advertisement true
}
}
mpls 1 {
admin-state enable
auto-bind-tunnel {
resolution any
}
}
}
sap lag-1:1 {
}
When PE-3 receives an ARP request from CE-11, it adds a dynamic entry to the ARP table for VPRN-10, as follows:
[/]
A:admin@PE-3# show router service-name "VPRN-10" arp
===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.1.11 00:00:5e:00:53:11 03h59m33s Dyn[I] int-BD-1
172.16.1.223 00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================
When PE-3 receives an ARP request from CE-11, it does not program MAC address 00:00:5e:00:53:11 in the FDB because the FDB table size is limited to 1:
[/]
A:admin@PE-2# show service id "BD-1" fdb mac 00:00:5e:00:53:11
===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================
PE-3 does not advertise an EVPN MAC route for a non-existing entry in the FDB, so PE-2 does not receive any EVPN MAC routes for MAC address 00:00:5e:00:53:11, as follows:
[/]
A:admin@PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
BGP Router ID:192.0.2.2 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
No Matching Entries Found.
===============================================================================
The ARP table for VPRN-10 on PE-2 does not contain an entry for CE-11 because PE-2 did not receive any EVPN MAC route for MAC address 00:00:5e:00:53:11 from PE-3:
[/]
A:admin@PE-2# show router service-name "VPRN-10" arp
===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address MAC Address Expiry Type Interface
-------------------------------------------------------------------------------
172.16.1.223 00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 1
===============================================================================
The preceding example only shows that EVPN MAC routes are not advertised when the CE-11 MAC is not programmed in the FDB. However, when the CE MAC address is learned in the FDB, the EVPN MAC routes are advertised with ESI 01:00:00:00:00:23:00:00:00:01, as in the normal operation.
Conclusion
In EVPN L3 services with all-active multihoming, occasional MAC mobility can be prevented when EVPN MAC routes are only advertised for MAC addresses that are programmed in the FDB.