Advertising ARP for FDB Entries Only in EVPN L3 All-Active Multihoming

This chapter provides information about advertising Address Resolution Protocol (ARP) for MAC entries in EVPN L3 all-active multihoming.

Topics in this chapter include:

Applicability

The information and the configuration in this chapter are based on SR OS Release 24.3.R1. Advertising ARP or ND for MAC entries in the FDB only in EVPN L3 all-active multihoming (AA MH) is supported in SR OS Release 23.10.R3 and later.

Overview

Example topology shows an EVPN L3 service with AA MH on PE-2 and PE-3. Multiple CEs are connected to VPLS-1 on PE-1, which is multihomed to PE-2 and PE-3.

Figure 1. Example topology

The CEs are connected to VPLS-1 on PE-1; an EVPN L3 service with all-active multihoming is configured on PE-2 and PE-3. When CE-11 sends an ARP request to retrieve the MAC address for IP address 172.16.1.12 of CE-12, these ARP requests may be hashed toward the DF or NDF in the AA MH "AA-ES-23". For example, the ARP request is hashed toward the DF PE-3, so the CE-11 MAC address 00:00:5e:00:53:11 is dynamically learned on PE-3. When CE-11 sends another ARP request, the ARP request may be hashed toward NDF PE-2, so the CE-11 MAC address 00:00:5e:00:53:11 is dynamically learned on PE-2 instead of PE-3.

If no previous EVPN MAC/IP or MAC-only route for MAC address 00:00:5e:00:53:11 was advertised with the ESI 01:00:00:00:00:23:00:00:00:01 of AA-ES-23, ARP messages trigger the advertisement of EVPN MAC/IP routes with ESI-0 because, at the time of advertisement, the router has not yet determined the ESI associated with the learned MAC address. As a result, the advertised EVPN MAC/IP routes may be flagged as MAC moves, even though the MAC address remains within the ES SAPs. When this happens, the MAC mobility sequence number is incremented and eventually, the CE-11 MAC address 00:00:5e:00:53:11 may be marked as duplicate, because the MAC address is bouncing between the MH PEs.

This occasional MAC mobility can be prevented by configuring arp-nd-only-with-fdb-advertisement in the VPLS "BD-1" on PE-2 and PE-3. With this configuration, EVPN MAC/IP routes for ARP entries are only advertised when the MAC address is programmed as FDB entry and with ESI 01:00:00:00:00:23:00:00:00:01, so the MAC address is not subject to mobility.

Configuration

The initial configuration on the PEs includes the following:

  • Cards, MDAs, ports
  • LAG-1 on PE-1, PE-2, PE-3
  • Router interfaces between PE-2 and PE-3
  • SR-ISIS between PE-2 and PE-3

BGP is configured for the EVPN address family between PE-2 and PE-3, as follows:

# on PE-2:
configure {
    router "Base" {
        autonomous-system 64500
        bgp {
            vpn-apply-export true
            vpn-apply-import true
            rapid-withdrawal true
            peer-ip-tracking true
            split-horizon true
            rapid-update {
                evpn true
            }
            group "internal" {
                peer-as 64500
                family {
                    evpn true
                }
            }
            neighbor "192.0.2.3" {        # on PE-3: 192.0.2.2
                group "internal"
            }

Initial service configuration

On PE-1, VPLS-1 is configured with different SAPs for each connected CE and one SAP using LAG-1 toward the PEs:

# on PE-1:
configure {
    service {
        vpls "VPLS-1" {
            admin-state enable
            service-id 1
            customer "1"
            sap 1/1/c4/1:1 {
                description "SAP to CE-12"
            }
            sap 1/1/c6/1:1 {
                description "SAP to CE-13"
            }
            sap 1/1/c8/1:1 {
                description "SAP to CE-14"
            }
            sap 1/1/c10/1:1 {
                description "SAP to CE-11"
            }
            sap 1/1/c12/1:1 {
                description "SAP to CE-15"
            }
            sap 1/1/c14/1:1 {
                description "SAP to CE-16"
            }
            sap lag-1:1 {
                description "SAP to PEs"
            }
        }
On PE-2 and PE-3, the service configuration is as follows:
  • Ethernet segment "AA-ES-23" associated with LAG 1
  • VPLS "BD-1" with SAP using LAG 1
  • VPRN-10 with interface "int-BD-1" using VPLS "BD-1".
# on PE-2, PE-3 (identical):
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "AA-ES-23" {
                        admin-state enable
                        esi 0x01000000002300000001
                        multi-homing-mode all-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            lag "lag-1" {
                            }
                        }
                    }
                }
            }
        }
        vpls "BD-1" {
            admin-state enable
            service-id 1
            customer "1"
            routed-vpls {
            }
            bgp 1 {
            }
            bgp-evpn {
                evi 1
                mpls 1 {
                    admin-state enable
                    auto-bind-tunnel {
                        resolution any
                    }
                }
            }
            sap lag-1:1 {
            }
        }
        vprn "VPRN-10" {
            admin-state enable
            service-id 10
            customer "1"
            interface "int-BD-1" {
                ipv4 {
                    primary {
                        address 172.16.1.223
                        prefix-length 24
                    }
                    neighbor-discovery {
                        learn-unsolicited true
                    }
                    vrrp 1 {
                        backup [172.16.1.223]
                        owner true
                        passive true
                    }
                }
                vpls "BD-1" {
                    evpn {
                        arp {
                            learn-dynamic false
                            advertise dynamic {
                            }
                        }
                    }
                }
            }
        }

With ipv4 neighbor-discovery learn-unsolicited true configured in VPRN-10, the ARP application learns new entries based on received ARP messages, such as Gratuitous ARP (GARP), ARP request, or ARP reply. The arp advertise dynamic command enables the advertisement of MAC/IP routes for the dynamic ARP entries.

Normal operation - CE MAC entry in FDB and EVPN MAC routes with ESI

CE-11 is multihomed to the R-VPLS on PE-2 and PE-3. When CE-11 sends an ARP request, it may be hashed to PE-3 and PE-3 learns the MAC address of CE-11 dynamically (L), as follows:

[/]
A:admin@PE-3# show service id "BD-1" fdb mac 00:00:5e:00:53:11

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:53:11 sap:lag-1:1             LT/330   11/21/24 14:36:54
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

With ipv4 neighbor-discovery learn-unsolicited true configured in VPRN-10 on PE-3, the ARP application learns the IP address and MAC address of CE-11 from the ARP request and adds a dynamic entry for CE-11:

[/]
A:admin@PE-3# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 03h54m58s Dyn[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

PE-3 advertises an EVPN MAC-only and an EVPN MAC/IP route for MAC address 00:00:5e:00:53:11 with ESI 01:00:00:00:00:23:00:00:00:01 to PE-2:

[/]
A:admin@PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         00:00:5e:00:53:11 01:00:00:00:00:23:00:00:00:01
      0                   Seq:0             LABEL 524286
                          n/a
                          192.0.2.3

u*>i  192.0.2.3:1         00:00:5e:00:53:11 01:00:00:00:00:23:00:00:00:01
      0                   Seq:0             LABEL 524286
                          172.16.1.11
                          192.0.2.3

-------------------------------------------------------------------------------
Routes : 2
===============================================================================

PE-3 does not receive any EVPN MAC routes for MAC address 00:00:5e:00:53:11 from PE-2, as follows:

[/]
A:admin@PE-3# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.3        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
No Matching Entries Found.
===============================================================================

The ARP table on PE-2 shows an EVPN entry for CE-11, which is added upon receiving an EVPN MAC/IP route:

[/]
A:admin@PE-2# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 00h00m00s Evp[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

The FDB on PE-2 shows an EVPN entry for MAC address 00:00:5e:00:53:11:

[/]
A:admin@PE-2# show service id "BD-1" fdb mac 00:00:5e:00:53:11

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:53:11 sap:lag-1:1             Evpn     11/21/24 14:36:54
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

In this scenario, the advertised MAC/IP routes have ESI 01:00:00:00:00:23:00:00:00:01. Different ARP requests from CE-11 may get hashed toward the DF or the NDF, but that will not be considered as MAC moves because the MAC address 00:00:5e:00:53:11 stays within the ES SAPs.

MAC move scenario - no CE MAC entry in FDB and EVPN MAC routes with ESI-0

To simulate a situation where no MAC learning takes place, the FDB table size is reduced to 1, as follows:

# on PE-2, PE-3:
configure {
    service {
        vpls "BD-1"
            fdb {
                table {
                    size 1            

With the FDB table size reduced to 1, the CE-11 MAC address 00:00:5e:00:53:11 is not programmed in the FDB of PE-3:

[/]
A:admin@PE-3# show service id "BD-1" fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:01:01 cpm                     Intf     11/21/24 14:36:50
1          00:02:fe:ff:ff:3e mpls-1:                 EvpnS:P  11/21/24 14:36:52
                             192.0.2.2:524286
           isis:524290
1          00:03:fe:ff:ff:3e cpm                     Intf     11/21/24 14:36:50
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

However, the FDB on PE-2 contains an EVPN entry for the CE-11 MAC address 00:00:5e:00:53:11:

[/]
A:admin@PE-2# show service id "BD-1" fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:01:01 cpm                     Intf     11/21/24 15:14:01
1          00:00:5e:00:53:11 mpls-1:                 Evpn     11/21/24 15:14:32
                             192.0.2.3:524286
           isis:524290
1          00:02:fe:ff:ff:3e cpm                     Intf     11/21/24 14:36:07
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

Even though PE-3 did not program MAC address 00:00:5e:00:53:11 to the FDB of BD-1, PE-3 advertised the following EVPN MAC/IP route with ESI-0 (instead of ESI 01:00:00:00:00:23:00:00:00:01) to PE-2:

[/]
A:admin@PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         00:00:5e:00:53:11 ESI-0
      0                   Seq:0             LABEL 524286
                          172.16.1.11
                          192.0.2.3

-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The ARP table on PE-3 contains a dynamic entry after receiving the ARP request from CE-11:

[/]
A:admin@PE-3# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 03h58m48s Dyn[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

The ARP table on PE-2 shows an EVPN entry for MAC address 00:00:5e:00:53:11, as follows:

[/]
A:admin@PE-2# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 00h00m00s Evp[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

In this scenario, the MAC/IP routes are advertised with ESI-0. Different ARP requests from CE-11 may get hashed toward the DF or the NDF, which could be wrongly considered as MAC moves even though the MAC address stays within the ES SAPs (because the ESI is not taken into account).

Preventing MAC move - EVPN MAC routes for FDB entries only

When the PEs only advertise EVPN MAC routes for MAC addresses that are programmed in the FDB, the EVPN MAC routes are advertised with the correct ESI and there are no incorrect MAC mobility events. On PE-2 and PE-3, BD-1 is configured as follows:

# on PE-2, PE-3:
configure {
    service {
        vpls "BD-1" {
            admin-state enable
            service-id 1
            customer "1"
            fdb {
                table {
                    size 1
                }
            }
            routed-vpls {
            }
            bgp 1 {
            }
            bgp-evpn {
                evi 1
                routes {
                    mac-ip {
                        arp-nd-only-with-fdb-advertisement true
                    }
                }
                mpls 1 {
                    admin-state enable
                    auto-bind-tunnel {
                        resolution any
                    }
                }
            }
            sap lag-1:1 {
            }
    

When PE-3 receives an ARP request from CE-11, it adds a dynamic entry to the ARP table for VPRN-10, as follows:

[/]
A:admin@PE-3# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 03h59m33s Dyn[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

When PE-3 receives an ARP request from CE-11, it does not program MAC address 00:00:5e:00:53:11 in the FDB because the FDB table size is limited to 1:

[/]
A:admin@PE-2# show service id "BD-1" fdb mac 00:00:5e:00:53:11

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

PE-3 does not advertise an EVPN MAC route for a non-existing entry in the FDB, so PE-2 does not receive any EVPN MAC routes for MAC address 00:00:5e:00:53:11, as follows:

[/]
A:admin@PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
No Matching Entries Found.
===============================================================================

The ARP table for VPRN-10 on PE-2 does not contain an entry for CE-11 because PE-2 did not receive any EVPN MAC route for MAC address 00:00:5e:00:53:11 from PE-3:

[/]
A:admin@PE-2# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 1
===============================================================================

The preceding example only shows that EVPN MAC routes are not advertised when the CE-11 MAC is not programmed in the FDB. However, when the CE MAC address is learned in the FDB, the EVPN MAC routes are advertised with ESI 01:00:00:00:00:23:00:00:00:01, as in the normal operation.

Conclusion

In EVPN L3 services with all-active multihoming, occasional MAC mobility can be prevented when EVPN MAC routes are only advertised for MAC addresses that are programmed in the FDB.