Advertising ARP for FDB Entries Only in EVPN L3 All-Active Multihoming

This chapter provides information about advertising Address Resolution Protocol (ARP) for MAC entries in EVPN L3 all-active multihoming.

Topics in this chapter include:

Applicability

The information and the configuration in this chapter are based on SR OS Release 24.3.R1. Advertising ARP or ND for MAC entries in the FDB only in EVPN L3 all-active multihoming (AA MH) is supported in SR OS Release 23.10.R3 and later.

Overview

Example topology shows an EVPN L3 service with AA MH on PE-2 and PE-3. Multiple CEs are connected to VPLS-1 on PE-1, which is multihomed to PE-2 and PE-3.

Figure 1. Example topology

The CEs are connected to VPLS-1 on PE-1; an EVPN L3 service with all-active multihoming is configured on PE-2 and PE-3. When CE-11 sends an ARP request to retrieve the MAC address for IP address 172.16.1.12 of CE-12, these ARP requests may be hashed toward the DF or NDF in the AA MH "AA-ES-23". For example, the ARP request is hashed toward the DF PE-3, so the CE-11 MAC address 00:00:5e:00:53:11 is dynamically learned on PE-3. When CE-11 sends another ARP request, the ARP request may be hashed toward NDF PE-2, so the CE-11 MAC address 00:00:5e:00:53:11 is dynamically learned on PE-2 instead of PE-3.

If no previous EVPN MAC/IP or MAC-only route for MAC address 00:00:5e:00:53:11 was advertised with the ESI 01:00:00:00:00:23:00:00:00:01 of AA-ES-23, ARP messages trigger the advertisement of EVPN MAC/IP routes with ESI-0 because, at the time of advertisement, the router has not yet determined the ESI associated with the learned MAC address. As a result, the advertised EVPN MAC/IP routes may be flagged as MAC moves, even though the MAC address remains within the ES SAPs. When this happens, the MAC mobility sequence number is incremented and eventually, the CE-11 MAC address 00:00:5e:00:53:11 may be marked as duplicate, because the MAC address is bouncing between the MH PEs.

This occasional MAC mobility can be prevented by configuring arp-nd-only-with-fdb-advertisement in the VPLS "BD-1" on PE-2 and PE-3. With this configuration, EVPN MAC/IP routes for ARP entries are only advertised when the MAC address is programmed as FDB entry and with ESI 01:00:00:00:00:23:00:00:00:01, so the MAC address is not subject to mobility.

Configuration

The initial configuration on the PEs includes the following:

  • Cards, MDAs, ports
  • LAG-1 on PE-1, PE-2, PE-3
  • Router interfaces between PE-2 and PE-3
  • SR-ISIS between PE-2 and PE-3

BGP is configured for the EVPN address family between PE-2 and PE-3, as follows:

# on PE-2:
configure
    router Base
        autonomous-system 64500
        bgp
            vpn-apply-import
            vpn-apply-export
            enable-peer-tracking
            rapid-withdrawal
            split-horizon
            rapid-update evpn
            group "internal"
                family evpn
                peer-as 64500
                neighbor 192.0.2.3        # on PE-3: 192.0.2.2
                exit
            exit

Initial service configuration

On PE-1, VPLS-1 is configured with different SAPs for each connected CE and one SAP using LAG-1 toward the PEs:

# on PE-1:
configure
    service
        vpls 1 name "VPLS-1" customer 1 create
            sap 1/1/c10/1:1 create
                description "SAP to CE-11"
                no shutdown
            exit
            sap 1/1/c4/1:1 create
                description "SAP to CE-12"
                no shutdown
            exit
            ---snip---          # SAPs to other CEs
            sap lag-1:1 create
                description "SAP to PEs"
                no shutdown
            exit
            no shutdown
On PE-2 and PE-3, the service configuration is as follows:
  • Ethernet segment "AA-ES-23" associated with LAG 1
  • VPLS "BD-1" with SAP using LAG 1
  • VPRN-10 with interface "int-BD-1" using VPLS "BD-1".
# on PE-2, PE-3 (identical):
configure
    service
        system
            bgp-evpn
                ethernet-segment "AA-ES-23" create
                    esi 01:00:00:00:00:23:00:00:00:01
                    es-activation-timer 3
                    service-carving
                        mode auto
                    exit
                    multi-homing all-active
                    lag 1
                    no shutdown
                exit
            exit
        exit
        vpls 1 name "BD-1" customer 1 create
            allow-ip-int-bind
            bgp 1
            exit
            bgp-evpn
                evi 1
                mpls bgp 1
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            sap lag-1:1 create
            exit
            no shutdown
        exit
        vprn 10 name "VPRN-10" customer 1 create
            interface "int-BD-1" create
                address 172.16.1.223/24
                arp-learn-unsolicited
                vrrp 1 owner passive
                    backup 172.16.1.223
                exit
                vpls "BD-1"
                    evpn
                        arp 
                            no learn-dynamic    # required for advertise command
                            advertise dynamic 
                        exit
                    exit
                exit
            exit
            no shutdown
        exit

With arp-learn-unsolicited enabled in VPRN-10, the ARP application learns new entries based on received ARP messages, such as Gratuitous ARP (GARP), ARP request, or ARP reply. The arp advertise dynamic command enables the advertisement of MAC/IP routes for the dynamic ARP entries. The advertise command must be used along with the no learn-dynamic command.

Normal operation - CE MAC entry in FDB and EVPN MAC routes with ESI

CE-11 is multihomed to the R-VPLS on PE-2 and PE-3. When CE-11 sends an ARP request, it may be hashed to PE-3 and PE-3 learns the MAC address of CE-11 dynamically (L), as follows:

*A:PE-3# show service id "BD-1" fdb mac 00:00:5e:00:53:11

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:53:11 sap:lag-1:1             L/60     11/27/24 08:30:48
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

With arp-learn-unsolicited enabled in VPRN-10 on PE-3, the ARP application learns the IP address and MAC address of CE-11 from the ARP request and adds a dynamic entry for CE-11:

*A:PE-3# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 03h59m48s Dyn[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

PE-3 advertises an EVPN MAC-only and an EVPN MAC/IP route for MAC address 00:00:5e:00:53:11 with ESI 01:00:00:00:00:23:00:00:00:01 to PE-2:

*A:PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         00:00:5e:00:53:11 01:00:00:00:00:23:00:00:00:01
      0                   Seq:0             LABEL 524285
                          n/a
                          192.0.2.3

u*>i  192.0.2.3:1         00:00:5e:00:53:11 01:00:00:00:00:23:00:00:00:01
      0                   Seq:0             LABEL 524285
                          172.16.1.11
                          192.0.2.3

-------------------------------------------------------------------------------
Routes : 2
===============================================================================

PE-3 does not receive any EVPN MAC routes for MAC address 00:00:5e:00:53:11 from PE-2, as follows:

*A:PE-3# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.3        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
No Matching Entries Found.
===============================================================================

The ARP table on PE-2 shows an EVPN entry for CE-11, which is added upon receiving an EVPN MAC/IP route:

*A:PE-2# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 00h00m00s Evp[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

The FDB on PE-2 shows an EVPN entry for MAC address 00:00:5e:00:53:11: 

*A:PE-2# show service id "BD-1" fdb mac 00:00:5e:00:53:11

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:53:11 sap:lag-1:1             Evpn     11/27/24 08:30:48
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

In this scenario, the advertised MAC/IP routes have ESI 01:00:00:00:00:23:00:00:00:01. Different ARP requests from CE-11 may get hashed toward the DF or the NDF, but that will not be considered as MAC moves because the MAC address 00:00:5e:00:53:11 stays within the ES SAPs.

MAC move scenario - no CE MAC entry in FDB and EVPN MAC routes with ESI-0

To simulate a situation where no MAC learning takes place, the FDB table size is reduced to 1, as follows:

# on PE-2, PE-3:
configure
    service
        vpls "BD-1"
            fdb-table-size 1

With the FDB table size reduced to 1, the CE-11 MAC address 00:00:5e:00:53:11 is not programmed in the FDB of PE-3:

*A:PE-3# show service id "BD-1" fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:01:01 cpm                     Intf     11/27/24 08:29:36
1          00:02:fe:ff:ff:3e mpls-1:                 EvpnS:P  11/27/24 08:29:42
                             192.0.2.2:524285
           isis:524290
1          00:03:fe:ff:ff:3e cpm                     Intf     11/27/24 08:29:36
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

However, the FDB on PE-2 contains an EVPN entry for the CE-11 MAC address 00:00:5e:00:53:11:

*A:PE-2# show service id "BD-1" fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:5e:00:01:01 cpm                     Intf     11/27/24 08:29:42
1          00:00:5e:00:53:11 mpls-1:                 Evpn     11/27/24 08:33:21
                             192.0.2.3:524285
           isis:524290
1          00:02:fe:ff:ff:3e cpm                     Intf     11/27/24 08:29:42
1          00:03:fe:ff:ff:3e mpls-1:                 EvpnS:P  11/27/24 08:29:44
                             192.0.2.3:524285
           isis:524290
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

Even though PE-3 did not program MAC address 00:00:5e:00:53:11 to the FDB of BD-1, PE-3 advertised the following EVPN MAC/IP route with ESI-0 (instead of ESI 01:00:00:00:00:23:00:00:00:01) to PE-2: 

*A:PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         00:00:5e:00:53:11 ESI-0
      0                   Seq:0             LABEL 524285
                          172.16.1.11
                          192.0.2.3

-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The ARP table on PE-3 contains a dynamic entry after receiving the ARP request from CE-11:

*A:PE-3# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 03h59m19s Dyn[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

The ARP table on PE-2 shows an EVPN entry for MAC address 00:00:5e:00:53:11, as follows:

*A:PE-2# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 00h00m00s Evp[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

In this scenario, the MAC/IP routes are advertised with ESI-0. Different ARP requests from CE-11 may get hashed toward the DF or the NDF, which could be wrongly considered as MAC moves even though the MAC address stays within the ES SAPs (because the ESI is not taken into account).

Preventing MAC move - EVPN MAC routes for FDB entries only

When the PEs only advertise EVPN MAC routes for MAC addresses that are programmed in the FDB, the EVPN MAC routes are advertised with the correct ESI and there are no incorrect MAC mobility events. On PE-2 and PE-3, BD-1 is configured as follows:

# on PE-2, PE-3:
configure
    service
        vpls "BD-1"
            allow-ip-int-bind
            exit
            fdb-table-size 1
            bgp 1
            exit
            bgp-evpn
                arp-nd-only-with-fdb-advertisement  # fails when BGP-EVPN is enabled
                evi 1
                mpls bgp 1
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            sap lag-1:1 create
            exit
            no shutdown

The following error message is raised when attempting to configure arp-nd-only-wifh-fdb-advertisement when bgp-evpn mpls bgp 1 is enabled:

*A:PE-2>config>service>vpls>bgp-evpn# arp-nd-only-with-fdb-advertisement
MINOR: SVCMGR #7886 cannot modify evpn - Evpn not shut

When PE-3 receives an ARP request from CE-11, it adds a dynamic entry to the ARP table for VPRN-10, as follows:

*A:PE-3# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.11     00:00:5e:00:53:11 03h59m46s Dyn[I] int-BD-1
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 2
===============================================================================

When PE-3 receives an ARP request from CE-11, it does not program MAC address 00:00:5e:00:53:11 in the FDB because the FDB table size is limited to 1:

*A:PE-3# show service id "BD-1" fdb mac 00:00:5e:00:53:11

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

PE-3 does not advertise an EVPN MAC route for a non-existing entry in the FDB, so PE-2 does not receive any EVPN MAC routes for MAC address 00:00:5e:00:53:11, as follows:

*A:PE-2# show router bgp routes evpn mac mac-address 00:00:5e:00:53:11
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
No Matching Entries Found.
===============================================================================

The ARP table for VPRN-10 on PE-2 does not contain an entry for CE-11 because PE-2 did not receive any EVPN MAC route for MAC address 00:00:5e:00:53:11 from PE-3:

*A:PE-2# show router service-name "VPRN-10" arp

===============================================================================
ARP Table (Service: 10)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
172.16.1.223    00:00:5e:00:01:01 00h00m00s Oth[I] int-BD-1
-------------------------------------------------------------------------------
No. of ARP Entries: 1
===============================================================================

The preceding example only shows that EVPN MAC routes are not advertised when the CE-11 MAC is not programmed in the FDB. However, when the CE MAC address is learned in the FDB, the EVPN MAC routes are advertised with ESI 01:00:00:00:00:23:00:00:00:01, as in the normal operation.

Conclusion

In EVPN L3 services with all-active multihoming, occasional MAC mobility can be prevented when EVPN MAC routes are only advertised for MAC addresses that are programmed in the FDB.