EVPN ARP/ND synchronization on Layer 3 interfaces

Note: This feature is supported on 7250 IXR (excluding mixed) and 7730 SXR platforms.

In typical access networks, the use of a single LAG to multi-home a CE to two Layer 3 PE routers is a common configuration. However, this approach presents challenges related to the synchronization of ARP/ND entries across the multihomed PEs, as illustrated in the following figure.

Figure 1. Multi-homed PEs with and without EVPN ARP/ND synchronization enabled

In the left-hand diagram, no synchronization of ARP/ND entries occurs between the PEs, which can lead to dropped traffic as follows:

Only the left-hand PE learns the ARP entry for 10.0.0.2 because of how the CE hashes traffic.
If downstream traffic arrives at the right-hand PE, it does not have a resolved entry for 10.0.0.2 and issues an ARP request.
If the ARP request is again hashed to the left-hand PE, the right-hand PE never receives a reply and is unable to resolve the 10.0.0.2 destination IP address.
As a result, the right-hand PE silently drops the downstream traffic.

In the right-hand diagram, this problem is addressed using synchronization of ARP/ND entries with EVPN between the multi-homed PEs (based on approach 2 in draft-ietf-bess-evpn-l3mh-proto). The solution operates as follows:

A BGP EVPN session is established between the PEs.
ARP/ND entries are synchronized between the two PEs using EVPN MAC/IP Advertisement routes carrying Ethernet Segment Identifier (ESI), ES‑import route target, and EVI route target information.
If downstream traffic arrives at the right-hand PE, it now has the ARP entry for 10.0.0.2 and can forward the traffic without issue.
IP-VRF Layer 3 interfaces are used to connect to the CE LAG ports, with no requirement for MAC-VRFs or IRBs on the PEs.

This solution applies to VXLAN, SRv6, MPLS BGP EVPN and IP-VPN services, and the default network instance. It supports up to four PEs on the same Ethernet Segment and interoperates with other platforms that implement the same standard.

Configuration prerequisites

As a prerequisite, multihomed PEs must have BGP EVPN enabled at the BGP level, either through a direct BGP neighbor relationship or via a route reflector (RR). This requirement applies whether or not EVPN is used in any service. This is necessary because EVPN is used to synchronize ARP/ND entries between the multihomed PEs.

Configuration

ARP/ND synchronization on Layer 3 interfaces requires configuration in two places:

The routing instance where synchronization is needed
The Ethernet Segments connected to the CEs for which synchronization is required

Configure Layer 3 EVPN ARP/ND synchronization and associated parameters in the network instance

To configure the synchronization of ARP and ND entries on all non-IRB Layer 3 interfaces on a network instance (default or IP-VRF), use the network-instance protocols layer-3-evpn-neighbor-synchronization command.

--{ candidate shared default }--[  ]--
# info with-context network-instance default protocols layer-3-evpn-neighbor-synchronization
    network-instance default {
        protocols {
            layer-3-evpn-neighbor-synchronization {
                admin-state enable
                route-distinguisher 10.4.4.4:6000
                evi-route-target evi-target:17676:6000
                retention-timer 300
                internal-tags {
                    set-tag-set [
                        set-1
                    ]
                }
            }
        }
    }

This command includes the following parameters:

route-distinguisher (mandatory)

Contrary to the BGP-VPN route distinguisher, the neighbor synchronization route-distinguisher command does not support autoderivation and must always be configured. In addition:
- The RDs must be unique per routing instance.
- The RDs referenced in layer-3-evpn-neighbor-synchronization must not be used in any other EVPN instance.
evi-route-target (mandatory)

Unlike regular BGP-VPN route targets, the evi-route-target value must be unique across network instances.
retention-timer

The retention-timer determines how long a synchronized ARP/ND entry of type EVPN is retained after the EVPN route is withdrawn by the remote ES peer, as follows:
- When the EVPN MAC/IP Advertisement route that originated an ARP-ND entry is withdrawn, the retention timer is started.
- The system attempts to refresh the entry by sending an ARP request or a neighbor solicitation (NS) message to the owner of the IP until the retention timer expires.
- When the timer expires, the entry is removed.
internal-tag

Similar to IRB subinterfaces, the internal-tag command allows MAC/IP routes generated for the purpose of ARP/ND synchronization to be matched on export policies.

Enable Layer 3 ARP/ND synchronization on the EVPN Ethernet Segment

To enable Layer 3 ARP/ND synchronization on the Ethernet Segment associated with the Layer 3 interfaces, use the layer-3-neighbor-synchronization true command. This command makes Layer 3 services in the ES subject to synchronization.

--{ candidate shared default }--[  ]--
# info with-context system network-instance protocols evpn ethernet-segments bgp-instance 1 ethernet-segment ESI-value layer-3-neighbor-synchronization
    system {
        network-instance {
            protocols {
                evpn {
                    ethernet-segments {
                        bgp-instance 1 {
                            ethernet-segment ESI-value {
                                layer-3-neighbor-synchronization true
                            }
                        }
                    }
                }
            }
        }
    }

The following Ethernet Segment configuration considerations apply:

ethernet-segment type must be none
Only non-virtual Ethernet Segment type is supported
The ES can only be associated with an interface and not with a next-hop
Multi-homing mode can be set to any mode, but it is expected as all-active for Layer 3 interfaces. Even if Layer 3 interfaces are configured as single-active, they are not impacted by the DF election procedures.
Routed subinterfaces and Ethernet interfaces are oper-up regardless of the Ethernet Segment multi-homing mode.
ESI label allocation depends on the value of use-esi-label (true or false).
The DF election commands have no impact on the state of Layer 3 interfaces.
The routes next-hop and routes ethernet-segment originating-ip commands under ethernet-segments are supported.
The ethernet-segments timers activation-timer command does not impact these Ethernet Segments for Layer 3 interfaces (since there is no DF election for Layer 3 services), but the ethernet-segments timers boot-timer works the same as for any ES: the ES route is advertised after the boot timer expires.
Up to four nodes can be attached to the same ES used for Layer 3 neighbor synchronization.
The ethernet-segment admin-state disable does not bring down the Layer 3 interface.

MAC/IP Advertisement routes for ARP/ND synchronization

When EVPN ARP/ND synchronization is enabled at the network instance level, it triggers the advertisement of the MAC/IP Advertisement routes for the static or dynamic ARP/ND entries on the Layer 3 subinterface.

MAC/IP routes are advertised with the following attributes:

the RD configured under layer-3-evpn-neighbor-synchronization
the ESI of the associated Ethernet Segment
the Ethernet tag ID of the Layer 3 interface encapsulation (which can only be 0 or VLAN-ID X)
the MAC/IP binding of the ARP/ND static/dynamic entry
Label-1 = 0 and no Label-2
Extended communities:
- ES-import extended community derived from the ESI, which ensures that MAC/IP routes are not imported in PEs that are not attached to the same ES.
- EVI-RT extended community derived from the configured layer-3-evpn-neighbor-synchronization evi-route-target. This is the same extended community used in EVPN route types 7 and 8. The EVI-RT extended community value, along with the ESI, is used to identify the network instance context and interface for which the ARP/ND information must be synchronized.
No encapsulation extended community is advertised with these routes.
No ARP/ND extended community is advertised with these routes.

Ethernet Segment route advertisement with Layer 3 synchronization

When Layer 3 ARP/ND synchronization is enabled on the Ethernet Segment:

The ES route includes the configured ESI, next hop and originating IP of the Ethernet Segment, as well as the ES-import extended community.
The ES route is advertised when the associated interface transitions to oper-up, and it is withdrawn when the interface transitions to oper-down. (This is not the case in regular Ethernet Segments, where the ES route is advertised with the first subinterface up. It is the AD per ES route that is advertised based on the interface oper-state.)

Programming of ARP/ND entries

ARP/ND synchronization enables the advertisement of EVPN ARP/ND synchronization routes (type 2) for all static and dynamic ARP/ND entries. No ARP/ND entry types other than static and dynamic are synchronized. The synchronized ARP or ND entries on Layer 3 interfaces display with origin evpn.

Programming of ARP/ND entries on the Layer 3 interface is based on the imported MAC/IP routes or locally received ARP/ND messages.

The MAC/IP routes used for ARP/ND synchronization are imported based on the evi-route-target matching the one configured in the network instance.
The ESI and Ethernet Tag ID encoded in the route determine the Layer 3 subinterface in which the ARP/ND entry is programmed.
The ESI determines the interface and the Ethernet Tag ID specifies the VLAN ID associated with the Layer 3 subinterface.
This means multiple tagged Layer 3 interfaces per ESI and router context are supported.

As with any ARP/ND entry of type evpn, these entries are subject to all the procedures supported for the EVPN entries. For instance, the command interface ethernet-1/1 subinterface 0 ipv4 arp host-route populate evpn creates ARP/ND host routes for these entries.

For IPv6 and ND synchronization, Link-Local Addresses are synchronized as well.

Ethernet Tag ID in EVPN MAC/IP routes

The support for multiple Layer 3 interfaces per router context and per ES is based on the use of the Ethernet Tag ID to encode the VLAN ID associated with the Layer 3 interface.

The Ethernet Tag ID encoded in the MAC/IP route used for synchronization is autoderived from the Layer 3 interface encapsulation 802.1Q tags and does not involve CLI configuration. For correct operation, the Layer 3 interface encapsulation 802.1Q tags must be configured consistently in all the PEs attached to the same ES.

When a new ARP/ND entry is learned locally, it is linked to only those Layer 3 interfaces (or LAGs) that have the following encapsulation types:

vlan-tagging false
single-tagged vlan-id X

No other Layer 3 interface encapsulations are supported.

The use of the Ethernet Tag ID in the EVPN MAC/IP routes used for synchronization allows the synchronization of multiple IP-MAC bindings with the same value in different Layer 3 subinterfaces, as in the following example:

Some router CEs connected to different Layer 3 subinterfaces within the same network instance use the same local Link-Local Address (LLA) and the same or different MAC addresses.
This leads to a situation where the same Layer 3 network instance context contains multiple ND entries with the same LLA (using the same or different MACs), although each interface has only one such entry.
This behavior is supported. Each entry is advertised in a separate MAC/IP Advertisement route, with each route tied to a different ESI/Ethernet Tag ID.
The case where the same VLAN is used in different Ethernet Segments does not support the same IP and MAC. For example:
- IP1/MAC1 is learned on interface ethernet-1/1 with VLAN-ID 100 and ESI-1 and on interface ethernet-1/2 with VLAN-ID 100 and ESI-2, in the same IP-VRF.
- In this case, BGP only advertises a single MAC/IP route for IP1/MAC1, because the route key of the two routes is the same.

Route selection

A local IP/MAC dynamic entry is always preferred over an EVPN entry, irrespective of the MAC address of the entry.
If multiple MAC/IP routes with the same IP (but different route keys) are received for the same ES, only one is selected for IP/MAC programming. The selection is based on the following, in priority order:
1. lowest RD route
2. lowest MAC address

Support of the same IP and MAC addresses on PE interfaces

In all PEs attached to the same ES, the same IP and MAC addresses are required on the Layer 3 interfaces of the same Ethernet Segment. On 7250 IXR and 7730 SXR platforms, you can configure the same MAC address on all ES PEs via configuration of VRRP or using the interface ethernet mac-address command, as shown in the following example output.

--{ candidate shared default }--[  ]--
# info with-context interface ethernet-1/11
    interface ethernet-1/11 {
        admin-state enable
        vlan-tagging true
        ethernet {
            mac-address 00:00:5E:00:53:EF
        }
    }
# Enable Layer 3 ARP/ND synchronization on the EVPN Ethernet Segment