EVPN VXLAN Anycast Multi-Homing

This chapter describes the implementation of Anycast Multi-Homing (MH).

EVPN VXLAN Anycast Multi-Homing for Layer 2 services

This chapter describes the implementation of Anycast Multi-Homing (MH) for Layer 2 services.

Anycast MH

EVPN VXLAN Anycast Multi-Homing (MH) leverages Anycast VXLAN Termination Endpoints (VTEPs) shared by the multi-homed leaf routers as defined in draft-rabnag-bess-evpn-anycast-aliasing (Anycast Multi-Homing for EVPN).

The following figure shows a leaf–spine fabric where Leaf1 and Leaf2 act as egress leaf nodes that multihome Server-1 and Server-2, while Leaf3 node serves as the ingress leaf forwarding traffic toward them. This figure serves as a reference point for all subsequent sections throughout this chapter.

Figure 1. EVPN Anycast MH
  • The egress leaf is the system attached to the Ethernet Segment (ES) working in the Anycast MH mode.
  • The ingress leaf is the system that performs the Ethernet Aliasing function (Overlay ECMP).

Supported platforms

Anycast Multi-Homing is supported on the following platforms:

  • 7220 IXR-Dx (ingress and egress function)
  • 7250 IXR Gen 2c+ (ingress and egress leaf function)
  • 7250 IXR Gen 3 (ingress and egress leaf function)
  • 7220 IXR-H5 (ingress and egress function)

Configuring Anycast MH

Anycast Multi-Homing (MH) is enabled per Ethernet Segment (ES) on egress leaf nodes (such as Leaf1 and Leaf2 in EVPN Anycast MH).

Note:

The ingress leaf node (such as Leaf3 in EVPN Anycast MH) requires no specific configuration to enable the Anycast MH mode. For backward compatibility within the same MAC-VRF, the ingress leaf supports two types of overlay ECMP.

  • ECMP to all egress leaf nodes advertising the Anycast VTEP
  • Ethernet Aliasing to all nodes of the same ES that advertise A-D per EVI routes and are operating in a non-anycast mode.

To configure Anycast MH per ES on the egress leaf node, use the command ethernet-segment.anycast-multi-homing. The configuration of ethernet-segment.anycast-multi-homing.ip-address <ip-address> is mandatory.

The IP address parameter (ip-address <ip-address>) should match an IPv4 address configured on a loopback interface in the default network instance. The system does not verify that the address exists on a loopback interface; you must ensure the correct IP address is configured.

The Anycast MH mode is supported in the following scenarios:

  • All-active multihoming ES
  • ES associated with Ethernet or LAG interfaces

The following configuration example enables Anycast MH for the ES named srl_ES within BGP instance 1, assigning 12.12.12.12 as the shared Anycast IP address for that segment.

--{ * candidate shared default }--[  ]--
# info with-context system network-instance protocols evpn ethernet-segments
    system {
        network-instance {
            protocols {
                evpn {
                    ethernet-segments {
                        bgp-instance 1 {
                            ethernet-segment srl_ES {
                                anycast-multi-homing {
                                    ip-address 12.12.12.12
                                }
                            }
                        }
                    }
                }
            }
        }
    }

Egress leaf in Anycast MH mode

The following sections describe the egress leaf working in Anycast Multi-Homing (MH) mode (such as Leaf1 and Leaf2 in EVPN Anycast MH).

Prerequisites

Following are the prerequisites for egress leaf nodes (such as Leaf1 and Leaf2 in EVPN Anycast MH) to operate in Anycast Multi-Homing (MH) mode:

  • Each egress leaf node must be configured with the same loopback IP address (ethernet-segment.anycast-multi-homing.ip-address <ip-address>). This IP address is referred to as the Anycast VTEP. This Anycast VTEP must be advertised in the underlay routing protocol (IS-IS, OSPF, or BGP) to ensure reachability from ingress leaf and spine routers.

    Note: The Anycast VTEP's loopback state is independent of the Ethernet Segment (ES) state. If the loopback goes down, the A-D per ES route is still advertised.
  • All egress leaf routers attached to the same ES in Anycast MH mode must use the same Anycast VTEP address.
    Note: If the user wants to bring down the loopback interface when an ES goes down, Event Handling System (EHS) can be used.
  • All egress leaf routers connected to the same ES in Anycast MH mode must use the same ingress VNI for each MAC-VRF.
    Note: Asymmetric VNIs are not supported for egress leaf nodes in Anycast MH mode.
  • The Anycast VTEP feature requires VXLAN termination on at least two distinct IP addresses: the system IP and a non-system IPv4 address. Egress leaf nodes must support VXLAN termination across multiple loopback interfaces.

Egress leaf behavior in Anycast MH mode

When an egress leaf (such as Leaf1 and Leaf2 in EVPN Anycast MH) operates in Anycast MH mode for a specific ES, the following behaviors apply:

  • Designated Forwarder (DF) Election and Local Bias behavior are unchanged. However, because A-D per-EVI routes are absent in this mode, the ac-df exclude capability must be configured, which then uses the preference algorithm.
  • The advertisement of the Ethernet Segment Identifier (ESI) with EVPN routes remains unchanged.
  • The processing and advertisement of BUM (Broadcast, Unknown Unicast, and Multicast) bindings and Inclusive Multicast Ethernet Tag (IMET) routes are not changed. The forwarding procedures for BUM traffic remain unchanged.
  • Only the procedures related to the forwarding of unicast traffic to a remote ES are updated.

Control plane behavior

From a control-plane perspective, an egress leaf working in Anycast MH mode (such as Leaf1 and Leaf2 in EVPN Anycast MH) for a given ES works as follows:

  • AD per-EVI routes for Anycast MH ESes are suppressed based on the Anycast MH mode configuration.
    • AD per-EVI routes are suppressed only for MAC-VRFs with VXLAN encapsulation.
    • This applies to nodes sharing ESs by mac-vrfs with vxlan and mpls encap.
  • AD per-ES routes for Anycast MH ESes are advertised with the A=1 flag in ESI label extended community (as defined in draft-rabnag-bess-evpn-anycast-aliasing).
    • The anycast-multi-homing flag indicates to remote NVEs that EVPN MAC/IP Advertisement routes with matching Ethernet Segment Identifier are resolved only by A-D per ES routes for the ES. This flag signals to the ingress leaf that no A-D per EVI routes are advertised for the Ethernet Segment. The A flag is bit 2 in the Flags field of the ESI label extended community.
      Figure 2. ESI label extended community
    • The BGP tunnel-encapsulation attribute ( per draft-rabnag-bess-evpn-anycast-aliasing) carries the Anycast VTEP; specifically, it is encoded in the BGP Tunnel Encapsulation Attribute Tunnel Egress Endpoint Sub-TLV (code point 6) [RFC9012]. The Tunnel Egress Endpoint Sub-TLV Value Field is shown below.
      Figure 3. Tunnel egress endpoint Sub-TLV
  • MAC/IP routes for MACs learned on the ES working in Anycast MH mode are advertised with ESI. The next-hop is either the regular system0.0 IPv4 address or the one determined by network-instance[=mac-vrf].protocols.bgp-evpn.bgp-instance.routes.bridge-table.next-hop <ip-address>

Data plane behavior

From a data-plane perspective, an egress leaf working in Anycast MH mode (such as Leaf1 and Leaf 2 in EVPN Anycast MH) for a given ES works as follows:

  • All VXLAN traffic is sourced from system0.0.
  • The Anycast VTEP is never used as a source VTEP.

Ingress leaf in Anycast MH mode

The ingress leaf node (such as Leaf3 in EVPN Anycast MH) function is mainly a control-plane function and is supported on all platforms, without requiring any datapath extensions.

For creating the Ethernet Segment (ES) destination, the following four operation modes are supported.

Non-Anycast mode

The ingress leaf operates in this mode when the ESI is identified as a non-Anycast MH ES. A non-Anycast MH ES is an ES where all AD per ES routes are received with A=0.

Anycast mode with multiple AD per-ES routes

In this mode, the ingress leaf receives multiple AD per-ES routes for the same ESI, all with the 'A' flag set to 1 and a consistent Anycast VTEP.

The ES destination is created using the Anycast VTEP learned from these routes and the VNI from the first received MAC/IP Advertisement or IP Prefix route.

Anycast mode with a single AD per-ES route

This mode applies when only a single AD per-ES route is received for the ES, or when all but one of the AD per-ES routes have been withdrawn.

The ES destination is created using the next-hop from the AD per-ES route and the VNI from the first received MAC/IP Advertisement or IP Prefix route.

Mismatch mode

This mode occurs when multiple AD per-ES routes are received for the same ESI, but the 'A' flag or Anycast VTEP information is inconsistent. A single route with A=1 is enough for the ingress leaf to be in "anycast mode".

If the ingress leaf receives AD per ES routes with an inconsistent "A" flag or Anycast VTEP information, the ES destination is created using the VTEPs of the AD per-ES routes NLRI next hops (allows load balancing) and the VNI from the first received MAC/IP Advertisement or IP Prefix route.

Only resolved AD per-ES routes (those with a reachable next hop and Anycast VTEP) are considered.

If a route cannot be resolved, BGP applies treat-as-withdraw behavior.

Ingress leaf BGP resolution of the Anycast VTEP

For EVPN routes advertised with an Anycast VTEP, successful resolution requires both the Network Layer Reachability Information (NLRI) Next Hop and the Anycast VTEP to be reachable. If either of these addresses becomes unreachable, the routes are not installed.

Comparing underlay failure behavior

The diagram below shows network behavior under default aliasing (left) versus Anycast aliasing (right).

Figure 4. Underlay failure: default aliasing versus Anycast aliasing
  • Default aliasing mode: When an underlay network failure occurs, in-flight packets may path hunt until the underlay network converges. If the underlay uses eBGP and the spines share the same ASN, in-flight packets are dropped.
  • Anycast aliasing mode: When an underlay network failure occurs, in-flight packets go directly from Spine1 to Leaf2. Therefore, convergence is faster and more efficient than in the default aliasing mode.

Egress leaf node failure behavior in Anycast ES

Anycast VTEPs ensure node failure events are handled smoothly and efficiently. The following diagram shows the situation before the node failure (left) and after (right).

Figure 5. Egress leaf node failure behavior in Anycast ESes
Note:

The following behavior assumes that Leaf3 keeps at least one MAC/IP route for Server-1 and ESI-1 after the node failure. Otherwise, the MAC is flushed, and unknown unicast traffic is flooded.

  • Non-failure state: The ES destination on Leaf3 points to the Anycast VTEP IP address 12.12.12.12. Per flow load balancing occurs based on the underlay set to 12.12.12.12 without using overlay ECPM sets.
  • Egress leaf failure (such as when Leaf1 goes down): In-flight packets are immediately rerouted to Leaf2, which also advertises the Anycast VTEP address. When Leaf3 receives the A-D per-ES MP_NLRI_UNREACH message from Leaf1, Leaf3 reprograms the ES destination to use the unicast VTEP of Leaf2, as specified by the AD per-ES route of the surviving ES leaf, Leaf2.

EVPN VXLAN Anycast Multi-Homing for Layer 3 services

Note: The components of EVPN VXLAN Anycast Multi-Homing for Layer 3 services are supported on 7250 IXR Gen 2c+, 7250 IXR Gen 3, 7220 IXR-Dx, 7220 IXR-H5, and 7220 IXR-H6 platforms.

This chapter describes the implementation of Anycast Multi-Homing (MH) for Layer 3 services.

EVPN VXLAN anycast aliasing for Layer 3 routes with ESI-0

Anycast MH can be applied to Ethernet Segment Identifier (ESI)-0 EVPN IFL routes advertised from two or more egress leaf routers.

The following figure illustrates anycast MH for ESI-0 EVPN IFL routes.
Figure 6. Anycast MH for ESI-0 EVPN IFL routes

Configuring anycast MH for EVPN IFL subnets

The following configuration enables the use of anycast MH for EVPN IFL subnets.
--{ * candidate shared default }--[  ]--
# info with-context network-instance ip-vrf
    network-instance ip-vrf {
        protocols {
            bgp-evpn {
                bgp-instance 1 {
                    routes {
                        route-table {
                            ip-prefix {
                                anycast-multi-homing {
                                    ip-address 12.12.12.12
                                }
                            }
                        }
                    }
                }
            }
        }
    }

Egress leaf in anycast MH mode for ESI-0

Two or more egress leafs that are attached to the same IP subnet and operate in anycast MH mode must use the same VNI to identify the IP-VRF associated with that IP subnet.

A single anycast VTEP is shared by the group of NVEs associated with the same IP subnets. These NVEs advertise reachability for the anycast VTEP in the underlay routing protocol. ESI-0 EVPN IFL routes are advertised with the anycast VTEP encoded in a BGP tunnel encapsulation attribute.

The presence of an anycast VTEP in the IP prefix route indicates to the ingress NVE that the prefix is using anycast MH. ESI-0 EVPN IFL routes do not carry the ESI extended community label; therefore they do not carry the A flag. For more information, see Anycast MH.

The router MAC used for anycast MH in RT5s (and IP A-D per ES routes in non-zero ESI Layer 3 cases; see EVPN VXLAN anycast aliasing for Layer 3 routes with non-zero ESI) is configured using the system network-instance protocols vxlan router-mac-address <mac> command and overrides the system MAC used in routed VXLAN interfaces as inner source-mac and router-mac in the EVPN routes. Only one router-mac per system can be configured and the router-mac is exposed in the state for routed VXLAN interfaces in the inner-ethernet-header used-source-mac <mac> context.

Ingress leaf in anycast MH mode for ESI-0

If GW-IP and anycast VTEP are both present in an RT5 route, BGP uses GW-IP for path selection and ignores anycast VTEP received in the RT5 route. If non-zero ESI and anycast VTEP are both present in an RT5 route, BGP prefers ESI and ignores anycast VTEP.

When receiving and importing ESI-0 EVPN IFL routes with an anycast VTEP for the same prefix, the following rules apply:
  • When all contributing RIB-INs are consistent, the anycast VTEP is used for installing the route.
  • For a specific prefix, consistency requires that all RIB-INs advertise the same anycast VTEP address (BGP Tunnel Encapsulation Attribute) and the same EVPN router's MAC extended community (GW-MAC). A differing VNI across RIB-INs does not affect consistency.
  • If the anycast VTEP address differs across RIB-INs or the router's MAC differs across RIB-INs, the ingress leaf enters mismatch mode and falls back to installing the route using the BGP NLRI next hops of the contributing routes.
  • If some RIB-INs carry an anycast VTEP and others do not (no Tunnel Encapsulation Attribute), the ingress leaf programs each contributing path individually. Routes without an anycast VTEP use the BGP NLRI next hop, and routes with an anycast VTEP use the anycast VTEP (ECMP across the resulting next-hop set).

When using anycast forwarding, the ingress leaf encapsulates the packets destined for a multihomed IP subnet into a VXLAN packet with the destination VTEP set to the anycast VTEP and the VNI set to the VNI of the best IP prefix route. All egress leafs attached to the multi-homed IP subnet have previously advertised reachability to the anycast VTEP, so the ingress leaf creates an underlay ECMP set for the anycast VTEP (assuming multiple equal-cost underlay paths).

The ingress leaf does not use an anycast VTEP as the outer source IP address of the VXLAN tunnel.

The anycast MH solution for Layer 3 does not support BGP fast reroute (FRR) backup-paths. FRR is not blocked in IP-VRFs that use anycast MH, but FRR only applies to RT5 routes without the anycast VTEP attribute.

ESI-0 EVPN routes with non-zero GW-IPs resolved to anycast MH EVPN-IFL-HOST routes are supported.

EVPN VXLAN anycast aliasing for Layer 3 routes with non-zero ESI

The following figure illustrates IP aliasing for non-zero Ethernet Segment Identifier (ESI) EVPN IFL routes.
Figure 7. Anycast MH for non-zero ESI EVPN IFL routes

Support for IP aliasing for non-zero ESI EVPN IFL routes is the same as for EVPN-IFL-HOST routes; for more information, see Multi-homed non-zero ESI EVPN-IFL-HOST routes

Anycast Ethernet Segments in the egress leaf routers are associated with a Layer 3 next hop (ethernet-segment <string> next-hop <l3-next-hop>) instead of an Ethernet or LAG interface.

Egress leaf in anycast MH mode for non-zero ESI

Anycast MH mode for non-zero ESI EVPN-IFL IP prefix routes is configured using the ethernet-segment anycast-multi-homing command.

The egress leaf in anycast MH mode for non-zero ESI EVPN-IFL IP prefix routes adheres to the following conditions:
  • Advertises the IP A-D per ES routes with the anycast MH flag set and anycast VTEP included
  • Suppresses IP A-D per EVI routes
  • Includes the EVPN router's MAC extended community in IP A-D per ES routes (required for VXLAN)
    • All egress leafs on the same anycast MH ES must advertise the same router MAC.
    • The router MAC is configured using the system network-instance protocols vxlan router-mac-address command, which overrides the system MAC as the inner source/destination MAC; only one is allowed per system.
    • This can be viewed in state under inner-ethernet-header used-source-mac.

Ingress leaf in anycast MH mode for non-zero ESI

The ingress leaf in anycast MH mode for non-zero ESI EVPN-IFL IP prefix routes adheres to the following conditions:
  • Searches for an IP A-D per ES route with the same ESI in the same IP-VRF
  • If found, it checks the anycast MH flag:
    • Flag not set – falls back to legacy IP aliasing behavior
    • Flag set – programs the EVPN-IFL entry toward an ES destination using the anycast VTEP
  • In all other cases, it follows the same IP A-D per ES procedures as the Layer 2 ingress NVE, including error handling.
  • For Layer 3 services, A-D per ES consistency requires that all contributing IP A-D per ES routes advertise the same anycast VTEP (BGP Tunnel Encapsulation Attribute) and the same EVPN router's MAC extended community (GW-MAC). If the anycast VTEP address or GW-MAC differs across contributing routes, the ingress leaf enters mismatch mode and falls back to installing the route using the BGP NLRI next hops of the contributing routes. A differing VNI across IP prefix routes does not affect consistency; when the anycast VTEP address and GW-MAC are consistent, the anycast VTEP is still used and the VNI is taken from the best IP prefix route.

Multi-homed non-zero ESI EVPN-IFL-HOST routes

Anycast MH can be applied to non-zero Ethernet Segment Identifier (ESI) EVPN-IFL-HOST routes, as shown in the following figure.

Figure 8. Anycast multi-homing for non-zero ESI EVPN-IFL-HOST routes
Configuring anycast multi-homing for non-zero ESI

The egress leafs attached to the anycast MH Ethernet Segments (Leaf1 and Leaf2 in the preceding figure) are configured with the ethernet-segment anycast-multi-homing command.

--{ * candidate shared default }--[  ]--
# info with-context system network-instance
    system {
        network-instance {
            protocols {
                evpn {
                    ethernet-segments {
                        bgp-instance 2 {
                            ethernet-segment esi1 {
                                anycast-multi-homing {
                                    ip-address 12.12.12.12
                                }
                            }
                        }
                    }
                }
            }
        }
    }
Egress leaf in anycast multi-homing mode for non-zero ESI
The egress leafs attached to the anycast MH Ethernet Segments (Leaf1 and Leaf2 in the preceding figure):
  • Advertise both sets of Ethernet A-D per ES and IP A-D per ES routes with the anycast MH mode flag and the anycast VTEP
  • Suppress the advertisement of both Ethernet A-D per EVI and IP A-D per EVI routes for ES configured in anycast MH mode
  • Include the EVPN router's MAC extended community along with the IP A-D per ES routes (required for VXLAN)
    • When advertised with the IP A-D per ES routes, the EVPN router's MAC extended community must contain the same MAC address value in all the IP A-D per ES routes advertised by all the egress leafs attached to the same anycast MH ES.
    • The router MAC used for anycast MH in MAC/IP routes (EVPN-IFL-HOST routes) and IP A-D per ES routes is configured using the system network-instance vxlan router-mac-address <mac> command and overrides the system MAC used in routed VXLAN interfaces as inner source-mac and router-mac in the EVPN routes. The configured router's MAC is used as the inner source MAC address of the sent packets and is expected as the inner destination MAC address of the received packets on termination.
    • Only one router-mac per system can be configured.
    • The router-mac is exposed in the state for routed VXLAN interfaces under inner-ethernet-header used-source-mac <mac>.
Ingress leaf in anycast multi-homing mode for non-zero ESI
When receiving and importing an EVPN MAC/IP advertisement route with a non-zero ESI, the ingress leaf searches for an IP A-D per ES route with the same ESI imported into the same IP-VRF, and then checks whether the anycast MH flag is set.
  • Flag not set – ingress NVE follows the IP aliasing procedures supported before this feature
  • Flag set – ingress NVE programs the EVPN-IFL-HOST entry (from the MAC/IP advertisement route with two VNIs) to be associated with the ES destination that uses an anycast VTEP.

Other than the preceding conditions, the same procedures can be applied to IP A‑D per ES routes as those defined for Ethernet A‑D per‑ES routes in the Layer 2 ingress NVE procedures, including all error‑handling cases.

For Layer 3 services, IP A-D per-ES consistency requires that all contributing IP A-D per ES routes advertise the same anycast VTEP (BGP Tunnel Encapsulation Attribute) and the same EVPN router's MAC extended community (GW-MAC). If the anycast VTEP address or GW-MAC differs across contributing routes, the ingress leaf enters mismatch mode and falls back to installing the route using the BGP NLRI next hops of the contributing routes. A differing VNI across MAC/IP advertisement routes does not affect consistency; when the anycast VTEP address and GW-MAC are consistent, the anycast VTEP is still used and the VNI is taken from the best MAC/IP advertisement route.