EVPN for MPLS Tunnels

This chapter provides information about EVPN for MPLS tunnels.

Topics in this chapter include:

Applicability

This chapter was initially written for SR OS Release 13.0.R6, but the MD-CLI in the current edition corresponds to Release 21.2.R1. A prerequisite is to read the EVPN for VXLAN Tunnels (Layer 2) chapter.

Overview

EVPN-MPLS is standardized in RFC 7432, BGP MPLS-Based Ethernet VPN, as a Layer 2 VPN technology that can supplement VPLS for E-LAN services. Besides the optimizations introduced by EVPN, a significant number of service providers offering E-LAN services today are requesting EVPN for their multi-homing capabilities. EVPN supports all-active multi-homing (per-flow load-balancing multi-homing) as well as single-active multi-homing (per-service load-balancing multi-homing). In addition to those superior multi-homing capabilities, EVPN also provides a number of significant benefits, such as:

  • IP-VPN-like operation and control for E-LAN services.

  • Reduction and (in some cases) suppression of the Broadcast, Unknown unicast, and Multicast (BUM) traffic in the network.

  • Simple provisioning and management.

  • New set of tools to control the distribution of MAC addresses and Address Resolution Protocol (ARP) entries in the network.

The EVPN for Virtual eXtensible Local Area Network (VXLAN) tunnels (Layer 2) chapter focuses on the use of EVPN as a control plane for VXLAN tunnels, whereas this chapter provides configuration guidelines for EVPN when used for MPLS tunnels. Similar to EVPN-VXLAN services, VPLS services with EVPN for MPLS tunnels are referred to as EVPN-MPLS services.

As a reference, the EVPN route types and NLRIs (Network Layer Reachability Information messages) used by the EVPN family in RFC 7432 are shown in EVPN route types and NLRIs.

Figure 1. EVPN route types and NLRIs

When no EVPN multi-homing is used in the network, only the base routes are used. Route types 2 and 3 are considered the base and mandatory routes:

  • Route type 2 - MAC/IP route: This route advertises MAC addresses to be installed in the remote FDBs, or MAC/IP address pairs to be installed in the remote proxy-ARP/ND (Neighbor Discovery) tables.

  • Route type 3 - Inclusive multicast route: This route advertises the multicast tree that the advertising PE intends to use for sending BUM traffic for an EVPN Instance (EVI). Ingress Replication, Point-to-multipoint multicast Label Distribution Protocol (P2MP mLDP), and composite tunnels are supported as tunnel types in route type 3 when BGP-EVPN MPLS is enabled. The ingress replication information, as well as the downstream MPLS label (for remote PEs to send BUM traffic to the advertising PE) are encoded in the Provider Multicast Service Interface Tunnel Attribute (PTA).

When EVPN multi-homing is used in an EVI, routes type 1 and 4 are used (where type 1 has two different purposes):

  • Route type 1 - Auto-discovery per Ethernet segment (AD per ES) route: This route is advertised per ES from the PE, carries the Ethernet Segment Identifier (ESI) label (used for split-horizon) in multi-homing mode, and can affect procedures such as the Designated Forwarder (DF) election, as well as the aliasing/backup path/mass withdrawal on remote PEs.

  • Route type 1 - Auto-discovery per EVPN instance (AD per-EVI) route: This route allows the remote PEs to provide aliasing and a backup path to the PEs part of the ES.

  • Route type 4 - Ethernet Segment (ES) route: This route advertises a local configured ES. The exchange of this route can discover remote PEs that are part of the same ES and the DF election algorithm among them.

The AD per-EVI, MAC/IP, and inclusive multicast routes are considered service-level BGP-EVPN routes. Their RT/RD (Route-Target/Route-Distinguisher) are taken from the VPLS configuration.

The AD per-ES and the ES routes are considered base-level BGP-EVPN routes. However, their RT/RD are taken differently:

  • The ES route RD is taken from the service>system>bgp>evpn configuration. The ES route RT is auto-derived from the Ethernet segment.

  • The AD per-ES route RD is taken from the system level RD or service level RD. The RT extended community is taken from the service level RT or an RT set for the services defined on the Ethernet segment.

Configuration

This section describes the configuration of EVPN-MPLS for Layer 2 services on SR OS, as well as the available troubleshooting and show commands, and EVPN multi-homing.

EVPN-MPLS for VPLS services shows the topology used throughout this chapter. The network consists of a core with four EVPN PEs (PE-2, PE-3, PE-4, and PE-5) and two MTU devices that are dual-homed to the EVPN network. For MTU-1, all-active multi-homing is used, whereas MTU-6 is connected via single-active multi-homing to the EVPN network. Three CEs are connected to VPLS 1 in MTU-1, PE-3, and MTU-6 in order to test the connectivity.

Figure 2. EVPN-MPLS for VPLS services

As part of the network infrastructure configuration, the following settings and protocols must be added to the configuration before starting with the EVPN-specific configuration for the services:

  • The ports interconnecting the four PEs in the core are configured as network ports (or hybrid) and will have router network interfaces defined in them. The ports on PE-2/PE-3 connected to MTU-1 can be access or hybrid ports, whereas the ports on PE-4/PE-5 connected to MTU-6 can be network or hybrid ports. In case of hybrid ports, no LACP can be configured.

  • The four PEs in the core (as well as MTU-6 in the access MPLS network) are running IS-IS and establishing point-to-point adjacencies for the exchange of the system IP addresses.

  • LDP is used as the MPLS protocol to signal transport tunnel labels among PE-2, PE-3, PE-4, PE-5, and MTU-6. There is no LDP running between MTU-1 and the rest of the network, that is, MTU-1 is a pure Ethernet aggregation device.

  • EVPN uses MP-BGP for exchanging reachability at service level. Therefore, BGP peering sessions must be established among the core PEs for the EVPN family. Although typically a separate router is used, in this chapter, PE-2 is used as BGP RR (route reflector) for EVPN routes. For example, the following output shows the configuration of BGP in the RR and one of the BGP clients. The relevant commands for EVPN are shown in bold.

    The configuration on the route reflector PE-2 is as follows:

    # on RR PE-2:
    configure {
        router "Base" {
            autonomous-system 64500
            bgp {
                vpn-apply-export true
                vpn-apply-import true
                rapid-withdrawal true
                peer-ip-tracking true
                split-horizon true
                rapid-update {
                    evpn true
                }
                group "internal" {
                    peer-as 64500
                    family {
                        evpn true
                    }
                    cluster {
                        cluster-id 1.1.1.1
                    }
                }
                neighbor "192.0.2.3" {
                    group "internal"
                }
                neighbor "192.0.2.4" {
                    group "internal"
                }
                neighbor "192.0.2.5" {
                    group "internal"
                }
            }
    

    The BGP configuration on the RR clients is as follows:

    # on RR clients PE-3, PE-4, PE-5:
    configure {
        router "Base" {
            autonomous-system 64500
            bgp {
                vpn-apply-export true
                vpn-apply-import true
                rapid-withdrawal true
                peer-ip-tracking true
                split-horizon true
                rapid-update {
                    evpn true
                }
                group "internal" {
                    peer-as 64500
                    family {
                        evpn true
                    }
                }
                neighbor "192.0.2.2" {
                    group "internal"
                }
            }
    
Note:

The def-recv-evpn-encap command is not used in the preceding configuration because the default MPLS configuration is sufficient to have a correct interpretation of the received EVPN encapsulations.

The EVPN encapsulation type can be configured as MPLS or VXLAN in the general BGP configuration, the BGP group, or the BGP neighbor. For example, on PE-3, the EVPN encapsulation type for neighbor 192.0.2.2 can be configured using the following command:

[ex:configure router "Base" bgp neighbor "192.0.2.2"]
A:admin@PE-3# def-recv-evpn-encap

 def-recv-evpn-encap <keyword>
 <keyword> - (mpls|vxlan)

    Default EVPN encapsulation type

EVPN routes type 1 (auto-discovery per-EVI route), type 2 (MAC/IP route), type 3 (inclusive multicast route), and type 5 (IP-prefix route) are always sent with the RFC 5512, the BGP Encapsulation Subsequent Address Family Identifier (SAFI) and the BGP Tunnel Encapsulation Attribute, BGP encapsulation extended community that indicates the associated encapsulation of the route. Because the use of this extended community is not mandatory in RFC 7432, the def-recv-evpn-encap command indicates to the system what encapsulation is associated with routes received without any encapsulation. When interoperating with third-party EVPN vendors in mixed MPLS and EVPN-VXLAN networks, this command should be revised accordingly.

EVPN-MPLS configuration without multi-homing

After the base infrastructure (interfaces, IGP, LDP, BGP protocols) is configured, the service and EVPN can be enabled. When no multi-homing is used, the EVPN-MPLS configuration in a VPLS service looks similar to the configuration of EVPN-VXLAN for Layer 2, except for the commands related to the MPLS data plane. The following output shows the VPLS-1 configuration on PE-3 as an example:

# on PE-3:
configure {
    service {
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            bgp 1 {
            }
            bgp-evpn {
                evi 1
                mpls 1 {
                    admin-state enable
                    ingress-replication-bum-label true
                    ecmp 2
                    auto-bind-tunnel {
                        resolution any
                    }
                }
            }
            sap 1/2/1:1 {
            }
            sap lag-1:1 {
            }
        }

Where the following commands are relevant for a basic EVPN configuration:

  • bgp enables the context for the BGP configuration relevant to the service. Two BGP instances can be configured, but only BGP instance 1 is configured in this example. If a manual (non-auto-derived) RD/RT, as well as import/export policies, are needed for the service, the commands in the bgp context must be configured. When bgp-evpn is enabled in a VPLS instance, other families are supported within the same service (bgp-ad, bgp-mh, bgp-vpls). This bgp context configures the common BGP parameters for all the BGP families in the service. Even if the general BGP parameters for the service are auto-derived (as in this example), the bgp context must be enabled.

    
    [ex:configure service vpls "VPLS1"]
    A:admin@PE-3# bgp ?
    
     [bgp-instance] <number>
     <number> - <1..2>
    
        BGP instance
    
    [ex:configure service vpls "VPLS1"]
    A:admin@PE-3# bgp 1 ?
    
     bgp
    
     apply-groups          - Apply a configuration group at this level
     apply-groups-exclude  - Exclude a configuration group at this level
     pw-template-binding   + Enter the pw-template-binding list instance
     route-distinguisher   - High-order 6 bytes that are used as string to compose 
                             VSI-ID for use in NLRI
     route-target          + Enter the route-target context
     vsi-export            - VSI export policies
     vsi-import            - VSI import policies
    
  • bgp-evpn evi <1..65535> — The EVPN instance or EVI is a 2-byte identifier used for the auto-derivation of the service RD, service RT, and for the service-carving algorithm when multi-homing is used. The EVI can be used for both bgp-evpn vxlan and bgp-evpn mpls when the user needs to auto-derive the RD and RT for the service. The auto-derivation is always based on:

    • RD system-ip:evi

    • RT autonomous-system:evi

    The configured and operating RD/RT values can be checked with the following show command (in this example, the evi value is 1):

    
    [/]
    A:admin@PE-3# show service id 1 bgp
    
    ===============================================================================
    BGP Information
    ===============================================================================
    Bgp Instance         : 1
    Vsi-Import           : None
    Vsi-Export           : None
    Route Dist           : None
    Oper Route Dist      : 192.0.2.3:1
    Oper RD Type         : derivedEvi
    Rte-Target Import    : None                 Rte-Target Export: None
    Oper RT Imp Origin   : derivedEvi           Oper RT Import   : 64500:1
    Oper RT Exp Origin   : derivedEvi           Oper RT Export   : 64500:1
    
    PW-Template Id       : None                 
    -------------------------------------------------------------------------------
    ===============================================================================
    

Although not required for a basic BGP-EVPN MPLS configuration, some other parameters may be used at the bgp-evpn context level, when EVPN-MPLS services are deployed. Some examples are listed here:

  • bgp-evpn>routes>mac-ip>cfm-mac must be enabled when eth-cfm is used across an EVPN-MPLS service among different PEs. If a Maintenance Endpoint (MEP) or Maintenance domain Intermediate Point (MIP) is configured in any of the SAP/SDP bindings in the VPLS and has to exchange eth-cfm packets with a remote MEP/MIP across the EVPN-MPLS core, this command must be enabled. In that way, the MEP/MIP MAC address can be advertised in EVPN (otherwise, the MEP/MIP MAC address would not be learned on remote EVPN-MPLS PEs and eth-cfm would not work correctly).

  • bgp-evpn>routes>mac-ip>advertise and bgp-evpn>mac-duplication — See the EVPN for VXLAN Tunnels (Layer 2) chapter for a description of these two commands.

  • bgp-evpn>mpls <bgp-instance> must be enabled.

When two BGP instances are added to a VPLS service, both BGP-EVPN MPLS and BGP-EVPN VXLAN can be configured at the same time in the service. A maximum of two BGP instances are supported in the same VPLS. In this chapter, only BGP instance 1 is used.

After the relevant VPLS parameters, BGP and BGP-EVPN attributes are added, the specific commands for bgp-evpn mpls <bgp-instance> can be configured as follows:

[ex:configure service vpls "VPLS1" bgp-evpn mpls 1]
A:admin@PE-3# info
    admin-state enable
    ingress-replication-bum-label true
    ecmp 2
    auto-bind-tunnel {
        resolution any
    }
  • ingress-replication-bum-label controls whether the system will advertise different service labels for unicast and BUM traffic. If no EVPN multi-homing is configured in the network, this command can be disabled (ingress-replication-bum-label false) and the same MPLS label will be advertised for the unicast and BUM traffic for the VPLS instance. If EVPN multi-homing is configured in the PE, this command is strongly recommended to avoid potential transient issues. See the EVPN-MPLS multi-homing section.

  • ecmp controls the number of remote PEs to which the local PE can load balance the unicast traffic. See the EVPN-MPLS multi-homing section.

  • auto-bind-tunnel controls the resolution of EVPN destinations to MPLS transport tunnels. This command is also in VPRN services and works in the same way.

    • If the auto-bind-tunnel resolution any is configured, as in the example, EVPN destinations in the service are resolved based on the best tunnel in the Tunnel Table Manager (TTM). For instance, the following command shows the existing EVPN destinations for VPLS 1 in PE-3. The EVPN-MPLS destination (Termination Endpoint (TEP) 192.0.2.2, label 524281) is resolved to LDP transport tunnel 65537 because the (best) LDP tunnel to 192.0.2.2 shown in the show router tunnel-table is LDP. If there was more than one tunnel type in the TTM to 192.0.2.2, the system would pick the lowest Pref (preference) tunnel.

      [/]
      A:admin@PE-3# show service id 1 evpn-mpls
      
      ===============================================================================
      BGP EVPN-MPLS Dest
      ===============================================================================
      TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                      Transport:Tnl                             Sup BCast Domain
      -------------------------------------------------------------------------------
      192.0.2.2       524281        0           bum             02/25/2021 16:46:38
                      ldp:65537                                 No
      192.0.2.4       524281        0           bum             02/25/2021 16:46:45
                      ldp:65538                                 No
      192.0.2.5       524281        0           bum             02/25/2021 16:46:52
                      ldp:65539                                 No
      -------------------------------------------------------------------------------
      Number of entries : 3
      -------------------------------------------------------------------------------
      ===============================================================================
      ---snip---
      
      [/]
      A:admin@PE-3# show router tunnel-table
      
      ===============================================================================
      IPv4 Tunnel Table (Router: Base)
      ===============================================================================
      Destination           Owner     Encap TunnelId  Pref   Nexthop        Metric
         Color 
      -------------------------------------------------------------------------------
      192.0.2.2/32          ldp       MPLS  65537     9      192.168.23.1   10
      192.0.2.4/32          ldp       MPLS  65538     9      192.168.34.2   10
      192.0.2.5/32          ldp       MPLS  65539     9      192.168.35.2   10
      192.0.2.6/32          ldp       MPLS  65540     9      192.168.34.2   20
      -------------------------------------------------------------------------------
      Flags: B = BGP or MPLS backup hop available
             L = Loop-Free Alternate (LFA) hop available
             E = Inactive best-external BGP route
             k = RIB-API or Forwarding Policy backup hop
      ===============================================================================
      
    • If resolution is set to any, the following tunnel types are selected in order of preference: RSVP, LDP, Segment Routing, and BGP. The user can configure the preference of the segment-routing tunnel type in the TTM for a specific IGP instance.

    • If one or more explicit tunnel types are specified using the resolution-filter option, then only these tunnel types will be selected again following the TTM preference.

    • The user must set the resolution to filter to activate the list of tunnel-types configured under resolution-filter.

Although not shown in the bgp-evpn mpls basic configuration for PE-3, there are other parameters that can be modified:

[ex:/configure service vpls "VPLS1" bgp-evpn]
A:admin@PE-3# mpls 1 ?

 mpls

 admin-state           - Administrative state of BGP EVPN MPLS
 apply-groups          - Apply a configuration group at this level
 apply-groups-exclude  - Exclude a configuration group at this level
 auto-bind-tunnel      + Enter the auto-bind-tunnel context
 control-word          - Enable/disable setting the CW bit in the label message.
 default-route-tag     - Default route tag
 ecmp                  - Maximum ECMP routes information
 entropy-label         - Enable/disable use of entropy-label.
 fdb                   + Enter the fdb context
 force-vc-forwarding   - VC forwarding action
 ingress-replication-  - Use the same label as the one advertised for unicast traffic
  bum-label
 oper-group            - Operational-Group identifier.
 route-next-hop        + Enter the route-next-hop context
 send-tunnel-encap     + Enter the send-tunnel-encap context
 split-horizon-group   - Split horizon group
bgp <bgp>
defines the BGP instance: 1 or 2.
control-word
enables/disables the insertion of the control-word in the data path. The control-word is disabled by default and is not signaled in EVPN (based on RFC 7432) and has to be consistently configured in all the PEs in the network. The use of the control-word prevents packet reordering from happening in P routers that misinterpret the first nibble of the payload in the packets they receive. In some third-party EVPN vendors, the control-word is enabled by default, so it is recommended to enable it when interoperating with other vendors.
entropy-label
enables the use of entropy labels, as described in the Entropy Label chapter in the 7450 ESS, 7750 SR, and 7950 XRS MPLS Advanced Configuration Guide for Classic CLI.
force-vc-forwarding <vlan>
allows the system to preserve the VLAN ID and p-bits of the service-delimiting q-tag in a new tag added in the customer frame before sending it to the EVPN core. This command may be used with the sap ingress qtag-manipulation command: the configured translated VLAN ID will be sent to the EVPN binds, as opposed to the service-delimiting tag VLAN ID. If the ingress SAP/SDP-binding is null encapsulated, the output VLAN ID and p-bits will be zero.
fdb protected-src-mac-violation-action
is by default disabled. When enabled, all packets entering the object will be verified not to contain a protected source MAC address. In combination with the keyword discard, the packets that contain a protected MAC address will be discarded and an alarm is generated.
send-tunnel-encap
configures the encapsulation to be advertised with the EVPN routes for the service. The encapsulation is encoded in RFC 5512-based tunnel encapsulation extended communities. When configured in the bgp-evpn>mpls context, the supported options are none (delete send-tunnel-encap), mpls, mpls-over-udp, or both.
admin-state
enables/disables the use of MPLS for EVPN. When enabled, a BGP route-refresh message is sent for the EVPN family.
split-horizon-group <group-name>
configures an explicit split-horizon-group (SHG) for all the EVPN destinations that can be shared with other SAP/SDP-bindings. See the VPLS to EVPN-MPLS integration section.

After bgp-evpn mpls is configured and enabled in the service, an inclusive multicast route is sent to the RR. The remote PEs receiving and importing that route will create an EVPN destination to the sending PE. An EVPN destination is identified by a TEP and MPLS label. Use the following show commands to view the service and the EVPN destinations created:

  • show service evpn-mpls

  • show service id 1 evpn-mpls

  • show service id 1 bgp-evpn

An example of the output is shown for PE-2 when there is no traffic in the network. Therefore, only inclusive multicast routes have been exchanged among the four PEs.

[/]
A:admin@PE-2# show service evpn-mpls

===============================================================================
EVPN MPLS Tunnel Endpoints
===============================================================================
EvpnMplsTEP Address EVPN-MPLS Dest      ES Dest             ES BMac Dest
-------------------------------------------------------------------------------
192.0.2.3           1                   0                   0
192.0.2.4           1                   0                   0
192.0.2.5           1                   0                   0
-------------------------------------------------------------------------------
Number of EvpnMpls Tunnel Endpoints: 3
-------------------------------------------------------------------------------
===============================================================================
[/]
A:admin@PE-2# show service id 1 evpn-mpls

===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                Transport:Tnl                             Sup BCast Domain
-------------------------------------------------------------------------------
192.0.2.3       524281        0           bum             02/25/2021 16:46:36
                ldp:65537                                 No
192.0.2.4       524281        0           bum             02/25/2021 16:46:45
                ldp:65538                                 No
192.0.2.5       524281        0           bum             02/25/2021 16:46:52
                ldp:65539                                 No
-------------------------------------------------------------------------------
Number of entries : 3
-------------------------------------------------------------------------------
===============================================================================

===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                       Num. Macs               Last Change
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

===============================================================================
BGP EVPN-MPLS ES BMAC Dest
===============================================================================
ES BMAC Addr                            Last Change
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================
[/]
A:admin@PE-2# show service id 1 bgp-evpn

===============================================================================
BGP EVPN Table
===============================================================================
MAC Advertisement  : Enabled            Unknown MAC Route  : Disabled
CFM MAC Advertise  : Disabled
Creation Origin    : manual
MAC Dup Detn Moves : 5                  MAC Dup Detn Window: 3
MAC Dup Detn Retry : 9                  Number of Dup MACs : 0
MAC Dup Detn BH    : Disabled
IP Route Advert    : Disabled
Sel Mcast Advert   : Disabled

EVI                : 1
Ing Rep Inc McastAd: Enabled
Accept IVPLS Flush : Disabled

-------------------------------------------------------------------------------
Detected Duplicate MAC Addresses             Time Detected
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
===============================================================================

===============================================================================
BGP EVPN MPLS Information
===============================================================================
Admin Status       : Enabled            Bgp Instance       : 1
Force Vlan Fwding  : Disabled
Route NextHop Type : system-ipv4
Control Word       : Disabled
Max Ecmp Routes    : 2
Entropy Label      : Disabled
Default Route Tag  : none
Split Horizon Group: (Not Specified)
Ingress Rep BUM Lbl: Enabled
Ingress Ucast Lbl  : 524282             Ingress Mcast Lbl  : 524281
RestProtSrcMacAct  : none
Evpn Mpls Encap    : Enabled            Evpn MplsoUdp      : Disabled
Oper Group         :
===============================================================================

===============================================================================
BGP EVPN MPLS Auto Bind Tunnel Information
===============================================================================
Allow-Flex-Algo-Fallback  : false
Resolution                : any                Strict Tnl Tag   : false
Max Ecmp Routes           : 1
Bgp Instance              : 1
Filter Tunnel Types       : (Not Specified)
===============================================================================

When traffic is generated, the PEs will start learning MAC addresses and advertising them in BGP so that the remote PEs learn those MAC addresses against EVPN destinations. For instance, when CE-13 sends traffic, PE-3 learns its MAC address and advertises it. The remote PEs (for instance, PE-2) will learn the MAC address and associate it with their EVPN destination to PE-3 (192.0.2.3:524282 in this example):

[/]
A:admin@PE-2# show service id 1 fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:11:11:11:11 sap:lag-1:1             L/0      02/25/21 16:54:31
1          00:00:13:13:13:13 mpls:                   Evpn     02/25/21 16:54:31
                             192.0.2.3:524282
           ldp:65537
-------------------------------------------------------------------------------
No. of MAC Entries: 2
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================

When the ingress-replication-bum-label is enabled in the PEs, the advertisement of MAC addresses will create new EVPN destinations, because the label is different from the one previously sent by the inclusive multicast route that created an EVPN destination. In the preceding example, when PE-3 advertises the CE-13 MAC address, PE-2 will create a new binding (see in the following output in bold) that shows one MAC address that is not Mcast (multicast) capable:

[/]
A:admin@PE-2# show service id 1 evpn-mpls
 
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                Transport:Tnl                             Sup BCast Domain
-------------------------------------------------------------------------------
192.0.2.3       524281        0           bum             02/25/2021 16:46:36
                ldp:65537                                 No
192.0.2.3       524282        1           none            02/25/2021 16:54:31
                ldp:65537                                 No
192.0.2.4       524281        0           bum             02/25/2021 16:46:45
                ldp:65538                                 No
192.0.2.5       524281        0           bum             02/25/2021 16:46:52
                ldp:65539                                 No
-------------------------------------------------------------------------------
Number of entries : 4
-------------------------------------------------------------------------------
===============================================================================
---snip---

When an EVPN-MPLS destination or MAC address is not created/installed correctly, the user may check the BGP-EVPN routes received and the routes kept in the RIB. The routes that the PE receives are shown when debug router bgp update is enabled. These routes are shown even before any BGP processing is carried out.

# on PE-2: 
30 2021/02/25 16:54:31.213 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 81
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
                       mac: 00:00:13:13:13:13, IP len: 0, IP: NULL, label1: 8388512
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:1
        bgp-tunnel-encap:MPLS
"
[/]
A:admin@PE-2# show router bgp routes evpn mac
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         00:00:13:13:13:13 ESI-0
      0                   Seq:0             LABEL 524282
                          n/a
                          192.0.2.3

-------------------------------------------------------------------------------
Routes : 1
===============================================================================

If the route is successfully imported, it can be shown in the RIB (show router bgp routes commands). The route shown in the debug and the same route in a show command do not necessarily have the same label value. The reason for this expected mismatch is that the debug command shows the complete 24-bit field value because the route is shown before BGP can decide and decipher whether the label value is an MPLS label (high-order 20-bits of the label field) or a VNI (all 24 bits of the Label field for VXLAN). When the label in the debug command (8388512) is divided by 16 (24), the result is the MPLS label (524282), as follows: 8388512:16=524282.

VPLS to EVPN-MPLS integration

The SR OS EVPN implementation supports RFC 8560, (PBB-)EVPN Seamless Integration with (PBB-)EVPN, so that EVPN-MPLS and VPLS can be integrated into the same network and within the same service.

The following behavior enables the integration of EVPN and SDP-bindings in the same VPLS network:

  • Systems with EVPN endpoints and SDP-bindings to the same far-end bring down the SDP-bindings.

    • SR OS will allow the establishment of an EVPN destination and an SDP-binding to the same far-end but the SDP-binding will be kept operationally down. Only the EVPN endpoint will be operationally up. This is true for spoke-SDPs (manual and BGP-AD) and mesh-SDPs. It is also true between VXLAN and SDP-bindings.

    • If there is an EVPN endpoint to a specified far-end and a spoke-SDP establishment is attempted, the spoke-SDP will be set up but kept down with an operational flag indicating that there is an EVPN route to the same far-end.

    • If there is a spoke-SDP and a valid/used EVPN route arrives, the EVPN endpoint will be set up and the spoke-SDP will be brought down with an operational flag indicating that there is an EVPN route to the same far-end.

    • In the case of an SDP-binding and EVPN endpoint to different far-end IPs on the same remote PE, both links will be up. This can happen if the SDP-binding is terminated in an IPv6 address or IPv4 address different from the system address where the EVPN endpoint is terminated.

    The following example illustrates the preceding description. A spoke-SDP is added to the VPLS 1 configuration on PE-2:

    # on PE-2:
    configure {
        service {
            sdp 24 {
                admin-state enable
                delivery-type mpls
                ldp true
                far-end {
                    ip-address 192.0.2.4
                }
            }
            vpls "VPLS1" {
                spoke-sdp 24:1 {
                }
            }
    

    The service configuration on PE-4 is as follows:

    # on PE-4:
    configure {
        service {
            sdp 42 {
                admin-state enable
                delivery-type mpls
                ldp true
                far-end {
                    ip-address 192.0.2.2
                }
            }
            sdp 46 {
                admin-state enable
                delivery-type mpls
                ldp true
                far-end {
                    ip-address 192.0.2.6
                }
            }
            vpls "VPLS1" {
                admin-state enable
                service-id 1
                customer "1"
                bgp 1 {
                }
                bgp-evpn {
                    evi 1
                    mpls 1 {
                        admin-state enable
                        ingress-replication-bum-label true
                        ecmp 2
                        auto-bind-tunnel {
                            resolution any
                        }
                    }
                }
                spoke-sdp 42:1 {
                }
                spoke-sdp 46:1 {
                }
            }
    

    Spoke SDP 24:1 is operationally down, as can be verified as follows:

    [/]
    A:admin@PE-2# show service id 1 sdp
    
    ===============================================================================
    Services: Service Destination Points
    ===============================================================================
    SdpId            Type     Far End addr    Adm     Opr       I.Lbl     E.Lbl
    -------------------------------------------------------------------------------
    24:1             Spok     192.0.2.4       Up      Down      524280    524279
    -------------------------------------------------------------------------------
    Number of SDPs : 1
    -------------------------------------------------------------------------------
    ===============================================================================
    

    Spoke SDP 24:1 is down because of an EVPN route conflict, as indicated by the flags:

    [/]
    A:admin@PE-2# show service id 1 sdp 24 detail | match Flag post-lines 1
    Flags              : PWPeerFaultStatusBits
                         EvpnRouteConflict
    
  • The user can add spoke-SDPs and all the EVPN-MPLS endpoints in the same SHG.

    • A CLI command exists in the bgp-evpn>mpls context so that the EVPN-MPLS endpoints can be added to an SHG.

    • The bgp-evpn mpls split-horizon-group must reference a user-configured SHG. User-configured SHGs can be configured within the service context.

    • The same group name can be associated with SAPs, spoke-SDPs, PW-templates, PW-template-bindings, and EVPN-MPLS endpoints.

    • If the split-horizon-group command in bgp-evpn>mpls is not used, the default SHG (in which all the EVPN endpoints are) is still used, but it will not be possible to refer to it on SAPs/spoke-SDPs.

  • The system disables the advertisement of MAC addresses learned on spoke- SDPs/SAPs that are part of an EVPN SHG.

    • When the SAPs or spoke-SDPs (manual or BGP-AD-discovered) are configured within the same SHG as the EVPN endpoints, MAC addresses will still be learned on them, but will not be advertised in EVPN.

    • The preceding statement is also true if proxy-ARP/ND is enabled and an IP-->MAC address pair is learned on a SAP/SDP-binding that belongs to the EVPN SHG.

    • The SAPs and/or spoke-SDPs added to an EVPN SHG should not be part of any EVPN multi-homed ES. If that happened, the PE would still advertise the AD per-EVI route for the SAP and/or spoke-SDP, attracting EVPN traffic that could not be forwarded to that SAP and/or SDP-binding.

    • Similar to the preceding statement, an SHG composed of SAPs/SDP-bindings used in a BGP-MH site should not be configured under bgp-evpn>mpls>split-horizon-group. This misconfiguration would prevent traffic being forwarded from the EVPN to the BGP-MH site, regardless of the DF/Non-DF state.

An example of a shared SHG configuration on PE-2 is as follows. Because the SAP and EVPN-MPLS are in the same SHG, no MAC addresses learned over SAP 1/2/1:2 will be advertised in EVPN (not even static MAC addresses).

# on PE-2:
configure {
    service {
        vpls "VPLS2" {
            admin-state enable
            service-id 2
            customer "1"
            bgp 1 {
            }
            bgp-evpn {
                evi 2
                mpls 1 {
                    admin-state enable
                    split-horizon-group "CORE"
                    ingress-replication-bum-label true
                    ecmp 2
                    auto-bind-tunnel
                        resolution any
                    }
                }
            }
            split-horizon-group "CORE" {
            }
            sap 1/2/1:2 {
                split-horizon-group "CORE"
            }
            sap lag-1:2 {
            }

EVPN-MPLS multi-homing

SR OS supports EVPN multi-homing as per RFC 7432.

The EVPN multi-homing implementation is based on the concept of the ES. An ES is a logical structure that can be defined in one or more PEs and identifies the CE (or access network) multi-homed to the EVPN PEs. An ES is associated with a port, LAG, or SDP object, and is shared by all the services defined on those objects.

Each ES has a unique identifier called ESI (Ethernet Segment Identifier) that is 10 bytes and is manually configured. The ESI is advertised in the control plane to all the PEs in an EVPN network; therefore, it is very important to ensure that the 10-byte ESI value is unique throughout the entire network. Single-homed CEs are assumed to be connected to an ES with ESI = 0 (single-homed ESs are not explicitly configured).

The ES is part of the base BGP-EVPN configuration and is not applied to any EVPN-MPLS service, by default. An ES can be shared by multiple services; the association of a specific SAP or spoke-SDP to an ES is automatically made when the SAP is defined in the same LAG or port configured in the ES, or when the spoke-SDP is defined in the same SDP configured in the ES. The following sections show the configuration of:

  • an all-active multi-homing ES with a LAG associated with it

  • a single-active multi-homing ES linked to an SDP

All-active multi-homing concepts

EVPN all-active multi-homing is built around three concepts: DF election, split-horizon (with an ESI-label), and aliasing, as shown in EVPN-MPLS all-active multi-homing concepts, from left to right.

Figure 3. EVPN-MPLS all-active multi-homing concepts
  • With DF election, when PE-4 sends BUM traffic to the remote ES (CE-2), only one PE segment sends the BUM packets to the ES (PE-3 is the DF in the preceding example, and is elected to send BUM packets to CE-2). The non-DF, PE-2, removes the LAG SAP from the default multicast list (PE-2 does not bring CE-2 down, because it still needs to send upstream/downstream unicast traffic). PE-2 and PE-3 elect a DF for each service, based on the ES routes and the service-carving algorithm.

  • With split-horizon, the PE part of the ES (PE-3 in the preceding example) identifies the BUM packets coming from the PE for the remote (PE-2), but within the same ES (ESI-2), and filters the packets so that they are not sent back to the ES, creating duplication. When PE-2 (non-DF) sends BUM traffic to PE-3 (DF), it uses a special MPLS label in the data path that PE-3 previously advertised for ESI-2 in an AD per-ES route. When PE-3 does an ingress lookup, it recognizes the ESI-label and filters the traffic (PE-3 still sends the BUM traffic to other SAPs/SDP-bindings).

  • With aliasing, remote PEs that are not part of the ES can load-balance unicast traffic to all the PEs that are part of the ES, irrespective of from which PE a destination MAC address was learned. PE-4 will create an EVPN destination to ESI-2 that will be resolved to the two next-hops: PE-2 and PE-3. Unicast load-balancing will happen as long as ECMP > 1 is enabled in PE-4.

Nokia recommends the use of ingress-replication-bum-label on the PEs that are part of an all-active ES. In an all-active multi-homing scenario, if a specified MAC address (for example, the CE-2 MAC address in the left-hand-side diagram), is not learned yet in a remote PE (for example, PE-4), but is known in the two PEs of the ES (for example, PE-2 and PE-3), the latter PEs might send duplicated packets to the CE.

This issue is solved by the use of ingress-replication-bum-label in PE-2 and PE-3. If configured, PE-2/PE-3 will know that the received packet is an unknown unicast packet; therefore, the Non-DF (PE-2) will not send the packet to CE-2 and there will not be duplication.

All-active multi-homing configuration

The all-active multi-homing configuration example is based on EVPN-MPLS for VPLS services.

MTU-1 is connected to the EVPN network using all-active multi-homing. According to RFC 7432, MTU-1 will be able to send traffic to both PEs for VPLS-1. Regular LAG load-balancing is used in MTU-1. Remote PEs such as PE-4 or PE-5 will be able to load-balance the unicast traffic to PE-2 and PE-3. PE-2 and PE-3 will discover that both are part of ESI-23 (due to the exchange of ES routes) and will elect a DF for VPLS-1. The non-DF for VPLS-1, in this case PE-2, will remove lag-1:1 from the VPLS-1 default multicast list. Also, when PE-2 and PE-3 send BUM traffic to each other, they will insert an ESI-label so that they can identify that the source of the BUM packet is ESI-23.

The following output shows the configuration of ESI-23 in PE-2 and PE-3, as well as the LAG interfaces for all-active multi-homing (see EVPN-MPLS for VPLS services). The configuration of LAG-1 in MTU-1 is also shown. Per RFC 7432, only a CE/MTU with a LAG can be connected to an all-active multi-homing ES. No other configuration is permitted on the CE for all-active multi-homing.

LAG 1 is configured on MTU-1, PE-2, and PE-3, as follows:

# on MTU-1:
configure {
    lag "lag-1" {
        admin-state enable
        encap-type dot1q
        mode access
        max-ports 64
        lacp {
            mode active
            administrative-key 32768
        }
        port 1/1/1 {
        }
        port 1/1/2 {
        }
    }
# on PE-2:
configure {
    lag "lag-1" {
        admin-state enable
        encap-type dot1q
        mode access
        max-ports 64
        lacp {
            mode active
            system-id 00:00:00:00:02:03
            administrative-key 1
        }
        port 1/1/2 {
        }
    }
# on PE-3:
configure {
    lag "lag-1" {
        admin-state enable
        encap-type dot1q
        mode access
        max-ports 64
        lacp {
            mode active
            system-id 00:00:00:00:02:03
            administrative-key 1
        }
        port 1/1/1 {
        }
    }

Ethernet segment ‟ESI-23” is configured in the service system bgp-evpn context on PE-2 and PE-3, as follows:

# on PE-2, PE-3:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-23" {
                        admin-state enable
                        esi 01:00:00:00:00:23:00:00:00:01
                        multi-homing-mode all-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            lag "lag-1" {
                            }
                        }
                    }
                }
            }
        }

When configuring an ES, the following must be considered:

  • Any EVPN parameter that is not specific to any particular VPLS service, and is common to all the EVIs, is configured in a base BGP-EVPN instance located at config>service>system>bgp-evpn. In this base instance, the following attributes may be configured:

    • ethernet-segments

    • the base BGP-EVPN instance route-distinguisher that will be used for the ES routes. If this route-distinguisher is not configured, by default a type-1 RD will be derived as system-ip:0. The ES route distinguisher can be configured using the following command:

    [ex:configure service system bgp evpn]
    A:admin@PE-2# route-distinguisher ?
    
     route-distinguisher <string>
     <string> - <0..255 characters>
    
        Route distinguisher for ES routes
    
  • The ES must be configured with a name and can contain the following parameters when configured for all-active multi-homing:

    • esi — 10-byte identifier that represents the ES in the BGP control plane. The same ESI must be configured in all the PEs connected to the same CE/MTU (using a unique value that cannot be associated with any other CE/MTU/access network). RFC 7432 defines five different types of ESI. In SR OS, the type byte, as well as the other 9 bytes can be arbitrarily configured.

    • multi-homing-mode all-active — This command indicates that the ES is in all-active mode.

    • association>lag <lag-id> — The LAG connected to the CE/MTU must be added to the ES. In this example, LAG "lag-1" is added to ESI-23, on both PE-2 and PE-3. Although a different LAG-id may have been assigned to the same ES on PE-2 and PE-3, PE-2 and PE-3 must have the same configuration on the ES LAG; that is, encap-type. Also, if LACP is added (it is not mandatory), both PEs must have the same admin-key, system-id, and system-priority. MTU-1 will see PE-2 and PE-3 as a single LAG peer. For all-active multi-homing, only the lag option is accepted by the system; port or sdp are not accepted.

    • admin-state — This command controls the administrative state of the ES.

  • The preceding parameters are the minimum necessary so that the ES can be activated. In addition to those parameters, there are a few more that the user can configure if requiring values different from the default ones:

    • activation-timer [0..100] can be configured at redundancy>bgp-evpn-ethernet-segment>activation-timer or at service>system>bgp>evpn>ethernet-segment>df-election>es-activation-timer level (the most specific value is used).

      The ES activation timer operation is as follows:

      • Upon reception of an ES, AD per-ES/EVI route update/withdrawal for a local ESI, the DF-candidate list of IP addresses is updated and the DF election algorithm is run without waiting for any timer.

      • If the result of the DF election requires the PE to be promoted from non-DF to DF, the ES activation timer will start, and only after its expiration will the PE add the SAP to the default multicast list. Transitions from non-DF to non-DF, or from DF to non-DF, are immediate and do not wait for any timer.

      • This use of an ES activation timer value minimizes the risks of loops and packet duplication due to transient multiple DFs.

      • The same ES activation timer must be configured in all the PEs that are part of the same ESI. The user must configure either a long timer to minimize the risks of loops/duplication, or (es-)activation-timer=0 to speed up the convergence for NDF to DF transitions. The default value is 3 seconds.

    • service-carving-mode — As defined in RFC 7432, service carving controls the distribution of DF/non-DF roles across the different services defined in an ES.

      [ex:configure service system bgp evpn ethernet-segment "ESI-23" df-election]
      A:admin@PE-2# service-carving-mode ?
      
       service-carving-mode <keyword>
       <keyword> - (auto|manual|off)
       Default   - auto
      
          Mode of service carving enabled per EVPN associated with this Ethernet segment
          entry
      
      [ex:configure service system bgp evpn ethernet-segment "ESI-23" df-election]
      A:admin@PE-2# manual ?
      
       manual
      
       evi                  + Enter the evi context
       isid                 + Enter the isid context
       preference           + Enter the preference context
      

      As shown above, service-carving has three different modes:

    • service-carving-mode auto (default) — The DF election algorithm will run the function [V(evi) mod N(peers) = i(ordinal)] to know who the DF for a specified service and ESI is. In this example, ESI-23 is configured with mode auto; therefore, for VPLS-1 (with EVI-1), PE-3 will be elected as DF because evi(1) mod (2)peers = 1, and the ordinal 1 corresponds to the second lowest IP, PE-3. The algorithm takes the configured evi in the service; therefore, the evi is mandatory, and for the same service must match in all the PEs that are part of the ES. This guarantees that the election algorithm is consistent across all the PEs of the ESI.

    • service-carving-mode manual — The user can manually decide for which evi identifiers the PE is DF: service-carving-mode manual / manual evi <start> end <to>. The PE will be non-DF for the non-specified EVIs. If service-carving-mode manual is configured, but no range is defined, all the services are considered to be non-DF. If a range is configured, but the service-carving-mode is not manual, the range has no effect. Only two PEs are supported when service-carving-mode manual is configured.

    • service-carving-mode off — The lowest originator IP will win the election for a specified service and ES.

    • Because the evi is used for the service carving algorithm, it must always be configured in a service with SAPs/SDP bindings created in an ES, regardless of the service-carving mode (service-carving off, auto, or manual).

Although not configured as part of the ES, the config>redundancy>bgp-evpn-ethernet-segment>boot-timer allows the necessary time for the control plane protocols to come up after the PE has rebooted, and before bringing up the ESs and running the DF algorithm. Some considerations about the boot timer:

  • The boot timer should use a value long enough to allow the IOMs and BGP sessions to come up before exchanging ES routes and run the DF election for each EVI (it is 10 s, by default).

  • The boot timer runs per EVI on the ESs in the system. While system-up-time < boot-timer, the system will not run the DF election for any EVI. When the boot timer expires, the DF election for the EVI is run and, if the system is elected DF for the EVI, the ES activation timer will start.

  • The system will not advertise ES routes until the boot timer expires. This guarantees that the peer ES PEs do not run the DF election either, until the PE is ready to become the DF, if needed.

  • The following show command displays the configured boot timer, as well as the remaining timer if the system is still in boot stage.

    
    [/]
    A:admin@PE-2# show redundancy bgp-evpn-multi-homing 
    
    ===============================================================================
    Redundancy BGP EVPN Multi-homing Information
    ===============================================================================
    Boot-Timer              : 10 secs                 
    Boot-Timer Remaining    : 0 secs                  
    ES Activation Timer     : 3 secs                  
    ===============================================================================
    

After ESI-23 is configured in PE-2 and PE-3, the lag-1 SAPs in both PEs can be added to the VPLS-1 service. Until the ESI-23 is successfully enabled, the LAG SAPs will be kept down with a StandByForMHProtocol flag. This is illustrated in the following example for PE-2 where the LAG SAP is added and ESI-23 is disabled:

# on PE-2:
configure exclusive
    service {
        vpls "VPLS1" {
            sap lag-1:1 {
            }
        }
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-23" {
                        admin-state disable
                    }
                    commit
[/]
A:admin@PE-2# show service id 1 sap lag-1:1 detail | match "  Oper State"
Admin State        : Up                       Oper State        : Down
[/]
A:admin@PE-2# show service id 1 sap lag-1:1 detail | match Flag 
Flags              : StandByForMHProtocol

ESI-23 is enabled and SAP lag-1:1 is operationally up, as follows:

# on PE-2:
configure exclusive
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-23" {
                        admin-state enable
                    }
                    commit
[/]
A:admin@PE-2# show log log-id "99" 
---snip---

107 2021/02/25 17:06:03.759 CET MINOR: SVCMGR #2203 Base
"Status of SAP lag-1:1 in service 1 (customer 1) changed to admin=up oper=up flags="

All-active multi-homing operation

To confirm that all-active multi-homing is working correctly for ESI-23, the user can use the following commands:

  • show service system bgp-evpn — Shows the RD is used for the ES route.

  • show service system bgp-evpn ethernet-segment — Shows all the ESs configured in the PE and their admin/operational status.

  • show service system bgp-evpn ethernet-segment name ESI-23 evi evi-1 1 — Shows the DF candidate PEs for EVI 1 and whether the system is DF for EVI.

  • show service system bgp-evpn ethernet-segment name ESI-23 all — Shows all the information related to a specific ESI.

The base BGP-EVPN information includes the RD:

[/]
A:admin@PE-2# show service system bgp-evpn

===============================================================================
System BGP EVPN Information
===============================================================================
Eth Seg Route Dist.               : <none>
Eth Seg Oper Route Dist.          : 192.0.2.2:0
Eth Seg Oper Route Dist Type      : default
Ad Per ES Route Target            : evi-rt
Leaf Label                        : 0
Mcast Leave Sync Prop             : 5
Attribute Uniform Prop            : Disabled
BGP Path Selection                : Disabled
===============================================================================

The following command shows the configured ESs in the PE and their status:

[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment 

===============================================================================
Service Ethernet Segment
===============================================================================
Name                             ESI                           Admin     Oper
-------------------------------------------------------------------------------
ESI-23                           01:00:00:00:00:23:00:00:00:01 Enabled   Up
-------------------------------------------------------------------------------
Entries found: 1
===============================================================================

The following command shows that PE-2 is not the DF and the DF candidate PEs for EVI 1 are PE-2 and PE-3:

[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" evi evi-1 1

===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
1             1             0                   no  02/25/2021 17:06:04
===============================================================================

===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.2                               02/25/2021 17:06:04
192.0.2.3                               02/25/2021 17:06:06
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================

The following command shows all information related to ESI-23 on PE-2:

[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" all

===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-23
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:23:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
ES SHG Label            : 524279
Source BMAC LSB         : <none>
Lag                     : lag-1
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================

===============================================================================
EVI Information
===============================================================================
EVI                 SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
1                   1                   0                   no
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================

-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI                                     DF Address
-------------------------------------------------------------------------------
1                                       192.0.2.2
1                                       192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---

The following command shows all information related to ESI-23 on PE-3:

[/]
A:admin@PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" all

===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-23
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:23:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
ES SHG Label            : 524279
Source BMAC LSB         : <none>
Lag                     : lag-1
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================

===============================================================================
EVI Information
===============================================================================
EVI                 SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
1                   1                   0                   yes
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================

-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI                                     DF Address
-------------------------------------------------------------------------------
1                                       192.0.2.2
1                                       192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---

The preceding commands show the ESI-23 configuration on both PEs and the result of the DF election for EVI 1.

The following output shows the ES route received on PE-2:

# on PE-2:
63 2021/02/25 17:04:23.069 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 71
    Flag: 0x90 Type: 14 Len: 34 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.3:0
              ESI: 01:00:00:00:00:23:00:00:00:01, IP-Len: 4 Orig-IP-Addr: 192.0.2.3
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        df-election::DF-Type:Auto/DP:0/DF-Preference:0/AC:1
        target:00:00:00:00:23:00
"

The ES RT as shown as target:00:00:00:00:23:00 in the extended community is auto-derived from the ESI bytes 2 to 7 (with the type byte being byte 1). Only PE-2 and PE-3 generate this RT and therefore import each other’s ES route.

The following message in log 99 on PE-3 shows the result of the DF election for EVI 1.

# log "99" on PE-3:
99 2021/02/25 17:04:27.467 CET MINOR: SVCMGR #2094 Base
"Ethernet Segment:ESI-23, EVI:1, Designated Forwarding state changed to:true"

The show service system bgp-evpn ethernet-segment name ESI-23 all command shows the ESI-label allocated to the PE: ES SHG Label 524282 in the CLI output for PE-3. In this example, this label is allocated by PE-3 for ESI-23 (a different one is allocated per ESI) and advertised in the AD per-ES route for ESI-23. The following output shows the AD per-ES and AD per-EVI (for evi 1) routes sent by PE-3 and received by PE-2.

  • The AD per-ES route can be identified by the MAX-ET in the ethernet-tag field (as per RFC 7432) and carries the ESI-label as well as the multi-homing mode (all-active in this case) in the ESI-label extended community (see EVPN route types and NLRIs).

    The user can enable the aggregation of AD per-ES routes by using the following command: configure service system bgp evpn ad-per-es-route route-target-type evi-route-target-set route-distinguisher-ip-address ip-address. If enabled, a single AD per-ES route with the associated RD and a set of EVI route-targets will be advertised (to a maximum of 128). When there are more than 128 EVIs defined in the ES, more than one route will be sent by the system.

    [ex:/configure service system bgp evpn ad-per-es-route]
    A:admin@PE-2# route-distinguisher-ip-address ?
    
     route-distinguisher-ip-address <ipv4-address>
     <ipv4-address>  - <d.d.d.d>
    
        IP address for route distinguisher for EVPN AD-ES routes
    

    The following AD per-ES route is received on PE-2:

    # AD per-ES route received on PE-2: 
    101 2021/02/25 17:06:06.205 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
    "Peer 1: 192.0.2.3: UPDATE
    Peer 1: 192.0.2.3 - Received BGP UPDATE:
        Withdrawn Length = 0
        Total Path Attr Length = 73
        Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
            Address Family EVPN
            NextHop len 4 NextHop 192.0.2.3
            Type: EVPN-AD Len: 25 RD: 192.0.2.3:1 ESI: 01:00:00:00:00:23:00:00:00:01,
                          tag: MAX-ET Label: 0
        Flag: 0x40 Type: 1 Len: 1 Origin: 0
        Flag: 0x40 Type: 2 Len: 0 AS Path:
        Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
        Flag: 0xc0 Type: 16 Len: 16 Extended Community:
            target:64500:1
            esi-label:524279/All-Active
    "
    
  • The AD per-EVI route has an eth-tag 0 and carries the service label in the NLRI.

# AD per-EVI route received on PE-2:
100 2021/02/25 17:06:06.204 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 73
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-AD Len: 25 RD: 192.0.2.3:1 ESI: 01:00:00:00:00:23:00:00:00:01,
                      tag: 0 Label: 8388512
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:1
        bgp-tunnel-encap:MPLS
"
[/]
A:admin@PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
      0                                                 LABEL 524282

u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
      MAX-ET                                            LABEL 0

-------------------------------------------------------------------------------
Routes : 2
===============================================================================
[/]
A:admin@PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01 hunt 
---snip---
===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Network        : n/a
Nexthop        : 192.0.2.3
From           : 192.0.2.3
Res. Nexthop   : 192.168.23.2
---snip---
Community      : target:64500:1 bgp-tunnel-encap:MPLS
---snip--- 
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:00:00:00:01
Tag            : 0                      
Route Dist.    : 192.0.2.3:1            
MPLS Label     : LABEL 524282 

---snip---

Network        : n/a 
Nexthop        : 192.0.2.3
From           : 192.0.2.3
Res. Nexthop   : 192.168.23.2
---snip---
Community      : target:64500:1 esi-label:524279/All-Active
---snip---
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:00:00:00:01
Tag            : MAX-ET                 
Route Dist.    : 192.0.2.3:1            
MPLS Label     : LABEL 0                
---snip---

From a service perspective, as soon as CE-11 sends some traffic, the PE learning the CE-11 MAC address will advertise it to the network. The remote PEs (PE-4 and PE-5) will create a new EVPN-MPLS ES destination to ESI-23, with two next-hops: PE-2 and PE-3. The following outputs show the following information:

  • PE-4 has learned AD per-EVI/ES routes for ESI-23 from PE-2 and PE-3, as well as the CE-11 MAC address from PE-3 (because MTU-1 picked up its link to PE-3 to send CE-11 frames).

    [/]
    A:admin@PE-4# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01
    ===============================================================================
     BGP Router ID:192.0.2.4        AS:64500       Local AS:64500
    ===============================================================================
     Legend -
     Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                     l - leaked, x - stale, > - best, b - backup, p - purge
     Origin codes  : i - IGP, e - EGP, ? - incomplete
    
    ===============================================================================
    BGP EVPN Auto-Disc Routes
    ===============================================================================
    Flag  Route Dist.         ESI                           NextHop
          Tag                                               Label
    -------------------------------------------------------------------------------
    u*>i  192.0.2.2:1         01:00:00:00:00:23:00:00:00:01 192.0.2.2
          0                                                 LABEL 524282
    
    u*>i  192.0.2.2:1         01:00:00:00:00:23:00:00:00:01 192.0.2.2
          MAX-ET                                            LABEL 0
    
    u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
          0                                                 LABEL 524282
    
    u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
          MAX-ET                                            LABEL 0
    
    -------------------------------------------------------------------------------
    Routes : 4
    ===============================================================================
    

    PE-4 has learned MAC address 00:00:11:11:11:11 of CE-11 in ESI-23. The BGP EVPN MAC route has PE-3 as next hop:

    [/]
    A:admin@PE-4# show router bgp routes evpn mac rd 192.0.2.3:1
    ===============================================================================
     BGP Router ID:192.0.2.4        AS:64500       Local AS:64500
    ===============================================================================
     Legend -
     Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                     l - leaked, x - stale, > - best, b - backup, p - purge
     Origin codes  : i - IGP, e - EGP, ? - incomplete
    
    ===============================================================================
    BGP EVPN MAC Routes
    ===============================================================================
    Flag  Route Dist.         MacAddr           ESI
          Tag                 Mac Mobility      Label1
                              Ip Address
                              NextHop
    -------------------------------------------------------------------------------
    u*>i  192.0.2.3:1         00:00:11:11:11:11 01:00:00:00:00:23:00:00:00:01
          0                   Seq:0             LABEL 524282
                              n/a
                              192.0.2.3
    
    -------------------------------------------------------------------------------
    Routes : 1
    ===============================================================================
    
  • In the FDB for VPLS-1, PE-4 has learned the CE-11 MAC address associated with a newly created EVPN-MPLS ES destination:

    [/]
    A:admin@PE-4# show service id 1 fdb mac 00:00:11:11:11:11
    
    ===============================================================================
    Forwarding Database, Service 1
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    1          00:00:11:11:11:11 eES:                    Evpn     02/25/21 17:14:43
                                 01:00:00:00:00:23:00:00:00:01
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    
  • Due to the aliasing function, the newly created EVPN-MPLS ES destination to ESI-23 has two next-hops (PE-2 and PE-3), to which PE-4 can load-balance the unicast traffic because ecmp 2 is configured in the VPLS-1 of PE-4.

    [/]
    A:admin@PE-4# show service id 1 evpn-mpls
    
    ===============================================================================
    BGP EVPN-MPLS Dest
    ===============================================================================
    TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                    Transport:Tnl                             Sup BCast Domain
    -------------------------------------------------------------------------------
    192.0.2.2       524281        0           bum             02/25/2021 16:46:48
                    ldp:65538                                 No
    192.0.2.3       524281        0           bum             02/25/2021 16:46:48
                    ldp:65537                                 No
    192.0.2.5       524281        0           bum             02/25/2021 16:46:52
                    ldp:65539                                 No
    -------------------------------------------------------------------------------
    Number of entries : 3
    -------------------------------------------------------------------------------
    ===============================================================================
    
    ===============================================================================
    BGP EVPN-MPLS Ethernet Segment Dest
    ===============================================================================
    Eth SegId                       Num. Macs               Last Change
    -------------------------------------------------------------------------------
    01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:14:43
    -------------------------------------------------------------------------------
    Number of entries: 1
    -------------------------------------------------------------------------------
    ===============================================================================
    ---snip---
    

    The show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01 command shows the next-hops that the EVPN-MPLS ES destination is resolved to.

    [/]
    A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01
    
    ===============================================================================
    BGP EVPN-MPLS Ethernet Segment Dest
    ===============================================================================
    Eth SegId                       Num. Macs               Last Change
    -------------------------------------------------------------------------------
    01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:14:43
    ===============================================================================
    
    ===============================================================================
    BGP EVPN-MPLS Dest TEP Info
    ===============================================================================
    TEP Address              Egr Label                Last Change
                             Transport:Tnl-Id
    -------------------------------------------------------------------------------
    192.0.2.2                524282                   02/25/2021 17:14:43
                             ldp:65538
    192.0.2.3                524282                   02/25/2021 17:14:43
                             ldp:65537
    -------------------------------------------------------------------------------
    Number of entries : 2
    -------------------------------------------------------------------------------
    ===============================================================================
    
  • PE-2 will show the CE-11 MAC address as learned locally in SAP lag-1:1 (because the data plane learning of the CE-11 MAC address happened in PE-2). For PE-3, even though it learned the MAC address from EVPN, it will install it as associated with SAP lag-1:1 because the EVPN route came with ESI-23, which is a local ESI. Because of this, whenever PE-3 receives a frame with MAC DA equal to the CE-11 MAC address, it will be able to forward the frame locally to the SAP lag-1:1. The following output shows the CE-11 MAC address as it is installed in PE-2 and PE-3:

    [/]
    A:admin@PE-2# show service id 1 fdb mac 00:00:11:11:11:11
    
    ===============================================================================
    Forwarding Database, Service 1
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    1          00:00:11:11:11:11 sap:lag-1:1             L/90     02/25/21 17:14:43
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    
    [/]
    A:admin@PE-3# show service id 1 fdb mac 00:00:11:11:11:11
    
    ===============================================================================
    Forwarding Database, Service 1
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    1          00:00:11:11:11:11 sap:lag-1:1             Evpn     02/25/21 17:14:43
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    

Single-active multi-homing concepts

EVPN-MPLS single-active multi-homing: mass-withdraw, backup path illustrates two concepts in EVPN single-active multi-homing: mass-withdraw and backup path.

Figure 4. EVPN-MPLS single-active multi-homing: mass-withdraw, backup path
  • With mass-withdraw, when ESI-45 goes down, PE-2 does not have to wait for all the MAC routes to be withdrawn to converge all the services. Instead, PE-4 will withdraw the AD per-ES routes (also the AD per-EVI and MAC routes) and that will be used at PE-2 as a notification to stop sending traffic to PE-4 for any MAC address associated with ESI-45.

  • With backup path, when PE-2 is notified of the ESI-45 failure due to the withdrawn AD routes, it will not flush any MAC address associated with ESI-45. Instead, it will change the next-hop of the EVPN-MPLS ES destination to the remaining PE in the ESI-45. Backup path only works when there are two PEs in the same ES. If there were more than two PEs in ESI-45, PE-2 would flush all the MAC addresses upon receiving a mass-withdraw notification, because it would not know who the new active PE is.

Single-active multi-homing configuration

The single-active multi-homing configuration example is based on EVPN-MPLS for VPLS services:

MTU-6 is connected to the EVPN network using single-active multi-homing. With the MTU-6 configuration, a VPLS service with active-standby spoke-SDP to PE-4 and PE-5 is configured. In PE-4 and PE-5, the SDP connected to MTU-6 is linked to ESI-45. Both will run the DF election algorithm for EVI 1, and the non-DF PE (PE-4 in this example) will bring down the spoke-SDP and notify MTU-6.

The following output shows the configuration of ESI-45 in PE-4 and PE-5, as well as the SDPs. The configuration of MTU-6 is also shown for completeness. It is important to keep the default — ignore-standby-signaling false — on MTU-6 spoke-SDPs because the PW switchover in MTU-6 will be triggered based on the PW status bits sent by PE-4 and PE-5.

SDP 46 with far-end MTU-6 is configured on PE-4:

# on PE-4:
configure {
    service {
        sdp 46 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.6
            }
        }

Ethernet segment ‟ESI-45” is configured on PE-4 as follows:

# on PE-4:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-45" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:01
                        multi-homing-mode single-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            sdp 46 {
                            }
                        }
                    }
                }
            }
        }

On PE-5, SDP 56 is configured as follows:

# on PE-5:
configure {
    service {
        sdp 56 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.6
            }
        }

Ethernet segment ‟ESI-45” is configured as follows on PE-5:

# on PE-5:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-45" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:01
                        multi-homing-mode single-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            sdp 56 {
                            }
                        }
                    }
                }
            }
        }

On MTU-6, the service configuration is as follows:

# on MTU-6:
configure {
    service {
        sdp 64 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.4
            }
        }
        sdp 65 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.5
            }
        }
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            endpoint "CORE" {
            }
            spoke-sdp 64:1 {
                endpoint {
                    name "CORE"
                }
                stp {
                    admin-state disable 
                }
            }
            spoke-sdp 65:1 {
                endpoint {
                    name "CORE"
                }
                stp {
                    admin-state disable 
                }
            }
            sap 1/2/1:1 {
            }
        }

For a detailed description of the base BGP-EVPN instance and ES configuration, see the All-active multi-homing configuration section. The es-activation-timer, esi, service-carving-mode, boot-timer, and admin-state commands are used in the same way as for all-active multi-homing. Only the differences compared to all-active multi-homing are described here:

  • multi-homing-mode single-active must be configured so that the ES acts as single-active. Optionally, multi-homing-mode single-active-no-esi-label can be configured, which controls the use of the ESI-label for single-active multi-homing. Although the ESI-label is always used in all-active multi-homing when sending BUM traffic between the PEs in the ES, it is configurable for single-active. However, Nokia recommends to use the default option (using ESI-label) to avoid potential transient issues when there is a DF switchover.

  • association>sdp <sdp-id> is configured so that the ES can be associated with the SDP connected to MTU-6. Although all-active multi-homing only allows LAG associations to the ES, single-active allows LAG, port, and SDP. In this example, SDP is the option, because the access network is MPLS-based.

Similar to the all-active multi-homing case, when configuring the service in PE-4 and PE-5, the service objects are automatically associated with the ESI-45, because they are defined in the SDPs linked to the ESI. The configuration for VPLS 1 on PE-5 is as follows:

# on PE-5:
configure {
    service {
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            bgp 1 {
            }
            bgp-evpn {
                evi 1
                mpls 1 {
                    admin-state enable
                    ingress-replication-bum-label true
                    ecmp 2
                    auto-bind-tunnel {
                        resolution any
                    }
                }
            }
            spoke-sdp 56:1 {
            }
        }

In all-active multi-homing, the non-DF does not bring down the service SAP associated with the ES (it only removes it from the default multicast list). However, in single-active multi-homing, the service spoke-SDP (or SAP, if that was the object associated) is brought operationally down. The following output shows the spoke-SDP state in PE-4 (non-DF), as operationally down with the StandbyForMHProtocol flag and the Local Pw Bits that are signaled to MTU-6:

[/]
A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" evi evi-1 1

===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
1             1             0                   no  02/25/2021 17:19:02
===============================================================================

===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.4                               02/25/2021 17:18:56
192.0.2.5                               02/25/2021 17:19:02
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================

Spoke-SDP 46:1 is operationally down on PE-4:

[/]
A:admin@PE-4# show service id 1 sdp

===============================================================================
Services: Service Destination Points
===============================================================================
SdpId            Type     Far End addr    Adm     Opr       I.Lbl     E.Lbl
-------------------------------------------------------------------------------
46:1             Spok     192.0.2.6       Up      Down      524280    524281
-------------------------------------------------------------------------------
Number of SDPs : 1
-------------------------------------------------------------------------------
===============================================================================

Spoke-SDP 46:1 is operationally down with the StandbyForMHProtocol flag:

[/]
A:admin@PE-4# show service id 1 sdp 46:1 detail | match Flag
Flags              : StandbyForMHProtocol

The local PW bits (pwFwdingStandby) are sent to MTU-6:

[/]
A:admin@PE-4# show service id 1 sdp 46:1 detail | match Pw 
Local Pw Bits      : pwFwdingStandby
Peer Pw Bits       : None

Single-active multi-homing operation

The same commands used in the All-active multi-homing operation section can be used for single-active; see that section.

The show service system bgp-evpn ethernet-segment name ESI-45 command shows an Ethernet-segment Oper Multi-homing in addition to the configured Multi-homing mode. This occurs because, in spite of configuring the ES as all-active, it may operate as single-active if there is a mismatch between the modes advertised by PE-4 and PE-5 in the AD per-ES routes (per RFC 7432). In this example, the configured and the operational value are the same:

[/]
A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45"

===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-45
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:45:00:00:00:01
Multi-homing            : singleActive       Oper Multi-homing  : singleActive
ES SHG Label            : 524278
Source BMAC LSB         : <none>
Sdp Id                  : 46
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================

As soon as CE-16 sends some traffic, the DF PE (PE-5) will learn the CE-16 MAC address and will advertise it to the network. The remote PEs (PE-2 and PE-3) will create a new EVPN-MPLS ES destination to ESI-45, but this time with only one next-hop, PE-5, because this is single-active multi-homing. The following outputs show the following information:

  • PE-2 has learned AD per-EVI/ES routes for ESI-45 from PE-4 and PE-5, as well as the CE-16 MAC address from an ES EVPN-MPLS destination, which is resolved to PE-5 (the DF for ESI-45).

    [/]
    A:admin@PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:45:00:00:00:01
    ===============================================================================
     BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
    ===============================================================================
     Legend -
     Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                     l - leaked, x - stale, > - best, b - backup, p - purge
     Origin codes  : i - IGP, e - EGP, ? - incomplete
    
    ===============================================================================
    BGP EVPN Auto-Disc Routes
    ===============================================================================
    Flag  Route Dist.         ESI                           NextHop
          Tag                                               Label
    -------------------------------------------------------------------------------
    u*>i  192.0.2.4:1         01:00:00:00:00:45:00:00:00:01 192.0.2.4
          0                                                 LABEL 524282
    
    u*>i  192.0.2.4:1         01:00:00:00:00:45:00:00:00:01 192.0.2.4
          MAX-ET                                            LABEL 0
    
    u*>i  192.0.2.5:1         01:00:00:00:00:45:00:00:00:01 192.0.2.5
          0                                                 LABEL 524282
    
    u*>i  192.0.2.5:1         01:00:00:00:00:45:00:00:00:01 192.0.2.5
          MAX-ET                                            LABEL 0
    
    -------------------------------------------------------------------------------
    Routes : 4
    ===============================================================================
    

    PE-2 has learned the CE-16 MAC address from an ES EVPN-MPLS destination:

    [/]
    A:admin@PE-2# show service id 1 fdb mac 00:00:16:16:16:16
    
    ===============================================================================
    Forwarding Database, Service 1
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    1          00:00:16:16:16:16 eES:                    Evpn     02/25/21 17:20:53
                                 01:00:00:00:00:45:00:00:00:01
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    

    On PE-2, the ES EVPN-MPLS destination is resolved to DF PE-5:

    [/]
    A:admin@PE-2# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:45:00:00:00:01
    
    ===============================================================================
    BGP EVPN-MPLS Ethernet Segment Dest
    ===============================================================================
    Eth SegId                       Num. Macs               Last Change
    -------------------------------------------------------------------------------
    01:00:00:00:00:45:00:00:00:01   1                       02/25/2021 17:20:53
    ===============================================================================
    
    ===============================================================================
    BGP EVPN-MPLS Dest TEP Info
    ===============================================================================
    TEP Address              Egr Label                Last Change
                             Transport:Tnl-Id
    -------------------------------------------------------------------------------
    192.0.2.5                524282                   02/25/2021 17:20:53
                             ldp:65539
    -------------------------------------------------------------------------------
    Number of entries : 1
    -------------------------------------------------------------------------------
    ===============================================================================
    
  • In this case, the local PEs, PE-4 and PE-5, will learn the CE MAC address from an EVPN-MPLS destination and a local spoke-SDP, respectively.

    [/]
    A:admin@PE-4# show service id 1 fdb mac 00:00:16:16:16:16
    ===============================================================================
    Forwarding Database, Service 1
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    1          00:00:16:16:16:16 eES:                    Evpn     02/25/21 17:20:53
                                 01:00:00:00:00:45:00:00:00:01
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    

    The ES EVPN-MPLS destination is resolved to DF PE-5:

    [/]
    A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:45:00:00:00:01
    
    ===============================================================================
    BGP EVPN-MPLS Ethernet Segment Dest
    ===============================================================================
    Eth SegId                       Num. Macs               Last Change
    -------------------------------------------------------------------------------
    01:00:00:00:00:45:00:00:00:01   1                       02/25/2021 17:20:53
    ===============================================================================
    
    ===============================================================================
    BGP EVPN-MPLS Dest TEP Info
    ===============================================================================
    TEP Address              Egr Label                Last Change
                             Transport:Tnl-Id
    -------------------------------------------------------------------------------
    192.0.2.5                524282                   02/25/2021 17:20:53
                             ldp:65539
    -------------------------------------------------------------------------------
    Number of entries : 1
    -------------------------------------------------------------------------------
    ===============================================================================
    

    DF PE-5 learns the CE-16 MAC address from a local spoke SDP:

    [/]
    A:admin@PE-5# show service id 1 fdb mac 00:00:16:16:16:16
    
    ===============================================================================
    Forwarding Database, Service 1
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    1          00:00:16:16:16:16 sdp:56:1                L/180    02/25/21 17:20:53
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    

Ethernet-segment failures

If either ES fails, a DF re-election will happen and the corresponding AD per-ES/EVI routes will be withdrawn, causing the remote PEs to modify the list of next-hops for the EVPN-MPLS ES destination. The following example illustrates a failure on the SDP between MTU-6 and PE-5 (the DF).

  1. A failure occurs in the LSP between MTU-6 and PE-5. This can be any event that brings the SDP down.

    # log "99" on PE-5:
    94 2021/02/25 17:25:00.632 CET MINOR: SVCMGR #2303 Base
    "Status of SDP 56 changed to admin=up oper=down"
    
  2. Immediately, PE-5 gives up the DF role and withdraws the ES route, as well as the AD routes and MAC routes. As soon as PE-4 receives any ES or AD withdraw, it will re-run the DF algorithm and, when the es-activation-timer expires, it will become the DF and activate its spoke-SDP.

    # log 99 on PE-5: 
    96 2021/02/25 17:25:00.633 CET MINOR: SVCMGR #2094 Base
    "Ethernet Segment:ESI-45, EVI:1, Designated Forwarding state changed to:false"
    

    The ES in PE-5 is operational down:

    [/]
    A:admin@PE-5# show service system bgp-evpn ethernet-segment name "ESI-45"
      
    ===============================================================================
    Service Ethernet Segment
    ===============================================================================
    Name                    : ESI-45
    Eth Seg Type            : None
    Admin State             : Enabled            Oper State         : Down
    ESI                     : 01:00:00:00:00:45:00:00:00:01
    Multi-homing            : singleActive       Oper Multi-homing  : singleActive
    ES SHG Label            : 524279
    Source BMAC LSB         : <none>
    Sdp Id                  : 56
    ES Activation Timer     : 3 secs
    Oper Group              : (Not Specified)
    Svc Carving             : auto               Oper Svc Carving   : auto
    Cfg Range Type          : primary
    ===============================================================================
    

    PE-5 is no longer the DF and the only DF candidate is PE-4:

    [/]
    A:admin@PE-5# show service system bgp-evpn ethernet-segment name "ESI-45" evi evi-1 1
    
    ===============================================================================
    EVI DF and Candidate List
    ===============================================================================
    EVI           SvcId         Actv Timer Rem      DF  DF Last Change
    -------------------------------------------------------------------------------
    1             1             0                   no  02/25/2021 17:25:01
    ===============================================================================
    
    ===============================================================================
    DF Candidates                           Time Added
    -------------------------------------------------------------------------------
    192.0.2.4                               02/25/2021 17:19:03
    -------------------------------------------------------------------------------
    Number of entries: 1
    ===============================================================================
    

    PE-4 becomes the DF and the spoke-SDP 46:1 is brought up.

    # log "99" on PE-4: 
    102 2021/02/25 17:25:03.598 CET MINOR: SVCMGR #2094 Base
    "Ethernet Segment:ESI-45, EVI:1, Designated Forwarding state changed to:true"
    
    103 2021/02/25 17:25:03.598 CET MINOR: SVCMGR #2326 Base
    "Status of SDP Bind 46:1 in service 1 (customer 1) local PW status bits changed to none"
    
    104 2021/02/25 17:25:03.598 CET MINOR: SVCMGR #2306 Base
    "Status of SDP Bind 46:1 in service 1 (customer 1) changed to admin=up oper=up flags="
    

    The ES is up in PE-4:

    [/]
    A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45"
    
    ===============================================================================
    Service Ethernet Segment
    ===============================================================================
    Name                    : ESI-45
    Eth Seg Type            : None
    Admin State             : Enabled            Oper State         : Up
    ESI                     : 01:00:00:00:00:45:00:00:00:01
    Multi-homing            : singleActive       Oper Multi-homing  : singleActive
    ES SHG Label            : 524278
    Source BMAC LSB         : <none>
    Sdp Id                  : 46
    ES Activation Timer     : 3 secs
    Oper Group              : (Not Specified)
    Svc Carving             : auto               Oper Svc Carving   : auto
    Cfg Range Type          : primary
    ===============================================================================
    

    PE-4 is the DF and there are no other DF candidates:

    [/]
    A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" evi evi-1 1
    
    ===============================================================================
    EVI DF and Candidate List
    ===============================================================================
    EVI           SvcId         Actv Timer Rem      DF  DF Last Change
    -------------------------------------------------------------------------------
    1             1             0                   yes 02/25/2021 17:25:04
    ===============================================================================
    
    ===============================================================================
    DF Candidates                           Time Added
    -------------------------------------------------------------------------------
    192.0.2.4                               02/25/2021 17:18:56
    -------------------------------------------------------------------------------
    Number of entries: 1
    ===============================================================================
    
  3. The remote PEs, PE-2 and PE-3, receive the BGP-EVPN routes withdrawal and modify the next-hop for the EVPN-MPLS ES destination.

    # on PE-2:
    186 2021/02/25 17:25:00.634 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
    "Peer 1: 192.0.2.5: UPDATE
    Peer 1: 192.0.2.5 - Received BGP UPDATE:
        Withdrawn Length = 0
        Total Path Attr Length = 86
        Flag: 0x90 Type: 15 Len: 82 Multiprotocol Unreachable NLRI:
            Address Family EVPN
            Type: EVPN-AD Len: 25 RD: 192.0.2.5:1 ESI: 01:00:00:00:00:45:00:00:00:01, 
                  tag: MAX-ET Label: 0 
            Type: EVPN-AD Len: 25 RD: 192.0.2.5:1 ESI: 01:00:00:00:00:45:00:00:00:01, 
                  tag: 0 Label: 0 
            Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.5:0 
                  ESI: 01:00:00:00:00:45:00:00:00:01, IP-Len: 4 Orig-IP-Addr: 192.0.2.5 
    "
    

    The ES EVPN-MPLS destination is resolved to the DF PE-4:

    [/]
    A:admin@PE-2# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:45:00:00:00:01
    
    ===============================================================================
    BGP EVPN-MPLS Ethernet Segment Dest
    ===============================================================================
    Eth SegId                       Num. Macs               Last Change
    -------------------------------------------------------------------------------
    01:00:00:00:00:45:00:00:00:01   1                       02/25/2021 17:25:19
    ===============================================================================
    
    ===============================================================================
    BGP EVPN-MPLS Dest TEP Info
    ===============================================================================
    TEP Address              Egr Label                Last Change
                             Transport:Tnl-Id
    -------------------------------------------------------------------------------
    192.0.2.4                524282                   02/25/2021 17:25:19
                             ldp:65538
    -------------------------------------------------------------------------------
    Number of entries : 1
    -------------------------------------------------------------------------------
    ===============================================================================
    

The following must be considered:

  • The DF election procedure is revertive, that is, when the failed SDP comes back up, PE-5 will take over again as DF and the network will re-converge.

  • The DF election is triggered by the following events:

    • Enabling an ES (admin-state enable) triggers the DF election for all the services in the ES.

    • A new update/withdrawal of an ES route (containing an ESI configured locally) triggers the DF election for all the services in the ESI.

    • A new update/withdrawal of an AD per-ES route (containing an ESI configured locally) triggers the DF election for all the services associated with the list of RTs received along with the route.

    • A new update of an AD per-ES route with a change in the ESI-label extended community (single-active bit or MPLS label) triggers the DF election for all the services associated with the list of RTs received along with the route.

    • A new update/withdrawal of an AD route per-EVI (containing an ESI configured locally) triggers the DF election for that service.

BGP-EVPN route selection in EVPN networks

The selection of the best route for a MAC address is as follows:

  • If a PE receives more than one route for the same MAC address, the best MAC route is chosen:

    • If the route key is equal in two or more routes (that is, the mac, mac-length, ip, ip-length, RD, eth-tag), then regular BGP selection applies:

      • If local-pref, AS-path, origin, and MED are equal, the lowest IGP distance to the BGP next-hop is chosen (unless ignore-nh-metric is configured). If the BGP next-hop is resolved by an LSP, the cost from the tunnel-table is used.

      • As a last resort tie-breaker, the route with the lowest originator ID, or received from the peer with the lowest BGP Identifier, is chosen (unless ignore-router-id is configured and the routes being compared are EBGP routes).

    • If the mac-length, mac, ip-length, ip, eth-tag are equal, and the RD is different, the EVPN selection process is applied in the following order:

      • Conditional static MAC addresses (local protected MAC addresses)

      • EVPN static MAC addresses (remote protected MAC addresses)

      • Data plane learned MAC addresses (regular learning on SAPs/SDP-bindings)

      • EVPN MAC addresses with higher sequence number

      • Lowest IP address (next-hop IP of the EVPN NLRI)

      • Lowest Ethernet tag (will be normally zero)

      • Lowest RD

  • After a MAC route is selected, the system checks for an associated ES.

    • If it has an ES, the system uses the MAC address as the EVPN-MPLS ES destination. The ES destination is constructed based on the AD per-EVI routes received for that ES (regardless of MAC address priorities with the ES).

    • The system selects the first ECMP number of AD per-EVI routes arranged by the IP address of PEs (lower IPs are selected first).

    • If the same PE has advertised multiple RDs, the system selects the route with the lowest RD for that PE.

In the example of EVPN-MPLS for VPLS services, PE-4 resolves the next-hops for ESI-23 as described in the second choice above, that is, because ECMP=2, the two available next-hops are chosen. If ECMP is changed to 1, PE-4 will pick up the lower IP (in the BGP next-hop). This is illustrated in the following output:

[/]
A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01

===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                       Num. Macs               Last Change
-------------------------------------------------------------------------------
01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:14:43
===============================================================================

===============================================================================
BGP EVPN-MPLS Dest TEP Info
===============================================================================
TEP Address              Egr Label                Last Change
                         Transport:Tnl-Id
-------------------------------------------------------------------------------
192.0.2.2                524282                   02/25/2021 17:14:43
                         ldp:65538
192.0.2.3                524282                   02/25/2021 17:14:43
                         ldp:65537
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================

When ECMP equals 1, only the BGP next hop with the lower IP is chosen:

# on PE-4:
configure {
    service {
        vpls "VPLS1" {
            bgp-evpn {
                mpls 1 {
                    ecmp 1
[/]
A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01

===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                       Num. Macs               Last Change
-------------------------------------------------------------------------------
01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:37:29
===============================================================================

===============================================================================
BGP EVPN-MPLS Dest TEP Info
===============================================================================
TEP Address              Egr Label                Last Change
                         Transport:Tnl-Id
-------------------------------------------------------------------------------
192.0.2.2                524282                   02/25/2021 17:14:43
                         ldp:65538
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
===============================================================================

Comparing EVPN multi-homing and BGP multi-homing

EVPN-MPLS services support EVPN-MH (EVPN multi-homing) and also BGP-MH as in chapter BGP Multi-Homing for VPLS Networks. While EVPN-MH is the standard way of providing access resiliency in RFC 7432, BGP-MH is also a standard mechanism supported in VPLS or EVPN networks. The following table provides some comparison between both technologies.

Table 1. Comparing EVPN multi-homing and BGP multi-homing

VPN Requirements

EVPN-MH

BGP-MH

Comments

All-active MH (flow-based load-balancing)

Yes

No

EVPN-MH provides better bandwidth utilization

Single-active MH (service-based load-balancing)

Yes

Yes

DF PE election - automatic service balancing

Yes

Service-carving

No

Requires vsi policies and LP manipulation

EVPN-MH provides better automation

DF PE election – manual configuration per service

Yes

No

EVPN-MH allows for manual DF config for EVIs and ISIDs (2 PEs)

Split-horizon indication in the data plane

Yes

ESI-label

No

Prevents transient loops when dual-active DFs show up

DF indication in the control plane

No

Yes

BGP MH guarantees one DF at a time. EVPN relies on timers to ensure one DF at a time

Allows multiple SAPs or SDP-bindings per service on the same site

No

Yes

Through the use of SHGs

Boot timer and site(es)-activation-timers

Yes

Yes

BGP-MH supports more granular configuration (service level)

Support for oper-groups

No

Yes

Non-DF notification to the CE (MPLS and CFM)

Yes

Yes

Avoids blackholing

In addition to the preceding comparison, the following configuration excerpt compares EVPN-MH with BGP-MH on a bgp-evpn VPLS service and shows that, while EVPN-MH does not have any configuration at service level, BGP-MH is configured within the VPLS context, which gives a more granular control over the redundancy provided. See the BGP Multi-Homing for VPLS Networks chapter for more information about BGP-MH.

[ex:/configure service system]
A:admin@PE-4# info
    bgp {
        evpn {
            ethernet-segment "ESI-45" {
                admin-state enable
                esi 0x01000000004500000001
                multi-homing-mode single-active
                df-election {
                    es-activation-timer 3
                }
                association {
                    sdp 46 {
                    }
                }
            }
        }
    }
[ex:/configure service vpls "VPLS1"]
A:admin@PE-4# info
    admin-state enable
    service-id 1
    customer "1"
    bgp 1 {
    }
    bgp-evpn {
        evi 1
        mpls 1 {
            admin-state enable
            ingress-replication-bum-label true
            ecmp 2
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    spoke-sdp 46:1 {
    }

For BGP multi-homing, site ‟site-1” is configured, as follows. The RD needs to be configured in the bgp context.

[ex:/configure service vpls "VPLS1"]
A:admin@PE-4# info
    admin-state enable
    service-id 1
    customer "1"
    bgp 1 {
        route-distinguisher "192.0.2.4:1"
    }
    bgp-evpn {
        evi 1
        mpls 1 {
            admin-state enable
            ingress-replication-bum-label true
            ecmp 2
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    spoke-sdp 46:1 {
    }
    bgp-mh-site "site-1" {
        admin-state enable
        id 1
        activation-timer 3
        spoke-sdp 46:1
    }

Proxy-ARP/ND configuration for EVPN-MPLS networks

Although not strictly a BGP-EVPN configuration, vpls>proxy-arp and vpls>proxy-nd functions are typically enabled along with EVPN-MPLS in order to reduce the amount of flooding in the network. The proxy-ARP/ND agent in the VPLS service will snoop ARP-requests and/or Neighbor Solicitation messages and will reply to those messages locally (if the information is known) without having to flood the requests to the network.

The configuration options for proxy-ARP are the following:

[ex:/configure service vpls "VPLS1"]
A:admin@PE-2# proxy-arp ?

 proxy-arp

 admin-state           - Administrative state of the proxy
 age-time              - Aging timer for proxy entries, where entries are flushed
                         upon timer expiry
 apply-groups          - Apply a configuration group at this level
 apply-groups-exclude  - Exclude a configuration group at this level
 duplicate-detect      + Enter the duplicate-detect context
 dynamic-arp           + Enter the dynamic-arp context
 dynamic-populate      - Populate proxy ARP entries from snooped GARP/ARP/ND 
                         messages on SAPs/SDP-bindings
 evpn                  + Enter the evpn context
 send-refresh          - Time at which to send a refresh message
 static-arp            + Enter the static-arp context
 table-size            - Maximum number of learned and static entries allowed in 
                         the proxy table of this service

The configuration options for proxy-ND are the following:

[ex:/configure service vpls "VPLS1"]
A:admin@PE-2# proxy-nd ?

 proxy-nd

 admin-state           - Administrative state of the proxy
 age-time              - Aging timer for proxy entries, where entries are flushed 
                         upon timer expiry
 apply-groups          - Apply a configuration group at this level
 apply-groups-exclude  - Exclude a configuration group at this level
 duplicate-detect      + Enter the duplicate-detect context
 dynamic-neighbor      + Enter the dynamic-neighbor context
 dynamic-populate      - Populate proxy ARP entries from snooped GARP/ARP/ND 
                         messages on SAPs/SDP-bindings
 evpn                  + Enter the evpn context
 send-refresh          - Time at which to send a refresh message
 static-neighbor       + Enter the static-neighbor context
 table-size            - Maximum number of learned and static entries allowed in 
                         the proxy table of this service

When proxy-ARP/ND is enabled, the following configuration guidelines must be followed:

  • dynamic-populate should be used only in networks with a consistent configuration of this command in all PEs.

  • When using dynamic-populate, the age-time value should be configured to a value equal to three times the send-refresh value. This will help reduce the EVPN withdrawals and re-advertisements in the network.

  • With large age-time values, it would be sufficient to configure the send-refresh value to half of the proxy-ARP/ND age-time or FDB age-time.

  • In scaled environments (with thousands of services), it is not recommended to set the send-refresh value to less than 300 s. In such scenarios, Nokia recommends using a minimum proxy-ARP/ND age-time and FDB age of 900 s.

  • The use of the following commands reduces or suppresses the ARP/ND flooding in an EVPN network, because EVPN MAC routes replace the function of the regular data plane ARP/ND messages:

    • proxy-arp>evpn>flood> gratuitous-arp false

    • proxy-arp>evpn>flood> unknown-arp-req false

    • proxy-nd>evpn>flood> unknown-neighbor-solicitation false

    • proxy-nd>evpn>flood> unknown-neighbor-advertise-router false

    • proxy-nd>evpn>flood> unknown-neighbor-advertise-host false

  • Nokia recommends using the preceding commands only in EVPN networks where the CEs are routers directly connected to an SR OS node acting as the PE. Networks using aggregation switches between the host/routers and the PEs should flood GARP/ND messages in EVPN to make sure the remote caches are updated and BGP does not miss the advertisement of these entries.

  • When duplicate-detect anti-spoof-mac is used with proxy-ARP/ND, ingress filters (in the access SAPs/SDP-bindings) should be configured to drop all traffic with destination anti-spoof-mac. The same MAC address should be configured in all PEs where duplicate-detect is active.

  • When proxy-ND is used, the configuration of the following commands should be consistent in all the PEs in the network:

    • proxy-nd>evpn>flood> unknown-neighbor-advertise-router

    • proxy-nd>evpn>flood> unknown-neighbor-advertise-host

    • proxy-nd>evpn>advertise-neighbor-type

  • Because EVPN does not propagate the router flag in IPv6--> MAC address advertisements, in a mixed network with hosts and routers where evpn>advertise-neighbor-type router is configured, unsolicited host NA messages should be flooded so that the entire network gets to learn all of the host and router ND entries. In the same way, evpn>advertise-neighbor-type host should be configured so that unsolicited router NA messages are flooded.

Finally, along with proxy-ARP/ND, vpls>fdb>discard-unknown true may be used in some EVPN-MPLS deployments where all the CEs are routers and they announce themselves to the network by sending GARPs or NAs (Neighbor Solicitation messages). According to RFC 7432, whether or not to flood packets to unknown destination MAC addresses should be an administrative choice, depending on how learning happens between CEs and PEs. Discard-unknown provides that administrative choice in case all the MAC addresses in an EVI can be learned even before any traffic is exchanged.

Proxy-ARP/ND along with discard-unknown helps reduce the BUM traffic in an EVPN network significantly; however, their use must be analyzed and considered, depending on the type of CEs in the EVI.

An example of proxy-ARP configuration is as follows. This configuration should be added to all PEs. When a new ARP message is received on any of the PEs, they will learn the IP-MAC address pair and will advertise it to the network.

# on PE-2, PE-3, PE-4, PE-5:
configure {
    service {
        vpls "VPLS1" {
            proxy-arp {
                admin-state enable
                dynamic-populate true
                age-time 900
                send-refresh 300
            }

Enabling proxy-ARP increases the number of MAC/IP routes being sent by the PEs. This is due to the following reasons:

  • An additional MAC/IP route will be advertised per new learned IP-MAC address pair, regardless of having advertised the same MAC address already.

  • A MAC per VPLS service will be advertised with a system MAC address. That MAC address will be used as MAC SA for proxy-ARP confirm messages when an IP moves to a different PE.

The following output shows the MAC/IP routes on PE-2 when proxy-ARP is enabled in the network.

[/]
A:admin@PE-2# show router bgp routes evpn mac
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         02:17:ff:00:03:3a ESI-0
      0                   Static            LABEL 524282
                          n/a
                          192.0.2.3

u*>i  192.0.2.4:1         02:1b:ff:00:03:3a ESI-0
      0                   Static            LABEL 524282
                          n/a
                          192.0.2.4

u*>i  192.0.2.5:1         00:00:16:16:16:16 01:00:00:00:00:45:00:00:00:01
      0                   Seq:0             LABEL 524282
                          n/a
                          192.0.2.5

u*>i  192.0.2.5:1         02:1f:ff:00:03:3a ESI-0
      0                   Static            LABEL 524282
                          n/a
                          192.0.2.5

-------------------------------------------------------------------------------
Routes : 4
===============================================================================

Troubleshooting and debug commands

When troubleshooting an EVPN-MPLS network, the following show commands and debug commands are recommended, as already discussed throughout this chapter:

  • show redundancy bgp-evpn-multi-homing

  • show router bgp routes evpn (and filters)

  • show service evpn-mpls [<TEP ip-address>]

  • show service id bgp-evpn

  • show service id evpn-mpls (and modifiers)

  • show service id fdb (and modifiers)

  • show service system bgp-evpn

  • show service system bgp-evpn ethernet-segment (and modifiers)

  • debug router bgp update (in classic CLI)

  • log-id "99"

In addition to the preceding commands, the following tools dump commands may also help:

  • tools dump service evpn usage — This command shows the amount of EVPN-MPLS (and EVPN-VXLAN) destinations consumed in the system.

  • tools dump service system bgp-evpn ethernet-segment <name> evi <[1..65535]> df — This command computes the DF election for a specific ESI and EVI. Note: The show service system bgp-evpn ethernet-segment commands shows whether the local PE is DF or non-DF for a specific EVI, but it does not show who the DF is if it is not the local PE. In case of more than 2 PEs in the ES, this command may be especially useful.

Some examples are provided below for PE-2. PE-2 is showing seven EVPN-MPLS destinations due to the following:

  • Each remote PE consumes one EVPN-MPLS destination for unicast (if they advertise MAC/IP routes to PE-2 and the ingress-replication-bum-label is configured in all the PEs). PE-2 has three remote unicast EVPN-MPLS destinations.

  • Each remote PE consumes one EVPN-MPLS destination for multicast (if they advertise inclusive multicast routes to PE-2). PE-2 has three remote multicast EVPN-MPLS destinations.

  • Each remote ES consumes one EVPN-MPLS destination (it is only one per ES, regardless of the multi-homing mode and the number of PEs in the ES). PE-2 has one remote ES (ESI-45).

[/]
A:admin@PE-2# tools dump service evpn usage

vxlan-evpn-mpls usage statistics at 02/25/2021 17:40:37:

MPLS-TEP                                        :             3
VXLAN-TEP                                       :             0
Total-TEP                                       :      3/ 16383

Mpls Dests (TEP, Egress Label + ES + ES-BMAC)   :             7
Mpls Etree Leaf Dests                           :             0
Vxlan Dests (TEP, Egress VNI + ES)              :             0
Total-Dest                                      :      7/196607

Sdp Bind +  Evpn Dests                          :      8/245759
ES L2/L3 PBR                                    :      0/ 32767
Evpn Etree Remote BUM Leaf Labels               :             0

To compute the DF election for EVI 1:

[/]
A:admin@PE-2# tools dump service system bgp-evpn ethernet-segment "ESI-23" evi 1 df

[02/25/2021 17:40:47] Computed DF: 192.0.2.3 (Remote) (Boot Timer Expired: Yes) 

Conclusion

SR OS has a full RFC 7432 EVPN-MPLS implementation including single-active and all-active multi-homing. This example has shown how to configure and operate EVPN-MPLS for a simple non multi-homing configuration as well as a multi-homing configuration. Other topics, such as the integration of VPLS objects with EVPN-MPLS and proxy-ARP/ND, have also been discussed.