EVPN for MPLS Tunnels

EVPN all-active multi-homing is built around three concepts: DF election, split-horizon (with an ESI-label), and aliasing, as shown in EVPN-MPLS all-active multi-homing concepts, from left to right.

• With DF election, when PE-4 sends BUM traffic to the remote ES (CE-2), only one PE segment sends the BUM packets to the ES (PE-3 is the DF in the preceding example, and is elected to send BUM packets to CE-2). The non-DF, PE-2, removes the LAG SAP from the default multicast list (PE-2 does not bring CE-2 down, because it still needs to send upstream/downstream unicast traffic). PE-2 and PE-3 elect a DF for each service, based on the ES routes and the service-carving algorithm.

• With split-horizon, the PE part of the ES (PE-3 in the preceding example) identifies the BUM packets coming from the PE for the remote (PE-2), but within the same ES (ESI-2), and filters the packets so that they are not sent back to the ES, creating duplication. When PE-2 (non-DF) sends BUM traffic to PE-3 (DF), it uses a special MPLS label in the data path that PE-3 previously advertised for ESI-2 in an AD per-ES route. When PE-3 does an ingress lookup, it recognizes the ESI-label and filters the traffic (PE-3 still sends the BUM traffic to other SAPs/SDP-bindings).

• With aliasing, remote PEs that are not part of the ES can load-balance unicast traffic to all the PEs that are part of the ES, irrespective of from which PE a destination MAC address was learned. PE-4 will create an EVPN destination to ESI-2 that will be resolved to the two next-hops: PE-2 and PE-3. Unicast load-balancing will happen as long as ECMP > 1 is enabled in PE-4.

Nokia recommends the use of ingress-replication-bum-label on the PEs that are part of an all-active ES. In an all-active multi-homing scenario, if a specified MAC address (for example, the CE-2 MAC address in the left-hand-side diagram), is not learned yet in a remote PE (for example, PE-4), but is known in the two PEs of the ES (for example, PE-2 and PE-3), the latter PEs might send duplicated packets to the CE.

This issue is solved by the use of ingress-replication-bum-label in PE-2 and PE-3. If configured, PE-2/PE-3 will know that the received packet is an unknown unicast packet; therefore, the Non-DF (PE-2) will not send the packet to CE-2 and there will not be duplication.

The all-active multi-homing configuration example is based on EVPN-MPLS for VPLS services.

MTU-1 is connected to the EVPN network using all-active multi-homing. According to RFC 7432, MTU-1 will be able to send traffic to both PEs for VPLS-1. Regular LAG load-balancing is used in MTU-1. Remote PEs such as PE-4 or PE-5 will be able to load-balance the unicast traffic to PE-2 and PE-3. PE-2 and PE-3 will discover that both are part of ESI-23 (due to the exchange of ES routes) and will elect a DF for VPLS-1. The non-DF for VPLS-1, in this case PE-2, will remove lag-1:1 from the VPLS-1 default multicast list. Also, when PE-2 and PE-3 send BUM traffic to each other, they will insert an ESI-label so that they can identify that the source of the BUM packet is ESI-23.

The following output shows the configuration of ESI-23 in PE-2 and PE-3, as well as the LAG interfaces for all-active multi-homing (see EVPN-MPLS for VPLS services). The configuration of LAG-1 in MTU-1 is also shown. Per RFC 7432, only a CE/MTU with a LAG can be connected to an all-active multi-homing ES. No other configuration is permitted on the CE for all-active multi-homing.

LAG 1 is configured on MTU-1, PE-2, PE-3, as follows:

# on MTU-1:
configure
    lag 1
        mode access
        encap-type dot1q
        port 1/1/1
        port 1/1/2
        lacp active administrative-key 32768
        no shutdown

# on PE-2:
configure
    lag 1
        mode access
        encap-type dot1q
        port 1/1/2
        lacp active administrative-key 1 system-id 00:00:00:00:02:03
        no shutdown

# on PE-3:
configure
    lag 1
        mode access
        encap-type dot1q
        port 1/1/1
        lacp active administrative-key 1 system-id 00:00:00:00:02:03
        no shutdown

Ethernet segment ‟ESI-23” is configured in the service system bgp-evpn context on PE-2 and PE-3, as follows:

# on PE-2, PE-3:
configure
    service
        system
            bgp-evpn
                ethernet-segment "ESI-23" create
                    esi 01:00:00:00:00:23:00:00:00:01
                    es-activation-timer 3
                    service-carving
                        mode auto
                    exit
                    multi-homing all-active
                    lag 1
                    no shutdown
                exit

When configuring an ES, the following must be considered:

• Any EVPN parameter that is not specific to any particular VPLS service, and is common to all the EVIs, is configured in a base BGP-EVPN instance located at config>service>system>bgp-evpn. In this base instance, the following attributes may be configured:

  • ethernet-segments

  • the base BGP-EVPN instance route-distinguisher that will be used for the ES routes. If this route-distinguisher is not configured, by default a type-1 RD will be derived as system-ip:0, as shown in the command help:

  *A:PE-2>config>service>system>bgp-evpn# route-distinguisher ?
    - no route-distinguisher
    - route-distinguisher <rd>
  
   <rd>                 : <ip-addr:comm-val>
                          ip-addr        - a.b.c.d
                          comm-val       - [0..65535]
                          default: system-ip:0
  

• The ES must be configured with a name and can contain the following parameters when configured for all-active multi-homing:

  • esi — 10-byte identifier that represents the ES in the BGP control plane. The same ESI must be configured in all the PEs connected to the same CE/MTU (using a unique value that cannot be associated with any other CE/MTU/access network). RFC 7432 defines five different types of ESI. In SR OS, the type byte, as well as the other 9 bytes can be arbitrarily configured.

  • multi-homing all-active — This command indicates that the ES is in all-active mode.

  • lag <lag-id> — The LAG connected to the CE/MTU must be added to the ES. In this example, lag-1 is added to ESI-23, on both PE-2 and PE-3. Although a different LAG-id may have been assigned to the same ES on PE-2 and PE-3, PE-2 and PE-3 must have the same configuration on the ES LAG; that is, encap-type. Also, if LACP is added (it is not mandatory), both PEs must have the same admin-key, system-id, and system-priority. MTU-1 will see PE-2 and PE-3 as a single LAG peer. For all-active multi-homing, only the lag option is accepted by the system; port or sdp are not accepted.

  • [no] shutdown — This command controls the administrative state of the ES.

• The preceding parameters are the minimum necessary so that the ES can be activated. In addition to those parameters, there are a few more that the user can configure if requiring values different from the default ones:

  • es-activation-timer [0..100] can be configured at redundancy>bgp-evpn-multi-homing>es-activation-timer or at service>system>bgp-evpn>eth-seg>es-activation-timer level (the most specific value is used).

    The es-activation-timer operation is as follows:

    • Upon reception of an ES, AD per-ES/EVI route update/withdrawal for a local ESI, the DF-candidate list of IPs is updated and the DF election algorithm is run without waiting for any timer.

    • If the result of the DF election requires the PE to be promoted from non-DF to DF, the es-activation-timer will start, and only after its expiration will the PE add the SAP to the default-multicast list. Transitions from non-DF to non-DF, or from DF to non-DF, are immediate and do not wait for any timer.

    • This use of an es-activation-timer value minimizes the risks of loops and packet duplication due to transient multiple DFs.

    • The same es-activation-timer must be configured in all the PEs that are part of the same ESI. The user must configure either a long timer to minimize the risks of loops/duplication, or es-activation-timer = 0 to speed up the convergence for NDF to DF transitions. The default value is 3 seconds.

  • service-carving — As defined in RFC 7432, service-carving controls the distribution of DF/non-DF roles across the different services defined in an ES.

    *A:PE-2>config>service>system>bgp-evpn>eth-seg>service-carving# mode ?
     - mode {auto|manual|off}
    
     <auto|manual|off>    : auto|manual|off
    

    *A:PE-2>config>service>system>bgp-evpn>eth-seg>service-carving# manual ?
      - manual
    
     [no] evi             - Configure EVI range (primary for non-preference based DF
                            election and lowest-preference for preference based DF
                            election)
     [no] isid            - Configure ISID range (primary for non-preference based DF 
                            election and lowest-preference for preference based DF
                            election)
     [no] preference      + Configure DF preference election information
    

    As shown above, service-carving has three different modes:

  • service-carving mode auto (default) — The DF election algorithm will run the function [V(evi) mod N(peers) = i(ordinal)] to know who the DF for a specified service and ESI is. In this example, ESI-23 is configured with mode auto; therefore, for VPLS-1 (with EVI-1), PE-3 will be elected as DF because evi(1) mod (2)peers = 1, and the ordinal 1 corresponds to the second lowest IP, PE-3. The algorithm takes the configured evi in the service; therefore, the evi is mandatory, and for the same service must match in all the PEs that are part of the ES. This guarantees that the election algorithm is consistent across all the PEs of the ESI.

  • service-carving mode manual — The user can manually decide for which evi identifiers the PE is DF: service-carving mode manual / manual evi <start> [to <to>]. The PE will be non-DF for the non-specified EVIs. If service-carving mode manual is configured, but no range is defined, all the services are considered to be non-DF. If a range is configured, but the service-carving is not mode manual, the range has no effect. Only two PEs are supported when service-carving mode manual is configured.

  • service-carving mode off — The lowest originator IP will win the election for a specified service and ES.

  • Because the evi is used for the service-carving algorithm, it must always be configured in a service with SAPs/SDP bindings created in an ES, regardless of the service-carving mode (service-carving off, auto, or manual).

Although not configured as part of the ES, the config>redundancy>bgp-evpn-multi-homing>boot-timer allows the necessary time for the control plane protocols to come up after the PE has rebooted, and before bringing up the ESs and running the DF algorithm. Some considerations about the boot timer:

• The boot timer should use a value long enough to allow the IOMs and BGP sessions to come up before exchanging ES routes and run the DF election for each EVI (it is 10 s, by default).

• The boot timer runs per EVI on the ESs in the system. While system-up-time <boot-timer>, the system will not run the DF election for any EVI. When the boot timer expires, the DF election for the EVI is run and, if the system is elected DF for the EVI, the es-activation-timer will start.

• The system will not advertise ES routes until the boot timer expires. This guarantees that the peer ES PEs do not run the DF election either, until the PE is ready to become the DF, if needed.

• The following show command displays the configured boot timer, as well as the remaining timer if the system is still in boot stage.

  
  *A:PE-2# show redundancy bgp-evpn-multi-homing 
  
  ===============================================================================
  Redundancy BGP EVPN Multi-homing Information
  ===============================================================================
  Boot-Timer              : 10 secs                 
  Boot-Timer Remaining    : 0 secs                  
  ES Activation Timer     : 3 secs                  
  ===============================================================================
  

After ESI-23 is configured in PE-2 and PE-3, the lag-1 SAPs in both PEs can be added to the VPLS-1 service. Until the ESI-23 is successfully enabled, the LAG SAPs will be kept down with a StandByForMHProtocol flag. This is illustrated in the following example for PE-2.

# on PE-2:
configure 
    service 
        system 
            bgp-evpn 
                ethernet-segment "ESI-23" 
                    shutdown
                exit
            exit
        exit
    exit
    service 
        vpls "VPLS1"
            sap lag-1:1 create
                no shutdown
            exit

*A:PE-2# show service id 1 sap lag-1:1 detail | match "  Oper State"
Admin State        : Up                       Oper State        : Down

*A:PE-2# show service id 1 sap lag-1:1 detail | match Flag 
Flags              : StandByForMHProtocol

# on PE-2:
configure 
    service 
        system 
            bgp-evpn 
                ethernet-segment "ESI-23" 
                    no shutdown
                exit 

*A:PE-2# show log log-id 99 

===============================================================================
Event Log 99 log-name 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=107  (not wrapped)]
 
106 2021/02/17 14:38:48.980 UTC MINOR: SVCMGR #2203 Base
"Status of SAP lag-1:1 in service 1 (customer 1) changed to admin=up oper=up flags="

To confirm that all-active multi-homing is working correctly for ESI-23, the user can use the following commands:

• show service system bgp-evpn — Shows the RD is used for the ES route.

• show service system bgp-evpn ethernet-segment — Shows all the ESs configured in the PE and their admin/operational status.

• show service system bgp-evpn ethernet-segment name ESI-23 evi 1 — Shows the DF candidate PEs for EVI 1 and whether the system is DF for EVI.

• show service system bgp-evpn ethernet-segment name ESI-23 all — Shows all the information related to a specific ESI.

The base BGP-EVPN information includes the RD:

*A:PE-2# show service system bgp-evpn
 
===============================================================================
System BGP EVPN Information
===============================================================================
Eth Seg Route Dist.               : <none>
Eth Seg Oper Route Dist.          : 192.0.2.2:0
Eth Seg Oper Route Dist Type      : default
Ad Per ES Route Target            : evi-rt
Leaf Label                        : 0
Mcast Leave Sync Prop             : 5
Attribute Uniform Prop            : Disabled
BGP Path Selection                : Disabled
===============================================================================

The following command shows the configured ESs in the PE and their status:

*A:PE-2# show service system bgp-evpn ethernet-segment 

===============================================================================
Service Ethernet Segment
===============================================================================
Name                             ESI                           Admin     Oper
-------------------------------------------------------------------------------
ESI-23                           01:00:00:00:00:23:00:00:00:01 Enabled   Up
-------------------------------------------------------------------------------
Entries found: 1
===============================================================================

The following command shows that PE-2 is not the DF and the DF candidate PEs for EVI 1 are PE-2 and PE-3:

*A:PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" evi 1 

===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
1             1             0                   no  02/17/2021 14:38:49
===============================================================================
 
===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.2                               02/17/2021 14:38:49
192.0.2.3                               02/17/2021 14:38:51
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================

The following command shows all information related to ESI-23 on PE-2:

*A:PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" all 
 
===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-23
Eth Seg Type            : None 
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:23:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
ES SHG Label            : 524279 
Source BMAC LSB         : <none>             
Lag Id                  : 1
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================
 
===============================================================================
EVI Information 
===============================================================================
EVI                 SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
1                   1                   0                   no
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
 
-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI                                     DF Address
-------------------------------------------------------------------------------
1                                       192.0.2.2
1                                       192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---

The following command shows all information related to ESI-23 on PE-3:

*A:PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" all 

===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-23
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:23:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
ES SHG Label            : 524280
Source BMAC LSB         : <none>             
Lag Id                  : 1
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary 
===============================================================================
===============================================================================
EVI Information 
===============================================================================
EVI                 SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
1                   1                   0                   yes
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI                                     DF Address
-------------------------------------------------------------------------------
1                                       192.0.2.2
1                                       192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---

The preceding commands show the ESI-23 configuration on both PEs and the result of the DF election for EVI 1.

The following output shows the ES route received on PE-2:

# on PE-2:
129 2021/02/17 14:31:29.996 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 71
    Flag: 0x90 Type: 14 Len: 34 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.3:0 ESI: 01:00:00:00:00:23:00:00:00:01,
                          IP-Len: 4 Orig-IP-Addr: 192.0.2.3
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        df-election::DF-Type:Auto/DP:0/DF-Preference:0/AC:1
        target:00:00:00:00:23:00
"

The ES RT as shown as target:00:00:00:00:23:00 in the extended community is auto-derived from the ESI bytes 2 to 7 (with the type byte being byte 1). Only PE-2 and PE-3 generate this RT and therefore import each other’s ES route.

The following message in log 99 on PE-3 shows the result of the DF election for EVI 1.

86 2021/02/17 14:31:34.395 UTC MINOR: SVCMGR #2094 Base
"Ethernet Segment:ESI-23, EVI:1, Designated Forwarding state changed to:true"

The show service system bgp-evpn ethernet-segment name ESI-23 all command shows the ESI-label allocated to the PE: ES SHG Label 524280 in the CLI output for PE-3. In this example, this label is allocated by PE-3 for ESI-23 (a different one is allocated per ESI) and advertised in the AD per-ES route for ESI-23. The following output shows the AD per-ES and AD per-EVI (for evi 1) routes sent by PE-3 and received by PE-2.

• The AD per-ES route can be identified by the MAX-ET in the Ethernet-tag field (as per RFC 7432) and carries the ESI-label as well as the multi-homing mode (all-active in this case) in the ESI-label extended community (see EVPN route types and NLRIs).

  The user can enable the aggregation of AD per-ES routes by using the following command: config service system bgp-evpn ad-per-es-route-target evi-rt-set route-distinguisher ip-address. If enabled, a single AD per-ES route with the associated RD and a set of EVI route-targets will be advertised (to a maximum of 128). When there are more than 128 EVIs defined in the Ethernet-segment, more than one route will be sent by the system.

• The AD per-EVI route will have an eth-tag 0 and will carry the service label in the NLRI.

# AD per-ES route received on PE-2: 
156 2021/02/24 08:39:37.907 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 73
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-AD Len: 25 RD: 192.0.2.3:1 ESI: 01:00:00:00:00:23:00:00:00:01, 
                      tag: MAX-ET Label: 0
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:1
        esi-label:524282/All-Active
"

# AD per-EVI route received on PE-2:
155 2021/02/24 08:39:37.906 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 73
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-AD Len: 25 RD: 192.0.2.3:1 ESI: 01:00:00:00:00:23:00:00:00:01, 
                      tag: 0 Label: 8388464
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:1
        bgp-tunnel-encap:MPLS
"

*A:PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01 
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
      0                                                 LABEL 524282

u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
      MAX-ET                                            LABEL 0

-------------------------------------------------------------------------------
Routes : 2
===============================================================================

*A:PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01 hunt
---snip---
===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Network        : n/a
Nexthop        : 192.0.2.3
From           : 192.0.2.3
Res. Nexthop   : 192.168.23.2
---snip---
Community      : target:64500:1 bgp-tunnel-encap:MPLS
---snip--- 
EVPN type      : AUTO-DISC
ESI            : 01:00:00:00:00:23:00:00:00:01
Tag            : 0
Route Dist.    : 192.0.2.3:1
MPLS Label     : LABEL 524282

---snip---

Network        : n/a 
Nexthop        : 192.0.2.3
From           : 192.0.2.3
Res. Nexthop   : 192.168.23.2
---snip---
Community      : target:64500:1 esi-label:524280/All-Active
---snip---
EVPN type      : AUTO-DISC
ESI            : 01:00:00:00:00:23:00:00:00:01
Tag            : MAX-ET
Route Dist.    : 192.0.2.3:1
MPLS Label     : LABEL 0
---snip---

From a service perspective, as soon as CE-11 sends some traffic, the PE learning the CE-11 MAC address will advertise it to the network. The remote PEs (PE-4 and PE-5) will create a new EVPN-MPLS ES destination to ESI-23, with two next-hops: PE-2 and PE-3. The following outputs on PE-4 show the following information:

• PE-4 has learned AD per-EVI/ES routes for ESI-23 from PE-2 and PE-3, as well as the CE-11 MAC address from PE-3 (because MTU-1 picked up its link to PE-3 to send CE-11 frames).

  
  *A:PE-4# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01 
  ===============================================================================
   BGP Router ID:192.0.2.4        AS:64500       Local AS:64500      
  ===============================================================================
   Legend -
   Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                   l - leaked, x - stale, > - best, b - backup, p - purge
   Origin codes  : i - IGP, e - EGP, ? - incomplete
  
  ===============================================================================
  BGP EVPN Auto-Disc Routes
  ===============================================================================
  Flag  Route Dist.         ESI                           NextHop
        Tag                                               Label
  -------------------------------------------------------------------------------
  u*>i  192.0.2.2:1         01:00:00:00:00:23:00:00:00:01 192.0.2.2
        0                                                 LABEL 524281
  
  u*>i  192.0.2.2:1         01:00:00:00:00:23:00:00:00:01 192.0.2.2
        MAX-ET                                            LABEL 0
  
  u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
        0                                                 LABEL 524282
  
  u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
        MAX-ET                                            LABEL 0
  
  -------------------------------------------------------------------------------
  Routes : 4
  ===============================================================================
  

  PE-4 has learned MAC address 00:00:11:11:11:11 of CE-11 in ESI-23. The BGP EVPN MAC route has PE-3 as next hop:

  *A:PE-4# show router bgp routes evpn mac rd 192.0.2.3:1
  ===============================================================================
   BGP Router ID:192.0.2.4        AS:64500       Local AS:64500
  ===============================================================================
   Legend -
   Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                   l - leaked, x - stale, > - best, b - backup, p - purge
   Origin codes  : i - IGP, e - EGP, ? - incomplete
   
  ===============================================================================
  BGP EVPN MAC Routes
  ===============================================================================
  Flag  Route Dist.         MacAddr           ESI
        Tag                 Mac Mobility      Label1
                            Ip Address
                            NextHop
  -------------------------------------------------------------------------------
  u*>i  192.0.2.3:1         00:00:11:11:11:11 01:00:00:00:00:23:00:00:00:01
        0                   Seq:0             LABEL 524282
                            n/a
                            192.0.2.3
   
  -------------------------------------------------------------------------------
  Routes : 1
  ===============================================================================
  

• In the FDB for VPLS-1, PE-4 has learned the CE-11 MAC address associated with a newly created EVPN-MPLS ES destination:

  
  *A:PE-4# show service id 1 fdb mac 00:00:11:11:11:11
   
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:11:11:11:11 eES:                    Evpn     02/17/21 14:54:42
                               01:00:00:00:00:23:00:00:00:01
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

• Due to the aliasing function, the newly created EVPN-MPLS ES destination to ESI-23 has two next-hops (PE-2 and PE-3), to which PE-4 can load-balance the unicast traffic because ecmp 2 is configured in the VPLS-1 of PE-4.

  
  *A:PE-4# show service id 1 evpn-mpls 
   
  ===============================================================================
  BGP EVPN-MPLS Dest
  ===============================================================================
  TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                  Transport:Tnl                             Sup BCast Domain
  -------------------------------------------------------------------------------
  192.0.2.2       524280        0           bum             02/17/2021 14:17:45
                  ldp:65538                                 No
  192.0.2.3       524278        0           bum             02/17/2021 14:17:45
                  ldp:65537                                 No
  192.0.2.5       524279        0           bum             02/17/2021 14:17:45
                  ldp:65539                                 No
  -------------------------------------------------------------------------------
  Number of entries : 3
  -------------------------------------------------------------------------------
  ===============================================================================
   
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:23:00:00:00:01   1                       02/17/2021 14:54:42
  -------------------------------------------------------------------------------
  Number of entries: 1
  -------------------------------------------------------------------------------
  ===============================================================================
  ---snip---
  

  The show service id 1 evpn-mpls esi 01:00:00:00:00:23:00:00:00:01 command shows the next-hops that the EVPN-MPLS ES destination is resolved to.

  
  *A:PE-4# show service id 1 evpn-mpls esi 01:00:00:00:00:23:00:00:00:01
   
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:23:00:00:00:01   1                       02/17/2021 14:54:42
  ===============================================================================
   
  ===============================================================================
  BGP EVPN-MPLS Dest TEP Info
  ===============================================================================
  TEP Address              Egr Label                Last Change
                           Transport:Tnl-Id
  -------------------------------------------------------------------------------
  192.0.2.2                524281                   02/17/2021 14:54:42
                           ldp:65538
  192.0.2.3                524282                   02/17/2021 14:54:42
                           ldp:65537
  -------------------------------------------------------------------------------
  Number of entries : 2
  -------------------------------------------------------------------------------
  ===============================================================================
  

• PE-2 will show the CE-11 MAC address as learned locally in SAP lag-1:1 (because the data plane learning of the CE-11 MAC address happened in PE-2). For PE-3, even though it learned the MAC address from EVPN, it will install it as associated with SAP lag-1:1 because the EVPN route came with ESI-23, which is a local ESI. Because of this, whenever PE-3 receives a frame with MAC DA equal to the CE-11 MAC address, it will be able to forward the frame locally to the SAP lag-1:1. The following output shows the CE-11 MAC address as it is installed in PE-2 and PE-3:

  
  *A:PE-2# show service id 1 fdb mac 00:00:11:11:11:11
   
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:11:11:11:11 sap:lag-1:1             L/0      02/17/21 15:02:44
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

  *A:PE-3# show service id 1 fdb mac 00:00:11:11:11:11
   
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:11:11:11:11 sap:lag-1:1             Evpn     02/17/21 15:02:44
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

EVPN-MPLS single-active multi-homing: mass-withdraw, backup path illustrates two concepts in EVPN single-active multi-homing: mass-withdraw and backup path.

• With mass-withdraw, when ESI-45 goes down, PE-2 does not have to wait for all the MAC routes to be withdrawn to converge all the services. Instead, PE-4 will withdraw the AD per-ES routes (also the AD per-EVI and MAC routes) and that will be used at PE-2 as a notification to stop sending traffic to PE-4 for any MAC address associated with ESI-45.

• With backup path, when PE-2 is notified of the ESI-45 failure due to the withdrawn AD routes, it will not flush any MAC address associated with ESI-45. Instead, it will change the next-hop of the EVPN-MPLS ES destination to the remaining PE in the ESI-45. Backup path only works when there are two PEs in the same ES. If there were more than two PEs in ESI-45, PE-2 would flush all the MAC addresses upon receiving a mass-withdraw notification, because it would not know who the new active PE is.

The single-active multi-homing configuration example is based on EVPN-MPLS for VPLS services:

MTU-6 is connected to the EVPN network using single-active multi-homing. With the MTU-6 configuration, a VPLS service with active-standby spoke-sdp to PE-4 and PE-5 is configured. In PE-4 and PE-5, the SDP connected to MTU-6 is linked to ESI-45. Both will run the DF election algorithm for EVI 1, and the non-DF PE (PE-4 in this example) will bring down the spoke-SDP and notify MTU-6.

The following output shows the configuration of ESI-45 in PE-4 and PE-5, as well as the SDPs. The configuration of MTU-6 is also shown for completeness. It is important to keep the default no ignore-standby-signaling command on MTU-6 spoke-SDPs because the PW switchover in MTU-6 will be triggered based on the PW status bits sent by PE-4 and PE-5.

SDP 46 with far-end MTU-6 is configured on PE-4:

# on PE-4:
configure
    service
        sdp 46 mpls create
            far-end 192.0.2.6
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

Ethernet segment ‟ESI-45” is configured on PE-4 as follows:

# on PE-4:
configure
    service
        system
            bgp-evpn
                ethernet-segment "ESI-45" create
                    esi 01:00:00:00:00:45:00:00:00:01
                    es-activation-timer 3
                    service-carving
                        mode auto
                    exit
                    multi-homing single-active
                    sdp 46
                    no shutdown
                exit

On PE-5, SDP 56 is configured as follows:

# on PE-5:
configure
    service
        sdp 56 mpls create
            far-end 192.0.2.6
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit

Ethernet segment ‟ESI-45” is configured as follows on PE-5:

# on PE-5:
configure
    service
        system
            bgp-evpn
                ethernet-segment "ESI-45" create
                    esi 01:00:00:00:00:45:00:00:00:01
                    es-activation-timer 3
                    service-carving
                        mode auto
                    exit
                    multi-homing single-active
                    sdp 56
                    no shutdown
                exit

On MTU-6, the service configuration is as follows:

# on MTU-6:
configure
    service
        sdp 64 mpls create
            far-end 192.0.2.4
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        sdp 65 mpls create
            far-end 192.0.2.5
            ldp
            keep-alive
                shutdown
            exit
            no shutdown
        exit
        vpls 1 name "VPLS1" customer 1 create
            endpoint "CORE" create
            exit
            stp
                shutdown
            exit
            sap 1/2/1:1 create
                no shutdown
            exit
            spoke-sdp 64:1 endpoint "CORE" create
                stp
                    shutdown
                exit
                no shutdown
            exit
            spoke-sdp 65:1 endpoint "CORE" create
                stp
                    shutdown
                exit
                no shutdown
            exit
            no shutdown
        exit

For a detailed description of the base BGP-EVPN instance and ES configuration, see the All-active multi-homing configuration section. The es-activation-timer, esi, service-carving, boot-timer, and shutdown commands are used in the same way as for all-active multi-homing. Only the differences compared to all-active multi-homing are described here:

• multi-homing single-active must be configured so that the ES acts as single-active. Optionally, the no-esi-label attribute can be added to the multi-homing single-active command. This attribute controls the use of the ESI-label for single-active multi-homing. Although the ESI-label is always used in all-active multi-homing when sending BUM traffic between the PEs in the ES, it is configurable for single-active. However, Nokia recommends to use the default option (using ESI-label) to avoid potential transient issues when there is a DF switchover.

• sdp <sdp-id> is configured so that the ES can be associated with the SDP connected to MTU-6. Although all-active multi-homing only allows LAG associations to the ES, single-active allows LAG, port, and SDP. In this example, SDP is the option, because the access network is MPLS-based.

Similar to the all-active multi-homing case, when configuring the service in PE-4 and PE-5, the service objects are automatically associated with the ESI-45, because they are defined in the SDPs linked to the ESI. The configuration for VPLS 1 on PE-5 is as follows:

# on PE-5:
configure
    service
        vpls 1 name "VPLS1" customer 1 create
            bgp
            exit
            bgp-evpn
                evi 1
                mpls bgp 1
                    ingress-replication-bum-label
                    ecmp 2
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            spoke-sdp 56:1 create
                no shutdown
            exit
            no shutdown

In all-active multi-homing, the non-DF does not bring down the service SAP associated with the ES (it only removes it from the default-multicast-list). However, in single-active multi-homing, the service spoke-SDP (or SAP, if that was the object associated) is brought operationally down. The following output shows the spoke-SDP state in PE-4 (non-DF), as operationally down with the StandbyForMHProtocol flag and the Local Pw Bits that are signaled to MTU-6:

*A:PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" evi 1
 
===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
1             1             0                   no  02/17/2021 15:04:43
===============================================================================
 
===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.4                               02/17/2021 15:04:28
192.0.2.5                               02/17/2021 15:04:43
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================

Spoke-SDP 46:1 is operationally down on PE-4:

*A:PE-4# show service id 1 sdp
 
===============================================================================
Services: Service Destination Points
===============================================================================
SdpId            Type     Far End addr    Adm     Opr       I.Lbl     E.Lbl
-------------------------------------------------------------------------------
46:1             Spok     192.0.2.6       Up      Down      524281    524281
-------------------------------------------------------------------------------
Number of SDPs : 1
-------------------------------------------------------------------------------
===============================================================================

Spoke-SDP 46:1 is operationally down with the StandbyForMHProtocol flag:

*A:PE-4# show service id 1 sdp 46:1 detail | match Flag        
Flags              : StandbyForMHProtocol

The local PW bits (pwFwdingStandby) are sent to MTU-6:

*A:PE-4# show service id 1 sdp 46:1 detail | match Pw   
Local Pw Bits      : pwFwdingStandby
Peer Pw Bits       : None

The same commands used in the All-active multi-homing operation section can be used for single-active; see that section.

The show service system bgp-evpn ethernet-segment name "ESI-45" command shows an Ethernet-segment Oper Multi-homing in addition to the configured Multi-homing mode. This occurs because, in spite of configuring the ES as all-active, it may operate as single-active if there is a mismatch between the modes advertised by PE-4 and PE-5 in the AD per-ES routes (per RFC 7432). In this example, the configured and the operational value are the same:

*A:PE-4# show service system bgp-evpn ethernet-segment name "ESI-45"
 
===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-45
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:45:00:00:00:01
Multi-homing            : singleActive       Oper Multi-homing  : singleActive
ES SHG Label            : 524278
Source BMAC LSB         : <none>
Sdp Id                  : 46
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================

As soon as CE-16 sends some traffic, the DF PE (PE-5) will learn the CE-16 MAC address and will advertise it to the network. The remote PEs (PE-2 and PE-3) will create a new EVPN-MPLS ES destination to ESI-45, but this time with only one next-hop, PE-5, because this is single-active multi-homing. The following outputs show the following information:

• PE-2 has learned AD per-EVI/ES routes for ESI-45 from PE-4 and PE-5, as well as the CE-16 MAC address from an ES EVPN-MPLS destination, which is resolved to PE-5 (the DF for ESI-45).

  
  *A:PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:45:00:00:00:01 
  ===============================================================================
   BGP Router ID:192.0.2.2        AS:64500       Local AS:64500      
  ===============================================================================
   Legend -
   Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                   l - leaked, x - stale, > - best, b - backup, p - purge
   Origin codes  : i - IGP, e - EGP, ? - incomplete
  
  ===============================================================================
  BGP EVPN Auto-Disc Routes
  ===============================================================================
  Flag  Route Dist.         ESI                           NextHop
        Tag                                               Label
  -------------------------------------------------------------------------------
  u*>i  192.0.2.4:1         01:00:00:00:00:45:00:00:00:01 192.0.2.4
        0                                                 LABEL 524280
  
  u*>i  192.0.2.4:1         01:00:00:00:00:45:00:00:00:01 192.0.2.4
        MAX-ET                                            LABEL 0
  
  u*>i  192.0.2.5:1         01:00:00:00:00:45:00:00:00:01 192.0.2.5
        0                                                 LABEL 524280
  
  u*>i  192.0.2.5:1         01:00:00:00:00:45:00:00:00:01 192.0.2.5
        MAX-ET                                            LABEL 0
  
  -------------------------------------------------------------------------------
  Routes : 4
  ===============================================================================
  

  PE-2 has learned the CE-16 MAC address from an ES EVPN-MPLS destination:

  
  *A:PE-2# show service id 1 fdb mac 00:00:16:16:16:16
   
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:16:16:16:16 eES:                    Evpn     02/17/21 15:10:23
                               01:00:00:00:00:45:00:00:00:01
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

  On PE-2, the ES EVPN-MPLS destination is resolved to DF PE-5:

  *A:PE-2# show service id 1 evpn-mpls esi 01:00:00:00:00:45:00:00:00:01
   
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:45:00:00:00:01   1                       02/17/2021 15:10:23
  ===============================================================================
   
  ===============================================================================
  BGP EVPN-MPLS Dest TEP Info
  ===============================================================================
  TEP Address              Egr Label                Last Change
                           Transport:Tnl-Id
  -------------------------------------------------------------------------------
  192.0.2.5                524280                   02/17/2021 15:10:23
                           ldp:65539
  -------------------------------------------------------------------------------
  Number of entries : 1
  -------------------------------------------------------------------------------
  ===============================================================================
  

• In this case, the local PEs, PE-4 and PE-5, will learn the CE MAC address from an EVPN-MPLS destination and a local spoke-SDP, respectively.

  *A:PE-4# show service id 1 fdb mac 00:00:16:16:16:16
   
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:16:16:16:16 eES:                    Evpn     02/17/21 15:10:23
                               01:00:00:00:00:45:00:00:00:01
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

  The ES EVPN-MPLS destination is resolved to DF PE-5:

  *A:PE-4# show service id 1 evpn-mpls esi 01:00:00:00:00:45:00:00:00:01
   
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:45:00:00:00:01   1                       02/17/2021 15:10:23
  ===============================================================================
   
  ===============================================================================
  BGP EVPN-MPLS Dest TEP Info
  ===============================================================================
  TEP Address              Egr Label                Last Change
                           Transport:Tnl-Id
  -------------------------------------------------------------------------------
  192.0.2.5                524280                   02/17/2021 15:10:23
                           ldp:65539
  -------------------------------------------------------------------------------
  Number of entries : 1
  -------------------------------------------------------------------------------
  ===============================================================================
  

  DF PE-5 learns the CE-16 MAC address from a local spoke SDP:

  *A:PE-5# show service id 1 fdb mac 00:00:16:16:16:16
   
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:16:16:16:16 sdp:56:1                L/210    02/17/21 15:10:23
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

If either ES fails, a DF re-election will happen and the corresponding AD per-ES/EVI routes will be withdrawn, causing the remote PEs to modify the list of next-hops for the EVPN-MPLS ES destination. The following example illustrates a failure on the SDP between MTU-6 and PE-5 (the DF).

1. A failure occurs in the LSP between MTU-6 and PE-5. This can be any event that brings the SDP down.

   # log 99 on PE-5:
   85 2021/02/17 15:14:45.918 UTC MINOR: SVCMGR #2303 Base
   "Status of SDP 56 changed to admin=up oper=down"
   

2. Immediately, PE-5 gives up the DF role and withdraws the ES route, as well as the AD routes and MAC routes. As soon as PE-4 receives any ES or AD withdraw, it will re-run the DF algorithm and, when the es-activation-timer expires, it will become the DF and activate its spoke-SDP.

   # log 99 on PE-5: 
   87 2021/02/17 15:14:45.920 UTC MINOR: SVCMGR #2094 Base 
   "Ethernet Segment:ESI-45, EVI:1, Designated Forwarding state changed to:false"
   

   The ES in PE-5 is operational down:

   *A:PE-5# show service system bgp-evpn ethernet-segment name "ESI-45"
    
   ===============================================================================
   Service Ethernet Segment
   ===============================================================================
   Name                    : ESI-45
   Eth Seg Type            : None
   Admin State             : Enabled            Oper State         : Down
   ESI                     : 01:00:00:00:00:45:00:00:00:01
   Multi-homing            : singleActive       Oper Multi-homing  : singleActive
   ES SHG Label            : 524282
   Source BMAC LSB         : <none>
   Sdp Id                  : 56
   ES Activation Timer     : 3 secs
   Oper Group              : (Not Specified)
   Svc Carving             : auto               Oper Svc Carving   : auto
   Cfg Range Type          : primary
   ===============================================================================
   

   PE-5 is no longer the DF and the only DF candidate is PE-4:

   *A:PE-5# show service system bgp-evpn ethernet-segment name "ESI-45" evi 1
    
   ===============================================================================
   EVI DF and Candidate List
   ===============================================================================
   EVI           SvcId         Actv Timer Rem      DF  DF Last Change
   -------------------------------------------------------------------------------
   1             1             0                   no  02/17/2021 15:14:46
   ===============================================================================
    
   ===============================================================================
   DF Candidates                           Time Added
   -------------------------------------------------------------------------------
   192.0.2.4                               02/17/2021 15:04:45
   -------------------------------------------------------------------------------
   Number of entries: 1
   ===============================================================================
   

   PE-4 becomes the DF and the spoke-SDP 46:1 is brought up.

   # log 99 on PE-4: 
   88 2021/02/17 15:14:48.951 UTC MINOR: SVCMGR #2094 Base
   "Ethernet Segment:ESI-45, EVI:1, Designated Forwarding state changed to:true"
   
   89 2021/02/17 15:14:48.951 UTC MINOR: SVCMGR #2326 Base
   "Status of SDP Bind 46:1 in service 1 (customer 1) local PW status bits changed to none"
   
   90 2021/02/17 15:14:48.951 UTC MINOR: SVCMGR #2306 Base
   "Status of SDP Bind 46:1 in service 1 (customer 1) changed to admin=up oper=up flags="
   

   The ES is up in PE-4:

   *A:PE-4# show service system bgp-evpn ethernet-segment name "ESI-45"
    
   ===============================================================================
   Service Ethernet Segment
   ===============================================================================
   Name                    : ESI-45
   Eth Seg Type            : None
   Admin State             : Enabled            Oper State         : Up
   ESI                     : 01:00:00:00:00:45:00:00:00:01
   Multi-homing            : singleActive       Oper Multi-homing  : singleActive
   ES SHG Label            : 524278
   Source BMAC LSB         : <none>
   Sdp Id                  : 46
   ES Activation Timer     : 3 secs
   Oper Group              : (Not Specified)
   Svc Carving             : auto               Oper Svc Carving   : auto
   Cfg Range Type          : primary
   ===============================================================================
   

   PE-4 is the DF and there are no other DF candidates:

   *A:PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" evi 1
    
   ===============================================================================
   EVI DF and Candidate List
   ===============================================================================
   EVI           SvcId         Actv Timer Rem      DF  DF Last Change
   -------------------------------------------------------------------------------
   1             1             0                   yes 02/17/2021 15:14:49
   ===============================================================================
    
   ===============================================================================
   DF Candidates                           Time Added
   -------------------------------------------------------------------------------
   192.0.2.4                               02/17/2021 15:04:28
   -------------------------------------------------------------------------------
   Number of entries: 1
   ===============================================================================
   

3. The remote PEs, PE-2 and PE-3, receive the BGP-EVPN routes withdrawal and modify the next-hop for the EVPN-MPLS ES destination.

   241 2021/02/17 15:14:45.921 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
   "Peer 1: 192.0.2.5: UPDATE
   Peer 1: 192.0.2.5 - Received BGP UPDATE:
       Withdrawn Length = 0
       Total Path Attr Length = 86
       Flag: 0x90 Type: 15 Len: 82 Multiprotocol Unreachable NLRI:
           Address Family EVPN
           Type: EVPN-AD Len: 25 RD: 192.0.2.5:1 ESI: 01:00:00:00:00:45:00:00:00:01, 
                         tag: MAX-ET Label: 0
           Type: EVPN-AD Len: 25 RD: 192.0.2.5:1 ESI: 01:00:00:00:00:45:00:00:00:01, 
                         tag: 0 Label: 0
           Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.5:0 ESI: 01:00:00:00:00:45:00:00:00:01,
                         IP-Len: 4 Orig-IP-Addr: 192.0.2.5
   "
   

   The ES EVPN-MPLS destination is resolved to the DF PE-4:

   *A:PE-2# show service id 1 evpn-mpls esi 01:00:00:00:00:45:00:00:00:01
    
   ===============================================================================
   BGP EVPN-MPLS Ethernet Segment Dest
   ===============================================================================
   Eth SegId                       Num. Macs               Last Change
   -------------------------------------------------------------------------------
   01:00:00:00:00:45:00:00:00:01   1                       02/17/2021 15:15:06
   ===============================================================================
    
   ===============================================================================
   BGP EVPN-MPLS Dest TEP Info
   ===============================================================================
   TEP Address              Egr Label                Last Change
                            Transport:Tnl-Id
   -------------------------------------------------------------------------------
   192.0.2.4                524280                   02/17/2021 15:15:06
                            ldp:65538
   -------------------------------------------------------------------------------
   Number of entries : 1
   -------------------------------------------------------------------------------
   ===============================================================================
   

The following must be considered:

• The DF election procedure is revertive, that is, when the failed SDP comes back up, PE-5 will take over again as DF and the network will re-converge.

• The DF election is triggered by the following events:

  • configure service system bgp-evpn ethernet-segment ESI-45 no shutdown triggers the DF election for all the services in the ES.

  • A new update/withdrawal of an ES route (containing an ESI configured locally) triggers the DF election for all the services in the ESI.

  • A new update/withdrawal of an AD per-ES route (containing an ESI configured locally) triggers the DF election for all the services associated with the list of RTs received along with the route.

  • A new update of an AD per-ES route with a change in the ESI-label extended community (single-active bit or MPLS label) triggers the DF election for all the services associated with the list of RTs received along with the route.

  • A new update/withdrawal of an AD route per-EVI (containing an ESI configured locally) triggers the DF election for that service.

The selection of the best route for a MAC address is as follows:

• If a PE receives more than one route for the same MAC address, the best MAC route is chosen:

  • If the route key is equal in two or more routes (that is, the mac, mac-length, ip, ip-length, RD, eth-tag), then regular BGP selection applies:

    • If local-pref, AS-path, origin, and MED are equal, the lowest IGP distance to the BGP next-hop is chosen (unless ignore-nh-metric is configured). If the BGP next-hop is resolved by an LSP, the cost from the tunnel-table is used.

    • As a last resort tie-breaker, the route with the lowest originator ID, or received from the peer with the lowest BGP Identifier, is chosen (unless ignore-router-id is configured and the routes being compared are EBGP routes).

  • If the mac-length, mac, ip-length, ip, eth-tag are equal, and the RD is different, the EVPN selection process is applied in the following order:

    • Conditional static MAC addresses (local protected MAC addresses)

    • EVPN static MAC addresses (remote protected MAC addresses)

    • Data plane learned MAC addresses (regular learning on SAPs/SDP-bindings)

    • EVPN MAC addresses with higher sequence number

    • Lowest IP address (next-hop IP of the EVPN NLRI)

    • Lowest Ethernet tag (will be normally zero)

    • Lowest RD

• After a MAC route is selected, the system checks for an associated ES.

  • If it has an ES, the system uses the MAC address as the EVPN-MPLS ES destination. The ES destination is constructed based on the AD per-EVI routes received for that ES (regardless of MAC address priorities with the ES).

  • The system selects the first ECMP number of AD per-EVI routes arranged by the IP address of PEs (lower IPs are selected first).

  • If the same PE has advertised multiple RDs, the system selects the route with the lowest RD for that PE.

In the example of EVPN-MPLS for VPLS services, PE-4 resolves the next-hops for ESI-23 as described in the second choice above, that is, because ECMP=2, the two available next-hops are chosen. If ECMP is changed to 1, PE-4 will pick up the lower IP (in the BGP next-hop). This is illustrated in the following output:

*A:PE-4# show service id 1 evpn-mpls esi 01:00:00:00:00:23:00:00:00:01
 
===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                       Num. Macs               Last Change
-------------------------------------------------------------------------------
01:00:00:00:00:23:00:00:00:01   1                       02/17/2021 15:10:16
===============================================================================
 
===============================================================================
BGP EVPN-MPLS Dest TEP Info
===============================================================================
TEP Address              Egr Label                Last Change
                         Transport:Tnl-Id
-------------------------------------------------------------------------------
192.0.2.2                524281                   02/17/2021 15:10:16
                         ldp:65538
192.0.2.3                524282                   02/17/2021 15:10:16
                         ldp:65537
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================

When ECMP equals 1, only the BGP next hop with the lower IP is chosen:

# on PE-4:
configure 
    service 
        vpls "VPLS1"
            bgp-evpn 
                mpls 
                    ecmp 1
                exit

*A:PE-4# show service id 1 evpn-mpls esi 01:00:00:00:00:23:00:00:00:01
 
===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                       Num. Macs               Last Change
-------------------------------------------------------------------------------
01:00:00:00:00:23:00:00:00:01   1                       02/17/2021 15:22:05
===============================================================================
 
===============================================================================
BGP EVPN-MPLS Dest TEP Info
===============================================================================
TEP Address              Egr Label                Last Change
                         Transport:Tnl-Id
-------------------------------------------------------------------------------
192.0.2.2                524281                   02/17/2021 15:10:16
                         ldp:65538
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
===============================================================================

EVPN-MPLS services support EVPN-MH (EVPN multi-homing) and also BGP-MH as in chapter BGP Multi-Homing for VPLS Networks. While EVPN-MH is the standard way of providing access resiliency in RFC 7432, BGP-MH is also a standard mechanism supported in VPLS or EVPN networks. The following table provides some comparison between both technologies.

In addition to the preceding comparison, the following configuration excerpt compares EVPN-MH with BGP-MH on a bgp-evpn VPLS service and shows that, while EVPN-MH does not have any configuration at service level, BGP-MH is configured within the VPLS context, which gives a more granular control over the redundancy provided. See the BGP Multi-Homing for VPLS Networks chapter for more information about BGP-MH.

*A:PE-4>config>service>system>bgp-evpn# info
----------------------------------------------
                ethernet-segment "ESI-45"  create
                    esi 01:00:00:00:00:45:00:00:00:01
                    es-activation-timer 3
                    service-carving
                        mode auto
                    exit
                    multi-homing single-active
                    sdp 46
                    no shutdown
                exit
----------------------------------------------

*A:PE-4>config>service>vpls# info
----------------------------------------------
            bgp
            exit
            bgp-evpn
                evi 1
                mpls bgp 1
                    ingress-replication-bum-label
                    ecmp 2
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            spoke-sdp 46:1 create
                no shutdown
            exit
            no shutdown
----------------------------------------------

For BGP multi-homing, site ‟site-1” is configured, as follows. The RD needs to be configured in the bgp context.

*A:PE-4>config>service>vpls# info
----------------------------------------------
            bgp
                route-distinguisher 192.0.2.4:1
            exit
            bgp-evpn
                evi 1
                mpls bgp 1
                    ingress-replication-bum-label
                    ecmp 2
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            site "site-1" create
                site-id 1
                spoke-sdp 46:1
                site-activation-timer 3
                no shutdown
            exit
            spoke-sdp 46:1 create
                no shutdown
            exit
            no shutdown
----------------------------------------------

Although not strictly a BGP-EVPN configuration, vpls>proxy-arp and vpls>proxy-nd functions are typically enabled along with EVPN-MPLS in order to reduce the amount of flooding in the network. The proxy-ARP/ND agent in the VPLS service will snoop ARP-requests and/or Neighbor Solicitation messages and will reply to those messages locally (if the information is known) without having to flood the requests to the network.

The configuration options for proxy-ARP are the following:

*A:PE-2>config>service>vpls# proxy-arp ?
  - no proxy-arp
  - proxy-arp
 
 [no] age-time        - Configure aging timer for proxy ARP entries
      dup-detect      - Configure anti-spoofing MAC address information
 [no] dynamic         + Configure dynamic entry information
 [no] dynamic-arp-po* - Configure population of dynamic proxy ARP entries
 [no] evpn-route-tag  - Configure EVPN Route Tag information
 [no] garp-flood-evpn - Configure to flood GARP request/replys into EVPN
 [no] send-refresh    - Configure send refresh time
 [no] shutdown        - Administratively enable/disable proxy ARP configuration
 [no] static          - Configure static IP address to MAC address associations
      table-size      - Configure the maximum number of entries in the proxy ARP table
 [no] unknown-arp-re* - Configure to flood unknown ARP request

The configuration options for proxy-ND are the following:

*A:PE-2>config>service>vpls# proxy-nd ?
  - no proxy-nd
  - proxy-nd
 
 [no] age-time        - Configure aging timer for proxy ND entries
      dup-detect      - Configure anti-spoofing MAC address information
 [no] dynamic         + Configure dynamic entry information
 [no] dynamic-nd-pop* - Configure population of dynamic proxy ND entries
      evpn-nd-advert* - Configure EVPN Neighbor Discovery advertisements
 [no] evpn-route-tag  - Configure EVPN Route Tag information
 [no] host-unsolicit* - Configure whether to flood evpn with host  neighbor
                        advertisement
 [no] router-unsolic* - Configure whether to flood evpn with router neighbor
                        advertisement
 [no] send-refresh    - Configure send refresh time
 [no] shutdown        - Administratively enable/disable proxy ND configuration
 [no] static          - Configure static IP address to MAC address associations
      table-size      - Configure the maximum number of entries in the proxy ND table
 [no] unknown-ns-flo* - Configure to flood unknown ND solicitation

When proxy-ARP/ND is enabled, the following configuration guidelines must be followed:

• dynamic-arp-populate or dynamic-nd-populate should be used only in networks with a consistent configuration of this command in all PEs.

• When using dynamic-arp-populate/dynamic-nd-populate, the age-time value should be configured to a value equal to three times the send-refresh value. This will help reduce the EVPN withdrawals and re-advertisements in the network.

• With large age-time values, it would be sufficient to configure the send-refresh value to half of the proxy-ARP/ND age-time or FDB age-time.

• In scaled environments (with thousands of services), it is not recommended to set the send-refresh value to less than 300 s. In such scenarios, Nokia recommends using a minimum proxy-ARP/ND age-time and FDB age-time of 900 s.

• The use of the following commands reduces or suppresses the ARP/ND flooding in an EVPN network, because EVPN MAC routes replace the function of the regular data plane ARP/ND messages:

  • no garp-flood-evpn

  • no unknown-arp-request-flood-evpn

  • no unknown-ns-flood-evpn

  • no host-unsolicited-na-flood-evpn

  • no router-unsolicited-na-flood-evpn

• Nokia recommends using the preceding commands only in EVPN networks where the CEs are routers directly connected to an SR OS node acting as the PE. Networks using aggregation switches between the host/routers and the PEs should flood GARP/ND messages in EVPN to make sure the remote caches are updated and BGP does not miss the advertisement of these entries.

• When the anti-spoof-mac is used with proxy-ARP/ND, ingress filters (in the access SAPs/SDP-bindings) should be configured to drop all traffic with destination anti-spoof-mac. The same MAC address should be configured in all PEs where dup-detect is active.

• When proxy-ND is used, the configuration of the following commands should be consistent in all the PEs in the network:

  • router-unsolicited-na-flood-evpn

  • host-unsolicited-na-flood-evpn

  • evpn-nd-advertise

• Because EVPN does not propagate the router flag in IPv6--> MAC address advertisements, in a mixed network with hosts and routers where evpn-nd-advertise router is configured, unsolicited host NA messages should be flooded so that the entire network gets to learn all of the host and router ND entries. In the same way, evpn-nd-advertise host should be configured so that unsolicited router NA messages are flooded.

Finally, along with proxy-ARP/ND, vpls>discard-unknown may be used in some EVPN-MPLS deployments where all the CEs are routers and they announce themselves to the network by sending GARPs or NAs (Neighbor Solicitation messages). According to RFC 7432, whether or not to flood packets to unknown destination MAC addresses should be an administrative choice, depending on how learning happens between CEs and PEs. Discard-unknown provides that administrative choice in case all the MAC addresses in an EVI can be learned even before any traffic is exchanged.

Proxy-ARP/ND along with discard-unknown helps reduce the BUM traffic in an EVPN network significantly; however, their use must be analyzed and considered, depending on the type of CEs in the EVI.

An example of proxy-ARP configuration is as follows. This configuration should be added to all PEs. When a new ARP message is received on any of the PEs, they will learn the IP-MAC address pair and will advertise it to the network.

# on PE-2, PE-3, PE-4, PE-5:
configure
    service
        vpls "VPLS1"
            proxy-arp
                age-time 900
                send-refresh 300
                dynamic-arp-populate
                no shutdown
            exit

Enabling proxy-ARP increases the number of MAC/IP routes being sent by the PEs. This is due to the following reasons:

• An additional MAC/IP route will be advertised per new learned IP-MAC address pair, regardless of having advertised the same MAC address already.

• A MAC per VPLS service will be advertised with a system MAC address. That MAC address will be used as MAC SA for proxy-ARP confirm messages when an IP moves to a different PE.

The following output shows the MAC/IP routes on PE-2 when proxy-ARP is enabled in the network.

*A:PE-2# show router bgp routes evpn mac
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete
 
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         02:17:ff:00:03:3a ESI-0
      0                   Static            LABEL 524282
                          n/a
                          192.0.2.3
 
u*>i  192.0.2.4:1         02:1b:ff:00:03:3a ESI-0
      0                   Static            LABEL 524280
                          n/a
                          192.0.2.4
 
u*>i  192.0.2.5:1         00:00:16:16:16:16 01:00:00:00:00:45:00:00:00:01
      0                   Seq:0             LABEL 524280
                          n/a
                          192.0.2.5
 
u*>i  192.0.2.5:1         02:1f:ff:00:03:3a ESI-0
      0                   Static            LABEL 524280
                          n/a
                          192.0.2.5
 
-------------------------------------------------------------------------------
Routes : 4
===============================================================================