EVPN for MPLS Tunnels

EVPN all-active multi-homing is built around three concepts: DF election, split-horizon (with an ESI-label), and aliasing, as shown in EVPN-MPLS all-active multi-homing concepts, from left to right.

• With DF election, when PE-4 sends BUM traffic to the remote ES (CE-2), only one PE segment sends the BUM packets to the ES (PE-3 is the DF in the preceding example, and is elected to send BUM packets to CE-2). The non-DF, PE-2, removes the LAG SAP from the default multicast list (PE-2 does not bring CE-2 down, because it still needs to send upstream/downstream unicast traffic). PE-2 and PE-3 elect a DF for each service, based on the ES routes and the service-carving algorithm.

• With split-horizon, the PE part of the ES (PE-3 in the preceding example) identifies the BUM packets coming from the PE for the remote (PE-2), but within the same ES (ESI-2), and filters the packets so that they are not sent back to the ES, creating duplication. When PE-2 (non-DF) sends BUM traffic to PE-3 (DF), it uses a special MPLS label in the data path that PE-3 previously advertised for ESI-2 in an AD per-ES route. When PE-3 does an ingress lookup, it recognizes the ESI-label and filters the traffic (PE-3 still sends the BUM traffic to other SAPs/SDP-bindings).

• With aliasing, remote PEs that are not part of the ES can load-balance unicast traffic to all the PEs that are part of the ES, irrespective of from which PE a destination MAC address was learned. PE-4 will create an EVPN destination to ESI-2 that will be resolved to the two next-hops: PE-2 and PE-3. Unicast load-balancing will happen as long as ECMP > 1 is enabled in PE-4.

Nokia recommends the use of ingress-replication-bum-label on the PEs that are part of an all-active ES. In an all-active multi-homing scenario, if a specified MAC address (for example, the CE-2 MAC address in the left-hand-side diagram), is not learned yet in a remote PE (for example, PE-4), but is known in the two PEs of the ES (for example, PE-2 and PE-3), the latter PEs might send duplicated packets to the CE.

This issue is solved by the use of ingress-replication-bum-label in PE-2 and PE-3. If configured, PE-2/PE-3 will know that the received packet is an unknown unicast packet; therefore, the Non-DF (PE-2) will not send the packet to CE-2 and there will not be duplication.

The all-active multi-homing configuration example is based on EVPN-MPLS for VPLS services.

MTU-1 is connected to the EVPN network using all-active multi-homing. According to RFC 7432, MTU-1 will be able to send traffic to both PEs for VPLS-1. Regular LAG load-balancing is used in MTU-1. Remote PEs such as PE-4 or PE-5 will be able to load-balance the unicast traffic to PE-2 and PE-3. PE-2 and PE-3 will discover that both are part of ESI-23 (due to the exchange of ES routes) and will elect a DF for VPLS-1. The non-DF for VPLS-1, in this case PE-2, will remove lag-1:1 from the VPLS-1 default multicast list. Also, when PE-2 and PE-3 send BUM traffic to each other, they will insert an ESI-label so that they can identify that the source of the BUM packet is ESI-23.

The following output shows the configuration of ESI-23 in PE-2 and PE-3, as well as the LAG interfaces for all-active multi-homing (see EVPN-MPLS for VPLS services). The configuration of LAG-1 in MTU-1 is also shown. Per RFC 7432, only a CE/MTU with a LAG can be connected to an all-active multi-homing ES. No other configuration is permitted on the CE for all-active multi-homing.

LAG 1 is configured on MTU-1, PE-2, and PE-3, as follows:

# on MTU-1:
configure {
    lag "lag-1" {
        admin-state enable
        encap-type dot1q
        mode access
        max-ports 64
        lacp {
            mode active
            administrative-key 32768
        }
        port 1/1/1 {
        }
        port 1/1/2 {
        }
    }

# on PE-2:
configure {
    lag "lag-1" {
        admin-state enable
        encap-type dot1q
        mode access
        max-ports 64
        lacp {
            mode active
            system-id 00:00:00:00:02:03
            administrative-key 1
        }
        port 1/1/2 {
        }
    }

# on PE-3:
configure {
    lag "lag-1" {
        admin-state enable
        encap-type dot1q
        mode access
        max-ports 64
        lacp {
            mode active
            system-id 00:00:00:00:02:03
            administrative-key 1
        }
        port 1/1/1 {
        }
    }

Ethernet segment ‟ESI-23” is configured in the service system bgp-evpn context on PE-2 and PE-3, as follows:

# on PE-2, PE-3:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-23" {
                        admin-state enable
                        esi 01:00:00:00:00:23:00:00:00:01
                        multi-homing-mode all-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            lag "lag-1" {
                            }
                        }
                    }
                }
            }
        }

When configuring an ES, the following must be considered:

• Any EVPN parameter that is not specific to any particular VPLS service, and is common to all the EVIs, is configured in a base BGP-EVPN instance located at config>service>system>bgp-evpn. In this base instance, the following attributes may be configured:

  • ethernet-segments

  • the base BGP-EVPN instance route-distinguisher that will be used for the ES routes. If this route-distinguisher is not configured, by default a type-1 RD will be derived as system-ip:0. The ES route distinguisher can be configured using the following command:

  [ex:configure service system bgp evpn]
  A:admin@PE-2# route-distinguisher ?
  
   route-distinguisher <string>
   <string> - <0..255 characters>
  
      Route distinguisher for ES routes
  

• The ES must be configured with a name and can contain the following parameters when configured for all-active multi-homing:

  • esi — 10-byte identifier that represents the ES in the BGP control plane. The same ESI must be configured in all the PEs connected to the same CE/MTU (using a unique value that cannot be associated with any other CE/MTU/access network). RFC 7432 defines five different types of ESI. In SR OS, the type byte, as well as the other 9 bytes can be arbitrarily configured.

  • multi-homing-mode all-active — This command indicates that the ES is in all-active mode.

  • association>lag <lag-id> — The LAG connected to the CE/MTU must be added to the ES. In this example, LAG "lag-1" is added to ESI-23, on both PE-2 and PE-3. Although a different LAG-id may have been assigned to the same ES on PE-2 and PE-3, PE-2 and PE-3 must have the same configuration on the ES LAG; that is, encap-type. Also, if LACP is added (it is not mandatory), both PEs must have the same admin-key, system-id, and system-priority. MTU-1 will see PE-2 and PE-3 as a single LAG peer. For all-active multi-homing, only the lag option is accepted by the system; port or sdp are not accepted.

  • admin-state — This command controls the administrative state of the ES.

• The preceding parameters are the minimum necessary so that the ES can be activated. In addition to those parameters, there are a few more that the user can configure if requiring values different from the default ones:

  • activation-timer [0..100] can be configured at redundancy>bgp-evpn-ethernet-segment>activation-timer or at service>system>bgp>evpn>ethernet-segment>df-election>es-activation-timer level (the most specific value is used).

    The ES activation timer operation is as follows:

    • Upon reception of an ES, AD per-ES/EVI route update/withdrawal for a local ESI, the DF-candidate list of IP addresses is updated and the DF election algorithm is run without waiting for any timer.

    • If the result of the DF election requires the PE to be promoted from non-DF to DF, the ES activation timer will start, and only after its expiration will the PE add the SAP to the default multicast list. Transitions from non-DF to non-DF, or from DF to non-DF, are immediate and do not wait for any timer.

    • This use of an ES activation timer value minimizes the risks of loops and packet duplication due to transient multiple DFs.

    • The same ES activation timer must be configured in all the PEs that are part of the same ESI. The user must configure either a long timer to minimize the risks of loops/duplication, or (es-)activation-timer=0 to speed up the convergence for NDF to DF transitions. The default value is 3 seconds.

  • service-carving-mode — As defined in RFC 7432, service carving controls the distribution of DF/non-DF roles across the different services defined in an ES.

    [ex:configure service system bgp evpn ethernet-segment "ESI-23" df-election]
    A:admin@PE-2# service-carving-mode ?
    
     service-carving-mode <keyword>
     <keyword> - (auto|manual|off)
     Default   - auto
    
        Mode of service carving enabled per EVPN associated with this Ethernet segment
        entry
    

    [ex:configure service system bgp evpn ethernet-segment "ESI-23" df-election]
    A:admin@PE-2# manual ?
    
     manual
    
     evi                  + Enter the evi context
     isid                 + Enter the isid context
     preference           + Enter the preference context
    

    As shown above, service-carving has three different modes:

  • service-carving-mode auto (default) — The DF election algorithm will run the function [V(evi) mod N(peers) = i(ordinal)] to know who the DF for a specified service and ESI is. In this example, ESI-23 is configured with mode auto; therefore, for VPLS-1 (with EVI-1), PE-3 will be elected as DF because evi(1) mod (2)peers = 1, and the ordinal 1 corresponds to the second lowest IP, PE-3. The algorithm takes the configured evi in the service; therefore, the evi is mandatory, and for the same service must match in all the PEs that are part of the ES. This guarantees that the election algorithm is consistent across all the PEs of the ESI.

  • service-carving-mode manual — The user can manually decide for which evi identifiers the PE is DF: service-carving-mode manual / manual evi <start> end <to>. The PE will be non-DF for the non-specified EVIs. If service-carving-mode manual is configured, but no range is defined, all the services are considered to be non-DF. If a range is configured, but the service-carving-mode is not manual, the range has no effect. Only two PEs are supported when service-carving-mode manual is configured.

  • service-carving-mode off — The lowest originator IP will win the election for a specified service and ES.

  • Because the evi is used for the service carving algorithm, it must always be configured in a service with SAPs/SDP bindings created in an ES, regardless of the service-carving mode (service-carving off, auto, or manual).

Although not configured as part of the ES, the config>redundancy>bgp-evpn-ethernet-segment>boot-timer allows the necessary time for the control plane protocols to come up after the PE has rebooted, and before bringing up the ESs and running the DF algorithm. Some considerations about the boot timer:

• The boot timer should use a value long enough to allow the IOMs and BGP sessions to come up before exchanging ES routes and run the DF election for each EVI (it is 10 s, by default).

• The boot timer runs per EVI on the ESs in the system. While system-up-time < boot-timer, the system will not run the DF election for any EVI. When the boot timer expires, the DF election for the EVI is run and, if the system is elected DF for the EVI, the ES activation timer will start.

• The system will not advertise ES routes until the boot timer expires. This guarantees that the peer ES PEs do not run the DF election either, until the PE is ready to become the DF, if needed.

• The following show command displays the configured boot timer, as well as the remaining timer if the system is still in boot stage.

  
  [/]
  A:admin@PE-2# show redundancy bgp-evpn-multi-homing 
  
  ===============================================================================
  Redundancy BGP EVPN Multi-homing Information
  ===============================================================================
  Boot-Timer              : 10 secs                 
  Boot-Timer Remaining    : 0 secs                  
  ES Activation Timer     : 3 secs                  
  ===============================================================================
  

After ESI-23 is configured in PE-2 and PE-3, the lag-1 SAPs in both PEs can be added to the VPLS-1 service. Until the ESI-23 is successfully enabled, the LAG SAPs will be kept down with a StandByForMHProtocol flag. This is illustrated in the following example for PE-2 where the LAG SAP is added and ESI-23 is disabled:

# on PE-2:
configure exclusive
    service {
        vpls "VPLS1" {
            sap lag-1:1 {
            }
        }
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-23" {
                        admin-state disable
                    }
                    commit

[/]
A:admin@PE-2# show service id 1 sap lag-1:1 detail | match "  Oper State"
Admin State        : Up                       Oper State        : Down

[/]
A:admin@PE-2# show service id 1 sap lag-1:1 detail | match Flag 
Flags              : StandByForMHProtocol

ESI-23 is enabled and SAP lag-1:1 is operationally up, as follows:

# on PE-2:
configure exclusive
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-23" {
                        admin-state enable
                    }
                    commit

[/]
A:admin@PE-2# show log log-id "99" 
---snip---

107 2021/02/25 17:06:03.759 CET MINOR: SVCMGR #2203 Base
"Status of SAP lag-1:1 in service 1 (customer 1) changed to admin=up oper=up flags="

To confirm that all-active multi-homing is working correctly for ESI-23, the user can use the following commands:

• show service system bgp-evpn — Shows the RD is used for the ES route.

• show service system bgp-evpn ethernet-segment — Shows all the ESs configured in the PE and their admin/operational status.

• show service system bgp-evpn ethernet-segment name ESI-23 evi evi-1 1 — Shows the DF candidate PEs for EVI 1 and whether the system is DF for EVI.

• show service system bgp-evpn ethernet-segment name ESI-23 all — Shows all the information related to a specific ESI.

The base BGP-EVPN information includes the RD:

[/]
A:admin@PE-2# show service system bgp-evpn

===============================================================================
System BGP EVPN Information
===============================================================================
Eth Seg Route Dist.               : <none>
Eth Seg Oper Route Dist.          : 192.0.2.2:0
Eth Seg Oper Route Dist Type      : default
Ad Per ES Route Target            : evi-rt
Leaf Label                        : 0
Mcast Leave Sync Prop             : 5
Attribute Uniform Prop            : Disabled
BGP Path Selection                : Disabled
===============================================================================

The following command shows the configured ESs in the PE and their status:

[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment 

===============================================================================
Service Ethernet Segment
===============================================================================
Name                             ESI                           Admin     Oper
-------------------------------------------------------------------------------
ESI-23                           01:00:00:00:00:23:00:00:00:01 Enabled   Up
-------------------------------------------------------------------------------
Entries found: 1
===============================================================================

The following command shows that PE-2 is not the DF and the DF candidate PEs for EVI 1 are PE-2 and PE-3:

[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" evi evi-1 1

===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
1             1             0                   no  02/25/2021 17:06:04
===============================================================================

===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.2                               02/25/2021 17:06:04
192.0.2.3                               02/25/2021 17:06:06
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================

The following command shows all information related to ESI-23 on PE-2:

[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" all

===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-23
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:23:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
ES SHG Label            : 524279
Source BMAC LSB         : <none>
Lag                     : lag-1
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================

===============================================================================
EVI Information
===============================================================================
EVI                 SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
1                   1                   0                   no
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================

-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI                                     DF Address
-------------------------------------------------------------------------------
1                                       192.0.2.2
1                                       192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---

The following command shows all information related to ESI-23 on PE-3:

[/]
A:admin@PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" all

===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-23
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:23:00:00:00:01
Multi-homing            : allActive          Oper Multi-homing  : allActive
ES SHG Label            : 524279
Source BMAC LSB         : <none>
Lag                     : lag-1
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================

===============================================================================
EVI Information
===============================================================================
EVI                 SvcId               Actv Timer Rem      DF
-------------------------------------------------------------------------------
1                   1                   0                   yes
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================

-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
EVI                                     DF Address
-------------------------------------------------------------------------------
1                                       192.0.2.2
1                                       192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---

The preceding commands show the ESI-23 configuration on both PEs and the result of the DF election for EVI 1.

The following output shows the ES route received on PE-2:

# on PE-2:
63 2021/02/25 17:04:23.069 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 71
    Flag: 0x90 Type: 14 Len: 34 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.3:0
              ESI: 01:00:00:00:00:23:00:00:00:01, IP-Len: 4 Orig-IP-Addr: 192.0.2.3
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        df-election::DF-Type:Auto/DP:0/DF-Preference:0/AC:1
        target:00:00:00:00:23:00
"

The ES RT as shown as target:00:00:00:00:23:00 in the extended community is auto-derived from the ESI bytes 2 to 7 (with the type byte being byte 1). Only PE-2 and PE-3 generate this RT and therefore import each other’s ES route.

The following message in log 99 on PE-3 shows the result of the DF election for EVI 1.

# log "99" on PE-3:
99 2021/02/25 17:04:27.467 CET MINOR: SVCMGR #2094 Base
"Ethernet Segment:ESI-23, EVI:1, Designated Forwarding state changed to:true"

The show service system bgp-evpn ethernet-segment name ESI-23 all command shows the ESI-label allocated to the PE: ES SHG Label 524282 in the CLI output for PE-3. In this example, this label is allocated by PE-3 for ESI-23 (a different one is allocated per ESI) and advertised in the AD per-ES route for ESI-23. The following output shows the AD per-ES and AD per-EVI (for evi 1) routes sent by PE-3 and received by PE-2.

• The AD per-ES route can be identified by the MAX-ET in the ethernet-tag field (as per RFC 7432) and carries the ESI-label as well as the multi-homing mode (all-active in this case) in the ESI-label extended community (see EVPN route types and NLRIs).

  The user can enable the aggregation of AD per-ES routes by using the following command: configure service system bgp evpn ad-per-es-route route-target-type evi-route-target-set route-distinguisher-ip-address ip-address. If enabled, a single AD per-ES route with the associated RD and a set of EVI route-targets will be advertised (to a maximum of 128). When there are more than 128 EVIs defined in the ES, more than one route will be sent by the system.

  [ex:/configure service system bgp evpn ad-per-es-route]
  A:admin@PE-2# route-distinguisher-ip-address ?
  
   route-distinguisher-ip-address <ipv4-address>
   <ipv4-address>  - <d.d.d.d>
  
      IP address for route distinguisher for EVPN AD-ES routes
  

  The following AD per-ES route is received on PE-2:

  # AD per-ES route received on PE-2: 
  101 2021/02/25 17:06:06.205 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
  "Peer 1: 192.0.2.3: UPDATE
  Peer 1: 192.0.2.3 - Received BGP UPDATE:
      Withdrawn Length = 0
      Total Path Attr Length = 73
      Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
          Address Family EVPN
          NextHop len 4 NextHop 192.0.2.3
          Type: EVPN-AD Len: 25 RD: 192.0.2.3:1 ESI: 01:00:00:00:00:23:00:00:00:01,
                        tag: MAX-ET Label: 0
      Flag: 0x40 Type: 1 Len: 1 Origin: 0
      Flag: 0x40 Type: 2 Len: 0 AS Path:
      Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
      Flag: 0xc0 Type: 16 Len: 16 Extended Community:
          target:64500:1
          esi-label:524279/All-Active
  "
  

• The AD per-EVI route has an eth-tag 0 and carries the service label in the NLRI.

# AD per-EVI route received on PE-2:
100 2021/02/25 17:06:06.204 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 73
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-AD Len: 25 RD: 192.0.2.3:1 ESI: 01:00:00:00:00:23:00:00:00:01,
                      tag: 0 Label: 8388512
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:1
        bgp-tunnel-encap:MPLS
"

[/]
A:admin@PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
      0                                                 LABEL 524282

u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
      MAX-ET                                            LABEL 0

-------------------------------------------------------------------------------
Routes : 2
===============================================================================

[/]
A:admin@PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01 hunt 
---snip---
===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Network        : n/a
Nexthop        : 192.0.2.3
From           : 192.0.2.3
Res. Nexthop   : 192.168.23.2
---snip---
Community      : target:64500:1 bgp-tunnel-encap:MPLS
---snip--- 
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:00:00:00:01
Tag            : 0                      
Route Dist.    : 192.0.2.3:1            
MPLS Label     : LABEL 524282 

---snip---

Network        : n/a 
Nexthop        : 192.0.2.3
From           : 192.0.2.3
Res. Nexthop   : 192.168.23.2
---snip---
Community      : target:64500:1 esi-label:524279/All-Active
---snip---
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:00:00:00:01
Tag            : MAX-ET                 
Route Dist.    : 192.0.2.3:1            
MPLS Label     : LABEL 0                
---snip---

From a service perspective, as soon as CE-11 sends some traffic, the PE learning the CE-11 MAC address will advertise it to the network. The remote PEs (PE-4 and PE-5) will create a new EVPN-MPLS ES destination to ESI-23, with two next-hops: PE-2 and PE-3. The following outputs show the following information:

• PE-4 has learned AD per-EVI/ES routes for ESI-23 from PE-2 and PE-3, as well as the CE-11 MAC address from PE-3 (because MTU-1 picked up its link to PE-3 to send CE-11 frames).

  [/]
  A:admin@PE-4# show router bgp routes evpn auto-disc esi 01:00:00:00:00:23:00:00:00:01
  ===============================================================================
   BGP Router ID:192.0.2.4        AS:64500       Local AS:64500
  ===============================================================================
   Legend -
   Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                   l - leaked, x - stale, > - best, b - backup, p - purge
   Origin codes  : i - IGP, e - EGP, ? - incomplete
  
  ===============================================================================
  BGP EVPN Auto-Disc Routes
  ===============================================================================
  Flag  Route Dist.         ESI                           NextHop
        Tag                                               Label
  -------------------------------------------------------------------------------
  u*>i  192.0.2.2:1         01:00:00:00:00:23:00:00:00:01 192.0.2.2
        0                                                 LABEL 524282
  
  u*>i  192.0.2.2:1         01:00:00:00:00:23:00:00:00:01 192.0.2.2
        MAX-ET                                            LABEL 0
  
  u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
        0                                                 LABEL 524282
  
  u*>i  192.0.2.3:1         01:00:00:00:00:23:00:00:00:01 192.0.2.3
        MAX-ET                                            LABEL 0
  
  -------------------------------------------------------------------------------
  Routes : 4
  ===============================================================================
  

  PE-4 has learned MAC address 00:00:11:11:11:11 of CE-11 in ESI-23. The BGP EVPN MAC route has PE-3 as next hop:

  [/]
  A:admin@PE-4# show router bgp routes evpn mac rd 192.0.2.3:1
  ===============================================================================
   BGP Router ID:192.0.2.4        AS:64500       Local AS:64500
  ===============================================================================
   Legend -
   Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                   l - leaked, x - stale, > - best, b - backup, p - purge
   Origin codes  : i - IGP, e - EGP, ? - incomplete
  
  ===============================================================================
  BGP EVPN MAC Routes
  ===============================================================================
  Flag  Route Dist.         MacAddr           ESI
        Tag                 Mac Mobility      Label1
                            Ip Address
                            NextHop
  -------------------------------------------------------------------------------
  u*>i  192.0.2.3:1         00:00:11:11:11:11 01:00:00:00:00:23:00:00:00:01
        0                   Seq:0             LABEL 524282
                            n/a
                            192.0.2.3
  
  -------------------------------------------------------------------------------
  Routes : 1
  ===============================================================================
  

• In the FDB for VPLS-1, PE-4 has learned the CE-11 MAC address associated with a newly created EVPN-MPLS ES destination:

  [/]
  A:admin@PE-4# show service id 1 fdb mac 00:00:11:11:11:11
  
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:11:11:11:11 eES:                    Evpn     02/25/21 17:14:43
                               01:00:00:00:00:23:00:00:00:01
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

• Due to the aliasing function, the newly created EVPN-MPLS ES destination to ESI-23 has two next-hops (PE-2 and PE-3), to which PE-4 can load-balance the unicast traffic because ecmp 2 is configured in the VPLS-1 of PE-4.

  [/]
  A:admin@PE-4# show service id 1 evpn-mpls
  
  ===============================================================================
  BGP EVPN-MPLS Dest
  ===============================================================================
  TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                  Transport:Tnl                             Sup BCast Domain
  -------------------------------------------------------------------------------
  192.0.2.2       524281        0           bum             02/25/2021 16:46:48
                  ldp:65538                                 No
  192.0.2.3       524281        0           bum             02/25/2021 16:46:48
                  ldp:65537                                 No
  192.0.2.5       524281        0           bum             02/25/2021 16:46:52
                  ldp:65539                                 No
  -------------------------------------------------------------------------------
  Number of entries : 3
  -------------------------------------------------------------------------------
  ===============================================================================
  
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:14:43
  -------------------------------------------------------------------------------
  Number of entries: 1
  -------------------------------------------------------------------------------
  ===============================================================================
  ---snip---
  

  The show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01 command shows the next-hops that the EVPN-MPLS ES destination is resolved to.

  [/]
  A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01
  
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:14:43
  ===============================================================================
  
  ===============================================================================
  BGP EVPN-MPLS Dest TEP Info
  ===============================================================================
  TEP Address              Egr Label                Last Change
                           Transport:Tnl-Id
  -------------------------------------------------------------------------------
  192.0.2.2                524282                   02/25/2021 17:14:43
                           ldp:65538
  192.0.2.3                524282                   02/25/2021 17:14:43
                           ldp:65537
  -------------------------------------------------------------------------------
  Number of entries : 2
  -------------------------------------------------------------------------------
  ===============================================================================
  

• PE-2 will show the CE-11 MAC address as learned locally in SAP lag-1:1 (because the data plane learning of the CE-11 MAC address happened in PE-2). For PE-3, even though it learned the MAC address from EVPN, it will install it as associated with SAP lag-1:1 because the EVPN route came with ESI-23, which is a local ESI. Because of this, whenever PE-3 receives a frame with MAC DA equal to the CE-11 MAC address, it will be able to forward the frame locally to the SAP lag-1:1. The following output shows the CE-11 MAC address as it is installed in PE-2 and PE-3:

  [/]
  A:admin@PE-2# show service id 1 fdb mac 00:00:11:11:11:11
  
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:11:11:11:11 sap:lag-1:1             L/90     02/25/21 17:14:43
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

  [/]
  A:admin@PE-3# show service id 1 fdb mac 00:00:11:11:11:11
  
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:11:11:11:11 sap:lag-1:1             Evpn     02/25/21 17:14:43
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

EVPN-MPLS single-active multi-homing: mass-withdraw, backup path illustrates two concepts in EVPN single-active multi-homing: mass-withdraw and backup path.

• With mass-withdraw, when ESI-45 goes down, PE-2 does not have to wait for all the MAC routes to be withdrawn to converge all the services. Instead, PE-4 will withdraw the AD per-ES routes (also the AD per-EVI and MAC routes) and that will be used at PE-2 as a notification to stop sending traffic to PE-4 for any MAC address associated with ESI-45.

• With backup path, when PE-2 is notified of the ESI-45 failure due to the withdrawn AD routes, it will not flush any MAC address associated with ESI-45. Instead, it will change the next-hop of the EVPN-MPLS ES destination to the remaining PE in the ESI-45. Backup path only works when there are two PEs in the same ES. If there were more than two PEs in ESI-45, PE-2 would flush all the MAC addresses upon receiving a mass-withdraw notification, because it would not know who the new active PE is.

The single-active multi-homing configuration example is based on EVPN-MPLS for VPLS services:

MTU-6 is connected to the EVPN network using single-active multi-homing. With the MTU-6 configuration, a VPLS service with active-standby spoke-SDP to PE-4 and PE-5 is configured. In PE-4 and PE-5, the SDP connected to MTU-6 is linked to ESI-45. Both will run the DF election algorithm for EVI 1, and the non-DF PE (PE-4 in this example) will bring down the spoke-SDP and notify MTU-6.

The following output shows the configuration of ESI-45 in PE-4 and PE-5, as well as the SDPs. The configuration of MTU-6 is also shown for completeness. It is important to keep the default — ignore-standby-signaling false — on MTU-6 spoke-SDPs because the PW switchover in MTU-6 will be triggered based on the PW status bits sent by PE-4 and PE-5.

SDP 46 with far-end MTU-6 is configured on PE-4:

# on PE-4:
configure {
    service {
        sdp 46 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.6
            }
        }

Ethernet segment ‟ESI-45” is configured on PE-4 as follows:

# on PE-4:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-45" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:01
                        multi-homing-mode single-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            sdp 46 {
                            }
                        }
                    }
                }
            }
        }

On PE-5, SDP 56 is configured as follows:

# on PE-5:
configure {
    service {
        sdp 56 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.6
            }
        }

Ethernet segment ‟ESI-45” is configured as follows on PE-5:

# on PE-5:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ESI-45" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:01
                        multi-homing-mode single-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            sdp 56 {
                            }
                        }
                    }
                }
            }
        }

On MTU-6, the service configuration is as follows:

# on MTU-6:
configure {
    service {
        sdp 64 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.4
            }
        }
        sdp 65 {
            admin-state enable
            delivery-type mpls
            ldp true
            far-end {
                ip-address 192.0.2.5
            }
        }
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            endpoint "CORE" {
            }
            spoke-sdp 64:1 {
                endpoint {
                    name "CORE"
                }
                stp {
                    admin-state disable 
                }
            }
            spoke-sdp 65:1 {
                endpoint {
                    name "CORE"
                }
                stp {
                    admin-state disable 
                }
            }
            sap 1/2/1:1 {
            }
        }

For a detailed description of the base BGP-EVPN instance and ES configuration, see the All-active multi-homing configuration section. The es-activation-timer, esi, service-carving-mode, boot-timer, and admin-state commands are used in the same way as for all-active multi-homing. Only the differences compared to all-active multi-homing are described here:

• multi-homing-mode single-active must be configured so that the ES acts as single-active. Optionally, multi-homing-mode single-active-no-esi-label can be configured, which controls the use of the ESI-label for single-active multi-homing. Although the ESI-label is always used in all-active multi-homing when sending BUM traffic between the PEs in the ES, it is configurable for single-active. However, Nokia recommends to use the default option (using ESI-label) to avoid potential transient issues when there is a DF switchover.

• association>sdp <sdp-id> is configured so that the ES can be associated with the SDP connected to MTU-6. Although all-active multi-homing only allows LAG associations to the ES, single-active allows LAG, port, and SDP. In this example, SDP is the option, because the access network is MPLS-based.

Similar to the all-active multi-homing case, when configuring the service in PE-4 and PE-5, the service objects are automatically associated with the ESI-45, because they are defined in the SDPs linked to the ESI. The configuration for VPLS 1 on PE-5 is as follows:

# on PE-5:
configure {
    service {
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            bgp 1 {
            }
            bgp-evpn {
                evi 1
                mpls 1 {
                    admin-state enable
                    ingress-replication-bum-label true
                    ecmp 2
                    auto-bind-tunnel {
                        resolution any
                    }
                }
            }
            spoke-sdp 56:1 {
            }
        }

In all-active multi-homing, the non-DF does not bring down the service SAP associated with the ES (it only removes it from the default multicast list). However, in single-active multi-homing, the service spoke-SDP (or SAP, if that was the object associated) is brought operationally down. The following output shows the spoke-SDP state in PE-4 (non-DF), as operationally down with the StandbyForMHProtocol flag and the Local Pw Bits that are signaled to MTU-6:

[/]
A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" evi evi-1 1

===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
1             1             0                   no  02/25/2021 17:19:02
===============================================================================

===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.4                               02/25/2021 17:18:56
192.0.2.5                               02/25/2021 17:19:02
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================

Spoke-SDP 46:1 is operationally down on PE-4:

[/]
A:admin@PE-4# show service id 1 sdp

===============================================================================
Services: Service Destination Points
===============================================================================
SdpId            Type     Far End addr    Adm     Opr       I.Lbl     E.Lbl
-------------------------------------------------------------------------------
46:1             Spok     192.0.2.6       Up      Down      524280    524281
-------------------------------------------------------------------------------
Number of SDPs : 1
-------------------------------------------------------------------------------
===============================================================================

Spoke-SDP 46:1 is operationally down with the StandbyForMHProtocol flag:

[/]
A:admin@PE-4# show service id 1 sdp 46:1 detail | match Flag
Flags              : StandbyForMHProtocol

The local PW bits (pwFwdingStandby) are sent to MTU-6:

[/]
A:admin@PE-4# show service id 1 sdp 46:1 detail | match Pw 
Local Pw Bits      : pwFwdingStandby
Peer Pw Bits       : None

The same commands used in the All-active multi-homing operation section can be used for single-active; see that section.

The show service system bgp-evpn ethernet-segment name ESI-45 command shows an Ethernet-segment Oper Multi-homing in addition to the configured Multi-homing mode. This occurs because, in spite of configuring the ES as all-active, it may operate as single-active if there is a mismatch between the modes advertised by PE-4 and PE-5 in the AD per-ES routes (per RFC 7432). In this example, the configured and the operational value are the same:

[/]
A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45"

===============================================================================
Service Ethernet Segment
===============================================================================
Name                    : ESI-45
Eth Seg Type            : None
Admin State             : Enabled            Oper State         : Up
ESI                     : 01:00:00:00:00:45:00:00:00:01
Multi-homing            : singleActive       Oper Multi-homing  : singleActive
ES SHG Label            : 524278
Source BMAC LSB         : <none>
Sdp Id                  : 46
ES Activation Timer     : 3 secs
Oper Group              : (Not Specified)
Svc Carving             : auto               Oper Svc Carving   : auto
Cfg Range Type          : primary
===============================================================================

As soon as CE-16 sends some traffic, the DF PE (PE-5) will learn the CE-16 MAC address and will advertise it to the network. The remote PEs (PE-2 and PE-3) will create a new EVPN-MPLS ES destination to ESI-45, but this time with only one next-hop, PE-5, because this is single-active multi-homing. The following outputs show the following information:

• PE-2 has learned AD per-EVI/ES routes for ESI-45 from PE-4 and PE-5, as well as the CE-16 MAC address from an ES EVPN-MPLS destination, which is resolved to PE-5 (the DF for ESI-45).

  [/]
  A:admin@PE-2# show router bgp routes evpn auto-disc esi 01:00:00:00:00:45:00:00:00:01
  ===============================================================================
   BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
  ===============================================================================
   Legend -
   Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                   l - leaked, x - stale, > - best, b - backup, p - purge
   Origin codes  : i - IGP, e - EGP, ? - incomplete
  
  ===============================================================================
  BGP EVPN Auto-Disc Routes
  ===============================================================================
  Flag  Route Dist.         ESI                           NextHop
        Tag                                               Label
  -------------------------------------------------------------------------------
  u*>i  192.0.2.4:1         01:00:00:00:00:45:00:00:00:01 192.0.2.4
        0                                                 LABEL 524282
  
  u*>i  192.0.2.4:1         01:00:00:00:00:45:00:00:00:01 192.0.2.4
        MAX-ET                                            LABEL 0
  
  u*>i  192.0.2.5:1         01:00:00:00:00:45:00:00:00:01 192.0.2.5
        0                                                 LABEL 524282
  
  u*>i  192.0.2.5:1         01:00:00:00:00:45:00:00:00:01 192.0.2.5
        MAX-ET                                            LABEL 0
  
  -------------------------------------------------------------------------------
  Routes : 4
  ===============================================================================
  

  PE-2 has learned the CE-16 MAC address from an ES EVPN-MPLS destination:

  [/]
  A:admin@PE-2# show service id 1 fdb mac 00:00:16:16:16:16
  
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:16:16:16:16 eES:                    Evpn     02/25/21 17:20:53
                               01:00:00:00:00:45:00:00:00:01
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

  On PE-2, the ES EVPN-MPLS destination is resolved to DF PE-5:

  [/]
  A:admin@PE-2# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:45:00:00:00:01
  
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:45:00:00:00:01   1                       02/25/2021 17:20:53
  ===============================================================================
  
  ===============================================================================
  BGP EVPN-MPLS Dest TEP Info
  ===============================================================================
  TEP Address              Egr Label                Last Change
                           Transport:Tnl-Id
  -------------------------------------------------------------------------------
  192.0.2.5                524282                   02/25/2021 17:20:53
                           ldp:65539
  -------------------------------------------------------------------------------
  Number of entries : 1
  -------------------------------------------------------------------------------
  ===============================================================================
  

• In this case, the local PEs, PE-4 and PE-5, will learn the CE MAC address from an EVPN-MPLS destination and a local spoke-SDP, respectively.

  [/]
  A:admin@PE-4# show service id 1 fdb mac 00:00:16:16:16:16
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:16:16:16:16 eES:                    Evpn     02/25/21 17:20:53
                               01:00:00:00:00:45:00:00:00:01
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

  The ES EVPN-MPLS destination is resolved to DF PE-5:

  [/]
  A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:45:00:00:00:01
  
  ===============================================================================
  BGP EVPN-MPLS Ethernet Segment Dest
  ===============================================================================
  Eth SegId                       Num. Macs               Last Change
  -------------------------------------------------------------------------------
  01:00:00:00:00:45:00:00:00:01   1                       02/25/2021 17:20:53
  ===============================================================================
  
  ===============================================================================
  BGP EVPN-MPLS Dest TEP Info
  ===============================================================================
  TEP Address              Egr Label                Last Change
                           Transport:Tnl-Id
  -------------------------------------------------------------------------------
  192.0.2.5                524282                   02/25/2021 17:20:53
                           ldp:65539
  -------------------------------------------------------------------------------
  Number of entries : 1
  -------------------------------------------------------------------------------
  ===============================================================================
  

  DF PE-5 learns the CE-16 MAC address from a local spoke SDP:

  [/]
  A:admin@PE-5# show service id 1 fdb mac 00:00:16:16:16:16
  
  ===============================================================================
  Forwarding Database, Service 1
  ===============================================================================
  ServId     MAC               Source-Identifier       Type     Last Change
              Transport:Tnl-Id                         Age
  -------------------------------------------------------------------------------
  1          00:00:16:16:16:16 sdp:56:1                L/180    02/25/21 17:20:53
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
  ===============================================================================
  

If either ES fails, a DF re-election will happen and the corresponding AD per-ES/EVI routes will be withdrawn, causing the remote PEs to modify the list of next-hops for the EVPN-MPLS ES destination. The following example illustrates a failure on the SDP between MTU-6 and PE-5 (the DF).

1. A failure occurs in the LSP between MTU-6 and PE-5. This can be any event that brings the SDP down.

   # log "99" on PE-5:
   94 2021/02/25 17:25:00.632 CET MINOR: SVCMGR #2303 Base
   "Status of SDP 56 changed to admin=up oper=down"
   

2. Immediately, PE-5 gives up the DF role and withdraws the ES route, as well as the AD routes and MAC routes. As soon as PE-4 receives any ES or AD withdraw, it will re-run the DF algorithm and, when the es-activation-timer expires, it will become the DF and activate its spoke-SDP.

   # log 99 on PE-5: 
   96 2021/02/25 17:25:00.633 CET MINOR: SVCMGR #2094 Base
   "Ethernet Segment:ESI-45, EVI:1, Designated Forwarding state changed to:false"
   

   The ES in PE-5 is operational down:

   [/]
   A:admin@PE-5# show service system bgp-evpn ethernet-segment name "ESI-45"
     
   ===============================================================================
   Service Ethernet Segment
   ===============================================================================
   Name                    : ESI-45
   Eth Seg Type            : None
   Admin State             : Enabled            Oper State         : Down
   ESI                     : 01:00:00:00:00:45:00:00:00:01
   Multi-homing            : singleActive       Oper Multi-homing  : singleActive
   ES SHG Label            : 524279
   Source BMAC LSB         : <none>
   Sdp Id                  : 56
   ES Activation Timer     : 3 secs
   Oper Group              : (Not Specified)
   Svc Carving             : auto               Oper Svc Carving   : auto
   Cfg Range Type          : primary
   ===============================================================================
   

   PE-5 is no longer the DF and the only DF candidate is PE-4:

   [/]
   A:admin@PE-5# show service system bgp-evpn ethernet-segment name "ESI-45" evi evi-1 1
   
   ===============================================================================
   EVI DF and Candidate List
   ===============================================================================
   EVI           SvcId         Actv Timer Rem      DF  DF Last Change
   -------------------------------------------------------------------------------
   1             1             0                   no  02/25/2021 17:25:01
   ===============================================================================
   
   ===============================================================================
   DF Candidates                           Time Added
   -------------------------------------------------------------------------------
   192.0.2.4                               02/25/2021 17:19:03
   -------------------------------------------------------------------------------
   Number of entries: 1
   ===============================================================================
   

   PE-4 becomes the DF and the spoke-SDP 46:1 is brought up.

   # log "99" on PE-4: 
   102 2021/02/25 17:25:03.598 CET MINOR: SVCMGR #2094 Base
   "Ethernet Segment:ESI-45, EVI:1, Designated Forwarding state changed to:true"
   
   103 2021/02/25 17:25:03.598 CET MINOR: SVCMGR #2326 Base
   "Status of SDP Bind 46:1 in service 1 (customer 1) local PW status bits changed to none"
   
   104 2021/02/25 17:25:03.598 CET MINOR: SVCMGR #2306 Base
   "Status of SDP Bind 46:1 in service 1 (customer 1) changed to admin=up oper=up flags="
   

   The ES is up in PE-4:

   [/]
   A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45"
   
   ===============================================================================
   Service Ethernet Segment
   ===============================================================================
   Name                    : ESI-45
   Eth Seg Type            : None
   Admin State             : Enabled            Oper State         : Up
   ESI                     : 01:00:00:00:00:45:00:00:00:01
   Multi-homing            : singleActive       Oper Multi-homing  : singleActive
   ES SHG Label            : 524278
   Source BMAC LSB         : <none>
   Sdp Id                  : 46
   ES Activation Timer     : 3 secs
   Oper Group              : (Not Specified)
   Svc Carving             : auto               Oper Svc Carving   : auto
   Cfg Range Type          : primary
   ===============================================================================
   

   PE-4 is the DF and there are no other DF candidates:

   [/]
   A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" evi evi-1 1
   
   ===============================================================================
   EVI DF and Candidate List
   ===============================================================================
   EVI           SvcId         Actv Timer Rem      DF  DF Last Change
   -------------------------------------------------------------------------------
   1             1             0                   yes 02/25/2021 17:25:04
   ===============================================================================
   
   ===============================================================================
   DF Candidates                           Time Added
   -------------------------------------------------------------------------------
   192.0.2.4                               02/25/2021 17:18:56
   -------------------------------------------------------------------------------
   Number of entries: 1
   ===============================================================================
   

3. The remote PEs, PE-2 and PE-3, receive the BGP-EVPN routes withdrawal and modify the next-hop for the EVPN-MPLS ES destination.

   # on PE-2:
   186 2021/02/25 17:25:00.634 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
   "Peer 1: 192.0.2.5: UPDATE
   Peer 1: 192.0.2.5 - Received BGP UPDATE:
       Withdrawn Length = 0
       Total Path Attr Length = 86
       Flag: 0x90 Type: 15 Len: 82 Multiprotocol Unreachable NLRI:
           Address Family EVPN
           Type: EVPN-AD Len: 25 RD: 192.0.2.5:1 ESI: 01:00:00:00:00:45:00:00:00:01, 
                 tag: MAX-ET Label: 0 
           Type: EVPN-AD Len: 25 RD: 192.0.2.5:1 ESI: 01:00:00:00:00:45:00:00:00:01, 
                 tag: 0 Label: 0 
           Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.5:0 
                 ESI: 01:00:00:00:00:45:00:00:00:01, IP-Len: 4 Orig-IP-Addr: 192.0.2.5 
   "
   

   The ES EVPN-MPLS destination is resolved to the DF PE-4:

   [/]
   A:admin@PE-2# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:45:00:00:00:01
   
   ===============================================================================
   BGP EVPN-MPLS Ethernet Segment Dest
   ===============================================================================
   Eth SegId                       Num. Macs               Last Change
   -------------------------------------------------------------------------------
   01:00:00:00:00:45:00:00:00:01   1                       02/25/2021 17:25:19
   ===============================================================================
   
   ===============================================================================
   BGP EVPN-MPLS Dest TEP Info
   ===============================================================================
   TEP Address              Egr Label                Last Change
                            Transport:Tnl-Id
   -------------------------------------------------------------------------------
   192.0.2.4                524282                   02/25/2021 17:25:19
                            ldp:65538
   -------------------------------------------------------------------------------
   Number of entries : 1
   -------------------------------------------------------------------------------
   ===============================================================================
   

The following must be considered:

• The DF election procedure is revertive, that is, when the failed SDP comes back up, PE-5 will take over again as DF and the network will re-converge.

• The DF election is triggered by the following events:

  • Enabling an ES (admin-state enable) triggers the DF election for all the services in the ES.

  • A new update/withdrawal of an ES route (containing an ESI configured locally) triggers the DF election for all the services in the ESI.

  • A new update/withdrawal of an AD per-ES route (containing an ESI configured locally) triggers the DF election for all the services associated with the list of RTs received along with the route.

  • A new update of an AD per-ES route with a change in the ESI-label extended community (single-active bit or MPLS label) triggers the DF election for all the services associated with the list of RTs received along with the route.

  • A new update/withdrawal of an AD route per-EVI (containing an ESI configured locally) triggers the DF election for that service.

The selection of the best route for a MAC address is as follows:

• If a PE receives more than one route for the same MAC address, the best MAC route is chosen:

  • If the route key is equal in two or more routes (that is, the mac, mac-length, ip, ip-length, RD, eth-tag), then regular BGP selection applies:

    • If local-pref, AS-path, origin, and MED are equal, the lowest IGP distance to the BGP next-hop is chosen (unless ignore-nh-metric is configured). If the BGP next-hop is resolved by an LSP, the cost from the tunnel-table is used.

    • As a last resort tie-breaker, the route with the lowest originator ID, or received from the peer with the lowest BGP Identifier, is chosen (unless ignore-router-id is configured and the routes being compared are EBGP routes).

  • If the mac-length, mac, ip-length, ip, eth-tag are equal, and the RD is different, the EVPN selection process is applied in the following order:

    • Conditional static MAC addresses (local protected MAC addresses)

    • EVPN static MAC addresses (remote protected MAC addresses)

    • Data plane learned MAC addresses (regular learning on SAPs/SDP-bindings)

    • EVPN MAC addresses with higher sequence number

    • Lowest IP address (next-hop IP of the EVPN NLRI)

    • Lowest Ethernet tag (will be normally zero)

    • Lowest RD

• After a MAC route is selected, the system checks for an associated ES.

  • If it has an ES, the system uses the MAC address as the EVPN-MPLS ES destination. The ES destination is constructed based on the AD per-EVI routes received for that ES (regardless of MAC address priorities with the ES).

  • The system selects the first ECMP number of AD per-EVI routes arranged by the IP address of PEs (lower IPs are selected first).

  • If the same PE has advertised multiple RDs, the system selects the route with the lowest RD for that PE.

In the example of EVPN-MPLS for VPLS services, PE-4 resolves the next-hops for ESI-23 as described in the second choice above, that is, because ECMP=2, the two available next-hops are chosen. If ECMP is changed to 1, PE-4 will pick up the lower IP (in the BGP next-hop). This is illustrated in the following output:

[/]
A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01

===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                       Num. Macs               Last Change
-------------------------------------------------------------------------------
01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:14:43
===============================================================================

===============================================================================
BGP EVPN-MPLS Dest TEP Info
===============================================================================
TEP Address              Egr Label                Last Change
                         Transport:Tnl-Id
-------------------------------------------------------------------------------
192.0.2.2                524282                   02/25/2021 17:14:43
                         ldp:65538
192.0.2.3                524282                   02/25/2021 17:14:43
                         ldp:65537
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================

When ECMP equals 1, only the BGP next hop with the lower IP is chosen:

# on PE-4:
configure {
    service {
        vpls "VPLS1" {
            bgp-evpn {
                mpls 1 {
                    ecmp 1

[/]
A:admin@PE-4# show service id 1 evpn-mpls esi esi-1 01:00:00:00:00:23:00:00:00:01

===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId                       Num. Macs               Last Change
-------------------------------------------------------------------------------
01:00:00:00:00:23:00:00:00:01   1                       02/25/2021 17:37:29
===============================================================================

===============================================================================
BGP EVPN-MPLS Dest TEP Info
===============================================================================
TEP Address              Egr Label                Last Change
                         Transport:Tnl-Id
-------------------------------------------------------------------------------
192.0.2.2                524282                   02/25/2021 17:14:43
                         ldp:65538
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
===============================================================================

EVPN-MPLS services support EVPN-MH (EVPN multi-homing) and also BGP-MH as in chapter BGP Multi-Homing for VPLS Networks. While EVPN-MH is the standard way of providing access resiliency in RFC 7432, BGP-MH is also a standard mechanism supported in VPLS or EVPN networks. The following table provides some comparison between both technologies.

Table 1. Comparing EVPN multi-homing and BGP multi-homing

VPN Requirements	EVPN-MH	BGP-MH	Comments
All-active MH (flow-based load-balancing)	Yes	No	EVPN-MH provides better bandwidth utilization
Single-active MH (service-based load-balancing)	Yes	Yes
DF PE election - automatic service balancing	Yes Service-carving	No Requires vsi policies and LP manipulation	EVPN-MH provides better automation
DF PE election – manual configuration per service	Yes	No	EVPN-MH allows for manual DF config for EVIs and ISIDs (2 PEs)
Split-horizon indication in the data plane	Yes ESI-label	No	Prevents transient loops when dual-active DFs show up
DF indication in the control plane	No	Yes	BGP MH guarantees one DF at a time. EVPN relies on timers to ensure one DF at a time
Allows multiple SAPs or SDP-bindings per service on the same site	No	Yes Through the use of SHGs
Boot timer and site(es)-activation-timers	Yes	Yes	BGP-MH supports more granular configuration (service level)
Support for oper-groups	No	Yes
Non-DF notification to the CE (MPLS and CFM)	Yes	Yes	Avoids blackholing

In addition to the preceding comparison, the following configuration excerpt compares EVPN-MH with BGP-MH on a bgp-evpn VPLS service and shows that, while EVPN-MH does not have any configuration at service level, BGP-MH is configured within the VPLS context, which gives a more granular control over the redundancy provided. See the BGP Multi-Homing for VPLS Networks chapter for more information about BGP-MH.

[ex:/configure service system]
A:admin@PE-4# info
    bgp {
        evpn {
            ethernet-segment "ESI-45" {
                admin-state enable
                esi 0x01000000004500000001
                multi-homing-mode single-active
                df-election {
                    es-activation-timer 3
                }
                association {
                    sdp 46 {
                    }
                }
            }
        }
    }

[ex:/configure service vpls "VPLS1"]
A:admin@PE-4# info
    admin-state enable
    service-id 1
    customer "1"
    bgp 1 {
    }
    bgp-evpn {
        evi 1
        mpls 1 {
            admin-state enable
            ingress-replication-bum-label true
            ecmp 2
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    spoke-sdp 46:1 {
    }

For BGP multi-homing, site ‟site-1” is configured, as follows. The RD needs to be configured in the bgp context.

[ex:/configure service vpls "VPLS1"]
A:admin@PE-4# info
    admin-state enable
    service-id 1
    customer "1"
    bgp 1 {
        route-distinguisher "192.0.2.4:1"
    }
    bgp-evpn {
        evi 1
        mpls 1 {
            admin-state enable
            ingress-replication-bum-label true
            ecmp 2
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    spoke-sdp 46:1 {
    }
    bgp-mh-site "site-1" {
        admin-state enable
        id 1
        activation-timer 3
        spoke-sdp 46:1
    }

Although not strictly a BGP-EVPN configuration, vpls>proxy-arp and vpls>proxy-nd functions are typically enabled along with EVPN-MPLS in order to reduce the amount of flooding in the network. The proxy-ARP/ND agent in the VPLS service will snoop ARP-requests and/or Neighbor Solicitation messages and will reply to those messages locally (if the information is known) without having to flood the requests to the network.

The configuration options for proxy-ARP are the following:

[ex:/configure service vpls "VPLS1"]
A:admin@PE-2# proxy-arp ?

 proxy-arp

 admin-state           - Administrative state of the proxy
 age-time              - Aging timer for proxy entries, where entries are flushed
                         upon timer expiry
 apply-groups          - Apply a configuration group at this level
 apply-groups-exclude  - Exclude a configuration group at this level
 duplicate-detect      + Enter the duplicate-detect context
 dynamic-arp           + Enter the dynamic-arp context
 dynamic-populate      - Populate proxy ARP entries from snooped GARP/ARP/ND 
                         messages on SAPs/SDP-bindings
 evpn                  + Enter the evpn context
 send-refresh          - Time at which to send a refresh message
 static-arp            + Enter the static-arp context
 table-size            - Maximum number of learned and static entries allowed in 
                         the proxy table of this service

The configuration options for proxy-ND are the following:

[ex:/configure service vpls "VPLS1"]
A:admin@PE-2# proxy-nd ?

 proxy-nd

 admin-state           - Administrative state of the proxy
 age-time              - Aging timer for proxy entries, where entries are flushed 
                         upon timer expiry
 apply-groups          - Apply a configuration group at this level
 apply-groups-exclude  - Exclude a configuration group at this level
 duplicate-detect      + Enter the duplicate-detect context
 dynamic-neighbor      + Enter the dynamic-neighbor context
 dynamic-populate      - Populate proxy ARP entries from snooped GARP/ARP/ND 
                         messages on SAPs/SDP-bindings
 evpn                  + Enter the evpn context
 send-refresh          - Time at which to send a refresh message
 static-neighbor       + Enter the static-neighbor context
 table-size            - Maximum number of learned and static entries allowed in 
                         the proxy table of this service

When proxy-ARP/ND is enabled, the following configuration guidelines must be followed:

• dynamic-populate should be used only in networks with a consistent configuration of this command in all PEs.

• When using dynamic-populate, the age-time value should be configured to a value equal to three times the send-refresh value. This will help reduce the EVPN withdrawals and re-advertisements in the network.

• With large age-time values, it would be sufficient to configure the send-refresh value to half of the proxy-ARP/ND age-time or FDB age-time.

• In scaled environments (with thousands of services), it is not recommended to set the send-refresh value to less than 300 s. In such scenarios, Nokia recommends using a minimum proxy-ARP/ND age-time and FDB age of 900 s.

• The use of the following commands reduces or suppresses the ARP/ND flooding in an EVPN network, because EVPN MAC routes replace the function of the regular data plane ARP/ND messages:

  • proxy-arp>evpn>flood> gratuitous-arp false

  • proxy-arp>evpn>flood> unknown-arp-req false

  • proxy-nd>evpn>flood> unknown-neighbor-solicitation false

  • proxy-nd>evpn>flood> unknown-neighbor-advertise-router false

  • proxy-nd>evpn>flood> unknown-neighbor-advertise-host false

• Nokia recommends using the preceding commands only in EVPN networks where the CEs are routers directly connected to an SR OS node acting as the PE. Networks using aggregation switches between the host/routers and the PEs should flood GARP/ND messages in EVPN to make sure the remote caches are updated and BGP does not miss the advertisement of these entries.

• When duplicate-detect anti-spoof-mac is used with proxy-ARP/ND, ingress filters (in the access SAPs/SDP-bindings) should be configured to drop all traffic with destination anti-spoof-mac. The same MAC address should be configured in all PEs where duplicate-detect is active.

• When proxy-ND is used, the configuration of the following commands should be consistent in all the PEs in the network:

  • proxy-nd>evpn>flood> unknown-neighbor-advertise-router

  • proxy-nd>evpn>flood> unknown-neighbor-advertise-host

  • proxy-nd>evpn>advertise-neighbor-type

• Because EVPN does not propagate the router flag in IPv6--> MAC address advertisements, in a mixed network with hosts and routers where evpn>advertise-neighbor-type router is configured, unsolicited host NA messages should be flooded so that the entire network gets to learn all of the host and router ND entries. In the same way, evpn>advertise-neighbor-type host should be configured so that unsolicited router NA messages are flooded.

Finally, along with proxy-ARP/ND, vpls>fdb>discard-unknown true may be used in some EVPN-MPLS deployments where all the CEs are routers and they announce themselves to the network by sending GARPs or NAs (Neighbor Solicitation messages). According to RFC 7432, whether or not to flood packets to unknown destination MAC addresses should be an administrative choice, depending on how learning happens between CEs and PEs. Discard-unknown provides that administrative choice in case all the MAC addresses in an EVI can be learned even before any traffic is exchanged.

Proxy-ARP/ND along with discard-unknown helps reduce the BUM traffic in an EVPN network significantly; however, their use must be analyzed and considered, depending on the type of CEs in the EVI.

An example of proxy-ARP configuration is as follows. This configuration should be added to all PEs. When a new ARP message is received on any of the PEs, they will learn the IP-MAC address pair and will advertise it to the network.

# on PE-2, PE-3, PE-4, PE-5:
configure {
    service {
        vpls "VPLS1" {
            proxy-arp {
                admin-state enable
                dynamic-populate true
                age-time 900
                send-refresh 300
            }

Enabling proxy-ARP increases the number of MAC/IP routes being sent by the PEs. This is due to the following reasons:

• An additional MAC/IP route will be advertised per new learned IP-MAC address pair, regardless of having advertised the same MAC address already.

• A MAC per VPLS service will be advertised with a system MAC address. That MAC address will be used as MAC SA for proxy-ARP confirm messages when an IP moves to a different PE.

The following output shows the MAC/IP routes on PE-2 when proxy-ARP is enabled in the network.

[/]
A:admin@PE-2# show router bgp routes evpn mac
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address
                          NextHop
-------------------------------------------------------------------------------
u*>i  192.0.2.3:1         02:17:ff:00:03:3a ESI-0
      0                   Static            LABEL 524282
                          n/a
                          192.0.2.3

u*>i  192.0.2.4:1         02:1b:ff:00:03:3a ESI-0
      0                   Static            LABEL 524282
                          n/a
                          192.0.2.4

u*>i  192.0.2.5:1         00:00:16:16:16:16 01:00:00:00:00:45:00:00:00:01
      0                   Seq:0             LABEL 524282
                          n/a
                          192.0.2.5

u*>i  192.0.2.5:1         02:1f:ff:00:03:3a ESI-0
      0                   Static            LABEL 524282
                          n/a
                          192.0.2.5

-------------------------------------------------------------------------------
Routes : 4
===============================================================================