EVPN for VXLAN Tunnels (Layer 2)

EVPN-VXLAN example topology shows the topology used in this example.

The example topology shows two overlay (VXLAN) networks interconnected by an MPLS network:

• PE-1, PE-2, and PE-3 are part of Overlay-Network-1

• PE-4, PE-5, and PE-6 are part of Overlay-Network-2

CE-1, CE-3, and CE-6 belong to the same IP subnet, therefore, Layer 2 connectivity must be provided to them.

The example topology can illustrate a Data Center Interconnect (DCI) example, where Overlay-Network-1 and Overlay-Network-2 are two data centers interconnected through an MPLS WAN. In this application, CE-1, CE-3, and CE-6 simulate virtual machines or appliances, PE-2/3/4/5 act as DC GWs and PE-1/6 as NVEs (or virtual PEs running on compute infrastructure).

The following protocols and objects are configured beforehand:

• The ports interconnecting the six PEs in EVPN-VXLAN example topology are configured as network ports (or hybrid) and have router network interfaces defined on them. Only the ports connected to the CEs are configured as access ports.

• The six PEs shown in the EVPN-VXLAN example topology are running IS-IS for the global routing table with the four core PEs interconnected using IS-IS Level-2 point-to-point interfaces and each overlay network is using IS-IS Level-1 point-to-point interfaces.

• LDP is used as the MPLS protocol to signal transport tunnel labels among PE-2, PE-3, PE-4, and PE-5. There is no LDP running in the two overlay networks.

• The network port MTU (in all the ports sending/receiving VXLAN packets) must be at least 50 bytes (54 if dot1q encapsulation is used) greater than the service MTU in order to accommodate the size of the VXLAN header.

Once the IGP infrastructure and LDP are enabled in the core, BGP has to be configured. In this example, two BGP families have to be enabled: EVPN within each overlay network for the exchange of MAC/IP addresses and setting up the flooding domains, and L2-VPN for the use of BGP-MH and BGP-AD in the VPLS-MPLS network.

As an example, the following CLI output shows the relevant BGP configuration of PE-1, which only needs the EVPN family. PE-6 would have a similar BGP configuration. The use of route reflectors (RRs) in this type of scenarios is common. Although this example does not use RRs, an EVPN RR could have been used in Overlay-Network-1 and Overlay-Network-2 and an L2-VPN RR could have been used in the core VPLS-MPLS network.

# on PE-1:
configure {
    router "Base" {
        autonomous-system 64500
        bgp {
            vpn-apply-export true
            vpn-apply-import true
            rapid-withdrawal true
            peer-ip-tracking true
            rapid-update {
                evpn true
            }
            group "DC" {
                peer-as 64500
                family {
                    evpn true
                }
            }
            neighbor "192.0.2.2" {
                group "DC"
            }
            neighbor "192.0.2.3" {
                group "DC"
            }
        }

The BGP configuration on PE-2 is as follows:

# on PE-2:
configure {
    router "Base" {
        autonomous-system 64500
        bgp {
            vpn-apply-export true
            vpn-apply-import true
            rapid-withdrawal true
            peer-ip-tracking true
            rapid-update {
                l2-vpn true
                evpn true
            }
            group "DC" {
                peer-as 64500
                family {
                    l2-vpn true
                    evpn true
                }
            }
            group "WAN" {
                peer-as 64500
                family {
                    l2-vpn true
                }
            }
            neighbor "192.0.2.1" {
                group "DC"
            }
            neighbor "192.0.2.3" {
                group "DC"
            }
            neighbor "192.0.2.4" {
                group "WAN"
            }
            neighbor "192.0.2.5" {
                group "WAN"
            }

The BGP configuration on PE-3 is as follows:

# on PE-3:
configure {
    router "Base" {
        autonomous-system 64500
        bgp {
            vpn-apply-export true
            vpn-apply-import true
            rapid-withdrawal true
            peer-ip-tracking true
            rapid-update {
                l2-vpn true
                evpn true
            }
            group "DC" {
                peer-as 64500
                family {
                    l2-vpn true
                    evpn true
                }
            }
            group "WAN" {
                peer-as 64500
                family {
                    l2-vpn true
                }
            }
            neighbor "192.0.2.1" {
                group "DC"
            }
            neighbor "192.0.2.2" {
                group "DC"
            }
            neighbor "192.0.2.4" {
                group "WAN"
            }
            neighbor "192.0.2.5" {
                group "WAN"
            }

The BGP configuration on PE-4 and PE-5 is equivalent.

BGP adjacencies and enabled families shows the BGP peering sessions among the PEs and the enabled BGP families. PE-1 will only establish an EVPN peering session with its peers (only the EVPN family is enabled on PE-1), even though PE-2 and PE-3 have EVPN and L2-VPN families configured.

Once the network infrastructure is running properly, the actual service configuration can be carried out. The following CLI outputs show the configuration of VPLS 1 in PE-1, PE-2, and PE-3 as per the topology illustrated in EVPN-VXLAN example topology.

VPLS 1 in those three PEs are interconnected using VXLAN bindings, whereas PE-2 and PE-3 are connected to the remote PEs by means of BGP-AD SDP-bindings. Although BGP-AD SDP-bindings are used in this example for the connectivity of the EVPN-VXLAN PEs to a regular VPLS network, SAPs, BGP-VPLS spoke-SDPs, manual spoke-SDPs, or mesh-SDPs could have been used instead.

VPLS 1 is configured on PE-1, as follows:

# on PE-1:
configure {
    service {
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            vxlan {
                instance 1 {
                    vni 1
                }
            }
            bgp 1 {
                route-distinguisher "192.0.2.1:1"
                route-target {
                    export "target:64500:12"
                    import "target:64500:12"
                }
            }
            bgp-evpn {
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
            sap 1/2/1:1 {
            }
        }

EVPN-VXLAN is enabled by the configuration of a valid VXLAN Network Identifier (VNI) and the bgp-evpn>vxlan>admin-state enable command. These two commands, along with the required BGP Route Distinguisher (RD) and Route Target (RT) information, are the minimum mandatory attributes:

• The VNI is a 24-bit identifier with valid values in the [1..16777215] range. This defines the VNI that SR OS will use in the EVPN routes generated for the VPLS service, and therefore the VNI that the system expects to see in the VXLAN packets destined to that particular VPLS service. The configured VNI determines the VNI that has to be received in the packets for the VPLS service, but not the VNI that will be sent in VXLAN packets to remote PEs for the service. In other words, in this example, VPLS 1 is configured with VNI=1 in all the PEs; however, each PE could have used a different VNI. The VNI is a system-wide significant value and two VPLS services cannot be configured with the same VNI.

• The bgp-evpn>vxlan>admin-state enable command enables the use of EVPN for VXLAN. It requires the previous configuration of the VNI, RD, and RT. As soon as this command is executed, EVPN will advertise an inclusive multicast route to all the BGP EVPN peers (regardless of the existing SAP/SDP-binding operational status). The exchange of inclusive multicast routes allows the establishment of the VXLAN bindings among the PEs.

Upon the reception of the EVPN inclusive multicast routes from PE-2 and PE-3, PE-1 will automatically set up its VXLAN bindings for VPLS 1. A VXLAN binding is represented by an (egress VTEP, egress VNI) pair, where VTEP is a VXLAN Termination End Point. This can be shown with the following show commands on PE-1:

[/]
A:admin@PE-1# show service vxlan

===============================================================================
VXLAN Tunnel Endpoints (VTEPs)
===============================================================================
VTEP Address                                 VXLAN Dest        ES Dest
-------------------------------------------------------------------------------
192.0.2.2                                    1                 0
192.0.2.3                                    1                 0
-------------------------------------------------------------------------------
Number of VTEPs: 2
-------------------------------------------------------------------------------
===============================================================================

[/]
A:admin@PE-1# show service id 1 vxlan instance 1 destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
Instance    VTEP Address                            Egress VNI  EvpnStatic Num
 Mcast       Oper State                              L2 PBR     SupBcasDom MACs
-------------------------------------------------------------------------------
1           192.0.2.2                               1           evpn       1
 BUM         Up                                      No          No
1           192.0.2.3                               1           evpn       1
 BUM         Up                                      No          No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 2
-------------------------------------------------------------------------------
===============================================================================
---snip---

To actually see this output, the VPLS service needs to be configured on all PEs, with import and export policy ‟vsi-policy-1” defined on the core PEs; see further. As can be seen in the CLI output, PE-1 has two VXLAN bindings: one to PE-2 and one to PE-3. Both use egress VNI=1 (the actual VNI used in its egress VXLAN packets) and both are part of the flooding multicast list (BUM) for VPLS 1 and are up. There is no layer 2 Policy-Based Routing (L2 PBR).

• The Mcast= BUM entry is set when the proper inclusive multicast route is received from the remote VTEP. The VXLAN binding will be used to flood BUM packets.

• The Oper State is based on the existence of the VTEP in the global routing table.

The VPLS 1 configuration of PE-2 and PE-3 is as follows:

# on PE-2:
configure {
    service {
        pw-template "PW1" {
            pw-template-id 1
        }
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            vxlan {
                instance 1 {
                    vni 1
                }
            }
            bgp 1 {
                route-distinguisher "192.0.2.2:1"
                vsi-import ["vsi-policy-1"]
                vsi-export ["vsi-policy-1"]
                pw-template-binding "PW1" {
                    split-horizon-group "CORE"
                }
            }
             bgp-ad {
                admin-state enable
                vpls-id "64500:1"
            }
            bgp-evpn {
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
            bgp-mh-site "site-1" {
                admin-state enable
                id 1
                shg-name "CORE"
            }
        }

# on PE-3:
configure {
    service {
        pw-template "PW1" {
            pw-template-id 1
        }
        vpls "VPLS1" {
            admin-state enable
            service-id 1
            customer "1"
            vxlan {
                instance 1 {
                    vni 1
                }
            }
            bgp 1 {
                route-distinguisher "192.0.2.3:1"
                vsi-import ["vsi-policy-1"]
                vsi-export ["vsi-policy-1"]
                pw-template-binding "PW1" {
                    split-horizon-group "CORE"
                }
            }
            bgp-ad {
                admin-state enable
                vpls-id "64500:1"
            }
            bgp-evpn {
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
            bgp-mh-site "site-1" {
                admin-state enable
                id 1
                shg-name "CORE"
                }
            }
            sap 1/2/1:1 {
            }

In addition to the VNI and bgp-evpn>vxlan>admin-state enable commands for enabling EVPN-VXLAN in VPLS 1, PE-2 and PE-3 require the configuration of BGP-AD for the discovery and establishment of FEC129 spoke-SDPs to the remote PEs in the core, as well as BGP-MH for redundancy. As described in EVPN-VXLAN example topology, there are two BGP-MH sites defined in the network: site-1 is used on PE-2/PE-3 and site-2 is used on PE-4/PE-5. Only one of the two gateway PEs in each overlay network will be the Designated Forwarder (DF) for VPLS 1, and only the DF will send/receive traffic for VPLS 1 in the overlay network. The following considerations must be taken into account when configuring the connectivity of EVPN-VXLAN services to regular VPLS objects:

• As discussed, in this example, BGP-AD spoke-SDPs are used, but SAPs, BGP-VPLS spoke-SDPs, manual spoke-SDPs, or mesh-SDPs are also supported.

• In this example, BGP-AD spoke-SDPs are auto-instantiated using pw-template-binding "PW1" split-horizon-group “CORE”.

  • This requires the creation of the pw-template "PW1" (config>service>pw-template "PW1").

• The split-horizon group CORE is added to the BGP-MH site ‟site-1”. This statement will ensure that all the spoke-SDPs automatically established to the remote PEs are part of the BGP-MH site.

• Although the route targets for the overlay network and the VPLS-MPLS network can have the same value for the same VPLS service, they are usually different. This example assumes the use of RT-DC-1 in Overlay-Network-1 and RT-WAN-1 in the VPLS-MPLS core for VPLS 1. The ‟vsi-policy-1” allows the system to export and import the right RTs for VPLS 1 on the core PEs:

  # on PE-2 and PE-3:
  configure {
      policy-options {
          community "RT-DC-1" {
              member "target:64500:12" { }
          }
          community "RT-WAN-1" {
              member "target:64500:11" { }
          }
          policy-statement "vsi-policy-1" {
              entry 10 {          # to import all the EVPN routes with RT-DC-1
                  from {
                      family [evpn]
                      community {
                          name "RT-DC-1"
                      }
                  }
                  action {
                      action-type accept
                  }
              }
              entry 20 {           # to import all the BGP-AD/MH routes from the WAN
                  from {
                      family [l2-vpn]
                      community {
                          name "RT-WAN-1"
                      }
                  }
                  action {
                      action-type accept
                  }
              }
              entry 30 {          # to export all the EVPN routes with "RT-DC-1"
                  from {
                      family [evpn]
                  }
                  action {
                      action-type accept
                      community {
                          add ["RT-DC-1"]
                      }
                  }
              }
              entry 40 {           # to export all the BGP-AD/MH routes with "RT-WAN-1"
                  from {
                      family [l2-vpn]
                  }
                  action {
                      action-type accept
                      community {
                          add ["RT-WAN-1"]
                      }
                  }
              }
              default-action {
                  action-type reject
              }
          }
  

Once PE-2 and PE-3 are configured as shown, they will set up the spoke-SDPs and will run the DF election algorithm to determine the operational status of those spoke-SDPs. See chapters LDP VPLS Using BGP Auto-Discovery and BGP Multi-Homing for VPLS Networks for more information about the use of BGP-AD and BGP-MH.

In the configuration for VPLS 1, both gateway PEs, PE-2 and PE-3, will attempt to establish two parallel Layer 2 paths between each other (a BGP-AD spoke-SDP and an EVPN VXLAN binding). Because that would create a Layer 2 loop, the SR OS implementation gives priority to the EVPN path and only the VXLAN binding will be active. In other words, when a VXLAN (egress VTEP, VNI) and a spoke-SDP are attempted to be set up to the same far-end IP address at the same time, the VXLAN path will prevail and the spoke-SDP will be kept down. The spoke-SDP will only be brought up if the VXLAN (egress VTEP, VNI) goes down.

This behavior can be easily observed in this setup by using the following show commands. In PE-2, the spoke-SDP to far-end PE-3 will be down with an EvpnRouteConflict Flag. The (egress VTEP, VNI) = (192.0.2.3, 1) VXLAN bind will be up.

[/]
A:admin@PE-2# show service id 1 base 

===============================================================================
Service Basic Information
===============================================================================
Service Id        : 1                   Vpn Id            : 0
Service Type      : VPLS                
---snip---

Admin State       : Up                  Oper State        : Up
MTU               : 1514 
SAP Count         : 0                   SDP Bind Count    : 3
---snip---

-------------------------------------------------------------------------------
Service Access & Destination Points
-------------------------------------------------------------------------------
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sdp:32765:4294967293 SB(192.0.2.5)       BgpAd        0       8978    Up   Up
sdp:32766:4294967294 SB(192.0.2.4)       BgpAd        0       8978    Up   Up
sdp:32767:4294967295 SB(192.0.2.3)       BgpAd        0       8978    Up   Down
===============================================================================

[/]
A:admin@PE-2# show service id 1 sdp 32767 detail | match 'Flag' post-lines 1
Flags              : PWPeerFaultStatusBits
                     EvpnRouteConflict

[/]
A:admin@PE-2# show service id 1 vxlan destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
Instance    VTEP Address                            Egress VNI  EvpnStatic Num
 Mcast       Oper State                              L2 PBR     SupBcasDom MACs
-------------------------------------------------------------------------------
1           192.0.2.1                               1           evpn       1
 BUM         Up                                      No          No
1           192.0.2.3                               1           evpn       1
 BUM         Up                                      No          No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 2
-------------------------------------------------------------------------------
===============================================================================
---snip---

At the non-DF, PE-3, all the spoke-SDPs will be down due to BGP-MH, but for the SDP 32767 toward PE-2, an additional reason is an EVPN route conflict:

[/]
A:admin@PE-3# show service id 1 base 

===============================================================================
Service Basic Information
===============================================================================
Service Id        : 1                   Vpn Id            : 0
Service Type      : VPLS                
---snip---

Admin State       : Up                  Oper State        : Up
MTU               : 1514
SAP Count         : 1                   SDP Bind Count    : 3
---snip---

-------------------------------------------------------------------------------
Service Access & Destination Points
-------------------------------------------------------------------------------
Identifier                               Type         AdmMTU  OprMTU  Adm  Opr
-------------------------------------------------------------------------------
sap:1/2/1:1                              q-tag        1518    1518    Up   Up
sdp:32765:4294967293 SB(192.0.2.5)       BgpAd        0       8978    Up   Down
sdp:32766:4294967294 SB(192.0.2.4)       BgpAd        0       8978    Up   Down
sdp:32767:4294967295 SB(192.0.2.2)       BgpAd        0       8978    Up   Down
===============================================================================

[/]
A:admin@PE-3# show service id 1 sdp 32767 detail | match 'Flag' post-lines 2
Flags              : StandbyForMHProtocol
                     PWPeerFaultStatusBits
                     EvpnRouteConflict

# on PE-1:
11 2021/02/10 15:48:52.094 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 88
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
                       mac: 00:00:03:03:03:03, IP len: 0, IP: NULL, label1: 1
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:12
        bgp-tunnel-encap:VXLAN
"

[/]
A:admin@PE-1# show service id 1 fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:01:01:01:01 sap:1/2/1:1             L/120    02/10/21 15:47:35
1          00:00:03:03:03:03 vxlan-1:                Evpn     02/10/21 15:48:52
                             192.0.2.3:1
1          00:00:06:06:06:06 vxlan-1:                Evpn     02/10/21 15:52:31
                             192.0.2.2:1
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================

# on PE-2, PE-3:
configure {
    service {
        vpls "VPLS1" {
            bgp-evpn {
                routes {
                    mac-ip {
                        unknown-mac true
                    }
                }
            }

47 2021/02/10 16:00:36.068 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 88
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.2
        Type: EVPN-MAC Len: 33 RD: 192.0.2.2:1 ESI: ESI-0, tag: 0, mac len: 48 
                       mac: 00:00:00:00:00:00, IP len: 0, IP: NULL, label1: 1
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:12
        bgp-tunnel-encap:VXLAN
"

# on DC GWs PE-2 and PE-3:
configure {
    service {
        vpls "VPLS1" { 
            bgp-evpn {
                routes {
                    mac-ip {
                        advertise false
                        unknown-mac true
                    }
                }
            }

# on PE-2:
configure {
    service {
        oper-group "control-vpls-1" {
        }
        vpls "VPLS1" {
            description "control-VPLS"
            bgp 1 {
                pw-template-binding "PW1" {
                    oper-group "control-vpls-1"
                }
            }
        }
        vpls "VPLS2" {
            admin-state enable
            description "data-VPLS"
            service-id 2
            customer "1"
            vxlan {
                instance 1 {
                    vni 2
                }
            }
            bgp 1 {
                route-distinguisher "192.0.2.2:2"
                vsi-import ["vsi-policy-2"]
                vsi-export ["vsi-policy-2"]
                pw-template-binding "PW1" {
                    monitor-oper-group "control-vpls-1"
                }
            }
            bgp-ad {                                                          
                admin-state enable
                vpls-id "64500:2"
            }
            bgp-evpn {
                routes {
                    mac-ip {
                        unknown-mac true
                    }
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }

# on PE-3:
configure {
    service {
        vpls "VPLS1" {
            proxy-arp {
                admin-state enable
            }

120 2021/02/10 16:12:53.542 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.1
"Peer 1: 192.0.2.1: UPDATE
Peer 1: 192.0.2.1 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 127
    Flag: 0x90 Type: 14 Len: 83 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.1
        Type: EVPN-MAC Len: 37 RD: 192.0.2.1:1 ESI: ESI-0, tag: 0, mac len: 48
                       mac: 00:00:01:01:01:01, IP len: 4, IP: 172.16.0.1, label1: 1 
        Type: EVPN-MAC Len: 33 RD: 192.0.2.1:1 ESI: ESI-0, tag: 0, mac len: 48 
                       mac: 00:00:01:01:01:01, IP len: 0, IP: NULL, label1: 1 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:12
        bgp-tunnel-encap:VXLAN
"

[/]
A:admin@PE-3# show service id 1 proxy-arp detail
-------------------------------------------------------------------------------
Proxy Arp
-------------------------------------------------------------------------------
Admin State       : enabled
Dyn Populate      : disabled
Age Time          : disabled            Send Refresh      : disabled
Table Size        : 250                 Total             : 1
Static Count      : 0                   EVPN Count        : 1
Dynamic Count     : 0                   Duplicate Count   : 0

Dup Detect
-------------------------------------------------------------------------------
Detect Window     : 3 mins              Num Moves         : 5
Hold down         : 9 mins
Anti Spoof MAC    : None

EVPN
-------------------------------------------------------------------------------
Garp Flood        : enabled             Req Flood         : enabled
Static Black Hole : disabled
EVPN Route Tag    : 0
-------------------------------------------------------------------------------

===============================================================================
VPLS Proxy Arp Entries
===============================================================================
IP Address          Mac Address         Type      Status    Last Update
-------------------------------------------------------------------------------
172.16.0.1          00:00:01:01:01:01   evpn      active    02/10/2021 16:12:54
-------------------------------------------------------------------------------
Number of entries : 1
===============================================================================

MAC mobility, duplication and protection are fully supported as specified in RFC 7432. EVPN MAC mobility illustrates the concept of mobility (Virtual Machine VM-1 moves from PE-1 to PE-3).

MAC mobility is handled in EVPN by the use of sequence numbers in the MAC routes. When 00:00:01:01:01:01 moves from PE-1 to PE-3, SR OS will gracefully handle it in this way:

• 00:00:01:01:01:01 moves to PE-3 SAP 1/2/1:1

• PE-3 advertises 00:00:01:01:01:01 using a higher sequence number (the first time a MAC is advertised, EVPN uses sequence number 0).

• PE-2 at this point has two valid MAC routes for 00:00:01:01:01:01. It picks up the one coming from PE-3 because the sequence number is higher.

• PE-1 receives the MAC route, and because the sequence number is higher than the one for its own route, it updates the FDB and withdraws its own MAC route.

However, if MAC 00:00:01:01:01:01 is constantly learned on the PE-1 and PE-3 SAPs, the preceding process causes an endless exchange of MAC route advertisements and withdraws that has a negative impact on all the PEs in the EVPN network. This issue is known as MAC duplication and is originated by a loop at the access or a duplicated MAC address in two hosts of the same service. SR OS solves this issue through the use of the MAC duplication detection feature. MAC duplication is always enabled with the following default settings:

[ex:/configure service vpls "VPLS1" bgp-evpn]
A:admin@PE-3# info detail | match 'mac-duplication' post-lines 7
    mac-duplication {
        retry 9
        blackhole false
        detect {
            num-moves 5
            window 3
        }
    }

Where:

num-moves

Identifies the number of MAC moves in a VPLS service. The counter is incremented when a MAC is locally relearned in the FDB or flushed from the FDB because of the reception of a better remote EVPN route for that MAC. When the threshold is reached for a MAC address, this MAC address is put in hold-down state (this ‛hold-down’ state is described below). Range: <3..10>. Default value: 5.

window

Identifies the timer within which a MAC is considered duplicate if it reaches the configured num-moves. Range: <1..15> minutes. Default value: 3 minutes.

retry

The timer after which the MAC in hold-down state is automatically flushed and the mac-duplication process starts again. This value is expected to be equal to two times or more than the window. If no retry is configured, this implies that, once MAC duplication is detected, MAC updates for that MAC will be held down until the user intervenes or a network event (that flushes the MAC) occurs. Range: <2..60> minutes. Default value: 9 minutes.

blackhole

If enabled and a duplicate MAC address is detected, the router adds the MAC address to the duplicate MAC list and it programs the MAC in the FDB as a protected MAC associated with a black-hole (with type EvpnD:P and source ID ‟black-hole”)

When a MAC address is considered a duplicate or in the hold-down state, no further BGP advertisements are issued for this MAC and an alarm is triggered (by the first MAC address in hold-down state). The following CLI output shows how PE-3 detects that MAC 00:00:01:01:01:01 is a duplicate (after reaching the num-moves in window) and the corresponding alarm.

# on PE-3:
144 2021/02/10 16:16:44.974 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.1
"Peer 1: 192.0.2.1: UPDATE
Peer 1: 192.0.2.1 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 96
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.3
        Type: EVPN-MAC Len: 33 RD: 192.0.2.3:1 ESI: ESI-0, tag: 0, mac len: 48
                       mac: 00:00:01:01:01:01, IP len: 0, IP: NULL, label1: 1
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x80 Type: 4 Len: 4 MED: 0
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64500:12
        bgp-tunnel-encap:VXLAN
        mac-mobility:Seq:5
"

Log 99 on PE-3 shows the following message when EVPN has detected a duplicate MAC address in VPLS 1:

# on PE-3:
154 2021/02/10 16:18:58.902 UTC MINOR: SVCMGR #2331 Base 
"VPLS Service 1 has MAC(s) detected as duplicates by EVPN mac-duplication detection."

The show service id bgp-evpn command shows the MAC duplication settings and the list of duplicate MAC addresses on hold-down.

[/]
A:admin@PE-3# show service id 1 bgp-evpn

===============================================================================
BGP EVPN Table
===============================================================================
MAC Advertisement  : Enabled            Unknown MAC Route  : Enabled
CFM MAC Advertise  : Disabled
Creation Origin    : manual
MAC Dup Detn Moves : 3                  MAC Dup Detn Window: 1
MAC Dup Detn Retry : 2                  Number of Dup MACs : 1
MAC Dup Detn BH    : Disabled
IP Route Advert    : Disabled
Sel Mcast Advert   : Disabled

EVI                : n/a
Ing Rep Inc McastAd: Enabled
Accept IVPLS Flush : Disabled

-------------------------------------------------------------------------------
Detected Duplicate MAC Addresses             Time Detected
-------------------------------------------------------------------------------
00:00:01:01:01:01                            02/10/2021 16:18:58
-------------------------------------------------------------------------------
===============================================================================
---snip---

SR OS stops sending and processing any BGP MAC advertisement routes for that MAC address until:

• The MAC is flushed because of a local event (SAP/SDP-binding associated with the MAC fails) or the reception of a remote withdraw for the MAC (because of a MAC flush at the remote 7x50).

• The retry <in_minutes> timer expires, which flushes the MAC and restart the process.

When the last duplicate MAC address is removed from the duplicate list, log 99 on PE-3 will show the following message:

155 2021/02/10 16:21:58.885 UTC MINOR: SVCMGR #2332 Base 
"VPLS Service 1 no longer has MAC(s) detected as duplicates by EVPN mac-duplication
detection."

EVPN also provides a mechanism to protect specific MAC addresses that do not move for which connectivity must be guaranteed. These addresses must be protected in case there is an attempt to dynamically learn them in a different place in the EVPN-VXLAN VPLS service (on the same or different PE).

The protected MAC addresses are configured in SR OS as conditional static MAC addresses. A conditional static MAC address defined in an EVPN-VXLAN VPLS service is advertised by BGP-EVPN as a static address. An example of the configuration of a conditional static MAC address is as follows:

# on PE-1:
configure {
    service {
        vpls "VPLS1"
            fdb {
                static-mac {
                    mac 00:00:05:05:05:05 {
                        sap 1/2/1:1
                        monitor forward-status
                    }
                }
            }
        }

The protected MAC addresses advertised in EVPN are shown in the receiving BGP RIB as Static (MAC mobility extended community with Sequence 0 and sticky bit set) and EvpnS:P (Evpn Static: Protected) in the FDB. The advertising PE shows the protected MAC as CStatic:P (Conditional Static: Protected) in the FDB:

On the advertising PE:

[/]
A:admin@PE-1# show service id 1 fdb mac 00:00:05:05:05:05

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:05:05:05:05 sap:1/2/1:1             CStatic: 02/10/21 16:31:03
                                                     P
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================

On the receiving PE:

[/]
A:admin@PE-3# show service id 1 fdb mac 00:00:05:05:05:05

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age
-------------------------------------------------------------------------------
1          00:00:05:05:05:05 vxlan-1:                EvpnS:P  02/10/21 16:31:03
                             192.0.2.1:1
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================

[/]
A:admin@PE-3# show router bgp routes evpn mac hunt mac-address 00:00:05:05:05:05
===============================================================================
 BGP Router ID:192.0.2.3       AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Network        : n/a
Nexthop        : 192.0.2.1
From           : 192.0.2.1
Res. Nexthop   : 192.168.13.1
Local Pref.    : 100                    Interface Name : int-PE-3-PE-1
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : 0
AIGP Metric    : None                   IGP Cost       : 10
Connector      : None
Community      : target:64500:12 bgp-tunnel-encap:VXLAN
                 mac-mobility:Seq:0/Static
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 192.0.2.1
Flags          : Used Valid Best IGP
Route Source   : Internal
AS-Path        : No As-Path
EVPN type      : MAC
ESI            : ESI-0
Tag            : 0
IP Address     : n/a
Route Dist.    : 192.0.2.1:1
Mac Address    : 00:00:05:05:05:05
MPLS Label1    : VNI 1                  MPLS Label2    : n/a
Route Tag      : 0
Neighbor-AS    : n/a
Orig Validation: N/A
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default
Last Modified  : 00h02m32s

-------------------------------------------------------------------------------
RIB Out Entries
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The following procedures are supported to protect the configured static MAC addresses:

• All the SAP/SDP-bindings are internally configured as MAC protect restrict-protected-src as soon as BGP-EVPN is enabled in the VPLS service.

• Local static MAC addresses or remote EVPN static MAC addresses are considered as protected.

• If a frame with a source MAC address matching one of the protected MAC addresses is received on a different SAP/SDP-binding than the owner of the protected MAC address, the frame is discarded and an alarm triggered. This MAC protection is not performed for frames received on VXLAN bindings.

• The same throttled alarm mechanism used in MAC protect for restrict-protected-src with discard-frame is used here: the offending frames are captured to a list to be polled by the CPM every ~10min.

In this example, PE-3 has 00:00:05:05:05:05 in its FDB as EvpnS. If SAP 1/2/1:1 on PE-3 receives a frame with source MAC address 00:00:05:05:05:05, the frame is discarded and an alarm triggered. The following is logged in log 99 on PE-3:

164 2021/02/10 16:44:03.736 UTC MINOR: SVCMGR #2208 Base Slot 1
"Protected MAC 00:00:05:05:05:05 received on SAP 1/2/1:1 in service 1. "

In addition to the previously mentioned show service id vxlan destinations, show service id bgp-evpn and show service id fdb detail commands, the following commands provide valuable information when troubleshooting an EVPN-VXLAN VPLS service.

The show router bgp routes evpn command supports filtering by route type as well as many other route fields.

[/]
A:admin@PE-1# show router bgp routes evpn ?

 auto-disc             - Display BGP EVPN Auto-Disc Routes
 eth-seg               - Display BGP EVPN Eth-Seg Routes
 incl-mcast            - Display BGP EVPN Inclusive-Mcast Routes
 ip-prefix             - Display BGP EVPN IPv4-Prefix Routes
 ipv6-prefix           - Display BGP EVPN IPv6-Prefix Routes
 mac                   - Display BGP EVPN Mac Routes
 mcast-join-synch      - Display BGP EVPN Mcast Join Sync Routes
 mcast-leave-synch     - Display BGP EVPN Mcast Leave Sync Routes
 smet                  - Display BGP EVPN Smet Routes
 spmsi-ad              - Display BGP EVPN Spmsi AD Routes

[/]
A:admin@PE-1# show router bgp routes evpn mac ?

 mac [<keyword>] [rd <string>] [next-hop <string>] [mac-address <string>] [community
 <string>] [tag <string>]
  [aspath-regex <string>]

 [hunt-detail] <keyword>
 <keyword>  - (hunt|detail)

    keywords

 [hunt-detail]         - keywords
 aspath-regex          - string '<1..80 characters>'
 community             - <as-number1:comm-val1>|<ext-comm>|, <well-known-comm>, 
                         ext-comm -  <type>:{<ip-address:comm-val1>|, 
                         <as-number1:comm-val2>|,<as-number2:comm-val1>},
                         as-number1 - [0..65535], comm-val1 - [0..65535], 
                         type  - target|origin, ip-address - a.b.c.d, 
                         comm-val2 - [0..4294967295], as-number2  - [0..4294967295],
                         well-known-comm - null|no-export|no-export-subconfed|,
                         no-advertise
 mac-address           - string '<0..255 characters>'
 next-hop              - Attribute next-hop for mac
 rd                    - {<ip-addr:comm-val>|, <2byte-asnumber:ext-comm-val>|, 
                                               <4byte-asnumber:comm-val>}
 tag                   - Attribute tag for mac

[/]
A:admin@PE-3# show router bgp routes evpn mac tag 0 
===============================================================================
 BGP Router ID:192.0.2.3        AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag  Route Dist.         MacAddr           ESI
      Tag                 Mac Mobility      Label1
                          Ip Address        
                          NextHop           
-------------------------------------------------------------------------------
u*>i  192.0.2.1:1         00:00:05:05:05:05 ESI-0
      0                   Static            VNI 1
                          n/a
                          192.0.2.1

u*>i  192.0.2.1:1         04:09:ff:00:03:3a ESI-0
      0                   Static            VNI 1
                          n/a
                          192.0.2.1

u*>i  192.0.2.2:1         00:00:00:00:00:00 ESI-0
      0                   Seq:0             VNI 1
                          n/a
                          192.0.2.2

-------------------------------------------------------------------------------
Routes : 3
===============================================================================

The tools dump service id vxlan displays the number of times a service could not add a VXLAN binding or <VTEP, Egress VNI> because of the following limits:

• The per system VTEP limit has been reached

• The per system (egress VTEP, egress VNI) limit has been reached

• The per service (egress VTEP, egress VNI) limit has been reached

• The per system Bind limit: Total bind limit or VXLAN bind limit has been reached.

[/]
A:admin@PE-1# tools dump service id 1 vxlan

VTEP, Egress VNI Failure statistics at 02/10/2021 17:03:07:

statistics last cleared at 02/10/2021 10:43:55:

Failures: None

[/]
A:admin@PE-1# tools dump service id 1 evpn usage

Evpn Tunnel Interface IP Next Hop: N/A

Tools dump service evpn usage displays the consumed resources in the system, as follows:

[/]
A:admin@PE-1# tools dump service evpn usage

vxlan-evpn-mpls usage statistics at 02/10/2021 17:03:07:

MPLS-TEP                                        :             0
VXLAN-TEP                                       :             2
Total-TEP                                       :      2/ 16383

Mpls Dests (TEP, Egress Label + ES + ES-BMAC)   :             0
Mpls Etree Leaf Dests                           :             0
Vxlan Dests (TEP, Egress VNI + ES)              :             2
Total-Dest                                      :      2/196607

Sdp Bind +  Evpn Dests                          :      2/245759
ES L2/L3 PBR                                    :      0/ 32767
Evpn Etree Remote BUM Leaf Labels               :             0