EVPN-VXLAN VPWS

This chapter provides information about EVPN-VXLAN VPWS.

Topics in this chapter include:

Applicability

This chapter was initially written for SR OS Release 16.0.R7, but the MD-CLI in the current edition is based on SR OS Release 21.5.R2.

Overview

Some service providers use VXLAN as a next-generation access technology between Multi-Service Access Node (MSAN) PE and core PE routers. VXLAN allows any IP router in the aggregation core and provides a simple alternative to MPLS. Static VXLAN bindings can be used when the MSAN PEs do not support any control plane. However, EVPN offers a control plane protocol for the VXLAN bindings for faster convergence and fault propagation. In this chapter, the focus is on EVPN-VPWS, which provides a lighter control plane compared to full-blown EVPN when point-to-point services need to be extended to the Data Center (DC).

EVPN-VXLAN VPWS is similar to EVPN-MPLS VPWS, including support of Equal Cost Multi-Path (ECMP), and EVPN All-Active (AA) and Single-Active (SA) Multi-Homing (MH). The configuration resembles the EVPN-MPLS Epipe configuration, as described in the EVPN for MPLS Tunnels in Epipe Services (EVPN-VPWS) chapter. As an example, the following configures EVPN-VXLAN Epipe 4 with SA MH.

# on PE-4:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "ES45" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:04
                        multi-homing-mode single-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            sdp 460 {
                            }
                        }
                    }
                }
            }
        }
        epipe "Epipe-4" {
            admin-state enable
            service-id 4
            customer "1"
            bgp 1 {
            }
            spoke-sdp 460:4 {
            }
            vxlan {
                instance 1 {
                    vni 4
                }
            }
            bgp-evpn {
                evi 4
                local-attachment-circuit "AC-45" {
                    eth-tag 145
                }
                remote-attachment-circuit "AC-23" {
                    eth-tag 123
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                    ecmp 2
                }
            }
        }
        sdp 460 {
            admin-state enable
            description "GRE SDP for SA MH"
            far-end {
                ip-address 192.0.2.6
            }
        }

The SDP is a GRE SDP, because no MPLS is configured in the network. The VNI is 4, and the local Attachment Circuit (AC) name is "AC-45" with Ethernet tag 145, whereas the remote AC name is "AC-23" with Ethernet tag 123. An ES can contain up to four nodes. Each of these nodes will have the same local Ethernet tag.

On Epipe services, the BGP instance is 1 and the VXLAN instance is 1. ECMP is configured with a value of 2, so the traffic flows can be sprayed over two paths with equal cost (a value greater than 2 can be configured if aliasing to more than two nodes is needed). By default, send-tunnel-encap is enabled, which determines whether the RFC 5512 encapsulation extended community is sent with VXLAN value (if enabled) or not sent.

EVPN-VPWS uses BGP-EVPN route type 1 (autodiscovery (AD) per-EVI routes and AD per-ES routes) and route type 4 (Ethernet Segment (ES) routes); it does not use route types 2 (MAC/IP routes), 3 (Inclusive Multicast routes), or 5 (IP Prefix routes). BGP-EVPN AD per-EVI route shows the fields in a BGP-EVPN AD per-EVI route.

Figure 1. BGP-EVPN AD per-EVI route

The Route Distinguisher (RD) is encoded as specified in RFC 7432; in this example, the system IP address is followed by the service ID, such as 192.0.2.2:1 for Epipe 1 on PE-2. The MPLS label field is encoded as the VXLAN Network Identifier (VNI) and the Ethernet tag field defines the local Attachment Circuit (AC) ID. The ES ID (ESI) is the 10 bytes configured ESI for MH and equals zero for single-homed services.

The EVPN L2 attributes extended community has type 0x06 (EVPN) and subtype 0x04 (EVPN L2 attributes). The flags are defined as follows:

  • Flag C (control word) is set if control word is configured in the service. For EVPN-MPLS VPWS, the control word can be configured in the bgp-evpn>mpls context, but for EVPN-VXLAN VPWS, the control word cannot be configured in the bgp-evpn>vxlan context, so flag C is always zero (C=0).

  • Flag P (primary) is set in MH scenarios: all nodes in an AA MH ES send P=1, but in an SA MH ES, only the Designated Forwarder (DF) sends P=1, while the NDFs send P=0. In single-homed scenarios, all nodes send P=0.

  • Flag B (backup) is set in SA MH scenarios: the NDF that will take the primary role after the original primary node has failed is the backup, so it sends B=1. All other NDFs have B=0. In AA MH scenarios, all nodes send B=0. Also, in single-homed scenarios, all nodes except for the backup DF send B=0.

If the received L2 MTU does not match the configured service MTU, the EVPN binding is not set up. However, if the received L2 MTU is zero, the MTU is ignored.

AD per-EVI routes are responsible for aliasing. The following BGP update shows an AD per-EVI route received from DF 192.0.2.4 (PE-4) in an SA MH ES with ESI 01:00:00:00:00:45:00:00:00:04, Ethernet tag 145 for the local AC on PE-4, and MPLS label 4 for Epipe 4. The primary flag is set: P=1.

102 2021/06/30 17:30:58.043 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 81
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.4
        Type: EVPN-AD Len: 25 RD: 192.0.2.4:4 ESI: 01:00:00:00:00:45:00:00:00:04,
                      tag: 145 Label: 4
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64500:4
        l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
        bgp-tunnel-encap:VXLAN
"

As per RFC 8214, in an AD per-ES route, the Ethernet tag is MAX-ET (all bits are set), the MPLS label is zero, and the BGP extended community contains the single-active flag (1 for SA and 0 for AA) and ESI label. BGP-EVPN AD per-ES route shows the fields in a BGP-EVPN AD per-ES route.

Figure 2. BGP-EVPN AD per-ES route

The following AD per-ES route is received by PE-2 from PE-4, which is in an SA MH ES with ESI 01:00:00:00:00:45:00:00:00:04.

62 2021/06/30 17:30:25.151 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 73
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.4
        Type: EVPN-AD Len: 25 RD: 192.0.2.4:4 ESI: 01:00:00:00:00:45:00:00:00:04,
                      tag: MAX-ET Label: 0
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:4
        esi-label:524284/Single-Active
"

BGP-EVPN ES route shows a BGP-EVPN route type 4 (ES route) that is used for MH ES discovery and DF election.

Figure 3. BGP-EVPN ES route

The RD is taken from the system level RD; by default, the RD is derived as system-IP:0, such as 192.0.2.4:0 for PE-4. The ESI contains the 10-byte identifier as configured in the ES. The ES import route target community has type 0x06 (EVPN) and subtype 0x02 (ES import route target), and is derived from the MAC address portion of the ESI. This extended community is treated as a route target, such as: target:00:00:00:00:45:00. Only the PEs attached to the ES will import the ES route.

The following BGP update shows a BGP-EVPN ES route sent by PE-4. The RD is defined as 192.0.2.4:0, the ESI is 01:00:00:00:00:45:00:00:00:04, and the originating IP address is 192.0.2.4 for PE-4. The ES import route target is target:00:00:00:00:45:00.

45 2021/06/30 17:30:55.107 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 71
    Flag: 0x90 Type: 14 Len: 34 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.4
        Type: EVPN-ETH-SEG Len: 23 RD: 192.0.2.4:0 ESI: 01:00:00:00:00:45:00:00:00:04,
                          IP-Len: 4 Orig-IP-Addr: 192.0.2.4
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        df-election::DF-Type:Auto/DP:0/DF-Preference:0/AC:1
        target:00:00:00:00:45:00
"

By default, the system IP addresses are used for the VXLAN tunnel termination. However, it is possible to use non-system IPv4 or IPv6 termination for EVPN-VXLAN VPWS, both for single-homed and multi-homed services. In that case, Forwarding Path Extension (FPE) needs to be defined with VXLAN termination, as described in chapter Static VXLAN Termination in Epipe Services.

The following shows the configuration of the single-homed Epipe 2 using non-system IPv4 source VXLAN Tunnel Endpoint (VTEP) 10.0.3.1 on PE-3. Likewise, it is possible to use a non-system IPv6 source VTEP, such as vxlan>source-vtep 2001::3:1. Unlike the source VTEP, the egress VTEP cannot be configured when BGP-EVPN is enabled. The egress VTEP is dynamically learned via BGP instead.

# on PE-3:
configure {
    service {
        epipe "Epipe-2" {
            admin-state enable
            service-id 2
            customer "1"
            bgp 1 {
            }
            sap 1/1/1:2 {
            }
            vxlan {
                source-vtep 10.0.3.1
                instance 1 {
                    vni 2
                }
            }
            bgp-evpn {
                evi 2
                local-attachment-circuit "AC-3" {
                    eth-tag 103
                }
                remote-attachment-circuit "AC-5" {
                    eth-tag 105
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }

Configuration

The following use cases are included in the configuration section:

  • Single-homed EVPN-VXLAN Epipe using IPv4 system addresses

  • Single-homed EVPN-VXLAN Epipe using non-system IPv4 addresses

  • Single-homed EVPN-VXLAN Epipe using non-system IPv6 addresses

  • AA and SA multi-homed EVPN-VXLAN Epipe using IPv4 system addresses

  • AA and SA multi-homed EVPN-VXLAN Epipe using non-system IPv4 addresses

  • AA and SA multi-homed EVPN-VXLAN Epipe using non-system IPv6 addresses

Example topology shows the example topology with six PEs. EVPN-VXLAN Epipe services will be configured on the core PEs PE-2, PE-3, PE-4, and PE-5. On the access nodes PE-1 and PE-6, ordinary Epipe services will be configured, without EVPN-VXLAN. The CEs are emulated by VPRN services configured on PE-1 or PE-6.

Figure 4. Example topology

The initial configuration includes:

  • Cards, MDAs, ports

  • Router interfaces

  • IS-IS on all router interfaces: level 2 between the core PEs and level 1 in the access networks

No MPLS protocol is configured.

BGP is configured on the core PEs for the EVPN address family with RR PE-2. The BGP configuration on RR PE-2 is as follows:

# on PE-2:
configure {
    router "Base" {
        autonomous-system 64500
        bgp {
            vpn-apply-export true
            vpn-apply-import true
            rapid-update {
                evpn true
            }
            group "iBGP" {
                type internal
                split-horizon true
                family {
                    evpn true
                }
                cluster {
                    cluster-id 192.0.2.2
                }
            }
            neighbor "192.0.2.3" {
                group "iBGP"
            }
            neighbor "192.0.2.4" {
                group "iBGP"
            }
            neighbor "192.0.2.5" {
                group "iBGP"
            }

Single-homed EVPN-VXLAN Epipe using system IPv4 addresses

Single-homed EVPN-VXLAN Epipe 1 using system IP addresses shows the routers PE-1, PE-2, PE-4, and PE-6 configured with Epipe 1. VXLAN-EVPN is only configured on the core PEs PE-2 and PE-4.

Figure 5. Single-homed EVPN-VXLAN Epipe 1 using system IP addresses

Configuration of Epipe 1

On PE-1, Epipe 1 is configured without EVPN-VXLAN, as follows.

# on PE-1:
configure {
    service {
        epipe "Epipe 1" {
            admin-state enable
            service-id 1
            customer "1"
            sap 1/1/1:1 {
            }
            sap 1/2/1:1 {
            }
        }

On PE-2, Epipe 1 is configured with EVPN-VXLAN. The local AC "AC-2" has Ethernet tag 102 and the remote AC is "AC-4" with Ethernet tag 104, as follows:

# on PE-2:
configure {
    service {
        epipe "Epipe-1" {
            admin-state enable
            service-id 1
            customer "1"
            bgp 1 {
            }
            sap 1/1/2:1 {
            }
            vxlan {
                instance 1 {
                    vni 1
                }
            }
            bgp-evpn {
                evi 1
                local-attachment-circuit "AC-2" {
                    eth-tag 102
                }
                remote-attachment-circuit "AC-4" {
                    eth-tag 104
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }

The Epipe configuration on PE-4 is similar, but the local AC and remote AC are swapped, as follows. Instead of a SAP, a spoke-SDP is configured toward PE-6. The SDP itself is GRE-based.

# on PE-4:
configure {
    service {
        epipe "Epipe-1" {
            admin-state enable
            service-id 1
            customer "1"
            bgp 1 {
            }
            spoke-sdp 46:1 {
            }
            vxlan {
                instance 1 {
                    vni 1
                }
            }
            bgp-evpn {
                evi 1
                local-attachment-circuit "AC-4" {
                    eth-tag 104
                }
                remote-attachment-circuit "AC-2" {
                    eth-tag 102
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }
        sdp 46 {
            admin-state enable
            description "GRE SDP for single-homing"
            far-end {
                ip-address 192.0.2.6
            }
        } 

On PE-6, Epipe 1 is an ordinary Epipe with spoke-SDP 64:1 toward PE-4 and SAP 1/2/1:1 toward a CE, as follows:

# on PE-6:
configure {
    service {
        epipe "Epipe 1" {
            admin-state enable
            service-id 1
            customer "1"
            spoke-sdp 64:1 {
            }
            sap 1/2/1:1 {
            }
        }
        sdp 64 {
            admin-state enable
            description "GRE SDP for single-homing"
            far-end {
                ip-address 192.0.2.4
            }
        }

Verification

VPRN 11 on PE-1 and PE-6 simulates the CEs CE-11 and CE-61. The connectivity between the CEs can be verified as follows:

[/]
A:admin@PE-1# ping 172.16.11.61 router-instance "VPRN 11" interval 0.1 
                                                              output-format summary
PING 172.16.11.61 56 data bytes
!!!!!
---- 172.16.11.61 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 4.42ms, avg = 5.89ms, max = 10.9ms, stddev = 2.49ms
[/]
A:admin@PE-1# ping 2001:db8::11:61 router-instance "VPRN 11" interval 0.1
                                                              output-format summary
PING 2001:db8::11:61 56 data bytes
!!!!!
---- 2001:db8::11:61 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 4.14ms, avg = 5.63ms, max = 11.2ms, stddev = 2.79ms

On PE-2, the VXLAN destination for Epipe 1 is the system address of PE-4: 192.0.2.4, as follows. There are no VXLAN ES destinations for Epipe 1, because the service is single-homed.

[/]
A:admin@PE-2# show service id 1 vxlan destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI          Oper      Vxlan
                                                            State     Type
-------------------------------------------------------------------------------
192.0.2.4                               1                   Up        evpn
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 1
-------------------------------------------------------------------------------
===============================================================================

===============================================================================
BGP EVPN VXLAN ES Dest
===============================================================================
I Eth Seg Id                    TEP Address       VNI       Last Changed
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

The following BGP-EVPN information for Epipe 1 on PE-2 includes the EVI and the AC names and Ethernet tags. For Epipes, the BGP instance ID and VXLAN instance ID always equal 1.

[/]
A:admin@PE-2# show service id 1 bgp-evpn

===============================================================================
BGP EVPN Table
===============================================================================
EVI                : 1                  Creation Origin    : manual
Local AC Name      : AC-2
Eth Tag            : 102
Endpoint           : (Not Specified)
Ingress Label      : 0
Remote AC Name     : AC-4
Eth Tag            : 104
Endpoint           : (Not Specified)

===============================================================================
BGP EVPN VXLAN Information
===============================================================================
Admin Status       : Enabled            Bgp Instance       : 1
Vxlan Instance     : 1
Max Ecmp Routes    : 1
Default Route Tag  : none
Send EVPN Encap    : Enabled
===============================================================================

PE-2 has received the following BGP-EVPN AD per-EVI route with RD 192.0.2.4:1 and Ethernet tag 104 from PE-4. Epipe 1 is single-homed, so ESI=0 and there is no primary or backup node (P=B=0). Also, no control word is used, so C=0.

5 2021/06/30 17:14:44.605 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 81
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.4
        Type: EVPN-AD Len: 25 RD: 192.0.2.4:1 ESI: ESI-0, tag: 104 Label: 1
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64500:1
        l2-attribute:MTU: 1514 C: 0 P: 0 B: 0
        bgp-tunnel-encap:VXLAN
"

The following shows the received BGP-EVPN AD per-EVI routes with RD 192.0.2.4:1 on PE-2.

[/]
A:admin@PE-2# show router bgp routes evpn auto-disc rd 192.0.2.4:1
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  192.0.2.4:1         ESI-0                         192.0.2.4
      104                                               VNI 1

-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Single-homed EVPN-VXLAN Epipe using non-system IPv4 addresses

Single-homed EVPN-VXLAN Epipe 2 using non-system IP addresses shows the single-homed service Epipe 2 configured on PE-1, PE-3, PE-5, and PE-6. On PE-3, a loopback interface is created in the base router with IPv4 address 10.0.3.3/28. Epipe 2 uses VXLAN termination 10.0.3.1 from the same subnet.

Figure 6. Single-homed EVPN-VXLAN Epipe 2 using non-system IP addresses

Configuration of Epipe 2

On PE-1 and PE-6, the configuration of Epipe 2 is similar to the configuration of Epipe 1.

On PE-3, FPE needs to be configured using PXC, as described in chapter Static VXLAN Termination in Epipe Services. The following configuration is included without further explanation about FPE or PXC. The same configuration is required on PE-5.

# on PE-3:
configure {
    fwd-path-ext {
        sdp-id-range {
            start 10000
            end 10127
        }
        fpe 1 {
            path {
                pxc 1
            }
            application {
                vxlan-termination {
                }
            }
        }
    }
    port 1/2/5 {
        admin-state enable
        ethernet {
            mode hybrid
            encap-type dot1q
            dot1x {
                tunneling true
            }
        }
    }
    port pxc-1.a {
        admin-state enable
    }
    port pxc-1.b {
        admin-state enable
    }
    port-xc {
        pxc 1 {
            admin-state enable
            port 1/2/5
        }
    }

On PE-3, the following loopback interface is created and IS-IS is enabled on it. The subnet must allow multiple IP addresses; one other IP address from the subnet will be defined as VXLAN tunnel termination. The IPv6 address is only required in the next use-case, but this configuration will not be repeated in that section.

# on PE-3:
configure {
    router "Base" {
        interface "lo1" {
            loopback
            ipv4 {
                primary {
                    address 10.0.3.3
                    prefix-length 28
                }
            }
            ipv6 {
                address 2001::3:3 {
                    prefix-length 124
                }
            }
        }
        isis 0 {
            interface "lo1" {
                passive true
            }
        }

Up to three VXLAN tunnel terminations can be defined per system. On PE-3, the following two VXLAN tunnel terminations are configured. For Epipe 2, only the first VXLAN tunnel termination is required; the second (IPv6) VXLAN tunnel termination is used in Epipe 3. The VXLAN tunnel termination is used as VXLAN source VTEP in Epipe 2. No egress VTEP can be defined when BGP-EVPN is configured in the service; egress VTEPs are configured in static VXLAN tunnels instead.

# on PE-3:
configure {
    service {
        system {
            vxlan {
                tunnel-termination 10.0.3.1 {
                    fpe-id 1
                }
                tunnel-termination 2001::3:1 {
                    fpe-id 1
                }
            }
        }
        epipe "Epipe-2" {
            admin-state enable
            service-id 2
            customer "1"
            bgp 1 {
            }
            sap 1/1/1:2 {
            }
            vxlan {
                source-vtep 10.0.3.1
                instance 1 {
                    vni 2
                }
            }
            bgp-evpn {
                evi 2
                local-attachment-circuit "AC-3" {
                    eth-tag 103
                }
                remote-attachment-circuit "AC-5" {
                    eth-tag 105
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }

The configuration on PE-5 is similar. The following is the service configuration on PE-5.

# on PE-5:
configure {
    service {
        system {
            vxlan {
                tunnel-termination 10.0.5.1 {
                    fpe-id 1
                }
                tunnel-termination 2001::5:1 {
                    fpe-id 1
                }
            }
        }
        epipe "Epipe-2" {
            admin-state enable
            service-id 2
            customer "1"
            bgp 1 {
            }
            spoke-sdp 56:2 {
            }
            vxlan {
                source-vtep 10.0.5.1
                instance 1 {
                    vni 2
                }
            }
            bgp-evpn {
                evi 2
                local-attachment-circuit "AC-5" {
                    eth-tag 105
                }
                remote-attachment-circuit "AC-3" {
                    eth-tag 103
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }
        sdp 56 {
            admin-state enable
            description "GRE SDP for single-homing"
            far-end {
                ip-address 192.0.2.6
            }
        }

It is possible to use a system IPv4 address as a VXLAN tunnel termination on one of the nodes and a non-system IPv4 address on another, but that is not configured here.

Verification

The connectivity between the CEs that are emulated by VPRN 12 can be verified as follows:

[/]
A:admin@PE-1# ping 172.16.12.62 router-instance "VPRN 12" interval 0.1 
                                                             output-format summary
PING 172.16.12.62 56 data bytes
!!!!!
---- 172.16.12.62 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 4.31ms, avg = 4.67ms, max = 5.13ms, stddev = 0.268ms
[/]
A:admin@PE-1# ping 2001:db8::12:62 router-instance "VPRN 12" interval 0.1 
                                                              output-format summary
PING 2001:db8::12:62 56 data bytes
!!!!!
---- 2001:db8::12:62 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 4.39ms, avg = 4.84ms, max = 5.20ms, stddev = 0.276ms

On PE-3, the VXLAN destination for Epipe 2 is the non-system address 10.0.5.1 on PE-5, as follows:

[/]
A:admin@PE-3# show service id 2 vxlan destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI          Oper      Vxlan
                                                            State     Type
-------------------------------------------------------------------------------
10.0.5.1                                2                   Up        evpn
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 1
-------------------------------------------------------------------------------
===============================================================================

===============================================================================
BGP EVPN VXLAN ES Dest
===============================================================================
I Eth Seg Id                    TEP Address       VNI       Last Changed
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

The following BGP-EVPN information for Epipe 2 on PE-3 includes the EVI, AC names, and Ethernet tags.

[/]
A:admin@PE-3# show service id 2 bgp-evpn

===============================================================================
BGP EVPN Table
===============================================================================
EVI                : 2                  Creation Origin    : manual
Local AC Name      : AC-3
Eth Tag            : 103
Endpoint           : (Not Specified)
Ingress Label      : 0
Remote AC Name     : AC-5
Eth Tag            : 105
Endpoint           : (Not Specified)

===============================================================================
BGP EVPN VXLAN Information
===============================================================================
Admin Status       : Enabled            Bgp Instance       : 1
Vxlan Instance     : 1
Max Ecmp Routes    : 1
Default Route Tag  : none
Send EVPN Encap    : Enabled
===============================================================================

PE-3 received the following BGP-EVPN AD per-EVI route with RD 192.0.2.5:2 from PE-5. The Ethernet tag is 105 and the next-hop is the non-system address 10.0.5.1. ESI=0 for single-homed services.

[/]
A:admin@PE-3# show router bgp routes evpn auto-disc rd 192.0.2.5:2
===============================================================================
 BGP Router ID:192.0.2.3        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  192.0.2.5:2         ESI-0                         10.0.5.1
      105                                               VNI 2

-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Single-homed EVPN-VXLAN Epipe using non-system IPv6 addresses

Single-homed EVPN-VXLAN Epipe 3 using non-system IPv6 addresses shows the example topology for single-homed EVPN-VXLAN Epipe 3 using non-system IPv6 addresses for VXLAN tunnel termination.

Figure 7. Single-homed EVPN-VXLAN Epipe 3 using non-system IPv6 addresses

Configuration of Epipe 3

The following single-homed Epipe 3 using non-system IPv6 addresses for the VXLAN tunnel terminations is configured on PE-3.

# on PE-3:
configure {
    service {
        system {
            vxlan {
                tunnel-termination 10.0.3.1 {
                    fpe-id 1
                }
                tunnel-termination 2001::3:1 {
                    fpe-id 1
                }
            }
        }
        epipe "Epipe-3" {
            admin-state enable
            service-id 3
            customer "1"
            bgp 1 {
            }
            sap 1/1/1:3 {
            }
            vxlan {
                source-vtep 2001::3:1
                instance 1 {
                    vni 3
                }
            }
            bgp-evpn {
                evi 3
                local-attachment-circuit "AC-3_v6" {
                    eth-tag 163
                }
                remote-attachment-circuit "AC-5_v6" {
                    eth-tag 165
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }

The service configuration on PE-5 is similar, as follows:

# on PE-5:
configure {
    service {
        system {
            vxlan {
                tunnel-termination 10.0.5.1 {
                    fpe-id 1
                }
                tunnel-termination 2001::5:1 {
                    fpe-id 1
                }
            }
        }
        epipe "Epipe-3" {
            admin-state enable
            service-id 3
            customer "1"
            bgp 1 {
            }
            spoke-sdp 56:3 {
            }
            vxlan {
                source-vtep 2001::5:1
                instance 1 {
                    vni 3
                }
            }
            bgp-evpn {
                evi 3
                local-attachment-circuit "AC-5_v6" {
                    eth-tag 165
                }
                remote-attachment-circuit "AC-3_v6" {
                    eth-tag 163
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                }
            }
        }
        sdp 56 {
            admin-state enable
            description "GRE SDP for single-homing"
            far-end {
                ip-address 192.0.2.6
            }
        }

Verification

The connectivity between the CEs that are emulated by VPRN 13 is verified as follows:

[/]
A:admin@PE-1# ping 172.16.13.63 router-instance "VPRN 13" interval 0.1
                                                              output-format summary
PING 172.16.13.63 56 data bytes
!!!!!
---- 172.16.13.63 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 4.37ms, avg = 4.68ms, max = 5.41ms, stddev = 0.377ms
[/]
A:admin@PE-1# ping 2001:db8::13:63 router-instance "VPRN 13" interval 0.1 
                                                             output-format summary
PING 2001:db8::13:63 56 data bytes
!!!!!
---- 2001:db8::13:63 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 3.87ms, avg = 4.54ms, max = 5.88ms, stddev = 0.702ms

On PE-3, the VXLAN destination for Epipe 3 is the non-system IPv6 address 2001::5:1 on PE-5, as follows:

[/]
A:admin@PE-3# show service id 3 vxlan destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI          Oper      Vxlan
                                                            State     Type
-------------------------------------------------------------------------------
2001::5:1                               3                   Up        evpn
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 1
-------------------------------------------------------------------------------
===============================================================================

===============================================================================
BGP EVPN VXLAN ES Dest
===============================================================================
I Eth Seg Id                    TEP Address       VNI       Last Changed
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

The following BGP-EVPN information for Epipe 3 on PE-3 includes the EVI and the AC names and Ethernet tags.

[/]
A:admin@PE-3# show service id 3 bgp-evpn

===============================================================================
BGP EVPN Table
===============================================================================
EVI                : 3                  Creation Origin    : manual
Local AC Name      : AC-3_v6
Eth Tag            : 163
Endpoint           : (Not Specified)
Ingress Label      : 0
Remote AC Name     : AC-5_v6
Eth Tag            : 165
Endpoint           : (Not Specified)

===============================================================================
BGP EVPN VXLAN Information
===============================================================================
Admin Status       : Enabled            Bgp Instance       : 1
Vxlan Instance     : 1
Max Ecmp Routes    : 1
Default Route Tag  : none
Send EVPN Encap    : Enabled
===============================================================================

PE-3 received the following BGP-EVPN AD per-EVI route with RD 192.0.2.5:3 and next-hop 2001::5:1.

[/]
A:admin@PE-3# show router bgp routes evpn auto-disc rd 192.0.2.5:3
===============================================================================
 BGP Router ID:192.0.2.3        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  192.0.2.5:3         ESI-0                         2001::5:1
      165                                               VNI 3
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

AA and SA multi-homed EVPN-VXLAN Epipe using system IPv4 addresses

EVPN-VXLAN Epipe 4 with AA MH and SA MH using system IPv4 addresses shows the example topology for EVPN-VXLAN Epipe 4 with AA MH ES "vES23_4" between PE-2 and PE-3 and SA MH ES "ES45" between PE-4 and PE-5.

Figure 8. EVPN-VXLAN Epipe 4 with AA MH and SA MH using system IPv4 addresses

Configuration of Epipe 4

On PE-1, Epipe 4 is configured as follows:

# on PE-1:
configure {
    service {
        epipe "Epipe-4" {
            admin-state enable
            service-id 4
            customer "1"
            sap 1/2/1:4 {
            }
            sap lag-1:4 {
            }
        }

On PE-2 and PE-3, the AA MH ES "vES23_4" is configured as a virtual ES for LAG 1 and dot1q-tag 4, so it only affects Epipe 4.

# on PE-2, PE-3:
configure {
    service {
        system {
            bgp {
                evpn {
                    ethernet-segment "vES23_4" {
                        admin-state enable
                        type virtual
                        esi 01:00:00:00:00:23:00:00:00:04
                        multi-homing-mode all-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            lag "lag-1" {
                                virtual-ranges {
                                    dot1q {
                                        q-tag 4 {
                                            end 4
                                        }
                                    }
                                }
                            }
                        }
                    }

On PE-2 and PE-3, Epipe 4 is configured as follows. The system IPv4 address is used as VXLAN termination, the local AC Ethernet tag is 123, and the remote AC Ethernet tag is 145.

# on PE-2, PE-3:
configure {
    service {
        epipe "Epipe-4" {
            admin-state enable
            service-id 4
            customer "1"
            bgp 1 {
            }
            sap lag-1:4 {
            }
            vxlan {
                instance 1 {
                    vni 4
                }
            }
            bgp-evpn {
                evi 4
                local-attachment-circuit "AC-23" {
                    eth-tag 123
                }
                remote-attachment-circuit "AC-45" {
                    eth-tag 145
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                    ecmp 2
                }
            }
        }

On PE-4 and PE-5, the SA MH ES "ES45" is configured with a GRE SDP toward PE-6: SDP 460 on PE-4 and SDP 560 on PE-6. The following is the configuration of "ES45" on PE-4:

# on PE-4:
configure {
    service {
        sdp 460 {
            admin-state enable
            description "GRE SDP for SA MH"
            far-end {
                ip-address 192.0.2.6
            }
        }
        system {
            bgp {
                evpn {
                    ethernet-segment "ES45" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:04
                        multi-homing-mode single-active
                        df-election {
                            es-activation-timer 3
                        }
                        association {
                            sdp 460 {
                            }
                        }
                    }
                }
            }
        }

On PE-4, Epipe 4 is configured as follows. The configuration on PE-5 is similar, but with spoke-SDP 560:4 instead.

# on PE-4:
configure {
    service {
        epipe "Epipe-4" {
            admin-state enable
            service-id 4
            customer "1"
            bgp 1 {
            }
            spoke-sdp 460:4 {
            }
            vxlan {
                instance 1 {
                    vni 4
                }
            }
            bgp-evpn {
                evi 4
                local-attachment-circuit "AC-45" {
                    eth-tag 145
                }
                remote-attachment-circuit "AC-23" {
                    eth-tag 123
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                    ecmp 2
                }
            }
        }

On PE-6, Epipe 4 is configured as follows:

# on PE-6:
configure {
    service {
        epipe "Epipe-4" {
            admin-state enable
            description "Epipe-4 with active/standby pseudowire"
            service-id 4
            customer "1"
            endpoint "EP" {
            }
            spoke-sdp 640:4 {
                endpoint {
                    name "EP"
                }
            }
            spoke-sdp 650:4 {
                endpoint {
                    name "EP"
                }
            }
            sap 1/2/1:4 {
            }
        }

Verification

The connectivity between the CEs emulated by VPRN 14 can be verified as follows:

[/]
A:admin@PE-1# ping 172.16.14.64 router-instance "VPRN 14" interval 0.1 
                                                              output-format summary
PING 172.16.14.64 56 data bytes
!!!!!
---- 172.16.14.64 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 4.17ms, avg = 5.79ms, max = 11.5ms, stddev = 2.88ms
[/]
A:admin@PE-1# ping 2001:db8::14:64 router-instance "VPRN 14" interval 0.1 
                                                              output-format summary
PING 2001:db8::14:64 56 data bytes
!!!!!
---- 2001:db8::14:64 PING Statistics ----
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min = 4.29ms, avg = 6.41ms, max = 14.4ms, stddev = 4.00ms

The following BGP-EVPN information for Epipe 4 includes the EVI and the AC names and Ethernet tags:

[/]
A:admin@PE-2# show service id 4 bgp-evpn

===============================================================================
BGP EVPN Table
===============================================================================
EVI                : 4                  Creation Origin    : manual
Local AC Name      : AC-23
Eth Tag            : 123
Endpoint           : (Not Specified)
Ingress Label      : 0
Remote AC Name     : AC-45
Eth Tag            : 145
Endpoint           : (Not Specified)

===============================================================================
BGP EVPN VXLAN Information
===============================================================================
Admin Status       : Enabled            Bgp Instance       : 1
Vxlan Instance     : 1
Max Ecmp Routes    : 2
Default Route Tag  : none
Send EVPN Encap    : Enabled
===============================================================================

PE-4 received the following BGP-EVPN ES route with ESI 01:00:00:00:00:45:00:00:00:04 from PE-5:

[/]
A:admin@PE-4# show router bgp routes evpn eth-seg
===============================================================================
 BGP Router ID:192.0.2.4        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Eth-Seg Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      OrigAddr
-------------------------------------------------------------------------------
u*>i  192.0.2.5:0         01:00:00:00:00:45:00:00:00:04 192.0.2.5
      192.0.2.5

-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Furthermore, PE-4 received the following AD per-EVI (with Ethernet tag 123 or 145) and AD per-ES (MAX-ET) routes for Epipe 4 from its three BGP peers. The ESI is non-zero for multi-homed services.

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc
===============================================================================
 BGP Router ID:192.0.2.4        AS:64500       Local AS:64500
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
---snip---

u*>i  192.0.2.2:4         01:00:00:00:00:23:00:00:00:04 192.0.2.2
      123                                               VNI 4

u*>i  192.0.2.2:4         01:00:00:00:00:23:00:00:00:04 192.0.2.2
      MAX-ET                                            LABEL 0

u*>i  192.0.2.3:4         01:00:00:00:00:23:00:00:00:04 192.0.2.3
      123                                               VNI 4

u*>i  192.0.2.3:4         01:00:00:00:00:23:00:00:00:04 192.0.2.3
      MAX-ET                                            LABEL 0

u*>i  192.0.2.5:4         01:00:00:00:00:45:00:00:00:04 192.0.2.5
      145                                               VNI 4

u*>i  192.0.2.5:4         01:00:00:00:00:45:00:00:00:04 192.0.2.5
      MAX-ET                                            LABEL 0

-------------------------------------------------------------------------------

In AA MH ESs, the DF for VPLS services is the forwarder for Broadcast, Unknown unicast, and Multicast (BUM) traffic. In Epipes, however, all traffic is treated as unicast. The following tools commands on PE-2 and PE-3 show that DF is not applicable for AA MH ES "vES23_4.

[/]
A:admin@PE-2# tools dump service system bgp-evpn ethernet-segment "vES23_4" evi 4 df

[06/30/2021 17:33:54] All Active VPWS - DF N/A
[/]
A:admin@PE-3# tools dump service system bgp-evpn ethernet-segment "vES23_4" evi 4 df

[06/30/2021 17:33:56] All Active VPWS - DF N/A

The following command on PE-2 shows no DF candidates for ES "vES23_4", even though PE-2 (as well as PE-3) is considered as DF (DF=yes):

[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment name "vES23_4" evi evi-1 4

===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
4             4             0                   yes 06/30/2021 17:28:14
===============================================================================

===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
No entries found
===============================================================================

In the SA MH ES "ES45", PE-4 is DF out of a list of two candidates, as follows:

[/]
A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ES45" evi evi-1 4

===============================================================================
EVI DF and Candidate List
===============================================================================
EVI           SvcId         Actv Timer Rem      DF  DF Last Change
-------------------------------------------------------------------------------
4             4             0                   yes 06/30/2021 17:30:25
===============================================================================

===============================================================================
DF Candidates                           Time Added
-------------------------------------------------------------------------------
192.0.2.4                               06/30/2021 17:30:55
192.0.2.5                               06/30/2021 17:30:55
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================

On NDF PE-5, the spoke-SDP is operationally down with flag StandbyForMHProtocol, as follows:

[/]
A:admin@PE-5# show service id 4 sdp

===============================================================================
Services: Service Destination Points
===============================================================================
SdpId            Type     Far End addr    Adm     Opr       I.Lbl     E.Lbl
-------------------------------------------------------------------------------
560:4            Spok     192.0.2.6       Up      Down      524282    524281
-------------------------------------------------------------------------------
Number of SDPs : 1
-------------------------------------------------------------------------------
===============================================================================
[/]
A:admin@PE-5# show service id 4 sdp detail | match "Flags"
Flags              : StandbyForMHProtocol

The following command on PE-2 shows that the VXLAN destination for Epipe 4 is the ES "ES45" with ESI 01:00:00:00:00:45:00:00:00:04 and TEP address 192.0.2.4, which is the system IP address of the DF.

[/]
A:admin@PE-2# show service id 4 vxlan destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI          Oper      Vxlan
                                                            State     Type
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

===============================================================================
BGP EVPN VXLAN ES Dest
===============================================================================
I Eth Seg Id                    TEP Address       VNI       Last Changed
-------------------------------------------------------------------------------
1 01:00:00:00:00:45:00:00:00:04 192.0.2.4         4         06/30/2021 17:30:58
-------------------------------------------------------------------------------
===============================================================================

On PE-2, the following command shows that BGP-EVPN AD per-EVI routes with Ethernet tag 145 from PE-4 (RD 192.0.2.4:4) are sent with primary flag P=1 and AD per-EVI routes with Ethernet tag 145 from PE-5 (RD 192.0.2.5:4) are sent with primary flag P=0 and backup flag B=1.

[/]
A:admin@PE-3# show router bgp routes evpn auto-disc tag 145 detail 
                                                            | match 'C:|Route Dist'
Community      : target:64500:4 l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
Route Dist.    : 192.0.2.4:4            
---snip---            
Community      : target:64500:4 l2-attribute:MTU: 1514 C: 0 P: 0 B: 1
Route Dist.    : 192.0.2.5:4            
---snip---

AA and SA multi-homed EVPN-VXLAN Epipe using non-system IPv4 addresses

EVPN-VXLAN Epipe 5 with AA MH and SA MH using non-system IPv4 addresses shows the example topology for EVPN-VXLAN Epipe 5 with AA MH ES "vES23_5" between PE-2 and PE-3 and SA MH ES "ES45_5" between PE-4 and PE-5.

Figure 9. EVPN-VXLAN Epipe 5 with AA MH and SA MH using non-system IPv4 addresses

The configuration of Epipe 5 on PE-1 is similar to the configuration of Epipe 4 on PE-1, so it is not shown here. The same applies for Epipe 5 on PE-6.

On PE-2, VTEP 10.0.2.1 is used instead of the system IP address. The ES must include two additional parameters for the DF selection: orig-ip and route-next-hop, which are both equal to the VTEP. Without these parameters, the DF selection will not work. The orig-ip command modifies the originator IP address of the ES route and the route-next-hop modifies the next-hop of the AD per-ES routes for the ES. The service configuration on PE-2 is as follows:

# on PE-2:
configure {
    service {
        system {
            vxlan {
                tunnel-termination 10.0.2.1 {
                    fpe-id 1
                }
                tunnel-termination 2001::2:1 {
                    fpe-id 1
                }
            }
            bgp {
                evpn {
                    ethernet-segment "vES23_5" {
                        admin-state enable
                        type virtual
                        esi 01:00:00:00:00:23:00:00:00:05
                        orig-ip 10.0.2.1
                        route-next-hop 10.0.2.1
                        multi-homing-mode all-active
                        association {
                            lag "lag-1" {
                                virtual-ranges {
                                    dot1q {
                                        q-tag 5 {
                                            end 5
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
        epipe "Epipe-5" {
            admin-state enable
            service-id 5
            customer "1"
            bgp 1 {
            }
            sap lag-1:5 {
            }
            vxlan {
                source-vtep 10.0.2.1
                instance 1 {
                    vni 5
                }
            }
            bgp-evpn {
                evi 5
                local-attachment-circuit "AC-23_2" {
                    eth-tag 223
                }
                remote-attachment-circuit "AC-45_2" {
                    eth-tag 245
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                    ecmp 2
                }
            }
        }

The service configuration on PE-3 is similar.

On PE-4, the service configuration is as follows:

# on PE-4:
configure {
    service {
        sdp 465 {
            admin-state enable
            description "GRE SDP for SA MH_Epipe5"
            far-end {
                ip-address 192.0.2.6
            }
        }
        system {
            vxlan {
                tunnel-termination 10.0.4.1 {
                    fpe-id 1
                }
                tunnel-termination 2001::4:1 {
                    fpe-id 1
                }
            }
            bgp {
                evpn {
                    ethernet-segment "ES45_5" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:05
                        orig-ip 10.0.4.1
                        route-next-hop 10.0.4.1
                        multi-homing-mode single-active
                        association {
                            sdp 465 {
                            }
                        }
                    }
                }
            }
        }
        epipe "Epipe-5" {
            admin-state enable
            service-id 5
            customer "1"
            bgp 1 {
            }
            spoke-sdp 465:5 {
            }
            vxlan {
                source-vtep 10.0.4.1
                instance 1 {
                    vni 5
                }
            }
            bgp-evpn {
                evi 5
                local-attachment-circuit "AC-45_2" {
                    eth-tag 245
                }
                remote-attachment-circuit "AC-23_2" {
                    eth-tag 223
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                    ecmp 2
                }
            }
        }

In the AA MH ES, both PE-2 and PE-3 are DF. PE-4 receives BGP-EVPN autodiscovery routes with Ethernet tag 223 from PE-2 and PE-3 with the primary flag set to 1, as follows:

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc tag 223 detail 
                                                            | match 'C:|Route Dist'
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
Route Dist.    : 192.0.2.2:5            
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
Route Dist.    : 192.0.2.2:5            
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
Route Dist.    : 192.0.2.3:5            
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
Route Dist.    : 192.0.2.3:5      

The VXLAN destinations for Epipe 5 on PE-4 are the non-system TEP addresses 10.0.2.1 and 10.0.3.1 in ES "vES23_5" with ESI 01:00:00:00:00:23:00:00:00:05, as follows:

[/]
A:admin@PE-4# show service id 5 vxlan destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI          Oper      Vxlan
                                                            State     Type
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

===============================================================================
BGP EVPN VXLAN ES Dest
===============================================================================
I Eth Seg Id                    TEP Address       VNI       Last Changed
-------------------------------------------------------------------------------
1 01:00:00:00:00:23:00:00:00:05 10.0.2.1          5         06/30/2021 17:42:03
1 01:00:00:00:00:23:00:00:00:05 10.0.3.1          5         06/30/2021 17:42:03
-------------------------------------------------------------------------------
===============================================================================

In the SA MH ES, PE-5 is DF and PE-4 is NDF. PE-2 receives BGP-EVPN autodiscovery routes with Ethernet tag 245 from PE-4 with backup flag 1 and from PE-5 with primary flag 1, as follows:

[/]
A:admin@PE-2# show router bgp routes evpn auto-disc tag 245 detail 
                                                            | match 'C:|Route Dist'
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 0 B: 1
Route Dist.    : 192.0.2.4:5            
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 0 B: 1
Route Dist.    : 192.0.2.4:5            
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
Route Dist.    : 192.0.2.5:5            
Community      : target:64500:5 l2-attribute:MTU: 1514 C: 0 P: 1 B: 0
Route Dist.    : 192.0.2.5:5 

The VXLAN destination for Epipe 5 on PE-2 is the non-system TEP address 10.0.5.1 of DF PE-5 in ES "ES45_5" with ESI 01:00:00:00:00:45:00:00:00:05, as follows:

[/]
A:admin@PE-2# show service id 5 vxlan destinations

===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI          Oper      Vxlan
                                                            State     Type
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

===============================================================================
BGP EVPN VXLAN ES Dest
===============================================================================
I Eth Seg Id                    TEP Address       VNI       Last Changed
-------------------------------------------------------------------------------
1 01:00:00:00:00:45:00:00:00:05 10.0.5.1          5         06/30/2021 17:42:39
-------------------------------------------------------------------------------
===============================================================================

AA and SA multi-homed EVPN-VXLAN Epipe using non-system IPv6 addresses

EVPN-VXLAN Epipe 6 with AA MH and SA MH using non-system IPv6 addresses shows the example topology for EVPN-VXLAN Epipe 6 with AA MH ES "vES23_6" between PE-2 and PE-3 and SA MH ES "ES45_6" between PE-4 and PE-5.

Figure 10. EVPN-VXLAN Epipe 6 with AA MH and SA MH using non-system IPv6 addresses

The service configuration on PE-2 is as follows:

# on PE-2:
configure {
    service {
        system {
            vxlan {
                tunnel-termination 2001::2:1 {
                    fpe-id 1
                }
            }
            bgp {
                evpn {
                    ethernet-segment "vES23_6" {
                        admin-state enable
                        type virtual
                        esi 01:00:00:00:00:23:00:00:00:06
                        orig-ip 2001::2:1
                        route-next-hop 2001::2:1
                        multi-homing-mode all-active
                        association {
                            lag "lag-1" {
                                virtual-ranges {
                                    dot1q {
                                        q-tag 6 {
                                            end 6
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
        epipe "Epipe-6" {
            admin-state enable
            service-id 6
            customer "1"
            bgp 1 {
            }
            sap lag-1:6 {
            }
            vxlan {
                source-vtep 2001::2:1
                instance 1 {
                    vni 6
                }
            }
            bgp-evpn {
                evi 6
                local-attachment-circuit "AC-23_v6" {
                    eth-tag 623
                }
                remote-attachment-circuit "AC-45_v6" {
                    eth-tag 645
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                    ecmp 2
                }
            }
        }

The service configuration on PE-4 is as follows:

# on PE-4:
configure {
    service {
        system {
            vxlan {
                tunnel-termination 10.0.4.1 {
                    fpe-id 1
                }
                tunnel-termination 2001::4:1 {
                    fpe-id 1
                }
            }
            bgp {
                evpn {
                    ethernet-segment "ES45_6" {
                        admin-state enable
                        esi 01:00:00:00:00:45:00:00:00:06
                        orig-ip 2001::4:1
                        route-next-hop 2001::4:1
                        multi-homing-mode single-active
                        association {
                            sdp 466 {
                            }
                        }
                    }
                }
            }
        }
        epipe "Epipe-6" {
            admin-state enable
            service-id 6
            customer "1"
            bgp 1 {
            }
            spoke-sdp 466:6 {
            }
            vxlan {
                source-vtep 2001::4:1
                instance 1 {
                    vni 6
                }
            }
            bgp-evpn {
                evi 6
                local-attachment-circuit "AC-45_v6" {
                    eth-tag 645
                }
                remote-attachment-circuit "AC-23_v6" {
                    eth-tag 623
                }
                vxlan 1 {
                    admin-state enable
                    vxlan-instance 1
                    ecmp 2
                }
            }
        }
        sdp 466 {
            admin-state enable
            description "GRE SDP for SA MH_Epipe6"
            far-end {
                ip-address 192.0.2.6
            }
        }

Conclusion

EVPN-VXLAN VPWS is similar to EVPN-MPLS VPWS, and can be used in networks without MPLS.