EVPN for VXLAN Tunnels (Layer 3)

This chapter provides information about EVPN for VXLAN tunnels (Layer 3).

Topics in this chapter include:

Applicability

This chapter is applicable to SR OS and was initially written for Release 12.0.R4. The CLI in the current edition is based on SR OS Release 21.10.R3. Ethernet Virtual Private Network (EVPN) is a control plane technology and does not have line card hardware dependencies.

Chapter EVPN for VXLAN Tunnels (Layer 2) is prerequisite reading.

Overview

As discussed in the EVPN for VXLAN Tunnels (Layer 2) chapter, EVPN and VXLAN can be enabled on VPLS or R-VPLS services in SR OS. Where that chapter focuses on the use of EVPN-VXLAN layer 2 services, in other words, how EVPN-VXLAN is configured in VPLS services, this chapter describes how EVPN-VXLAN can be used to provide inter-subnet forwarding in R-VPLS and VPRN services. Inter-subnet forwarding can be provided by regular R-VPLS and VPRN services. However, EVPN provides an efficient and unified way to populate Forwarding Databases (FDBs), Address Resolution Protocol (ARP) tables and routing tables using a single BGP address family. Inter-subnet forwarding in overlay networks would otherwise require data plane learning and the use of routing protocols on a per VPRN basis.

The SR OS solution for inter-subnet forwarding using EVPN is based on building blocks described in draft-ietf-bess-evpn-inter-subnet-forwarding and the use of the EVPN IP-prefix routes (route type 5) as explained in RFC 9136. This example describes three supported common scenarios and provides the CLI configuration and required tools to troubleshoot EVPN-VXLAN in each case. The scenarios configured and explained are:

  • EVPN-VXLAN in R-VPLS services

  • EVPN-VXLAN in Integrated Routing Bridging (IRB) backhaul R-VPLS services

  • EVPN-VXLAN in EVPN tunnel R-VPLS services

In all these scenarios, redundant PEs are usually deployed. If that is the case, the interaction of EVPN, IP-VPN, and the Routing Table Manager (RTM) may lead to some routing loop situations that must be avoided by using routing policies (this also may happen in traditional IP-VPN deployments when eBGP and MP-BGP interact to populate VPRN routing tables in multi-homed networks). This chapter explains when those routing loops can happen and how to avoid them.

The term IRB interface refers to an R-VPLS service bound to a VPRN IP interface. The terms IRB interface and R-VPLS interface are used interchangeably throughout this chapter.

Configuration

This section describes the configuration of EVPN-VXLAN for Layer 3 services on SR OS, as well as the available troubleshooting and show commands. The three scenarios described in the overview are analyzed independently.

EVPN-VXLAN in an R-VPLS service

EVPN-VXLAN for R-VPLS services shows the topology used in the first scenario.

Figure 1. EVPN-VXLAN for R-VPLS services

The network topology shows two overlay (VXLAN) networks interconnected by an MPLS network:

  • PE-1, PE-2, and PE-3 are part of Overlay-Network-1

  • PE-4, PE-5, and PE-6 are part of Overlay-Network-2

A Layer 2/Layer 3 service is provided to a customer to connect CE-1, CE-3, and CE-6. In this scenario, Layer 2 connectivity is provided within each overlay network and inter-subnet connectivity (Layer 3) is provided between the overlay networks. VPLS 101 is defined within each overlay network and VPRN 10 connects both Layer 2 services through an IP-VPN MPLS network.

This topology can illustrate a Data Center Interconnect (DCI) example, where Overlay-Network-1 and Overlay-Network-2 are two data centers interconnected through an MPLS WAN. In this application, CE-1, CE-3, and CE-6 simulate virtual machines or appliances, PE-2/3/4/5 act as Data Center Gateways (DC GWs) and PE-1/6 as Network Virtualization Edge devices (or virtual PEs running on a compute infrastructure).

The following protocols and objects are configured beforehand:

  • The ports interconnecting the six PEs in EVPN-VXLAN for R-VPLS services are configured as network or hybrid ports and have router network interfaces defined in them. Only the ports connected to the CEs are configured as access ports.

  • The six PEs are running IS-IS for the global routing table with the four core PEs interconnected using IS-IS Level-2 point-to-point interfaces and each overlay network using IS-IS Level-1 point-to-point interfaces.

  • LDP is used as the MPLS protocol to signal transport tunnel labels among PE-2, PE-3, PE-4, and PE-5. There is no LDP running within each overlay network.

  • The network port MTU (in all the ports sending/receiving VXLAN packets) must be at least 50 bytes (54 if dot1q encapsulation is used) greater than the service MTU to accommodate the size of the VXLAN header.

Once the IGP infrastructure and LDP in the core are enabled, BGP is configured. In this scenario, two BGP families must be enabled: EVPN within each overlay network for the exchange of MAC/IP addresses and setting up the flooding domains, and VPN-IPv4 among the four core PEs so that IP prefixes can be exchanged and resolved to MPLS tunnels in the core.

The following CLI output shows the BGP configuration of PE-1, which only needs the EVPN family. PE-6 has a similar BGP configuration, that is, only EVPN family is configured for its peers. The use of Route Reflectors (RRs) in these scenarios is common. Although this scenario does not use RRs, an EVPN RR could have been used in Overlay-Network-1 and Overlay-Network-2 and a separate VPN-IPv4 RR could have been used in the core IP-VPN MPLS network.

# on PE-1:
configure
    router Base
        autonomous-system 64500
        bgp
            vpn-apply-import
            vpn-apply-export
            enable-peer-tracking
            rapid-withdrawal
            rapid-update evpn
            group "DC"
                family evpn
                peer-as 64500
                neighbor 192.0.2.2
                exit
                neighbor 192.0.2.3
                exit
            exit
            no shutdown
        exit

The BGP configuration on the DC GWs is as follows:

# on PE-2:
configure
    router
        autonomous-system 64500
        bgp
            vpn-apply-import
            vpn-apply-export
            enable-peer-tracking
            rapid-withdrawal
            rapid-update evpn
            group "DC"
                family vpn-ipv4 evpn
                peer-as 64500
                neighbor 192.0.2.1
                exit
                neighbor 192.0.2.3
                exit
            exit
            group "WAN"
                family vpn-ipv4
                peer-as 64500
                neighbor 192.0.2.4
                exit
                neighbor 192.0.2.5
                exit
            exit
            no shutdown
        exit
# on PE-3:
configure
    router
        autonomous-system 64500
        bgp
            vpn-apply-import
            vpn-apply-export
            enable-peer-tracking
            rapid-withdrawal
            rapid-update evpn
            group "DC"
                family vpn-ipv4 evpn
                peer-as 64500
                neighbor 192.0.2.1
                exit
                neighbor 192.0.2.2
                exit
            exit
            group "WAN"
                family vpn-ipv4
                peer-as 64500
                neighbor 192.0.2.4
                exit
                neighbor 192.0.2.5
                exit
            exit
            no shutdown
        exit

The DC GWs PE-4 and PE-5 have an equivalent BGP configuration.

BGP adjacencies and enabled families shows the BGP peering sessions among the PEs and the enabled BGP families. PE-1 and PE-6 only establish an EVPN peering session with their peers (only the EVPN family is enabled on PE-1 and PE-6, even if the peer PEs are VPN-IPv4 capable as well).

Figure 2. BGP adjacencies and enabled families

Once the network infrastructure is running properly, the actual service configuration, as illustrated in EVPN-VXLAN for R-VPLS services, can be carried out. The following CLI shows the configuration for VPLS 101 and VPRN 10 in PE-1, PE-2, and PE-3. The other overlay network has a similar configuration.

# on PE-1:
configure
    service
        vpls 101 name "evi-101" customer 1 create
            vxlan instance 1 vni 101 create
            exit
            bgp
                route-distinguisher 192.0.2.1:101
                route-target export target:64500:101 import target:64500:101
            exit
            bgp-evpn
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            sap 1/2/1:101 create
                no shutdown
            exit
            proxy-arp
                no shutdown
            exit
            no shutdown

Proxy-ARP is disabled (default) on PE-2, as well as on the other core PEs:

# on PE-2:
configure
    service
        vpls 101 name "evi-101" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 101 create
            exit
            bgp
                route-distinguisher 192.0.2.2:101
                route-target export target:64500:101 import target:64500:101
            exit
            bgp-evpn
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit
        vprn 10 name "VPRN10" customer 1 create
            ecmp 2
            interface "int-1" create
                address 172.16.0.2/24
                mac 00:ca:fe:ca:fe:02
                vrrp 1
                    backup 172.16.0.254
                    priority 254
                    ping-reply
                    traceroute-reply
                    mac 00:ca:fe:ca:fe:54
                exit
                vrrp 2
                    backup 172.16.0.253
                    ping-reply
                    traceroute-reply
                    mac 00:ca:fe:ca:fe:53
                exit
                vpls "evi-101"
                exit
            exit
            bgp-ipvpn
                mpls
                    auto-bind-tunnel
                        resolution-filter
                            ldp
                        exit
                        resolution filter
                    exit
                    route-distinguisher 192.0.2.2:10
                    vrf-target target:64500:10
                    no shutdown
                exit
            exit
            no shutdown
        exit
# on PE-3:
configure
    service
        vpls 101 name "evi-101" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 101 create
            exit
            bgp
                route-distinguisher 192.0.2.3:101
                route-target export target:64500:101 import target:64500:101
            exit
            bgp-evpn
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            sap 1/2/1:101 create
                no shutdown
            exit
            no shutdown
        exit
        vprn 10 name "VPRN10" customer 1 create
            ecmp 2
            interface "int-1" create
                address 172.16.0.3/24
                mac 00:ca:fe:ca:fe:03
                vrrp 1
                    backup 172.16.0.254
                    ping-reply
                    traceroute-reply
                    mac 00:ca:fe:ca:fe:54
                exit
                vrrp 2
                    backup 172.16.0.253
                    priority 254
                    ping-reply
                    traceroute-reply
                    mac 00:ca:fe:ca:fe:53
                exit
                vpls "evi-101"
                exit
            exit
            bgp-ipvpn
                mpls
                    auto-bind-tunnel
                        resolution-filter
                            ldp
                        exit
                        resolution filter
                    exit
                    route-distinguisher 192.0.2.3:10
                    vrf-target target:64500:10
                    no shutdown
                exit
            exit
            no shutdown
        exit

For details about the EVPN and VXLAN configuration on PE-1 VPLS 101, see chapter EVPN for VXLAN Tunnels (Layer 2). The configuration of VPLS 101 on PE-2 and PE-3 has the following important aspects:

  • The allow-ip-int-bind command is required so that the R-VPLS can be bound to VPRN 10.

  • The service name "evi-101" is configured when the service is created and cannot be modified afterward. The service name must match the name configured in the VPRN 10 VPLS interface.

  • Even though EVPN and VXLAN are properly configured, proxy-ARP cannot be enabled in VPLS 101. In an R-VPLS with EVPN-VXLAN, proxy-ARP is not supported and the VPRN ARP table is used instead. When an EVPN MAC route that includes an IP address is received in an R-VPLS, the MAC-IP pair encoded in the route is added to the ARP table of the VPRN, as opposed to the proxy-ARP table.

    *A:PE-2>config>service>vpls>proxy-arp$ no shutdown
    MINOR: SVCMGR #8007 Cannot modify proxy arp - service is routed
    

When configuring VPRN 10 on PE-2 and PE-3, the following considerations must be taken into account:

  • When trying to enable existing VPRN features on interfaces linked to EVPN-VXLAN R-VPLS interfaces, the authentication-policy command is not supported:

    *A:PE-2>config>service>vprn>if# authentication-policy "authPol1"
    INFO: PIP #1875 Cannot configure auth-policy on routed-vpls interface
    
  • Dynamic routing protocols such as IS-IS, RIP, or OSPF are not supported.

  • In general, no SR OS control plane generated packets are sent to the egress VXLAN bindings except for ARP, VRRP, ICMP, BFD, and Eth-CFM.

  • As shown in EVPN-VXLAN for R-VPLS services and in the CLI excerpts, VRRP can be configured on the VPRN 10 VPLS interfaces to provide default gateway redundancy to the hosts connected to VPLS 101. Two VRRP instances are configured so that VPLS 101 upstream traffic can be load-balanced to PE-2 and PE-3. With VRRP on EVPN-VXLAN R-VPLS interfaces:

    • Ping-reply and traceroute-reply can be configured and are supported. BFD is also supported to speed up the fault detection.

    • standby-forwarding, even if it were configured for VRRP, would not have any effect in this configuration: the standby PE will never see any flooded traffic sent to it, so this command is not applicable to this scenario.

  • When a VPRN 10 VPLS interface is bound to VPLS 101, EVPN advertises all the IP addresses configured for that VPLS interface as MAC routes with a static MAC indication. For the remote EVPN peers, that means that those MAC addresses linked to remote IP interfaces are protected. VRRP virtual IP/MACs are also advertised by EVPN as "static" and so protected. In the example of EVPN-VXLAN for R-VPLS services, the VPLS 101 FDB in PE-1 shows the IP interface MAC addresses and VRRP MAC addresses as EvpnS:P (Static and protected MAC) as shown in the following output:

    
    *A:PE-1# show service id 101 fdb detail
     
    ===============================================================================
    Forwarding Database, Service 101
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    101        00:00:01:01:01:01 sap:1/2/1:101           L/0      03/02/22 11:34:55
    101        00:00:03:03:03:03 vxlan-1:                Evpn     03/02/22 11:35:37
                                 192.0.2.3:101
    101        00:ca:fe:ca:fe:02 vxlan-1:                EvpnS:P  03/02/22 11:35:05
                                 192.0.2.2:101
    101        00:ca:fe:ca:fe:03 vxlan-1:                EvpnS:P  03/02/22 11:35:37
                                 192.0.2.3:101
    101        00:ca:fe:ca:fe:53 vxlan-1:                EvpnS:P  03/02/22 11:35:40
                                 192.0.2.3:101
    101        00:ca:fe:ca:fe:54 vxlan-1:                EvpnS:P  03/02/22 11:35:08
                                 192.0.2.2:101
    -------------------------------------------------------------------------------
    No. of MAC Entries: 6
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    

    The VPRN 10 VRRP instances on PE-2 are the following:

    *A:PE-2# show router 10 vrrp instance
     
    ===============================================================================
    VRRP Instances
    ===============================================================================
    Interface Name                   VR Id Own Adm  State       Base Pri   Msg Int
                                     IP        Opr  Pol Id      InUse Pri  Inh Int
    -------------------------------------------------------------------------------
    int-1                            1     No  Up   Master       254       1
                                     IPv4      Up   n/a         254        No
      Backup Addr: 172.16.0.254
    int-1                            2     No  Up   Backup       100       1
                                     IPv4      Up   n/a         100        No
      Backup Addr: 172.16.0.253
    -------------------------------------------------------------------------------
    Instances : 2
    ===============================================================================
    

    The ARP entries for PE-2 are the following:

    *A:PE-2# show router 10 arp
     
    ===============================================================================
    ARP Table (Service: 10)
    ===============================================================================
    IP Address      MAC Address       Expiry    Type   Interface
    -------------------------------------------------------------------------------
    172.16.0.2      00:ca:fe:ca:fe:02 00h00m00s Oth[I] int-1
    172.16.0.3      00:ca:fe:ca:fe:03 00h00m00s Evp[I] int-1
    172.16.0.253    00:ca:fe:ca:fe:53 00h00m00s Oth    int-1
    172.16.0.254    00:ca:fe:ca:fe:54 00h00m00s Oth[I] int-1
    -------------------------------------------------------------------------------
    No. of ARP Entries: 4
    ===============================================================================
    

EVPN-VXLAN in IRB backhaul R-VPLS services

EVPN-VXLAN for IRB backhaul R-VPLS services illustrates the second inter-subnet forwarding scenario, where Layer 3 connectivity must be provided not only between the overlay networks but also within each overlay network. In the example shown in EVPN-VXLAN for IRB backhaul R-VPLS services, a customer (tenant) has different subnets and connectivity must be provided across all of them (CE-1, CE-3, and CE-6 must be able to communicate), bearing in mind that EVPN-VXLAN is enabled in each overlay network and IP-VPN MPLS is used to interconnect both overlay networks. VPLS 201 is an IRB Backhaul R-VPLS service because it provides connectivity to the VPRN instances.

Figure 3. EVPN-VXLAN for IRB backhaul R-VPLS services

From a BGP peering perspective, there is no change in this scenario compared to the previous one: PE-1 and PE-6 only support the EVPN address family. However, in this scenario, CE-1 is now connected to an R-VPLS directly linked to the VPRN instances in PE-2/PE-3. As a result of that, IP prefixes must be exchanged between PE-1 and PE-2/PE-3. EVPN can advertise not only MAC routes and Inclusive Multicast routes, but also IP prefix routes that contain IP prefixes that can be installed in the attached VPRN routing table.

As an example, the VPRN 20 and VPLS "evi-201" configurations on PE-1, PE-2, and PE-3 are shown. Similar configurations are needed in PE-4, PE-5, and PE-6.

On PE-1, VPRN 20 and VPLS "evi-201" are configured as follows:

# on PE-1:
configure
    service
        vprn 20 name "VPRN20" customer 1 create
            interface "int-evi-201" create
                address 172.16.0.1/24
                vpls "evi-201"
                exit
            exit
            interface "int-PE-1-CE-1" create
                address 172.16.1.254/24
                sap 1/2/1:20 create
                exit
            exit
            no shutdown
        exit
        vpls 201 name "evi-201" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 201 create
            exit
            bgp
                route-distinguisher 192.0.2.1:201
                route-target export target:64500:201 import target:64500:201
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit

On PE-2, VPRN 20 and VPLS "evi-201" are configured as follows:

# on PE-2:
configure
    service
        vprn 20 name "VPRN20" customer 1 create
            interface "int-evi-201" create
                address 172.16.0.2/24
                vpls "evi-201"
                exit
            exit
            bgp-ipvpn
                mpls
                    auto-bind-tunnel
                        resolution any
                    exit
                    route-distinguisher 192.0.2.2:20
                    vrf-target target:64500:20
                    no shutdown
                exit
            exit
            no shutdown
        exit
        vpls 201 name "evi-201" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 201 create
            exit
            bgp
                route-distinguisher 192.0.2.2:201
                route-target export target:64500:201 import target:64500:201
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            no shutdown
        exit

On PE-3, VPRN 20 and VPLS "evi-201" are configured as follows:

# on PE-3:
configure
    service
        vprn 20 name "VPRN20" customer 1 create
            interface "int-evi-201" create
                address 172.16.0.3/24
                vpls "evi-201"
                exit
            exit
            bgp-ipvpn
                mpls
                    auto-bind-tunnel
                        resolution any
                    exit
                    route-distinguisher 192.0.2.3:20
                    vrf-target target:64500:20
                    no shutdown
                exit
            exit
            no shutdown
        exit
        vpls 201 name "evi-201" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 201 create
            exit
            bgp
                route-distinguisher 192.0.2.3:201
                route-target export target:64500:201 import target:64500:201
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            sap 1/2/1:20 create
                no shutdown
            exit
            no shutdown
        exit

As shown in the CLI excerpt, the configuration in the three nodes (PE-1, PE-2, and PE-3) for VPLS "evi-201" and VPRN 20 is very similar. The main difference is the auto-bind-tunnel command in VRPN 20 on PE-2 and PE-3. This command allows the VPRN 20 on PE-2 and PE-3 to receive IP-VPN routes from the core and resolve them to MPLS tunnels. VPRN 20 on PE-1 does not require such command because all its IP prefixes are resolved to local interfaces or to EVPN peers.

The ip-route-advertisement command enables:

  • The advertisement of IP prefixes in EVPN, in routes type 5. All the existing IP prefixes in the attached VPRN 20 routing table are advertised in EVPN within the VPLS 201 context (except for the ones associated to VPLS 201 itself).

  • The installation of IP prefixes in the attached VPRN 20 routing table with a preference of 169 (BGP-VPN routes for IP-VPN have a preference of 170) and a next-hop of the gateway IP (GW IP) address included in the EVPN IP prefix route.

For instance, the following output shows that PE-1 advertises the IP prefix 172.16.1.0/24 as an EVPN route to PE-3 (a similar route is sent to PE-2), captured by a debug router bgp update session.

44 2022/03/02 11:38:45.956 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.3
"Peer 1: 192.0.2.3: UPDATE
Peer 1: 192.0.2.3 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 82
    Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.1
        Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.1:201, tag: 0, 
              ip_prefix: 172.16.1.0/24 gw_ip 172.16.0.1 Label: 201 (Raw Label: 0xc9)
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 16 Extended Community:
        target:64500:201
        bgp-tunnel-encap:VXLAN
"

The VPRN 20 routing table in PE-1 includes two EVPN Interface-ful (EVPN-IFF) routes with preference 169, as follows:

*A:PE-1# show router 20 route-table
 
===============================================================================
Route Table (Service: 20)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.0.0/24                                 Local   Local     00h22m22s  0
       int-evi-201                                                  0
172.16.1.0/24                                 Local   Local     00h22m22s  0
       int-PE-1-CE-1                                                0
172.16.2.0/24                                 Remote  EVPN-IFF  00h01m41s  169
       172.16.0.2                                                   0
172.16.6.0/24                                 Remote  EVPN-IFF  00h01m41s  169
       172.16.0.2                                                   0
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

The subnet 172.16.0.0/24 is used on the interfaces "int-evi-201" in overlay network 1 and subnet 172.16.2.0/24 is used on similar interfaces in overlay network 2. CE-1 has an IP address in subnet 172.16.1.0/24 and CE-6 has an IP address in subnet 172.16.6.0/24. The next hop to reach 172.16.2.0/24 (overlay network 2) or CE-6, is 172.16.0.2 (PE-2), but it could have been PE-3.

There is redundancy in the example setup and therefore, loops can occur. To avoid loops, routing policies need to be configured on the core PEs (PE-2, PE-3, PE-4, and PE-5). These policies are described in the Use of routing policies to avoid routing loops in redundant PEs section for routing loop use case 1.

The routing table on PE-2 shows a EVPN-IFF route toward CE-1 (subnet 172.16.1.0/24) via PE-1. The route toward CE-6 uses a tunnel toward PE-4 in overlay network 2.

*A:PE-2# show router 20 route-table
 
===============================================================================
Route Table (Service: 20)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.0.0/24                                 Local   Local     00h20m36s  0
       int-evi-201                                                  0
172.16.1.0/24                                 Remote  EVPN-IFF  00h02m17s  169
       172.16.0.1                                                   0
172.16.2.0/24                                 Remote  BGP VPN   00h01m43s  170
       192.0.2.4 (tunneled)                                         10
172.16.6.0/24                                 Remote  BGP VPN   00h01m43s  170
       192.0.2.4 (tunneled)                                         10
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

The routing table on PE-3 is as follows:

*A:PE-3# show router 20 route-table
 
===============================================================================
Route Table (Service: 20)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.0.0/24                                 Local   Local     00h09m20s  0
       int-evi-201                                                  0
172.16.1.0/24                                 Remote  EVPN-IFF  00h02m46s  169
       172.16.0.1                                                   0
172.16.2.0/24                                 Remote  BGP VPN   00h02m20s  170
       192.0.2.4 (tunneled)                                         10
172.16.6.0/24                                 Remote  BGP VPN   00h01m53s  170
       192.0.2.4 (tunneled)                                         10
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

When checking the operation of EVPN in this scenario, it is important to observe that the right next hops and prefixes are successfully installed in the VPRN 20 routing table:

  • EVPN IP prefixes are sent using a GW IP matching the primary IP interface address of the R-VPLS for which the routes are sent. For instance, as shown above, IP prefix 172.16.1.0/24 is advertised from PE-1 with GW IP 172.16.0.1, which is the IP address configured for the VPRN 20 VPLS interface in PE-1. In the VPRN 20 routing tables on PE-2 and PE-3, IP prefix 172.16.1.0/24 is installed with next hop 172.16.0.1. Traffic arriving at PE-2 or PE-3 on VPRN 20 with IP Destination Address (DA) in the 172.16.1.0/24 subnet matches the mentioned routing table entry. As usual, the next-hop is resolved by the ARP table to a MAC address and the MAC address resolved by the FDB table to an egress VTEP, VNI.

  • IP prefixes in the VPRN 20 routing table are advertised in IP-VPN to the remote IP-VPN MPLS peers. Received IP-VPN prefixes are installed in the VPRN 20 routing table using the remote PE system IP address as the next hop, as usual. For instance, 172.16.6.0/24 is installed in the routing table of VPRN 20 on PE-2 with next-hop (tunneled) 192.0.2.4 and preference 170.

The following considerations of how the routing table manager (RTM) handles EVPN and IP-VPN prefixes must be taken into account:

  • Only VPRN interface primary addresses are advertised as GW IP in EVPN IP prefix routes. Secondary addresses are never sent as GW IP addresses.

  • EVPN IP prefixes are advertised by default as soon as the ip-route-advertisement command is enabled and there are active IP prefixes in the attached VPRN routing table.

  • If the same IP prefix is received on a PE via EVPN and IP-VPN at the same time for the same VPRN, by default, the EVPN prefix is selected because its preference (169) is better than the IP-VPN preference (170).

  • Because EVPN has a better preference compared to IP-VPN, when the VPRNs on redundant PEs are attached to the same R-VPLS service, routing loops may occur. The use case described here is an example where routing loops can occur. Check Use of routing policies to avoid routing loops in redundant PEs to avoid routing loops in redundant PEs for more information.

  • When the command ip-route-advertisement is enabled, the subnet IP prefixes are advertised in EVPN but not the host IP prefixes (/32 prefixes associated with the local interfaces). If the user wants to advertise the host IP prefixes as well, the incl-host keyword must be added to the ip-route-advertisement command. The following example illustrates this.


*A:PE-1# show router 20 route-table
 
===============================================================================
Route Table (Service: 20)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.0.0/24                                 Local   Local     00h10m12s  0
       int-evi-201                                                  0
172.16.1.0/24                                 Local   Local     00h10m12s  0
       int-PE-1-CE-1                                                0
172.16.2.0/24                                 Remote  EVPN-IFF  00h02m51s  169
       172.16.0.2                                                   0
172.16.6.0/24                                 Remote  EVPN-IFF  00h02m51s  169
       172.16.0.2                                                   0
-------------------------------------------------------------------------------
No. of Routes: 4
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

The host routes can be shown with the show router route-table all command:

*A:PE-1# show router 20 route-table all
 
===============================================================================
Route Table (Service: 20)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                         Active     Metric
-------------------------------------------------------------------------------
172.16.0.0/24                                 Local   Local     00h10m49s  0
       int-evi-201                                     Y            0
172.16.0.1/32                                 Local   Host      00h10m49s  0
       int-evi-201                                     Y            0
172.16.1.0/24                                 Local   Local     00h10m49s  0
       int-PE-1-CE-1                                   Y            0
172.16.1.254/32                               Local   Host      00h10m49s  0
       int-PE-1-CE-1                                   Y            0
172.16.2.0/24                                 Remote  EVPN-IFF  00h03m28s  169
       172.16.0.2                                      Y            0
172.16.6.0/24                                 Remote  EVPN-IFF  00h03m28s  169
       172.16.0.2                                      Y            0
-------------------------------------------------------------------------------
No. of Routes: 6
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
       E = Inactive best-external BGP route
===============================================================================

When the incl-host keyword is added to VPLS "evi-201" on PE-1, PE-1 advertises the host routes as well and these are installed in the routing tables on the remote PEs.

# on PE-1:
configure 
    service 
        vpls "evi-201"
            bgp-evpn 
                ip-route-advertisement incl-host
            exit
*A:PE-2# show router 20 route-table
 
===============================================================================
Route Table (Service: 20)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.0.0/24                                 Local   Local     00h11m40s  0
       int-evi-201                                                  0
172.16.1.0/24                                 Remote  EVPN-IFF  00h04m59s  169
       172.16.0.1                                                   0
172.16.1.254/32                               Remote  EVPN-IFF  00h00m11s  169
       172.16.0.1                                                   0
172.16.2.0/24                                 Remote  BGP VPN   00h04m27s  170
       192.0.2.4 (tunneled)                                         10
172.16.6.0/24                                 Remote  BGP VPN   00h04m27s  170
       192.0.2.4 (tunneled)                                         10
-------------------------------------------------------------------------------
No. of Routes: 5
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
  • ECMP is fully supported for the VPRN for EVPN IP prefix routes coming from different GW IP next-hops. However, ECMP is not supported for IP prefixes routes belonging to different owners (EVPN and IP-VPN). ECMP is enabled in VPRN 20 on PE-1, as follows:

    # on PE-1:
    configure 
        service 
            vprn "VPRN20" 
                ecmp 2
    

    When policies are applied that prevent routing loops, as described in section Use of routing policies to avoid routing loops in redundant PEs, both PE-2 and PE-3 have IP-VPN tunnels for IP prefixes 172.16.2.0/24 and 172.16.6.0/24. In that case, an additional route with a different GW IP as next hop is installed in the routing table for these IP prefixes:

    
    *A:PE-1# show router 20 route-table
     
    ===============================================================================
    Route Table (Service: 20)
    ===============================================================================
    Dest Prefix[Flags]                            Type    Proto     Age        Pref
          Next Hop[Interface Name]                                    Metric
    -------------------------------------------------------------------------------
    172.16.0.0/24                                 Local   Local     00h12m39s  0
           int-evi-201                                                  0
    172.16.1.0/24                                 Local   Local     00h12m39s  0
           int-PE-1-CE-1                                                0
    172.16.2.0/24                                 Remote  EVPN-IFF  00h00m08s  169
           172.16.0.2                                                   0
    172.16.2.0/24                                 Remote  EVPN-IFF  00h00m08s  169
           172.16.0.3                                                   0
    172.16.6.0/24                                 Remote  EVPN-IFF  00h00m08s  169
           172.16.0.2                                                   0
    172.16.6.0/24                                 Remote  EVPN-IFF  00h00m08s  169
           172.16.0.3                                                   0
    -------------------------------------------------------------------------------
    No. of Routes: 6
    Flags: n = Number of times nexthop is repeated
           B = BGP backup route available
           L = LFA nexthop available
           S = Sticky ECMP requested
    ===============================================================================
    

EVPN-VXLAN in EVPN tunnel R-VPLS services

The previous scenario shows how to use EVPN-VXLAN to provide inter-subnet forwarding for a tenant, where R-VPLS services can contain hosts and also offer transit services between VPRN instances. For example, in the use case shown in EVPN-VXLAN for IRB backhaul R-VPLS services, VPLS 201 in Overlay-Network-1 is an R-VPLS that can provide intra-subnet connectivity to all the hosts in subnet 172.16.0.0/24 (for example, CE-3 belongs to this subnet) but it can also provide transit or backhaul connectivity to hosts in subnet 172.16.1.0/24 (for example, CE-1) sending packets to subnets 172.16.2.0/24 or 172.16.6.0/24.

In some cases, the R-VPLS where EVPN-VXLAN is enabled does not need to provide intra-subnet connectivity and it is purely a transit or backhaul service where VPRN IRB interfaces are connected. EVPN-VXLAN in EVPN-tunnel R-VPLS services illustrates this use case.

Figure 4. EVPN-VXLAN in EVPN-tunnel R-VPLS services

Compared to the preceding use case in EVPN-VXLAN for IRB backhaul R-VPLS services, in this case the R-VPLS connecting the IRB interfaces in Overlay-Network-1 (VPLS 301) does not have any connected host. If that is the case, VPLS 301 can be configured as an EVPN tunnel.

EVPN tunnels are enabled using the evpn-tunnel command under the R-VPLS interface configured on the VPRN. EVPN tunnels bring the following benefits to EVPN-VXLAN IRB backhaul R-VPLS services:

  • Easier and simpler provisioning of the tenant service: if an EVPN tunnel is configured in an IRB backhaul R-VPLS, there is no need to provision the IRB IP addresses in the VPRN. This makes the provisioning easier to automate and saves IP addresses from the tenant IP space.

  • Higher scalability of the IRB backhaul R-VPLS: if EVPN tunnels are enabled, BUM traffic is suppressed in the EVPN-VXLAN IRB backhaul R-VPLS service (it is not required). As a result, the number of VXLAN bindings in IRB backhaul R-VPLS services with EVPN tunnels can be much higher.

As an example, the VPRN 30 and VPLS 301 configurations on PE-1, PE-2, and PE-3 are shown. Similar configurations are needed in PE-4, PE-5, and PE-6.

# on PE-1:
configure
    service
        vprn 30 name "VPRN30" customer 1 create
            interface "int-PE-1-CE-1" create
                address 172.16.0.254/24
                sap 1/2/1:30 create
                exit
            exit
            interface "int-evi-301" create
                vpls "evi-301"
                    evpn-tunnel
                exit
            exit
            no shutdown
        exit
        vpls 301 name "evi-301" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 301 create
            exit
            bgp
                route-distinguisher 192.0.2.1:301
                route-target export target:64500:301 import target:64500:301
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            no shutdown
        exit
# on PE-2:
configure
    service
        vprn 30 name "VPRN30" customer 1 create
            interface "int-evi-301" create
                vpls "evi-301"
                    evpn-tunnel
                exit
            exit
            bgp-ipvpn
                mpls
                    auto-bind-tunnel
                        resolution-filter
                            ldp
                        exit
                        resolution filter
                    exit
                    route-distinguisher 192.0.2.2:30
                    vrf-target target:64500:30
                    no shutdown
                exit
            exit
            no shutdown
        exit
        vpls 301 name "evi-301" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 301 create
            exit
            bgp
                route-distinguisher 192.0.2.2:301
                route-target export target:64500:301 import target:64500:301
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            no shutdown
        exit
# on PE-3:
configure
    service
        vprn 30 name "VPRN30" customer 1 create
            interface "int-evi-301" create
                vpls "evi-301"
                    evpn-tunnel
                exit
            exit
            bgp-ipvpn
                mpls
                    auto-bind-tunnel
                        resolution-filter
                            ldp
                        exit
                        resolution filter
                    exit
                    route-distinguisher 192.0.2.3:30
                    vrf-target target:64500:30
                    no shutdown
                exit
            exit
            no shutdown
        exit
        vpls 301 name "evi-301" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 301 create
            exit
            bgp
                route-distinguisher 192.0.2.3:301
                route-target export target:64500:301 import target:64500:301
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            no shutdown
        exit

As shown in the preceding output, the configuration in the three nodes (PE-1/2/3) for VPLS 301 and VPRN 30 is similar to the configuration of VPLS 201 and VPRN 20 in the previous scenario, however, when the evpn-tunnel command is added to the VPRN interface, there is no need to configure an IP interface address. The option evpn-tunnel can be enabled independently of ip-route-advertisement (although no route type 5 advertisements are sent in that case).

A VPRN supports regular IRB backhaul R-VPLS services as well as EVPN tunnel R-VPLS services. A maximum of eight R-VPLS services with ip-route-advertisement enabled per VPRN is supported (in any combination of regular IRB R-VPLS or EVPN tunnel R-VPLS services). EVPN tunnel R-VPLS services do not support SAPs or SDP-bindings. No frames are flooded in an EVPN tunnel R-VPLS service, and, in fact no inclusive multicast routes are exchanged in R-VPLS services that are configured as EVPN tunnels.

The show service id vxlan destinations command for an R-VPLS service configured as an EVPN tunnel shows <egress VTEP, VNI> bindings excluded from Mcast, in other words, the VXLAN bindings are not used to flood BUM traffic:

*A:PE-2# show service id 301 vxlan destinations
 
===============================================================================
Egress VTEP, VNI
===============================================================================
Instance    VTEP Address                            Egress VNI  EvpnStatic Num
 Mcast       Oper State                              L2 PBR     SupBcasDom MACs
-------------------------------------------------------------------------------
1           192.0.2.1                               301         evpn       1
 -           Up                                      No          No
1           192.0.2.3                               301         evpn       1
 -           Up                                      No          No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 2
-------------------------------------------------------------------------------
===============================================================================
 
===============================================================================
BGP EVPN-VXLAN Ethernet Segment Dest
===============================================================================
Instance  Eth SegId                       Num. Macs     Last Change
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================

The process followed upon receiving a route type 5 on a regular IRB R-VPLS interface (previous scenario) differs from the one for an EVPN tunnel type (this scenario):

  • IRB backhaul R-VPLS VPRN interface:

    • When a route type 2 that includes an IP address is received and it becomes active, the MAC/IP information is added to the FDB and ARP tables. This can be checked with the show router arp command and the show service id fdb detail command.

    • When a route type 5 is received on (for instance) PE-2, and becomes active for the R-VPLS service, the IP prefix is added to the VPRN routing table regardless of the existence of a route type 2 that can resolve the GW IP address. If a packet is received from the WAN side and the IP lookup hits an entry for which the GW IP (IP next-hop) does not have an active ARP entry, the system will ARP to get the MAC. If the ARP is resolved but the MAC is unknown in the FDB table, the system will flood the ARP message into the R-VPLS multicast list. Routes type 5 can be checked in the routing table with the show router route-table command and the show router fib command.

  • EVPN tunnel R-VPLS VPRN interface:

    • When a route type 2 is received and becomes active, the MAC address is added to the FDB (only). This MAC address is normally a GW MAC.

    • When a route type 5 is received on (for instance) PE-1, the system looks for the GW MAC. The IP prefix is added to the VPRN routing table with next hop equal to EVPN-tunnel GW MAC; for example, ET-02:13:ff:00:00:6a is an EVPN tunnel with GW MAC 02:13:ff:00:00:6a. The GW MAC is added from the GW MAC extended community sent along with the route type 5 for prefix 172.16.6.0/24. If a packet is received from CE-1 and the IP lookup hits an entry for which the next hop is an EVPN tunnel: GW MAC, the system looks up the GW MAC in the FDB. Normally a route type 2 with the GW MAC has already been received so that the GW MAC has been added to the FDB. If the GW MAC is not present in the FDB, the packet will be dropped.

    • The IP prefixes with GW MACs as next hops for the setup in EVPN-VXLAN in EVPN-tunnel R-VPLS services are displayed in the show router route-table command, as follows:

    
    *A:PE-1# show router 30 route-table
     
    ===============================================================================
    Route Table (Service: 30)
    ===============================================================================
    Dest Prefix[Flags]                            Type    Proto     Age        Pref
          Next Hop[Interface Name]                                    Metric
    -------------------------------------------------------------------------------
    172.16.1.0/24                                 Local   Local     00h02m40s  0
           int-PE-1-CE-1                                                0
    172.16.6.0/24                                 Remote  EVPN-IFF  00h00m36s  169
           int-evi-301 (ET-02:13:ff:00:00:6a)                           0
    -------------------------------------------------------------------------------
    No. of Routes: 2
    Flags: n = Number of times nexthop is repeated
           B = BGP backup route available
           L = LFA nexthop available
           S = Sticky ECMP requested
    ===============================================================================
    

    The same routing policies are applied on the core PEs to prevent loops; see Use of routing policies to avoid routing loops in redundant PEs.

    The show service id 301 fdb detail command can be used to look for the forwarding information for a GW MAC:

    
    *A:PE-1# show service id 301 fdb detail
     
    ===============================================================================
    Forwarding Database, Service 301
    ===============================================================================
    ServId     MAC               Source-Identifier       Type     Last Change
                Transport:Tnl-Id                         Age
    -------------------------------------------------------------------------------
    301        02:0f:ff:00:00:6a cpm                     Intf     03/02/22 11:52:54
    301        02:13:ff:00:00:6a vxlan-1:                EvpnS:P  03/02/22 11:53:02
                                 192.0.2.2:301
    301        02:17:ff:00:00:6a vxlan-1:                EvpnS:P  03/02/22 11:53:09
                                 192.0.2.3:301
    -------------------------------------------------------------------------------
    No. of MAC Entries: 3
    -------------------------------------------------------------------------------
    Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
    ===============================================================================
    

IP prefix routes sent for EVPN tunnel R-VPLS services do not contain a GW IP (the GW IP will be zero) but convey a GW MAC address that is used in the peer VPRN routing table. The following output shows PE-2’s VPRN 30 interface MAC address sent to PE-1:

*A:PE-2# show router 30 interface "int-evi-301" detail | match "MAC "
MAC Address      : 02:13:ff:00:00:6a    Mac Accounting    : Disabled

When ip-route-advertisement is configured, PE-2 sends route type 5 messages to PE-1, as can be seen in the following BGP update for the route toward subnet 172.16.6.0/24 in overlay network 2, using the MAC as GW MAC:

# on PE-2:
configure
    service
        vpls "evi-301"
            bgp-evpn
                ip-route-advertisement
            exit
221 2022/03/02 11:54:51.734 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.1
"Peer 1: 192.0.2.1: UPDATE
Peer 1: 192.0.2.1 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 90
    Flag: 0x90 Type: 14 Len: 45 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.2
        Type: EVPN-IP-PREFIX Len: 34 RD: 192.0.2.2:301, tag: 0,
              ip_prefix: 172.16.6.0/24 gw_ip 0.0.0.0 Label: 301 (Raw Label: 0x12d)
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64500:301
        mac-nh:02:13:ff:00:00:6a
        bgp-tunnel-encap:VXLAN
"

In the VPRN 30 routing table on PE-2, IP prefixes are shown with an EVPN tunnel next-hop (GW MAC) as opposed to an IP next-hop, therefore, the user may think that no ARP entries are consumed by VPRN 30. However, internal ARP entries are still consumed in VPRN 30. Although not shown in the show router 30 arp command, the summary option shows the consumption of internal ARP entries for EVPN.

*A:PE-2# show router 30 route-table
 
===============================================================================
Route Table (Service: 30)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric
-------------------------------------------------------------------------------
172.16.1.0/24                                 Remote  EVPN-IFF  00h04m28s  169
       int-evi-301 (ET-02:0f:ff:00:00:6a)                           0
172.16.6.0/24                                 Remote  BGP VPN   00h02m40s  170
       192.0.2.4 (tunneled)                                         10
-------------------------------------------------------------------------------
No. of Routes: 2
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================

There are no entries in the ARP table:

*A:PE-2# show router 30 arp
 
===============================================================================
ARP Table (Service: 30)
===============================================================================
IP Address      MAC Address       Expiry    Type   Interface
-------------------------------------------------------------------------------
No Matching Entries Found
===============================================================================

One internal BGP-EVPN ARP entry is consumed, as can be seen as follows:

*A:PE-2# show router 30 arp summary
 
============================================================
ARP Table Summary (Service: 30)
============================================================
Local ARP Entries    : 1
Static ARP Entries   : 0
Dynamic ARP Entries  : 0
Managed ARP Entries  : 0
Internal ARP Entries : 0
BGP-EVPN ARP Entries : 1
------------------------------------------------------------
No. of ARP Entries   : 2
============================================================

The number of BGP-EVPN ARP entries in the show router 30 arp summary command matches the number of remote valid GW MACs for VPRN 30.

Routing policies for IP prefixes in EVPN

Routing policies are supported for IP prefixes imported or exported through BGP EVPN. The default import and export behavior for IP prefixes in EVPN can be modified by the use of routing policies applied either at peer level (config router bgp group/group neighbor import/export) or VPLS level (config service vpls bgp vsi-import/vsi-export).

When applying routing policies to control the distribution of prefixes between EVPN and IP-VPN, the user must take into account that both families are completely separated as far as BGP is concerned and that when prefixes from a family are imported in the RTM, the BGP attributes are lost to the other family. The use of tags allows the controlled distribution of prefixes across the two families.

Routing policies for egress EVPN routes illustrates how VPN-IPv4 routes are imported into the RTM and then passed onto EVPN for its own processing. VPN-IPv4 routes can be tagged at ingress and this tag is preserved throughout the RTM and EVPN processing so that the tag can be matched by the egress BGP routing policy. In this example, egress EVPN routes matching tag 10, are modified to add a site-of-origin (SOO) community origin:64500:1.

Figure 5. Routing policies for egress EVPN routes

Policy tags can be used to match EVPN IP-prefixes that were learned not only from BGP VPN-IPv4 but also from other routing protocols. The tag range supported for each protocol is different:

*A:PE-2>config>router>policy-options>policy-statement>entry>action$ tag ?
  - no tag
  - tag <tag>
 
 <tag>                : tag-value      - accepts in decimal or hex
                                         [0x1..0xFFFFFFFF]H (for OSPF and ISIS)
                                         [0x1..0xFFFF]H (for RIP)
                                         [0x1..0xFFFFFFFF]H (for BGP)
                        param-name     - [32 chars max] - Must start and end with an at-sign
                                         (@)

Routing policies for ingress EVPN routes illustrates the reverse workflow: routes imported from EVPN and exported from RTM to BGPVPN-IPv4. In this example, EVPN routes received with community VM-mob are tagged with tag 200. At the egress VPN-IPv4 peers, only the routes with tag 200 are advertised.

Figure 6. Routing policies for ingress EVPN routes

The preceding behavior and the use of tags is also valid for vsi-import and vsi-export policies. The behavior can be summarized in the following statements:

  • For EVPN prefix routes received and imported in RTM:

    • Routes can be matched on communities and tags can be added to them. This works at peer level or vsi-import level.

    • Well-known communities [no-export | no-export-subconfed | no-advertise] also require that the routing policies add a tag if the user wants to modify the behavior when exporting to BGP.

    • Routes can be matched based on family EVPN.

    • Routes cannot be matched on prefix list.

  • For exporting RTM to EVPN prefix routes:

    • Routes can be matched on tags and based on that, communities added, or routes accepted or rejected (dropped), and so on. This works at peer level or vsi-export level.

    • Tags can be added for static routes, RIP, OSPF, IS-IS, and BGP and then be matched in the vsi-export policy for EVPN IP-prefix route advertisement.

    • Tags cannot be added for direct routes.

Use of routing policies to avoid routing loops in redundant PEs

When redundant PE VPRN instances are connected to the same R-VPLS service (IRB backhaul or EVPN tunnel R-VPLS) with the ip-route-advertisement command enabled, routing loops can occur in two different use cases:

  1. Routing loop caused by EVPN and IP-VPN interaction in the RTM.

  2. Routing loop caused by EVPN in parallel R-VPLS services.

Policy configuration examples for both cases are provided in the following sections.

Routing loop use-case 1: EVPN and IP-VPN interaction

This use case refers to scenarios with redundant PEs and VPRNs connected to the same R-VPLS with ip-route-advertisement. The scenarios in EVPN-VXLAN for IRB backhaul R-VPLS services (EVPN-VXLAN for IRB Backhaul R-VPLS services) and EVPN-VXLAN in EVPN-tunnel R-VPLS services (EVPN-VXLAN in EVPN tunnel R-VPLS services) are examples of this use case. In both scenarios, the following process causes a routing loop:

  1. PE-4 advertises IP prefix 172.16.6.0/24 with preference 170 (IP-VPN) to PE-2 and PE-3.

  2. PE-2 and PE-3 import prefix 172.16.6.0/24 in the VPRN routing table. PE-2 re-advertises prefix 172.16.6.0/24 with preference 169 (EVPN) to PE-1 and PE-3; PE-3 re-advertises the IP prefix in EVPN to PE-1 and PE-2.

  3. PE-2 and PE-3 already have the 172.16.6.0/24 prefix in the VPRN routing table with preference 170 (IP-VPN) but because the IP prefix from EVPN has a lower preference (169), the RTM installs the EVPN prefix in the VPRN routing table.

  4. PE-2 advertises the EVPN-learned IP prefix 172.16.6.0/24 to all MP-BGP VPN-IPv4 peers, including PE-3; PE-3 advertises the prefix 172.16.6.0/24 to all MP-BGP VPN-IPv4 peers, including PE-2.

  5. PE-2 receives the IP prefix 172.16.6.0/24 again from PE-3 and advertises it in EVPN again, creating a routing loop. The same thing happens in PE-3.

This routing loop also happens in traditional multi-homed IP-VPN scenarios where the PE-CE eBGP and MP-BGP VPN-IPv4/v6 protocols interact in the same VPRN RTM, with different router preferences. In either case (EVPN or eBGP interaction with MP-BGP) the issue can be solved by using routing policies and site-of-origin communities.

Routing policies are applied to PE-2 and PE-3 (also to PE-4 and PE-5) and allow the redundant PEs to reject their own generated routes to avoid the loops. These routing policies can be applied at vsi-import/export level or BGP group/neighbor level. The following output shows an example of routing policies applied at BGP neighbor level for PE-2 (similar policies are applied on PE-3/4/5). Neighbor or group level policies are the preferred way in this kind of use case: a single set of policies is sufficient, as opposed to a set of policies per service (if the policies are applied at vsi-import/export level).

The following policies are applied in the BGP group or BGP group/neighbor context on PE-2:

# on PE-2:
configure
    router Base
        policy-options
            begin
            community "SOO-PE-2"
                members "origin:2:1"
            exit
            community "SOO-PE-3"
                members "origin:3:1"
            exit
            policy-statement "add-SOO_on_export"
                entry 10
                    from
                        tag 2
                    exit
                    action accept
                        community add "SOO-PE-2"
                    exit
                exit
                entry 20
                    from
                        tag 3
                    exit
                    action accept
                        community add "SOO-PE-3"
                    exit
                exit
            exit
            policy-statement "reject_based_on_SOO"
                entry 10
                    from
                        community "SOO-PE-2"
                    exit
                    action drop
                    exit
                exit
                entry 20
                    from
                        community "SOO-PE-3"
                    exit
                    action drop
                    exit
                exit
            exit
            policy-statement "add-tag_to_bgp-vpn_routes"
                entry 10
                    from
                        protocol bgp-vpn
                    exit
                    action accept
                        tag 2
                    exit
                exit
            exit
            policy-statement "add-tag_to_bgp-evpn_routes"
                entry 10
                    from
                        family evpn
                    exit
                    action accept
                        tag 2
                    exit
                exit
            exit
            commit
        exit
        bgp
            group "DC"
                neighbor 192.0.2.1
                    import "add-tag_to_bgp-evpn_routes"
                exit
                neighbor 192.0.2.3
                    import "reject_based_on_SOO"
                    export "add-SOO_on_export"
                exit
            exit
            group "WAN"
                import "add-tag_to_bgp-vpn_routes"
            exit

EVPN and MP-BGP routes are tagged at import; on export, a site-of-origin community is added. Routes exchanged between the two redundant PEs are dropped if they are received by a PE with its own site-of-origin.

Routing loop use-case 2: EVPN in parallel R-VPLS services

If a VPRN is connected to more than one R-VPLS with ip-route-advertisement enabled, IP prefixes that belong to one R-VPLS are advertised into the other R-VPLS and vice versa. When redundant PEs are used, a routing loop will occur. EVPN in parallel R-VPLS services illustrates this use case. The example shows R-VPLS with an EVPN tunnel configuration, but the same routing loop occurs for regular IRB backhaul R-VPLS services.

Figure 7. EVPN in parallel R-VPLS services

The configuration of VPRN 50 as well as VPLS 501/502 and the required policies are as follows. For this use case, policies must be applied at vsi-import/export level because more granularity is required when modifying the imported/exported routes.

# on PE-2:
configure
    service
        vprn 50 name "VPRN50" customer 1 create
            interface "int-evi-501" create
                vpls "evi-501"
                    evpn-tunnel
                exit
            exit
            interface "int-evi-502" create
                vpls "evi-502"
                    evpn-tunnel
                exit
            exit
            no shutdown
        exit
        vpls 501 name "evi-501" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 501 create
            exit
            bgp
                route-distinguisher 192.0.2.2:501
                vsi-export "vsi-export-policy-501"
                vsi-import "vsi-import-policy-501"
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit
        vpls 502 name "evi-502" customer 1 create
            allow-ip-int-bind
            exit
            vxlan instance 1 vni 502 create
            exit
            bgp
                route-distinguisher 192.0.2.2:502
                vsi-export "vsi-export-policy-502"
                vsi-import "vsi-import-policy-502"
            exit
            bgp-evpn
                ip-route-advertisement
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            exit
            no shutdown
        exit
    exit
    router Base
        policy-options
            begin
            community "exp_RVPLS501"
                members "origin:2:11" "target:64500:501"
            exit
            community "exp_RVPLS502"
                members "origin:2:11" "target:64500:502"
            exit
            community "SOO-PE-2-RVPLS"
                members "origin:2:11"
            exit
            community "SOO-PE-3-RVPLS"
                members "origin:3:11"
            exit
            community "SOO_PE-3_RVPLS501"
                members "origin:3:11" "target:64500:501"
            exit
            community "SOO_PE-3_RVPLS502"
                members "origin:3:11" "target:64500:502"
            exit
            policy-statement "vsi-export-policy-501"
                entry 10
                    from
                        tag 12
                    exit
                    action accept
                        community add "SOO_PE-3_RVPLS501"
                    exit
                exit
                entry 20
                    action accept
                        community add "exp_RVPLS501"
                    exit
                exit
            exit
            policy-statement "vsi-export-policy-502"
                entry 10
                    from
                        tag 12
                    exit
                    action accept
                        community add "SOO_PE-3_RVPLS502"
                    exit
                exit
                entry 20
                    action accept
                        community add "exp_RVPLS502"
                    exit
                exit
            exit
            policy-statement "vsi-import-policy-501"
                entry 10
                    from
                        community "SOO-PE-2-RVPLS"
                    exit
                    action drop
                    exit
                exit
                entry 20
                    from
                        community "SOO_PE-3_RVPLS501"
                    exit
                    action accept
                        tag 12
                    exit
                exit
                default-action accept 
                exit
            exit
            policy-statement "vsi-import-policy-502"
                entry 10
                    from
                        community "SOO-PE-2-RVPLS"
                    exit
                    action drop
                    exit
                exit
                entry 20
                    from
                        community "SOO_PE-3_RVPLS502"
                    exit
                    action accept
                        tag 12
                    exit
                exit
                default-action accept
                exit
            exit
            commit

Troubleshooting and debug commands

For general information on EVPN and VXLAN troubleshooting and debug commands, see chapter EVPN for VXLAN Tunnels (Layer 2). The following information focuses on specific commands for Layer-3 applications.

When troubleshooting and operating an EVPN-VXLAN scenario with inter-subnet forwarding, it is important to check the IP prefixes and next-hops, as well as ARP tables and FDB tables:

  • show router <..> route-table

  • show router <..> arp

  • show service id <..> fdb detail

ICMP commands can also help checking the connectivity. When traceroute is used on EVPN-VXLAN in EVPN tunnel interfaces, EVPN tunnel interface hops in the traceroute commands are showing the VPRN loopback address or the other non EVPN-tunnel interface address. In VPRN services where all the interfaces are EVPN tunnels, ICMP packets fail until an IP address is configured. The following output shows a traceroute from VPRN 30 in PE-1 to CE-6 and from PE-2 to CE-1 (see EVPN-VXLAN in EVPN-tunnel R-VPLS services):

*A:PE-1# traceroute router 30 172.16.6.6
traceroute to 172.16.6.6, 30 hops max, 40 byte packets
  1  0.0.0.0  * * *
  2  0.0.0.0  * * *
  3  172.16.6.254 (172.16.6.254)    4.98 ms  4.77 ms  4.97 ms
  4  172.16.6.6 (172.16.6.6)    7.64 ms  4.88 ms  5.14 ms
*A:PE-2# traceroute router 30 172.16.1.1
traceroute to 172.16.1.1, 30 hops max, 0 byte packets
No route to destination. Address: 172.16.1.1, Service: 30

When troubleshooting R-VPLS services, specifically R-VPLS services configured as EVPN tunnels, the limit of peer PEs per EVPN tunnel service is much higher than for a regular R-VPLS service because the egress <VTEP, VNI> bindings do not have to be added to the multicast flooding list. For this reason, the following tools dump command has been added to check the consumed/total EVPN tunnel next hops. The number of EVPN tunnel next hops matches the number of remote GW MAC addresses per EVPN tunnel R-VPLS service.

*A:PE-1# tools dump service id 501 evpn usage 

Evpn Tunnel Interface IP Next Hop: 2/8189

Finally, when troubleshooting EVPN routes and routing policies, the show router bgp routes evpn command and its filters can help:

  • Check that the expected routes are received, properly imported, and communities/tags added/replaced/removed.

  • Check that the expected routes are sent, properly exported, and communities added/replaced/removed.

Examples of EVPN IP prefix routes including communities and tags are the following.

*A:PE-2# show router bgp routes evpn ?
  - evpn <evpn-type>
 
      auto-disc       - Display BGP EVPN Auto-Disc Routes
      eth-seg         - Display BGP EVPN Eth-Seg Routes
      incl-mcast      - Display BGP EVPN Inclusive-Mcast Routes
      ip-prefix       - Display BGP EVPN IPv4-Prefix Routes
      ipv6-prefix     - Display BGP EVPN IPv6-Prefix Routes
      mac             - Display BGP EVPN Mac Routes
      mcast-join-syn* - Display BGP EVPN Mcast Join Sync Routes
      mcast-leave-sy* - Display BGP EVPN Mcast Leave Sync Routes
      smet            - Display BGP EVPN Smet Routes
      spmsi-ad        - Display BGP EVPN Spmsi AD Routes
*A:PE-2# show router bgp routes evpn ip-prefix ?
  - ip-prefix [hunt|detail] [rd <rd>] [prefix <ip-prefix/ip-prefix-length>] 
      [community <comm-id>] [tag <tag>] [next-hop <next-hop>] [aspath-regex <reg-exp>]
 
 <hunt|detail>        : keywords
 <rd>                 : {<ip-addr:comm-val>|
                        <2byte-asnumber:ext-comm-val>|
                        <4byte-asnumber:comm-val>}
 <ip-prefix/ip-pref*> : ip-address     - a.b.c.d (host bits must be 0)
                        mask           - [0..32]
 <comm-id>            : <as-number1:comm-val1>|<ext-comm>|
                        <well-known-comm>
                        ext-comm        - <type>:{<ip-address:comm-val1>|
                                                  <as-number1:comm-val2>|
                                                  <as-number2:comm-val1>}
                        as-number1      - [0..65535]
                        comm-val1       - [0..65535]
                        type            - target|origin
                        ip-address      - a.b.c.d
                        comm-val2       - [0..4294967295]
                        as-number2      - [0..4294967295]
                        well-known-comm - null|no-export|no-export-subconfed|
                                          no-advertise
 <tag>                : [0..4294967295] | MAX-ET
 <next-hop>           : ipv4-address   - a.b.c.d
                        ipv6-address   - x:x:x:x:x:x:x:x   (eight 16-bit pieces)
                                         x:x:x:x:x:x:d.d.d.d
                                         x - [0..FFFF]H
                                         d - [0..255]D
 <reg-exp>            : [80 chars max]

Routing policy "vsi-export-policy-502" adds community "origin:2:11 target:64500:502" to the outgoing routes, as can be verified as follows:

*A:PE-2# show router bgp routes evpn ip-prefix hunt prefix 172.16.1.0/24
===============================================================================
 BGP Router ID:192.0.2.2        AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
---snip---
-------------------------------------------------------------------------------
RIB Out Entries
-------------------------------------------------------------------------------
Network        : n/a
Nexthop        : 192.0.2.2
Path Id        : None
To             : 192.0.2.1
Res. Nexthop   : n/a
Local Pref.    : 100                    Interface Name : NotAvailable
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : None
AIGP Metric    : None                   IGP Cost       : n/a
Connector      : None
Community      : origin:2:11 target:64500:502
                 mac-nh:02:13:ff:00:01:33 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 192.0.2.1
Origin         : IGP
AS-Path        : No As-Path
EVPN type      : IP-PREFIX
ESI            : n/a
Tag            : 0
Gateway Address: 02:13:ff:00:01:33
Prefix         : 172.16.1.0/24
Route Dist.    : 192.0.2.2:502
MPLS Label     : VNI 502
Route Tag      : 0
Neighbor-AS    : n/a
Orig Validation: N/A
Source Class   : 0                      Dest Class     : 0
---snip---

On PE-2, policy "add-tag_to_bgp-evpn_routes" adds route tag 2 to all BGP EVPN routes, as can be verified in the following output:

*A:PE-2# show router bgp routes evpn ip-prefix prefix 172.16.1.0/24 detail 
===============================================================================
 BGP Router ID:192.0.2.2       AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete
 
===============================================================================
BGP EVPN IP-Prefix Routes
===============================================================================
Original Attributes
  
Network        : n/a
Nexthop        : 192.0.2.1
Path Id        : None                   
From           : 192.0.2.1
Res. Nexthop   : 192.168.12.1
Local Pref.    : 100                    Interface Name : int-PE-2-PE-1
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : None
AIGP Metric    : None                   IGP Cost       : 10
Connector      : None
Community      : target:64500:201 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 192.0.2.1
Flags          : Used Valid Best IGP 
Route Source   : Internal
AS-Path        : No As-Path
EVPN type      : IP-PREFIX              
ESI            : n/a
Tag            : 0                      
Gateway Address: 172.16.0.1
Prefix         : 172.16.1.0/24
Route Dist.    : 192.0.2.1:201          
MPLS Label     : VNI 201                
Route Tag      : 0                      
Neighbor-AS    : n/a
Orig Validation: N/A                    
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 00h04m30s              
  
Modified Attributes
  
Network        : n/a
Nexthop        : 192.0.2.1
Path Id        : None                   
From           : 192.0.2.1
Res. Nexthop   : 192.168.12.1
Local Pref.    : 100                    Interface Name : int-PE-2-PE-1
Aggregator AS  : None                   Aggregator     : None
Atomic Aggr.   : Not Atomic             MED            : None
AIGP Metric    : None                   IGP Cost       : 10
Connector      : None
Community      : target:64500:201 bgp-tunnel-encap:VXLAN
Cluster        : No Cluster Members
Originator Id  : None                   Peer Router Id : 192.0.2.1
Flags          : Used Valid Best IGP 
Route Source   : Internal
AS-Path        : No As-Path
EVPN type      : IP-PREFIX              
ESI            : n/a
Tag            : 0                      
Gateway Address: 172.16.0.1
Prefix         : 172.16.1.0/24
Route Dist.    : 192.0.2.1:201          
MPLS Label     : VNI 201                
Route Tag      : 2                      
Neighbor-AS    : n/a
Orig Validation: N/A                    
Source Class   : 0                      Dest Class     : 0
Add Paths Send : Default                
Last Modified  : 00h04m30s              
  
-------------------------------------------------------------------------------
---snip---

Conclusion

SR OS supports not only the EVPN control plane for VXLAN tunnels in Layer 2 applications but also the simultaneous use of EVPN and VXLAN for VPN customers (tenants) with intra and inter-subnet connectivity requirements. R-VPLS services can be configured to provide default gateway connectivity to hosts, IRB backhaul connectivity to VPRN services, and EVPN tunnel connectivity to VPRN services.

When configured to do so, EVPN can advertise IP prefixes and interact with the VPRN RTM to propagate IP prefix connectivity between EVPN and other routing protocols in the VPRN, including IP-VPN. This example has shown how to configure R-VPLS services for all these functions, as well as how to configure routing policies for EVPN-based IP prefixes.