EVPN-VXLAN for layer 3

Check IP-VRF-10 state and connectivity

--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance default protocols bgp routes evpn route-type 5 prefix 22.22.22.22/32 detail
--------------------------------------------------------------------------------------------------
Show report for the EVPN routes in network-instance  "default"
-------------------------------------------------------------------------------------------------
Route Distinguisher: 2.2.2.2:10
Tag-ID             : 0
ip-prefix-len      : 32
ip-prefix          : 22.22.22.22/32
neighbor           : 2.2.2.2
Gateway IP         : 0.0.0.0
Received paths     : 1
  Path 1: <Best,Valid,Used,>
    ESI             : 00:00:00:00:00:00:00:00:00:00
    VNI             : 10
    Route source    : neighbor 2.2.2.2 (last modified 46m16s ago)
    Route preference: No MED, LocalPref is 100
    Atomic Aggr     : false
    BGP next-hop    : 2.2.2.2
    AS Path         :  i
    Communities     : [target:64500:10, mac-nh:00:01:02:ff:00:00, bgp-tunnel-encap:VXLAN]
    RR Attributes   : No Originator-ID, Cluster-List is []
    Aggregation     : None
    Unknown Attr    : None
    Invalid Reason  : None
    Tie Break Reason: none
---------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance default protocols bgp routes evpn route-type 5 prefix 44.44.44.44/32 summary
---------------------------------------------------------------------------------------------------
Show report for the BGP route table of network-instance "default"
--------------------------------------------------------------------------------------------------
Status codes: u=used, *=valid, >=best, x=stale
Origin codes: i=IGP, e=EGP, ?=incomplete
-------------------------------------------------------------------------------------------------
BGP Router ID: 3.3.3.3      AS: 3333      Local AS: 3333
------------------------------------------------------------------------------------------------
Type 5 IP Prefix Routes
+--------+---------------+-----+----------------+----------+----------+-----+---------+
| Status |   Route-      | Tag | IP-address     | neighbor | Next-Hop | VNI | Gateway |
|        | distinguisher | -ID |                |          |          |     |         |
+========+===============+=====+================+==========+==========+=====+=========+
| u*>    | 4.4.4.4:10    | 0   | 44.44.44.44/32 | 4.4.4.4  | 4.4.4.4  | 10  | 0.0.0.0 |
+--------+---------------+-----+----------------+----------+----------+-----+---------+
---------------------------------------------------------------------------------------
1 IP Prefix routes 1 used, 1 valid
---------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]-- 

Check for VXLAN tunnel creation

--{ [FACTORY] + candidate shared default }--[  ]--
# show network-instance default tunnel-table all
-----------------------------------------------------------------------------------------------------
Show report for network instance "default" tunnel table
------------------------------------------------------------------------------------------------------
+-------------+-----------+-------+-------+--------+------------+---------+--------------------------+
| IPv4 Prefix |   Owner   | Type  | Index | Metric | Preference |Fib-prog |       Last Update        |
+=============+===========+=======+=======+========+============+=========+==========================+
| 2.2.2.2/32  | vxlan_mgr | vxlan | 7     | 0      | 0          | Y       | 2021-04-13T10:43:09.483Z |
| 4.4.4.4/32  | vxlan_mgr | vxlan | 6     | 0      | 0          | Y       | 2021-04-13T10:43:09.144Z |
+-------------+-----------+-------+-------+--------+------------+---------+--------------------------+
------------------------------------------------------------------------------------------------------
2 VXLAN tunnels, 2 active, 0 inactive
-----------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------
Show report for network instance "default" tunnel table
-----------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]-- 

Check for remote VTEPs and associated destinations

# show tunnel vxlan-tunnel vtep 2.2.2.2 
----------------------------------------------------------------------------
Show report for vxlan-tunnels vtep
---------------------------------------------------------------------------
VTEP Address: 2.2.2.2
Index       : 202418627561
Last Change : 2021-04-13T11:47:09.000Z
---------------------------------------------------------------------------
Destinations
---------------------------------------------------------------------------
+------------------+-----------------+------------+-----------------------+
| Tunnel Interface | VXLAN Interface | Egress VNI |         Type          |
+==================+=================+============+=======================+
| vxlan1           | 1               | 1          | multicast-destination |
| vxlan2           | 10              | 10         | ip-destination        |
+------------------+-----------------+------------+-----------------------+
---------------------------------------------------------------------------
1 bridged destinations, 1 multicast, 0 unicast, 0 es
1 routed destinations
--------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]--
# show tunnel vxlan-tunnel vtep 4.4.4.4
---------------------------------------------------------------------------
Show report for vxlan-tunnels vtep
---------------------------------------------------------------------------
VTEP Address: 4.4.4.4
Index       : 202418627553
Last Change : 2021-04-14T15:40:23.000Z
--------------------------------------------------------------------------
Destinations
---------------------------------------------------------------------------
+------------------+-----------------+------------+-----------------------+
| Tunnel Interface | VXLAN Interface | Egress VNI |         Type          |
+==================+=================+============+=======================+
| vxlan1           | 1               | 1          | multicast-destination |
| vxlan1           | 1               | 1          | unicast-destination   |
| vxlan2           | 10              | 10         | ip-destination        |
+------------------+-----------------+------------+-----------------------+
---------------------------------------------------------------------------
2 bridged destinations, 1 multicast, 1 unicast, 0 es
1 routed destinations
--------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[  ]-- 

Check IP-VRF-10 route table

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# show route-table ipv4-unicast summary
------------------------------------------------------------------------------------------------
IPv4 Unicast route table of network instance IP-VRF-10
------------------------------------------------------------------------------------------------
+-----------------+-----+--------+------------+--------+------+--------------------+-----------+
|     Prefix      | ID  | Active | Route Type | Metric | Pref |  Next-hop (Type)   | Next-hop  |
|                 |     |        |            |        |      |                    | Interface |
+=================+=====+========+============+========+======+====================+===========+
| 31.31.31.31/32  | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 20.1.1.0/24     | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 20.1.1.3/32     | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 22.22.22.22/32  | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 33.33.33.33/32  | 0   | true   | host       | 0      | 0    | None (extract)     | None      |
| 44.44.44.44/32  | 0   | true   | bgp-evpn   | 0      | 170  | 4.4.4.4 (indirect) | None      |
| 101.1.1.0/24    | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
|                 |     |        |            |        |      | 4.4.4.4 (indirect) | None      |
| 101.1.1.1/32    | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 102.1.1.0/24    | 0   | true   | bgp-evpn   | 0      | 170  | 2.2.2.2 (indirect) | None      |
| 103.1.1.0/24    | 0   | true   | local      | 0      | 0    | 103.1.1.3 (direct) | irb0.3    |
| 103.1.1.1/32    | 0   | true   | arp-nd     | 0      | 1    | 103.1.1.1 (direct) | irb0.3    |
| 103.1.1.3/32    | 0   | true   | host       | 0      | 0    | None (extract)     | None      |
| 103.1.1.254/32  | 0   | true   | host       | 0      | 0    | None (extract)     | None      |
| 103.1.1.255/32  | 0   | true   | host       | 0      | 0    | None (broadcast)   | None      |
| 104.1.1.0/24    | 0   | true   | bgp-evpn   | 0      | 170  | 4.4.4.4 (indirect) | None      |
+-----------------+-----+--------+------------+--------+------+--------------------+-----------+
------------------------------------------------------------------------------------------------
15 IPv4 routes total
15 IPv4 prefixes with active routes
1 IPv4 prefixes with active ECMP routes
-----------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--

Check route-table state for a RT5

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# info from state route-table ipv4-unicast route 101.1.1.0/24 id 0 
    route-table {
        ipv4-unicast {
            route 101.1.1.0/24 id 0 {
                route-type bgp-evpn
                route-owner bgp_evpn_mgr
                metric 0
                preference 170
                active true
                last-app-update "a day ago"
                next-hop-group 202418627581
                resilient-hash false
                fib-programming {
                    status success
                }
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# info from state route-table next-hop-group 202418627581 next-hop *
    route-table {
        next-hop-group 202418627581 {
            next-hop 0 {
                next-hop 202418627569
                resolved true
            }
            next-hop 1 {
                next-hop 202418627565
                resolved true
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# info from state route-table next-hop 202418627569 
    route-table {
        next-hop 202418627569 {
            type indirect
            ip-address 2.2.2.2
            resolving-tunnel {
                ip-prefix 2.2.2.2/32
                tunnel-type vxlan
                tunnel-owner vxlan_mgr
            }
            vxlan {
                vni 10
                source-mac 00:01:03:FF:00:00
                destination-mac 00:01:02:FF:00:00
            }
        }
    }

Monitor pings

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
// a ping is sent from Leaf-3 to 22.22.22.22 (the loopback on Leaf-2’s IP-VRF-10)
# network-instance IP-VRF-10

--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]--
# ping 22.22.22.22 
Using network instance IP-VRF-10
PING 22.22.22.22 (22.22.22.22) 56(84) bytes of data.
64 bytes from 22.22.22.22: icmp_seq=1 ttl=64 time=4.88 ms
64 bytes from 22.22.22.22: icmp_seq=2 ttl=64 time=4.76 ms
^C
--- 22.22.22.22 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 4.767/4.827/4.888/0.092 ms
// the monitor command catches the inner packet sent on VXLAN
--{ [FACTORY] +! candidate shared default }--[ network-instance IP-VRF-10 ]--
# /tools system traffic-monitor destination-address 33.33.33.33 protocol icmp
Running as user "root" and group "root". This could be dangerous.
Capturing on 'monit'
1       0.000000000     ethernet-1/1    00:01:02:ff:00:00       00:01:03:ff:00:00
         22.22.22.22     33.33.33.33    ICMP    146     Echo (ping) reply    id=0x580c,
         seq=1/256, ttl=64

// the source MAC is the local chassis mac:

--{ [FACTORY] + candidate shared default }--[  ]-- 
# info from state platform chassis mac-address
    platform {
        chassis {
            mac-address 00:01:03:FF:00:00
        }
    }

// that source MAC is also advertised in the RT5’s mac-nh

--{ [FACTORY] + candidate shared default }--[  ]--
# info from state network-instance default bgp-rib evpn rib-in-out rib-out-post ip-prefix-routes 3.3.3.3:10 ethernet-tag-id 0 ip-prefix-length 32 ip-prefix 33.33.33.33/32 neighbor 2.2.2.2
    network-instance default {
        bgp-rib {
            evpn {
                rib-in-out {
                    rib-out-post {
                        ip-prefix-routes 3.3.3.3:10 ethernet-tag-id 0 ip-prefix-length 32
                        ip-prefix 33.33.33.33/32 neighbor 2.2.2.2 {
                            esi 00:00:00:00:00:00:00:00:00:00
                            gateway-ip 0.0.0.0
                            vni 10
                            attr-id 132
                            next-hop 3.3.3.3
                        }
                    }
                }
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[  ]--
# info from state network-instance default bgp-rib attr-sets attr-set rib-out index 132
    network-instance default {
        bgp-rib {
            attr-sets {
                attr-set rib-out index 132 {
                    origin igp
                    atomic-aggregate false
                    next-hop 3.3.3.3
                    med 0
                    local-pref 100
                    aggregator {
                    }
                    pmsi-tunnel {
                    }
                    communities {
                        ext-community [
                            target:64500:10
                            mac-nh:00:01:03:ff:00:00
                            bgp-tunnel-encap:VXLAN
                        ]
                    }
                    unknown-attributes {
                    }
                }
            }
        }
    }

The received EVPN IFL IP Prefix routes are only installed in the IP-VRF-10 route-table if:

• the import route-target matches the RT5 route-target

• the RT5's VNI matches the VNI of the vxlan-interface in the IP-VRF-10

• the RT5's gateway-ip is zero

Additional guidelines:

• Importing an RT5 into multiple ip-vrf network-instances is not supported due to the VNI restriction: an ip-vrf can only use a single VNI for ingress and egress VXLAN packets. This is a TD3 limitation.

• The route next-hop cannot be changed in ip-vrf network-instances. It is always the system-ip in this release.

• The ip-vrf bgp-evpn bgp instance can be oper-down for the same reasons as the bgp-evpn bgp instance can be down in mac-vrfs. See EVPN-VXLAN for layer-2 and multi-homing for details.

• VXLAN statistics are also accounted for when EVPN-IFL is used.

• No MTU checks are done for VXLAN in EVPN-IFL. If the routed packet plus the VXLAN overhead exceeds the underlay interface MTU of the egress interface in the default network-instance, the packet is still encapsulated and sent to the remote leaf. No statistics increment or drops occur.

• Outer TTL for VXLAN packets is always initialized to 255 and not copied or propagated from or to the inner IP packet.

Check PE-CE routing on IP-VRF

// IP-VRF-10 in Leaf-2
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF* ]--
# info 
    network-instance IP-VRF-10 {
        type ip-vrf
        interface irb0.2 {
        }
        interface irb0.24 {
        }
        interface lo10.2 {
        }
        vxlan-interface vxlan2.10 {
        }
        protocols {
            bgp-evpn {
                bgp-instance 1 {
                    vxlan-interface vxlan2.10
                    evi 10
                    ecmp 2
                }
            }
            bgp {
                admin-state enable
                afi-safi ipv4-unicast {
                    admin-state enable
                }
                autonomous-system 645002
                router-id 2.2.2.2
                group eBGP-PE-CE {
                    admin-state enable
                    export-policy export-all
                    import-policy import-all
                    afi-safi ipv4-unicast {
                        admin-state enable
                    }
                }
                neighbor 20.1.1.3 {
                    peer-as 645003
                    peer-group eBGP-PE-CE
                    local-as as-number 645002 {
                    }
                    transport {
                        local-address 20.1.1.2
                    }
                }
            }
            bgp-vpn {
                bgp-instance 1 {
                    route-target {
                        export-rt target:64500:10
                        import-rt target:64500:10
                    }
                }
            }
        }
    }
--{ [FACTORY] + candidate shared default }--[ network-instance IP-VRF-10 ]-- 
# show protocols bgp neighbor 20.1.1.3 
---------------------------------------------------------------------------------------------------
BGP neighbor summary for network-instance "IP-VRF-10"
Flags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow
--------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------
+-----------+----------+------------+--------+---------+------------+---------+---------+------------+
|  Net-Inst |    Peer  |   Group    |  Flags | Peer-AS |   State    |  Uptime | AFI/    | [Rx/Active |
|           |          |            |        |         |            |         | SAFI    |    /Tx]    |
+===========+============+==========+========+=========+============+=========+======================+
| IP-VRF-10 | 20.1.1.3 | eBGP-PE-CE | S      | 645003  | established| 1d:21h: | ipv4-   | [2/1/11]   |
|           |          |            |        |         |            | 46m:56s | unicast |            |
+------------+----------+-----------+--------+---------+------------+---------+---------+------------+
-----------------------------------------------------------------------------------------------------
1 configured neighbors, 1 configured sessions are established,0 disabled peers
0 dynamic peers