EVPN-VXLAN for layer-2 and multi-homing
Ethernet Virtual Private Network (EVPN) is a standard technology in multi-tenant Data Centers (DCs). EVPN provides a control frame framework for many functions. This chapter details the configuration and operation of an EVPN and VXLAN (EVPN-VXLAN) solution for the following components:
Bridged sub-Interfaces associated with a specific vlan-id or default sub-interfaces, that capture untagged and non-explicitly configured vlan-tagged frames in tagged sub-interfaces.
MAC-VRF type network-instances that are EVPN-enabled so that they can use Virtual Extended LAN (VXLAN) tunnels to connect to other MAC-VRFs of the same Broadcast Domains (BD).
EVPN-VXLAN control and data plane extensions as in [RFC8365], including EVPN route type 2 (MAC/IP) and route type 2 (Inclusive Multicast Ethernet Tag [IMET]).
Distributed security and protection for static-macs.
The MAC duplication mechanism, extended to support EVPN, to provide loop protection.
EVPN L2 multi-homing, including Ethernet Segment (ES) model configuration for all-active multi-homing.
Applicability
The information and configuration in this chapter are based on SR Linux Release 21.3.
Overview
EVPN-VXLAN provides Layer-2 connectivity in multi-tenant DCs. EVPN-VXLAN Broadcast Domains (BD) can span several leaf routers connected to the same IP fabric, allowing hosts attached to the same BD to communicate as though they were connected to the same layer-2 switch. VXLAN tunnels bridge the layer-2 frames between leaf routers with EVPN providing the control plane to automatically setup tunnels and use them efficiently.
The following diagram shows this concept. In this example, four leaf routers are attached to the same BD that is instantiated by a MAC-VRF on each leaf. SR Linux leaf routers support standard-based EVPN-VXLAN [RFC8365]; therefore third-party leaf routers (LEAF-4 in this example) can be attached to the same BD as the SR Linux leaf routers as long as they follow standard [RFC8365].
Configuration of EVPN-VXLAN broadcast domains
The following figure shows a configuration example of an EVPN-VXLAN BD that is distributed in multiple leaf nodes in the same DC. The BD is instantiated by MAC-VRF-1 in each of the three leaf nodes. The sections that follow describe how to configure and operate MAC-VRF-1 in each node.
Configuring the underlay network
-
ISIS
-
OSPF
-
eBGP
The EVPN family must also be enabled for the distribution of EVPN routes among leaf routers of the same tenant. EVPN is enabled using iBGP and typically a Route Reflector (RR), or eBGP.
As an example, the following configuration on LEAF-3 shows an eBGP-underlay BGP group to enable the IPv4 and IPv6 unicast families, and an iBGP-evpn group for the distribution of the EVPN routes. A full mesh of iBGP EVPN sessions is established among the three leaf routers, but a pair of RRs is typical.
Underlay Configuration
--{ [FACTORY] + candidate shared default }--[ network-instance default ]--
A:dut3# info
type default
admin-state enable
description "Default network instance"
router-id 3.3.3.3
interface ethernet-1/1.1 {
}
interface ethernet-1/3.1 {
}
interface system0.0 {
}
protocols {
bgp {
admin-state enable
autonomous-system 3333
router-id 3.3.3.3
group eBGP-underlay {
admin-state enable
export-policy export-all
import-policy import-all
timers {
connect-retry 5
hold-time 5
keepalive-interval 2
minimum-advertisement-interval 2
}
}
group iBGP-evpn {
admin-state enable
export-policy export-all
import-policy import-all
evpn {
admin-state enable
}
local-as 1234 {
}
timers {
minimum-advertisement-interval 1
}
}
ipv4-unicast {
admin-state enable
}
ipv6-unicast {
admin-state enable
}
neighbor 1.1.1.1 {
admin-state enable
peer-as 1234
peer-group iBGP-evpn
transport {
local-address 3.3.3.3
}
}
neighbor 2.2.2.2 {
admin-state enable
peer-as 1234
peer-group iBGP-evpn
transport {
local-address 3.3.3.3
}
}
neighbor 4.4.4.4 {
admin-state enable
peer-as 1234
peer-group iBGP-evpn
transport {
local-address 3.3.3.3
}
}
neighbor 10.2.3.2 {
admin-state enable
peer-as 2222
peer-group eBGP-underlay
}
neighbor 10.3.4.4 {
admin-state enable
peer-as 4444
peer-group eBGP-underlay
}
trace-options {
flag packets {
modifier detail
}
flag update {
modifier detail
}
flag route {
modifier detail
}
flag socket {
modifier detail
}
flag notification {
modifier detail
}
}
}
linux {
export-routes true
export-neighbors true
}
}
In the example above, eBGP is used for underlay reachability, and iBGP for overlay EVPN route distribution. The command local-as overrides the configuration of the bgp>autonomous-system so that the overlay BGP sessions are established using the same autonomous system in the three leaf routers.
The system0.0 interface hosts the loopback address used to originate and typically terminate VXLAN packets. This address is also used by default as the next-hop of all EVPN routes.
system0.0 interface configuration
The following example shows the configuration of the system0.0 interface in LEAF-3.
--{ [FACTORY] + candidate shared default }--[ interface system0 ]--
A:dut3# info
admin-state enable
subinterface 0 {
admin-state enable
ipv4 {
address 3.3.3.3/32 {
}
}
}
Configuring LEAF-3 with an EVPN-VXLAN enabled MAC-VRF
After LEAF-3 is configured as defined in Configuring the underlay network, use the following steps to enable EVPN-VXLAN on LEAF-3.
In this example, Ethernet-1/2 connects HOST-3 to LEAF-3. Although this interface could be defined untagged, this example configures the interface as tagged and using vlan-id (vlan-tagging true).
A subinterface with index 1 is created under the interface. The subinterface must be configured as type bridged. Bridged subinterfaces can be associated with MAC-VRF instances so that MAC learning and layer-2 forwarding can be enabled on each.
-
In candidate mode, create the interfaces and bridged subinterfaces to connect
LEAF-3 to HOST-3.
creation of interfaces/bridged subinterfaces
--{ [FACTORY] + candidate shared default }--[ interface ethernet-1/2 ]-- A:dut3# info description dut3_host3 admin-state enable vlan-tagging true subinterface 1 { type bridged admin-state enable vlan { encap { single-tagged { vlan-id 1 } } } }In the above example, the subinterface uses vlan-id 1 since this is the VLAN ID used by HOST-3 to send and receive frames. If you wanted HOST-3 to sent and received untagged traffic, the vlan encap command can be configured with either of these options:
- vlan encap single-tagged vlan-id any - where 'any' captures all traffic when no specific vlan-id has been defined.
- vlan encap untagged - where 'untagged' captures traffic with no tags or vlan-tag 0.
-
After creating the access subinterfaces, create the vxlan-interfaces.
This allows MC-VRFs of the same BD to be connected throughout the IP fabric.The SR Linux models VXLAN as tunnels and vxlan-interfaces exist within them. The network-instance and main property is the VNI or VXLAN network identifier. SR Linux VXLAN model characteristics include:
- The tunnel-interface for vxlan is configured as vxlan<N> where the value of N is 0-255.
- Multiple tunnel-interfaces can be configured. The tunnel-interface can host multiple vxlan-interfaces.
- vxlan-interfaces are configured under tunnel-interfaces with an associated number in the range 0-4294967295. Minimally, the vxlan-interface must have an index, type, and ingress VNI.
- A vxlan-interface can only be associated with one network-instance, and in the R21.3, a network-instance can have only one vxlan-interface.
- The vxlan-interface type can be routed or bridged. When used for EVPN-VXLAN Layer-2 in MAC-VRFs, the type must be "bridged".
- The ingress VNI must be configured. The VNI is used to find the MAC-VRF where the inner MAC lookup is performed. The egress VNI is not configured and is determined by the imported EVPN routes. SR Linux requires that the egress VNI (discovered) matches the configured ingress VNI so that two leaf routers attached to the same BD can exchange packets.
vxlan1 vxlan-interface configuration--{ [FACTORY] + candidate shared default }--[ tunnel-interface * ]-- A:dut3# info tunnel-interface vxlan1 { vxlan-interface 1 { type bridged ingress { vni 1 } } }Outer VLAN tagging is supported (one VLAN tag only), assuming that the egress subinterface in the default network-instance uses vlan-tagging. No inner VLAN tags can be pushed or popped on vxlan-interfaces, but vlan tags that are not stripped-off at the ingress bridged subinterfaces are transparently carried over the VXLAN tunnels.
The following applies for MTU and fragmentation for VXLAN interfaces:
- No specific MTU checks are performed in network-instances with VXLAN.
- The default network-instance interface MTU should be made large enough to allow room for the VXLAN overhead.
- The Don't Fragment (DF) flag is always set in the VXLAN outer IP header.
- Reassembly is not supported for VXLAN packets.
-
Configure the network-instance type mac-vrf and associate it with the bridged
interfaces and vxlan-interface to.
A bgp-evpn enabled mac-vrf requires the association of at least one bridged subinterface and one bridged vxlan-interface.mac-vrf configuration and bridged interface association
--{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 ]-- A:dut3# info type mac-vrf admin-state enable interface ethernet-1/2.1 { } vxlan-interface vxlan1.1 { } -
Enable EVPN in the mac-vrf by configuring the bgp-vpn and the bgp-evpn protocol
containers:
- bgp-vpn - Provides the configuration of the bgp-instances where the route-distinguisher and the import/export route-targets used for the EVPN routes exist. Import and export policies can be used instead of explicit route-targets. In the current release, only one bgp-instance per network-instance is supported.
- bgp-evpn - Hosts all the commands required to enable EVPN in the
network-instance. At a minimum, a reference to bgp-instance 1 is
configured, along with the reference to the vxlan-interface (where EVPN
is enabled) and the EVI. The EVI or EVPN Instance identifier is a
two-byte value that is mandatory, and is used for:
-
The auto-derivation of the route-distinguisher (RD). If a manual RD is not configured, the RD is auto-derived as system-ip:evi. Where the system-ip is the IP address configured in the system0.0 subinterface.
-
The auto-derivation of the route-target (RT). If a manual RT is not configured, the RT is auto-derived as autonomous-system:evi. The autonomous-system is configured in the default network-instance.
-
The value used to represent the MAC-VRF in the DF Election algorithm. See Multi-homing configuration for EVPN broadcast domains.
-
Configure bgp-vpn and bgp-evpn protocol containers--{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 ]-- A:dut3# info type mac-vrf admin-state enable interface ethernet-1/2.1 { } vxlan-interface vxlan1.1 { } protocols { bgp-evpn { bgp-instance 1 { admin-state enable vxlan-interface vxlan1.1 evi 1 ecmp 2 } } bgp-vpn { bgp-instance 1 { route-target { export-rt target:1234:1 import-rt target:1234:1 } } } }Each leaf routers is configured with a different autonomous-system number. Therefore, EVI-based auto-derived RTs cannot be used or the three leaf routers would not produce the same import and export route-targets for the MAC-VRFs of the same BD. Therefore, RTs are configured manually.
-
Review the changes. If correct, commit the changes.
A:dut2# commit stay --{ candidate shared default }--[ ]--
Checking the EVPN-VXLAN operation in MAC-VRFs
After all leaf routers attached to the same BD are configured, the state of the MAC-VRF and connectivity to LEAF-2 and LEAF-3 should be checked.
Checking vxlan-interface configuration
The following command checks that the vxlan-interface is properly configured and associated with the network-instance. If the network-instance vxlan-interface is oper-down, a reason is shown. The egress source-ip shown in the command should match the IPv4 address configured in the subinterface system0.0.
Check vxlan-interface configuration
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show tunnel-interface vxlan1 vxlan-interface 1 brief
---------------------------------------------------------------------------------
Show report for vxlan-tunnels
---------------------------------------------------------------------------------
+------------------+-----------------+---------+-------------+------------------+
| Tunnel Interface | VxLAN Interface | Type | Ingress VNI | Egress source-ip |
+==================+=================+=========+=============+==================+
| vxlan1 | vxlan1.1 | bridged | 1 | 3.3.3.3/32 |
+------------------+-----------------+---------+-------------+------------------+
--------------------------------------------------------------------------------
Summary
1 tunnel-interfaces, 1 vxlan interfaces
5 vxlan-destinations, 2 unicast, 1 es, 2 multicast, 0 ip
------------------------------------------------------------
A:dut3# show network-instance MAC-VRF-1 vxlan-interface vxlan1.1
------------------------------------------------------------------------------------
Show report for network instance "MAC-VRF-1" VxLAN interface table
------------------------------------------------------------------------------------
====================================================================================
Network-instance: MAC-VRF-1
VxLAN-Interface : vxlan1.1
Type : bridged
Oper state : up
Oper-down-reason: None
====================================================================================
Checking mac-vrf, bgp-vpn, and bgp-evpn parameters
The following command checks that the mac-vrf, bgp-vpn, and bgp-evpn parameters are properly configured. A manual or auto-derived RD/RT must exist, or the bgp-evpn bgp-instance will be oper-down.
Check mac-vrf, bgp-vpn, and bgp-evpn parameters
A:dut3# show network-instance MAC-VRF-1 protocols bgp-vpn bgp-instance 1
===================================================================================
Net Instance : MAC-VRF-1
bgp Instance 1
-----------------------------------------------------------------------------------
route-distinguisher: 3.3.3.3:1, auto-derived-from-evi
export-route-target: target:1234:1, manual
import-route-target: target:1234:1, manual
====================================================================================
================================================================================
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show network-instance MAC-VRF-1 protocols bgp-evpn bgp-instance 1
====================================================================================
Net Instance : MAC-VRF-1
bgp Instance 1 is enabled and None
------------------------------------------------------------------------------------
VXLAN-Interface : vxlan1.1
evi : 1
ecmp : 2
default-admin-tag : 0
oper-down-reason : N/A
EVPN Routes
Next hop : None
MAC/IP Routes : None
IMET Routes : None, originating-ip None
====================================================================================
--{ [FACTORY] + candidate shared default }--[ ]--
Checking VXLAN tunnels
This example checks for the creation of VXLAN tunnels to the remote VTEPs. After receiving EVPN routes from the remote leaf routers with VXLAN encapsulation, the vxlan_mgr creates VTEPs from the EVPN routes next-hops. Each VTEP gets an index allocated by fib_mgr (per source and destination tunnel IP addresses) if the next-hop is resolved in the default network-instance. The state of the two remote VTEPs is shown with their own indexes. EVPN routes are received with Next Hops 2.2.2.2 and 4.4.4.4 respectively.
Check VXLAN tunnels
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show tunnel vxlan-tunnel all
---------------------------------------------------------------------------------
Show report for vxlan-tunnels
--------------------------------------------------------------------------------
+--------------+--------------+--------------------------+
| VTEP Address | Index | Last Change |
+==============+==============+==========================+
| 2.2.2.2 | 278779228830 | 2021-02-15T11:07:23.000Z |
| 4.4.4.4 | 278779228829 | 2021-02-15T18:15:28.000Z |
+--------------+--------------+--------------------------+
2 VXLAN tunnels, 2 active, 0 inactive
-----------------------------------------------------------
Checking tunnel-table entries
The following command displays all tunnel-table entries. When a VTEP is created in the vxlan-tunnel table and a non-zero index is allocated, a tunnel-table entry is created in the tunnel-table of the default network-instance.
If the next hop is not resolved to a route in the default network-instance route-table, the index in the vxlan-tunnel table shows as ‟0” for the VTEP and no tunnel-table is created. If the tunnel prefix in the tunnel-table is resolved, but the system runs out of hardware index resources, the tunnel shows in the tunnel-table, but is not programmed. A non-programmed-reason is displayed.
Check tunnel-table entries
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show network-instance default tunnel-table all
-------------------------------------------------------
Show report for network instance "default" tunnel table
------------------------------------------------------------------------------------------------------
+-------------+-----------+-------+-------+--------+------------+----------+-------------------------+
| IPv4 Prefix | Owner | Type | Index | Metric | Preference | Fib-prog | Last Update |
+=============+===========+=======+=======+========+============+==========+=========================+
| 2.2.2.2/32 | vxlan_mgr | vxlan | 4 | 0 | 0 | Y | 2021-03-01T10:41:38.590Z|
| 4.4.4.4/32 | vxlan_mgr | vxlan | 5 | 0 | 0 | Y | 2021-03-01T10:45:08.633Z|
+-------------+-----------+-------+-------+--------+------------+----------+-------------------------+
-----------------------------------------------------------------------------------------------------
2 VXLAN tunnels, 2 active, 0 inactive
-----------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------
Show report for network instance "default" tunnel table
-----------------------------------------------------------------------------------------------------
Checking statistics
When the three leaf routers exchange packets over the VXLAN, LEAF-3 displays statistics for all individual VTEPs. Statistics include:
-
Global-level ingress/egress packets and octets. Global in/out octets and packets are aggregations of the individual statistics per VTEP. ‟in-discarded-packets” are vxlan packets discarded as a result of a non-existent local VNI, packets from a source VTEP are not discovered in the control plan, and packets are not aggregations of individual per VTEP dropped packets
-
Per VTEP packets and octets with in/out discarded packets.
Check statistics
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# info from state tunnel vxlan-tunnel
tunnel {
vxlan-tunnel {
vtep 2.2.2.2 {
index 278779228830
last-change "8 hours ago"
statistics {
in-octets 0
in-packets 0
in-discarded-packets 0
out-octets 0
out-packets 0
out-discarded-packets 0
}
}
vtep 4.4.4.4 {
index 278779228829
last-change "an hour ago"
statistics {
in-octets 555720
in-packets 5052
in-discarded-packets 0
out-octets 0
out-packets 0
out-discarded-packets 0
}
}
statistics {
in-octets 555720
in-packets 5052
in-discarded-packets 0
out-octets 0
out-packets 0
}
}
}
Statistics can be cleared using the command: tools tunnel vxlan-tunnel vtep 2.2.2.2 statistics clear
Checking for received IMET routes and multicast destination creation
IMET routes are used for auto-discovery and the creation of the default flood list for vxlan in the MAC-VRF. When LEAF-3 receives and imports the IMET routes from LEAF-2 and LEAF-4, it creates a VXLAN default flood list. BUM frames received on a bridged subinterface are ingress-replicated to the VTEPs on the list.
The following command checks that the IMET routes for the BD from LEAF-2 and LEAF-4 have been received and they have created multicast destinations in the MAC-VRF. Note that the VNI is received in the PMSI tunnel attribute and not in the route’s Network Layer Reachability Information (NLRI).
Check for received IMET routes and multicast destination creation
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show network-instance default protocols bgp routes evpn route-type 3 detail
------------------------------------------------------------------------------------
Show report for the EVPN routes to network "*" network-instance "default"
------------------------------------------------------------------------------------
Route Distinguisher: 2.2.2.2:1
Tag-ID : 0
Originating router : 2.2.2.2
neighbor : 2.2.2.2
Received paths : 1
Path 1: <Best,Valid,Used,>
VNI : 1
Route source : neighbor 2.2.2.2 (last modified 9h11m44s ago)
Route preference: No MED, LocalPref is 100
Atomic Aggr : false
BGP next-hop : 2.2.2.2
AS Path : i
Communities : [target:1234:1, bgp-tunnel-encap:VXLAN]
RR Attributes : No Originator-ID, Cluster-List is []
Aggregation : None
Unknown Attr : None
Invalid Reason : None
Tie Break Reason: none
------------------------------------------------------------------------------------
Route Distinguisher: 4.4.4.4:1
Tag-ID : 0
Originating router : 4.4.4.4
neighbor : 4.4.4.4
Received paths : 1
Path 1: <Best,Valid,Used,>
VNI : 1
Route source : neighbor 4.4.4.4 (last modified 9h11m44s ago)
Route preference: No MED, LocalPref is 100
Atomic Aggr : false
BGP next-hop : 4.4.4.4
AS Path : i
Communities : [target:1234:1, bgp-tunnel-encap:VXLAN]
RR Attributes : No Originator-ID, Cluster-List is []
Aggregation : None
Unknown Attr : None
Invalid Reason : None
Tie Break Reason: none
------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# info from state network-instance default bgp-rib evpn rib-in-out rib-in-post imet-routes * originating-router * ethernet-tag-id * neighbor *
network-instance default {
bgp-rib {
evpn {
rib-in-out {
rib-in-post {
imet-routes 2.2.2.2:1 originating-router 2.2.2.2 ethernet- tag-id 0 neighbor 2.2.2.2 {
attr-id 193
last-modified 2021-02-15T11:07:21.500Z
used-route true
valid-route true
best-route true
stale-route false
pending-delete false
tie-break-reason none
invalid-reason {
rejected-route false
as-loop false
next-hop-unresolved false
cluster-loop false
}
}
imet-routes 4.4.4.4:1 originating-router 4.4.4.4 ethernet-tag-id 0 neighbor 4.4.4.4 {
attr-id 197
last-modified 2021-02-15T11:07:21.500Z
used-route true
valid-route true
best-route true
stale-route false
pending-delete false
tie-break-reason none
invalid-reason {
rejected-route false
as-loop false
next-hop-unresolved false
cluster-loop false
}
}
}
}
}
}
}
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# info from state network-instance default bgp-rib attr-sets attr-set rib-in index {193,197}
network-instance default {
bgp-rib {
attr-sets {
attr-set rib-in index 193 {
origin igp
atomic-aggregate false
next-hop 2.2.2.2
med 0
local-pref 100
aggregator {
}
pmsi-tunnel {
tunnel-type ingress-replication
vni 1
tunnel-endpoint 2.2.2.2
}
communities {
ext-community [
target:1234:1
bgp-tunnel-encap:VXLAN
]
}
unknown-attributes {
}
}
attr-set rib-in index 197 {
origin igp
atomic-aggregate false
next-hop 4.4.4.4
med 0
local-pref 100
aggregator {
}
pmsi-tunnel {
tunnel-type ingress-replication
vni 1
tunnel-endpoint 4.4.4.4
}
communities {
ext-community [
target:1234:1
bgp-tunnel-encap:VXLAN
]
}
unknown-attributes {
}
}
}
}
}
Checking for multicast-destinations
If the IMET routes from LEAF-2 and LEAF-4 are imported for MAC-VRF-1, the corresponding multicast VXLAN destinations are added and can be checked with the following command:
Check for multicast-destinations
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show tunnel-interface vxlan1 vxlan-interface 1 bridge-table multicast-destinations destination *
------------------------------------------------------------------------------------
Show report for vxlan-interface vxlan1.1 multicast destinations (flooding-list)
------------------------------------------------------------------------------------
+--------------+------------+-------------------+----------------------+
| VTEP Address | Egress VNI | Destination-index | Multicast-forwarding |
+==============+============+===================+======================+
| 2.2.2.2 | 1 | 278779228840 | BUM |
| 4.4.4.4 | 1 | 278779228838 | BUM |
+--------------+------------+-------------------+----------------------+
------------------------------------------------------------------------
Summary
2 multicast-destinations
------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking mac-table and MAC/IP routes
When traffic is exchanged between HOST-3 and HOST-12, the MACs are learned on the access bridged sub-interfaces and advertised in MAC/IP routes. The MAC/IP routes are imported, and the MACs programmed in the mac-table. The following command can check the MAC/IP routes and the programmed MACs.
Check mac-table and MAC/IP routes
--{ [FACTORY] + candidate shared default )--[ ]--
A:dut3# show network-instance MAC-VRF-1 bridge-table mac-table all
-----------------------------------------------------------------------------------------------------
Mac-table of network instance MAC-VRF-1
-----------------------------------------------------------------------------------------------------
+--------------+---------------------------+------------+------+--------+-------+-------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+==============+===========================+============+======+========+=======+===================+
| 00:00:00:00: | vxlan-interface:vxlan1.1 | 2787792288 | evpn | true | N/A | 2021-02-16T10:54: |
| 00:01 | esi:01:24:24:24:24:24:24: | 46 | | | | 54.000Z |
| | 00:00:01 | | | | | |
| 00:00:00:00: | vxlan-interface:vxlan1.1 | 2787792288 | evpn | true | N/A | 2021-02-16T10:54: |
| 00:02 | vtep:2.2.2.2 vni:1 | 40 | | | | 54.000Z |
| 00:00:00:00: | ethernet-1/2.1 | 12 |learnt| true | 300 | 2021-02-16T10:54: |
| 00:03 | | | | | | 54.000Z |
| 00:00:00:00: | vxlan-interface:vxlan1.1 | 2787792288 | evpn | true | N/A | 2021-02-16T10:54: |
| 00:04 | vtep:4.4.4.4 vni:1 | 38 | | | | 54.000Z |
+--------------+---------------------------+------------+------+--------+-------+-------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 1 Total 1 Active
Total Evpn Macs : 3 Total 3 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 4 Total 4 Active
---------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show network-instance default protocols bgp routes evpn route-type 2 summary
------------------------------------------------------------------------------------
Show report for the BGP route table of network-instance "default"
------------------------------------------------------------------------------------
Status codes: u=used, *=valid, >=best, x=stale
Origin codes: i=IGP, e=EGP, ?=incomplete
-----------------------------------------------------------------------------------
BGP Router ID: 3.3.3.3 AS: 3333 Local AS: 3333
-----------------------------------------------------------------------------------
Type 2 MAC-IP Advertisement Routes
+------+-----------+-----+--------------+---------+---------+---------+---+-------------------+------+
|Status| Route-dis-| Tag | MAC-address | IP- | Neighbor| Next- |VNI| ESI | MAC |
| | tinguisher| ID | | address | | Hop | | | Mob. |
+======+===========+=====+==============+=========+=========+=========+===+===================+======+
| u*> | 2.2.2.2:1 | 0 | 00:00:00:00: | 0.0.0.0 | 2.2.2.2 | 2.2.2.2 | 1 | 00:00:00:00:00:00 | - |
| | | | 00:02 | | | | | 00:00:00:00 | |
| u*> | 4.4.4.4:1 | 0 | 00:00:00:00: | 0.0.0.0 | 4.4.4.4 | 4.4.4.4 | 1 | 01:24:24:24:24:24 | - |
| | | | 00:01 | | | | | 24:00:00:01 | |
| u*> | 4.4.4.4:1 | 0 | 00:00:00:00: | 0.0.0.0 | 4.4.4.4 | 4.4.4.4 | 1 | 00:00:00:00:00:00:| - |
| | | | 00:04 | | | | | 00:00:00:0 0 | |
+------+-----------+-----+--------------+---------+---------+---------+---+-------------------+------+
------------------------------------------------------------------------------------------------------
3 MAC-IP Advertisement routes 3 used, 3 valid
------------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking unicast destinations
The reception of MAC/IP routes also creates unicast destinations in the vxlan-interface. In some cases, the unicast destinations are Ethernet Segment (ES) destinations if the MAC/IP routes are advertised from an ES. See Multi-homing configuration for EVPN broadcast domains for details. The following command displays the unicast destinations.
Check unicast destinations
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show tunnel-interface vxlan1 vxlan-interface 1 bridge-table unicast-destinations destination *
-----------------------------------------------------------------------------------------------------
Show report for vxlan-interface vxlan1.1 unicast destinations
----------------------------------------------------------------------------------------------------
Destinations
-------------------------------------------------------------------------------
+--------------+------------+-------------------+-----------------------------+
| VTEP Address | Egress VNI | Destination-index | Number MACs (Active/Failed) |
+==============+============+===================+=============================+
| 2.2.2.2 | 1 | 278779228840 | 1(1/0) |
| 4.4.4.4 | 1 | 278779228838 | 1(1/0) |
+--------------+------------+-------------------+-----------------------------+
-------------------------------------------------------------------------------
Ethernet Segment Destinations
------------------------------------------------------------------------------------------------------
+-------------------------------+-------------------+------------------+-----------------------------+
| ESI | Destination-index | VTEPs | Number MACs (Active/Failed) |
+===============================+===================+==================+=============================+
| 01:24:24:24:24:24:24:00:00:01 | | 2.2.2.2, 4.4.4.4 | 0(0/0) |
+-------------------------------+-------------------+------------------+-----------------------------+
------------------------------------------------------------------------------------------------------
Summary
3 unicast-destinations, 2 non-es, 1 es
2 MAC addresses, 2 active, 0 non-active
-----------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking MAC mobility, MAC protection and MAC loop protection in EVPN-VXLAN BDs
MAC mobility and MAC protection are implemented following [RFC7432]. MAC loop protection follows [draft-ietf-bess-rfc7432bis].
Checking MAC mobility
MAC Mobility is an event that triggers the fast move and re-learn of a MAC in a different leaf router. Mobility is common in DCs with some workloads moving between racks in the same DC. EVPN provides tools for fast mobility because MAC/IP routes are advertised with a sequence number that indicates the latest location of a MAC. This sequence number is used by the leaf routers to program the MAC with the correct VXLAN destination.
MAC mobility (1 of 2)
Using Example of EVPN-VXLAN broadcast domain as reference, if HOST-2 moves from LEAF-2 to LEAF-3, and you review the programming of the MAC in LEAF-4, MAC 00:00:00:00:00:02 is learned against VXLAN destination vtep:2.2.2.2 vni:1:
--{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]--
A:dut4# /show network-instance MAC-VRF-1 bridge-table mac-table all
----------------------------------------------------------------------------------------------------
Mac-table of network i
----------------------------------------------------------------------------------------------------
+--------------------+-------------------------+------------+-------+--------+-------+--------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================+============+=======+========+=======+==============+
| 00:00:00:00:00:01 |lag1.1 | 11 | learnt| true | 300 |2021-02-16T11:|
| | | | | | | 32:14.000Z |
| 00:00:00:00:00:02 |vxlan-interface:vxlan1.1 | 2787793311 | evpn | true | N/A |2021-02-16T11:|
| | vtep:2.2.2.2 vni:1 | 82 | | | | 32:15.000z |
| 00:00:00:00:00:03 |vxlan-interface:vxlan1.1 | 2787793311 | evpn | true | N/A |2021-02-16T11:|
| | vtep:3.3.3.3 vni:1 | 84 | | | | 32:15.000Z |
| 00:00:00:00:00:04 |ethernet-1/13.1 | 13 | learnt| true | 300 |2021-02-16T11:|
| | | | | | | 32:14.000Z |
| 00:CA:FE:CA:FE:04 |ethernet-1/13.1 | 13 | static| true | N/A |2021-02-16T11:|
| | | | | | | 10:26.000Z |
+--------------------+-------------------------+------------+-------+--------+-------+--------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 1 Total 1 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 2 Total 2 Active
Total Evpn Macs : 2 Total 2 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 5 Total 5 Active
-----------------------------------------------------------------------------------------------------
MAC mobility (2 of 2)
After the mobility event, LEAF-4 receives the MAC with a higher sequence number. This makes LEAF-4 reprogram the MAC against LEAF-3 as shown below:
2021-02-16T03:33:53.505253-08:00 dut4 local6|DEBU sr_bgp_mgr: bgp|4959|5178|402506|D:
VR default (1) Peer 1: 3.3.3.3 UPDATE: Peer 1: 3.3.3.3 - Received BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 96
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 3.3.3.3
Type: EVPN-MAC Len: 33 RD: 3.3.3.3:1 ESI: ESI-0, tag: 0, mac len: 48 mac: 00:00:00:00:00:02,
IP len: 0, IP: NULL, label1: 1
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x80 Type: 4 Len: 4 MED: 0
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:1234:1
bgp-tunnel-encap:VXLAN
mac-mobility:Seq:1
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut4# show network-instance MAC-VRF-1 bridge-table mac-table all
-------------------------------------------------------------------------------------------------------
Mac-table of network instance MAC-VRF-1
-------------------------------------------------------------------------------------------------------
+--------------------+-------------------------+------------+-------+--------+--------+---------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+====================+=========================+============+=======+========+========+===============+
| 00:00:00:00:00:01 |lag1.1 | 11 | learnt| true | 180 |2021-02-16T11: |
| | | | | | | 32:14.000Z |
| 00:00:00:00:00:02 |vxlan-interface:vxlan1.1 | 2787793311 | evpn | true | N/A |2021-02-16T11: |
| | vtep:3.3.3.3 vni:1 | 84 | | | | 33:54.000Z |
| 00:00:00:00:00:03 |vxlan-interface:vxlan1.1 | 2787793311 | evpn | true | N/A |2021-02-16T11: |
| | vtep:3.3.3.3 vni:1 | 84 | | | | 32:15.000Z |
| 00:00:00:00:00:04 |ethernet-1/13.1 | 13 | learnt| true | 180 |2021-02-16T11: |
| | | | | | | 32:14.000Z |
| 00:CA:FE:CA:FE:04 |ethernet-1/13.1 | 13 | static| true | N/A |2021-02-16T11: |
| | | | | | | 10:26.000Z |
+--------------------+-------------------------+------------+-------+--------+--------+---------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 1 Total 1 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 2 Total 2 Active
Total Evpn Macs : 3 Total 3 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 5 Total 5 Active
-------------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking MAC protection
MAC protection refers to the property of a MAC that does not move between leaf routers. It is always learned against a bridged subinterface. You can configure this MAC as static, but should observe the following:
-
When a MAC is programmed as static, the same MAC cannot be learned in another sub-interace or via EVPN. If frames arriving on an interface are different than the ones associated with the static MAC, they are discarded.
-
The MAC is now advertised as ‟static” in EVPN and installed as "evpn-static" in the leaf routers attached to the same BD. If programmed, evpn-static MACs are also protected. Therefore, frames arriving on a local subinterface are discarded if their source MAC matches an evpn-static MAC.
MAC protection (1 of 2)
Using Example of EVPN-VXLAN broadcast domain as reference, the following commands show MAC 00:ca:fe:ca:fe:04 is configured as static in LEAF-4.
--{ [FACTORY] +* candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]--
A:dut4# info
static-mac {
mac 00:CA:FE:CA:FE:04 {
destination ethernet-1/13.1
}
}
--{ [FACTORY] +* candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]--
A:dut4# commit stay
All changes have been committed. Starting new transaction.
--{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]--
A:dut4#
--{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]--
A:dut4# /show network-instance MAC-VRF-1 bridge-table mac-table all
-----------------------------------------------------------------------------------------
Mac-table of network instance MAC-VRF-1
--------------------------------------------------------------------------------------------
+--------------+--------------------------+------------+-------+--------+-------+------------------+
| Address | Destination | Dest Index | Type | Active | Aging | Last Update |
+==============+==========================+============+=======+========+=======+==================+
| 00:00:00:00: | lag1.1 | 11 | learnt| true | 180 | 2021-02-16T11:08:|
| 00:01 | | | | | | 10.000Z |
| 00:00:00:00: | vxlan-interface:vxlan1.1 | 2787793311 | evpn | true | N/A | 2021-02-16T11:08:|
| 00:02 | vtep:2.2.2.2 vni:1 | 82 | | | | 11.000Z |
| 00:00:00:00: | vxlan-interface:vxlan1.1 | 2787793311 | evpn | true | N/A | 2021-02-16T11:08:|
| 00:03 | vtep:3.3.3.3 vni:1 | 84 | | | | 11.000Z |
| 00:00:00:00: | ethernet-1/13.1 | 13 | learnt| true | 180 | 2021-02-16T11:08:|
| 00:04 | | | | | | 10.000Z |
| 00:CA:FE:CA: | ethernet-1/13.1 | 13 | static| true | N/A | 2021-02-16T11:10:|
| FE:04 | | | | | | 26.000Z |
+--------------+--------------------------+------------+-------+--------+-------+------------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 1 Total 1 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 2 Total 2 Active
Total Evpn Macs : 2 Total 2 Active
Total Evpn static Macs : 0 Total 0 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 5 Total 5 Active
---------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]--
MAC protection (2 of 2)
On the remote leaf routers, the MAC is received as evpn-static and programmed this way. For example, LEAF-4 receives the route and programs it as follows:
2021-02-16T03:10:27.402653-08:00 dut3 local6|DEBU sr_bgp_mgr: bgp|4628|4789|270039|D:
VR default (1) Peer 1: 4.4.4.4 UPDATE: Peer 1: 4.4.4.4 - Received BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 96
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 4.4.4.4
Type: EVPN-MAC Len: 33 RD: 4.4.4.4:1 ESI: ESI-0, tag: 0, mac len: 48 mac:
00:ca:fe:ca:fe:04, IP len: 0, IP: NULL, label1: 1
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x80 Type: 4 Len: 4 MED: 0
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:1234:1
bgp-tunnel-encap:VXLAN
mac-mobility:Seq:0/Static
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show network-instance MAC-VRF-1 bridge-table mac-table all
-------------------------------------------------------------------------------------------------------
Mac-table of network instance MAC-VRF-1
-------------------------------------------------------------------------------------------------------
+---------------+--------------------------+------------+------------+-------+-------+----------------+
| Address | Destination | Dest Index | Type | Active| Aging | Last Update |
+===============+==========================+============+============+=======+=======+================+
| 00:CA:FE:CA: | vxlan-interface:vxlan1.1 | 2787792288 | evpn-static| true | N/A | 2021-02-16T11: |
| FE:04 | vtep:4.4.4.4 vni:1 | 38 | | | | 10:27.000Z |
+---------------+--------------------------+------------+------------+-------+-------+----------------+
Total Irb Macs : 0 Total 0 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 0 Total 0 Active
Total Learnt Macs : 0 Total 0 Active
Total Evpn Macs : 0 Total 0 Active
Total Evpn static Macs : 1 Total 1 Active
Total Irb anycast Macs : 0 Total 0 Active
Total Macs : 1 Total 1 Active
-----------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[
Note that the static MACs state depends on the state of the subinterface which they are configured against. If the subinterface goes oper-down, the static MAC and EVPN route are removed. Static Blackhole MACs (where the configured destination is "blackhole") also behave as static MACs and are advertised as "evpn-static".
MAC loop protection
MAC loop protection in EVPN BDs is based on the SR Linux MAC Duplication feature. This feature detects MAC duplication for MACs moving:
among bridge sub-interfaces of the same MAC-VRF
between bridge sub-interfaces and EVPN (in the same MAC-VRF)
It does not detect MAC duplication for MACs moving from one VTEP to a different VTEP in the same MAC-VRF. In addition, when a MAC is declared as a duplicate:
If the blackhole configuration option is added to the interface, incoming frames on bridged sub-interfaces are discarded if their MAC SA or DA match the blackhole MAC. Frames encapsulated in VXLAN packets are discarded if their inner source MAC or destination MAC match the blackhole MAC in the mac-table.
The "duplicate" MAC can be overwritten by a higher priority type (for example, static or evpn-static) or flushed by a tools command. Blackhole MACs that result out of duplicate MACs are advertised as regular MACs (non-static).
Multi-homing configuration for EVPN broadcast domains
SR Linux supports all-active multi-homing for multi-homed peers connected using VXLAN, as per [RFC8365].
Using Example of EVPN-VXLAN broadcast domain as a reference, LEAF-2 and LEAF-3 are multi-homed to server-1 using all-active multi-homing. The representation of the multi-homed device in the EVPN control plane is referred to as an Ethernet Segment (ES). It is considered "all-active" and not "active-active", because SR Linux supports up to four leaf routers multi-homed to the same CE or server with all links being active (not just two).
The all-active multi-homing function relies on three different procedures to handle multi-homing in the ES:
Designated Forwarder (DF) election
Split-Horizon (also known as Local-Bias)
Aliasing
The DF is the leaf that forwards BUM traffic received from the VXLAN into the ES (to the server). Only one DF can exist per ES at a time, and it is elected based on the exchange of ES routes (EVPN routes type 4) and the subsequent DF election algorithm. All leaf routers, DF and non-DF, forward known-unicast traffic to the multi-homed server.
The split-horizon or local bias is the procedure that avoids looped packets on the server. If server-1 hashes the BUM traffic to the non-DF leaf (for example, LEAF-2), without any split-horizon technique, the flooded BUM packets move to the DF (LEAF-4) and back to server-1. Split-horizon prevents these flooded packets from forwarding back to server-1. When the data plane is VXLAN, the split-horizon mechanism is based on a "local-bias" forwarding mode as defined in RFC8365. This implies that:
BUM traffic from a local subinterface is always forwarded to the ES, irrespective of the PE being DF. For example, if HOST-2 sends a broadcast frame, it is sent to the ES (lag1.1) even though LEAF- is non-DF for ES-1.
BUM traffic received over VXLAN is never be forwarded to the ES if the source VTEP matches a leaf that is attached to the same ES. For example, BUM traffic from HOST-2 that is forwarded via VXLAN to LEAF-4 is not forwarded to lag1.1, only to Ethernet-1/13.1.
Aliasing is the procedure that allows ecmp (load-balancing) from remote leaf routers (LEAF-3) to all leaf routers attached to the same ES, even though the MAC is only advertised by one of the leaf routers in the ES. For example, in Example of EVPN-VXLAN broadcast domain, flows from 00:00:00:00:00:01 can only be hashed to LEAF-2. LEAF-2 is the only router in the ES advertising the MAC. However, because LEAF-2 and LEAF-4 advertise the association to the same Ethernet Segment Identifier (ESI), and the MAC/IP route for 00:00:00:00:00:01 is tagged with ESI-1, LEAF-3 can "alias" the unicast traffic to both leaf routers.
Note that a leaf advertises its association to an ES via AD per ES routes, and its association to an ES for a specified MAC-VRF via AD per EVI routes. Both are EVPN routes type 1.
All-active multi-homing configurations
Configuration of all-active multi-homing involves four major steps:
Configuring the server/CE with a single LAG that connects it to the leaf routers.
Configuring an ES (ES-1) on the leaf routers (LEAF-2 and LEAF-4)
Associating the ES interface with the MAC-VRF.
-
Optional: Configuring the ecmp with a value greater than 1 in all the leaf routers attached to the same BD (for aliasing). This step is only required if multi-homing is used in an EVPN-VXLAN BD distributed among multiple leaf routers.
Ethernet segment configuration details
With SR Linux, ESs are control plane entities that reside in the system network-instance. The system network-instance contains a BGP-VPN instance similar to the one in mac-vrfs. This instance hosts the bgp information used by EVPN for multi-homing routes, and the ES configuration and state.
The following example provides ES configuration details.
--{ [FACTORY] + candidate shared default }--
[ system network-instance protocols evpn ethernet-segments ]--
A:dut4# tree detail
ethernet-segments!+ evpn_mgr
+-- timers evpn_mgr
| +-- boot-timer? evpn_mgr
| +-- activation-timer? evpn_mgr
+-- bgp-instance* [id] evpn_mgr
+-- ethernet-segment* [name] evpn_mgr
+-- admin-state? evpn_mgr
+-- esi? evpn_mgr
+-- interface? evpn_mgr
+-- multi-homing-mode? evpn_mgr
+-- df-election evpn_mgr
| +-- timers evpn_mgr
| | +-- activation-timer? evpn_mgr
The ES model uses a BGP-VPN instance where the route-distinguisher and export/import route-targets are taken by BGP and used for the ES routes. Only one instance is allowed, and all ESs live under this BGP instance. The default route-distinguisher for the instance is automatically derived from system-ip:0 and used in ES routes.
The default import/export route-target is automatically derived from the ESI (bytes 1 to 6; from the second highest order byte up to the seventh byte). This route-target is of type ESI-import route-target (as per [RFC7432]) and is used in ES routes to ensure they are imported on Leaf routers attached to the ES.
ESs have an admin-state this is disabled by default. It must be toggled to change any of the parameters affecting the EVPN control plane.
The following timers can be configures for ESs: general boot, per ES boot, and activation:
The boot-timer is configured globally for all ESs. This allows the system to synchronize with the rest of the network at reboot, and before the ES is brought up and its route is advertised.
Before a boot-timer expires, the ES subinterfaces are oper-up and the AD routes advertised. In all-active mode, they forward unicast traffic; BUM is not forwarded until the ES subinterfaces become DF. When the boot-timer expires, the ES route is advertised, and the DF election takes place.
The boot-timer can be configured with a value of 0-6000 seconds. Because it is linked to the evpn_mgr application, the boot-timer kicks in when the evpn_mgr restarts. It is recommended that you configure a timer that is long enough for the node to establish its BGP sessions and underlay connectivity before it expires, use some transient BUM duplication).and ES routes are exchanged.
The ES activation timer allows collecting ES routes for the same ES from other leaf routers before promoting a node subinterface as DF. This prevents multiple transient DF leaf routers on the same ES, and BUM duplication to the server/CE. The ES activation timer defaults to 3 seconds, but can be set to 0 if fast convergence is needed (although this may cause some transient BUM duplication).
The ES requires a manual 10-byte ESI configuration. Reserved ESI values such as ESI-0 or MAX-ESI (0xFF..FF) are not allowed. ESI values with 00-00-00-00-00-00 in bytes 1-6 are not allowed. This prevents the auto-derivation of ESI-import route-targets as all 0s.
SR Linux supports the default DF election algorithm, as per [RFC8584]. No configuration is required.
The algorithm (also known as type Default or type 0) is a modulo-based operation that uses the number of leafs in the ES and the configured EVI values in the contained mac-vrfs.
The default alg orders the candidate list from lowest to highest IP address (where the IP address is taken from the originating-ip of the ES routes), and picks up an ordinal of the list based on the outcome of the modulo operation.
If the mac-vrf instances in the ES have consecutive EVI values, load balancing of the DF function occurs. For example, if mac-vrf-1 has a value of EVI=1, mac-vrf-2 is EVI=2. Both have subinterfaces in lag1 that belong to ES-1; one of the mac-vrfs is DF and the other is non-DF for ES-1.
The ES association to an interface must be configured. The interface type can be Ethernet or LAG. If LACP is used on the CE, as shown in Example of EVPN-VXLAN broadcast domain, only a LAG can be associated.
Configuring LEAF-2 and LEAF-4 as multi-homed nodes to server-1
LEAF-2 and LEAF-4 behave as a single system to server-1. Therefore, they must be configured with the same LACP parameters for the LAG on server-1 to come up. The admin-key, system-id-mac, and system-priority must match on both leaf routers so that the LAG comes up.
-
In candidate mode, configure the LAG that connects to server-1.
The LAG can be LACP enabled or static. In this example, LACP is used.
LAG configuration (LAG1 on LEAF-4)--{ [FACTORY] + candidate shared default }--[ ]-- A:dut4# interface ethernet-1/4 --{ [FACTORY] + candidate shared default }--[ interface ethernet-1/4 ]-- A:dut4# info description ES-1 ethernet { aggregate-id lag1 } --{ [FACTORY] + candidate shared default }--[ interface ethernet-1/4 ]-- A:dut4# /interface lag1 --{ [FACTORY] + candidate shared default }--[ interface lag1 ]-- A:dut4# info admin-state enable vlan-tagging true subinterface 1 { type bridged vlan { encap { single-tagged { vlan-id 1 } } } } lag { lag-type lacp member-speed 100G lacp { interval FAST lacp-mode ACTIVE admin-key 24 system-id-mac 00:00:00:00:00:24 system-priority 24 } } --{ [FACTORY] + candidate shared default }--[ interface lag1 ]--LAG configuration (LAG1 on LEAF-2)--{ [FACTORY] + candidate shared default }--[ interface ethernet-1/11 ]-- A:dut2# info description ES-1 ethernet { aggregate-id lag1 } --{ [FACTORY] + candidate shared default }--[ interface ethernet-1/11 ]-- A:dut2# /interface lag1 --{ [FACTORY] + candidate shared default }--[ interface lag1 ]-- A:dut2# info admin-state enable vlan-tagging true subinterface 1 { type bridged vlan { encap { single-tagged { vlan-id 1 } } } } lag { lag-type lacp member-speed 100G lacp { interval FAST lacp-mode ACTIVE admin-key 24 system-id-mac 00:00:00:00:00:24 system-priority 24 } } -
Configure Ethernet Segment (ES-1) on LEAF-2 and LEAF-4.
See Ethernet segment configuration details for an in-depth overview of ES functionality and configuration.
ES configuration (ES-1 on LEAF-2)--{ [FACTORY] + candidate shared default }--[ interface lag1 ]-- A:dut2# /system network-instance --{ [FACTORY] + candidate shared default }--[ system network-instance ]-- A:dut2# info protocols { evpn { ethernet-segments { bgp-instance 1 { ethernet-segment ES-1 { admin-state enable esi 01:24:24:24:24:24:24:00:00:01 interface lag1 multi-homing-mode all-active } } } } bgp-vpn { bgp-instance 1 { } } }The ESI and mode must match on both Leaf routers. The following is the minimum configuration for ES-1 to function across the two leaf routers.
ES configuration (ES-1 on LEAF-4)--{ [FACTORY] + candidate shared default }--[ interface lag1 ]-- A:dut4# /system network-instance --{ [FACTORY] + candidate shared default }--[ system network-instance ]-- A:dut4# info protocols { evpn { ethernet-segments { bgp-instance 1 { ethernet-segment ES-1 { admin-state enable esi 01:24:24:24:24:24:24:00:00:01 interface lag1 multi-homing-mode all-active } } } } bgp-vpn { bgp-instance 1 { } } } -
Configure the association of the ES interface with the MAC-VRF.
In this example, interface lag1.1 is added to the MAC-VRF. The association is based on the configuration under the ES and no further configuration is needed at the MAC-VRF level.
ES interface association with the MAC-VRFA:dut2# /network-instance MAC-VRF-1 --{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 ]-- A:dut2# info type mac-vrf admin-state enable interface ethernet-1/12.1 { } interface lag1.1 { } vxlan-interface vxlan1.1 { } protocols { bgp-evpn { bgp-instance 1 { admin-state enable vxlan-interface vxlan1.1 evi 1 } } bgp-vpn { bgp-instance 1 { route-target { export-rt target:1234:1 import-rt target:1234:1 } } } } A:dut4# /network-instance MAC-VRF-1 --{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 ]-- A:dut4# info type mac-vrf admin-state enable interface ethernet-1/13.1 { } interface lag1.1 { } vxlan-interface vxlan1.1 { } protocols { bgp-evpn { bgp-instance 1 { admin-state enable vxlan-interface vxlan1.1 evi 1 } } bgp-vpn { bgp-instance 1 { route-target { export-rt target:1234:1 import-rt target:1234:1 } } } } bridge-table { static-mac { mac 00:CA:FE:CA:FE:04 { destination ethernet-1/13.1 } } } -
Configure the ecmp with a value greater than 1 in all the leaf routers attached
to the same BD to allow for aliasing.
Note:
This step is only required if Multi-Homing is used in an EVPN-VXLAN BD distributed among multiple Leaf routers. If the Multi-Homing ES is used locally as a Layer-2 MLAG (Multi-chassis Link Aggregation Group) technique, this step can be skipped.
In the following example, LEAF-3 is configured with ecmp 2.
Configure ecmp--{ [FACTORY] + candidate shared default }--[ network-instance MAC-VRF-1 ]-- A:dut3# info detail flat | grep ecmp set / network-instance MAC-VRF-1 protocols bgp-evpn bgp-instance 1 ecmp 2 -
Review the configuration and commit the changes.
A:dut2# commit stay --{ candidate shared default }--[ ]--
Use of multi-homing as all-active MLAG for non-EVPN layer-2 BDs
An example of ESs in a non-EVPN layer-2 BD is shown in the following diagram. In this scenario, EVPN only runs locally between the leaf routers of the two pairs, but not globally in the network. The two leaf tiers (LEAF-4/5 and LEAF-2/3) are connected via layer-2 sub-interfaces, and not VXLAN. Each leaf is configured with two ESs. LEAF-4 is configured with ES-6 for multi-homing to server-6, and ES-45 for multi-homing to the LEAF-2/3 tier. The configuration of the ES previously described also apply to these topologies, with the exception that each ES would use a different name, ESI, and lag interface.
As noted in the multi-homing configuration procedure, configuring the ecmp with a value greater than 1 in all the leaf routers attached to the same BD is not required If the multi-homing ES is used locally as a layer-2 MLAG.
Checking the multi-homing operation
After the multi-homed leaf routers and the remote leaf are configured, the ES operation must be checked.
Checking the ES status
Use the following to check the status of the ES on LEAF-2 and LEAF-4. The output from this command must show the same DF for the same mac-vrf on both leaf routers, and the same candidate list for the ES on both leaf routers. The detail form of this command also provides information about timers and the DF election.
Check the ES status
// Leaf-4
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut4# show system network-instance ethernet-segments ES-1
-------------------------------------------------------------------------------
ES-1 is up, all-active
ESI : 01:24:24:24:24:24:24:00:00:01
Alg : default
Peers: 2.2.2.2
Interface: lag1
Network-instances:
MAC-VRF-1
Candidates : 2.2.2.2, 4.4.4.4 (DF)
Interface : lag1.1
-------------------------------------------------------------------------------
Summary
1 Ethernet Segments Up
0 Ethernet Segments Down
--------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut4# show system network-instance ethernet-segments ES-1 detail
=================================================================================
Ethernet Segment
=================================================================================
Name : ES-1
4.4.4.4 (DF)
Admin State : enable Oper State : up
ESI : 01:24:24:24:24:24:24:00:00:01
Multi-homing : all-active Oper Multi-homing : all-active
Interface : lag1
ES Activation Timer : None
DF Election : default Oper DF Election : default
Last Change : 2021-02-15T11:07:01.412Z
=================================================================================
MAC-VRF Actv Timer Rem DF
ES-1 0 Yes
--------------------------------------------------------------------------------
DF Candidates
--------------------------------------------------------------------------------
Network-instance ES Peers
MAC-VRF-1 2.2.2.2
MAC-VRF-1 4.4.4.4 (DF)
================================================================================
--{ [FACTORY] + candidate shared default }--[ ]—
// Leaf-2
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut2# show system network-instance ethernet-segments ES-1
--------------------------------------------------------------------------------
ES-1 is up, all-active
ESI : 01:24:24:24:24:24:24:00:00:01
Alg : default
Peers: 4.4.4.4
show system network-instance ethernet-segments ES-1
Network-instances:
MAC-VRF-1
Candidates : 2.2.2.2, 4.4.4.4 (DF)
Interface : lag1.1
--------------------------------------------------------------------------------
Summary
1 Ethernet Segments Up
0 Ethernet Segments Down
--------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking ES and EVI routes
The exchange of ES routes (used for DF election and ES discovery) and AD per ES/EVI routes (used to indicate the association of Leaf services to ES) can be checked with the following commands.
Note that LEAF-2 should have a type 4 route for the ES originating from LEAF-4. There should also be one AD per EVI and one AD per ES route from LEAF-4 for MAC-VRF-1. AD routes are advertised for each MAC-VRF that are part of the ES.
Check ES and EVI routes
// Received ES routes or routes type 4 on Leaf-2
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut2# show network-instance default protocols bgp routes evpn route-type 4 summary
------------------------------------------------------------------------------------
Show report for the BGP route table of network-instance "default"
------------------------------------------------------------------------------------
Status codes: u=used, *=valid, >=best, x=stale
Origin codes: i=IGP, e=EGP, ?=incomplete
------------------------------------------------------------------------------------
BGP Router ID: 2.2.2.2 AS: 2222 Local AS: 2222
-----------------------------------------------------------------------------------
Type 4 Ethernet Segment Routes
+--------+---------------+-------------------+--------------+---------+-----------+
| Status | Route- | ESI | originating | neighbor| Next-Hop |
| | distinguisher | | -router | | |
+========+===============+===================+==============+=========+===========+
| u*> | 4.4.4.4:0 | 01:24:24:24:24:24:| 4.4.4.4 | 4.4.4.4 | 4.4.4.4 |
| | | 24:00:00:01 | | | |
+--------+---------------+-------------------+------------------------+-----------+
-----------------------------------------------------------------------------------
1 Ethernet Segment routes 1 used, 1 valid
----------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut2#
/* Received AD routes or routes type 1 on Leaf-2. The AD per ES route uses Eth-Tag= MAX-ET (all FFs). */
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut2# show network-instance default protocols bgp routes evpn route-type 1 summary
Show report for the BGP route table of network-instance "default"
-------------------------------------------------------------------------------------
Status codes: u=used, *=valid, >=best, x=stale
Origin codes: i=IGP, e=EGP, ?=incomplete
------------------------------------------------------------------------------------
BGP Router ID: 2.2.2.2 AS: 2222 Local AS: 2222
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
Type 1 Ethernet Auto-Discovery Routes
+--------+---------------+------------------+------------+---------+---------+-----+
| Status | Route- | ESI | Tag-ID | neighbor|Next-hop | VNI |
| | distinguisher | | | | | |
+========+===============+==================+============+===================+=====+
| u*> | 4.4.4.4:1 | 01:24:24:24:24: | 0 | 4.4.4.4 | 4.4.4.4 | - |
| | | 24:24:00:00:01 | | | | |
| u*> | 4.4.4.4:1 | 01:24:24:24:24: | 4294967295 | 4.4.4.4 | 4.4.4.4 | - |
| | | 24:24:00:00:01 | | | | |
+--------+---------------+------------------+------------+---------+---------+-----+
2 Ethernet Auto-Discovery routes 2 used, 2 valid
------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut2# show network-instance default protocols bgp routes evpn route-type 1 detail
-------------------------------------------------------------------------------------
Show report for the EVPN routes to network "*" network-instance "default"
------------------------------------------------------------------------------------
Route Distinguisher: 4.4.4.4:1
Tag-ID : 0
ESI : 01:24:24:24:24:24:24:00:00:01
neighbor : 4.4.4.4
Received paths : 1
Path 1: <Best,Valid,Used,>
ESI : 01:24:24:24:24:24:24:00:00:01
Route source : neighbor 4.4.4.4 (last modified 4h38m27s ago)
Route preference: No MED, LocalPref is 100
Atomic Aggr : false
BGP next-hop : 4.4.4.4
AS Path : i
Communities : [target:1234:1, bgp-tunnel-encap:VXLAN]
RR Attributes : No Originator-ID, Cluster-List is []
Aggregation : None
Unknown Attr : None
Invalid Reason : None
Tie Break Reason: none
------------------------------------------------------------------------------------
Route Distinguisher: 4.4.4.4:1
Tag-ID : 4294967295
ESI : 01:24:24:24:24:24:24:00:00:01
neighbor : 4.4.4.4
Received paths : 1
Path 1: <Best,Valid,Used,>
ESI : 01:24:24:24:24:24:24:00:00:01
Route source : neighbor 4.4.4.4 (last modified 4h38m27s ago)
Route preference: No MED, LocalPref is 100
Atomic Aggr : false
BGP next-hop : 4.4.4.4
AS Path : i
Communities : [target:1234:1, esi-label:0/All-Active]
RR Attributes : No Originator-ID, Cluster-List is []
Aggregation : None
Unknown Attr : None
Invalid Reason : None
Tie Break Reason: none
------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking the MAC/IP route
MAC/IP routes advertised for MACs learned on the ES sub-interfaces are advertised with the ESI of ES-1. The RIB state for the MAC/IP routes can be used to check this. The following command checks the MAC/IP route for MAC 00:00:00:00:00:01 on LEAF-3:
Check the MAC/IP route
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show network-instance default protocols bgp routes evpn route-type 2 summary
-----------------------------------------------------------------------------------------------------
Show report for the BGP route table of network-instance "default"
-----------------------------------------------------------------------------------------------------
Status codes: u=used, *=valid, >=best, x=stale
Origin codes: i=IGP, e=EGP, ?=incomplete
-----------------------------------------------------------------------------------------------------
BGP Router ID: 3.3.3.3 AS: 3333 Local AS: 3
-----------------------------------------------------------------------------------------------------
Type 2 MAC-IP Advertisement Routes
+--------+-----------+-----+-------------+--------+---------+---------+---+-----------------+-------+
| Status |Route-dis- | Tag |MAC-address |IP- | neighbor| Next-Hop|VNI| ESI | MAC |
| |tinguisher | -ID | |address | | | | | Mob. |
+========+===========+=====+=============+========+=========+=========+===+=================+=======+
| u*> |2.2.2.2:1 | 0 |00:00:00:00: |0.0.0.0 | 2.2.2.2 | 2.2.2.2 | 1 | 00:00:00:00:00: | - |
| | | | 00:02 | | | | | 00:00:00:00:0 0| |
| u*> |4.4.4.4:1 | 0 |00:00:00:00: |0.0.0.0 | 4.4.4.4 | 4.4.4.4 | 1 | 01:24:24:24:24: | - |
| | | | 00:01 | | | | | 24:24:00:00:01 | |
| u*> |4.4.4.4:1 | 0 |00:00:00:00: |0.0.0.0 | 4.4.4.4 | 4.4.4.4 | 1 | 00:00:00:00:00: | - |
| | | | 00:0 4 | | | | | 00:00:00:00:00 | |
| u*> |4.4.4.4:1 | 0 |00:CA:FE:CA: |0.0.0.0 | 4.4.4.4 | 4.4.4.4 | 1 | 00:00:00:00:00: |Seq:0/ |
| | | | FE:04 | | | | | 00:00:00:00:00 |Static |
+--------+-----------+-----+-------------+--------+---------+---------+---+-----------------+-------+
-----------------------------------------------------------------------------------------------------
4 MAC-IP Advertisement routes 4 used, 4 valid
-----------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking the ES destination
When AD routes are received from LEAF-2 and LEAF-4, and the MAC/IP route is tagged with the ESI, LEAF-3 creates an ES destination resolved to as many VTEPs as leafs advertising the AD routes (up to a maximum of the ecmp setting on LEAF-3). Use the following to check the ES destination for ES-1 created on LEAF-3.
Check the ES destination
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show tunnel-interface vxlan1 vxlan-interface 1 bridge-table unicast-destinations destination *
------------------------------------------------------------------------------------------------------
Show report for vxlan-interface vxlan1.1 unicast destinations
------------------------------------------------------------------------------------------------------
Destinations
------------------------------------------------------------------------------
+--------------+------------+-------------------+-----------------------------+
| VTEP Address | Egress VNI | Destination-index | Number MACs (Active/Failed) |
+==============+============+===================+=============================+
| 2.2.2.2 | 1 | 278779228840 | 1(1/0) |
| 4.4.4.4 | 1 | 278779228838 | 2(2/0) |
+--------------+------------+-------------------+-----------------------------+
-------------------------------------------------------------------------------
Ethernet Segment Destinations
------------------------------------------------------------------------------------------------------
+-------------------------------+-------------------+------------------+-----------------------------+
| ESI | Destination-index | VTEPs | Number MACs (Active/Failed) |
+===============================+===================+==================+=============================+
| 01:24:24:24:24:24:24:00:00:01 | | 2.2.2.2, 4.4.4.4 | 0(0/0) |
+-------------------------------+-------------------+------------------+-----------------------------+
------------------------------------------------------------------------------------------------------
Summary
3 unicast-destinations, 2 non-es, 1 es
3 MAC addresses, 3 active, 0 non-active
-----------------------------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking MAC programming
The following command shows how LEAF-3 programs the MACs received for the ES as associated with an ES destination
Check MAC programming
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut3# show network-instance MAC-VRF-1 bridge-table mac-table mac 00:00:00:00:00:01
------------------------------------------------------------------------------------
Mac-table of network instance MAC-VRF-1
-----------------------------------------------------------------------------------
Mac : 00:00:00:00:00:01
Destination : vxlan-interface:vxlan1.1 esi:01:24:24:24:24:24:24:00:00:01
Dest Index : 278779228850
Type : evpn
Programming Status : Success
Aging : N/A
Last Update : 2021-02-16T15:51:09.000Z
Duplicate Detect time : N/A
Hold down time remaining: N/A
-----------------------------------------------------------------------------------
--{ [FACTORY] + candidate shared default }--[ ]--
Checking subinterface ES association (LEAF-2)
On the non-DF (in this example, LEAF-2), the MAC-VRF interface included in ES-1 (lag1.1) shows the "multicast-forwarding" flag as none. This means that the interface does not forward BUM traffic to the CE/server when it is received on the VXLAN. The following shows how the state for the subinterface itself shows the association with ES-1 and the DF state:
Check subinterface ES association (LEAF-2)
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut2# info from state network-instance MAC-VRF-1 interface lag1.1
network-instance MAC-VRF-1 {
interface lag1.1 {
oper-state up
oper-mac-learning up
index 12
multicast-forwarding none
}
}
--{ [FACTORY] + candidate shared default }--[ ]--
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut2# info from state interface lag1 subinterface 1 ethernet-segment-association
interface lag1 {
subinterface 1 {
ethernet-segment-association {
ethernet-segment ES-1
es-managed true
designated-forwarder false
}
}
Checking subinterface ES association (LEAF-4)
On the DF (in this example, LEAF-4), the MAC-VRF interface in ES-1 (lag1.1) shows the "multicast-forwarding" flag as BUM. The following shows how the state for the subinterface itself shows the association with ES-1 and the DF state.
Check subinterface ES association (LEAF-4)
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut4# info from state network-instance MAC-VRF-1 interface lag1.1
network-instance MAC-VRF-1 {
interface lag1.1 {
oper-state up
oper-mac-learning up
index 11
multicast-forwarding BUM
}
}
--{ [FACTORY] + candidate shared default }--[ ]--
--{ [FACTORY] + candidate shared default }--[ ]--
A:dut4# info from state interface lag1 subinterface 1 ethernet-segment-association
interface lag1 {
subinterface 1 {
ethernet-segment-association {
ethernet-segment ES-1
es-managed true
designated-forwarder true
}
}
}
EVPN related logs
The evpn_mgr application has logs that are triggered when the DF state changes on an ES as shown in the following example: