MAC-VRF network-instances for server aggregation
MAC-VRF network-instances can provide aggregation for a group of servers into the same subnet. This chapter defines concepts and procedures for configuring MAC-VRF network-instances and Integrated Routing and Bridging (IRB) subinterfaces.
Applicability
The information and configuration in this chapter are based on SR Linux Release 20.6.
Overview
Data Center (DC) servers or hosts are connected to TOR routers so that they can be reached from other TOR routers in the same IP fabric. The TOR nodes use BGP to learn and propagate subnet reachability in the underlaying routing infrastructure. The servers or hosts connected to these TOR BGP routers use routed subinterfaces on the TOR, and static routes or a PE-CE BGP session, to learn or advertise reachability to the rest of the DC.
Each server requires a separate routed subinterface and subnet on the TOR, and the number of subinterfaces and local routes in the route-table grows linearly as the number of servers increases. The use of a MAC-VRF network-instance provides aggregation for a group of servers into the same subnet. This saves routes and subinterfaces in the TOR. A MAC-VRF is attached to the default network-instance by a single IRB interface and subnet, instead of a separate subinterface and route per server. The following figure shows an example of MAC-VRF network-instances for server aggregation.
In this example, Leaf-1 and Leaf-2 are configured with MAC-VRF instances that aggregate a group of servers. These servers are assigned IP addresses on the same subnet and are connected to the leaf default network-instance by a single IRB subinterface. The servers use a PE-CE BGP session with the IRB IP address to exchange reachability. The use of the MAC-VRF with an IRB subinterface saves routed subinterfaces on the default network-instance; only one routed subinterface is needed instead of one per server.
Configuration of MAC-VRF network-instances and IRB subinterfaces
The following figure shows an example of how to configure MAC-VRF network-instances and their IRB subinterfaces to the default network-instance, and how EBGP sessions are configured with the servers. In this example, DUT2 is the TOR being configured. DUT1 and DUT3 are servers that are running BGP against the DUT2 IRB subinterface.
Example: Configure DUT2 with MAC-VRF, IRB, and static BGP on IRB
This example assumes DUT2 is pre-configured with a default network-instance that runs BGP sessions to the spine routers, as defined in the section: BGP for underlay routing.
This example shows how to configure the DUT2 with a MAC-VRF, bridged subinterfaces to DUT1 and DUT3, and an IRB subinterface (see MAC-VRF and IRB example in DUT2).
-
In candidate mode, create the interfaces and bridged subinterfaces to connect
to DUT1 and DUT3.
In this example:
- Connect ethernet-1/1 and ethernet-1/2 to DUT1 and DUT3, respectively. Although these interfaces could be defined untagged, this example configures them as tagged (vlan-tagging true).
- Create a subinterface with index 1 under each interface. The subinterface must be configured as type bridged. Bridged subinterfaces can be associated with MAC-VRF instances so that MAC learning and layer-2 forwarding can be enabled on them.
- The subinterfaces use vlan-id 100 because this is the VLAN ID used by the servers (DUT1 and DUT2) to send and receive frames.
--{ candidate shared default }--[ interface * ]-- A:dut2# info interface ethernet-1/1 { description dut2-dut1 vlan-tagging true subinterface 1 { type bridged vlan { encap { single-tagged { vlan-id 100 } } } } } interface ethernet-1/2 { description dut2-dut3 vlan-tagging true subinterface 1 { type bridged vlan { encap { single-tagged { vlan-id 100 } } } } } -
Configure an IRB interface and subinterface to connect the MAC-VRF to the
existing default network-instance.
The IRB is configured in a similar way to a loopback interface and subinterfaces. The IRB subinterface must be type routed, but does not need to be explicitly configured as routed.
--{ candidate shared default }--[ interface irb* ]-- A:dut2# info interface irb1 { subinterface 1 { ipv4 { address 10.0.0.2/24 { } } ipv6 { address 2001:db8::2/64 { } } } } -
Configure the network-instance type mac-vrf and associate it with the bridged
and IRB interfaces.
--{ candidate shared default }--[ network-instance MAC-VRF-1 ]-- A:dut2# info type mac-vrf admin-state enable interface ethernet-1/1.1 { } interface ethernet-1/2.1 { } interface irb1.1 { } -
Associate the same IRB interface with the network-instance default and
configure the BGP IPv4 and IPv6 neighbors to DUT1 and DUT3.
See BGP for underlay routing for more information about configuring BGP sessions.
--{ candidate shared default }--[ network-instance default ]-- A:dut2# info type default admin-state enable router-id 2.2.2.2 interface irb1.1 { } interface lo0.1 { } protocols { bgp { admin-state enable autonomous-system 64502 router-id 10.0.0.2 ebgp-default-policy { import-reject-all false } failure-detection { enable-bfd true fast-failover true } group tor { admin-state enable export-policy pass-all ipv4-unicast { admin-state enable } ipv6-unicast { admin-state enable } local-as 64502 { } timers { minimum-advertisement-interval 1 } trace-options { flag update { } flag graceful-restart { } flag events { } flag keepalive { } flag notification { } flag open { } flag packets { } flag route { } flag socket { } flag timers { } } } ipv4-unicast { admin-state enable } ipv6-unicast { admin-state enable } neighbor 10.0.0.1 { peer-as 64501 peer-group tor transport { local-address 10.0.0.2 } } neighbor 10.0.0.3 { peer-as 64503 peer-group tor transport { local-address 10.0.0.2 } } neighbor 2001:db8::1 { peer-as 64501 peer-group tor transport { local-address 2001:db8::2 } } neighbor 2001:db8::3 { peer-as 64503 peer-group tor transport { local-address 2001:db8::2 } } } } -
Review all the changes and commit if correct.
--{ candidate shared default }--[ ]-- A:dut2# commit stay -
Check the state of the MAC-VRF and the connectivity to DUT1 and DUT3 using the
following commands:
- show network-instance MAC-VRF-1 interfaces
- show network-instance default interfaces
- show network-instance MAC-VRF-1 bridge-table mac-table all
- show arpnd arp-entries interface irb1
- show arpnd neighbors interface irb1
- show network-instance default protocols bgp neighbor
A:dut2# show network-instance MAC-VRF-1 interfaces ==================================================================== Net instance : MAC-VRF-1 Interface : ethernet-1/1.1 Type : bridged Oper state : up ==================================================================== Net instance : MAC-VRF-1 Interface : ethernet-1/2.1 Type : bridged Oper state : up ==================================================================== Net instance : MAC-VRF-1 Interface : irb1.1 Oper state : up Ip mtu : 1500 Prefix Origin Status ================================================================== 10.0.0.2/24 static 2001:db8::2/64 static preferred fe80::201:2ff:feff:41/64 link-layer preferred ====================================================================A:dut2# show network-instance default interfaces ==================================================================== Net instance : default Interface : irb1.1 Oper state : up Ip mtu : 1500 Prefix Origin Status ==================================================================== 10.0.0.2/24 static 2001:db8::2/64 static preferred fe80::201:2ff:feff:41/64 link-layer preferred ===================================================================== Net instance : default Interface : lo0.1 Oper state : up Prefix Origin Status ==================================================================== 2.2.2.2/32 static 2001:db8:1::2/128 static preferred =====================================================================A:dut2# show network-instance MAC-VRF-1 bridge-table mac-table all ------------------------------------------------------------------------------------------------------ Mac-table of network instance MAC-VRF-1 ------------------------------------------------------------------------------------------------------ +-------------------+----------------+-------+-----------+--------+-------+--------------------------+ | address | Destination | Dest | Type | Active | Aging | Last Update | | | | Index | | | | | +===================+================+=======+===========+========+=======+==========================+ | 00:01:01:FF:00:00 | ethernet-1/1.1 | 16 | learnt | true | 287 | 2020-06-03T13:40:25.000Z | | 00:01:02:FF:00:41 | irb-interface | 0 | irb- | true | N/A | 2020-06-02T13:53:50.000Z | | | | | interface | | | | | 00:01:03:FF:00:00 | ethernet-1/2.1 | 17 | learnt | true | 287 | 2020-06-03T13:40:25.000Z | +-------------------+----------------+-------+-----------+--------+-------+--------------------------+ Total Irb Macs : 1 Total 1 Active Total Static Macs : 0 Total 0 Active Total Duplicate Macs : 0 Total 0 Active Total Learnt Macs : 2 Total 2 Active Total Macs : 3 Total 3 Active ---------------------------------------------------------------------------------------------------A:dut2# show arpnd arp-entries interface irb1 +-----------+--------------+----------+---------+----------------------+-------------------+ | Interface | Subinterface | Neighbor | Origin | Link layer address | Expiry | +===========+==============+=================+=========+===============+===================+ | irb1 | 1 | 10.0.0.1 | dynamic | 00:01:01:FF:00:00 | 3 hours from now | | irb1 | 1 | 10.0.0.3 | dynamic | 00:01:03:FF:00:00 | 3 hours from now | +-----------+--------------+----------+---------+----------------------+-------------------+ -------------------------------------------------------------------------------------------- Total entries : 2 (0 static, 2 dynamic) --------------------------------------------------------------------------------------------A:dut2# show arpnd neighbors interface irb1 +-----------+-----------+----------------+---------+--------------------+---------+-----------+------+ | Interface | Sub | Neighbor | Origin | Link layer address | Current | Next state| Is | | | interface | | | | state | change |Router +===========+===========+================+=========+====================+=========+===========+======+ | irb1 | 1 | 2001:db8::1 | dynamic | 00:01:01:FF:00:00 | stale | 2 hours | true | | | | | | | | from now | | | irb1 | 1 | 2001:db8::3 | dynamic | 00:01:03:FF:00:00 | stale | 2 hours | true | | | | | | | | from now | | | irb1 | 1 | fe80::201:1ff: | dynamic | 00:01:01:FF:00:00 | stale | 2 hours | true | | | | feff:0 | | | | from now | | | irb1 | 1 | fe80::201:3ff: | dynamic | 00:01:03:FF:00:00 | stale | 2 hours | true | | | | feff:0 | | | | from now | | +-----------+-----------+----------------+---------+--------------------+---------+-----------+------+ ------------------------------------------------------------------------------------------------------ Total entries : 4 (0 static, 4 dynamic) ------------------------------------------------------------------------------------------------------A:dut2# show network-instance default protocols bgp neighbor ----------------------------------------------------------------------------------------------------- BGP neighbor summary for network-instance "default" Flags: S static, D dynamic, L discovered by LLDP, B BFD enabled, - disabled, * slow ----------------------------------------------------------------------------------------------------- +----------+-------------+-----+----+-------+-------------+---------------+------------+------------+ | Net-Inst | Peer |Group|Flag| Peer | State | Uptime | AFI/SAFI | [Rx/Active | | | | | | -AS | | | | /Tx] | +==========+=============+=====+====+=======+=============+===============+============+============+ | default | 10.0.0.1 | tor | SB | 64501 | established | 0d:3h:12m:33s | ipv4-unicast | [3/2/4] | | | | | | | | | ipv6-unicast | [3/2/4] | | default | 10.0.0.3 | tor | SB | 64503 | established | 0d:3h:10m:55s | ipv4-unicast | [3/2/4] | | | | | | | | | ipv6-unicast | [3/2/4] | | default | 2001:db8::1 | tor | SB | 64501 | established | 0d:3h:12m:31s | ipv4-unicast | [0/0/0] | | | | | | | | | ipv6-unicast | [6/0/6] | | default | 2001:db8::3 | tor | SB | 64503 | established | 0d:3h:10m:52s | ipv4-unicast | [0/0/0] | | | | | | | | | ipv6-unicast | [6/0/6] | +----------+-------------+-----+----+-------+-------------+---------------+--------------+----------+ ----------------------------------------------------------------------------------------------------- Summary: 4 configured neighbors, 4 configured sessions are established,0 disabled peers 0 dynamic peers
Advanced configuration: bridge-table settings
A MAC-VRF network-instance uses a bridge-table to forward frames between its subinterfaces. Some bridge-table properties can be configured. For example:
--{ * candidate shared default }--[ network-instance MAC-VRF-1 bridge-table ]
A:dut2# tree detail
bridge-table! net_inst_mgr
+-- discard-unknown-dest-mac? net_inst_mgr
+-- mac-learning net_inst_mgr
| +-- admin-state? net_inst_mgr
| +-- aging net_inst_mgr
| +-- admin-state? net_inst_mgr
| +-- age-time? net_inst_mgr
+-- mac-duplication net_inst_mgr
| +-- admin-state? net_inst_mgr
| +-- monitoring-window? net_inst_mgr
| +-- num-moves? net_inst_mgr
| +-- hold-down-time? net_inst_mgr
| +-- action? net_inst_mgr
+-- mac-limit net_inst_mgr
| +-- maximum-entries? net_inst_mgr
| +-- warning-threshold-pct? net_inst_mgr
+-- static-mac l2_static_mac_mgr
+-- mac* [address] l2_static_mac_mgr
+-- address l2_static_mac_mgr
+-- destination?M l2_static_mac_mgr
Where:
-
The ‟mac-learning” container provides control over how MACs are dynamically learned on the subinterfaces, including whether learning is enabled (admin-state) or controlled by the aging timer for the mac-table.
-
The ‟mac-duplication” container controls how the system behaves when duplicate MACs are detected.
-
The ‟mac-limit” container provides parameters for limiting the maximum number of MACs installed for a specific mac-vrf.
-
The ‟static-macs” provides control to configure and associate to either a subinterface destination or with a blackhole. Incoming frames with the source or destination MAC matching a configured "blackholed mac" are discarded by the system.
Advanced configuration: MAC-duplication for loop protection
SR Linux supports MAC-duplication detection and associated procedures to protect the system against network loops. The following figure shows a simple loop and describes the associated configuration.
In this example, MAC-VRF 2 is connected using two bridged subinterfaces to a layer 2 switch. When a host with MAC M1 sends a broadcast frame, a loop is created. MAC-duplication is, by default, enabled in mac-vrf network-instances with the following parameters:
--{ * candidate shared default }--[ network-instance MAC-VRF-1 bridge-table mac-
duplication ]--
A:dut2# info detail
admin-state enable
monitoring-window 3
num-moves 5
hold-down-time 10
action stop-learning
The loops shown in this example are resolved in the following sequence:
-
MAC-duplication detection.
-
A MAC M1 is declared as ‟duplicate” when the number of moves across two or more subinterfaces exceeds the configured num-moves in the configured monitoring-window.
-
When M1 is ‟duplicate”, it is kept in a duplicate-entries list and stays associated with the last subinterface where the MAC was learned before the number of moves exceed the num-moves value.
-
-
MAC-duplication action.
-
When the MAC M1 is declared ‟duplicate” in a subinterface, an action is taken in that subinterface. The action is configurable per network-instance and can be overridden on a per-subinterface basis.
-
Possible actions on the subinterface are oper-down, blackhole, and stop-learning.
-
oper-down - Brings down the subinterface, breaks the loop, and discards all the frames arriving on the subinterface.
-
blackhole - Discards frames with a source or destination MAC that matches the duplicate MAC. but allows the remaining frames to forward normally on the subinterface.
-
stop-learning - Does not discard any frame on the subinterface and keeps the existing MACs learned against it. No new MACs are learned on the subinterface until the action is cleared.
-
--{ * candidate shared default }--[ network-instance MAC-VRF-1 bridge-table mac-duplication ]-- A:dut2# action <value> usage: action <blackhole|oper-down|stop-learning> Action to take on the subinterface whose action is use-net-instance-action, upon detecting one or more mac addresses as duplicate In particular: - Oper-down: if configured, upon detecting a duplicate mac on the subinterface, the subinterface will be brought oper-down. - Blackhole: upon detecting a duplicate mac on the subinterface, the mac will be blackholed. Any frame received on this or any other subinterface with MAC SA matching a blackhole mac is discarded. - Stop-learning: this is the default action, compliant with RFC7432. Upon detecting a duplicate mac on the subinterface, the mac will not be relearned anymore on this or any subinterface. Positional arguments: value -
-
MAC-duplication hold-down-time and process restart.
-
When the configured hold-down-time expires, the duplicate MAC is flushed from the mac-table and the entire process restarts for the MAC.
-
The duplicate action on a subinterface clears when there are no longer duplicate MAC addresses in the subinterface.
-
As a loop protection mechanism, MAC-duplication is self-contained and does not require a control plane protocol that runs network-wide among network devices.
Example: Configure MAC-duplication and troubleshoot loops in DUT2
Use this example to assist in configuring MAC-duplication. We assume MAC-VRF 1 is connected to a layer 2 switch (not shown) using two bridge subinterfaces (ethernet-1/1.2 and ethernet-1/1.3). This creates a loop. DUT2 is configured with the following MAC-duplication settings:
DUT2 MAC-duplications settings
--{ candidate shared default }--[ network-instance MAC-VRF-1 ]--
A:dut2# info
type mac-vrf
admin-state enable
interface ethernet-1/1.1 {
}
interface ethernet-1/1.2 {
}
interface ethernet-1/1.3 {
}
interface ethernet-1/2.1 {
}
interface irb1.1 {
}
bridge-table {
mac-duplication {
num-moves 3
hold-down-time 300
action stop-learning
}
}
The subinterfaces are also configured with the following actions:
Subinterface mac-duplication settings
--{ * candidate shared default }--[ interface * subinterface * bridge-table mac-
duplication ]--
A:dut2# info
interface ethernet-1/1 {
subinterface 1 {
bridge-table {
mac-duplication {
}
}
}
subinterface 2 {
bridge-table {
mac-duplication {
action oper-down
}
}
}
subinterface 3 {
bridge-table {
mac-duplication {
action oper-down
}
}
In this example, the MAC-duplication action configured under the network-instance is overridden by the more specific action under the subinterfaces 2 and 3. When traffic is generated by the remote layer 2 switch, the same MAC address moves between ethernet-1/1.2 and ethernet-1/1.3. After the third move, the MAC is declared a duplicate and appears in the duplicate-entries list:
Display duplicate MAC entries
A:dut2# show network-instance MAC-VRF-1 bridge-table mac-table all
------------------------------------------------------------------------------------------------------
Mac-table of network instance MAC-VRF-1
------------------------------------------------------------------------------------------------------
+-------------------+----------------+-------+-----------+--------+-------+--------------------------+
| address | Destination | Dest | Type | Active | Aging | Last Update |
| | | Index | | | | |
+===================+================+=======+===========+========+=======+==========================+
| 00:01:01:FF:00:00 | ethernet-1/1.1 | 16 | learnt | true | 287 | 2020-06-03T13:40:25.000Z |
| 00:01:01:FF:00:41 | ethernet-1/1.3 | 20 | duplicate | true | N/A | 2020-06-05T20:07:24.000Z |
| 00:01:02:FF:00:41 | irb-interface | 0 | irb- | true | N/A | 2020-06-02T13:53:50.000Z |
| | | | interface | | | |
| 00:01:03:FF:00:00 | ethernet-1/2.1 | 17 | learnt | true | 287 | 2020-06-03T13:40:25.000Z |
+-------------------+----------------+-------+-----------+--------+-------+--------------------------+
Total Irb Macs : 1 Total 1 Active
Total Static Macs : 0 Total 0 Active
Total Duplicate Macs : 1 Total 1 Active
Total Learnt Macs : 2 Total 2 Active
Total Macs : 4 Total 4 Active
----------------------------------------------------------------------------------------------------
A:dut2# show network-instance MAC-VRF-1 bridge-table mac-duplication duplicate-entries
----------------------------------------------------------------------------------------------------
Mac-Duplication in network instance MAC-VRF-1
----------------------------------------------------------------------------------------------------
Admin state : enable
Monitoring window : 3
Number of moves allowed: 3
Hold down time : 300
Action : stop-learning
----------------------------------------------------------------------------------------------------
+-------------------+----------------+------------+--------------------------+---------------------+
| Duplicate MAC | Destination | Dest Index | Detect Time | Hold Time Remaining |
+===================+================+============+==========================+=====================+
| 00:01:01:FF:00:41 | ethernet-1/1.3 | 20 | 2020-06-05T20:07:24.000Z | 270 |
+-------------------+----------------+------------+--------------------------+---------------------+
Total Duplicate Macs : 1 Total 0 Active
---------------------------------------------------------------------------------------------------
Using logs to detect duplicate MACs
A log event can help to troubleshoot when MACs are detected as duplicate, or when they are deleted after the hold-down-timer. For example:
[root@dut2 srlinux]# tail -f /var/log/srlinux/debug/sr_l2_mac_mgr.log
2020-06-06T06:22:48.679070+00:00 dut2 local6|NOTI sr_l2_mac_mgr: bridgetable|2608|2608|00035|N:
A duplicate MAC address 00:01:01:FF:00:00 was detected on MAC-VRF-1.
2020-06-06T06:27:48.933312+00:00 dut2 local6|NOTI sr_l2_mac_mgr: bridgetable|2608|2608|00036|N:
A duplicate MAC address 00:01:01:FF:00:00 detected on MAC-VRF-1 is now deleted.
The network-instance manager logs also show when the subinterfaces go down as a result of MAC-duplication. For example:
[root@dut2 srlinux]# tail -f /var/log/srlinux/debug/sr_net_inst_mgr.log
2020-06-06T06:22:48.680609+00:00 dut2 local6|WARN sr_net_inst_mgr: netinst|2663|2663|00080|W:
The interface ethernet-1/1.3 in network-instance MAC-VRF-1 is now down for reason:
A duplicate MAC address has been detected