EVPN for PBB over MPLS (PBB-EVPN)
This chapter provides information about EVPN for PBB over MPLS (PBB-EVPN).
Topics in this chapter include:
Applicability
This chapter was initially written for SR OS Release 13.0.R6. The MD-CLI in the current edition is based on SR OS Release 21.2.R1.
A prerequisite is to read the EVPN for MPLS Tunnels chapter.
Overview
EVPN for Provider Backbone Bridging (PBB) over MPLS (hereafter called PBB-EVPN) is specified in RFC 7623, Provider Backbone Bridging Combined with Ethernet VPN (PBB-EVPN). It provides a simplified version of EVPN-MPLS for cases where the network requires very high scalability and does not need all the advanced features supported by EVPN-MPLS (but still requires single-active and all-active multi-homing capabilities). EVPN and PBB-EVPN SR OS feature comparison provides a comparison between the capabilities of EVPN and PBB-EVPN in SR OS, and may help to choose between them when designing a VPN service.
|
VPN requirements |
EVPN |
PBB-EVPN |
Comments |
|---|---|---|---|
|
All-active Multi-Homing (MH) (flow-based load-balancing) |
Yes |
Yes |
Allows better bandwidth utilization |
|
Single-active MH (service-based load-balancing) |
Yes |
Yes |
|
|
Ethernet Local Area Network (E-LAN) and point-to-point E-Line services |
Yes |
Yes |
|
|
Inter-subnet-forwarding |
Yes |
No |
Allows combined Layer 2 / Layer 3 services. EVPN |
|
Proxy-Address Resolution Protocol / Neighbor Discovery (Proxy-ARP/ND) and IP-duplication protection |
Yes |
No |
Allows Broadcast, Unknown unicast and Multicast (BUM) traffic reduction and better security |
|
Customer MAC (CMAC) protection |
Yes |
No |
Allows protecting key static CMACs |
|
Data Center integration |
Yes |
No |
Integration with VXLAN and Nuage Virtualized Services Directory (VSD) |
|
Control plane overhead |
Medium |
Low |
PBB-EVPN only advertises Backbone MACs (BMACs) and no route type 1s |
|
Confinement of CMAC learning |
No |
Yes |
CMACs are only learned on PEs with flows using those CMACs |
|
CMAC summarization |
No |
Yes |
Aggregation of CMACs into BMACs |
PBB-EVPN is a combination of 802.1ah PBB and RFC 7432, BGP MPLS-Based Ethernet VPN (EVPN-MPLS), and reuses the PBB-Virtual Private LAN Service (VPLS) service model, where Border Gateway Protocol BGP-EVPN is enabled in the backbone VPLS (B-VPLS) domain. EVPN is used as the control plane in the B-VPLS domain to control the distribution of BMACs and set up per-backbone service instance identifier (ISID) flooding trees for service instance VPLS (I-VPLS) services. The learning of the CMACs, either on local SAPs/SDP-bindings or associated with remote BMACs, is still performed in the data plane. Only the learning of BMACs in the B-VPLS is performed through BGP.
The SR OS PBB-EVPN implementation supports I-VPLS and PBB-Epipe services, including single-active and all-active multi-homing.
Because PBB-EVPN is based on the same control plane model as EVPN for MPLS, it is recommended to read the EVPN for MPLS Tunnels chapter before configuring PBB-EVPN. PBB-EVPN uses a subset of the BGP-EVPN routes described in EVPN for MPLS Tunnels as shown in EVPN route types.
When no EVPN multi-homing is used in the network, only the base routes are used. Route types 2 and 3 are considered the base and mandatory routes:
-
Route type 2 — (B) MAC route — In PBB-EVPN, this route type is used for the advertisement of BMAC addresses that will be installed in the remote Forwarding Data Bases (FDBs). There are no IP addresses advertised in PBB-EVPN. The MAC mobility extended community is used for advertising system BMACs as protected (with the sticky bit set) and it is also used for CMAC flush in some single-homing scenarios that will be described later.
-
Route type 3 — Inclusive Multicast route — This route type is used for the advertisement of the I-VPLS ISIDs (no Epipes) and the desired multicast tree for each of them. The ISIDs are encoded in the Ethernet-tag field of the Network Layer Reachability Information (NLRI). When the B-VPLS is created and enabled, an Inclusive Multicast route with ISID = 0 is advertised. This is for the creation of the default multicast tree.
When EVPN multi-homing is used in an ISID, route type 4 (Ethernet Segment (ES) route) is used. In PBB-EVPN, there is no route type 1 advertised when multi-homing is used on the ISID services (I-VPLS and Epipes). Only route type 4 is used, and in the same way as it is for EVPN-MPLS. See the EVPN for MPLS Tunnels example for more information about ES routes, how they are formed, and how their RT/RD values are populated.
Configuration
This example describes the basic PBB-EVPN configuration first (without multi-homing) and how the flood containment is handled in PBB-EVPN. Flood containment refers to the efficient distribution of the BUM traffic generated for an ISID.
Networks are not always greenfield, so a smooth migration of PBB-EVPN from PBB-VPLS is required to minimize the effect on existing services. This example also describes this migration, starting from a common PBB-VPLS configuration.
Finally, this example describes the configuration of PBB-EVPN multi-homing.
The same setup described in the VPN for MPLS tunnels example is used:
-
Four PEs in the core (PE-2, PE-3, PE-4, and PE-5).
-
The PEs are interconnected in the same way as explained in EVPN for MPLS Tunnels with the same IP addressing, IS-IS, transport LDP, and BGP peering configuration. There is not any difference with the basic infrastructure. See the EVPN for MPLS Tunnels chapter if more information is required.
-
When configuring multi-homing, MTU-1 and MTU-6 are connected to the core.
PBB-EVPN configuration without multi-homing
PBB-EVPN network without multi-homing shows the example topology used in this chapter.
When configuring PBB-EVPN:
-
There is no difference at the access side (I-VPLS and Epipe configuration) compared to other PBB technologies supported in SR OS, such as Shortest Path Bridging for MAC (SPBM) or PBB-VPLS.
-
The B-VPLS becomes an EVPN-MPLS service, where bgp-evpn mpls is added.
The following output shows an example of a basic configuration in PE-3. B-VPLS 1000 is bgp-evpn enabled and I-VPLS 1001 and Epipe 1002 are linked to B-VPLS 1000.
# on PE-3:
configure {
service {
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:03
}
}
bgp 1 {
}
bgp-evpn {
evi 1000
mpls 1 {
admin-state enable
auto-bind-tunnel {
resolution any
}
}
}
}
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
sap 1/2/1:1001 {
}
}
epipe "Epipe 1002" {
admin-state enable
service-id 1002
customer "1"
pbb {
tunnel {
backbone-vpls-service-name "B-VPLS 1000"
isid 1002
backbone-dest-mac 00:00:00:00:00:05
}
}
sap 1/2/1:1002 {
}
}
In the preceding output, there is no new configuration needed for I-VPLS/Epipe services. As for the B-VPLS, the output shows the minimum configuration required. If needed, the following parameters can be modified in the bgp-evpn context:
[ex:/configure service vpls "B-VPLS 1000"]
A:admin@PE-2# bgp-evpn ?
bgp-evpn
accept-ivpls-evpn- - Accept non-zero ethernet-tag MAC routes and process for CMAC
flush flushing
apply-groups - Apply a configuration group at this level
apply-groups-exclude - Exclude a configuration group at this level
evi - EVPN ID
incl-mcast-orig-ip - Originating IP address
isid-route-target + Enter the isid-route-target context
mac-duplication + Enter the mac-duplication context
mpls + Enter the mpls list instance
routes + Enter the routes context
vxlan + Enter the vxlan list instance
The following parameters can be modified in the bgp-evpn mpls 1 context:
[ex:/configure service vpls "B-VPLS 1000" bgp-evpn]
A:admin@PE-2# mpls 1 ?
mpls
admin-state - Administrative state of BGP EVPN MPLS
apply-groups - Apply a configuration group at this level
apply-groups-exclude - Exclude a configuration group at this level
auto-bind-tunnel + Enter the auto-bind-tunnel context
control-word - Enable/disable setting the CW bit in the label message.
default-route-tag - Default route tag
ecmp - Maximum ECMP routes information
entropy-label - Enable/disable use of entropy-label.
fdb + Enter the fdb context
force-vc-forwarding - VC forwarding action
ingress-replication- - Use the same label as the one advertised for unicast traffic
bum-label
oper-group - Operational-Group identifier.
route-next-hop + Enter the route-next-hop context
send-tunnel-encap + Enter the send-tunnel-encap context
split-horizon-group - Split horizon group
A detailed description of these commands is included in the EVPN for MPLS Tunnels chapter. In addition to the preceding commands, the following service>(b-)vpls>pbb commands are relevant for PBB-EVPN in the B-VPLS service:
-
force-qtag-forwarding allows the transparent transport of the customer 802.1p bits across the B-VPLS services.
-
source-bmac>address can modify the source BMAC for all the PBB packets containing traffic from non-multi-homed I-VPLS and Epipe services.
-
source-bmac>use-es-bmac-lsb true instructs the system to use an ES-specific BMAC for traffic coming from an ES on an I-VPLS or Epipe.
-
source-bmac>use-mclag-bmac-lsb true instructs the system to use a SAP-specific BMAC for traffic coming from an MC-LAG I-VPLS/Epipe SAP.
Flood containment for I-VPLS services
In general, PBB technologies in SR OS support a way to contain flooding for a specified I-VPLS ISID, so that BUM traffic for that ISID only reaches the PEs where the ISID is locally defined. Each PE creates a Multicast Forwarding Information Base (MFIB) per I-VPLS ISID on the B-VPLS instance. That MFIB supports SAP/SDP-binding endpoints that can be populated by:
-
Multiple MAC Registration Protocol (MMRP) in regular PBB-VPLS
-
IS-IS in SPBM
In PBB-EVPN, B-VPLS EVPN destinations can be added to the MFIBs using EVPN Inclusive Multicast Ethernet tag routes when they include the ISID in the Ethernet-tag. By default, when a B-VPLS is successfully enabled (admin-state enable), the PE advertises:
-
An Inclusive Multicast route for ISID = 0 — This allows the remote PEs to add the advertising PE to the default-multicast-list for the B-VPLS.
-
An Inclusive Multicast route for each local ISID defined in the system (a local ISID includes configured I-VPLS and static-ISIDs) — This allows the remote PEs to create MFIB entries in the B-VPLS for the received ISIDs.
Because EVPN destinations, B-SAPs, and B-spoke-SDPs can coexist in the same B-VPLS, be aware of the different flooding lists created and how they are used in a B-VPLS. PBB-EVPN — flooding lists illustrates this concept with an example for B-VPLS 1000 in PE-1. The assumptions are:
-
I-VPLS 1001 is created in PE-1, PE-2, and PE-4 only.
-
PE-1, PE-2, PE-3, PE-4, and PE-5 support BGP-EVPN in B-VPLS 1000.
-
PE-6 and PE-7 only support spoke-SDPs.
-
PE-1 is connected to all six PEs.
Figure 3. PBB-EVPN — flooding lists 
In this situation, PE-1 creates two flooding lists in B-VPLS 1000:
-
Default-multicast-list — composed of:
-
All the EVPN PEs that advertised ISID = 0 (PE-2, PE-3, PE-4, PE-5).
-
All the B-spoke-SDPs (or B-SAPs) (PE-6, PE-7).
-
All the EVPN PEs that advertised ISID 1001 and no ISID 0 (if an isid-policy is created in PE-1 stating use-def-mcast for ISID 1001). Note: third-party PEs may not advertise ISID = 0, but only non-zero ISIDs.
-
-
MFIB for ISID 1001 is composed of:
-
All the EVPN PEs that advertised ISID 1001 (PE-2 and PE-4) unless there is an ISID-policy in PE-1 stating use-def-mcast for ISID 1001.
-
Static-ISIDs defined in manual B-spoke-SDPs and B-SAPs (static-ISIDs cannot be created on BGP-AD auto-discovered B-spoke-SDPs).
-
Based on the above, when BUM traffic is sent to I-VPLS 1001 on PE-1:
-
The traffic is encapsulated in PBB with the group BMAC for ISID 1001 and sent (by default) to the MFIB created for ISID 1001 (PE-2 and PE-4).
-
If an ISID-policy is added with use-def-mcast for ISID 1001, the BUM traffic is encapsulated in PBB with the group BMAC for ISID 1001 and sent to the default-multicast-list, that is, all six remote PEs.
Referring to PBB-EVPN network without multi-homing, the following output illustrates the use of the ISID-policy in PBB-EVPN. PE-2 does not have any ISID-policy configured; when it receives BUM traffic from the local I-VPLS 1001, it uses the MFIB for ISID 1001:
# on PE-2:
configure {
service {
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:02
}
}
bgp 1 {
}
bgp-evpn {
evi 1000
mpls 1 {
admin-state enable
auto-bind-tunnel {
resolution any
}
}
}
}
[/]A:admin@PE-2# show service id 1000 mfib
===============================================================================
Multicast FIB, Service 1000
===============================================================================
Source Address Group Address Port Id Svc Id Fwd
Blk
-------------------------------------------------------------------------------
* 01:1e:83:00:03:e9 b-mpls:192.0.2.3:524282 Local Fwd
b-mpls:192.0.2.4:524282 Local Fwd
b-mpls:192.0.2.5:524282 Local Fwd
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
An ISID-policy can be added to modify this behavior and allow PE-2 to use the default multicast list. If I-VPLS 1001 exists in all the remote PEs (as in this example), using the default multicast list is as efficient as using the MFIB and saves expensive MFIB resources. In the following output, as soon as the ISID-policy is added, the MFIB entries for ISID 1001 are removed and PE-2 starts using the default multicast list.
# on PE-2:
configure {
service {
vpls "B-VPLS 1000" {
isid-policy {
entry 10 {
use-def-mcast true
range {
start 1001
end 2000
}
}
The MFIB on PE-2 does not contain any entries for ISID 1001 anymore, as follows:
[/]
A:admin@PE-2# show service id 1000 mfib
===============================================================================
Multicast FIB, Service 1000
===============================================================================
Source Address Group Address Port Id Svc Id Fwd
Blk
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Number of entries: 0
===============================================================================
PBB-VPLS to PBB-EVPN migration
The principles required for migrating a PBB-VPLS network to PBB-EVPN are explained in the VPLS to EVPN-MPLS integration section of the EVPN for MPLS Tunnels chapter. Those principles are also applicable to EVPN destinations and spoke-SDPs in the B-VPLS and can be summarized in three points:
-
Systems with an EVPN destination and SDP-binding to the same far-end IP bring down the SDP-binding. This avoids loops when both constructs exist in the same network.
-
SDP-bindings and EVPN destinations can be placed in the same Split-Horizon Group (SHG). When traffic from an SDP-binding/EVPN destination belonging to that SHG is received on a PE, it is never forwarded to another SDP-binding or EVPN destination on the same SHG.
-
MAC addresses learned on an SDP-binding or SAP, that belong to an SHG where EVPN destinations are also created, are not advertised in BGP-EVPN.
Based on those principles, this section describes how to migrate a PBB-VPLS network to PBB-EVPN. The network in PBB-EVPN network without multi-homing represents a regular PBB-VPLS network that needs to be migrated to PBB-EVPN.
In that network, the four PEs are running BGP-AD and TLDP for the discovery and setup of the pseudowires in the B-VPLS instance. The advantage of this configuration is that the migration can be done node by node and with minimum impact on customer service.
Initial configuration
Initially, the network is configured for PBB-VPLS with BGP-AD in B-VPLS 1000. The EVPN family is to be added. At the access, I-VPLS 1001 is connected to the CEs. As an example, the configuration in PE-3 is shown. An equivalent configuration exists in the other three PEs.
The EVPN family is added to the BGP configuration because PBB-EVPN uses this address family. Assuming there are redundant Route Reflectors (RRs), the addition of EVPN can be done without service impact. In this example, the assumption is that the PEs are already configured with the EVPN family.
# on PE-3:
configure {
router "Base" {
autonomous-system 64500
bgp {
vpn-apply-export true
vpn-apply-import true
rapid-withdrawal true
peer-ip-tracking true
split-horizon true
rapid-update {
evpn true
}
group "internal" {
peer-as 64500
family {
l2-vpn true
evpn true
}
}
neighbor "192.0.2.2" {
group "internal"
}
}
# on PE-3:
configure {
service {
pw-template "PW1" {
pw-template-id 1
split-horizon-group {
name "CORE"
}
}
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:03
}
}
bgp 1 {
pw-template-binding "PW1" {
}
}
bgp-ad {
admin-state enable
vpls-id "64500:1000"
}
}
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
sap 1/2/1:1001 {
}
}
[/]
A:admin@PE-3# show service id 1000 base
===============================================================================
Service Basic Information
===============================================================================
Service Id : 1000 Vpn Id : 0
Service Type : b-VPLS
---snip---
Oper Backbone Src : 00:00:00:00:00:03
Use SAP B-MAC : Disabled
i-Vpls Count : 1
Epipe Count : 1
Use ESI B-MAC : Disabled
-------------------------------------------------------------------------------
Service Access & Destination Points
-------------------------------------------------------------------------------
Identifier Type AdmMTU OprMTU Adm Opr
-------------------------------------------------------------------------------
sdp:32765:4294967293 SB(192.0.2.5) BgpAd 0 8978 Up Up
sdp:32766:4294967294 SB(192.0.2.4) BgpAd 0 8978 Up Up
sdp:32767:4294967295 SB(192.0.2.2) BgpAd 0 8978 Up Up
===============================================================================
* indicates that the corresponding row element may have been truncated.
Multiple MAC Registration Protocol (MMRP) is not used in the B-VPLS instance. If it were enabled, MMRP would have to be disabled in the network before this migration. If there are ISIDs using B-VPLS SDP-bindings to reach some remote locations and B-VPLS EVPN destinations to reach others, the default multicast list must be used in the current release (the MFIB cannot be used if there is a mix of both types). Therefore, during the migration process, the ISIDs must be added to the default multicast list.
-
Add service-level SHG (if not already there).
From the first node being migrated to PBB-EVPN to all nodes migrated, PBB-VPLS and PBB-EVPN have to coexist within the same meshed network. That is, EVPN-MPLS destinations and SDP-bindings need to be defined in the same split-horizon group. Therefore, if there is no split-horizon group defined in the B-VPLS, the first step is to add it. In this example, the split-horizon group is defined at the config>service>pw-template>level; therefore, it has to be added at the B-VPLS level.
Note:When the service>split-horizon-group is removed, an eval-pw-template must be performed.
Note:After adding the split-horizon-group at the service level, an eval-pw-template must be performed again so that the SDP-bindings take the new SHG configuration.
Note:During the time between the split-horizon-group being removed and added back again, the SDP-bindings can forward BUM traffic to each other, so this operation must be done carefully to avoid loops.
Assuming that the first node to be migrated is PE-3, the following output shows the procedure for adding the split-horizon-group at the service level.
# on PE-3: configure exclusive service { pw-template "PW1" { pw-template-id 1 delete split-horizon-group } commit[/] A:admin@PE-3# tools perform service id 1000 eval-pw-template 1 allow-service-impact eval-pw-template succeeded for Svc 1000 32765:4294967293 Policy 1 eval-pw-template succeeded for Svc 1000 32766:4294967294 Policy 1 eval-pw-template succeeded for Svc 1000 32767:4294967295 Policy 1# on PE-3: configure exclusive service { vpls "B-VPLS 1000" { bgp 1 { pw-template-binding "PW1" { split-horizon-group "CORE" } } split-horizon-group "CORE" { } commit[/] A:admin@PE-3# tools perform service id 1000 eval-pw-template 1 allow-service-impact eval-pw-template succeeded for Svc 1000 32765:4294967293 Policy 1 eval-pw-template succeeded for Svc 1000 32766:4294967294 Policy 1 eval-pw-template succeeded for Svc 1000 32767:4294967295 Policy 1[ex:configure service vpls "B-VPLS 1000"] A:admin@PE-3# info admin-state enable service-id 1000 customer "1" service-mtu 2000 pbb-type b-vpls pbb { source-bmac { address 00:00:00:00:00:03 } } bgp 1 { pw-template-binding "PW1" { split-horizon-group "CORE" } } bgp-ad { admin-state enable vpls-id "64500:1000" } split-horizon-group "CORE" { } -
Add BGP-EVPN and ISID-policy configuration to the B-VPLS.
After the B-VPLS is configured with the split horizon group, the BGP-EVPN configuration (with admin-state disabled) and ISID-policy can be added, as follows.
# on PE-3: configure { service { vpls "B-VPLS 1000" { bgp-evpn { evi 1000 mpls 1 { admin-state disable # must be disabled split-horizon-group "CORE" auto-bind-tunnel { resolution any } } } isid-policy { entry 10 { use-def-mcast true range { start 1001 end 3000 } } } -
Enable BGP-EVPN MPLS on the PE.
When the configuration is ready, the bgp-evpn mpls 1 context can be enabled, as follows:
# on PE-3: configure { service { vpls "B-VPLS 1000" { bgp-evpn { mpls 1 { admin-state enable } }Enabling the bgp-evpn mpls 1 context triggers a route-refresh message for the EVPN family from PE-3, but no changes happen because PE-3 does not create any EVPN destinations until it imports EVPN routes from the other PEs. The three spoke-SDPs to the remote PEs are still up.
-
Repeat steps 1 to 3 for the second PE (PE-5).
The same steps 1 to 3 are repeated for PE-5. When the bgp-evpn mpls 1 context is enabled, PE-5 sends a route-refresh and gets the BGP-EVPN routes from PE-3. As a result of that, PE-3 brings down the spoke-SDP to PE-5 and creates an EVPN destination to PE-5. The same process happens in PE-5. The following CLI output shows the received routes in PE-3 and spoke-SDP going down.
45 2021/03/05 09:57:37.206 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2 "Peer 1: 192.0.2.2: UPDATE Peer 1: 192.0.2.2 - Received BGP UPDATE: Withdrawn Length = 0 Total Path Attr Length = 110 Flag: 0x90 Type: 14 Len: 47 Multiprotocol Reachable NLRI: Address Family EVPN NextHop len 4 NextHop 192.0.2.5 Type: EVPN-INCL-MCAST Len: 17 RD: 64500:1000, tag: 1001, orig_addr len: 32, orig_addr: 192.0.2.5 Type: EVPN-INCL-MCAST Len: 17 RD: 64500:1000, tag: 0, orig_addr len: 32, orig_addr: 192.0.2.5 Flag: 0x40 Type: 1 Len: 1 Origin: 0 Flag: 0x40 Type: 2 Len: 0 AS Path: Flag: 0x40 Type: 5 Len: 4 Local Preference: 100 Flag: 0x80 Type: 9 Len: 4 Originator ID: 192.0.2.5 Flag: 0x80 Type: 10 Len: 4 Cluster ID: 1.1.1.1 Flag: 0xc0 Type: 16 Len: 16 Extended Community: target:64500:1000 bgp-tunnel-encap:MPLS Flag: 0xc0 Type: 22 Len: 9 PMSI: Tunnel-type Ingress Replication (6) Flags: (0x0)[Type: None BM: 0 U: 0 Leaf: not required] MPLS Label 8388464 Tunnel-Endpoint 192.0.2.5 "Log 99 shows that spoke SDP 32765:4294967293 is operationally down:
184 2021/03/05 09:57:39.472 CET MINOR: SVCMGR #2313 Base "Status of SDP Bind 32765:4294967293 in service 1000 (customer 1) peer PW status bits changed to pwNotForwarding " 183 2021/03/05 09:57:37.207 CET MINOR: SVCMGR #2306 Base "Status of SDP Bind 32765:4294967293 in service 1000 (customer 1) changed to admin=up oper=down flags=evpnRouteConflict " 182 2021/03/05 09:57:37.207 CET MINOR: SVCMGR #2326 Base "Status of SDP Bind 32765:4294967293 in service 1000 (customer 1) local PW status bits changed to pwNotForwarding "Spoke SDP 32765:4294967293 is the spoke SDP toward PE-5 and it is kept down:
[/] A:admin@PE-3# show service id 1000 base ---snip--- ------------------------------------------------------------------------------- Service Access & Destination Points ------------------------------------------------------------------------------- Identifier Type AdmMTU OprMTU Adm Opr ------------------------------------------------------------------------------- sdp:32765:4294967293 SB(192.0.2.5) BgpAd 0 8978 Up Down sdp:32766:4294967294 SB(192.0.2.4) BgpAd 0 8978 Up Up sdp:32767:4294967295 SB(192.0.2.2) BgpAd 0 8978 Up Up =============================================================================== * indicates that the corresponding row element may have been truncated.The reason why the spoke SDP toward PE-5 is down is an EVPN route conflict:
[/] A:admin@PE-3# show service id 1000 sdp 32765:4294967293 detail | match Flag post-lines 1 Flags : PWPeerFaultStatusBits EvpnRouteConflictAn EVPN destination to PE-5 is created:
[/] A:admin@PE-3# show service id 1000 evpn-mpls =============================================================================== BGP EVPN-MPLS Dest =============================================================================== TEP Address Egr Label Num. MACs Mcast Last Change Transport:Tnl Sup BCast Domain ------------------------------------------------------------------------------- 192.0.2.5 524279 1 bum 03/05/2021 09:57:37 ldp:65539 No ------------------------------------------------------------------------------- Number of entries : 1 ------------------------------------------------------------------------------- =============================================================================== ---snip--- -
Repeat Steps 1 to 3 for the rest of the PEs (PE-2, PE-4).
The same process is repeated in all the PEs, node by node. The service impact for the I-VPLS 1001 is minimal.
-
(Optional) Remove the ISID policy.
When all the PEs in the B-VPLS 1000 are migrated, the ISID policy can optionally be removed, node by node. This forces the B-VPLS instance to start using the MFIB to send I-VPLS BUM traffic to the remote nodes. This has no effect on Epipes (traffic is always unicast for Epipes).
Before removing the ISID policy and starting to use the MFIB, it is recommended to check that the Inclusive Multicast routes for an ISID to the remote PEs are all active. Otherwise, connectivity for BUM traffic could be interrupted if any of the expected routes are not active. This is illustrated for PE-3.
[/] A:admin@PE-3# show service id 1000 evpn-mpls =============================================================================== BGP EVPN-MPLS Dest =============================================================================== TEP Address Egr Label Num. MACs Mcast Last Change Transport:Tnl Sup BCast Domain ------------------------------------------------------------------------------- 192.0.2.2 524282 1 bum 03/05/2021 10:00:32 ldp:65537 No 192.0.2.4 524282 1 bum 03/05/2021 10:00:33 ldp:65538 No 192.0.2.5 524279 1 bum 03/05/2021 09:57:37 ldp:65539 No ------------------------------------------------------------------------------- Number of entries : 3 ------------------------------------------------------------------------------- =============================================================================== ---snip---The routes for ISID 1001 are valid and used by BGP (flags u*>i):
[/] A:admin@PE-3# show router bgp routes evpn incl-mcast tag 1001 =============================================================================== BGP Router ID:192.0.2.3 AS:64500 Local AS:64500 =============================================================================== Legend - Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid l - leaked, x - stale, > - best, b - backup, p - purge Origin codes : i - IGP, e - EGP, ? - incomplete =============================================================================== BGP EVPN Inclusive-Mcast Routes =============================================================================== Flag Route Dist. OrigAddr Tag NextHop ------------------------------------------------------------------------------- u*>i 64500:1000 192.0.2.2 1001 192.0.2.2 u*>i 64500:1000 192.0.2.4 1001 192.0.2.4 u*>i 64500:1000 192.0.2.5 1001 192.0.2.5 ------------------------------------------------------------------------------- Routes : 3 ===============================================================================There are no entries in the MFIB:
[/] A:admin@PE-3# show service id 1000 mfib =============================================================================== Multicast FIB, Service 1000 =============================================================================== Source Address Group Address Port Id Svc Id Fwd Blk ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- Number of entries: 0 ===============================================================================The ISID policy is removed as follows:
# on all PEs: configure { service { vpls "B-VPLS 1000" { delete isid-policyAfter removing the ISID-policy, the MFIB is populated with entries for the ISID 1001 group BMAC to the three remote PEs where ISID 1001 is defined:
[/] A:admin@PE-3# show service id 1000 mfib =============================================================================== Multicast FIB, Service 1000 =============================================================================== Source Address Group Address Port Id Svc Id Fwd Blk ------------------------------------------------------------------------------- * 01:1e:83:00:03:e9 b-mpls:192.0.2.2:524282 Local Fwd b-mpls:192.0.2.4:524282 Local Fwd b-mpls:192.0.2.5:524279 Local Fwd ------------------------------------------------------------------------------- Number of entries: 1 =============================================================================== -
(Optional) Remove the BGP-AD configuration.
The BGP-AD configuration can stay in the B-VPLS services. However, when the entire network is migrated to PBB-EVPN, all the spoke-SDPs will be operationally down and, even if they are not forwarding traffic, they consume resources in the system. Consider removing the BGP-AD configuration and, therefore, the spoke-SDPs.
The following example shows the removal of BGP-AD in PE-4. Be aware that when BGP-AD is removed from the configuration, if the RD/RT was derived from the VPLS ID (as in this example), a new RD/RT must be auto-derived for the service. Therefore, new updates will be sent for all the EVPN NLRIs, as shown in the following output.
[/] A:admin@PE-4# show service id 1000 bgp =============================================================================== BGP Information =============================================================================== Bgp Instance : 1 Vsi-Import : None Vsi-Export : None Route Dist : None Oper Route Dist : 64500:1000 Oper RD Type : derivedVpls Rte-Target Import : None Rte-Target Export: None Oper RT Imp Origin : derivedVpls Oper RT Import : 64500:1000 Oper RT Exp Origin : derivedVpls Oper RT Export : 64500:1000 PW-Template Id : 1 PW-Template SHG : CORE Oper Group : None Mon Oper Group : None BFD Template : None BFD-Enabled : no BFD-Encap : ipv4 Import Rte-Tgt : None ------------------------------------------------------------------------------- ===============================================================================On all PEs, the BGP-AD configuration and the PW template binding are removed, as follows:
# on all PEs: configure { service { vpls "B-VPLS 1000" { delete bgp-ad bgp 1 { delete pw-template-binding "PW1"After BGP-AD is disabled, the spoke SDP bindings are deleted. The following messages are logged in log 99 on PE-4:
179 2021/03/05 10:05:41.204 CET MAJOR: SVCMGR #2319 Base "Dynamic bgp-l2vpn SDP 32765 (192.0.2.5) was deleted." 178 2021/03/05 10:05:41.204 CET MINOR: SVCMGR #2303 Base "Status of SDP 32765 changed to admin=down oper=down" 177 2021/03/05 10:05:41.204 CET MAJOR: SVCMGR #2320 Base "Service Id 1000, Dynamic bgp-l2vpn SDP Bind Id 32765:4294967292 was deleted." 176 2021/03/05 10:05:41.194 CET MINOR: SVCMGR #2306 Base "Status of SDP Bind 32765:4294967292 in service 1000 (customer 1) changed to admin=down oper=down flags="Initially, the RD/RT was derived from the VPLS ID (64500:1000). After the BGP-AD configuration is removed, a new RD and RT must be auto-derived from the EVI:
[/] A:admin@PE-4# show service id 1000 bgp =============================================================================== BGP Information =============================================================================== Bgp Instance : 1 Vsi-Import : None Vsi-Export : None Route Dist : None Oper Route Dist : 192.0.2.4:1000 Oper RD Type : derivedEvi Rte-Target Import : None Rte-Target Export: None Oper RT Imp Origin : derivedEvi Oper RT Import : 64500:1000 Oper RT Exp Origin : derivedEvi Oper RT Export : 64500:1000 PW-Template Id : None ------------------------------------------------------------------------------- ===============================================================================In this case, the system picks up the RD in the following order:
-
Manual RD or auto-RD always take precedence when configured.
-
If no manual/auto-RD, the RD is derived from the bgp-ad vpls-id.
-
If no manual/auto-rd/vpls-id configuration, the RD is derived from the bgp evpn evi.
-
If no manual/auto-rd/vpls-id/evi configuration, there will be no RD and the service will fail.
-
If in the migration from BGP-AD to BGP-EVPN, the advertisement of new updates is not needed, the initial configuration must include manual/auto-RDs. If manual/auto-RDs were not included, disabling BGP-AD would not cause the change of RD and the consequent BGP updates.
PBB-EVPN multi-homing
This section provides configuration guidelines for PBB-EVPN multi-homing. In the same way that EVPN-MPLS supports single-active and all-active multi-homing, PBB-EVPN can also be configured to support both modes. The same Ethernet segment that is used for regular EVPN-MPLS service SAPs and spoke-SDPs can be shared with I-VPLS/Epipe SAPs and spoke-SDPs.
PBB-EVPN multi-homing shows the example topology used in this section.
MTU-1 and MTU-6 have been added to the network (compared to PBB-EVPN network without multi-homing). I-VPLS 1001 has two new sites that are multi-homed to the PBB-EVPN network. MTU-1 uses all-active multi-homing, whereas MTU-6 is connected to a single-active ES. As with EVPN-MPLS, all-active multi-homing is only supported when a LAG is used at the access. Single-active multi-homing can be supported with regular Ethernet ports (that can form an independent LAG per PE) or SDPs.
RFC 7623 describes two types of system BMAC assignments that a PE can implement in a B-VPLS when ESs are present:
-
Shared BMAC addresses that can be used for all the single-homed CEs and a number of multi-homed CEs connected to Ethernet-segments.
-
Dedicated BMAC addresses per Ethernet segment.
In this chapter and in SR OS terminology:
-
A shared BMAC address (in IETF) is a source BMAC address as configured in service>(b)vpls>pbb>source-bmac. All the I-VPLS/Epipe traffic coming from single-homed CEs is sent encapsulated in a PBB packet with that source BMAC address.
-
A dedicated-BMAC per ES (in IETF) is an ES BMAC address as activated in service>(b)vpls>pbb>source-bmac>use-es-bmac-lsb and generated from the combination of vpls>pbb>source-bmac plus ethernet-segment>pbb>source-bmac-lsb. If configured, any I-VPLS/Epipe traffic coming from an ES is encapsulated in a PBB packet with the ES-BMAC address as the source BMAC address.
The system allows the following user choices per B-VPLS and ES:
-
A dedicated ES BMAC address per ES can be used. In that case, the pbb>source-bmac>use-es-bmac-lsb command is configured in the B-VPLS. In all-active multi-homing, all the PEs that are part of the ES source the PBB packets with the same source ES BMAC address; single-active multi-homing requires the use of a different ES BMAC address per PE.
-
A non-dedicated source BMAC address can be used (this is only possible in single-active multi-homing). In this case, the user does not configure pbb>source-bmac>use-es-bmac-lsb and the regular source BMAC address is used for the traffic. A different source BMAC address has to be advertised per PE.
As discussed, single-active multi-homing can use source BMAC addresses or ES BMAC addresses. Using one type or another has a different impact on CMAC flushing, as illustrated in The use of ES BMAC to minimize CMAC flush.
-
If ES BMAC addresses are used, as shown on the right-hand side of The use of ES BMAC to minimize CMAC flush, a less-impacting CMAC flush is achieved, therefore minimizing the flooding after ES failures. In the case of ES failure, PE-1 withdraws the ES BMAC address 00:12 and the remote PE-3 only flushes the CMACs associated with that ES BMAC address (only the CMAC addresses behind the CE are flushed).
-
If source BMAC addresses are used, as shown on the left-hand side of The use of ES BMAC to minimize CMAC flush, in the case of ES failure, a BGP update with higher sequence number is issued by PE-1 and the remote PE-3 flushes all the CMAC addresses associated with the source BMAC address. Therefore, all the CMAC addresses behind the B-VPLS of the PEs will be flushed, as opposed to only the CMAC addresses behind the CE of the Ethernet Service Instances (ESIs).
Figure 5. The use of ES BMAC to minimize CMAC flush 
PBB-EVPN multi-homing supported combinations in SR OS shows the PBB-EVPN multi-homing combinations supported in the current release in the topology of PBB-EVPN multi-homing.
|
CE Connectivity |
PE Connectivity |
PE Redundancy |
BMAC Assignment |
I-VPLS Support |
Epipe Support |
|---|---|---|---|---|---|
|
LAG (LACP optional) |
LAG SAP |
EVPN MH all-active |
use-es-bmac-lsb (shared BMAC) |
Yes |
Yes |
|
Ethernet ports (no LAG) |
LAG SAP or port SAP |
EVPN MH single-active |
use-es-bmac-lsb (dedicated per PE) |
Yes |
No |
|
Ethernet ports (no LAG) |
LAG SAP or port SAP |
EVPN MH single-active |
source-bmac address (dedicated per PE) |
Yes |
No |
|
MPLS |
spoke-SDP |
EVPN MH single-active |
source-bmac address (dedicated per PE) |
Yes |
No |
|
MPLS |
spoke-SDP |
EVPN MH single-active |
use-es-bmac-lsb (dedicated per PE) |
Yes |
No |
As an example, the configurations of the first, and last two, rows (LAG SAP all-active, MPLS source-BMAC, and MPLS ES-BMAC, respectively) will be discussed in the following three sections.
PBB-EVPN all-active multi-homing for I-VPLS and Epipes
PBB-EVPN multi-homing shows a PBB-EVPN network where ESI-23 is configured as an all-active multi-homing ES on PE-2 and PE-3. Two services are using ESI-23: I-VPLS 1001 and Epipe 1003. The following output shows the relevant configuration in PE-2:
# on PE-2:
configure {
service {
pbb {
mac "PE-5" {
address 00:00:00:00:00:05
}
}
system {
bgp {
evpn {
ethernet-segment "ESI-23" {
admin-state enable
esi 01:00:00:00:00:23:00:00:00:01
multi-homing-mode all-active
df-election {
es-activation-timer 3
}
association {
lag "lag-1" {
}
}
pbb {
source-bmac-lsb 23-23
}
}
}
}
}
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:02
use-es-bmac-lsb true
}
}
bgp 1 {
}
bgp-evpn {
evi 1000
mpls 1 {
admin-state enable
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel {
resolution any
}
}
}
split-horizon-group "CORE" {
}
}
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
sap lag-1:1001 {
}
}
epipe "Epipe 1003" {
admin-state enable
service-id 1003
customer "1"
pbb {
tunnel {
backbone-vpls-service-name "B-VPLS 1000"
isid 1003
backbone-dest-mac-name "PE-5"
}
}
sap lag-1:1003 {
}
}
The following output shows the relevant configuration in PE-3:
# on PE-3:
configure {
service {
pbb {
mac "PE-5" {
address 00:00:00:00:00:05
}
}
system {
bgp {
evpn {
ethernet-segment "ESI-23" {
admin-state enable
esi 01:00:00:00:00:23:00:00:00:01
multi-homing-mode all-active
df-election {
es-activation-timer 3
}
association {
lag "lag-1" {
}
}
pbb {
source-bmac-lsb 23-23
}
}
}
}
}
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:03
use-es-bmac-lsb true
}
}
bgp 1 {
}
bgp-evpn {
evi 1000
mpls 1 {
admin-state enable
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel {
resolution any
}
}
}
split-horizon-group "CORE" {
}
}
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
sap 1/2/1:1001 {
}
sap lag-1:1001 {
}
}
epipe "Epipe 1003" {
admin-state enable
service-id 1003
customer "1"
pbb {
tunnel {
backbone-vpls-service-name "B-VPLS 1000"
isid 1003
backbone-dest-mac-name "PE-5"
}
}
sap lag-1:1003 {
}
}
The preceding configuration shows that Epipe 1003 has a PBB tunnel pointing at the PE-5 source-BMAC. Epipe 1003 has the following configuration in PE-5 (the PBB tunnel points at the ESI-23 ES-BMAC):
# on PE-5:
configure {
service {
pbb {
mac "ES-MAC-23" {
address 00:00:00:00:23:23
}
}
epipe "Epipe 1003" {
admin-state enable
service-id 1003
customer "1"
pbb {
tunnel {
backbone-vpls-service-name "B-VPLS 1000"
isid 1003
backbone-dest-mac-name "ES-MAC-23"
}
}
sap 1/2/1:1003 {
}
}
Source-BMAC addresses and ES-BMAC addresses are distributed in BGP-EVPN. PE-2 and PE-3 will each advertise their own source-BMAC in a MAC route with ESI-0 and the shared ES-BMAC address with ESI-MAX (as per the RFC 7623). The ES-BMAC address that each PE uses in a B-VPLS is derived from the configured service>(b)vpls>pbb>source-bmac (four high-order bytes) and the ESI-23 configured source-bmac-lsb. In this example, PE-2 and PE-3 will both derive ES-BMAC 00:00:00:00:23:23. For both PEs to derive the required same ES-BMAC address, the four high-order bytes of the source-BMAC address must match on both PEs.
The es-bmac-table-size parameter modifies the default value (8) for the maximum number of ES-BMAC addresses that can be associated with the Ethernet segment across different B-VPLS services. When source-bmac-lsb is configured, the associated es-bmac-table-size is reserved out of the total FDB space.
The following outputs show the source-BMAC addresses and ES-BMAC address and how they are advertised and installed in the B-VPLS FDB.
[/]
A:admin@PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" | match BMAC
Source BMAC LSB : 23-23
The following output shows that ES-BMAC is used and that the operational source-BMAC is 00:00:00:00:00:02.
[/]
A:admin@PE-2# show service id 1000 base
===============================================================================
Service Basic Information
===============================================================================
Service Id : 1000 Vpn Id : 0
Service Type : b-VPLS
---snip---
Oper Backbone Src : 00:00:00:00:00:02
Use SAP B-MAC : Disabled
i-Vpls Count : 1
Epipe Count : 1
Use ESI B-MAC : Enabled
---snip---
The source BMAC LSB is configured with the same value on PE-2 and PE-3. The two low-order bytes of the ES-BMAC will be 23:23.
[/]
A:admin@PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" | match BMAC
Source BMAC LSB : 23-23
On PE-3, ES-BMAC is used and the operational source BMAC is 00:00:00:00:00:03, as follows:
[/]
A:admin@PE-3# show service id 1000 base
===============================================================================
Service Basic Information
===============================================================================
Service Id : 1000 Vpn Id : 0
Service Type : b-VPLS
---snip---
Oper Backbone Src : 00:00:00:00:00:03
Use SAP B-MAC : Disabled
i-Vpls Count : 1
Epipe Count : 2
Use ESI B-MAC : Enabled
---snip---
On PE-2, the FDB for B-VPLS 1000 has an entry for each of the other PEs. PEs do not show their own system BMAC addresses in the FDB:
[/]
A:admin@PE-2# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/05/21 10:05:43
192.0.2.3:524282
ldp:65537
1000 00:00:00:00:00:04 mpls: EvpnS:P 03/05/21 10:05:41
192.0.2.4:524282
ldp:65538
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/05/21 10:05:42
192.0.2.5:524279
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
On PE-4, the FDB for B-VPLS 1000 has an entry for each of the other PEs and an entry for the ES-BMAC address of ES ‟ESI-23”:
[/]
A:admin@PE-4# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:02 mpls: EvpnS:P 03/05/21 10:05:44
192.0.2.2:524282
ldp:65538
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/05/21 10:05:43
192.0.2.3:524282
ldp:65537
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/05/21 10:05:42
192.0.2.5:524279
ldp:65539
1000 00:00:00:00:23:23 eES: EvpnS:P 03/05/21 10:07:25
MAX-ESI
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
On PE-4, there are two BGP routes for ES-BMAC address 00:00:00:00:23:23: one with next hop PE-2 and the other with next hop PE-3, as follows:
[/]
A:admin@PE-4# show router bgp routes evpn mac mac-address 00:00:00:00:23:23
===============================================================================
BGP Router ID:192.0.2.4 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.2:1000 00:00:00:00:23:23 ESI-MAX
0 Static LABEL 524282
n/a
192.0.2.2
u*>i 192.0.2.3:1000 00:00:00:00:23:23 ESI-MAX
0 Static LABEL 524282
n/a
192.0.2.3
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
PBB-EVPN all-active multi-homing is based on the same concepts as EVPN-MPLS all-active multi-homing: DF election, split-horizon, and aliasing.
Designated forwarder (DF) election
Only the DF PE for an ISID will send multicast traffic to the ES. The following command shows that PE-3 is the DF PE in ES ‟ESI-23” for ISID 1003:
[/]
A:admin@PE-3# show service system bgp-evpn ethernet-segment name "ESI-23"
isid isid-2 1003
===============================================================================
ISID DF and Candidate List
===============================================================================
Isid SvcId Actv Timer Rem DF DF Last Change
-------------------------------------------------------------------------------
1003 1003 0 yes 03/05/2021 10:07:45
===============================================================================
---snip---
The following command shows the DF and DF candidate list in the ES for all EVIs and all ISIDs:
[/]
A:admin@PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" all
===============================================================================
Service Ethernet Segment
===============================================================================
Name : ESI-23
Eth Seg Type : None
Admin State : Enabled Oper State : Up
ESI : 01:00:00:00:00:23:00:00:00:01
Multi-homing : allActive Oper Multi-homing : allActive
---snip---
===============================================================================
ISID Information
===============================================================================
ISID SvcId Actv Timer Rem DF
-------------------------------------------------------------------------------
1001 1001 0 yes
1003 1003 0 yes
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================
-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
ISID DF Address
-------------------------------------------------------------------------------
1001 192.0.2.2
1001 192.0.2.3
1003 192.0.2.2
1003 192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 4
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---
The DF PE identifies multicast traffic by looking at either the destination BMAC or the EVPN label (which can be unicast or multicast).
In the case of Epipes, there are also DF and non-DF PEs. However, traffic is usually unicast (sent to the PBB tunnel backbone-destination BMAC). The non-DF PE will usually not discard Epipe traffic to the ES, unless the packet comes with an EVPN multicast label. To avoid packet duplication at the CE for Epipes, it is recommended to either:
-
configure discard-unknown on all the B-VPLS instances where there are PBB-Epipes. This will prevent the ingress PE from flooding Epipe traffic if the PBB tunnel BMAC is unknown in the FDB.
-
configure ingress-replication-bum-label true so that, when the PBB tunnel BMAC is unknown in the FDB, the ingress PE sends traffic with a multicast label. The non-DF will discard traffic identified as multicast at Epipes.
Ethernet segment split-horizon
In PBB-EVPN all-active multi-homing, the split-horizon function is not based in the ESI label but in a source BMAC check. When BUM traffic is received on an I-VPLS, the PE will encapsulate it in PBB using the ES-BMAC as source BMAC and the group BMAC for the ISID. When the DF PE for the ISID receives that packet, it will not send it back to the ES if the packet is identified as being originated from the ES itself (based on the ES-BMAC shared between the PEs).
Aliasing
Aliasing is based on the advertisement of the same ES-BMAC with MAX-ESI from the PEs part of the same ES. PE-2 and PE-3 advertise the ES-BMAC 00:00:00:00:23:23 with MAX-ESI (ESI = all FFs, as per the RFC 7623) and as Static (protected). When the remote PEs, PE-4, and PE-5, receive the two routes for the same BMAC and MAX-ESI, they will create a single EVPN-MPLS destination that will give more than one next-hop (in this case 2), as long as ECMP > 1:
[/]
A:admin@PE-4# show service id 1000 evpn-mpls
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address Egr Label Num. MACs Mcast Last Change
Transport:Tnl Sup BCast Domain
-------------------------------------------------------------------------------
192.0.2.2 524282 1 bum 03/05/2021 10:05:44
ldp:65538 No
192.0.2.3 524282 1 bum 03/05/2021 10:05:43
ldp:65537 No
192.0.2.5 524279 1 bum 03/05/2021 10:05:42
ldp:65539 No
-------------------------------------------------------------------------------
Number of entries : 3
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId Num. Macs Last Change
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================
===============================================================================
BGP EVPN-MPLS ES BMAC Dest
===============================================================================
ES BMAC Addr Last Change
-------------------------------------------------------------------------------
00:00:00:00:23:23 03/05/2021 10:07:45
-------------------------------------------------------------------------------
Number of entries: 1
-------------------------------------------------------------------------------
===============================================================================
The EVPN-MPLS ES BMAC destination has two next hops: PE-2 and PE-3.
[/]
A:admin@PE-4# show service id 1000 evpn-mpls es-bmac ieee-address 00:00:00:00:23:23
===============================================================================
BGP EVPN-MPLS ES BMAC Dest
===============================================================================
ES BMAC Addr Last Change
-------------------------------------------------------------------------------
00:00:00:00:23:23 03/05/2021 10:07:45
===============================================================================
===============================================================================
BGP EVPN-MPLS ES BMAC Dest TEP Info
===============================================================================
TEP Address Egr Label Last Change Tunnel-
Transport Id
-------------------------------------------------------------------------------
192.0.2.2 524282 03/05/2021 10:07:25 65538
ldp
192.0.2.3 524282 03/05/2021 10:07:45 65537
ldp
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================
A similar output will be obtained in PE-5. Unicast traffic entering I-VPLS 1001 in either PE-4 or PE-5 will be hashed and load-balanced to PE-2 and PE-3 if the destination CMAC lookup yields an es-bmac-dest:
[/]
A:admin@PE-5# show service id 1001 fdb detail pbb
==============================================================================
Forwarding Database, i-Vpls Service 1001
==============================================================================
MAC Source-Identifier B-Svc b-Vpls MAC Type/Age
Transport:Tnl-Id
------------------------------------------------------------------------------
00:00:10:10:10:10 eES-BMAC: 1000 00:00:00:00:23:23 L/0
00:00:00:00:23:23
00:00:30:30:30:30 b-mpls: 1000 00:00:00:00:00:03 L/0
192.0.2.3:524282
00:00:50:50:50:50 sap:1/2/1:1001 1000 N/A L/0
00:00:60:60:60:60 sdp:56:1001 1000 N/A L/0
==============================================================================
Verify the FDB of I-VPLS 1001 for ES BMAC destination 00:00:00:00:23:23 as follows:
[/]
A:admin@PE-5# show service id 1001 fdb evpn-mpls es-bmac-dest 00:00:00:00:23:23
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:10:10:10:10 eES-BMAC: L/30 03/05/21 10:30:52
00:00:00:00:23:23
-------------------------------------------------------------------------------
No. of Entries: 1
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
If a failure occurs in the ES, the PE will withdraw the ES-BMAC and the remote PEs will remove one next-hop of the ES-BMAC EVPN-MPLS destination.
For PBB-Epipes, aliasing will also work, as long as shared-queuing or policing are enabled on the ingress PE Epipe. In PBB-EVPN multi-homing, Epipe 1003 on PE-5 requires shared-queuing or policing at the ingress SAP. Otherwise, the traffic will be sent to only one PE of the ES (usually to the lower-IP PE).
For more information about the configuration of the Ethernet segment and its parameters, see the EVPN for MPLS Tunnels chapter.
PBB-EVPN single-active multi-homing for I-VPLS with source BMAC addresses
ESI-45 is a single-active Ethernet-segment (see PBB-EVPN multi-homing) with SDPs linked to it. As indicated in PBB-EVPN multi-homing supported combinations in SR OS , only I-VPLS services can be used in this configuration. As described in section PBB-EVPN multi-homing, single-active ES and B-VPLS services can be configured to either use source-BMAC addresses or ES-BMAC addresses. The following configuration shows the former option on PE-4:
# on PE-4:
configure {
service {
system {
bgp {
evpn {
ethernet-segment "ESI-45" {
admin-state enable
esi 01:00:00:00:00:45:00:00:00:01
multi-homing-mode single-active
df-election {
es-activation-timer 3
}
association {
sdp 46 {
}
}
pbb {
source-bmac-lsb 45-04
}
}
}
}
}
sdp 46 {
admin-state enable
delivery-type mpls
ldp true
far-end {
ip-address 192.0.2.6
}
}
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:04
}
}
bgp 1 {
}
bgp-evpn {
evi 1000
mpls 1 {
admin-state enable
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel {
resolution any
}
}
}
split-horizon-group "CORE" {
}
}
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
spoke-sdp 46:1001 {
}
sap 1/2/1:1001 {
}
}
The configuration on PE-5 is similar:
# on PE-5:
configure {
service {
system {
bgp {
evpn {
ethernet-segment "ESI-45" {
admin-state enable
esi 01:00:00:00:00:45:00:00:00:01
multi-homing-mode single-active
df-election {
es-activation-timer 3
}
association {
sdp 56 {
}
}
pbb {
source-bmac-lsb 45-05
}
}
}
}
}
sdp 56 {
admin-state enable
delivery-type mpls
ldp true
far-end {
ip-address 192.0.2.6
}
}
vpls "B-VPLS 1000" {
admin-state enable
service-id 1000
customer "1"
service-mtu 2000
pbb-type b-vpls
pbb {
source-bmac {
address 00:00:00:00:00:05
}
}
bgp 1 {
}
bgp-evpn {
evi 1000
mpls 1 {
admin-state enable
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel {
resolution any
}
}
}
split-horizon-group "CORE" {
}
}
vpls "I-VPLS 1001" {
admin-state enable
service-id 1001
customer "1"
pbb-type i-vpls
pbb {
backbone-vpls "B-VPLS 1000" {
isid 1001
}
}
spoke-sdp 56:1001 {
}
sap 1/2/1:1001 {
}
}
With the preceding configuration, PE-4 and PE-5 will not advertise ES-BMAC addresses with MAX-ESI. Therefore, all the remote BMACs on PE-2 and PE-3 are associated with regular backbone EVPN-MPLS destinations. The CMAC addresses will be learned in the data plane associated with local SAP/SDP-bindings or remote BMAC addresses. An example for the I-VPLS and B-VPLS FDB in PE-2 follows:
[/]
A:admin@PE-2# show service id 1001 fdb detail pbb
==============================================================================
Forwarding Database, i-Vpls Service 1001
==============================================================================
MAC Source-Identifier B-Svc b-Vpls MAC Type/Age
Transport:Tnl-Id
------------------------------------------------------------------------------
00:00:10:10:10:10 sap:lag-1:1001 1000 N/A L/60
00:00:60:60:60:60 b-mpls: 1000 00:00:00:00:00:05 L/60
192.0.2.5:524279
==============================================================================
The B-VPLS FDB on PE-2 looks as follows:
[/]
A:admin@PE-2# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/05/21 10:05:43
192.0.2.3:524282
ldp:65537
1000 00:00:00:00:00:04 mpls: EvpnS:P 03/05/21 10:05:41
192.0.2.4:524282
ldp:65538
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/05/21 10:05:42
192.0.2.5:524279
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
In the preceding example, the DF for ISID 1001 is PE-5. With a failure event on the SDP to MTU-6, PE-5 will not withdraw the advertised source BMAC (because it is still being used as source BMAC for other services and even CEs within the same service). PE-5 will send an update of the same source BMAC instead, increasing the sequence number in the MAC mobility extended community. That will be a flush-all-from-me indication for the remote PEs (they will flush all the CMACs associated with the updated source BMAC, regardless of the service).
When the former DF (PE-5) comes back up, PE-4 will become non-DF and will send a CMAC flush indication using the same mechanism as described above.
The following example shows a failure of SDP 56 in PE-5 and the corresponding DF switchover and CMAC flush.
# on PE-5:
221 2021/03/05 10:34:56.487 CET MINOR: SVCMGR #2095 Base
"Ethernet Segment:ESI-45, ISID:1001, Designated Forwarding state changed to:false"
219 2021/03/05 10:34:56.486 CET MINOR: SVCMGR #2303 Base
"Status of SDP 56 changed to admin=up oper=down"
PE-5 sends a BGP update with the same source BMAC, increasing the sequence number in the MAC mobility extended community—CMAC flush:
# on PE-5:
97 2021/03/05 10:34:56.487 CET MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388464
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:2/Static
"
Individual SAP or spoke-SDP failures do not trigger any MAC flush or DF re-election. This is as per RFC 7623. In EVPN-MPLS, individual SAP/spoke-SDP failures are captured by the AD per-EVI withdrawal, which triggers a DF switchover.
PBB-EVPN single-active multi-homing for I-VPLS with ES-BMACs
As discussed throughout this chapter, the use of ES-BMACs for single-active multi-homing can minimize the number of CMACs flushed in a network. A simple change is necessary: configure the use-es-bmac-lsb true command and ensure that the generated ES-BMACs in PE-4 and PE-5 are different (the source-bmac-lsb in the previous configuration had different values for PE-4 and PE-5 already):
# on PE-4 and PE-5:
configure {
service {
vpls "B-VPLS 1000" {
pbb {
source-bmac {
use-es-bmac-lsb true
}
}
On PE-4, the source BMAC LSB in ESI-45 is configured with a value of 45-04:
[/]
A:admin@PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" | match BMAC
Source BMAC LSB : 45-04
On PE-5, the source BMAC LSB in ESI-45 is configured with a value of 45-05:
[/]
A:admin@PE-5# show service system bgp-evpn ethernet-segment name "ESI-45" | match BMAC
Source BMAC LSB : 45-05
The remote PEs (such as PE-2 in the following output) will receive additional BGP EVPN-MAC routes for the ES-BMACs. The following FDB on PE-2 shows the source BMAC addresses of PE-4 and PE-5 and the ES BMAC address of DF PE-5.
[/]
A:admin@PE-2# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/05/21 10:05:43
192.0.2.3:524282
ldp:65537
1000 00:00:00:00:00:04 mpls: EvpnS:P 03/05/21 10:05:41
192.0.2.4:524282
ldp:65538
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/05/21 10:05:42
192.0.2.5:524279
ldp:65539
1000 00:00:00:00:45:05 mpls: EvpnS:P 03/05/21 10:37:14
192.0.2.5:524279
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The benefit is that in case of a failure in ESI-45 (as before) the ES-BMAC is withdrawn and the remote PEs will only flush the CMACs associated with the remote ESI-45, as opposed to all the CMACs associated with PE-5.
PBB-EVPN single-active multi-homing for Epipes
In the network in PBB-EVPN multi-homing, Epipes can only support single-homing or all-active multi-homing but not single-active. For non-local-switching PBB-Epipes (there is a single SAP per Epipe), only all-active multi-homing is supported. Single-active multi-homing for local-switching enabled PBB-Epipes (two SAPs are defined within the PBB-Epipe instance) is only supported in the following scenarios.
Single-active multi-homing is supported for redundancy in a two-node, three or four SAP, scenario, as displayed in PBB-EVPN single-active support for Epipes. In these two cases, the Epipe PBB tunnel will be configured with the source BMAC of the remote PE node. When two SAPs are active in the same Epipe, local-switching is used to exchange frames between the CEs.
All-active multi-homing is not supported for redundancy in this scenario because the PE-1 PBB tunnel cannot point at a locally defined ES-BMAC.
PBB-EVPN multi-homing operation
See the EVPN for MPLS Tunnels chapter for the commands to operate Ethernet segments. Consider that there are no AD routes in PBB-EVPN. Also, the DF election algorithm will be based on the ISID values as opposed to EVIs.
Troubleshooting and debug commands
When troubleshooting PBB-EVPN networks, most of the troubleshooting commands discussed in EVPN for MPLS Tunnels can be used in the B-VPLS service and the base service>system>bgp-evpn instance. Some examples of useful commands are:
-
show redundancy bgp-evpn-multi-homing
-
show router bgp routes evpn (and filters)
-
show service evpn-mpls [<TEP ip-address>]
-
show service id bgp-evpn
-
show service id evpn-mpls (and modifiers)
-
show service id fdb pbb (and modifiers)
-
show service system bgp-evpn
-
show service system bgp-evpn ethernet-segment (and modifiers)
-
debug router bgp update (in classic CLI)
-
log-id "99"
In addition, the following tools dump commands also discussed in EVPN for MPLS Tunnels can help too:
-
tools dump service evpn usage
-
tools dump service system bgp-evpn ethernet-segment <name> isid <isid> df (Note: isid is used instead of evi.)
There are two aspects that are specific to PBB-EVPN and not EVPN:
-
Consumption of virtual BMAC addresses in the system— source BMACs, SAP BMACs, SDP BMACs, and ES BMACs are system BMACs that use FDB space but are not shown in the FDB together with the rest of the learned MAC addresses. The following command provides information about the virtual system MAC addresses consumed in the system.
[/] A:admin@PE-3# tools dump redundancy src-bmac-lsb Src-bmac-lsb: 3 (00-03) User: B-Vpls - 1 service(s) Src-bmac-lsb: 8995 (23-23) User: Evpn Mpls Total Src-bmac-lsbs = 2 -
Consumption of MFIBs — when ISIDs are not using the default-multicast list in the B-VPLS context for sending BUM traffic, an MFIB is consumed per ISID. The following command provides information about the consumption of MFIBs per system and per B-VPLS.
[/] A:admin@PE-3# tools dump service vpls-pbb-mfib-stats detail Service Manager VPLS PBB MFIB statistics at 03/05/2021 10:39:28: Usage per Service ServiceId MFIB User Count ------------+--------------+------- 1000 Evpn 1 ------------+--------------+------- Total 1 MMRP Current Usage : 0 System Limit : 8191 Full, 40959 ESOnly Per Service Limit : 2048 Full, 8192 ESOnly SPB Current Usage : 0 System Limit : 8191 Per Service Limit : 8191 Evpn Current Usage : 1 System Limit : 40959 Per Service Limit : 8191
Conclusion
In addition to a full RFC 7432 EVPN-MPLS implementation, SR OS supports PBB-EVPN as per RFC 7623 for large Layer 2 deployments, including single-active and all-active multi-homing. This example has shown how to configure and operate a PBB-EVPN network focusing on the specific aspects of PBB-EVPN compared to EVPN-MPLS.