EVPN for PBB over MPLS (PBB-EVPN)
This chapter provides information about EVPN for PBB over MPLS (PBB-EVPN).
Topics in this chapter include:
Applicability
This chapter was initially written for SR OS Release 13.0.R6. The CLI in the current edition is based on SR OS Release 21.2.R1.
Important note: A prerequisite is to read the EVPN for MPLS Tunnels chapter.
Overview
EVPN for Provider Backbone Bridging (PBB) over MPLS (hereafter called PBB-EVPN) is specified in RFC 7623, Provider Backbone Bridging Combined with Ethernet VPN (PBB-EVPN). It provides a simplified version of EVPN-MPLS for cases where the network requires very high scalability and does not need all the advanced features supported by EVPN-MPLS (but still requires single-active and all-active multi-homing capabilities). EVPN and PBB-EVPN SR OS feature comparison provides a comparison between the capabilities of EVPN and PBB-EVPN in SR OS, and may help to choose between them when designing a VPN service.
VPN requirements |
EVPN |
PBB-EVPN |
Comments |
|---|---|---|---|
All-active Multi-Homing (MH) (flow-based load-balancing) |
Yes |
Yes |
Allows better bandwidth utilization |
Single-active MH (service-based load-balancing) |
Yes |
Yes |
|
Ethernet Local Area Network (E-LAN) and point-to-point E-Line services |
Yes |
Yes |
|
Inter-subnet-forwarding |
Yes |
No |
Allows combined Layer 2 / Layer 3 services. EVPN |
Proxy-Address Resolution Protocol / Neighbor Discovery (Proxy-ARP/ND) and IP-duplication protection |
Yes |
No |
Allows Broadcast, Unknown unicast and Multicast (BUM) traffic reduction and better security |
Customer MAC (CMAC) protection |
Yes |
No |
Allows protecting key static CMACs |
Data Center integration |
Yes |
No |
Integration with VXLAN and Nuage Virtualized Services Directory (VSD) |
Control plane overhead |
Medium |
Low |
PBB-EVPN only advertises Backbone MACs (BMACs) and no route type 1s |
Confinement of CMAC learning |
No |
Yes |
CMACs are only learned on PEs with flows using those CMACs |
CMAC summarization |
No |
Yes |
Aggregation of CMACs into BMACs |
PBB-EVPN is a combination of 802.1ah PBB and RFC 7432, BGP MPLS-Based Ethernet VPN (EVPN-MPLS), and reuses the PBB-Virtual Private LAN Service (VPLS) service model, where Border Gateway Protocol BGP-EVPN is enabled in the backbone VPLS (B-VPLS) domain. EVPN is used as the control plane in the B-VPLS domain to control the distribution of BMACs and set up per-backbone service instance identifier (ISID) flooding trees for service instance VPLS (I-VPLS) services. The learning of the CMACs, either on local SAPs/SDP-bindings or associated with remote BMACs, is still performed in the data plane. Only the learning of BMACs in the B-VPLS is performed through BGP.
The SR OS PBB-EVPN implementation supports I-VPLS and PBB-Epipe services, including single-active and all-active multi-homing.
Because PBB-EVPN is based on the same control plane model as EVPN for MPLS, it is recommended to read the EVPN for MPLS Tunnels chapter before configuring PBB-EVPN. PBB-EVPN uses a subset of the BGP-EVPN routes described in EVPN for MPLS Tunnels as shown in EVPN route types.
When no EVPN multi-homing is used in the network, only the base routes are used. Route types 2 and 3 are considered the base and mandatory routes:
Route type 2 — (B) MAC route — In PBB-EVPN, this route type is used for the advertisement of BMACs that will be installed in the remote Forwarding Data Bases (FDBs). There are no IP addresses advertised in PBB-EVPN. The MAC mobility extended community is used for advertising system BMACs as protected (with the sticky bit set) and it is also used for CMAC flush in some single-homing scenarios that will be described later.
Route type 3 — Inclusive Multicast route — This route type is used for the advertisement of the I-VPLS ISIDs (no Epipes) and the desired multicast tree for each of them. The ISIDs are encoded in the Ethernet-tag field of the Network Layer Reachability Information (NLRI). When the B-VPLS is created and enabled, an Inclusive Multicast route with ISID = 0 is advertised. This is for the creation of the default multicast tree.
When EVPN multi-homing is used in an ISID, route type 4 (Ethernet Segment (ES) route) is used. In PBB-EVPN, there is no route type 1 advertised when multi-homing is used on the ISID services (I-VPLS and Epipes). Only route type 4 is used, and in the same way as it is for EVPN-MPLS. See the EVPN for MPLS Tunnels example for more information about ES routes, how they are formed, and how their RT/RD values are populated.
Configuration
This example describes the basic PBB-EVPN configuration first (without multi-homing) and how the flood containment is handled in PBB-EVPN. Flood containment refers to the efficient distribution of the BUM traffic generated for an ISID.
Networks are not always greenfield, so a smooth migration of PBB-EVPN from PBB-VPLS is required to minimize the effect on existing services. This example also describes this migration, starting from a common PBB-VPLS configuration.
Finally, this example describes the configuration of PBB-EVPN multi-homing.
The same setup described in the EVPN for MPLS Tunnels example is used:
Four PEs in the core (PE-2, PE-3, PE-4, and PE-5).
The PEs are interconnected in the same way as explained in EVPN for MPLS Tunnels with the same IP addressing, IS-IS, transport LDP, and BGP peering configuration. There is not any difference with the basic infrastructure. See the EVPN for MPLS Tunnels chapter if more information is required.
When configuring multi-homing, MTU-1 and MTU-6 are connected to the core.
PBB-EVPN configuration without multi-homing
PBB-EVPN network without multi-homing shows the example topology used in this chapter.
When configuring PBB-EVPN:
There is no difference at the access side (I-VPLS and Epipe configuration) compared to other PBB technologies supported in SR OS, such as Shortest Path Bridging for MAC (SPBM) or PBB-VPLS.
The B-VPLS becomes an EVPN-MPLS service, where bgp-evpn mpls is added.
The following output shows an example of a basic configuration in PE-3. B-VPLS 1000 is bgp-evpn enabled and I-VPLS 1001 and Epipe 1002 are linked to B-VPLS 1000.
# on PE-3:
configure
service
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:03
exit
bgp
exit
bgp-evpn
evi 1000
mpls bgp 1
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
no shutdown
exit
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
shutdown
exit
sap 1/2/1:1001 create
no shutdown
exit
no shutdown
exit
epipe 1002 name "Epipe 1002" customer 1 create
pbb
tunnel 1000 backbone-dest-mac 00:00:00:00:00:05 isid 1002
exit
sap 1/2/1:1002 create
no shutdown
exit
no shutdown
exit
In the preceding output, there is no new configuration needed for I-VPLS/Epipe services. As for the B-VPLS, the output shows the minimum configuration required. If needed, the following parameters can be modified under bgp-evpn:
*A:PE-2>config>service>vpls# bgp-evpn ?
- bgp-evpn
- no bgp-evpn
[no] accept-ivpls-e* - Configure to accept non-zero ethernet-tag MAC routes and
process for CMAC flushing
[no] cfm-mac-advert* - Enable/disable the advertisement of MEP, MIP, and VMEP MAC
addresses over the BGP EVPN
[no] evi - EVPN Identifier
[no] incl-mcast-ori* - Configure originating IP address
[no] ingress-repl-i* - Configure BGP EVPN IMET-IR route advertisement
[no] ip-route-adver* - Configure BGP EVPN IP Route Advertisement
isid-route-tar* + configure ISID route target information
[no] mac-advertisem* - Configure BGP EVPN MAC Advertisement
mac-duplication + Configure BGP EVPN MAC Duplication
[no] mpls + Configure BGP EVPN mpls
[no] sel-mcast-adve* - Enable/disable selective multicast advertisements
[no] unknown-mac-ro* - Configure BGP EVPN Unknown MAC Route
[no] vxlan + Configure BGP EVPN vxlan
*A:PE-2>config>service>vpls>bgp-evpn# mpls ?
- no mpls [bgp <bgp>]
- mpls [bgp <bgp>]
<bgp> : [1..2]
auto-bind-tunn* + Configure BGP EVPN mpls auto-bind-tunnel
[no] control-word - Enable/disable setting the CW bit in the label message
[no] default-route-* - Configure default-route-tag to match against export policies
ecmp - Configure maximum ECMP routes information
[no] entropy-label - Enable/disable use of entropy-label
[no] force-vlan-vc-* - Forces vlan-vc-type forwarding in the data-path
[no] ingress-replic* - Use the same label as the one advertised for unicast traffic
[no] oper-group - Configure oper-group
[no] restrict-prote* - Enable/disable protected src MAC restriction
route-next-hop - Configure route next-hop
[no] send-tunnel-en* - Configure encapsulation for this service
[no] shutdown - Administratively Enable/Disable BGP-EVPN mpls
[no] split-horizon-* - Configure a split-horizon-group
A detailed description of these commands is included in the EVPN for MPLS Tunnels chapter. In addition to the preceding commands, the following service (b-)vpls pbb commands are relevant for PBB-EVPN in the B-VPLS service:
force-qtag-forwarding allows the transparent transport of the customer 802.1p bits across the B-VPLS services.
source-bmac can modify the source BMAC for all the PBB packets containing traffic from non-multi-homed I-VPLS and Epipe services.
use-es-bmac instructs the system to use an ES-specific BMAC for traffic coming from an ES on an I-VPLS or Epipe.
use-sap-bmac instructs the system to use a SAP-specific BMAC for traffic coming from an MC-LAG I-VPLS/Epipe SAP.
Flood containment for I-VPLS services
In general, PBB technologies in SR OS support a way to contain flooding for a specified I-VPLS ISID, so that BUM traffic for that ISID only reaches the PEs where the ISID is locally defined. Each PE creates a Multicast Forwarding Information Base (MFIB) per I-VPLS ISID on the B-VPLS instance. That MFIB supports SAP/SDP-binding endpoints that can be populated by:
Multiple MAC Registration Protocol (MMRP) in regular PBB-VPLS
IS-IS in SPBM
In PBB-EVPN, B-VPLS EVPN destinations can be added to the MFIBs using EVPN Inclusive Multicast Ethernet tag routes when they include the ISID in the Ethernet-tag. By default, when a B-VPLS is successfully enabled (no shutdown), the PE advertises:
An Inclusive Multicast route for ISID = 0 — This allows the remote PEs to add the advertising PE to the default-multicast-list for the B-VPLS.
An Inclusive Multicast route for each local ISID defined in the system (a local ISID includes configured I-VPLS and static-ISIDs) — This allows the remote PEs to create MFIB entries in the B-VPLS for the received ISIDs.
Because EVPN destinations, B-SAPs, and B-spoke-SDPs can coexist in the same B-VPLS, be aware of the different flooding lists created and how they are used in a B-VPLS. PBB-EVPN — flooding lists illustrates this concept with an example for B-VPLS 1000 in PE-1. The assumptions are:
I-VPLS 1001 is created in PE-1, PE-2, and PE-4 only.
PE-1, PE-2, PE-3, PE-4, and PE-5 support BGP-EVPN in B-VPLS 1000.
PE-6 and PE-7 only support spoke-SDPs.
PE-1 is connected to all six PEs.
Figure 3. PBB-EVPN — flooding lists 
In this situation, PE-1 creates two flooding lists in B-VPLS 1000:
Default-multicast-list — composed of:
All the EVPN PEs that advertised ISID = 0 (PE-2, PE-3, PE-4, PE-5).
All the B-spoke-SDPs (or B-SAPs) (PE-6, PE-7).
All the EVPN PEs that advertised ISID 1001 and no ISID 0 (if an isid-policy is created in PE-1 stating use-def-mcast for ISID 1001). Note: third-party PEs may not advertise ISID = 0, but only non-zero ISIDs.
MFIB for ISID 1001 is composed of:
All the EVPN PEs that advertised ISID 1001 (PE-2 and PE-4) unless there is an ISID-policy in PE-1 stating use-def-mcast for ISID 1001.
Static-ISIDs defined in manual B-spoke-SDPs and B-SAPs (static-ISIDs cannot be created on BGP-AD auto-discovered B-spoke-SDPs).
Based on the above, when BUM traffic is sent to I-VPLS 1001 on PE-1:
The traffic is encapsulated in PBB with the group BMAC for ISID 1001 and sent (by default) to the MFIB created for ISID 1001 (PE-2 and PE-4).
If an ISID-policy is added with use-def-mcast for ISID 1001, the BUM traffic is encapsulated in PBB with the group BMAC for ISID 1001 and sent to the default-multicast-list, that is, all six remote PEs.
Referring to PBB-EVPN network without multi-homing, the following output illustrates the use of the ISID-policy in PBB-EVPN. PE-2 does not have any ISID-policy configured; when it receives BUM traffic from the local I-VPLS 1001, it uses the MFIB for ISID 1001:
# on PE-2:
configure
service
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:02
exit
bgp
exit
bgp-evpn
evi 1000
mpls bgp 1
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
no shutdown
*A:PE-2# show service id 1000 mfib
===============================================================================
Multicast FIB, Service 1000
===============================================================================
Source Address Group Address Port Id Svc Id Fwd
Blk
-------------------------------------------------------------------------------
* 01:1e:83:00:03:e9 b-mpls:192.0.2.3:524279 Local Fwd
b-mpls:192.0.2.4:524280 Local Fwd
b-mpls:192.0.2.5:524279 Local Fwd
-------------------------------------------------------------------------------
Number of entries: 1
===============================================================================
An ISID-policy can be added to modify this behavior and allow PE-2 to use the default multicast list. If I-VPLS 1001 exists in all the remote PEs (as in this example), using the default multicast list is as efficient as using the MFIB and saves expensive MFIB resources. In the following output, as soon as the ISID-policy is added, the MFIB entries for ISID 1001 are removed and PE-2 starts using the default multicast list.
# on PE-2:
configure
service
vpls "B-VPLS 1000"
isid-policy
entry 10 create
use-def-mcast
range 1001 to 2000
exit
exit
The MFIB on PE-2 does not contain any entries for ISID 1001 anymore, as follows:
*A:PE-2# show service id 1000 mfib
===============================================================================
Multicast FIB, Service 1000
===============================================================================
Source Address Group Address Port Id Svc Id Fwd
Blk
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Number of entries: 0
===============================================================================
PBB-VPLS to PBB-EVPN migration
The principles required for migrating a PBB-VPLS network to PBB-EVPN are explained in the VPLS to EVPN-MPLS Integration section of the EVPN for MPLS Tunnels chapter. Those principles are also applicable to EVPN destinations and spoke-SDPs in the B-VPLS and can be summarized in three points:
Systems with an EVPN destination and SDP-binding to the same far-end IP bring down the SDP-binding. This avoids loops when both constructs exist in the same network.
SDP-bindings and EVPN destinations can be placed in the same Split-Horizon Group (SHG). When traffic from an SDP-binding/EVPN destination belonging to that SHG is received on a PE, it is never forwarded to another SDP-binding or EVPN destination on the same SHG.
MAC addresses learned on an SDP-binding or SAP, that belong to an SHG where EVPN destinations are also created, are not advertised in BGP-EVPN.
Based on those principles, this section describes how to migrate a PBB-VPLS network to PBB-EVPN. The network in PBB-EVPN network without multi-homing represents a regular PBB-VPLS network that needs to be migrated to PBB-EVPN.
In that network, the four PEs are running BGP-AD and TLDP for the discovery and setup of the pseudowires in the B-VPLS instance. The advantage of this configuration is that the migration can be done node by node and with minimum impact on customer service.
Initial configuration
Initially, the network is configured for PBB-VPLS with BGP-AD in B-VPLS 1000. The EVPN family is to be added. At the access, I-VPLS 1001 is connected to the CEs. As an example, the configuration in PE-3 is shown. An equivalent configuration exists in the other three PEs.
The EVPN family is added to the BGP configuration because PBB-EVPN uses this address family. Assuming there are redundant Route Reflectors (RRs), the addition of EVPN can be done without service impact. In this example, the assumption is that the PEs are already configured with the EVPN family.
*A:PE-3#
configure
router Base
bgp
vpn-apply-import
vpn-apply-export
enable-peer-tracking
rapid-withdrawal
split-horizon
rapid-update evpn
group "internal"
family l2-vpn evpn
peer-as 64500
neighbor 192.0.2.2
exit
exit
# on PE-3:
configure
service
pw-template 1 name "PW1" create
split-horizon-group "CORE"
exit
exit
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:03
exit
bgp
pw-template-binding 1
exit
exit
bgp-ad
vpls-id 64500:1000
no shutdown
exit
stp
shutdown
exit
no shutdown
exit
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
shutdown
exit
sap 1/2/1:1001 create
exit
no shutdown
exit
*A:PE-3# show service id 1000 base
===============================================================================
Service Basic Information
===============================================================================
Service Id : 1000 Vpn Id : 0
Service Type : b-VPLS
---snip---
Oper Backbone Src : 00:00:00:00:00:03
Use SAP B-MAC : Disabled
i-Vpls Count : 1
Epipe Count : 1
Use ESI B-MAC : Disabled
-------------------------------------------------------------------------------
Service Access & Destination Points
-------------------------------------------------------------------------------
Identifier Type AdmMTU OprMTU Adm Opr
-------------------------------------------------------------------------------
sdp:32765:4294967293 SB(192.0.2.5) BgpAd 0 8978 Up Up
sdp:32766:4294967294 SB(192.0.2.4) BgpAd 0 8978 Up Up
sdp:32767:4294967295 SB(192.0.2.2) BgpAd 0 8978 Up Up
===============================================================================
* indicates that the corresponding row element may have been truncated.
Multiple MAC Registration Protocol (MMRP) is not used in the B-VPLS instance. If it were enabled, MMRP would have to be disabled in the network before this migration. If there are ISIDs using B-VPLS SDP-bindings to reach some remote locations and B-VPLS EVPN destinations to reach others, the default multicast list must be used in the current release (the MFIB cannot be used if there is a mix of both types). Therefore, during the migration process, the ISIDs must be added to the default multicast list.
Add service-level SHG (if not already there).
From the first node being migrated to PBB-EVPN to all nodes migrated, PBB-VPLS and PBB-EVPN have to coexist within the same meshed network. That is, EVPN-MPLS destinations and SDP-bindings need to be defined in the same split-horizon group. Therefore, if there is no split-horizon group defined in the B-VPLS, the first step is to add it. In this example, the split-horizon group is defined at the config>service>pw-template>level; therefore, it has to be added at the B-VPLS level.
Note:When the service>split-horizon-group is removed, an eval-pw-template must be performed.
Note:After adding the split-horizon-group at the service level, an eval-pw-template must be performed again so that the SDP-bindings take the new SHG configuration.
Note:During the time between the split-horizon-group being removed and added back again, the SDP-bindings can forward BUM traffic to each other, so this operation must be done carefully to avoid loops.
Assuming that the first node to be migrated is PE-3, the following output shows the procedure for adding the split-horizon-group at the service level.
# on PE-3: configure service pw-template 1 no split-horizon-group*A:PE-3# tools perform service id 1000 eval-pw-template 1 allow-service-impact eval-pw-template succeeded for Svc 1000 32765:4294967293 Policy 1 eval-pw-template succeeded for Svc 1000 32766:4294967294 Policy 1 eval-pw-template succeeded for Svc 1000 32767:4294967295 Policy 1# on PE-3: configure service vpls "B-VPLS 1000" split-horizon-group "CORE" create exit bgp pw-template-binding 1 split-horizon-group "CORE" exit exit*A:PE-3# tools perform service id 1000 eval-pw-template 1 allow-service-impact eval-pw-template succeeded for Svc 1000 32765:4294967293 Policy 1 eval-pw-template succeeded for Svc 1000 32766:4294967294 Policy 1 eval-pw-template succeeded for Svc 1000 32767:4294967295 Policy 1*A:PE-3>config>service>vpls# info ---------------------------------------------- service-mtu 2000 pbb source-bmac 00:00:00:00:00:03 exit split-horizon-group "CORE" create exit bgp pw-template-binding 1 split-horizon-group "CORE" exit exit bgp-ad vpls-id 64500:1000 no shutdown exit stp shutdown exit no shutdownAdd BGP-EVPN and ISID-policy configuration to the B-VPLS.
After the B-VPLS is configured with the split horizon group, the BGP-EVPN configuration (still in shutdown) and an ISID-policy can be added, as follows.
# on PE-2, PE-3, PE-4, PE-5: configure service vpls "B-VPLS 1000" bgp-evpn evi 1000 mpls bgp 1 shutdown split-horizon-group "CORE" auto-bind-tunnel resolution any exit exit exit isid-policy entry 10 create use-def-mcast range 1001 to 3000 exit exitEnable BGP-EVPN MPLS on the PE.
When the configuration is ready, the bgp-evpn mpls bgp 1 context can be enabled, as follows:
# on PE-3: configure service vpls "B-VPLS 1000" bgp-evpn mpls bgp 1 no shutdownEnabling the bgp-evpn mpls bgp 1 context triggers a route-refresh message for the EVPN family from PE-3, but no changes happen because PE-3 does not create any EVPN destinations until it imports EVPN routes from the other PEs. The three spoke-SDPs to the remote PEs are still up.
Repeat steps 1 to 3 for the second PE (PE-5).
The same steps 1 to 3 are repeated for PE-5. When the bgp-evpn mpls bgp 1 context is enabled, PE-5 sends a route-refresh and gets the BGP-EVPN routes from PE-3. As a result of that, PE-3 brings down the spoke-SDP to PE-5 and creates an EVPN destination to PE-5. The same process happens in PE-5. The following CLI output shows the received routes in PE-3 and spoke-SDP going down.
25 2021/03/03 11:00:17.445 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2 "Peer 1: 192.0.2.2: UPDATE Peer 1: 192.0.2.2 - Received BGP UPDATE: Withdrawn Length = 0 Total Path Attr Length = 110 Flag: 0x90 Type: 14 Len: 47 Multiprotocol Reachable NLRI: Address Family EVPN NextHop len 4 NextHop 192.0.2.5 Type: EVPN-INCL-MCAST Len: 17 RD: 64500:1000, tag: 1001, orig_addr len: 32, orig_addr: 192.0.2.5 Type: EVPN-INCL-MCAST Len: 17 RD: 64500:1000, tag: 0, orig_addr len: 32, orig_addr: 192.0.2.5 Flag: 0x40 Type: 1 Len: 1 Origin: 0 Flag: 0x40 Type: 2 Len: 0 AS Path: Flag: 0x40 Type: 5 Len: 4 Local Preference: 100 Flag: 0x80 Type: 9 Len: 4 Originator ID: 192.0.2.5 Flag: 0x80 Type: 10 Len: 4 Cluster ID: 1.1.1.1 Flag: 0xc0 Type: 16 Len: 16 Extended Community: target:64500:1000 bgp-tunnel-encap:MPLS Flag: 0xc0 Type: 22 Len: 9 PMSI: Tunnel-type Ingress Replication (6) Flags: (0x0)[Type: None BM: 0 U: 0 Leaf: not required] MPLS Label 8388480 Tunnel-Endpoint 192.0.2.5 "Log 99 shows that spoke SDP 32765:4294967293 is operationally down:
171 2021/03/03 11:00:17.766 UTC MINOR: SVCMGR #2313 Base "Status of SDP Bind 32765:4294967293 in service 1000 (customer 1) peer PW status bits changed to pwNotForwarding " 170 2021/03/03 11:00:16.687 UTC MINOR: SVCMGR #2306 Base "Status of SDP Bind 32765:4294967293 in service 1000 (customer 1) changed to admin=up oper=down flags=evpnRouteConflict " 169 2021/03/03 11:00:16.687 UTC MINOR: SVCMGR #2326 Base "Status of SDP Bind 32765:4294967293 in service 1000 (customer 1) local PW status bits changed to pwNotForwarding "Spoke SDP 32765:4294967293 is the spoke SDP toward PE-5 and it is kept down:
*A:PE-3# show service id 1000 base ---snip--- ------------------------------------------------------------------------------- Service Access & Destination Points ------------------------------------------------------------------------------- Identifier Type AdmMTU OprMTU Adm Opr ------------------------------------------------------------------------------- sdp:32765:4294967293 SB(192.0.2.5) BgpAd 0 8978 Up Down sdp:32766:4294967294 SB(192.0.2.4) BgpAd 0 8978 Up Up sdp:32767:4294967295 SB(192.0.2.2) BgpAd 0 8978 Up Up =============================================================================== * indicates that the corresponding row element may have been truncated.The reason why the spoke SDP toward PE-5 is down is an EVPN route conflict:
*A:PE-3# show service id 1000 sdp 32765:4294967293 detail | match Flag context all Flags : PWPeerFaultStatusBits EvpnRouteConflictAn EVPN destination to PE-5 is created:
*A:PE-3# show service id 1000 evpn-mpls =============================================================================== BGP EVPN-MPLS Dest =============================================================================== TEP Address Egr Label Num. MACs Mcast Last Change Transport:Tnl Sup BCast Domain ------------------------------------------------------------------------------- 192.0.2.5 524280 1 bum 03/03/2021 11:00:17 ldp:65539 No ------------------------------------------------------------------------------- Number of entries : 1 ------------------------------------------------------------------------------- =============================================================================== ---snip---Repeat Steps 1 to 3 for the rest of the PEs (PE-2, PE-4).
The same process is repeated in all the PEs, node by node. The service impact for the I-VPLS 1001 is minimal.
(Optional) Remove the ISID policy.
When all the PEs in the B-VPLS 1000 are migrated, the ISID policy can optionally be removed, node by node. This forces the B-VPLS instance to start using the MFIB to send I-VPLS BUM traffic to the remote nodes. This has no effect on Epipes (traffic is always unicast for Epipes).
Before removing the ISID policy and starting to use the MFIB, it is recommended to check that the Inclusive Multicast routes for an ISID to the remote PEs are all active. Otherwise, connectivity for BUM traffic could be interrupted if any of the expected routes are not active. This is illustrated for PE-3.
*A:PE-3# show service id 1000 evpn-mpls =============================================================================== BGP EVPN-MPLS Dest =============================================================================== TEP Address Egr Label Num. MACs Mcast Last Change Transport:Tnl Sup BCast Domain ------------------------------------------------------------------------------- 192.0.2.2 524281 1 bum 03/03/2021 11:02:31 ldp:65537 No 192.0.2.4 524280 1 bum 03/03/2021 11:02:32 ldp:65538 No 192.0.2.5 524280 1 bum 03/03/2021 11:00:17 ldp:65539 No ------------------------------------------------------------------------------- Number of entries : 3 ------------------------------------------------------------------------------- =============================================================================== ---snip---The routes for ISID 1001 are valid and used by BGP (flags u*>i):
*A:PE-3# show router bgp routes evpn incl-mcast tag 1001 =============================================================================== BGP Router ID:192.0.2.3 AS:64500 Local AS:64500 =============================================================================== Legend - Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid l - leaked, x - stale, > - best, b - backup, p - purge Origin codes : i - IGP, e - EGP, ? - incomplete =============================================================================== BGP EVPN Inclusive-Mcast Routes =============================================================================== Flag Route Dist. OrigAddr Tag NextHop ------------------------------------------------------------------------------- u*>i 64500:1000 192.0.2.2 1001 192.0.2.2 u*>i 64500:1000 192.0.2.4 1001 192.0.2.4 u*>i 64500:1000 192.0.2.5 1001 192.0.2.5 ------------------------------------------------------------------------------- Routes : 3 ===============================================================================There are no entries in the MFIB:
*A:PE-3# show service id 1000 mfib =============================================================================== Multicast FIB, Service 1000 =============================================================================== Source Address Group Address Port Id Svc Id Fwd Blk ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- Number of entries: 0 ===============================================================================The ISID policy is removed as follows:
# on PE-2, PE-3, PE-4, PE-5: configure service vpls "B-VPLS 1000" isid-policy no entry 10After removing the ISID-policy, the MFIB is populated with entries for the ISID 1001 group BMAC to the three remote PEs where ISID 1001 is defined:
*A:PE-3# show service id 1000 mfib =============================================================================== Multicast FIB, Service 1000 =============================================================================== Source Address Group Address Port Id Svc Id Fwd Blk ------------------------------------------------------------------------------- * 01:1e:83:00:03:e9 b-mpls:192.0.2.2:524281 Local Fwd b-mpls:192.0.2.4:524280 Local Fwd b-mpls:192.0.2.5:524280 Local Fwd ------------------------------------------------------------------------------- Number of entries: 1 ===============================================================================(Optional) Remove the BGP-AD configuration.
The BGP-AD configuration can stay in the B-VPLS services. However, when the entire network is migrated to PBB-EVPN, all the spoke-SDPs will be operationally down and, even if they are not forwarding traffic, they consume resources in the system. Consider removing the BGP-AD configuration and, therefore, the spoke-SDPs.
The following example shows the removal of BGP-AD in PE-4. Be aware that when BGP-AD is removed from the configuration, if the RD/RT was derived from the VPLS ID (as in this example), a new RD/RT must be auto-derived for the service. Therefore, new updates will be sent for all the EVPN NLRIs, as shown in the following output.
*A:PE-4# show service id 1000 bgp =============================================================================== BGP Information =============================================================================== Bgp Instance : 1 Vsi-Import : None Vsi-Export : None Route Dist : None Oper Route Dist : 64500:1000 Oper RD Type : derivedVpls Rte-Target Import : None Rte-Target Export: None Oper RT Imp Origin : derivedVpls Oper RT Import : 64500:1000 Oper RT Exp Origin : derivedVpls Oper RT Export : 64500:1000 PW-Template Id : 1 PW-Template SHG : CORE Oper Group : None Mon Oper Group : None BFD Template : None BFD-Enabled : no BFD-Encap : ipv4 Import Rte-Tgt : None ------------------------------------------------------------------------------- ===============================================================================BGP-AD is disabled as follows:
# on PE-4: configure service vpls "B-VPLS 1000" bgp-ad shutdownAfter BGP-AD is disabled, the spoke SDP bindings are deleted.
# log "99" on PE-4: 163 2021/03/03 11:05:42.890 UTC MAJOR: SVCMGR #2319 Base "Dynamic bgp-l2vpn SDP 32765 (192.0.2.5) was deleted." 162 2021/03/03 11:05:42.890 UTC MINOR: SVCMGR #2303 Base "Status of SDP 32765 changed to admin=down oper=down" 161 2021/03/03 11:05:42.890 UTC MAJOR: SVCMGR #2320 Base "Service Id 1000, Dynamic bgp-l2vpn SDP Bind Id 32765:4294967293 was deleted." 160 2021/03/03 11:05:42.880 UTC MINOR: SVCMGR #2306 Base "Status of SDP Bind 32765:4294967293 in service 1000 (customer 1) changed to admin=down oper=down flags="The PW template binding is removed as follows:
# on PE-4: configure service vpls "B-VPLS 1000" bgp no pw-template-binding 1The BGP-AD configuration is removed as follows:
# on PE-4: configure service vpls "B-VPLS 1000" no bgp-adInitially, the RD/RT was derived from the VPLS ID (64500:1000). After the BGP-AD configuration is removed, a new RD and RT must be auto-derived from the EVI:
*A:PE-4# show service id 1000 bgp =============================================================================== BGP Information =============================================================================== Bgp Instance : 1 Vsi-Import : None Vsi-Export : None Route Dist : None Oper Route Dist : 192.0.2.4:1000 Oper RD Type : derivedEvi Rte-Target Import : None Rte-Target Export: None Oper RT Imp Origin : derivedEvi Oper RT Import : 64500:1000 Oper RT Exp Origin : derivedEvi Oper RT Export : 64500:1000 PW-Template Id : None ------------------------------------------------------------------------------- ===============================================================================In this case, the system picks up the RD in the following order:
Manual RD or auto-RD always take precedence when configured.
If no manual/auto-RD, the RD is derived from the bgp-ad vpls-id.
If no manual/auto-rd/vpls-id configuration, the RD is derived from the bgp evpn evi.
If no manual/auto-rd/vpls-id/evi configuration, there will be no RD and the service will fail.
If in the migration from BGP-AD to BGP-EVPN, the advertisement of new updates is not needed, the initial configuration must include manual/auto-RDs. If manual/auto-RDs were not included, disabling BGP-AD would not cause the change of RD and the consequent BGP updates.
PBB-EVPN multi-homing
This section provides configuration guidelines for PBB-EVPN multi-homing. In the same way that EVPN-MPLS supports single-active and all-active multi-homing, PBB-EVPN can also be configured to support both modes. The same Ethernet segment that is used for regular EVPN-MPLS service SAPs and spoke-SDPs can be shared with I-VPLS/Epipe SAPs and spoke-SDPs.
PBB-EVPN multi-homing shows the example topology used in this section.
MTU-1 and MTU-6 have been added to the network (compared to PBB-EVPN network without multi-homing). I-VPLS 1001 has two new sites that are multi-homed to the PBB-EVPN network. MTU-1 uses all-active multi-homing, whereas MTU-6 is connected to a single-active ES. As with EVPN-MPLS, all-active multi-homing is only supported when a LAG is used at the access. Single-active multi-homing can be supported with regular Ethernet ports (that can form an independent LAG per PE) or SDPs.
RFC 7623 describes two types of system BMAC assignments that a PE can implement in a B-VPLS when ESs are present:
Shared BMAC addresses that can be used for all the single-homed CEs and a number of multi-homed CEs connected to Ethernet-segments.
Dedicated BMAC addresses per Ethernet-segment.
In this chapter and in SR OS terminology:
A shared BMAC address (in IETF) is a source BMAC address as configured in service>(b)vpls>pbb>source-bmac. All the I-VPLS/Epipe traffic coming from single-homed CEs is sent encapsulated in a PBB packet with that source BMAC address.
A dedicated-BMAC per ES (in IETF) is an ES BMAC address as activated in service>(b)vpls>pbb>use-es-bmac and generated from the combination of vpls>pbb>source-bmac plus ethernet-segment>source-bmac-lsb. If configured, any I-VPLS/Epipe traffic coming from an ES is encapsulated in a PBB packet with the ES-BMAC address as the source BMAC address.
The system allows the following user choices per B-VPLS and ES:
A dedicated ES BMAC address per ES can be used. In that case, the pbb use-es-bmac command is configured in the B-VPLS. In all-active multi-homing, all the PEs that are part of the ES source the PBB packets with the same source ES BMAC address; single-active multi-homing requires the use of a different ES BMAC address per PE.
A non-dedicated source BMAC address can be used (this is only possible in single-active multi-homing). In this case, the user does not configure pbb>use-es-bmac and the regular source BMAC address is used for the traffic. A different source BMAC address has to be advertised per PE.
As discussed, single-active multi-homing can use source BMAC addresses or ES BMAC addresses. Using one type or another has a different impact on CMAC flushing, as illustrated in The use of ES BMAC to minimize CMAC flush.
If ES BMAC addresses are used, as shown on the right-hand side of The use of ES BMAC to minimize CMAC flush, a less-impacting CMAC flush is achieved, therefore minimizing the flooding after ES failures. In the case of ES failure, PE-1 withdraws the ES BMAC address 00:12 and the remote PE-3 only flushes the CMACs associated with that ES BMAC address (only the CMAC addresses behind the CE are flushed).
If source BMAC addresses are used, as shown on the left-hand side of The use of ES BMAC to minimize CMAC flush, in the case of ES failure, a BGP update with higher sequence number is issued by PE-1 and the remote PE-3 flushes all the CMAC addresses associated with the source BMAC address. Therefore, all the CMAC addresses behind the B-VPLS of the PEs will be flushed, as opposed to only the CMAC addresses behind the CE of the Ethernet Service Instances (ESIs).
Figure 5. The use of ES BMAC to minimize CMAC flush 
PBB-EVPN multi-homing supported combinations in SR OS shows the PBB-EVPN multi-homing combinations supported in the current release in the topology of PBB-EVPN multi-homing.
CE Connectivity |
PE Connectivity |
PE Redundancy |
BMAC Assignment |
I-VPLS Support |
Epipe Support |
|---|---|---|---|---|---|
LAG (LACP optional) |
LAG SAP |
EVPN MH all-active |
use-es-bmac (shared BMAC) |
Yes |
Yes |
Ethernet ports (no LAG) |
LAG SAP or port SAP |
EVPN MH single-active |
use-es-bmac (dedicated per PE) |
Yes |
No |
Ethernet ports (no LAG) |
LAG SAP or port SAP |
EVPN MH single-active |
source-bmac (dedicated per PE) |
Yes |
No |
MPLS |
spoke-SDP |
EVPN MH single-active |
source-bmac (dedicated per PE) |
Yes |
No |
MPLS |
spoke-SDP |
EVPN MH single-active |
use-es-bmac (dedicated per PE) |
Yes |
No |
As an example, the configurations of the first, and last two, rows (LAG SAP all-active, MPLS source-BMAC, and MPLS ES-BMAC, respectively) will be discussed in the following three sections.
PBB-EVPN all-active multi-homing for I-VPLS and Epipes
PBB-EVPN multi-homing shows a PBB-EVPN network where ESI-23 is configured as an all-active multi-homing ES on PE-2 and PE-3. Two services are using ESI-23: I-VPLS 1001 and Epipe 1003. The following output shows the relevant configuration in PE-2:
# on PE-2:
configure
service
pbb
mac-name "PE-5" 00:00:00:00:00:05
exit
system
bgp-evpn
ethernet-segment "ESI-23" create
esi 01:00:00:00:00:23:00:00:00:01
source-bmac-lsb 23-23 es-bmac-table-size 8
es-activation-timer 3
service-carving
mode auto
exit
multi-homing all-active
lag 1
no shutdown
exit
exit
exit
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:02
use-es-bmac
exit
split-horizon-group "CORE" create
exit
bgp
exit
bgp-evpn
evi 1000
mpls bgp 1
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
no shutdown
exit
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
shutdown
exit
sap lag-1:1001 create
no shutdown
exit
no shutdown
exit
epipe 1003 name "Epipe 1003" customer 1 create
pbb
tunnel 1000 backbone-dest-mac "PE-5" isid 1003
exit
sap lag-1:1003 create
no shutdown
exit
no shutdown
exit
The following output shows the relevant configuration in PE-3:
# on PE-3:
configure
service
pbb
mac-name "PE-5" 00:00:00:00:00:05
exit
system
bgp-evpn
ethernet-segment "ESI-23" create
esi 01:00:00:00:00:23:00:00:00:01
source-bmac-lsb 23-23 es-bmac-table-size 8
es-activation-timer 3
service-carving
mode auto
exit
multi-homing all-active
lag 1
no shutdown
exit
exit
exit
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:03
use-es-bmac
exit
split-horizon-group "CORE" create
exit
bgp
exit
bgp-evpn
evi 1000
mpls bgp 1
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
no shutdown
exit
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
shutdown
exit
sap 1/2/1:1001 create
no shutdown
exit
sap lag-1:1001 create
no shutdown
exit
no shutdown
exit
epipe 1003 name "Epipe 1003" customer 1 create
pbb
tunnel 1000 backbone-dest-mac "PE-5" isid 1003
exit
sap lag-1:1003 create
no shutdown
exit
no shutdown
exit
The preceding configuration shows that Epipe 1003 has a PBB tunnel pointing at the PE-5 source-BMAC. Epipe 1003 has the following configuration in PE-5 (the PBB tunnel points at the ESI-23 ES-BMAC):
# on PE-5:
configure
service
pbb
mac-name "ES-MAC-23" 00:00:00:00:23:23
exit
epipe 1003 name "Epipe 1003" customer 1 create
pbb
tunnel 1000 backbone-dest-mac "ES-MAC-23" isid 1003
exit
sap 1/2/1:1003 create
no shutdown
exit
no shutdown
exit
Source-BMAC addresses and ES-BMAC addresses are distributed in BGP-EVPN. PE-2 and PE-3 will each advertise their own source-BMAC in a MAC route with ESI-0 and the shared ES-BMAC address with ESI-MAX (as per the RFC 7623). The ES-BMAC address that each PE uses in a B-VPLS is derived from the configured service>(b)vpls>pbb>source-bmac (four high-order bytes) and the ESI-23 configured source-bmac-lsb. In this example, PE-2 and PE-3 will both derive ES-BMAC 00:00:00:00:23:23. For both PEs to derive the required same ES-BMAC address, the four high-order bytes of the source-BMAC address must match on both PEs.
The es-bmac-table-size parameter modifies the default value (8) for the maximum number of ES-BMAC addresses that can be associated with the Ethernet segment across different B-VPLS services. When source-bmac-lsb is configured, the associated es-bmac-table-size is reserved out of the total FDB space.
The following outputs show the source-BMAC addresses and ES-BMAC address and how they are advertised and installed in the B-VPLS FDB.
*A:PE-2# show service system bgp-evpn ethernet-segment name "ESI-23" | match BMAC
Source BMAC LSB : 23-23
The following output shows that ES-BMAC is used and that the operational source-BMAC is 00:00:00:00:00:02.
*A:PE-2# show service id 1000 base
===============================================================================
Service Basic Information
===============================================================================
Service Id : 1000 Vpn Id : 0
Service Type : b-VPLS
---snip---
Oper Backbone Src : 00:00:00:00:00:02
Use SAP B-MAC : Disabled
i-Vpls Count : 1
Epipe Count : 1
Use ESI B-MAC : Enabled
---snip---
The source BMAC LSB is configured with the same value on PE-2 and PE-3. The two low-order bytes of the ES-BMAC will be 23:23.
*A:PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" | match BMAC
Source BMAC LSB : 23-23
On PE-3, ES-BMAC is used and the operational source BMAC is 00:00:00:00:00:03, as follows:
*A:PE-3# show service id 1000 base
===============================================================================
Service Basic Information
===============================================================================
Service Id : 1000 Vpn Id : 0
Service Type : b-VPLS
---snip---
Oper Backbone Src : 00:00:00:00:00:03
Use SAP B-MAC : Disabled
i-Vpls Count : 1
Epipe Count : 2
Use ESI B-MAC : Enabled
---snip---
On PE-2, the FDB for B-VPLS 1000 has an entry for each of the other PEs. PEs do not show their own system BMAC addresses in the FDB:
*A:PE-2# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/03/21 11:06:41
192.0.2.3:524277
ldp:65537
1000 00:00:00:00:00:04 mpls: EvpnS:P 03/03/21 11:06:37
192.0.2.4:524280
ldp:65538
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/03/21 11:06:39
192.0.2.5:524280
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
On PE-4, the FDB for B-VPLS 1000 has an entry for each of the other PEs and an entry for the ES-BMAC address of ES ‟ESI-23”:
*A:PE-4# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:02 mpls: EvpnS:P 03/03/21 11:06:42
192.0.2.2:524281
ldp:65537
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/03/21 11:06:41
192.0.2.3:524277
ldp:65538
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/03/21 11:06:39
192.0.2.5:524280
ldp:65539
1000 00:00:00:00:23:23 eES: EvpnS:P 03/03/21 11:07:33
MAX-ESI
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
On PE-4, there are two BGP routes for ES-BMAC address 00:00:00:00:23:23: one with next hop PE-2 and the other with next hop PE-3, as follows:
*A:PE-4# show router bgp routes evpn mac mac-address 00:00:00:00:23:23
===============================================================================
BGP Router ID:192.0.2.4 AS:64500 Local AS:64500
===============================================================================
Legend -
Status codes : u - used, s - suppressed, h - history, d - decayed, * - valid
l - leaked, x - stale, > - best, b - backup, p - purge
Origin codes : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN MAC Routes
===============================================================================
Flag Route Dist. MacAddr ESI
Tag Mac Mobility Label1
Ip Address
NextHop
-------------------------------------------------------------------------------
u*>i 192.0.2.2:1000 00:00:00:00:23:23 ESI-MAX
0 Static LABEL 524281
n/a
192.0.2.2
u*>i 192.0.2.3:1000 00:00:00:00:23:23 ESI-MAX
0 Static LABEL 524277
n/a
192.0.2.3
-------------------------------------------------------------------------------
Routes : 2
===============================================================================
PBB-EVPN all-active multi-homing is based on the same concepts as EVPN-MPLS all-active multi-homing: DF election, split-horizon, and aliasing.
Designated forwarder (DF) election
Only the DF PE for an ISID will send multicast traffic to the ES. The following command shows that PE-3 is the DF PE in ES ‟ESI-23” for ISID 1003:
*A:PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" isid 1003
===============================================================================
ISID DF and Candidate List
===============================================================================
Isid SvcId Actv Timer Rem DF DF Last Change
-------------------------------------------------------------------------------
1003 1003 0 yes 03/03/2021 11:07:45
===============================================================================
---snip---
The following command shows the DF and DF candidate list in the ES for all EVIs and all ISIDs:
*A:PE-3# show service system bgp-evpn ethernet-segment name "ESI-23" all
===============================================================================
Service Ethernet Segment
===============================================================================
Name : ESI-23
Eth Seg Type : None
Admin State : Enabled Oper State : Up
ESI : 01:00:00:00:00:23:00:00:00:01
Multi-homing : allActive Oper Multi-homing : allActive
---snip---
===============================================================================
ISID Information
===============================================================================
ISID SvcId Actv Timer Rem DF
-------------------------------------------------------------------------------
1001 1001 0 yes
1003 1003 0 yes
-------------------------------------------------------------------------------
Number of entries: 2
===============================================================================
-------------------------------------------------------------------------------
DF Candidate list
-------------------------------------------------------------------------------
ISID DF Address
-------------------------------------------------------------------------------
1001 192.0.2.2
1001 192.0.2.3
1003 192.0.2.2
1003 192.0.2.3
-------------------------------------------------------------------------------
Number of entries: 4
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
---snip---
The DF PE identifies multicast traffic by looking at either the destination BMAC or the EVPN label (which can be unicast or multicast).
In the case of Epipes, there are also DF and non-DF PEs. However, traffic is usually unicast (sent to the PBB tunnel backbone-destination BMAC). The non-DF PE will usually not discard Epipe traffic to the ES, unless the packet comes with an EVPN multicast label. To avoid packet duplication at the CE for Epipes, it is recommended to either:
configure discard-unknown on all the B-VPLS instances where there are PBB-Epipes. This will prevent the ingress PE from flooding Epipe traffic if the PBB tunnel BMAC is unknown in the FDB.
configure ingress-replication-bum-label so that, when the PBB tunnel BMAC is unknown in the FDB, the ingress PE sends traffic with a multicast label. The non-DF will discard traffic identified as multicast at Epipes.
Ethernet segment split-horizon
In PBB-EVPN all-active multi-homing, the split-horizon function is not based in the ESI label but in a source BMAC check. When BUM traffic is received on an I-VPLS, the PE will encapsulate it in PBB using the ES-BMAC as source BMAC and the group BMAC for the ISID. When the DF PE for the ISID receives that packet, it will not send it back to the ES if the packet is identified as being originated from the ES itself (based on the ES-BMAC shared between the PEs).
Aliasing
Aliasing is based on the advertisement of the same ES-BMAC with MAX-ESI from the PEs part of the same ES. PE-2 and PE-3 advertise the ES-BMAC 00:00:00:00:23:23 with MAX-ESI (ESI = all FFs, as per the RFC 7623) and as Static (protected). When the remote PEs, PE-4, and PE-5, receive the two routes for the same BMAC and MAX-ESI, they will create a single EVPN-MPLS destination that will give more than one next-hop (in this case 2), as long as ECMP > 1:
*A:PE-4# show service id 1000 evpn-mpls
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address Egr Label Num. MACs Mcast Last Change
Transport:Tnl Sup BCast Domain
-------------------------------------------------------------------------------
192.0.2.2 524281 1 bum 03/03/2021 11:06:42
ldp:65537 No
192.0.2.3 524277 1 bum 03/03/2021 11:06:41
ldp:65538 No
192.0.2.5 524280 1 bum 03/03/2021 11:06:39
ldp:65539 No
-------------------------------------------------------------------------------
Number of entries : 3
-------------------------------------------------------------------------------
===============================================================================
===============================================================================
BGP EVPN-MPLS Ethernet Segment Dest
===============================================================================
Eth SegId Num. Macs Last Change
-------------------------------------------------------------------------------
No Matching Entries
===============================================================================
===============================================================================
BGP EVPN-MPLS ES BMAC Dest
===============================================================================
ES BMAC Addr Last Change
-------------------------------------------------------------------------------
00:00:00:00:23:23 03/03/2021 11:07:49
-------------------------------------------------------------------------------
Number of entries: 1
-------------------------------------------------------------------------------
===============================================================================
The EVPN-MPLS ES BMAC destination has two next hops: PE-2 and PE-3.
*A:PE-4# show service id 1000 evpn-mpls es-bmac 00:00:00:00:23:23
===============================================================================
BGP EVPN-MPLS ES BMAC Dest
===============================================================================
ES BMAC Addr Last Change
-------------------------------------------------------------------------------
00:00:00:00:23:23 03/03/2021 11:07:49
===============================================================================
===============================================================================
BGP EVPN-MPLS ES BMAC Dest TEP Info
===============================================================================
TEP Address Egr Label Last Change Tunnel-
Transport Id
-------------------------------------------------------------------------------
192.0.2.2 524281 03/03/2021 11:07:33 65537
ldp
192.0.2.3 524277 03/03/2021 11:07:49 65538
ldp
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================
A similar output will be obtained in PE-5. Unicast traffic entering I-VPLS 1001 in either PE-4 or PE-5 will be hashed and load-balanced to PE-2 and PE-3 if the destination CMAC lookup yields an es-bmac-dest:
*A:PE-5# show service id 1001 fdb detail pbb
==============================================================================
Forwarding Database, i-Vpls Service 1001
==============================================================================
MAC Source-Identifier B-Svc b-Vpls MAC Type/Age
Transport:Tnl-Id
------------------------------------------------------------------------------
00:00:10:10:10:10 eES-BMAC: 1000 00:00:00:00:23:23 L/0
00:00:00:00:23:23
00:00:30:30:30:30 b-mpls: 1000 00:00:00:00:00:03 L/0
192.0.2.3:524277
00:00:50:50:50:50 sap:1/2/1:1001 1000 N/A L/0
00:00:60:60:60:60 sdp:56:1001 1000 N/A L/0
==============================================================================
Verify the FDB of I-VPLS 1001 for ES BMAC destination 00:00:00:00:23:23 as follows:
*A:PE-5# show service id 1001 fdb evpn-mpls es-bmac-dest 00:00:00:00:23:23
===============================================================================
Forwarding Database, Service 1001
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1001 00:00:10:10:10:10 eES-BMAC: L/30 03/03/21 11:13:15
00:00:00:00:23:23
-------------------------------------------------------------------------------
No. of Entries: 1
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
If a failure occurs in the ES, the PE will withdraw the ES-BMAC and the remote PEs will remove one next-hop of the ES-BMAC EVPN-MPLS destination.
For PBB-Epipes, aliasing will also work, as long as shared-queuing or policing are enabled on the ingress PE Epipe. In PBB-EVPN multi-homing, Epipe 1003 on PE-5 requires shared-queuing or policing at the ingress SAP. Otherwise, the traffic will be sent to only one PE of the ES (usually to the lower-IP PE).
For more information about the configuration of the Ethernet segment and its parameters, see the EVPN for MPLS Tunnels chapter.
PBB-EVPN single-active multi-homing for I-VPLS with source BMAC addresses
ESI-45 is a single-active Ethernet-segment (see PBB-EVPN multi-homing) with SDPs linked to it. As indicated in PBB-EVPN multi-homing supported combinations in SR OS , only I-VPLS services can be used in this configuration. As described in section PBB-EVPN multi-homing, single-active ES and B-VPLS services can be configured to either use source-BMAC addresses or ES-BMAC addresses. The following configuration shows the former option on PE-4:
# on PE-4:
configure
service
sdp 46 mpls create
far-end 192.0.2.6
ldp
keep-alive
shutdown
exit
no shutdown
exit
system
bgp-evpn
ethernet-segment "ESI-45" create
esi 01:00:00:00:00:45:00:00:00:01
source-bmac-lsb 45-04 es-bmac-table-size 8
es-activation-timer 3
service-carving
mode auto
exit
multi-homing single-active
sdp 46
no shutdown
exit
exit
exit
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:04
exit
split-horizon-group "CORE" create
exit
bgp
exit
bgp-evpn
evi 1000
mpls bgp 1
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
shutdown
exit
no shutdown
exit
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
shutdown
exit
spoke-sdp 46:1001 create
exit
no shutdown
exit
The configuration on PE-5 is similar:
# on PE-5:
configure
service
sdp 56 mpls create
far-end 192.0.2.6
ldp
keep-alive
shutdown
exit
no shutdown
exit
system
bgp-evpn
ethernet-segment "ESI-45" create
esi 01:00:00:00:00:45:00:00:00:01
source-bmac-lsb 45-05 es-bmac-table-size 8
es-activation-timer 3
service-carving
mode auto
exit
multi-homing single-active
sdp 56
no shutdown
exit
exit
exit
vpls 1000 name "B-VPLS 1000" customer 1 b-vpls create
service-mtu 2000
pbb
source-bmac 00:00:00:00:00:05
exit
split-horizon-group "CORE" create
exit
bgp
exit
bgp-evpn
evi 1000
mpls bgp 1
split-horizon-group "CORE"
ecmp 2
auto-bind-tunnel
resolution any
exit
no shutdown
exit
exit
stp
no shutdown
exit
no shutdown
exit
vpls 1001 name "I-VPLS 1001" customer 1 i-vpls create
pbb
backbone-vpls 1000
exit
exit
stp
no shutdown
exit
sap 1/2/1:1001 create
no shutdown
exit
spoke-sdp 56:1001 create
no shutdown
exit
no shutdown
exit
With the preceding configuration, PE-4 and PE-5 will not advertise ES-BMAC addresses with MAX-ESI. Therefore, all the remote BMACs on PE-2 and PE-3 are associated with regular backbone EVPN-MPLS destinations. The CMAC addresses will be learned in the data plane associated with local SAP/SDP-bindings or remote BMAC addresses. An example for the I-VPLS and B-VPLS FDB in PE-2 follows:
*A:PE-2# show service id 1001 fdb detail pbb
==============================================================================
Forwarding Database, i-Vpls Service 1001
==============================================================================
MAC Source-Identifier B-Svc b-Vpls MAC Type/Age
Transport:Tnl-Id
------------------------------------------------------------------------------
00:00:10:10:10:10 sap:lag-1:1001 1000 N/A L/90
00:00:60:60:60:60 b-mpls: 1000 00:00:00:00:00:05 L/90
192.0.2.5:524280
==============================================================================
The B-VPLS FDB on PE-2 looks as follows:
*A:PE-2# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/03/21 11:06:41
192.0.2.3:524277
ldp:65537
1000 00:00:00:00:00:04 mpls: EvpnS:P 03/03/21 11:06:37
192.0.2.4:524280
ldp:65538
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/03/21 11:06:39
192.0.2.5:524280
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
In the preceding example, the DF for ISID 1001 is PE-5. With a failure event on the SDP to MTU-6, PE-5 will not withdraw the advertised source BMAC (because it is still being used as source BMAC for other services and even CEs within the same service). PE-5 will send an update of the same source BMAC instead, increasing the sequence number in the MAC mobility extended community. That will be a flush-all-from-me indication for the remote PEs (they will flush all the CMACs associated with the updated source BMAC, regardless of the service).
When the former DF (PE-5) comes back up, PE-4 will become non-DF and will send a CMAC flush indication using the same mechanism as described above.
The following example shows a failure of SDP 56 in PE-5 and the corresponding DF switchover and CMAC flush.
# on PE-5:
186 2021/03/03 11:18:28.912 UTC MINOR: SVCMGR #2095 Base
"Ethernet Segment:ESI-45, ISID:1001, Designated Forwarding state changed to:false"
185 2021/03/03 11:18:28.912 UTC MINOR: SVCMGR #2303 Base
"Status of SDP 56 changed to admin=up oper=down"
PE-5 sends a BGP update with the same source BMAC, increasing the sequence number in the MAC mobility extended community—CMAC flush:
# on PE-5:
73 2021/03/03 11:18:28.912 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.2
"Peer 1: 192.0.2.2: UPDATE
Peer 1: 192.0.2.2 - Send BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 89
Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
Address Family EVPN
NextHop len 4 NextHop 192.0.2.5
Type: EVPN-MAC Len: 33 RD: 192.0.2.5:1000 ESI: ESI-0, tag: 0, mac len: 48
mac: 00:00:00:00:00:05, IP len: 0, IP: NULL, label1: 8388480
Flag: 0x40 Type: 1 Len: 1 Origin: 0
Flag: 0x40 Type: 2 Len: 0 AS Path:
Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
Flag: 0xc0 Type: 16 Len: 24 Extended Community:
target:64500:1000
bgp-tunnel-encap:MPLS
mac-mobility:Seq:2/Static
"
Individual SAP or spoke-SDP failures do not trigger any MAC flush or DF re-election. This is as per RFC 7623. In EVPN-MPLS, individual SAP/spoke-SDP failures are captured by the AD per-EVI withdrawal, which triggers a DF switchover.
PBB-EVPN single-active multi-homing for I-VPLS with ES-BMACs
As discussed throughout this chapter, the use of ES-BMACs for single-active multi-homing can minimize the number of CMACs flushed in a network. A simple change is necessary: activate the use-es-bmac command and ensure that the generated ES-BMACs in PE-4 and PE-5 are different (the source-bmac-lsb in the previous configuration had different values for PE-4 and PE-5 already):
# on PE-4, PE-5:
configure
service
vpls "B-VPLS 1000"
pbb
use-es-bmac
On PE-4, the source BMAC LSB in ESI-45 is configured with a value of 45-04:
*A:PE-4# show service system bgp-evpn ethernet-segment name "ESI-45" | match BMAC
Source BMAC LSB : 45-04
On PE-5, the source BMAC LSB in ESI-45 is configured with a value of 45-05:
*A:PE-5# show service system bgp-evpn ethernet-segment name "ESI-45" | match BMAC
Source BMAC LSB : 45-05
The remote PEs (such as PE-2 in the following output) will receive additional BGP EVPN-MAC routes for the ES-BMACs. The following FDB on PE-2 shows the source BMAC addresses of PE-4 and PE-5 and the ES BMAC address of DF PE-5.
*A:PE-2# show service id 1000 fdb detail
===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId MAC Source-Identifier Type Last Change
Transport:Tnl-Id Age
-------------------------------------------------------------------------------
1000 00:00:00:00:00:03 mpls: EvpnS:P 03/03/21 11:06:41
192.0.2.3:524277
ldp:65537
1000 00:00:00:00:00:04 mpls: EvpnS:P 03/03/21 11:06:37
192.0.2.4:524280
ldp:65538
1000 00:00:00:00:00:05 mpls: EvpnS:P 03/03/21 11:06:39
192.0.2.5:524280
ldp:65539
1000 00:00:00:00:45:05 mpls: EvpnS:P 03/03/21 11:19:55
192.0.2.5:524280
ldp:65539
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend: L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf
===============================================================================
The benefit is that in case of a failure in ESI-45 (as before) the ES-BMAC is withdrawn and the remote PEs will only flush the CMACs associated with the remote ESI-45, as opposed to all the CMACs associated with PE-5.
PBB-EVPN single-active multi-homing for Epipes
In the network in PBB-EVPN multi-homing, Epipes can only support single-homing or all-active multi-homing but not single-active. For non-local-switching PBB-Epipes (there is a single SAP per Epipe), only all-active multi-homing is supported. Single-active multi-homing for local-switching enabled PBB-Epipes (two SAPs are defined within the PBB-Epipe instance) is only supported in the following scenarios.
Single-active multi-homing is supported for redundancy in a two-node, three or four SAP, scenario, as displayed in PBB-EVPN single-active support for Epipes. In these two cases, the Epipe PBB tunnel will be configured with the source BMAC of the remote PE node. When two SAPs are active in the same Epipe, local-switching is used to exchange frames between the CEs.
All-active multi-homing is not supported for redundancy in this scenario because the PE-1 PBB tunnel cannot point at a locally defined ES-BMAC.
PBB-EVPN multi-homing operation
See the EVPN for MPLS Tunnels chapter for the commands to operate Ethernet-segments. Consider that there are no AD routes in PBB-EVPN. Also, the DF election algorithm will be based on the ISID values as opposed to EVIs.
Troubleshooting and debug commands
When troubleshooting PBB-EVPN networks, most of the troubleshooting commands discussed in EVPN for MPLS Tunnels can be used in the B-VPLS service and the base service>system>bgp-evpn instance. Some examples of useful commands are:
show redundancy bgp-evpn-multi-homing
show router bgp routes evpn (and filters)
show service evpn-mpls [<TEP ip-address>]
show service id bgp-evpn
show service id evpn-mpls (and modifiers)
show service id fdb pbb (and modifiers)
show service system bgp-evpn
show service system bgp-evpn ethernet-segment (and modifiers)
debug router bgp update
log-id 99
In addition, the following tools dump commands also discussed in EVPN for MPLS Tunnels can help too:
tools dump service evpn usage
tools dump service system bgp-evpn ethernet-segment <name> isid <isid> df (Note: isid is used instead of evi.)
There are two aspects that are specific to PBB-EVPN and not EVPN:
Consumption of virtual BMAC addresses in the system— source BMACs, SAP BMACs, SDP BMACs, and ES BMACs are system BMACs that use FDB space but are not shown in the FDB together with the rest of the learned MAC addresses. The following command provides information about the virtual system MAC addresses consumed in the system.
*A:PE-3# tools dump redundancy src-bmac-lsb Src-bmac-lsb: 3 (00-03) User: B-Vpls - 1 service(s) Src-bmac-lsb: 8995 (23-23) User: Evpn Mpls Total Src-bmac-lsbs = 2Consumption of MFIBs — when ISIDs are not using the default-multicast list in the B-VPLS context for sending BUM traffic, an MFIB is consumed per ISID. The following command provides information about the consumption of MFIBs per system and per B-VPLS.
*A:PE-3# tools dump service vpls-pbb-mfib-stats detail Service Manager VPLS PBB MFIB statistics at 03/03/2021 11:21:15: Usage per Service ServiceId MFIB User Count ------------+--------------+------- 1000 Evpn 1 ------------+--------------+------- Total 1 MMRP Current Usage : 0 System Limit : 8191 Full, 40959 ESOnly Per Service Limit : 2048 Full, 8192 ESOnly SPB Current Usage : 0 System Limit : 8191 Per Service Limit : 8191 Evpn Current Usage : 1 System Limit : 40959 Per Service Limit : 8191
Conclusion
In addition to a full RFC 7432 EVPN-MPLS implementation, SR OS supports PBB-EVPN as per RFC 7623 for large Layer 2 deployments, including single-active and all-active multi-homing. This example has shown how to configure and operate a PBB-EVPN network focusing on the specific aspects of PBB-EVPN compared to EVPN-MPLS.