EVPN-MPLS E-Tree

Frame Forwarding in a VPLS E-Tree without EVPN shows how frames are forwarded in an E-Tree. The ingress node PE-2 knows whether the frame comes from a leaf AC or a root AC and adds a tag indicating "from root" or "from leaf". Specific VLAN IDs are used to indicate "from root" or "from leaf". The egress node PE-4 forwards the frame based on the "from root" or "from leaf" tag, as follows:

• A frame with the "from root" tag can be forwarded to any AC, leaf or root.

• A frame with the "from leaf" tag can only be forwarded to a root AC, not to a leaf AC.

SAPs and SDP bindings are considered as root AC automatically (in the following example, SAP 1/2/c1/1:4 is a root AC); leaf ACs get the keyword leaf-ac, and NNI SAPs and SDP bindings get the keyword root-leaf-tag. The root tag equals the service delimiting VLAN ID (VID) in the SAP and the leaf tag can only be configured with a different value.

On PE-2:
configure {
    service {
        vpls "VPLS 4" {
            admin-state enable
            service-id 4
            customer "1"
            etree true
            sap 1/2/c1/1:4 { }
            sap 1/2/c3/1:4 {
                etree-leaf true
            }
            sap 1/2/c5/1:4 {
                etree-root-leaf-tag {
                    leaf 44
                }
            }
            spoke-sdp 24:4 {
                etree-root-leaf-tag true
                vc-type vlan
            }
            spoke-sdp 210:4 {
                etree-leaf true
            }
        }
    }
}

VLAN ranges are not allowed in a VPLS E-Tree, as shown for the following connection profile VLAN, which is configured on PE-2:

On PE-2:
configure {
    connection-profile {
        vlan 10 {
            qtag-range 10 {
                end 19
            }
            qtag-range 110 {
                end 110
            }
        }
    }
}

The following error is raised when attempting to configure a SAP with VLAN range cp-10:

configure {
    service {
        vpls "VPLS 4" {
            sap 1/2/c3/1:cp-10 { }
MINOR: MGMT_CORE #4001: configure service vpls "VPLS 4" sap 1/2/c3/1:cp-10
 - vlan-range not allowed with etree vpls

configure {
    service {
        vpls "VPLS 4" {
            sap 1/2/c3/1:cp-10.* {
                etree-leaf true
            }
MINOR: MGMT_CORE #4001: configure service vpls "VPLS 4" sap 1/2/c3/1:cp-10.*
 - SAP and port encapsulation values are incompatible
 - configure port 1/2/c3/1 ethernet encap-type
MINOR: MGMT_CORE #4001: configure service vpls "VPLS 4" sap 1/2/c3/1:cp-10.*
 - vlan-range not allowed with etree vpls
 - configure service vpls "VPLS 4" etree

All incoming frames on a SAP or SDP binding in a VPLS have their dot1q/qinq encapsulation removed by the local PE. In a VPLS E-Tree, the local PE then adds a VLAN tag with a dedicated VID indicating whether the frame originates from a root AC or a leaf AC.

• For dot1q/qinq-based L2 services, a VLAN tag with VID x is added for root and VID y for leaf. Frames with VID x are forwarded to any type of AC, while frames with VID y are only forwarded to root ACs at the remote node, as shown in VLAN Tags Added by Ingress Node and Filtered by Egress Node in VPLS E-Tree.

• For pseudowire-based L2 services, a VLAN tag with VID 1 is hard-coded for frames received on a root AC and a VLAN tag with VID 2 for frames received on a leaf AC.

No leaf tag needs to be added to frames forwarded to EVPN destinations. Instead, the BGP EVPN control plane for EVPN E-Tree advertises a leaf indication bit and a leaf label in the E-tree extended community, as shown in BGP EVPN Control Plane for EVPN E-Tree.

The BGP EVPN control plane is extended with the EVPN E-Tree extended community, as per RFC 8317. The low-order bit of the flags field contains the L-bit (L=1 indicates a leaf AC). The leaf label contains a 20-bit MPLS label that is non-zero for Ethernet Auto Discovery (AD) per Ethernet Segment (per-ES) routes (tag MAX-ET), but it equals zero for MAC/IP routes and Ethernet AD per EVPN Instance (per-EVI) routes (tag 0). The following BGP EVPN AD per-ES route contains an EVPN E-Tree extended community with L=0 and leaf label 524282, and is used for egress BUM filtering. RFC 8317 states that the leaf indication bit L must be ignored on reception and should be zero on transmission.

On PE-2:
1 2023/08/01 14:28:10.587 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 81
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.2
        Type: EVPN-AD Len: 25 RD: 192.0.2.2:1 ESI: ESI-0, tag: MAX-ET Label: 0 (Raw Label: 0x0) PathId:
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64496:1
        etree::L:0/Leaf-Label:524282
        bgp-tunnel-encap:MPLS
"

The following BGP EVPN MAC route contains an EVPN E-Tree extended community with L=1 and leaf label 0, and is used for known unicast ingress filtering:

On PE-2:
4 2023/08/01 14:28:14.229 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Send BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 89
    Flag: 0x90 Type: 14 Len: 44 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.2
        Type: EVPN-MAC Len: 33 RD: 192.0.2.2:1 ESI: ESI-0, tag: 0, mac len: 48 mac: ca:fe:09:29:29:29, IP len: 0, IP: NULL, label1: 8388496 (Raw Label: 0x7fff90) 
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64496:1
        etree::L:1/Leaf-Label:0
        bgp-tunnel-encap:MPLS
"

The following BGP EVPN AD per-EVI route contains an EVPN E-Tree extended community with L=1 and leaf label 0, and is used for ES consistency checking:

On PE-4:
80 2023/08/01 15:12:36.304 CEST MINOR: DEBUG #2001 Base Peer 1: 192.0.2.5
"Peer 1: 192.0.2.5: UPDATE
Peer 1: 192.0.2.5 - Received BGP UPDATE:
    Withdrawn Length = 0
    Total Path Attr Length = 81
    Flag: 0x90 Type: 14 Len: 36 Multiprotocol Reachable NLRI:
        Address Family EVPN
        NextHop len 4 NextHop 192.0.2.5
        Type: EVPN-AD Len: 25 RD: 192.0.2.5:2 ESI: 01:00:00:00:00:45:01:00:00:01, tag: 0 Label: 8388480 (Raw Label: 0x7fff80) PathId:
    Flag: 0x40 Type: 1 Len: 1 Origin: 0
    Flag: 0x40 Type: 2 Len: 0 AS Path:
    Flag: 0x40 Type: 5 Len: 4 Local Preference: 100
    Flag: 0xc0 Type: 16 Len: 24 Extended Community:
        target:64496:2
        etree::L:1/Leaf-Label:0
        bgp-tunnel-encap:MPLS
"

When PE-2 receives a BGP EVPN MAC route with an E-Tree extended community with leaf indication bit L=1, the PE imports the route and installs the MAC address in the forwarding database (FDB) with an EVPN leaf (Lf) flag, as follows:

[/]
A:admin@PE-2# show service id 1 fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age      
-------------------------------------------------------------------------------
1          ca:fe:01:01:01:01 sdp:210:1               L/0      08/01/23 14:33:40
1          ca:fe:06:46:46:46 mpls-1:                 Evpn     08/01/23 14:31:37
                             192.0.2.4:524281
           ldp:65538
1          ca:fe:07:47:47:47 mpls-1:                 Evpn,Lf  08/01/23 14:31:37
                             192.0.2.4:524281
           ldp:65538
1          ca:fe:08:28:28:28 sap:1/2/c1/1:1          LT/0     08/01/23 14:28:14
1          ca:fe:09:29:29:29 sap:1/2/c3/1:1          LT/0     08/01/23 14:28:14
-------------------------------------------------------------------------------
No. of MAC Entries: 5
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

If receiving the same MAC route as root from PE-1 and as leaf from PE-2, the MAC route from PE-1 is selected: root MAC routes have higher priority than leaf MAC routes. Root static MAC routes take precedence over leaf static MAC routes.

EVPN MAC routes with a higher sequence number have a higher priority than root or leaf MAC routes. MAC mobility procedures take precedence to first identify the location of the MAC before associating that MAC with a root or a leaf site. The EVPN MAC route selection criteria in tie-break order are as follows:

1. Conditional static MACs (local protected MACs)

2. Auto-learned protected MACs (locally learned MACs on SAPs or mesh/spoke SDPs because of the configuration of auto-learn-mac-protect)

3. EVPN ES PBR MACs

4. EVPN static MACs (remote protected MACs)

5. Data plane learned MACs (regular MAC learning on SAPs/SDP-bindings)

6. EVPN MACs with a higher sequence number

7. EVPN E-Tree root MACs

8. Lowest IP (next-hop IP of the EVPN NLRI)

9. Lowest Ethernet tag (Ethernet tag is zero for MPLS and non-zero for VXLAN)

10. Lowest RD

EVPN-MPLS E-Tree is the only E-Tree technology able to do unicast ingress filtering, as opposed to the usual unicast egress filtering that, for example, VPLS does. Remote MAC addresses are learned in the control plane, so EVPN can optimize the forwarding by filtering known unicast traffic at the ingress:

• Unicast frames entering a root AC at the ingress PE are not filtered. The MAC destination address (DA) is looked up in the FDB and the frames are forwarded. The MAC source address (SA) is learned and advertised in BGP EVPN without the E-Tree extended community.

• Unicast frames entering a leaf AC at the ingress PE are filtered. The MAC DA is looked up in the FDB. When the MAC DA is learned from an EVPN leaf (or a leaf AC), the frame is dropped. When the MAC DA is learned from an EVPN root (or root AC), the frame is forwarded. The MAC SA is learned and advertised in BGP EVPN with leaf indication bit L=1.

Ingress Leaf Filtering for Known Unicast Traffic shows that PE-4 advertises MAC2 with leaf indication bit L=1. When a frame is sent with MAC SA MAC1 on a leaf AC of PE-2, PE-2 does a MAC lookup in the FDB to find out that the DA MAC2 is learned from an EVPN leaf. Therefore, PE-2 does not forward the frame to PE-4, but drops it at the ingress.

The ingress filtering blocks E-Tree leaf-to-leaf traffic and requires the implementation of an extra leaf EVPN-MPLS destination per remote PE containing leaf ACs per E-Tree service. Therefore, a dedicated EVPN-MPLS binding is created per leaf unicast traffic in the service. This additional internal EVPN-MPLS destination is created per remote PE that contains a leaf and that advertises at least one leaf MAC. The MPLS E-Tree leaf destination is created when a MAC route with L=1 is received. Any EVPN E-Tree service could potentially use one additional EVPN-MPLS destination for leaf unicast traffic per remote PE. This additional EVPN-MPLS leaf destination in the E-Tree is only unicast and not part of the flooding list. The EVPN-MPLS leaf destination consumes EVPN resources, as can be verified as follows:

[/]
A:admin@PE-2# tools dump service evpn usage | match "Mpls Etree"
Mpls Etree Leaf Dests                           :             1

All MAC addresses received with L=1 point to this EVPN-MPLS E-Tree leaf destination, whereas root MAC addresses point to the root destination.

Egress Leaf Filtering for BUM Traffic shows that leaf-to-leaf BUM traffic is filtered at the egress, based on the EVPN leaf label advertised in the E-Tree extended community of the zero ESI AD per-ES route (tag=MAX-ET).

• BUM frames that enter a root AC at the ingress PE are not filtered; the BUM frames follow regular EVPN data plane procedures.

• BUM frames that enter a leaf AC at the ingress PE are marked as leaf and forwarded or replicated to the egress IOM. At the egress IOM, the frame is flooded in the default multicast list, subject to the following:

  • Leaf entries are skipped when BUM traffic is forwarded, so no BUM traffic is forwarded to local leaf ACs.

  • BUM traffic to remote BGP EVPN PEs is encapsulated with the EVPN label stack.

    • If the remote PE has advertised an AD per-ES route with E-Tree leaf label L1, this leaf label L1 is added at the bottom of the stack. At the egress PE, when the leaf label L1 matches the leaf label of the PE, the BUM traffic is only forwarded to the root ACs, not to the leaf ACs.

    • If the egress PE does not have any E-Tree enabled service, it has not advertised any AD per-ES route with E-Tree leaf label. The local PE forwards the BUM traffic with BGP EVPN encapsulation, but without an additional label. Even when the egress PE does not have E-Tree enabled, it can still work with the VPLS E-Tree service available in the ingress PE. No traffic is dropped at the egress PE where no E-Tree is configured.

The following command is used to monitor the ESI label entries consumed by the EVPN E-Tree application:

[/]
A:admin@PE-2# tools dump service evpn usage | match "BUM"
Evpn Etree Remote BUM Leaf Labels               :             1

Unicast traffic can be exchanged between CE-8 (root AC) and any other CE. However, unicast traffic from CE-9 on leaf AC can only be exchanged with CE-8 and CE-6 on root ACs, but not with CE-7 (via leaf AC SAP 1/2/c3/1:1) or CE-1 (via leaf AC spoke SDP 210:1), as follows:

[/]
A:admin@CE-9# ping 172.16.0.28 interval 0.1 count 20 output-format summary     # succeeds - leaf AC can send to root AC
PING 172.16.0.28 56 data bytes
!!!!!!!!!!!!!!!!!!!!
---- 172.16.0.28 PING Statistics ----
20 packets transmitted, 20 packets received, 0.00% packet loss
round-trip min = 2.45ms, avg = 2.66ms, max = 2.95ms, stddev = 0.101ms

[/]
A:admin@CE-9# ping 172.16.0.46 interval 0.1 count 20 output-format summary     # succeeds - leaf AC can send to root AC
PING 172.16.0.46 56 data bytes
!!!!!!!!!!!!!!!!!!!!
---- 172.16.0.46 PING Statistics ----
20 packets transmitted, 20 packets received, 0.00% packet loss
round-trip min = 3.48ms, avg = 3.76ms, max = 5.80ms, stddev = 0.477ms

[/]
A:admin@CE-9# ping 172.16.0.47 interval 0.1 count 20 output-format summary     # fails - leaf AC cannot send to leaf AC!
PING 172.16.0.47 56 data bytes
...   ...   ...   ...   . ....................
---- 172.16.0.47 PING Statistics ----
20 packets transmitted, 0 packets received, 100% packet loss

[/]
A:admin@CE-9# ping 172.16.0.110 interval 0.1 count 20 output-format summary    # fails - leaf AC cannot send to leaf AC!
PING 172.16.0.110 56 data bytes
...   ...   ...   ...   . ....................
---- 172.16.0.110 PING Statistics ----
20 packets transmitted, 0 packets received, 100% packet loss

The following FDB for VPLS 1 on PE-2 shows that MAC address ca:fe:07:47:47:47 of CE-7 is learned as EVPN leaf, whereas MAC address ca:fe:01:01:01:01 of CE-1 is learned on the local root spoke SDP.

[/]
A:admin@PE-2# show service id 1 fdb detail

===============================================================================
Forwarding Database, Service 1
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age      
-------------------------------------------------------------------------------
1          ca:fe:01:01:01:01 sdp:210:1               L/0      08/01/23 14:33:40
1          ca:fe:06:46:46:46 mpls-1:                 Evpn     08/01/23 14:31:37
                             192.0.2.4:524281
           ldp:65538
1          ca:fe:07:47:47:47 mpls-1:                 Evpn,Lf  08/01/23 14:31:37
                             192.0.2.4:524281
           ldp:65538
1          ca:fe:08:28:28:28 sap:1/2/c1/1:1          LT/0     08/01/23 14:28:14
1          ca:fe:09:29:29:29 sap:1/2/c3/1:1          LT/0     08/01/23 14:28:14
-------------------------------------------------------------------------------
No. of MAC Entries: 5
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

When multicast traffic is sent from a video source via CE-8 (root AC), both CE-6 and CE-7 receive this traffic; for multicast traffic sent via CE-9 (leaf AC), only CE-6 (root AC) receives this traffic. PE-2 received leaf label 524282 in an AD per-ES route from PE-4, as follows:

[/]
A:admin@PE-2# show router bgp routes evpn auto-disc rd 192.0.2.4:1 detail
===============================================================================
 BGP Router ID:192.0.2.2        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Original Attributes
 
Network        : n/a
Nexthop        : 192.0.2.4
Path Id        : None                   
From           : 192.0.2.4
Res. Nexthop   : 192.168.24.2
Local Pref.    : 100                    Interface Name : int-PE-2-PE-4
---snip---
Community      : target:64496:1 etree::L:0/Leaf-Label:524282
                 bgp-tunnel-encap:MPLS
---snip---
EVPN type      : AUTO-DISC              
ESI            : ESI-0
Tag            : MAX-ET                 
Route Dist.    : 192.0.2.4:1            
MPLS Label     : LABEL 0                
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

Multicast traffic is sent with three labels: MPLS (LDP), EVPN, and leaf label. The EVPN label is 524280 for multicast, as follows:

[/]
A:admin@PE-2# show service id 1 evpn-mpls

===============================================================================
BGP EVPN-MPLS Dest (Instance 1)
===============================================================================
TEP Address                     Transport:Tnl     Egr Label  Oper  Mcast  Num
                                                             State        MACs
-------------------------------------------------------------------------------
192.0.2.4                       ldp:65538         524280     Up    bum    0
192.0.2.4                       ldp:65538         524281     Up    none   2
-------------------------------------------------------------------------------
Number of entries: 2
-------------------------------------------------------------------------------
---snip---
===============================================================================

The MPLS transport label is 524287, as follows:

[/]
A:admin@PE-2# show router ldp bindings active prefixes prefix 192.0.2.4/32

===============================================================================
---snip---
===============================================================================
LDP IPv4 Prefix Bindings (Active)
===============================================================================
Prefix                                      Op
IngLbl                                      EgrLbl
EgrNextHop                                  EgrIf/LspId
-------------------------------------------------------------------------------
192.0.2.4/32                                Push
  --                                        524287
192.168.24.2                                1/1/c1/1
                                             
192.0.2.4/32                                Swap
524285                                      524287
192.168.24.2                                1/1/c1/1
                                             
-------------------------------------------------------------------------------
No. of IPv4 Prefix Active Bindings: 2
===============================================================================

The video source sends the following multicast stream via CE-9 (leaf AC):

[/]
A:admin@CE-9# show router pim group detail

===============================================================================
PIM Source Group ipv4
===============================================================================
Group Address      : 232.1.1.1
Source Address     : 192.168.55.2
---snip---
Rpf Neighbor       : 192.168.19.1
Incoming Intf      : int-CE-9-CE-1
Outgoing Intf List : int-CE-9-PE-2
 
Curr Fwding Rate   : 9751.560 kbps      
Forwarded Packets  : 29664              Discarded Packets  : 0
Forwarded Octets   : 43962048           RPF Mismatches     : 0
Spt threshold      : 0 kbps             ECMP opt threshold : 7
Admin bandwidth    : 1 kbps             
-------------------------------------------------------------------------------
Groups : 1
===============================================================================

Receiver H-6 has joined the multicast stream and CE-6 (root AC) receives the following multicast group:

[/]
A:admin@CE-6# show router pim group detail

===============================================================================
PIM Source Group ipv4
===============================================================================
Group Address      : 232.1.1.1
Source Address     : 192.168.55.2
---snip---
Rpf Neighbor       : 172.16.0.29
Incoming Intf      : int-CE-6-PE-4
Outgoing Intf List : int-CE-6-H-6
 
Curr Fwding Rate   : 9751.560 kbps      
Forwarded Packets  : 168421             Discarded Packets  : 0
Forwarded Octets   : 249599922          RPF Mismatches     : 0
Spt threshold      : 0 kbps             ECMP opt threshold : 7
Admin bandwidth    : 1 kbps             
-------------------------------------------------------------------------------
Groups : 1
===============================================================================

Receiver H-7 has also joined the multicast stream, but CE-7 (leaf AC) cannot receive BUM traffic from a leaf AC, so the forwarding rate is 0 kbps, as follows:

[/]
A:admin@CE-7# show router pim group detail

===============================================================================
PIM Source Group ipv4
===============================================================================
Group Address      : 232.1.1.1
Source Address     : 192.168.55.2
---snip---
Rpf Neighbor       : 
Incoming Intf      : 
Outgoing Intf List : int-CE-7-H-7
 
Curr Fwding Rate   : 0.000 kbps         
Forwarded Packets  : 0                  Discarded Packets  : 0
Forwarded Octets   : 0                  RPF Mismatches     : 0
Spt threshold      : 0 kbps             ECMP opt threshold : 7
Admin bandwidth    : 1 kbps             
-------------------------------------------------------------------------------
Groups : 1
===============================================================================

Egress filtering on MAC SA is required to cover cases when the ingress PE sends traffic received on a leaf AC, but without leaf indication. EVPN E-Tree Egress Filtering Based on MAC SA shows that CE-9 sends traffic with MAC SA ca:fe:09:29:29:29 on a leaf AC.

When CE-9 sends unicast traffic to CE-6 with root MAC DA ca:fe:06:46:46:46, the ingress PE-2 forwards the frames to this root MAC DA to egress PE-4. However, if PE-4 does not have the MAC DA in its FDB (because of aging or MAC flush and the MAC route has not made it yet to PE-2), it may flood the frame to all the root and leaf ACs, even if the frame originated from a leaf AC. EVPN E-Tree egress filtering based on MAC SA prevents this from happening, so the traffic is only forwarded to the root AC.

The data path does the egress filtering based on MAC SA as follows:

• First, frames are identified as leaf frames in one of the following cases:

  • Frames arriving on a leaf SAP

  • EVPN traffic arriving with a leaf label

  • Frames arriving with a MAC SA that is flagged as being a leaf SA

• At the egress PE, frames identified as leaf are filtered depending on the type of traffic:

  • For known unicast traffic, the FDB is consulted. If the MAC DA FDB entry is marked as being from a leaf, the frame is dropped to prevent leaf-to-leaf forwarding.

  • For BUM traffic, the leaf frames are filtered at the egress IOM to suppress leaf-to-leaf forwarding.

Traffic can be sent between CE-12 (root AC lag-1:2) and CE-62 (leaf AC spoke SDP 46:2), but traffic between CE-13 (leaf AC lag-2:2) and CE-63 (leaf AC spoke SDP 46:2) is filtered. The following FDB for VPLS 2 on PE-2 shows two EVPN leaf MAC addresses: ca:fe:06:00:20:62 for CE-62 and ca:fe:06:00:30:63 for CE-63.

[/]
A:admin@PE-2# show service id 2 fdb detail

===============================================================================
Forwarding Database, Service 2
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age      
-------------------------------------------------------------------------------
2          ca:fe:01:00:20:12 sap:lag-1:2             L/0      08/01/23 15:14:06
2          ca:fe:01:00:30:13 sap:lag-2:2             Evpn     08/01/23 15:14:09
2          ca:fe:06:00:20:62 eES:                    Evpn,Lf  08/01/23 15:12:37
                             01:00:00:00:00:45:01:00:00:01
2          ca:fe:06:00:30:63 eES:                    Evpn,Lf  08/01/23 15:14:31
                             01:00:00:00:00:45:01:00:00:01
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

The FDB for VPLS 2 on PE-3 shows the same EVPN leaf MAC addresses. For all PEs in the all-active MH ESs, the MAC addresses ca:fe:06:00:20:12 and ca:fe:06:00:30:13 from the locally attached ACs can be learned on the SAPs or via EVPN from the ES peer where they are learned on the SAPs. In this case, they are learned on the SAPs on PE-2 and PE-3.

[/]
A:admin@PE-3# show service id 2 fdb detail

===============================================================================
Forwarding Database, Service 2
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age      
-------------------------------------------------------------------------------
2          ca:fe:01:00:20:12 sap:lag-1:2             L/0      08/01/23 15:12:17
2          ca:fe:01:00:30:13 sap:lag-2:2             L/0      08/01/23 15:14:09
2          ca:fe:06:00:20:62 eES:                    Evpn,Lf  08/01/23 15:12:36
                             01:00:00:00:00:45:01:00:00:01
2          ca:fe:06:00:30:63 eES:                    Evpn,Lf  08/01/23 15:14:31
                             01:00:00:00:00:45:01:00:00:01
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

The following FDB for VPLS 2 on DF PE-4 shows one EVPN leaf MAC address: ca:fe:01:00:30:13 for CE-13 on a remote ES.

[/]
A:admin@PE-4# show service id 2 fdb detail

===============================================================================
Forwarding Database, Service 2
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age      
-------------------------------------------------------------------------------
2          ca:fe:01:00:20:12 eES:                    Evpn     08/01/23 15:12:25
                             01:00:00:00:00:23:01:00:00:01
2          ca:fe:01:00:30:13 eES:                    Evpn,Lf  08/01/23 15:14:09
                             01:00:00:00:00:23:02:00:00:01
2          ca:fe:06:00:20:62 sdp:46:2                L/0      08/01/23 15:12:36
2          ca:fe:06:00:30:63 sdp:46:2                L/0      08/01/23 15:14:31
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

PE-5 is NDF, and the following FDB shows three MAC routes of type EVPN leaf, for CE-13, CE-62, and CE-63.

[/]
A:admin@PE-5# show service id 2 fdb detail

===============================================================================
Forwarding Database, Service 2
===============================================================================
ServId     MAC               Source-Identifier       Type     Last Change
            Transport:Tnl-Id                         Age      
-------------------------------------------------------------------------------
2          ca:fe:01:00:20:12 eES:                    Evpn     08/01/23 15:12:38
                             01:00:00:00:00:23:01:00:00:01
2          ca:fe:01:00:30:13 eES:                    Evpn,Lf  08/01/23 15:14:09
                             01:00:00:00:00:23:02:00:00:01
2          ca:fe:06:00:20:62 eES:                    Evpn,Lf  08/01/23 15:12:38
                             01:00:00:00:00:45:01:00:00:01
2          ca:fe:06:00:30:63 eES:                    Evpn,Lf  08/01/23 15:14:31
                             01:00:00:00:00:45:01:00:00:01
-------------------------------------------------------------------------------
No. of MAC Entries: 4
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

Each PE advertises zero ESI AD per-ES routes (with tag MAX-ET) that are needed for egress BUM filtering.

BUM frames received on an ES root AC are flooded to the EVPN, based on regular EVPN procedures. The regular ESI label is sent for split horizon when frames are sent to the DF or NDF PEs in the same ES.

BUM frames received on an ES leaf AC are flooded in the default multicast list. The egress PE does not forward BUM traffic to any leaf ACs, including the ES leaf ACs. However, in the unlikely event that some ACs in a specific ES for an EVI have an inconsistent E-Tree configuration, these ACs are treated as root ACs, and the traffic is forwarded.

The remote PE-4 receives the following EVPN AD routes from DF PE-2: a zero ESI AD per-ES (tag MAX-ET), two AD per-EVI (tag 0) routes with a non-zero label, and two AD per-ES routes (tag MAX-ET).

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc next-hop 192.0.2.2
===============================================================================
 BGP Router ID:192.0.2.4        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  2.2.2.2:1           01:00:00:00:00:23:01:00:00:01 192.0.2.2
      MAX-ET                                            LABEL 0

u*>i  2.2.2.2:1           01:00:00:00:00:23:02:00:00:01 192.0.2.2
      MAX-ET                                            LABEL 0

u*>i  192.0.2.2:2         ESI-0                         192.0.2.2
      MAX-ET                                            LABEL 0

u*>i  192.0.2.2:2         01:00:00:00:00:23:01:00:00:01 192.0.2.2
      0                                                 LABEL 524277

u*>i  192.0.2.2:2         01:00:00:00:00:23:02:00:00:01 192.0.2.2
      0                                                 LABEL 524277

-------------------------------------------------------------------------------
Routes : 5
===============================================================================

The same remote PE-4 receives similar EVPN AD routes from NDF PE-3: a zero ESI AD per-ES (tag MAX-ET), two AD per-EVI (tag 0) routes with a non-zero label, and two AD per-ES routes (tag MAX-ET).

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc next-hop 192.0.2.3
===============================================================================
 BGP Router ID:192.0.2.4        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  3.3.3.3:1           01:00:00:00:00:23:01:00:00:01 192.0.2.3
      MAX-ET                                            LABEL 0

u*>i  3.3.3.3:1           01:00:00:00:00:23:02:00:00:01 192.0.2.3
      MAX-ET                                            LABEL 0

u*>i  192.0.2.3:2         ESI-0                         192.0.2.3
      MAX-ET                                            LABEL 0

u*>i  192.0.2.3:2         01:00:00:00:00:23:01:00:00:01 192.0.2.3
      0                                                 LABEL 524280

u*>i  192.0.2.3:2         01:00:00:00:00:23:02:00:00:01 192.0.2.3
      0                                                 LABEL 524280

-------------------------------------------------------------------------------
Routes : 5
===============================================================================

The following detailed information about the AD per-ES route (tag MAX-ET) for mass withdraw on PE-4 shows that no E-Tree extended community is sent by PE-2; only the ESI-label extended community is sent.

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc rd 2.2.2.2:1 tag MAX-ET esi 01:00:00:00:00:23:01:00:00:01 detail
===============================================================================
 BGP Router ID:192.0.2.4        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Original Attributes
 
Network        : n/a
Nexthop        : 192.0.2.2
Path Id        : None                   
From           : 192.0.2.2
Res. Nexthop   : 192.168.24.1
---snip---
Community      : target:64496:2 esi-label:524275/All-Active
---snip---
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:01:00:00:01
Tag            : MAX-ET                 
Route Dist.    : 2.2.2.2:1              
MPLS Label     : LABEL 0                
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

A similar result is seen for the other vES:

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc rd 2.2.2.2:1 tag MAX-ET esi 01:00:00:00:00:23:02:00:00:01 detail
===============================================================================
 BGP Router ID:192.0.2.4        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Original Attributes
 
Network        : n/a
Nexthop        : 192.0.2.2
Path Id        : None                   
From           : 192.0.2.2
Res. Nexthop   : 192.168.24.1
---snip---
Community      : target:64496:2 esi-label:524274/All-Active
---snip---
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:02:00:00:01
Tag            : MAX-ET                 
Route Dist.    : 2.2.2.2:1              
MPLS Label     : LABEL 0                
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The following detailed information about the AD per-EVI (tag 0) on PE-4 shows that if the ES is root (as for vES-23_1), the regular extended community is sent, not the E-Tree extended community.

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc rd 192.0.2.2:2 tag 0 esi 01:00:00:00:00:23:01:00:00:01 detail
===============================================================================
 BGP Router ID:192.0.2.4        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Original Attributes
 
Network        : n/a
Nexthop        : 192.0.2.2
Path Id        : None                   
From           : 192.0.2.2
Res. Nexthop   : 192.168.24.1
---snip---
Community      : target:64496:2 bgp-tunnel-encap:MPLS
---snip---
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:01:00:00:01
Tag            : 0                      
Route Dist.    : 192.0.2.2:2            
MPLS Label     : LABEL 524277           
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The following detailed information about the AD per-EVI (tag 0) on PE-4 shows that if the ES is leaf (as for vES-23_2), the E-Tree extended community is sent, along with the regular extended community.

[/]
A:admin@PE-4# show router bgp routes evpn auto-disc rd 192.0.2.2:2 tag 0 esi 01:00:00:00:00:23:02:00:00:01 detail
===============================================================================
 BGP Router ID:192.0.2.4        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Original Attributes
 
Network        : n/a
Nexthop        : 192.0.2.2
Path Id        : None                   
From           : 192.0.2.2
Res. Nexthop   : 192.168.24.1
---snip---
Community      : target:64496:2 etree::L:1/Leaf-Label:0
                 bgp-tunnel-encap:MPLS
---snip---
EVPN type      : AUTO-DISC              
ESI            : 01:00:00:00:00:23:02:00:00:01
Tag            : 0                      
Route Dist.    : 192.0.2.2:2            
MPLS Label     : LABEL 524277           
---snip---
-------------------------------------------------------------------------------
Routes : 1
===============================================================================

The tools dump service evpn usage command shows that there are three EVPN E-Tree remote BUM leaf labels:

[/]
A:admin@PE-2# tools dump service evpn usage | match "BUM"
Evpn Etree Remote BUM Leaf Labels               :             3

This corresponds to the following three ESI-0 AD per-ES routes (tag MAX-ET) on PE-2:

[/]
A:admin@PE-2# show router bgp routes evpn auto-disc esi ESI-0
===============================================================================
 BGP Router ID:192.0.2.2        AS:64496       Local AS:64496      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup, p - purge
 Origin codes  : i - IGP, e - EGP, ? - incomplete

===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------
u*>i  192.0.2.3:2         ESI-0                         192.0.2.3
      MAX-ET                                            LABEL 0

u*>i  192.0.2.4:2         ESI-0                         192.0.2.4
      MAX-ET                                            LABEL 0

u*>i  192.0.2.5:2         ESI-0                         192.0.2.5
      MAX-ET                                            LABEL 0

-------------------------------------------------------------------------------
Routes : 3
===============================================================================