Ethernet Virtual Private Networks

The DGW supports ECMP load balancing to reach the destination VTEP. Also, any intermediate core node in the DC should be able to provide further load balancing across ECMP paths, because the source UDP port of each tunneled packet is derived from a hash of the customer inner packet. The following must be considered:

• ECMP for VXLAN is supported on VPLS services but not for BUM traffic. Unicast spraying is based on the packet contents.

• ECMP for VXLAN on R-VPLS services is supported for VXLAN IPv6 tunnels.

• ECMP for VXLAN IPv4 tunnels on R-VPLS is only supported if the configure service vpls allow-ip-int-bind vxlan-ipv4-tep-ecmp command is enabled on the R-VPLS (as well as configure router ecmp).

• ECMP for Layer 3 multicast traffic on R-VPLS services with EVPN-VXLAN destinations is only supported if the vpls allow-ip-int-bind ip-multicast-ecmp command is enabled (as well as configure router ecmp).

• In the cases where ECMP is not supported (BUM traffic in VPLS and ECMP on R-VPLS if not enabled), each VXLAN binding is tied to a single (different) ECMP path, so that in a normal deployment with a reasonable number of remote VTEPs, there should be a fair distribution of the traffic across the paths. That is, only per-VTEP load-balancing is supported, instead of per-flow load-balancing.

• LAG spraying based on the packet hash is supported in all the cases (VPLS unicast, VPLS BUM, and R-VPLS).

The following describes the behavior with respect to VLAN tag handling for VXLAN VPLS services:

• Dot1q, QinQ, and null SAPs, as well as regular VLAN-handling procedures at the WAN side, are supported on VXLAN VPLS services.

• No ‟vc-type vlan” like VXLAN VNI bindings are supported. Therefore, at the egress of the VXLAN network port, the router does not add any inner VLAN tag on top of the VXLAN encapsulation, and at the ingress network port, the router ignores any VLAN tag received and handles it as part of the payload.

For VXLAN VPLS services, the network port MTU must be at least 50 Bytes (54 Bytes if dot1q) greater than the service MTU to allow enough room for the VXLAN encapsulation.

The service MTU is only enforced on SAPs (any SAP ingress packet with MTU greater than the service MTU is discarded) and not on VXLAN termination (any VXLAN ingress packet makes it to the egress SAP regardless of the configured service MTU).

If BGP-EVPN is enabled in a VXLAN VPLS service, the service MTU can be advertised in the Inclusive Multicast Ethernet Tag routes and enforce that all the routers attached to the same EVPN service have the same service MTU configured.

VXLAN is a network port encapsulation; therefore, the QoS settings for VXLAN are controlled from the network QoS policies.

A new VXLAN troubleshooting tool, VXLAN Ping, is available to verify VXLAN VTEP connectivity. The VXLAN Ping command is available from interactive CLI and SNMP.

This tool allows the user to specify a wide range of variables to influence how the packet is forwarded from the VTEP source to VTEP termination. The ping function requires the user to specify a different test-id (equates to originator handle) for each active and outstanding test. The required local service identifier from which the test is launched determines the source IP (the system IP address) to use in the outer IP header of the packet. This IP address is encoded into the VXLAN header Source IP TLV. The service identifier also encodes the local VNI. The outer-ip-destination must equal the VTEP termination point on the remote node, and the dest-vni must be a valid VNI within the associated service on the remote node. The outer source IP address is automatically detected and inserted in the IP header of the packet. The outer source IP address uses the IPv4 system address by default.

If the VTEP is created using a non-system source IP address through the vxlan-src-vtep command, the outer source IP address uses the address specified by vxlan-src-vtep. The remainder of the variables are optional.

The VXLAN PDU is encapsulated in the appropriate transport header and forwarded within the overlay to the appropriate VTEP termination. The VXLAN router alert (RA) bit is set to prevent forwarding OAM PDU beyond the terminating VTEP. Because handling of the router alert bit was not defined in some early releases of VXLAN implementations, the VNI Informational bit (I-bit) is set to ‟0” for OAM packets. This indicates that the VNI is invalid, and the packet should not be forwarded. This safeguard can be overridden by including the i-flag-on option that sets the bit to ‟1”, valid VNI. Ensure that OAM frames meant to be contained to the VTEP are not forwarded beyond its endpoints.

The supporting VXLAN OAM ping draft includes a requirement to encode a reserved IEEE MAC address as the inner destination value. However, at the time of implementation, that IEEE MAC address had not been assigned. The inner IEEE MAC address defaults to 00:00:00:00:00:00, but may be changed using the inner-l2 option. Inner IEEE MAC addresses that are included with OAM packets are not learned in the local Layer 2 forwarding databases.

The echo responder terminates the VXLAN OAM frame, and takes the appropriate response action, and include relevant return codes. By default, the response is sent back using the IP network as an IPv4 UDP response. The user can choose to override this default by changing the reply-mode to overlay. The overlay return mode forces the responder to use the VTEP connection representing the source IP and source VTEP. If a return overlay is not available, the echo response is dropped by the responder.

Support is included for:

• IPv4 VTEP

• Optional specification of the outer UDP Source, which helps downstream network elements along the path with ECMP to hash to flow to the same path

• Optional configuration of the inner IP information, which helps the user test different equal paths where ECMP is deployed on the source. A test only validates a single path where ECMP functions are deployed. The inner IP information is processed by a hash function, and there is no guarantee that changing the IP information between tests selects different paths.

• Optional end system validation for a single L2 IEEE MAC address per test. This function checks the remote FDB for the configured IEEE MAC Address. Only one end system IEEE MAC Address can be configured per test.

• Reply mode UDP (default) or Overlay

• Optional additional padding can be added to each packet. There is an option that indicates how the responder should handle the pad TLV. By default, the padding is not reflected to the source. The user can change this behavior by including the reflect-pad option. The reflect-pad option is not supported when the reply mode is set to UDP.

• Configurable send counts, intervals, times outs, and forwarding class

The VXLAN OAM PDU includes two timestamps. These timestamps are used to report forward direction delay. Unidirectional delay metrics require accurate time of day clock synchronization. Negative unidirectional delay values are reported as ‟0.000”. The round trip value includes the entire round trip time including the time that the remote peer takes to process that packet. These reported values may not be representative of network delay.

The following example commands and outputs show how the VXLAN Ping function can be used to validate connectivity. The echo output includes a new header to better describe the VXLAN ping packet headers and the various levels.

oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 
interval
0.1 send-count 10 

TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

! ! ! ! ! ! ! ! ! ! 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 1.097ms, avg = 2.195ms, max = 2.870ms, stddev = 0.735ms
round-trip-delay min = 1.468ms, avg = 1.693ms, max = 2.268ms, stddev = 0.210ms



oam vxlan-ping test-id 2 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 outer-
ip-source-udp 65000 outer-ip-ttl 64 inner-l2 d0:0d:1e:00:00:01 inner-ip-source
 192.168.1.2 inner-ip-destination 127.0.0.8 reply-mode overlay send-count 20 
interval
 1 timeout 3 padding 1000 reflect-pad fc nc profile out 

TestID 2, Service 1, DestVNI 2, ReplyMode overlay, IFlag Off, PadSize 1000, ReflectP
ad
 Yes, SendCount 20, Interval 1, Timeout 3
Outer: SourceIP 10.20.1.3, SourcePort 65000, DestIP 10.20.1.4, TTL 64, FC nc, Profil
e
 out
Inner: DestMAC d0:0d:1e:00:00:01, SourceIP 192.168.1.2, DestIP 127.0.0.8

===================================================================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
===================================================================================
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=1.733ms fwd
-time=0.302ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.549ms fwd
-time=1.386ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=3.243ms fwd
-time=0.643ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.551ms fwd
-time=2.350ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.644ms fwd
-time=1.080ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.670ms fwd
-time=1.307ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.636ms fwd
-time=0.490ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.649ms fwd
-time=0.005ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.401ms fwd
-time=0.685ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.634ms fwd
-time=0.373ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=11 ttl=255 rtt-time=1.559ms fwd
-time=0.679ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=12 ttl=255 rtt-time=1.666ms fwd
-time=0.880ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=13 ttl=255 rtt-time=1.629ms fwd
-time=0.669ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=14 ttl=255 rtt-time=1.280ms fwd
-time=1.029ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=15 ttl=255 rtt-time=1.458ms fwd
-time=0.268ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=16 ttl=255 rtt-time=1.659ms fwd
-time=0.786ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=17 ttl=255 rtt-time=1.636ms fwd
-time=1.071ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=18 ttl=255 rtt-time=1.568ms fwd
-time=2.129ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=19 ttl=255 rtt-time=1.657ms fwd
-time=1.326ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=20 ttl=255 rtt-time=1.762ms fwd
-time=1.335ms. rc=4

---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
20 packets transmitted, 20 packets received, 0.00% packet loss
   20 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 0.005ms, avg = 0.939ms, max = 2.350ms, stddev = 0.577ms
round-trip-delay min = 1.280ms, avg = 1.679ms, max = 3.243ms, stddev = 0.375ms



oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 interval 0.1 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

2 2 2 2 2 2 2 2 2 2 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present(1), 10 end-system not present(2)
forward-delay min = 0.467ms, avg = 0.979ms, max = 1.622ms, stddev = 0.504ms
round-trip-delay min = 1.501ms, avg = 1.597ms, max = 1.781ms, stddev = 0.088ms



oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

===================================================================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
mac=1 End System Present, mac=2 End System Not Present
===================================================================================

92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=2.883ms fwd
-time=4.196ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.596ms fwd
-time=1.536ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=1.698ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.687ms fwd
-time=1.766ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.679ms fwd
-time=0.799ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.678ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.709ms fwd
-time=0.031ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.757ms fwd
-time=1.441ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.613ms fwd
-time=2.570ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.631ms fwd
-time=2.130ms. rc=4 mac=2

---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present, 10 end-system not present
forward-delay min = 0.000ms, avg = 1.396ms, max = 4.196ms, stddev = 1.328ms
round-trip-delay min = 1.596ms, avg = 1.793ms, max = 2.883ms, stddev = 0.366ms

IPv4 and IPv6 multicast routing is supported in an EVPN-VXLAN VPRN and IES routed VPLS service through its IP interface when the source of the multicast stream is on one side of its IP interface and the receivers are on either side of the IP interface. For example, the source for multicast stream G1 could be on the IP side, sending to receivers on both other regular IP interfaces and the VPLS of the routed VPLS service, while the source for group G2 could be on the VPLS side sending to receivers on both the VPLS and IP side of the routed VPLS service. See IPv4 and IPv6 multicast routing support for more details.

The delivery of IP multicast in VXLAN services can be optimized with IGMP and MLD snooping. IGMP and MLD snooping are supported in EVPN-VXLAN VPLS services and in EVPN-VXLAN VPRN/IES R-VPLS services. When enabled, IGMP and MLD reports are snooped on SAPs or SDP bindings, but also on VXLAN bindings, to create or modify entries in the MFIB for the VPLS service.

When configuring IGMP and MLD snooping in EVPN-VXLAN VPLS services, consider the following:

• To enable IGMP snooping in the VPLS service on VXLAN, use the configure service vpls igmp-snooping no shutdown command.

• To enable MLD snooping in the VPLS service on VXLAN, use the configure service vpls mld-snooping no shutdown command.

• The VXLAN bindings only support basic IGMP/MLD snooping functionality. Features configurable under SAPs or SDP bindings are not available for VXLAN (VXLAN bindings are configured with the default values used for SAPs and SDP bindings). By default, a specified VXLAN binding only becomes a dynamic Mrouter when it receives IGMP or MLD queries and adds a specified multicast group to the MFIB when it receives an IGMP or MLD report for that group.

  Alternatively, it is possible to configure all VXLAN bindings for a particular VXLAN instance to be Mrouter ports using the configure service vpls vxlan igmp-snooping mrouter-port and configure service vpls vxlan mld-snooping mrouter-port commands.

• The show service id igmp-snooping, clear service id igmp-snooping, show service id mld-snooping, and clear service id mld-snooping commands are also available for VXLAN bindings.

  Note: MLD snooping uses MAC-based forwarding. See MAC-based IPv6 multicast forwarding for more details.

  The following CLI commands show how the system displays IGMP snooping information and statistics on VXLAN bindings (the equivalent MLD output is similar).

*A:PE1# show service id 1 igmp-snooping port-db vxlan vtep  192.0.2.72 vni 1 detail 
===============================================================================
IGMP Snooping VXLAN 192.0.2.72/1 Port-DB for service 1
===============================================================================
-------------------------------------------------------------------------------
IGMP Group 239.0.0.1
-------------------------------------------------------------------------------
Mode             : exclude              Type             : dynamic
Up Time          : 0d 19:07:05          Expires          : 137s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
No sources.
-------------------------------------------------------------------------------
IGMP Group 239.0.0.2
-------------------------------------------------------------------------------
Mode             : include              Type             : dynamic
Up Time          : 0d 19:06:39          Expires          : 0s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
10.0.0.232      0d 19:06:39  137s     dynamic  Fwd              
-------------------------------------------------------------------------------
Number of groups: 2
===============================================================================

*A:PE1# show service id 1 igmp-snooping 
statistics vxlan vtep  192.0.2.72 vni 1        
 
===============================================================================
IGMP Snooping Statistics for VXLAN 192.0.2.72/1 (service 1)
===============================================================================
Message Type            Received      Transmitted   Forwarded
-------------------------------------------------------------------------------
General Queries         0             0             556
Group Queries           0             0             0
Group-Source Queries    0             0             0
V1 Reports              0             0             0
V2 Reports              0             0             0
V3 Reports              553           0             0
V2 Leaves               0             0             0
Unknown Type            0             N/A           0
-------------------------------------------------------------------------------
Drop Statistics
-------------------------------------------------------------------------------
Bad Length               : 0
Bad IP Checksum          : 0
Bad IGMP Checksum        : 0
Bad Encoding             : 0
No Router Alert          : 0
Zero Source IP           : 0
Wrong Version            : 0
Lcl-Scope Packets        : 0
Rsvd-Scope Packets       : 0
 
Send Query Cfg Drops     : 0
Import Policy Drops      : 0
Exceeded Max Num Groups  : 0
Exceeded Max Num Sources : 0
Exceeded Max Num Grp Srcs: 0
MCAC Policy Drops        : 0
===============================================================================
*A:PE1# show service id 1 mfib 
===============================================================================
Multicast FIB, Service 1
===============================================================================
Source Address  Group Address         SAP or SDP Id               Svc Id   Fwd/Blk
-------------------------------------------------------------------------------
*               *                     sap:1/1/1:1              Local    Fwd
*               239.0.0.1             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
10.0.0.232      239.0.0.2             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
-------------------------------------------------------------------------------
Number of entries: 3
===============================================================================

PIM snooping for IPv4 and IPv6 are supported in an EVPN-EVPN-VXLAN VPLS or R-VPLS service (with the R-VPLS attached to a VPRN or IES service). The snooping operation is similar to that within a VPLS service (see PIM snooping for VPLS) and supports both PIM snooping and PIM proxy modes.

PIM snooping for IPv4 is enabled using the configure service vpls pim-snooping command.

PIM snooping for IPv6 is enabled using the configure service vpls pim-snooping no ipv6-multicast-disable command.

When using PIM snooping for IPv6, the default forwarding is MAC-based with optional support for SG-based (see IPv6 multicast forwarding). SG-based forwarding requires FP3- or higher-based hardware.

It is not possible to configure max-num-groups for VXLAN bindings.

By default, the system IP address is used to terminate and generate VXLAN traffic. The following configuration example shows an Epipe service that supports static VXLAN termination:

config service epipe 1 name "epipe1" customer 1 create
  sap 1/1/1:1 create
  exit
  vxlan vni 100 create
    egr-vtep 192.0.2.1
      oper-group op-grp-1
    exit
  no shutdown

Where:

• vxlan vni vni create specifies the ingress VNI the router uses to identify packets for the service. The following considerations apply:

  • In services that use EVPN, the configured VNI is only used as the ingress VNI to identify packets that belong to the service. Egress VNIs are learned from the BGP EVPN. In the case of Static VXLAN, the configured VNI is also used as egress VNI (because there is no BGP EVPN control plane).

  • The configured VNI is unique in the system, and as a result, it can only be configured in one service (VPLS or Epipe).

• egr-vtep ip-address specifies the remote VTEP the router uses when encapsulating frames into VXLAN packets. The following consideration apply:

  • When the PE receives VXLAN packets, the source VTEP is not checked against the configured egress VTEP.

  • The ip-address must be present in the global routing table so that the VXLAN destination is operationally up.

• The oper-group may be added under egr-vtep. The expected behavior for the operational group and service status is as follows:

  • If the egr-vtep entry is not present in the routing table, the VXLAN destination (in the show service id vxlan command) and the provisioned operational group under egr-vtep enters into the operationally down state.

  • If the Epipe SAP goes down, the service goes down, but it is not affected if the VXLAN destination goes down.

  • If the service is admin shutdown, then in addition to the SAP, the VXLAN destination and the oper-group also enters the operationally down state.

  Note: The operational group configured under egr-vtep cannot be monitored on the SAP of the Epipe where it is configured.

The following features are not supported by Epipe services with VXLAN destinations:

• per-service hashing

• SDP-binds

• PBB context

• BGP-VPWS

• spoke SDP-FEC

• PW-port

VXLAN instances in VPLS and R-VPLS can be configured with egress VTEPs. This is referred as static vxlan-instances. The following configuration example shows a VPLS service that supports a static vxlan-instance:

config service vpls 1 name "vpls-1" customer 1 create
  sap 1/1/1:1 create
  exit
  vxlan instance 1 vni 100 create
    source-vtep-security
    no disable-aging /* default: disable-aging    
    no disable-learning  /* default: disable-learning
    no discard-unknown-source
    no max-nbr-mac-addr <table-size>
    restrict-protected-src discard-frame
    egr-vtep 192.0.2.1 create
    exit
    egr-vtep 192.0.2.2 create
    exit
  vxlan instance 2 vni 101 create
    egr-vtep 192.0.2.3 create
    exit

 vxlan instance 2 vni 101 create
    egr-vtep 192.0.2.3 create
    exit
  no shutdown

Specifically the following can be stated:

• Each VPLS service can have up to two static VXLAN instances. Each instance is an implicit split-horizon-group, and up to 255 static VXLAN binds are supported in total, shared between the two VXLAN instances.

• Single VXLAN instance VPLS services with static VXLAN are supported along with SAPs and SDP bindings. Therefore:

  • VNIs configured in static VXLAN instances are ‟symmetric”, that is, the same ingress and egress VNIs are used for VXLAN packets using that instance. Note that asymmetric VNIs are actually possible in EVPN VXLAN instances.

  • The addresses can be IPv4 or IPv6 (but not a mix within the same service).

  • A specified VXLAN instance can be configured with static egress VTEPs, or be associated with BGP EVPN, but the same instance cannot be configured to support both static and BGP-EVPN based VXLAN bindings.

• Up to two VXLAN instances are supported per VPLS (up to two).

  • When two VXLAN instances are configured in the same VPLS service, any combination of static and BGP-EVPN enabled instances are supported. That is, the two VXLAN instances can be static, or BGP-EVPN enabled, or one of each type.

  • When a service is configured with EVPN and there is a static BGP-EVPN instance in the same service, the user must configure restrict-protected-src discard-frame along with no disable-learning in the static BGP-EVPN instance, service>vpls>vxlan.

• MAC addresses are learned also on the VXLAN bindings of the static VXLAN instance. Therefore, they are shown in the FDB commands. Note that disable-learning and disable-aging are by default enabled in static vxlan-instance.

  • The learned MAC addresses are subject to the remote-age, and not the local-age (only MACs learned on SAPs use the local-age setting).

  • MAC addresses are learned on a VTEP as long as no disable-learning is configured, and the VXLAN VTEP is present in the base route table. When the VTEP disappears from the route table, the associated MACs are flushed.

• The vpls vxlan source-vtep-security command can be configured per VXLAN instance on VPLS services. When enabled, the router performs an IPv4 source-vtep lookup to discover if the VXLAN packet comes from a trusted VTEP. If not, the router discards the frame. If the lookup yields a trusted source VTEP, then the frame is accepted.

  • A trusted VTEP is an egress VTEP that has been statically configured, or dynamically learned (through EVPN) in any service, Epipe or VPLS

  • The command show service vxlan shows the list of trusted VTEPs in the router.

  • The command source-vtep-security works for static VXLAN instances or BGP-EVPN enabled VXLAN instances, but only for IPv4 VTEPs.

  • The command is mutually exclusive with assisted-replication (replicator or leaf) in the VNI instance. AR can still be configured in a different instance.

Static VXLAN instances can use non-system IPv4/IPv6 termination.

The EVPN-VXLAN service is designed around current VPLS objects and the additional VXLAN construct.

Layer 2 DC PE with VPLS to the WAN shows a DC with a Layer 2 service that carries the traffic for a tenant who wants to extend a subnet beyond the DC. The DC PE function is carried out by the 7705 SAR Gen 2 where a VPLS instance exists for that particular tenant. Within the DC, the tenant has VPLS instances in all the Network Virtualization Edge (NVE) devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The VPLS instances in the redundant DGW and the DC NVEs are connected by VXLAN bindings. BGP-EVPN provides the required control plane for such VXLAN connectivity.

The DGW routers are configured with a VPLS per tenant that provides the VXLAN connectivity to the Nuage VPLS instances. On the router, each tenant VPLS instance is configured with:

• WAN-related parameters (SAPs, spoke SDPs, mesh-SDPs, BGP-AD, and so on).

• BGP-EVPN and VXLAN (VNI) parameters. The following CLI output is an example of an EVPN-VXLAN VPLS service.

*A:DGW1>config>service>vpls# info 
----------------------------------------------
            description "vxlan-service"
            vxlan instance 1 vni 1 create
            exit
            bgp
                route-distinguisher 65001:1
                route-target export target:65000:1 import target:65000:1
            exit
            bgp-evpn
                unknown-mac-route
                mac-advertisement 
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            sap 1/1/1:1 create
            exit
            no shutdown
----------------------------------------------

Gateway IRB on the DC PE for an L2 EVPN/VXLAN DC shows a DC with a Layer 2 service that carries the traffic for a tenant who extends a subnet within the DC, while the DGW is the default gateway for all the hosts in the subnet. The DGW function is carried out by the 7705 SAR Gen 2 where an R-VPLS instance exists for that particular tenant. Within the DC, the tenant has VPLS instances in all the NVE devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The WAN connectivity is based on existing IP-VPN features.

In this model, the DGW routers are configured with a R-VPLS (bound to the VPRN that provides the WAN connectivity) per tenant that provides the VXLAN connectivity to the Nuage VPLS instances. This model provides inter-subnet forwarding for L2-only TORs and other L2 DC NVEs.

On the router:

• The VPRN is configured with an interface bound to the backhaul R-VPLS. That interface is a regular IP interface (IP address configured or possibly a Link Local Address if IPv6 is added).

• The VPRN can support other numbered interfaces to the WAN or even to the DC.

• The R-VPLS is configured with the BGP, BGP-EVPN and VXLAN (VNI) parameters.

The Nuage VSGs and NVEs use a regular VPLS service model with BGP EVPN and VXLAN parameters.

Consider the following:

• Route-type 2 routes with MACs and IPs are advertised. Some considerations about MAC+IP and ARP/ND entries are:

  • The node advertises its IRB MAC+IP in a route type 2 route and possibly the VRRP vMAC+vIP if it runs VRRP and the node is the active router. In both cases, the MACs are advertised as static MACs, therefore, protected by the receiving PEs.

  • If the VPRN interface is configured with one or more additional secondary IP addresses, they are all advertised in routes type 2, as static MACs.

  • The node processes route-type 2 routes as usual, populating the FDB with the received MACs and the VPRN ARP/ND table with the MAC and IPs, respectively.

    Note: ND entries received from the EVPN are installed as Router entries. The ARP/ND entries coming from the EVPN are tagged as evpn.

  • When a VPLS containing proxy-ARP/proxy-ND entries is bound to a VPRN (allow-ip-int-bind) all the proxy-ARP/proxy-ND entries are moved to the VPRN ARP/ND table. ARP/ND entries are also moved to proxy-ARP/proxy-ND entries if the VPLS is unbound.

  • EVPN does not program EVPN-received ARP/ND entries if the receiving VPRN has no IP addresses for the same subnet. The entries are added when the IP address for the same subnet is added.

  • Static ARP/ND entries have precedence over dynamic and EVPN ARP/ND entries.

• VPRN interface binding to VPLS service brings down the VPRN interface operational status, if the VPRN interface MAC or the VRRP MAC matches a static-mac or OAM MAC configured in the associated VPLS service. If that is the case, a trap is generated.

• Redundancy is handled by VRRP. The active node advertises vMAC and vIP, as discussed, including the MAC mobility extended community and the sticky bit.

EVPN-enabled R-VPLS services are also supported on IES interfaces.

An AR-R configuration is shown in the following example.

*A:PE-2>config>service>system>vxlan# info 
----------------------------------------------
            assisted-replication-ip 10.2.2.2
----------------------------------------------
*A:PE-2>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication replicator
            exit
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
<snip>
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with tunnel-type = AR, type (T) = AR-R, and tunnel-id = originating-ip = next-hop = assisted-replication-ip (IP address 10.2.2.2 in the preceding example). In addition to the AR route, the AR-R sends a regular IR route if ingress-repl-inc-mcast-advertisement is enabled.

The AR-R builds a flooding list composed of ACs (SAPs and SDP bindings) and VXLAN tunnels to remote nodes in the VPLS. All objects in the flooding list are broadcast/multicast (BM) and unknown unicast (U) capable. The following example output of the show service id vxlan command shows that the VXLAN destinations in the flooding list are tagged as ‟BUM”.

*A:PE-2# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: replicator
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
192.0.2.3                               4000        0           BUM   Up    No
192.0.2.5                               4000        0           BUM   Up    No
192.0.2.6                               4000        0           BUM   Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================

When the AR-R receives a BUM packet on an AC, the AR-R forwards the packet to its flooding list (including the local ACs and remote VTEPs).

When the AR-R receives a BM packet on a VXLAN tunnel, it checks the IP DA of the underlay IP header and performs the following BM packet processing.

• If the destination IP matches its AR-IP, the AR-R forwards the BM packet to its flooding list (ACs and VXLAN tunnels). The AR-R performs source suppression to ensure that the traffic is not sent back to the originating Leaf.

• If the destination IP matches its regular VXLAN termination IP (IR-IP), the AR-R skips all the VXLAN tunnels from the flooding list and only replicates to the local ACs. This is the default Ingress Replication (IR) behavior.

An AR-L is configured as shown in the following example.

A:PE-3>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication leaf replicator-activation-time 30
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
                mpls
                    shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:4000 create
                no shutdown
            exit
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with a tunnel-type = IR, type (T) = AR-L and tunnel-id = originating-ip = next-hop = IR-IP (IP address terminating VXLAN normally, either system-ip or vxlan-src-vtep address).

The AR-L builds a single flooding list per service but controlled by the BM and U flags. These flags are displayed in the following show service id vxlan command example output.

A:PE-3# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: leaf      Replicator-Activation-Time: 30
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
10.2.2.2                                 4000        0           BM    Up    No
10.4.4.4                                 4000        0           -     Up    No
192.0.2.2                               4000        0           U     Up    No
192.0.2.5                               4000        0           U     Up    No
192.0.2.6                               4000        0           U     Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 5
-------------------------------------------------------------------------------
===============================================================================

The AR-L creates the following VXLAN destinations when it receives and selects a Replicator-AR route or the Regular-IR routes:

• A VXLAN destination to each remote PE that sent an IR route. These bindings have the U flag set.

• A VXLAN destination to the selected AR-R. These bindings have only the BM flag set; the U flag is not set.

• The non-selected AR-Rs create a binding with flag ‟-” (in the CPM) that is displayed by the show service id vxlan command. Although the VXLAN destinations to non-selected AR-Rs do not carry any traffic, the destinations count against the total limit and must be considered when accounting for consumed VXLAN destinations in the router.

The BM traffic is only sent to the selected AR-R, whereas the U (unknown unicast) traffic is sent to all the destinations with the U flag.

The AR-L performs per-service load-balancing of the BM traffic when two or more AR-Rs exist in the same service. The AR Leaf creates a list of candidate PEs for each AR-R (ordered by IP and VNI; candidate 0 being the lowest IP and VNI). The replicator is selected out of a modulo function of the service-id and the number of replicators, as shown in the following example output.

A:PE-3# show service id 4000 vxlan assisted-replication replicator 
===============================================================================
Vxlan AR Replicator Candidates
===============================================================================
VTEP Address           Egress VNI     In Use  In Candidate List Pending Time
-------------------------------------------------------------------------------
10.2.2.2                4000           yes     yes               0
10.4.4.4                4000           no      yes               0
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================

A change in the number of Replicator-AR routes (for example, if a route is withdrawn or a new route appears) affects the result of the hashing, which may cause a different AR-R to be selected.

The following list summarizes other aspects of the AR-L behavior:

• When a Leaf receives a BM packet on an AC, it sends the packet to its flood list that includes access SAP or SDP bindings and VXLAN destinations with BM or BUM flags. If a single AR-R is selected, only a VXLAN destination includes the BM flags.

• Control plane-generated BM packets, such as ARP/ND (when proxy-ARP/ND is enabled) or Eth-CFM, follow the behavior of regular data plane BM packets.

• When a Leaf receives an unknown unicast packet on an AC, it sends the packet to the flood-list, skipping the AR destination because the U flag is set to 0. To avoid packet re-ordering, the unknown unicast packets do not go through the AR-R.

• When a Leaf receives a BUM packet on an overlay tunnel, it forwards the packet to the flood list, skipping the VXLAN tunnels (that is, the packet is sent to the local ACs and never to a VXLAN tunnel). This is the default IR behavior.

• When the last Replicator-AR route is withdrawn, the AR-L removes the AR destination from the flood list and falls back to ingress replication.

AR BM replication behavior for a BM packet shows the expected replication behavior for BM traffic when received at the access on an AR-R, AR-L, or RNVE router. Unknown unicast follows regular ingress replication behavior regardless of the role of the ingress node for the specific service.

The Assisted-Replication feature has the following limitations:

• The following features are not supported on the same service where the Assisted-Replication feature is enabled.

  • Aggregate QoS per VNI

  • VXLAN IPv6 transport

  • IGMP/MLD/PIM-snooping

• Assisted-Replication Leaf and Replicator functions are mutually exclusive within the same VPLS service.

• The Assisted-Replication feature is supported with IPv4 non-system-ip VXLAN termination. However, the configured assisted-replication-ip (AR-IP) must be different from the tunnel termination IP address.

• The AR-IP address must be a /32 loopback interface on the base router.

• The Assisted-Replication feature is only supported in EVPN-VXLAN services (VPLS with BGP-EVPN vxlan enabled). Although services with a combination of EVPN-MPLS and EVPN-VXLAN are supported, the Assisted-Replication configuration is only relevant to the VXLAN.

EVPN MPLS, as described in EVPN for MPLS tunnels, uses ESI-labels to identify the BUM traffic sourced from a specified ES. The egress PE performs a label lookup to find the ESI label below the EVI label and to determine if a frame can be forwarded to a local ES. Because VXLAN does not support ESI-labels, or any MPLS label for that matter, the split-horizon filtering must be based on the tunnel source IP address. This also implies that the SAP-to-SAP forwarding rules must be changed when the SAPs belong to local ESs, irrespective of the DF state. This new forwarding is what RFC 8365 refers to as local bias. EVPN-VXLAN multihoming with local bias illustrates the local bias forwarding behavior.

Local bias is based on the following principles:

• Every PE knows the IP addresses associated with the other PEs with which it has shared multihomed ESs.

• When the PE receives a BUM frame from a VXLAN bind, it looks up the source IP address in the tunnel header and filters out the frame on all local interfaces connected to ESs that are shared with the ingress PE.

With this approach, the ingress PE must perform replication locally to all directly-attached ESs (regardless of the DF Election state) for all flooded traffic coming from the access interfaces. BUM frames received on any SAP are flooded to:

• local non-ES SAPs and non-ES SDP-binds

• local all-active ES SAPs (DF and NDF)

• local single-active ES SDP-binds and SAPs (DF only)

• EVPN-VXLAN destinations

As an example, in EVPN-VXLAN multihoming with local bias, PE2 receives BUM traffic from Host-3 and it forwards it to the remote PEs and the local ES SAP, even though the SAP is in NDF state.

The following rules apply to egress PE forwarding for EVPN-VXLAN services:

• The source VTEP is looked up for BUM frames received on EVPN-VXLAN.

• If the source VTEP matches one of the PEs with which the local PE shares both an ES and a VXLAN service:

  • the local PE is not forwarded to the shared ES local SAPs

  • the local PE forwards normally to ES SAPs unless they are in NDF state

• Because there is no multicast label or multicast B-MAC in VXLAN, the egress PE only identifies BUM traffic using the customer MAC DA; as a result, BM or unknown MAC DAs identify BUM traffic.

For example, in EVPN-VXLAN multihoming with local bias, PE3 receives BUM traffic on VXLAN. PE3 identifies the source VTEP as a PE with which two ESs are shared, therefore it does not forward the BUM frames to the two shared ESs. It forwards to the non-shared ES (Host-5) because it is in DF state. PE4 receives BUM traffic and forwards it based on normal rules because it does not share any ESs with PE2.

The following command can be used to check whether the local PE has enabled the local bias procedures for a specific ES:

A:PE-2# tools dump service system bgp-evpn ethernet-segment "ES-1" local-bias 
-------------------------------------------------------------------------------
[04/01/2019 08:45:08] Vxlan Local Bias Information 
----------------------------------------------------------------------+--------
Peer                                                                  | Enabled   
----------------------------------------------------------------------+--------
192.0.2.3                                                             | Yes       
-------------------------------------------------------------------------------

In EVPN MPLS networks, an ingress PE that uses ingress replication to flood unknown unicast traffic pushes a BUM MPLS label that is different from a unicast label. The egress PEs use this BUM label to identify such BUM traffic to apply DF filtering for All-Active multihomed sites. In PBB-EVPN, in addition to the multicast label, the egress PE can also rely on the multicast B-MAC DA to identify customer BUM traffic.

In VXLAN there are no BUM labels or any tunnel indication that can assist the egress PE in identifying the BUM traffic. As such, the egress PE must solely rely on the C-MAC destination address, which may create some transient issues that are depicted in EVPN-VXLAN multihoming and unknown unicast issues.

As shown in EVPN-VXLAN multihoming and unknown unicast issues, top diagram, in absence of the mentioned unknown unicast traffic indication there can be transient duplicate traffic to All-Active multihomed sites under the following condition: CE1’s MAC address is learned by the egress PEs (PE1 and PE2) and advertised to the ingress PE3; however, the MAC advertisement has not been received or processed by the ingress PE, resulting in the host MAC address to be unknown on the ingress PE3 but known on the egress PEs. Therefore, when a packet destined for CE1 address arrives on PE3, it floods it through ingress replication to PE1 or PE2 and, because CE1’s MAC is known to PE1 and PE2, multiple copies are sent to CE1.

Another issue is shown at the bottom of EVPN-VXLAN multihoming and unknown unicast issues. In this case, CE1’s MAC address is known on the ingress PE3 but unknown on PE1 and PE2. If PE3’s aliasing hashing picks up the path to the ES’ NDF, a black-hole occurs.

The above two issues are solved in MPLS, as unicast known and unknown frames are identified with different labels.

Finally, another issue is described in Blackhole created by a remote SAP shutdown. Under normal circumstances, when CE3 sends BUM traffic to PE3, the traffic is ‟local-biased” to PE3’s SAP3 even though it is NDF for the ES. The flooded traffic to PE2 is forwarded to CE2, but not to SAP2 because the local bias split-horizon filtering takes place.

The right side of the diagram in Blackhole created by a remote SAP shutdown shows an issue when SAP3 is manually shutdown. In this case, PE3 withdraws the AD per-EVI route corresponding to SAP3; however, this does not change the local bias filtering for SAP2 in PE2. Therefore, when CE3 sends BUM traffic, it can neither be forwarded to CE23 via local SAP3 nor can it be forwarded by PE2.

EVPN VXLAN multihoming is supported on VPLS and R-VPLS services when the PEs use non-system IPv4 or IPv6 termination, however, as with EVPN VPWS services, additional configuration steps are required.

• The configure service system bgp-evpn eth-seg es-orig-ip ip-address command must be configured with the non-system IPv4 or IPv6 address used for the EVPN-VXLAN service. This command modifies the originating-ip field in the ES routes advertised for the Ethernet Segment, and makes the system use this IP address when adding the local PE as DF candidate.

• The configure service system bgp-evpn eth-seg route-next-hop ip-address command must also be configured with the non-system IP address. This command changes the next-hop of the ES and AD per-ES routes to the configured address.

• Finally, the non-system IP address (in each of the PEs in the ES) must match in these three commands for the local PE to be considered suitable for DF election:

  • es-orig-ip ip-address

  • route-next-hop ip-address

  • vxlan-src-vtep ip-address

The 7705 SAR Gen 2 EVPN implementation supports RFC 8560 so that EVPN-MPLS and VPLS can be integrated into the same network and within the same service. Because EVPN is not deployed in green-field networks, this feature is useful for the integration between both technologies and even for the migration of VPLS services to EVPN-MPLS.

The following behavior enables the integration of EVPN and SDP-bindings in the same VPLS network.

Systems with EVPN endpoints and SDP-bindings to the same far-end bring down the SDP-bindings.

• The router allows the establishment of an EVPN endpoint and an SDP-binding to the same far-end, but the SDP-binding is kept operationally down. Only the EVPN endpoint is operationally up. This applies to spoke-SDPs (manual, BGP-AD, and BGP-VPLS) and mesh-SDPs. It is also possible between VXLAN and SDP bindings.

• If there is an existing EVPN endpoint to a specified far-end and a spoke-SDP establishment is attempted, the spoke-SDP is set up but kept down with an operational flag indicating that there is an EVPN route to the same far-end.

• If there is an existing spoke-SDP and a valid or used EVPN route arrives, the EVPN endpoint is set up and the spoke-SDP is brought down with an operational flag indicating that there is an EVPN route to the same far-end.

• In the case of an SDP-binding and EVPN endpoint to different far-end IPs on the same remote PE, both links are up. This may occur if the SDP-binding is terminated in an IPv6 or IPv4 address that is different from the system address where the EVPN endpoint is terminated.

The user can add spoke-SDPs and all the EVPN-MPLS endpoints in the same SHG.

• A CLI command is added under the bgp-evpn>mpls context so that the EVPN-MPLS endpoints can be added to an SHG:

  bgp-evpn mpls [no] split-horizon-group group-name

• The bgp-evpn mpls split-horizon-group command must reference a user-configured SHG. User-configured SHGs can be configured within the service context. The same group-name can be associated with SAPs, spoke-SDPs, pw-templates, pw-template-bindings, and EVPN-MPLS endpoints.

• If the bgp-evpn mpls split-horizon-group command is not used, the default SHG (that contains all the EVPN endpoints) is still used but cannot be referred to on SAPs/spoke-SDPs.

• SAPs and SDP-bindings that share the same SHG of the EVPN-MPLS provider-tunnel are brought operationally down if the point-to-multipoint tunnel is operationally up.

The system disables the advertisement of MACs learned on spoke-SDPs or SAPs that are part of an EVPN SHG.

• When the SAPs and spoke-SDPs (manual or BGP-AD/VPLS-discovered) are configured within the same SHG as the EVPN endpoints, MAC addresses are still learned on them but they are not advertised in EVPN.

• The preceding statement is also true if proxy-ARP/ND is enabled and an IP→MAC pair is learned on a SAP or SDP-binding that belongs to the EVPN SHG.

• The SAPs and spoke-SDPs added to an EVPN SHG should not be part of any EVPN multihomed ES. If that occurs, the PE would still advertise the AD per-EVI route for the SAP or spoke-SDP, attracting EVPN traffic that could not possibly be forwarded to that SAP or SDP-binding.

• Similar to the preceding statement, an SHG composed of SAPs/SDP-bindings used in a BGP-MH site should not be configured under bgp-evpn>mpls>split-horizon-group. This misconfiguration would prevent forwarding of traffic from the EVPN to the BGP-MH site, regardless of the DF/NDF state.

  The following figure shows an example of EVPN-VPLS integration.

  

  The following is an example configuration output of PE1, PE5, and PE2.

*A:PE1>config>service# info 
----------------------------------------------
pw-template 1 create
vpls 1 customer 1 create
  split-horizon-group "SHG-1" create 
  bgp
    route-target target:65000:1
    pw-template-binding 1 split-horizon-group SHG-1 
  bgp-ad
    no shutdown
    vpls-id 65000:1
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      split-horizon-group SHG-1
  spoke-sdp 12:1 create
  exit
  sap 1/1/1:1 create
  exit

*A:PE5>config>service# info 
----------------------------------------------
pw-template 1 create
vpls 1 customer 1 create
  bgp
    route-target target:65000:1
    pw-template-binding 1 split-horizon-group SHG-1 # auto-created SHG
  bgp-ad
    no shutdown
    vpls-id 65000:1
  spoke-sdp 52:1 create
  exit

*A:PE2>config>service# info 
----------------------------------------------
vpls 1 customer 1 create
  end-point CORE create
    no suppress-standby-signaling
  spoke-sdp 21:1 end-point CORE
    precedence primary
  spoke-sdp 25:1 end-point CORE

The following applies to the configuration described in the preceding example.

• PE1, PE3, and PE4 have BGP-EVPN and BGP-AD enabled in VPLS-1. PE5 has BGP-AD enabled and PE2 has active/standby spoke-SDPs to PE1 and PE5. The following applies to this configuration.

  • PE1, PE3, and PE4 attempt to establish BGP-AD spoke-SDPs, but they are kept operationally down as long as there are EVPN endpoints active among them.

  • BGP-AD spoke-SDPs and EVPN endpoints are instantiated within the same split horizon group, for example, SHG-1.

  • Manual spoke-SDPs from PE1 and PE5 to PE2 are not part of SHG-1.

• EVPN MAC advertisements

  • MACs learned on FEC128 spoke-SDPs are advertised normally in EVPN.

  • MACs learned on FEC129 spoke-SDPs are not advertised in EVPN (because they are part of SHG-1, which is the split horizon group used for bgp-evpn mpls). This prevents any data plane MACs learned on the SHG from being advertised in EVPN.

• BUM operation on PE1

  • When CE1 sends BUM, PE1 floods to all the active bindings.

  • When CE2 sends BUM, PE2 sends it to PE1 (active spoke-SDP) and PE1 floods to all the bindings and SAPs.

  • When CE5 sends BUM, PE5 floods to the three EVPN PEs. PE1 floods to the active spoke-SDP and SAPs, never to the EVPN PEs because they are part of the same SHG.

The operation in services with BGP-VPLS and BGP-EVPN is equivalent to the operation for BGP-AD and BGP EVPN described in EVPN and VPLS integration.

In a VPLS service to which multiple EVPN PEs and BGP-VPLS PEs are attached, single-active multihoming is supported on two or more of the EVPN PEs with no special considerations. However, all-active multihoming is not supported because traffic from the all-active multihomed CE could cause a MAC flip-flopping effect on remote BGP-VPLS PEs, asymmetric flows, or other issues.

The following figure shows a scenario with a single-active ES used in a service where EVPN PEs and BGP-VPLS are integrated.

Although other single-active examples are supported, in the preceding figure, CE1 is connected to the EVPN PEs via a single LAG (lag-1). The LAG is associated with eth-segment-1 on PE1 and PE2, which is configured as single-active and with oper-group 1. PE1 and PE2 use lag>monitor-oper-group 1 so that the non-DF PE can signal the non-DF state to CE1 (in the form of Link Aggregation Control Protocol (LACP) out-of-synch or power-off).

In addition to the BGP-VPLS routes sent for the service virtual-edge (VE) ID, the multihoming PEs, in this case, must generate additional BGP-VPLS routes per ES (per VPLS service) to generate a MAC flush on the remote BGP-VPLS PEs in case of failure.

The sap>bgp-vpls-mh-veid number command should be configured on the SAPs that are part of an EVPN single-active ES. This command allows the advertisement of L2VPN routes that indicate the state of the multihomed SAPs to the remote BGP-VPLS PEs. After a DF switchover, the F and D bits of the generated L2VPN routes for the SAP VE ID are updated so that the remote BGP-VPLS PEs can perform a MAC flush operation on the service and avoid blackholes.

For example, in case of a failure on the ES SAP on PE1, PE1 must indicate to PE3 and PE4 the need to flush MAC addresses learned from PE1 (flush-all-from-me message). Otherwise, PE3 continues to send traffic with MAC DA = CE1 to PE1, and PE1 blackholes the traffic.

The following applies to the example in BGP-VPLS to EVPN integration and single-active MH.

• Both ES peers (PE1 and PE2) should be configured with the same VE ID for the ES SAP. However, this is not mandatory.

• In addition to the regular service ve-id L2VPN route, based on the sap>bgp-vpls-mh-ve-id configuration and on BGP VPLS being enabled, the PE advertises an L2VPN route with the following fields:

  • VE ID that contains the value configured using the sap bgp-vpls-mh-ve-id command

  • RD, RT, next hop, and other attributes that are the same as the service BGP VPLS route

  • L2VPN information extended community with the following flags:

    • D=0

      This value applies If the SAP is oper-up or oper-down with a flag MHStandby (for example, the PE is non-DF in single-active MH).

      This value also applies if there is an ES oper-group and the port is down as a result of the oper-group.

    • D=1

      This value applies if the SAP is oper-down with a different flag (for example, port-down or admin-down).

    • F (DF bit) =1

      This value applies if the SAP is oper-up. Otherwise, the value is F=0.

• After a failure on the access SAP, only MAC flush messages are triggered if the bgp-vpls-mh-ve-id command is configured in the failing SAP. If the SAP is configured with VE ID 1, the following applies.

  • If the non-DF PE has a failure on the access SAP, PE2 sends an update with VE ID=1/D=1/F=0. This is an indication for PE3 and PE4 that the PE2 SAP is oper-down but triggering a MAC flush on PE3 and PE4 is not required.

  • If the DF PE has a failure on the SAP, PE1 advertises VE ID=1/D=1/F=0. After receiving this update, PE3 and PE4 flush all their MACs associated with the PE1 spoke-SDP. The failure on PE1 triggers an EVPN DF election on PE2, which becomes DF and advertises VE ID=1/D=0/F=1. This message does not trigger any MAC flush procedures.

The following considerations apply for EVPN single-active multihoming and BGP-VPLS integration.

• PE3 and PE4 can be Nokia nodes running SR OS or any third-party PEs that support the procedures defined in draft-ietf-bess-vpls-multihoming, such that BGP-VPLS MAC flush signaling is understood.

• PE1 and PE2 are expected to run an SR OS version that supports the configuration of the sap bgp-vpls-mh-veid number command on multihomed SAPs. Otherwise, the MAC flush behavior does not work as expected.

• The procedures described in this section are also supported if the EVPN PEs use MC-LAG instead of an ES for CE1 redundancy. In this case, the SAP VE ID route for the standby PE is sent as VE ID=1/D=1/F=0, whereas the active chassis advertises VE ID=1/D=0/F=1. A switchover triggers a MAC flush on the remote PEs, as described in EVPN single-active multihoming and BGP-VPLS integration.

• L2VPN routes generated for ESs or SAPs with the sap bgp-vpls-mh-veid number command are decoded in the remote nodes as BGP-MH routes (because they do not have label information) in the show router bgp routes l2vpn command and debug.

Multiple BGP families and protocols can be enabled at the same time in a VPLS. When bgp-evpn is enabled, bgp-ad and bgp-mh are also supported. A single RD is used per service and not per BGP family or protocol.

The following rules apply.

• The VPLS RD is selected based on the following precedence.

  • Manual RD or automatic RD always take precedence when configured.

  • If manual or automatic RD is not configured, the RD is derived from the bgp-ad>vpls-id.

  • If manual RD, automatic RD, or VPLS ID are not configured, the RD is derived from the bgp-evpn>evi, except for bgp-mh, which does not support EVI-derived RD, and except when the EVI is greater than 65535. In these two cases, no EVI-derived RD is possible.

  • If manual RD, automatic RD, VPLS ID, or EVI is not configured, there is no RD and the service fails.

• The selected RD (see the preceding selection criteria) is displayed by the Oper Route Dist field of the show service id bgp command.

• The service supports dynamic RD changes. For example, the CLI allows the dynamic update of the VPLS ID, even if it is used to automatically derive the service RD for bgp-ad, bgp-vpls, or bgp-mh.

  Note: When the RD changes, the active routes for that VPLS are withdrawn and readvertised with the new RD.

• If one of the mechanisms to derive the RD for a specified service is removed from the configuration, the system selects a new RD based on the preceding rules. For example, if the VPLS ID is removed from the configuration, the routes are withdrawn, the new RD selected from the EVI, and the routes readvertised with the new RD.

  Note: The reconfiguration fails if the new RD already exists in a different VPLS or Epipe service.

• Because the vpls-id takes precedence over the evi when the RD is auto-derived, the existing RD is not affected when evpn is added to an existing bgp-ad service. This is important to support bgp-ad to evpn migration.

EVPN-VPWS for MPLS tunnels uses the RFC 8214 BGP extensions described in EVPN-VPWS for VXLAN tunnels, with the following differences for the Ethernet AD per-EVI routes:

• The MPLS field encodes an MPLS label as opposed to a VXLAN VNI.

• The C flag is set if the control word is configured in the service.

• The F flag is set if the hash label is configured in the service.

The use and configuration of EVPN-VPWS services is described in EVPN-VPWS for VXLAN tunnels with the following differences when the EVPN-VPWS services use MPLS tunnels instead of VXLAN.

When MPLS tunnels are used, the bgp-evpn>mpls context must be configured in the Epipe. As an example, if Epipe 2 is an EVPN-VPWS service that uses MPLS tunnels between PE2 and PE4, this would be its configuration:

PE2>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-attachment-circuit "AC-1" 
    eth-tag 200
exit
 remote-attachment-circuit "AC-2" 
    eth-tag 200
exit
  mpls bgp 1
    ecmp 2
    no shutdown
exit
sap 1/1/1:1 create

PE4>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-attachment-circuit "AC-2" 
    eth-tag 200
exit
  remote-attachment-circuit "AC-1" 
    eth-tag 100
exit
  mpls bgp 1
    ecmp 2
    no shutdown
exit
spoke-sdp 1:1

Where the following BGP-EVPN commands, specific to MPLS tunnels, are supported in the same way as in VPLS services:

• mpls auto-bind-tunnel

• mpls control-word

• mpls hash-label

• mpls entropy-label

• mpls force-vlan-vc-forwarding

• mpls shutdown

EVPN-VPWS Epipes with MPLS tunnels can also be configured with the following characteristics:

• Access attachment circuits can be SAPs or spoke SDPs. Manually configured and BGP-VPWS spoke SDPs are supported. The VC switching configuration is not supported on BGP-EVPN-enabled pipes.

• EVPN-VPWS Epipes using null SAPs can be configured with sap>ethernet>llf. When enabled, upon removing the EVPN destination, the port is brought oper-down with flag LinkLossFwd, however the AD per EVI route for the SAP is still advertised (the SAP is kept oper-up). When the EVPN destination is created, the port is brought oper-up and the flag cleared.

• EVPN-VPWS Epipes for MPLS tunnels support endpoints. The parameter endpoint endpoint name is configurable along with bgp-evpn>local-attachment-circuit and bgp-evpn>remote-attachment-circuit. The following conditions apply to endpoints on EVPN-VPWS Epipes with MPLS tunnels:

  • Up to two explicit endpoints are allowed per Epipe service with BGP-EVPN configured.

  • A limited endpoint configuration is allowed in Epipes with BGP-EVPN. Specifically, neither active-hold-delay nor revert-time are configurable.

  • When bgp-evpn>remote-attachment-circuit is added to an explicit endpoint with a spoke SDP, the spoke-sdp>precedence command is not allowed. The spoke SDP always has a precedence of four, which is always higher than the EVPN precedence. Therefore, the EVPN-MPLS destination is used for transmission if it is created, and the spoke SDP is only used when the EVPN-MPLS destination is removed.

• EVPN-VPWS Epipes for MPLS tunnels support control word, hash label and entropy labels.

  • When the control word is configured, the PE sets the C bit in its AD per-EVI advertisement and sends the control word in the datapath. In this case, the PE expects the control word to be received. If there is a mismatch between the received control word and the configured control word, the system does not set up the EVPN destination and the service does not come up.
  • When the hash-label command is configured, the PE sets the F bit in its AD per-EVI routes and sends the hash label in the datapath. The PE expects the hash-label to also be received. In case of a mismatch between the received F flag and the locally configured hash-label, the router does not create the EVPN destination and the service does not come up. For the service, the use of the hash label and entropy labels are mutually exclusive.

• EVPN-VPWS Epipes support force-qinq-vc-forwarding [c-tag-c-tag | s-tag-c-tag] command under bgp-evpn mpls and the qinq-vlan-translation s-tag.c-tag command on ingress QinQ SAPs.

  When QinQ VLAN translation is configured at the ingress QinQ or dot1q SAP, the service-delimiting outer and inner VLAN values can be translated to the configured values. The force-qinq-vc-forwarding s-tag-c-tag command must be configured to preserve the translated QinQ tags in the payload when sending EVPN packets. This translation and preservation behavior is aligned with the ‟normalization” concept described in draft-ietf-bess-evpn-vpws-fxc. The VLAN tag processing described in Epipe service pseudowire VLAN tag processing applies to EVPN destinations in EVPN-VPWS services too.

The following features, described in EVPN-VPWS for VXLAN tunnels, are also supported for MPLS tunnels:

• Advertisement of the Layer-2 MTU and consistency checking of the MTU of the AD per-EVI routes.

• Use of A/S PW and MC-LAG at access.

• EVPN multihoming, including:

  • Single-active and all-active

  • Regular or virtual ESs

  • All existing DF election modes

Epipes with BGP-EVPN MPLS support the following configurations:

• up to two endpoints

• up to two SAPs, each associated with a different configured endpoint

• two pairs of local/remote attachment circuit Ethernet tags, also associated with different configured endpoints

• EVPN destinations used as Inter-Chassis Backup (ICB) links

The support of endpoints and up to two SAPs with local switching allows two-node and three-node topologies for EVPN-VPWS. See sections EVPN-VPWS endpoints example 1, EVPN-VPWS endpoints example 2, and EVPN-VPWS endpoints example 3 for example topologies.

FXC, specified in draft-ietf-bess-evpn-vpws-fxc, extends EVPN-VPWS services to support multiple SAPs at the access, in contrast with only one SAP in regular EVPN-VPWS services. Multiplexing multiple SAPs into a single EVPN-VPWS tunnel allows the user to save MPLS labels and reduce the number of EVPN Auto-Discovery per EVI routes.

Use the following command to turn an Epipe into a FXC service.

• MD-CLI

  configure service epipe flexible-cross-connect true

• classic CLI

  configure service epipe flexible-cross-connect create

Since there are multiple active egress SAPs in FXC services, EVPN labels are no longer enough to identify the egress SAP on the egress node. Therefore, payload VLAN tags are transmitted into EVPN-encapsulated FXC packets, and the router looks up these inner VLAN tags to forward the decapsulated traffic to the correct egress SAP.

configure service epipe sap qtag-normalization

VLAN normalization configuration and the two FXC modes of operation (default and VLAN aware mode) are described in FXC VLAN normalization, Default FXC mode, and VLAN-aware bundle FXC mode.

SAP and spoke-SDP based ESs are supported on R-VPLS services where bgp-evpn mpls is enabled.

EVPN-MPLS multihoming in R-VPLS services shows an example of EVPN-MPLS multihoming in R-VPLS services, with the following assumptions:

• There are two subnets for a specific customer (for example, EVI1 and EVI2 in EVPN-MPLS multihoming in R-VPLS services), and a VPRN is instantiated in all the PEs for efficient inter-subnet forwarding.

• A ‟backhaul” R-VPLS with evpn-tunnel mode enabled is used in the core to interconnect all the VPRNs. EVPN IP-prefix routes are used to exchange the prefixes corresponding to the two subnets.

• An all-active ES is configured for EVI1 on PE1 and PE2.

• A single-active ES is configured for EVI2 on PE3 and PE4.

In the example in the preceding figure, the hosts connected to CE1 and CE4 can use regular VRRP for default gateway redundancy; however, this may not be the most efficient way to provide upstream routing.

For example, if PE1 and PE2 are using regular VRRP, the upstream traffic from CE1 may be hashed to the backup IRB VRRP interface, instead of being hashed to the master. The same may occur for single-active multihoming and regular VRRP for PE3 and PE4. The traffic from CE4 is sent to PE3, when PE4 may be the VRRP master. In that case, PE3 must send the traffic to PE4, instead of routing it directly.

Both cases use unnecessary bandwidth between the PEs to get to the master IRB interface. In addition, VRRP scaling is limited if aggressive keepalive timers are used.

Because of these issues, passive VRRP is the recommended method when EVPN-MPLS multihoming is used in combination with R-VPLS redundant interfaces.

Passive VRRP is a VRRP setting in which the transmission and reception of keepalive messages is completely suppressed, and therefore, the VPRN interface always behaves as the master. Passive VRRP is enabled by adding the passive keyword to the VRRP instance at creation, as shown in the following examples:

configure service vprn 1 interface int-1 vrrp 1 passive

configure service vprn 1 interface int-1 ipv6 vrrp 1 passive

For example, if PE1, PE2, and PE5 in EVPN-MPLS multihoming in R-VPLS services use passive VRRP, even if each individual R-VPLS interface has a different MAC/IP address, because they share the same VRRP instance 1 and the same backup IP, the three PEs will own the same virtual MAC and virtual IP address (for example, 00-00-5E-00-00-01 and 10.0.0.254). The virtual MAC is auto-derived from 00-00-5E-00-00-VRID per RFC 3768. The following is the expected behavior when passive VRRP is used in this example:

• All R-VPLS IRB interfaces for EVI1 have their own physical MAC/IP address; they also own the same default gateway virtual MAC and IP address.

• All EVI1 hosts have a unique configured default gateway; for example, 10.0.0.254.

• When CE1 or CE2 send upstream traffic to a remote subnet, the packets are routed by the closest PE because the virtual MAC is always local to the PE.

  For example, the packets from CE1 hashed to PE1 are routed at PE1. The packets from CE1 hashed to PE2 are routed directly at PE2.

• Downstream packets (for example, packets from CE3 to CE1), are routed directly by the PE to CE1, regardless of the PE to which PE5 routed the packets.

  For example, the packets from CE3 sent to PE1 are routed at PE1. The packets from CE3 sent to PE2 are routed at PE2.

• In case of ES failure in one of the PEs, the traffic is forwarded by the available PE.

  For example, if the packets routed by PE5 arrive at PE1 and the link to CE1 is down, PE1 sends the packets to PE2. PE2 forwards the packets to CE1 even if the MAC source address of the packets matches PE2's virtual MAC address. Virtual MACs bypass the R-VPLS interface MAC protection.

The following list summarizes the advantages of using passive VRRP mode versus regular VRRP for EVPN-MPLS multihoming in R-VPLS services:

• Passive VRRP does not require multiple VRRP instances to achieve default gateway load-balancing. Only one instance per R-VPLS, therefore only one default gateway, is needed for all the hosts.

• The convergence time for link/node failures is not impacted by the VRRP convergence, because all the nodes in the VRRP instance are master routers.

• Passive VRRP scales better than VRRP, as it does not use keepalive or BFD messages to detect failures and allow the backup to take over.

In EVPN all-active multi-homing scenarios with R-VPLS services where the advertisement of the ARP/ND entries is enabled, use the following command to avoid issues with MAC mobility caused by the MAC/IP advertisement route for the ARP/ND entry being sent with ESI=0:

• MD-CLI

  configure service vpls bgp-evpn routes mac-ip arp-nd-only-with-fdb-advertisement true

• classic CLI

  configure service vpls bgp-evpn arp-nd-only-with-fdb-advertisement

When this command is enabled, the local ARP/ND entries of VPRN interfaces using this VPLS are advertised in this BGP-EVPN service only when the corresponding local MAC is programmed in the FDB.

In an EVPN multi-homing scenario, this command prevents the router from advertising a MAC/IP advertisement route with the MAC and IP binding but without the correct ESI value (which is taken only when the MAC is properly programmed in the FDB against the ESI).

In addition, if an Ethernet Segment SAP receives a frame, the MAC address can be re-programmed as type learned, even if the MAC was previously programmed as type EVPN.

When single-active multihoming is used, the IGMP snooping state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IGMP messages to be received to instantiate the IGMP snooping state after the ES activation timer expires; this could result in an increased outage.

The outage can be reduced by using MCS synchronization, which is supported for IGMP snooping in both EVPN-MPLS and PBB-EVPN services (see Multichassis synchronization for Layer 2 snooping states). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs. Also, IGMP snooping state can be synchronized only on a SAP.

An increased outage would also occur when using all-active EVPN multihoming. The IGMP snooping state on an ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE may not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, IGMP snooping messages can be synchronized between the PEs on an ES using data-driven IGMP snooping state synchronization, which is supported in EVPN-MPLS services, PBB-EVPN services, EVPN-MPLS VPRN and IES R-VPLS services. The IGMP messages received on an ES SAP or spoke SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES B-MAC (for PBB-EVPN) and these are used to synchronize the IGMP snooping state on the ES SAP or spoke SDP on the receiving PE.

Data-driven IGMP snooping state synchronization is supported for both all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS, EVPN-MPLS VPRN and IES R-VPLS services, and for all-active multihoming in PBB-EVPN services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability. PBB-EVPN with IGMP snooping using single-active multihoming is not supported.

Data-driven IGMP snooping state synchronization is also supported with P2MP mLDP LSPs in both EVPN-MPLS and PBB-EVPN services. When P2MP mLDP LSPs are used in EVPN-MPLS services, all PEs (including the PEs not connected to a multihomed ES) in the EVPN-MPLS service must be running an SR OS version supporting this capability with IGMP snooping enabled and all network interfaces must be configured on FP3 or higher-based line cards.

Data-driven IGMP snooping synchronization with EVPN multihoming shows the processing of an IGMP message for EVPN-MPLS. In PBB-EVPN services, the ES B-MAC is used instead of the ESI label to synchronize the state.

Data-driven synchronization is enabled by default when IGMP snooping is enabled within an EVPN-MPLS service using all-active multihoming or single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If IGMP snooping MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP then MCS synchronization takes precedence over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS IGMP synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down which prevents state synchronization to the PE with the ES down, as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it may not have any IGMP snooping state until the next IGMP messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive can avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services:

1. Upgrade SR OS on all ES PEs to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.

2. Enable IGMP snooping in the required services on all ES PEs. Traffic loss occurs until all ES PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

   Note: There is no action required on the non-ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services:

1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.

2. Enable IGMP snooping in EVPN-MPLS and PBB-EVPN services.

   • Perform the following steps for EVPN-MPLS:

     • Enable IGMP snooping on all non-ES PEs. Traffic loss occurs until the first set of join/query messages are processed by the non-ES PEs.

     • Then enable IGMP snooping on all ES PEs. Traffic loss occurs until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

   • Perform the following steps for PBB-EVPN:

     • Enable IGMP snooping on all ES PEs. Traffic loss occurs until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

     • There is no action required on the non-ES PEs.

To aid with troubleshooting, the debug packet output displays the IGMP packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows.

6 2017/06/16 18:00:07.819 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: TX packet on svc 1
  from chaddr 5e:00:00:16:d8:2e
  send towards ES:esi-1
  Port  : evpn-mpls
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1


4 2017/06/16 18:00:07.820 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: RX packet on svc 1
  from chaddr d8:2e:ff:00:01:41
  received via evpn-mpls on ES:esi-1
  Port  : sap lag-1:1
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1

When single-active multihoming is used, PIM snooping for IPv4 state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IPv4 PIM messages to be received to instantiate the PIM snooping for IPv4 state after the ES activation timer expires, which could result in an increased outage.

This outage can be reduced by using MCS synchronization, which is supported for PIM snooping for IPv4 in both EVPN-MPLS and PBB-EVPN services (see Multichassis synchronization for Layer 2 snooping states). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs.

An increased outage would also occur when using all-active EVPN multihoming. The PIM snooping for IPv4 state on an all-active ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE may not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, snooped IPv4 PIM messages should be synchronized between the PEs on an ES using data-driven PIM snooping for IPv4 state synchronization, which is supported in both EVPN-MPLS and PBB-EVPN services. The IPv4 PIM messages received on an ES SAP or spoke SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES B-MAC (for PBB-EVPN) and are used to synchronize the PIM snooping for IPv4 state on the ES SAP or spoke SDP on the receiving PE.

Data-driven PIM snooping state synchronization is supported for all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability with PIM snooping for IPv4 enabled. It is also supported with P2MP mLDP LSPs in the EVPN-MPLS services, in which case all PEs (including the PEs not connected to a multihomed ES) must have PIM snooping for IPv4 enabled and all network interfaces must be configured on FP3 or higher-based line cards.

In addition, data-driven PIM snooping state synchronization is supported for all-active multihoming in PBB-EVPN services and with P2MP mLDP LSPs in PBB-EVPN services. All PEs participating in a multihomed ES, and all PEs using PIM proxy mode (including the PEs not connected to a multihomed ES) in the PBB-EVPN service must be running an SR OS version supporting this capability and must have PIM snooping for IPv4 enabled. PBB-EVPN with PIM snooping for IPv4 using single-active multihoming is not supported.

Data-driven PIM snooping for IPv4 synchronization with EVPN multihoming shows the processing of an IPv4 PIM message for EVPN-MPLS. In PBB-EVPN services, the ES B-MAC is used instead of the ESI label to synchronize the state.

Data-driven synchronization is enabled by default when PIM snooping for IPv4 is enabled within an EVPN-MPLS service using all-active multihoming and single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If PIM snooping for IPv4 MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP or spoke SDP, then MCS synchronization takes preference over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS PIM synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down, which prevents state synchronization to the PE with the ES down as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it may not have any PIM snooping for IPv4 state until the next PIM messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive can avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling PIM snooping for IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services:

• PIM snooping mode

  1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

     Note: There is no action required on the non-ES PEs.

• PIM proxy mode

  • EVPN-MPLS

    1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

    2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

       Note: There is no action required on the non-ES PEs.

  • PBB-EVPN

    1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

    2. Enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.

    3. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling PIM snooping or IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services.

• PIM snooping mode

  1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Then enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.

     Note: There is no action required on the non-ES PEs.

• PIM proxy mode

  1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.

  3. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.

In the above steps, when PIM snooping for IPv4 is enabled, the traffic loss can be reduced or eliminated by configuring a larger hold-time (up to 300 seconds), during which multicast traffic is flooded.

To aid with troubleshooting, the debug packet output displays the PIM packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows:

6 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimVplsFwdJPToEvpn
Forwarding to remote peer on bgp-evpn ethernet-segment esi-1"
7 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:37.040] PIM-TX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>


4 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimProcessPdu
Received from remote peer on bgp-evpn ethernet-segment esi-1, will be applied on
 lag-1:1
"
5 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:30.740] PIM-RX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>

When enable-rr-vpn-forwarding or enable-inter-as-vpn is configured, only EVPN-MPLS routes are processed for label swap and the next hop is changed. EVPN-VXLAN routes are re-advertised without a change in the next hop.

The following shows how the router processes and re-advertises the different EVPN route types. For more information about the route fields, see the BGP-EVPN control plane for MPLS tunnels Guide.

• Auto-discovery (AD) routes (type 1)

  For AD per EVI routes, the MPLS label is extracted from the route NLRI. The route is re-advertised with Next-Hop-Self (NHS) and a new label. No modifications are made for the remaining attributes.

  For AD per ES routes, the MPLS label in the NLRI is zero. The route is re-advertised with NHS and the MPLS label remains zero. No modifications are made for the remaining attributes.

• MAC/IP routes (type 2)

  The MPLS label (Label-1) is extracted from the NLRI. The route is re-advertised with NHS and a new Label-1. No modifications are made for the remaining attributes.

• Inclusive Multicast Ethernet Tag (IMET) routes (type 3)

  Because there is no MPLS label present in the NLRI, the MPLS label is extracted from the PMSI Tunnel Attribute (PTA) if needed, and the route is then re-advertised with NHS, with the following considerations:

  • For IMET routes with tunnel-type Ingress Replication, the router extracts the IR label from the PTA. The router programs the label swap and re-advertises the route with a new label in the PTA.

  • For tunnel-type P2MP mLDP, the router re-advertises the route with NHS. No label is extracted; therefore, no swap operation occurs.

  • For tunnel-type Composite, the IR label is extracted from the PTA, the swap operation is programmed and the route re-advertised with NHS. A new label is encoded in the PTA’s IR label with no other changes in the remaining fields.

  • For tunnel-type AR, the routes are always considered VXLAN routes and are re-advertised with the next-hop unchanged.

• Ethernet-Segment (ES) routes (type 4)

  Because ES routes do not contain an MPLS label, the route is re-advertised with NHS and no modifications to the remaining attributes. Although an ASBR or ABR re-advertises ES routes, EVPN multihoming for ES PEs located in different ASs or IGMP domains is not supported.

• IP-Prefix routes (type 5)

  The MPLS label is extracted from the NLRI and the route is re-advertised with NHS and a new label. No modifications are made to the remaining attributes.

Inter-AS Option B and VPN-NH-RR support the use of non-segmented trees for forwarding BUM traffic in EVPN.

For ingress replication and non-segmented trees, the ASBR or ABR performs an EVPN BUM label swap without any aggregation or further replication. This concept is shown in VPN-NH-RR and ingress replication for BUM traffic.

In VPN-NH-RR and ingress replication for BUM traffic, when PE2, PE3, and PE4 advertise their IMET routes, the ABRs re-advertise the routes with NHS and a different label. However, IMET routes are not aggregated; therefore, PE1 sets up three different EVPN multicast destinations and sends three copies of every BUM packet, even if they are sent to the same ABR. This example is also applicable to ASBRs and Inter-AS Option B.

P2MP mLDP may also be used with VPN-NH-RR, but not with Inter-AS Option B. The ABRs, however, do not aggregate or change the mLDP root IP addresses in the IMET routes. The root IP addresses must be leaked across IGP domains. For example, if PE2 advertises an IMET route with mLDP or composite tunnel type, PE1 is able to join the mLDP tree if the root IP is leaked into PE1’s IGP domain.

When proxy-ARP/proxy-ND is enabled, the system starts populating the proxy table and responding to ARP-requests/NS messages. To keep the active IP→MAC entries alive and ensure that all the host/routers in the service update their ARP/ND caches, the system may generate the following three types of ARP/ND messages for a specified IP→MAC entry:

• periodic refresh messages (ARP-requests or NS for a specified IP)

  These messages are activated by the send-refresh command and their objective is to keep the existing FDB and Proxy-ARP/ND entries alive, to minimize EVPN withdrawals and re-advertisements.

• unsolicited refresh messages (unsolicited GARP or NA messages)

  These messages are sent by the system when a new entry is learned or updated. Their objective is to update the attached host/router caches.

• confirm messages (unicast ARP-requests or unicast NS messages)

  These messages are sent by the system when a new MAC is learned for an existing IP. The objective of the confirm messages is to verify that a specified IP has really moved to a different part of the network and is associated with the new MAC. If the IP has not moved, it forces the owners of the duplicate IP to reply and cause dup-detect to kick in.

In order for the ingress and egress EVPN PEs to install the proxy-ARP/ND entries with the same property flags, RFC 9047 describes how those flags (R, O and I) can be conveyed in the EVPN ARP/ND extended community advertised along with the EVPN MAC/IP Advertisement routes. The format of the EVPN ARP/ND extended community follows:

By default, the router does not advertise the ARP/ND extended community. Use the following command to configure the router to advertise all the proxy ARP/ND MAC/IP Advertisement routes with the extended community:

configure service vpls bgp-evpn arp-nd-extended-community

SR OS supports the association of configured MAC lists with a configured dynamic proxy-ARP or proxy-ND IP address. The actual proxy-ARP or proxy-ND entry is not created until an ARP or Neighbor Advertisement message is received for the IP and one of the MACs in the associated MAC-list. This is in accordance with IETF RFC 9161, which states that a proxy-ARP or proxy-ND IP entry can be associated with one MAC among a list of allowed MACs.

The following example shows the use of MAC lists for dynamic entries.

A:PE-2>config>service#
  proxy-arp-nd
    mac-list ISP-1 create 
      mac 00:de:ad:be:ef:01 
      mac 00:de:ad:be:ef:02 
      mac 00:de:ad:be:ef:03
 
A:PE-2>config>service>vpls>proxy-arp#
  dynamic 1.1.1.1 create
    mac-list ISP-1
    resolve 30
 
A:PE-2>config>service>vpls>proxy-nd#
  dynamic 2001:db8:1000::1 create
    mac-list ISP-1 
    resolve 30

where:

• A dynamic IP (dynamic ip create) is configured and associated with a MAC list (mac-list name).

• The MAC list is created in the config>service context and can be reused by multiple configured dynamic IPs as follows:

  • in different services

  • in the same service, for proxy-ARP and proxy-ND entries

• If the MAC list is empty, the proxy-ARP or proxy-ND entry is not created for the configured IP.

• The same MAC list can be applied to multiple configured dynamic entries even within the same service.

• The new proxy-ARP and proxy-ND entries behave as dynamic entries and are displayed as type dyn in the show commands.

  The following output example displays the entry corresponding to the configured dynamic IP.

show service id 1 proxy-arp detail

-------------------------------------------------------------------------------
Proxy Arp
-------------------------------------------------------------------------------
Admin State       : enabled             
Dyn Populate      : enabled             
Age Time          : 900 secs            Send Refresh      : 300 secs
Table Size        : 250                 Total             : 1
Static Count      : 0                   EVPN Count        : 0
Dynamic Count     : 1                   Duplicate Count   : 0
Dup Detect
-------------------------------------------------------------------------------
Detect Window     : 3 mins              Num Moves         : 5
Hold down         : 9 mins              
Anti Spoof MAC    : None
EVPN
-------------------------------------------------------------------------------
Garp Flood        : enabled             Req Flood         : enabled
Static Black Hole : disabled            
-------------------------------------------------------------------------------
===============================================================================
VPLS Proxy Arp Entries
===============================================================================
IP Address          Mac Address       Type Status    Last Update
-------------------------------------------------------------------------------
1.1.1.1            00:de:ad:be:ef:01  dyn active    02/23/2016 09:05:49
-------------------------------------------------------------------------------
Number of entries : 1
===============================================================================

show service proxy-arp-nd mac-list "ISP-1" associations

===============================================================================
MAC List Associations
===============================================================================
Service Id                    IP Addr
-------------------------------------------------------------------------------
1                             1.1.1.1
1                             2001:db8:1000::1
-------------------------------------------------------------------------------
Number of Entries: 2
===============================================================================

The EVPN loop detection procedure, described in the preceding section, is compliant with draft-ietf-bess-rfc7432bis and is an efficient way of detecting and blocking loops in EVPN networks. Contrary to other intrusive methods that inject Ethernet beacons into the customer network and detect loops depending on whether the beacon messages get back to the PEs, the EVPN loop detection mechanism is non intrusive because it relies entirely on learning the same MAC on different nodes. However, the mechanism lacks determinism, as shown in the following figure.

Assume PE1, PE2 and PE3 are attached to the same EVPN VPLS service, and there is an accidental backdoor link between the Base Stations connected to PE2 and PE3. When the Controller with MAC M1 issues a broadcast frame, PE1 forwards it to PE2 and PE3, and the frame is looped back via the backdoor link. The mac-duplication procedure kicks in and M1 is detected as duplicate and turned into a blackhole MAC in the FDB, effectively solving the loop. However, the user does not know beforehand if M1 is blackholed in PE1, PE2, PE3 or multiple of them at the same time. If M1 is blackholed in PE1, this represents an issue for the hosts connected to other PEs (not shown) attached to the same service. Therefore, in the example, we want to influence the mac-duplication procedure so that M1 gets blackholed in PE2, PE3, or both, but not in PE1. To make the procedure more deterministic, the Trusted MAC concept is used.

configure service vpls bgp-evpn mac-duplication trusted-mac-time

Trusted MACs are affected by the mac-duplication procedures in a different way from the non-trusted MACs. Trusted MACs require a different number of moves (during the mac-duplication window) to be declared as duplicate, specified by num-moves <number> * trusted-mac-move-factor <number> in the following context.

configure service vpls bgp-evpn mac-duplication detect

While non-trusted MACs are detected as duplicate after num-moves, trusted MACs need more moves to be declared as duplicate.

// Applicable to PE1, PE2 and PE3

[ex:/configure service vpls "bd-1000" bgp-evpn mac-duplication]
A:admin@node-2# info
  blackhole true
  trusted-mac-time 5 // value 1..15, default: 5
  detect {
    num-moves 5
    window 3
    trusted-mac-move-factor 3 // value 1..10, default: 1
  }

// Applicable to PE1, PE2 and PE3

A:node-2>config>service>vpls>bgp-evpn>mac-duplication# info 
----------------------------------------------
  detect num-moves 5 window 3 trusted-mac-move-factor 3 // value 1..10, default: 1
  black-hole-dup-mac
  trusted-mac-time 5 // value 1..15, default: 5

show service id 1000 fdb detail

===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId   MAC               Source-Identifier   Type     Last Change
          Transport:Tnl-Id                     Age 
-------------------------------------------------------------------------------
1000     00:de:fe:da:da:04 sap:1/1/1:1000      LT/0     05/18/23 10:54:54
-------------------------------------------------------------------------------
No. of MAC Entries: 1
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

This section describes the configuration aspects of a VPLS/R-VPLS with EVPN-VXLAN and EVPN-MPLS.

In a service where EVPN-VXLAN and EVPN-MPLS are configured together, the configure service vpls bgp-evpn vxlan bgp 1 and configure service vpls bgp-evpn mpls bgp 2 commands allow the user to associate EVPN-MPLS to a different instance from that associated with EVPN-VXLAN, and have both encapsulations simultaneously enabled in the same service. At the control plane level, EVPN MAC/IP advertisement routes received in one instance are consumed and readvertised in the other instance as long as the route is the best route for a specific MAC. Inclusive multicast routes are independently generated for each BGP instance. In the data plane, the EVPN-MPLS and EVPN-VXLAN destinations are instantiated in different implicit Split Horizon Groups (SHGs) so that traffic can be forwarded between them.

The following example shows a VPLS service with two BGP instances and both VXLAN and MPLS encapsulations configured for the same BGP-EVPN service.

*A:PE-1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls and evpn-vxlan in the same service"
  vxlan instance 1 vni 7000 create
  exit
  bgp 
    route-distinguisher 10:2
    route-target target:64500:1
  exit
  bgp 2 
    route-distinguisher 10:1
    route-target target:64500:1
  exit
  bgp-evpn
    evi 7000
    incl-mcast-orig-ip 10.12.12.12
    vxlan bgp 1 vxlan-instance 1
      no shutdown 
    mpls bgp 2
      control-word
      auto-bind-tunnel 
        resolution any
      exit
      force-vlan-vc-forwarding
      no shutdown
    exit
  exit
  no shutdown

The following list describes the preceding example:

• bgp 1 or bgp is the default BGP instance

• bgp 2 is the additional instance required when both bgp-evpn vxlan and bgp-evpn mpls are enabled in the service

• The commands supported in instance 1 are also available in instance 2 with the following considerations:

  • pw-template-binding

    The pw-template-binding can only exist in instance 1; it is not supported in instance 2.

  • route-distinguisher

    The operating route-distinguisher in both BGP instances must be different.

  • route-target

    The route target in both instances can be the same or different.

  • vsi-import and vsi-export

    Import and export policies can also be defined for either BGP instance.

• MPLS and VXLAN can use either BGP instance, and the instance is associated when bgp-evpn mpls or bgp-evpn vxlan is created. The bgp-evpn vxlan command must include not only the association to a BGP instance, but also to a vxlan-instance (because the VPLS services support two VXLAN instances).

  Note: The bgp-evpn vxlan no shutdown command is only allowed if bgp-evpn mpls shutdown is configured, or if the BGP instance associated with the MPLS has a different route distinguisher than the VXLAN instance.

The following features are not supported when two BGP instances are enabled on the same VPLS/R-VPLS service:

• SDP bindings

• M-VPLS, I-VPLS, B-VPLS, or E-Tree VPLS

• Proxy-ARP and proxy-ND

• BGP Multihoming

• IGMP, MLD, and PIM snooping

• BGP-VPLS or BGP-AD (SDP bindings are not created)

The service>vpls>bgp-evpn>ip-route-advertisement command is not supported on R-VPLS services with two BGP instances.

EVPN-SRv6 and EVPN-MPLS or EVPN-VXLAN can be simultaneously configured in the same VPLS service (but not R-VPLS), in different instances. In addition, EVPN-SRv6 and EVPN-MPLS can be simultaneously configured in the same Epipe service, so that border routers can stitch SRv6 and MPLS domains for point-to-point services.

The following sections describe BGP-EVPN routes in EVPN and VPLS services configured with two BGP instances.

The following sections describe the anycast redundant solution for dual BGP instances in VPLS and Epipe services.

Anycast multihoming and mLDP shows an example of a common BGP EVPN service configured in redundant anycast DCGWs and mLDP used in the MPLS instance.

When mLDP is used with multiple anycast multihoming DCGWs, the same originating IP address must be used by all the DCGWs. Failure to do so may result in packet duplication.

In the example shown in Anycast multihoming and mLDP, each pair of DCGWs (DCGW1/DCGW2 and DCGW3/DCGW4) is configured with a different originating IP address, which causes the following behavior:

1. DCGW3 and DCGW4 receive the inclusive multicast routes with the same route key from DCGW1 and DCGW2.

2. Both DCGWs (DCGW3 and DCGW4) select only one route, which is generally the same, for example, DCGW1's inclusive multicast route.

3. As a result, DCGW3 and DCGW4 join the mLDP tree with root in DCGW1, creating packet duplication when DCGW1 sends BUM traffic.

4. Remote PE nodes with a single BGP-EVPN instance join the mLDP tree without any problem.

To avoid the packet duplication shown in Anycast multihoming and mLDP, Nokia recommends to configure the same originating IP address in all four DCGWs (DCGW1/DCGW2 and DCGW3/DCGW4). However, the route distinguishers can be different per pair.

The following behavior occurs if the same originating IP address is configured on the DCGW pairs shown in Anycast multihoming and mLDP.

• DCGW3 and DCGW4 do not join any mLDP tree sourced from DCGW1 or DCGW2, which prevents any packet duplication. This is because a router ignore inclusive multicast routes received with its own originating-ip, regardless of the route-distinguisher.

• PE1 joins the mLDP trees from the two DCs.

SR OS supports Interconnect ESs (I-ES) for VXLAN as per RFC9014. An I-ES is a virtual ES that allows DCGWs with two BGP instances to handle VXLAN access networks as any other type of ES. I-ESs support the RFC 7432 multihoming functions, including single-active and all-active, ESI-based split-horizon filtering, DF election, and aliasing and backup on remote EVPN-MPLS PEs.

In addition to the EVPN multihoming features, the main advantages of the I-ES redundant solution compared to the redundant solution described in Anycast redundant solution for dual BGP-instance services are as follows:

• The use of I-ES for redundancy in dual BGP-instance services allows local SAPs on the DCGWs.

• P2MP mLDP can be used to transport BUM traffic between DCs that use I-ES without any risk of packet duplication. As described in Using P2MP mLDP in redundant anycast DCGWs, packet duplication may occur in the anycast DCGW solution when mLDP is used in the WAN.

Where EVPN-MPLS networks are interconnected to EVPN-VXLAN networks, the I-ES concept applies only to the access VXLAN network; the EVPN-MPLS network does not modify its existing behavior.

The Interconnect ES concept shows the use of I-ES for Layer 2 EVPN DCI between VXLAN and MPLS networks.

The following example shows how I-ES-1 would be provisioned on DCGW1 and the association between I-ES to a specified VPLS service. A similar configuration would occur on DCGW2 in the I-ES.

New I-ES configuration:

DCGW1#config>service>system>bgp-evpn# 
ethernet-segment I-ES-1 virtual create
 esi 01:00:00:00:12:12:12:12:12:00
 service-carving
  mode auto
 multi-homing all-active
 network-interconnect-vxlan 1
 service-id
   service-range 1 to 1000
 no shutdown

Service configuration:

DCGW1#config>service>vpls(1)# 
 vxlan instance 1 vni 1 instance 1 create
 exit
bgp
 route-distinguisher 1:1 
bgp 2
 route-distinguisher 2:2               
bgp-evpn        
  evi 1
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any                     
   no shutdown

...

DCGW1#config>service>vpls(2)#  
 vxlan instance 1 vni 2 create
 exit
bgp
 route-distinguisher 3:3 
bgp 2
 route-distinguisher 4:4               
 bgp-evpn        
  evi 2
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any   
   no shutdown
 sap 1/1/1:1 create
 exit 

The above configuration associates I-ES-1 to the VXLAN instance in services VPLS1 and VPLS 2. The I-ES is modeled as a virtual ES, with the following considerations:

• The commands network-interconnect-vxlan and service-id service-range svc-id [to svc-id] are required within the ES.

  • The network-interconnect-vxlan parameter identifies the VXLAN instance associated with the virtual ES. The value of the parameter must be set to 1. This command is rejected in a non-virtual ES.

  • The service-range parameter associates the specific service range to the ES. The ES must be configured as network-interconnect-vxlan before any service range can be added.

  • The ES parameters port, lag, sdp, vc-id-range, dot1q, and qinq cannot be configured in the ES when a network-interconnect-vxlan instance is configured. The source-bmac-lsb option is blocked, as the I-ES cannot be associated with an I-VPLS or PBB-Epipe service. The remaining ES configuration options are supported.

  • All services with two BGP instances associate the VXLAN destinations and ingress VXLAN instances to the ES.

• Multiple services can be associated with the same ES, with the following considerations:

  • In a DC with two DCGWs (as in The Interconnect ES concept), only two I-ESs are needed to load-balance, where one half of the dual BGP-instance services would be associated with one I-ES (for example, I-ES-1, in the above configuration) and one half to another I-ES.

  • Up to eight service ranges per VXLAN instance can be configured. Ranges may overlap within the same ES, but not between different ESs.

  • The service range can be configured before the service.

• After the I-ES is configured using network-interconnect-vxlan, the ES operational state depends exclusively on the ES administrative state. Because the I-ES is not associated with a physical port or SDP, when testing the non-revertive service carving manual mode, an ES shutdown and no shutdown event results in the node sending its own administrative preference and DP bit and taking control if the preference and DP bit are higher than the current DF. This is because the peer ES routes are not present at the EVPN application layer when the ES is configured for no shutdown; therefore, the PE sends its own administrative preference and DP values. For I-ESs, the non-revertive mode works only for node failures.

• A VXLAN instance may be placed in MhStandby under any of the following situations:

  • if the PE is single-active NDF for that I-ES

  • if the VXLAN service is added to the I-ES and either the ES or BGP-EVPN MPLS is shut down in all the services included in the ES

    The following example shows the change of the MhStandby flag from false to true when BGP-EVPN MPLS is shut down for all the services in the I-ES.

    A:PE-4# show service id 500 vxlan instance 1 oper-flags 
    ===============================================================================
    VPLS VXLAN oper flags
    ===============================================================================
    MhStandby                              : false
    ===============================================================================
    A:PE-4# configure service vpls 500 bgp-evpn vxlan shutdown 
    *A:PE-4# show service id 500 vxlan instance 1 oper-flags    
    ===============================================================================
    VPLS VXLAN oper flags
    ===============================================================================
    MhStandby                              : true
    ===============================================================================
    

If two BGP instances with the same encapsulation (VXLAN or MPLS) are configured in the same VPLS/R-VPLS service, different import route targets in each BGP instance are mandatory (although this is not enforced).

BGP-EVPN routes in services configured with two BGP instances describes the use of policies to avoid sending WAN routes (routes meant to be redistributed from DC to WAN) to the DC again and DC routes (routes meant to be redistributed from WAN to DC) to the WAN again. Those policies are based on export policy statements that match on the RFC 9012 BGP encapsulation extended community (MPLS and VXLAN respectively).

When the two BGP instances are of the same encapsulation (VXLAN or MPLS), the policies matching on different BGP encapsulation extended community are not feasible because both instances advertise routes with the same encapsulation value. Because the export route targets in the two BGP instances must be different, the policies, to avoid sending WAN routes back to the WAN and DC routes back to the DC, can be based on export policies that prevent routes with a DC route target from being sent to the WAN peers (and opposite for routes with a WAN route target).

In scaled scenarios, matching based on route targets, does not scale well. An alternative and preferred solution is to configure a default-route-tag that identifies all the EVPN instances connected to the DC (or one domain), and a different default-route-tag in all the EVPN instances connected to the WAN (or the other domain). Anycast redundant solution for multi-instance EVPN services with the same encapsulation shows an example that demonstrates the use of default-route-tags.

Other than the specifications described in this section, the processing of MAC/IP routes and inclusive multicast Ethernet tag routes in multi-instance EVPN services of the same encapsulation follow the rules described in BGP-EVPN routes in services configured with two BGP instances.

The solution described in Anycast redundant solution for dual BGP-instance services is also supported in multi-instance EVPN VPLS/R-VPLS services with the same encapsulation.

The following CLI example output shows the configuration of DCGW-1 and DCGW-2 in Multihomed anycast solution where VPLS 500 is a multi-instance EVPN-VXLAN service and BGP instance 2 is associated with VXLAN instead of MPLS.

Different default-route-tags are used in BGP instance 1 and instance 2, so that in the export route policies, DC routes are not advertised to the WAN, and WAN routes are not advertised to the DC, respectively.

*A:DCGW-1(and DCGW-2)>config>service>vpls(500)# info
----------------------------------------------
vxlan instance 1 vni 500 create
exit
vxlan instance 2 vni 501 create
exit
bgp
 route-distinguisher 192.0.2.2:500
 route-target target:64500:500
exit
bgp 2
 route-distinguisher 192.0.2.2:501
 route-target target:64500:501
exit
bgp-evpn
 incl-mcast-orig-ip 23.23.23.23
 evi 500
 vxlan bgp 1 vxlan-instance 1
 default-route-tag 500
 no shutdown
 exit
 vxlan bgp 2 vxlan-instance 2
   default-route-tag 501
   no shutdown
 exit
exit
stp
shutdown
exit
no shutdown
----------------------------------------------
config>router>bgp#
 vpn-apply-import
 vpn-apply-export
 group "WAN"
  family evpn
  type internal
  export "allow only mpls"
  neighbor 192.0.2.6
 group "DC"
  family evpn
  type internal
export "allow only vxlan"
  neighbor 192.0.2.2

config>router>policy-options# info
----------------------------------------------
            policy-statement "allow only mpls"
                entry 10
                  from
                    family evpn 
                    tag 500 
                  action drop
                  exit
                exit
            exit
            policy-statement "allow only vxlan"
                entry 10
                  from
                    family evpn 
                    tag 501 
                  action drop
       exit
                  exit
            exit

The same Anycast redundant solution can be applied to VPLS/R-VPLS with two instances of EVPN-MPLS encapsulation. The configuration would be identical, other than replacing the VXLAN aspects with the EVPN-MPLS-specific parameters.

For a full description of this solution, see the Anycast redundant solution for dual BGP-instance services

The I-ES of network-interconnect VXLAN Ethernet segment is described in I-ES solution for dual BGP instance services. I-ES’s are also supported on VPLS and R-VPLS services with two EVPN-VXLAN instances.

I-ES in dual EVPN-VXLAN services shows the use of an I-ES in a dual EVPN-VXLAN instance service.

Similar to (single-instance) EVPN-VXLAN all-active multihoming, the BUM forwarding procedures follow the ‟Local Bias” behavior.

At the ingress PE, the forwarding rules for EVPN-VXLAN services are as follows:

• The no send-imet-ir-on-ndf or rx-discard-on-ndf bum command must be enabled so that the NDF does not forward any BUM traffic.

• BUM frames received on any SAP or I-ES VXLAN binding are flooded to:

  • local non-ES and single-active DF ES SAPs

  • local all-active ES SAPs (DF and NDF)

  • EVPN-VXLAN destinations

    BUM received on an I-ES VXLAN binding follows SHG rules, for example, it can only be forwarded to EVPN-VXLAN destinations that belong to the other VXLAN instance (instance 2), which is a different SHG.

• As an example, in I-ES in dual EVPN-VXLAN services:

  • GW1 and GW2 are configured with no send-imet-ir-on-ndf.

  • TOR1 generates BUM traffic that only reaches GW1 (DF).

  • GW1 forwards to CE1 and EVPN-VXLAN destinations.

The forwarding rules at the egress PE are as follows:

• The source VTEP is looked up for BUM frames received on EVPN-VXLAN.

• If the source VTEP matches one of the PEs with which the local PE shares an ES _AND_ a VXLAN service:

  • Then the local PE does not forward to the shared local ESs (this includes port, lag, or network-interconnect-vxlan ESs). It forwards though to non-shared ES SAPs unless they are in NDF state.

  • Else, the local PE forwards normally to local ESs unless they are in NDF state.

• Because there is no multicast label or multicast B-MAC in VXLAN, the only way the egress PE can identify BUM traffic is by looking at the customer MAC DA. Therefore, BM or unknown MAC DAs identify BUM traffic.

• As an example, in I-ES in dual EVPN-VXLAN services:

  • GW2 receives BUM on EVPN-VXLAN. GW2 identifies the source VTEP as a PE with which the I-ES-1 is shared, therefore it does not forward the BUM frames to the local I-ES. It forwards to the non-shared ES and local SAPs though (CE2).

  • GW3 receives BUM on EVPN-VXLAN, however the source VTEP does not match any PE with which GW3 shares an ES. Hence GW3 forwards to all local ESs that are DF, in other words, CE3.

The following configuration example shows how I-ES-1 would be provisioned on DCGW1 and the association between I-ES to a specified VPLS service. A similar configuration would occur on DCGW2 in the I-ES.

I-ES configuration:

*A:GW1>config>service>system>bgp-evpn>eth-seg# info 
----------------------------------------------
                esi 00:23:23:23:23:23:23:00:00:01
                service-carving
                    mode manual
                    manual
                        preference non-revertive create
                            value 150
                        exit
                        evi 101 to 200
                    exit
                exit
                multi-homing all-active
                network-interconnect-vxlan 1
                service-id
                    service-range 1
                    service-range 1000 to 1002
                    service-range 2000
                exit
                no shutdown

Service configuration:

*A:GW1>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 1000 create
                rx-discard-on-ndf bum
            exit
            vxlan instance 2 vni 1002 create
            exit
            bgp
                route-target export target:64500:1000 import target:64500:1000
            exit
            bgp 2
                route-distinguisher auto-rd
                route-target export target:64500:1002 import target:64500:1002
            exit
            bgp-evpn
                evi 1000
                vxlan bgp 1 vxlan-instance 1
                    ecmp 2
                    default-route-tag 100
                    auto-disc-route-advertisement
                    no shutdown
                exit
                vxlan bgp 2 vxlan-instance 2
                    ecmp 2
                    default-route-tag 200
                    auto-disc-route-advertisement
                    mh-mode network
                    no shutdown
                exit
            exit
            no shutdown 

Multi-instance EVPN VPLS/R-VPLS services with two EVPN-MPLS instances do not support I-ESs.

For information about how the EVPN routes are processed and advertised in an I-ES, see the I-ES solution for dual BGP instance services.

The SBD is equivalent to an R-VPLS that connects all the PEs that are attached to the same tenant VPRN. Interface-ful refers to the use of a full IRB interface between the VPRN and the SBD (an interface object with MAC and IP addresses, over which interface parameters can be configured).

The following figure shows an example of the interface-ful IP-VRF-to-IP-VRF with SBD IRB model.

In the preceding figure, an SR OS router and a third-party router are using the interface-ful IP-VRF-to-IP-VRF with SBD IRB model. The two routers are attached to a VPRN for the same tenant, and those VPRNs are connected by R-VPLS-2, or SBD. Both routers exchange IP prefix routes with a non-zero gateway IP, which is the IP address of the SBD IRB. The SBD IRB MAC and IP are advertised in a MAC/IP route. On reception, the IP prefix route creates a route-table entry in the VPRN, where the gateway IP must be recursively resolved to the information provided by the MAC/IP route and installed in the ARP and FDB tables.

This model is detailed in EVPN for VXLAN in IRB backhaul R-VPLS services and IP prefixes. The following is an example of the configuration of SBD and VPRN, as shown in Interface-ful IP-VRF-to-IP-VRF with SBD IRB model.

A:node-2>config>service

vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
 evi 2
 ip-route-advertisement
 mpls bgp 1
 auto-bind-tunnel resolution any
 no shutdown

vprn 1 customer 1 name "vprn1" create
 route-distinguisher auto-rd
 interface "sbd" create
 address 192.168.0.1/16
 ipv6
   30::3/64
 exit
 vpls "sbd"

The interface-ful IP-VRF-to-IP-VRF with SBD IRB model is also supported for IPv6 prefixes. There are no configuration differences except the ability to configure an IPv6 address and interface.

Interface-ful refers to the use of a full IRB interface between the VPRN and the SBD. However, the SBD IRB is unnumbered in this model, which means no IP address is configured on it. In SR OS, an unnumbered SBD IRB is equivalent to an R-VPLS that is linked to a VPRN interface through an EVPN tunnel. See EVPN for VXLAN in EVPN tunnel R-VPLS services for more information.

The following figure shows an example of the interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model.

In the preceding figure, an SR OS router and a third-party router are using the interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model. The IP prefix routes are expected to have a zero gateway IP, and the MAC in the router's MAC extended community is used for the recursive resolution to a MAC/IP route.

The corresponding configuration of the VPRN and SBD in the example could be:

A:node-2>config>service

vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
evi 2
ip-route-advertisement
mpls bgp 1
auto-bind-tunnel resolution any
no shutdown

vprn 1 customer 1 create
 route-distinguisher auto-rd
 interface "sbd" create
  ipv6
  exit
 vpls "sbd" 
  evpn-tunnel ipv6-gateway-address mac

The evpn-tunnel command controls the use of the router's MAC extended community and the zero gateway IP in the IPv4-prefix route. For IPv6, the ipv6-gateway-address mac command allows the router to advertise the IPv6-prefix routes with a router's MAC extended community and zero gateway IP.

The interface-less model does not require a Supplementary Broadcast Domain (SBD) connecting the VPRNs for the tenant, and no recursive resolution is required upon receiving an IP prefix route. In other words, the next-hop of the IP prefix route is directly resolved to an EVPN tunnel without the need for any other route. The standard specification RFC 9136 supports two variants of this model that are not interoperable.

• EVPN IFL for Ethernet Network Virtualization Overlay (NVO) tunnels

  Ethernet NVO indicates that the EVPN packets contain an inner Ethernet header. This is the case for tunnels such as VXLAN

  In the Ethernet NVO option, the ingress PE uses the received router’s MAC extended community address (received along with the route type 5) as the inner destination MAC address for the EVPN packets sent to the prefix.

• EVPN IFL for IP NVO tunnels

  IP NVO indicates that the EVPN packets contain an inner IP packet, but without Ethernet header. This is similar to the IPVPN packets exchanged between PEs.

The following figure shows an example of the interface-less IP-VRF-to-IP-VRF model.

SR OS supports the interoperable Interface-less IP-VRF-to-IP-VRF Model for Ethernet NVO tunnels. In the preceding figure, this interoperable is shown on the left-most PE router. The following is the model implementation.

• There is no datapath difference between this model and the existing R-VPLS EVPN tunnel model or the model described in Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model.

• This model is enabled by configuring the config service vprn if vpls evpn-tunnel (with ipv6-gateway-address mac for IPv6) and bgp-evpn ip-route-advertisement commands. In addition, because the SBD IRB MAC/IP route is no longer needed, the config service vpls bgp-evpn no mac-advertisement command prevents the advertisement of the MAC/IP route.

• The following IP prefix routes are processed as follows.

  • On transmission, there is no change in the IP prefix route processing compared to the configuration of the Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model.

    • IPv4 or IPv6 prefix routes are advertised, based on the information in the route-table for IPv4 and IPv6, with GW-IP=0 and the corresponding MAC extended community.

    • If the bgp-evpn no mac-advertisement command is configured, no MAC/IP route is sent for the R-VPLS.

  • The received IPv4 or IPv6 prefix routes are processed as follows.

    • Upon receiving an IPv4 or IPv6 prefix route with a MAC extended community for the router, an internal MAC/IP route is generated with the encoded MAC and the RD, Ethernet tag, ESI, Label/VNI and next hop from the IP prefix route itself.

    • If no competing received MAC/IP routes exists for the same MAC, this IP prefix-derived MAC/IP route is selected and the MAC installed in the R-VPLS FDB with type "Evpn".

    • After the MAC is installed in FDB, there are no differences between this interoperable interface-less model and the interface-ful with unnumbered SBD IRB model. Therefore SR OS is compatible with the received IP prefix routes for both models.

The following is an example of a typical configuration of a PE's SBD and VPRN that work in interface-less model for IPv4 and IPv6.

A:node-2>config>service

vpls 2 customer 1 name "sbd" create
       allow-ip-int-bind
       exit
       bgp
       exit
       bgp-evpn
            evi 2
            no mac-advertisement
            ip-route-advertisement
            mpls bgp 1
                 auto-bind-tunnel resolution any
                 no shutdown
vprn 1 customer 1 create
       route-distinguisher auto-rd
       interface "sbd" create
            ipv6
            exit
            vpls "sbd"
                 evpn-tunnel ipv6-gateway-address mac

In addition to the Interface-ful and interoperable Interface-less models described in Interface-ful IP-VRF-to-IP-VRF with SBD IRB model, Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model, and Interoperable interface-less IP-VRF-to-IP-VRF model (Ethernet encapsulation) sections, SR OS also supports the Interface-less Model (EVPN IFL) with IP encapsulation for MPLS tunnels. The RFC 9136 standard specification refers to this model as the EVPN IFL model for IP NVO tunnels.

Compared to the Ethernet NVO model in which the ingress PE pushes an inner Ethernet header, the IP packet in this EVPN IFL model is directly encapsulated with an EVPN service label and the transport labels.

The following figure shows the EVPN IFL model with IP encapsulation for MPLS tunnels.

EVPN IFL uses EVPN IP Prefix routes to exchange prefixes between PEs without the need for an R-VPLS service, termed Supplementary Broadcast Domain (SBD) in the standards, and any destination MAC lookup. The EVPN IFL uses the same datapath that is used for IP-VPN services in the VPRN.

In the example shown in Interface-less IP-VRF-to-IP-VRF model for IP encapsulation in MPLS tunnels, the following applies.

• PE2 advertises IP Prefix 20.0/24 (shorthand for 20.0.0.0/24) in an EVPN IP Prefix route that no longer contains a router's MAC extended community. In step 1, per usual processing, arriving frames with IP destination 20.0.0.1 on PE1's R-VPLS-1 are processed for a route lookup on VPRN-1.

• In step 2, in contrast to and as opposed to the other EVPN Layer 3 models, the lookup yields a route-table entry that does not point at an SBD R-VPLS, but points instead to an MPLS tunnel terminated on PE2. PE1 then pushes the EVPN service label received on the IP Prefix route at the top of the IP packet, and the packet is sent on the wire without an inner Ethernet header.

• In step 3, the MPLS tunnel is terminated on PE2 and the EVPN label identifies the VPRN-1 service for a route lookup.

• In Step 4, the processing corresponds to the regular R-VPLS forwarding that occurs in the other EVPN Layer 3 models.

Use the vprn>bgp-evpn>mpls context to configure a VPRN service for EVPN IFL with MPLS encapsulation. This context which is similar to the existing contexts in VPLS and Epipe services, enables the use of EVPN IFL in the VPRN service. When this context is enabled, no R-VPLS with evpn-tunnel should be added to the VPRN; that is, the user must ensure that no SBD is configured. For example, in Interface-less IP-VRF-to-IP-VRF model for IP encapsulation in MPLS tunnels, PE1 and PE2 VPRN-1 services are configured as follows.

[ex:configure service vprn "vprn-1"]
A:admin@PE1# info
    admin-state enable
    ecmp 2
    bgp-evpn {
        mpls 1 {
            admin-state enable
            route-distinguisher "192.0.2.1:12"
            vrf-target {
                community "target:64500:2"
            }
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    interface "irb-1" {
        ipv4 {
            primary {
                address 10.0.0.254
                prefix-length 24
            }
        }
        vpls "r-vpls-1" {
        }
    }

[ex:configure service vprn "vprn-1"]
A:admin@PE2# info
    admin-state enable
    ecmp 2
    bgp-evpn {
        mpls 1 {
            admin-state enable
            route-distinguisher "192.0.2.2:21"
            vrf-target {
                community "target:64500:2"
            }
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    interface "irb-2" {
        ipv4 {
            primary {
                address 20.0.0.254
                prefix-length 24
            }
        }
        vpls "r-vpls-1" {
        }
    }

SR OS supports the symmetric IRB model specified in RFC 9135, by which, when enabled, the router advertises the learned MAC and IP addresses of a host in an EVPN MAC/IP Advertisement route. This route contains not only the label and route target of the R-VPLS service where the host is learned, but also the label and the route target of the VPRN attached to the R-VPLS service. This is possible because the MAC/IP Advertisement route contains two labels in the NLRI: label-1 (for Layer 2 purposes) and label-2 (for Layer 3 purposes).

On reception, the remote router receives the EVPN MAC/IP Advertisement route and programs the IP address as a host route (/32 or /128) in the VPRN route table. Although host routes can be advertised in IP-Prefix routes, the symmetric IRB model allows the advertisement of the MAC and host IP information in a single NLRI. The following figure shows this model.

configure service interface vpls evpn arp advertise interface-less-routing bgp-evpn-instance
configure service interface vpls evpn nd advertise interface-less-routing bgp-evpn-instance

show router 200 route-table 10.0.0.202/32

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
10.0.0.202/32                                 Remote  EVPN-IFL* 00h04m11s  170
       2001:db8::2 (tunneled)                                       10
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
* indicates that the corresponding row element may have been truncated.

show router 200 route-table 10.0.0.202/32 extensive

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix             : 10.0.0.202/32
  Protocol              : EVPN-IFL-HOST
  Age                   : 00h04m18s
  Preference            : 170
  Indirect Next-Hop     : 2001:db8::2
    Label               : 524280
    VPN Next-Hop Index  : 10
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : N/A
    Resolving Next-Hop  : 2001:db8::2 (LDP tunnel)
      Metric            : 10
      ECMP-Weight       : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================

The following figure shows an example of a host connected to a source PE, PE1, that moved to the target, PE2. The figure shows the expected configuration on the VPRN interface, where R-VPLS 1 is attached (for both PE1 and PE2). PE1 and PE2 are configured with an anycast gateway, that is, a VRRP passive instance with the same backup MAC and IP in both PEs.

In this initial phase:

• PE1 learns Host-1 IP to MAC (10.1-M1) in the ARP table and generates a host route (RT5) for 10.1/32 because Host-1 is locally connected to PE1. In particular:

  • arp-learn-unsolicited triggers the learning of 10.1-M1 upon receiving a GARP from Host-1 or any other ARP

  • arp-proactive-refresh triggers the refresh of the host-1 ARP entry 30 seconds before the entry ages out

  • local-proxy-arp ensures that PE1 replies to any received ARP request on behalf of other hosts in the R-VPLS

  • arp-host-route populate dynamic ensures that only the dynamically learned ARP entries create a host route, for example, 10.1

  • no flood-garp-and-unknown-req suppresses ARP flooding (from the CPM) within the R-VPLS1 context, and because ARP entries are synchronized via EVPN, reduces significantly the unnecessary ARP flooding

  • advertise dynamic triggers the advertisement of MAC/IP routes for the dynamic ARP entries, including the IP and MAC addresses, for example, 10.1-M1; a MAC/IP route for M1-only that has been previously advertised as M1 is learned on the FDB as local or dynamic

• PE2 learns Host-1 10.1-M1 in the ARP and FDB tables as EVPN type. PE2 must not learn 10.1-M1 as dynamic so that PE2 is prevented from advertising an RT5 for 10.1/32. If PE2 advertises 10.1/32, PE3 could select PE2 as the next hop to reach Host-1, creating an undesired hair-pinning forwarding behavior. PE2 is expected to have the same configuration as PE1, including the following commands, as well as those described for PE1:

  • no learn-dynamic prevents PE2 from learning ARP entries from ARP traffic received on an EVPN tunnel

  • populate dynamic, as in PE1, ensures PE2 only creates route-table ARP-ND host routes for dynamic entries. Therefore, 10.1-M1 does not create a host route as long as it is learned via EVPN only.

The configuration described in this section and the cases described in Host mobility case 1, Host mobility case 2, and Host mobility case 3 apply to IPv4 hosts; however, the functionality is also supported for IPv6 hosts. The IPv6 configuration requires equivalent commands that use the prefix ‟nd-” instead of ‟arp-”. The only exception is the flood-garp-and-unknown-req command, which does not have an equivalent command for ND.

SR OS uses the Eth-CFM fault-propagation to support CE-to-CE fault propagation in EVPN-VPWS services. That is, upon detecting a CE failure, an EVPN-VPWS PE withdraws the corresponding Auto-Discovery per-EVI route, which then triggers a down maintenance endpoint (MEP) on the remote PE that signals the fault to the connected CE. In cases where the CE connected to EVPN-VPWS services does not support Eth-CFM, the fault can be propagated to the remote CE by using LAG standby-signaling, which can be LACP-based or simply power-off.

The following figure shows an example of link loss forwarding for EVPN-VPWS.

In this example, PE1 is configured as follows:

A:PE1>config>lag(1)# info 
----------------------------------------------
mode access
encap-type null 
port 1/1/1
port 1/1/2
standby-signaling power-off
monitor-oper-group "llf-1"
no shutdown
----------------------------------------------
*A:PE1>config>service>epipe# info
----------------------------------------------
bgp
exit
bgp-evpn
    evi 1
    local-attachment-circuit ac-1  
        eth-tag 1
        exit
    remote-attachment-circuit ac-2 
        eth-tag 2
        exit
    mpls bgp 1
        oper-group "llf-1"
        auto-bind-tunnel
            resolution any
        exit
        no shutdown
    exit
sap lag-1 create
no shutdown
exit
no shutdown

The following applies to the PE1 configuration.

• The EVPN Epipe service is configured on PE1 with a null LAG SAP and the oper-group ‟llf-1” under bgp-evpn>mpls. This is the only member of oper-group ‟llf-1”.

  Note: Do not configure the oper-group under config>service>epipe, because circular dependencies are created when the access SAPs go down as a result of the LAG monitor-oper-group command.

• The oper-group monitors the status of the BGP-EVPN instance in the Epipe service. The status of the BGP-EVPN instance is determined by the existence of an EVPN destination at the Epipe.

• The LAG, in access mode and encap-type null, is configured using the command monitor-oper-group ‟llf-1”.

  Note: The configure lag monitor-oper-group name command is only supported in access mode. Any encapsulation type can be used.

As shown in Link loss forwarding for EVPN-VPWS, upon failure on CE2, the following events occur.

• PE2 withdraws the EVPN route.

• The EVPN destination is removed in PE1 and oper-group ‟llf-1” also goes down.

• Because lag-1 is monitoring ‟llf-1”, the oper-group that is becoming inactive triggers standby signaling on the LAG; that is, power-off or LACP out-of-sync signaling to the CE1.

  Because the SAP or port is down as a result of the LAG monitoring of the oper-group, PE1 does not trigger an AD per-EVI route withdrawal, even if the SAP is brought operationally down.

• After CE2 recovers and PE2 re-advertises the AD per-EVI route, PE1 creates the EVPN destination and oper-group ‟llf-1” comes up. As a result, the monitoring LAG stops signaling standby and the LAG is brought up.

The following figure shows how black holes can be avoided when a PE becomes isolated from the core.

In this example, consider PE2 and PE1 are single-active multihomed to CE1. If PE2 loses all its core links, PE2 must somehow notify CE1 so that PE2 does not continue attracting traffic and so that PE1 can take over. This notification is achieved by using oper-groups under the BGP-EVPN instance in the service. The following is an example output of the PE2 configuration.

*[ex:configure service vpls ”evi1"]
A:admin@PE-2# info
    admin-state enable 
    bgp-evpn { 
        evi 1 
        mpls 1 { 
            admin-state enable 
            oper-group ‟evpn-mesh”
            auto-bind-tunnel { 
                resolution any 
            } 
        } 
    } 
    sap lag-1:351 { 
        monitor-oper-group ‟evpn-mesh”
            } 
*[ex:configure service oper-group ”evpn-mesh"]
A:admin@PE-2# info detail 
    hold-time { 
        up 4 
    } 

With the PE2 configuration and Core isolation blackhole avoidance example, the following steps occur.

1. PE2 loses all its core links, therefore, it removes its EVPN-MPLS destinations. This causes oper-group ‟evpn-mesh” to go down.

2. Because PE2 is the DF in the Ethernet Segment (ES) ES-1 and sap lag-1:351 is monitoring the oper-group, the SAP becomes operationally down. If ETH-CFM fault propagation is enabled on a down MEP configured on the SAP, CE1 is notified of the failure.

3. PE1 takes over as the DF, based on the withdrawal of the ES (and AD) routes from PE2, and CE1 begins sending traffic immediately to PE1 only, therefore, avoiding a traffic black hole.

Generally, when oper-groups are associated with EVIs, the following applies.

• The oper-group state is determined by the existence of at least one EVPN destination in the EVPN instance.

• The oper-group that is configured under a BGP EVPN instance cannot be configured under any other object (for example, SAP, SDP binding, and so on) of the same or different service.

• The status of an oper-group associated with an EVPN instance does not go down if all the EVPN destinations are operationally down because of a control word or MTU mismatch.

• The status of an oper-group associated with an EVPN instance goes down in the following cases:

  • the service admin-state is disabled (only for VPLS services, not for Epipes)

  • the BGP EVPN VXLAN or MPLS admin-state are disabled

  • there are no EVPN destinations associated with the instance

As described in EVPN for MPLS tunnels, EVPN single-active multihoming PEs that are elected as non-DF must notify their attached CEs so the CE does not send traffic to the non-DF PE. This can be performed on a per-service basis that is based on the ETH-CFM and fault-propagation. However, sometimes ETH-CFM is not supported in multihomed CEs and other notification mechanisms are needed, such as LACP standby or power-off. This scenario is shown in the following figure.

As shown in the preceding figure, the multihomed PEs are configured with multiple EVPN services that use ES-1. ES-1 and its associated LAG is configured as follows.

*[ex:configure lag 1]
A:admin@PE-2# info
    admin-state enable 
    standby-signaling {power-off|lacp} 
    monitor-oper-group ”DF-signal-1" 
    mode access 
    port 1/1/c2/1 { 
    }
<snip>
ex:configure service system bgp evpn]
A:admin@PE-2# info 
    ethernet-segment "ES-1" { 
        admin-state enable 
        esi 0x01010000000000000000 
        multi-homing-mode single-active 
        oper-group ‟DF-signal-1” 
        association { 
            lag 1 { 
            } 
<snip>

The following applies when the operational group is configured on the ES and monitored on the associated LAG.

• The operational group status is driven by the ES DF status (defined by the number of DF SAPs or oper-up SAPs owned by the ES).

• The operational group goes down if all the SAPs in the ES go down (this happens in PE2 in LACP standby signaling from the non-DF). The ES operational group goes up when at least one SAP in the ES goes up.

• As a result, if PE2 becomes non-DF on all the SAPs in the ES, they all go oper-down, including the ES-1 operational group.

• Because LAG-1 is monitoring the operational group, when its status goes down, LAG-1 signals LAG standby state to the CE. The standby signaling can be configured as LACP or power-off.

• The ES and AD routes for the ES are not withdrawn because the router recognizes that the LAG becomes standby as a result of the ES operational group.

If the single-active ES is associated with a port instead of a LAG, the config>port>monitor-oper-group DF-signal-1 command can be configured. In this case, the port monitors the ES operational group and the following rules apply:

• As in the case of the LAG, if the ES goes non-DF, its operational group also goes down.
• The port that is monitoring the ES operational group signals standby state by powering off the port itself.
• As in the case of the LAG, the ES and AD routes for the ES are not withdrawn because the router recognizes that the port is in standby state because of the ES operational group.

Operational groups cannot be assigned to ESs that are configured as virtual, all-active or service-carving mode auto.

The AC-influenced designated forwarder (AC-DF) election capability, as described in RFC 8584, is supported in SR OS. By default, the ac-df-capability command is set to the include option. This configuration addresses the need to consider EVPN Auto-discovery per EVI/ES (AD per EVI/ES) routes for a specific PE, which ensures that the PE is included on the candidate DF list.

Configuring ac-df-capability to exclude disables the AC-DF capability. When ac-df-capability exclude is configured on a specific ES, the presence or absence of the AD per EVI/ES routes from the ES peers does not modify the DF Election candidate list for the ES. The exclude option is recommended in ESs that use an oper-group, which is monitored by the access LAG, to signal standby lacp or power-off, as described in LAG or port standby signaling to the CE on non-DF EVPN PEs (single-active). All PE routers attached to the same ES must be configured consistently for the specific ac-df-capability.

When enabling existing VPLS features in an EVPN-MPLS enabled service, the following considerations apply.

• EVPN-MPLS is only supported in regular VPLS. Other VPLS types, such as m-vpls, are not supported with EVPN-MPLS.

• In general, no router-generated control packets are sent to the EVPN destination bindings, except for proxy-ARP/proxy-ND confirm messages and Eth-CFM for EVPN-MPLS.

• For xSTP and M-VPLS services:

  • xSTP can be configured in BGP-EVPN services. BPDUs are not sent over the EVPN bindings.

  • BGP-EVPN is blocked in M-VPLS services; however, a different M-VPLS service can manage a SAP or spoke-SDP in a BGP-EVPN-enabled service.

  • xSTP is not supported in BGP-EVPN services that use Ethernet segments for multihoming; an M-VPLS must not drive the state of a BGP-EVPN service that uses Ethernet segments.

• For BGP-EVPN-enabled VPLS services, mac-move can be used in SAPs/SDP-bindings; however, the MACs being learned through BGP-EVPN are not considered.

  Note: MAC duplication already provides a protection against MAC moves between EVPN and SAPs/SDP-bindings.

• The disable-learning command and other FDB-related tools work only for data plane learned MAC addresses.

• The mac-protect command cannot be used in conjunction with EVPN.

  Note: EVPN provides its own protection mechanism for static MAC addresses.

• MAC OAM tools (mac-ping, mac-trace, mac-populate, mac-purge, and cpe-ping) are not supported for BGP-EVPN services

• EVPN multihoming and BGP-MH can be enabled in the same VPLS service, as long as they are not enabled in the same SAP-SDP or spoke-SDP. There is no limitation on the number of BGP-MH sites supported per EVPN-MPLS service.

• SAPs/SDP-bindings that belong to a specified ES but are configured on non-BGP-EVPN-MPLS-enabled VPLS or Epipe services are kept down using the StandByForMHProtocol flag.

• CPE ping is not supported on EVPN services but it is supported in PBB-EVPN services (including I-VPLS and PBB-Epipe). CPE ping packets are not sent over EVPN destinations.

• Other features not supported in conjunction with BGP-EVPN:

  • subscriber management commands under service, SAP, and SDP-binding interfaces

  • BPDU translation

  • L2PT termination

  • MAC-pinning

• Other features not supported in conjunction with the bgp-evpn mpls command are:

  • SPB configuration and attributes

When enabling existing VPRN features on interfaces linked to EVPN-MPLS R-VPLS interfaces, consider that the following are not supported:

• the arp-populate and authentication-policy commands

• dynamic routing protocols such as IS-IS, RIP, and OSPF

When enabling existing IES features on interfaces linked to EVPN-MPLS R-VPLS interfaces, the following commands are not supported:

• if vpls evpn-tunnel

• bgp-evpn ip-route-advertisement

• arp-populate

• authentication-policy

• dynamic routing protocols such as IS-IS, RIP, and OSPF

A VPRN can receive and install routes for a specific BGP for a specific BGP owner. The routes may be re-exported in the context of the same VPRN and to the same BGP owner or a different one. For example, an EVPN-IFL route can be received from peer N, installed in VPRN 1, and re-exported to peer M using family VPN-IPv4.

When re-exporting BGP routes, the original BGP path attributes are preserved without any configuration in the following cases:

• EVPN-IFL or EVPN-IFL-HOST route re-exported into an IPVPN route, and an IPVPN route re-exported into an EVPN-IFL route

• EVPN-IFL or EVPN-IFL-HOST route re-exported into a BGP IP route (PE-CE), and a BGP IP routes re-exported into an EVPN-IFL route

• IPVPN route re-exported into a BGP IP route (PE-CE), and the other way around

• EVPN-IFL or EVPN-IFL-HOST, IPVPN or BGP IP routes re-exported into a route of the same owner. For example, EVPN-IFL to EVPN-IFL, when the allow-export-bgp-vpn command is configured. Received EVPN-IFL-HOST routes are re-exported into an EVPN-IFL route when allow-export-bgp-vpn command is enabled.

  Note: allow-export-bgp-vpn must never be used in a VPRN service with a route distinguisher that is used in other PEs attached to the same service. If the same route distinguisher is used in this case, constant route flaps occur.

BGP path attributes to or from EVPN-IFF are not preserved by default. If BGP Path Attribute propagation is required, the configure service system bgp-evpn ip-prefix-routes iff-attribute-uniform-propagation command must be configured. The following figure shows an example of BGP Path Attribute propagation from EVPN-IFF to the other BGP owners in the VPRN when the iff-attribute-uniform-propagation command is configured.

In the example in BGP path attribute propagation when iff-attribute-uniform-propagation is configured, DGW1 propagates the received LP and communities on an EVPN-IFF route, when advertising the same prefix into any type of BGP owner route, including VPN-IPv4/6, EVPN-IFL, EVPN-IFF, IPv4, or IPv6. If the iff-attribute-uniform-propagation command is not configured on DCW1, no BGP path attributes are propagated, but are re-originated instead. The propagation in the opposite direction follows the same rules; configuration of the iff-attribute-uniform-propagation command is required.

When propagating BGP path attributes, the following criteria are considered.

• The propagation is compliant with the uniform propagation described in draft-ietf-bess-evpn-ipvpn-interworking.

• The following extended communities are filtered or excluded when propagating attributes:

  • all extended communities of type 0x06 (EVPN type). In particular, all those that are supported by routes type 5:

    • MAC Mobility extended community (sub-type 0x00)

    • EVPN Router's MAC extended community (sub-type 0x03)

  • BGP encapsulation extended community

  • all Route Target extended communities

• The BGP Path Attribute propagation within the same owner is supported in the following cases:

  • EVPN-IFF to EVPN-IFF (route received on R-VPLS and advertised in a different R-VPLS context), assuming the iff-attribute-uniform-propagation command is configured

  • EVPN-IFL or EVPN-IFL-HOST to EVPN-IFL (route received on a VPRN and re-advertised based on the configuration of vprn>allow-export-bgp-vpn)

  • VPN-IPv4/6 to VPN-IPv4/6 (route received on a VPRN and re-advertised based on the configuration of vprn>allow-export-bgp-vpn)

• The propagation is supported for iBGP and eBGP as follows:

  • iBGP-only attributes can only be propagated to iBGP peers

  • non-transitive attributes are propagated based on existing rules

  • when peering an eBGP neighbor, the AS_PATH is prepended by the VPRN ASN

• If ECMP is enabled in the VPRN and multiple routes of the same BGP owner with different Route Distinguishers are installed in the route table, only the BGP path attributes of the best route are subject for propagation.

The SR OS has a full implementation of the D-PATH attribute as described in draft-ietf-bess-evpn-ipvpn-interworking.

D-PATH is composed of a sequence of domain segments (similar to AS_PATH). Each domain segment is graphically represented as shown in the following figure.

Where:

• Each domain segment is comprised of <domain_segment_length, domain_segment_value>, where the domain segment value is a sequence of one or more domains.

• Each domain is represented by <DOMAIN-ID:ISF_SAFI_TYPE>, where the newly added domain is added by a GW, is always prepended at the left of the existing last domain.

• The supported ISF_SAFI_TYPE values are:

  • 0 = Local ISF route
  • 1 = safi 1 (typically identifies PE-CE BGP domains)
  • 70 = evpn
  • 128 = safi 128 (IPVPN domains)

• Labeled unicast IP routes do not support D-PATH.

• The D-PATH attribute is only modified by a gateway and not by an ABR/ASBR or RR. A gateway is defined as a PE where a VPRN is instantiated, and that VPRN advertises or receives routes from multiple BGP owners (for example, EVPN-IFL and BGP-IPVPN) or multiple instances of the same owner (for example, VPRN with two BGP-IPVPN instances

Suppose a router receives prefix P in an EVPN-IFL instance with the following D-PATH from neighbor N.

+----------+------------+
|Seg Len=1 | 65000:1:128|
+----------+------------+

If the router imports the route in VPRN-1, BGP-EVPN SRv6 instance with domain 65000:2, the router readvertises the route to its BGP-IPVPN MPLS instance as follows:

+----------+----------+-----------+
|Seg Len=2 |65000:2:70|65000:1:128|
+----------+----------+-----------+

If the router imports the route in VPRN-1, BGP-EVPN SRv6 instance with domain 65000:3, the router readvertises the route to its BGP-EVPN MPLS instance as follows:

+----------+----------+-----------+
|Seg Len=2 |65000:3:70|65000:1:128|
+----------+----------+-----------+

If the router imports the route in VPRN-1, BGP-EVPN MPLS instance with domain 65000:4, the router readvertises the route to its PE-CE BGP neighbor as follows:

+----------+----------+-----------+
|Seg Len=2 |65000:4:70|65000:1:128|
+----------+----------+-----------+

When a BGP route of families that support D-PATH is received and must be imported in a VPRN, the following rules apply:

• All domain IDs included in the D-PATH are compared with the local domain IDs configured in the VPRN. The local domain IDs for the VPRN include a list of (up to four) domain IDs configured at the vprn or vprn bgp instance level, including the domain IDs in local attached R-VPLS instances.

• If one or more D-PATH domain IDs match any local domain IDs for the VPRN, the route is not installed in the VPRN’s route table.

• In the case where the IP-VPN or EVPN route matches the import route target in multiple VRFs, the D-PATH loop detection works per VPRN. For example, for each VPRN, BGP checks if the received domain IDs match any locally configured (maximum 4) domain IDs for that VPRN. A route may have a looped domain for one VPRN and not the other. In this case, BGP installs a route only in the VPRN route table that does not have a loop; the route is not installed in the VPRN that has the loop.

• A route that is not installed in any VPRN RTM (due to the domain ID matching any of the local domain IDs in the importing VPRNs) is still kept in the RIB-IN. The route is displayed in the show router bgp routes command with a DPath Loop VRFs field, indicating the VPRN in which the route is not installed due to a loop.

• Route target-based leaking between VPRNs and D-PATH loop detection is described in the following example.

  Consider an EVPN-IFL or EVPN-IFL-HOST route to prefix P imported in VPRN 20 (configured with domain 65000:20) is leaked into VPRN 30.

  When the route to prefix P is readvertised in the context of VPRN 30, which is enabled for BGP-IPVPN MPLS and BGP-EVPN MPLS, the readvertised BGP-IPVPN and BGP-EVPN routes have a D-PATH with a prepended domain 65000:20:0. That is, leaked routes are readvertised with the domain ID of the VPRN of origin and an ISF_SAFI_TYPE = 0, as described in draft-ietf-bess-evpn-ipvpn-interworking.

In the D-PATH example shown in the following figure, the different gateway PEs along the domains modify the D-PATH attribute by adding the source domain and family. If PE4 receives a route for the prefix with the domain of PE4 included in the D-PATH, PE4 does not install the route in order to avoid control plane loops.

In the D-PATH example shown in the following figure, DGW1 and DGW2 rely on the D-PATH attribute to automatically discard the prefixes received from the peer gateway in IPVPN and avoid loops by reinjecting the route back into the EVPN domain.

This section describes configuration examples for stitching IPVPN and EVPN-IFL domains and the propagation of BGP path attributes for EVPN-IFF.

BGP routing policies are supported for IP prefixes imported or exported through BGP-EVPN in R-VPLS services (EVPN-IFF routes) or VPRN services (EVPN-IFL routes).

When applying routing policies to control the distribution of prefixes between EVPN-IFF and IP-VPN (or EVPN-IFL), the user must consider that these owners are completely separate as far as BGP is concerned and when prefixes are imported in the VPRN routing table, the BGP attributes are lost to the other owner, unless the iff-attribute-uniform-propagation command is configured on the router.

If the iff-attribute-uniform-propagation command is disabled, the use of route tags allows the controlled distribution of prefixes across the two families.

The following figure shows an example of how VPN-IPv4 routes are imported into the RTM, and then passed to EVPN for its own process.

Policy tags can be used to match EVPN IP prefixes that were learned not only from BGP VPN-IPv4 but also from other routing protocols. The tag range supported for each protocol is different:

<tag>  : accepts in decimal or hex
        [0x1..0xFFFFFFFF]H (for OSPF and IS-IS)
        [0x1..0xFFFF]H (for RIP)
        [0x1..0xFF]H (for BGP)

The following figure shows an example of the reverse workflow: routes imported from EVPN and exported from RTM to BGP VPN-IPv4.

The preceding described behavior and the use of tags is also valid for VSI import and VSI export policies in the R-VPLS.

The following is a summary of the policy behavior for EVPN-IFF IP-prefixes when iff-attribute-uniform-propagation is disabled.

• For EVPN-IFF routes received and imported in RTM, policy entries (peer or VSI-import) match on communities or any of the following fields, and can add tags (as action):

  • communities, extended-communities or large communities

  • well-known communities

  • family EVPN

  • protocol bgp-vpn

  • prefix-lists

  • EVPN route type

  • BGP attributes (as-path, local-preference, next-hop)

• For exporting RTM to EVPN-IFF prefix routes, policy entries only match on tags, and based on this matching, add communities, accept, or reject. This applies to the peer level or on the VSI export level. Policy entries can also add tags for static routes, RIP, OSPF, IS-IS, BGP, and ARP-ND routes, which can then be matched on the BGP peer export policy, or on the VSI export policy for EVPN-IFF routes.

The following applies if the iff-attribute-uniform-propagation command is enabled.

For exporting RTM to EVPN-IFF prefix routes, in addition to matching on tags, matching path attributes on EVPN-IFF routes is supported in the following:

• vrf-export (when exporting the prefixes in VPN-IP or EVPN IFL or IP routes)

• vsi-export policies (when exporting the prefixes in EVPN-IFF routes)

• for non-BGP route-owners (RIP, OSPF, IS-IS, static, ARP-ND), there are no changes and the only match criterion in vsi-export for EVPN-IFF routes is tags

This section shows a configuration example for three 7705 SAR Gen 2 PEs, all the following assumptions are considered:

• PE-1 and PE-2 are multihomed to CE-12 that uses a LAG to get connected to the network. CE-12 is connected to LAG SAPs configured with all-active multihoming.

• PE-3 is a remote PE that performs aliasing for traffic destined for the CE-12

A:PE1# configure lag 1 
A:PE1>config>lag# info 
----------------------------------------------
        mode access
        encap-type dot1q
        port 1/1/2 
        lacp active administrative-key 1 system-id 00:00:00:00:69:72 
        no shutdown
----------------------------------------------
A:PE1>config>lag# /configure service system bgp-evpn 
A:PE1>config>service>system>bgp-evpn# info 
----------------------------------------------
                route-distinguisher 192.0.2.69:0
                exit
----------------------------------------------
A:PE1>config>service>system>bgp-evpn# /configure service vpls 1 
A:PE1>config>service>vpls# info 
----------------------------------------------
            bgp
            exit
            bgp-evpn
                cfm-mac-advertisement
                evi 1
                vxlan
                    shutdown
                exit
                mpls bgp 1
                    ingress-replication-bum-label
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap lag-1:1 create
              
            exit
            no shutdown
----------------------------------------------

A:PE2# configure lag 1 
A:PE2>config>lag# info 
----------------------------------------------
        mode access
        encap-type dot1q
        port 1/1/3 
        lacp active administrative-key 1 system-id 00:00:00:00:69:72 
        no shutdown
----------------------------------------------
A:PE2>config>lag# configure service vpls 1 
A:PE2>config>service>vpls# info 
----------------------------------------------
            bgp
            exit
            bgp-evpn
                cfm-mac-advertisement
                evi 1
                vxlan
                    shutdown
                exit
                mpls bgp 1
                    ingress-replication-bum-label
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap lag-1:1 create
            exit
            no shutdown
----------------------------------------------

*A:PE3>config>service>vpls# info 
----------------------------------------------
            bgp
            exit
            bgp-evpn
                cfm-mac-advertisement
                evi 1
                mpls bgp 1
                    ingress-replication-bum-label
                    ecmp 4
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:1 create
            exit
            spoke-sdp 4:13 create
                no shutdown
            exit
            no shutdown
----------------------------------------------

To use single-active multihoming on PE-1 and PE-2 instead of all-active multihoming:

• change the LAG configuration to multi-homing single-active

  The CE-12 is now configured with two different LAGs; therefore the key, system ID, and system priority values must be different on PE-1 and PE-2

No changes are needed at the service level on any of the three PEs.

A:PE1# configure lag 1 
A:PE1>config>lag# info 
----------------------------------------------
        mode access
        encap-type dot1q
        port 1/1/2 
        lacp active administrative-key 1 system-id 00:00:00:00:69:69 
        no shutdown
----------------------------------------------
A:PE1>config>lag# /configure service system bgp-evpn 
A:PE1>config>service>system>bgp-evpn# info 
----------------------------------------------
                route-distinguisher 192.0.2.69:0
                exit
----------------------------------------------
A:PE2# configure lag 1 
A:PE2>config>lag# info 
----------------------------------------------
        mode access
        encap-type dot1q
        port 1/1/3 
        lacp active administrative-key 1 system-id 00:00:00:00:72:72 
        no shutdown
----------------------------------------------
A:PE2>config>lag# /configure service system bgp-evpn 
A:PE2>config>service>system>bgp-evpn# info 
----------------------------------------------
                route-distinguisher 192.0.2.72:0
                exit
----------------------------------------------