EVPN

The DGW supports ECMP load balancing to reach the destination VTEP. Also, any intermediate core node in the Data Center should be able to provide further load balancing across ECMP paths because the source UDP port of each tunneled packet is derived from a hash of the customer inner packet. The following must be considered:

• ECMP for VXLAN is supported on VPLS services, but not for BUM traffic. Unicast spraying is based on the packet contents.

• ECMP for VXLAN on R-VPLS services is supported for VXLAN IPv6 tunnels.

• ECMP for VXLAN IPv4 tunnels on R-VPLS is only supported if the command configure service vpls allow-ip-int-bind vxlan-ipv4-tep-ecmp is enabled on the R-VPLS (as well as config>router>ecmp).

• ECMP for Layer 3 multicast traffic on R-VPLS services with EVPN-VXLAN destinations is only supported if the vpls allow-ip-int-bind ip-multicast-ecmp command is enabled (as well as config>router>ecmp).

• In the cases where ECMP is not supported (BUM traffic in VPLS and ECMP on R-VPLS if not enabled), each VXLAN binding is tied to a single (different) ECMP path, so that in a normal deployment with a reasonable number of remote VTEPs, there should be a fair distribution of the traffic across the paths. In other words, only per-VTEP load-balancing is supported, instead of per-flow load-balancing.

• LAG spraying based on the packet hash is supported in all the cases (VPLS unicast, VPLS BUM, and R-VPLS).

The following describes the behavior on the 7750 SR, 7450 ESS, and 7950 XRS with respect to VLAN tag handling for VXLAN VPLS services:

• Dot1q, QinQ, and null SAPs, as well as regular VLAN handling procedures at the WAN side, are supported on VXLAN VPLS services.

• No ‟vc-type vlan” like VXLAN VNI bindings are supported. Therefore, at the egress of the VXLAN network port, the router does not add any inner VLAN tag on top of the VXLAN encapsulation, and at the ingress network port, the router ignores any VLAN tag received and considers it as part of the payload.

For VXLAN VPLS services, the network port MTU must be at least 50 Bytes (54 Bytes if dot1q) greater than the Service-MTU to allow enough room for the VXLAN encapsulation.

The Service-MTU is only enforced on SAPs, (any SAP ingress packet with MTU greater than the service-mtu is discarded) and not on VXLAN termination (any VXLAN ingress packet makes it to the egress SAP regardless of the configured service-mtu).

VXLAN is a network port encapsulation; therefore, the QoS settings for VXLAN are controlled from the network QoS policies.

A new VXLAN troubleshooting tool, VXLAN Ping, is available to verify VXLAN VTEP connectivity. The VXLAN Ping command is available from interactive CLI and SNMP.

This tool allows the operator to specify a wide range of variables to influence how the packet is forwarded from the VTEP source to VTEP termination. The ping function requires the operator to specify a different test-id (equates to originator handle) for each active and outstanding test. The required local service identifier from which the test is launched determines the source IP (the system IP address) to use in the outer IP header of the packet. This IP address is encoded into the VXLAN header Source IP TLV. The service identifier also encodes the local VNI. The outer-ip-destination must equal the VTEP termination point on the remote node, and the dest-vni must be a valid VNI within the associated service on the remote node. The outer source IP address is automatically detected and inserted in the IP header of the packet. The outer source IP address uses the IPv4 system address by default.

If the VTEP is created using a non-system source IP address through the vxlan-src-vtep command, the outer source IP address uses the address specified by vxlan-src-vtep. The remainder of the variables are optional.

The VXLAN PDU is encapsulated in the appropriate transport header and forwarded within the overlay to the appropriate VTEP termination. The VXLAN router alert (RA) bit is set to prevent forwarding OAM PDU beyond the terminating VTEP. Because handling of the router alert bit was not defined in some early releases of VXLAN implementations, the VNI Informational bit (I-bit) is set to ‟0” for OAM packets. This indicates that the VNI is invalid, and the packet should not be forwarded. This safeguard can be overridden by including the i-flag-on option that sets the bit to ‟1”, valid VNI. Ensure that OAM frames meant to be contained to the VTEP are not forwarded beyond its endpoints.

The supporting VXLAN OAM ping draft includes a requirement to encode a reserved IEEE MAC address as the inner destination value. However, at the time of implementation, that IEEE MAC address had not been assigned. The inner IEEE MAC address defaults to 00:00:00:00:00:00, but may be changed using the inner-l2 option. Inner IEEE MAC addresses that are included with OAM packets are not learned in the local Layer 2 forwarding databases.

The echo responder terminates the VXLAN OAM frame, and takes the appropriate response action, and include relevant return codes. By default, the response is sent back using the IP network as an IPv4 UDP response. The operator can choose to override this default by changing the reply-mode to overlay. The overlay return mode forces the responder to use the VTEP connection representing the source IP and source VTEP. If a return overlay is not available, the echo response is dropped by the responder.

Support is included for:

• IPv4 VTEP

• Optional specification of the outer UDP Source, which helps downstream network elements along the path with ECMP to hash to flow to the same path

• Optional configuration of the inner IP information, which helps the operator test different equal paths where ECMP is deployed on the source. A test only validates a single path where ECMP functions are deployed. The inner IP information is processed by a hash function, and there is no guarantee that changing the IP information between tests selects different paths.

• Optional end system validation for a single L2 IEEE MAC address per test. This function checks the remote FDB for the configured IEEE MAC Address. Only one end system IEEE MAC Address can be configured per test.

• Reply mode UDP (default) or Overlay

• Optional additional padding can be added to each packet. There is an option that indicates how the responder should handle the pad TLV. By default, the padding is not reflected to the source. The operator can change this behavior by including the reflect-pad option. The reflect-pad option is not supported when the reply mode is set to UDP.

• Configurable send counts, intervals, times outs, and forwarding class

The VXLAN OAM PDU includes two timestamps. These timestamps are used to report forward direction delay. Unidirectional delay metrics require accurate time of day clock synchronization. Negative unidirectional delay values are reported as ‟0.000”. The round trip value includes the entire round trip time including the time that the remote peer takes to process that packet. These reported values may not be representative of network delay.

The following example commands and outputs show how the VXLAN Ping function can be used to validate connectivity. The echo output includes a new header to better describe the VXLAN ping packet headers and the various levels.

oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 
interval
0.1 send-count 10 

TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

! ! ! ! ! ! ! ! ! ! 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 1.097ms, avg = 2.195ms, max = 2.870ms, stddev = 0.735ms
round-trip-delay min = 1.468ms, avg = 1.693ms, max = 2.268ms, stddev = 0.210ms



oam vxlan-ping test-id 2 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 outer-
ip-source-udp 65000 outer-ip-ttl 64 inner-l2 d0:0d:1e:00:00:01 inner-ip-source
 192.168.1.2 inner-ip-destination 127.0.0.8 reply-mode overlay send-count 20 
interval
 1 timeout 3 padding 1000 reflect-pad fc nc profile out 

TestID 2, Service 1, DestVNI 2, ReplyMode overlay, IFlag Off, PadSize 1000, ReflectP
ad
 Yes, SendCount 20, Interval 1, Timeout 3
Outer: SourceIP 10.20.1.3, SourcePort 65000, DestIP 10.20.1.4, TTL 64, FC nc, Profil
e
 out
Inner: DestMAC d0:0d:1e:00:00:01, SourceIP 192.168.1.2, DestIP 127.0.0.8

===================================================================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
===================================================================================
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=1.733ms fwd
-time=0.302ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.549ms fwd
-time=1.386ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=3.243ms fwd
-time=0.643ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.551ms fwd
-time=2.350ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.644ms fwd
-time=1.080ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.670ms fwd
-time=1.307ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.636ms fwd
-time=0.490ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.649ms fwd
-time=0.005ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.401ms fwd
-time=0.685ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.634ms fwd
-time=0.373ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=11 ttl=255 rtt-time=1.559ms fwd
-time=0.679ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=12 ttl=255 rtt-time=1.666ms fwd
-time=0.880ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=13 ttl=255 rtt-time=1.629ms fwd
-time=0.669ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=14 ttl=255 rtt-time=1.280ms fwd
-time=1.029ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=15 ttl=255 rtt-time=1.458ms fwd
-time=0.268ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=16 ttl=255 rtt-time=1.659ms fwd
-time=0.786ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=17 ttl=255 rtt-time=1.636ms fwd
-time=1.071ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=18 ttl=255 rtt-time=1.568ms fwd
-time=2.129ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=19 ttl=255 rtt-time=1.657ms fwd
-time=1.326ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=20 ttl=255 rtt-time=1.762ms fwd
-time=1.335ms. rc=4

---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
20 packets transmitted, 20 packets received, 0.00% packet loss
   20 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 0.005ms, avg = 0.939ms, max = 2.350ms, stddev = 0.577ms
round-trip-delay min = 1.280ms, avg = 1.679ms, max = 3.243ms, stddev = 0.375ms



oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 interval 0.1 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

2 2 2 2 2 2 2 2 2 2 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present(1), 10 end-system not present(2)
forward-delay min = 0.467ms, avg = 0.979ms, max = 1.622ms, stddev = 0.504ms
round-trip-delay min = 1.501ms, avg = 1.597ms, max = 1.781ms, stddev = 0.088ms



oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

===================================================================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
mac=1 End System Present, mac=2 End System Not Present
===================================================================================

92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=2.883ms fwd
-time=4.196ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.596ms fwd
-time=1.536ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=1.698ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.687ms fwd
-time=1.766ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.679ms fwd
-time=0.799ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.678ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.709ms fwd
-time=0.031ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.757ms fwd
-time=1.441ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.613ms fwd
-time=2.570ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.631ms fwd
-time=2.130ms. rc=4 mac=2

---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present, 10 end-system not present
forward-delay min = 0.000ms, avg = 1.396ms, max = 4.196ms, stddev = 1.328ms
round-trip-delay min = 1.596ms, avg = 1.793ms, max = 2.883ms, stddev = 0.366ms

IPv4 and IPv6 multicast routing is supported in an EVPN-VXLAN VPRN and IES routed VPLS service through its IP interface when the source of the multicast stream is on one side of its IP interface and the receivers are on either side of the IP interface. For example, the source for multicast stream G1 could be on the IP side, sending to receivers on both other regular IP interfaces and the VPLS of the routed VPLS service, while the source for group G2 could be on the VPLS side sending to receivers on both the VPLS and IP side of the routed VPLS service. See IPv4 and IPv6 multicast routing support for more details.

The delivery of IP multicast in VXLAN services can be optimized with IGMP and MLD snooping. IGMP and MLD snooping are supported in EVPN-VXLAN VPLS services and in EVPN-VXLAN VPRN/IES R-VPLS services. When enabled, IGMP and MLD reports are snooped on SAPs or SDP bindings, but also on VXLAN bindings, to create or modify entries in the MFIB for the VPLS service.

When configuring IGMP and MLD snooping in EVPN-VXLAN VPLS services, consider the following:

• To enable IGMP snooping in the VPLS service on VXLAN, use the configure service vpls igmp-snooping no shutdown command.

• To enable MLD snooping in the VPLS service on VXLAN, use the configure service vpls mld-snooping no shutdown command.

• The VXLAN bindings only support basic IGMP/MLD snooping functionality. Features configurable under SAPs or SDP bindings are not available for VXLAN (VXLAN bindings are configured with the default values used for SAPs and SDP bindings). By default, a specified VXLAN binding only becomes a dynamic Mrouter when it receives IGMP or MLD queries and adds a specified multicast group to the MFIB when it receives an IGMP or MLD report for that group.

  Alternatively, it is possible to configure all VXLAN bindings for a particular VXLAN instance to be Mrouter ports using the configure service vpls vxlan igmp-snooping mrouter-port and configure service vpls vxlan mld-snooping mrouter-port commands.

• The show service id igmp-snooping, clear service id igmp-snooping, show service id mld-snooping, and clear service id mld-snooping commands are also available for VXLAN bindings.

  Note: MLD snooping uses MAC-based forwarding. See MAC-based IPv6 multicast forwarding for more details.

  The following CLI commands show how the system displays IGMP snooping information and statistics on VXLAN bindings (the equivalent MLD output is similar).

*A:PE1# show service id 1 igmp-snooping port-db vxlan vtep  192.0.2.72 vni 1 detail 
===============================================================================
IGMP Snooping VXLAN 192.0.2.72/1 Port-DB for service 1
===============================================================================
-------------------------------------------------------------------------------
IGMP Group 239.0.0.1
-------------------------------------------------------------------------------
Mode             : exclude              Type             : dynamic
Up Time          : 0d 19:07:05          Expires          : 137s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
No sources.
-------------------------------------------------------------------------------
IGMP Group 239.0.0.2
-------------------------------------------------------------------------------
Mode             : include              Type             : dynamic
Up Time          : 0d 19:06:39          Expires          : 0s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
10.0.0.232      0d 19:06:39  137s     dynamic  Fwd              
-------------------------------------------------------------------------------
Number of groups: 2
===============================================================================

*A:PE1# show service id 1 igmp-snooping 
statistics vxlan vtep  192.0.2.72 vni 1        
 
===============================================================================
IGMP Snooping Statistics for VXLAN 192.0.2.72/1 (service 1)
===============================================================================
Message Type            Received      Transmitted   Forwarded
-------------------------------------------------------------------------------
General Queries         0             0             556
Group Queries           0             0             0
Group-Source Queries    0             0             0
V1 Reports              0             0             0
V2 Reports              0             0             0
V3 Reports              553           0             0
V2 Leaves               0             0             0
Unknown Type            0             N/A           0
-------------------------------------------------------------------------------
Drop Statistics
-------------------------------------------------------------------------------
Bad Length               : 0
Bad IP Checksum          : 0
Bad IGMP Checksum        : 0
Bad Encoding             : 0
No Router Alert          : 0
Zero Source IP           : 0
Wrong Version            : 0
Lcl-Scope Packets        : 0
Rsvd-Scope Packets       : 0
 
Send Query Cfg Drops     : 0
Import Policy Drops      : 0
Exceeded Max Num Groups  : 0
Exceeded Max Num Sources : 0
Exceeded Max Num Grp Srcs: 0
MCAC Policy Drops        : 0
===============================================================================
*A:PE1# show service id 1 mfib 
===============================================================================
Multicast FIB, Service 1
===============================================================================
Source Address  Group Address         SAP or SDP Id               Svc Id   Fwd/Blk
-------------------------------------------------------------------------------
*               *                     sap:1/1/1:1              Local    Fwd
*               239.0.0.1             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
10.0.0.232      239.0.0.2             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
-------------------------------------------------------------------------------
Number of entries: 3
===============================================================================

PIM snooping for IPv4 and IPv6 are supported in an EVPN-EVPN-VXLAN VPLS or R-VPLS service (with the R-VPLS attached to a VPRN or IES service). The snooping operation is similar to that within a VPLS service (see PIM snooping for VPLS) and supports both PIM snooping and PIM proxy modes.

PIM snooping for IPv4 is enabled using the configure service vpls pim-snooping command.

PIM snooping for IPv6 is enabled using the configure service vpls pim-snooping no ipv6-multicast-disable command.

When using PIM snooping for IPv6, the default forwarding is MAC-based with optional support for SG-based (see IPv6 multicast forwarding). SG-based forwarding requires FP3- or higher-based hardware.

It is not possible to configure max-num-groups for VXLAN bindings.

By default, the system IP address is used to terminate and generate VXLAN traffic. The following configuration example shows an Epipe service that supports static VXLAN termination:

config service epipe 1 name "epipe1" customer 1 create
  sap 1/1/1:1 create
  exit
  vxlan vni 100 create
    egr-vtep 192.0.2.1
      oper-group op-grp-1
    exit
  no shutdown

Where:

• vxlan vni vni create specifies the ingress VNI the router uses to identify packets for the service. The following considerations apply:

  • In services that use EVPN, the configured VNI is only used as the ingress VNI to identify packets that belong to the service. Egress VNIs are learned from the BGP EVPN. In the case of Static VXLAN, the configured VNI is also used as egress VNI (because there is no BGP EVPN control plane).

  • The configured VNI is unique in the system, and as a result, it can only be configured in one service (VPLS or Epipe).

• egr-vtep ip-address specifies the remote VTEP the router uses when encapsulating frames into VXLAN packets. The following consideration apply:

  • When the PE receives VXLAN packets, the source VTEP is not checked against the configured egress VTEP.

  • The ip-address must be present in the global routing table so that the VXLAN destination is operationally up.

• The oper-group may be added under egr-vtep. The expected behavior for the operational group and service status is as follows:

  • If the egr-vtep entry is not present in the routing table, the VXLAN destination (in the show service id vxlan command) and the provisioned operational group under egr-vtep enters into the operationally down state.

  • If the Epipe SAP goes down, the service goes down, but it is not affected if the VXLAN destination goes down.

  • If the service is admin shutdown, then in addition to the SAP, the VXLAN destination and the oper-group also enters the operationally down state.

  Note: The operational group configured under egr-vtep cannot be monitored on the SAP of the Epipe where it is configured.

The following features are not supported by Epipe services with VXLAN destinations:

• per-service hashing

• SDP-binds

• PBB context

• BGP-VPWS

• spoke SDP-FEC

• PW-port

VXLAN instances in VPLS and R-VPLS can be configured with egress VTEPs. This is referred as static vxlan-instances. The following configuration example shows a VPLS service that supports a static vxlan-instance:

config service vpls 1 name "vpls-1" customer 1 create
  sap 1/1/1:1 create
  exit
  vxlan instance 1 vni 100 create
    source-vtep-security
    no disable-aging /* default: disable-aging    
    no disable-learning  /* default: disable-learning
    no discard-unknown-source
    no max-nbr-mac-addr <table-size>
    restrict-protected-src discard-frame
    egr-vtep 192.0.2.1 create
    exit
    egr-vtep 192.0.2.2 create
    exit
  vxlan instance 2 vni 101 create
    egr-vtep 192.0.2.3 create
    exit

 vxlan instance 2 vni 101 create
    egr-vtep 192.0.2.3 create
    exit
  no shutdown

Specifically the following can be stated:

• Each VPLS service can have up to two static VXLAN instances. Each instance is an implicit split-horizon-group, and up to 255 static VXLAN binds are supported in total, shared between the two VXLAN instances.

• Single VXLAN instance VPLS services with static VXLAN are supported along with SAPs and SDP bindings. Therefore:

  • VNIs configured in static VXLAN instances are ‟symmetric”, that is, the same ingress and egress VNIs are used for VXLAN packets using that instance. Note that asymmetric VNIs are actually possible in EVPN VXLAN instances.

  • The addresses can be IPv4 or IPv6 (but not a mix within the same service).

  • A specified VXLAN instance can be configured with static egress VTEPs, or be associated with BGP EVPN, but the same instance cannot be configured to support both static and BGP-EVPN based VXLAN bindings.

• Up to two VXLAN instances are supported per VPLS (up to two).

  • When two VXLAN instances are configured in the same VPLS service, any combination of static and BGP-EVPN enabled instances are supported. That is, the two VXLAN instances can be static, or BGP-EVPN enabled, or one of each type.

  • When a service is configured with EVPN and there is a static BGP-EVPN instance in the same service, the user must configure restrict-protected-src discard-frame along with no disable-learning in the static BGP-EVPN instance, service>vpls>vxlan.

• MAC addresses are learned also on the VXLAN bindings of the static VXLAN instance. Therefore, they are shown in the FDB commands. Note that disable-learning and disable-aging are by default enabled in static vxlan-instance.

  • The learned MAC addresses are subject to the remote-age, and not the local-age (only MACs learned on SAPs use the local-age setting).

  • MAC addresses are learned on a VTEP as long as no disable-learning is configured, and the VXLAN VTEP is present in the base route table. When the VTEP disappears from the route table, the associated MACs are flushed.

• The vpls vxlan source-vtep-security command can be configured per VXLAN instance on VPLS services. When enabled, the router performs an IPv4 source-vtep lookup to discover if the VXLAN packet comes from a trusted VTEP. If not, the router discards the frame. If the lookup yields a trusted source VTEP, then the frame is accepted.

  • A trusted VTEP is an egress VTEP that has been statically configured, or dynamically learned (through EVPN) in any service, Epipe or VPLS

  • The command show service vxlan shows the list of trusted VTEPs in the router.

  • The command source-vtep-security works for static VXLAN instances or BGP-EVPN enabled VXLAN instances, but only for IPv4 VTEPs.

  • The command is mutually exclusive with assisted-replication (replicator or leaf) in the VNI instance. AR can still be configured in a different instance.

Static VXLAN instances can use non-system IPv4/IPv6 termination.

The EVPN-VXLAN service is designed around the current VPLS objects and the additional VXLAN construct.

Layer 2 DC PE with VPLS to the WAN shows a DC with a Layer 2 service that carries the traffic for a tenant who wants to extend a subnet beyond the DC. The DC PE function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where a VPLS instance exists for that particular tenant. Within the DC, the tenant has VPLS instances in all the Network Virtualization Edge (NVE) devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The VPLS instances in the redundant DGW and the DC NVEs are connected by VXLAN bindings. BGP-EVPN provides the required control plane for such VXLAN connectivity.

The DGW routers are configured with a VPLS per tenant that provides the VXLAN connectivity to the Nuage VPLS instances. On the router, each tenant VPLS instance is configured with:

• The WAN-related parameters (SAPs, spoke SDPs, mesh-SDPs, BGP-AD, and so on).

• The BGP-EVPN and VXLAN (VNI) parameters. The following CLI output shows an example for an EVPN-VXLAN VPLS service.

*A:DGW1>config>service>vpls# info 
----------------------------------------------
            description "vxlan-service"
            vxlan instance 1 vni 1 create
            exit
            bgp
                route-distinguisher 65001:1
                route-target export target:65000:1 import target:65000:1
            exit
            bgp-evpn
                unknown-mac-route
                mac-advertisement 
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            sap 1/1/1:1 create
            exit
            no shutdown
----------------------------------------------

The bgp-evpn context specifies the encapsulation type (only vxlan is supported) to be used by EVPN and other parameters like the unknown-mac-route and mac-advertisement commands. These commands are typically configured in three different ways:

• If the operator configures no unknown-mac-route and mac-advertisement (default option), the router advertises new learned MACs (on the SAPs or SDP bindings) or new conditional static MACs.

• If the operator configures unknown-mac-route and no mac-advertisement, the router only advertises an unknown-mac-route as long as the service is operationally up (if no BGP-MH site is configured in the service) or the router is the DF (if BGP-MH is configured in the service).

• If the operator configures unknown-mac-route and mac-advertisement, the router advertises new learned MACs, conditional static MACs, and the unknown-mac-route. The unknown-mac-route is only advertised under the preceding described conditions.

Other parameters related to EVPN or VXLAN are:

• MAC duplication parameters

• VXLAN VNI (defines the VNI that the router uses in the EVPN routes generated for the VPLS service)

After the VPLS is configured and operationally up, the router sends or receives inclusive multicast Ethernet Tag routes, and a full-mesh of VXLAN connections are automatically created. These VXLAN ‟auto-bindings” can be characterized as follows:

• The VXLAN auto-binding model is based on an IP-VPN-like design, where no SDPs or SDP binding objects are created by or visible to the user. The VXLAN auto-binds are composed of remote VTEPs and egress VNIs, and can be displayed with the following command:

  A:PE-1# show service id 40 vxlan destinations  
  ===============================================================================
  Egress VTEP, VNI
  ===============================================================================
  Instance    VTEP Address                            Egress VNI  Evpn/   Num.
  Mcast       Oper State                              L2 PBR     Static  MACs
  -------------------------------------------------------------------------------
  1           192.0.2.1                               40          evpn    0
  BUM         Up                                      No                 
  1           192.0.2.3                               40          evpn    1
  BUM         Up                                      No                 
  -------------------------------------------------------------------------------
  Number of Egress VTEP, VNI : 2
  -------------------------------------------------------------------------------
  ===============================================================================
   
  ===============================================================================
  BGP EVPN-VXLAN Ethernet Segment Dest
  ===============================================================================
  Instance  Eth SegId                       Num. Macs     Last Change
  -------------------------------------------------------------------------------
  No Matching Entries
  ===============================================================================
  

• The VXLAN bindings observe the VPLS split-horizon rule. This is performed automatically without the need for any split-horizon configuration.

• BGP Next-Hop Tracking for EVPN is fully supported. If the BGP next-hop for a specified received BGP EVPN route disappears from the routing table, the BGP route is not marked as ‟used” and the respective entry in show service id vxlan destinations is removed.

After the flooding domain is setup, the routers and DC NVEs start advertising MAC addresses, and the routers can learn MACs and install them in the FDB. Some considerations are the following:

• All the MAC addresses associated with remote VTEP/VNIs are always learned in the control plane by EVPN. Data plane learning on VXLAN auto-bindings is not supported.

• When unknown-mac-route is configured, it is generated when no (BGP-MH) site is configured, or a site is configured AND the site is DF in the PE.

  Note: The unknown-mac-route is not installed in the FDB (therefore, does not show up in the show service id svc-id fdb detail command).

• While the router can be configured with only one VNI (and signals a single VNI per VPLS), it can accept any VNI in the received EVPN routes as long as the route target is properly imported. The VTEPs and VNIs show up in the FDB associated with MAC addresses:

  A:PE65# show service id 1000 fdb detail 
  ===============================================================================
  Forwarding Database, Service 1000
  ===============================================================================
  ServId    MAC               Source-Identifier        Type     Last Change
                                                       Age      
  -------------------------------------------------------------------------------
  1000      00:00:00:00:00:01 vxlan-1:                 Evpn     10/05/13 23:25:57
                              192.0.2.63:1063                   
  1000      00:00:00:00:00:65 sap:1/1/1:1000           L/30     10/05/13 23:25:57
  1000      00:ca:ca:ca:ca:00 vxlan-1:                 EvpnS    10/04/13 17:35:43
                              192.0.2.63:1063                   
  -------------------------------------------------------------------------------
  No. of MAC Entries: 3
  -------------------------------------------------------------------------------
  Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
  ===============================================================================
  

Gateway IRB on the DC PE for an L2 EVPN/VXLAN DC shows a DC with a Layer 2 service that carries the traffic for a tenant who extends a subnet within the DC, while the DGW is the default gateway for all the hosts in the subnet. The DGW function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where an R-VPLS instance exists for that particular tenant. Within the DC, the tenant has VPLS instances in all the NVE devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The WAN connectivity is based on existing IP-VPN features.

In this model, the DGW routers are configured with a R-VPLS (bound to the VPRN that provides the WAN connectivity) per tenant that provides the VXLAN connectivity to the Nuage VPLS instances. This model provides inter-subnet forwarding for L2-only TORs and other L2 DC NVEs.

On the router:

• The VPRN is configured with an interface bound to the backhaul R-VPLS. That interface is a regular IP interface (IP address configured or possibly a Link Local Address if IPv6 is added).

• The VPRN can support other numbered interfaces to the WAN or even to the DC.

• The R-VPLS is configured with the BGP, BGP-EVPN and VXLAN (VNI) parameters.

The Nuage VSGs and NVEs use a regular VPLS service model with BGP EVPN and VXLAN parameters.

Consider the following:

• Route-type 2 routes with MACs and IPs are advertised. Some considerations about MAC+IP and ARP/ND entries are:

  • The 7750 SR advertises its IRB MAC+IP in a route type 2 route and possibly the VRRP vMAC+vIP if it runs VRRP and the 7750 SR is the active router. In both cases, the MACs are advertised as static MACs, therefore, protected by the receiving PEs.

  • If the 7750 SR VPRN interface is configured with one or more additional secondary IP addresses, they are all advertised in routes type 2, as static MACs.

  • The 7750 SR processes route-type 2 routes as usual, populating the FDB with the received MACs and the VPRN ARP/ND table with the MAC and IPs, respectively.

    Note: ND entries received from the EVPN are installed as Router entries. The ARP/ND entries coming from the EVPN are tagged as evpn.

  • When a VPLS containing proxy-ARP/proxy-ND entries is bound to a VPRN (allow-ip-int-bind) all the proxy-ARP/proxy-ND entries are moved to the VPRN ARP/ND table. ARP/ND entries are also moved to proxy-ARP/proxy-ND entries if the VPLS is unbound.

  • EVPN does not program EVPN-received ARP/ND entries if the receiving VPRN has no IP addresses for the same subnet. The entries are added when the IP address for the same subnet is added.

  • Static ARP/ND entries have precedence over dynamic and EVPN ARP/ND entries.

• VPRN interface binding to VPLS service brings down the VPRN interface operational status, if the VPRN interface MAC or the VRRP MAC matches a static-mac or OAM MAC configured in the associated VPLS service. If that is the case, a trap is generated.

• Redundancy is handled by VRRP. The active 7750 SR advertises vMAC and vIP, as discussed, including the MAC mobility extended community and the sticky bit.

EVPN-enabled R-VPLS services are also supported on IES interfaces.

The Extensible Messaging and Presence Protocol (XMPP) is an open technology for real-time communication using XML (Extensible Markup Language) as the base format for exchanging information. The XMPP provides a way to send small pieces of XML from one entity to another in close to real time.

In a Nuage DC, an XMPP ejabberd server has an interface to the Nuage VSD as well as the Nuage VSC/VSG and the 7750 SR, 7450 ESS, or 7950 XRS DGW.

Basic XMPP architecture shows the basic XMPP architecture in the data center. While a single XMPP server is represented in the diagram, XMPP allows for easy server clustering and performs message replication to the cluster. It is similar to how BGP can scale and replicate the messages through the use of route reflectors.

Also the VSD is represented as a single server, but a cluster of VSD servers (using the same data base) is a very common configuration in a DC.

In the Nuage solution, each XMPP client, including the 7750 SR, 7450 ESS, and 7950 XRS, is referred to with a JID (JabberID) in the following format: username@xmppserver.domain. The xmppserver.domain points to the XMPP Server.

To enable the XMPP interface on the 7750 SR, 7450 ESS, or 7950 XRS, the following command must be added to indicate to which XMPP server address the DGW has to register, as well as the router’s JID:

*A:PE-2# configure system xmpp server
  - server <xmpp-server-name> [domain-name <fqdn>] [username <user-name>]
    [password <password>] [create] [service-name <service-name>]
  - no server <xmpp-server-name>
  - server <xmpp-server-name> [domain-name <fqdn>] [username <user-name>]
    [password <password>] [create] [router <router-instance>]
 <xmpp-server-name>   : [32 chars max]
 <fqdn>               : [256 chars max]
 <user-name>          : [32 chars max]
 <password>           : [32 chars max]
 <create>             : keyword - mandatory while creating an entry.
 <router-instance>    : <router-name>|<vprn-svc-id>
                        router-name    - "Base"|"management"
                        vprn-svc-id    - [1..2147483647]
 <service-name>       : [64 chars max]
 [no] shutdown        - Administratively enable or disable XMPP server

Where:

• [domain-name <fqdn>] is the domain portion of the JID.

• <user-name> and <password> is the username:password portion of the JID of the router acting as an XMPP client. Plain/MD5/anonymous authentication is supported.

• The user can choose not to configure the username portion of the JID. In that case, an in-band registration is attempted, using the chassis MAC as username.

• The user has the option to try to establish an XMPP TCP session over a router instance by using the router router-instance command. The router name can be ‟Base”, ‟management”, or a specified VPRN service identifier.

• When the xmpp server is properly configured and no shutdown, the 7750 SR tries to establish a TCP session with the XMPP server through the management interface first. If it fails to establish communication, the 7750 SR uses an in-band communication and uses its system IP as source IP address. Shutdown does not remove the dynamic configs in all the services. No server removes all the dynamic configs in all the services.

• Only one xmpp server can be configured.

After the XMPP server is properly configured, the router can generate or receive XMPP stanza elements, such as presence and IQ (Information/Query) messages. IQ messages are used between the VSD and the router to request and receive configuration parameters. The status of the XMPP communication channel can be checked with the following command:

Dut# show system xmpp server "vsd1-hy" 

===============================================================================
XMPP Server Table
===============================================================================
XMPP FQDN          : vsd1-hy.alu.us
XMPP Admin User    : csproot
XMPP Oper User     : csproot
State Lst Chg Since: 0d 02:56:44        State              : Functional
Admin State        : Up                 Connection Mode    : outOfBand
Auth Type          : md5
IQ Tx.             : 47                 IQ Rx.             : 47
IQ Error           : 0                  IQ Timed Out       : 0
IQ Min. Rtt        : 0 ms               IQ Max. Rtt        : 180 ms
IQ Ack Rcvd.       : 47                 
Push Updates Rcvd  : 1                  VSD list Upd Rcvd  : 12
Msg Tx.            : 27                 Msg Rx.            : 27
Msg Ack. Rx.       : 27                 Msg Error          : 0
Msg Min. Rtt       : 0 ms               Msg Max. Rtt       : 180 ms
Sub Tx.            : 1                  UnSub Tx.          : 0
Msg Timed Out      : 0                  
 
===============================================================================

In addition to the XMPP server, the router must be configured with a VSD system-id that uniquely identifies the router in the VSD:

*B:Dut>config>system>vsd# info 
----------------------------------------------
                system-id "SR12U-46-PE"
---------------------------------------------- 

After the above configuration is complete, the router subscribes to a VSD XMPP PubSub node to discover the available VSD servers. Then, the router is discovered in the VSD UIs. On the router, the available VSD servers can be shown with the following command.

B:Dut#show system xmpp vsd 

===============================================================================
Virtual Services Directory Table
===============================================================================
Id User Name                    Uptime             Status
-------------------------------------------------------------------------------
1  cna@vsd1-hy.alu-srpm.us/nua* 0d 00:44:36        Available
-------------------------------------------------------------------------------
No. of VSD's: 1
===============================================================================
* indicates that the corresponding row element may have been truncated.
*B:Dut#show system xmpp vsd 1 

===============================================================================
VSD Server Table
===============================================================================
VSD User Name      : cna@vsd1-hy.alu-srpm.us/nuage
Uptime             : 0d 00:44:39        Status             : Available
Msg Tx.            : 16                 Msg Rx.            : 10
Msg Ack. Rx.       : 4                  Msg Error          : 6
Msg TimedOut       : 0                  Msg MinRtt         : 80 ms
Msg MaxRtt         : 240 ms             
 
===============================================================================

In the Static-Dynamic integration model, the DC and DGW management entities can be the same or different. The DGW operator provisions the required VPRN and VPLS services with all the parameters needed for the connectivity to the WAN. VSD only pushes the required parameters so that those WAN services can be attached to the existing DC domains.

WAN services attachment workflow shows the workflow for the attachment of the WAN services defined on the DGW to the DC domains.

The Static-Dynamic VSD integration model can be summarized in the steps shown in WAN services attachment workflow and described in the following procedure.

1. The WAN or SP (Service Provider) administrator (which can be also the DC or Cloud Service Provider administrator) provisions the WAN services with all the parameters required for the connectivity to the WAN. This configuration is performed through the regular management interfaces, for example, CLI or SNMP. In the example above, there are two services created by the SP:

   • VPRN 1

     This service is associated with vsd-domain Ent-1, which is a VRF-GRE domain.

   • VPLS 2

     This service is associated with vsd-domain Ent-2, which is an L2-DOMAIN.

   Note: The parameters between brackets ‟[..]” are not configured at this step. They are pushed by the VSD through XMPP.

2. The router communicates with the VSD through the XMPP channel and allows VSD to know about its presence and available domains, Ent-1 and Ent-2. In the VSD’s User Interface (UI), the router shows up as DGW with its System ID, personality (for example, router) and the available WAN resources, that is, vsd-domains Ent-1 and Ent-2.

3. At VSD, the Cloud Service Provider administrator assigns the available WAN resources to Enterprises defined in VSD. In this example, VRF-GRE Ent-1 is assigned to Enterprise-1 and L2-DOMAIN Ent-2 to Enterprise-2.

4. Each Enterprise administrator has visibility of their own assigned WAN resource and attaches it to an existing DC Domain, assuming that both the DC domain and WAN resource are compatible. For instance, a VRF-GRE domain can only be attached to an L3 domain in the DC that uses GRE as transport.

5. When the Enterprise administrator attaches the WAN resource to the DC domain, the VSD sends the required configuration parameters to the DGW through the XMPP channel:

   • In the case of the VRF-GRE domain, VSD only sends the vrf-target required for the service attachment to the DC domain.

   • In the case of the L2-DOMAIN, VSD send the route-target (in the service>bgp or vsi-import/export contexts) as well as the vxlan vni and the bgp-evpn vxlan no shutdown commands.

   WAN resources can also be detached from the DC domains.

In the Static-Dynamic integration model, VSD can only provision specific parameters in VPLS and, or VPRN services. When VSD and the DGW exchange XMPP messages for a specified service, they use vsd-domains to identify those services. A vsd-domain is a tag that is used by the 7750 SR, 7450 ESS, or 7950 XRS router and the VSD to correlate a configuration to a service. When redundant DC GWs are available, the vsd-domain for the same service can have the same or a different name in the two redundant DC GWs.

There are four different types of vsd-domains that can be configured in the router:

• L2-DOMAIN

  It is associated with a VPLS service in the router and in VSD, it is attached to an existing Nuage L2-DOMAIN. This type of domain is used for extending Layer 2 subnets to the WAN connected to the DGW.

• L2-DOMAIN-IRB

  It is associated with a R-VPLS service in the router and in VSD, it is attached to an existing Nuage L2-DOMAIN. In this case, the DGW is the default gateway for all the VMs and hosts in the Nuage L2-DOMAIN.

• VRF-GRE

  This domain type is associated with a VPRN service in the router that uses GRE tunnels and MP-BGP VPN-IPv4 to provide connectivity to the DC. In VSD, it is attached to an existing Nuage L3-DOMAIN, when GRE is configured as tunnel-type for L3-DOMAINs.

• VRF-VXLAN

  This domain type is associated with a router R-VPLS service (connected to a VPRN with an evpn-tunnel VPLS interface) that uses VXLAN tunnels and EVPN to provide connectivity to the DC. In VSD, it is attached to an existing Nuage L3-DOMAIN, when VXLAN is configured as the tunnel-type for L3-DOMAINs.

The domains are configured in the config>service# context and assigned to each service.

# configure service vsd domain 
  - domain <name> [type {l2-domain|vrf-gre|vrf-vxlan|l2-domain-irb}] [create]
  - no domain <name>
 <name>               : [32 chars max]
 <create>             : keyword
 [no] description     - Set VSD Domain Description
 [no] shutdown        - Administratively enable/disable the domain

In the Static-Dynamic VSD integration model, the VPLS/VPRN service, as well as most of the parameters, must be provisioned ‟statically” through usual procedures (CLI, SNMP, and so on). VSD ‟dynamically” sends the parameters that are required for the attachment of the VPLS/VPRN service to the L2/L3 domain in the Data Center. In the Fully-Dynamic VSD integration model, the entire VPLS/VPRN service configuration is dynamically driven from VSD and no static configuration is required. Through the existing XMPP interface, the VSD provides the 7750 SR, 7450 ESS, and 7950 XRS routers with a handful of parameters that are translated into a service configuration by a python-script. This python-script provides an intermediate abstraction layer between VSD and the router, translating the VSD data model into a 7750 SR, 7450 ESS, or 7950 XRS CLI data model.

In this Fully-Dynamic integration model, the DC and DGW management entities are usually the same. The DGW operator provisions the required VPRN and VPLS services with all the parameters needed for the connectivity to the WAN and the DC. VSD pushes the required parameters so that the router can create the service completely and get it attached to an existing DC domain.

The workflow of the Fully-Dynamic integration model is shown in Fully-Dynamic VSD integration model workflow.

The Fully-Dynamic VSD integration model can be summarized in the steps shown in Fully-Dynamic VSD integration model workflow and described in the following procedure:

1. The 7750 SR, 7450 ESS, or 7950 XRS administrator only needs to provision parameters required for connectivity to the VSD, a service-range, and configure the python script/policy in the system. Provisioning of service parameters is not required.

   The service-range defines the service identifiers to include for VSD provisioned services and, when configured, they are protected from CLI changes. The vsd-policy defines the script to be used:

   *A:PE1>config>python# info 
   ----------------------------------------------
           python-script "l2-domain_services" create
               primary-url "ftp://1.1.1.1/cf2/l2domain_service.py"
               no shutdown
           exit
           python-policy "py-l2" create
               description "Python script to create L2 domains"
               vsd script "l2-domain_services"
           exit
   ----------------------------------------------
   
   *A:PE1>config>service>vsd# info 
   ----------------------------------------------
     service-range 64000 to 65000
   ----------------------------------------------
   

   When the router boots up or the gateway configuration is changed, the router sends a message to the VSD indicating its capabilities:

   • System-ID

   • Name and gateway type

   The VSD uses this information to register the router on its list of router GWs.

   When registered, the VSD and router exchange messages where the VSD communicates its list of service-names and their domain-type to the router. Based on this list, the router sends an XMPP IQ message to request the configuration of a specified service.

   The 7750 SR, 7450 ESS, or 7950 XRS router periodically audits the VSD and request a ‟DIFF” list of Full-Dynamic VSD domains. The VSD keeps a ‟DIFF” list of domains, that contains the Fully-Dynamic domain names for which the VSD has not received an IQ request from the router for a long time.

   The 7750 SR, 7450 ESS, or 7950 XRS CLI user can also audit the VSD to get the DIFF list, or even the ‟FULL” list of all the domains in the VSD. The following command triggers this audit: tools perform service vsd fd-domain-sync {full | diff}.

2. Concurrently at the VSD, the Cloud Service Provider administrator assigns WAN resources to Enterprises defined in the VSD. In this example, a VRF-GRE domain is assigned to Enterprise-1.

3. Each Enterprise administrator has visibility of their own assigned WAN resource and attaches it to an existing DC Domain, assuming that both the DC domain and WAN resource are compatible. For instance, a VRF-GRE domain can only be attached to an L3 domain in the DC that uses GRE as transport.

4. When the Enterprise administrator attaches the service requested by the 7750 SR, 7450 ESS, or 7950 XRS router to the DC domain, the VSD sends the required configuration parameters for that service to the DGW through the XMPP channel in an IQ Service message, including:

   • service name

   • service type

   • domain type; which can be one of the following:

     • L2-DOMAIN

     • L2-DOMAIN-IRB

     • VRF-GRE

     • VRF-VXLAN

   Other required configuration parameters include:

   Configuration type

   static (for Static-Dynamic model) or Dynamic (for Fully-Dynamic model)

   Internal route-target (RT-i)

   used to export/import the BGP routes sent/received from/to the DC route-reflector

   External route-target (RT-e)

   used to export/import the BGP routes sent/received from/to the WAN route-reflector. The value can be the same as the RT-i.

   VNI (VXLAN Network Identifier)

   used to configure the EVPN-VXLAN VPLS service on the router (if the domain type is L2-DOMAIN, L2-DOMAIN-IRB, or VRF-VXLAN)

   Metadata

   a collection of 'opaque' <key=value> pairs including the rest of the service parameters required for the service configuration at the router

   Note: The keys or values do not need to follow any specific format. The python script interprets and translates them into the router data model.

   Python-policy

   used by the router to find the Python script that translates the VSD parameters into configuration.

5. When the 7750 SR, 7450 ESS, or 7950 XRS router receives the IQ Service message, it builds a string with all the parameters and passes it to the Python module. The Python module is responsible for creating and activating the service, and, therefore, provides connectivity between the tenant domain and the WAN.

   Note: The Python script cannot access all the CLI commands and nodes in the system. An allowlist of nodes and commands is built and Python only has access to those nodes/commands. The list can be displayed using the tools dump service vsd-services command-list command.

   In addition to the allowlist, the user can further restrict the allowed CLI nodes to the VSD by using a separate CLI user for the XMPP interface, and associating that user to a profile where the commands are limited. The CLI user for the XMPP interface is configurable:

   
   config>system>security>cli-script>authorization>
      vsd   
   [no] cli-user <username>
   

   When the system executes a python-script for VSD commands, the vsd cli-user profile is checked to allow the resulting commands. A single CLI user is supported for VSD, therefore, the operator can configure a single 'profile' to restrict (even further than the allowlist) the CLI commands that can be executed by the VSD Python scripts.

   No cli-user means that the system does not perform any authorization checks and the VSD scripts can access any commands that are supported in the allowlist.

An AR-R configuration is shown in the following example.

*A:PE-2>config>service>system>vxlan# info 
----------------------------------------------
            assisted-replication-ip 10.2.2.2
----------------------------------------------
*A:PE-2>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication replicator
            exit
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
<snip>
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with tunnel-type = AR, type (T) = AR-R, and tunnel-id = originating-ip = next-hop = assisted-replication-ip (IP address 10.2.2.2 in the preceding example). In addition to the AR route, the AR-R sends a regular IR route if ingress-repl-inc-mcast-advertisement is enabled.

The AR-R builds a flooding list composed of ACs (SAPs and SDP bindings) and VXLAN tunnels to remote nodes in the VPLS. All objects in the flooding list are broadcast/multicast (BM) and unknown unicast (U) capable. The following example output of the show service id vxlan command shows that the VXLAN destinations in the flooding list are tagged as ‟BUM”.

*A:PE-2# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: replicator
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
192.0.2.3                               4000        0           BUM   Up    No
192.0.2.5                               4000        0           BUM   Up    No
192.0.2.6                               4000        0           BUM   Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================

When the AR-R receives a BUM packet on an AC, the AR-R forwards the packet to its flooding list (including the local ACs and remote VTEPs).

When the AR-R receives a BM packet on a VXLAN tunnel, it checks the IP DA of the underlay IP header and performs the following BM packet processing.

• If the destination IP matches its AR-IP, the AR-R forwards the BM packet to its flooding list (ACs and VXLAN tunnels). The AR-R performs source suppression to ensure that the traffic is not sent back to the originating Leaf.

• If the destination IP matches its regular VXLAN termination IP (IR-IP), the AR-R skips all the VXLAN tunnels from the flooding list and only replicates to the local ACs. This is the default Ingress Replication (IR) behavior.

An AR-L is configured as shown in the following example.

A:PE-3>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication leaf replicator-activation-time 30
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
                mpls
                    shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:4000 create
                no shutdown
            exit
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with a tunnel-type = IR, type (T) = AR-L and tunnel-id = originating-ip = next-hop = IR-IP (IP address terminating VXLAN normally, either system-ip or vxlan-src-vtep address).

The AR-L builds a single flooding list per service but controlled by the BM and U flags. These flags are displayed in the following show service id vxlan command example output.

A:PE-3# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: leaf      Replicator-Activation-Time: 30
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
10.2.2.2                                 4000        0           BM    Up    No
10.4.4.4                                 4000        0           -     Up    No
192.0.2.2                               4000        0           U     Up    No
192.0.2.5                               4000        0           U     Up    No
192.0.2.6                               4000        0           U     Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 5
-------------------------------------------------------------------------------
===============================================================================

The AR-L creates the following VXLAN destinations when it receives and selects a Replicator-AR route or the Regular-IR routes:

• A VXLAN destination to each remote PE that sent an IR route. These bindings have the U flag set.

• A VXLAN destination to the selected AR-R. These bindings have only the BM flag set; the U flag is not set.

• The non-selected AR-Rs create a binding with flag ‟-” (in the CPM) that is displayed by the show service id vxlan command. Although the VXLAN destinations to non-selected AR-Rs do not carry any traffic, the destinations count against the total limit and must be considered when accounting for consumed VXLAN destinations in the router.

The BM traffic is only sent to the selected AR-R, whereas the U (unknown unicast) traffic is sent to all the destinations with the U flag.

The AR-L performs per-service load-balancing of the BM traffic when two or more AR-Rs exist in the same service. The AR Leaf creates a list of candidate PEs for each AR-R (ordered by IP and VNI; candidate 0 being the lowest IP and VNI). The replicator is selected out of a modulo function of the service-id and the number of replicators, as shown in the following example output.

A:PE-3# show service id 4000 vxlan assisted-replication replicator 
===============================================================================
Vxlan AR Replicator Candidates
===============================================================================
VTEP Address           Egress VNI     In Use  In Candidate List Pending Time
-------------------------------------------------------------------------------
10.2.2.2                4000           yes     yes               0
10.4.4.4                4000           no      yes               0
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================

A change in the number of Replicator-AR routes (for example, if a route is withdrawn or a new route appears) affects the result of the hashing, which may cause a different AR-R to be selected.

The following list summarizes other aspects of the AR-L behavior:

• When a Leaf receives a BM packet on an AC, it sends the packet to its flood list that includes access SAP or SDP bindings and VXLAN destinations with BM or BUM flags. If a single AR-R is selected, only a VXLAN destination includes the BM flags.

• Control plane-generated BM packets, such as ARP/ND (when proxy-ARP/ND is enabled) or Eth-CFM, follow the behavior of regular data plane BM packets.

• When a Leaf receives an unknown unicast packet on an AC, it sends the packet to the flood-list, skipping the AR destination because the U flag is set to 0. To avoid packet re-ordering, the unknown unicast packets do not go through the AR-R.

• When a Leaf receives a BUM packet on an overlay tunnel, it forwards the packet to the flood list, skipping the VXLAN tunnels (that is, the packet is sent to the local ACs and never to a VXLAN tunnel). This is the default IR behavior.

• When the last Replicator-AR route is withdrawn, the AR-L removes the AR destination from the flood list and falls back to ingress replication.

AR BM replication behavior for a BM packet shows the expected replication behavior for BM traffic when received at the access on an AR-R, AR-L, or RNVE router. Unknown unicast follows regular ingress replication behavior regardless of the role of the ingress node for the specific service.

The Assisted-Replication feature has the following limitations:

• The following features are not supported on the same service where the Assisted-Replication feature is enabled.

  • Aggregate QoS per VNI

  • VXLAN IPv6 transport

  • IGMP/MLD/PIM-snooping

• Assisted-Replication Leaf and Replicator functions are mutually exclusive within the same VPLS service.

• The Assisted-Replication feature is supported with IPv4 non-system-ip VXLAN termination. However, the configured assisted-replication-ip (AR-IP) must be different from the tunnel termination IP address.

• The AR-IP address must be a /32 loopback interface on the base router.

• The Assisted-Replication feature is only supported in EVPN-VXLAN services (VPLS with BGP-EVPN vxlan enabled). Although services with a combination of EVPN-MPLS and EVPN-VXLAN are supported, the Assisted-Replication configuration is only relevant to the VXLAN.

PBF to ESI function shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on VPLS services. In this example, the DC gateway, PE1, is connected to an L2-DOMAIN that exists in the DC and must redirect the traffic to the Service Function SF-1. The regular Layer 2 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

An operator must configure a PBF match/action filter policy entry in an IPv4 or MAC ingress access or network filter deployed on a VPLS interface using CLI/SNMP/NETCONF management interfaces. The PBF target is the first service function in the chain (SF-1) that is identified by an ESI.

In the example shown in PBF to ESI function, the PBF filter redirects the matching packets to ESI 0x01 in VPLS-1.

As soon as the redirection target is configured and associated with the vport connected to SF-1, the Nuage VSC (Virtual Services Controller, or the remote PE3 in the example) advertises the location of SF-1 via an Auto-Discovery Ethernet Tag route (route type 1) per-EVI. In this AD route, the ESI associated with SF-1 (ESI 0x01) is advertised along with the VTEP (PE3's IP) and VNI (VNI-1) identifying the vport where SF-1 is connected. PE1 sends all the frames matching the ingress filter to PE3's VTEP and VNI-1.

The following filter configuration shows an example of PBF rule redirecting all the frames to an ESI.

A:PE1>config>filter>mac-filter# info 
----------------------------------------------
            default-action forward
            entry 10 create
                action
                    forward esi ff:00:00:00:00:00:00:00:00:01 service-id 301
                exit
            exit

When the filter is properly applied to the VPLS service (VPLS-301 in this example), it shows 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported.

A:PE1# show filter mac 1 
===============================================================================
Mac Filter
===============================================================================
Filter Id   : 1                                Applied         : Yes
Scope       : Template                         Def. Action     : Forward
Entries     : 1                                Type            : normal
Description : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : Mac
-------------------------------------------------------------------------------
Entry       : 10                               FrameType       : Ethernet
Description : (Not Specified)
Log Id      : n/a                              
Src Mac     : Undefined
Dest Mac    : Undefined
Dot1p       : Undefined                        Ethertype       : Undefined
DSAP        : Undefined                        SSAP            : Undefined
Snap-pid    : Undefined                        ESnap-oui-zero  : Undefined
Match action: Forward (ESI) Active             
  ESI       : ff:00:00:00:00:00:00:00:00:01
  Svc Id    : 301                              
PBR Down Act: Forward (entry-default)          
Ing. Matches: 3 pkts
Egr. Matches: 0 pkts
===============================================================================

A:PE1# show service id 301 es-pbr 
===============================================================================
L2 ES PBR
===============================================================================
ESI                           Users      Status
                                         VTEP:VNI
-------------------------------------------------------------------------------
ff:00:00:00:00:00:00:00:00:01 1          Active
                                         192.0.2.72:7272
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
===============================================================================

Details of the received AD route that resolves the filter forwarding are shown in the following show router bgp routes command.

A:PE1# show router bgp routes evpn auto-
disc esi ff:00:00:00:00:00:00:00:00:01                                    
===============================================================================
 BGP Router ID:192.0.2.71       AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup
 Origin codes  : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------

u*>i  192.0.2.72:100      ff:00:00:00:00:00:00:00:00:01 192.0.2.72
      0                                                 VNI 7272

-------------------------------------------------------------------------------
Routes : 1
=============================================================

This AD route, when used for PBF redirection, is added to the list of EVPN-VXLAN bindings for the VPLS service and shown as 'L2 PBR' type:

A:PE1# show service id 301 vxlan 
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 301
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address           Egress VNI     Num. MACs    Mcast   Oper State   L2 PBR
-------------------------------------------------------------------------------
192.0.2.69             301            1            Yes     Up           No
192.0.2.72             301            1            Yes     Up           No
192.0.2.72             7272           0            No      Up           Yes
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================

If the AD route is withdrawn, the binding disappears and the filter is inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

PBR to ESI function shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on L2-DOMAIN-IRB domains. In this example, the DC gateway, PE1, is connected to an L2-DOMAIN-IRB that exists in the DC and must redirect the traffic to the Service Function SF-1 with IP address 10.10.10.1. The regular Layer 3 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

In this case, an operator must configure a PBR match/action filter policy entry in an IPv4 ingress access or network filter deployed on IES/VPRN interface using CLI, SNMP or NETCONF management interfaces. The PBR target identifies first service function in the chain (ESI 0x01 in PBR to ESI function, identifying where the Service Function is connected and the IPv4 address of the SF) and EVPN VXLAN egress interface on the PE (VPRN routing instance and R-VPLS interface name). The BGP control plane together with ESI PBR configuration are used to forward the matching packets to the next-hop in the EVPN-VXLAN data center chain (through resolution to a VNI and VTEP). If the BGP control plane information is not available, the packets matching the ESI PBR entry is, by default, forwarded using regular routing. Optionally, an operator can select to drop the packets when the ESI PBR target is not reachable.

The following filter configuration shows an example of a PBR rule redirecting all the matching packets to an ESI.

*A:PE1>config>filter>ip-filter# info 
----------------------------------------------
            default-action forward
            entry 10 create
                match 
                    dst-ip 10.10.10.253/32
                exit 
                action
                    forward esi ff:00:00:00:00:21:5f:00:df:e5 sf-ip 10.10.10.1 vas-
interface "evi-301" router 300
                exit
                pbr-down-action-override filter-default-action
            exit 
----------------------------------------------

In this use case, the following are required in addition to the ESI: the sf-ip (10.10.10.1 in the example above), router instance (300), and vas-interface.

The sf-ip is used by the system to know which inner MAC DA it has to use when sending the redirected packets to the SF. The SF-IP is resolved to the SF MAC following regular ARP procedures in EVPN-VXLAN.

The router instance may be the same as the one where the ingress filter is configured or may be different: for instance, the ingress PBR filter can be applied on an IES interface pointing at a VPRN router instances that is connected to the DC fabric.

The vas-interface refers to the R-VPLS interface name through which the SF can be found. The VPRN instance may have more than one R-VPLS interface, therefore, it is required to specify which R-VPLS interface to use.

When the filter is properly applied to the VPRN or IES service (VPRN-300 in this example), it shows 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported and the SF-IP resolved to a MAC address.

*A:PE1# show filter ip 1 

===============================================================================
IP Filter
===============================================================================
Filter Id    : 1                                Applied        : Yes
Scope        : Template                         Def. Action    : Forward
System filter: Unchained                        
Radius Ins Pt: n/a                              
CrCtl. Ins Pt: n/a                              
RadSh. Ins Pt: n/a                              
PccRl. Ins Pt: n/a                              
Entries      : 1                                
Description  : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry        : 10
Description  : (Not Specified)
Log Id       : n/a                              
Src. IP      : 0.0.0.0/0
Src. Port    : n/a
Dest. IP     : 10.16.0.253/32
Dest. Port   : n/a
Protocol     : Undefined                        Dscp           : Undefined
ICMP Type    : Undefined                        ICMP Code      : Undefined
Fragment     : Off                              Src Route Opt  : Off
Sampling     : Off                              Int. Sampling  : On
IP-Option    : 0/0                              Multiple Option: Off
TCP-syn      : Off                              TCP-ack        : Off
Option-pres  : Off                              
Egress PBR   : Undefined                        
Match action : Forward (ESI) Active
  ESI        : ff:00:00:00:00:21:5f:00:df:e5    
  SF IP      : 10.10.10.1
  VAS If name: evi-301                          
  Router     : 300                              
PBR Down Act : Forward (filter-default-action) Ing. Matches : 3 pkts (318 bytes)
Egr. Matches : 0 pkts
===============================================================================

*A:PE1# show service id 300 es-pbr 
===============================================================================
L3 ES PBR
===============================================================================
SF IP              ESI                                 Users Status
                   Interface                                 MAC
                                                             VTEP:VNI
-------------------------------------------------------------------------------
10.10.10.1         ff:00:00:00:00:21:5f:00:df:e5       1     Active
                   evi-301                                   d8:47:01:01:00:0a
                                                             192.0.2.71:7171
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
=================================================================================

In the FDB for the R-VPLS 301, the MAC address is associated with the VTEP and VNI specified by the AD route, and not by the MAC/IP route anymore. When a PBR filter with a forward action to an ESI and SF-IP (Service Function IP) exists, a MAC route is auto-created by the system and this route has higher priority that the remote MAC, or IP routes for the MAC (see BGP and EVPN route selection for EVPN routes).

The following shows that the AD route creates a new EVPN-VXLAN binding and the MAC address associated with the SF-IP uses that 'binding':

*A:PE1# show service id 301 vxlan 
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 301
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address           Egress VNI     Num. MACs    Mcast   Oper State   L2 PBR
-------------------------------------------------------------------------------
192.0.2.69             301            1            Yes     Up           No
192.0.2.71             301            0            Yes     Up           No
192.0.2.71             7171           1            No      Up           No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================
*A:PE1# show service id 301 fdb detail 
===============================================================================
Forwarding Database, Service 301
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
301       d8:45:ff:00:00:6a vxlan-1:                 EvpnS    06/15/15 21:55:27
                            192.0.2.69:301
301       d8:47:01:01:00:0a vxlan-1:                 EvpnS    06/15/15 22:32:56
                            192.0.2.71:7171
301       d8:48:ff:00:00:6a cpm                      Intf     06/15/15 21:54:12
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

For Layer 2, if the AD route is withdrawn or the SF-IP ARP not resolved, the filter is inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

EVPN MPLS, as described in EVPN for MPLS tunnels, uses ESI-labels to identify the BUM traffic sourced from a specified ES. The egress PE performs a label lookup to find the ESI label below the EVI label and to determine if a frame can be forwarded to a local ES. Because VXLAN does not support ESI-labels, or any MPLS label for that matter, the split-horizon filtering must be based on the tunnel source IP address. This also implies that the SAP-to-SAP forwarding rules must be changed when the SAPs belong to local ESs, irrespective of the DF state. This new forwarding is what RFC 8365 refers to as local bias. EVPN-VXLAN multihoming with local bias illustrates the local bias forwarding behavior.

Local bias is based on the following principles:

• Every PE knows the IP addresses associated with the other PEs with which it has shared multihomed ESs.

• When the PE receives a BUM frame from a VXLAN bind, it looks up the source IP address in the tunnel header and filters out the frame on all local interfaces connected to ESs that are shared with the ingress PE.

With this approach, the ingress PE must perform replication locally to all directly-attached ESs (regardless of the DF Election state) for all flooded traffic coming from the access interfaces. BUM frames received on any SAP are flooded to:

• local non-ES SAPs and non-ES SDP-binds

• local all-active ES SAPs (DF and NDF)

• local single-active ES SDP-binds and SAPs (DF only)

• EVPN-VXLAN destinations

As an example, in EVPN-VXLAN multihoming with local bias, PE2 receives BUM traffic from Host-3 and it forwards it to the remote PEs and the local ES SAP, even though the SAP is in NDF state.

The following rules apply to egress PE forwarding for EVPN-VXLAN services:

• The source VTEP is looked up for BUM frames received on EVPN-VXLAN.

• If the source VTEP matches one of the PEs with which the local PE shares both an ES and a VXLAN service:

  • the local PE is not forwarded to the shared ES local SAPs

  • the local PE forwards normally to ES SAPs unless they are in NDF state

• Because there is no multicast label or multicast B-MAC in VXLAN, the egress PE only identifies BUM traffic using the customer MAC DA; as a result, BM or unknown MAC DAs identify BUM traffic.

For example, in EVPN-VXLAN multihoming with local bias, PE3 receives BUM traffic on VXLAN. PE3 identifies the source VTEP as a PE with which two ESs are shared, therefore it does not forward the BUM frames to the two shared ESs. It forwards to the non-shared ES (Host-5) because it is in DF state. PE4 receives BUM traffic and forwards it based on normal rules because it does not share any ESs with PE2.

The following command can be used to check whether the local PE has enabled the local bias procedures for a specific ES:

A:PE-2# tools dump service system bgp-evpn ethernet-segment "ES-1" local-bias 
-------------------------------------------------------------------------------
[04/01/2019 08:45:08] Vxlan Local Bias Information 
----------------------------------------------------------------------+--------
Peer                                                                  | Enabled   
----------------------------------------------------------------------+--------
192.0.2.3                                                             | Yes       
-------------------------------------------------------------------------------

In EVPN MPLS networks, an ingress PE that uses ingress replication to flood unknown unicast traffic pushes a BUM MPLS label that is different from a unicast label. The egress PEs use this BUM label to identify such BUM traffic to apply DF filtering for All-Active multihomed sites. In PBB-EVPN, in addition to the multicast label, the egress PE can also rely on the multicast B-MAC DA to identify customer BUM traffic.

In VXLAN there are no BUM labels or any tunnel indication that can assist the egress PE in identifying the BUM traffic. As such, the egress PE must solely rely on the C-MAC destination address, which may create some transient issues that are depicted in EVPN-VXLAN multihoming and unknown unicast issues.

As shown in EVPN-VXLAN multihoming and unknown unicast issues, top diagram, in absence of the mentioned unknown unicast traffic indication there can be transient duplicate traffic to All-Active multihomed sites under the following condition: CE1’s MAC address is learned by the egress PEs (PE1 and PE2) and advertised to the ingress PE3; however, the MAC advertisement has not been received or processed by the ingress PE, resulting in the host MAC address to be unknown on the ingress PE3 but known on the egress PEs. Therefore, when a packet destined for CE1 address arrives on PE3, it floods it through ingress replication to PE1 or PE2 and, because CE1’s MAC is known to PE1 and PE2, multiple copies are sent to CE1.

Another issue is shown at the bottom of EVPN-VXLAN multihoming and unknown unicast issues. In this case, CE1’s MAC address is known on the ingress PE3 but unknown on PE1 and PE2. If PE3’s aliasing hashing picks up the path to the ES’ NDF, a black-hole occurs.

The above two issues are solved in MPLS, as unicast known and unknown frames are identified with different labels.

Finally, another issue is described in Blackhole created by a remote SAP shutdown. Under normal circumstances, when CE3 sends BUM traffic to PE3, the traffic is ‟local-biased” to PE3’s SAP3 even though it is NDF for the ES. The flooded traffic to PE2 is forwarded to CE2, but not to SAP2 because the local bias split-horizon filtering takes place.

The right side of the diagram in Blackhole created by a remote SAP shutdown shows an issue when SAP3 is manually shutdown. In this case, PE3 withdraws the AD per-EVI route corresponding to SAP3; however, this does not change the local bias filtering for SAP2 in PE2. Therefore, when CE3 sends BUM traffic, it can neither be forwarded to CE23 via local SAP3 nor can it be forwarded by PE2.

EVPN VXLAN multihoming is supported on VPLS and R-VPLS services when the PEs use non-system IPv4 or IPv6 termination, however, as with EVPN VPWS services, additional configuration steps are required.

• The configure service system bgp-evpn eth-seg es-orig-ip ip-address command must be configured with the non-system IPv4 or IPv6 address used for the EVPN-VXLAN service. This command modifies the originating-ip field in the ES routes advertised for the Ethernet Segment, and makes the system use this IP address when adding the local PE as DF candidate.

• The configure service system bgp-evpn eth-seg route-next-hop ip-address command must also be configured with the non-system IP address. This command changes the next-hop of the ES and AD per-ES routes to the configured address.

• Finally, the non-system IP address (in each of the PEs in the ES) must match in these three commands for the local PE to be considered suitable for DF election:

  • es-orig-ip ip-address

  • route-next-hop ip-address

  • vxlan-src-vtep ip-address

The 7750 SR, 7450 ESS, or 7950 XRS router SR OS EVPN implementation supports RFC 8560 so that EVPN-MPLS and VPLS can be integrated into the same network and within the same service. Because EVPN is not deployed in green-field networks, this feature is useful for the integration between both technologies and even for the migration of VPLS services to EVPN-MPLS.

The operation in services with BGP-VPLS and BGP-EVPN is equivalent to the one described above for BGP-AD and BGP-EVPN.

In a VPLS service to which multiple EVPN PEs and BGP-VPLS PEs are attached, single-active multihoming is supported on two or more of the EVPN PEs with no special considerations. All-active multihoming is not supported, because the traffic from the all-active multihomed CE could cause a MAC flip-flop effect on remote BGP-VPLS PEs, asymmetric flows, or other issues.

BGP-VPLS to EVPN integration and single-active MH illustrates a scenario with a single-active Ethernet-segment used in a service where EVPN PEs and BGP-VPLS are integrated.

Although other single-active examples are supported, in BGP-VPLS to EVPN integration and single-active MH, CE1 is connected to the EVPN PEs through a single LAG (lag-1). The LAG is associated with the Ethernet-segment 1 on PE1 and PE2, which is configured as single-active and with oper-group 1. PE1 and PE2 make use of lag>monitor-oper-group 1 so that the non-DF PE can signal the non-DF state to CE1 (in the form of LACP out-of-synch or power-off).

In addition to the BGP-VPLS routes sent for the service ve-id, the multihoming PEs in this case need to generate additional BGP-VPLS routes per Ethernet Segment (per VPLS service) for the purpose of MAC flush on the remote BGP-VPLS PEs in case of failure.

The sap>bgp-vpls-mh-veid number command should be configured on the SAPs that are part of an EVPN single-active Ethernet Segment, and allows the advertisement of L2VPN routes that indicate the state of the multihomed SAPs to the remote BGP-VPLS PEs. Upon a Designated Forwarder (DF) switchover, the F and D bits of the generated L2VPN routes for the SAP ve-id are updated so that the remote BGP-VPLS PEs can perform a mac-flush operation on the service and avoid blackholes.

As an example, in case of a failure on the Ethernet-segment sap on PE1, PE1 must indicate PE3 and PE4 the need to flush MAC addresses learned from PE1 (flush-all-from-me message). Otherwise, for example, PE3 continues sending traffic with MAC DA = CE1 to PE1, and PE1 blackholes the traffic.

In the BGP-VPLS to EVPN integration and single-active MH example:

• Both ES peers (PE1 and PE2) should be configured with the same ve-id for the ES SAP. However, this is not mandatory.

• In addition to the regular service ve-id L2VPN route, based on the sap>bgp-vpls-mh-ve-id configuration and upon BGP VPLS being enabled, the PE advertises an L2VPN route with the following fields:

  • ve-id = sap>bgp-vpls-mh-ve-id identifier

  • RD, RT, next hop and other attributes same as the service BGP VPLS route

  • L2VPN information extended community with the following flags:

    • D=0 if the SAP is oper-up or oper-down with a flag MHStandby (for example, the PE is non-DF in single-active MH)

    • D=0 also if there is an ES oper-group and the port is down because of the oper-group

    • D=1 if the SAP is oper-down with a different flag (for example, port-down or admin-down)

    • F (DF bit) =1 if the SAP is oper-up, F=0 otherwise

• Upon a failure on the access SAP, there are only mac-flush messages triggered in case the command bgp-vpls-mh-ve-id is configured in the failing SAP. In case it is configured with ve-id 1:

  • If the non-DF PE has a failure on the access SAP, PE2 sends an update with ve-id=1/D=1/F=0. This is an indication for PE3/PE4 that PE2's SAP is oper-down but it should not trigger a mac-flush on PE3/PE4.

  • If the DF PE has a failure on the SAP, PE1 advertises ve-id=1/D=1/F=0. Upon receiving this update, PE3 and PE4 flushes all their MACs associated with the PE1's spoke SDP. Note, that the failure on PE1, triggers an EVPN DF Election on PE2, which becomes DF and advertises ve-id=1/D=0/F=1. This message does not trigger any mac-flush procedures.

Other considerations:

• PE3/PE4 are SR OS or any third-party PEs that support the procedures in draft-ietf-bess-vpls-multihoming, so that BGP-VPLS mac-flush signaling is understood.

• PE1 and PE2 are expected to run an SR OS version that supports the sap>bgp-vpls-mh-veid number configuration on the multihomed SAPs. Otherwise, the mac-flush behavior would not work as expected.

• The procedures described above are also supported if the EVPN PEs use MC-LAG instead of an ES for the CE1 redundancy. In this case, the SAP ve-id route for the standby PE is sent as ve-id=1/D=1/F=0, whereas the active chassis advertises ve-id=1/D=0/F=1. A switchover triggers mac-flush on the remote PEs as described earlier.

• The L2VPN routes generated for the ES and SAPs with the sap bgp-vpls-mh-veid number command are decoded in the remote nodes as bgp-mh routes (because they do not have label information) in the show router bgp routes l2-vpn command and debug.

In a VPLS service, multiple BGP families and protocols can be enabled at the same time. When bgp-evpn is enabled, bgp-ad and bgp-mh are also supported. A single RD is used per service and not per BGP family or protocol.

The following rules apply:

• The VPLS RD is selected based on the following precedence:

  • Manual RD or automatic RD always take precedence when configured.

  • If no manual or automatic RD configuration, the RD is derived from the bgp-ad>vpls-id.

  • If manual RD, automatic RD, or VPLS ID are not configured, the RD is derived from the bgp-evpn>evi, except for bgp-mh and except when the EVI is greater than 65535. In these two cases, no EVI-derived RD is possible.

  • If manual RD, automatic RD, VPLS ID, or EVI is not configured, there is no RD and the service fails.

• The selected RD (see preceding rules) is displayed by the Oper Route Dist field of the show service id bgp command.

• The service supports dynamic RD changes. For example, the CLI allows the dynamic update of VPLS ID be , even if it is used to automatically derive the service RD for bgp-ad, bgp-vpls, or bgp-mh.

  Note: When the RD changes, the active routes for that VPLS are withdrawn and readvertised with the new RD.

• If one of the mechanisms to derive the RD for a specified service is removed from the configuration, the system selects a new RD based on the preceding rules. For example, if the VPLS ID is removed from the configuration, the routes are withdrawn, the new RD selected from the EVI, and the routes readvertised with the new RD.

  Note: This reconfiguration fails if the new RD already exists in a different VPLS or Epipe.

• Because the vpls-id takes precedence over the EVI when deriving the RD automatically, adding evpn to an existing bgp-ad service does not impact the existing RD. The latter is important to support bgp-ad to evpn migration.

EVPN multihoming implementation is based on the concept of the ethernet-segment. An ethernet-segment is a logical structure that can be defined in one or more PEs and identifies the CE (or access network) multihomed to the EVPN PEs. An ethernet-segment is associated with port, LAG, PW port, or SDP objects and is shared by all the services defined on those objects. In the case of virtual ESs, individual VID or VC-ID ranges can be associated with the port, LAG, or PW port, SDP objects defined in the ethernet-segment.

Each ethernet-segment has a unique Ethernet Segment Identifier (ESI) that is 10 bytes long and is manually configured in the router.

This section describes the behavior of the EVPN multihoming implementation in an EVPN-MPLS service.

PBB-EVPN uses a reduced subset of the routes and procedures described in RFC 7432. The supported routes are:

• ES routes

• MAC/IP routes

• Inclusive Multicast Ethernet Tag routes

The 7750 SR, 7450 ESS, and 7950 XRS SR OS implementation of PBB-EVPN reuses the existing PBB-VPLS model, where N I-VPLS (or Epipe) services can be linked to a B-VPLS service. BGP-EVPN is enabled in the B-VPLS and the B-VPLS becomes an EVI (EVPN Instance). PBB-EVPN for I-VPLS and PBB Epipe services shows the PBB-EVPN model in the SR OS.

Each PE in the B-VPLS domain advertises its source-bmac as either configured in vpls>pbb>source-bmac or auto-derived from the chassis MAC. The remote PEs install the advertised B-MACs in the B-VPLS FDB. If a specified PE is configured with an ethernet-segment associated with an I-VPLS or PBB Epipe, it may also advertise an es-bmac for the Ethernet-Segment.

In the example shown in PBB-EVPN for I-VPLS and PBB Epipe services, when a frame with MAC DA = AA gets to PE1, a MAC lookup is performed on the I-VPLS FDB and B-MAC-34 is found. A B-MAC lookup on the B-VPLS FDB yields the next-hop (or next-hops if the destination is in an all-active Ethernet-Segment) to which the frame is sent. As in PBB-VPLS, the frame is encapsulated with the corresponding PBB header. A label specified by EVPN for the B-VPLS and the MPLS transport label are also added.

If the lookup on the I-VPLS FDB fails, the system sends the frame encapsulated into a PBB packet with B-MAC DA = Group B-MAC for the ISID. That packet is distributed to all the PEs where the ISID is defined and contains the EVPN label distributed by the Inclusive Multicast routes for that ISID, as well as the transport label.

For PBB-Epipes, all the traffic is sent in a unicast PBB packet to the B-MAC configured in the pbb-tunnel.

The following CLI output shows an example of the configuration of an I-VPLS, PBB-Epipe, and their corresponding B-VPLS.

*A:PE-1>config# 

service vpls 1 name "b-vpls1" b-vpls create
  description "pbb-evpn-service"
  service-mtu 2000
  pbb   
    source-bmac 00:00:00:00:00:03 
  bgp
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  sap 1/1/1:1 create 
  exit
  spoke-sdp 1:1 create


*A:PE-1>config# 

service vpls 101 name "vpls101" i-vpls create
  pbb
    backbone-vpls 1
  sap 1/2/1:101 create
  spoke-sdp 1:102 create

*A:PE-1>config# 

service epipe 102 name "epipe102" create
  pbb
    tunnel 1 backbone-dest-mac 00:00:00:00:00:01 isid 102
  sap 1/2/1:102 create

Configure the bgp-evpn context as described in section EVPN for MPLS tunnels in VPLS services (EVPN-MPLS).

Some EVPN configuration options are not relevant to PBB-EVPN and are not supported when BGP-EVPN is configured in a B-VPLS; these are as follows:

• bgp-evpn> [no] ip-route-advertisement

• bgp-evpn> [no] unknown-mac-route

• bgp-evpn> vxlan [no] shutdown

• bgp-evpn>mpls>force-vlan-vc-forwarding

When bgp-evpn>mpls no shutdown is added to a specified B-VPLS instance, the following considerations apply:

• BGP-AD is supported along with EVPN in the same B-VPLS instance.

• The following B-VPLS and BGP-EVPN commands are fully supported:

  • vpls>backbone-vpls

  • vpls>backbone-vpls>send-flush-on-bvpls-failure

  • vpls>backbone-vpls>source-bmac

  • vpls>backbone-vpls>use-sap-bmac

  • vpls>backbone-vpls>use-es-bmac (For more infomration, see PBB-EVPN multihoming in I-VPLS and PBB Epipe services)

  • vpls>isid-policies

  • vpls>static-mac

  • vpls>SAP or SDP-binding>static-isid

  • bgp-evpn>mac-advertisement - this command affects the 'learned' B-MACs on SAPs or SDP bindings and not on the system B-MAC or SAP/es-bmacs being advertised.

  • bgp-evpn>mac-duplication and settings.

  • bgp-evpn>mpls>auto-bind-tunnel and options.

  • bgp-evpn>mpls>ecmp

  • bgp-evpn>mpls>control-word

  • bgp-evpn>evi

  • bgp-evpn>mpls>ingress-replication-bum-label

EVPN-VPWS for MPLS tunnels uses the RFC 8214 BGP extensions described in EVPN-VPWS for VXLAN tunnels, with the following differences for the Ethernet AD per-EVI routes:

• The MPLS field encodes an MPLS label as opposed to a VXLAN VNI.

• The C flag is set if the control word is configured in the service.

The use and configuration of EVPN-VPWS services is described in EVPN-VPWS for VXLAN tunnels with the following differences when the EVPN-VPWS services use MPLS tunnels instead of VXLAN.

When MPLS tunnels are used, the bgp-evpn>mpls context must be configured in the Epipe. As an example, if Epipe 2 is an EVPN-VPWS service that uses MPLS tunnels between PE2 and PE4, this would be its configuration:

PE2>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-attachment-circuit "AC-1" 
    eth-tag 200
exit
 remote-attachment-circuit "AC-2" 
    eth-tag 200
exit
  mpls bgp 1
    ecmp 2
    no shutdown
exit
sap 1/1/1:1 create

PE4>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-attachment-circuit "AC-2" 
    eth-tag 200
exit
  remote-attachment-circuit "AC-1" 
    eth-tag 100
exit
  mpls bgp 1
    ecmp 2
    no shutdown
exit
spoke-sdp 1:1

Where the following BGP-EVPN commands, specific to MPLS tunnels, are supported in the same way as in VPLS services:

• mpls auto-bind-tunnel

• mpls control-word

• mpls entropy-label

• mpls force-vlan-vc-forwarding

• mpls shutdown

EVPN-VPWS Epipes with MPLS tunnels can also be configured with the following characteristics:

• Access attachment circuits can be SAPs or spoke SDPs. Manually configured and BGP-VPWS spoke SDPs are supported. The VC switching configuration is not supported on BGP-EVPN-enabled pipes.

• EVPN-VPWS Epipes using null SAPs can be configured with sap>ethernet>llf. When enabled, upon removing the EVPN destination, the port is brought oper-down with flag LinkLossFwd, however the AD per EVI route for the SAP is still advertised (the SAP is kept oper-up). When the EVPN destination is created, the port is brought oper-up and the flag cleared.

• EVPN-VPWS Epipes for MPLS tunnels support endpoints. The parameter endpoint endpoint name is configurable along with bgp-evpn>local-attachment-circuit and bgp-evpn>remote-attachment-circuit. The following conditions apply to endpoints on EVPN-VPWS Epipes with MPLS tunnels:

  • Up to two explicit endpoints are allowed per Epipe service with BGP-EVPN configured.

  • A limited endpoint configuration is allowed in Epipes with BGP-EVPN. Specifically, neither active-hold-delay nor revert-time are configurable.

  • When bgp-evpn>remote-attachment-circuit is added to an explicit endpoint with a spoke SDP, the spoke-sdp>precedence command is not allowed. The spoke SDP always has a precedence of four, which is always higher than the EVPN precedence. Therefore, the EVPN-MPLS destination is used for transmission if it is created, and the spoke SDP is only used when the EVPN-MPLS destination is removed.

• EVPN-VPWS Epipes for MPLS tunnels support control word and entropy labels. When a control word is configured, the PE sets the C bit in its AD per-EVI advertisement and sends the control word in the data path. In this case, the PE expects the control word to be received. If there is a mismatch between the received control word and the configured control word, the system does not set up the EVPN destination; as a result, the service does not come up.

• EVPN-VPWS Epipes support force-qinq-vc-forwarding [c-tag-c-tag | s-tag-c-tag] command under bgp-evpn mpls and the qinq-vlan-translation s-tag.c-tag command on ingress QinQ SAPs.

  When QinQ VLAN translation is configured at the ingress QinQ or dot1q SAP, the service-delimiting outer and inner VLAN values can be translated to the configured values. The force-qinq-vc-forwarding s-tag-c-tag command must be configured to preserve the translated QinQ tags in the payload when sending EVPN packets. This translation and preservation behavior is aligned with the ‟normalization” concept described in draft-ietf-bess-evpn-vpws-fxc. The VLAN tag processing described in Epipe service pseudowire VLAN tag processing applies to EVPN destinations in EVPN-VPWS services too.

The following features, described in EVPN-VPWS for VXLAN tunnels, are also supported for MPLS tunnels:

• Advertisement of the Layer-2 MTU and consistency checking of the MTU of the AD per-EVI routes.

• Use of A/S PW and MC-LAG at access.

• EVPN multihoming, including:

  • Single-active and all-active

  • Regular or virtual ESs

  • All existing DF election modes

Epipes with BGP-EVPN MPLS support the following configurations:

• up to two endpoints

• up to two SAPs associated with a different configured endpoint each

• two pairs of local/remote attachment circuit Ethernet tags, also associated with different configured endpoints

• EVPN destinations that can be used as Inter-Chassis Backup (ICB) links

The support of endpoints and up to two SAPs with local-switching allows two and three-node topologies for EVPN-VPWS. EVPN-VPWS endpoints example 1, EVPN-VPWS endpoints example 2, and EVPN-VPWS endpoints example 3 show examples of these topologies.

SAP and spoke SDP based ESs are supported on R-VPLS services where bgp-evpn mpls is enabled.

EVPN-MPLS multihoming in R-VPLS services shows an example of EVPN-MPLS multihoming in R-VPLS services, with the following assumptions:

• There are two subnets for a specific customer (for example, EVI1 and EVI2 in EVPN-MPLS multihoming in R-VPLS services), and a VPRN is instantiated in all the PEs for efficient inter-subnet forwarding.

• A ‟backhaul” R-VPLS with evpn-tunnel mode enabled is used in the core to interconnect all the VPRNs. EVPN IP-prefix routes are used to exchange the prefixes corresponding to the two subnets.

• An all-active ES is configured for EVI1 on PE1 and PE2.

• A single-active ES is configured for EVI2 on PE3 and PE4.

In the example in EVPN-MPLS multihoming in R-VPLS services, the hosts connected to CE1 and CE4 could use regular VRRP for default gateway redundancy; however, this may not be the most efficient way to provide upstream routing.

For example, if PE1 and PE2 are using regular VRRP, the upstream traffic from CE1 may be hashed to the backup IRB VRRP interface, instead of being hashed to the active interface. The same thing may occur for single-active multihoming and regular VRRP for PE3 and PE4. The traffic from CE4 is sent to PE3, while PE4 may be the active VRRP router. In that case, PE3 has to send the traffic to PE4, instead of route it directly.

In both cases, unnecessary bandwidth between the PEs is used to get to the active IRB interface. In addition, VRRP scaling is limited if aggressive keepalive timers are used.

Because of these issues, passive VRRP is recommended as the best method when EVPN-MPLS multihoming is used in combination with R-VPLS redundant interfaces.

Passive VRRP is a VRRP setting in which the transmission and reception of keepalive messages is completely suppressed, and therefore the VPRN interface always behaves as the active router. Passive VRRP is enabled by adding the passive keyword to the VRRP instance at creation, as shown in the following examples:

• config service vprn 1 interface int-1 vrrp 1 passive

• config service vprn 1 interface int-1 ipv6 vrrp 1 passive

For example, if PE1, PE2, and PE5 in EVPN-MPLS multihoming in R-VPLS services use passive VRRP, even if each individual R-VPLS interface has a different MAC/IP address, because they share the same VRRP instance 1 and the same backup IP, the three PEs own the same virtual MAC and virtual IP address (for example, 00-00-5E-00-00-01 and 10.0.0.254). The virtual MAC is auto-derived from 00-00-5E-00-00-VRID per RFC 3768. The following is the expected behavior when passive VRRP is used in this example:

• All R-VPLS IRB interfaces for EVI1 have their own physical MAC/IP address; they also own the same default gateway virtual MAC and IP address.

• All EVI1 hosts have a unique configured default gateway; for example, 10.0.0.254.

• When CE1 or CE2 send upstream traffic to a remote subnet, the packets are routed by the closest PE because the virtual MAC is always local to the PE.

  For example, the packets from CE1 hashed to PE1 are routed at PE1. The packets from CE1 hashed to PE2 are routed directly at PE2.

• Downstream packets (for example, packets from CE3 to CE1), are routed directly by the PE to CE1, regardless of the PE to which PE5 routed the packets.

  For example, the packets from CE3 sent to PE1 are routed at PE1. The packets from CE3 sent to PE2 are routed at PE2.

• In case of ES failure in one of the PEs, the traffic is forwarded by the available PE.

  For example, if the packets routed by PE5 arrive at PE1 and the link to CE1 is down, then PE1 sends the packets to PE2. PE2 forwards the packets to CE1 even if the MAC source address of the packets matches PE2's virtual MAC address. Virtual MACs bypass the R-VPLS interface MAC protection.

The following list summarizes the advantages of using passive VRRP mode versus regular VRRP for EVPN-MPLS multihoming in R-VPLS services:

• Passive VRRP does not require multiple VRRP instances to achieve default gateway load-balancing. Only one instance per R-VPLS, therefore only one default gateway, is needed for all the hosts.

• The convergence time for link/node failures is not impacted by the VRRP convergence, as all the nodes in the VRRP instance are acting as active routers.

• Passive VRRP scales better than VRRP, as it does not use keepalive or BFD messages to detect failures and allow the backup to take over.

When single-active multihoming is used, the IGMP snooping state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IGMP messages to be received to instantiate the IGMP snooping state after the ES activation timer expires; this could result in an increased outage.

The outage can be reduced by using MCS synchronization, which is supported for IGMP snooping in both EVPN-MPLS and PBB-EVPN services (see Multi-chassis synchronization for Layer 2 snooping states). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs. Also, IGMP snooping state can be synchronized only on a SAP.

An increased outage would also occur when using all-active EVPN multihoming. The IGMP snooping state on an ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE may not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, IGMP snooping messages can be synchronized between the PEs on an ES using data-driven IGMP snooping state synchronization, which is supported in EVPN-MPLS services, PBB-EVPN services, EVPN-MPLS VPRN and IES R-VPLS services. The IGMP messages received on an ES SAP or spoke SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES B-MAC (for PBB-EVPN) and these are used to synchronize the IGMP snooping state on the ES SAP or spoke SDP on the receiving PE.

Data-driven IGMP snooping state synchronization is supported for both all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS, EVPN-MPLS VPRN and IES R-VPLS services, and for all-active multihoming in PBB-EVPN services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability. PBB-EVPN with IGMP snooping using single-active multihoming is not supported.

Data-driven IGMP snooping state synchronization is also supported with P2MP mLDP LSPs in both EVPN-MPLS and PBB-EVPN services. When P2MP mLDP LSPs are used in EVPN-MPLS services, all PEs (including the PEs not connected to a multihomed ES) in the EVPN-MPLS service must be running an SR OS version supporting this capability with IGMP snooping enabled and all network interfaces must be configured on FP3 or higher-based line cards.

Data-driven IGMP snooping synchronization with EVPN multihoming shows the processing of an IGMP message for EVPN-MPLS. In PBB-EVPN services, the ES B-MAC is used instead of the ESI label to synchronize the state.

Data-driven synchronization is enabled by default when IGMP snooping is enabled within an EVPN-MPLS service using all-active multihoming or single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If IGMP snooping MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP then MCS synchronization takes precedence over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS IGMP synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down which prevents state synchronization to the PE with the ES down, as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it may not have any IGMP snooping state until the next IGMP messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive can avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services:

1. Upgrade SR OS on all ES PEs to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.

2. Enable IGMP snooping in the required services on all ES PEs. Traffic loss occurs until all ES PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

   Note: There is no action required on the non-ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services:

1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.

2. Enable IGMP snooping in EVPN-MPLS and PBB-EVPN services.

   • Perform the following steps for EVPN-MPLS:

     • Enable IGMP snooping on all non-ES PEs. Traffic loss occurs until the first set of join/query messages are processed by the non-ES PEs.

     • Then enable IGMP snooping on all ES PEs. Traffic loss occurs until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

   • Peform the following steps for PBB-EVPN:

     • Enable IGMP snooping on all ES PEs. Traffic loss occurs until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

     • There is no action required on the non-ES PEs.

To aid with troubleshooting, the debug packet output displays the IGMP packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows.

6 2017/06/16 18:00:07.819 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: TX packet on svc 1
  from chaddr 5e:00:00:16:d8:2e
  send towards ES:esi-1
  Port  : evpn-mpls
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1


4 2017/06/16 18:00:07.820 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: RX packet on svc 1
  from chaddr d8:2e:ff:00:01:41
  received via evpn-mpls on ES:esi-1
  Port  : sap lag-1:1
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1

When single-active multihoming is used, PIM snooping for IPv4 state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IPv4 PIM messages to be received to instantiate the PIM snooping for IPv4 state after the ES activation timer expires, which could result in an increased outage.

This outage can be reduced by using MCS synchronization, which is supported for PIM snooping for IPv4 in both EVPN-MPLS and PBB-EVPN services (see Multi-chassis synchronization for Layer 2 snooping states). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs.

An increased outage would also occur when using all-active EVPN multihoming. The PIM snooping for IPv4 state on an all-active ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE may not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, snooped IPv4 PIM messages should be synchronized between the PEs on an ES using data-driven PIM snooping for IPv4 state synchronization, which is supported in both EVPN-MPLS and PBB-EVPN services. The IPv4 PIM messages received on an ES SAP or spoke SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES B-MAC (for PBB-EVPN) and are used to synchronize the PIM snooping for IPv4 state on the ES SAP or spoke SDP on the receiving PE.

Data-driven PIM snooping state synchronization is supported for all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability with PIM snooping for IPv4 enabled. It is also supported with P2MP mLDP LSPs in the EVPN-MPLS services, in which case all PEs (including the PEs not connected to a multihomed ES) must have PIM snooping for IPv4 enabled and all network interfaces must be configured on FP3 or higher-based line cards.

In addition, data-driven PIM snooping state synchronization is supported for all-active multihoming in PBB-EVPN services and with P2MP mLDP LSPs in PBB-EVPN services. All PEs participating in a multihomed ES, and all PEs using PIM proxy mode (including the PEs not connected to a multihomed ES) in the PBB-EVPN service must be running an SR OS version supporting this capability and must have PIM snooping for IPv4 enabled. PBB-EVPN with PIM snooping for IPv4 using single-active multihoming is not supported.

Data-driven PIM snooping for IPv4 synchronization with EVPN multihoming shows the processing of an IPv4 PIM message for EVPN-MPLS. In PBB-EVPN services, the ES B-MAC is used instead of the ESI label to synchronize the state.

Data-driven synchronization is enabled by default when PIM snooping for IPv4 is enabled within an EVPN-MPLS service using all-active multihoming and single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If PIM snooping for IPv4 MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP or spoke SDP, then MCS synchronization takes preference over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS PIM synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down, which prevents state synchronization to the PE with the ES down as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it may not have any PIM snooping for IPv4 state until the next PIM messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive can avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling PIM snooping for IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services:

• PIM snooping mode

  1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

     Note: There is no action required on the non-ES PEs.

• PIM proxy mode

  • EVPN-MPLS

    1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

    2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

       Note: There is no action required on the non-ES PEs.

  • PBB-EVPN

    1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

    2. Enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.

    3. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling PIM snooping or IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services.

• PIM snooping mode

  1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Then enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.

     Note: There is no action required on the non-ES PEs.

• PIM proxy mode

  1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.

  3. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.

In the above steps, when PIM snooping for IPv4 is enabled, the traffic loss can be reduced or eliminated by configuring a larger hold-time (up to 300 seconds), during which multicast traffic is flooded.

To aid with troubleshooting, the debug packet output displays the PIM packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows:

6 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimVplsFwdJPToEvpn
Forwarding to remote peer on bgp-evpn ethernet-segment esi-1"
7 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:37.040] PIM-TX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>


4 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimProcessPdu
Received from remote peer on bgp-evpn ethernet-segment esi-1, will be applied on
 lag-1:1
"
5 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:30.740] PIM-RX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>

BGP EVPN control plane is extended and aligned with IETF RFC 8317 to support EVPN E-Tree services. EVPN E-Tree BGP routes shows the main EVPN extensions for the EVPN E-Tree information model.

The following BGP extensions are implemented for EVPN E-Tree services:

• An EVPN E-Tree extended community (EC) sub-type 0x5 is defined. The following information is included:

  • The lower bit of the Flags field contains the L bit (where L=1 indicates leaf AC).

  • The leaf label contains a 20-bit MPLS label in the high-order 20 bits of the label field. This leaf label is automatically allocated by the system or statically assigned by the evpn-etree-leaf-label <value> command.

• The new E-Tree EC is sent with the following routes:

  • AD per-ES per PE route for BUM egress filtering:

    Each EVPN E-Tree capable PE advertises an AD per-ES route with the E-Tree EC, and the following information:

    • Service RD and route-target; if ad-per-es-route-target evi-rt-set is configured, then non-zero ESI AD per-ES routes (used for multihoming) are sent per the evi-rt-set configuration, but E-Tree zero-ESI routes (used for E-Tree) are sent based on the default evi-rt configuration

    • ESI = 0

    • Eth Tag = MAX-ET

    • MPLS label = zero

  • AD per-EVI route for root or leaf configuration consistency check as follows:

    • The E-Tree EC is sent with the AD per-EVI routes for a specific ES. In this case, no validation is performed by the implementation, and the leaf indication is only used for troubleshooting on the remote PEs.

    • The MPLS label value is zero.

    • All attachment circuits (ACs) in each EVI for a specific ES must be configured as either a root or leaf AC, but not a combination. In case of a configuration error, for example, where the AC in PE1 is configured as root and in PE2 as leaf AC, the remote PE3 receives the AD per-EVI routes with inconsistent leaf indication. However, the unicast filtering remains unaffected and is still handled by the FDB lookup information.

  • MAC/IP routes for known unicast ingress filtering as follows:

    • An egress PE sends all MAC/IP routes learned over a leaf AC SAP or spoke SDP with this E-Tree EC indicating that the MAC/IP belongs to a leaf AC.

    • The MPLS label value in the EC is 0.

    • Upon receiving a route with E-Tree EC, the ingress PE imports the route and installs the MAC in the FDB with a leaf flag (if there is a leaf indication in the route). Any frame coming from a leaf AC for which the MAC destination address (DA) matches a leaf AC MAC is discarded at the ingress.

    • If two PEs send the same MAC with the same ESI but inconsistent root or leaf AC indication, the MAC is installed in the FDB as root.

EVPN E-Tree services are modeled as VPLS services configured as E-Trees with the bgp-evpn mpls context enabled.

The following example shows a CLI configuration of a VPLS E-Tree service with EVPN E-Tree service enabled.

*A:PE1>config>service>system>bgp-evpn#
  evpn-etree-leaf-label
*A:PE1>config>service# vpls 1 customer 1 etree create 
*A:PE1>config>service>vpls(etree)# info 
----------------------------------------------
  description "ETREE-enabled evpn-mpls-service"
  bgp-evpn
    evi 10
    mpls bgp 1
      no shutdown
      ecmp 2
      auto-bind-tunnel resolution any
      ingress-replication-bum-label
  sap lag-1:1 leaf-ac create 
  exit
  sap 2/1/1:1 leaf-ac create 
  exit
  sap 2/2/2:1 create 
  exit
  spoke-sdp 3:1 leaf-ac create 
  exit

The following configuration guidelines apply to the EVPN E-Tree service:

• Before configuring an EVPN E-Tree service, the user must first run the evpn-etree-leaf-label <value> command. This is relevant for EVPN E-Tree services only. The command allocates an E-Tree leaf label on the system and, when a specific value is configured, the leaf label must match on all other PE nodes attached to the same EVPN service.

  Optionally, the evpn-etree-leaf-label <value> command may be configured with a static label value (within the static label range configured in the system using the config>router>mpls>mpls-label>static-label-range context). The static label is used when global leaf labels are needed in the network. For example, the case where at least one 7250 IXR Gen 1 router is attached to the EVPN E-Tree service.

• The configure service vpls create etree command is compatible with the bgp-evpn mpls context.

• As in VPLS E-Tree services, an AC that is not configured as a leaf AC is treated as root AC.

• MAC addresses learned over a leaf AC SAP or SDP binding are advertised as leaf MAC addresses.

• Any PE with one or more bgp-evpn enabled VPLS E-Tree service advertises an AD per-ES per-PE route with the leaf indication and leaf label used for BUM egress filtering.

• Any leaf AC SAP or SDP binding defined in an ES triggers the advertisement of an AD per-EVI route with the leaf indication.

• EVPN E-Tree services use the following CLI commands:

  • sap sap-id leaf-ac create command using the configure service vpls context

  • mesh-sdp sdp-id:vc-id create leaf-ac command using the configure service vpls context

  • spoke-sdp sdp-id:vc-id create leaf-ac command using the configure service vpls context

• The root-leaf-tag command is blocked in VPLS E-Tree services where bgp-evpn mpls context is enabled.

EVPN E-Tree supports all operations related to flows among local root AC and leaf AC objects in accordance with IETF RFC 8317. This section describes the extensions required to forward traffic from (or to) root AC and leaf AC objects to (or from) BGP EVPN destinations.

EVPN E-Tree procedures support all-active and single-active EVPN multihoming. Ingress filtering can handle MACs learned on ES leaf-ac SAP or SDP bindings. If a MAC associated with an ES leaf-ac is advertised with a different E-Tree indication or if the AD per-EVI routes have inconsistent leaf indications, then the remote PEs performing the aliasing treat the MAC as root.

EVPN E-Tree BUM egress filtering and multihoming shows the expected behavior for multihoming and egress BUM filtering.

Multihoming and egress BUM filtering in EVPN E-Tree BUM egress filtering and multihoming is handled as follows:

• BUM frames received on an ES leaf-ac are flooded to the EVPN based on EVPN E-Tree procedures. The leaf ESI label is sent when flooding to other PEs in the same ES, and additional labels are not added to the stack.

  When flooding in the default multicast list, the egress PE skips all the leaf-acs (including the ES leaf-acs) on the assumption that all ACs in a specific ES for a specified EVI have a consistent E-Tree configuration, and they send an AD per-EVI route with a consistent E-Tree indication.

• BUM frames received on an ES root-ac are flooded to the EVPN based on regular EVPN procedures. The regular ES label is sent for split-horizon when packets are sent to the DF or NDF PEs in the same ES. When flooding in the default multicast list, the egress PE skips the ES SAPs based on the ES label lookup.

If the PE receives an ES MAC from a peer that shares the ES and decides to install it against the local ES SAP that is oper-up, it checks the E-Tree configuration (root or leaf) of the local ES SAP against the received MAC route. The MAC route is processed as follows:

• If the E-Tree configuration does not match, then the MAC is not installed against any destination until the misconfiguration is resolved.

• If the SAP is oper-down, the MAC is installed against the EVPN destination to the peer.

SR OS supports PBB-EVPN E-Tree services in accordance with IETF RFC 8317. PBB-EVPN E-Tree services are modeled as PBB-EVPN services where some I-VPLS services are configured as etree and some of their SAP or spoke SDPs are configured as leaf-acs.

The procedures for the PBB-EVPN E-Tree are similar to those for the EVPN E-Tree, except that the egress leaf-to-leaf filtering for BUM traffic is based on the B-MAC source address. Also, the leaf label and the EVPN AD routes are not used.

The PBB-EVPN E-Tree operation is as follows:

• When one or more I-VPLS E-Tree services are linked to a B-VPLS, the leaf backbone source MAC address (leaf-source-bmac parameter) is used for leaf-originated traffic in addition to the source B-VPLS MAC address (source-bmac parameter) that is used for sourcing root traffic.

• The leaf backbone source MAC address for PBB must be configured using the command config>service>pbb>leaf-source-bmac ieee-address before the configuration of any I-VPLS E-Tree service.

• The leaf-source-bmac address is advertised in a B-MAC route with a leaf indication.

• Known unicast filtering occurs at the ingress PE. When a frame enters an I-VPLS leaf-ac, a MAC lookup is performed. If the C-MAC DA is associated with a leaf B-MAC, the frame is dropped.

• Leaf-to-leaf BUM traffic filtering occurs at the egress PE. When flooding BUM traffic with the B-MAC SA matching a leaf B-MAC, the egress PE skips the I-VPLS leaf-acs.

The following CLI example shows an I-VPLS E-Tree service that uses PBB-EVPN E-Tree. The leaf-source-bmac address must be configured before the configuration of the I-VPLS E-Tree. As is the case in regular E-Tree services, SAP and spoke SDPs that are not explicitly configured as leaf-acs are considered root-ac objects.

A:PE-2>config>service# info
----------------------------------------------
        pbb
            leaf-source-bmac 00:00:00:00:00:22
        exit
        vpls 1000 customer 1 name "vpls1000" b-vpls create
            service-mtu 2000
            bgp
            exit
            bgp-evpn
                evi 1000
                exit
                mpls bgp 1
                    ingress-replication-bum-label
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp                      
                shutdown
            exit
            no shutdown
        exit
        vpls 1001 customer 1 i-vpls etree create
            pbb
                backbone-vpls 1000
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:1001 leaf-ac create
                no shutdown
            exit
            sap 1/1/1:1002 create
                no shutdown
            exit
            no shutdown
        exit

The following considerations apply to PBB-EVPN E-Trees and multihoming:

• All-active multihoming is not supported on leaf-ac I-VPLS SAPs.

• Single-active multihoming is supported on leaf-ac I-VPLS SAPs and spoke SDPs.

• ISID- and RFC 7623-based C-MAC flush are supported in addition to PBB-EVPN E-Tree services and single-active multihoming.

When enable-rr-vpn-forwarding or enable-inter-as-vpn is configured, only EVPN-MPLS routes are processed for label swap and the next hop is changed. EVPN-VXLAN routes are re-advertised without a change in the next hop.

The following shows how the router processes and re-advertises the different EVPN route types. For more information about the route fields, see the BGP-EVPN control plane for MPLS tunnels Guide.

• Auto-discovery (AD) routes (type 1)

  For AD per EVI routes, the MPLS label is extracted from the route NLRI. The route is re-advertised with Next-Hop-Self (NHS) and a new label. No modifications are made for the remaining attributes.

  For AD per ES routes, the MPLS label in the NLRI is zero. The route is re-advertised with NHS and the MPLS label remains zero. No modifications are made for the remaining attributes.

• MAC/IP routes (type 2)

  The MPLS label (Label-1) is extracted from the NLRI. The route is re-advertised with NHS and a new Label-1. No modifications are made for the remaining attributes.

• Inclusive Multicast Ethernet Tag (IMET) routes (type 3)

  Because there is no MPLS label present in the NLRI, the MPLS label is extracted from the PMSI Tunnel Attribute (PTA) if needed, and the route is then re-advertised with NHS, with the following considerations:

  • For IMET routes with tunnel-type Ingress Replication, the router extracts the IR label from the PTA. The router programs the label swap and re-advertises the route with a new label in the PTA.

  • For tunnel-type P2MP mLDP, the router re-advertises the route with NHS. No label is extracted; therefore, no swap operation occurs.

  • For tunnel-type Composite, the IR label is extracted from the PTA, the swap operation is programmed and the route re-advertised with NHS. A new label is encoded in the PTA’s IR label with no other changes in the remaining fields.

  • For tunnel-type AR, the routes are always considered VXLAN routes and are re-advertised with the next-hop unchanged.

• Ethernet-Segment (ES) routes (type 4)

  Because ES routes do not contain an MPLS label, the route is re-advertised with NHS and no modifications to the remaining attributes. Although an ASBR or ABR re-advertises ES routes, EVPN multihoming for ES PEs located in different ASs or IGMP domains is not supported.

• IP-Prefix routes (type 5)

  The MPLS label is extracted from the NLRI and the route is re-advertised with NHS and a new label. No modifications are made to the remaining attributes.

Inter-AS Option B and VPN-NH-RR support the use of non-segmented trees for forwarding BUM traffic in EVPN.

For ingress replication and non-segmented trees, the ASBR or ABR performs an EVPN BUM label swap without any aggregation or further replication. This concept is shown in VPN-NH-RR and ingress replication for BUM traffic.

In VPN-NH-RR and ingress replication for BUM traffic, when PE2, PE3, and PE4 advertise their IMET routes, the ABRs re-advertise the routes with NHS and a different label. However, IMET routes are not aggregated; therefore, PE1 sets up three different EVPN multicast destinations and sends three copies of every BUM packet, even if they are sent to the same ABR. This example is also applicable to ASBRs and Inter-AS Option B.

P2MP mLDP may also be used with VPN-NH-RR, but not with Inter-AS Option B. The ABRs, however, do not aggregate or change the mLDP root IP addresses in the IMET routes. The root IP addresses must be leaked across IGP domains. For example, if PE2 advertises an IMET route with mLDP or composite tunnel type, PE1 is able to join the mLDP tree if the root IP is leaked into PE1’s IGP domain.

In general, EVPN multihoming is supported in Inter-AS Option B or VPN-NH-RR networks with the following limitations:

• An ES PE can only process a remote ES route correctly if the received next hop and origination IP address match. EVPN multihoming is not supported when the ES PEs are in different ASs or IGP domains, or if there is an NH-RR peering the ES PEs and overriding the ES route next hops.

• EVPN multihoming ESs are not supported on EVPN PEs that are also ABRs or ASBRs.

• Mass-withdraw based on the AD per-ES routes is not supported for a PE that is in a different AS or IGP domain that the ES PEs. EVPN multihoming with inter-AS Option B or VPN-NH-RR shows an EVPN multihoming scenario where the ES PEs, PE2 and PE3, and the remote PE, PE1, are in different ASs or IGP domains.

In EVPN multihoming with inter-AS Option B or VPN-NH-RR, PE1’s aliasing and backup functions to the remote ES-1 are supported. However, PE1 cannot identify the originating PE for the received AD per-ES routes because they are both arriving with the same next hop (ASBR/ABR4) and RDs may not help to correlate each AD per-ES route to a specified PE. Therefore, if there is a failure on PE2’s ES link, PE1 cannot remove PE2 from the destinations list for ES-1 based on the AD per-ES route. PE1 must wait for the AD per-EVI route withdrawals to remove PE2 from the list. In summary, when the ES PEs and the remote PE are in different ASs or IGP domains, per-service withdrawal based on AD per-EVI routes is supported, but mass-withdrawal based on AD per-ES routes is not supported.

Unicast procedures known to EVPN-MPLS E-Tree are supported in Inter-AS Option B or VPN-NH-RR scenarios, however, the BUM filtering procedures are affected.

As described in EVPN E-Tree, leaf-to-leaf BUM filtering is based on the Leaf Label identification at the egress PE. In a non-Inter-AS or non-VPN-NH-RR scenario, EVPN E-tree AD per-ES (ESI-0) routes carrying the Leaf Label are distinguished by the advertised next hop. In Inter-AS or VPN-NH-RR scenarios, all the AD per-ES routes are received with the ABR or ASBR next hop. Therefore, AD per-ES routes originating from different PEs would all have the same next hop, and the ingress PE would not be able to determine which leaf label to use for a specific EVPN multicast destination.

A simplified EVPN E-Tree solution is supported, where an E-Tree Leaf Label is not installed in the IOM if the PE receives more than one E-Tree AD per-ES route, with different RDs, for the same next hop. In this case, leaf BUM traffic is transmitted without a Leaf Label and the leaf-to-leaf traffic filtering depends on the egress source MAC filtering on the egress PE. See EVPN E-Tree egress filtering based on MAC source address.

PBB-EVPN E-tree services are not affected by Inter-AS or VPN-NH-RR scenarios, as AD per-ES routes are not used.

When proxy-ARP/proxy-ND is enabled, the system starts populating the proxy table and responding to ARP-requests/NS messages. To keep the active IP-MAC entries alive and ensure that all the host/routers in the service update their ARP/ND caches, the system may generate the following three types of ARP/ND messages for a specified IP-MAC entry:

• periodic refresh messages (ARP-requests or NS for a specified IP)

  These messages are activated by the send-refresh command and their objective is to keep the existing FDB and Proxy-ARP/ND entries alive to minimize EVPN withdrawals and re-advertisements.

• unsolicited refresh messages (unsolicited GARP or NA messages)

  These messages are sent by the system when a new entry is learned or updated. Their objective is to update the attached host/router caches.

• confirm messages (unicast ARP-requests or unicast NS messages)

  These messages are sent by the system when a new MAC is learned for an existing IP. The objective of the confirm messages is to verify that a specified IP has really moved to a different part of the network and is associated with the new MAC. If the IP has not moved, it forces the owners of the duplicate IP to reply and cause dup-detect to kick in.

RFC 4861 describes the use of the (R) or ‟Router” flag in NA messages as follows:

• A node capable of routing IPv6 packets must reply to NS messages with NA messages where the R flag is set (R=1).

• Hosts must reply with NA messages where R=0.

The use of the ‟R” flag in NA messages impacts how the hosts select their default gateways when sending packets ‟off-link”. Therefore, the proxy-ND function on the 7750 SR, 7450 ESS, or 7950 XRS does one of the following, depending on whether it can provide the appropriate R flag information:

• Proxy-ND provides the appropriate R flag information in proxy-ND NA replies

• If proxy-ND cannot provide the appropriate R flag when replying, it floods the received NA messages.

Because of the ‟R” flag, the procedure for learning proxy-ND entries and replying to NS messages differs from the procedures for proxy-ARP in IPv4: the router or host flag is added to each entry, and that determines the flag to use when responding to a NS.

The procedure to add the R flag to a specified entry is as follows:

• Dynamic entries are learned based on received NA messages. The R flag is also learned and added to the proxy-ND entry so that the appropriate R flag is used in response to NS requests for a specified IP.

• Static entries are configured as host or router as per the command [no] static <ip-address> <ieee-address> {host | router}.

• EVPN entries are learned from BGP and the command evpn-nd-advertise {host | router} determines the R flag added to them.

• In addition, the evpn-nd-advertise {host | router} command indicates what static and dynamic IP-MAC entries the system advertises in EVPN. If evpn-nd-advertise router is configured, the system should flood the received unsolicited NA messages for hosts. This is controlled by the [no] host-unsolicited-na-flood-evpn command. The opposite is also recommended so that the evpn-nd-advertise host is configured with the router-unsolicited-na-flood-evpn.

SR OS supports the association of configured MAC lists with a configured dynamic proxy-ARP or proxy-ND IP address. The actual proxy-ARP or proxy-ND entry is not created until an ARP or Neighbor Advertisement message is received for the IP and one of the MACs in the associated MAC-list. This is in accordance with IETF Draft draft-ietf-bess-evpn-proxy-arp-nd, which states that a proxy-ARP or proxy-ND IP entry can be associated with one MAC among a list of allowed MACs.

The following example shows the use of MAC lists for dynamic entries.

A:PE-2>config>service#
  proxy-arp-nd
    mac-list ISP-1 create 
      mac 00:de:ad:be:ef:01 
      mac 00:de:ad:be:ef:02 
      mac 00:de:ad:be:ef:03
 
A:PE-2>config>service>vpls>proxy-arp#
  dynamic 1.1.1.1 create
    mac-list ISP-1
    resolve 30
 
A:PE-2>config>service>vpls>proxy-nd#
  dynamic 2001:db8:1000::1 create
    mac-list ISP-1 
    resolve 30

where:

• A dynamic IP (dynamic ip create) is configured and associated with a MAC list (mac-list name).

• The MAC list is created in the config>service context and can be reused by multiple configured dynamic IPs as follows:

  • in different services

  • in the same service, for proxy-ARP and proxy-ND entries

• If the MAC list is empty, the proxy-ARP or proxy-ND entry is not created for the configured IP.

• The same MAC list can be applied to multiple configured dynamic entries even within the same service.

• The new proxy-ARP and proxy-ND entries behave as dynamic entries and are displayed as type dyn in the show commands.

  The following output example displays the entry corresponding to the configured dynamic IP.

  *A:PE-2# show service id 1 proxy-arp detail                                  
  -------------------------------------------------------------------------------
  Proxy Arp
  -------------------------------------------------------------------------------
  Admin State       : enabled             
  Dyn Populate      : enabled             
  Age Time          : 900 secs            Send Refresh      : 300 secs
  Table Size        : 250                 Total             : 1
  Static Count      : 0                   EVPN Count        : 0
  Dynamic Count     : 1                   Duplicate Count   : 0
  Dup Detect
  -------------------------------------------------------------------------------
  Detect Window     : 3 mins              Num Moves         : 5
  Hold down         : 9 mins              
  Anti Spoof MAC    : None
  EVPN
  -------------------------------------------------------------------------------
  Garp Flood        : enabled             Req Flood         : enabled
  Static Black Hole : disabled            
  -------------------------------------------------------------------------------
  ===============================================================================
  VPLS Proxy Arp Entries
  ===============================================================================
  IP Address          Mac Address       Type Status    Last Update
  -------------------------------------------------------------------------------
  1.1.1.1            00:de:ad:be:ef:01  dyn active    02/23/2016 09:05:49
  -------------------------------------------------------------------------------
  Number of entries : 1
  ===============================================================================
  *A:PE-2# show service proxy-arp-nd mac-list "ISP-1" associations 
  ===============================================================================
  MAC List Associations
  ===============================================================================
  Service Id                    IP Addr
  -------------------------------------------------------------------------------
  1                             1.1.1.1
  1                             2001:db8:1000::1
  -------------------------------------------------------------------------------
  Number of Entries: 2
  ===============================================================================
  

Although no new proxy-ARP or proxy-ND entries are created when a dynamic IP is configured, the router triggers the following resolve procedure:

1. The router sends a resolve message with a configurable frequency of 1 to 60 minutes; the default value is five minutes.

   Note: The resolve message is an ARP-request or NS message flooded to all the non-EVPN endpoints in the service.

2. The router sends resolve messages at the configured frequency until a dynamic entry for the IP is created.

   Note: The dynamic entry is created only if an ARP, GARP, or NA message is received for the configured IP, and the associated MAC belongs to the configured MAC list of the IP. If the MAC list is empty, the proxy-ARP or proxy-ND entry is not created for the configured IP.

After a dynamic entry (with a MAC address included in the list) is successfully created, its behavior (for send-refresh, age-time, and other activities) is the same as a configured dynamic entry with the following exceptions.

• Regular dynamic entries may override configured dynamic entries, but static or EVPN entries cannot override configured dynamic entries.

• If the corresponding MAC is flushed from the FDB after the entry is successfully created, the entry becomes inactive in the proxy-ARP or proxy-ND table and the resolve process is restarted.

• If the MAC list is changed, all the IPs that point to the list delete the proxy entries and the resolve process is restarted.

• If there is an existing configured dynamic entry and the router receives a GARP, ARP, or NA for the IP with a MAC that is not contained in the MAC list, the message is discarded and the proxy-ARP or proxy-ND entry is deleted. The resolve process is restarted.

• If there is an existing configured dynamic entry and the router receives a GARP, ARP, or NA for the IP with a MAC contained in the MAC list, the existing entry is overridden by the IP and new MAC, assuming the confirm procedure passes.

• The dup-detect and confirm procedures work for the configured dynamic entries when the MAC changes are between MACs in the MAC list. Changes to an off-list MAC cause the entry to be deleted and the resolve process is restarted.

This section describes the configuration aspects of a VPLS/R-VPLS with EVPN-VXLAN and EVPN-MPLS.

In a service where EVPN-VXLAN and EVPN-MPLS are configured together, the configure service vpls bgp-evpn vxlan bgp 1 and configure service vpls bgp-evpn mpls bgp 2 commands allow the user to associate EVPN-MPLS to a different instance from that associated with EVPN-VXLAN, and have both encapsulations simultaneously enabled in the same service. At the control plane level, EVPN MAC/IP advertisement routes received in one instance are consumed and readvertised in the other instance as long as the route is the best route for a specific MAC. Inclusive multicast routes are independently generated for each BGP instance. In the data plane, the EVPN-MPLS and EVPN-VXLAN destinations are instantiated in different implicit Split Horizon Groups (SHGs) so that traffic can be forwarded between them.

The following example shows a VPLS service with two BGP instances and both VXLAN and MPLS encapsulations configured for the same BGP-EVPN service.

*A:PE-1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls and evpn-vxlan in the same service"
  vxlan instance 1 vni 7000 create
  exit
  bgp 
    route-distinguisher 10:2
    route-target target:64500:1
  exit
  bgp 2 
    route-distinguisher 10:1
    route-target target:64500:1
  exit
  bgp-evpn
    evi 7000
    incl-mcast-orig-ip 10.12.12.12
    vxlan bgp 1 vxlan-instance 1
      no shutdown 
    mpls bgp 2
      control-word
      auto-bind-tunnel 
        resolution any
      exit
      force-vlan-vc-forwarding
      no shutdown
    exit
  exit
  no shutdown

The following list describes the preceding example:

• bgp 1 or bgp is the default BGP instance

• bgp 2 is the additional instance required when both bgp-evpn vxlan and bgp-evpn mpls are enabled in the service

• The commands supported in instance 1 are also available in instance 2 with the following considerations:

  • pw-template-binding

    The pw-template-binding can only exist in instance 1; it is not supported in instance 2.

  • route-distinguisher

    The operating route-distinguisher in both BGP instances must be different.

  • route-target

    The route target in both instances can be the same or different.

  • vsi-import and vsi-export

    Import and export policies can also be defined for either BGP instance.

• MPLS and VXLAN can use either BGP instance, and the instance is associated when bgp-evpn mpls or bgp-evpn vxlan is created. The bgp-evpn vxlan command must include not only the association to a BGP instance, but also to a vxlan-instance (because the VPLS services support two VXLAN instances).

  Note: The bgp-evpn vxlan no shutdown command is only allowed if bgp-evpn mpls shutdown is configured, or if the BGP instance associated with the MPLS has a different route distinguisher than the VXLAN instance.

The following features are not supported when two BGP instances are enabled on the same VPLS/R-VPLS service:

• SDP bindings

• M-VPLS, I-VPLS, B-VPLS, or E-Tree VPLS

• Proxy-ARP and proxy-ND

• BGP Multihoming

• IGMP, MLD, and PIM snooping

• BGP-VPLS or BGP-AD (SDP bindings are not created)

The service>vpls>bgp-evpn>ip-route-advertisement command is not supported on R-VPLS services with two BGP instances.

EVPN-SRv6 and EVPN-MPLS or EVPN-VXLAN can be simultaneously configured in the same VPLS service (but not R-VPLS), in different instances.

EVPN-SRv6 and EVPN-VXLAN instances in the same VPLS service follow the same configuration rules as described in EVPN-VXLAN to EVPN-MPLS interworking, and the same processing of MAC/IP Advertisement routes and Inclusive Multicast Ethernet Tag routes is applied.

The following example shows a VPLS service with two BGP instances, with both VXLAN and SRv6 encapsulations configured under BGP-EVPN.

A:node-2>config>service>vpls "evpn-srv6-vxlan-1"> info
    admin-state enable
    description "evpn-srv6 and evpn-vxlan in the same service"
    vxlan {
        instance 1 {
            vni 12340
        }
    }
    segment-routing-v6 1 {
        locator "loc-1" {
            function {
                end-dt2u {
                }
                end-dt2m {
                }
            }
        }
    }
    bgp 1 {
        route-distinguisher "12340:1"
        route-target {
            export "target:64500:12340"
            import "target:64500:12340"
        }
    }
    bgp 2 {
        route-distinguisher "12340:2"
        route-target {
            export "target:64500:12341"
            import "target:64500:12341"
        }
    }
    bgp-evpn {
        evi 12340
        incl-mcast-orig-ip 10.12.12.12
        segment-routing-v6 2 {
            admin-state enable
            ecmp 4
            force-vc-forwarding vlan
            srv6 {
                default-locator "loc-1"
            }
        }
        vxlan 1 {
            admin-state enable
            vxlan-instance 1
        }
    }

A:node-2>config>service>vpls# info
----------------------------------------------
            description "evpn-srv6 and evpn-vxlan in the same service"
            vxlan instance 1 vni 12340 create
            exit
            segment-routing-v6 1 create
                locator "loc-1"
                    function
                        end-dt2u
                        end-dt2m
                    exit
                exit
            exit
            bgp
                route-distinguisher 12340:1
                route-target export target:64500:12340 import target:64500:12340
            exit
            bgp 2
                route-distinguisher 12340:2
                route-target export target:64500:12341 import target:64500:12341
            exit
            bgp-evpn
                incl-mcast-orig-ip 10.12.12.12
                evi 12340
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
                segment-routing-v6 bgp 2 srv6-instance 1 default-locator "loc-1" create
                    ecmp 4
                    force-vlan-vc-forwarding
                    route-next-hop 2001:db8::76
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            no shutdown
----------------------------------------------

configure service vpls "evpn-srv6-mpls-1" >info
admin-state enable
    description "evpn-srv6 and evpn-mpls in the same service"
    segment-routing-v6 1 {
        locator "loc-1" {
            function {
                end-dt2u {
                }
                end-dt2m {
                }
            }
        }
    }
    bgp 1 {
        route-distinguisher "12341:1"
        route-target {
            export "target:64500:12342"
            import "target:64500:12342"
        }
    }
    bgp 2 {
        route-distinguisher "12341:2"
        route-target {
            export "target:64500:12343"
            import "target:64500:12343"
        }
    }
    bgp-evpn {
        evi 12340
        incl-mcast-orig-ip 10.12.12.12
        segment-routing-v6 2 {
            admin-state enable
            ecmp 4
            force-vc-forwarding vlan
            srv6 {
                default-locator "loc-1"
            }
            route-next-hop {
                ip-address 2001:db8::76
            }
        }
        mpls 1 {
            admin-state enable
            force-vc-forwarding vlan
            split-horizon-group "SHG-1"
            ingress-replication-bum-label true
            ecmp 4
            mh-mode access
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    split-horizon-group "SHG-1" {
    }

A:node-2>config>service>vpls# info
----------------------------------------------
            description "evpn-srv6 and evpn-mpls in the same service"
            split-horizon-group "SHG-1" create
            exit
            segment-routing-v6 1 create
                locator "loc-1"
                    function
                        end-dt2u
                        end-dt2m
                    exit
                exit
            exit
            bgp
                route-distinguisher 12341:1
                route-target export target:64500:12342 import target:64500:12342
            exit
            bgp 2
                route-distinguisher 12341:2
                route-target export target:64500:12343 import target:64500:12343
            exit
            bgp-evpn
                incl-mcast-orig-ip 10.12.12.12
                evi 12341
                mpls bgp 1
                    mh-mode access
                    force-vlan-vc-forwarding
                    split-horizon-group "SHG-1"
                    ingress-replication-bum-label
                    ecmp 4
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
                segment-routing-v6 bgp 2 srv6-instance 1 default-locator "loc-1" create
                    ecmp 4
                    force-vlan-vc-forwarding
                    route-next-hop 2001:db8::76
                    split-horizon-group "SHG-1"
                    no shutdown
                exit
            exit
            stp
                shutdown
            exit
            no shutdown
----------------------------------------------

From a BGP perspective, the two BGP instances configured in the service are independent of each other. The redistribution of routes between the BGP instances is resolved at the EVPN application layer.

By default, if EVPN-VXLAN and EVPN-MPLS are both enabled in the same service, BGP sends the generated EVPN routes twice: with the RFC 9012 BGP encapsulation extended community set to VXLAN and a second time with the encapsulation type set to MPLS.

Usually, a DCGW peers a pair of Route Reflectors (RRs) in the DC and a pair of RRs in the WAN. For this reason, the user needs to add router policies so that EVPN-MPLS routes are only sent to the WAN RRs and EVPN-VXLAN routes are only sent to the DC RRs. The following examples show how to configure router policies.

config>router>bgp# 
 vpn-apply-import
 vpn-apply-export
 group "WAN"                
  family evpn                
  type internal                
  export "allow only mpls"                 
  neighbor 192.0.2.6           
 group "DC"               
  family evpn                
  type internal                
  export "allow only vxlan"                 
  neighbor 192.0.2.2
config>router>policy-options# info 
----------------------------------------------
         community "vxlan" members "bgp-tunnel-encap:VXLAN"
         community "mpls" members "bgp-tunnel-encap:MPLS"
            policy-statement "allow only mpls"
                entry 10
                  from
                    family evpn
                    community vxlan
                  action drop
                  exit
                exit
            exit
            policy-statement "allow only vxlan"
                entry 10
             from
              family evpn
              community mpls
                    action drop
                    exit
                exit
            exit

In a BGP instance, the EVPN routes are imported based on the route-targets and regular BGP selection procedures, regardless of their encapsulation.

The BGP-EVPN routes are generated and redistributed between BGP instances based on the following rules:

• Auto-discovery (AD) routes (type 1) are not generated by services with two BGP EVPN instances, unless a local Ethernet segment is present on the service. However, AD routes received from the EVPN-MPLS peers are processed for aliasing and backup functions as usual.

• MAC/IP routes (type 2) received in one of the two BGP instances are imported and the MACs added to the FDB according to the existing selection rules. If the MAC is installed in the FDB, it is readvertised in the other BGP instance with the new BGP attributes corresponding to the BGP instance (route target, route distinguisher, and so on). The following considerations apply to these routes:

  • The mac-advertisement command governs the advertisement of any MACs (even those learned from BGP).

  • A MAC route is redistributed only if it is the best route based on the EVPN selection rules.

  • If a MAC route is the best route and has to be redistributed, the MAC/IP information, along with the MAC mobility extended community, is propagated in the redistribution.

  • The router redistributes any MAC route update for which any attribute has changed. For example, a change in the SEQ or sticky bit in one instance is updated in the other instance for a route that is selected as the best MAC route.

• EVPN inclusive multicast routes are generated independently for each BGP instance with the corresponding BGP encapsulation extended community (VXLAN or MPLS). Also, the following considerations apply to these routes:

  • Ingress Replication (IR) and Assisted Replication (AR) routes are supported in the EVPN-VXLAN instance. If AR is configured, the AR IP address must be a loopback address different from the system-ip and the configured originating-ip address.

  • IR, P2MP mLDP, and composite inclusive multicast routes are supported in the EVPN-MPLS instance.

  • The modification of the incl-mcast-orig-ip command is supported, subject to the following considerations:

    • The configured IP in the incl-mcast-orig-ip command is encoded in the originating-ip field of the inclusive multicast Routes for IR, P2MP, and composite tunnels.

    • The originating-ip field of the AR routes is still derived from the service>system>vxlan>assisted-replication-ip configured value.

  • EVPN handles the inclusive multicast routes in a service based on the following rules:

    • For IR routes, the EVPN destination is set up based on the NLRI next hop.

    • For P2MP mLDP routes, the PMSI Tunnel Attribute tunnel-id is used to join the mLDP tree.

    • For composite P2MP-IR routes, the PMSI Tunnel Attribute tunnel-id is used to join the tree and create the P2MP bind. The NLRI next-hop is used to build the IR destination.

    • For AR routes, the NLRI next-hop is used to build the destination.

    • The following applies if a router receives two inclusive multicast routes in the same instance:

      • If the routes have the same originating-ip but different route distinguishers and next-hops, the router processes both routes. In the case of IR routes, it sets up two destinations: one to each next-hop.

      • If the routes have the same originating-ip, different route distinguishers, but same next hops, the router sets up only one binding for IR routes.

      • The router ignores inclusive multicast routes received with its own originating-ip, regardless of the route distinguisher.

• IP-Prefix routes (type 5) are not generated or imported by a service with two BGP instances.

The rules in this section can be extrapolated to VPLS services where SRv6 and MPLS or VXLAN are configured in different instances of the same VPLS with different split horizon groups.

Multihomed anycast solution shows the anycast mechanism used to support gateway redundancy for dual BGP-instance services. The example shows two redundant DC gateways (DCGWs) where the VPLS services contain two BGP instances: one each for EVPN-VXLAN and EVPN-MPLS.

The example shown in Multihomed anycast solution depends on the ability of the two DCGWs to send the same inclusive multicast route to the remote PE or NVEs, such that:

• The remote PE or NVEs create a single BUM destination to one of the DCGWs (because the BGP selects only the best route to the DCGWs).

• The DCGWs do not create a destination between each other.

This solution avoids loops for BUM traffic, and known unicast traffic can use either DCGW router, depending on the BGP selection. The following CLI example output shows the configuration of each DCGW.

/* bgp configuration on DCGW1 and DCGW2 */
config>router>bgp# 
 group ”WAN"                
  family evpn                
  type internal                
  neighbor 192.0.2.6          
 group ”DC"                
  family evpn                
  type internal                
  neighbor 192.0.2.2
/* vpls service configuration */
DCGW-1# config>service>vpls(1)#
-----------------------
bgp 
  route-distinguisher 64501:12
  route-target target:64500:1
exit
bgp 2
  route-distinguisher 64502:12
  route-target target:64500:1
exit
vxlan instance 1 vni 1 create
exit
bgp-evpn 
  evi 1
  incl-mcast-orig-ip 10.12.12.12
  vxlan bgp 1 vxlan-instance 1
   no shutdown 
  mpls bgp 2
   no shutdown
   auto-bind-tunnel 
     resolution any
   exit
<snip>
DCGW-2# config>service>vpls(1)#
-----------------------
bgp 
  route-distinguisher 64501:12
  route-target target:64500:1
exit
bgp 2
  route-distinguisher 64502:12
  route-target target:64500:1
exit
vxlan instance 1 vni 1 create
exit
bgp-evpn 
  evi 1
  incl-mcast-orig-ip 10.12.12.12
  vxlan bgp 1 vxlan-instance 1
   no shutdown 
  mpls bgp 2
   no shutdown
   auto-bind-tunnel 
     resolution any
<snip>

Based on the preceding configuration example, the behavior of the DCGWs in this scenario is as follows:

• DCGW-1 and DCGW-2 send inclusive multicast routes to the DC RR and WAN RR with the same route key. For example:

  • DCGW-1 and DCGW-2 both send an IR route to DC RR with RD=64501:12, orig-ip=10.12.12.12, and a different next-hop and tunnel ID

  • DCGW-1 and DCGW-2 both send an IR route to WAN RR with RD=64502:12, orig-ip=10.12.12.12, and different next-hop and tunnel ID

• DCGW-1 and DCGW-2 both receive MAC/IP routes from DC and WAN that are redistributed to the other BGP instances, assuming that the route is selected as best route and the MAC is installed in the FDB.

  As described in section BGP-EVPN routes in services configured with two BGP instances, router peer policies are required so that only VXLAN or MPLS routes are sent or received for a specific peer.

• Configuration of the same incl-mcast-orig-ip address in both DCGWs enables the anycast solution for BUM traffic for all the following reasons:

  • The configured originating-ip is not required to be a reachable IP address and this forces the remote PE or NVEs to select only one of the two DCGWs.

  • The BGP next-hops are allowed to be the system-ip or even a loopback address. In both cases, the BGP next-hops are not required to be reachable in their respective networks.

In the example shown in Multihomed anycast solution, PE-1 picks up DCGW-1's inclusive multicast route (because of its lower BGP next-hop) and creates a BUM destination to 192.0.2.4. When sending BUM traffic for VPLS-1, it only sends the traffic to DCGW-1. In the same way, the DCGWs do no set up BUM destinations between each other as they use the same originating-ip in their inclusive multicast routes.

The remote PE or NVEs perform a similar BGP selection for MAC/IP routes, as a specific MAC is sent by the two DCGWs with the same route-key. A PE or NVE sends known unicast traffic for a specific MAC to only one DCGW.

Anycast multihoming and mLDP shows an example of a common BGP EVPN service configured in redundant anycast DCGWs and mLDP used in the MPLS instance.

When mLDP is used with multiple anycast multihoming DCGWs, the same originating IP address must be used by all the DCGWs. Failure to do so may result in packet duplication.

In the example shown in Anycast multihoming and mLDP, each pair of DCGWs (DCGW1/DCGW2 and DCGW3/DCGW4) is configured with a different originating IP address, which causes the following behavior:

1. DCGW3 and DCGW4 receive the inclusive multicast routes with the same route key from DCGW1 and DCGW2.

2. Both DCGWs (DCGW3 and DCGW4) select only one route, which is generally the same, for example, DCGW1's inclusive multicast route.

3. As a result, DCGW3 and DCGW4 join the mLDP tree with root in DCGW1, creating packet duplication when DCGW1 sends BUM traffic.

4. Remote PE nodes with a single BGP-EVPN instance join the mLDP tree without any problem.

To avoid the packet duplication shown in Anycast multihoming and mLDP, Nokia recommends to configure the same originating IP address in all four DCGWs (DCGW1/DCGW2 and DCGW3/DCGW4). However, the route distinguishers can be different per pair.

The following behavior occurs if the same originating IP address is configured on the DCGW pairs shown in Anycast multihoming and mLDP.

• DCGW3 and DCGW4 do not join any mLDP tree sourced from DCGW1 or DCGW2, which prevents any packet duplication. This is because a router ignore inclusive multicast routes received with its own originating-ip, regardless of the route-distinguisher.

• PE1 joins the mLDP trees from the two DCs.

SR OS supports Interconnect ESs (I-ES) for VXLAN as per RFC9014. An I-ES is a virtual ES that allows DCGWs with two BGP instances to handle VXLAN access networks as any other type of ES. I-ESs support the RFC 7432 multihoming functions, including single-active and all-active, ESI-based split-horizon filtering, DF election, and aliasing and backup on remote EVPN-MPLS PEs.

In addition to the EVPN multihoming features, the main advantages of the I-ES redundant solution compared to the redundant solution described in Anycast redundant solution for dual BGP-instance services are as follows:

• The use of I-ES for redundancy in dual BGP-instance services allows local SAPs on the DCGWs.

• P2MP mLDP can be used to transport BUM traffic between DCs that use I-ES without any risk of packet duplication. As described in Using P2MP mLDP in redundant anycast DCGWs, packet duplication may occur in the anycast DCGW solution when mLDP is used in the WAN.

Where EVPN-MPLS networks are interconnected to EVPN-VXLAN networks, the I-ES concept applies only to the access VXLAN network; the EVPN-MPLS network does not modify its existing behavior.

The Interconnect ES concept shows the use of I-ES for Layer 2 EVPN DCI between VXLAN and MPLS networks.

The following example shows how I-ES-1 would be provisioned on DCGW1 and the association between I-ES to a specified VPLS service. A similar configuration would occur on DCGW2 in the I-ES.

New I-ES configuration:

DCGW1#config>service>system>bgp-evpn# 
ethernet-segment I-ES-1 virtual create
 esi 01:00:00:00:12:12:12:12:12:00
 service-carving
  mode auto
 multi-homing all-active
 network-interconnect-vxlan 1
 service-id
   service-range 1 to 1000
 no shutdown

Service configuration:

DCGW1#config>service>vpls(1)# 
 vxlan instance 1 vni 1 instance 1 create
 exit
bgp
 route-distinguisher 1:1 
bgp 2
 route-distinguisher 2:2               
bgp-evpn        
  evi 1
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any                     
   no shutdown

...

DCGW1#config>service>vpls(2)#  
 vxlan instance 1 vni 2 create
 exit
bgp
 route-distinguisher 3:3 
bgp 2
 route-distinguisher 4:4               
 bgp-evpn        
  evi 2
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any   
   no shutdown
 sap 1/1/1:1 create
 exit 

The above configuration associates I-ES-1 to the VXLAN instance in services VPLS1 and VPLS 2. The I-ES is modeled as a virtual ES, with the following considerations:

• The commands network-interconnect-vxlan and service-id service-range svc-id [to svc-id] are required within the ES.

  • The network-interconnect-vxlan parameter identifies the VXLAN instance associated with the virtual ES. The value of the parameter must be set to 1. This command is rejected in a non-virtual ES.

  • The service-range parameter associates the specific service range to the ES. The ES must be configured as network-interconnect-vxlan before any service range can be added.

  • The ES parameters port, lag, sdp, vc-id-range, dot1q, and qinq cannot be configured in the ES when a network-interconnect-vxlan instance is configured. The source-bmac-lsb option is blocked, as the I-ES cannot be associated with an I-VPLS or PBB-Epipe service. The remaining ES configuration options are supported.

  • All services with two BGP instances associate the VXLAN destinations and ingress VXLAN instances to the ES.

• Multiple services can be associated with the same ES, with the following considerations:

  • In a DC with two DCGWs (as in The Interconnect ES concept), only two I-ESs are needed to load-balance, where one half of the dual BGP-instance services would be associated with one I-ES (for example, I-ES-1, in the above configuration) and one half to another I-ES.

  • Up to eight service ranges per VXLAN instance can be configured. Ranges may overlap within the same ES, but not between different ESs.

  • The service range can be configured before the service.

• After the I-ES is configured using network-interconnect-vxlan, the ES operational state depends exclusively on the ES administrative state. Because the I-ES is not associated with a physical port or SDP, when testing the non-revertive service carving manual mode, an ES shutdown and no shutdown event results in the node sending its own administrative preference and DP bit and taking control if the preference and DP bit are higher than the current DF. This is because the peer ES routes are not present at the EVPN application layer when the ES is configured for no shutdown; therefore, the PE sends its own administrative preference and DP values. For I-ESs, the non-revertive mode works only for node failures.

• A VXLAN instance may be placed in MhStandby under any of the following situations:

  • if the PE is single-active NDF for that I-ES

  • if the VXLAN service is added to the I-ES and either the ES or BGP-EVPN MPLS is shut down in all the services included in the ES

    The following example shows the change of the MhStandby flag from false to true when BGP-EVPN MPLS is shut down for all the services in the I-ES.

    A:PE-4# show service id 500 vxlan instance 1 oper-flags 
    ===============================================================================
    VPLS VXLAN oper flags
    ===============================================================================
    MhStandby                              : false
    ===============================================================================
    A:PE-4# configure service vpls 500 bgp-evpn vxlan shutdown 
    *A:PE-4# show service id 500 vxlan instance 1 oper-flags    
    ===============================================================================
    VPLS VXLAN oper flags
    ===============================================================================
    MhStandby                              : true
    ===============================================================================
    

If two BGP instances with the same encapsulation (VXLAN or MPLS) are configured in the same VPLS/R-VPLS service, different import route targets in each BGP instance are mandatory (although this is not enforced).

BGP-EVPN routes in services configured with two BGP instances describes the use of policies to avoid sending WAN routes (routes meant to be redistributed from DC to WAN) to the DC again and DC routes (routes meant to be redistributed from WAN to DC) to the WAN again. Those policies are based on export policy statements that match on the RFC 9012 BGP encapsulation extended community (MPLS and VXLAN respectively).

When the two BGP instances are of the same encapsulation (VXLAN or MPLS), the policies matching on different BGP encapsulation extended community are not feasible because both instances advertise routes with the same encapsulation value. Because the export route targets in the two BGP instances must be different, the policies, to avoid sending WAN routes back to the WAN and DC routes back to the DC, can be based on export policies that prevent routes with a DC route target from being sent to the WAN peers (and opposite for routes with a WAN route target).

In scaled scenarios, matching based on route targets, does not scale well. An alternative and preferred solution is to configure a default-route-tag that identifies all the EVPN instances connected to the DC (or one domain), and a different default-route-tag in all the EVPN instances connected to the WAN (or the other domain). Anycast redundant solution for multi-instance EVPN services with the same encapsulation shows an example that demonstrates the use of default-route-tags.

Other than the specifications described in this section, the processing of MAC/IP routes and inclusive multicast Ethernet tag routes in multi-instance EVPN services of the same encapsulation follow the rules described in BGP-EVPN routes in services configured with two BGP instances.

The solution described in Anycast redundant solution for dual BGP-instance services is also supported in multi-instance EVPN VPLS/R-VPLS services with the same encapsulation.

The following CLI example output shows the configuration of DCGW-1 and DCGW-2 in Multihomed anycast solution where VPLS 500 is a multi-instance EVPN-VXLAN service and BGP instance 2 is associated with VXLAN instead of MPLS.

Different default-route-tags are used in BGP instance 1 and instance 2, so that in the export route policies, DC routes are not advertised to the WAN, and WAN routes are not advertised to the DC, respectively.

*A:DCGW-1(and DCGW-2)>config>service>vpls(500)# info
----------------------------------------------
vxlan instance 1 vni 500 create
exit
vxlan instance 2 vni 501 create
exit
bgp
 route-distinguisher 192.0.2.2:500
 route-target target:64500:500
exit
bgp 2
 route-distinguisher 192.0.2.2:501
 route-target target:64500:501
exit
bgp-evpn
 incl-mcast-orig-ip 23.23.23.23
 evi 500
 vxlan bgp 1 vxlan-instance 1
 default-route-tag 500
 no shutdown
 exit
 vxlan bgp 2 vxlan-instance 2
   default-route-tag 501
   no shutdown
 exit
exit
stp
shutdown
exit
no shutdown
----------------------------------------------
config>router>bgp#
 vpn-apply-import
 vpn-apply-export
 group "WAN"
  family evpn
  type internal
  export "allow only mpls"
  neighbor 192.0.2.6
 group "DC"
  family evpn
  type internal
export "allow only vxlan"
  neighbor 192.0.2.2

config>router>policy-options# info
----------------------------------------------
            policy-statement "allow only mpls"
                entry 10
                  from
                    family evpn 
                    tag 500 
                  action drop
                  exit
                exit
            exit
            policy-statement "allow only vxlan"
                entry 10
                  from
                    family evpn 
                    tag 501 
                  action drop
       exit
                  exit
            exit

The same Anycast redundant solution can be applied to VPLS/R-VPLS with two instances of EVPN-MPLS encapsulation. The configuration would be identical, other than replacing the VXLAN aspects with the EVPN-MPLS-specific parameters.

For a full description of this solution, see the Anycast redundant solution for dual BGP-instance services

The I-ES of network-interconnect VXLAN Ethernet segment is described in I-ES solution for dual BGP instance services. I-ES’s are also supported on VPLS and R-VPLS services with two EVPN-VXLAN instances.

I-ES in dual EVPN-VXLAN services shows the use of an I-ES in a dual EVPN-VXLAN instance service.

Similar to (single-instance) EVPN-VXLAN all-active multihoming, the BUM forwarding procedures follow the ‟Local Bias” behavior.

At the ingress PE, the forwarding rules for EVPN-VXLAN services are as follows:

• The no send-imet-ir-on-ndf or rx-discard-on-ndf bum command must be enabled so that the NDF does not forward any BUM traffic.

• BUM frames received on any SAP or I-ES VXLAN binding are flooded to:

  • local non-ES and single-active DF ES SAPs

  • local all-active ES SAPs (DF and NDF)

  • EVPN-VXLAN destinations

    BUM received on an I-ES VXLAN binding follows SHG rules, for example, it can only be forwarded to EVPN-VXLAN destinations that belong to the other VXLAN instance (instance 2), which is a different SHG.

• As an example, in I-ES in dual EVPN-VXLAN services:

  • GW1 and GW2 are configured with no send-imet-ir-on-ndf.

  • TOR1 generates BUM traffic that only reaches GW1 (DF).

  • GW1 forwards to CE1 and EVPN-VXLAN destinations.

The forwarding rules at the egress PE are as follows:

• The source VTEP is looked up for BUM frames received on EVPN-VXLAN.

• If the source VTEP matches one of the PEs with which the local PE shares an ES _AND_ a VXLAN service:

  • Then the local PE does not forward to the shared local ES’es (this includes port, lag, or network-interconnect-vxlan ES’es). It forwards though to non-shared ES SAPs unless they are in NDF state.

  • Else, the local PE forwards normally to local ES’es unless they are in NDF state.

• Because there is no multicast label or multicast B-MAC in VXLAN, the only way the egress PE can identify BUM traffic is by looking at the customer MAC DA. Therefore, BM or unknown MAC DAs identify BUM traffic.

• As an example, in I-ES in dual EVPN-VXLAN services:

  • GW2 receives BUM on EVPN-VXLAN. GW2 identifies the source VTEP as a PE with which the I-ES-1 is shared, therefore it does not forward the BUM frames to the local I-ES. It forwards to the non-shared ES and local SAPs though (CE2).

  • GW3 receives BUM on EVPN-VXLAN, however the source VTEP does not match any PE with which GW3 shares an ES. Hence GW3 forwards to all local ES’es that are DF, in other words, CE3.

The following configuration example shows how I-ES-1 would be provisioned on DCGW1 and the association between I-ES to a specified VPLS service. A similar configuration would occur on DCGW2 in the I-ES.

I-ES configuration:

*A:GW1>config>service>system>bgp-evpn>eth-seg# info 
----------------------------------------------
                esi 00:23:23:23:23:23:23:00:00:01
                service-carving
                    mode manual
                    manual
                        preference non-revertive create
                            value 150
                        exit
                        evi 101 to 200
                    exit
                exit
                multi-homing all-active
                network-interconnect-vxlan 1
                service-id
                    service-range 1
                    service-range 1000 to 1002
                    service-range 2000
                exit
                no shutdown

Service configuration:

*A:GW1>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 1000 create
                rx-discard-on-ndf bum
            exit
            vxlan instance 2 vni 1002 create
            exit
            bgp
                route-target export target:64500:1000 import target:64500:1000
            exit
            bgp 2
                route-distinguisher auto-rd
                route-target export target:64500:1002 import target:64500:1002
            exit
            bgp-evpn
                evi 1000
                vxlan bgp 1 vxlan-instance 1
                    ecmp 2
                    default-route-tag 100
                    auto-disc-route-advertisement
                    no shutdown
                exit
                vxlan bgp 2 vxlan-instance 2
                    ecmp 2
                    default-route-tag 200
                    auto-disc-route-advertisement
                    mh-mode network
                    no shutdown
                exit
            exit
            no shutdown 

Multi-instance EVPN VPLS/R-VPLS services with two EVPN-MPLS instances do not support I-ESs.

For information about how the EVPN routes are processed and advertised in an I-ES, see the I-ES solution for dual BGP instance services.

The SBD is equivalent to an R-VPLS that connects all the PEs that are attached to the same tenant VPRN. Interface-ful refers to the fact that there is a full IRB interface between the VPRN and the SBD (an interface object with MAC and IP addresses, over which interface parameters can be configured).

Interface-ful IP-VRF-to-IP-VRF with SBD IRB model illustrates this model.

Interface-ful IP-VRF-to-IP-VRF with SBD IRB model shows a 7750 SR and a third-party router using interface-ful IP-VRF-to-IP-VRF with SBD IRB model. The two routers are attached to a VPRN for the same tenant, and those VPRNs are connected by R-VPLS-2, or SBD. Both routers exchange IP prefix routes with a non-zero gateway IP (this is the IP address of the SBD IRB). The SBD IRB MAC and IP are advertised in a MAC/IP route. On reception, the IP prefix route creates a route-table entry in the VPRN, where the gateway IP must be recursively resolved to the information provided by the MAC/IP route and installed in the ARP and FDB tables.

This model is described in detail in EVPN for VXLAN in IRB backhaul R-VPLS services and IP prefixes. As an example, and based on Interface-ful IP-VRF-to-IP-VRF with SBD IRB model above, the following CLI output shows the configuration of a 7750 SR SBD and VPRN, using on this interface-ful with SBD IRB mode:

7750SR#config>service#

vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
 evi 2
 ip-route-advertisement
 mpls bgp 1
 auto-bind-tunnel resolution any
 no shutdown

vprn 1 customer 1 name "vprn1" create
 route-distinguisher auto-rd
 interface "sbd" create
 address 192.168.0.1/16
 ipv6
   30::3/64
 exit
 vpls "sbd"

The model is, also, supported for IPv6 prefixes. There are no configuration differences except the ability to configure an IPv6 address and interface.

Interface-ful refers to the fact that there is a full IRB interface between the VPRN and the SBD. However, the SBD IRB is unnumbered in this model, which means no IP address is configured on it. In SR OS, an unnumbered SBD IRB is equivalent to an R-VPLS linked to a VPRN interface through an EVPN tunnel. See EVPN for VXLAN in EVPN tunnel R-VPLS services for more information.

Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model illustrates this model.

Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model shows a 7750 SR and a third-party router running interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model. The IP prefix routes are now expected to have a zero gateway IP and the MAC in the router's MAC extended community used for the recursive resolution to a MAC/IP route.

The corresponding configuration of the 7750 SR VPRN and SBD in the example could be:

7750SR#config>service#

vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
evi 2
ip-route-advertisement
mpls bgp 1
auto-bind-tunnel resolution any
no shutdown

vprn 1 customer 1 create
 route-distinguisher auto-rd
 interface "sbd" create
  ipv6
  exit
 vpls "sbd" 
  evpn-tunnel ipv6-gateway-address mac

Note that the evpn-tunnel command controls the use of the Router's MAC extended community and the zero gateway IP in the IPv4-prefix route. For IPv6, the ipv6-gateway-address mac option makes the router advertise the IPv6-prefix routes with a Router's MAC extended community and zero gateway IP.

This model is interface-less because no Supplementary Broadcast Domain (SBD) is required to connect the VPRNs of the tenant, and no recursive resolution is required upon receiving an IP prefix route. In other words, the next-hop of the IP prefix route is directly resolved to an EVPN tunnel, without the need for any other route. The standard specification draft-ietf-bess-evpn-ip-prefix supports two variants of this model that are not interoperable with each other:

• EVPN IFL for Ethernet NVO (Network Virtualization Overlay) tunnels

  Ethernet NVO indicates that the EVPN packets contain an inner Ethernet header. This is the case for tunnels such as VXLAN.

  In the Ethernet NVO option, the ingress PE uses the received router's MAC extended community address (received along with the route type 5) as the inner destination MAC address for the EVPN packets sent to the prefix

• EVPN IFL for IP NVO tunnels

  IP NVO indicates that the EVPN packets contain an inner IP packet, but without Ethernet header. This is similar to the IPVPN packets exchanged between PEs.

Interface-less IP-VRF-to-IP-VRF model illustrates the Interface-less IP-VRF-to-IP-VRF model.

SR OS supports the interoperable Interface-less IP-VRF-to-IP-VRF Model for Ethernet NVO tunnels. In Interface-less IP-VRF-to-IP-VRF model this interoperable model is shown on the left side PE router. The following is the model implementation:

• There is no data path difference between this model and the existing R-VPLS EVPN tunnel model or the model described in Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model.

• This model is enabled by configuring config>service>vprn>if>vpls>evpn-tunnel (with ipv6-gateway-address mac for IPv6), and bgp-evpn>ip-route-advertisement. In addition, because the SBD IRB MAC/IP route is no longer needed, the bgp-evpn no mac-advertisement command prevents the advertisement of the MAC/IP route.

• The IP prefix routes are processed as follows:

  • On transmission, there is no change in the IP prefix route processing compared to the configuration of the Interface-ful IP-VRF-to-IP-VRF with Unnumbered SBD IRB Model.

    • IPv4/IPv6 prefix routes are advertised based on the information in the route-table for IPv4 and IPv6, with GW-IP=0 and the corresponding MAC extended community.

    • If bgp-evpn no mac-advertisement is configured, no MAC/IP route is sent for the R-VPLS.

  • The received IPv4/IPv6 prefix routes are processed as follows:

    1. Upon receiving an IPv4/IPv6 prefix route with a MAC extended community for the router, an internal MAC/IP route is generated with the encoded MAC and the RD, Ethernet tag, ESI, Label/VNI and next hop derived from the IP prefix route itself.

    2. If no competing received MAC/IP routes exist for the same MAC, this IP prefix-derived MAC/IP route is selected and the MAC is installed in the R-VPLS FDB with type ‟Evpn”.

    3. After the MAC is installed in FDB, there are no differences between this interoperable interface-less model and the interface-ful with unnumbered SBD IRB model. Therefore, SR OS is compatible with the received IP prefix routes for both models.

The following is an example of a typical configuration of a PE's SBD and VPRN that work in interface-less model for IPv4 and IPv6:

7750SR#config>service#

vpls 2 customer 1 name "sbd" create
       allow-ip-int-bind
       exit
       bgp
       exit
       bgp-evpn
            evi 2
            no mac-advertisement
            ip-route-advertisement
            mpls bgp 1
                 auto-bind-tunnel resolution any
                 no shutdown
vprn 1 customer 1 create
       route-distinguisher auto-rd
       interface "sbd" create
            ipv6
            exit
            vpls "sbd"
                 evpn-tunnel ipv6-gateway-address mac

In addition to the Interface-ful and interoperable Interface-less models described in the previous sections, SR OS also supports Interface-less Model (EVPN IFL) with IP encapsulation for MPLS tunnels. In the standard specification - draft-ietf-bess-evpn-ip-prefix - this refers to the EVPN IFL model for IP NVO tunnels.

Compared to the Ethernet NVO option, the ingress PE no longer pushes an inner Ethernet header, but the IP packet is directly encapsulated with an EVPN service label and the transport labels.

Interface-less IP-VRF-to-IP-VRF model for IP encapsulation in MPLS tunnels illustrates the Interface-less Model (EVPN IFL) with IP encapsulation for MPLS tunnels.

EVPN IFL uses EVPN IP Prefix routes to exchange prefixes between PEs without the need for an R-VPLS service, termed Supplementary Broadcast Domain (SBD) in the standards, and any destination MAC lookup. The data path used in EVPN IFL is the same as that is used for IP-VPN services in the VPRN.

In the example of Interface-less IP-VRF-to-IP-VRF model for IP encapsulation in MPLS tunnels:

1. PE2 advertises IP Prefix 20.0/24 (shorthand for 20.0.0.0/24) in an EVPN IP Prefix route that does not contain a Router's MAC extended community anymore. As usual, and depicted in step 1, arriving frames with IP destination of 20.0.0.1 on PE1's R-VPLS-1 are processed for a route lookup on VPRN-1.

2. However, in step 2 and as opposed to the previous models, the lookup yields a route-table entry that does not point at an SBD R-VPLS, but rather to an MPLS tunnel terminated on PE2. PE1 then pushes the EVPN service label that was received on the IP Prefix route at the top of the IP packet, and the packet is sent on the wire without any inner Ethernet header.

3. In step 3, the MPLS tunnel is terminated on PE2 and the EVPN label identifies the VPRN-1 service for a route lookup.

4. Step 4 corresponds to the regular R-VPLS forwarding that happens in the other EVPN L3 models.

A new vprn>bgp-evpn>mpls context has been added to configure a VPRN service for EVPN IFL. This context is like the one existing in VPLS and Epipe services and enables the use of EVPN IFL in the VPRN service. When configured, no R-VPLS with evpn-tunnel should be added to the VPRN, that is, no SBD is configured. As an example, in Interface-less IP-VRF-to-IP-VRF model for IP encapsulation in MPLS tunnels PE1 and PE2 VPRN-1 service are configured as follows:

[ex:configure service vprn "vprn-1"]
A:admin@PE1# info
    admin-state enable
    ecmp 2
    bgp-evpn {
        mpls 1 {
            admin-state enable
            route-distinguisher "192.0.2.1:12"
            vrf-target {
                community "target:64500:2"
            }
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    interface "irb-1" {
        ipv4 {
            primary {
                address 10.0.0.254
                prefix-length 24
            }
        }
        vpls "r-vpls-1" {
        }
    }

[ex:configure service vprn "vprn-1"]
A:admin@PE2# info
    admin-state enable
    ecmp 2
    bgp-evpn {
        mpls 1 {
            admin-state enable
            route-distinguisher "192.0.2.2:21"
            vrf-target {
                community "target:64500:2"
            }
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    interface "irb-2" {
        ipv4 {
            primary {
                address 20.0.0.254
                prefix-length 24
            }
        }
        vpls "r-vpls-1" {
        }
    }

Host mobility within the same R-VPLS – initial phase shows an example of a host connected to a source PE, PE1, that moved to the target, PE2. The figure shows the expected configuration on the VPRN interface, where R-VPLS 1 is attached (for both PE1 and PE2). PE1 and PE2 are configured with an ‟anycast gateway”, that is, a VRRP passive instance with the same backup MAC and IP in both PEs.

In this initial phase:

1. PE1 learns Host-1 IP to MAC (10.1-M1) in the ARP table and generates a host route (RT5) for 10.1/32, because Host-1 is locally connected to PE1. In particular:

   • arp-learn-unsolicited triggers the learning of 10.1-M1 upon receiving a GARP from Host-1 or any other ARP

   • arp-proactive-refresh triggers the refresh of host-1’s ARP entry 30 seconds before the entry ages out

   • local-proxy-arp makes sure PE1 replies to any received ARP request on behalf of other hosts in the R-VPLS

   • arp-host-route populate dynamic ensures that only the dynamically learned ARP entries create a host route, for example, 10.1

   • no flood-garp-and-unknown-req suppresses ARP flooding (from CPM) within the R-VPLS1 context and reduces significantly the unnecessary ARP flooding because the ARP entries are synchronized through EVPN

   • advertise dynamic triggers the advertisement of MAC/IP routes for the dynamic ARP entries, including the IP and MAC addresses, for example, 10.1-M1; a MAC/IP route for M1-only that has been previously advertised as M1 is learned on the FDB as local or dynamic

2. PE2 learns Host-1 10.1-M1 in the ARP and FDB tables as EVPN type. PE2 must not learn 10.1-M1 as dynamic, so that PE2 is prevented from advertising an RT5 for 10.1/32. If PE2 advertises 10.1/32, then PE3 could select PE2 as the next-hop to reach Host-1, creating an unwanted hair-pinning forwarding behavior. PE2 is expected to have the same configuration as PE1, including the following commands, as well as those described for PE1:

   • no learn-dynamic prevents PE2 from learning ARP entries from ARP traffic received on an EVPN tunnel.

   • populate dynamic, as in PE1, makes sure PE2 only creates route-table ARP-ND host routes for dynamic entries. Hence, 10.1-M1 does not create a host route as long as it is learned via EVPN only.

The configuration described in this section and the cases in the following sections are for IPv4 hosts, however, the functionality is also supported for IPv6 hosts. The IPv6 configuration requires equivalent commands, that use the prefix "nd-" instead of "arp-". The only exception is the flood-garp-and-unknown-req command, which does not have an equivalent command for ND.

SR OS uses Eth-CFM fault-propagation to support CE-to-CE fault propagation in EVPN-VPWS services. That is, upon detecting a CE failure, an EVPN-VPWS PE withdraws the corresponding Auto-Discovery per-EVI route, which then triggers a down MEP on the remote PE that signals the fault to the connected CE. In cases where the CE connected to EVPN-VPWS services does not support Eth-CFM, the fault can be propagated to the remote CE by using LAG standby-signaling, which can be LACP-based or simply power-off.

Link loss forwarding for EVPN-VPWS shows an example of link loss forwarding for EVPN-VPWS.

In this example, PE1 is configured as follows:

A:PE1>config>lag(1)# info 
----------------------------------------------
mode access
encap-type null 
port 1/1/1
port 1/1/2
standby-signaling power-off
monitor-oper-group "llf-1"
no shutdown
----------------------------------------------
*A:PE1>config>service>epipe# info
----------------------------------------------
bgp
exit
bgp-evpn
    evi 1
    local-attachment-circuit ac-1  
        eth-tag 1
        exit
    remote-attachment-circuit ac-2 
        eth-tag 2
        exit
    mpls bgp 1
        oper-group "llf-1"
        auto-bind-tunnel
            resolution any
        exit
        no shutdown
    exit
sap lag-1 create
no shutdown
exit
no shutdown

The following applies to the PE1 configuration:

• The EVPN Epipe service is configured on PE1 with a null LAG SAP and the oper-group ‟llf-1” under bgp-evpn>mpls. This is the only member of oper-group ‟llf-1”.

  Note: Do not configure the oper-group under config>service>epipe, because circular dependencies are created when the access SAPs go down because of the LAG monitor-oper-group command.

• The operational group monitors the status of the BGP-EVPN instance in the Epipe service. The status of the BGP-EVPN instance is determined by the existence of an EVPN destination at the Epipe.

• The LAG, in access mode and encap-type null, is configured with the command monitor-oper-group ‟llf-1”.

  Note: The configure>lag>monitor-oper-group name command is only supported in access mode. Any encap-type can be used.

As shown in Link loss forwarding for EVPN-VPWS, upon failure on CE2, the following events occur:

1. PE2 withdraws the EVPN route.

2. The EVPN destination is removed in PE1 and oper-group ‟llf-1” also goes down.

3. Because lag-1 is monitoring ‟llf-1”, the oper-group that is becoming inactive triggers standby signaling on the LAG; that is, power-off or LACP out-of-sync signaling to the CE1.

   When the SAP or port is down because of the LAG monitoring of the oper-group, PE1 does not trigger an AD per-EVI route withdrawal, even if the SAP is brought operationally down.

4. After CE2 recovers and PE2 re-advertises the AD per-EVI route, PE1 creates the EVPN destination and oper-group ‟llf-1” comes up. As a result, the monitoring LAG stops signaling standby and the LAG is brought up.

Core isolation blackhole avoidance shows how blackholes can be avoided when a PE becomes isolated from the core.

In this example, consider that PE2 and PE1 are single-active multihomed to CE1. If PE2 loses all its core links, PE2 must somehow notify CE1 so that PE2 does not continue attracting traffic and so that PE1 can take over. This notification is achieved by using oper-groups under the BGP-EVPN instance in the service. The following is an example output of the PE2 configuration.

*[ex:configure service vpls ”evi1"]
A:admin@PE-2# info
    admin-state enable 
    bgp-evpn { 
        evi 1 
        mpls 1 { 
            admin-state enable 
            oper-group ‟evpn-mesh”
            auto-bind-tunnel { 
                resolution any 
            } 
        } 
    } 
    sap lag-1:351 { 
        monitor-oper-group ‟evpn-mesh”
            } 
*[ex:configure service oper-group ”evpn-mesh"]
A:admin@PE-2# info detail 
    hold-time { 
        up 4 
    } 

With the PE2 configuration and Core isolation blackhole avoidance example, the following steps occur:

1. PE2 loses all its core links, therefore, it removes its EVPN-MPLS destinations. This causes oper-group ‟evpn-mesh” to go down.

2. Because PE2 is the DF in the Ethernet Segment (ES) ES-1 and sap lag-1:351 is monitoring the oper-group, the SAP becomes operationally down. If ETH-CFM fault propagation is enabled on a down MEP configured on the SAP, CE1 is notified of the failure.

3. PE1 takes over as the DF based on the withdrawal of the ES (and AD) routes from PE2, and CE1 begins sending traffic immediately to PE1 only, therefore, avoiding a traffic blackhole.

Generally, when oper-groups are associated with EVPN instances:

• The oper-group state is determined by the existence of at least one EVPN destination in the EVPN instance.

• The oper-group that is configured under a BGP EVPN instance cannot be configured under any other object (for example, SAP, SDP binding, and so on) of the same or different service.

• The status of an oper-group associated with an EVPN instance goes down in the following cases:

  • the service admin-state is disabled (only for VPLS services, not for Epipes)

  • the BGP EVPN VXLAN or MPLS admin-state are disabled

  • there are no EVPN destinations associated with the instance

As described in EVPN for MPLS tunnels, EVPN single-active multihoming PEs that are elected as non-DF must notify their attached CEs so the CE does not send traffic to the non-DF PE. This can be performed on a per-service basis that is based on the ETH-CFM and fault-propagation. However, sometimes ETH-CFM is not supported in multihomed CEs and other notification mechanisms are needed, such as LACP standby or power-off. This scenario is shown in the following figure.

As shown in the preceding figure, the multihomed PEs are configured with multiple EVPN services that use ES-1. ES-1 and its associated LAG is configured as follows:

*[ex:configure lag 1]
A:admin@PE-2# info
    admin-state enable 
    standby-signaling {power-off|lacp} 
    monitor-oper-group ”DF-signal-1" 
    mode access 
    port 1/1/c2/1 { 
    }
<snip>
ex:configure service system bgp evpn]
A:admin@PE-2# info 
    ethernet-segment "ES-1" { 
        admin-state enable 
        esi 0x01010000000000000000 
        multi-homing-mode single-active 
        oper-group ‟DF-signal-1” 
        association { 
            lag 1 { 
            } 
<snip>

When the operational group is configured on the ES and monitored on the associated LAG:

• The operational group status is driven by the ES DF status (defined by the number of DF SAPs or oper-up SAPs owned by the ES).

• The operational group goes down if all the SAPs in the ES go down (this happens in PE2 in LACP standby signaling from the non-DF). The ES operational group goes up when at least one SAP in the ES goes up.

  As a result, if PE2 becomes non-DF on all the SAPs in the ES, they all go operationally down, including the ES-1 operational group.

• Because LAG-1 is monitoring the operational group, when its status goes down, LAG-1 signals LAG standby state to the CE. The standby signaling can be configured as LACP or power-off.

• The ES and AD routes for the ES are not withdrawn because the router recognizes that the LAG becomes standby for the ES operational group.

If the Single-Active ES is associated with a port instead of a LAG, the config>port> monitor-oper-group DF-signal-1 command can be configured. In this case, the port monitors the ES operational group and the following rules apply:

• As in the case of the LAG, if the ES goes non-DF, its operational group also goes down.
• The port that is monitoring the ES operational group signals standby state by powering off the port itself.
• As in the case of the LAG, the ES and AD routes for the ES are not withdrawn because the router recognizes that the port is in standby state because of the ES operational group.

Operational groups cannot be assigned to ESs that are configured as virtual, all-active or service-carving mode auto.

The Attachment Circuit Influenced (AC-Influenced) Designated Forwarder Election Capability (AC-DF), as described in RFC8584, is supported in SR OS. By default, the ac-df-capability command is set to the include option. This configuration addresses the need to consider EVPN Auto-discovery per EVI/ES (AD per EVI/ES) routes for a specific PE, which ensures that the PE is included on the candidate DF list.

Configuring ac-df-capability to exclude disables the AC-DF capability. When ac-df-capability exclude is configured on a specific ES, the presence or absence of the AD per EVI/ES routes from the ES peers does not modify the DF Election candidate list for the ES. The exclude option is recommended in ESs that use an oper-group, that is monitored by the access LAG, to signal standby lacp or power-off, as described in LAG or port standby signaling to the CE on non-DF EVPN PEs (single-active). All PE routers attached to the same ES must be configured consistently for the specific ac-df-capability.

EVPN OISM is similar to Multicast VPNs (MVPN) in some aspects, because it does IP multicast routing in VPNs, uses MP-BGP to signal the interest of a PE in a specified multicast group and uses Provider Multicast Service Interface (PMSI) trees among the PEs to send and receive the IP multicast traffic.

However, OISM is simpler than MVPN and allows efficient multicast in networks that integrate Layer 2 and Layer 3; that is, networks where PEs may be attached to different subnets, but could also be attached to the same subnet.

OISM is simpler than MVPN since:

• it does not need to setup shared trees (that need to switchover to shortest path trees)

• it does not require of the MVPN Any Source Multicast (ASM) complex procedures or the Rendezvous Point (RP) function

• it does not require Upstream Multicast Hop (UMH) selection and therefore does not have the UMH potential issues and limitations described in RFC6513 and RFC6514

• multiple PEs can be attached to the same Receiver subnet or Source subnet, which provides full flexibility when designing the multicast network

EVPN OISM is defined by draft-ietf-bess-evpn-irb-mcast and uses the following terminology that is also used in the rest of this section:

BD with IRB

Broadcast Domain with an Integrated Routing and Bridging interface. It is an R-VPLS service in SR OS.

Ordinary BD

refers to an R-VPLS where sources and, or receivers are connected

SBD

Supplementary Broadcast Domain. It is a backhaul R-VPLS that connects the PEs' VPRN services and is configured as an evpn-tunnel interface in the VPRN services. The SBD is mandatory in OISM and is needed to receive multicast traffic on the PEs that are not attached to the source ordinary BD.

EVPN Tenant Domain

refers to the group of BDs and IP-VRFs (VPRNs) of the same tenant

SMET route or EVPN route type 6

the EVPN route that the PEs use to signal interest for a specific multicast group (S ,G) or (*,G)

IIF and OIF

refers to Incoming Interface and Outgoing Interface. A multicast enabled VPRN has Layer 3 IIF and OIFs. A multicast enabled R-VPLS have Layer 2 OIFs.

Upstream and Downstream PEs

refers to the PEs that are connected to sources and receivers respectively

In an EVPN OISM network, it is assumed that the sources and receivers are connected to ordinary BDs and EVPN is the only multicast control plane protocol used among the PEs. Also, the subnets (and optionally hosts) are advertised normally by the EVPN IP Prefix routes. The IP-Prefix routes are installed in the PEs' VPRN route tables and are used for multicast RPF checks when routing multicast packets. EVPN OISM forwarding plane illustrates a simple EVPN OISM network.

In EVPN OISM forwarding plane, and from the perspective of the multicast flow (S1,G1), PE1 is considered an upstream PE, whereas PE2 and PE3 are downstream PEs. The OISM forwarding rules are as follows.

• On the upstream PE (PE1), the multicast traffic is sent to local receivers irrespective of the receivers being attached to the source BD (BD1) or not (BD2).

  Note: OISM does not use any multicast Designated Router (DR) concept, therefore the upstream PE always routes locally as long as it has local receivers.

• On downstream PEs that are attached to the source BD (PE2), the multicast traffic is always received on the source BD (BD1) and forwarded locally to receivers in the same or different ordinary BD (as in the case of Receiver-22 or Receiver-21). Multicast traffic received on this PE is never sent back to the SBD or remote EVPN PEs.

• On downstream PEs that are not attached to the source BD (PE3), the multicast traffic is always received on the SBD and sent to local receivers. Multicast received on this PE is never sent to remote EVPN PEs.

  Note: In order for PE3 to receive the multicast traffic on the SBD, the source PE, PE1, forms an EVPN destination from BD1 to PE3's SBD. This EVPN destination on PE1 is referred to as an SBD destination.

OISM uses the Selective Multicast Ethernet Tag (SMET) route or route type 6 to signal interest on a specific (S,G) or (*,G). Use of the SMET route provides an example.

As shown in Use of the SMET route, a PE with local receivers interested in a multicast group G1 issues an SMET route encoding the source and group information (upon receiving local IGMP join messages for that group). EVPN OISM uses the SMET route in the following way:

• A route type-6 (SMET) can carry information for IPv4 or IPv6 multicast groups, for (S,G) or (*,G) or even wildcard groups (*,*).

  Note: MVPN uses different route types or even families to address the different multicast group types.

• The SMET routes are advertised with the route-target of the SBD, that guarantees that the SMET routes are imported by all the PEs of the tenant.

• The SMET routes also help minimize the control plane overhead because they aggregate the multicast state created on the downstream PEs. This is illustrated in Use of the SMET route, where PE2 sends the minimum number of SMET routes to pull multicast traffic for G1. That is, if PE2 has state for (S1,G1) and (*,G1), the SMET route for (*,G1) is enough to attract the multicast traffic required by the local receivers. There is no need to send an SMET route for (S1,G1) and a different route for (*,G1). Only (*,G1) SMET route is advertised.

• The SMET routes also provide an "implicit" S-PMSI (Selective Provider Multicast Service Interface) tree. That is, PE1 sends the multicast traffic only to the PEs requesting it, for example, PE2 and not to PE3. In MVPN, even for Ingress Replication, a separate S-PMSI tree is setup to avoid PE1 from sending multicast to PE3.

EVPN OISM supports multihomed multicast sources and receivers.

While MVPN requires complex UMH (Upstream Multicast Hop) selection procedures to provide multihoming for sources, EVPN simply reuses the existing EVPN multihoming procedures. EVPN OISM and multihomed sources illustrates an example of a multihomed source that makes use of EVPN all-active multihoming.

The source S1 is attached to a switch SW1 that is connected via single LAG to PE1 and PE2, a pair of EVPN OISM PEs. PE1 and PE2 define Ethernet Segment ES-1 for SW1, where ES-1 is all-active in this case (single-active multihoming being supported too). Even in case of all-active, the multicast flow for (S1,G1) is only sent to one OISM PE, and the regular all-active multihoming procedures (Split-Horizon) make sure that PE3 does not send the multicast traffic back to SW1. This is true for EVPN-MPLS and EVPN-VXLAN BDs.

Convergence, in case of failure, is very fast because the downstream PEs, for example, PE2, advertise the SMET route for (*,G1) with the SBD route target and it is imported by both PE1 and PE3. In case of failure on PE2, PE3 already has state for (*,G1) and can forward the multicast traffic immediately.

EVPN OISM also supports multihomed receivers. EVPN OISM and multihomed receivers illustrates an example of multihomed receivers.

Multi-homed receivers as depicted in EVPN OISM and multihomed receivers, require the support of multicast state synchronization on the multihoming PEs to avoid blackholes. As an example, consider that SW1 hashes an IGMP join (*,G1) to PE2, and PE2 adds the ES-1 SAP to the OIF list for (*,G1). Consider PE1 is the ES-1 DF. Unless the (*,G1) state is synchronized on PE1, the multicast traffic is pulled to PE2 only and then discarded. The state synchronization on PE1 pulls the multicast traffic to PE1 too, and PE1 forwards to the receiver using its DF SAP.

In SR OS, the IGMP/MLD-snooping state is synchronized across ES peers using EVPN Multicast Synch routes, as specified in draft-ietf-bess-evpn-igmp-mld-proxy.

The same mechanism must be used in all the PEs attached to the same Ethernet Segment. MCS takes precedence when both mechanisms are simultaneously used.

EVPN Multicast Synch routes are supported as specified in draft-ietf-bess-evpn-igmp-mld-proxy for OISM services. They use EVPN route types 7 and 8, and are known as the Multicast Join Synch and Multicast Leave Synch routes, respectively.

When a PE that is attached to an EVPN Ethernet Segment receives an IGMP or MLD join, it creates multicast state and advertises a Multicast Join Synch route so that the peer ES PEs can synchronize the state. Similarly, when a PE in the Ethernet Segment receives a leave message, it advertises a Multicast Leave Synch route so that all the PEs in the Ethernet Segment can synchronize the Last Member Query procedures.

The Multicast Join Synch route or EVPN route type 7 is similar to the SMET route, but also includes the ESI. The Multicast Join Synch route indicates the multicast group that must be synchronized in all objects of the Ethernet Segment. Multicast join synch route depicts the format of the Multicast Join Synch route.

In accordance with draft-ietf-bess-evpn-igmp-mld-proxy, the following rules pertain:

• All fields except for the Flags are part of the route key for BGP processing purposes.

• Synch routes are resolved by BGP auto-bind resolution, as any other service route.

• The Flags are advertised and processed based on the received igmp or mld report that triggered the advertisement of the route (this includes the versions for IGMP or MLD and Include/Exclude bit for IGMPv3).

• The Route Distinguisher (RD) is the service RD.

• This route is only distributed to the ES peers - it is advertised with the ES-import route target, which limits its distribution to ES peers only.

• In addition, the route is sent with only one EVI-RT extended community. The EVI-RT EC does not use a route target type/sub-type, therefore, it does not affect the distribution of the route, for example, it is not considered for route target constraint filtering; only the ES-import route target is. However, its value is still taken from the configured service route target or evi auto-derived route target.

The Multicast Leave Synch route or EVPN route type 8 indicates the multicast group Leave states that must be synchronized in all objects of the Ethernet Segment. Multicast leave synch route depicts the format of the Multicast Leave Synch route.

In accordance with draft-ietf-bess-evpn-igmp-mld-proxy, the following rules pertain:

• All fields except for the Flags, the Maximum Response Time and ‟reserved” field are part of the route key for BGP processing purposes.

• Synch routes are resolved by BGP auto-bind resolution, as any other service route.

• The Flags are generated based on the version of the leave message that triggered the advertisement of the route.

• As with the Multicast Join Synch route, this is a service level route sent with one ES-import route target and one EVI-RT EC. RD, Flags, ES-import and EVI-RT EC are advertised and processed in the same way as for the Multicast Join Synch route.

The EVI-RT is automatically added to the routes type 7 and 8, depending on the type of route target being configured on the service.

• If the service is configured with target:2byte-asnumber:ext-comm-val as route target, an EVI-RT type 0 is automatically added to routes type 7 and 8. No route target (other than the ES-import route target) is added to the route.

• If the service is configured with target:ip-addr:comm-val as route target, an EVI-RT type 1 is automatically added to routes type 7 and 8. No route target (other than the ES-import route target) is added to the route.

• If the service is configured with target:4byte-asnumber:comm-val as route target, an EVI-RT type 2 is automatically added to routes type 7 and 8. No route target (other than the ES-import route target) is added to the route.

• If auto-derived service RTs are used in the service, the corresponding operating route target is used as the EVI-RT.

• EVI-RT type 3 is not supported (type 3 is specified in draft-ietf-bess-evpn-igmp-mld-proxy).

• In general, vsi-import and vsi-export must not be used in OISM mode services or when the Multicast Synch routes are used. Using vsi-import or vsi-export policies instead of the route target command or the EVI-derived route target leads to issues when advertising and processing the Multicast Synch routes.

The following are additional considerations about the Multicast Synch routes:

• The routes are advertised without the need to configure any command as long as igmp-snooping or mld-snooping are enabled on an R-VPLS in OISM mode attached to a regular or virtual Ethernet Segment.

• The reception of Multicast Join or Leave Synch routes triggers the synchronization of states and the associated procedures in draft-ietf-bess-evpn-igmp-mld-proxy.

• Upon receiving a Leave message, the triggered Multicast Synch route encodes the configured Last Member Query interval times robust-count (LMQ ✕ robust-count) in the Maximum Response Time field. The local PE expires the multicast state after the usual time plus an additional time that accounts for the BGP propagation to the remote ES peers and can be configured with the command config>service>system>bgp-evpn>multicast-leave-sync-propagation number. This timer should be configured the same in all the PEs attached to the same ES.

This section shows a configuration example for the network illustrated in EVPN OISM example.

The following CLI excerpt shows the configuration required on PE4 for services 2000 (VPRN), BD-2003 and BD-2004 (ordinary BDs) and BD-2002 (SBD).

vprn 2000 name "tenant-2k" customer 1 create
    route-distinguisher auto-rd
    interface "bd-2003" create
        address 10.41.0.1/24
        vpls "bd-2003"
        exit                  
    exit
    interface "bd-2004" create
        address 10.42.0.1/24
        vpls "bd-2004"
        exit
    exit
    interface "bd-2002" create
        vpls "bd-2002"
            evpn-tunnel supplementary-broadcast-domain  <------
        exit
    exit
    igmp                      <------
        interface "bd-2003"   <------
            no shutdown
        exit
        interface "bd-2004"   <------
            no shutdown
        exit
        no shutdown
    exit
    pim                       <------
        interface "bd-2002"      <------
            multicast-senders always     <------
        exit
        apply-to all             <------
        no shutdown
    exit
    no shutdown
exit

As shown in the previous configuration commands, the VPRN must be configured as follows:

• The SBD interface in the VPRN must be configured as evpn-tunnel supplementary-broadcast-domain so that the OISM forwarding mode is enabled.

• IGMP must be enabled on the ordinary BD (R-VPLS) interfaces so that the PEs can process the received IGMP messages from the receivers.

• Even though the protocol itself is not used, PIM is enabled in the VPRN on all the IRB interfaces so that the multicast source addresses can be resolved. Also, multicast-senders always must be enabled on the SBD interface; this is because the SBD interface is unnumbered (it does not have an IP address associated) and the multicast traffic source RPF-check would discard the multicast traffic arriving at the SBD interface unless the system is informed that legal multicast traffic may be expected on the SBD. The multicast-senders always command is allows the system to process multicast on the unnumbered SBD interface.

Besides the VPRN, BD-2003, BD-2004 and BD-2002 (SBD) must be configured as follows.

vpls 2003 name "bd-2003" customer 1 create
    allow-ip-int-bind
        forward-ipv4-multicast-to-ip-int    <------
    exit
    bgp
    exit
    bgp-evpn
        evi 2003
        mpls bgp 1
            ingress-replication-bum-label
            auto-bind-tunnel
                resolution any
            exit
            no shutdown
        exit
    exit
    igmp-snooping                            <------
        no shutdown                          <------
    exit
    sap 1/1/1:2003 create
        igmp-snooping
            mrouter-port
        exit                  
        no shutdown
    exit
    no shutdown
exit
vpls 2004 name "bd-2004" customer 1 create
    allow-ip-int-bind
        forward-ipv4-multicast-to-ip-int     <------
    exit
    bgp
    exit
    bgp-evpn
        evi 2004
        mpls bgp 1
            ingress-replication-bum-label
            auto-bind-tunnel
                resolution any
            exit
            no shutdown
        exit
    exit
    igmp-snooping                            <------
        no shutdown                          <------
    exit
    sap 1/1/1:2004 create
        igmp-snooping
            fast-leave
        exit
        no shutdown
    exit
    no shutdown
exit
vpls 2002 name "bd-2002" customer 1 create
    allow-ip-int-bind
        forward-ipv4-multicast-to-ip-int     <------
    exit
    bgp
    exit
    bgp-evpn
        no mac-advertisement
        ip-route-advertisement
        sel-mcast-advertisement              <------
        evi 2002
        mpls bgp 1
            auto-bind-tunnel
                resolution any
            exit
            no shutdown
        exit
    exit
    igmp-snooping                            <------
        no shutdown                          <------
    exit
    no shutdown
exit

As shown in the previous configuration commands, igmp-snooping no shutdown must be configured in ordinary and SBD R-VPLS services so that IGMP messages or SMET routes are processed by the IGMP module. Also, the command forward-ipv4-multicast-to-ip-int allows the system to forward multicast traffic from the R-VPLS to the VPRN interface.

Finally, bgp-evpn>sel-mcast-advertisement must be enabled on the SBD R-VPLS, so that the SBD aggregates the multicast state for all the ordinary BDs and advertises the corresponding SMET routes with the SBD route-target. In OISM mode, the SMET route is only advertised from the SBD R-VPLS and not from the other ordinary BD R-VPLSes.

PE2 and PE3 are configured with the VPRN (2000), ordinary BD (BD-2001) and SBD (BD-2002) as above. In addition, PE2 and PE3 are attached to ES-1 where a receiver is connected. Multicast state synchronization through BGP Multicast Synch routes is automatically enabled in R-VPLS services in OISM mode and no additional configuration is needed:

/* Example of ES-1 configuration and MCS on PE3. Similar configuration is needed in 
PE2.
bgp-evpn
    ethernet-segment "ES-1" virtual create
        esi 01:00:00:00:00:00:01:00:00:00
        service-carving
            mode manual
            manual
                preference non-revertive create
                    value 30
                exit
            exit
        exit
        multi-homing single-active
        lag 1
        dot1q
            q-tag-range 2001
        exit
        no shutdown
    exit

When the previous configuration is executed in the three nodes, the EVPN routes are exchanged. BD2003 in PE4 receives IMET routes from the remote SBD PEs and creates "SBD" destinations to PE2 and PE3. Those SBD destinations are used to forward multicast traffic to PE2 and PE3, following the OISM forwarding procedures described in OISM forwarding plane. The following command shows an example of IMET route (flagged as SBD route working on OISM mode) and SMET route received on PE4 from PE2.

/* IMET route received from PE2 on PE4.
*A:PE-4# show router bgp routes evpn incl-mcast community target:64500:2002 hunt 
<snip>
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Nexthop        : 192.0.2.2
From           : 192.0.2.2
Res. Nexthop   : 192.168.24.1
Local Pref.    : 100                    Interface Name : int-PE-4-PE-2
<snip>
Community      : target:64500:2002
                 mcast-flags:SBD/NO-MEG/NO-PEG/OISM/NO-MLD-Proxy/NO-IGMP-Proxy <---
                 bgp-tunnel-encap:MPLS
<snip>
EVPN type      : INCL-MCAST             
Tag            : 0                      
Originator IP  : 192.0.2.2                   <------
Route Dist.    : 192.0.2.2:2002
<snip>
-------------------------------------------------------------------------------
PMSI Tunnel Attributes : 
Tunnel-type    : Ingress Replication    
Flags          : Type: RNVE(0) BM: 0 U: 0 Leaf: not required
MPLS Label     : LABEL 524241           
Tunnel-Endpoint: 192.0.2.2
-------------------------------------------------------------------------------
/* SMET route from PE2 received on PE4.
*A:PE-4# show router bgp routes evpn smet community target:64500:2002 hunt       
<snip>
-------------------------------------------------------------------------------
RIB In Entries
-------------------------------------------------------------------------------
Nexthop        : 192.0.2.2
From           : 192.0.2.2
Res. Nexthop   : 192.168.24.1
Local Pref.    : 100                    Interface Name : int-PE-4-PE-2
<snip>
Community      : target:64500:2002 bgp-tunnel-encap:MPLS
<snip>
EVPN type      : SMET                   
Tag            : 0                      
Src IP         : 0.0.0.0                 <------
Grp IP         : 239.0.0.4               <------
Originator IP  : 192.0.2.2               <------
Route Dist.    : 192.0.2.2:2002         
<snip>

When PE4 receives the IMET routes from PE2 and PE3 SBDs, it identifies the routes as SBD routes in OISM mode, and PE4 creates special EVPN destinations on the BD-2003 service that are used to forward the multicast traffic. The SBD destinations are shown as Sup BCast Domain in the show command output.

*A:PE-4# show service id 2003 evpn-mpls 
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                Transport:Tnl                             Sup BCast Domain <---
-------------------------------------------------------------------------------
192.0.2.2       524241        0           m               11/26/2019 10:38:04
                ldp:65537                                 Yes   <------
192.0.2.3       524253        0           m               11/26/2019 10:38:28
                ldp:65539                                 Yes   <------
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------

Based on the reception of the SMET routes from PE2 and PE3, PE4 adds the SBD EVPN destinations to its MFIB on BD-2003.

*A:PE-4# show service id 2003 igmp-snooping base 
===============================================================================
IGMP Snooping Base info for service 2003
===============================================================================
Admin State : Up
Querier     : 10.41.0.1 on rvpls bd-2003
SBD service : 2002
-------------------------------------------------------------------------------
Port                      Oper MRtr Pim  Send Max   Max  Max   MVR       Num
Id                        Stat Port Port Qrys Grps  Srcs Grp   From-VPLS Grps
                                                         Srcs            
-------------------------------------------------------------------------------
sap:1/1/1:2003            Up   Yes  No   No   None  None None  Local     0
rvpls                     Up   Yes  No   N/A  N/A   N/A  N/A   N/A       N/A
sbd-mpls:192.0.2.2:524241 Up   No   No   N/A  N/A   N/A  N/A   N/A       1 <------
sbd-mpls:192.0.2.3:524253 Up   No   No   N/A  N/A   N/A  N/A   N/A       1 <------
===============================================================================
*A:PE-4# show service id 2003 igmp-snooping statistics 
===============================================================================
IGMP Snooping Statistics for service 2003
===============================================================================
Message Type            Received      Transmitted   Forwarded
-------------------------------------------------------------------------------
<snip>
EVPN SMET Routes        2             0             N/A    <------
-------------------------------------------------------------------------------
*A:PE-4# show service id 2003 mfib                                               
<snip>
-------------------------------------------------------------------------------
*               *                     sap:1/1/1:2003               Local    Fwd
*               239.0.0.4             sap:1/1/1:2003               Local    Fwd
                                      sbd-eMpls:192.0.2.2:524241   Local    Fwd
                                      sbd-eMpls:192.0.2.3:524253   Local    Fwd

PE2 and PE3 also creates regular destinations and SBD destinations based on the reception of IMET routes. As an example, the show service id 2001 evpn-mpls command shows the destinations created by PE3 in the ordinary BD-2001.

*A:PE-3# show service id 2001 evpn-mpls 
===============================================================================
BGP EVPN-MPLS Dest
===============================================================================
TEP Address     Egr Label     Num. MACs   Mcast           Last Change
                Transport:Tnl                             Sup BCast Domain <------
-------------------------------------------------------------------------------
192.0.2.2       524241        0           m               11/26/2019 10:38:28
                ldp:65538                                 Yes              <------
192.0.2.2       524242        0           bum             11/26/2019 10:38:28
                ldp:65538                                 No 
192.0.2.2       524243        1           none            11/26/2019 10:38:28
                ldp:65538                                 No
192.0.2.4       524245        0           m               11/26/2019 10:38:28
                ldp:65537                                 Yes              <------
-------------------------------------------------------------------------------
Number of entries : 4
-------------------------------------------------------------------------------
===============================================================================

In case of an SBD destination and a non-SBD destination to the same PE (PE2), IGMP only uses the non-SBD one in the MFIB. The non-SBD destination always has priority over the SBD destination. This can be seen in the show service id igmp-snooping base command in PE3, where the SBD destination to PE2 is down as long as the non-SBD destination is up.

*A:PE-3# show service id 2001 igmp-snooping base 
===============================================================================
IGMP Snooping Base info for service 2001
===============================================================================
Admin State : Up
Querier     : 10.0.0.3 on rvpls bd-2001
SBD service : 2002
-------------------------------------------------------------------------------
Port                      Oper MRtr Pim  Send Max   Max  Max   MVR       Num
Id                        Stat Port Port Qrys Grps  Srcs Grp   From-VPLS Grps
                                                         Srcs            
-------------------------------------------------------------------------------
sap:lag-1:2001            Down No   No   No   None  None None  Local     1
rvpls                     Up   Yes  No   N/A  N/A   N/A  N/A   N/A       N/A
sbd-mpls:192.0.2.2:524241 Down No   No   N/A  N/A   N/A  N/A   N/A       0 <------
mpls:192.0.2.2:524242     Up   No   No   N/A  N/A   N/A  N/A   N/A       1 <------
sbd-mpls:192.0.2.4:524245 Up   No   No   N/A  N/A   N/A  N/A   N/A       0 
===============================================================================
*A:PE-3# show service id 2001 mfib 
===============================================================================
Multicast FIB, Service 2001
===============================================================================
Source Address  Group Address         Port Id                      Svc Id   Fwd
                                                                            Blk
-------------------------------------------------------------------------------
*               239.0.0.4             sap:lag-1:2001               Local    Fwd
                                      eMpls:192.0.2.2:524242       Local    Fwd <---

Finally, to check the Layer 3 IIF and OIF entries on the VPRN services, enter the show router id pim group detail command. As an example, the command is executed in PE2:

*A:PE-2# show router 2000 pim group detail         
===============================================================================
PIM Source Group ipv4
===============================================================================
Group Address      : 239.0.0.4
Source Address     : *
<snip>
===============================================================================
PIM Source Group ipv4
===============================================================================
Group Address      : 239.0.0.4
Source Address     : 10.41.0.41
<snip>
Up Time            : 0d 00:13:20        Resolved By        : rtable-u 
Up JP State        : Joined             Up JP Expiry       : 0d 00:00:00
Up JP Rpt          : Pruned             Up JP Rpt Override : 0d 00:00:00
 
Rpf Neighbor       : 10.41.0.41
Incoming Intf      : bd-2002
Outgoing Intf List : bd-2001
 
Curr Fwding Rate   : 0.000 kbps         
Forwarded Packets  : 1000               Discarded Packets  : 0
Forwarded Octets   : 84000              RPF Mismatches     : 0
Spt threshold      : 0 kbps             ECMP opt threshold : 7
Admin bandwidth    : 1 kbps             
-------------------------------------------------------------------------------
Groups : 2
===============================================================================

Inclusive provider tunnels of type mLDP are supported in OISM PEs. These tunnels can be used to transport multicast flows from root PEs to leaf PEs while preventing multiple copies of the same multicast packet on the same link.

OISM with IR versus inclusive mLDP illustrates the difference between using Ingress Replication (IR) and inclusive mLDP provider tunnels in OISM. With a source S1 connected to BD1 and sending a flow to G1, if IR is used, the multicast traffic is only sent to PEs with receivers for (S1,G1). However, if an inclusive mLDP tunnel on PE1 is used (right side of OISM with IR versus inclusive mLDP) the multicast flow is sent to all the PEs in the tenant domain. For example, PE3 receives the flow only to drop it because there are no local receivers.

mLDP tunnels are referred to as Inclusive BUM tunnels, because, although IP multicast traffic uses these tunnels, any BUM frame is also distributed to all PEs in the tenant. For example, in OISM with IR versus inclusive mLDP (right hand side), any BUM frame generated by any host connected to BD1 in PE1 uses the mLDP tunnel and is also sent to PE3.

The use of mLDP-inclusive provider tunnels in OISM requires the following configuration and procedures to be enabled on the PEs:

• All the PEs in the OISM tenant domain that need to transmit or receive multicast traffic on an mLDP tree in a BD, are configured with the config>service>vpls>provider-tunnel>inclusive>owner bgp-evpn-mpls and config>service>vpls>provider-tunnel>inclusive>mldp commands.

• The PEs attached to the sources (root PEs) should be configured with the config>service>vpls>provider-tunnel>inclusive>root-and-leaf command on the ordinary BDs, and the PEs attached to the receivers should be configured as root-and-leaf or leaf-only.

• The PEs attached to the receivers (leaf PEs) need to be configured using the ingress-repl-inc-mcast-advertisement command on the BDs or SBDs. This ensures the leaf PEs advertise a label in the IMET routes so that the root PEs can create EVPN-MPLS destinations to the leaf PEs and add them to their MFIB. Having EVPN-MPLS destinations in the MFIB is required on the root PE to use the mLDP tunnel for the multicast traffic.

• The SBD must always be configured as leaf-only in all PEs, because the SBD mLDP tree is not used to transmit IP multicast.

• For the IMET and SMET routes to be exported and imported with the correct route targets, no vsi-import or vsi-export policies should be configured on the ordinary BDs and the SBDs.

Assuming the above guidelines are followed, and as illustrated in OISM with IR versus inclusive mLDP (right side), the root PE (PE1) that is attached to the source in BD1 sends the multicast traffic in an mLDP tree that is joined by leaf PEs either on BD1 (if BD1 is exists) or on the SBD (if BD1 does not exist on the leaf PE).

OISM with inclusive mLDP example illustrates an example of the OISM procedures with mLDP trees.

Consider three PEs, PE1, PE2, and PE3, attached to BD1/BD2, BD1, and BD3 respectively, as in OISM with inclusive mLDP example. Assume that the source S1 is connected to BD1 in PE1. PE2 and PE3 are leaf PEs, because they have receivers but no sources. In this example:

• BD and SBD services must be configured for provider tunnel as follows:

  • To have PE1 sending multicast traffic in P2MP mLDP tunnels on BD1 and BD2, both BDs are configured as provider-tunnel inclusive root-and-leaf. They are also configured with ingress-repl-inc-mcast-advertisement. The following is an example configuration of BD1 in PE1.

    *A:PE-1>config>service>vpls# info 
    ----------------------------------------------
                allow-ip-int-bind
                exit
                bgp
                exit
                bgp-evpn
                    evi 1
                    ingress-repl-inc-mcast-advertisement // default value
                    mpls bgp 1
                        auto-bind-tunnel
                            resolution any
                        exit
                        no shutdown       
                    exit
                exit
                provider-tunnel
                    inclusive
                        owner bgp-evpn-mpls
                        root-and-leaf
                        data-delay-interval 10
                        mldp
                        no shutdown
                    exit
                exit
                igmp-snooping / mld-snooping
                  no shutdown
                exit
    <snip>
    

  • PE2 and PE3 BDs are configured as leaf (no root-and-leaf) as they must be able to join mLDP trees but not set up an mLDP tree themselves. It is important that these BDs are configured with the ingress-repl-inc-mcast-advertisement command that allows upstream PEs to create destinations to them.

    Multicast traffic cannot use the mLDP tree unless there is an EVPN-MPLS destination in the MFIB for the multicast stream.

  • The SBDs in all PEs must be configured as provider-tunnel>inclusive>no root-and-leaf and ingress-repl-inc-mcast-advertisement.

• When the configuration is added, the PEs create EVPN-MPLS destinations as follows, where a destination is represented as {pe, label} with ‟pe” being the IP address of the remote PE and ‟label” being the EVPN label advertised by the remote PE.

  • PE1 creates the following EVPN-MPLS destinations:

    • On BD1: {pe2,bd1-L21}, {pe2,sbd-L22}, {pe3,sbd-L32}

    • On BD2: {pe2,sbd-L22}, {pe3,sbd-L32}

    • On SBD: {pe2,sbd-L22}, {pe3,sbd-L32}

  • PE2 creates destinations as follows:

    • On BD1: {pe1,bd1-L11}, {pe1,sbd-L13}, {pe3,sbd-L32}

    • On SBD: {pe1,sbd-L13}, {pe3,sbd-L32}

  • PE3 creates destinations as follows:

    • On BD3: {pe1,sbd-L13}, {pe2,sbd-L22}

    • On SBD: {pe1,sbd-L13}, {pe2,sbd-L22}

  • PE2's BD1 and PE3's BD3 does not create an EVPN-MPLS destination to PE1's BD2. Also, PE3's BD3 does not create a destination to PE1's BD1. This is in spite of receiving IMET-Composite routes for those BDs with the SBD-RT, which is imported in PE2/PE3 ordinary BDs.

• As an example, on BD1, PE1's IGMP process adds the EVPN-MPLS destinations {pe2,bd1-L21}, {pe3,sbd-L32} to the MFIB. The third destination {pe2,sbd-L22} is kept down because the EVPN-MPLS destination in BD1 has higher priority.

  1. Upon receiving the SMET route from PE2, PE1 adds {pe2,bd1-L21} as OIF for the MFIB (*,G1).

  2. In the meantime, PE2 and PE3 have joined the mLDP tree with tunnel-id 1.

  3. When multicast to G1 is received from S1, because there is an MFIB EVPN OIF entry, the multicast traffic is forwarded. At the IOM level, PE1 replaces the MFIB EVPN destination with the P2MP tunnel with tunnel-id 1, as long as the P2MP tree is operationally up.

  4. The multicast traffic is sent along the mLDP tree and arrives at PE2/BD1 and PE3/SBD. Then local forwarding or routing is performed in PE2 and PE3, as normally in OISM.