Ethernet Virtual Private Networks

The DGW supports ECMP load balancing to reach the destination VTEP. Also, any intermediate core node in the DC should be able to provide further load balancing across ECMP paths, because the source UDP port of each tunneled packet is derived from a hash of the customer inner packet. The following must be considered:

• ECMP for VXLAN is supported on VPLS services but not for BUM traffic. Unicast spraying is based on the packet contents.

• ECMP for VXLAN on R-VPLS services is supported for VXLAN IPv6 tunnels.

• ECMP for VXLAN IPv4 tunnels on R-VPLS is only supported if the configure service vpls allow-ip-int-bind vxlan-ipv4-tep-ecmp command is enabled on the R-VPLS (as well as configure router ecmp).

• ECMP for Layer 3 multicast traffic on R-VPLS services with EVPN-VXLAN destinations is only supported if the vpls allow-ip-int-bind ip-multicast-ecmp command is enabled (as well as configure router ecmp).

• In the cases where ECMP is not supported (BUM traffic in VPLS and ECMP on R-VPLS if not enabled), each VXLAN binding is tied to a single (different) ECMP path, so that in a normal deployment with a reasonable number of remote VTEPs, there should be a fair distribution of the traffic across the paths. That is, only per-VTEP load-balancing is supported, instead of per-flow load-balancing.

• LAG spraying based on the packet hash is supported in all the cases (VPLS unicast, VPLS BUM, and R-VPLS).

The following describes the behavior on the 7750 SR, 7450 ESS, and 7950 XRS with respect to VLAN tag handling for VXLAN VPLS services:

• Dot1q, QinQ, and null SAPs, as well as regular VLAN-handling procedures at the WAN side, are supported on VXLAN VPLS services.

• No ‟vc-type vlan” like VXLAN VNI bindings are supported. Therefore, at the egress of the VXLAN network port, the router does not add any inner VLAN tag on top of the VXLAN encapsulation, and at the ingress network port, the router ignores any VLAN tag received and handles it as part of the payload.

For VXLAN VPLS services, the network port MTU must be at least 50 Bytes (54 Bytes if dot1q) greater than the service MTU to allow enough room for the VXLAN encapsulation.

The service MTU is only enforced on SAPs (any SAP ingress packet with MTU greater than the service MTU is discarded) and not on VXLAN termination (any VXLAN ingress packet makes it to the egress SAP regardless of the configured service MTU).

If BGP-EVPN is enabled in a VXLAN VPLS service, the service MTU can be advertised in the Inclusive Multicast Ethernet Tag routes and enforce that all the routers attached to the same EVPN service have the same service MTU configured.

VXLAN is a network port encapsulation; therefore, the QoS settings for VXLAN are controlled from the network QoS policies.

A new VXLAN troubleshooting tool, VXLAN Ping, is available to verify VXLAN VTEP connectivity. The VXLAN Ping command is available from interactive CLI and SNMP.

This tool allows the user to specify a wide range of variables to influence how the packet is forwarded from the VTEP source to VTEP termination. The ping function requires the user to specify a different test-id (equates to originator handle) for each active and outstanding test. The required local service identifier from which the test is launched determines the source IP (the system IP address) to use in the outer IP header of the packet. This IP address is encoded into the VXLAN header Source IP TLV. The service identifier also encodes the local VNI. The outer-ip-destination must equal the VTEP termination point on the remote node, and the dest-vni must be a valid VNI within the associated service on the remote node. The outer source IP address is automatically detected and inserted in the IP header of the packet. The outer source IP address uses the IPv4 system address by default.

If the VTEP is created using a non-system source IP address through the vxlan-src-vtep command, the outer source IP address uses the address specified by vxlan-src-vtep. The remainder of the variables are optional.

The VXLAN PDU is encapsulated in the appropriate transport header and forwarded within the overlay to the appropriate VTEP termination. The VXLAN router alert (RA) bit is set to prevent forwarding OAM PDU beyond the terminating VTEP. Because handling of the router alert bit was not defined in some early releases of VXLAN implementations, the VNI Informational bit (I-bit) is set to ‟0” for OAM packets. This indicates that the VNI is invalid, and the packet should not be forwarded. This safeguard can be overridden by including the i-flag-on option that sets the bit to ‟1”, valid VNI. Ensure that OAM frames meant to be contained to the VTEP are not forwarded beyond its endpoints.

The supporting VXLAN OAM ping draft includes a requirement to encode a reserved IEEE MAC address as the inner destination value. However, at the time of implementation, that IEEE MAC address had not been assigned. The inner IEEE MAC address defaults to 00:00:00:00:00:00, but may be changed using the inner-l2 option. Inner IEEE MAC addresses that are included with OAM packets are not learned in the local Layer 2 forwarding databases.

The echo responder terminates the VXLAN OAM frame, and takes the appropriate response action, and include relevant return codes. By default, the response is sent back using the IP network as an IPv4 UDP response. The user can choose to override this default by changing the reply-mode to overlay. The overlay return mode forces the responder to use the VTEP connection representing the source IP and source VTEP. If a return overlay is not available, the echo response is dropped by the responder.

Support is included for:

• IPv4 VTEP

• Optional specification of the outer UDP Source, which helps downstream network elements along the path with ECMP to hash to flow to the same path

• Optional configuration of the inner IP information, which helps the user test different equal paths where ECMP is deployed on the source. A test only validates a single path where ECMP functions are deployed. The inner IP information is processed by a hash function, and there is no guarantee that changing the IP information between tests selects different paths.

• Optional end system validation for a single L2 IEEE MAC address per test. This function checks the remote FDB for the configured IEEE MAC Address. Only one end system IEEE MAC Address can be configured per test.

• Reply mode UDP (default) or Overlay

• Optional additional padding can be added to each packet. There is an option that indicates how the responder should handle the pad TLV. By default, the padding is not reflected to the source. The user can change this behavior by including the reflect-pad option. The reflect-pad option is not supported when the reply mode is set to UDP.

• Configurable send counts, intervals, times outs, and forwarding class

The VXLAN OAM PDU includes two timestamps. These timestamps are used to report forward direction delay. Unidirectional delay metrics require accurate time of day clock synchronization. Negative unidirectional delay values are reported as ‟0.000”. The round trip value includes the entire round trip time including the time that the remote peer takes to process that packet. These reported values may not be representative of network delay.

The following example commands and outputs show how the VXLAN Ping function can be used to validate connectivity. The echo output includes a new header to better describe the VXLAN ping packet headers and the various levels.

oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 
interval
0.1 send-count 10 

TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

! ! ! ! ! ! ! ! ! ! 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 1.097ms, avg = 2.195ms, max = 2.870ms, stddev = 0.735ms
round-trip-delay min = 1.468ms, avg = 1.693ms, max = 2.268ms, stddev = 0.210ms



oam vxlan-ping test-id 2 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 outer-
ip-source-udp 65000 outer-ip-ttl 64 inner-l2 d0:0d:1e:00:00:01 inner-ip-source
 192.168.1.2 inner-ip-destination 127.0.0.8 reply-mode overlay send-count 20 
interval
 1 timeout 3 padding 1000 reflect-pad fc nc profile out 

TestID 2, Service 1, DestVNI 2, ReplyMode overlay, IFlag Off, PadSize 1000, ReflectP
ad
 Yes, SendCount 20, Interval 1, Timeout 3
Outer: SourceIP 10.20.1.3, SourcePort 65000, DestIP 10.20.1.4, TTL 64, FC nc, Profil
e
 out
Inner: DestMAC d0:0d:1e:00:00:01, SourceIP 192.168.1.2, DestIP 127.0.0.8

===================================================================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
===================================================================================
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=1.733ms fwd
-time=0.302ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.549ms fwd
-time=1.386ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=3.243ms fwd
-time=0.643ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.551ms fwd
-time=2.350ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.644ms fwd
-time=1.080ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.670ms fwd
-time=1.307ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.636ms fwd
-time=0.490ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.649ms fwd
-time=0.005ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.401ms fwd
-time=0.685ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.634ms fwd
-time=0.373ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=11 ttl=255 rtt-time=1.559ms fwd
-time=0.679ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=12 ttl=255 rtt-time=1.666ms fwd
-time=0.880ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=13 ttl=255 rtt-time=1.629ms fwd
-time=0.669ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=14 ttl=255 rtt-time=1.280ms fwd
-time=1.029ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=15 ttl=255 rtt-time=1.458ms fwd
-time=0.268ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=16 ttl=255 rtt-time=1.659ms fwd
-time=0.786ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=17 ttl=255 rtt-time=1.636ms fwd
-time=1.071ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=18 ttl=255 rtt-time=1.568ms fwd
-time=2.129ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=19 ttl=255 rtt-time=1.657ms fwd
-time=1.326ms. rc=4
1132 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=20 ttl=255 rtt-time=1.762ms fwd
-time=1.335ms. rc=4

---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
20 packets transmitted, 20 packets received, 0.00% packet loss
   20 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
forward-delay min = 0.005ms, avg = 0.939ms, max = 2.350ms, stddev = 0.577ms
round-trip-delay min = 1.280ms, avg = 1.679ms, max = 3.243ms, stddev = 0.375ms



oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 interval 0.1 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 0.1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

2 2 2 2 2 2 2 2 2 2 
---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 non-errored responses(!), 0 out-of-order(*), 0 malformed echo responses(.)
   0 send errors(.), 0 time outs(.)
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present(1), 10 end-system not present(2)
forward-delay min = 0.467ms, avg = 0.979ms, max = 1.622ms, stddev = 0.504ms
round-trip-delay min = 1.501ms, avg = 1.597ms, max = 1.781ms, stddev = 0.088ms



oam vxlan-ping test-id 1 service 1 dest-vni 2 outer-ip-destination 10.20.1.4 send
-count 10 end-system 00:00:00:00:00:01 
TestID 1, Service 1, DestVNI 2, ReplyMode UDP, IFlag Off, PadSize 0, ReflectPad No,
 EndSystemMAC 00:00:00:00:00:01, SendCount 10, Interval 1, Timeout 5
Outer: SourceIP 10.20.1.3, SourcePort Dynamic, DestIP 10.20.1.4, TTL 10, FC be, Prof
ile
 In
Inner: DestMAC 00:00:00:00:00:00, SourceIP 10.20.1.3, DestIP 127.0.0.1

===================================================================================
rc=1 Malformed Echo Request Received, rc=2 Overlay Segment Not Present, rc=3 Overlay
 Segment Not Operational, rc=4 Ok
mac=1 End System Present, mac=2 End System Not Present
===================================================================================

92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=1 ttl=255 rtt-time=2.883ms fwd
-time=4.196ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=2 ttl=255 rtt-time=1.596ms fwd
-time=1.536ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=3 ttl=255 rtt-time=1.698ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=4 ttl=255 rtt-time=1.687ms fwd
-time=1.766ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=5 ttl=255 rtt-time=1.679ms fwd
-time=0.799ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=6 ttl=255 rtt-time=1.678ms fwd
-time=0.000ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=7 ttl=255 rtt-time=1.709ms fwd
-time=0.031ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=8 ttl=255 rtt-time=1.757ms fwd
-time=1.441ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=9 ttl=255 rtt-time=1.613ms fwd
-time=2.570ms. rc=4 mac=2
92 bytes from vxlan-id 2 10.20.1.4: vxlan_seq=10 ttl=255 rtt-time=1.631ms fwd
-time=2.130ms. rc=4 mac=2

---- vxlan-id 2 ip-address 10.20.1.4 PING Statistics ----
10 packets transmitted, 10 packets received, 0.00% packet loss
   10 valid responses, 0 out-of-order, 0 malformed echo responses
   0 send errors, 0 time outs
   0 overlay segment not found, 0 overlay segment not operational
   0 end-system present, 10 end-system not present
forward-delay min = 0.000ms, avg = 1.396ms, max = 4.196ms, stddev = 1.328ms
round-trip-delay min = 1.596ms, avg = 1.793ms, max = 2.883ms, stddev = 0.366ms

IPv4 and IPv6 multicast routing is supported in an EVPN-VXLAN VPRN and IES routed VPLS service through its IP interface when the source of the multicast stream is on one side of its IP interface and the receivers are on either side of the IP interface. For example, the source for multicast stream G1 could be on the IP side, sending to receivers on both other regular IP interfaces and the VPLS of the routed VPLS service, while the source for group G2 could be on the VPLS side sending to receivers on both the VPLS and IP side of the routed VPLS service. See IPv4 and IPv6 multicast routing support for more details.

The delivery of IP multicast in VXLAN services can be optimized with IGMP and MLD snooping. IGMP and MLD snooping are supported in EVPN-VXLAN VPLS services and in EVPN-VXLAN VPRN/IES R-VPLS services. When enabled, IGMP and MLD reports are snooped on SAPs or SDP bindings, but also on VXLAN bindings, to create or modify entries in the MFIB for the VPLS service.

When configuring IGMP and MLD snooping in EVPN-VXLAN VPLS services, consider the following:

• To enable IGMP snooping in the VPLS service on VXLAN, use the configure service vpls igmp-snooping no shutdown command.

• To enable MLD snooping in the VPLS service on VXLAN, use the configure service vpls mld-snooping no shutdown command.

• The VXLAN bindings only support basic IGMP/MLD snooping functionality. Features configurable under SAPs or SDP bindings are not available for VXLAN (VXLAN bindings are configured with the default values used for SAPs and SDP bindings). By default, a specified VXLAN binding only becomes a dynamic Mrouter when it receives IGMP or MLD queries and adds a specified multicast group to the MFIB when it receives an IGMP or MLD report for that group.

  Alternatively, it is possible to configure all VXLAN bindings for a particular VXLAN instance to be Mrouter ports using the configure service vpls vxlan igmp-snooping mrouter-port and configure service vpls vxlan mld-snooping mrouter-port commands.

• The show service id igmp-snooping, clear service id igmp-snooping, show service id mld-snooping, and clear service id mld-snooping commands are also available for VXLAN bindings.

  Note: MLD snooping uses MAC-based forwarding. See MAC-based IPv6 multicast forwarding for more details.

  The following CLI commands show how the system displays IGMP snooping information and statistics on VXLAN bindings (the equivalent MLD output is similar).

*A:PE1# show service id 1 igmp-snooping port-db vxlan vtep  192.0.2.72 vni 1 detail 
===============================================================================
IGMP Snooping VXLAN 192.0.2.72/1 Port-DB for service 1
===============================================================================
-------------------------------------------------------------------------------
IGMP Group 239.0.0.1
-------------------------------------------------------------------------------
Mode             : exclude              Type             : dynamic
Up Time          : 0d 19:07:05          Expires          : 137s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
No sources.
-------------------------------------------------------------------------------
IGMP Group 239.0.0.2
-------------------------------------------------------------------------------
Mode             : include              Type             : dynamic
Up Time          : 0d 19:06:39          Expires          : 0s
Compat Mode      : IGMP Version 3       
V1 Host Expires  : 0s                   V2 Host Expires  : 0s
-------------------------------------------------------
Source Address  Up Time      Expires  Type     Fwd/Blk
-------------------------------------------------------
10.0.0.232      0d 19:06:39  137s     dynamic  Fwd              
-------------------------------------------------------------------------------
Number of groups: 2
===============================================================================

*A:PE1# show service id 1 igmp-snooping 
statistics vxlan vtep  192.0.2.72 vni 1        
 
===============================================================================
IGMP Snooping Statistics for VXLAN 192.0.2.72/1 (service 1)
===============================================================================
Message Type            Received      Transmitted   Forwarded
-------------------------------------------------------------------------------
General Queries         0             0             556
Group Queries           0             0             0
Group-Source Queries    0             0             0
V1 Reports              0             0             0
V2 Reports              0             0             0
V3 Reports              553           0             0
V2 Leaves               0             0             0
Unknown Type            0             N/A           0
-------------------------------------------------------------------------------
Drop Statistics
-------------------------------------------------------------------------------
Bad Length               : 0
Bad IP Checksum          : 0
Bad IGMP Checksum        : 0
Bad Encoding             : 0
No Router Alert          : 0
Zero Source IP           : 0
Wrong Version            : 0
Lcl-Scope Packets        : 0
Rsvd-Scope Packets       : 0
 
Send Query Cfg Drops     : 0
Import Policy Drops      : 0
Exceeded Max Num Groups  : 0
Exceeded Max Num Sources : 0
Exceeded Max Num Grp Srcs: 0
MCAC Policy Drops        : 0
===============================================================================
*A:PE1# show service id 1 mfib 
===============================================================================
Multicast FIB, Service 1
===============================================================================
Source Address  Group Address         SAP or SDP Id               Svc Id   Fwd/Blk
-------------------------------------------------------------------------------
*               *                     sap:1/1/1:1              Local    Fwd
*               239.0.0.1             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
10.0.0.232      239.0.0.2             sap:1/1/1:1              Local    Fwd
                                      vxlan:192.0.2.72/1       Local    Fwd
-------------------------------------------------------------------------------
Number of entries: 3
===============================================================================

PIM snooping for IPv4 and IPv6 are supported in an EVPN-EVPN-VXLAN VPLS or R-VPLS service (with the R-VPLS attached to a VPRN or IES service). The snooping operation is similar to that within a VPLS service (see PIM snooping for VPLS) and supports both PIM snooping and PIM proxy modes.

PIM snooping for IPv4 is enabled using the configure service vpls pim-snooping command.

PIM snooping for IPv6 is enabled using the configure service vpls pim-snooping no ipv6-multicast-disable command.

When using PIM snooping for IPv6, the default forwarding is MAC-based with optional support for SG-based (see IPv6 multicast forwarding). SG-based forwarding requires FP3- or higher-based hardware.

It is not possible to configure max-num-groups for VXLAN bindings.

By default, the system IP address is used to terminate and generate VXLAN traffic. The following configuration example shows an Epipe service that supports static VXLAN termination:

config service epipe 1 name "epipe1" customer 1 create
  sap 1/1/1:1 create
  exit
  vxlan vni 100 create
    egr-vtep 192.0.2.1
      oper-group op-grp-1
    exit
  no shutdown

Where:

• vxlan vni vni create specifies the ingress VNI the router uses to identify packets for the service. The following considerations apply:

  • In services that use EVPN, the configured VNI is only used as the ingress VNI to identify packets that belong to the service. Egress VNIs are learned from the BGP EVPN. In the case of Static VXLAN, the configured VNI is also used as egress VNI (because there is no BGP EVPN control plane).

  • The configured VNI is unique in the system, and as a result, it can only be configured in one service (VPLS or Epipe).

• egr-vtep ip-address specifies the remote VTEP the router uses when encapsulating frames into VXLAN packets. The following consideration apply:

  • When the PE receives VXLAN packets, the source VTEP is not checked against the configured egress VTEP.

  • The ip-address must be present in the global routing table so that the VXLAN destination is operationally up.

• The oper-group may be added under egr-vtep. The expected behavior for the operational group and service status is as follows:

  • If the egr-vtep entry is not present in the routing table, the VXLAN destination (in the show service id vxlan command) and the provisioned operational group under egr-vtep enters into the operationally down state.

  • If the Epipe SAP goes down, the service goes down, but it is not affected if the VXLAN destination goes down.

  • If the service is admin shutdown, then in addition to the SAP, the VXLAN destination and the oper-group also enters the operationally down state.

  Note: The operational group configured under egr-vtep cannot be monitored on the SAP of the Epipe where it is configured.

The following features are not supported by Epipe services with VXLAN destinations:

• per-service hashing

• SDP-binds

• PBB context

• BGP-VPWS

• spoke SDP-FEC

• PW-port

VXLAN instances in VPLS and R-VPLS can be configured with egress VTEPs. This is referred as static vxlan-instances. The following configuration example shows a VPLS service that supports a static vxlan-instance:

config service vpls 1 name "vpls-1" customer 1 create
  sap 1/1/1:1 create
  exit
  vxlan instance 1 vni 100 create
    source-vtep-security
    no disable-aging /* default: disable-aging    
    no disable-learning  /* default: disable-learning
    no discard-unknown-source
    no max-nbr-mac-addr <table-size>
    restrict-protected-src discard-frame
    egr-vtep 192.0.2.1 create
    exit
    egr-vtep 192.0.2.2 create
    exit
  vxlan instance 2 vni 101 create
    egr-vtep 192.0.2.3 create
    exit

 vxlan instance 2 vni 101 create
    egr-vtep 192.0.2.3 create
    exit
  no shutdown

Specifically the following can be stated:

• Each VPLS service can have up to two static VXLAN instances. Each instance is an implicit split-horizon-group, and up to 255 static VXLAN binds are supported in total, shared between the two VXLAN instances.

• Single VXLAN instance VPLS services with static VXLAN are supported along with SAPs and SDP bindings. Therefore:

  • VNIs configured in static VXLAN instances are ‟symmetric”, that is, the same ingress and egress VNIs are used for VXLAN packets using that instance. Note that asymmetric VNIs are actually possible in EVPN VXLAN instances.

  • The addresses can be IPv4 or IPv6 (but not a mix within the same service).

  • A specified VXLAN instance can be configured with static egress VTEPs, or be associated with BGP EVPN, but the same instance cannot be configured to support both static and BGP-EVPN based VXLAN bindings.

• Up to two VXLAN instances are supported per VPLS (up to two).

  • When two VXLAN instances are configured in the same VPLS service, any combination of static and BGP-EVPN enabled instances are supported. That is, the two VXLAN instances can be static, or BGP-EVPN enabled, or one of each type.

  • When a service is configured with EVPN and there is a static BGP-EVPN instance in the same service, the user must configure restrict-protected-src discard-frame along with no disable-learning in the static BGP-EVPN instance, service>vpls>vxlan.

• MAC addresses are learned also on the VXLAN bindings of the static VXLAN instance. Therefore, they are shown in the FDB commands. Note that disable-learning and disable-aging are by default enabled in static vxlan-instance.

  • The learned MAC addresses are subject to the remote-age, and not the local-age (only MACs learned on SAPs use the local-age setting).

  • MAC addresses are learned on a VTEP as long as no disable-learning is configured, and the VXLAN VTEP is present in the base route table. When the VTEP disappears from the route table, the associated MACs are flushed.

• The vpls vxlan source-vtep-security command can be configured per VXLAN instance on VPLS services. When enabled, the router performs an IPv4 source-vtep lookup to discover if the VXLAN packet comes from a trusted VTEP. If not, the router discards the frame. If the lookup yields a trusted source VTEP, then the frame is accepted.

  • A trusted VTEP is an egress VTEP that has been statically configured, or dynamically learned (through EVPN) in any service, Epipe or VPLS

  • The command show service vxlan shows the list of trusted VTEPs in the router.

  • The command source-vtep-security works for static VXLAN instances or BGP-EVPN enabled VXLAN instances, but only for IPv4 VTEPs.

  • The command is mutually exclusive with assisted-replication (replicator or leaf) in the VNI instance. AR can still be configured in a different instance.

Static VXLAN instances can use non-system IPv4/IPv6 termination.

The EVPN-VXLAN service is designed around current VPLS objects and the additional VXLAN construct.

Layer 2 DC PE with VPLS to the WAN shows a DC with a Layer 2 service that carries the traffic for a tenant who wants to extend a subnet beyond the DC. The DC PE function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where a VPLS instance exists for that particular tenant. Within the DC, the tenant has VPLS instances in all the Network Virtualization Edge (NVE) devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The VPLS instances in the redundant DGW and the DC NVEs are connected by VXLAN bindings. BGP-EVPN provides the required control plane for such VXLAN connectivity.

The DGW routers are configured with a VPLS per tenant that provides the VXLAN connectivity to the Nuage VPLS instances. On the router, each tenant VPLS instance is configured with:

• WAN-related parameters (SAPs, spoke SDPs, mesh-SDPs, BGP-AD, and so on).

• BGP-EVPN and VXLAN (VNI) parameters. The following CLI output is an example of an EVPN-VXLAN VPLS service.

*A:DGW1>config>service>vpls# info 
----------------------------------------------
            description "vxlan-service"
            vxlan instance 1 vni 1 create
            exit
            bgp
                route-distinguisher 65001:1
                route-target export target:65000:1 import target:65000:1
            exit
            bgp-evpn
                unknown-mac-route
                mac-advertisement 
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
            sap 1/1/1:1 create
            exit
            no shutdown
----------------------------------------------

Gateway IRB on the DC PE for an L2 EVPN/VXLAN DC shows a DC with a Layer 2 service that carries the traffic for a tenant who extends a subnet within the DC, while the DGW is the default gateway for all the hosts in the subnet. The DGW function is carried out by the 7750 SR, 7450 ESS, and 7950 XRS where an R-VPLS instance exists for that particular tenant. Within the DC, the tenant has VPLS instances in all the NVE devices where they require connectivity (such VPLS instances can be instantiated in TORs, Nuage VRS, VSG, and so on). The WAN connectivity is based on existing IP-VPN features.

In this model, the DGW routers are configured with a R-VPLS (bound to the VPRN that provides the WAN connectivity) per tenant that provides the VXLAN connectivity to the Nuage VPLS instances. This model provides inter-subnet forwarding for L2-only TORs and other L2 DC NVEs.

On the router:

• The VPRN is configured with an interface bound to the backhaul R-VPLS. That interface is a regular IP interface (IP address configured or possibly a Link Local Address if IPv6 is added).

• The VPRN can support other numbered interfaces to the WAN or even to the DC.

• The R-VPLS is configured with the BGP, BGP-EVPN and VXLAN (VNI) parameters.

The Nuage VSGs and NVEs use a regular VPLS service model with BGP EVPN and VXLAN parameters.

Consider the following:

• Route-type 2 routes with MACs and IPs are advertised. Some considerations about MAC+IP and ARP/ND entries are:

  • The 7750 SR advertises its IRB MAC+IP in a route type 2 route and possibly the VRRP vMAC+vIP if it runs VRRP and the 7750 SR is the active router. In both cases, the MACs are advertised as static MACs, therefore, protected by the receiving PEs.

  • If the 7750 SR VPRN interface is configured with one or more additional secondary IP addresses, they are all advertised in routes type 2, as static MACs.

  • The 7750 SR processes route-type 2 routes as usual, populating the FDB with the received MACs and the VPRN ARP/ND table with the MAC and IPs, respectively.

    Note: ND entries received from the EVPN are installed as Router entries. The ARP/ND entries coming from the EVPN are tagged as evpn.

  • When a VPLS containing proxy-ARP/proxy-ND entries is bound to a VPRN (allow-ip-int-bind) all the proxy-ARP/proxy-ND entries are moved to the VPRN ARP/ND table. ARP/ND entries are also moved to proxy-ARP/proxy-ND entries if the VPLS is unbound.

  • EVPN does not program EVPN-received ARP/ND entries if the receiving VPRN has no IP addresses for the same subnet. The entries are added when the IP address for the same subnet is added.

  • Static ARP/ND entries have precedence over dynamic and EVPN ARP/ND entries.

• VPRN interface binding to VPLS service brings down the VPRN interface operational status, if the VPRN interface MAC or the VRRP MAC matches a static-mac or OAM MAC configured in the associated VPLS service. If that is the case, a trap is generated.

• Redundancy is handled by VRRP. The active 7750 SR advertises vMAC and vIP, as discussed, including the MAC mobility extended community and the sticky bit.

EVPN-enabled R-VPLS services are also supported on IES interfaces.

An AR-R configuration is shown in the following example.

*A:PE-2>config>service>system>vxlan# info 
----------------------------------------------
            assisted-replication-ip 10.2.2.2
----------------------------------------------
*A:PE-2>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication replicator
            exit
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
<snip>
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with tunnel-type = AR, type (T) = AR-R, and tunnel-id = originating-ip = next-hop = assisted-replication-ip (IP address 10.2.2.2 in the preceding example). In addition to the AR route, the AR-R sends a regular IR route if ingress-repl-inc-mcast-advertisement is enabled.

The AR-R builds a flooding list composed of ACs (SAPs and SDP bindings) and VXLAN tunnels to remote nodes in the VPLS. All objects in the flooding list are broadcast/multicast (BM) and unknown unicast (U) capable. The following example output of the show service id vxlan command shows that the VXLAN destinations in the flooding list are tagged as ‟BUM”.

*A:PE-2# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: replicator
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
192.0.2.3                               4000        0           BUM   Up    No
192.0.2.5                               4000        0           BUM   Up    No
192.0.2.6                               4000        0           BUM   Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================

When the AR-R receives a BUM packet on an AC, the AR-R forwards the packet to its flooding list (including the local ACs and remote VTEPs).

When the AR-R receives a BM packet on a VXLAN tunnel, it checks the IP DA of the underlay IP header and performs the following BM packet processing.

• If the destination IP matches its AR-IP, the AR-R forwards the BM packet to its flooding list (ACs and VXLAN tunnels). The AR-R performs source suppression to ensure that the traffic is not sent back to the originating Leaf.

• If the destination IP matches its regular VXLAN termination IP (IR-IP), the AR-R skips all the VXLAN tunnels from the flooding list and only replicates to the local ACs. This is the default Ingress Replication (IR) behavior.

An AR-L is configured as shown in the following example.

A:PE-3>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 4000 create
                assisted-replication leaf replicator-activation-time 30
            bgp
            exit
            bgp-evpn
                evi 4000
                vxlan bgp 1 vxlan-instance 1
                    no shutdown
                exit
                mpls
                    shutdown
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:4000 create
                no shutdown
            exit
            no shutdown
----------------------------------------------

In this example configuration, the BGP advertises a new inclusive multicast route with a tunnel-type = IR, type (T) = AR-L and tunnel-id = originating-ip = next-hop = IR-IP (IP address terminating VXLAN normally, either system-ip or vxlan-src-vtep address).

The AR-L builds a single flooding list per service but controlled by the BM and U flags. These flags are displayed in the following show service id vxlan command example output.

A:PE-3# show service id 4000 vxlan 
===============================================================================
Vxlan Src Vtep IP: N/A
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 4000
Creation Origin: manual
Assisted-Replication: leaf      Replicator-Activation-Time: 30
RestProtSrcMacAct: none
===============================================================================
VPLS VXLAN service Network Specifics
===============================================================================
Ing Net QoS Policy : none                            Vxlan VNI Id     : 4000
Ingress FP QGrp    : (none)                          Ing FP QGrp Inst : (none)
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address                            Egress VNI  Num. MACs   Mcast Oper  L2
                                                                      State PBR
-------------------------------------------------------------------------------
10.2.2.2                                 4000        0           BM    Up    No
10.4.4.4                                 4000        0           -     Up    No
192.0.2.2                               4000        0           U     Up    No
192.0.2.5                               4000        0           U     Up    No
192.0.2.6                               4000        0           U     Up    No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 5
-------------------------------------------------------------------------------
===============================================================================

The AR-L creates the following VXLAN destinations when it receives and selects a Replicator-AR route or the Regular-IR routes:

• A VXLAN destination to each remote PE that sent an IR route. These bindings have the U flag set.

• A VXLAN destination to the selected AR-R. These bindings have only the BM flag set; the U flag is not set.

• The non-selected AR-Rs create a binding with flag ‟-” (in the CPM) that is displayed by the show service id vxlan command. Although the VXLAN destinations to non-selected AR-Rs do not carry any traffic, the destinations count against the total limit and must be considered when accounting for consumed VXLAN destinations in the router.

The BM traffic is only sent to the selected AR-R, whereas the U (unknown unicast) traffic is sent to all the destinations with the U flag.

The AR-L performs per-service load-balancing of the BM traffic when two or more AR-Rs exist in the same service. The AR Leaf creates a list of candidate PEs for each AR-R (ordered by IP and VNI; candidate 0 being the lowest IP and VNI). The replicator is selected out of a modulo function of the service-id and the number of replicators, as shown in the following example output.

A:PE-3# show service id 4000 vxlan assisted-replication replicator 
===============================================================================
Vxlan AR Replicator Candidates
===============================================================================
VTEP Address           Egress VNI     In Use  In Candidate List Pending Time
-------------------------------------------------------------------------------
10.2.2.2                4000           yes     yes               0
10.4.4.4                4000           no      yes               0
-------------------------------------------------------------------------------
Number of entries : 2
-------------------------------------------------------------------------------
===============================================================================

A change in the number of Replicator-AR routes (for example, if a route is withdrawn or a new route appears) affects the result of the hashing, which may cause a different AR-R to be selected.

The following list summarizes other aspects of the AR-L behavior:

• When a Leaf receives a BM packet on an AC, it sends the packet to its flood list that includes access SAP or SDP bindings and VXLAN destinations with BM or BUM flags. If a single AR-R is selected, only a VXLAN destination includes the BM flags.

• Control plane-generated BM packets, such as ARP/ND (when proxy-ARP/ND is enabled) or Eth-CFM, follow the behavior of regular data plane BM packets.

• When a Leaf receives an unknown unicast packet on an AC, it sends the packet to the flood-list, skipping the AR destination because the U flag is set to 0. To avoid packet re-ordering, the unknown unicast packets do not go through the AR-R.

• When a Leaf receives a BUM packet on an overlay tunnel, it forwards the packet to the flood list, skipping the VXLAN tunnels (that is, the packet is sent to the local ACs and never to a VXLAN tunnel). This is the default IR behavior.

• When the last Replicator-AR route is withdrawn, the AR-L removes the AR destination from the flood list and falls back to ingress replication.

AR BM replication behavior for a BM packet shows the expected replication behavior for BM traffic when received at the access on an AR-R, AR-L, or RNVE router. Unknown unicast follows regular ingress replication behavior regardless of the role of the ingress node for the specific service.

The Assisted-Replication feature has the following limitations:

• The following features are not supported on the same service where the Assisted-Replication feature is enabled.

  • Aggregate QoS per VNI

  • VXLAN IPv6 transport

  • IGMP/MLD/PIM-snooping

• Assisted-Replication Leaf and Replicator functions are mutually exclusive within the same VPLS service.

• The Assisted-Replication feature is supported with IPv4 non-system-ip VXLAN termination. However, the configured assisted-replication-ip (AR-IP) must be different from the tunnel termination IP address.

• The AR-IP address must be a /32 loopback interface on the base router.

• The Assisted-Replication feature is only supported in EVPN-VXLAN services (VPLS with BGP-EVPN vxlan enabled). Although services with a combination of EVPN-MPLS and EVPN-VXLAN are supported, the Assisted-Replication configuration is only relevant to the VXLAN.

PBF to ESI function shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on VPLS services. In this example, the DC gateway, PE1, is connected to an L2-DOMAIN that exists in the DC and must redirect the traffic to the Service Function SF-1. The regular Layer 2 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

An operator must configure a PBF match/action filter policy entry in an IPv4 or MAC ingress access or network filter deployed on a VPLS interface using CLI/SNMP/NETCONF management interfaces. The PBF target is the first service function in the chain (SF-1) that is identified by an ESI.

In the example shown in PBF to ESI function, the PBF filter redirects the matching packets to ESI 0x01 in VPLS-1.

As soon as the redirection target is configured and associated with the vport connected to SF-1, the Nuage VSC (Virtual Services Controller, or the remote PE3 in the example) advertises the location of SF-1 via an Auto-Discovery Ethernet Tag route (route type 1) per-EVI. In this AD route, the ESI associated with SF-1 (ESI 0x01) is advertised along with the VTEP (PE3's IP) and VNI (VNI-1) identifying the vport where SF-1 is connected. PE1 sends all the frames matching the ingress filter to PE3's VTEP and VNI-1.

The following filter configuration shows an example of PBF rule redirecting all the frames to an ESI.

A:PE1>config>filter>mac-filter# info 
----------------------------------------------
            default-action forward
            entry 10 create
                action
                    forward esi ff:00:00:00:00:00:00:00:00:01 service-id 301
                exit
            exit

When the filter is properly applied to the VPLS service (VPLS-301 in this example), it shows 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported.

A:PE1# show filter mac 1 
===============================================================================
Mac Filter
===============================================================================
Filter Id   : 1                                Applied         : Yes
Scope       : Template                         Def. Action     : Forward
Entries     : 1                                Type            : normal
Description : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : Mac
-------------------------------------------------------------------------------
Entry       : 10                               FrameType       : Ethernet
Description : (Not Specified)
Log Id      : n/a                              
Src Mac     : Undefined
Dest Mac    : Undefined
Dot1p       : Undefined                        Ethertype       : Undefined
DSAP        : Undefined                        SSAP            : Undefined
Snap-pid    : Undefined                        ESnap-oui-zero  : Undefined
Match action: Forward (ESI) Active             
  ESI       : ff:00:00:00:00:00:00:00:00:01
  Svc Id    : 301                              
PBR Down Act: Forward (entry-default)          
Ing. Matches: 3 pkts
Egr. Matches: 0 pkts
===============================================================================

A:PE1# show service id 301 es-pbr 
===============================================================================
L2 ES PBR
===============================================================================
ESI                           Users      Status
                                         VTEP:VNI
-------------------------------------------------------------------------------
ff:00:00:00:00:00:00:00:00:01 1          Active
                                         192.0.2.72:7272
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
===============================================================================

Details of the received AD route that resolves the filter forwarding are shown in the following show router bgp routes command.

A:PE1# show router bgp routes evpn auto-
disc esi ff:00:00:00:00:00:00:00:00:01                                    
===============================================================================
 BGP Router ID:192.0.2.71       AS:64500       Local AS:64500      
===============================================================================
 Legend -
 Status codes  : u - used, s - suppressed, h - history, d - decayed, * - valid
                 l - leaked, x - stale, > - best, b - backup
 Origin codes  : i - IGP, e - EGP, ? - incomplete
===============================================================================
BGP EVPN Auto-Disc Routes
===============================================================================
Flag  Route Dist.         ESI                           NextHop
      Tag                                               Label
-------------------------------------------------------------------------------

u*>i  192.0.2.72:100      ff:00:00:00:00:00:00:00:00:01 192.0.2.72
      0                                                 VNI 7272

-------------------------------------------------------------------------------
Routes : 1
=============================================================

This AD route, when used for PBF redirection, is added to the list of EVPN-VXLAN bindings for the VPLS service and shown as 'L2 PBR' type:

A:PE1# show service id 301 vxlan 
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 301
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address           Egress VNI     Num. MACs    Mcast   Oper State   L2 PBR
-------------------------------------------------------------------------------
192.0.2.69             301            1            Yes     Up           No
192.0.2.72             301            1            Yes     Up           No
192.0.2.72             7272           0            No      Up           Yes
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================

If the AD route is withdrawn, the binding disappears and the filter is inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

PBR to ESI function shows the 7750 SR, 7450 ESS, and 7950 XRS Service Chaining integration with the Nuage VSP on L2-DOMAIN-IRB domains. In this example, the DC gateway, PE1, is connected to an L2-DOMAIN-IRB that exists in the DC and must redirect the traffic to the Service Function SF-1 with IP address 10.10.10.1. The regular Layer 3 forwarding procedures would have taken the packets to PE2, as opposed to SF-1.

In this case, an operator must configure a PBR match/action filter policy entry in an IPv4 ingress access or network filter deployed on IES/VPRN interface using CLI, SNMP or NETCONF management interfaces. The PBR target identifies first service function in the chain (ESI 0x01 in PBR to ESI function, identifying where the Service Function is connected and the IPv4 address of the SF) and EVPN VXLAN egress interface on the PE (VPRN routing instance and R-VPLS interface name). The BGP control plane together with ESI PBR configuration are used to forward the matching packets to the next-hop in the EVPN-VXLAN data center chain (through resolution to a VNI and VTEP). If the BGP control plane information is not available, the packets matching the ESI PBR entry is, by default, forwarded using regular routing. Optionally, an operator can select to drop the packets when the ESI PBR target is not reachable.

The following filter configuration shows an example of a PBR rule redirecting all the matching packets to an ESI.

*A:PE1>config>filter>ip-filter# info 
----------------------------------------------
            default-action forward
            entry 10 create
                match 
                    dst-ip 10.10.10.253/32
                exit 
                action
                    forward esi ff:00:00:00:00:21:5f:00:df:e5 sf-ip 10.10.10.1 vas-
interface "evi-301" router 300
                exit
                pbr-down-action-override filter-default-action
            exit 
----------------------------------------------

In this use case, the following are required in addition to the ESI: the sf-ip (10.10.10.1 in the example above), router instance (300), and vas-interface.

The sf-ip is used by the system to know which inner MAC DA it has to use when sending the redirected packets to the SF. The SF-IP is resolved to the SF MAC following regular ARP procedures in EVPN-VXLAN.

The router instance may be the same as the one where the ingress filter is configured or may be different: for instance, the ingress PBR filter can be applied on an IES interface pointing at a VPRN router instances that is connected to the DC fabric.

The vas-interface refers to the R-VPLS interface name through which the SF can be found. The VPRN instance may have more than one R-VPLS interface, therefore, it is required to specify which R-VPLS interface to use.

When the filter is properly applied to the VPRN or IES service (VPRN-300 in this example), it shows 'Active' in the following show commands as long as the Auto-Discovery route for the ESI is received and imported and the SF-IP resolved to a MAC address.

*A:PE1# show filter ip 1 

===============================================================================
IP Filter
===============================================================================
Filter Id    : 1                                Applied        : Yes
Scope        : Template                         Def. Action    : Forward
System filter: Unchained                        
Radius Ins Pt: n/a                              
CrCtl. Ins Pt: n/a                              
RadSh. Ins Pt: n/a                              
PccRl. Ins Pt: n/a                              
Entries      : 1                                
Description  : (Not Specified)
-------------------------------------------------------------------------------
Filter Match Criteria : IP
-------------------------------------------------------------------------------
Entry        : 10
Description  : (Not Specified)
Log Id       : n/a                              
Src. IP      : 0.0.0.0/0
Src. Port    : n/a
Dest. IP     : 10.16.0.253/32
Dest. Port   : n/a
Protocol     : Undefined                        Dscp           : Undefined
ICMP Type    : Undefined                        ICMP Code      : Undefined
Fragment     : Off                              Src Route Opt  : Off
Sampling     : Off                              Int. Sampling  : On
IP-Option    : 0/0                              Multiple Option: Off
TCP-syn      : Off                              TCP-ack        : Off
Option-pres  : Off                              
Egress PBR   : Undefined                        
Match action : Forward (ESI) Active
  ESI        : ff:00:00:00:00:21:5f:00:df:e5    
  SF IP      : 10.10.10.1
  VAS If name: evi-301                          
  Router     : 300                              
PBR Down Act : Forward (filter-default-action) Ing. Matches : 3 pkts (318 bytes)
Egr. Matches : 0 pkts
===============================================================================

*A:PE1# show service id 300 es-pbr 
===============================================================================
L3 ES PBR
===============================================================================
SF IP              ESI                                 Users Status
                   Interface                                 MAC
                                                             VTEP:VNI
-------------------------------------------------------------------------------
10.10.10.1         ff:00:00:00:00:21:5f:00:df:e5       1     Active
                   evi-301                                   d8:47:01:01:00:0a
                                                             192.0.2.71:7171
-------------------------------------------------------------------------------
Number of entries : 1
-------------------------------------------------------------------------------
=================================================================================

In the FDB for the R-VPLS 301, the MAC address is associated with the VTEP and VNI specified by the AD route, and not by the MAC/IP route anymore. When a PBR filter with a forward action to an ESI and SF-IP (Service Function IP) exists, a MAC route is auto-created by the system and this route has higher priority that the remote MAC, or IP routes for the MAC (see BGP and EVPN route selection for EVPN routes).

The following shows that the AD route creates a new EVPN-VXLAN binding and the MAC address associated with the SF-IP uses that 'binding':

*A:PE1# show service id 301 vxlan 
===============================================================================
VPLS VXLAN, Ingress VXLAN Network Id: 301
===============================================================================
Egress VTEP, VNI
===============================================================================
VTEP Address           Egress VNI     Num. MACs    Mcast   Oper State   L2 PBR
-------------------------------------------------------------------------------
192.0.2.69             301            1            Yes     Up           No
192.0.2.71             301            0            Yes     Up           No
192.0.2.71             7171           1            No      Up           No
-------------------------------------------------------------------------------
Number of Egress VTEP, VNI : 3
-------------------------------------------------------------------------------
===============================================================================
*A:PE1# show service id 301 fdb detail 
===============================================================================
Forwarding Database, Service 301
===============================================================================
ServId    MAC               Source-Identifier        Type     Last Change
                                                     Age      
-------------------------------------------------------------------------------
301       d8:45:ff:00:00:6a vxlan-1:                 EvpnS    06/15/15 21:55:27
                            192.0.2.69:301
301       d8:47:01:01:00:0a vxlan-1:                 EvpnS    06/15/15 22:32:56
                            192.0.2.71:7171
301       d8:48:ff:00:00:6a cpm                      Intf     06/15/15 21:54:12
-------------------------------------------------------------------------------
No. of MAC Entries: 3
-------------------------------------------------------------------------------
Legend:  L=Learned O=Oam P=Protected-MAC C=Conditional S=Static
===============================================================================

For Layer 2, if the AD route is withdrawn or the SF-IP ARP not resolved, the filter is inactive again. The user can control whether the matching packets are dropped or forwarded if the PBF target cannot be resolved by BGP.

EVPN MPLS, as described in EVPN for MPLS tunnels, uses ESI-labels to identify the BUM traffic sourced from a specified ES. The egress PE performs a label lookup to find the ESI label below the EVI label and to determine if a frame can be forwarded to a local ES. Because VXLAN does not support ESI-labels, or any MPLS label for that matter, the split-horizon filtering must be based on the tunnel source IP address. This also implies that the SAP-to-SAP forwarding rules must be changed when the SAPs belong to local ESs, irrespective of the DF state. This new forwarding is what RFC 8365 refers to as local bias. EVPN-VXLAN multihoming with local bias illustrates the local bias forwarding behavior.

Local bias is based on the following principles:

• Every PE knows the IP addresses associated with the other PEs with which it has shared multihomed ESs.

• When the PE receives a BUM frame from a VXLAN bind, it looks up the source IP address in the tunnel header and filters out the frame on all local interfaces connected to ESs that are shared with the ingress PE.

With this approach, the ingress PE must perform replication locally to all directly-attached ESs (regardless of the DF Election state) for all flooded traffic coming from the access interfaces. BUM frames received on any SAP are flooded to:

• local non-ES SAPs and non-ES SDP-binds

• local all-active ES SAPs (DF and NDF)

• local single-active ES SDP-binds and SAPs (DF only)

• EVPN-VXLAN destinations

As an example, in EVPN-VXLAN multihoming with local bias, PE2 receives BUM traffic from Host-3 and it forwards it to the remote PEs and the local ES SAP, even though the SAP is in NDF state.

The following rules apply to egress PE forwarding for EVPN-VXLAN services:

• The source VTEP is looked up for BUM frames received on EVPN-VXLAN.

• If the source VTEP matches one of the PEs with which the local PE shares both an ES and a VXLAN service:

  • the local PE is not forwarded to the shared ES local SAPs

  • the local PE forwards normally to ES SAPs unless they are in NDF state

• Because there is no multicast label or multicast B-MAC in VXLAN, the egress PE only identifies BUM traffic using the customer MAC DA; as a result, BM or unknown MAC DAs identify BUM traffic.

For example, in EVPN-VXLAN multihoming with local bias, PE3 receives BUM traffic on VXLAN. PE3 identifies the source VTEP as a PE with which two ESs are shared, therefore it does not forward the BUM frames to the two shared ESs. It forwards to the non-shared ES (Host-5) because it is in DF state. PE4 receives BUM traffic and forwards it based on normal rules because it does not share any ESs with PE2.

The following command can be used to check whether the local PE has enabled the local bias procedures for a specific ES:

A:PE-2# tools dump service system bgp-evpn ethernet-segment "ES-1" local-bias 
-------------------------------------------------------------------------------
[04/01/2019 08:45:08] Vxlan Local Bias Information 
----------------------------------------------------------------------+--------
Peer                                                                  | Enabled   
----------------------------------------------------------------------+--------
192.0.2.3                                                             | Yes       
-------------------------------------------------------------------------------

In EVPN MPLS networks, an ingress PE that uses ingress replication to flood unknown unicast traffic pushes a BUM MPLS label that is different from a unicast label. The egress PEs use this BUM label to identify such BUM traffic to apply DF filtering for All-Active multihomed sites. In PBB-EVPN, in addition to the multicast label, the egress PE can also rely on the multicast B-MAC DA to identify customer BUM traffic.

In VXLAN there are no BUM labels or any tunnel indication that can assist the egress PE in identifying the BUM traffic. As such, the egress PE must solely rely on the C-MAC destination address, which may create some transient issues that are depicted in EVPN-VXLAN multihoming and unknown unicast issues.

As shown in EVPN-VXLAN multihoming and unknown unicast issues, top diagram, in absence of the mentioned unknown unicast traffic indication there can be transient duplicate traffic to All-Active multihomed sites under the following condition: CE1’s MAC address is learned by the egress PEs (PE1 and PE2) and advertised to the ingress PE3; however, the MAC advertisement has not been received or processed by the ingress PE, resulting in the host MAC address to be unknown on the ingress PE3 but known on the egress PEs. Therefore, when a packet destined for CE1 address arrives on PE3, it floods it through ingress replication to PE1 or PE2 and, because CE1’s MAC is known to PE1 and PE2, multiple copies are sent to CE1.

Another issue is shown at the bottom of EVPN-VXLAN multihoming and unknown unicast issues. In this case, CE1’s MAC address is known on the ingress PE3 but unknown on PE1 and PE2. If PE3’s aliasing hashing picks up the path to the ES’ NDF, a black-hole occurs.

The above two issues are solved in MPLS, as unicast known and unknown frames are identified with different labels.

Finally, another issue is described in Blackhole created by a remote SAP shutdown. Under normal circumstances, when CE3 sends BUM traffic to PE3, the traffic is ‟local-biased” to PE3’s SAP3 even though it is NDF for the ES. The flooded traffic to PE2 is forwarded to CE2, but not to SAP2 because the local bias split-horizon filtering takes place.

The right side of the diagram in Blackhole created by a remote SAP shutdown shows an issue when SAP3 is manually shutdown. In this case, PE3 withdraws the AD per-EVI route corresponding to SAP3; however, this does not change the local bias filtering for SAP2 in PE2. Therefore, when CE3 sends BUM traffic, it can neither be forwarded to CE23 via local SAP3 nor can it be forwarded by PE2.

EVPN VXLAN multihoming is supported on VPLS and R-VPLS services when the PEs use non-system IPv4 or IPv6 termination, however, as with EVPN VPWS services, additional configuration steps are required.

• The configure service system bgp-evpn eth-seg es-orig-ip ip-address command must be configured with the non-system IPv4 or IPv6 address used for the EVPN-VXLAN service. This command modifies the originating-ip field in the ES routes advertised for the Ethernet Segment, and makes the system use this IP address when adding the local PE as DF candidate.

• The configure service system bgp-evpn eth-seg route-next-hop ip-address command must also be configured with the non-system IP address. This command changes the next-hop of the ES and AD per-ES routes to the configured address.

• Finally, the non-system IP address (in each of the PEs in the ES) must match in these three commands for the local PE to be considered suitable for DF election:

  • es-orig-ip ip-address

  • route-next-hop ip-address

  • vxlan-src-vtep ip-address

The 7750 SR, 7450 ESS, or 7950 XRS router SR OS EVPN implementation supports RFC 8560 so that EVPN-MPLS and VPLS can be integrated into the same network and within the same service. Because EVPN is not deployed in green-field networks, this feature is useful for the integration between both technologies and even for the migration of VPLS services to EVPN-MPLS.

The following behavior enables the integration of EVPN and SDP-bindings in the same VPLS network.

Systems with EVPN endpoints and SDP-bindings to the same far-end bring down the SDP-bindings.

• The router allows the establishment of an EVPN endpoint and an SDP-binding to the same far-end, but the SDP-binding is kept operationally down. Only the EVPN endpoint is operationally up. This applies to spoke-SDPs (manual, BGP-AD, and BGP-VPLS) and mesh-SDPs. It is also possible between VXLAN and SDP bindings.

• If there is an existing EVPN endpoint to a specified far-end and a spoke-SDP establishment is attempted, the spoke-SDP is set up but kept down with an operational flag indicating that there is an EVPN route to the same far-end.

• If there is an existing spoke-SDP and a valid or used EVPN route arrives, the EVPN endpoint is set up and the spoke-SDP is brought down with an operational flag indicating that there is an EVPN route to the same far-end.

• In the case of an SDP-binding and EVPN endpoint to different far-end IPs on the same remote PE, both links are up. This may occur if the SDP-binding is terminated in an IPv6 or IPv4 address that is different from the system address where the EVPN endpoint is terminated.

The user can add spoke-SDPs and all the EVPN-MPLS endpoints in the same SHG.

• A CLI command is added under the bgp-evpn>mpls context so that the EVPN-MPLS endpoints can be added to an SHG:

  bgp-evpn mpls [no] split-horizon-group group-name

• The bgp-evpn mpls split-horizon-group command must reference a user-configured SHG. User-configured SHGs can be configured within the service context. The same group-name can be associated with SAPs, spoke-SDPs, pw-templates, pw-template-bindings, and EVPN-MPLS endpoints.

• If the bgp-evpn mpls split-horizon-group command is not used, the default SHG (that contains all the EVPN endpoints) is still used but cannot be referred to on SAPs/spoke-SDPs.

• SAPs and SDP-bindings that share the same SHG of the EVPN-MPLS provider-tunnel are brought operationally down if the point-to-multipoint tunnel is operationally up.

The system disables the advertisement of MACs learned on spoke-SDPs or SAPs that are part of an EVPN SHG.

• When the SAPs and spoke-SDPs (manual or BGP-AD/VPLS-discovered) are configured within the same SHG as the EVPN endpoints, MAC addresses are still learned on them but they are not advertised in EVPN.

• The preceding statement is also true if proxy-ARP/ND is enabled and an IP→MAC pair is learned on a SAP or SDP-binding that belongs to the EVPN SHG.

• The SAPs and spoke-SDPs added to an EVPN SHG should not be part of any EVPN multihomed ES. If that occurs, the PE would still advertise the AD per-EVI route for the SAP or spoke-SDP, attracting EVPN traffic that could not possibly be forwarded to that SAP or SDP-binding.

• Similar to the preceding statement, an SHG composed of SAPs/SDP-bindings used in a BGP-MH site should not be configured under bgp-evpn>mpls>split-horizon-group. This misconfiguration would prevent forwarding of traffic from the EVPN to the BGP-MH site, regardless of the DF/NDF state.

  The following figure shows an example of EVPN-VPLS integration.

  

  The following is an example configuration output of PE1, PE5, and PE2.

*A:PE1>config>service# info 
----------------------------------------------
pw-template 1 create
vpls 1 customer 1 create
  split-horizon-group "SHG-1" create 
  bgp
    route-target target:65000:1
    pw-template-binding 1 split-horizon-group SHG-1 
  bgp-ad
    no shutdown
    vpls-id 65000:1
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      split-horizon-group SHG-1
  spoke-sdp 12:1 create
  exit
  sap 1/1/1:1 create
  exit

*A:PE5>config>service# info 
----------------------------------------------
pw-template 1 create
vpls 1 customer 1 create
  bgp
    route-target target:65000:1
    pw-template-binding 1 split-horizon-group SHG-1 # auto-created SHG
  bgp-ad
    no shutdown
    vpls-id 65000:1
  spoke-sdp 52:1 create
  exit

*A:PE2>config>service# info 
----------------------------------------------
vpls 1 customer 1 create
  end-point CORE create
    no suppress-standby-signaling
  spoke-sdp 21:1 end-point CORE
    precedence primary
  spoke-sdp 25:1 end-point CORE

The following applies to the configuration described in the preceding example.

• PE1, PE3, and PE4 have BGP-EVPN and BGP-AD enabled in VPLS-1. PE5 has BGP-AD enabled and PE2 has active/standby spoke-SDPs to PE1 and PE5. The following applies to this configuration.

  • PE1, PE3, and PE4 attempt to establish BGP-AD spoke-SDPs, but they are kept operationally down as long as there are EVPN endpoints active among them.

  • BGP-AD spoke-SDPs and EVPN endpoints are instantiated within the same split horizon group, for example, SHG-1.

  • Manual spoke-SDPs from PE1 and PE5 to PE2 are not part of SHG-1.

• EVPN MAC advertisements

  • MACs learned on FEC128 spoke-SDPs are advertised normally in EVPN.

  • MACs learned on FEC129 spoke-SDPs are not advertised in EVPN (because they are part of SHG-1, which is the split horizon group used for bgp-evpn mpls). This prevents any data plane MACs learned on the SHG from being advertised in EVPN.

• BUM operation on PE1

  • When CE1 sends BUM, PE1 floods to all the active bindings.

  • When CE2 sends BUM, PE2 sends it to PE1 (active spoke-SDP) and PE1 floods to all the bindings and SAPs.

  • When CE5 sends BUM, PE5 floods to the three EVPN PEs. PE1 floods to the active spoke-SDP and SAPs, never to the EVPN PEs because they are part of the same SHG.

The operation in services with BGP-VPLS and BGP-EVPN is equivalent to the operation for BGP-AD and BGP EVPN described in EVPN and VPLS integration.

In a VPLS service to which multiple EVPN PEs and BGP-VPLS PEs are attached, single-active multihoming is supported on two or more of the EVPN PEs with no special considerations. However, all-active multihoming is not supported because traffic from the all-active multihomed CE could cause a MAC flip-flopping effect on remote BGP-VPLS PEs, asymmetric flows, or other issues.

The following figure shows a scenario with a single-active ES used in a service where EVPN PEs and BGP-VPLS are integrated.

Although other single-active examples are supported, in the preceding figure, CE1 is connected to the EVPN PEs via a single LAG (lag-1). The LAG is associated with eth-segment-1 on PE1 and PE2, which is configured as single-active and with oper-group 1. PE1 and PE2 use lag>monitor-oper-group 1 so that the non-DF PE can signal the non-DF state to CE1 (in the form of Link Aggregation Control Protocol (LACP) out-of-synch or power-off).

In addition to the BGP-VPLS routes sent for the service virtual-edge (VE) ID, the multihoming PEs, in this case, must generate additional BGP-VPLS routes per ES (per VPLS service) to generate a MAC flush on the remote BGP-VPLS PEs in case of failure.

The sap>bgp-vpls-mh-veid number command should be configured on the SAPs that are part of an EVPN single-active ES. This command allows the advertisement of L2VPN routes that indicate the state of the multihomed SAPs to the remote BGP-VPLS PEs. After a DF switchover, the F and D bits of the generated L2VPN routes for the SAP VE ID are updated so that the remote BGP-VPLS PEs can perform a MAC flush operation on the service and avoid blackholes.

For example, in case of a failure on the ES SAP on PE1, PE1 must indicate to PE3 and PE4 the need to flush MAC addresses learned from PE1 (flush-all-from-me message). Otherwise, PE3 continues to send traffic with MAC DA = CE1 to PE1, and PE1 blackholes the traffic.

The following applies to the example in BGP-VPLS to EVPN integration and single-active MH.

• Both ES peers (PE1 and PE2) should be configured with the same VE ID for the ES SAP. However, this is not mandatory.

• In addition to the regular service ve-id L2VPN route, based on the sap>bgp-vpls-mh-ve-id configuration and on BGP VPLS being enabled, the PE advertises an L2VPN route with the following fields:

  • VE ID that contains the value configured using the sap bgp-vpls-mh-ve-id command

  • RD, RT, next hop, and other attributes that are the same as the service BGP VPLS route

  • L2VPN information extended community with the following flags:

    • D=0

      This value applies If the SAP is oper-up or oper-down with a flag MHStandby (for example, the PE is non-DF in single-active MH).

      This value also applies if there is an ES oper-group and the port is down as a result of the oper-group.

    • D=1

      This value applies if the SAP is oper-down with a different flag (for example, port-down or admin-down).

    • F (DF bit) =1

      This value applies if the SAP is oper-up. Otherwise, the value is F=0.

• After a failure on the access SAP, only MAC flush messages are triggered if the bgp-vpls-mh-ve-id command is configured in the failing SAP. If the SAP is configured with VE ID 1, the following applies.

  • If the non-DF PE has a failure on the access SAP, PE2 sends an update with VE ID=1/D=1/F=0. This is an indication for PE3 and PE4 that the PE2 SAP is oper-down but triggering a MAC flush on PE3 and PE4 is not required.

  • If the DF PE has a failure on the SAP, PE1 advertises VE ID=1/D=1/F=0. After receiving this update, PE3 and PE4 flush all their MACs associated with the PE1 spoke-SDP. The failure on PE1 triggers an EVPN DF election on PE2, which becomes DF and advertises VE ID=1/D=0/F=1. This message does not trigger any MAC flush procedures.

The following considerations apply for EVPN single-active multihoming and BGP-VPLS integration.

• PE3 and PE4 can be Nokia nodes running SR OS or any third-party PEs that support the procedures defined in draft-ietf-bess-vpls-multihoming, such that BGP-VPLS MAC flush signaling is understood.

• PE1 and PE2 are expected to run an SR OS version that supports the configuration of the sap bgp-vpls-mh-veid number command on multihomed SAPs. Otherwise, the MAC flush behavior does not work as expected.

• The procedures described in this section are also supported if the EVPN PEs use MC-LAG instead of an ES for CE1 redundancy. In this case, the SAP VE ID route for the standby PE is sent as VE ID=1/D=1/F=0, whereas the active chassis advertises VE ID=1/D=0/F=1. A switchover triggers a MAC flush on the remote PEs, as described in EVPN single-active multihoming and BGP-VPLS integration.

• L2VPN routes generated for ESs or SAPs with the sap bgp-vpls-mh-veid number command are decoded in the remote nodes as BGP-MH routes (because they do not have label information) in the show router bgp routes l2vpn command and debug.

Multiple BGP families and protocols can be enabled at the same time in a VPLS. When bgp-evpn is enabled, bgp-ad and bgp-mh are also supported. A single RD is used per service and not per BGP family or protocol.

The following rules apply.

• The VPLS RD is selected based on the following precedence.

  • Manual RD or automatic RD always take precedence when configured.

  • If manual or automatic RD is not configured, the RD is derived from the bgp-ad>vpls-id.

  • If manual RD, automatic RD, or VPLS ID are not configured, the RD is derived from the bgp-evpn>evi, except for bgp-mh, which does not support EVI-derived RD, and except when the EVI is greater than 65535. In these two cases, no EVI-derived RD is possible.

  • If manual RD, automatic RD, VPLS ID, or EVI is not configured, there is no RD and the service fails.

• The selected RD (see the preceding selection criteria) is displayed by the Oper Route Dist field of the show service id bgp command.

• The service supports dynamic RD changes. For example, the CLI allows the dynamic update of the VPLS ID, even if it is used to automatically derive the service RD for bgp-ad, bgp-vpls, or bgp-mh.

  Note: When the RD changes, the active routes for that VPLS are withdrawn and readvertised with the new RD.

• If one of the mechanisms to derive the RD for a specified service is removed from the configuration, the system selects a new RD based on the preceding rules. For example, if the VPLS ID is removed from the configuration, the routes are withdrawn, the new RD selected from the EVI, and the routes readvertised with the new RD.

  Note: The reconfiguration fails if the new RD already exists in a different VPLS or Epipe service.

• Because the vpls-id takes precedence over the evi when the RD is auto-derived, the existing RD is not affected when evpn is added to an existing bgp-ad service. This is important to support bgp-ad to evpn migration.

EVPN multihoming implementation is based on the concept of the Ethernet segment (ES). An ES is a logical structure that can be defined in one or more PEs and identifies the CE (or access network) multihomed to the EVPN PEs. An ES is associated with port, LAG, PW port, or SDP objects and shared by all the services defined on those objects. In the case of virtual ESs, individual VID or VC-ID ranges can be associated with the port, LAG, or PW port, SDP objects defined in the ES.

Each ES has a unique Ethernet Segment Identifier (ESI) that is 10 bytes long and is manually configured in the router.

This section describes the behavior of the EVPN multihoming implementation in an EVPN-MPLS service.

PBB-EVPN uses a reduced subset of the routes and procedures described in RFC 7432. The supported routes are:

• ES routes

• MAC/IP routes

• Inclusive Multicast Ethernet Tag routes

The 7750 SR, 7450 ESS, and 7950 XRS SR OS implementation of PBB-EVPN reuses the existing PBB-VPLS model, where N I-VPLS (or Epipe) services can be linked to a B-VPLS service. BGP-EVPN is enabled in the B-VPLS and the B-VPLS becomes an EVI (EVPN Instance). PBB-EVPN for I-VPLS and PBB Epipe services shows the PBB-EVPN model in the SR OS.

Each PE in the B-VPLS domain advertises its source-bmac as either configured in vpls>pbb>source-bmac or auto-derived from the chassis MAC. The remote PEs install the advertised B-MACs in the B-VPLS FDB. If a specified PE is configured with an ethernet-segment associated with an I-VPLS or PBB Epipe, it may also advertise an es-bmac for the Ethernet-Segment.

In the example shown in PBB-EVPN for I-VPLS and PBB Epipe services, when a frame with MAC DA = AA gets to PE1, a MAC lookup is performed on the I-VPLS FDB and B-MAC-34 is found. A B-MAC lookup on the B-VPLS FDB yields the next-hop (or next-hops if the destination is in an all-active Ethernet-Segment) to which the frame is sent. As in PBB-VPLS, the frame is encapsulated with the corresponding PBB header. A label specified by EVPN for the B-VPLS and the MPLS transport label are also added.

If the lookup on the I-VPLS FDB fails, the system sends the frame encapsulated into a PBB packet with B-MAC DA = Group B-MAC for the ISID. That packet is distributed to all the PEs where the ISID is defined and contains the EVPN label distributed by the Inclusive Multicast routes for that ISID, as well as the transport label.

For PBB-Epipes, all the traffic is sent in a unicast PBB packet to the B-MAC configured in the pbb-tunnel.

The following CLI output shows an example of the configuration of an I-VPLS, PBB-Epipe, and their corresponding B-VPLS.

*A:PE-1>config# 

service vpls 1 name "b-vpls1" b-vpls create
  description "pbb-evpn-service"
  service-mtu 2000
  pbb   
    source-bmac 00:00:00:00:00:03 
  bgp
  bgp-evpn
    evi 1
    mpls bgp 1
      no shutdown
      auto-bind-tunnel resolution any
  sap 1/1/1:1 create 
  exit
  spoke-sdp 1:1 create


*A:PE-1>config# 

service vpls 101 name "vpls101" i-vpls create
  pbb
    backbone-vpls 1
  sap 1/2/1:101 create
  spoke-sdp 1:102 create

*A:PE-1>config# 

service epipe 102 name "epipe102" create
  pbb
    tunnel 1 backbone-dest-mac 00:00:00:00:00:01 isid 102
  sap 1/2/1:102 create

Configure the bgp-evpn context as described in section EVPN for MPLS tunnels in VPLS services.

Some EVPN configuration options are not relevant to PBB-EVPN and are not supported when BGP-EVPN is configured in a B-VPLS; these are as follows:

• bgp-evpn> [no] ip-route-advertisement

• bgp-evpn> [no] unknown-mac-route

• bgp-evpn> vxlan [no] shutdown

• bgp-evpn>mpls>force-vlan-vc-forwarding

When bgp-evpn>mpls no shutdown is added to a specified B-VPLS instance, the following considerations apply:

• BGP-AD is supported along with EVPN in the same B-VPLS instance.

• The following B-VPLS and BGP-EVPN commands are fully supported:

  • vpls>backbone-vpls

  • vpls>backbone-vpls>send-flush-on-bvpls-failure

  • vpls>backbone-vpls>source-bmac

  • vpls>backbone-vpls>use-sap-bmac

  • vpls>backbone-vpls>use-es-bmac (For more information, see PBB-EVPN multihoming in I-VPLS and PBB Epipe services)

  • vpls>isid-policies

  • vpls>static-mac

  • vpls>SAP or SDP-binding>static-isid

  • bgp-evpn>mac-advertisement - this command affects the 'learned' B-MACs on SAPs or SDP bindings and not on the system B-MAC or SAP/es-bmacs being advertised.

  • bgp-evpn>mac-duplication and settings.

  • bgp-evpn>mpls>auto-bind-tunnel and options.

  • bgp-evpn>mpls>ecmp

  • bgp-evpn>mpls>control-word

  • bgp-evpn>evi

  • bgp-evpn>mpls>ingress-replication-bum-label

EVPN-VPWS for MPLS tunnels uses the RFC 8214 BGP extensions described in EVPN-VPWS for VXLAN tunnels, with the following differences for the Ethernet AD per-EVI routes:

• The MPLS field encodes an MPLS label as opposed to a VXLAN VNI.

• The C flag is set if the control word is configured in the service.

• The F flag is set if the hash label is configured in the service.

The use and configuration of EVPN-VPWS services is described in EVPN-VPWS for VXLAN tunnels with the following differences when the EVPN-VPWS services use MPLS tunnels instead of VXLAN.

When MPLS tunnels are used, the bgp-evpn>mpls context must be configured in the Epipe. As an example, if Epipe 2 is an EVPN-VPWS service that uses MPLS tunnels between PE2 and PE4, this would be its configuration:

PE2>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-attachment-circuit "AC-1" 
    eth-tag 200
exit
 remote-attachment-circuit "AC-2" 
    eth-tag 200
exit
  mpls bgp 1
    ecmp 2
    no shutdown
exit
sap 1/1/1:1 create

PE4>config>service>epipe(2)#
-----------------------
bgp
exit
bgp-evpn
  evi 2
  local-attachment-circuit "AC-2" 
    eth-tag 200
exit
  remote-attachment-circuit "AC-1" 
    eth-tag 100
exit
  mpls bgp 1
    ecmp 2
    no shutdown
exit
spoke-sdp 1:1

Where the following BGP-EVPN commands, specific to MPLS tunnels, are supported in the same way as in VPLS services:

• mpls auto-bind-tunnel

• mpls control-word

• mpls hash-label

• mpls entropy-label

• mpls force-vlan-vc-forwarding

• mpls shutdown

EVPN-VPWS Epipes with MPLS tunnels can also be configured with the following characteristics:

• Access attachment circuits can be SAPs or spoke SDPs. Manually configured and BGP-VPWS spoke SDPs are supported. The VC switching configuration is not supported on BGP-EVPN-enabled pipes.

• EVPN-VPWS Epipes using null SAPs can be configured with sap>ethernet>llf. When enabled, upon removing the EVPN destination, the port is brought oper-down with flag LinkLossFwd, however the AD per EVI route for the SAP is still advertised (the SAP is kept oper-up). When the EVPN destination is created, the port is brought oper-up and the flag cleared.

• EVPN-VPWS Epipes for MPLS tunnels support endpoints. The parameter endpoint endpoint name is configurable along with bgp-evpn>local-attachment-circuit and bgp-evpn>remote-attachment-circuit. The following conditions apply to endpoints on EVPN-VPWS Epipes with MPLS tunnels:

  • Up to two explicit endpoints are allowed per Epipe service with BGP-EVPN configured.

  • A limited endpoint configuration is allowed in Epipes with BGP-EVPN. Specifically, neither active-hold-delay nor revert-time are configurable.

  • When bgp-evpn>remote-attachment-circuit is added to an explicit endpoint with a spoke SDP, the spoke-sdp>precedence command is not allowed. The spoke SDP always has a precedence of four, which is always higher than the EVPN precedence. Therefore, the EVPN-MPLS destination is used for transmission if it is created, and the spoke SDP is only used when the EVPN-MPLS destination is removed.

• EVPN-VPWS Epipes for MPLS tunnels support control word, hash label and entropy labels.

  • When the control word is configured, the PE sets the C bit in its AD per-EVI advertisement and sends the control word in the datapath. In this case, the PE expects the control word to be received. If there is a mismatch between the received control word and the configured control word, the system does not set up the EVPN destination and the service does not come up.
  • When the hash-label command is configured, the PE sets the F bit in its AD per-EVI routes and sends the hash label in the datapath. The PE expects the hash-label to also be received. In case of a mismatch between the received F flag and the locally configured hash-label, the router does not create the EVPN destination and the service does not come up. For the service, the use of the hash label and entropy labels are mutually exclusive.

• EVPN-VPWS Epipes support force-qinq-vc-forwarding [c-tag-c-tag | s-tag-c-tag] command under bgp-evpn mpls and the qinq-vlan-translation s-tag.c-tag command on ingress QinQ SAPs.

  When QinQ VLAN translation is configured at the ingress QinQ or dot1q SAP, the service-delimiting outer and inner VLAN values can be translated to the configured values. The force-qinq-vc-forwarding s-tag-c-tag command must be configured to preserve the translated QinQ tags in the payload when sending EVPN packets. This translation and preservation behavior is aligned with the ‟normalization” concept described in draft-ietf-bess-evpn-vpws-fxc. The VLAN tag processing described in Epipe service pseudowire VLAN tag processing applies to EVPN destinations in EVPN-VPWS services too.

The following features, described in EVPN-VPWS for VXLAN tunnels, are also supported for MPLS tunnels:

• Advertisement of the Layer-2 MTU and consistency checking of the MTU of the AD per-EVI routes.

• Use of A/S PW and MC-LAG at access.

• EVPN multihoming, including:

  • Single-active and all-active

  • Regular or virtual ESs

  • All existing DF election modes

Epipes with BGP-EVPN MPLS support the following configurations:

• up to two endpoints

• up to two SAPs, each associated with a different configured endpoint

• two pairs of local/remote attachment circuit Ethernet tags, also associated with different configured endpoints

• EVPN destinations used as Inter-Chassis Backup (ICB) links

The support of endpoints and up to two SAPs with local switching allows two-node and three-node topologies for EVPN-VPWS. See sections EVPN-VPWS endpoints example 1, EVPN-VPWS endpoints example 2, and EVPN-VPWS endpoints example 3 for example topologies.

FXC, specified in draft-ietf-bess-evpn-vpws-fxc, extends EVPN-VPWS services to support multiple SAPs at the access, in contrast with only one SAP in regular EVPN-VPWS services. Multiplexing multiple SAPs into a single EVPN-VPWS tunnel allows the user to save MPLS labels and reduce the number of EVPN Auto-Discovery per EVI routes.

Use the following command to turn an Epipe into a FXC service.

• MD-CLI

  configure service epipe flexible-cross-connect true

• classic CLI

  configure service epipe flexible-cross-connect create

Since there are multiple active egress SAPs in FXC services, EVPN labels are no longer enough to identify the egress SAP on the egress node. Therefore, payload VLAN tags are transmitted into EVPN-encapsulated FXC packets, and the router looks up these inner VLAN tags to forward the decapsulated traffic to the correct egress SAP.

configure service epipe sap qtag-normalization

VLAN normalization configuration and the two FXC modes of operation (default and VLAN aware mode) are described in FXC VLAN normalization, Default FXC mode, and VLAN-aware bundle FXC mode.

SAP and spoke-SDP based ESs are supported on R-VPLS services where bgp-evpn mpls is enabled.

EVPN-MPLS multihoming in R-VPLS services shows an example of EVPN-MPLS multihoming in R-VPLS services, with the following assumptions:

• There are two subnets for a specific customer (for example, EVI1 and EVI2 in EVPN-MPLS multihoming in R-VPLS services), and a VPRN is instantiated in all the PEs for efficient inter-subnet forwarding.

• A ‟backhaul” R-VPLS with evpn-tunnel mode enabled is used in the core to interconnect all the VPRNs. EVPN IP-prefix routes are used to exchange the prefixes corresponding to the two subnets.

• An all-active ES is configured for EVI1 on PE1 and PE2.

• A single-active ES is configured for EVI2 on PE3 and PE4.

In the example in the preceding figure, the hosts connected to CE1 and CE4 can use regular VRRP for default gateway redundancy; however, this may not be the most efficient way to provide upstream routing.

For example, if PE1 and PE2 are using regular VRRP, the upstream traffic from CE1 may be hashed to the backup IRB VRRP interface, instead of being hashed to the master. The same may occur for single-active multihoming and regular VRRP for PE3 and PE4. The traffic from CE4 is sent to PE3, when PE4 may be the VRRP master. In that case, PE3 must send the traffic to PE4, instead of routing it directly.

Both cases use unnecessary bandwidth between the PEs to get to the master IRB interface. In addition, VRRP scaling is limited if aggressive keepalive timers are used.

Because of these issues, passive VRRP is the recommended method when EVPN-MPLS multihoming is used in combination with R-VPLS redundant interfaces.

Passive VRRP is a VRRP setting in which the transmission and reception of keepalive messages is completely suppressed, and therefore, the VPRN interface always behaves as the master. Passive VRRP is enabled by adding the passive keyword to the VRRP instance at creation, as shown in the following examples:

configure service vprn 1 interface int-1 vrrp 1 passive

configure service vprn 1 interface int-1 ipv6 vrrp 1 passive

For example, if PE1, PE2, and PE5 in EVPN-MPLS multihoming in R-VPLS services use passive VRRP, even if each individual R-VPLS interface has a different MAC/IP address, because they share the same VRRP instance 1 and the same backup IP, the three PEs will own the same virtual MAC and virtual IP address (for example, 00-00-5E-00-00-01 and 10.0.0.254). The virtual MAC is auto-derived from 00-00-5E-00-00-VRID per RFC 3768. The following is the expected behavior when passive VRRP is used in this example:

• All R-VPLS IRB interfaces for EVI1 have their own physical MAC/IP address; they also own the same default gateway virtual MAC and IP address.

• All EVI1 hosts have a unique configured default gateway; for example, 10.0.0.254.

• When CE1 or CE2 send upstream traffic to a remote subnet, the packets are routed by the closest PE because the virtual MAC is always local to the PE.

  For example, the packets from CE1 hashed to PE1 are routed at PE1. The packets from CE1 hashed to PE2 are routed directly at PE2.

• Downstream packets (for example, packets from CE3 to CE1), are routed directly by the PE to CE1, regardless of the PE to which PE5 routed the packets.

  For example, the packets from CE3 sent to PE1 are routed at PE1. The packets from CE3 sent to PE2 are routed at PE2.

• In case of ES failure in one of the PEs, the traffic is forwarded by the available PE.

  For example, if the packets routed by PE5 arrive at PE1 and the link to CE1 is down, PE1 sends the packets to PE2. PE2 forwards the packets to CE1 even if the MAC source address of the packets matches PE2's virtual MAC address. Virtual MACs bypass the R-VPLS interface MAC protection.

The following list summarizes the advantages of using passive VRRP mode versus regular VRRP for EVPN-MPLS multihoming in R-VPLS services:

• Passive VRRP does not require multiple VRRP instances to achieve default gateway load-balancing. Only one instance per R-VPLS, therefore only one default gateway, is needed for all the hosts.

• The convergence time for link/node failures is not impacted by the VRRP convergence, because all the nodes in the VRRP instance are master routers.

• Passive VRRP scales better than VRRP, as it does not use keepalive or BFD messages to detect failures and allow the backup to take over.

In EVPN all-active multi-homing scenarios with R-VPLS services where the advertisement of the ARP/ND entries is enabled, use the following command to avoid issues with MAC mobility caused by the MAC/IP advertisement route for the ARP/ND entry being sent with ESI=0:

• MD-CLI

  configure service vpls bgp-evpn routes mac-ip arp-nd-only-with-fdb-advertisement true

• classic CLI

  configure service vpls bgp-evpn arp-nd-only-with-fdb-advertisement

When this command is enabled, the local ARP/ND entries of VPRN interfaces using this VPLS are advertised in this BGP-EVPN service only when the corresponding local MAC is programmed in the FDB.

In an EVPN multi-homing scenario, this command prevents the router from advertising a MAC/IP advertisement route with the MAC and IP binding but without the correct ESI value (which is taken only when the MAC is properly programmed in the FDB against the ESI).

In addition, if an Ethernet Segment SAP receives a frame, the MAC address can be re-programmed as type learned, even if the MAC was previously programmed as type EVPN.

When single-active multihoming is used, the IGMP snooping state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IGMP messages to be received to instantiate the IGMP snooping state after the ES activation timer expires; this could result in an increased outage.

The outage can be reduced by using MCS synchronization, which is supported for IGMP snooping in both EVPN-MPLS and PBB-EVPN services (see Multichassis synchronization for Layer 2 snooping states). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs. Also, IGMP snooping state can be synchronized only on a SAP.

An increased outage would also occur when using all-active EVPN multihoming. The IGMP snooping state on an ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE may not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, IGMP snooping messages can be synchronized between the PEs on an ES using data-driven IGMP snooping state synchronization, which is supported in EVPN-MPLS services, PBB-EVPN services, EVPN-MPLS VPRN and IES R-VPLS services. The IGMP messages received on an ES SAP or spoke SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES B-MAC (for PBB-EVPN) and these are used to synchronize the IGMP snooping state on the ES SAP or spoke SDP on the receiving PE.

Data-driven IGMP snooping state synchronization is supported for both all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS, EVPN-MPLS VPRN and IES R-VPLS services, and for all-active multihoming in PBB-EVPN services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability. PBB-EVPN with IGMP snooping using single-active multihoming is not supported.

Data-driven IGMP snooping state synchronization is also supported with P2MP mLDP LSPs in both EVPN-MPLS and PBB-EVPN services. When P2MP mLDP LSPs are used in EVPN-MPLS services, all PEs (including the PEs not connected to a multihomed ES) in the EVPN-MPLS service must be running an SR OS version supporting this capability with IGMP snooping enabled and all network interfaces must be configured on FP3 or higher-based line cards.

Data-driven IGMP snooping synchronization with EVPN multihoming shows the processing of an IGMP message for EVPN-MPLS. In PBB-EVPN services, the ES B-MAC is used instead of the ESI label to synchronize the state.

Data-driven synchronization is enabled by default when IGMP snooping is enabled within an EVPN-MPLS service using all-active multihoming or single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If IGMP snooping MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP then MCS synchronization takes precedence over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS IGMP synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down which prevents state synchronization to the PE with the ES down, as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it may not have any IGMP snooping state until the next IGMP messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive can avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services:

1. Upgrade SR OS on all ES PEs to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.

2. Enable IGMP snooping in the required services on all ES PEs. Traffic loss occurs until all ES PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

   Note: There is no action required on the non-ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling IGMP snooping in EVPN-MPLS and PBB-EVPN services:

1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven IGMP snooping synchronization with EVPN multihoming.

2. Enable IGMP snooping in EVPN-MPLS and PBB-EVPN services.

   • Perform the following steps for EVPN-MPLS:

     • Enable IGMP snooping on all non-ES PEs. Traffic loss occurs until the first set of join/query messages are processed by the non-ES PEs.

     • Then enable IGMP snooping on all ES PEs. Traffic loss occurs until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

   • Perform the following steps for PBB-EVPN:

     • Enable IGMP snooping on all ES PEs. Traffic loss occurs until all PEs have IGMP snooping enabled and the first set of join/query messages are processed by the ES PEs.

     • There is no action required on the non-ES PEs.

To aid with troubleshooting, the debug packet output displays the IGMP packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows.

6 2017/06/16 18:00:07.819 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: TX packet on svc 1
  from chaddr 5e:00:00:16:d8:2e
  send towards ES:esi-1
  Port  : evpn-mpls
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1


4 2017/06/16 18:00:07.820 PDT MINOR: DEBUG #2001 Base IGMP
"IGMP: RX packet on svc 1
  from chaddr d8:2e:ff:00:01:41
  received via evpn-mpls on ES:esi-1
  Port  : sap lag-1:1
  SrcIp : 0.0.0.0
  DstIp : 239.0.0.22
  Type  : V3 REPORT
    Num Group Records: 1
        Group Record Type: MODE_IS_EXCL (2), AuxDataLen 0, Num Sources 0
          Group Addr: 239.0.0.1

When single-active multihoming is used, PIM snooping for IPv4 state is learned on the active multihoming object. If a failover occurs, the system with the newly active multihoming object must wait for IPv4 PIM messages to be received to instantiate the PIM snooping for IPv4 state after the ES activation timer expires, which could result in an increased outage.

This outage can be reduced by using MCS synchronization, which is supported for PIM snooping for IPv4 in both EVPN-MPLS and PBB-EVPN services (see Multichassis synchronization for Layer 2 snooping states). However, MCS only supports synchronization between two PEs, whereas EVPN multihoming is supported between a maximum of four PEs.

An increased outage would also occur when using all-active EVPN multihoming. The PIM snooping for IPv4 state on an all-active ES LAG SAP or virtual ES to the attached CE must be synchronized between all the ES PEs, as the LAG link used by the DF PE may not be the same as that used by the attached CE. MCS synchronization is not applicable to all-active multihoming as MCS only supports active/standby synchronization.

To eliminate any additional outage on a multihoming failover, snooped IPv4 PIM messages should be synchronized between the PEs on an ES using data-driven PIM snooping for IPv4 state synchronization, which is supported in both EVPN-MPLS and PBB-EVPN services. The IPv4 PIM messages received on an ES SAP or spoke SDP are sent to the peer ES PEs with an ESI label (for EVPN-MPLS) or ES B-MAC (for PBB-EVPN) and are used to synchronize the PIM snooping for IPv4 state on the ES SAP or spoke SDP on the receiving PE.

Data-driven PIM snooping state synchronization is supported for all-active multihoming and single-active with an ESI label multihoming in EVPN-MPLS services. All PEs participating in a multihomed ES must be running an SR OS version supporting this capability with PIM snooping for IPv4 enabled. It is also supported with P2MP mLDP LSPs in the EVPN-MPLS services, in which case all PEs (including the PEs not connected to a multihomed ES) must have PIM snooping for IPv4 enabled and all network interfaces must be configured on FP3 or higher-based line cards.

In addition, data-driven PIM snooping state synchronization is supported for all-active multihoming in PBB-EVPN services and with P2MP mLDP LSPs in PBB-EVPN services. All PEs participating in a multihomed ES, and all PEs using PIM proxy mode (including the PEs not connected to a multihomed ES) in the PBB-EVPN service must be running an SR OS version supporting this capability and must have PIM snooping for IPv4 enabled. PBB-EVPN with PIM snooping for IPv4 using single-active multihoming is not supported.

Data-driven PIM snooping for IPv4 synchronization with EVPN multihoming shows the processing of an IPv4 PIM message for EVPN-MPLS. In PBB-EVPN services, the ES B-MAC is used instead of the ESI label to synchronize the state.

Data-driven synchronization is enabled by default when PIM snooping for IPv4 is enabled within an EVPN-MPLS service using all-active multihoming and single-active with an ESI label multihoming, or in a PBB-EVPN service using all-active multihoming. If PIM snooping for IPv4 MCS synchronization is enabled on an EVPN-MPLS or PBB-EVPN (I-VPLS) multihoming SAP or spoke SDP, then MCS synchronization takes preference over the data-driven synchronization and the MCS information is used. Mixing data-driven and MCS PIM synchronization within the same ES is not supported.

When using EVPN-MPLS, the ES should be configured as non-revertive to avoid an outage when a PE takes over the DF role. The Ethernet A-D per ESI route update is withdrawn when the ES is down, which prevents state synchronization to the PE with the ES down as it does not advertise an ESI label. The lack of state synchronization means that if the ES comes up and that PE becomes DF after the ES activation timer expires, it may not have any PIM snooping for IPv4 state until the next PIM messages are received, potentially resulting in an additional outage. Configuring the ES as non-revertive can avoid this potential outage. Configuring the ES to be non-revertive would also avoid an outage when PBB-EVPN is used, but there is no outage related to the lack of the ESI label as it is not used in PBB-EVPN.

The following steps can be used when enabling PIM snooping for IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services:

• PIM snooping mode

  1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

     Note: There is no action required on the non-ES PEs.

• PIM proxy mode

  • EVPN-MPLS

    1. Upgrade SR OS on all ES PEs to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

    2. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

       Note: There is no action required on the non-ES PEs.

  • PBB-EVPN

    1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

    2. Enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.

    3. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by the ES PEs.

If P2MP mLDP LSPs are also configured, the following steps can be used when enabling PIM snooping or IPv4 (using PIM snooping and PIM proxy modes) in EVPN-MPLS and PBB-EVPN services.

• PIM snooping mode

  1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Then enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.

     Note: There is no action required on the non-ES PEs.

• PIM proxy mode

  1. Upgrade SR OS on all PEs (both ES and non-ES) to a version supporting data-driven PIM snooping for IPv4 synchronization with EVPN multihoming.

  2. Enable PIM snooping for IPv4 on all non-ES PEs. Traffic loss occurs until all PEs have PIM snooping for IPv4 enabled and the first set of join/hello messages are processed by each non-ES PE.

  3. Enable PIM snooping for IPv4 on all ES PEs. Traffic loss occurs until all PEs have PIM snooping enabled and the first set of join/hello messages are processed by the ES PEs.

In the above steps, when PIM snooping for IPv4 is enabled, the traffic loss can be reduced or eliminated by configuring a larger hold-time (up to 300 seconds), during which multicast traffic is flooded.

To aid with troubleshooting, the debug packet output displays the PIM packets used for the snooping state synchronization. An example of a join sent on ES esi-1 from one ES PE and the same join received on another ES PE follows:

6 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimVplsFwdJPToEvpn
Forwarding to remote peer on bgp-evpn ethernet-segment esi-1"
7 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:37.040] PIM-TX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>


4 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: pimProcessPdu
Received from remote peer on bgp-evpn ethernet-segment esi-1, will be applied on
 lag-1:1
"
5 2017/06/16 17:36:37.144 PDT MINOR: DEBUG #2001 Base PIM[vpls 1 ]
"PIM[vpls 1 ]: Join/Prune
[000 00:19:30.740] PIM-RX ifId 1071394 ifName EVPN-MPLS-ES:esi-1 10.0.0.10 -> 22
10.0.0.13 Length: 34
PIM Version: 2 Msg Type: Join/Prune Checksum: 0xd2de
Upstream Nbr IP : 10.0.0.1 Resvd: 0x0, Num Groups 1, HoldTime 210
        Group: 239.0.0.10/32 Num Joined Srcs: 1, Num Pruned Srcs: 0
        Joined Srcs:
                10.0.0.1/32 Flag SWR  <*,G>

BGP EVPN control plane is extended and aligned with IETF RFC 8317 to support EVPN E-Tree services.

The following figure shows the main EVPN extensions for the EVPN E-Tree information model.

The following BGP extensions are implemented for EVPN E-Tree services.

• An EVPN E-Tree extended community (EC) sub-type 0x5 is defined. The following information is included.

  • The lower bit of the Flags field contains the L bit (where L=1 indicates leaf AC).

  • The leaf label contains a 20-bit MPLS label in the high-order 20 bits of the label field. This leaf label is statically assigned by the evpn-etree-leaf-label value command.

• The new E-Tree EC is sent with the following routes.

  • AD per-ES per PE route for BUM egress filtering

    Each EVPN E-Tree capable PE advertises an AD per-ES route with the E-Tree EC, and the following information:

    • service RD and route-target

      If ad-per-es-route-target evi-rt-set is configured, then non-zero ESI AD per-ES routes (used for multihoming) are sent per the evi-rt-set configuration, but E-Tree zero-ESI routes (used for E-Tree) are sent based on the default evi-rt configuration.

    • ESI = 0

      Eth Tag = MAX-ET

      MPLS label = zero

  • consistency check for AD per-EVI route for root or leaf configuration

    • The E-Tree EC is sent with the AD per-EVI routes for a specific ES. In this case, no validation is performed by the implementation, and the leaf indication is only used for troubleshooting on the remote PEs.

    • The MPLS label value is zero.

    • All ACs in each EVI for a specific ES must be configured as either a root AC or leaf AC, but not a combination. In case of a configuration error, for example where the AC in PE1 is configured as root AC and in PE2 as leaf AC, the remote PE3 receives the AD per-EVI routes with inconsistent leaf indication. However, the unicast filtering remains unaffected and is still handled by the FDB lookup information.

  • MAC or IP routes for known unicast ingress filtering

    • An egress PE sends all MAC or IP routes learned over a leaf AC SAP or spoke SDP with this E-Tree EC indicating that the MAC or IP belongs to a leaf AC.

    • The MPLS label value in the EC is 0.

    • Upon receiving a route with E-Tree EC, the ingress PE imports the route and installs the MAC in the FDB with a leaf flag (if there is a leaf indication in the route). Any frame coming from a leaf AC for which the MAC DA matches a leaf AC MAC is discarded at the ingress.

    • If two PEs send the same MAC with the same ESI but inconsistent root or leaf indication, the MAC is installed in the FDB as root.

EVPN E-Tree services are modeled as VPLS services configured as E-Trees with the bgp-evpn mpls context enabled.

[ex:/configure service system bgp evpn]
A:admin@node-2# info
    etree-leaf-label true

[ex:/configure service vpls "1"]
A:admin@node-2# info
    admin-state enable
    description "ETREE-enabled evpn-mpls-service"
    customer "1"
    etree true
    bgp-evpn {
        evi 10
        mpls 1 {
            admin-state enable
            ingress-replication-bum-label true
            ecmp 2
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    sap 1/1/c1/1:1 {
        etree-leaf true
    }
    sap lag-1:1 {
    }

A:node-2# configure service vpls 1 customer 1 etree create 
A:node-2# configure service system bgp-evpn evpn-etree-leaf-label
A:node-2>config>service>vpls$ info 
----------------------------------------------
            description "ETREE-enabled evpn-mpls-service"
            bgp-evpn
                evi 10
                mpls bgp 1
                    ingress-replication-bum-label
                    ecmp 2
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            sap 1/1/c1/1:1 leaf-ac create
                no shutdown
            exit
            sap lag-1:1 create
                no shutdown
            exit
            no shutdown
---------------------------------------------- 

EVPN E-Tree supports all operations related to flows among local root AC and leaf AC objects in accordance with IETF RFC 8317. This section describes the extensions required to forward traffic from (or to) root AC and leaf AC objects to (or from) BGP EVPN destinations.

EVPN E-Tree procedures support all-active and single-active EVPN multihoming. Ingress filtering can handle MACs learned on ES leaf AC SAPs. If a MAC associated with an ES leaf AC is advertised with a different E-Tree indication or if the AD per-EVI routes have inconsistent leaf indications, the remote PEs performing the aliasing treat the MAC as root.

The following figure shows the expected behavior for multihoming and egress BUM filtering, when ingress replication is used to forward BUM packets.

Multihoming and egress BUM filtering in the preceding figure is handled as follows.

• BUM frames received on an ES leaf AC are flooded to the EVPN based on EVPN E-Tree procedures. The leaf ESI label is sent when flooding to other PEs in the same ES, and additional labels are not added to the stack.

  When flooding in the default multicast list, the egress PE skips all the leaf ACs (including the ES leaf ACs) on the assumption that all ACs in a specific ES for a specified EVI have a consistent E-Tree configuration, and they send an AD per-EVI route with a consistent E-Tree indication.

• BUM frames received on an ES root AC are flooded to the EVPN based on regular EVPN procedures. The regular ES label is sent for split-horizon when packets are sent to the DF or NDF PEs in the same ES. When flooding in the default multicast list, the egress PE skips the ES SAPs based on the ES label lookup.

If the PE receives an ES MAC from a peer that shares the ES and decides to install it against the local ES SAP that is oper-up, it checks the E-Tree configuration (root or leaf) of the local ES SAP against the received MAC route. The MAC route is processed as follows.

• If the E-Tree configuration does not match, the MAC is not installed against any destination until the misconfiguration is resolved.

• If the SAP is oper-down, the MAC is installed against the EVPN destination to the peer.

EVPN E-Tree and multihoming is also supported when point-to-multipoint mLDP provider tunnels are used instead of ingress replication. If provider tunnels are used, leaf labels are upstream allocated instead of downstream allocated, and they are pushed immediately below the mLDP label (same as ESI labels with mLDP). Other than that, the procedures are equivalent to the ones described in this section for ingress replication.

SR OS supports PBB-EVPN E-Tree services in accordance with IETF RFC 8317. PBB-EVPN E-Tree services are modeled as PBB-EVPN services where some I-VPLS services are configured as etree and some of their SAP or spoke SDPs are configured as leaf-acs.

The procedures for the PBB-EVPN E-Tree are similar to those for the EVPN E-Tree, except that the egress leaf-to-leaf filtering for BUM traffic is based on the B-MAC source address. Also, the leaf label and the EVPN AD routes are not used.

The PBB-EVPN E-Tree operation is as follows:

• When one or more I-VPLS E-Tree services are linked to a B-VPLS, the leaf backbone source MAC address (leaf-source-bmac parameter) is used for leaf-originated traffic in addition to the source B-VPLS MAC address (source-bmac parameter) that is used for sourcing root traffic.

• The leaf backbone source MAC address for PBB must be configured using the command config>service>pbb>leaf-source-bmac ieee-address before the configuration of any I-VPLS E-Tree service.

• The leaf-source-bmac address is advertised in a B-MAC route with a leaf indication.

• Known unicast filtering occurs at the ingress PE. When a frame enters an I-VPLS leaf-ac, a MAC lookup is performed. If the C-MAC DA is associated with a leaf B-MAC, the frame is dropped.

• Leaf-to-leaf BUM traffic filtering occurs at the egress PE. When flooding BUM traffic with the B-MAC SA matching a leaf B-MAC, the egress PE skips the I-VPLS leaf-acs.

The following CLI example shows an I-VPLS E-Tree service that uses PBB-EVPN E-Tree. The leaf-source-bmac address must be configured before the configuration of the I-VPLS E-Tree. As is the case in regular E-Tree services, SAP and spoke SDPs that are not explicitly configured as leaf-acs are considered root-ac objects.

A:PE-2>config>service# info
----------------------------------------------
        pbb
            leaf-source-bmac 00:00:00:00:00:22
        exit
        vpls 1000 customer 1 name "vpls1000" b-vpls create
            service-mtu 2000
            bgp
            exit
            bgp-evpn
                evi 1000
                exit
                mpls bgp 1
                    ingress-replication-bum-label
                    auto-bind-tunnel
                        resolution any
                    exit
                    no shutdown
                exit
            exit
            stp                      
                shutdown
            exit
            no shutdown
        exit
        vpls 1001 customer 1 i-vpls etree create
            pbb
                backbone-vpls 1000
                exit
            exit
            stp
                shutdown
            exit
            sap 1/1/1:1001 leaf-ac create
                no shutdown
            exit
            sap 1/1/1:1002 create
                no shutdown
            exit
            no shutdown
        exit

The following considerations apply to PBB-EVPN E-Trees and multihoming:

• All-active multihoming is not supported on leaf-ac I-VPLS SAPs.

• Single-active multihoming is supported on leaf-ac I-VPLS SAPs and spoke SDPs.

• ISID- and RFC 7623-based C-MAC flush are supported in addition to PBB-EVPN E-Tree services and single-active multihoming.

When enable-rr-vpn-forwarding or enable-inter-as-vpn is configured, only EVPN-MPLS routes are processed for label swap and the next hop is changed. EVPN-VXLAN routes are re-advertised without a change in the next hop.

The following shows how the router processes and re-advertises the different EVPN route types. For more information about the route fields, see the BGP-EVPN control plane for MPLS tunnels Guide.

• Auto-discovery (AD) routes (type 1)

  For AD per EVI routes, the MPLS label is extracted from the route NLRI. The route is re-advertised with Next-Hop-Self (NHS) and a new label. No modifications are made for the remaining attributes.

  For AD per ES routes, the MPLS label in the NLRI is zero. The route is re-advertised with NHS and the MPLS label remains zero. No modifications are made for the remaining attributes.

• MAC/IP routes (type 2)

  The MPLS label (Label-1) is extracted from the NLRI. The route is re-advertised with NHS and a new Label-1. No modifications are made for the remaining attributes.

• Inclusive Multicast Ethernet Tag (IMET) routes (type 3)

  Because there is no MPLS label present in the NLRI, the MPLS label is extracted from the PMSI Tunnel Attribute (PTA) if needed, and the route is then re-advertised with NHS, with the following considerations:

  • For IMET routes with tunnel-type Ingress Replication, the router extracts the IR label from the PTA. The router programs the label swap and re-advertises the route with a new label in the PTA.

  • For tunnel-type P2MP mLDP, the router re-advertises the route with NHS. No label is extracted; therefore, no swap operation occurs.

  • For tunnel-type Composite, the IR label is extracted from the PTA, the swap operation is programmed and the route re-advertised with NHS. A new label is encoded in the PTA’s IR label with no other changes in the remaining fields.

  • For tunnel-type AR, the routes are always considered VXLAN routes and are re-advertised with the next-hop unchanged.

• Ethernet-Segment (ES) routes (type 4)

  Because ES routes do not contain an MPLS label, the route is re-advertised with NHS and no modifications to the remaining attributes. Although an ASBR or ABR re-advertises ES routes, EVPN multihoming for ES PEs located in different ASs or IGMP domains is not supported.

• IP-Prefix routes (type 5)

  The MPLS label is extracted from the NLRI and the route is re-advertised with NHS and a new label. No modifications are made to the remaining attributes.

Inter-AS Option B and VPN-NH-RR support the use of non-segmented trees for forwarding BUM traffic in EVPN.

For ingress replication and non-segmented trees, the ASBR or ABR performs an EVPN BUM label swap without any aggregation or further replication. This concept is shown in VPN-NH-RR and ingress replication for BUM traffic.

In VPN-NH-RR and ingress replication for BUM traffic, when PE2, PE3, and PE4 advertise their IMET routes, the ABRs re-advertise the routes with NHS and a different label. However, IMET routes are not aggregated; therefore, PE1 sets up three different EVPN multicast destinations and sends three copies of every BUM packet, even if they are sent to the same ABR. This example is also applicable to ASBRs and Inter-AS Option B.

P2MP mLDP may also be used with VPN-NH-RR, but not with Inter-AS Option B. The ABRs, however, do not aggregate or change the mLDP root IP addresses in the IMET routes. The root IP addresses must be leaked across IGP domains. For example, if PE2 advertises an IMET route with mLDP or composite tunnel type, PE1 is able to join the mLDP tree if the root IP is leaked into PE1’s IGP domain.

In general, EVPN multihoming is supported in Inter-AS Option B or VPN-NH-RR networks with the following limitations:

• An ES PE can only process a remote ES route correctly if the received next hop and origination IP address match. EVPN multihoming is not supported when the ES PEs are in different ASs or IGP domains, or if there is an NH-RR peering the ES PEs and overriding the ES route next hops.

• EVPN multihoming ESs are not supported on EVPN PEs that are also ABRs or ASBRs.

• Mass-withdraw based on the AD per-ES routes is not supported for a PE that is in a different AS or IGP domain that the ES PEs. EVPN multihoming with inter-AS Option B or VPN-NH-RR shows an EVPN multihoming scenario where the ES PEs, PE2 and PE3, and the remote PE, PE1, are in different ASs or IGP domains.

In EVPN multihoming with inter-AS Option B or VPN-NH-RR, PE1’s aliasing and backup functions to the remote ES-1 are supported. However, PE1 cannot identify the originating PE for the received AD per-ES routes because they are both arriving with the same next hop (ASBR/ABR4) and RDs may not help to correlate each AD per-ES route to a specified PE. Therefore, if there is a failure on PE2’s ES link, PE1 cannot remove PE2 from the destinations list for ES-1 based on the AD per-ES route. PE1 must wait for the AD per-EVI route withdrawals to remove PE2 from the list. In summary, when the ES PEs and the remote PE are in different ASs or IGP domains, per-service withdrawal based on AD per-EVI routes is supported, but mass-withdrawal based on AD per-ES routes is not supported.

Unicast procedures known to EVPN-MPLS E-Tree are supported in Inter-AS Option B or VPN-NH-RR scenarios, however, the BUM filtering procedures are affected.

As described in EVPN Ethernet-Tree, leaf-to-leaf BUM filtering is based on the Leaf Label identification at the egress PE. In a non-Inter-AS or non-VPN-NH-RR scenario, EVPN E-tree AD per-ES (ESI-0) routes carrying the Leaf Label are distinguished by the advertised next hop. In Inter-AS or VPN-NH-RR scenarios, all the AD per-ES routes are received with the ABR or ASBR next hop. Therefore, AD per-ES routes originating from different PEs would all have the same next hop, and the ingress PE would not be able to determine which leaf label to use for a specific EVPN multicast destination.

A simplified EVPN E-Tree solution is supported, where an E-Tree Leaf Label is not installed in the IOM if the PE receives more than one E-Tree AD per-ES route, with different RDs, for the same next hop. In this case, leaf BUM traffic is transmitted without a Leaf Label and the leaf-to-leaf traffic filtering depends on the egress source MAC filtering on the egress PE. See EVPN E-Tree egress filtering based on MAC source address.

PBB-EVPN E-tree services are not affected by Inter-AS or VPN-NH-RR scenarios, as AD per-ES routes are not used.

When proxy-ARP/proxy-ND is enabled, the system starts populating the proxy table and responding to ARP-requests/NS messages. To keep the active IP→MAC entries alive and ensure that all the host/routers in the service update their ARP/ND caches, the system may generate the following three types of ARP/ND messages for a specified IP→MAC entry:

• periodic refresh messages (ARP-requests or NS for a specified IP)

  These messages are activated by the send-refresh command and their objective is to keep the existing FDB and Proxy-ARP/ND entries alive, to minimize EVPN withdrawals and re-advertisements.

• unsolicited refresh messages (unsolicited GARP or NA messages)

  These messages are sent by the system when a new entry is learned or updated. Their objective is to update the attached host/router caches.

• confirm messages (unicast ARP-requests or unicast NS messages)

  These messages are sent by the system when a new MAC is learned for an existing IP. The objective of the confirm messages is to verify that a specified IP has really moved to a different part of the network and is associated with the new MAC. If the IP has not moved, it forces the owners of the duplicate IP to reply and cause dup-detect to kick in.

In order for the ingress and egress EVPN PEs to install the proxy-ARP/ND entries with the same property flags, RFC 9047 describes how those flags (R, O and I) can be conveyed in the EVPN ARP/ND extended community advertised along with the EVPN MAC/IP Advertisement routes. The format of the EVPN ARP/ND extended community follows:

By default, the router does not advertise the ARP/ND extended community. Use the following command to configure the router to advertise all the proxy ARP/ND MAC/IP Advertisement routes with the extended community:

configure service vpls bgp-evpn arp-nd-extended-community

SR OS supports the association of configured MAC lists with a configured dynamic proxy-ARP or proxy-ND IP address. The actual proxy-ARP or proxy-ND entry is not created until an ARP or Neighbor Advertisement message is received for the IP and one of the MACs in the associated MAC-list. This is in accordance with IETF RFC 9161, which states that a proxy-ARP or proxy-ND IP entry can be associated with one MAC among a list of allowed MACs.

The following example shows the use of MAC lists for dynamic entries.

A:PE-2>config>service#
  proxy-arp-nd
    mac-list ISP-1 create 
      mac 00:de:ad:be:ef:01 
      mac 00:de:ad:be:ef:02 
      mac 00:de:ad:be:ef:03
 
A:PE-2>config>service>vpls>proxy-arp#
  dynamic 1.1.1.1 create
    mac-list ISP-1
    resolve 30
 
A:PE-2>config>service>vpls>proxy-nd#
  dynamic 2001:db8:1000::1 create
    mac-list ISP-1 
    resolve 30

where:

• A dynamic IP (dynamic ip create) is configured and associated with a MAC list (mac-list name).

• The MAC list is created in the config>service context and can be reused by multiple configured dynamic IPs as follows:

  • in different services

  • in the same service, for proxy-ARP and proxy-ND entries

• If the MAC list is empty, the proxy-ARP or proxy-ND entry is not created for the configured IP.

• The same MAC list can be applied to multiple configured dynamic entries even within the same service.

• The new proxy-ARP and proxy-ND entries behave as dynamic entries and are displayed as type dyn in the show commands.

  The following output example displays the entry corresponding to the configured dynamic IP.

show service id 1 proxy-arp detail

-------------------------------------------------------------------------------
Proxy Arp
-------------------------------------------------------------------------------
Admin State       : enabled             
Dyn Populate      : enabled             
Age Time          : 900 secs            Send Refresh      : 300 secs
Table Size        : 250                 Total             : 1
Static Count      : 0                   EVPN Count        : 0
Dynamic Count     : 1                   Duplicate Count   : 0
Dup Detect
-------------------------------------------------------------------------------
Detect Window     : 3 mins              Num Moves         : 5
Hold down         : 9 mins              
Anti Spoof MAC    : None
EVPN
-------------------------------------------------------------------------------
Garp Flood        : enabled             Req Flood         : enabled
Static Black Hole : disabled            
-------------------------------------------------------------------------------
===============================================================================
VPLS Proxy Arp Entries
===============================================================================
IP Address          Mac Address       Type Status    Last Update
-------------------------------------------------------------------------------
1.1.1.1            00:de:ad:be:ef:01  dyn active    02/23/2016 09:05:49
-------------------------------------------------------------------------------
Number of entries : 1
===============================================================================

show service proxy-arp-nd mac-list "ISP-1" associations

===============================================================================
MAC List Associations
===============================================================================
Service Id                    IP Addr
-------------------------------------------------------------------------------
1                             1.1.1.1
1                             2001:db8:1000::1
-------------------------------------------------------------------------------
Number of Entries: 2
===============================================================================

The EVPN loop detection procedure, described in the preceding section, is compliant with draft-ietf-bess-rfc7432bis and is an efficient way of detecting and blocking loops in EVPN networks. Contrary to other intrusive methods that inject Ethernet beacons into the customer network and detect loops depending on whether the beacon messages get back to the PEs, the EVPN loop detection mechanism is non intrusive because it relies entirely on learning the same MAC on different nodes. However, the mechanism lacks determinism, as shown in the following figure.

Assume PE1, PE2 and PE3 are attached to the same EVPN VPLS service, and there is an accidental backdoor link between the Base Stations connected to PE2 and PE3. When the Controller with MAC M1 issues a broadcast frame, PE1 forwards it to PE2 and PE3, and the frame is looped back via the backdoor link. The mac-duplication procedure kicks in and M1 is detected as duplicate and turned into a blackhole MAC in the FDB, effectively solving the loop. However, the user does not know beforehand if M1 is blackholed in PE1, PE2, PE3 or multiple of them at the same time. If M1 is blackholed in PE1, this represents an issue for the hosts connected to other PEs (not shown) attached to the same service. Therefore, in the example, we want to influence the mac-duplication procedure so that M1 gets blackholed in PE2, PE3, or both, but not in PE1. To make the procedure more deterministic, the Trusted MAC concept is used.

configure service vpls bgp-evpn mac-duplication trusted-mac-time

Trusted MACs are affected by the mac-duplication procedures in a different way from the non-trusted MACs. Trusted MACs require a different number of moves (during the mac-duplication window) to be declared as duplicate, specified by num-moves <number> * trusted-mac-move-factor <number> in the following context.

configure service vpls bgp-evpn mac-duplication detect

While non-trusted MACs are detected as duplicate after num-moves, trusted MACs need more moves to be declared as duplicate.

// Applicable to PE1, PE2 and PE3

[ex:/configure service vpls "bd-1000" bgp-evpn mac-duplication]
A:admin@node-2# info
  blackhole true
  trusted-mac-time 5 // value 1..15, default: 5
  detect {
    num-moves 5
    window 3
    trusted-mac-move-factor 3 // value 1..10, default: 1
  }

// Applicable to PE1, PE2 and PE3

A:node-2>config>service>vpls>bgp-evpn>mac-duplication# info 
----------------------------------------------
  detect num-moves 5 window 3 trusted-mac-move-factor 3 // value 1..10, default: 1
  black-hole-dup-mac
  trusted-mac-time 5 // value 1..15, default: 5

show service id 1000 fdb detail

===============================================================================
Forwarding Database, Service 1000
===============================================================================
ServId   MAC               Source-Identifier   Type     Last Change
          Transport:Tnl-Id                     Age 
-------------------------------------------------------------------------------
1000     00:de:fe:da:da:04 sap:1/1/1:1000      LT/0     05/18/23 10:54:54
-------------------------------------------------------------------------------
No. of MAC Entries: 1
-------------------------------------------------------------------------------
Legend:L=Learned O=Oam P=Protected-MAC C=Conditional S=Static Lf=Leaf T=Trusted
===============================================================================

This section describes the configuration aspects of a VPLS/R-VPLS with EVPN-VXLAN and EVPN-MPLS.

In a service where EVPN-VXLAN and EVPN-MPLS are configured together, the configure service vpls bgp-evpn vxlan bgp 1 and configure service vpls bgp-evpn mpls bgp 2 commands allow the user to associate EVPN-MPLS to a different instance from that associated with EVPN-VXLAN, and have both encapsulations simultaneously enabled in the same service. At the control plane level, EVPN MAC/IP advertisement routes received in one instance are consumed and readvertised in the other instance as long as the route is the best route for a specific MAC. Inclusive multicast routes are independently generated for each BGP instance. In the data plane, the EVPN-MPLS and EVPN-VXLAN destinations are instantiated in different implicit Split Horizon Groups (SHGs) so that traffic can be forwarded between them.

The following example shows a VPLS service with two BGP instances and both VXLAN and MPLS encapsulations configured for the same BGP-EVPN service.

*A:PE-1>config>service>vpls# info 
----------------------------------------------
  description "evpn-mpls and evpn-vxlan in the same service"
  vxlan instance 1 vni 7000 create
  exit
  bgp 
    route-distinguisher 10:2
    route-target target:64500:1
  exit
  bgp 2 
    route-distinguisher 10:1
    route-target target:64500:1
  exit
  bgp-evpn
    evi 7000
    incl-mcast-orig-ip 10.12.12.12
    vxlan bgp 1 vxlan-instance 1
      no shutdown 
    mpls bgp 2
      control-word
      auto-bind-tunnel 
        resolution any
      exit
      force-vlan-vc-forwarding
      no shutdown
    exit
  exit
  no shutdown

The following list describes the preceding example:

• bgp 1 or bgp is the default BGP instance

• bgp 2 is the additional instance required when both bgp-evpn vxlan and bgp-evpn mpls are enabled in the service

• The commands supported in instance 1 are also available in instance 2 with the following considerations:

  • pw-template-binding

    The pw-template-binding can only exist in instance 1; it is not supported in instance 2.

  • route-distinguisher

    The operating route-distinguisher in both BGP instances must be different.

  • route-target

    The route target in both instances can be the same or different.

  • vsi-import and vsi-export

    Import and export policies can also be defined for either BGP instance.

• MPLS and VXLAN can use either BGP instance, and the instance is associated when bgp-evpn mpls or bgp-evpn vxlan is created. The bgp-evpn vxlan command must include not only the association to a BGP instance, but also to a vxlan-instance (because the VPLS services support two VXLAN instances).

  Note: The bgp-evpn vxlan no shutdown command is only allowed if bgp-evpn mpls shutdown is configured, or if the BGP instance associated with the MPLS has a different route distinguisher than the VXLAN instance.

The following features are not supported when two BGP instances are enabled on the same VPLS/R-VPLS service:

• SDP bindings

• M-VPLS, I-VPLS, B-VPLS, or E-Tree VPLS

• Proxy-ARP and proxy-ND

• BGP Multihoming

• IGMP, MLD, and PIM snooping

• BGP-VPLS or BGP-AD (SDP bindings are not created)

The service>vpls>bgp-evpn>ip-route-advertisement command is not supported on R-VPLS services with two BGP instances.

EVPN-SRv6 and EVPN-MPLS or EVPN-VXLAN can be simultaneously configured in the same VPLS service (but not R-VPLS), in different instances. In addition, EVPN-SRv6 and EVPN-MPLS can be simultaneously configured in the same Epipe service, so that border routers can stitch SRv6 and MPLS domains for point-to-point services.

The following sections describe BGP-EVPN routes in EVPN and VPLS services configured with two BGP instances.

The following sections describe the anycast redundant solution for dual BGP instances in VPLS and Epipe services.

Anycast multihoming and mLDP shows an example of a common BGP EVPN service configured in redundant anycast DCGWs and mLDP used in the MPLS instance.

When mLDP is used with multiple anycast multihoming DCGWs, the same originating IP address must be used by all the DCGWs. Failure to do so may result in packet duplication.

In the example shown in Anycast multihoming and mLDP, each pair of DCGWs (DCGW1/DCGW2 and DCGW3/DCGW4) is configured with a different originating IP address, which causes the following behavior:

1. DCGW3 and DCGW4 receive the inclusive multicast routes with the same route key from DCGW1 and DCGW2.

2. Both DCGWs (DCGW3 and DCGW4) select only one route, which is generally the same, for example, DCGW1's inclusive multicast route.

3. As a result, DCGW3 and DCGW4 join the mLDP tree with root in DCGW1, creating packet duplication when DCGW1 sends BUM traffic.

4. Remote PE nodes with a single BGP-EVPN instance join the mLDP tree without any problem.

To avoid the packet duplication shown in Anycast multihoming and mLDP, Nokia recommends to configure the same originating IP address in all four DCGWs (DCGW1/DCGW2 and DCGW3/DCGW4). However, the route distinguishers can be different per pair.

The following behavior occurs if the same originating IP address is configured on the DCGW pairs shown in Anycast multihoming and mLDP.

• DCGW3 and DCGW4 do not join any mLDP tree sourced from DCGW1 or DCGW2, which prevents any packet duplication. This is because a router ignore inclusive multicast routes received with its own originating-ip, regardless of the route-distinguisher.

• PE1 joins the mLDP trees from the two DCs.

SR OS supports Interconnect ESs (I-ES) for VXLAN as per RFC9014. An I-ES is a virtual ES that allows DCGWs with two BGP instances to handle VXLAN access networks as any other type of ES. I-ESs support the RFC 7432 multihoming functions, including single-active and all-active, ESI-based split-horizon filtering, DF election, and aliasing and backup on remote EVPN-MPLS PEs.

In addition to the EVPN multihoming features, the main advantages of the I-ES redundant solution compared to the redundant solution described in Anycast redundant solution for dual BGP-instance services are as follows:

• The use of I-ES for redundancy in dual BGP-instance services allows local SAPs on the DCGWs.

• P2MP mLDP can be used to transport BUM traffic between DCs that use I-ES without any risk of packet duplication. As described in Using P2MP mLDP in redundant anycast DCGWs, packet duplication may occur in the anycast DCGW solution when mLDP is used in the WAN.

Where EVPN-MPLS networks are interconnected to EVPN-VXLAN networks, the I-ES concept applies only to the access VXLAN network; the EVPN-MPLS network does not modify its existing behavior.

The Interconnect ES concept shows the use of I-ES for Layer 2 EVPN DCI between VXLAN and MPLS networks.

The following example shows how I-ES-1 would be provisioned on DCGW1 and the association between I-ES to a specified VPLS service. A similar configuration would occur on DCGW2 in the I-ES.

New I-ES configuration:

DCGW1#config>service>system>bgp-evpn# 
ethernet-segment I-ES-1 virtual create
 esi 01:00:00:00:12:12:12:12:12:00
 service-carving
  mode auto
 multi-homing all-active
 network-interconnect-vxlan 1
 service-id
   service-range 1 to 1000
 no shutdown

Service configuration:

DCGW1#config>service>vpls(1)# 
 vxlan instance 1 vni 1 instance 1 create
 exit
bgp
 route-distinguisher 1:1 
bgp 2
 route-distinguisher 2:2               
bgp-evpn        
  evi 1
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any                     
   no shutdown

...

DCGW1#config>service>vpls(2)#  
 vxlan instance 1 vni 2 create
 exit
bgp
 route-distinguisher 3:3 
bgp 2
 route-distinguisher 4:4               
 bgp-evpn        
  evi 2
  vxlan bgp 1 vxlan-instance 1
   no shutdown
  exit
  mpls bgp 2
   auto-bind-tunnel resolution any   
   no shutdown
 sap 1/1/1:1 create
 exit 

The above configuration associates I-ES-1 to the VXLAN instance in services VPLS1 and VPLS 2. The I-ES is modeled as a virtual ES, with the following considerations:

• The commands network-interconnect-vxlan and service-id service-range svc-id [to svc-id] are required within the ES.

  • The network-interconnect-vxlan parameter identifies the VXLAN instance associated with the virtual ES. The value of the parameter must be set to 1. This command is rejected in a non-virtual ES.

  • The service-range parameter associates the specific service range to the ES. The ES must be configured as network-interconnect-vxlan before any service range can be added.

  • The ES parameters port, lag, sdp, vc-id-range, dot1q, and qinq cannot be configured in the ES when a network-interconnect-vxlan instance is configured. The source-bmac-lsb option is blocked, as the I-ES cannot be associated with an I-VPLS or PBB-Epipe service. The remaining ES configuration options are supported.

  • All services with two BGP instances associate the VXLAN destinations and ingress VXLAN instances to the ES.

• Multiple services can be associated with the same ES, with the following considerations:

  • In a DC with two DCGWs (as in The Interconnect ES concept), only two I-ESs are needed to load-balance, where one half of the dual BGP-instance services would be associated with one I-ES (for example, I-ES-1, in the above configuration) and one half to another I-ES.

  • Up to eight service ranges per VXLAN instance can be configured. Ranges may overlap within the same ES, but not between different ESs.

  • The service range can be configured before the service.

• After the I-ES is configured using network-interconnect-vxlan, the ES operational state depends exclusively on the ES administrative state. Because the I-ES is not associated with a physical port or SDP, when testing the non-revertive service carving manual mode, an ES shutdown and no shutdown event results in the node sending its own administrative preference and DP bit and taking control if the preference and DP bit are higher than the current DF. This is because the peer ES routes are not present at the EVPN application layer when the ES is configured for no shutdown; therefore, the PE sends its own administrative preference and DP values. For I-ESs, the non-revertive mode works only for node failures.

• A VXLAN instance may be placed in MhStandby under any of the following situations:

  • if the PE is single-active NDF for that I-ES

  • if the VXLAN service is added to the I-ES and either the ES or BGP-EVPN MPLS is shut down in all the services included in the ES

    The following example shows the change of the MhStandby flag from false to true when BGP-EVPN MPLS is shut down for all the services in the I-ES.

    A:PE-4# show service id 500 vxlan instance 1 oper-flags 
    ===============================================================================
    VPLS VXLAN oper flags
    ===============================================================================
    MhStandby                              : false
    ===============================================================================
    A:PE-4# configure service vpls 500 bgp-evpn vxlan shutdown 
    *A:PE-4# show service id 500 vxlan instance 1 oper-flags    
    ===============================================================================
    VPLS VXLAN oper flags
    ===============================================================================
    MhStandby                              : true
    ===============================================================================
    

If two BGP instances with the same encapsulation (VXLAN or MPLS) are configured in the same VPLS/R-VPLS service, different import route targets in each BGP instance are mandatory (although this is not enforced).

BGP-EVPN routes in services configured with two BGP instances describes the use of policies to avoid sending WAN routes (routes meant to be redistributed from DC to WAN) to the DC again and DC routes (routes meant to be redistributed from WAN to DC) to the WAN again. Those policies are based on export policy statements that match on the RFC 9012 BGP encapsulation extended community (MPLS and VXLAN respectively).

When the two BGP instances are of the same encapsulation (VXLAN or MPLS), the policies matching on different BGP encapsulation extended community are not feasible because both instances advertise routes with the same encapsulation value. Because the export route targets in the two BGP instances must be different, the policies, to avoid sending WAN routes back to the WAN and DC routes back to the DC, can be based on export policies that prevent routes with a DC route target from being sent to the WAN peers (and opposite for routes with a WAN route target).

In scaled scenarios, matching based on route targets, does not scale well. An alternative and preferred solution is to configure a default-route-tag that identifies all the EVPN instances connected to the DC (or one domain), and a different default-route-tag in all the EVPN instances connected to the WAN (or the other domain). Anycast redundant solution for multi-instance EVPN services with the same encapsulation shows an example that demonstrates the use of default-route-tags.

Other than the specifications described in this section, the processing of MAC/IP routes and inclusive multicast Ethernet tag routes in multi-instance EVPN services of the same encapsulation follow the rules described in BGP-EVPN routes in services configured with two BGP instances.

The solution described in Anycast redundant solution for dual BGP-instance services is also supported in multi-instance EVPN VPLS/R-VPLS services with the same encapsulation.

The following CLI example output shows the configuration of DCGW-1 and DCGW-2 in Multihomed anycast solution where VPLS 500 is a multi-instance EVPN-VXLAN service and BGP instance 2 is associated with VXLAN instead of MPLS.

Different default-route-tags are used in BGP instance 1 and instance 2, so that in the export route policies, DC routes are not advertised to the WAN, and WAN routes are not advertised to the DC, respectively.

*A:DCGW-1(and DCGW-2)>config>service>vpls(500)# info
----------------------------------------------
vxlan instance 1 vni 500 create
exit
vxlan instance 2 vni 501 create
exit
bgp
 route-distinguisher 192.0.2.2:500
 route-target target:64500:500
exit
bgp 2
 route-distinguisher 192.0.2.2:501
 route-target target:64500:501
exit
bgp-evpn
 incl-mcast-orig-ip 23.23.23.23
 evi 500
 vxlan bgp 1 vxlan-instance 1
 default-route-tag 500
 no shutdown
 exit
 vxlan bgp 2 vxlan-instance 2
   default-route-tag 501
   no shutdown
 exit
exit
stp
shutdown
exit
no shutdown
----------------------------------------------
config>router>bgp#
 vpn-apply-import
 vpn-apply-export
 group "WAN"
  family evpn
  type internal
  export "allow only mpls"
  neighbor 192.0.2.6
 group "DC"
  family evpn
  type internal
export "allow only vxlan"
  neighbor 192.0.2.2

config>router>policy-options# info
----------------------------------------------
            policy-statement "allow only mpls"
                entry 10
                  from
                    family evpn 
                    tag 500 
                  action drop
                  exit
                exit
            exit
            policy-statement "allow only vxlan"
                entry 10
                  from
                    family evpn 
                    tag 501 
                  action drop
       exit
                  exit
            exit

The same Anycast redundant solution can be applied to VPLS/R-VPLS with two instances of EVPN-MPLS encapsulation. The configuration would be identical, other than replacing the VXLAN aspects with the EVPN-MPLS-specific parameters.

For a full description of this solution, see the Anycast redundant solution for dual BGP-instance services

The I-ES of network-interconnect VXLAN Ethernet segment is described in I-ES solution for dual BGP instance services. I-ES’s are also supported on VPLS and R-VPLS services with two EVPN-VXLAN instances.

I-ES in dual EVPN-VXLAN services shows the use of an I-ES in a dual EVPN-VXLAN instance service.

Similar to (single-instance) EVPN-VXLAN all-active multihoming, the BUM forwarding procedures follow the ‟Local Bias” behavior.

At the ingress PE, the forwarding rules for EVPN-VXLAN services are as follows:

• The no send-imet-ir-on-ndf or rx-discard-on-ndf bum command must be enabled so that the NDF does not forward any BUM traffic.

• BUM frames received on any SAP or I-ES VXLAN binding are flooded to:

  • local non-ES and single-active DF ES SAPs

  • local all-active ES SAPs (DF and NDF)

  • EVPN-VXLAN destinations

    BUM received on an I-ES VXLAN binding follows SHG rules, for example, it can only be forwarded to EVPN-VXLAN destinations that belong to the other VXLAN instance (instance 2), which is a different SHG.

• As an example, in I-ES in dual EVPN-VXLAN services:

  • GW1 and GW2 are configured with no send-imet-ir-on-ndf.

  • TOR1 generates BUM traffic that only reaches GW1 (DF).

  • GW1 forwards to CE1 and EVPN-VXLAN destinations.

The forwarding rules at the egress PE are as follows:

• The source VTEP is looked up for BUM frames received on EVPN-VXLAN.

• If the source VTEP matches one of the PEs with which the local PE shares an ES _AND_ a VXLAN service:

  • Then the local PE does not forward to the shared local ESs (this includes port, lag, or network-interconnect-vxlan ESs). It forwards though to non-shared ES SAPs unless they are in NDF state.

  • Else, the local PE forwards normally to local ESs unless they are in NDF state.

• Because there is no multicast label or multicast B-MAC in VXLAN, the only way the egress PE can identify BUM traffic is by looking at the customer MAC DA. Therefore, BM or unknown MAC DAs identify BUM traffic.

• As an example, in I-ES in dual EVPN-VXLAN services:

  • GW2 receives BUM on EVPN-VXLAN. GW2 identifies the source VTEP as a PE with which the I-ES-1 is shared, therefore it does not forward the BUM frames to the local I-ES. It forwards to the non-shared ES and local SAPs though (CE2).

  • GW3 receives BUM on EVPN-VXLAN, however the source VTEP does not match any PE with which GW3 shares an ES. Hence GW3 forwards to all local ESs that are DF, in other words, CE3.

The following configuration example shows how I-ES-1 would be provisioned on DCGW1 and the association between I-ES to a specified VPLS service. A similar configuration would occur on DCGW2 in the I-ES.

I-ES configuration:

*A:GW1>config>service>system>bgp-evpn>eth-seg# info 
----------------------------------------------
                esi 00:23:23:23:23:23:23:00:00:01
                service-carving
                    mode manual
                    manual
                        preference non-revertive create
                            value 150
                        exit
                        evi 101 to 200
                    exit
                exit
                multi-homing all-active
                network-interconnect-vxlan 1
                service-id
                    service-range 1
                    service-range 1000 to 1002
                    service-range 2000
                exit
                no shutdown

Service configuration:

*A:GW1>config>service>vpls# info 
----------------------------------------------
            vxlan instance 1 vni 1000 create
                rx-discard-on-ndf bum
            exit
            vxlan instance 2 vni 1002 create
            exit
            bgp
                route-target export target:64500:1000 import target:64500:1000
            exit
            bgp 2
                route-distinguisher auto-rd
                route-target export target:64500:1002 import target:64500:1002
            exit
            bgp-evpn
                evi 1000
                vxlan bgp 1 vxlan-instance 1
                    ecmp 2
                    default-route-tag 100
                    auto-disc-route-advertisement
                    no shutdown
                exit
                vxlan bgp 2 vxlan-instance 2
                    ecmp 2
                    default-route-tag 200
                    auto-disc-route-advertisement
                    mh-mode network
                    no shutdown
                exit
            exit
            no shutdown 

Multi-instance EVPN VPLS/R-VPLS services with two EVPN-MPLS instances do not support I-ESs.

For information about how the EVPN routes are processed and advertised in an I-ES, see the I-ES solution for dual BGP instance services.

The SBD is equivalent to an R-VPLS that connects all the PEs that are attached to the same tenant VPRN. Interface-ful refers to the use of a full IRB interface between the VPRN and the SBD (an interface object with MAC and IP addresses, over which interface parameters can be configured).

The following figure shows an example of the interface-ful IP-VRF-to-IP-VRF with SBD IRB model.

In the preceding figure, an SR OS router and a third-party router are using the interface-ful IP-VRF-to-IP-VRF with SBD IRB model. The two routers are attached to a VPRN for the same tenant, and those VPRNs are connected by R-VPLS-2, or SBD. Both routers exchange IP prefix routes with a non-zero gateway IP, which is the IP address of the SBD IRB. The SBD IRB MAC and IP are advertised in a MAC/IP route. On reception, the IP prefix route creates a route-table entry in the VPRN, where the gateway IP must be recursively resolved to the information provided by the MAC/IP route and installed in the ARP and FDB tables.

This model is detailed in EVPN for VXLAN in IRB backhaul R-VPLS services and IP prefixes. The following is an example of the configuration of SBD and VPRN, as shown in Interface-ful IP-VRF-to-IP-VRF with SBD IRB model.

7750SR#config>service#

vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
 evi 2
 ip-route-advertisement
 mpls bgp 1
 auto-bind-tunnel resolution any
 no shutdown

vprn 1 customer 1 name "vprn1" create
 route-distinguisher auto-rd
 interface "sbd" create
 address 192.168.0.1/16
 ipv6
   30::3/64
 exit
 vpls "sbd"

The interface-ful IP-VRF-to-IP-VRF with SBD IRB model is also supported for IPv6 prefixes. There are no configuration differences except the ability to configure an IPv6 address and interface.

Interface-ful refers to the use of a full IRB interface between the VPRN and the SBD. However, the SBD IRB is unnumbered in this model, which means no IP address is configured on it. In SR OS, an unnumbered SBD IRB is equivalent to an R-VPLS that is linked to a VPRN interface through an EVPN tunnel. See EVPN for VXLAN in EVPN tunnel R-VPLS services for more information.

The following figure shows an example of the interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model.

In the preceding figure, an SR OS router and a third-party router are using the interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model. The IP prefix routes are expected to have a zero gateway IP, and the MAC in the router's MAC extended community is used for the recursive resolution to a MAC/IP route.

The corresponding configuration of the 7750 SR VPRN and SBD in the example could be:

7750SR#config>service#

vpls 2 customer 1 name "sbd" create
 allow-ip-int-bind
 exit
 bgp
 exit
 bgp-evpn
evi 2
ip-route-advertisement
mpls bgp 1
auto-bind-tunnel resolution any
no shutdown

vprn 1 customer 1 create
 route-distinguisher auto-rd
 interface "sbd" create
  ipv6
  exit
 vpls "sbd" 
  evpn-tunnel ipv6-gateway-address mac

The evpn-tunnel command controls the use of the router's MAC extended community and the zero gateway IP in the IPv4-prefix route. For IPv6, the ipv6-gateway-address mac command allows the router to advertise the IPv6-prefix routes with a router's MAC extended community and zero gateway IP.

The interface-less model does not require a Supplementary Broadcast Domain (SBD) connecting the VPRNs for the tenant, and no recursive resolution is required upon receiving an IP prefix route. In other words, the next-hop of the IP prefix route is directly resolved to an EVPN tunnel without the need for any other route. The standard specification RFC 9136 supports two variants of this model that are not interoperable.

• EVPN IFL for Ethernet Network Virtualization Overlay (NVO) tunnels

  Ethernet NVO indicates that the EVPN packets contain an inner Ethernet header. This is the case for tunnels such as VXLAN

  In the Ethernet NVO option, the ingress PE uses the received router’s MAC extended community address (received along with the route type 5) as the inner destination MAC address for the EVPN packets sent to the prefix.

• EVPN IFL for IP NVO tunnels

  IP NVO indicates that the EVPN packets contain an inner IP packet, but without Ethernet header. This is similar to the IPVPN packets exchanged between PEs.

The following figure shows an example of the interface-less IP-VRF-to-IP-VRF model.

SR OS supports the interoperable Interface-less IP-VRF-to-IP-VRF Model for Ethernet NVO tunnels. In the preceding figure, this interoperable is shown on the left-most PE router. The following is the model implementation.

• There is no datapath difference between this model and the existing R-VPLS EVPN tunnel model or the model described in Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model.

• This model is enabled by configuring the config service vprn if vpls evpn-tunnel (with ipv6-gateway-address mac for IPv6) and bgp-evpn ip-route-advertisement commands. In addition, because the SBD IRB MAC/IP route is no longer needed, the config service vpls bgp-evpn no mac-advertisement command prevents the advertisement of the MAC/IP route.

• The following IP prefix routes are processed as follows.

  • On transmission, there is no change in the IP prefix route processing compared to the configuration of the Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model.

    • IPv4 or IPv6 prefix routes are advertised, based on the information in the route-table for IPv4 and IPv6, with GW-IP=0 and the corresponding MAC extended community.

    • If the bgp-evpn no mac-advertisement command is configured, no MAC/IP route is sent for the R-VPLS.

  • The received IPv4 or IPv6 prefix routes are processed as follows.

    • Upon receiving an IPv4 or IPv6 prefix route with a MAC extended community for the router, an internal MAC/IP route is generated with the encoded MAC and the RD, Ethernet tag, ESI, Label/VNI and next hop from the IP prefix route itself.

    • If no competing received MAC/IP routes exists for the same MAC, this IP prefix-derived MAC/IP route is selected and the MAC installed in the R-VPLS FDB with type "Evpn".

    • After the MAC is installed in FDB, there are no differences between this interoperable interface-less model and the interface-ful with unnumbered SBD IRB model. Therefore SR OS is compatible with the received IP prefix routes for both models.

The following is an example of a typical configuration of a PE's SBD and VPRN that work in interface-less model for IPv4 and IPv6.

7750SR#config>service#

vpls 2 customer 1 name "sbd" create
       allow-ip-int-bind
       exit
       bgp
       exit
       bgp-evpn
            evi 2
            no mac-advertisement
            ip-route-advertisement
            mpls bgp 1
                 auto-bind-tunnel resolution any
                 no shutdown
vprn 1 customer 1 create
       route-distinguisher auto-rd
       interface "sbd" create
            ipv6
            exit
            vpls "sbd"
                 evpn-tunnel ipv6-gateway-address mac

In addition to the Interface-ful and interoperable Interface-less models described in Interface-ful IP-VRF-to-IP-VRF with SBD IRB model, Interface-ful IP-VRF-to-IP-VRF with unnumbered SBD IRB model, and Interoperable interface-less IP-VRF-to-IP-VRF model (Ethernet encapsulation) sections, SR OS also supports the Interface-less Model (EVPN IFL) with IP encapsulation for MPLS tunnels. The RFC 9136 standard specification refers to this model as the EVPN IFL model for IP NVO tunnels.

Compared to the Ethernet NVO model in which the ingress PE pushes an inner Ethernet header, the IP packet in this EVPN IFL model is directly encapsulated with an EVPN service label and the transport labels.

The following figure shows the EVPN IFL model with IP encapsulation for MPLS tunnels.

EVPN IFL uses EVPN IP Prefix routes to exchange prefixes between PEs without the need for an R-VPLS service, termed Supplementary Broadcast Domain (SBD) in the standards, and any destination MAC lookup. The EVPN IFL uses the same datapath that is used for IP-VPN services in the VPRN.

In the example shown in Interface-less IP-VRF-to-IP-VRF model for IP encapsulation in MPLS tunnels, the following applies.

• PE2 advertises IP Prefix 20.0/24 (shorthand for 20.0.0.0/24) in an EVPN IP Prefix route that no longer contains a router's MAC extended community. In step 1, per usual processing, arriving frames with IP destination 20.0.0.1 on PE1's R-VPLS-1 are processed for a route lookup on VPRN-1.

• In step 2, in contrast to and as opposed to the other EVPN Layer 3 models, the lookup yields a route-table entry that does not point at an SBD R-VPLS, but points instead to an MPLS tunnel terminated on PE2. PE1 then pushes the EVPN service label received on the IP Prefix route at the top of the IP packet, and the packet is sent on the wire without an inner Ethernet header.

• In step 3, the MPLS tunnel is terminated on PE2 and the EVPN label identifies the VPRN-1 service for a route lookup.

• In Step 4, the processing corresponds to the regular R-VPLS forwarding that occurs in the other EVPN Layer 3 models.

Use the vprn>bgp-evpn>mpls context to configure a VPRN service for EVPN IFL with MPLS encapsulation. This context which is similar to the existing contexts in VPLS and Epipe services, enables the use of EVPN IFL in the VPRN service. When this context is enabled, no R-VPLS with evpn-tunnel should be added to the VPRN; that is, the user must ensure that no SBD is configured. For example, in Interface-less IP-VRF-to-IP-VRF model for IP encapsulation in MPLS tunnels, PE1 and PE2 VPRN-1 services are configured as follows.

[ex:configure service vprn "vprn-1"]
A:admin@PE1# info
    admin-state enable
    ecmp 2
    bgp-evpn {
        mpls 1 {
            admin-state enable
            route-distinguisher "192.0.2.1:12"
            vrf-target {
                community "target:64500:2"
            }
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    interface "irb-1" {
        ipv4 {
            primary {
                address 10.0.0.254
                prefix-length 24
            }
        }
        vpls "r-vpls-1" {
        }
    }

[ex:configure service vprn "vprn-1"]
A:admin@PE2# info
    admin-state enable
    ecmp 2
    bgp-evpn {
        mpls 1 {
            admin-state enable
            route-distinguisher "192.0.2.2:21"
            vrf-target {
                community "target:64500:2"
            }
            auto-bind-tunnel {
                resolution any
            }
        }
    }
    interface "irb-2" {
        ipv4 {
            primary {
                address 20.0.0.254
                prefix-length 24
            }
        }
        vpls "r-vpls-1" {
        }
    }

The interface-less model described in the previous section is also supported for VXLAN encapsulation. VXLAN tunnels are originated and terminated from and on a VPRN service, as opposed to on an R-VPLS service in the interface-ful models.

Use the following context to configure the characteristics of the VXLAN tunnels on the VPRN service.

configure service vprn vxlan instance number

Use the following context to configure the EVPN IFL parameters for VXLAN encapsulation.

configure service vprn bgp-evpn vxlan bgp-instance

[ex:/configure service vprn "evpn-ifl-vxlan-test-1" vxlan]
A:admin@PE-2# tree 
+-- instance
|   +-- network
|   |   +-- ingress
|   |       +-- qos
|   |           +-- network
|   |               +-- fp-redirect-group
|   |               |   +-- group-name
|   |               |   +-- instance
|   |               +-- policy-name
|   +-- source-vtep-security
|   +-- vni
+-- source-vtep

*A:PE1# show router 1 route-table all

===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                         Active     Metric   
-------------------------------------------------------------------------------
10.10.10.0/24                                 Remote  EVPN-IFL  06d03h12m  169
       192.168.0.1  (tunneled:VXLAN:1)                 Y            10
40.40.40.0/24                                 Remote  EVPN-IFL  06d03h04m  170
       192.168.0.2  (tunneled:VXLAN:1)                 Y            10
-------------------------------------------------------------------------------
No. of Routes: 2
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
       E = Inactive best-external BGP route
===============================================================================

*A:PE1# show router 1 route-table extensive 

===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix             : 10.10.10.0/24
  Protocol              : EVPN-IFL
  Age                   : 06d03h05m
  Preference            : 170
  Indirect Next-Hop     : 192.168.0.2
    VNI                 : 100
    MAC-NH              : 00:aa:aa:01:00:01
    VPN Next-Hop Index  : 65
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : N/A
    Resolving Next-Hop  : 192.0.2.1 (VXLAN tunnel)
      Metric            : 5
      ECMP-Weight       : N/A
-------------------------------------------------------------------------------
Dest Prefix             : 40.40.40.0/24
  Protocol              : EVPN-IFL
  Age                   : 06d03h05m
  Preference            : 170
  Indirect Next-Hop     : 192.168.0.2
    VNI                 : 200
    MAC-NH              : 00:bb:bb:01:00:01
    VPN Next-Hop Index  : 65
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : N/A
    Resolving Next-Hop  : 192.0.2.2 (VXLAN tunnel)
      Metric            : 5
      ECMP-Weight       : N/A
-------------------------------------------------------------------------------

*A:PE1# show router 1 route-table extensive 

===============================================================================
Route Table (Service: 1)
===============================================================================
Dest Prefix             : 10.10.10.0/24
  Protocol              : EVPN-IFL
  Age                   : 06d03h05m
  Preference            : 170
  Indirect Next-Hop     : 192.168.0.2
    VNI                 : 100
    MAC-NH              : 00:aa:aa:01:00:01
    VPN Next-Hop Index  : 65
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : N/A
    Resolving Next-Hop  : 192.0.2.1 (VXLAN tunnel)
      Metric            : 5
      ECMP-Weight       : N/A
-------------------------------------------------------------------------------
Dest Prefix             : 40.40.40.0/24
  Protocol              : EVPN-IFL
  Age                   : 06d03h05m
  Preference            : 170
  Indirect Next-Hop     : 192.168.0.2
    VNI                 : 200
    MAC-NH              : 00:bb:bb:01:00:01
    VPN Next-Hop Index  : 65
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : N/A
    Resolving Next-Hop  : 192.0.2.2 (VXLAN tunnel)
      Metric            : 5
      ECMP-Weight       : N/A
-------------------------------------------------------------------------------

SR OS supports the symmetric IRB model specified in RFC 9135, by which, when enabled, the router advertises the learned MAC and IP addresses of a host in an EVPN MAC/IP Advertisement route. This route contains not only the label and route target of the R-VPLS service where the host is learned, but also the label and the route target of the VPRN attached to the R-VPLS service. This is possible because the MAC/IP Advertisement route contains two labels in the NLRI: label-1 (for Layer 2 purposes) and label-2 (for Layer 3 purposes).

On reception, the remote router receives the EVPN MAC/IP Advertisement route and programs the IP address as a host route (/32 or /128) in the VPRN route table. Although host routes can be advertised in IP-Prefix routes, the symmetric IRB model allows the advertisement of the MAC and host IP information in a single NLRI. The following figure shows this model.

configure service interface vpls evpn arp advertise interface-less-routing bgp-evpn-instance
configure service interface vpls evpn nd advertise interface-less-routing bgp-evpn-instance

show router 200 route-table 10.0.0.202/32

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix[Flags]                            Type    Proto     Age        Pref
      Next Hop[Interface Name]                                    Metric   
-------------------------------------------------------------------------------
10.0.0.202/32                                 Remote  EVPN-IFL* 00h04m11s  170
       2001:db8::2 (tunneled)                                       10
-------------------------------------------------------------------------------
No. of Routes: 1
Flags: n = Number of times nexthop is repeated
       B = BGP backup route available
       L = LFA nexthop available
       S = Sticky ECMP requested
===============================================================================
* indicates that the corresponding row element may have been truncated.

show router 200 route-table 10.0.0.202/32 extensive

===============================================================================
Route Table (Service: 200)
===============================================================================
Dest Prefix             : 10.0.0.202/32
  Protocol              : EVPN-IFL-HOST
  Age                   : 00h04m18s
  Preference            : 170
  Indirect Next-Hop     : 2001:db8::2
    Label               : 524280
    VPN Next-Hop Index  : 10
    QoS                 : Priority=n/c, FC=n/c
    Source-Class        : 0
    Dest-Class          : 0
    ECMP-Weight         : N/A
    Resolving Next-Hop  : 2001:db8::2 (LDP tunnel)
      Metric            : 10
      ECMP-Weight       : N/A
-------------------------------------------------------------------------------
No. of Destinations: 1
===============================================================================