Service chaining for ESM hosts with L2-Aware NAT

This section describes service chaining for ESM hosts with L2-Aware NAT.

Steering to service chains for ESM hosts with L2-Aware NAT

This feature provides the steering of traffic flows for ESM hosts with L2-aware NAT (typically used in vRGW and WLAN-GW), to service-functions (SF) in a data-center, reachable over an IP underlay network with a tunnel. The supported tunnel encapsulation is VXLAN. The steering function on the gateway (vRGW or WLAN-GW) is configured through a PBR action in an ISA filter (also referred to as VAS filter) associated with an L2-aware NAT host. The PBR action specifies the IP address of the SF, the EVPN service instance through which the SF is reached, and optionally, an ESI. A controller in the data-center, and the subscriber edge (a gateway such as vRGW, WLAN-GW, or BNG) acting as a service-function-classifier (SFC) participate in BGP-EVPN to exchange reachability information for the SF in the DC, and NAT pools on the gateway. The gateway resolves the PBR target, in other words, SF IP@ in the EVPN service configured in the VAS filter via BGP EVPN routes received from the controller in the DC. The network virtualization edge (NVE) in the DC, such as a host stack running the SF in a VM can act as a bridge or a router. Nuage VRS or VSG is an example of an NVE.

The ISA filter used for steering can also be configured with an optional action to insert a network services header (NSH) in steered traffic, as described in RFC 8300, Network Service Header (NSH).

A group of NAT ISAs that provide per-host or per-flow steering functions for L2-aware NAT hosts on the gateway are configurable under the base router. The steered traffic from the gateway to the VAS is VXLAN encapsulated on these ISAs. A range of IP addresses used for local VXLAN VTEPs on ISAs is configured (and routable) under the base router.



A network element in the DC that peers with the subscriber edge to exchange BGP EVPN routes. The controller can advertise type-1 (ESI), type-2 (MAC advertisement, with an optional IP address), and type-5 (an IP prefix route containing an SF’s IP address or subnet) routes for each SF. The controller also learns EVPN type-2 and type-5 routes from the subscriber edge for reachability of NAT pools on the ISAs. The controller can program the NVE with these routes.

EVPN import-mode

A configurable attribute of the EVPN service on the subscriber edge that controls the type of EVPN route information that is imported or tracked from the EVPN peer and exported or advertised to the EVPN peer. The possible values are bridged, routed, or none. For import mode bridged, the subscriber edge imports EVPN type-2 routes containing the SF’s MAC and IP addresses and the associated VXLAN VTEP to reach the SF. With import mode routed, the EVPN type-5 route containing the SF’s IP address or subnet and the type-2 route containing the NVE’s IP and MAC address are imported or tracked. The NVE, in this case, routes traffic to or from the SF. Also, with import-mode routed, the subscriber edge advertises type-2 and type-5 routes for NAT prefixes (outside pools). If no import-mode is configured, then only type-2 and type-5 routes for NAT prefixes are advertised to the peer (no routes are imported). See EVPN route updates and tracking.


Network Services Header. Extra encapsulation carried in the steered traffic over VXLAN tunnels that contains information about the packet handling through the chain of value added service functions in the DC. In addition to the information about the service chain that the steered traffic traverses, it also contains optional meta-data (for example, the subscriber-id). More details on NSH usage and format are defined in VAS filters on the ISA.


Network virtualization edge. A network element in the DC (for example, TOR or virtual switch or router such as the Nuage VRS) that provides connectivity to the SFs (VM or appliance). The NVE can perform bridging or routing functions for traffic to or from the SF. EVPN service instances (including R-VPLS tied to a VPRN) can be configured on the NVE, depending on the configured forwarding mode (bridging or routing to connected SFs).

Service-chaining service

An EVPN control plane instance on the subscriber edge that is used to learn and track EVPN routes for reachability to the SFs in the DC. It is also used to advertise EVPN routes for NAT pools on the subscriber edge to the controller in the DC.


Service function. The value-added service (for example, a parental control service) running in a VM or on an appliance in the Data Center (DC). The SF can be part of a chain of SFs providing opt-in value added services.


Service function classifier. The component on the vRGW/WLAN-GW/BNG (subscriber edge) that matches flows belonging to the L2-aware NAT host that has opted-in for value-added services, and performs steering (PBR) functions to the first service in the service chain that is applicable to the host. SFC supports optional NSH insertion in the steered traffic. Flow matching and steering is specified in VAS filters that are applied upstream and downstream flows on the ISA. SFC also tracks EVPN routes (type-1, type-2, and type-5) to resolve the SF’s IP address, and optionally, ESI in an EVPN service, all of which is specified in the steering action in the VAS filters.

VAS filters on the ISA

A VAS filter can be associated with individual hosts of an L2-aware NAT subscriber either at creation time or via CoA. The name of the VAS filter associated with the host is provided by the AAA server. This VAS filter is applied on each packet that creates a new flow. This results in a pair of actions which are then used for all subsequent upstream and downstream packets in that flow. These actions allow forwarding different kinds of traffic to different service chains in both upstream and downstream directions.


A VAS filter contains a set of ordered entries. Each entry contains a single match criteria and either an upstream action, a downstream action, or both. Matching occurs independently for both directions, only considering the entries for which an action is configured in that direction. Matching stops at the first matching entry. Actions from different entries for either direction can result.

The VAS filter entries can match on:

  • a foreign IP or subnet (for example, a destination IP and subnet for upstream traffic, and source IP and subnet for downstream traffic match)

  • a foreign port (for example, a destination port for upstream and a source port for downstream traffic)

  • a protocol


The resulting forwarding action can be one of the following:

  • action forward

    The packet is forwarded as usual to its original destination.

  • action forward sf-ip sf-ip [esi esi] service-id svc-id

    The packet is tunneled toward the sf-ip over EVPN/VXLAN overlay.

    If the sf-ip cannot be resolved (see EVPN route updates and tracking), the fail-action is used. This can be either forward or drop. The default is forward. Optionally, an NSH header can be inserted.

NSH insertion

NSH insertion is an optional action of the VAS filter when forwarding to the service chain. An additional header is inserted between the tunneling Ethernet header and the IP payload, as defined in RFC 8300, Network Service Header (NSH). VXLAN-GPE is not supported.

IP => UDP (port 4789) => VXLAN => Ethernet =>NSH => IP

The NSH header is populated with following values:

  • TTL in the base header is set to 63.

  • The service-path-id (24-bit number) and service-index (8-bit number) are filled in from the filter entry’s action.

  • The MD-Type is set to 1 and the 16-byte metadata is filled in. The source of this metadata is discussed below. If no metadata is provided, this field is zero.

The metadata can be specified in the filter entry’s action. The metadata can either be a 16-byte opaque data hex string (zero-padded if it is smaller than 16 bytes), or it can be derived from the subscriber string (in the Alc-Subsc-Id-Str VSA). In the latter case, the subscriber string is truncated after the first 16 bytes, and therefore, these first 16 bytes should be unique.

Alternatively, the opaque data string can be provided by AAA. This source has precedence over the filter entry’s action.


Modifications to an existing VAS filter are applicable to all hosts using that VAS filter.

  • Adding filter entries is always possible and the new entry is considered on the next filter application at flow creation. Existing flows are not re-evaluated against the modified filter.

  • Removing filter entries is only possible when the filter entry is shut down.

  • Modifying the match criteria on a filter entry is only possible when the filter entry is shut down.

  • Adding or removing a filter entry action is only possible when the filter entry is shut down.

  • Modifying an existing filter entry’s action is always possible and takes effect immediately for all flows currently matched against that filter entry action.

When a filter entry is shut down, all flows matched against any of the filter entry actions is killed. The clear subscriber-mgmt isa-service-chaining vas-filter filter-name command can be executed to clear all flows of all hosts associated with the filter. This command can be used if all flows needing re-evaluation against a modified set of matching criteria.

A maximum of 512 VAS filters with up to 64 entries per filter are supported.

EVPN route updates and tracking

NVE bridging to SF

In this scenario, the NVE acts as a bridge for upstream traffic. The service chain (EVPN) service on subscriber edge is configured with import-mode bridged. In this mode, the SF IP address in the VAS filter action is resolved to obtain SF’s MAC address and VXLAN VTEP/VNI to reach it. If the filter action only specifies SF IP address and no ESI, then EVPN route type-2 (MAC route with MAC and IP address of SF) is used to resolve SF’s IP address. If a route is found, then the SF MAC address and VXLAN VTEP/VNI from the type-2 route are used. If ESI is specified in the filter action, then the ESI is resolved using EVPN type-1 route. In this case, the MAC address is taken from route type-2, and VXLAN and VNI are taken from route type-1. The resolution is downloaded to the ISA. The upstream packet from the host is VXLAN encapsulated. The destination MAC in the inner Ethernet frame is the SF’s MAC address. The source MAC is ISA’s MAC address.

      service-chain 500 import-mode bridged create
             route-distinguisher 65001:500
             route-target import 65000:500
          no shut

A single separate EVPN service (a backbone EVPN) can be configured between the controller and subscriber edge. This is used to advertise type-5 routes for NAT pools on the subscriber edge.

       service-chain 600 create                                          
             route-distinguisher 65001:600                                          
             route-target export 65000:600                                        
             vxlan vni 10
             nat-group 1
             gw-address-range start end
             ip-advertise-routes ipv4 nat 
                 outside-svc 400 outside-pool pool-1
                 outside-svc 400 outside-pool pool-2

A Layer 3 domain (overlay VPRN) must be configured on the NVE. The subscriber edge is connected to this Layer 3 domain on the NVE with an EVPN (the backbone EVPN or R-VPLS on NVE). The R-VPLS interface on the NVE (into this VPRN) must share the same subnet with the subscriber edge. NVE receives the type-5 routes in the backbone EVPN (R-VPLS), and adds the received type-5 prefixes to the FIB in the VPRN that the R-VPLS is connected to. The gateway IP address in the EVPN type-5 routes sent by the subscriber edge must be on the same subnet as the R-VPLS interface on NVE. A range of gateway IP addresses in this subnet are configured under the EVPN service on the subscriber edge, such that each individual ISA gets a gateway IP address to use for exported type-5 routes. NVE acts as a router for downstream traffic from the SF that is destined for NAT outside IP address of the subscriber (NVE bridging traffic to the SF).

Figure 1. NVE bridging traffic to the SF

NVE routing to SF

In this situation, the NVE acts as a router for both upstream (subscriber edge to SF) and downstream traffic (SF to the subscriber edge). The SF must be configured with NVE as the default router.

The service-chain (EVPN) service on subscriber edge is configured with import-mode routed. NAT pools are advertised in EVPN type-5 routes. The ISA’s gateway IP address is sent in a type-5 route (EVPN route type-5).

Figure 2. EVPN route type-5

A type-2 router advertising ISA’s MAC address and gateway IP address is generated (EVPN route type-2).

Figure 3. EVPN route type-2

This EVPN service (called a backbone EVPN) is configured on NVE as an R-VPLS that ties to a VPRN as described in NVE routing to SF. NVE adds the received type-5 prefixes to the FIB in VPRN.

The controller in the data-center also generates type-5 routes carrying IP address or subnet for the SF. The MAC address in a type-5 route must be the R-VPLS interface MAC or NVE’s system MAC address (because the NVE is routing traffic). NVE should also generate an EVPN route type-2 to advertise the MAC of the R-VPLS interface (or single MAC address of the NVE), and optionally the IP address of the R-VPLS interface. This is shown in NVE routing traffic to and from the SF.

On subscriber edge, SF’s IP address in the filter action is resolved in the configured service in the filter action via a best match IP lookup against EVPN route type-5. If the resolved route type-5 has nonzero GW-IP, then a recursive lookup (exact match) is done in the service. If it is resolved by EVPN route type-2, then the next hop MAC (DA MAC that is used in inner Ethernet header) and the VXLAN VNI/VTEP are taken from the route type-2. If GW-IP in route type-5 is zero, then the dest MAC and next hop (VTEP) is take from the route type-5.

Figure 4. NVE routing traffic to and from the SF

Data path on the subscriber edge

Upstream traffic (access to network)

In this scenario, L2-aware NAT is performed. Traffic is subjected to VAS filtering associated with the host. Based on the match entry, the output of the action contains an SF IP address, EVPN service instance, optional ESI, and optional NSH parameters (service-path-id, service-index, and optional metadata). SF IP address and optional ESI are resolved in the indicated EVPN service as per the configured import-mode of the EVPN service (described in the previous sections). The result of resolution is SF MAC address, VXLAN VTEP and VNI. The upstream packet is encapsulated as shown below:

With an optional NSH insertion, the encapsulation used is VXLAN, where the Ethernet header following the VXLAN carries Ether-Type NSH, as shown below. VXLAN-GPE is not supported.

IP => UDP (port 4789) => VXLAN => Ethernet => NSH => IP

  • Outer IP source address: Local VTEP (ISA’s local IP address)

  • Outer IP destination address: Remote VTEP (from EVPN route, as described in previous sections)

  • Destination MAC in Ethernet header: SF MAC address or NVE’s MAC address (depending on bridging or routing on NVE).

  • Source MAC in Ethernet header: MAC address of ISA (generated based on configured MAC prefix)

  • VNI in VXLAN: VNI from EVPN route resolution (as defined in previous sections)

  • Inner IP (payload) source address: NAT outside IP address

  • Inner IP (payload) destination address: original destination on the Internet

  • Without NSH the encapsulation is standard VXLAN, for example, IP => UDP => VXLAN => Ethernet => IP. The parameters in outer IP, inner Ethernet, and IP are as defined about (in other words, the same as for the encapsulation with NSH).

Downstream traffic — from network

The destination IP address in the downstream traffic is the subscriber’s NAT outside IP address. The traffic is revived on the right NAT ISA. NAT flow is looked up. Associated VAS filter action is executed. The action can be steering (with or without NSH), with steering parameters, such as an SF IP address, EVPN service instance, (optional) ESI, and (optional) NSH parameters (service-path-id, service-index, optional meta-data). SF IP address and optional ESI are resolved in the indicated EVPN service as per the configured import-mode of the EVPN service (described in previous sections). The result of resolution is SF MAC address, VXLAN VTEP and VNI. The downstream packet steered to the SF is encapsulated similarly to upstream traffic described in NVE bridging to SF.

Data path on NVE

Upstream traffic from the subscriber edge is either bridged or routed by the NVE. The EVPN service is located based on the VNI. The destination MAC address in the Ethernet frame is either the SF’s MAC address, in which case the traffic is bridged to the SF behind the NVE, or the destination MAC matches the local R-VPLS interface MAC address or single MAC address of the NVE. Then, the packet is routed (based on FIB lookup on the NVE) to the right SF.

Downstream traffic (network to subscriber edge to vas) after being processed by the VAS chain contains the NAT outside IP address in the IP destination address. SFs are configured with NVE as the default router. The packet is received on the NVE from the SF. A lookup is performed in the FIB on NVE (which contains IP prefixes populated by EVPN route type-5 updates). The next-hop IP address is the IP address of the ISA (GW-IP field in route type-5). An ARP and L2FDB lookup respectively provides the destination MAC address, and VXLAN VTEP and VNI. Traffic is VXLAN encapsulated and forwarded in the underlay toward the subscriber edge.