Port features

Multilink point-to-point protocol (MLPPP) is a method of splitting, recombining, and sequencing packets across multiple logical data links. MLPPP is defined in RFC 1990, The PPP Multilink Protocol (MP).

MLPPP allows multiple PPP links to be bundled together, providing a single logical connection between two routers. Data can be distributed across the multiple links within a bundle to achieve high bandwidth. As well, MLPPP allows for a single frame to be fragmented and transmitted across multiple links. This capability allows for lower latency and also for a higher maximum receive unit (MRU).

Multilink protocol is negotiated during the initial LCP option negotiations of a standard PPP session. A system indicates to its peer that it is willing to perform MLPPP by sending the MP option as part of the initial LCP option negotiation.

The system has the following capabilities:

• The system offering the option is capable of combining multiple physical links into one logical link.

• The system is capable of receiving upper layer protocol data units (PDUs) that are fragmented using the MP header and then reassembling the fragments back into the original PDU for processing.

• The system is capable of receiving PDUs of size N octets, where N is specified as part of the option, even if N is larger than the maximum receive unit (MRU) for a single physical link.

When MLPPP has been successfully negotiated, the sending system is free to send PDUs encapsulated and/or fragmented with the MP header.

MP introduces a new protocol type with a protocol ID (PID) of 0x003d. MLPPP 24-bit fragment format and MLPPP 12-bit fragment format show the MLPPP fragment frame structure. Framing to indicate the beginning and end of the encapsulation is the same as that used by PPP and described in RFC 1662, PPP in HDLC-like Framing.

MP frames use the same HDLC address and control pair value as PPP: Address – 0xFF and Control – 0x03. The 2-octet protocol field is also structured the same way as in PPP encapsulation.

The required and default format for MP is the 24-bit format. During the LCP state, the 12-bit format can be negotiated. The 7705 SAR is capable of supporting and negotiating the alternate 12-bit frame format.

The maximum differential delay supported for MLPPP is 25 ms.

The protocol field (PID) is 2 octets. Its value identifies the datagram encapsulated in the Information field of the packet. For MP, the PID also identifies the presence of a 4-octet MP header (or 2-octet, if negotiated).

A PID of 0x003d identifies the packet as MP data with an MP header.

The LCP packets and protocol states of the MLPPP session follow those defined by PPP in RFC 1661. The options used during the LCP state for creating an MLPPP NCP session are described in the sections that follow.

The B&E bits are used to indicate the start and end of a packet. Ingress packets to the MLPPP process have an MTU, which may or may not be larger than the maximum received reconstructed unit (MRRU) of the MLPPP network. The B&E bits manage the fragmentation of ingress packets when the packet exceeds the MRRU.

The B-bit indicates the first (or beginning) packet of a given fragment. The E-bit indicates the last (or ending) packet of a fragment. If there is no fragmentation of the ingress packet, both B&E bits are set to true (=1).

Sequence numbers can be either 12 or 24 bits long. The sequence number is 0 for the first fragment on a newly constructed bundle and increments by one for each fragment sent on that bundle. The receiver keeps track of the incoming sequence numbers on each link in a bundle and reconstructs the required unbundled flow through processing of the received sequence numbers and B&E bits. For a detailed description of the algorithm, see RFC 1990.

The Information field is 0 or more octets. The Information field contains the datagram for the protocol specified in the protocol field.

The MRRU has the same default value as the MTU for PPP. The MRRU is always negotiated during LCP.

On transmission, the Information field of the ending fragment may be padded with an arbitrary number of octets up to the MRRU. It is the responsibility of each protocol to distinguish padding octets from real information. Padding must only be added to the last fragment (E-bit set to true).

The FCS field of each MP packet is inherited from the normal framing mechanism from the member link on which the packet is transmitted. There is no separate FCS applied to the reconstituted packet as a whole if it is transmitted in more than one fragment.

The link control protocol (LCP) is used to establish the connection through an exchange of configure packets. This exchange is complete, and the LCP opened state entered, once a Configure-Ack packet has been both sent and received.

LCP allows for the negotiation of multiple options in a PPP session. MP is somewhat different from PPP, and therefore the following options are set for MP and are not negotiated:

• no async control character map

• no magic number

• no link quality monitoring

• address and control field compression

• protocol field compression

• no compound frames

• no self-describing padding

Any non-LCP packets received during this phase must be silently discarded.

T1/E1 link hold timers (or MLPPP link flap dampening) guard against the node reporting excessive interface transitions. Timers can be set to determine when link up and link down events are advertised; that is, up-to-down and down-to-up transitions of the interface are not advertised to upper layer protocols (are dampened) until the configured timer has expired.

MC-MLPPP on the 7705 SAR supports scheduling based on multiclass implementation. Instead of the standard profiled queue-type scheduling, an MC-MLPPP encapsulated access port performs class-based traffic servicing. The four MC-MLPPP classes are scheduled in a strict priority fashion, as shown in the following table.

For example, if a packet is sent to an MC-MLPPP class 3 queue and all other queues are empty, the 7705 SAR fragments the packet according to the configured fragment size and begins sending the fragments. If a new packet arrives at an MC-MLPPP class 2 queue while the class 3 fragment is still being serviced, the 7705 SAR finishes sending any fragments of the class 3 packet that are on the wire, then holds back the remaining fragments in order to service the higher-priority packet.

The fragments of the first packet remain at the top of the class 3 queue. For packets of the same class, MC-MLPPP class queues operate on a first-in, first-out basis.

The user configures the required number of MLPPP classes to use on a bundle. The forwarding class of the packet, as determined by the ingress QoS classification, is used to determine the MLPPP class for the packet. The mapping of forwarding class to MLPPP class is a function of the user-configurable number of MLPPP classes. The mapping for 4-class, 3-class, and 2-class MLPPP bundles is shown in the following table.

If one or more forwarding classes are mapped to a queue, the scheduling priority of the queue is based on the lowest forwarding class mapped to it. For example, if forwarding classes 0 and 7 are mapped to a queue, the queue is serviced by MC-MLPPP class 3 in a 4-class bundle model.

The 7705 SAR supports only the SLARP keepalive protocol.

For the SLARP keepalive protocol, each system sends the other a keepalive packet at a user configurable interval. The default interval is 10 seconds. Both systems must use the same interval to ensure reliable operation. Each system assigns sequence numbers to the keepalive packets it sends, starting with zero, independent of the other system. These sequence numbers are included in the keepalive packets sent to the other system. Also included in each keepalive packet is the sequence number of the last keepalive packet received from the other system, as assigned by the other system. This number is called the returned sequence number. Each system keeps track of the last returned sequence number it has received. Immediately before sending a keepalive packet, the system compares the sequence number of the packet it is about to send with the returned sequence number in the last keepalive packet it has received. If the two differ by 3 or more, it considers the line to have failed, and will not route higher-level data across it until an acceptable keepalive response is received.

Line timing mode provides physical layer timing (Layer 1) that can be used as an accurate reference for nodes in the network. This mode is immune to any packet delay variation (PDV) occurring on a Layer 2 or Layer 3 link. Physical layer timing provides the best synchronization performance through a synchronization distribution network.

On the 7705 SAR-A variant with T1/E1 ports, line timing is supported on T1/E1 ports. Line timing is also supported on all synchronous Ethernet ports on both 7705 SAR-A variants. Synchronous Ethernet is supported on the XOR ports (1 to 4), configured as either RJ45 ports or SFP ports. Synchronous Ethernet is also supported on SFP ports 5 to 8. Ports 9 to 12 do not support synchronous Ethernet and therefore do not support line timing.

On the 7705 SAR-Ax, line timing is supported on all Ethernet ports.

On the 7705 SAR-H, line timing is supported on:

• all Ethernet ports

• T1/E1 ports on a chassis equipped with a 4-port T1/E1 and RS-232 Combination module

On the 7705 SAR-Hc, line timing is supported on all Ethernet ports.

On the 7705 SAR-M variants with T1/E1 ports, line timing is supported on T1/E1 ports. Line timing is also supported on all RJ45 Ethernet ports and SFP ports on all 7705 SAR-M variants.

In addition, line timing is supported on the following 7705 SAR-M modules:

• 2-port 10GigE (Ethernet) module

• 6-port SAR-M Ethernet module

On the 7705 SAR-Wx, line timing is supported on:

• RJ45 Ethernet ports and optical SFP ports (these ports support synchronous Ethernet and IEEE 1588v2 PTP)

On the 7705 SAR-X, line timing is supported on T1/E1 ports and Ethernet ports.

On the 7705 SAR-8 Shelf V2 and 7705 SAR-18, line timing is supported on:

• 16-port T1/E1 ASAP Adapter card

• 32-port T1/E1 ASAP Adapter card

• 6-port Ethernet 10Gbps Adapter card

• 8-port Gigabit Ethernet Adapter card (dual-rate and copper SFPs do not support synchronous Ethernet)

• 2-port 10GigE (Ethernet) Adapter card

• 10-port 1GigE/1-port 10GigE X-Adapter card (supported on the 7705 SAR-18 only)

• 4-port DS3/E3 Adapter card

• 2-port OC3/STM1 Channelized Adapter card

• 4-port OC3/STM1 / 1-port OC12/STM4 Adapter card

• 4-port OC3/STM1 Clear Channel Adapter card

• Packet Microwave Adapter card on ports that support synchronous Ethernet and on ports that support PCR

Synchronous Ethernet is a variant of line timing and is automatically enabled on ports and SFPs that support it. The operator can select a synchronous Ethernet port as a candidate for the timing reference. The recovered timing from this port is then used to time the system. This ensures that any of the system outputs are locked to a stable, traceable frequency source.

Each SONET/SDH port can be independently configured to be loop-timed (recovered from an Rx line) or node-timed (recovered from the SSU in the active CSM).

A SONET/SDH port’s receive clock rate can be used as a synchronization source for the node.

Each clear channel DS3/E3 port on a 4-port DS3/E3 Adapter card can be independently configured to be loop-timed (recovered from an Rx line), node-timed (recovered from the SSU in the active CSM), or differential-timed (derived from the comparison of a common clock to the received RTP timestamp in TDM pseudowire packets). When a DS3 port is channelized, each DS1 or E1 channel can be independently configured to be loop-timed, node-timed, or differential-timed (differential timing on DS1/E1 channels is supported only on the first three ports of the card). When not configured for differential timing, a DS3/E3 port can be configured to be a timing source for the node.

Each DS3 CES circuit on a 2-port OC3/STM1 Channelized Adapter card card can be loop-timed (recovered from an Rx line) or free-run (timing source is from its own clock). A DS3 circuit can be configured to be a timing source for the node.

Each T1/E1 port can be independently configured for loop-timing (recovered from an Rx line) or node-timing (recovered from the SSU in the active CSM).

In addition, T1/E1 CES circuits on the following can be independently configured for adaptive timing (clocking is derived from incoming TDM pseudowire packets):

• 16-port T1/E1 ASAP Adapter card

• 32-port T1/E1 ASAP Adapter card

• 7705 SAR-M (variants with T1/E1 ports)

• 7705 SAR-X

• 7705 SAR-A (variant with T1/E1 ports)

• T1/E1 ports on the 4-port T1/E1 and RS-232 Combination module

T1/E1 CES circuits on the following can be independently configured for differential timing (recovered from RTP in TDM pseudowire packets):

• 16-port T1/E1 ASAP Adapter card

• 32-port T1/E1 ASAP Adapter card

• 4-port OC3/STM1 / 1-port OC12/STM4 Adapter card (DS1/E1 channels)

• 4-port DS3/E3 Adapter card (DS1/E1 channels on DS3 ports; E3 ports cannot be channelized); differential timing on DS1/E1 channels is supported only on the first three ports of the card

• 7705 SAR-M (variants with T1/E1 ports)

• 7705 SAR-X

• 7705 SAR-A (variant with T1/E1 ports)

• T1/E1 ports on the 4-port T1/E1 and RS-232 Combination module

A T1/E1 port can be configured to be a timing source for the node.

802.3ah Clause 57 (EFM OAM) defines the operations, administration, and maintenance (OAM) sublayer, which is a link level Ethernet OAM. It provides mechanisms for monitoring link operations such as remote fault indication and remote loopback control.

Ethernet OAM gives network operators the ability to monitor the status of Ethernet links and quickly determine the location of failing links or fault conditions.

Because some of the sites where the 7705 SAR will be deployed will only have Ethernet uplinks, this OAM functionality is mandatory. For example, mobile operators must be able to request remote loopbacks from the peer router at the Ethernet layer in order to debug any connectivity issues. EFM OAM provides this capability.

EFM OAM is supported on network and access Ethernet ports and is configured at the port level. The access ports can be configured to tunnel the OAM traffic originated by the far-end devices.

EFM OAM has the following characteristics:

• All EFM OAM, including loopbacks, operate on point-to-point links only.

• EFM loopbacks are always line loopbacks (line Rx to line Tx).

• When a port is in loopback, all frames (except EFM frames) are discarded. If dynamic signaling and routing is used (dynamic LSPs, OSPF, IS-IS, or BGP routing), all services also go down. If all signaling and routing protocols are static (static routes, LSPs, and service labels), the frames are discarded but services stay up.

The following EFM OAM functions are supported:

• OAM capability discovery

• configurable transmit interval with an Information OAMPDU

• active or passive mode

• OAM loopback

• OAMPDU tunneling and termination (for Epipe service)

• dying gasp at network and access ports

• non-zero vendor-specific information field – the 32-bit field is encoded using the format 00:PP:CC:CC and references TIMETRA-CHASSIS-MIB

  • 00 – must be zeros

  • PP – the platform type from tmnxHwEquippedPlatform

  • CC:CC – the chassis type index value from tmnxChassisType that is indexed in tmnxChassisTypeTable. The table identifies the specific chassis backplane.

    The value 00:00:00:00 is sent for all releases that do not support the non-zero value or are unable to identify the required elements. There is no decoding of the peer or local vendor information fields on the network element. The hexadecimal value is included in the show port port-id ethernet efm-oam output.

With ignore-efm-state configured, if the EFM OAM protocol cannot negotiate a peer session or an established session fails, the port will enter the link up state. The link up state is used by many protocols to indicate that the port is administratively up and there is physical connectivity but a protocol (such as EFM OAM) has caused the port operational state to be down. The show port slot/mda/port command output includes a Reason Down field to indicate if the protocol is the underlying reason for the link up state. For EFM OAM, the Reason Down code is efmOamDown. This is shown in the following command output example, where port 1/1/3 is in a link up state.

*A:ALU-1># show port
===============================================================================
Ports on Slot 1
===============================================================================
Port        Admin Link Port    Cfg  Oper LAG/ Port Port Port   C/QS/S/XFP/
Id          State      State   MTU  MTU  Bndl Mode Encp Type   MDIMDX
-------------------------------------------------------------------------------
1/1/1       Down  No   Down    1578 1578    - netw null xcme
1/1/2       Down  No   Down    1578 1578    - netw null xcme
1/1/3       Up    Yes  Link Up 1522 1522    - accs qinq xcme
1/1/4       Down  No   Down    1578 1578    - netw null xcme
1/1/5       Down  No   Down    1578 1578    - netw null xcme
1/1/6       Down  No   Down    1578 1578    - netw null xcme

*A:ALU-1># show port 1/1/3
===============================================================================
Ethernet Interface
===============================================================================
Description        : 10/100/Gig Ethernet SFP
Interface          : 1/1/3                      Oper Speed       : N/A
Link-level         : Ethernet                   Config Speed     : 1 Gbps
Admin State        : up                         Oper Duplex      : N/A
Oper State         : down                       Config Duplex    : full
Reason Down        : efmOamDown
Physical Link      : Yes                        MTU              : 1522
Single Fiber Mode  : No                         Min Frame Length : 64 Bytes
IfIndex            : 35749888                   Hold time up     : 0 seconds
Last State Change  : 12/18/2012 15:58:29        Hold time down   : 0 seconds
Last Cleared Time  : N/A                        DDM Events       : Enabled
Phys State Chng Cnt: 1

......

The EFM OAM protocol can be decoupled from the port state and operational state. In cases where an operator wants to remove the protocol, monitor only the protocol, migrate, or make changes, the ignore-efm-state command can be configured under the config>port>ethernet>efm-oam context.

When the ignore-efm-state command is configured on a port, the protocol behavior is normal. However, any failure in the EFM protocol state (discovery, configuration, time-out, loops, and so on) will not affect the port. Only a protocol warning message will be raised to indicate issues with the protocol. When the ignore-efm-state command is not configured on a port, the default behavior is that the port state will be affected by any EFM OAM protocol fault or clear conditions.

Enabling and disabling this command immediately affects the port state and operating state based on the active configuration, and this is displayed in the show port command output. For example, if the ignore-efm-state command is configured on a port that is exhibiting a protocol error, that protocol error does not affect the port state or operational state and there is no Reason Down code in the output. If the ignore-efm-state command is disabled on a port with an existing EFM OAM protocol error, the port will transition to port state link up, operational state down with reason code efmOamDown.

If the port is a member of a microwave link, the ignore-efm-state command must be enabled before the EFM OAM protocol can be activated. This restriction is required because EFM OAM is not compatible with microwave links.

Cyclic redundancy check (CRC) errors typically occur when Ethernet links are compromised due to optical fiber degradation, weak optical signals, bad optical connections, or problems on a third-party networking element. As well, higher-layer OAM options such as EFM and BFD may not detect errors and trigger appropriate alarms and switchovers if the errors are intermittent, since this does not affect the continuous operation of other OAM functions.

CRC error monitoring on Ethernet ports allows degraded links to be alarmed or failed in order to detect network infrastructure issues, trigger necessary maintenance, or switch to redundant paths. This is achieved through monitoring ingress error counts and comparing them to the configured error thresholds. The rate at which CRC errors are detected on a port can trigger two alarm states. Crossing the configured signal degrade (SD) threshold (sd-threshold) causes an event to be logged and an alarm to be raised, which alerts the operator to a potential issue on a link. Crossing the configured signal failure (SF) threshold (sf-threshold) causes the affected port to enter the operationally down state, and causes an event to be logged and an alarm to be raised.

The CRC error rates are calculated as M✕10E-N, which is the ratio of errored frames allowed for total frames received. The operator can configure both the threshold (N) and a multiplier (M). If the multiplier is not configured, the default multiplier (1) is used.

For example, setting the SD threshold to 3 results in a signal degrade error rate threshold of 1✕10E-3 (1 errored frame per 1000 frames). Changing the configuration to an SD threshold of 3 and a multiplier of 5 results in a signal degrade error rate threshold of 5✕10E-3 (5 errored frames per 1000 frames). The signal degrade error rate threshold must be lower than the signal failure error rate threshold because it is used to notify the operator that the port is operating in a degraded but not failed condition.

A sliding window (window-size) is used to calculate a statistical average of CRC error statistics collected every second. Each second, the oldest statistics are dropped from the calculation. For example, if the default 10-s sliding window is configured, at the 11th second the oldest second of statistical data is dropped and the 11th second is included. This sliding average is compared against the configured SD and SF thresholds to determine if the error rate over the window exceeds one or both of the thresholds, which will generate an alarm and log event.

When a port enters the failed condition as a result of crossing an SF threshold, the port is not automatically returned to service. Because the port is operationally down without a physical link, error monitoring stops. The operator can enable the port by using the shutdown and no shutdown port commands or by using other port transition functions such as clearing the MDA (clear mda command) or removing the cable. A port that is down due to crossing an SF threshold can also be re-enabled by changing or disabling the SD threshold. The SD state is self-clearing, and it clears if the error rate drops below 1/10th of the configured SD rate.

EFM OAM provides a link-layer frame loopback mode, which can be controlled remotely.

To initiate a remote loopback, the local EFM OAM client sends a loopback control OAMPDU by enabling the OAM remote loopback command. After receiving the loopback control OAMPDU, the remote OAM client puts the remote port into local loopback mode.

OAMPDUs are slow protocol frames that contain appropriate control and status information used to monitor, test, and troubleshoot OAM-enabled links.

To exit a remote loopback, the local EFM OAM client sends a loopback control OAMPDU by disabling the OAM remote loopback command. After receiving the loopback control OAMPDU, the remote OAM client puts the port back into normal forwarding mode.

When a port is in local loopback mode (the far end requested an Ethernet OAM loopback), any packets received on the port will be looped back, except for EFM OAMPDUs. No data will be transmitted from the node; only data that is received on the node will be sent back out.

When the node is in remote loopback mode, local data from the CSM is transmitted, but any data received on the node is dropped, except for EFM OAMPDUs.

Remote loopbacks should be used with caution; if dynamic signaling and routing protocols are used, all services go down when a remote loopback is initiated. If only static signaling and routing is used, the services stay up. On the 7705 SAR, the Ethernet port can be configured to accept or reject the remote-loopback command.

Customers who subscribe to Epipe service may have customer equipment running 802.3ah at both ends. The 7705 SAR can be configured to tunnel EFM OAMPDUs received from a customer device to the other end through the existing network using MPLS or GRE, or to terminate received OAMPDUs at a network or an access Ethernet port.

While tunneling offers the ability to terminate and process the OAM messages at the head-end, termination on the first access port at the cell site can be used to detect immediate failures or can be used to detect port failures in a timelier manner. The user can choose either tunneling or termination, but not both at the same time.

In the following figure, scenario 1 shows the termination of received EFM OAMPDUs from a customer device on an access port, while scenario 2 shows the same thing except for a network port. Scenario 3 shows tunneling of EFM OAMPDUs through the associated Ethernet PW. To configure termination (scenario 1), use the config>port>ethernet>efm-oam>no shutdown command.

Dying gasp is used to notify the far end that EFM-OAM is disabled or shut down on the local port. The dying gasp flag is set on the OAMPDUs that are sent to the peer. The far end can then take immediate action and inform upper layers that EFM-OAM is down on the port.

When a dying gasp is received from a peer, the node logs the event and generates an SNMP trap to notify the operator.

A line loopback loops frames received on the corresponding port back toward the transmit direction. Line loopbacks are supported on ports configured for access or network mode.

Similarly, a line loopback with MAC addressing loops frames received on the corresponding port back toward the transmit direction, and swaps the source and destination MAC addresses before transmission. See MAC swapping for more information.

An internal loopback loops frames from the local router back to the framer. This is usually referred to as an equipment loopback. The transmit signal is looped back and received by the interface. Internal loopbacks are supported on ports configured in access mode.

If a loopback is enabled on a port, the port mode cannot be changed until the loopback has been disabled.

A port can support only one loopback at a time. If a loopback exists on a port, it must be disabled or the timer must expire before another loopback can be configured on the same port. EFM-OAM cannot be enabled on a port that has an Ethernet loopback enabled on it. Similarly, an Ethernet loopback cannot be enabled on a port that has EFM-OAM enabled on it.

When an internal loopback is enabled on a port, autonegotiation is turned off silently. This is to allow an internal loopback when the operational status of a port is down. Any user modification to autonegotiation on a port configured with an internal Ethernet loopback will not take effect until the loopback is disabled.

The loopback timer can be configured from 30 s to 86400 s. All non-zero timed loopbacks are turned off automatically under the following conditions: an adapter card reset, an activity switch, or timer expiry. Line or internal loopback timers can also be configured as a latched loopback by setting the timer to 0 s, or as a persistent loopback with the persistent keyword. Latched and persistent loopbacks are enabled indefinitely until turned off by the user. Latched loopbacks survive adapter card resets and activity switches, but are lost if there is a system restart. Persistent loopbacks survive adapter card resets and activity switches and can survive a system restart if the admin save or admin save detail command was executed before the restart. Latched loopbacks (untimed) and persistent loopbacks can be enabled only on Ethernet access ports.

Persistent loopbacks are the only Ethernet loopbacks saved to the database by the admin-save and admin-save-detail commands.

An Ethernet port loopback may interact with other features. See Interaction of Ethernet port loopback with other features for more information.

This section contains information about the following topics:

• CFM loopback overview

• CFM loopback mechanics

Because of the services overhead (that is, pseudowire/VLL, MPLS tunnel, dot1q/qinq and dot1p overhead), it is crucial that configurable variable frame size be supported for end-to-end service delivery.

Observe the following general rules when planning your service and physical maximum transmission unit (MTU) configurations:

• The 7705 SAR must contend with MTU limitations at many service points. The physical (access and network) port, service, and SDP MTU values must be individually defined. MTU points on the 7705 SAR identifies the various MTU points on the 7705 SAR.

• The ports that will be designated as network ports intended to carry service traffic must be identified.

• MTU values should not be modified frequently.

• MTU values must conform to both of the following conditions:

  • the service MTU must be less than or equal to the SDP path MTU

  • the service MTU must be less than or equal to the access port (SAP) MTU

• When the allow-fragmentation command is enabled on an SDP, the current MTU algorithm is overwritten with the configured path MTU. The administrative MTU and operational MTU both show the specified MTU value. If the path MTU is not configured or available, the operational MTU is set to 2000 bytes, and the administrative MTU displays a value of 0. When allow-fragmentation is disabled, the operational MTU reverts to the previous value.

For more information, see the ‟MTU Settings” section in the 7705 SAR Services Guide. To configure various MTU points, use the following commands:

• port MTUs are set with the mtu command, under the config>port context, where the port type can be Ethernet, TDM, serial, or SONET/SDH

• service MTUs are set in the appropriate config>service context

• path MTUs are set with the path-mtu command under the config>service>sdp context

Frame size configuration is supported for an Ethernet port configured as an access or a network port.

For an Ethernet adapter card that does not support jumbo frames, all frames received at an ingress network or access port are policed against 1576 bytes (1572 + 4 bytes of FCS), regardless of the port MTU. Any frames longer than 1576 bytes are discarded and the ‟Too Long Frame” and ‟Error Stats” counters in the port statistics display are incremented. See Jumbo frames for more information.

At network egress, Ethernet frames are policed against the configured port MTU. If the frame exceeds the configured port MTU, the ‟Interface Out Discards” counter in the port statistics is incremented.

When the network group encryption (NGE) feature is used, additional bytes due to NGE packet overhead must be considered. See the ‟NGE Packet Overhead and MTU Considerations” section in the 7705 SAR Services Guide for more information.

IP fragmentation is used to fragment a packet that is larger than the MTU of the egress interface, so that the packet can be transported over that interface.

For IPv4, the router fragments or discards the IP packets based on whether the DF (Do not fragment) bit is set in the IP header. If the packet that exceeds the MTU cannot be fragmented, the packet is discarded and an ICMP message ‟Fragmentation Needed and Don’t Fragment was Set” is sent back to the source IP address.

For IPv6, the router cannot fragment the packet so must discard it. An ICMP message ‟Packet too big” is sent back to the source node.

As a source of self-generated traffic, the 7705 SAR can perform packet fragmentation.

Fragmentation can be enabled for GRE tunnels. See the ‟GRE Fragmentation” section in the 7705 SAR Services Guide for more information.

Jumbo frames are supported on all Ethernet ports.

The maximum MTU size for a jumbo frame on the 7705 SAR is 9732 bytes. The maximum MTU for a jumbo frame may vary depending on the Ethernet encapsulation type, as shown in the following table. The calculations of the other MTU values (service MTU, path MTU, and so on) are based on the port MTU. The values in the table are also maximum receive unit (MRU) values. MTU values are user-configured values. MRU values are the maximum MTU value that a user can configure on an adapter card that supports jumbo frames.

For an Ethernet adapter card, all frames received at an ingress network or access port are policed against the MRU for the ingress adapter card, regardless of the configured MTU. Any frames larger than the MRU are discarded and the ‟Too Long Frame” and ‟Error Stats” counters in the port statistics display are incremented.

At network egress, frames are checked against the configured port MTU. If the frame exceeds the configured port MTU and the DF bit is set, then the ‟MTU Exceeded” discard counter will be incremented on the ingress IP interface statistics display, or on the MPLS interface statistics display if the packet is an MPLS packet.

For example, on adapter cards that do not support an MTU greater than 2106 bytes, fragmentation is not supported for frames greater than the maximum supported MTU for that card (that is, 2106 bytes). If the maximum supported MTU is exceeded, the following occurs:

• An appropriate ICMP reply message (Destination Unreachable) is generated by the 7705 SAR. The router ensures that the ICMP generated message cannot be used as a DOS attack (that is, the router paces the ICMP message).

• The appropriate statistics are incremented.

Jumbo frames offer better utilization of an Ethernet link because as more payload is packed into an Ethernet frame of constant size, the ratio of overhead to payload is minimized.

From the traffic management perspective, large payloads may cause long delays, so a balance between link utilization and delay must be found. For example, for ATM VLLs, concatenating a large number of ATM cells when the MTU is set to a very high value could generate a 9-kB ATM VLL frame. Transmitting a frame that large would take more than 23 ms on a 3-Mb/s policed Ethernet uplink.

The following table displays the default and maximum port MTU values that are dependent upon the port type, mode, and encapsulation type.

Notes:

1. The maximum MTU value is supported only on cards that have buffer chaining enabled.
2. On the Packet Microwave Adapter card, the MWA ports support 4 bytes less than the Ethernet ports. MWA ports support a maximum MTU of 9720 bytes (null) or 9724 bytes (dot1q). MWA ports do not support QinQ.
3. QinQ is supported only on access ports.
4. For X.21 serial ports at super-rate speeds.

For more information, see the ‟MTU Settings” section in the 7705 SAR Services Guide.

The 7705 SAR supports link aggregation groups (LAGs) based on the IEEE 802.1ax standard (formerly 802.3ad). Link aggregation provides:

• increased bandwidth by combining multiple links into one logical link (in active/active mode)

• load sharing by distributing traffic across multiple links (in active/active mode)

• redundancy and increased resiliency between devices by having a standby link to act as backup if the active link fails (in active/standby mode)

In the 7705 SAR implementation, all links must operate at the same speed.

Packet sequencing must be maintained for any given session. The hashing algorithm deployed by Nokia routers is based on the type of traffic transported to ensure that all traffic in a flow remains in sequence while providing effective load sharing across the links in the LAG. See LAG and ECMP hashing for more information.

LAGs must be statically configured or formed dynamically with Link Aggregation Control Protocol (LACP). See LACP and active/standby operation for information about LACP.

All Ethernet-based supported services can benefit from LAG, including:

• network interfaces and SDPs

• spoke SDPs, mesh SDPs, and EVPN endpoints

• IES and VPRN interfaces and SAPs

• Ethernet and IP pseudowire SAPs

• routed VPLS (r-VPLS) SAPs

LAGs are supported on access, network, and hybrid ports. A LAG can be in active/active mode or in active/standby mode for access, network, or hybrid ports. Active/standby mode is a subset of active/active mode if subgroups are enabled.

LAGs are supported on access ports on the following:

• 8-port Gigabit Ethernet Adapter card

• 10-port 1GigE/1-port 10GigE X-Adapter card (10-port GigE mode)

• 6-port Ethernet 10Gbps Adapter card

• 4-port SAR-H Fast Ethernet module

• 6-port SAR-M Ethernet module

• Packet Microwave Adapter card (for ports not in a microwave link)

• all fixed platforms

LAGs are supported on network ports on the following:

• 8-port Gigabit Ethernet Adapter card

• 10-port 1GigE/1-port 10GigE X-Adapter card

• 6-port Ethernet 10Gbps Adapter card

• 4-port SAR-H Fast Ethernet module

• 6-port SAR-M Ethernet module

• Packet Microwave Adapter card (for ports not in a microwave link and ports in a 1+0 network microwave link; LAGs are not supported on ports in a 1+1 HSB microwave link)

• all fixed platforms

LAGs are supported on hybrid ports on the following:

• 8-port Gigabit Ethernet Adapter card

• 10-port 1GigE/1-port 10GigE X-Adapter card (10-port GigE mode)

• 6-port Ethernet 10Gbps Adapter card

• 6-port SAR-M Ethernet module

• Packet Microwave Adapter card (for ports not in a microwave link)

• all fixed platforms

On access ports, a LAG supports active/active and active/standby operation. For active/standby operation the links must be in different subgroups. Links can be on the same platform or adapter card/module or distributed over multiple components. Load sharing is supported among the active links in a LAG group.

On network ports, a LAG supports active/active and active/standby operation. For active/standby operation the links must be in different subgroups. Links can be on the same platform or adapter card/module or distributed over multiple components. Load sharing is supported among the active links in a LAG group. Any tunnel type (for example, IP, GRE, or MPLS) transporting any service type, any IP traffic, or any labeled traffic (LER, LSR) can use the LAG load-sharing, active/active, and active/standby functionality.

LAGs are supported on network 1+0 microwave links. Ports that are in a microwave link can be added to the same LAG as ports that are not in a microwave link. Ports belonging to a microwave link must have limited autonegotiation enabled before the link can be added to a LAG.

A LAG that contains ports in a microwave link must have LACP enabled for active/standby operation. Static LAG configuration (without LACP) is not supported for active/standby LAGs with microwave-enabled ports.

On hybrid ports, a LAG supports active/active and active/standby operation. For active/standby operation the links must be in different subgroups. Links can be on the same platform or adapter card/module or distributed over multiple components. Load sharing is supported among the active links in a LAG group.

A LAG group with assigned members can be converted from one mode to another as long as the number of member ports are supported in the new mode and the ports all support the new mode, none of the members belong to a microwave link, and the LAG group is not associated with a network interface or a SAP.

A subgroup is a group of links within a LAG. On access, network, or hybrid ports, a LAG can have a maximum of four subgroups and a subgroup can have links up to the maximum number supported on the LAG. The LAG is active/active if there is only one sub-group and is active/standby if there is more than one subgroup.

When configuring a LAG, most port features (port commands) can only be configured on the primary member port. The configuration, or any change to the configuration, is automatically propagated to any remaining ports within the same LAG. Operators cannot modify the configurations on non-primary ports. For more information, see Configuring LAG parameters.

If the LAG has one member link on a second-generation (Gen-2) Ethernet adapter card and the other link on a third-generation (Gen-3) Ethernet adapter card or platform, a mix-and-match scenario exists for traffic management on the LAG SAP. In this case, all QoS parameters for the LAG SAP are configured but only those parameters applicable to the active member link are used. See LAG support on mixed-generation hardware for more information.

Configuring a multiservice site (MSS) aggregate rate can restrict the use of LAG SAPs. For more information, see the ‟MSS and LAG interaction on the 7705 SAR-8 Shelf V2 and 7705 SAR-18” section in the 7705 SAR Quality of Service Guide.

On access, network, and hybrid ports, where multiple links in a LAG can be active at the same time, normal operation is that all non-failing links are active and traffic is load-balanced across all the active links. In some cases, however, it is desirable to have only some of the links active and the other links kept in standby mode. The Link Aggregation Control Protocol (LACP) is used to make the selection of the active links in a LAG predictable and compatible with any vendor equipment. The mechanism is based on the IEEE 802.1ax standard so that interoperability is ensured.

LACP is disabled by default and therefore must be enabled on the LAG if required. LACP can be used in either active mode or passive mode. The mode must match with connected CE devices for proper operation. For example, if the LAG on the 7705 SAR end is configured to be active, the CE end must be passive.

The following figure shows the interconnection between a DSLAM and a LAG aggregation node. In this configuration, LAG is used to protect against hardware failure. If the active link goes down, the link on standby takes over (see LAG on access failure switchover). The links are distributed across two different adapter cards to eliminate a single point of failure.

LACP handles active/standby operation of LAG subgroups as follows:

• Each link in a LAG is assigned to a subgroup. On access, network, and hybrid ports, a LAG can have a maximum of four subgroups and a subgroup can have up to the maximum number of links supported for the LAG. The selection algorithm implemented by LACP ensures that only one subgroup in a LAG is selected as active.

• The algorithm selects the active link as follows:

  • If multiple subgroups satisfy the selection criteria, the subgroup currently active remains active. Initially, the subgroup containing the highest-priority (lowest value) eligible link is selected as active.

  • An eligible member is a link that can potentially become active. This means it is operationally up, and if the slave-to-partner flag is set, the remote system did not disable its use (by signaling standby).

• The selection algorithm works in a revertive mode (for details, see the IEEE 802.1ax standard). This means that every time the configuration or status of a subgroup changes, the selection algorithm reruns. If multiple subgroups satisfy the selection criteria, the subgroup currently active remains active. This behavior does not apply if the selection-criteria hold-time parameter is set to infinite.

Log events and traps are generated at both the LAG and link level to indicate any LACP changes. See the TIMETRA-LAG-MIB for details.

QoS on access port LAGs (access ports and hybrid ports in access mode) is handled differently from QoS on network port LAGs (see QoS for LAG on network). Based on the configured hashing, traffic on a SAP can be sent over multiple LAG ports or can use a single port of a LAG. There are two user-selectable adaptive QoS modes (distribute and link) that allow the user to determine how the configured QoS rate is distributed to each of the active LAG port SAP queue schedulers, SAP schedulers (H-QoS), and MSS schedulers. These modes are:

• adapt-qos distribute

  For SAP queue schedulers, SAP schedulers (H-QoS), and SAP egress MSS schedulers, distribute mode divides the QoS rates (as specified by the SLA) equally among the active LAG links (ports). For example, if a SAP queue PIR and CIR are configured on an active/active LAG SAP to be 200 Mb/s and 100 Mb/s respectively, and there are four active LAG ports, the SAP queue on each LAG port will be configured with a PIR of 50 Mb/s (200/4) and a CIR of 25 Mb/s (100/4).

  For the SAP ingress MSS scheduler, the scheduler rate is configured on an MDA basis. Distributive adaptive QoS divides the QoS rates (as specified by the SLA) among the active link MDAs proportionally to the number of active links on each MDA.

  For example, if an MSS shaper group with an aggregate rate of 200 Mb/s and a CIR of 100 Mb/s is assigned to an active/active LAG SAP where the LAG has two ports on MDA 1 and three ports on MDA 2, the MSS shaper group on MDA 1 will have an aggregate rate of 80 Mb/s (200 ✕ 2/5 of the SLA) and a CIR of 40 Mb/s (100 ✕ 2/5 of the SLA). MDA 2 will have an aggregate rate of 120 Mb/s (200 ✕ 3/5) and a CIR of 60 Mb/s (100 ✕ 3/5).

• adapt-qos link (default)

  For SAP queue schedulers, SAP schedulers (H-QoS), and SAP egress MSS schedulers, link mode forces the full QoS rates (as specified by the SLA) to be configured on each of the active LAG links. For example, if a SAP queue PIR and CIR are configured on an active/active LAG SAP to be 200 Mb/s and 100 Mb/s respectively, and there are two active LAG ports, the SAP queue on each LAG port will be configured to the full SLA, which is a PIR of 200 Mb/s and a CIR of 100 Mb/s.

  For the SAP ingress MSS scheduler, the scheduler rate is configured on an MDA basis. In LAG link mode, each active LAG link MDA MSS shaper scheduler is configured with the full SLA. For example, if an MSS shaper group is configured with an aggregate rate of 200 Mb/s and CIR of 100 Mb/s and is assigned to an active/active LAG SAP with three ports on MDA 1 and two ports on MDA 2, the MSS shaper group on MDA 1 and MDA 2 are each configured with the full SLA of 200 Mb/s for the aggregate rate and 100 Mb/s for the CIR.

The following table shows examples of rate and bandwidth distributions based on the adapt-qos mode configuration.

The following restrictions apply to ingress MSS LAG adaptive QoS (distribute mode):

• A unique MSS shaper group must be used per LAG when a non-default ingress MSS shaper group is assigned to a LAG SAP using adaptive QoS.

• When a shaper group is assigned to a LAG SAP using adaptive QoS, all ports in the LAG group must have their MDAs assigned to the same shaper policy.

The following restrictions apply to egress MSS LAG:

• The shaper policy for all LAG ports in a LAG must be the same and can only be configured on the primary LAG port member.

The following limitations apply to adaptive QoS (distribute mode):

• The QoS rates for an ingress LAG using adaptive QoS are only distributed among the active links when a non-default shaper group is used. If a default shaper group is used, the full QoS rates are configured for each port in the LAG as if link mode is being used.

• The QoS rates for an ingress or egress LAG using adaptive QoS will not be distributed among the active links when a user sets the PIR/CIR on a SAP queue, or aggregate rate/CIR on a SAP scheduler or MSS scheduler, to the default values (max and 0).

QoS on network port LAGs is handled differently from QoS on access port LAGs. The adapt-qos command is not supported on network port LAGs. However, QoS behavior on network port LAGs is similar to QoS on access port LAGs configured for adapt-qos link mode. For network queue and per-VLAN shapers, the full QoS rates are configured on each of the active LAG links. For example, if a per-VLAN shaper agg-rate-limit aggregate rate (PIR) and CIR are configured on an active/active LAG interface to be 200 Mb/s and 100 Mb/s respectively, and there are two active LAG ports, the per-VLAN shaper on each LAG port will be configured to an aggregate rate of 200 Mb/s and a CIR of 100 Mb/s.

To prevent traffic congestion and ease the effects of possible bursts, a fabric shaper is implemented on each adapter card. Traffic being switched to a LAG SAP on an access interface goes through fabric shapers that are either in aggregate mode or destination mode. When in destination mode, the multipoint shaper is used to set the rate on all adapter cards. For more information about the modes used in fabric shaping, see the 7705 SAR Quality of Service Guide, ‟Configurable ingress shaping to fabric (access and network)”.

Hold-down timers control how quickly a LAG responds to operational port state changes. The following timers are supported:

• port-level hold-time (up/down) timer

  This timer controls the delay before a port is added to or removed from a LAG when the port comes up or goes down. Each port in the LAG has the same timer value, which is configured on the primary LAG link (port). The timer is set with the config>port>ethernet>hold-time command.

• subgroup-level hold-down timer

  This timer controls the delay before a switch from the current subgroup to a new candidate subgroup, selected by the LAG subgroup selection algorithm. The timer is set with the config>lag>selection-criteria command.

  The timer can be configured to never expire, which prevents a switch from an operationally up subgroup to a new candidate subgroup. This setting can be manually overridden by using the tools>perform>force>lag-id command (see the 7705 SAR OAM and Diagnostics Guide, ‟Tools Command Reference”, for information about this command).

  If the port-level timer is set, it must expire before the subgroup selection occurs and this timer is started. The subgroup-level timer is supported only for LAGs running LACP.

• LAG-level hold-down timer

  This timer controls the delay before a LAG is declared operationally down when the available links fall below the required port or bandwidth minimum. This timer is recommended for MC-LAG operation. The timer prevents a LAG from being brought down when an MC-LAG switchover executes a make-before-break switch. The LAG-level timer is set with the config>lag>hold-time down command.

  If the port-level timer is set, it must expire before the LAG operational status is processed and this timer is started.

Multi-chassis LAG (MC-LAG) is a redundancy feature on the 7705 SAR, useful for nodes that are taken out of service for maintenance, upgrades, or relocation. MC-LAG also provides redundancy for incidents of peer nodal failure. See the ‟Multi-chassis LAG redundancy” section in the 7705 SAR Basic System Configuration Guide.

Some Layer 2-capable network equipment devices support LAG protected links in an active/standby mode but without LACP. This is commonly referred to as static LAG. In order to interwork with these products, the 7705 SAR supports configuring LAG without LACP.

LACP provides a standard means of communicating health and status information between LAG peers. If LACP is not used, the peers must be initially configured in a way that ensures that the ports on each end are connected and communicating. Otherwise, LAG will not be active. Which LAG peer is made active is a local decision. If the port priority settings are the same for all ports, it is possible that the two ends will select ports on different physical links and LAG will not be active. Decide the primary link by setting the port priority for the LAG on each peer to ensure that the active ports on each end coincide with the same physical link.

The key parameters for configuring static LAG are selection-criteria (set to best-port) and standby-signaling (set to power-off). The selection criteria is used to determine which selection algorithm decides the primary port (the active port in a no-fault condition). It is always the subgroup with the best-port (the highest-priority port - lowest configured value) that is chosen as the active subgroup. The selection criteria must be set to best-port before standby signaling can be placed in power-off mode. Once the selection criteria is set to best-port, setting the standby-signaling parameter to power-off causes the transmitters on the standby ports to be powered down.

After a switchover caused by a failure on the active link, the transmitters on the standby link are powered on. The switch time for static LAG is typically longer than it is with LACP, due to the time it takes for the transmitters to come up and transmission to be established. When the fault is restored, static LAG causes a revertive switch to take place. The revertive switch is of shorter duration than the initial switchover since the system is able to prepare the other side for the switch and initiate the switchover once it is ready.

This section contains information about the following topics:

• LAG configuration at SAP level

• LAG configuration at port level

The 7705 SAR supports the application of BFD to monitor individual LAG link members in order to speed up the detection of link failures. When BFD is associated with an Ethernet LAG, BFD sessions are set up over each link member. These asynchronous independent sessions are referred to as micro-BFD sessions. When micro-BFD is configured, a link is not operational in the associated LAG until the associated micro-BFD session is fully established. The link member is taken out of the operational state in the LAG if the micro-BFD session fails.

Although ETH-EFM can be used on individual LAG links, EFM timers are limited to 100 ms. With micro-BFD, 10-ms timers are supported, which allows for much faster detection times. The micro-BFD sessions use the well-known destination UDP port 6784 over LAG links. The source MAC address is the local system MAC address for the LAG interface. The micro-BFD packets use the well-known destination MAC address 01:00:5e:90:00:01.

The 7705 SAR supports per-flow hashing for LAG and ECMP. Per-flow hashing uses information in a packet as an input to the hash function, ensuring that any given traffic flow maps to the same egress LAG port or ECMP path.

Depending on the type of traffic that needs to be distributed in an ECMP or LAG path, different variables are used as the input to the hashing algorithm that determines the selection of the next hop (ECMP) or port (LAG). The hashing result can be changed using the options described in Per-service hashing, LSR hashing, Layer 4 load balancing, TEID hashing for GTP-encapsulated traffic, and Entropy labels.

The following table summarizes the possible inputs to the hashing algorithm for ECMP and LAG.

Fragmented packets cannot use Layer 4 UDP/TCP ports or tunnel endpoint IDs (TEIDs). The datapath looks at IP source address and destination address only, even if configured to use Layer 4 UDP/TCP ports or TEID.

In the table, the hashing inputs in the Service ID column and the inputs in the other columns are mutually exclusive. Where checkmarks appear on both the per-service and per-flow sides of the table, see the table note in the Service ID column to determine when per-service hashing is used.

Notes:

1. The system IP address can be included as a hashing input using the system-ip-load-balancing command at the system level. For MPLS LSR, this configuration is ignored when the hashing algorithm is configured as lbl-only using the lsr-load-balancing command.

2. Optional hashing input that is included when the use-ingress-port option is enabled in the lsr-load-balancing command.

3. Optional hashing input that is included when the l4-load-balancing command is enabled (for all except MPLS LSR) or when the hashing algorithm is configured as lbl-ip-l4-teid using the lsr-load-balancing command (for MPLS LSR only). Layer 4 load balancing at the service level is not affected by Layer 4 load balancing at the system, router interface, or service interface levels (IES and VPRN).

4. Optional hashing input that is included when the teid-load-balancing command is enabled (for all except MPLS LSR) or when the hashing algorithm is configured as lbl-ip-l4-teid using the lsr-load-balancing command (for MPLS LSR only). TEID load balancing at the service level is not affected by TEID load balancing at the router interface or service interface levels (IES and VPRN).

5. Only applies to multicast traffic. The internal multicast group ID is generated from either the (S,G) record (IGMP snooping, MLD snooping, and PIM snooping), the point-to-multipoint label binding, or the VPLS service creation.

6. Only for Layer 3 traffic going to a Layer 3 spoke SDP interface.

7. Only included when the first 4 bits (first nibble) after the last MPLS header (bottom of stack = 1) has a value of 4 (decimal), in which case the next header encapsulation is considered to be an IPv4 header.

8. Optional hashing input that is included when LSR hashing is configured as lbl-ip or label-ip-l4-teid using the lsr-load-balancing command.

9. MPLS label stack and entropy label are mutually exclusive hashing inputs. When an entropy label indicator (ELI) and entropy label (EL) are found in the label stack, the MPLS labels are not used as hashing inputs.

10. When the per-service-hashing command is enabled in a VPLS service, the service ID and an internal spoke SDP binding ID are included as inputs to the hashing algorithm.

11. When the per-service-hashing command is enabled in a VPLS or Epipe service, only the service ID is included as an input to the hashing algorithm.

The 7705 SAR supports load balancing based on service ID, as shown in Hashing algorithm inputs (ECMP and LAG) , The 7705 SAR uses the service ID as the input to the hash function. Per-service and per-flow hashing are mutually exclusive features.

For IPv4 and IPv6 routed traffic under ECMP operation, the service ID is used as the hashing input for Layer 3 traffic going to a Layer 3 spoke SDP interface. Otherwise, per-flow load balancing is used.

For Epipe and VPLS services under LAG operation, the per-service-hashing command and the l4-load-balancing and teid-load-balancing commands are mutually exclusive. Load balancing via per-service hashing is configured under the config>service> epipe>load-balancing and config>service>vpls> load-balancing contexts.

LSR hashing operates on the label stack and can also include hashing on the IP header if the packet is an IPv4 packet. The label-IP hashing algorithm can also include the Layer 4 header and the TEID field. The default hash is on the label stack only. IPv4 is the only IP hashing supported on a 7705 SAR LSR.

When a 7705 SAR is acting as an LSR, it considers a packet to be IP if the first nibble following the bottom of the label stack is 4 (IPv4). This allows the user to include an IP header in the hashing routine at an LSR in order to spray labeled IP packets over multiple equal-cost paths in ECMP in an LDP LSP and/or over multiple links of a LAG group in all types of LSPs.

Other LSR hashing options include label stack profile options on the significance of the bottom-of-stack label (VC label), the inclusion or exclusion of the ingress port, and the inclusion or exclusion of the system IP address.

LSR load balancing is configured using the config>system>lsr-load-balancing or config>router>if>load-balancing>lsr-load-balancing command. Configuration at the router interface level overrides the system-level configuration for the specified interface.

If an ELI is found in the label stack, the entropy label is used as the hash result. Hashing continues based on the configuration of label-only (lbl-only), label-IP (lbl-ip), or label-IP with Layer 4 header and TEID (lbl-ip-l4-teid) options.

The IP Layer 4 load-balancing option includes the TCP/UDP source and destination port numbers in addition to the source and destination IP addresses in per-flow hashing of IP packets. By including the Layer 4 information, a source address/destination address default hash flow can be subdivided into multiple finer-granularity flows if the ports used between a source address and destination address vary.

Layer 4 load balancing is configured at the system level using the config>system>l4-load-balancing command. It can also be configured at the router interface level or the service interface level (IES and VPRN). Configuration at the router interface or service interface level overrides the system-level configuration for the specified interface or service.

For LSR LDP ECMP, Layer 4 load balancing is configured using the lbl-ip-l4-teid option in the lsr-load-balancing command at the system level or router interface level. Configuration at the router interface level overrides the system-level configuration for the specified interface.

Layer 4 load balancing can also be configured at the service level for Epipe and VPLS services. Layer 4 load balancing at the service level is not impacted by Layer 4 load balancing at the system, router interface, or service interface levels.

GTP is the GPRS (general packet radio service) tunneling protocol. The tunnel endpoint identifier (TEID) is a field in the GTP header. TEID hashing can be enabled on Layer 3 interfaces. The hash algorithm identifies the GTP-U protocol by checking the UDP destination port (2152) of an IP packet to be hashed. If the value of the port matches, the packet is assumed to be GTP-U. For GTPv1 packets, the TEID value from the expected header location is then included in the hash. For GTPv2 packets, the TEID flag value in the expected header is additionally checked to verify whether the TEID is present. If the TEID is present, it is included in the hash algorithm inputs.

TEID load balancing is configured at the router interface level using the config> router>if>teid-load-balancing command. It can also be configured at the IES or VPRN service interface level.

For LSR LDP ECMP, TEID load balancing is configured using the lbl-ip-l4-teid option in the lsr-load-balancing command at the system level or router interface level. Configuration at the router interface level overrides the system-level configuration for the specified interface.

TEID load balancing can also be configured at the service level for Epipe and VPLS services. TEID load balancing at the service level is not impacted by TEID load balancing at the router interface or service interface levels.

The 7705 SAR supports MPLS entropy labels on RSVP-TE and SR-TE LSPs, as per RFC 6790. The entropy label provides greater granularity for load balancing on an LSR where load balancing is typically based on the MPLS label stack.

If an ELI is found in the label stack, the entropy label is used as the hash result and hashing continues based on the configuration of label-only (lbl-only) or label-IP (lbl-ip) options. For information about the behavior of LSR hashing when entropy label is enabled, see LSR hashing.

To support entropy labels on RSVP-TE and SR-TE LSPs:

• the eLER must signal to the ingress node that entropy label capability (ELC) is enabled, meaning that the eLER can receive and process an entropy label for an LSP tunnel. Entropy labels are supported on RSVP-TE and SR-TE tunnels. Entropy labels are not supported on point-to-multipoint LSPs, BGP tunnels, or LDP FECs.

• the iLER must receive the entropy label capability signal and be configured to enable the insertion of entropy labels for the spoke SDP, mesh SDP, or EVPN endpoint. Inserting an entropy label adds two labels in the MPLS label stack: the entropy label itself and the ELI.

At the eLER, use the config>router>rsvp>entropy-label-capability command to enable entropy label capability on RSVP-TE LSPs.

At the iLER, use the entropy-label command to enable the insertion of the entropy label into the label stack. The command is found under the following services and protocols:

• Epipe and VPLS

  • config>service>epipe>spoke-sdp

  • config>service>epipe>bgp-evpn>mpls

  • config>service>vpls>spoke-sdp

  • config>service>vpls>mesh-sdp

  • config>service>vpls>bgp-evpn>mpls

• IS-IS, OSPF, and MPLS

  • config>router>isis>segment-routing

  • config>router>ospf>segment-routing

  • config>router>mpls

For details on entropy labels, see the ‟MPLS entropy labels” section in the 7705 SAR MPLS Guide.

Security parameter index (SPI) load balancing provides a mechanism to improve the hashing performance of IPSec encrypted traffic. IPSec-tunneled traffic transported over a LAG typically relies on IP header hashing only. For example, in LTE deployments, TEID hashing cannot be performed because of encryption, and the system performs IP-only tunnel-level hashing. Because each SPI in the IPSec header identifies a unique SA, and therefore a unique flow, these flows can be hashed individually without impacting packet ordering. In this way,

The 7705 SAR allows enabling SPI hashing per Layer 3 interface (this is the incoming interface for hash on system egress) or per Layer 2 VPLS service. When SPI hashing is enabled, an SPI value from the ESP/AH header is used in addition to any other IP hash input based on the per-flow hash configuration: source/destination IPv4/IPv6 addresses and Layer 4 source/destination ports in case NAT traversal is required and Layer 4 load balancing is enabled. If the ESP/AH header is not present in a packet received on a given interface, the SPI will not be part of the hash inputs and the packet is hashed as per other hashing configurations. SPI hashing is not used for fragmented traffic in order to ensure that first and subsequent fragments use the same hash inputs.

SPI hashing is supported for IPv4 and IPv6 tunnel unicast traffic.

The following figure shows the SONET hierarchy at STS-12 (OC-12).

A SONET multiplexing structure allows several combinations of signal transportation. For example, at the STS-3 (OC-3) level:

• STS-3 is achieved by interleaving three STS-1s byte by byte

• an STS-1 payload can be subdivided into virtual tributary groups (VTGs) and virtual tributaries (VTs). Each STS-1 may contain seven VTGs, which in turn carry sub-STS traffic in VTs. There are four VT sizes:

  • VT1.5 (1.728 Mb/s) (typically used for DS1, indicated in the CLI as vt15)

  • VT2 (2.304 Mb/s) (typically transports one E1)

  • VT3 (3.456 Mb/s) (not shown in the figure)

  • VT6 (6.912 Mb/s)

• each VTG can contain four VT1.5s, three VT2s, two VT3s, or one VT6

• each VTG can carry only VTs of the same size

The following figure shows the SDH hierarchy at STM-4 (OC-12).

An SDH multiplexing structure allows several combinations of signal transportation. For example, at the STM-1 (OC-3) level:

• one STM-1 payload supports one administrative unit group (AUG)

• each AUG can contain either three administrative units (AU-3s) or a single AU-4

For example, the hierarchical possibilities for a single AU-4 are:

• each AU-4 transports data via a virtual container (VC-4)

• a VC-4 consists of three tributary unit groups (TUG-3s), where either:

  • a single tributary unit (TU-3) can be multiplexed via a TUG-3, containing a VC-3 plus the VC-3 path overhead (POH) and TU-3 pointer

  • seven TUG-2s can be multiplexed via a TUG-3, where each TUG-2 can contain one TU-2, three TU-12s, or four TU-11s

• the AU-4 structure addresses the data as (K, L, M), where K is the TUG-3 number, L is the TUG-2 number, and M is the TU-11/TU-12 number

The 7705 SAR supports a subset of the standards-based channel mapping options. The following table shows path support on the 7705 SAR.

The 7705 SAR CLI always uses the SONET VT frame convention. For example, the same SONET CLI syntax and nomenclature would be used to configure both a VT1.5 and a VC11. The framing {sonet | sdh} command determines whether VTs or VCs are being configured. Use the show>port-tree port-id command to display the SONET/SDH path containers.

Automatic protection switching (APS) allows users to protect a SONET/SDH port or link with a backup (protection) facility of the same speed but from a different adapter card. APS provides protection against a port, signal, or adapter card failure. The 7705 SAR supports 1+1 APS protection in compliance with GR-253-CORE and ITU-T Recommendation G.841 to provide SONET/SDH carrier-grade reliability. All SONET/SDH paths and channels within a SONET/SDH port are protected.

When APS is enabled, the 7705 SAR constantly monitors the health of the APS links, APS ports, and APS-equipped adapter cards. If the signal on the active (working) port degrades or fails, the network proceeds through a predefined sequence of steps to transfer (or switch over) traffic processing to the protection port. This switchover is done very quickly to minimize traffic loss. Traffic is streamed from the protection port until the fault on the working port is cleared, at which time the traffic may optionally revert to the working port.

The 7705 SAR supports 1+1 single-chassis APS (SC-APS) and 1+1 multi-chassis APS (MC-APS). In an SC-APS group, both the working and protection circuit must be configured on the same node. In an MC-APS group, the working and protection circuits are configured on two separate nodes, providing protection from node failure in addition to protection from link and hardware failure.

Unidirectional and bidirectional modes are supported:

• unidirectional APS (Uni-1Plus1) – in unidirectional mode, only the port in the failed direction switches to the protection port. Unidirectional mode is supported only on SC-APS.

• bidirectional APS – in bidirectional mode, a failure in either direction causes both the near-end and far-end equipment to switch to the protection port in each direction. Bidirectional mode is the default mode and is supported on both SC-APS and MC-APS.

For SC-APS and MC-APS with MEF 8 services where the remote device performs source MAC validation, the MAC address of the channel group in each of the redundant interfaces may be configured to the same MAC address using the mac CLI command.

In an SC-APS group, both the working and protection circuits terminate on the same node. SC-APS is supported in unidirectional or bidirectional mode on:

• 2-port OC3/STM1 Channelized Adapter cards for TDM CES (Cpipes) and TDM CESoETH with MEF 8 with DS3/DS1/E1/DS0 channels

• 4-port OC3/STM1 / 1-port OC12/STM4 Adapter cards for MLPPP access ports or TDM CES (Cpipes) and TDM CESoETH (MEF 8) access ports with DS1/E1 channels, or on a network port configured for POS

• 4-port OC3/STM1 Clear Channel Adapter cards network side (configured for POS operation)

SC-APS with TDM access is supported on DS3, DS1, E1, and DS0 (64 kb/s) channels.

The working and protection circuits of an SC-APS group must be on two ports on different adapter cards.

The following figure shows an SC-APS group with physical port and adapter card failure protection. SC-APS application shows a packet network using SC-APS.

MC-APS extends the functionality offered by SC-APS to include protection against 7705 SAR node failure. MC-APS is supported in bidirectional mode on:

• 2-port OC3/STM1 Channelized Adapter cards for TDM CES (Cpipes) and TDM CESoETH with MEF 8 with DS3/DS1/E1/DS0 channels

• 4-port OC3/STM1 / 1-port OC12/STM4 Adapter cards for MLPPP access ports or CES (Cpipes) and TDM CESoETH (MEF 8) access ports with DS1/E1 channels

MC-APS with TDM access is supported on DS3, DS1, E1, and DS0 (64 kb/s) channels. TDM SAP-to-SAP with MC-APS is not supported.

With MC-APS, the working circuit of an APS group can be configured on one 7705 SAR node while the protection circuit of the same APS group is configured on a different 7705 SAR node. The working and protection nodes are connected by an IP link that establishes an MC-APS signaling path between the nodes.

The working and protection circuits must have compatible configurations, such as the same speed, framing, and port type. The circuits in an APS group on both the working and protection nodes must also have the same group ID, but they can have different port descriptions. In order for MC-APS to function correctly, pseudowire redundancy must be configured on both the working and protection circuits. For more information, see the 7705 SAR Services Guide. MC-APS with pseudowire redundancy also supports inter-chassis backup (ICB); see MC-APS and inter-chassis backup for more information.

The working and protection nodes can be different platforms, such as a 7705 SAR-8 Shelf V2 and a 7705 SAR-18. However, to prevent possible switchover performance issues, avoid mixing different platform types in the same MC-APS group. The 7705 SAR does not enforce configuration consistency between the working circuit and the protection circuit. Additionally, no service or network-specific configuration data is signaled or synchronized between the two routers.

An MC-APS signaling path is established using the IP link between the two routers by matching APS group IDs. A heartbeat protocol can also be used to add robustness. The signaling path verifies that one router is configured as the working circuit and the other is configured as the protection circuit. In case of a mismatch, an incompatible neighbor trap is generated. The protection router uses K1/K2 byte data, member circuit status, and the settings configured for the APS tools commands to select the working circuit. Changes in working circuit status are sent across the MC-APS signaling link from the working router to keep the protection router synchronized. External requests such as lockout, force, and manual switches are allowed only on the node with the protection circuit.

The following figure shows an MC-APS group with physical port, adapter card, and node protection. MC-APS application shows a packet network using MC-APS.

The APS protocol uses the K1 and K2 bytes of the SONET/SDH header to exchange commands and replies between the near end and far end.

The switch priority of a request is assigned by bits 1 through 4 of the K1 byte, as shown in the following table.

In unidirectional mode, the K1 and K2 bytes are not used to coordinate switch action; however, the K1 byte is still used to inform the other end of the local action, and bit 5 of the K2 byte is set to 0 to indicate 1+1 APS mode (see K2 byte functions ).

In bidirectional mode, the highest-priority local request is compared to the remote request (received from the far-end node using an APS command), and whichever request has the greater priority is selected. The requests can be automatically initiated (such as Signal Failure or Signal Degrade), external (such as Lockout, Forced Switch, Request Switch), or state requests (such as Revert-Time timers).

The channels requesting the switch action are assigned by bits 5 through 8. Only channel number codes 0 and 1 are supported on the 7705 SAR. If channel 0 is selected, the condition bits show the received protection channel status. If channel 1 is selected, the condition bits shows the received working channel status.

The K2 byte is used to indicate bridging actions performed at the line termination equipment (LTE), the provisioned architecture, and mode of operation, as shown in the following table.

1+1 APS provides revertive and non-revertive modes; non-revertive mode is the default option. In revertive mode, the activity is switched back to the working port after the working line has recovered from a failure (or the manual switch is cleared). In non-revertive mode, a switch to the protection line is maintained even after the working line has recovered from a failure (or the manual switch is cleared).

To prevent frequent automatic switches that result from intermittent failures, a revert-time is defined for revertive switching. The revert-time is configurable from 0 to 60 min in increments of 1 min; the default value is 5 min. In some scenarios, performance issues can occur if the revert-time is set to 0; therefore, it is recommended that the revert-time always be set to a value of 1 or higher. Any change in the revert-time value takes effect upon the next initiation of the wait-to-restore (WTR) timer. The change does not modify the duration of a WTR timer that has already been started. The WTR timer of a non-revertive switch can be assumed to be infinite.

If both working and protection lines fail, the line that has less-severe errors will be active. If there is signal degradation on both ports, the active port that failed last will stay active. If there is signal failure on both ports, the working port will always be active because signal failure on the protection line is a higher priority than on the working line.

T1/E1 line card redundancy (LCR) uses redundant adapter cards to protect T1/E1 services in case of hardware failures. T1/E1 LCR provides protection against adapter card or node failures. When T1/E1 LCR is used in conjunction with pseudowire redundancy, the network path between the endpoints is also protected. Protection is provided specifically for Cpipe services at the clear channel level and at the channelized level.

When T1/E1 LCR is enabled, the 7705 SAR constantly monitors the health of the adapter cards. If the active (working) adapter card fails (for example, because a card has been removed or due to a bus error), the system proceeds through a predefined sequence of steps to transfer (or switch over) traffic processing to the protection MDA. This switchover is done very quickly to minimize traffic loss. Traffic is moved to the protection adapter card until the fault on the working adapter card is cleared, at which time the traffic may optionally revert to the working adapter card.

T1/E1 LCR is supported on the following cards on the 7705 SAR-8 Shelf V2 and the 7705 SAR-18:

• 16-port T1/E1 ASAP Adapter card

• 32-port T1/E1 ASAP Adapter card

T1/E1 LCR includes support for single-chassis LCR (SC-LCR) and multi-chassis LCR (MC-LCR). In an SC-LCR group, both the working and protection adapter cards must be configured on the same node. In an MC-LCR group, the working adapter card and protection adapter card are configured on two separate nodes, providing protection from node failure in addition to protection from adapter card hardware failure.

In an SC-LCR group, both the working and protection adapter cards are configured with the same LCR group ID on the same node. The working and protection adapter cards are required to be the same type.

SC-LCR is supported for TDM CES (Cpipes). SC-LCR with TDM access is supported on DS1, E1, and DS0 (64 kb/s) channels.

SC-LCR supports TDM SAP-to-SAP connections when both SAPs are configured as LCR SAPs.

SC-LCR also supports TDM SAP-to-spoke SDP connections over an MPLS network. In this configuration, the far-end connection may or may not be configured for LCR.

MC-LCR extends the functionality offered by SC-LCR to include protection against 7705 SAR node failure. With MC-LCR, the working adapter card of an LCR group is configured on one 7705 SAR node while the protection adapter card of the same LCR group is configured on a different 7705 SAR node. The working and protection nodes are connected by an IP link (directly or indirectly) that establishes a multi-chassis protocol (MCP) link between the nodes.

MC-LCR is supported for TDM CES (Cpipes). MC-LCR with TDM access is supported on DS1, E1, and DS0 (64 kb/s) channels.

MC-LCR supports TDM SAP-to-SAP connections when both LCR SAPs are configured using the same adapter card on each node.

MC-LCR also supports TDM SAP-to-spoke SDP connections over an MPLS network. In this configuration, the far-end connection may or may not be configured for LCR.

The working and protection adapter cards must be the same type and must have compatible configurations, such as the same speed, framing, and port type. The adapter cards in an LCR group on both the working and protection nodes must also have the same group ID. The LCR groups can have different descriptions. In order for MC-LCR to function correctly, pseudowire redundancy must be configured on both the working and protection adapter cards. For information about pseudowire redundancy, see the 7705 SAR Services Guide, ‟Pseudowire Redundancy”. MC-LCR with pseudowire redundancy also supports inter-chassis backup (ICB); see MC-LCR and inter-chassis backup for more information.

An MCP link can be established using the IP link between the two nodes by matching LCR group IDs. The signaling path verifies that one node is configured as the working adapter card and the other is configured as the protection adapter card. In case of a mismatch, an incompatible neighbor trap is generated. The protection node uses member adapter card status and the settings configured in the LCR tools commands to select the working adapter card. Changes in working adapter card status are sent across the MC-LCR signaling link from the working node to keep the protection node synchronized. External requests such as lockout and force switch are allowed only on the node with the protection adapter card.

T1/E1 LCR provides revertive and non-revertive modes; non-revertive mode is the default option. In revertive mode, the activity is switched back to the working adapter card after it has recovered from a failure. In non-revertive mode, a switch to the protection adapter card is maintained even after the working adapter card has recovered from a failure.

To prevent frequent automatic switches that result from intermittent failures, a revert-time is defined for revertive switching. The revert-time is configurable from 0 to 60 min in increments of 1 min; the default value is 5 min. In some scenarios, performance issues can occur if the revert-time is set to 0; therefore, it is recommended that the revert-time always be set to a value of 1 or higher. Any change in the revert-time value takes effect upon the next initiation of the wait-to-restore (WTR) timer. The change does not modify the duration of a WTR timer that has already been started. The WTR timer of a non-revertive switch can be assumed to be infinite.

The LCR tools commands can only be executed on the node used in an SC-LCR group or on the protection node of an MC-LCR group. The commands cannot be executed on the working node of an MC-LCR group.

A microwave link allows a 7705 SAR-8 Shelf V2 or 7705 SAR-18 to be connected to a 9500 MPR-e radio node. The MPR-e is the zero-footprint (outdoor) microwave solution offered by Nokia that allows customers to migrate from TDM microwave to pure packet microwave. The following MPR-e radio variants are supported:

• MPT-MC - Microwave Packet Transport, Medium Capacity (ODU)

• MPT-HC V2/9558HC - Microwave Packet Transport, High Capacity Version 2 (ODU)

• MPT-XP - Microwave Packet Transport, High Capacity (very high power version of the MPT-HC V2/9558HC) (ODU)

• MPT-HQAM - Microwave Packet Transport, High Capacity (MPT-HC-QAM) or Extended Power (MPT-XP-QAM) with 512/1024 QAM (ODU)

• MPT-HLC and MPT-HLC plus - Microwave Packet Transport, High-Capacity Long-Haul Cubic (ANSI) (IDU)

A microwave link is configured on a 7705 SAR-8 Shelf V2 or 7705 SAR-18 as a virtual port object (not as a physical port) using the CLI command mw-link-id (for more information about how to configure a microwave link, see Microwave link commands).

The supported microwave link types are 1+0 and 1+1 Hot Standby (HSB). To deploy an N+0 link (with N ≥ 2), multiple links of 1+0 can be configured separately.

A microwave link connection is made from ports 1 through 4 on a Packet Microwave Adapter card to an MPR-e radio using one of the methods described in the 7705 SAR Packet Microwave Adapter Card Installation Guide, ‟Delivering Data to an MPR-e Radio”. The radio can be configured in standalone mode to provide a basic microwave connection as described in Standalone mode or in single network element (single NE) mode to provide the advanced networking capabilities described in Single NE mode. The default configuration is single NE mode.

When connected to an MPR-e radio, these ports, with a microwave link configured, operate as Gigabit Ethernet ports and provide the same features as the other ports (ports 5 through 8), except for the following:

• 802.1x authentication

• active/standby operation on Ethernet access ports configured as LAGs

• hard policing on Ethernet ports

If a microwave link is not configured on ports 1 through 4, they provide all of the same features as the other Gigabit Ethernet ports (ports 5 through 8).

A microwave link from ports 1 through 4 on a Packet Microwave Adapter card to an MPR-e radio that is configured in standalone mode provides a basic microwave connection to the MPR-e radio. In standalone mode, each MPR-e radio that is connected to a 7705 SAR-8 Shelf V2 or 7705 SAR-18 is managed as a separate standalone NE by the MPT Craft Terminal (MCT) Element Manager.

A microwave link from ports 1 through 4 on a Packet Microwave Adapter card to an MPR-e radio that is configured in single NE mode provides the following networking capabilities to the radio over the microwave link:

• Single NE management

• Microwave link fast fault detection

• 1+1 HSB

• 1+1 switching operation

Depending on the type of Gigabit Ethernet microwave link used to connect the Packet Microwave Adapter card and an MPR-e radio, different frequency synchronization mechanisms can be used.

When using optical 1000Base-SX to connect the Packet Microwave Adapter card and an MPR-e radio, synchronous Ethernet and SSM are the frequency synchronization mechanisms that are used. SSM is used as the mechanism to detect a microwave link failure, including loss of frame and MPR-e radio hardware failure.

When using electrical 1000Base-T to connect the Packet Microwave Adapter card and an MPR-e radio, PCR is the frequency synchronization mechanism that is used (a copper SFP is mandatory on ports 3 and 4).

For more information about PCR, synchronous Ethernet, and SSM, see the 7705 SAR Basic System Configuration Guide, ‟Node timing”.

An MPR-e radio that is connected to the 7705 SAR can automatically upload its received signal level (RSL) history file to the 7705 SAR host. The RSL file contains a history of radio attributes and alarms that radio operators can use to isolate and diagnose radio-layer problems that may exist in the network.

Up to 24 MPR-e radios can independently upload their RSL history file every 15 minutes when the rsl-history command is configured on the 7705 SAR for each radio. When uploaded, the file is stored on the 7705 SAR compact flash. Each RSL file can be up to 1 MB and contain up to 10 000 lines. Each time a new file from a specific MPR-e radio is sent to the 7705 SAR, the new file overwrites the previous version for that radio. When uploaded to the 7705 SAR, the operator can view the file in its raw format using the file>type command or FTP it to an external server.

The following table lists the attributes in the RSL history file.