ESM on High Scale QoS IOM
Overview
This section describes High Scale QoS (HSQ) IOM as it applies to ESM. HSQ IOM features a high scale egress traffic manager with an expansive set of QoS capabilities that are particularly suitable for ESM applications. Refer to the 7450 ESS, 7750 SR, 7950 XRS, and VSR Quality of Service Guide for detailed information about HSQ.
Subscriber queues on HSQ in the ESM context are always allocated in a set of eight queues per SLA profile instance (SPI). SPI has a particularly important role on HSQ that determines not only the scale of subscribers but also the subscriber’s QoS modes as it pertains to shaping hierarchy. At this point it suffices to say that SPI is an instantiation of SLA profile and each subscriber host or a PPPoE or IPoE session must be associated with an SPI. An SLA-profile, with its reference to a QoS policy, indirectly defines queuing configuration along with classification for the subscriber host or session. Multiple subscriber hosts or sessions can share the same SPI. The number of SPIs per subscriber determines the SLA profile mode in which ESM on HSQ operates.
HSQ traffic manager overview
One of the principal functionalities of HSQ egress traffic manager are hierarchical rate limiting (with buffering or queuing) and scheduling functions. Rate limiters are implemented by a single or multiple threshold token buckets. The bucket thresholds define traffic burstiness. This implies that packets within each token bucket (rate limiter) are always sent at the full line rate, and only when one of the thresholds is reached (or bucket is filled), the object (queue, scheduling-class, aggregate-shaper) associated with the token bucket is stopped being served. The rate limiter allows traffic bursts (no traffic smoothing function), with a queue at the end of the shaping hierarchy. Because of the ability of the queue to buffer traffic, traffic limiters in this context are referred to as shapers.
The HSQ egress traffic manager supports a seven-tier shaping hierarchy with six levels of scheduling. Six levels of scheduling are realized via six scheduling classes that are served in a strict priority order by the port scheduler. Level six is the highest priority and one is the lowest. Scheduling classes should not be confused with a QoS policy driven forwarding class. Forwarding classes within the system are used between the ingress and egress forwarding complexes and help the system to map a packet to per-hop and per-domain behavior. Scheduling classes are slices of scheduling opportunity within a port scheduling context.
Shaping hierarchy
The following describes the seven tiers of hierarchical shaping on HSQ:
First shaping tier:
This is the rate control tier. At the bottom of the shaping hierarchy is a queue that represents the basic entity from which a packet is extracted and sent out of the port by the port scheduler. At the SPI level, a queue can be attached either to a scheduler class or to a WRR group (but not to both simultaneously). Accordingly, the following shapers are supported at the bottom (first) tier:
Queue shaper — If a queue is directly attached to a scheduler class.
WRR group shaper – If a queue (possibly along with other queues) is attached to one of the two WRR groups at the SPI level. In this case, the queue shaper is ignored.
-
Second shaping tier:
This tier continues to operate at the SPI level. Eight queues form a queue set which is always allocated per SPI and the entire queue set can be shaped to an aggregate rate. In the ESM context, this tier is referred to as the SPI shaping tier. Depending on the SLA mode of operation, this tier can represent a subscriber aggregate rate (in a single SLA mode) or the aggregate rate for a subset of services within a subscriber (in an expanded SLA mode).
-
Third shaping tier:
Multiple SPIs within a single subscriber (supported only in expanded SLA mode) can be attached to a primary shaper which controls subscriber’s aggregate rate. The aggregate rate of the primary shaper is the third shaping tier.
-
Fourth shaping tier:
This tier is enforced within the context of the secondary shaper. The secondary shaper represents an aggregation of subscribers, for example, a group of subscribers that are attached to the same aggregation node (DSLAM or OLT).
Each scheduling class at the secondary shaper can be individually shaped.
-
Fifth shaping tier:
This tier is the aggregate rate of the secondary shaper.
-
Sixth shaping tier:
Sixth and seventh shaping tier are at the port level.
At the port level, scheduling classes can be either:
-
individually shaped
-
collapsed into a single WRR group, in which the collection of scheduling classes (a WRR group) is shaped.
-
These two possibilities (individual or group shaping) are mutually exclusive by configuration, and whichever is chosen represents the sixth shaping tier.
-
-
Seventh shaping tier:
This tier is the aggregate shaping rate of the physical port.
Scheduling
Each port scheduler can be viewed as six separate schedulers, one for each scheduling class. At the port level, there are six strict priorities serviced exhaustively (priority level six is highest).
Scheduling classes at the port level and queues at the SPI level can be mapped to WRR groups. Within the WRR group, each WRR group member (a queue or a scheduling class) is assigned a weight that is used for relative scheduling importance of the queue or scheduler class to other active queues or scheduler classes at the same scheduling service level.
Scheduling opportunity at each tier within the scheduling class is governed by a weighted mechanism. By default, primary shaper members in secondary shaper lists and secondary shaper members in port scheduling class lists use a dynamic weighting function that increases a member’s weight based on the amount of pending work associated with that member. This allows a primary shaper with 10,000 active queue sets (SPIs) to receive proportionately more scheduling opportunities at the secondary shaper level than another primary shaper with only 100 active queue sets (SPIs). The dynamic weight is based on actual activity and not simply based on number of potential (provisioned) membership.
A complete scheduling hierarchy must be maintained between each attached queue at the SPI level and its port priority. In other words, the HSQ has no provision to bypass the primary and secondary shaping and scheduling levels. Entities at the SPI level must be associated with a primary shaper and each primary shaper must be associated with a secondary shaper. All queues in the same queue set are mapped to the same primary shaper. All scheduling classes in the same primary shaper are mapped to the same secondary shaper. A default secondary shaper context is created for each port scheduler and a default primary shaper (this is called a managed bypass primary shaper) context is created for each secondary shaper context. Through normal provisioning events, other non-default secondary shapers and primary shapers can also be created.
HSQ and ESM SLA modes
HSQ supports two SLA modes for ESM, expanded and single. The SLA mode is configurable per subscriber profile.
In single mode, the subscriber aggregate rate and the SLA profile queuing are both performed using a single primary shaper context for the subscriber. This mode provides the highest subscriber scale.
In expanded mode, multiple SLA profile instances are supported by moving the subscriber aggregate shaping function to a primary shaper specifically created for the subscriber instance. The subscriber scale in this mode depends on the number of available primary shapers.
Both modes support routed and bridged homes (multiple subscriber hosts can be associated with a single SPI). However, multiple SPIs per subscribers support richer QoS granularity for bridged homes where a set of hosts using the same service can have their own queues and aggregate rate, separate from other services within the same home.
Single SPI mode and Expanded SLA mode display two ESM configurations that are designed to emphasize the distinction between the two SLA modes. These are examples to explore SLA modes with an arbitrary queue and WRR group assignments. For example, in both examples, the first four queues within the queue set (SPI) are assigned to two WRR groups at the SPI level, while scheduling classes three and four at the port level are collapsed into a single WRR group. Other configurations for queue, WRR group and scheduling classes are possible.
Both examples contain common parts generic to HSQ, independent of the SLA modes:
Some of the queues in a queue set (SPI) are directly assigned to scheduling classes, and some are assigned indirectly through two WRR groups that are supported per each SPI on HSQ. Note that configuring WRR groups on HSQ is optional.
-
On HSQ, only one queue or one WRR group can be mapped to a specific scheduling class. Queues either not mapped to a scheduling class or mapped to a WRR group that is not mapped to a scheduling class drops all frames incrementing the queue’s discard counters.
-
A queue’s WRR weight is only used when the queue is attached to a WRR group. The WRR weight specifies the relative scheduling importance of the queue relative to other active queues on the same group. A queue’s class attachment weight (CW) is only used when the queue is directly attached to a scheduler class. The CW of a WRR group specifies the relative scheduling importance of a queue or WRR group at the scheduling class to queues or WRR groups belonging to other SPIs. Each individual queue or a WRR group can be shaped to a configurable rate.
-
The secondary shaper in ESM represents an access node. A subscriber is associated with the secondary shaper by an intermediate-destination-id string which is supplied by RADIUS, LUDB, or Python during the authentication phase.
-
At the port level, multiple scheduling classes can be collapsed through a WRR group (only one WRR group is supported on the port level) and, in this example, scheduling classes three and four are collapsed in WRR group one, with a single scheduling priority that is numerically between the scheduling priorities two and five. Essentially, the WRR group collapses the member scheduling priorities into a single scheduling priority using each class’s WRR weight to manage scheduling opportunities between the classes. Because the member classes default priority levels are unused, the system simply maps the WRR group to one of the unused priorities (in this example, scheduling classes three or four). Because the unused priority levels are contiguous, and the remainder is unused, the priority chosen has no bearing on scheduler performance.
The following sections explore the two SLA modes focusing on the assignment of SPIs and the use of primary shapers as principal differentiators between the two SLA modes. Be aware that primary shapers on HSQ are not exposed as customer-defined objects and are explicitly created only when intermediate shaping is required by specific application (for example, as required by expanded SLA mode).
ESM single SLA mode
An example for single SLA mode is shown in Single SPI mode. In this mode, only a single SPI can be assigned to the subscriber. On the QoS level, the SPI represents a subscriber with the aggregate rate that is configured in the subscriber profile. In this mode of operation, the aggregate rate in the SLA profile should not be configured. If the aggregate rate is configured, the minimum of the rate configured in the subscriber profile and SLA profile are in effect for the SPI.
In single SLA mode, primary shapers have no functional role. However, because HSQ mandates the presence of the primary shaper in the hierarchy, the queue set under the SPI must be associated with a primary shaper construct. In such case, this enforced and futile primary shaper is implicitly created by the system. The number of primary shapers on an HSQ is finite, and consequently the number of implicitly created primary shapers should be tracked by the operator to ensure that the scaling limit of primary shapers in expanded SLA mode is not violated.
The system automatically creates the following primary shapers:
one default primary shaper per physical port
one managed bypass primary shaper per secondary shaper in single SLA mode
By default, the aggregate shaping rate of implicitly-created primary shapers is set to max which means that no rate limiting is performed at that level.
ESM expanded SLA mode
In expanded SLA mode, the primary shaper is used to limit the subscriber’s aggregate rate. Multiple SPIs can be associated with a subscriber and each SPI can be individually rate limited. An example of this scenario is shown in Expanded SLA mode.
In this example, two SPIs are associated with the subscriber (only the details of the SPI1 for subscriber 1 are shown in Expanded SLA mode). Each SPI is shaped to an aggregate rate while the aggregate rate of the subscriber is shaped by the aggregate rate of the primary shaper. This scaling configuration is limited by the number of available primary shapers per line card. See the scaling figures in the SR OS Scaling Guides.
The access to the aggregate rate of the primary shaper through a subscriber profile in expanded SLA mode is an important distinction between the two modes of operation. In single SLA mode aggregate rate of the primary shaper is not accessible and is set to max (no rate limiting function is performed by the primary shaper).
Configuration steps
There are QoS configurations common to both SLA modes. The configuration difference is rooted in subscriber management-specific commands where the SLA mode and the aggregate rate of the primary shaper are configured.
The following are the common configuration QoS blocks for both SLA modes:
A hs-scheduler-policy defines parameters at the port level:
-
the maximum rate of the port scheduler
-
the maximum rate of the WRR group (single WRR group is supported at this level)
-
the mapping of scheduling classes to WRR group 1 and their weights within this group.
-
the shaping rate of each scheduling class that is not mapped to a WRR group 1.
*A:BNG>config>qos> hs-scheduler-policy ‟hs1” create max-rate 10000 group 1 rate 3000 scheduling-class 1 rate 1000 scheduling-class 2 rate 1000 scheduling-class 3 group 1 weight 1 scheduling-class 4 group 1 weight 1 scheduling-class 5 rate 1000 scheduling-class 6 rate 1000 After it is defined, the hs-scheduler-policy is applied under the port. There is a single HSQ port scheduler per port.
*A:BNG>config> port 1/1/1 ethernet egress hs-secondary-shaper ‟1” create aggregate rate 1000000 exit class 1 rate 10000 exit class 2 rate 10000 exit class 3 rate 10000 exit class 4 rate 10000 exit class 5 rate 10000 exit class 6 rate 10000 exit
In this example, the msap-policy defines the default inter-dest-id to be the top qtag on the subscriber SAP. All subscriber hosts on a SAP with top qtag ‛1’ are mapped to this secondary shaper.
*A:BNG>config>
subscr-mgmt
msap-policy ‟msaps” create
sub-sla-mgmt
def-inter-dest-id use-top-q
exit
An HSQ attachment policy defines how the eight individual queues within the SPI attach to either a scheduling class or to one of two WRR groups local to the SPI.
The HSQ attachment policy also defines the maximum scheduling class (one through six) that uses the SPI aggregate shaper’s low burst tolerance threshold. If a value of three is configured, classes one through three is associated with the low burst tolerance threshold while classes four through six are associated with the high burst threshold. In this example, if the SPI aggregate shaper becomes non-conforming, classes one through three shuts off first (associated queues removed from scheduler). This allows classes four to six to keep being served without buffering and therefore avoiding packet delays. If the rate of classes four through six is enough to cause the high burst tolerance threshold to be crossed, the remaining classes are also shutoff.
*A:BNG>config>qos hs-attachment-policy ‟attach-1” create low-burst-max-class 3 queue 1 wrr-group 1 queue 2 wrr-group 1 queue 3 wrr-group 2 queue 4 wrr-group 2 queue 5 sched-class 3 queue 6 sched-class 4 queue 7 sched-class 5 queue 8 sched-class 6 wrr-group 1 sched-class 1 wrr-group 2 sched-class 2The HSQ attachment policy described in the previous step is referenced in the SAP egress QoS policy that is associated with the subscriber host or session.
The principal functionality provided by the SAP egress QoS policy is the following:
Forwarding class mappings that associate a forwarding class to one of the eight available egress SPI queues. A queue can service one, several, or all eight forwarding classes. A queue without a forwarding class mapping receives no user traffic.
WRR weight and class attachment weight parameters for each queue. A queue’s WRR weight is only used when the queue is attached to a WRR group. In this configuration, the queues’ WRR weight specifies the relative scheduling importance of the queue relative to other active queues in the same queue set. A queue’s class attachment weight is only used when the queue is directly attached to a scheduler class.
Class attachment weights for the WRR groups. The class attachment weight specifies the relative scheduling importance of a queue or WRR group at the primary shaper compared to queues or WRR groups belonging to other SPIs.
Queue rate. The following is an example of queue rate configurations.
*A:BNG>config>qos> sap-egress 10 create ---------------------------------------------- hs-attachment-policy "attach1" queue 1 create hs-wrr-weight 5 exit queue 2 create hs-wrr-weight 10 exit queue 3 create hs-wrr-weight 15 exit queue 4 create hs-wrr-weight 20 exit queue 5 create hs-class-weight 2 rate 20000 exit queue 6 create hs-class-weight 3 rate 20000 exit queue 7 create hs-class-weight 4 rate 20000 exit queue 8 create hs-class-weight 5 rate 20000 exit hs-wrr-group 1 rate 10000 hs-class-weight 8 exit hs-wrr-group 2 rate 20000 hs-class-weight 2 exit fc af create queue 3 exit fc be create queue 1 exit fc ef create queue 7 exit fc h1 create queue 5 exit fc h2 create queue 6 exit fc l1 create queue 4 exit fc l2 create queue 2 exit fc nc create queue 8 exit
The following commands are specific to ESM:
SLA mode as configured in the subscriber profile:
*A:BNG>config>subscr-mgmt
sub-profile ‟hs-sub” create
hs-sla-mode {expanded | single}
Single SLA mode:
In single SLA mode, only the sub-profile hs-aggregate-rate-limit should be configured:
*A:BNG>config>subscr-mgmt
sub-profile ‟hs-sub-single” create
hs-sla-mode single
egress
hs-agg-rate-limit 20000
exit
*A:BNG>config>subscr-mgmt
sla-profile ‟hs-single” create
ingress
qos 5
exit
exit
egress
qos 5
exit
exit
When both, subscriber profile egress hs-agg-rate-limit and SLA profile hs-agg-rate-limit are configured, the system uses the minimum value to program the hs-agg-rate of the SPI shaper.
Expanded SLA mode:
In expanded mode, the hs-agg-rate-limit in the SLA profile determines the aggregate shaping rate of each SPI associated with the subscriber:
*A:BNG>config>subscr-mgmt>
sla-profile ‟hs-expanded” create
ingress
qos 5
exit
exit
egress
qos 5
exit
hs-agg-rate-limit 1000
exit
The aggregate rate of the subscriber (primary shaper) is configured in the subscriber profile:
*A:BNG>config>subscr-mgmt>
sub-profile ‟hs-sub-expanded” create
hs-sla-mode expanded
egress
hs-agg-rate-limit 20000
exit
HSQ overrides:
*A:DUAL_HOME_2>config>subscr-mgmt>sla-prof# info
----------------------------------------------
egress
qos 2
queue 1
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
queue 2
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
queue 3
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
queue 4
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
queue 5
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
queue 6
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
queue 7
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
queue 8
rate 100
mbs 1000 kilobytes
hs-class-weight 8
hs-wred-queue-policy "_tmnx_hs_default"
hs-wrr-weight 20
exit
hs-wrr-group 1
hs-class-weight 2
rate 100
exit
hs-wrr-group 2
hs-class-weight 4
rate 100
exit
exit
exit
----------------------------------------------
Deployment considerations
For general restrictions for HSQ, see the HSQ Restrictions section in the 7450 ESS, 7750 SR, 7950 XRS, and VSR Quality of Service Guide
Deploy ESM on HSQ with the following considerations:
An on-line subscriber profile change (using CoA, RAR, re-authentication, or the tools perform command) for subscribers is allowed only for subscriber profiles with the same hs-sla-modes. In other words, an on-line change of the SLA mode for a subscriber on HSQ is not supported.
Change of a SLA mode within the subscriber profile is allowed only for subscriber profiles that are not associated with subscribers.
An on-line change of a SLA profile for a subscriber in single SLA mode with host-shared filters is not supported.
An on-line change of a SLA profile for subscribers in single SLA mode with more than one non-session based hosts or more than one session, is not supported. When a single subscriber session with multiple hosts within the session where SLA change is allowed, all the hosts within the session are associated with the new SLA.
An on-line change of an SLA profile for subscribers in single SLA mode with any static/l2 host is not supported.
A single SLA mode supports a single SPI and a single SAP. Any attempt to create an additional SPI, or to add an SPI with a different SAP for the subscriber is rejected.
Shared queueing on HSQ ingress is not supported and consequently the default shared-queuing setting in MSAP policy must be explicitly disabled.
To achieve maximum scaling on ingress, it is recommended that policing is deployed on ingress. This implies that all FCs on ingress are explicitly mapped to ingress policers within the QoS policy and that profiled-traffic-only command in the sap>sub-sla-mgmt or msap>sub-sla-mgmt context under is enabled.