High Scale QoS IOM in ESM Context: Expanded SLA Mode
This chapter describes the High Scale QoS (HSQ) IOM in the Enhanced Subscriber Management (ESM) context
Topics in this chapter include:
Applicability
This chapter is applicable to the 7750 SR-7/12/12e platforms and describes the High Scale QoS (HSQ) IOM in the Enhanced Subscriber Management (ESM) context. The information and configuration in this chapter are based on SR OS Release 15.0.R4.
Overview
This chapter describes the QoS operation and configuration of the High Scale QoS IOM (HSQ), with a focus on the expanded SLA mode in the Enhanced Subscriber Management (ESM) context.
This chapter can be considered as an extension of the High Scale QoS IOM in ESM Context: Single SLA Mode chapter and it is recommended that the High Scale QoS IOM in ESM Context: Single SLA Mode chapter be read first. Generic concepts and configuration that is common to both chapters will not be repeated in this chapter. For example, high-level HSQ description or generic ESM configuration (as in Appendix A in the High Scale QoS IOM in ESM Context: Single SLA Mode chapter) will not be repeated here. However, since QoS is a centerpiece of this chapter, the QoS-specific configuration will be provided, even for the parts that may overlap between the two chapters (for example, HS attachment policy configuration is the same in both chapters and it will be repeated here).
Expanded SLA mode on HSQ is the default mode of operation in the ESM context and in contrast to single SLA mode, it allows multiple SLA profile instances (SPIs) per subscriber. That is, expanded SLA mode allows multiple HSQ queue groups per subscriber. Multiple subscriber hosts\sessions are supported per each SPI and HSQ queue group.
While expanded SLA mode provides better QoS flexibility at the subscriber level than single SLA mode, its scale becomes restricted by the number of HS primary shapers on the HSQ IOM. In Expanded SLA Mode, one HS primary shaper is allocated per subscriber to limit the subscriber aggregate rate (which can now consist of multiple HSQ queue groups). Since the HSQ IOM supports 16k HS primary shapers, the scale of the subscribers on HSQ IOM becomes limited by the same number. These two factors (greater QoS flexibility at the subscriber level and reduced subscriber scale) are the two main differences between the two SLA modes of operation.
Some small number of HS primary shapers are used internally by the system, so are not available to subscribers.
The SLA mode of operation is enabled per subscriber and it cannot be changed online (while the subscriber is established). The reason why expanded SLA mode is the default mode, even though it reduces the subscriber scale, is that this mode adheres to the existing ESM principles in SR OS where the number of SPIs per subscriber is not restricted.
Like the Single SLA Mode chapter, this chapter is also conceptually divided into two parts:
The focus of the first part is on the (egress) QoS configuration for two subscribers, named ‟sub-1” and ‟sub-2”. Each subscriber has two hosts and consequently two SPIs and HSQ queue groups. Subscriber hosts will be referred to as:
Host 1-1: first host of sub-1
Host 1-2: second host of sub-1
Host 2-1: first host of sub-2
Host 2-2: second host of sub-2
Note:Subscriber sessions are also supported. This example is, however, based on subscriber hosts while the subscriber session concept is left disabled.
The second part focuses on examining HSQ traffic management capabilities in the ESM context by observing output traffic patterns in a congested system.
Topics related to the generic operation of HSQ IOM and ESM that are not directly described in this chapter are included in the following:
Configuring single SLA mode in ESM is in the chapter High Scale QoS IOM in ESM Context: Single SLA Mode.
Configuring HSQ in the service context is in the chapter High Scale QoS IOM: QoS, Service, and Network Configuration.
Generic ESM concepts are described in the 7450 ESS, 7750 SR, and VSR Triple Play Service Delivery Architecture Guide and in numerous chapters in ACG part III – TPSDA.
Test Environment
Test Environment Example shows the test environment example with two subscribers in expanded SLA mode, each with two DHCPv4 hosts set up in the BNG. DHCPv4 control traffic (simulated subscriber hosts) is initiated from port 103/1 on a traffic generator. Subscribers are authenticated via local user database (LUDB) and instantiated on managed service access points (MSAPs) in the IES 3 service. Subscriber IP addresses are assigned statically via LUDB.
To explore traffic management capabilities under congestion, a number of traffic streams are generated in the downstream direction, from port 203/1 on the traffic generator toward the subscriber hosts on port 103/1.
The QoS hierarchy associated with the subscribers is shown in QoS Hierarchy in Expanded SLA Mode. At a high level, this hierarchy can be described as follows:
Each of the two subscribers (sub-1 and ‟sub-2”) has two hosts, each of which is associated with an SPI and HSQ queue group. HSQ queue group is always comprised of eight queues. However, not all the queues are required to be used by the subscriber. In this example, only six queues are used by each subscriber while the two remaining queues, although allocated to the subscriber, remain unused.
Note:Subscriber sessions are also supported. This example is, however, based on subscriber hosts while the subscriber session concept is left disabled.
All four hosts use the same HS attachment policy template. This means that the mapping between the queues, scheduling classes (SC), and WRR groups is the same for all four hosts.
Queues 1 and 2 are attached to WRR group 1 at the local HSQ queue group level and served in 4:1 ratio. WRR group 1 is attached to the lowest scheduling class 1, whereas queues 3, 4, 5, and 6 are directly attached to scheduling classes 3, 4, 5, and 6, respectively.
Each HSQ queue group is rate limited at the aggregate level. In this example, there is only one host associated with each HSQ queue group. However, multiple hosts per SPI and HSQ queue group can be configured.
Each subscriber is rate limited at the aggregate level.
Subscribers are mapped to the same HS secondary shapers.
Mapping between the subscribers and their HS secondary shapers is achieved via outer VLANs. Sub-1 and sub-2 on SAPs with the outer VLAN 1 are mapped to the HS secondary shaper 1.
The HS secondary shaper is associated with the HS port scheduler that has its own aggregate rate limit set. Because there is only one HS secondary shaper configured per port in this example, the smaller configured aggregate rate limit of the two (HS secondary shaper and the port shaper) will affect traffic going out this port.
At the port level, the scheduling class 6 is rate limited while scheduling classes 4 and 5 are collapsed into a WRR group, which is rate limited.
Configuration
The configuration section is split into two parts:
ESM-specific configuration as it relates to HSQ
QoS-specific configuration as it relates to HSQ
ESM-specific Configuration
For repeating parts of common ESM configuration, see the High Scale QoS IOM in ESM Context: Single SLA Mode chapter.
Each of the two subscribers (sub-1 and sub-2) have their own sub-profile, which will determine the HS SLA mode in which the subscriber operates, as well as the aggregate rate limit in kb/s for each subscriber. The aggregate rate of the subscriber is configured via its HS primary shaper. The hs-agg-rate-limit command in sub-profile is the only command that is applicable to HS primary shaper. Since the Expanded SLA Mode is the default mode, its configuration is not shown in the CLI unless requested by the detail keyword:
configure
subscriber-mgmt
sub-profile "sub-prof-1" create
egress
hs-agg-rate-limit 100000
exit
exit
sub-profile "sub-prof-2" create
egress
hs-agg-rate-limit 150000
exit
exit
With info detail or admin display-config detail commands, the Expanded SLA Mode configuration is shown (irrelevant parts are removed from the output):
subscriber-mgmt
sub-profile "sub-prof-1" create
hs-sla-mode expanded
egress
hs-agg-rate-limit 100000
exit
Each subscriber host has its own SPI that references the QoS SAP policy on egress (in this case, QoS SAP egress policies 10, 20, 30, and 40). Each SLA profile defines the aggregate rate in kb/s that will be applied to the corresponding HSQ queue group, and in this case to each subscriber host.
The egress QoS SAP policy defines the QoS characteristics at the subscriber host level (or HSQ queue group level), such as:
traffic classification
queue/WRR rates and weights
mapping between the queues, WRR groups, and scheduling classes by referencing the HS attachment policy.
configure
subscriber-mgmt
sla-profile "sla-1-1" create
egress
qos 10
exit
hs-agg-rate-limit 50000
exit
exit
sla-profile "sla-1-2" create
egress
qos 20
exit
hs-agg-rate-limit 70000
exit
exit
sla-profile "sla-2-1" create
egress
qos 30
exit
hs-agg-rate-limit 80000
exit
exit
sla-profile "sla-2-2" create
egress
qos 40
exit
hs-agg-rate-limit 100000
exit
exit
exit
For detailed descriptions of the configuration options in MSAP policy, see the High Scale QoS IOM in ESM Context: Single SLA Mode chapter, because this configuration is shared:
configure
subscriber-mgmt
msap-policy "msaps" create
sub-sla-mgmt
def-inter-dest-id use-top-q
sub-ident-policy "sub_ident_pol"
no multi-sub-sap
single-sub-parameters
profiled-traffic-only
exit
exit
ies-vprn-only-sap-parameters
ingress
qos 1 service-queuing
exit
exit
exit
QoS-specific configuration
At the subscriber level (or HSQ queue group level), the HS attachment policy defines mapping between the queues, WRR groups, and scheduling classes. In this example, queues 1 and 2 are mapped to a WRR group 1, which is mapped into scheduling class 1. Queues 3 to 6 are mapped to respective scheduling classes (SC 3 to 6). Queues 7 and 8, although allocated, are unattached, so will drop any traffic that they receive. A maximum of two WRR groups are supported at the HSQ queue group level and in this example, only WRR 1 is used.
The low burst max class parameter is set to 3, which ensures that the three lower scheduling classes (1 to 3) are stopped being served first if HSQ queue group aggregate congestion occurs. The HSQ queue group aggregate shaper has two burst tolerance thresholds: when the first threshold is reached, scheduling classes 1 to 3 will be removed from the service list, followed by the higher-level scheduling classes (4 to 6) being removed when the second burst tolerance threshold is reached. This designation of scheduling classes in two tiers ensures lower latency for traffic associated with higher scheduling classes during short-lived congestion periods.
The following defined HS attachment policy is applied to the subscriber hosts through the SAP egress policy referenced in the SLA profile. In this example, all four subscriber hosts use the same HS attachment policy:
configure
qos
hs-attachment-policy "hs-attach-1-1" create
low-burst-max-class 3
queue 1 wrr-group 1
queue 2 wrr-group 1
queue 3 sched-class 3
queue 4 sched-class 4
queue 5 sched-class 5
queue 6 sched-class 6
queue 7 unattached
queue 8 unattached
wrr-group 1 sched-class 1
wrr-group 2 unattached
exit
exit
Besides referencing the HS attachment policy, the SAP egress policy defines traffic classification, as well as characteristics of queues and WRR groups at the subscriber level. In this example, the SAP egress policy 10 is associated with the host 1 of subscriber sub-1. The queues 1 and 4 are in the context of WRR group 1 serviced in the ratio 1:4.
Although the SAP egress policy syntax implies that such policy is applied per SAP, in the ESM context, this policy is instantiated via the SLA profile; therefore. the queue/policer instantiations are performed per SPI and not per SAP.
The aggregate rate of WRR 1 is set to 20Mb/s. Mapping of forwarding classes to queues is self-explanatory. Separate SAP egress policies (20, 30, and 40) are applied to the remaining hosts (host 1-2, host 2-1, and host 2-2). The only difference between the QoS SAP egress policies for the subscriber hosts is the rate of the WRR group 1, which is 40Mb/s for host 1-2, 60Mb/s for host 2-1, and 80Mb/s for host 2-2.
The SAP egress policy is applied in the SLA profile for the subscriber. The WRR group rate (along with other QoS parameters) can be dynamically overridden via RADIUS/Diameter during authentication or while the host/session is online. This functionality would allow having one SAP egress policy configured where parameters can be dynamically overridden, when the policy is applied to the subscriber host or session.
configure
qos
sap-egress 10 create
hs-attachment-policy "hs-attach-1-1"
queue 1 create
hs-wrr-weight 1
exit
queue 2 create
hs-wrr-weight 4
exit
queue 3 create
exit
queue 4 create
exit
queue 5 create
exit
queue 6 create
exit
hs-wrr-group 1
rate 20000
exit
fc af create
queue 3
exit
fc be create
queue 1
exit
fc h1 create
queue 5
exit
fc h2 create
queue 6
exit
fc l1 create
queue 4
exit
fc l2 create
queue 2
exit
dscp af12 fc "af"
dscp be fc "be"
dscp af22 fc "h1"
dscp ef fc "h2"
dscp af21 fc "l1"
dscp af11 fc "l2"
exit
It was already shown in the ESM-specific Configuration section that the HSQ queue group aggregate rate limit is configured in the SLA profile while the aggregate rate of the subscriber is managed by the HS primary shaper and is configured in the sub-profile.
The next hierarchy level in the chain (up from the subscriber level) is performed by the HS secondary shapers that are directly configured in the egress context of the port. In this example, there is only one configured HS secondary shaper that corresponds to the outer VLAN on the subscriber SAPs. The network representation of the HS secondary shaper is an access node downstream from the BNG.
The HS secondary shaper ‟1” is configured with the aggregate rate of 250Mb/s. The rates are configured in kb/s. Similarly, as at the subscriber level, the low-burst-max-class 3 command maps all scheduling classes at or below level 3 to the low burst tolerance threshold, while all scheduling classes above level 3 are mapped to the high burst tolerance threshold at the HS secondary shaper level. Therefore, in case of short congestion periods, objects associated with scheduling classes 3 and below will be removed from the service list before the objects associated with scheduling classes 4 and above.
configure
port 3/1/1
ethernet
mode access
encap-type qinq
egress
hs-scheduler-policy "hs1"
hs-secondary-shaper "1" create
aggregate
rate 250000
low-burst-max-class 3
exit
exit
exit
exit
no shutdown
exit
The last configuration block in the scheduling hierarchy is an HS port scheduler, which is associated with the port via the command hs-scheduler-policy ‟hs1” in the preceding CLI code.
The HS port scheduler characteristics in this example are defined as follows:
The maximum rate is set to 300Mb/s. However, in this example, this rate limit has no effect on traffic because it is higher than the 250Mb/s rate configured on HS secondary shaper 1. Because HS secondary shaper 1 is the only one configured under the port, the lower rate of this HS secondary shaper will limit traffic before it reaches the 300Mb/s limit configured on the port. More realistic deployment scenarios would contain multiple secondary shapers per port.
Scheduling classes 4 and 5 are collapsed into a single scheduling priority (5) and they are served in a 1:2 ratio. This is performed via WRR group 1.
This collapsing of scheduling classes 4 and 5 occurs at the port level, whereas scheduling classes 1 and 2 are collapsed at the subscriber level (HSQ queue group level).
The WRR 1 at the port level is rate limited to 60Mb/s.
The highest priority (6) scheduling class is rate limited to 20Mb/s at the port level.
All the rates under the HS port scheduler are configured in Mb/s.
configure
qos
hs-scheduler-policy "hs1" create
max-rate 200
group 1 rate 60
scheduling-class 4 group 1 weight 10
scheduling-class 5 group 1 weight 20
scheduling-class 6 rate 20
exit
exit
The delta between the low and high burst tolerance thresholds at the subscriber and HS secondary shaper levels can be adjusted with the hs-fixed-high-thresh-delta command that is configured at the card level. In this example, the difference between the thresholds is set to 12kbytes.
configure
card 3
card-type iom4-e-hs
mda 1
mda-type me10-10gb-sfp+
no shutdown
exit
fp 1
egress
hs-fixed-high-thresh-delta 12000
exit
exit
no shutdown
Operational Commands
Operational commands are used to troubleshoot the system and monitor its operational state. The focus of this section will be on show, clear, and tools dump commands. The debug commands are omitted because there are no debug commands related to QoS on HSQ IOM and the debug commands related to ESM are described in other chapters. Also, there are no log events related to QoS on HSQ IOM.
Show Commands
show commands in this section are divided into two groups:
show commands that display association between ESM and QoS objects where all displayed information is static and it does not change autonomously over time.
show commands that display QoS hierarchy with the running rates (where the state is changing autonomously).
show commands have filters (or CLI parameters) that can be used to control the amount of the output information. This section will provide a few show commands; it is left to the user to explore all the options available for a particular show command.
Subscriber Management Related Show Commands
Examining subscriber associations with QoS objects should begin with the show service active-subscribers command. The output of this command provides information about the subscriber context and the output can vary in detail depending on the options with which this command is run. Besides the subscriber name, underlying subscriber SAPs, and SLA/sub-profile names, the following information is provided:
SLA mode in which the subscriber operates on HSQ (single versus expanded SLA mode)
Aggregate rate of the subscriber
Ingress and egress QoS policies
Subscriber association with an HS secondary shaper and the inter dest id string that is used to make this association
Ingress queue/policer statistics
Egress queues statistics
For brevity, only the information for subscriber sub-1 (both hosts) is shown in the following output.
*A:PE-1# show service active-subscribers subscriber "sub-1" detail
===============================================================================
Active Subscribers
===============================================================================
-------------------------------------------------------------------------------
Subscriber sub-1 (sub-prof-1)
-------------------------------------------------------------------------------
I. Policer Ctrl. : N/A
E. Policer Ctrl. : N/A
I. vport-hashing : Disabled
I. sec-sh-hashing: Disabled
Q Frame-Based Ac*: Disabled
Acct. Policy : N/A Collect Stats : Disabled
ANCP Pol. : N/A
Accu-stats-pol : (Not Specified)
HostTrk Pol. : N/A
IGMP Policy : N/A
MLD Policy : N/A
PIM Policy : N/A
Sub. MCAC Policy : N/A
NAT Policy : N/A
Firewall Policy : N/A
UPnP Policy : N/A
NAT Prefix List : N/A
Def. Encap Offset: none Encap Offset Mode: none
Avg Frame Size : N/A
Vol stats type : full
Preference : 5
LAG hash class : 1
LAG hash weight : 1
Sub. ANCP-String : "sub-1"
Sub. Int Dest Id : "1"
Igmp Rate Adj : N/A
RADIUS Rate-Limit: N/A
Oper-Rate-Limit : 100000
-------------------------------------------------------------------------------
Radius Accounting
-------------------------------------------------------------------------------
Policy : N/A
Session Opti.Stop: False
-------------------------------------------------------------------------------
HS
-------------------------------------------------------------------------------
SLA-mode : expanded E Agg Rate Limit : 100000
Hs Second Shaper : "1"
* indicates that the corresponding row element may have been truncated.
-------------------------------------------------------------------------------
(1) SLA Profile Instance
- sap:[3/1/1:1.1] (IES 3 - group-int-1)
- sla:sla-1-1
-------------------------------------------------------------------------------
Description : (Not Specified)
Host Limits : No Limit
Egr Sched-Policy : N/A
Ingress Qos-Policy : 1 Egress Qos-Policy : 10
Ingress Queuing Type : Service-queuing (Not Applicable to Policer)
Ingr IP Fltr-Id : N/A Egr IP Fltr-Id : N/A
Ingr IPv6 Fltr-Id : N/A Egr IPv6 Fltr-Id : N/A
Ingress Report-Rate : Maximum
Egress Report-Rate : Maximum
Egress Remarking : from Sap Qos
Credit Control Pol. : N/A
Category Map : (Not Specified)
Use ing L2TP DSCP : false
Hs-Agg-Rate-Limit : 50000
Egress HS Q stat mode: no-override
Hs-Oper-Rate-Limit : 50000
Egr hqos mgmt status : disabled
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
IP Address
MAC Address Session Origin Svc Fwd
-------------------------------------------------------------------------------
10.10.1.1
00:00:64:01:01:01 N/A DHCP 3 Y
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
SLA Profile Instance statistics
-------------------------------------------------------------------------------
Packets Octets
Off. HiPrio : 0 0
Off. LowPrio : 0 0
Off. Uncolor : 0 0
Off. Managed : 0 0
Queueing Stats (Ingress QoS Policy 1)
Dro. HiPrio : 0 0
Dro. LowPrio : 0 0
For. InProf : 0 0
For. OutProf : 0 0
Queueing Stats (Egress QoS Policy 10)
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 89016218 89016218000
For. In/InplusProf : 0 0
For. Out/ExcProf : 74180626 74180626000
-------------------------------------------------------------------------------
SLA Profile Instance per Queue statistics
-------------------------------------------------------------------------------
Packets Octets
Ingress Queue 1 (Unicast) (Priority)
Off. HiPrio : 0 0
Off. LowPrio : 0 0
Dro. HiPrio : 0 0
Dro. LowPrio : 0 0
For. InProf : 0 0
For. OutProf : 0 0
Egress Queue 1
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 34006842 34006842000
For. In/InplusProf : 0 0
For. Out/ExcProf : 3083545 3083545000
Egress Queue 2
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 24756213 24756213000
For. In/InplusProf : 0 0
For. Out/ExcProf : 12334174 12334174000
Egress Queue 3
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 0 0
For. In/InplusProf : 0 0
For. Out/ExcProf : 29672356 29672356000
Egress Queue 4
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 14981296 14981296000
For. In/InplusProf : 0 0
For. Out/ExcProf : 7272638 7272638000
Egress Queue 5
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 7708660 7708660000
For. In/InplusProf : 0 0
For. Out/ExcProf : 14545275 14545275000
Egress Queue 6
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 7563207 7563207000
For. In/InplusProf : 0 0
For. Out/ExcProf : 7272638 7272638000
-------------------------------------------------------------------------------
(2) SLA Profile Instance
- sap:[3/1/1:1.1] (IES 3 - group-int-1)
- sla:sla-1-2
-------------------------------------------------------------------------------
Description : (Not Specified)
Host Limits : No Limit
Egr Sched-Policy : N/A
Ingress Qos-Policy : 1 Egress Qos-Policy : 20
Ingress Queuing Type : Service-queuing (Not Applicable to Policer)
Ingr IP Fltr-Id : N/A Egr IP Fltr-Id : N/A
Ingr IPv6 Fltr-Id : N/A Egr IPv6 Fltr-Id : N/A
Ingress Report-Rate : Maximum
Egress Report-Rate : Maximum
Egress Remarking : from Sap Qos
Credit Control Pol. : N/A
Category Map : (Not Specified)
Use ing L2TP DSCP : false
Hs-Agg-Rate-Limit : 70000
Egress HS Q stat mode: no-override
Hs-Oper-Rate-Limit : 70000
Egr hqos mgmt status : disabled
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
IP Address
MAC Address Session Origin Svc Fwd
-------------------------------------------------------------------------------
10.10.1.2
00:00:64:01:01:02 N/A DHCP 3 Y
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
SLA Profile Instance statistics
-------------------------------------------------------------------------------
Packets Octets
Off. HiPrio : 0 0
Off. LowPrio : 0 0
Off. Uncolor : 0 0
Off. Managed : 0 0
Queueing Stats (Ingress QoS Policy 1)
Dro. HiPrio : 0 0
Dro. LowPrio : 0 0
For. InProf : 0 0
For. OutProf : 0 0
Queueing Stats (Egress QoS Policy 20)
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 118692005 118692005000
For. In/InplusProf : 0 0
For. Out/ExcProf : 74182782 74182782000
-------------------------------------------------------------------------------
SLA Profile Instance per Queue statistics
-------------------------------------------------------------------------------
Packets Octets
Ingress Queue 1 (Unicast) (Priority)
Off. HiPrio : 0 0
Off. LowPrio : 0 0
Dro. HiPrio : 0 0
Dro. LowPrio : 0 0
For. InProf : 0 0
For. OutProf : 0 0
Egress Queue 1
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 48844432 48844432000
For. In/InplusProf : 0 0
For. Out/ExcProf : 3083637 3083637000
Egress Queue 2
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 39593533 39593533000
For. In/InplusProf : 0 0
For. Out/ExcProf : 12334536 12334536000
Egress Queue 3
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 0 0
For. In/InplusProf : 0 0
For. Out/ExcProf : 29673215 29673215000
Egress Queue 4
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 14981730 14981730000
For. In/InplusProf : 0 0
For. Out/ExcProf : 7272849 7272849000
Egress Queue 5
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 7708883 7708883000
For. In/InplusProf : 0 0
For. Out/ExcProf : 14545696 14545696000
Egress Queue 6
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 7563427 7563427000
For. In/InplusProf : 0 0
For. Out/ExcProf : 7272849 7272849000
To reveal the subscriber hierarchy in a terse form with respect to the sub/SLA-profiles and the SAP, the following command can be run:
*A:PE-1# show service active-subscribers hierarchy
===============================================================================
Active Subscribers Hierarchy
===============================================================================
-- sub-1 (sub-prof-1)
|
|-- sap:[3/1/1:1.1] - sla:sla-1-1
| |
| +-- 10.10.1.1 - mac:00:00:64:01:01:01 - DHCP - svc:3
|
+-- sap:[3/1/1:1.1] - sla:sla-1-2
|
+-- 10.10.1.2 - mac:00:00:64:01:01:02 - DHCP - svc:3
-- sub-2 (sub-prof-2)
|
|-- sap:[3/1/1:1.2] - sla:sla-2-1
| |
| +-- 10.10.1.3 - mac:00:00:64:01:01:03 - DHCP - svc:3
|
+-- sap:[3/1/1:1.2] - sla:sla-2-2
|
+-- 10.10.1.4 - mac:00:00:64:01:01:04 - DHCP - svc:3
-------------------------------------------------------------------------------
Number of active subscribers : 2
Flags: (N) = the host or the managed route is in non-forwarding state
===============================================================================
The following SAP related show command confirms that the SAP queues are removed from the underlying subscriber SAP (under the stats section at the end of the output). This was ensured by configuring the profiled-traffic-only command in the MSAP policy, with the purpose of reducing the queue consumption on ingress and egress.
*A:PE-1# show service id 3 sap 3/1/1:1.1 detail
===============================================================================
Service Access Points(SAP)
===============================================================================
Service Id : 3
SAP : 3/1/1:1.1 Encap : qinq
QinQ Dot1p : Default
Description : Managed SAP - Capture Svc 10 3/1/1:*.*
Admin State : Up Oper State : Up
Flags : None
Multi Svc Site : None
Last Status Change : 09/22/2017 17:17:22
Last Mgmt Change : 09/22/2017 17:21:50
Sub Type : managed
Capture Service Id : 10 Capture SAP : 3/1/1:*.*
MSAP Policy : msaps
Idle : no Sticky : no
Dot1Q Ethertype : 0x8100 QinQ Ethertype : 0x8100
Split Horizon Group: (Not Specified)
Admin MTU : 1522 Oper MTU : 1522
Ingr IP Fltr-Id : n/a Egr IP Fltr-Id : n/a
Ingr Mac Fltr-Id : n/a Egr Mac Fltr-Id : n/a
Ingr IPv6 Fltr-Id : n/a Egr IPv6 Fltr-Id : n/a
qinq-pbit-marking : both
Egr Agg Rate Limit: max
Q Frame-Based Acct : Disabled Limit Unused BW : Disabled
Acct. Pol : None Collect Stats : Disabled
Anti Spoofing : Ip-Mac Dynamic Hosts : Enabled
Avl Static Hosts : 0 Tot Static Hosts : 0
Calling-Station-Id : n/a
Application Profile: None
Transit Policy : None
AARP Id : None
Oper Group : (none) Monitor Oper Grp : (none)
Host Lockout Plcy : n/a
Lag Link Map Prof : (none)
Bandwidth : Not-Applicable
Oper DCpu Prot Pol*: _default-access-policy
-------------------------------------------------------------------------------
ETH-CFM SAP specifics
-------------------------------------------------------------------------------
Tunnel Faults : accept AIS : Disabled
MC Prop-Hold-Timer : n/a
Squelch Levels : None
Collect Lmm Stats : Disabled
LMM FC Stats : None
LMM FC In Prof : None
-------------------------------------------------------------------------------
QOS
-------------------------------------------------------------------------------
Ingress qos-policy : 1 Egress qos-policy : 1
Ingress FP QGrp : (none) Egress Port QGrp : (none)
Ing FP QGrp Inst : (none) Egr Port QGrp Inst: (none)
Shared Q plcy : n/a Multipoint shared : Disabled
I. Sched Pol : (Not Specified)
E. Sched Pol : (Not Specified)
I. Policer Ctl Pol : (Not Specified)
E. Policer Ctl Pol : (Not Specified)
E. HS Sec. Shaper : (Not Specified)
I. QGrp Redir. List: (Not Specified)
E. QGrp Redir. List: (Not Specified)
-------------------------------------------------------------------------------
Subscriber Management
-------------------------------------------------------------------------------
Admin State : Up MAC DA Hashing : False
Def Sub-Id : Use auto-sub-id
Def Sub-Profile : None
Def SLA-Profile : None
Def Inter-Dest-Id : (Use top-q-tag)
Def App-Profile : None
Sub-Ident-Policy : sub_ident_pol
Subscriber Limit : 1
Single-Sub-Parameters
Prof Traffic Only : True
Non-Sub-Traffic : N/A
Static host management
MAC learn options : N/A
-------------------------------------------------------------------------------
Sap Statistics
-------------------------------------------------------------------------------
Last Cleared Time : N/A
Packets Octets
CPM Ingress : 5 1077
Forwarding Engine Stats
Dropped : 0 0
Received Valid : 0 0
Off. HiPrio : 0 0
Off. LowPrio : 0 0
Off. Uncolor : 0 0
Off. Managed : 0 0
Queueing Stats(Ingress QoS Policy 1)
Dro. HiPrio : 0 0
Dro. LowPrio : 0 0
For. InProf : 0 0
For. OutProf : 0 0
Queueing Stats(Egress QoS Policy 1)
Dro. In/InplusProf : 0 0
Dro. Out/ExcProf : 0 0
For. In/InplusProf : 0 0
For. Out/ExcProf : 0 0
-------------------------------------------------------------------------------
Sap per Queue stats
-------------------------------------------------------------------------------
Packets Octets
No entries found
===============================================================================
* indicates that the corresponding row element may have been truncated.
QoS-related Show Commands in ESM Context
Examination of the subscriber QoS hierarchy on HSQ can start at the subscriber (HS queue group and SAP) level and gradually move through the HS secondary shaper, and finally the port level. The output of the show commands should confirm that the subscriber is associated with the QoS object as intended by the configuration.
For example, the following output shows that HS attachment policy ‟hs-attach-1-1” is associated with the QoS SAP egress policy 10. It was determined previously that QoS SAP egress policy 10 is associated with host 1 of the subscriber 1 (‟sub-1-1”). In this case, a two-step process was necessary to track the association between the subscriber and the HS attachment policy.
*A:PE-1# show qos sap-egress 10 association
===============================================================================
QoS Sap Egress
===============================================================================
-------------------------------------------------------------------------------
Sap Egress Policy (10)
-------------------------------------------------------------------------------
Policy-id : 10 Scope : Template
Ethernet-ctag : False Parent-loc : default
Name : (Not Specified)
Description : (Not Specified)
Policy Active : True Plcrs HQoS Managed : False
Post Plcr Mapping Policy: (Not Specified)
HS Attachment Policy : hs-attach-1-1
-------------------------------------------------------------------------------
Dynamic Configuration Information
-------------------------------------------------------------------------------
PccRule Insert Point : n/a DynPlcr Insert Point : n/a
CBS : Def MBS : Def
Parent : (Not Specified)
Level : 1 Weight : 1
Packet Byte Offset : 0
Stat Mode : minimal
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Associations
-------------------------------------------------------------------------------
SLA Profiles :
- sla-1-1
-------------------------------------------------------------------------------
HSMDA Associations
-------------------------------------------------------------------------------
No Associations Found.
===============================================================================
The output from the HS attachment policy reflects the QoS configuration state at the subscriber level that is shown in Figure 2:
Queues 1 and 2 are attached to WRR 1.
Queues 3 to 6 are directly attached to the corresponding scheduling classes (3 to 6).
Queues 7 and 8 are unattached.
WRR 2 is unattached.
HS attachment policy ‟hs-attach-1” is associated with QoS SAP egress policies 10, 20, 30, and 40 that correspond to subscriber hosts ‟sub-1-1”, ‟sub-1-2”, ‟sub-2-1”, and ‟sub-2-2”.
*A:PE-1# show qos hs-attachment-policy "hs-attach-1-1" detail
===============================================================================
HS Attachment Policy Information
===============================================================================
Policy Name : hs-attach-1-1
Description : (Not Specified)
Low Burst Max Class : 3
-------------------------------------------------------------------------------
Queue Scheduling Class WRR Group
-------------------------------------------------------------------------------
1 (Not-Applicable) 1
2 (Not-Applicable) 1
3 3 (Not-Applicable)
4 4 (Not-Applicable)
5 5 (Not-Applicable)
6 6 (Not-Applicable)
7 unattached unattached
8 unattached unattached
-------------------------------------------------------------------------------
WRR Group Scheduling Class
-------------------------------------------------------------------------------
1 1
2 unattached
-------------------------------------------------------------------------------
Associations
-------------------------------------------------------------------------------
Network-Queue Policy
-------------------------------------------------------------------------------
No Matching Entries
Sap-Egress Policy
-------------------------------------------------------------------------------
10
20
30
40
Egress Queue-Group Templates
-------------------------------------------------------------------------------
No Matching Entries
Association between the HS secondary shaper and the subscribers can be verified with the following command. HS secondary shaper is allocated per port (or LAG). The two subscribers sub-1 and sub-2 are instantiated on SAPs with the outer VLAN tag ‟1” and consequently they are both associated with the HS secondary shaper 1. The HS secondary shaper 1 is rate limited to 120Mb/s while its scheduling classes are left open (max rate).
*A:PE-1# show port 3/1/1 hs-secondary-shaper "1" associations
===============================================================================
Ethernet Port 3/1/1 Egress HS Secondary Shaper Information
===============================================================================
Policy Name : 1
Description : (Not Specified)
Rate : 250000 Kbps
Low Burst Max Class: 3
-------------------------------------------------------------------------------
Class Rate
-------------------------------------------------------------------------------
1 max
2 max
3 max
4 max
5 max
6 max
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Service Associations
-------------------------------------------------------------------------------
Service ID Service Type SAP
-------------------------------------------------------------------------------
No Service Associations Found.
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Subscriber Associations
-------------------------------------------------------------------------------
Subscriber ID
-------------------------------------------------------------------------------
sub-1
sub-2
-------------------------------------------------------------------------------
Number of subscriber associations : 2
The port scheduler information can be obtained with the following command. This command is run in the QoS context (as opposed to being run in the port context, which was the case for HS secondary schedulers). The reason for this is that the HS secondary scheduler is configured directly under the port, while the HS port scheduler is configured in an HS scheduler policy (in QoS context), which is then applied to a port.
*A:PE-1# show qos hs-scheduler-policy "hs1" detail
===============================================================================
HS Scheduler Policy Information
===============================================================================
Policy Name : hs1
Description : (Not Specified)
Max Rate : 300 Mbps
-------------------------------------------------------------------------------
Scheduling Class Rate Group Weight in Group
-------------------------------------------------------------------------------
1 max 0 1
2 max 0 1
3 max 0 1
4 max 1 10
5 max 1 20
6 20 Mbps 0 1
-------------------------------------------------------------------------------
Group Rate
-------------------------------------------------------------------------------
1 60 Mbps
-------------------------------------------------------------------------------
Port Ethernet Egress Associations
-------------------------------------------------------------------------------
3/1/1
Show Commands with Dynamically Changing Information
The following commands show the QoS hierarchy with the running rates of the objects (queues, WRR groups, scheduling classes, secondary shapers, and port shapers) and the queue buffer depths in the QoS hierarchy. The command output in this section is based on the scenario described in the Traffic Management on HSQ section.
Running rates are the dynamically calculated rates while traffic is running.
The following subscriber hierarchy (for sub-1) shows the rates per subscriber host, per scheduling priority (and consequently, scheduling class), starting at the queue level and moving up toward the port level. For example, scheduling priority 1 starts with the rates of two subscriber host queues (1 and 2) that are mapped to the WRR group and then it moves up to the rate at the HS secondary shaper level (the summed rate of all the entities at scheduling priority 1 at the HS secondary shaper level), ending with the rate of the scheduling class 1 at the port level. The aggregate rate of the HS secondary shaper and the port in the subscriber hierarchy are also provided.
In this case, the rates between the two HSQ queue groups (or subscriber hosts) are evenly spread. This distribution will be explained in the Traffic Management on HSQ section. The subscriber hosts can be differentiated by the two different SLA profiles with which they are associated.
*A:PE-1# show qos hs-scheduler-hierarchy subscriber "sub-1" egress
===============================================================================
Hs Scheduler Hierarchy Information
===============================================================================
PortId : 3/1/1
SAP : [3/1/1:1.1]
SLA Profile : sla-1-1
Hs Sched Policy Name : hs1
Port Max-Rate : 250 Mbps
Hs-Sec-Shaper:1 Agg-Rate : 250479 Kbps
Scheduler Priority 6
Scheduler Class 6 Rate : 20 Mbps
Hs-Sec-Shaper:1 Class 6 Rate : 20040 Kbps
Queue : 6 Rate : 4912 Kbps
Scheduler Priority 5 Group 1
Scheduler Class 5 Rate : 40 Mbps Weight : 20
Hs-Sec-Shaper:1 Class 5 Rate : 40098 Kbps
Queue : 5 Rate : 9824 Kbps
Scheduler Class 4 Rate : 20 Mbps Weight : 10
Hs-Sec-Shaper:1 Class 4 Rate : 20024 Kbps
Queue : 4 Rate : 4912 Kbps
Scheduler Priority 3
Scheduler Class 3 Rate : 81 Mbps
Hs-Sec-Shaper:1 Class 3 Rate : 81746 Kbps
Queue : 3 Rate : 20032 Kbps
Scheduler Priority 2
Scheduler Class 2 Rate : 0 Mbps
Hs-Sec-Shaper:1 Class 2 Rate : 0 Kbps
Scheduler Priority 1
Scheduler Class 1 Rate : 88 Mbps
Hs-Sec-Shaper:1 Class 1 Rate : 88568 Kbps
Queue : 1 Group : 1 Rate : 2080 Kbps
Queue : 2 Group : 1 Rate : 8336 Kbps
===============================================================================
PortId : 3/1/1
SAP : [3/1/1:1.1]
SLA Profile : sla-1-2
Hs Sched Policy Name : hs1
Port Max-Rate : 250 Mbps
Hs-Sec-Shaper:1 Agg-Rate : 250414 Kbps
Scheduler Priority 6
Scheduler Class 6 Rate : 20 Mbps
Hs-Sec-Shaper:1 Class 6 Rate : 20040 Kbps
Queue : 6 Rate : 4912 Kbps
Scheduler Priority 5 Group 1
Scheduler Class 5 Rate : 40 Mbps Weight : 20
Hs-Sec-Shaper:1 Class 5 Rate : 40065 Kbps
Queue : 5 Rate : 9808 Kbps
Scheduler Class 4 Rate : 20 Mbps Weight : 10
Hs-Sec-Shaper:1 Class 4 Rate : 20040 Kbps
Queue : 4 Rate : 4912 Kbps
Scheduler Priority 3
Scheduler Class 3 Rate : 81 Mbps
Hs-Sec-Shaper:1 Class 3 Rate : 81714 Kbps
Queue : 3 Rate : 20040 Kbps
Scheduler Priority 2
Scheduler Class 2 Rate : 0 Mbps
Hs-Sec-Shaper:1 Class 2 Rate : 0 Kbps
Scheduler Priority 1
Scheduler Class 1 Rate : 88 Mbps
Hs-Sec-Shaper:1 Class 1 Rate : 88552 Kbps
Queue : 1 Group : 1 Rate : 2080 Kbps
Queue : 2 Group : 1 Rate : 8336 Kbps
===============================================================================
The following command provides the running rates at the HS secondary shaper and port levels:
*A:PE-1# show qos hs-scheduler-hierarchy port 3/1/1 hs-secondary-shapers
===============================================================================
Hs Scheduler Hierarchy Information
===============================================================================
Hs Sched Policy Name : hs1
Port Max-Rate : 250 Mbps
Scheduler Priority 6
Scheduler Class 6 Rate : 20 Mbps
Scheduler Priority 5 Group 1
Scheduler Class 5 Rate : 40 Mbps Weight : 20
Scheduler Class 4 Rate : 20 Mbps Weight : 10
Scheduler Priority 3
Scheduler Class 3 Rate : 81 Mbps
Scheduler Priority 2
Scheduler Class 2 Rate : 0 Mbps
Scheduler Priority 1
Scheduler Class 1 Rate : 88 Mbps
-------------------------------------------------------------------------------
HS Secondary Shaper Rates
-------------------------------------------------------------------------------
Hs-Sec-Shaper:1 Agg-Rate : 249043 Kbps
Class 6 Rate : 19926 Kbps
Class 5 Rate : 39853 Kbps
Class 4 Rate : 19926 Kbps
Class 3 Rate : 81273 Kbps
Class 2 Rate : 0 Kbps
Class 1 Rate : 88062 Kbps
Hs-Sec-Shaper:default Agg-Rate : 0 Kbps
Class 6 Rate : 0 Kbps
Class 5 Rate : 0 Kbps
Class 4 Rate : 0 Kbps
Class 3 Rate : 0 Kbps
Class 2 Rate : 0 Kbps
Class 1 Rate : 0 Kbps
-------------------------------------------------------------------------------
Another important parameter to monitor is the depth of the queues. This provides information about the congestion at the queue level. The buffer space per queue is allocated automatically by the system and, in the following case, the buffers are rather large. Deep buffering causes longer delays. To avoid this, the queue buffers can be adjusted by the mbs command under the queue definition in the QoS SAP egress policy. For example, 15kbytes would accommodate roughly fifteen 1000byte packets in a buffer queue.
*A:PE-1>config>qos>sap-egress# info
----------------------------------------------
queue 1 create
mbs 15 kilobytes
The following output shows that, except for queues 3, all queues have their buffers fully used. Because there is no congestion on queues 3, their buffer depth is low.
*A:PE-1# show hs-pools port 3/1/1 egress subscriber "sub-1"
| match "Queue Information" pre-lines 1 post-lines 200
-------------------------------------------------------------------------------
Queue Information
-------------------------------------------------------------------------------
Queue Name : Sub=sub-1:sla-1-1 3->3/1/1:1.1->1
FC Map : be ef nc
Admin PIR : 20000 Oper PIR : 0
Admin MBS : 64 KB Oper MBS : 64 KB
HS Wrr Group : 1
HS Wrr Class Weight: 1 HS Wrr Weight : 1
Depth : 58 KB
HS Class : 1 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-1 3->3/1/1:1.1->2
FC Map : l2
Admin PIR : 20000 Oper PIR : 0
Admin MBS : 64 KB Oper MBS : 64 KB
HS Wrr Group : 1
HS Wrr Class Weight: 1 HS Wrr Weight : 4
Depth : 58 KB
HS Class : 1 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-1 3->3/1/1:1.1->3
FC Map : af
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 1 KB
HS Class : 3 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-1 3->3/1/1:1.1->4
FC Map : l1
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 333 KB
HS Class : 4 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-1 3->3/1/1:1.1->5
FC Map : h1
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 333 KB
HS Class : 5 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-1 3->3/1/1:1.1->6
FC Map : h2
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 332 KB
HS Class : 6 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-2 3->3/1/1:1.1->1
FC Map : be ef nc
Admin PIR : 40000 Oper PIR : 0
Admin MBS : 64 KB Oper MBS : 64 KB
HS Wrr Group : 1
HS Wrr Class Weight: 1 HS Wrr Weight : 1
Depth : 56 KB
HS Class : 1 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-2 3->3/1/1:1.1->2
FC Map : l2
Admin PIR : 40000 Oper PIR : 0
Admin MBS : 64 KB Oper MBS : 64 KB
HS Wrr Group : 1
HS Wrr Class Weight: 1 HS Wrr Weight : 4
Depth : 58 KB
HS Class : 1 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-2 3->3/1/1:1.1->3
FC Map : af
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 1 KB
HS Class : 3 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-2 3->3/1/1:1.1->4
FC Map : l1
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 333 KB
HS Class : 4 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-2 3->3/1/1:1.1->5
FC Map : h1
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 332 KB
HS Class : 5 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
Queue Name : Sub=sub-1:sla-1-2 3->3/1/1:1.1->6
FC Map : h2
Admin PIR : Max Oper PIR : Max
Admin MBS : 375000 B Oper MBS : 375296 B
HS Wrr Group : (not-applicable)
HS Wrr Class Weight: 1 HS Wrr Weight : 0
Depth : 333 KB
HS Class : 6 HS Alt Port Class Pool : No
HS Slope Policy : _tmnx_hs_default
-------------------------------------------------------------------------------
Clear Commands
Clear commands in the HSQ context are used to clear statistics associated with the HS secondary shaper:
clear port <port-id> hs-secondary-shaper <name> statistics
Resources Monitoring
The tools dump system-resources command is used to monitor resource on an HSQ IOM.
Some of the resources (HSQ queue groups, HS primary shapers, HS secondary shapers, HS turbo queue groups, and so on) are allocated for internal use, thereby reducing the number of resources in the Free column. Such internally consumed resources are not available to be part of the user configuration.
100G ports support port-based (access or network) HS queue groups on egress that can be configured to support higher throughput rates. Such port-based high-rate HS queue groups on egress are referred to as egress HS turbo queue groups.
The number of internally consumed resources depends on the configuration and MDA types. In the following example, the number of allocated HS queue groups is 40. Each of the four subscriber hosts consume one HS queue group, which means that 36 HS queue groups are internally allocated.
Out of 23 allocated HS primary shapers, two are allocated for the two subscribers and one is allocated to fulfill the HSQ hierarchy chain in conjunction with the default HS secondary shaper. This leaves 20 HS primary shapers consumed internally by the HSQ IOM.
Similarly, out of 22 allocated HS secondary shapers in total, only two are user related (and visible via show commands): a default HS secondary shaper and the HS secondary shaper 1. This means that 20 HS secondary shapers are internally consumed. The same logic can be followed for turbo HS queue groups, which are out of the scope in this chapter.
On HSQ IOM, a complete scheduling hierarchy must be maintained between each attached queue at the SPI level and its port priority. That is, the HSQ has no provision for bypassing the primary and secondary shaping/scheduling levels.
*A:PE-1# tools dump system-resources 3
Resource Manager info at 004 h 09/22/17 22:09:11.322:
Hardware Resource Usage for Slot #3, CardType iom4-e-hs, Cmplx #0:
| Total | Allocated | Free
-------------------------------|-----------|-----------|------------
SAP Ingress QoS Policies | 1791| 1| 1790
Dynamic Egress Classification + 2047| 5| 2042
SAP Egress QoS Policies - | 5|
Network Egress Classification - | 0|
Ingress Queues | 131072| 502| 130570
Egress Queues | 786432| 143| 786289
Egress HS Turbo Queue Groups | 64| 10| 54
Egress HS Queue Groups | 98240| 40| 98200
Primary Shapers + 16384| 23| 16361
Explicit Primary Shapers - | 2|
Managed Primary Shapers - | 21|
Secondary Shapers | 4096| 22| 4074
Ingress Policers | 511999| 1| 511998
Egress Policers | 262143| 1| 262142
Ingress Policer Stats | 511967| 0| 511967
Egress Policer Stats | 262111| 0| 262111
Qos Ingress Root Arbiters | 65535| 1| 65534
Qos Egress Root Arbiters | 65535| 1| 65534
Qos Intermediate Arbiters | 262143| 0| 262143
Egress QoS Bypass | 131071| 0| 131071
Ingress ACL Entries | 65536| 2| 65534
Ingress QoS Entries | 16384| 2| 16382
Ingress IPv6 ACL Entries | 28672| 2| 28670
Ingress IPv6 QoS Entries | 4096| 2| 4094
Egress ACL Entries | 32768| 2| 32766
Egress QoS Entries | 14336| 2| 14334
Egress IPv6 ACL Entries | 16384| 2| 16382
Egress IPv6 QoS Entries | 2048| 2| 2046
Ingress ACL Filters | 2047| 0| 2047
Ingress IPv6 ACL Filters | 2047| 0| 2047
Egress ACL Filters | 2047| 0| 2047
Egress IPv6 ACL Filters | 2047| 0| 2047
QoS User Schedulers | 98303| 0| 98303
QoS User Scheduler Overrides | 196607| 0| 196607
Sap IngQGrp RedirLst Entries | 31999| 0| 31999
Dynamic Service Entries + 131071| 4| 131067
Subscriber Hosts - 131071| 4| 131067
Encap Group Members - 65535| 0| 65535
Egr Network Queue Group Mappings - 131071| 0| 131071
SapInst EgrQGrp RedirLst Entries - 31999| 0| 31999
Dynamic Nexthop Entries + 511999| 4| 511995
Subscriber Nexthops - 511999| 4| 511995
Ipsec tunnels - 511999| 0| 511999
Subscriber SPI QoS Overrides | 131072| 0| 131072
Mac Fdb Entries | 511999| 0| 511999
Egress TLS Mcast Entries | 368639| 1| 368638
Traffic Management on HSQ
This section examines traffic output on HSQ IOM during congestion. Managing Congestion on HSQ in Expanded SLA Mode is a graphic representation of the configuration described previously, but with traffic streams running through the HSQ IOM.
Six traffic streams are sent in the downstream direction toward each of the four subscriber hosts (host 1-1, host 1-2, host 2-1, and host 2-2): in total, there are 24 traffic streams. The traffic streams are shown on the left side of Managing Congestion on HSQ in Expanded SLA Mode, with their names, offered rates (IN column) and measured output rates (OUT column). The traffic streams are sent and analyzed by the traffic generator.
The four shaded squares in the center of Managing Congestion on HSQ in Expanded SLA Mode represent the four subscriber hosts and their scheduling classes. The QoS hierarchy is shown on the right side. Red shaded areas (shapers) represent points of congestion on HSQ IOM caused by the 24 traffic streams.
Each of the six traffic streams per subscriber host is fed into the six queues of each subscriber host (or HSQ queue groups). The first digit in the traffic stream name represents the subscriber, the second digit represents the subscriber host, and the third digit represents the queue to which this stream is sent. For example, STRM 1-2-3 represents a traffic stream sent to queue 3 of the second host for subscriber 1.
To summarize the scenario shown in Managing Congestion on HSQ in Expanded SLA Mode:
Traffic streams 1 and 2 of each subscriber host are mapped to subscriber queues 1 and 2, which are in turn associated with WRR group 1 at the subscriber host (HSQ queue group) level.
WRR group 1 is, depending on the subscriber host, rate limited to 20Mb/s, 40Mb/s, 60Mb/s, and 80 Mb/s. Weight ratio between queues 2 and 1 is 4:1. WRR group 1 is then attached to scheduling class 1.
Traffic stream 3 is via queue 3 directly mapped to scheduling class 3.
Scheduling class 2 is unused in this example.
Traffic streams 4 and 5 are via queues 4 and 5 mapped to scheduling classes 4 and 5, which are at the port level collapsed into WRR group 1 with an aggregate rate limit of 60Mb/s.
Traffic stream 6 is the highest priority stream that is via queue 6 mapped to scheduling class 6. At the port level, scheduling class 6 is rate limited to 20Mb/s.
Subscriber host (HSQ queue group) aggregate rates are set to 50Mb/s, 70Mpbs, 80 Mb/s, and 100Mb/s, respectively.
Subscribers (sub-1 and sub-2) aggregate rates (HS primary shapers) are set to 100Mb/s and 150 Mb/s, respectively.
Subscribers sub-1 and sub-2 are mapped to HS secondary shaper ‟1” (via outer VLAN on their SAPs) with the aggregate rate of 250 Mb/s.
HS port scheduler is rate limited to 300Mb/s.
The configured rate limits at the subscriber level are L2 rates, while configured rate limits at the HS secondary shaper level and the HS port level include L1 overhead and are, therefore, on-the-wire rates. On-the-wire rates account for 20 additional bytes in each Ethernet frame (8 bytes preamble and 12 bytes IFG).
All traffic streams are sent with constant rates (no added burstiness) and fixed packet size (1000 bytes). Therefore, the difference between the L2 rates and the on-the-wire rates for the 1000byte packets is 1000/1020 = 0.98 or 2%. That is, on-the-wire rates are 2% higher than the L2 rates. The name for this 2% delta factor in this chapter will be the Rate Conversion Factor (RCF).
Input/Output Rates lists the input/output rates throughout the subscriber QoS hierarchy. Red shaded table cells represent congested objects. The numbers in blue represent the L2 output rates measured on the traffic generator. The numbers in red (in parentheses) are configured aggregate rates for the object in the hierarchy. The numbers above them are either the operational rates measured by SR OS and observed via the show qos hs-scheduler-hierarchy command, or the manually summed rates under the hierarchy.
Running aggregate rates per HSQ queue group or subscriber (HS primary shaper) are now available via show commands.
Some of the output columns in this table require additional explanation:
Per stream output rates (first column under the Output Rate section) are L2 rates as measured by the traffic generator. The analysis of the results will be based on these rates.
However, those rates can also be displayed within the system via the show qos hs-scheduler-hierarchy subscriber <sub-name> egress command. RCF must be used to correctly interpret the results in respective rate domains (L1 versus L2).
WRR rates at the subscriber level (second column under the Output Rate section) are populated by manually adding rates measured by the traffic generator for traffic streams 1 and 2. The 4:1 notation in parentheses represents the weight ratio between the two streams. There are no means to observe the WRR group rates at the subscriber level directly within the system (via show commands).
Aggregate rates per host (third column under the Output Rate section) are populated by manually adding the rates measured by the traffic generator from all six traffic streams for each subscriber. There are no means to observe the aggregate subscriber rate directly within the system (via show commands). Those rates also represent the HSQ queue group aggregate rate because each host in this example is associated with an HSQ queue group.
Aggregate rates per subscriber (fourth column under the Output Rate section) are populated by manually adding the rates measured by the traffic generator from all six traffic streams for each subscriber. There are no means to observe the aggregate subscriber rate directly within the system (via show commands).
Aggregate rates per HS secondary shaper (fifth column under the Output Rate section) are on-the-wire rates displayed via the show qos hs-scheduler-hierarchy port 3/1/1 hs-secondary-shapers command. The displayed rates are rounded to the nearest Mb/s.
Per scheduling class rates at the port level (sixth column under the Output Rate section) are displayed via the show qos hs-scheduler-hierarchy port 3/1/1 command. The displayed rates are rounded by the system to the nearest Mb/s. The discrepancy between the actual rates for scheduling class 3 (80Mb/s) and the displayed rate (81Mb/s) is due to the measuring and rounding inaccuracy at display time. The actual rate at scheduling class 3 at the port level can be calculated by summing the rates measured (by the traffic generator) of each traffic stream 3 for each subscriber and adjusting the sum for the on-the-wire rate (RCF).
The same logic applies to scheduling class 1 (actual rate 86.8Mb/s vs 88Mb/s measured rate).
The WRR-1 rate at the port level (seventh column under the Output Rate section) is manually calculated by adding the rates measured by the traffic generator of all streams mapped to scheduling classes 5 and 4 (8 traffic streams in total, two per each subscriber host), then converting this rate into the on-the-wire rate. At the port level, traffic on scheduling classes 5 and 4 is weighted in a 2:1 ratio (this is noted in parentheses).
The port rate (last column under the Output Rate section) is collected from two places:
The top number in obtained via the show qos hs-scheduler-hierarchy command and represents the on-the-wire rate.
The bottom number in blue is the number measured by the traffic generator, which only measures L2 rates. On-the-wire rates can be converted to L2 rates by multiplying the on-the-wire rates by the RCF.
Strm |
Input rate in [mbps] |
Output rate in [mbps] |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Per strm |
Agg per host |
Per strm (L2) |
WRR on subscr level |
Agg per host (L2) |
Agg per subscr (L2) |
Agg per secondary shaper (wire) |
Per sch class on port level (wire) |
WRR-1 on port level (wire) |
Port |
|
1-1-6 |
10 |
110 |
4.9 |
- |
50 (50) |
100 (100) |
250 (250) |
SC 6 20 (20) |
- |
250 (300) 245.1 (L2) |
1-1-5 |
15 |
9.8 |
- |
- |
||||||
1-1-4 |
15 |
4.9 |
- |
- |
||||||
1-1-3 |
20 |
20 |
- |
- |
||||||
1-1-2 |
25 |
8.3 |
10.4 (4:1) |
SC 5 40 |
60 (60) (2:1) |
|||||
1-1-1 |
25 |
2.1 |
||||||||
1-2-6 |
10 |
130 |
4.9 |
- |
50 (70) |
|||||
1-2-5 |
15 |
9.8 |
- |
|||||||
1-2-4 |
15 |
4.9 |
- |
SC 4 20 |
||||||
1-2-3 |
20 |
20 |
- |
|||||||
1-2-2 |
35 |
8.3 |
10.4 (4:1) |
|||||||
1-2-1 |
35 |
2.1 |
||||||||
2-1-6 |
10 |
160 |
4.9 |
- |
72.6 (80) |
145.2 (150) |
SC 3 81 |
- |
||
2-1-5 |
15 |
9.8 |
- |
- |
||||||
2-1-4 |
15 |
4.9 |
- |
- |
||||||
2-1-3 |
20 |
20 |
- |
- |
||||||
2-1-2 |
50 |
26.4 |
33 (4:1) |
SC 2 - |
- |
|||||
2-1-1 |
50 |
6.6 |
- |
|||||||
2-2-6 |
10 |
160 |
4.9 |
- |
72.6 (100) |
- |
||||
2-2-5 |
15 |
9.8 |
- |
- |
||||||
2-2-4 |
15 |
4.9 |
- |
SC 1 88 |
- |
|||||
2-2-3 |
20 |
20 |
- |
- |
||||||
2-2-2 |
50 |
26.4 |
33 (4:1) |
- |
||||||
2-2-1 |
50 |
6.6 |
- |
|||||||
Analysis of Results
The analysis of results begins with the rates per stream measured by the traffic generator. The expected behavior is that those rates agree with the theoretical rate calculations based on our understanding of QoS on HSQ IOM.
Considering the six strict priority classes in the HSQ scheduling mechanism, the expectation is that the traffic is serviced in the order of priority, from the highest scheduling class 6 to the lowest scheduling class 1. Consequently, the traffic analysis starts with the streams that are mapped to the highest priority scheduling class 6 (streams 1-1-6, 1-2-6, 2-1-6, 2-2-6, that is, stream 6 of each subscriber host).
Scheduling class 6 (streams 1-1-6, 1-2-6, 2-1-6, and 2-2-6) – the highest priority scheduling class
Due to the aggregate rate limit of 20Mb/s for scheduling class 6 at the port level, it is expected that each subscriber host receives an equal amount of traffic on scheduling class 6:
(20 Mb/s (aggregate rate of SC 6))÷(4 streams (one per subscriber host)) =5 Mb/s
The measured results on the traffic generator for traffic streams 6 in Table 1 show that each subscriber host receives 4.9Mb/s out of an offered 10Mb/s. The slight difference between the expected (5Mb/s) and the measured (4.9Mb/s) rate is caused by the discrepancy between the L2 rates at the subscriber level (as measured by the traffic generator), and on-the-wire rates enforcement (20Mb/s) at the scheduling class 6 at the port level. Multiplying 5Mb/s by the RCF (1000/1020) will align the results.
This slight discrepancy between the L2 and on-the-wire rates is common throughout the remaining analysis. Because this is well understood and expected, it will not be mentioned again.
Scheduling classes 5 and 4 (streams 1-1-5, 1-1-4, 1-2-5, 1-2-4, 2-1-5, 2-1-4, 2-2-5, and 2-2-4)
Scheduling classes 5 and 4 are collapsed at the port level into WRR group 1, which is at the next scheduling priority to be served. These two scheduling classes are served by WRR group 1 in a 2:1 ratio. The aggregate rate limit for WRR 1 at the port level is set to 60Mb/s.
Expected rates for combined scheduling classes 5 and 4 of each host are
60Mb/s÷4=15Mb/s
When 15Mb/s is distributed between the two scheduling classes in the 2:1 ratio, each host should receive 10Mb/s on scheduling class 5 and 5Mb/s on scheduling class 4, for the total of 15Mb/s (out of an offered 30Mb/s).
The measured results in Input/Output Rates are 9.8Mb/s on scheduling class 5 and 4.9Mb/s on scheduling class 4 for traffic streams 4 and 5 of each individual host. This is in line with the expected results (the slight difference is due to L2 rates measured by the traffic generator and enforced on-the-wire rates at the port level).
Scheduling class 3 (streams 1-1-3, 1-2-3, 2-1-3, and 2-2-3)
Traffic streams mapped to scheduling class 3 do not have any rate restriction at the scheduling class level. Those streams can be only limited by the congestion at the HSQ queue group aggregate level, the subscriber aggregate level, the HS secondary shaper aggregate level, or the port aggregate level. Because the total amount of traffic so far is below congestion level at each point in the hierarchy, it is expected that traffic stream 3 flows unimpeded. Consequently, each subscriber host should receive the full input rate of 20Mb/s on scheduling class 3. The actual results in Input/Output Rates are aligned with the expected results.
To confirm that there is no congestion at the aggregate level so far for the subscriber hosts, the subscribers, the HS secondary shapers, and the port, a calculation shows that each subscriber host has received an equal amount of bandwidth so far: 39.6Mb/s. This is below the configured aggregate rate limits at the host, the subscriber, the HS secondary scheduler and the port levels:
39.6Mb/s is below the configured limit of 50Mb/s for host 1-1.
39.6Mb/s is below the configured limit of 70Mb/s for host 1-2.
39.6Mb/s is below the configured limit of 80Mb/s for host 2-1.
39.6Mb/s is below the configured limit of 100Mb/s for host 2-2.
79.2Mb/s (39.6Mb/s x 2) is below the configured limit of 100Mb/s for sub-1.
79.2Mb/s (39.6Mb/s x 2) is below the configured limit of 150Mb/s for sub-2.
Sub-1 and sub-2 compete for the bandwidth at HS secondary shaper ‟1”, and their combined rate of 158.4 Mb/s (39.6Mb/s x 4) is below the configured aggregate rate of the HS secondary shaper ‟1” (250Mb/s) or the port shaper (300Mb/s).
Scheduling classes 2 and 1 (streams 1-1-2, 1-1-1, 1-2-2, 1-2-1, 2-1-2, 2-1-1, 2-2-2, 2-2-1) – the lowest priority scheduling classes
The total amount of offered (input) traffic for lowest priority scheduling classes 1 and 2 across all four subscriber hosts is 560Mb/s (110Mb/s for host 1-1, 130Mb/s for host 1-2, 160Mb/s for host 2-1, and 160Mb/s for host 2-2). This additional amount of traffic will cause congestion on the aggregate level for host 1-1, subscriber sub-1, and HS secondary shaper 1.
Up to this point, the spare capacity on the HS secondary shaper 1 is:
250 Mb/s (HS secondary shaper 1 aggregate on-the-wire rate limit)
- 40.3 Mb/s (on-the-wire traffic from host 1-1 up to this point)
- 40.3 Mb/s (on-the-wire traffic from host 1-2 up to this point)
- 40.3 Mb/s (on-the-wire traffic from host 2-1 up to this point)
- 40.3 Mb/s (on-the-wire traffic from host 2-2 up to this point)
_______________________________________________________
= 88.8 Mb/s (capacity left on the HS secondary shaper 1 up to this point)
40.3Mb/s corresponds to 39.6Mb/s converted to L2 rate.
Spare capacity for HS secondary shaper ‟1” is calculated based on on-the-wire rates while the subscriber aggregate capacity is calculated in L2 rates because the aggregate rate limit in HS secondary shaper in SR OS is configured in on-the-wire rates while the subscriber aggregate rates are configured in L2 rates.
Considering that each host has so far received 40.3Mb/s (on-the-wire rate), the 88.8Mb/s left on the HS secondary shaper will be distributed between scheduling classes 2 and 1 for each of the hosts (1-1, 1-2, 2-1, and 2-2) in the following manner:
Host 1-1, limited by its configured aggregate rate limit of 50Mb/s (L2 rate) will receive (50Mb/s – 39.6Mb/s =) 10.4Mb/s.
The amount of bandwidth that host 1-2 receives is limited by the sub-1 configured aggregate rate limit (100Mb/s). Because host 1-1 and host 1-2 belong to the same subscriber (sub-1) and host 1-1 is limited by its own configured limit of 50Mb/s, this will leave 50Mb/s of aggregate bandwidth for host 2-1. That is, host 1-2 is not limited by its own configured aggregate rate (70Mb/s) but by that of the sub-1, which is now congested. Therefore, host 1-2 will receive (50Mb/s – 39.6Mb/s =) 10.4Mb/s of bandwidth on scheduling classes 2 and 1.
The remaining 66.2Mb/s (L2 rate) is delegated to sub-2, and this will not be enough to congest any aggregate rate at the sub-2 level. Therefore, this bandwidth will be equally divided over host 2-1 and 2-2, each receiving 33.1Mb/s (L2 rate) on scheduling classes 2 and 1.
Considering the preceding bandwidth distribution for scheduling classes 2 and 1 for all four hosts and the 4:1 weight ratio between scheduling classes 2 and 1, the following conclusion can be reached:
Host 1-1 → 10.4Mb/s in 4:1 ratio:
scheduling class 2 will receive 8.32Mb/s
scheduling class 1 will receive 2.08Mb/s
Host 1-2 → 10.4Mb/s in 4:1 ratio:
scheduling class 2 will receive 8.32Mb/s
scheduling class 1 will receive 2.08Mb/s
Host 2-1 → 33.1Mb/s in 4:1 ratio:
scheduling class 2 will receive 26.5Mb/s
scheduling class 1 will receive 6.62Mb/s
Host 2-2 → 33.1Mb/s in 4:1 ratio:
scheduling class 2 will receive 26.5Mb/s
scheduling class 1 will receive 6.62Mb/s
These numbers match the rates measured by the traffic generator for streams <1-1-2>, <1-1-1>, <1-2-2>, <1-2-1>, <2-1-2>, <2-1-1>, <2-2-2>, <2-2-1> in Input/Output Rates.
Conclusion
This chapter described traffic management capabilities and configuration of HSQ IOM with ESM in expanded SLA mode. This chapter is an extension of the chapter High Scale QoS IOM in ESM Context: Single SLA Mode. Both single and expanded SLA modes rely on the unique properties of HSQ IOM, while each mode has a unique set of characteristics:
Single SLA mode provides higher subscriber scale per HSQ IOM, but allows only one SLA profile instance (SPI) per subscriber. This means that only a single SAP per subscriber (or residence) is supported (service per SAP model is not supported).
In contrast, expanded SLA mode supports multiple SPIs per subscriber (and SAPs), but with the reduced subscriber scale. The reduction in subscriber scale is caused by the tie-in between the subscribers and HS primary shapers. That is, the aggregate subscriber rate is enforced through HS primary shapers, which means that the mapping between a subscriber and an HS primary shaper is 1:1 Therefore, 16k HS primary shapers on HSQ will determine the subscriber scale in expanded SLA mode.
Note:On HSQ, a subscriber cannot be dissociated from an HS primary shaper, or from the QoS hierarchy.
Expanded SLA Mode can be used in service per SAP deployments where each subscriber is assigned to multiple SAPs, or it can be used on a single SAP where various services require that each of them is assigned its own HSQ queue group (or SPI).
HSQ IOM is a module of choice in an environment that demands high performance (200Gb/s per IOM) in combination with QoS functionality and a high number of egress queues.