Egress scheduling

On 7730 SXR platforms, the traffic management unit (TMU) uses egress schedulers to manage the distribution of scheduling resources available to process egress traffic. Two configurable egress scheduling policies are supported:

  • Queue scheduling policy (at subinterface [CVLAN] and interface-queue level)
  • Scheduling class scheduling policy (at interface level)

The following diagram shows the flow of traffic from queues through example queue schedulers and scheduling class schedulers.

Figure 1. Queue scheduling policy and scheduling class scheduling policy

Queue scheduling policy (at subinterface [CVLAN] and interface-queue level)

Queue scheduling policies manage the distribution of scheduling resources available to process traffic in egress queues. The following figure illustrates the default queue scheduling policy.

Figure 2. Default queue scheduling policy

The queue scheduling policies define the following hierarchy of scheduling tiers:

  • Tier 0: Up to 12 schedulers, each of which can provide shaping for single or multiple queues. The egress queues are processed initially by tier 0 schedulers, which in turn feed into the tier 1 scheduler.
  • Tier 1: One scheduler, which provides aggregate-level shaping for all queues.

A total of 13 tier 0 and tier 1 schedulers can be defined. Within a queue scheduling policy, each defined queue must be mapped as an input to one of the tier 0 schedulers. The tier 0 schedulers are in turn mapped as inputs to the tier 1 scheduler. For each queue scheduler, shaper bucket thresholds determine how the scheduling resources are distributed by the scheduler.

The queue scheduling policies are applied at subinterface-level (such as CVLANs) to manage the defined subinterface queues. They are also applied at interface-level for any traffic that does not match any policies defined for subinterfaces.

The queue scheduling policies also support the mapping of each of the 12 egress queues to one of four scheduling classes: 0, 2, 4, and 6. If multiple queues are assigned to the same tier 0 scheduler, they must also be assigned to the same scheduling class.

The queues within a tier 0 scheduler draw bandwidth proportional to their weights. The tier 0 scheduler draws bandwidth based on the scheduling class. The tier 0 schedulers with higher scheduling classes are scheduled ahead of the lower scheduling classes and are capped by their rate. The scheduler rate can be defined as an absolute value (from 64Kb/s to 400Gb/s), or as a percentage value of the line rate.

The interface-level scheduling class schedulers provide further shaping.

Scheduling class scheduling policy (at interface level)

Scheduling class scheduling policies manage the distribution of scheduling resources available to process traffic in scheduling classes. The following figure illustrates the default scheduling class scheduling policy.

Figure 3. Default scheduling class scheduling policy

Similar to queue scheduling policies, the scheduling class scheduling policies define the following hierarchy of scheduling tiers:

  • Tier 0: Up to 8 schedulers, each of which can provide shaping for single or multiple scheduling classes exiting the queue schedulers. The scheduling classes are processed initially by tier 0 schedulers, which in turn feed into the tier 1 scheduler.
  • Tier 1: One scheduler, which provides aggregate-level shaping for all scheduling classes.

A total of 9 tier 0 and tier 1 schedulers can be defined. Also similar to queue schedulers, shaper bucket thresholds determine how the scheduling resources are distributed by the scheduler within each scheduling class scheduler.

The scheduling class scheduling policies are applied at interface level.

Queue and scheduling class policies applied to interface and subinterface

The following diagram shows the flow of packet processing through the queue and scheduling class scheduler policies applied to a subinterface and to an interface.

Figure 4. Applied queue and scheduling class policies

When a packet arrives on a subinterface for queueing at egress, the system performs the following steps.

  1. Compare the packet FC (derived from one of the ingress classification policies) against the subinterface queue policies.

    • If a match is found, assign a queue for the packet based on the output class map policy.
    • If no match is found, assign the packet to a queue based on the interface-level FC-to-queue mapping.
  2. The queues are scheduled based on the configured queue and scheduling class policies. The tier-1 scheduler allocates available bandwidth to its children serving the higher scheduling classes first and allocating the remaining bandwidth until none remains. And within a scheduling class, the available bandwidth is allocated to children in a weighted fair manner.

Queue scheduling policy example

The queue scheduling hierarchy consists of queues feeding multiple tier 0 schedulers which in turn feed into a single tier 1 scheduler that aggregates all queues from tier 0.

The following example shows the queue scheduling policy configuration for three queues, where scheduler 4 in tier 1 is configured with the inputs auto-input parameter, which indicates that all tier 0 schedulers are inputs to scheduler 4. (The auto-input option cannot be set for a tier 0 scheduler.)

--{ candidate shared default }--[  ]--
# info qos scheduler-policies queue-scheduling-policy test-sub-interface
    qos {
        scheduler-policies {
            queue-scheduling-policy test-sub-interface {
                queue af1-queue {
                    scheduling {
                        scheduling-class 4
                        weight 2
                        packet-length-adjustment {
                            add 0
                        }
                    }
                }
                queue af2-queue {
                    scheduling {
                        scheduling-class 4
                        weight 4
                        packet-length-adjustment {
                            add 0
                        }
                    }
                }
                queue be-queue {
                    scheduling {
                        scheduling-class 0
                        packet-length-adjustment {
                            add 0
                        }
                    }
                }
                queue ef-queue {
                    scheduling {
                        scheduling-class 6
                        packet-length-adjustment {
                            add 0
                        }
                    }
                }
                scheduler 0 {
                    tier 0
                    rate {
                        peak-rate-percentage 100
                    }
                    inputs {
                        queue [
                            be-queue
                        ]
                    }
                }
                scheduler 1 {
                    tier 0
                    rate {
                        peak-rate-percentage 80
                    }
                    inputs {
                        queue [
                            af1-queue
                            af2-queue
                        ]
                    }
                }
                scheduler 2 {
                    tier 0
                    rate {
                        peak-rate-percentage 20
                    }
                    inputs {
                        queue [
                            expedite-queue
                        ]
                    }
                }
                scheduler 4 {
                    tier 1
                    rate {
                        peak-rate-kbps 7000
                    }
                    inputs {
                        inputs auto-input
                    }
                }
            }
        }
    }

Scheduling class scheduling policy example

At interface-level, the scheduling hierarchy consists of scheduling classes feeding multiple tier 0 schedulers which in turn feed into a single tier 1 scheduler that aggregates all scheduling-classes from tier 0.

The following example shows the scheduling class scheduling policy configuration for three scheduling classes.

Similar to the queue policy, the tier 1 scheduler 2 is configured with the inputs auto-input parameter, which indicates that all tier 0 schedulers are inputs to this scheduler. (If the auto-input option is set for a tier 0 scheduler, all defined queues are inputs to this scheduler. In that case, none of other tier 0 schedulers can have queues defined as explicit inputs.)

--{ candidate shared default }--[  ]--
# info qos scheduler-policies sched-class-scheduling-policy sched-class-scheduling-policy-name
    qos {
        scheduler-policies {
            sched-class-scheduling-policy sched-class-scheduling-policy-name {
                scheduler 1 {
                    tier 0
                    rate {
                        peak-rate-kbps 10000
                        pir-adaptation-rule closest
                    }
                    inputs {
                        scheduling-class [
                            0
                            4
                        ]
                    }
                }
                scheduler 2 {
                    tier 1
                    rate {
                        peak-rate-kbps 10000
                        pir-adaptation-rule closest
                    }
                    inputs {
                        inputs auto-input
                    }
                }
                scheduler 3 {
                    tier 0
                    rate {
                        peak-rate-kbps 2000
                        pir-adaptation-rule closest
                    }
                    inputs {
                        scheduling-class [
                            6
                        ]
                    }
                }
            }
        }
    }

Default scheduling policy settings

Default interface queues

The following table shows the default settings applied to the interface-level queues. At interface level, all 12 queues are created by default. The table lists default settings for all queues, including the scheduling-related parameters.

Table 1. Default interface queues
default-queue-name queue-index committed-burst-size maximum-burst-size peak-rate-percent weight scheduling-class
queue-0 0 default max 100 1 0
queue-1 1 default max 100 1 0
queue-2 2 default max 100 1 2
queue-3 3 default max 100 1 2
queue-4 4 default max 100 1 4
queue-5 5 default max 100 1 4
queue-6 6 default max 100 1 6
queue-7 7 default max 100 1 6
queue-8 8 default max 100 1 0
queue-9 9 default max 100 1 0
queue-10 10 default max 100 1 0
queue-11 11 default max 100 1 0

To change the queue scheduling parameters, assign a custom queue scheduling policy to the interface. Similarly, to modify default scheduling behavior between scheduling classes at the interface level, assign a custom scheduling class scheduling policy to an interface.

Default subinterface queues

At subinterface level, local queues are created only after an output class map is assigned to the subinterface. By default, the queue parameters are set as described in the interface table above, using the queue-index as a key. To change queue scheduling parameters, assign a custom queue scheduling policy to the subinterface.

Default scheduling threshold policies

By default, the system provides the following two non-configurable scheduling threshold policies:

  • per-scheduling-class
  • single

The following diagram shows the default scheduling threshold policies. Both tier 0 schedulers are assigned the default scheduling-threshold-policy single, while both tier 1 schedulers are assigned the default scheduling-threshold-policy per-scheduling-class.

Figure 5. Default scheduling threshold policies

Custom-defined scheduling threshold policies are not supported.

Scheduling threshold policy: per-scheduling-class

The per-scheduling-class policy ensures that each scheduling class is assigned a distinct priority level within a shaper bucket.

The shaping of scheduling classes occurs in relation to the shaper token level. This process can be illustrated by considering scheduling class 3, which is associated with threshold level 3. If the token level is below threshold level 3, but above threshold level 2, queues associated with scheduling class 3 are shaped to enforce the overall rate of the bucket. Queues associated with scheduling class 3 are not shaped by this bucket if the shaper token level is at or above threhold level 3, or if the token level drops below threshold level 2.

The per-scheduling-class policy is assigned to tier 1 schedulers by default, both in the queue scheduling policies and the scheduling class scheduling policies. These settings are non-configurable.

The default per-scheduling-class policy is as follows:

qos {
   scheduling-policies {
      scheduling-threshold-policy "per-scheduling-class" {
         threshold 0 {
            maximum-scheduling-class 0
         }
         threshold 2 {
            maximum-scheduling-class 2
         }
         threshold 4 {
            maximum-scheduling-class 4
         }
         threshold 6 {
            maximum-scheduling-class 6
         }
      }
   }
}

Scheduling threshold policy: single

The single scheduling threshold policy places all scheduling classes on the first threshold level, avoiding multiple priority levels and the corresponding increases in burst tolerance.

The single policy is assigned to tier 0 schedulers by default, in both the queue scheduling policy and scheduling class scheduling policy. This setting is non-configurable.

qos {
   scheduling-policies {
      scheduling-threshold-policy "single" {
         threshold 0 {
            maximum-scheduling-class 6
         }
      }
   }
}

Modifying threshold level bucket depths

The size of the shaper buckets for all threshold levels used in the default scheduling threshold policies are defined in relation to the burst-allowance and threshold-separation parameters. These parameters are defined for queues and scheduling classes using queue scheduling policies and scheduling class scheduler policies.

Configuring queue scheduling policies

To configure queue scheduling policies, use the qos scheduler-policies queue-scheduling-policy command.

Configure a tier 0 queue scheduling policy

The following example shows a custom configuration for one queue and one scheduler in a tier 0 queue scheduling policy. The queue is assigned a scheduling class and weight, while the scheduler is assigned tier, burst, threshold-separation, and rate parameters and associated queues.

--{ * candidate shared default }--[  ]--
# info qos scheduler-policies queue-scheduling-policy custom-tier-0-queue-policy
    qos {
        scheduler-policies {
            queue-scheduling-policy custom-tier-0-queue-policy {
                queue queue-01 {
                    scheduling {
                        scheduling-class 0
                        weight 10
                    }
                }
                scheduler 0 {
                    tier 0
                    burst-allowance 10000
                    threshold-separation 28672
                    rate {
                        peak-rate-percentage 80
                    }
                    inputs {
                        queue [
                            queue-01
                        ]
                    }
                }
            }
        }
    }

Configure a tier 1 queue scheduling policy

The following example shows a custom configuration for all queues in a tier 1 queue scheduling policy. The scheduler is assigned a tier, burst, threshold-separation, and rate parameters. All queues egressing the tier 0 schedulers feed into this single tier 1 scheduler using the inputs auto-input option.

--{ candidate shared default }--[  ]--
# info qos scheduler-policies queue-scheduling-policy custom-tier-1-queue-policy
    qos {
        scheduler-policies {
            queue-scheduling-policy custom-tier-1-scheduling-policy {
                scheduler 14 {
                    tier 1
                    burst-allowance 11000
                    threshold-separation 28672
                    rate {
                        peak-rate-percentage 100
                    }
                    inputs {
                        inputs auto-input
                    }
                }
            }
        }
    }

Applying queue scheduling policies to subinterfaces

To apply queue scheduling policies to egress traffic on a subinterface, specify the required policy using the qos interfaces interface output scheduler command.

Apply queue scheduling policies to a subinterface

The following example shows a tier 0 queue scheduling policy applied to a subinterface.

--{ * candidate shared default }--[  ]--
# info qos interfaces
    qos {
        interfaces {
            interface eth-1/1.1 {
                interface-ref {
                    interface ethernet-1/1
                    subinterface 1
                }
                output {
                    scheduler {
                        queue-scheduling-policy custom-tier-0-queue-policy
                    }
                }
            }
        }
    }

Configuring scheduling class scheduling policies

To configure scheduling class scheduling policies, use the qos scheduler-policies sched-class-scheduling-policy <name> command.

Configure a tier 0 scheduling class scheduling policy

The following example shows a custom configuration for one scheduler included in a tier 0 scheduling class scheduling policy. The scheduler is assigned tier, burst, threshold-separation, and rate parameters, and associated scheduling classes.

--{ * candidate shared default }--[  ]--
# info qos scheduler-policies sched-class-scheduling-policy custom-tier-0-sched-class-policy
    qos {
        scheduler-policies {
            sched-class-scheduling-policy custom-tier-0-sched-class-policy {
                scheduler 0 {
                    tier 0
                    burst-allowance 10000
                    threshold-separation 28672
                    rate {
                        peak-rate-percentage 90
                    }
                    inputs {
                        scheduling-class [
                            0
                            2
                        ]
                    }
                }
            }
        }
    }

Configure a tier 1 scheduling class scheduling policy

The following example shows a custom configuration for a tier 1 scheduling class scheduling policy. The scheduler is assigned tier, burst, threshold-separation, and rate parameters. All scheduling classes egressing the tier 0 schedulers feed into this single tier 1 scheduler using the inputs auto-input option.

--{ candidate shared default }--[  ]--
# info qos scheduler-policies queue-scheduling-policy custom-tier-1-queue-policy
    qos {
        scheduler-policies {
            queue-scheduling-policy custom-tier-1-scheduling-policy {
                scheduler 14 {
                    tier 1
                    burst-allowance 11000
                    threshold-separation 28672
                    rate {
                        peak-rate-percentage 100
                    }
                    inputs {
                        inputs auto-input
                    }
                }
            }
        }
    }

Applying scheduling class scheduling policies to interfaces

To apply scheduling class scheduling policies to egress traffic on an interface, specify the required policy using the qos interfaces interface output scheduler command.

Apply scheduling class scheduling policies to an interface

The following example shows a tier 0 scheduling class scheduling policy applied to a an interface.

--{ * candidate shared default }--[  ]--
# info qos interfaces
    qos {
        interfaces {
            interface eth-1/1 {
                interface-ref {
                    interface ethernet-1/1
                }
                output {
                    scheduler {
                        sched-class-scheduling-policy custom-tier-0-sched-class-policy
                    }
                }
            }
        }
    }

Scheduling priority mapping table

The scheduling priority mapping table is a global table that maps individual scheduling classes into three scheduling priorities at the hardware level. Scheduling priority 0 is the lowest priority, and priority 2 is the highest.

The following table shows the default settings:

Scheduling class Scheduling priority
0 0
2 0
4 1
6 2

Configuring the scheduling priority mapping table

To configure the scheduling priority mapping table, use the qos scheduler-policies scheduling-priority-mapping-table command.

Configure scheduling priority mapping table

--{ * candidate shared default }--[  ]--
# info qos scheduler-policies scheduling-priority-mapping-table
    qos {
        scheduler-policies {
            scheduling-priority-mapping-table {
                scheduling-class 0 {
                    scheduling-priority 0
                }
                scheduling-class 2 {
                    scheduling-priority 0
                }
                scheduling-class 4 {
                    scheduling-priority 1
                }
                scheduling-class 6 {
                    scheduling-priority 2
                }
            }
        }
    }

Egress queue scheduling resource management

The resource management of output queues, tier 0 queue schedulers, and tier 1 queue schedulers is based on resource sets consisting of:

  • 12 output queues
  • 12 tier 0 queue schedulers
  • one tier 1 queue scheduler

Every subinterface that is created is allocated a resource set regardless of the number of local queues configured in its associated output class map and regardless of the number of schedulers configured in its associated queue scheduling policy. This is to prevent changes in any of the scheduling policies from causing depletion of resources for the subinterface.

The following diagram shows a resource set consisting of queues, tier 0 schedulers, and a single tier 1 scheduler.

Figure 6. Resource set

At the highest level, all resource sets are collected into two resource set pools. Each resource set pool contains 62 resource groups. And each resource groups contains 256 resource sets.

These resource groups are allocated to interface group resource pools, defining the interface-level resource sets that subinterfaces require to process queues.

At interface level, every interface is statically mapped to an interface group resource pool. The resource groups are dynamically allocated to the interface group resource pool based on demand. On an interface, all subinterfaces associated with the given interface group resource pool share the available resource sets.

The following figure shows the organization of resource sets to interfaces and subinterfaces.

Figure 7. Resource sets to interfaces and subinterfaces

At system initialization, every interface group resource pool is allocated a single resource group from the corresponding resource set pool.

As the individual subinterfaces are created, the resource sets are allocated from the corresponding resource group. If a new subinterface is created and all resource sets in a group are already allocated, a new resource group is allocated to the interface group resource pool. This allocation is persistent, such that when a resource group is allocated to an interface group resource pool, it remains allocated even if all corresponding resource sets become free.

If subinterface queues are mapped to interface queues as a result of exhaustion of available resources, the output-class-map-pending flag indicates the interfaces that failed to allocate the queue resources. To view this flag, use the info from state qos interfaces interface output output-class-map-pending command.

In the system state information, a container is available under scheduling-resources-pools listing the scheduling resource pools individual interfaces are using resources from. This mapping is static, but it is platform dependent.

--{ candidate shared default }--[ ] --
# info from state qos interfaces interface eth1 output scheduler
    scheduling-resources-pools {   
        resource-set-pool 1
        interface-group-resource-pool 12
    }

There is also a state table under qos resource-set-pool that tracks the utilization of the individual resource groups and their allocation to interface group resource pools.

--{ candidate shared default }--[ ] --
# info from state platform linecard 1 forwarding-complex 1 qos
    qos {
        ...
        resource-set-pool 0 {
        ...
        }
        resource-set-pool 1 {
           interface-group-resource-pool 8 {
               resource-group 0 {
                   resource-set-used 253
                   resource-set-free 3
               }
           }
           interface-group-resource-pool 9 {
               resource-group 1 {
                   resource-set-used 0
                   resource-set-free 256
               }
           }
            interface-group-resource-pool 10 {
               resource-group 2 {
                   resource-set-used 0
                   resource-set-free 256
               }
           }
           interface-group-resource-pool 11 {
               resource-group 3 {
                   resource-set-used 255
                   resource-set-free 1
               }
               resource-group 8 {
                   resource-set-used 1
                   resource-set-free 255
               }
           }
           ...
           interface-group-resource-pool 15 {
               resource-group 7 {
                   resource-set-used 255
                   resource-set-free 1
               }
           }
        }
    }

Each configured subinterface requires one resource set at minimum. However, if the subinterface is created on a LAG, one resource set per LAG member is allocated. Depending on the configuration, this can lead to depletion of the resources. In such a case, the subinterface points to interface-based queues (remote-output-queue) overriding any output class map configuration. If the resource set in the corresponding interface group resource pool becomes available, the system automatically assigns it to the subinterface waiting for such a resource set.