TCAM allocation on SR Linux devices

Static TCAM allocation

Static TCAM refers to TCAM slices that are allocated at initialization and remain constantly reserved for their allocated purpose. Each static TCAM slice has a type, which refers to the type or types of entries that it stores. Examples of different entry types are IPv4 ingress ACL, IPv6 ingress ACL, and so on. A static TCAM slice consumes resources even if has no entries.

When the system is initialized (for example, after a restart) static TCAM slices are allocated first, before the configuration is evaluated and dynamic TCAM slices are created.

Dynamic TCAM allocation

Dynamic TCAM refers to TCAM slices that are taken from a free or unused pool of TCAM slices and released back to that pool when configuration determines that they are no longer needed.

Each dynamic TCAM slice has a type, which refers to the type or types of entries that it stores. When the configuration of the device has no entries of the type associated with a particular dynamic TCAM slice, there are no allocated slices of that type. The device does not hold onto empty TCAM banks to guarantee a minimum reservation.

The addition of the first entry associated with a particular dynamic TCAM slice attempts to create the required group of slices (slice group). Depending on the entry type (and resulting TCAM key length) the size of the slice group can be one TCAM slice (single-wide TCAM slice type), two TCAM slices (double-wide TCAM slice type), or three TCAM slices (triple-wide TCAM slice type). If a configuration change adds many entries of a new type, multiple slice groups may be needed.

Subsequent configuration changes may add more entries of a type that already has dynamic TCAM slices allocated. When the net number of new entries of that type (that is, additional entries minus deleted entries) exceeds the limit of the current allocation (after filling all empty holes in existing allocated slice groups), the system attempts to allocate new slice groups. There is no artificially imposed limit on the maximum number of slices that can be consumed for an entry type.

If a configuration change requires additional dynamic TCAM slices, and there are not enough free slices to accommodate the new slice groups, even if existing groups are moved to satisfy system constraints on the placement of groups, then the configuration commit is blocked.

If a configuration change requires additional dynamic TCAM slices, and there are enough free slices to accommodate the new slice groups, but only if existing groups are moved to satisfy system constraints on the placement of groups, then the commit is allowed and the TCAM slice relocation operations are initiated automatically. Moving a bank to a new location can take two or three minutes, so an entire operation involving multiple banks could delay the programming of the new entries for up to 20 minutes or more.

The deletion of the last entry associated with a particular dynamic slice group deletes the group and makes the freed slices available to other slice types. If a single configuration commit deletes entries of one type and adds entries of another type, the deleted entries (and any reclamation of slices) are processed first, so that there is a greater chance of successfully allocating the additional slices.

The deletion of some (but not all) entries of a specific type that already has dynamic TCAM slices allocated can leave empty entries in these TCAM slices. This could create a situation where there are N allocated slices for a specific entry type, but considering the empty entries in each of these N slices, fewer than N slices would be needed if the used entries could be moved between slices to consolidate them. This process, called slice compaction, is done automatically during the processing of a configuration transaction that results in a net deletion of entries; it should not be service impacting.

The maximum number of statistics resources that is available to dynamic TCAM entry types remains fixed, and does not scale to the maximum possible size for an entry type. If an ACL rule is configured to have per-entry statistics, but a statistics index resource cannot be allocated, this is indicated by statistics-resource-allocated = false in the YANG state for the entry.

Displaying static and dynamic TCAM usage

Use the info from state platform command to display the number of free static and dynamic TCAM entries available to each type of ACL. For example:

--{ running }--[  ]--
# info from state platform linecard 1 forwarding-complex 0 tcam resource *
    platform {
        linecard 1 {
            forwarding-complex 0 {
                tcam {
                    resource if-input-ipv4 {
                        free-static 6912
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource if-input-ipv4-qos {
                        free-static 6904
                        free-dynamic 0
                        reserved 8
                        programmed 8
                    }
                    resource if-input-ipv6 {
                        free-static 2304
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource if-input-ipv6-qos {
                        free-static 2262
                        free-dynamic 0
                        reserved 42
                        programmed 42
                    }
                    resource if-input-mac {
                        free-static 2304
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource if-input-policer {
                        free-static 1536
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource if-output-cpm-ipv4 {
                        free-static 1536
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource if-output-cpm-ipv6 {
                        free-static 512
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource if-output-cpm-mac {
                        free-static 1536
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource system-capture-ipv4 {
                        free-static 768
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                    resource system-capture-ipv6 {
                        free-static 384
                        free-dynamic 0
                        reserved 0
                        programmed 0
                    }
                }
            }
        }
    }

The free-static statistic is the number of free entries assuming the number of TCAM slices that are currently allocated to the type of entry remains constant.

The free-dynamic statistic is the number of free entries that are possible if all remaining unused TCAM slices are dynamically assigned to this one type of entry.