SR Linux applications

The SR Linux is a suite of modular, lightweight applications running like any others in a Linux environment. Each SR Linux application supports a different protocol or function, such as BGP, LLDP, AAA, and so on. These applications use gRPC and APIs to communicate with each other and external systems over TCP.

One SR Linux application, the application manager (app_mgr), is itself responsible for monitoring the health of the process IDs running each SR Linux applications, and restarting them if they fail. The application manager reads in application-specific YAML configuration and YANG models, and starts each application (or allows an application not to start if there no configuration exists for it). There is an instance of the app_mgr that handles applications running on the CPM, and an instance of the app_mgr on each IMM that handles applications running on the line card.

In addition to the Nokia-provided SR Linux applications, the SR Linux supports installation of user-defined applications, which are managed and configured in the same way as the default SR Linux applications. User-defined SR Linux applications can be loaded, reloaded, and unloaded from the system as necessary.

Installing an application

To install an application, copy the application files into the appropriate SR Linux directories, then reload the application manager and start the application.

The example in this topic installs an application called fib_agent. The application consists of files named fib_agent.yml, fib_agent.sh, fib_agent.py, and fib_agent.yang. The fib_agent.yml file is installed in the /etc/opt/srlinux/appmgr/ directory. The .yml file for a user-defined application must reside in this directory in order for the app_mgr to read its YAML configuration.

The .yml file defines the locations of the other application files. The other application files can reside anywhere in the system other than in the /opt/srlinux/ directory or any tempfs file system.

In this example, the fib_agent.sh and fib_agent.py files are installed in the directory /user_agents/, and the fib_agent.yang file is installed in the directory/yang/. The locations for these files are defined in the fib_agent.yml file.

  1. Copy the application files into the SR Linux directories.
    --{ candidate }--[  ]--
    # bash
    # cp fib_agent.yml /etc/opt/srlinux/appmgr/.
    # cp fib_agent.sh /user_agents/.
    # cp fib_agent.py /user_agents/.
    # cp fib_agent.yang /yang/.
    # exit
  2. From the SR Linux CLI, reload the application manager.
    --{ candidate }--[ ]--
    # tools system app-management application app_mgr reload
    
  3. Apply the changes to the configuration.
    --{ candidate }--[  ]--
    # fib-agent
    --{ candidate }--[ fib-agent ]--
    # commit stay
    All changes have been committed. Starting new transaction.
    --{ candidate }--[ fib-agent ]--
    
  4. Verify that the application is running.
    # show system application fib_agent
      +-----------+-----+---------+-----------------------+--------------------------+
      |  Name     | PID |  State  |        Version        |       Last Change        |
      +===========+=====+=========+=======================+==========================+
      | fib_agent | 227 | running | v21.3.0-61-gd19567393 | 2021-01-13T20:16:45.697Z |
      +-----------+-----+---------+-----------------------+--------------------------+
    

Starting an application

To start an SR Linux application instance, use the start option in the tools system app-management command.

To start an SR Linux application instance:

--{ running }--[  ]--
# tools system app-management application mpls_mgr start
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was started

Restarting applications

SR Linux applications that are currently running can be restarted; that is, stopped then started again. SR Linux supports two types of application restarts: warm restart and cold restart. The difference between the two types involves how the application state is maintained in the IDB:

  • Warm restart

    Warm restart allows forwarding to continue based on the previous state of the application. A warm restart causes the application to leave its state information in IDB during the restart, then recover it after the restart. This restart type results in less disruption to surrounding applications that depend on the restarting application's state.

  • Cold restart

    Cold restart results in a normal stop and start of the application, including purging its state in IDB.

Not all SR Linux applications support warm restart. For a list of applications that support warm restart, see the SR Linux Release Notes. You can also determine if an application supports warm restart by using an info from state command; for example:
# info from state system app-management application chassis_mgr supported-restart-types
    system {
        app-management {
            application chassis_mgr {
                supported-restart-types [
                    cold
                    warm
                ]
            }
        }
    }

Applications can be restarted automatically by app_mgr (for example, if an application fails and needs to be restarted) or you can restart an application manually from the CLI using a tools command. When app_mgr restarts an application, the application is warm-restarted if the application supports it; otherwise the application is cold-restarted. If you restart an application using a tools command, you can specify whether the application is cold or warm restarted. If you do not specify the restart type, the application is warm-restarted if the application supports it; otherwise the application is cold-restarted.

Restarting an application

You can restart an application from the SR Linux CLI. The application can be cold-restarted or warm-restarted (if the application supports warm-restart). Using an info from state command, you can display the type of restart (cold or warm) that was used the last time the application was restarted.

Restart an SR Linux application instance

--{ running }--[  ]--
# tools system app-management application chassis_mgr restart cold
/system/app-management/application[name=chassis_mgr]:
    Application 'chassis_mgr' was killed with signal 9

/system/app-management/application[name=chassis_mgr]:
    Application 'chassis_mgr' was restarted

Display the type of restart used the last time the application was restarted

--{ running }--[  ]--
# info from state system app-management application chassis_mgr last-start-type
    system {
        app-management {
            application chassis_mgr {
                last-start-type cold
            }
        }
    }

Terminating an application

You can use the stop, quit, or kill options in the tools system app-management command to terminate an SR Linux application.

  • stop gracefully terminates the application, allowing it to clean up before exiting.

  • quit terminates the application and generates a core dump. The core dump files are saved in the /var/log/srlinux/cores/ directory.

  • kill terminates the application immediately, without allowing it to clean up before exiting.

Terminate an application gracefully

--{ candidate }--[  ]--
# tools system app-management application mpls_mgr stop
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was killed with signal 15

Terminate an application and generate a core dump

--{ candidate }--[  ]--
# tools system app-management application mpls_mgr quit
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was killed with signal 3

Terminate an application immediately

--{ candidate }--[  ]--
# tools system app-management application mpls_mgr kill
/system/app-management/application[name=mpls_mgr]:
    Application 'mpls_mgr' was killed with signal 9

Reloading application configuration

To reload the application configuration, reload the application manager. Reloading the application manager causes it to process any changes in the installed applications' YAML configuration and restart the applications. Applications that are no longer present in the system are stopped, and their YANG modules are removed from the management server. Applications removed from the system have their nodes or augmentations removed from the system schema.

--{ * running }--[  ]--
# tools system app-management application app_mgr reload

Clearing application statistics

You can display statistics collected for an application with the info from state command. To reset the statistics counters for the application, use the statistics clear option in the tools system app-management command.

To reset the statistics counters for an application:

--{ running }--[  ]--
# tools system app-management application mpls_mgr statistics clear

Restricted operations for applications

An application may have one or more operations that are restricted by default. For example, the linux_mgr application has stop, quit, and kill as restricted operations, meaning that these options are not available when entering the tools system app-management command for the linux_mgr application.

Restricted operations for SR Linux applications lists the restricted operations for each SR Linux application.

Table 1. Restricted operations for SR Linux applications

Application

Restricted operations

aaa_mgr

reload

acl_mgr

reload

app_mgr

start, stop, restart, quit, kill

arp_nd_mgr

reload

bfd_mgr

reload

bgp_mgr

reload

chassis_mgr

stop, quit, kill, reload

device_mgr

reload

dhcp_client_mgr

stop, reload

fib_mgr

reload

gnmi_server

reload

idb_server

start, stop, restart, quit, kill, reload

json_rpc_config

reload

linux_mgr

stop, quit, kill

lldp_mgr

reload

log_mgr

reload

mgmt_server

start, stop, quit, kill, reload

mpls_mgr

reload

net_inst_mgr

start, stop, quit, kill, reload

oam_mgr

reload

plcy_mgr

reload

qos_mgr

reload

sdk_mgr

reload

static_route_mgr

reload

supportd

reload

xdp_cpm

stop, quit, kill, reload

xdp_lc

reload

Restricted options are specified in the restricted-operations setting in the YAML file for the application.

Configuring an application

To configure an SR Linux application, edit settings in the application YAML file, then reload the application manager to activate the changes.

The example in this section shows how to configure an application to specify the action the SR Linux device takes when the application fails. If an SR Linux application fails a specified number of times over a specified time period, the SR Linux device can reboot the system or attempt to restart the application after waiting a specified number of seconds.

For example, if the aaa_mgr application crashes 5 times within a 500-second window, the SR Linux device can be configured to wait 100 seconds, then restart the aaa_mgr application.

The following actions can be taken if an SR Linux application fails:

  • Reboot the system

  • Wait a specified number of seconds, then attempt to restart the failed application

  • Move the failed application to error state without rebooting the system or attempting to restart the application

If you stop or restart an application using the tools system app-management command in the SR Linux CLI, it is not considered an application failure; the failure action for the application, if one is configured, does not occur. However, if the failed application waits a specified period of time (or forever) to be restarted, or has been moved into error state, you can restart the application manually with the tools system app-management application restart CLI command.

To configure the failure action for an application:

  1. Check the status of the SR Linux applications:
    --{ running }--[  ]--
    # show system application
    +--------------+-----+--------------------+-----------------------+--------------------------+
    |    Name      | PID |       State        |        Version        |       Last Change        |
    +==============+=====+====================+=======================+==========================+
    | aaa_mgr      | 242 | error              | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.967Z |
    | acl_mgr      | 193 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.967Z |
    | app_mgr      | 118 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:11.161Z |
    | arp_nd_mgr   | 104 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.967Z |
    | bfd_mgr      |     | waiting-for-config |                       |                          |
    | bgp_mgr      | 109 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:13.156Z |
    | chassis_mgr  | 115 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.967Z |
    | dev_mgr      | 150 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.001Z |
    | fib_mgr      | 168 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.968Z |
    | gnmi_server  | 157 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:13.296Z |
    | idb_server   | 171 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.228Z |
    | igmp_mgr     |     | waiting-for-config |                       |                          |
    | isis_mgr     |     | waiting-for-config |                       |                          |
    | json_rpc     | 166 | running            |                       | 2022-01-21T20:15:13.234Z |
    | label_mgr    | 139 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:13.186Z |
    | lag_mgr      | 103 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.968Z |
    | ldp_mgr      |     | waiting-for-config |                       |                          |
    | linux_mgr    | 116 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.968Z |
    | lldp_mgr     | 149 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:13.206Z |
    | log_mgr      | 127 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.968Z |
    | mgmt_server  | 149 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.968Z |
    | mirror_mgr   |     | waiting-for-config |                       |                          |
    | mpls_mgr     |     | waiting-for-config |                       |                          |
    | net_inst_mgr | 160 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.968Z |
    | oam_mgr      | 160 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:13.231Z |
    | ospf_mgr     |     | waiting-for-config |                       |                          |
    | p4rt_server  | 641 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T21:56:47.935Z |
    | plcy_mgr     | 177 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:13.242Z |
    | qos_mgr      | 186 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:13.322Z |
    | sshd-mgmt    | 192 | running            |                       | 2022-01-21T20:15:19.710Z |
    | xdp_cpm      | 197 | running            | v22.3.0-34-ge0ee326f8 | 2022-01-21T20:15:10.968Z |
    | xdp_lc_1     | 108 | running            |                       | 2022-01-21T20:15:10.968Z |
    +--------------+-----+--------------------+-----------------------+--------------------------+
  2. Use the info from state command to check the current failure action settings for the application to configure. These settings are highlighted in the following example:
    --{ running }--[  ]--
    # info from state system app-management application aaa_mgr
        system {
            app-management {
                application aaa_mgr {
                    pid 242
                    state error
                    last-change 2022-01-21T20:15:10.967Z
                    author Nokia
                    failure-threshold 3
    failure-window 300
    failure-action reboot
                    path /opt/srlinux/bin
                    launch-command ./sr_aaa_mgr
                    search-command ./sr_aaa_mgr
                    version v22.3.0-34-ge0ee326f8
                    restricted-operations [
                        reload
                    ]
                    statistics {
                        restart-count 0
                    }
                    yang {
                        modules [
                            srl_nokia-aaa
                            srl_nokia-aaa-types
                        ]
                        source-directories [
                            /opt/srlinux/models/ietf
                            /opt/srlinux/models/srl_nokia/models/common
                            /opt/srlinux/models/srl_nokia/models/network-instance
                            /opt/srlinux/models/srl_nokia/models/system
                        ]
                    }
                }
            }
        }

    The following failure action settings can be configured for an application:

    • failure-threshold: number of times that the application must fail during the failure-window period before the failure-action is taken; the default is three times.

    • failure-window: number of seconds over which the application must fail the failure-threshold number of times before the failure-action is taken; the default is 300 seconds.

    • failure-action: action to take if the application fails failure-threshold times over failure-window seconds. This can be one of the following:

      • reboot: reboot the system; this is the default failure-action.

      • wait=seconds: wait this number of seconds, then attempt to restart the application.

      • wait=forever: move the application to error state and do not reboot the system or attempt to restart the application.

  3. Edit the YAML configuration for the application.

    The YAML configuration files for SR Linux applications are located in the directory /opt/srlinux/appmgr on the SR Linux device. They are named sr_application_name_config.yml; for example, sr_aaa_mgr_config.yml.

  4. In the .yml file, add or change the settings for the failure-threshold, failure-window, and failure-action parameters.

    In the following example, the failure action settings in the sr_aaa_mgr_config.yml file are configured so that if the aaa_mgr application fails 5 times over a 500-second period, the SR Linux device waits 100 seconds, then attempts to restart the aaa_mgr application:

    aaa_mgr_setup:
       path: /opt/srlinux/bin
       launch-command: ./aaamgr_set_env.sh
       run-as-user: root
       never-show: Yes
       never-restart: Yes
       start-order: 1
    aaa_mgr:
       path: /opt/srlinux/bin
       launch-command:  ./sr_aaa_mgr
       search-command: ./sr_aaa_mgr
       run-as-user: root
       restricted-operations: ['reload']
       failure-threshold: 5
    failure-window : 500
    failure-action: "wait=100"
       yang-modules:
           names:
               - "srl_nokia-aaa"
               - "srl_nokia-aaa-types"
           source-directories:
               - "/opt/srlinux/models/ietf"
               - "/opt/srlinux/models/srl_nokia/models/common"
               - "/opt/srlinux/models/srl_nokia/models/system"
               - "/opt/srlinux/models/srl_nokia/models/network-instance"
  5. Save and close the .yml configuration file.
  6. In the SR Linux CLI, reload the application manager:
    --{ running }--[  ]--
    # tools system app-management application app_mgr reload
    This command reloads any application whose .yml configuration file has changed. It does not affect any service.
  7. Use the info from state command to verify that the changes to the failure action settings are now in effect.
    --{ running }--[  ]--
    # info from state system app-management application aaa_mgr
        system {
            app-management {
                application aaa_mgr {
                    pid 242
                    state running
                    last-change 2022-01-21T20:15:10.967Z
                    author Nokia
                    failure-threshold 5
    failure-window 500
    failure-action wait=100
                    path /opt/srlinux/bin
                    launch-command ./sr_aaa_mgr
                    search-command ./sr_aaa_mgr
                    version v22.3.0-34-ge0ee326f8
                    restricted-operations [
                        reload
                    ]
                    statistics {
                        restart-count 0
                    }
                    yang {
                        modules [
                            srl_nokia-aaa
                            srl_nokia-aaa-types
                        ]
                        source-directories [
                            /opt/srlinux/models/ietf
                            /opt/srlinux/models/srl_nokia/models/common
                            /opt/srlinux/models/srl_nokia/models/network-instance
                            /opt/srlinux/models/srl_nokia/models/system
                        ]
                    }
                }
            }
        }

Removing an application from the system

To remove an application from the system, remove the application files from the SR Linux directories where they reside, then reload the application manager.

When an application is removed from the system, SR Linux stops sending updates for the paths that the application would populate. If the application is subsequently reloaded, SR Linux resumes sending updates.

For active CLI sessions, the schema is updated to remove the application. If an active CLI session exists in a context that is no longer present because of the application being removed, the CLI context is changed to the next-highest valid context.

User-defined applications can be removed from the system, but removing Nokia-provided SR Linux applications is not supported.

In the following example, the fib_agent application, consisting of files named fib_agent.yml, fib_agent.sh, fib_agent.py, and fib_agent.yang, is removed from the system.

  1. Remove the application files from the SR Linux directories.
    --{ candidate }--[  ]--
    # bash
    # rm /etc/opt/srlinux/appmgr/fib_agent.yml
    # rm /user_agents/fib_agent.sh
    # rm /user_agents/fib_agent.py
    # rm /yang/fib_agent.yang
    # exit
  2. From the SR Linux CLI, reload the application manager.
    --{ candidate }--[ ]--
    # tools system app-management application app_mgr reload
    
    The application manager stops any application that is no longer present, and removes the application's YANG module from the management server.

Partioning and isolating application resources

The SR Linux protects system processes through the use of control groups (cgroups), which impose resource consumption limits on resource-intensive customer applications.

Cgroup profiles

Cgroup profiles define how usage limits are applied. On the SR Linux, cgroup profiles are supported for CPU and memory and are defined in cgroup_profile.json configuration files.

SR Linux provides a default cgroup profile; customers can configure additional cgroup profiles.

Default cgroup profile

The SR Linux-provided default cgroup profile is located in the /opt/srlinux/appmgr/cgroup_profile.json directory.

Note: Editing the default cgroup profile is not recommended.

If the default cgroup profile fails to parse or be read by the app_mgr, the SR Linux does not start.

The default cgroup_profile.json file definition is shown below:

{
 "profiles": [
  {
   "name": "workload.primary",
   "path": "workload.slice/primary",
   "controller": {
    "memory": {
     "max": 0.8,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "10000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "",
     "mems": ""
    }
   }
  },
  {
   "name": "workload.datapath",
   "path": "workload.slice/datapath",
   "controller": {
    "memory": {
     "max": 0.8,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "10000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "all",
     "mems": ""
    }
   }
  },
  {
   "name": "workload.secondary",
   "path": "workload.slice/secondary",
   "controller": {
    "memory": {
     "max": 0.5,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "10000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "",
     "mems": ""
    }
   }
  },
  {
   "name": "user.default",
   "path": "user.slice/default",
   "controller": {
    "memory": {
     "max": 0.25,
     "swap_max": 0,
     "low": 0
    },
    "cpu": {
     "weight": "1000",
     "period": "100000",
     "quota": "0"
    },
    "cpuset": {
     "cpus": "",
     "mems": ""
    }
   }
  }
 ]
}

Default cgroup profile parameters describes the default cgroup profile parameters.

Table 2. Default cgroup profile parameters

Parameter

Description

name

The cgroup profile name.

Type: string

path

The cgroup directory path relative to a unified root path. A typical unified root path is /sys/fs/cgroup or /mnt/cgroup/unified

Type: string

controller

The memory controller configuration.

max This number denotes the percentage of total memory. The actual memory value is calculated as (max. × total_memory) and is set in the memory.max interface file of the cgroup. If the value is 0, this configuration is ignored. The range is from 0 to 1, the default is 0.8.

low This number denotes the percentage of total memory. The actual memory value is calculated as (max. × total_memory) and is set in the memory.low interface file of the cgroup. The range is from 0 to 1, the default is 0.8.

cpu

The CPU controller configuration.

weight This value is set in the cpu.weight interface file of the cgroup. The range is from 1 to 10 000, the default is 100.

period This value specifies a period of time, in microseconds, for how regularly a cgroup's access to CPU resources should be reallocated.This value is set in the cpu.max interface file of the cgroup. The range is from 1000 to 1 000 000, the default is 100 000.

quota This value specifies the total length of time, in microseconds, for which all tasks in a cgroup can run during one period (as defined in the period parameter). If quota is set to 0, it translates to ‟max” in the cpu.max interface file. The range is from 1000 to 1 000 000, the default is max.

cpuset

CPU usage information for the cgroup.

cpus This value indicates the CPUs used by the cgroup. This can be "", meaning use all CPUs except for the isolated CPUs; this is the default. The value all means include the isolated CPUs for cgroup usage. The value x, y-z, where x, y, and z are CPU numbers, means use a specific CPU or a range of CPUs.

mems This value is used for scheduling multiple NUMA (non-uniform memory access) aware applications in the cgroup.

Customer-defined cgroup profile

Customers can configure cgroup profiles in the /etc/opt/srlinux/appmgr/cgroup_profile.json directory. The app_mgr creates this directory at boot up if it does not exist.

If a customer-defined cgroup profile fails to load, the system continues to function and applications that are loaded into the customer cgroup are loaded using Nokia defaults, listed below.

  • Nokia-written applications run in the workload.slice/primary cgroup along with any processes that are started by linuxadmin, including sr_cli.

  • Non-Nokia-written applications run in the workload.slice/secondary cgroup. If a customer builds an application and launches it using the app_mgr without specifying a cgroup, the application runs in this cgroup

  • All interactive user applications run in the user.slice/default cgroup, including sr_cli when not started by linuxadmin.

The admin user is treated as any other user in the system. Its processes fall into the user.slice/default cgroup.

Configuring a cgroup

Customers can configure up to three cgroups in the /etc/opt/srlinux/appmgr/cgroup_profile.json directory. Customer applications are assigned to these groups. Any more than three configured cgroups are ignored. The depth of cgroups is limited to two levels where, for example, workload is one level, and primary/secondary are two levels. Any levels beyond this are also ignored.

If a cgroup with the same name is used in multiple customer-defined profiles, the system ignores it and uses the cgroup defined in the default profile.

Cgroup configuration example

The following example shows the configuration of two customer-defined cgroups: one for a lightweight database that needs priority access to resources, and one for storing the users of the database.

The steps and the outputs are described below.

The configuration above created two cgroup profiles: one for the database slice and one for the frontend slice. The profile for the database slice is configured to limit the database to 50% of system memory. The profile for the frontend slice is configured to limit the web front end to 20% of system memory.

In addition, both cgroup profiles are configured to limit CPU resources for their respective cgroup. The database server CPU is weighted at 10000 (the maximum CPU weight) and the frontend server CPU is weighted at 5000 (half the CPU weight of the database). The weights are added together and each group is allocated its ratio of CPU as a proportion of the sum. The periods are kept the same, and no guaranteed CPU is granted.

  1. Install the application, including the YAML binary file and optional YANG module.

    In this example, YAML defines two applications: dtw-database and dtw-frontend. These applications are placed into their own cgroups: distributetheweb.slice/database and distributetheweb.slice/frontend.

    The app-mgr creates the cgroups.

    The output below shows the installation of the database and a web front end.

    dtw-database:
       path: /opt/distributetheweb/bin/
       launch-command: ./run_db
       search-command: ./run_db
       failure-threshold: 100
       failure-action: "wait=60"
       cgroup: distributetheweb.slice/database
       oom-score-adj: -500
       yang-modules:
           names:
               - "database"
           source-directories:
               - "/opt/distributetheweb/yang/"
    
    dtw-frontend:
       path: /opt/distributetheweb/bin/
       launch-command: ./run_frontend
       search-command: ./run_frontend
       failure-threshold: 100
       failure-action: "wait=60"
       cgroup: distributetheweb.slice/frontend
       oom-score-adj: 200
       yang-modules:
           names:
               - "frontend"
           source-directories:
               - "/opt/distributetheweb/yang/"
    
  2. Configure the cgroup profiles in the /etc/opt/srlinux/appmgr/cgroup_profile.json directory.

    The output below shows the configuration.

    {
     "profiles": [
      {
       "name": "distributetheweb.database",
       "path": "distributetheweb.slice/database",
       "controller": {
        "memory": {
         "max": 0.5,
         "swap_max": 0,
         "low": 0
        },
        "cpu": {
         "weight": "10000",
         "period": "100000",
         "quota": "0"
        }
       }
      },
      {
       "name": "distributetheweb.frontend",
       "path": "distributetheweb.slice/frontend",
       "controller": {
        "memory": {
         "max": 0.2,
         "swap_max": 0,
         "low": 0
        },
        "cpu": {
         "weight": "5000",
         "period": "100000",
         "quota": "0"
        }
       }
      }
     ]
    }
    

Kernel low-memory killer

The kernel low-memory killer driver monitors the memory state of a running system. It reacts to high memory pressure by killing the least essential processes to keep the system performing at acceptable levels.

When the system is low in memory and cannot find free memory space, the out_of_memory function is called. The out_of_memory function makes memory available by killing one or more processes.

When an out-of-memory (OOM) failure occurs, the out_of_memory function is called and it obtains a score from the select_bad_process function. The process with the highest score is the one that is killed. Some of the criteria used to identify a bad process include the following:

  • The kernel needs a minimum amount of memory for itself.

  • Try to reclaim a large amount of memory.

  • Do not kill a process that is using a small amount of memory.

  • Try to kill the minimum number of processes.

  • Algorithms that elevate the sacrifice priority on processes the user wants to kill.

In addition to this list, the OOM killer checks the out-of-memory (OOM) score. The OOM killer sets the OOM score for each process and then multiplies that value by memory usage. The processes with higher values have a high probability of being terminated by the OOM killer.

SR Linux process kill strategy

The kernel calculates the oom_score using the formula 10 × percentage of memory used by the process. The maximum score is 10 × 100% = 1000. The oom_score of a process can be found in the /proc directory (/proc/$pid/oom_score). An oom_score of 1000 means the process is using all the memory, an oom_score of 500 means it is using half the memory, and an oom_score of 0 means it is using no memory.

The OOM killer checks the oom_score_adj file in the /proc/$pid/oom_score_adj directory to adjust its final calculated score. The default value is 0.

The oom_score_adj value can range from -1000 to 1000. A score of -1000 results in a process using 100% of the memory and in not being terminated by the OOM killer. However, a score of 1000 causes the Linux kernel to terminate the process even when it uses minimal memory. A score of -100 results in a process using 10% of the memory before it is considered for termination, as its score remains 0 until its unadjusted score reaches 100.

The oom_score_adj value for each process is defined in its corresponding YAML definition file. The system groups the processes based on their score, which the SR Linux OOM killer uses as the hierarchy for terminating a rogue process, as follows:

  • group1 = -998, for processes that should never be killed, such as app_mgr, idb_server, and all processes on the IMM.

  • group2 = -200, for processes that ideally should not be killed because doing so has a substantial impact on the system, such as mgmt_server, chassis_mgr, and net_inst_mgr. A score of -200 means the process gets to use 20% of the memory before their memory use starts being counted.

  • group3 = 0, for process that should be killed if they are using too much memory such as BGP, ISIS, and OSPF.

  • group4 = 500, for processes that should be preferentially killed, such as cli and oam_mgr

OOM adjust score per process lists the OOM adjust score for each process.

Table 3. OOM adjust score per process

Process name

OOM adjust score

aaa_mgr

0

acl_mgr

0

app_mgr

-998

sarp_nd_mgr

-200

bfd_mgr

-200

bgp_mgr

0

chassis_mgr

-200

dev_mgr

-200

dhcp_client_mgr

0

dhcp_relay_mgr

0

eth_switch_mgr

-200

evpn_mgr

0

fib_mgr

0

gnmi_server

500

idb_server

-998

isis_mgr

0

json_rpc

500

l2_mac_learn_mgr

0

l2_mac_mgr

0

l2_static_mac_mgr

0

lag_mgr

0

linux_mgr

0

lldp_mgr

0

log_mgr

0

mcid_mgr

0

mgmt_server

-200

mpls_mgr

0

net_inst_mgr

-200

oam_mgr

500

ospf_mgr

0

plcy_mgr

0

qos_mgr

0

sdk_mgr

500

sflow_sample_mgr

500

sshd-mgmt

0

static_route_mgr

0

supportd

0

timesrv

0

vrrp_mgr

0

vxlan_mgr

0

xdp_cpm

-200

Application manager extensions for cgroups

The cgroup feature provides two additional parameters in the app_mgr YAML file, the cgroup parameter and the oom-score-adj parameter.

The app_mgr uses the cgroup parameter to launch an application within a specific cgroup. A valid value for the cgroup parameter is the path of a cgroup as specified in the cgroup profile (equivalent to /profiles/name[]/path), from the cgroupv2 root. If this cgroup does not exist, the app_mgr launches the user application from the default cgroup profile path workload.slice/secondary.

The app_mgr uses the oom-score-adj parameter to set the out-of-memory adjust score for a process. This score is fed into the SR Linux OOM killer. Valid oom-score-adj scores are any value in the range of -1000 to 1000. A process with a score of -1000 is least likely to be killed while a process with a score of 1000 is most likely to be killed. At -1000, a process can use 100% of memory and still avoid being terminated by the OOM killer; however, SR Linux kills the process more frequently.

The cgroup parameter and the oom-score-adj parameter are shown in the output below.

bgp_mgr:
   path: @SRLINUX_BINARY_INSTALL_PREFIX@
   launch-command: @YAML_LAUNCH_ENVIRONMENT@ ./sr_bgp_mgr
   search-command: ./sr_bgp_mgr
   oom-score-adj: 0
   wait-for-config: Yes
   author: 'Nokia'
   restricted-operations: ['reload']
   cgroup: workload.slice/primary
   yang-modules:
       names:
           - "srl_nokia-bgp"
           - "srl_nokia-bgp-vpn"
           - "srl_nokia-rib-bgp"
           - "srl_nokia-system-network-instance-bgp-vpn"
       tools:
           - "srl_nokia-tools-bgp"
       source-directories:
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/ietf"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/common"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/interfaces"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/network-
instance"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/routing-policy"
           - "@SRLINUX_FILE_INSTALL_PREFIX@/models/srl_nokia/models/system"

Debugging cgroups

Cgroup debugging capability is available through:

  • SR Linux CLI commands

  • Linux-provided CLI commands

SR Linux cgroup debugging commands

The SR Linux provides CLI commands to perform the following:

  • check the usage of existing cgroups

  • show information about the OOM adjust score of applications managed by the app_mgr

  • show information about the cgroups that are associated with the applications that are managed by the app_mgr

  • list all of the applications associated with a specified cgroup

Checking existing cgroup usage

The output below is an example of checking existing cgroup usage.

--{ running }--[  ]--  
# info from state platform control A cgroup *
    platform {                    
        control A {                
            cgroup /mnt/cgroup/unified/user.slice/default {
                memory-statistics {
                    current 0 
                    current-swap 0    
                    anon 0        
                    kernel-stack 0  
                    slab 0      
                    sock 0
                    anon-thp 0      
                    file 0          
                    file-writeback 0  
                    file-dirty 0
                }
                cpuacct-statistics {
                    user 0
                    system 0
                }
            }
            cgroup /mnt/cgroup/unified/workload.slice/primary {
                memory-statistics {
                    current 1621110784
                    current-swap 0
                    anon 1327427584
                    kernel-stack 2396160
                    slab 12939264
                    sock 151552
                    anon-thp 855638016
                    file 250916864
                    file-writeback 0
                    file-dirty 0
                }
                cpuacct-statistics {
                    user 22082989551
                    system 11252275018
                }
            }
            cgroup /mnt/cgroup/unified/workload.slice/secondary {
                memory-statistics {
                    current 7827456
                    current-swap 0
                    anon 4501504
                    kernel-stack 73728
                    slab 2453504
                    sock 8192
                    anon-thp 0
                    file 0
                    file-writeback 0
                    file-dirty 0
                }
                cpuacct-statistics {
                    user 34323091
                    system 12342430
                }
            }
        }
    }
Showing current OOM adjust scores

The output below is an example of showing information about the current OOM adjust scores for all applications that are managed by the app_mgr.

--{ running }--[  ]-- 
# info from state system app-management application * oom-score-adj
    system {                     
        app-management {          
            application aaa_mgr {
                oom-score-adj 0          
            }                  
            application acl_mgr {
                oom-score-adj 0         
            }                  
            application app_mgr {
                oom-score-adj -998    
            }                  
            application arp_nd_mgr {
                oom-score-adj -200
            }                  
|
|
|
            application vxlan_mgr {
                oom-score-adj 0
            }
            application xdp_lc_1 {
                oom-score-adj -200
            }
        }
    }
Showing cgroup information

The output below is an example of showing information about cgroups that are associated with applications managed by the app_mgr.

--{ running }--[  ]--  
# info from state system app-management application * cgroup
    system {
        app-management {         
            application aaa_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary  
            }
            application acl_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application arp_nd_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application bgp_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }     
|
|
|
            application sshd-mgmt {
                cgroup /mnt/cgroup/unified/workload.slice/secondary  
            }
            application supportd {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application vxlan_mgr {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
            application xdp_lc_1 {
                cgroup /mnt/cgroup/unified/workload.slice/primary
            }
        }
    }
Listing all the applications associated with a specified cgroup

Use the tools system cgroup command pgrep cgroup cgroupname command to list all the applications associated with a specified group; the output below shows an example.

--{ running }--[  ]--
# tools system cgroup command pgrep cgroup workload.slice/primary
+-------+----------------------+
|  Pid  |       Process        |
+=======+======================+
| 3373  | sr_app_mgr           |
| 3385  | sr_supportd          |
| 3402  | sr_device_mgr        |
| 3460  | sr_idb_server        |
| 3471  | sr_aaa_mgr           |
| 3482  | sr_acl_mgr           |
| 3518  | sr_arp_nd_mgr        |
| 3542  | sr_chassis_mgr       |
| 3560  | sr_dhcp_client_mgr   |
| 3574  | sr_evpn_mgr          |
| 3586  | sr_fib_mgr           |
| 3598  | sr_l2_mac_learn_mgr  |
| 3611  | sr_l2_mac_mgr        |
| 3621  | sr_lag_mgr           |
| 3631  | sr_linux_mgr         |
| 3641  | sr_log_mgr           |
| 3651  | sr_mcid_mgr          |
| 3681  | sr_mgmt_server       |
| 3702  | sr_net_inst_mgr      |
| 3724  | sr_oam_mgr           |
| 3743  | sr_sdk_mgr           |
| 3753  | sr_sflow_sample_mgr  |
| 3772  | sr_xdp_lc_1          |
| 3874  | sudo                 |
| 3895  | sudo                 |
| 3902  | sudo                 |
| 3921  | sudo                 |
| 3930  | sr_qos_mgr           |
| 3953  | sr_gnmi_server       |
| 3979  | sr_json_rpc          |
| 3990  | sr_vxlan_mgr         |
| 16740 | sr_json_wkr          |
| 16742 | python3              |
| 17013 | python3              |
| 18474 | sr_json_wkr          |
| 18478 | python3              |
| 18671 | python3              |
| 20671 | sr_lldp_mgr          |
| 21177 | sr_bgp_mgr           |
| 21190 | sr_plcy_mgr          |
| 12753 | sr_l2_static_mac_mgr |
| 27353 | rsyslogd             |
| 9777  | python               |
+-------+----------------------+

Linux-provided cgroup debugging commands

The following Linux-provided CLI commands are available for debugging cgroups:

  • the systemd-cgls command

  • the systemd-cgtop command

The systemd-cgls command dumps the cgroup hierarchy. The output below shows an example of the systemd-cgls command.

[linuxadmin@srlinux unified]$ systemd-cgls
+-1 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
+-system.slice
  +-rngd.service
  | +-2725 /sbin/rngd -f
  +-systemd-udevd.service
  | +-868 /usr/lib/systemd/systemd-udevd
  +-system-serial\x2dgetty.slice
  | +-serial-getty@ttyS0.service
  |   +-1120 login -- linuxadmin
  |   +-8320 -bash
  |   +-9262 sendacct "systemd-cgls"
  |   +-9263 systemd-cgls
  |   +-9264 less
  +-srlinux.service
  | +- 3092 /usr/bin/sudo /opt/srlinux/bin/sr_linux
  | +- 3120 runuser --user srlinux --group srlinux -- /opt/srlinux/bin/sr_linux
  | +- 4019 /bin/bash /opt/srlinux/bin/sr_linux --user-env-switched
  | +- 4039 ./sr_app_mgr
  | +- 4051 ./sr_supportd --server-mode
  | +- 4068 ./sr_device_mgr
  | +- 4219 ./sr_idb_server
  | +- 4229 ./sr_eth_switch
  | +- 4288 ./sr_aaa_mgr
  | +- 4299 ./sr_acl_mgr
  | +- 4314 ./sr_arp_nd_mgr
  | +- 4324 ./sr_chassis_mgr
  | +- 4337 ./sr_dhcp_client_mgr
  | +- 4365 ./sr_evpn_mgr
  | +- 4381 ./sr_fib_mgr
  | +- 4395 ./sr_l2_mac_learn_mgr
  | +- 4411 ./sr_l2_mac_mgr
  | +- 4423 ./sr_lag_mgr
  | +- 4445 ./sr_linux_mgr
  | +- 4494 ./sr_log_mgr
  | +- 4510 ./ntpd -c /etc/ntps.conf -g
  | +- 4526 ./sr_mcid_mgr
  | +- 4574 ./sr_mgmt_server
  | +- 4608 ./sr_net_inst_mgr
  | +- 4626 ./sr_oam_mgr
  | +- 4640 ./sr_sdk_mgr
  | +- 4658 ./sr_sflow_sample_mgr
  | +- 4681 ./sr_xdp_cpm
  | +- 5197 /usr/bin/sudo -Enu root /usr/bin/sudo -Enu gnmirpc bash -c ./sr_gnmi
  | +- 5224 /usr/bin/sudo -Enu root /usr/bin/sudo -Enu jsonrpc bash -c ./sr_json
  | +- 5243 /usr/bin/sudo -Enu gnmirpc bash -c ./sr_gnmi_server
  | +- 5248 ./sr_qos_mgr
  | +- 5261 /usr/bin/sudo -Enu jsonrpc bash -c ./sr_json_rpc
  | +- 5290 ./sr_gnmi_server
  | +- 5302 ./sr_json_rpc
  | +- 5693 /usr/sbin/sshd -f /etc/ssh/sshd_config_mgmt
  | +- 6361 /usr/sbin/rsyslogd -i /var/run/srlinux/rsyslogd.pid
  | +-21936 ./sr_bfd_mgr
  | +-22018 ./sr_bgp_mgr
  | +-22132 ./sr_isis_mgr
  | +-22242 ./sr_lldp_mgr
  | +-22321 ./sr_mpls_mgr
  | +-22437 ./sr_ospf_mgr
  | +-22588 ./sr_plcy_mgr
  | +-22701 ./sr_static_route_mgr
  | +-22796 sr_json_wkr
  | +-22797 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  | +-22953 ./ntpd -c /etc/mntp.conf -g
  | +-23023 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  | +-23736 sr_json_wkr
  | +-23741 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  | +-24009 python3 -m srlinux.mgmt.json_rpc.json_rpc_server_main --bind=[::]:40
  | +-27488 ./dnsmasq --conf-file=/etc/dnsmasq.conf
  +-polkit.service
  | +-2701 /usr/lib/polkit-1/polkitd --no-debug
  +-ztpapi.service
  | +-1114 /opt/srlinux/ztp/virtual-env/bin/python -m ztp.ztphttp
  +-systemd-journald.service
  | +-840 /usr/lib/systemd/systemd-journald
  +-sshd.service
  | +-985 /usr/sbin/sshd -D
  +-crond.service
  | +-1133 /usr/sbin/crond -n
  +-sr_watchdog.service
  | +-1155 sr_wd
  +-dbus.service
  | +-991 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
  +-systemd-logind.service
    +-1062 /usr/lib/systemd/systemd-logind

The systemd-cgtop command dumps the current usage of each cgroup. The output below shows an example of the systemd-cgtop command.

Path                                     Tasks   %CPU   Memory  Input/s Output/s
 
/                                          186      -        -        -        -
/system.slice/crond.service                  1      -        -        -        -
/system.slice/dbus.service                   1      -        -        -        -
/system.slice/polkit.service                 1      -        -        -        -
/system.slice/rngd.service                   1      -        -        -        -
/system.slice/sr_watchdog.service            1      -        -        -        -
/system.slice/srlinux.service               53      -        -        -        -
/system.slice/sshd.service                   1      -        -        -        -
/system.slic...ial-getty@ttyS0.service       3      -        -        -        -
/system.slice/systemd-journald.service       1      -        -        -        -
/system.slice/systemd-logind.service         1      -        -        -        -
/system.slice/systemd-udevd.service          1      -        -        -        -
/system.slice/ztpapi.service                 1      -        -        -        -