Configuring event handler for operational groups

The key to preventing traffic from being black-holed is to not allow it to be forwarded to a leaf that has no active uplinks; for example, by disabling access links as soon as uplinks become operationally disabled.

The following sections provide an example of configuring event handler for an operational group (oper-group):

Configuring the event handler instance

To configure an event handler instance for the oper-group feature:

  1. Define a set of uplinks to monitor in the paths statement.

  2. Specify downlinks (access links) and other parameters in the options statement.

  3. Provide the name of a MicroPython script in the upython-script statement.

Defining uplinks to monitor in the paths statement

In the paths statement of an event handler instance used with an oper-group, you define the set of uplinks that are necessary to provide service for a set of downlinks. The oper-group feature works by monitoring the operational state of the uplinks, and uses the state information to determine whether the operational state of the access links must be changed.

In this example, the operational state of two uplink interfaces ethernet-1/49 and ethernet-1/50 on Leaf 1 are being monitored. If the operational state of the uplink interfaces changes to down, the oper-group feature changes the state of the access interface to down to avoid black-holing of traffic from the client.

Figure 1. Disabling downlink based on uplink state

To monitor the operational state for a set of interfaces, configure the paths statement in an event handler instance. For example:

--{ candidate shared default }--[  ]--
# info system event-handler instance opergroup path
    system {
        event-handler {
            instance opergroup {
                paths [
                    "interface ethernet-1/{49..50} oper-state"
                ]
            }
        }
    }

Specify the contents of the paths statement in SR Linux CLI format. In the example above, the paths statement is equivalent to the following CLI command:

--{ running }--[  ]--
# info from state interface ethernet-1/{49..50} oper-state

Specifying downlinks and other parameters in the options statement

The options statement in the event handler instance allows you to define objects that are passed to the script to be used as input parameters.

For an oper-group configuration, you can use the options statement to indicate the relationship between the monitored uplinks and the access links.

In the following example, the options define objects that specify the following:

  • The access links that react to state changes of the uplinks
  • The number of operationally up uplinks required for the access links to stay up
--{ candidate shared default }--[  ]--
# info system event-handler instance opergroup options
    system {
        event-handler {
            instance opergroup {
                options {
                    object down-links {
                        values [
                            ethernet-1/1
                        ]
                    }
                    object required-up-uplinks {
                        value 1
                    }
                    object debug {
                        value true
                    }
                }
            }
        }
    }

In this example, the down-links object specifies an interface name. When the object is passed to the script, it can be used as a parameter indicating the access link associated with the uplinks. For this oper-group configuration, the down-links object indicates the interface for which the operational state depends on the state of the uplinks defined in the paths statement.

The required-up-uplinks object specifies the number of uplinks that need to be operationally up before the access link is brought down. For this oper-group configuration, the value is 1, which means that at least one uplink must be up. The script calculates the number of uplinks that are operationally up, and compares that number to the value in the required-up-uplinks object.

The debug object is set to true, which directs the script to print the values of certain script variables.

Specifying script name in the upython-script statement

The upython-script statement in the event handler instance specifies the name of the MicroPython script to be invoked when SR Linux detects a change in the interfaces defined in the paths statement.

The MicroPython script must reside in one of the following locations:

  • /etc/opt/srlinux/eventmgr for user-provided scripts
  • /opt/srlinux/eventmgr for Nokia-provided scripts
--{ candidate shared default }--[  ]--
# info system event-handler instance opergroup upython-script
    system {
        event-handler {
            instance opergroup {
                upython-script oper-group.py
            }
        }
    }

For this oper-group configuration, whenever a change occurs to the oper-state of the interfaces defined in the paths statement, event handler invokes the oper-group.py script.

Event handler oper-group configuration

When administratively enabled, the full configuration for this event handler instance looks like the following:

--{ candidate shared default }--[  ]--
# info system event-handler instance opergroup
    system {
        event-handler {
            instance opergroup {
                admin-state enable
                upython-script oper-group.py
                paths [
                    "interface ethernet-1/{49..50} oper-state"
                ]
                options {
                    object debug {
                        value true
                    }
                    object down-links {
                        values [
                            ethernet-1/1
                        ]
                    }
                    object required-up-uplinks {
                        value 1
                    }
                }
            }
        }
    }

MicroPython script for oper-group

When there is a state change in any of the paths defined in the paths statement of the event handler instance, the script defined in the upython-script statement is invoked. Event handler calls the function event_handler_main() in the script, passing it a JSON string indicating the current state of the monitored paths, as well as the object:value pairs defined in the options statement.

The script receives this input, processes it, and returns a list of actions.

Script input

For the example in Event handler oper-group configuration, the input JSON string consists of the current state of the two uplinks and the provided options. The following JSON string is passed to the oper-group.py script if the operational state of interface ethernet-1/49 changes to down:

{
    "paths": [
        {
            "path": "interface ethernet-1/49 oper-state",
            "value": "down"
        },
        {
            "path": "interface ethernet-1/50 oper-state",
            "value": "up"
        }
    ],
    "options": {
        "debug": "true",
        "required-up-uplinks": "1",
        "down-links": [
            "ethernet-1/1"
        ]
    }
}

Script processing

The following is the oper-group.py script referenced in the event handler instance.

import sys
import json

# count_up_uplinks returns the number of monitored uplinks that have oper-state=up
def count_up_uplinks(paths):
    up_cnt = 0
    for path in paths:
        if path.get("value", "down") == "up":
            up_cnt = up_cnt + 1
    return up_cnt

# required_up_uplinks returns the value of the `required-up-uplinks` option
def required_up_uplinks(options):
    return int(options.get("required-up-uplinks", 1))

# main entry function for event handler
def event_handler_main(in_json_str):
    # parse input json string passed by event handler
    in_json = json.loads(in_json_str)
    paths = in_json["paths"]
    options = in_json["options"]

    num_up_uplinks = count_up_uplinks(paths)
    downlinks_new_state = (
        "down" if num_up_uplinks < required_up_uplinks(options) else "up"
    )

    # add `debug="true"` option to event-handler configuration to output parsed parameters
    if options.get("debug") == "true":
        print(
            f"num of required up uplinks = {required_up_uplinks(options)}\n\
detected num of up uplinks = {num_up_uplinks}\n\
downlinks new state = {downlinks_new_state}"
        )

    response_actions = []

    for downlink in options.get("down-links", []):
        response_actions.append(
            {
                "set-ephemeral-path": {
                    "path": f"interface {downlink} oper-state",
                    "value": downlinks_new_state,
                }
            }
        )

    response = {"actions": response_actions}
    return json.dumps(response)

The following sections describe how each part the script processes the input for this oper-group example.

Parsing input JSON

Starting with the event_handler_main function, the incoming JSON string is parsed and the relevant portions are extracted.

in_json = json.loads(in_json_str)
paths = in_json["paths"]
options = in_json["options"]

The paths and options are objects defined in the incoming JSON string, and they are saved in their respective like-named variables.

Populating the debug log

The debug option causes the script variables to appear in the debug log.

# add `debug="true"` option to event-handler configuration to output parsed parameters
if options.get("debug") == "true":
    print(
        f"num of required up uplinks = {required_up_uplinks(options)}\n\
detected num of up uplinks = {num_up_uplinks}\n\
downlinks new state = {downlinks_new_state}"
    )

The debug log is present only if the debug option is set to "true" in the event handler instance configuration.

You can display the debug log by using the following CLI command:

--{ running }--[  ]--
# info from state system event-handler instance opergroup last-stdout-stderr

Composing output

At this point, the script is able to define the correct state for the downlinks, based on the state of the monitored uplinks and the required number of healthy uplinks. For the event handler to take action, the script needs to output a JSON string following the format defined in Actions.

response_actions = []

for downlink in options.get("down-links", []):
    response_actions.append(
        {
            "set-ephemeral-path": {
                "path": f"interface {downlink} oper-state",
                "value": downlinks_new_state,
            }
        }
    )

response = {"actions": response_actions}
return json.dumps(response)

This example shows an output JSON string, using the calculated downlinks_new_state and the list of downlinks provided from the down-links option.

The output JSON string contains the set-ephemeral-path action, which sets the oper-state of the downlink to the correct value (up or down).

The output is provided via the response dictionary, and is JSON-encoded before returning from the function. This routine provides a JSON string back to the event handler, which processes and executes the actions passed to it.

The result of this processing shows the implementation of the oper-group feature: the event handler executes actions to set the state of a downlink based on the state of a group of uplinks.

Displaying oper-group information

When an event handler instance is configured and administratively enabled, an initial sync of the monitored paths state is performed. As a result of this initial sync, event handler immediately attempts to execute a script when it receives the state for the monitored paths.

You can display the status of an event handler instance by querying the state datastore. For example:

--{ running }--[  ]--
# info from state system event-handler instance opergroup
    system {
        event-handler {
            instance opergroup {
                admin-state enable
                upython-script oper-group.py
                oper-state up
                last-input "{\"paths\":[{\"path\":\"interface ethernet-1/49 oper-state\",\"value\":\"up\"},{\"path\":\"interface ethernet-1/50 oper-state\",\"value\":\"up\"}],\"options\":{\"down-links\":[\"ethernet-1/1\"],\"required-num-up-link\":\"1\",\"required-up-uplins\":\"1\"}}"
                last-output "{\"actions\": [{\"set-ephemeral-path\": {\"path\": \"interface ethernet-1/1 oper-state\", \"value\": \"up\"}}]}"
                last-stdout-stderr ""
                paths [
                    "interface ethernet-1/{49..50} oper-state"
                ]
                options {
                    object down-links {
                        values [
                            ethernet-1/1
                        ]
                    }
                    object required-num-up-link {
                        value 1
                    }
                    object required-up-uplins {
                        value 1
                    }
                }
                statistics {
                    execution-duration 0
                    last-execution "20 minutes ago"
                    total-execution-duration 0
                    execution-count 2
                    execution-successes 2
                }
            }
        }
    }

This command displays the following information:

  • oper-state

    The operational state of the event handler instance. In case of any errors in the script and, or configuration the state is down.

  • oper-down-reason and oper-down-reason-detail

    Information about why the event handler instance is operationally down.

  • last-input

    The input JSON string used during the last execution of the script.

  • last-stdout-stderr

    Outputs from the script, such as print statements and log messages.

  • last-output

    The output JSON string produced by a script during the last execution.

  • statistics

    Statistics related to the execution process.

For the oper-group example, the following output is displayed if one of the uplinks goes down:

--{ running }--[  ]--
# info from state system event-handler instance opergroup
    system {
        event-handler {
            instance opergroup {
                admin-state enable
                upython-script oper-group.py
                oper-state up
                last-input "{\"paths\":[{\"path\":\"interface ethernet-1/49 oper-state\",\"value\":\"down\"},{\"path\":\"interface ethernet-1/50 oper-state\",\"value\":\"up\"}],\"options\":{\"debug\":\"true\",\"down-links\":[\"ethernet-1/1\"],\"required-num-up-link\":\"1\",\"required-up-uplins\":\"1\"}}"
                last-output "{\"actions\": [{\"set-ephemeral-path\": {\"path\": \"interface ethernet-1/1 oper-state\", \"value\": \"up\"}}]}"
                last-stdout-stderr "num of required up uplinks = 1
detected num of up uplinks = 1
downlinks new state = up

The setting for downlinks new state is up because the detected num of up uplinks did not drop below the required number of 1. The downlink interface therefore remains operationally up.

If both of the uplinks go down, the following is displayed:

--{ running }--[  ]--
# info from state system event-handler instance opergroup
    system {
        event-handler {
            instance opergroup {
                admin-state enable
                upython-script oper-group.py
                oper-state up
                last-input "{\"paths\":[{\"path\":\"interface ethernet-1/49 oper-state\",\"value\":\"down\"},{\"path\":\"interface ethernet-1/50 oper-state\",\"value\":\"down\"}],\"options\":{\"debug\":\"true\",\"down-links\":[\"ethernet-1/1\"],\"required-num-up-link\":\"1\",\"required-up-uplins\":\"1\"}}"
                last-output "{\"actions\": [{\"set-ephemeral-path\": {\"path\": \"interface ethernet-1/1 oper-state\", \"value\": \"down\"}}]}"
                last-stdout-stderr "num of required up uplinks = 1
detected num of up uplinks = 0
downlinks new state = down

The detected num of up uplinks is 0, which is below the required number of 1. This causes event handler to set the downlink interface to operationally down.

In this way, event handler uses the oper-group feature to disable the access link when the uplink interfaces go down, therefore preventing traffic from black-holing.