Zero Touch Provisioning

ZTP requires the following components:

• DHCP server (IPv4 or IPv6) – To support the assignment of IP addresses through DHCP requests and offers.
• file server – For staging and transfer of RPMs, configurations, images, and scripts. HTTP, HTTPS, TFTP, and FTP are supported. For HTTPS, the default Mozilla certificate should be used.
• DHCP relay – Required if the server is outside the management interface broadcast domain.

ZTP works in the following network environments:

• nodes, HTTP file servers, and DHCP server in the same subnet
• HTTP file servers and DHCP server in the same subnet, separate from the nodes
• nodes, HTTP file servers, and DHCP server in different subnets

All components in the same subnet shows the first scenario where all components are in a Layer 2 broadcast domain. There is no DHCP relay and all IPs are assigned from a single pool.

HTTP file and DHCP servers in the same subnet shows the second scenario where only the HTTP file servers and DHCP server are in the same subnet. The DHCP relay is used to fill Option 82 as the gateway address. The gateway address is used to find the appropriate pool in the DHCP server to assign the correct subnet IP address to the SR Linux.

The DHCP offer allows the Option 3 router to define the default gateway. If multiple addresses are provided via Option 3, the first address is used for the default gateway.

All components in different subnets shows the third scenario where all components are in different subnets. The DHCP relay adds the Option 82 gateway address to the DHCP request, and the DHCP server adds the Option 3 with the gateway address of the HTTP file server.

DHCP discovery is sent out from a management port that is operationally up with un-tag format when OOB is used. IPv4 DHCP discovery is sent out first, and if no offer is received within the DHCP timeout, an IPv6 DHCP solicitation is then sent out. The DHCP timeout is 60 seconds.

• For DHCP IPv4, Option 61 is used for pool selection. By default, the node sends Option 61 with the serial number of the chassis.
  Note: The grub.cfg can be provisioned with a MAC option as well. When a MAC option is specified, Option 61 is populated with the chassis MAC address.
• For DHCP IPv6, Option 1 is used for pool selection. By default, the node uses RFC 3315 DUID Type 2 vendor-assigned unique ID. The value for enterprise-id is 6527 and the identifier is the chassis serial number.
  Note: Type 3 is configurable in the grub.cfg.

When the DHCP server receives the discovery packet, it assigns the IP address to the node. The DHCP offer for IPv4 requires the options shown in Required DHCP offer options.

DHCP IPv4 and IPv6 equivalents lists DHCP IPv4 and IPv6 equivalents.

OOB auto-boot supports both IPv4 discovery and IPv6 solicitation, but when the offer is received, the provisioning script follows the same address family format for file download as the DHCP offer.

The first DHCP offer received with the correct Option 66 and 67 or Option 43 is used. The Python provisioning script is downloaded from the location defined by Option 66 and 67 or Option 43.

With an IPv4 DHCP offer, the node IP address and other information, such as the default route, is included.

For IPv6, only the IP arrives from the DHCP server. This offer does not include a prefix, and when it is received, the route is installed with a prefix of /128. This means that the prefix received from the RA is ignored, and the node always acts as a host with a prefix of /128. The default route is received from the Route Advertisement (RA) from the IPv6 peer.

The Python provisioning file is downloaded from the location dictated by Option 66 and 67 or 43 using HTTP, HTTPS, TFTP, or FTP. Option 66 and 67 takes precedence over Option 43 when both are present in the offer, but Option 43 is used if there is a download error with Option 66 and 67. In addition, Option 66 and 67 can be summarized in Option 67 only. The URL of the provisioning file can be resolved via the DHCP-provided DNS. Up to three DNS servers can be offered by the DHCP.

The node downloads the Python script to storage device. The node then uses the Python provisioning script to download any RPMs, images, scripts, or configurations to the destination dictated by the script.

The URLs defined in the Python script define multiple levels of redundancy. If the primary location is unreachable and times out, two additional redundant servers can be configured. The node cycles through the primary, secondary, and tertiary locations and, when successful, downloads the files to the storage device where they are executed locally.

After successful completion, the provisioning script disables the auto-boot flag to ensure that additional reboots of the node do not enter auto-boot mode. When the nodes reboot, they come up in an operational state with the configuration and image.

For more information about the Python provisioning script, see Configuring the Python provisioning script.

The following are possible failure scenarios:

• No Option 66 and 67 or 43 is received (possibly because the format is a URL or no IP address was provided via the DHCP server).
• The download of the Python provisioning script failed or the server was not reachable.
• The download of the RPMs, scripts, or configurations failed (possibly because to the server is not available, or incorrect directory or credentials).
• Copying the RPMs, scripts, or configuration to the storage device failed.

If a failure occurs:

• Details of the failure display on the console and are recorded in the appropriate log files. Log files are stored in the storage device. There can be three log files, which are overwritten in a circular manner.
• The DHCP task is notified of the failure and releases the IP address on the port.
• The auto-boot task goes through the process cycle again until it succeeds, the timeout value is reached, or the auto-boot Option is removed from the grub.cfg, either by editing the grub.cfg or using the ZTP CLI.

ZTP log files are stored under /var/log/ztp.

[root@srlinux ztp]# ls -ltr
total 28528
-rw-r--r-- 1 root root    18220 Sep  4 23:04 ztp_2019-09-04_23-03-52_880867.log
-rw-r--r-- 1 root root     1789 Sep  4 23:29 ztp_2019-09-04_23-29-38_496995.log
-rw-r--r-- 1 root root    18220 Sep  4 23:31 ztp_2019-09-04_23-31-08_996284.log
-rw-r--r-- 1 root root     1791 Sep  5 17:42 ztp_2019-09-05_17-42-01_082002.log
-rw-r--r-- 1 root root        0 Sep  5 21:56 ztp_2019-09-05_21-56-13_730783.log
-rw-r--r-- 1 root root        0 Sep  5 21:56 ztp_2019-09-05_21-56-14_036070.log
[root@srlinux ztp]#

There are two CLIs where ZTP-related commands can be executed:

• SR Linux CLI (sr_cli)
• ZTP CLI

The SR Linux commands are available to use only when the SR Linux is operational at the sr_cli.

When the SR Linux is not operational, the ZTP CLI is a unified tool that can be used to manage ZTP tasks at the console.

See the ZTP CLI and SR Linux CLI command structures sections for a complete list of available ZTP-related commands.

The primary components of the Python provisioning script include:

• location of provider certificate and trust anchor when HTTPS is used to download RPMs and other bash/Python scripts
• the URL location for each RPM, script, and configuration
• DNS information for resolving the URL of a file. At least one DNS entry is needed for resolving the URL of downloaded files. This DNS can be different from the DHCP offered DNS. If a DNS is defined in the provisioning file, it takes precedence over the DHCP DNS. If there is no DNS in the provisioning script, the DHCP DNS will be used.
• a section to clear the auto-boot option from the grub.cfg kernel section

import errno
import os
import sys
import signal
import subprocess
from subprocess import Popen, PIPE
import threading
srlinux_image_url = 'http://135.227.249.116/srlinux/srlinux-20.6.1-10656.squashfs'
srlinux_image_md5_url = 'http://135.227.249.116/srlinux/srlinux-20.6.1-10656.md5'
srlinux_config_url = 'http://135.227.249.116/srlinux/config.json'
class ProcessError(Exception):
    def __init__(self, msg, errno=-1):
        Exception.__init__(self, msg)
        self.errno = errno
class ProcessOpen(Popen):
    def __init__(self, cmd, cwd=None, env=None, flags=None, stdin=None,
        stdout=None, stderr=None, universal_newlines=True,):
        self.__use_killpg = False
        shell = False
        if not isinstance(cmd, (list, tuple)):
            shell = True
        # Set flags to 0, subprocess raises an exception otherwise.
        flags = 0
        # Set a preexec function, this will make the sub-process create it's
        # own session and process group - bug 80651, bug 85693.
        preexec_fn = os.setsid
        self.__cmd = cmd
        self.__retval = None
        self.__hasTerminated = threading.Condition()
        Popen.__init__(self, cmd, cwd=cwd, env=env, shell=shell, stdin=stdin,
           stdout=PIPE, stderr=PIPE, close_fds=True, 
           universal_newlines=universal_newlines, creationflags=flags,)
        print("Process [{}] pid [{}]".format(cmd, self.pid))
    def _getReturncode(self):
        return self.__returncode
    def __finalize(self):
        # Any finalize actions
        pass
    def _setReturncode(self, value):
        self.__returncode = value
        if value is not None:
            # Notify that the process is done.
            self.__hasTerminated.acquire()
            self.__hasTerminated.notifyAll()
            self.__hasTerminated.release()
    returncode = property(fget=_getReturncode, fset=_setReturncode)
    def _getRetval(self):
        # Ensure the returncode is set by subprocess if the process is finished.
        self.poll()
        return self.returncode
    retval = property(fget=_getRetval)
    def wait_for(self, timeout=None):
        if timeout is None or timeout < 0:
            # Use the parent call.
            try:
                out, err = self.communicate()
                self.__finalize()
                return self.returncode, out, err
            except OSError as ex:
                # If the process has already ended, that is fine. This is
                # possible when wait is called from a different thread.
                if ex.errno != 10:  # No child process
                    raise
                return self.returncode, "", ""
        try:
            out, err = self.communicate(timeout=timeout)
            self.__finalize()
            return self.returncode, out, err
        except subprocess.TimeoutExpired:
            self.__finalize()
            raise ProcessError(
                "Process timeout: waited %d seconds, "
                "process not yet finished." % (timeout)
            )
    def kill(self, exitCode=-1, sig=None):
        if sig is None:
            sig = signal.SIGKILL
        try:
            if self.__use_killpg:
                os.killpg(self.pid, sig)
            else:
                os.kill(self.pid, sig)
        except OSError as ex:
            self.__finalize()
            if ex.errno != 3:
                # Ignore:   OSError: [Errno 3] No such process
                raise
        self.returncode = exitCode
        self.__finalize()
    def commandline(self):
        """returns string of command line"""
        if isinstance(self.__cmd, six.string):
            return self.__cmd
        return subprocess.list2cmdline(self.__cmd)
    __str__ = commandline
def execute_and_out(command, timeout=None):
    print("Executing command: {}".format(command))
    process = ProcessOpen(command)
    try:
        #logger.trace("Timeout = {}".format(timeout))
        ret, out, err = process.wait_for(timeout=timeout)
        return ret, out, err
    except ProcessError:
        print("{} command timeout".format(command))
        process.kill()
        return errno.ETIMEDOUT, "", ""
def execute(command, timeout=None):
    ret, _, _ = execute_and_out(command, timeout=timeout)
    return ret
def pre_tasks():
    pass
def srlinux():
    nos_install()
    nos_configure()
def post_tasks():
    pass
def nos_install():
    cmd = 'ztp image upgrade --imageurl {} --md5url {}'.format(srlinux_image_url, srlinux_image_md5_url)
    ret,out,err = execute_and_out(cmd)
def nos_configure():
    cmd = 'ztp configure-nos --configurl {}'.format(srlinux_config_url)
    ret,out,err = execute_and_out(cmd)
def main():
    pre_tasks()
    srlinux()
    post_tasks()
if __name__ == '__main__':
    main()

The ZTP process sends DHCP discovery messages on all ports within a ZTP cycle. Every time the DHCP discovery timeout expires and a DHCP offer has not been received, the DHCP discovery process reinitiates on the port until the ZTP timeout expires.

The timeout value can be set using the:

• ZTP CLI (see procedure Configuring options in the grub.cfg using ZTP CLI)
• SR Linux CLI (see procedure Configuring options in the grub.cfg using SR Linux CLI)

Several options can be manually configured in the grub.cfg using the ZTP CLI at the console. The command has the following format:

# ztp option <command> [<arguments>]

where command must be one of the following:

ZTP CLI: option command examples describes examples of ztp option commands and available arguments.

# ztp option list
+--------------+----------+
| Name         | Value    |
+--------------+----------+
| autoboot     | False    |
| nosinstall   | False    |
| bootintf     | mgmt0    |
| timeout      | 3600     |
| retry        | 3        |
| clientid     | serialid |
| downgrade    | True     |
| formatovl    | False    |
| formatsrlopt | True     |
| formatsrletc | False    |
| srlflags     | None     |
+--------------+----------+

The ZTP CLI ztp image command can be used to activate, delete, list, or perform an upgrade at the console. The following command format is used:

# ztp image <command> [<arguments>]

where command must be one of the following:

ZTP CLI: ztp image command examples describes examples of ztp image commands and available arguments.

[root@localhost ~]# ztp image list
[ 1638.035200] EXT4-
fs (sdb2): mounted filesystem with ordered data mode. Opts: (null)
+--------------+
| Versions     |
+--------------+
| 20.6.1-10654* |
| 20.6.1-10587  |
| 20.6.1        |
+--------------+

[root@localhost ~]# ztp image list 
[  227.172007] EXT4-
fs (sdb2): mounted filesystem with ordered data mode. Opts: (null)
+--------------+
| Versions     |
+--------------+
| 20.6.1-10587* |
| 20.6.1-10654  |
| 20.6.1        |
+--------------+

[root@localhost ~]# ztp image version --filename ./srlinux-20.6.1-12617.tar
+-------------+---------+
| version     | message |
+-------------+---------+
| 20.6.1-12617 | None   |
+-------------+---------+

The ZTP CLI image command can be used to push the configuration from the console when SR Linux is not operational.

The following is an example showing how to configure the SR Linux with a configuration downloaded with a user-provided URL:

ztp configure-nos --configurl <URL to download configuration>

Executable files (RPMs, scripts, and configurations) can be redownloaded if required. The redownload can be performed at the console using the ZTP CLI.

The following is an example showing how to run the provisioning script with a user-provided URL:

ztp provision --url <URL where files should be downloaded from>

The ZTP process can be manually started, stopped, and restarted using a ZTP CLI command at the console. The following command format is used:

# ztp service <command> [<arguments>]

where command must be one of the following:

ZTP CLI: ztp service command examples describes examples of ztp service commands and available arguments.

The ZTP process status can be manually checked using the ZTP CLI command at the console.

The following is an example showing how to check the status of the ZTP process:

ztp service status

# ztp service status
+---------+----------+
| Service | Status   |
+---------+----------+
| ztp     | Active   |
+---------+----------+
#

Several auto-boot related options can be manually configured in the grub.cfg using the SR Linux CLI when SR Linux is operational. The command has the following format:

# system boot autoboot <command> [<arguments>]

where command must be one of the following:

SR Linux CLI: autoboot commands for grub.cfg update examples describes examples of SR Linux autoboot commands and available arguments.

Users can specify an ordered list of local images, kernels, or initial RAM disks to boot the system using the SR Linux CLI when SR Linux is operational. This directly translates into boot configuration in the grub, where the images or kernels are tried in the order specified by the user. The command has the following format:

# system boot <command> [<arguments>]

where command must be the following:

SR Linux CLI: image and kernel boot command example describes an example of the SR Linux image and kernel boot command and available argument.

The ZTP process can be manually started, stopped, and restarted using the SR Linux CLI when SR Linux is operational. The following command format is used:

# tools system boot autoboot <command>

where command must be one of the following:

SR Linux CLI: start, stop, and restart process command examples describes examples of SR Linux start, stop, and restart commands and available arguments.

The ZTP process status can be manually checked using the SR Linux CLI when SR Linux is operational.

The following is an example showing how to check the status of the ZTP process:

# tools system boot autoboot status

The following ZTP CLI commands are available at the console:

ztp
    — chassis
        — control
            — [--format table | json]
        — linecards
            — [--format table | json]
    configure-nos
        — --configurl <URL to download configuration>
    image
        — activate
            — --version <build version>
                — [--no-reboot]
        — bootorder
            — --version <up to 3 build versions>
        — delete
            — --version <build version>
        — list
            — [--format table | json]
        — upgrade
            — --imageurl <URL to download image> --md5url <URL to download md5 file>
                — [--no-reboot]
                — [--skip-check]
                — [--not-active]
        — version
            — --filename <name>
                — [--format table | json]
    option
        — autoboot
            — --status enable | disable
        — bootintf
            — --intf <boot interface>
        — clientid
            — --type serialid
        — downgrade
            — --status enable | disable
        — duration
            — --timeout <seconds> --retry <integer>
        — formatall
            — --status enable | disable
        — formatovl
            — --status enable | disable
        — formatsrletc
            — --status enable | disable
        — formatsrlopt
            — --status enable | disable
        — grubopt
            — [--key <text>]
            — [--value <text>]
            — [--delete]
        — list
            — [--format table | json]
        — nosinstall
            — --status enable | disable
        — reload
        — srlflags
            — [--value <text>]
            — [--delete]
    provision
        — --url <URL to download provisioning script>
    service
        — canstart
            — [--format table | json]
        — restart
            — --autoboot enable | disable
        — start
            — --autoboot enable | disable
        — status
            — [--format table | json]
        — stop
            — --autoboot enable | disable

The following SR Linux auto-boot related tool commands are available when the SR Linux is operational:

system
   — boot
        — autoboot
            — admin-state
            — attempts
            — client-id
            — interface
            — status
            — timeout
        — image

tools
    — system
        — boot
            — autoboot
                — execute-script
                — restart
                — start
                — status
                — stop

Refer to the SR Linux Data Model Reference for additional information about SR Linux commands and parameter descriptions.