Download Kubernetes upgrade bundle

Download the following from the NSP downloads page on the Nokia Support portal to a local station that is not part of the CLM deployment:

Note: The download takes considerable time; while the download is in progress, you may proceed to Step 2.

• NSP_CLM_K8S_DEPLOYER_R_r.tar.gz—software bundle for installing the registry and deploying the container environment

• associated .cksum file

where

R_r is the CLM release ID, in the form Major_minor

Verify CLM cluster readiness

Perform the following steps on each CLM cluster to verify that the cluster is fully operational.

1. Log in as the root user on the CLM cluster host.

2. Open a console window.

3. Enter the following to display the status of the CLM cluster nodes:

   # kubectl get nodes -A ↵

   The status of each cluster node is displayed.

   The CLM cluster is fully operational if the status of each node is Ready.

4. If any node is not in the Ready state, you must correct the condition; contact technical support for assistance, if required.

   Do not proceed to the next step until the issue is resolved.

5. Enter the following to display the CLM pod status:

   # kubectl get pods -A ↵

   The status of each pod is displayed.

   The CLM cluster is operational if the status of each pod is Running or Completed.

6. If any pod is not in the Running or Completed state, you must correct the condition; see the NSP Troubleshooting Guide for information about troubleshooting an errored pod.

Back up CLM databases

On the standalone CLM cluster, or the primary cluster in a DR deployment, perform Back up the existing CLM.

Back up system configuration files

Perform the following on the CLM deployer host in each data center.

Note: In a DR deployment, you must clearly identify the source cluster of each set of backup files.

1. Log in as the root or CLM admin user.

2. Back up the Kubernetes deployer configuration file:

   /opt/nsp/nsp-k8s-deployer-release-ID/config/k8s-deployer.yml

3. Back up the CLM deployer configuration file:

   /opt/nsp/NSP-CN-DEP-release-ID/config/nsp-deployer.yml

4. Back up the CLM configuration file:

   /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/appliedConfigs/nspConfiguratorConfigs.zip

5. Copy the backed-up files to a separate station that is not part of the CLM deployment.

Back up Kubernetes secrets

If the CLM is at Release 24.8 or later, back up the Kubernetes secrets, if not done as part of the database backup in Step 3.

1. Enter the following on the CLM deployer host:

   # cd /opt/nsp/NSP-CN-DEP-old-release-ID/bin ↵

2. Enter the following:

   # ./nspdeployerctl secret -o backup_file backup ↵

   where backup_file is the absolute path and name of the backup file to create

   As the secrets are backed up, messages like the following are displayed for each Kubernetes namespace:

   Backing up secrets to /opt/backupfile...

     Including secret namespace:ca-key-pair-external

     Including secret namespace:ca-key-pair-internal

     Including secret namespace:nsp-tls-store-pass

   When the backup is complete, the following prompt is displayed:

   Please provide an encryption password for backup_file

   enter aes-256-ctr encryption password:

3. Enter a password.

   The following prompt is displayed:

   Verifying - enter aes-256-ctr encryption password:

4. Re-enter the password.

   The backup file is encrypted using the password.

5. Record the password for use when restoring the backup.

6. Record the name of the data center associated with the backup.

7. Transfer the backup file to a secure location in a separate facility for safekeeping.

Verify checksum of downloaded file

It is strongly recommended that you verify the message digest of each CLM file that you download from the Nokia Support portal. The downloaded .cksum file contains checksums for comparison with the output of the RHEL md5sum, sha256sum, or sha512sum commands.

When the file download is complete, verify the file checksum.

1. Enter the following:

   # command file ↵

   where

   command is md5sum, sha256sum, or sha512sum

   file is the name of the downloaded file

   A file checksum is displayed.

2. Compare the checksum and the associated value in the .cksum file.

3. If the values do not match, the file download has failed. Retry the download, and then repeat Step 6.

Upgrade CLM registry

Perform Step 8 to Step 17 on the CLM deployer host in each data center, and then go to Step 18.

Note: In a DR deployment, you must perform the steps first on the CLM deployer host in the primary data center.

If the CLM deployer host is deployed in a VM created using a RHEL OS disk image, perform To apply a RHEL update to a CLM image-based OS.

Copy the downloaded NSP_CLM_K8S_DEPLOYER_R_r.tar.gz file to the /opt/nsp directory.

Expand the software bundle file.

1. Enter the following:

   # cd /opt/nsp ↵

2. Enter the following:

   # tar -zxvf NSP_CLM_K8S_DEPLOYER_R_r.tar.gz ↵

   The bundle file is expanded, and the following directories are created:

   • /opt/nsp/nsp-registry-new-release-ID

   • /opt/nsp/nsp-k8s-deployer-new-release-ID

3. After the file expansion completes successfully, enter the following to remove the bundle file, which is no longer required:

   # rm -f NSP_CLM_K8S_DEPLOYER_R_r.tar.gz ↵

If you are not upgrading Kubernetes from the immediately previous version supported by the CLM, but from an earlier version, you must uninstall the Kubernetes registry; otherwise, you can skip this step. See the Host Environment Compatibility Guide for NSP and CLM for information about Kubernetes version support.

Enter the following:

# /opt/nsp/nsp-registry-old-release-ID/bin/nspregistryctl uninstall ↵

The Kubernetes software is uninstalled.

Enter the following:

# cd /opt/nsp/nsp-registry-new-release-ID/bin ↵

Enter the following to perform the registry upgrade:

Note: During the registry upgrade, the registry may be temporarily unavailable. During such a period, a CLM pod that restarts on a new cluster node, or a pod that starts. is in the ImagePullBackOff state until the registry upgrade completes. Any such pods recover automatically after the upgrade, and no user intervention is required.

# ./nspregistryctl install ↵

If you did not perform Step 11 to uninstall the Kubernetes registry, go to Step 17.

Enter the following to import the original Kubernetes images.

# /opt/nsp/NSP-CN-DEP-base_load/bin/nspdeployerctl import ↵

where base_load is the initially deployed version of the installed CLM release

If you have applied any CLM service pack since the original deployment of the installed release, you must import the Kubernetes images from the latest applied service pack.

Enter the following to import the Kubernetes images from the latest applied service pack.

# /opt/nsp/NSP-CN-DEP-latest_load/bin/nspdeployerctl import ↵

where latest_load is the version of the latest applied CLM service pack

Verify CLM cluster initialization

When the registry upgrade is complete, verify the cluster initialization.

1. Enter the following:

   # kubectl get nodes ↵

   CLM deployer node status information like the following is displayed:

   NAME        STATUS    ROLES                  AGE     VERSION

   node_name   status    control-plane,master   xxdnnh   version

2. Verify that status is Ready; do not proceed to the next step otherwise.

3. Enter the following periodically to monitor the CLM cluster initialization:

   # kubectl get pods -A ↵

   The status of each pod is displayed.

   The CLM cluster is fully operational when the status of each pod is Running or Completed.

4. If any pod fails to enter the Running or Completed state, correct the condition; see the NSP Troubleshooting Guide for information about troubleshooting an errored pod.

Prepare to upgrade CLM Kubernetes deployer

Perform Step 19 to Step 32 on the CLM deployer host in each cluster, and then go to Step 33.

Note: In a DR deployment, you can perform the steps on each CLM deployer host concurrently; the order is unimportant.

You must merge the current k8s-deployer.yml settings into the new k8s-deployer.yml file.

Open the following files using a plain-text editor such as vi:

• old configuration file—/opt/nsp/nsp-k8s-deployer-old-release-ID/config/k8s-deployer.yml

• new configuration file—/opt/nsp/nsp-k8s-deployer-new-release-ID/config/k8s-deployer.yml

Apply the settings in the old file to the same parameters in the new file.

Close the old k8s-deployer.yml file.

In the new k8s-deployer.yml file, edit the following line in the cluster section to read:

  hosts: "/opt/nsp/nsp-k8s-deployer-new-release-ID/config/hosts.yml"

If you have disabled remote root access to the CLM cluster VMs,configure the following parameters in the cluster section, sshAccess subsection:

  sshAccess:

    userName: "user"

    privateKey: "path"

where

user is the designated CLM ansible user

path is the SSH key path, for example, /home/user/.ssh/id_rsa

Each CLM cluster VM has a parameter block like the following in the hosts section; configure the parameters for each VM, as required:

  - isIngress: value

    nodeIp: private_IP

    accessIp: public_address

    nodeName: node_name

where

value is true or false, and indicates whether the node acts as a load-balancer endpoint

private_IP is the VM IP address

public_IP is the public VM address; required only in a NAT environment

node_name is the VM name

In the following section, specify the virtual IP addresses for the CLM to use as the internal load-balancer endpoints.

Note: A single-node CLM cluster requires at least the client_IP address.

The addresses are the following values for CLM client, internal, and mediation access that you specify in the platform—ingressApplications section of the nsp-config.yml file during CLM cluster deployment.

In the internalAddresses subsection, if configured, otherwise, in the clientAddresses subsection:

• if configured, the advertised value

• otherwise, the virtualIp value

loadBalancerExternalIps:

    - client_IP

    - internal_IP

Configure the following parameter, which specifies whether dual-stack NE management is enabled:

Note: Dual-stack NE management can function only when the network environment is appropriately configured, for example:

• Only valid, non-link-local static or DHCPv6-assigned addresses are used.

• A physical or virtual IPv6 subnet is configured for IPv6 communication with the NEs.

  enable_dual_stack_networks: value

where value must be set to true if the cluster VMs support both IPv4 and IPv6 addressing

Save and close the new k8s-deployer.yml file.

Enter the following:

# cd /opt/nsp/nsp-k8s-deployer-new-release-ID/bin ↵

Enter the following to create the new hosts.yml file:

# ./nspk8sctl config -c ↵

Enter the following to list the node entries in the new hosts.yml file:

# ./nspk8sctl config -l ↵

Output like the following example for a one-node cluster is displayed:

Note: If NAT is used in the cluster:

• The ip value is the private IP address of the cluster node.

• The access_ip value is the public address of the cluster node.

Otherwise:

• The ip value is the private IP address of the cluster node.

• The access_ip value matches the ip value.

Note: The ansible_host value must match the access_ip value.

Existing cluster hosts configuration is:

node_1_name:

  ansible_host: 203.0.113.11

  ip: private_IP

  access_ip: public_IP

  node_labels:

    isIngress: "true"

Verify the IP addresses.

Enter the following to import the Kubernetes images to the repository:

.# ./nspk8sctl import ↵

The images are imported.

Stop and undeploy CLM cluster

Perform Step 34 to Step 36 on each CLM cluster, and then go to Step 37.

Note: In a DR deployment, you must perform the steps first on the standby cluster.

Perform the following steps on the CLM deployer host to preserve the existing cluster data.

1. Open the following file using a plain-text editor such as vi:

   /opt/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/config/nsp-config.yml

2. Edit the following line in the platform section, kubernetes subsection to read:

     deleteOnUndeploy:false

3. Save and close the file.

Enter the following on the CLM deployer host to undeploy the CLM cluster:

Note: If you are upgrading a standalone CLM system, or the primary CLM cluster in a DR deployment, this step marks the beginning of the network management outage associated with the upgrade.

Note: If the CLM cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass uninstall --undeploy --clean

# /opt/nsp/NSP-CN-DEP-old-release-ID/bin/nspdeployerctl uninstall --undeploy --clean ↵

The CLM cluster is undeployed.

On the CLM cluster host, enter the following periodically to display the status of the Kubernetes system pods:

Note: You must not proceed to the next step until the output lists only the following:

• pods in kube-system namespace

• nsp-backup-storage pod

# kubectl get pods -A ↵

The pods are listed.

Deploy new CLM Kubernetes software

Perform Step 38 to the end of the procedure on each CLM cluster.

Note: In a DR deployment, you must perform the steps first on the primary cluster.

If the new Kubernetes version is more than one version later than the existing version, you cannot upgrade the software; instead, you must completely replace the existing version by uninstalling the software and then installing the new version.

Perform the following steps.

Note: The software replacement on a cluster takes approximately 30 minutes, and is the recommended option.

Note: If the CLM cluster VMs do not have the required SSH key, you must include the --ask-pass argument in a command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspk8sctl --ask-pass uninstall

1. Enter the following on the CLM deployer host:

   # cd /opt/nsp/nsp-k8s-deployer-old-release-ID/bin ↵

2. Enter the following:

   # ./nspk8sctl uninstall ↵

   The existing Kubernetes software is uninstalled.

Enter the following:

# cd /opt/nsp/nsp-k8s-deployer-new-release-ID/bin ↵

If the CLM is at Release 24.8 or later, restore the backed-up Kubernetes secrets.

1. Enter the following to restore the Kubernetes secrets:

   ./nspdeployerctl secret -i backup_file restore ↵

   where backup_file is the secrets backup file created earlier in the procedure

   The following prompt is displayed:

   Please provide the encryption password for /opt/backupfile

   enter aes-256-ctr decryption password:

2. Enter the password recorded during the backup creation.

   As the secrets are restored, messages like the following are displayed for each Kubernetes namespace:

   Restoring secrets from backup_file...

   secret/ca-key-pair-external created

     Restored secret namespace:ca-key-pair-external

   secret/ca-key-pair-internal created

     Restored secret namespace:ca-key-pair-internal

   secret/nsp-tls-store-pass created

     Restored secret namespace:nsp-tls-store-pass

Enter the following to install or upgrade the Kubernetes software:

Note: If you do not uninstall Kubernetes in Step 38, the software is upgraded rather than installed. An upgrade takes considerable time; during the upgrade process, each cluster node is individually cordoned, drained, upgraded, and uncordoned. The operation on each node may take 15 minutes or more.

# ./nspk8sctl install ↵

The new CLM Kubernetes environment is deployed.

Enter the following on the CLM cluster host periodically to display the status of the Kubernetes system pods:

Note: You must not proceed to the next step until each pod STATUS reads Running or Completed.

# kubectl get pods -A ↵

The pods are listed.

Enter the following periodically on the CLM cluster host to display the status of the CLM cluster nodes:

Note: You must not proceed to the next step until each node STATUS reads Ready.

# kubectl get nodes -o wide ↵

The CLM cluster nodes are listed, as shown in the following example:

NAME    STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   

node1   Ready    master   nd    version   int_IP        ext_IP

Update the CLM cluster configuration.

1. Open the following files using a plain-text editor such as vi:

   • old configuration file—/opt/nsp/NSP-CN-DEP-old-release-ID/config/nsp-deployer.yml

   • new configuration file—/opt/nsp/NSP-CN-DEP-new-release-ID/config/nsp-deployer.yml

2. Merge the settings from the old file into the new file.

3. Edit the following line in the new file to read:

     hosts: "/opt/nsp/nsp-k8s-deployer-new-release-ID/config/hosts.yml"

4. Save and close the new nsp-deployer.yml file.

5. Close the old nsp-deployer.yml file.

Redeploy CLM software

Enter the following on the CLM deployer host:

# cd /opt/nsp/NSP-CN-DEP-new-release-ID/bin ↵

Enter the following:

# ./nspdeployerctl config ↵

Enter the following:

Note: If the CLM cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass install --config --deploy

# ./nspdeployerctl install --config --deploy ↵

The CLM starts.

Verify CLM initialization

On the CLM cluster host, monitor and validate the CLM cluster initialization.

Note: You must not proceed to the next step until each CLM pod is operational.

1. Enter the following every few minutes:

   # kubectl get pods -A ↵

   The status of each CLM cluster pod is listed; all pods are running when the displayed STATUS value is Running or Completed.

   The nsp deployer log file is /var/log/nspdeployerctl.log.

2. If any pod fails to enter the Running or Completed state, see the NSP Troubleshooting Guide for information about troubleshooting an errored pod.

Enter the following on the CLM cluster host to display the status of the CLM cluster members:

Note: You must not proceed to the next step until each node is operational.

# kubectl get nodes ↵

The status of each node is listed; all nodes are operational when the displayed STATUS value is Ready.

The CLM Kubernetes deployer log file is /var/log/nspk8sctl.log.

Verify upgraded CLM cluster operation

Use a browser to open the CLM cluster URL.

Verify the following.

• In a DR deployment, if you specify the standby cluster address, the browser is redirected to the primary cluster address.

• The CLM sign-in page opens.

• The CLM UI opens after you sign in.

As required, use the CLM to monitor device discovery and to check network management functions.

Note: You do not need to perform the step on the standby CLM cluster.

Note: If you are upgrading Kubernetes in a standalone CLM cluster, or the primary CLM cluster in a DR deployment, the completed CLM cluster initialization marks the end of the network management outage.

Purge Kubernetes image files

Note: You must perform this and the following step only after you verify that the CLM system is operationally stable and that an upgrade rollback is not required.

Enter the following on the CLM deployer host:

# cd /opt/nsp/nsp-k8s-deployer-new-release-ID/bin ↵

Enter the following on the CLM deployer host:

# ./nspk8sctl purge-registry -e ↵

The images are purged.

Close the open console windows.

End of steps