Stop and undeploy NSP cluster

Log in as the root user on the appropriate station, based on the installed NSP release:

• Release 22.3—NSP configurator VM

• Release 22.6—NSP deployer host

Perform the following steps to preserve the existing cluster data.

1. Open the appropriate file, based on the installed NSP release, using a plain-text editor such as vi:

   • Release 22.3—/opt/nsp/NSP-CN-release-ID/config/nsp-config.yml

   • Release 22.6—/opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/config/nsp-config.yml

2. Edit the following line in the platform section, kubernetes subsection to read:

     deleteOnUndeploy:false

3. Save and close the file.

CAUTION Data Loss

Undeploying an NSP cluster as described in this step permanently removes the cluster data.

If you are upgrading a DR NSP system, you must ensure that you have the latest database backup from the primary cluster before you perform this step.

Undeploy the NSP cluster.

Note: If you are upgrading a standalone NSP system, or the primary NSP cluster in a DR deployment, this step marks the beginning of the network management outage associated with the upgrade.

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in a command, as shown in the following example, and are subsequently prompted for the common root password of each cluster member:

nsp-config.bash --ask-pass --undeploy --clean | nspdeployerctl uninstall --undeploy --clean

1. If you are upgrading from Release 22.3, enter the following:

   # /opt/nsp/NSP-CN-DEP-release-ID/bin/nsp-config.bash --undeploy --clean ↵

2. If you are upgrading from Release 22.6 or later, enter the following:

   # /opt/nsp/NSP-CN-DEP-release-ID/bin/nspdeployerctl uninstall --undeploy --clean ↵

The NSP cluster is undeployed.

Before you create new NSP cluster VMs, you must disable each existing VM in the NSP cluster. The following options are available.

• stop but do not delete existing VMs—simplifies upgrade rollback, but requires sufficient platform resources for new VMs

• delete existing VMs—complicates upgrade rollback, but conserves platform resources

1. Log in as the root user on the station that hosts the NSP cluster VM.

2. Enter the following:

   # virsh list ↵

   The NSP cluster VMs are listed.

3. Enter the following for each listed VM:

   # virsh destroy VM ↵

   where VM is the VM name

   The VM stops.

4. To delete the VM, enter the following:

   # virsh undefine VM ↵

   The VM is deleted.

Uninstall IPRC, CDRC

If you are upgrading from Release 22.3 and the NSP deployment includes IP resource control or cross-domain resource control, uninstall each IPRC and CDRC server.

1. Log in as the root user on the IPRC or CDRC server station that has the extracted NSP software bundle from the previous installation or upgrade.

2. Open a console window.

3. Navigate to the NSP installer directory; enter the following:

   # cd path/NSD_NRC_R_r ↵

   where

   path is the directory that contains the extracted NSP software bundle

   R_r is the NSP software release, in the form MAJOR_minor

4. Edit the hosts file in the directory to list only the addresses of the NSP components to uninstall.

   Note: The uninstaller uses the same root password on each server in the list; ensure that you specify only servers that have the same root password.

5. Enter the following; include the --ask-pass option only if each target station has the same root user password:

   # ./bin/uninstall.sh --ask-pass ↵

6. Enter the root user password each time you are prompted.

   The NSP software is removed from each server listed in the hosts file.

7. Close the console window.

Preserve NSP cluster configuration

Log in as the root user on the existing NSP deployer host.

Open a console window.

Perform one of the following.

1. If you are upgrading from Release 22.3, copy the following file to a separate station that is unaffected by the upgrade activity:

   /opt/nsp/kubespray/inventory/nsp-deployer-default/hosts.yml

2. If you are upgrading from NSP Release 22.6, copy the following file to a separate station that is unaffected by the upgrade activity:

   /opt/nsp/nsp-k8s-deployer-release-ID/config/k8s-deployer.yml

Copy the following file to a separate station that is unaffected by the upgrade activity:

/opt/nsp/NSP-CN-DEP-release-ID/config/nsp-deployer.yml

Create NSP deployer host

Log in as the root user on the station that will host the NSP deployer host VM.

Open a console window.

Enter the following:

# dnf -y install virt-install libguestfs-tools ↵

Before you create the new NSP deployer host VM, you must disable the existing VM; the following options are available.

• stop but do not delete existing VM—simplifies upgrade rollback, but VM consumes platform resources

• delete existing VM—complicates upgrade rollback, but conserves platform resources

1. Log in as the root user on the station that hosts the NSP deployer host VM.

2. Enter the following to list the VMs on the station:

   # virsh list ↵

   The VMs are listed.

3. Enter the following:

   # virsh destroy VM ↵

   where VM is the name of the NSP deployer host VM

   The NSP deployer host VM stops.

4. To delete the VM, enter the following:

   Note: If you intend to use the same VM name for the new NSP deployer host VM, you must delete the VM.

   # virsh undefine VM ↵

   The VM is deleted.

Perform one of the following to create the new NSP deployer host VM.

Note: The NSP deployer host VM requires a hostname; you must change the default of ‘localhost’ to an actual hostname.

1. Deploy the downloaded NSP_K8S_PLATFORM_RHEL8_yy_mm.qcow2 disk image; perform Step 6 to Step 16 of To deploy an NSP RHEL qcow2 disk image.

2. Deploy the NSP_K8S_PLATFORM_RHEL8_yy_mm.ova disk image; see the documentation for your virtualization environment for information.

   Note: For OVA-image deployment, it is strongly recommended that you mount the /opt directory on a separate hard disk that has sufficient capacity to allow for future expansion.

3. Manually install the RHEL OS and configure the disk partitions, as described in Manual NSP RHEL OS installation and Chapter 2, NSP disk setup and partitioning.

Configure NSP deployer host network interface

Enter the following to open a console session on the NSP deployer host VM:

# virsh console deployer_host ↵

You are prompted for credentials.

Enter the following credentials:

• username—root

• password—available from technical support

A virtual serial console session opens.

Enter the following:

# ip a ↵

The available network interfaces are listed; information like the following is displayed for each:

if_n: if_name: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

    link/ether MAC_address

    inet IPv4_address/v4_netmask brd broadcast_address scope global noprefixroute if_name

       valid_lft forever preferred_lft forever

    inet6 IPv6_address/v6_netmask scope link

       valid_lft forever preferred_lft forever

Record the if_name and MAC_address values of the interface that you intend to use.

Enter the following:

# nmcli con add con-name con_name ifname if_name type ethernet mac MAC_address ↵

where

con_name is a connection name that you assign to the interface for ease of identification

if_name is the interface name recorded in Step 18

MAC_address is the MAC address recorded in Step 18

Enter the following:

# nmcli con mod con_name ipv4.addresses IP_address/netmask ↵

where

con_name is the connection name assigned in Step 19

IP_address is the IP address to assign to the interface

netmask is the subnet mask to assign

Enter the following:

# nmcli con mod con_name ipv4.method static ↵

Enter the following:

# nmcli con mod con_name ipv4.gateway gateway_IP ↵

gateway_IP is the gateway IP address to assign

Enter the following:

Note: You must specify a DNS name server. If DNS is not deployed, you must use a non-routable IP address as a nameserver entry.

Note: Any hostnames used in an NSP deployment must be resolved by a DNS server.

Note: An NSP deployment that uses IPv6 networking for client communication must use a hostname configuration.

# nmcli con mod con_name ipv4.dns nameserver_1,nameserver_2...nameserver_n ↵

where nameserver_1 to nameserver_n are the available DNS name servers

To optionally specify one or more DNS search domains, enter the following:

# nmcli con mod con_name ipv4.dns-search search_domains ↵

where search_domains is a comma-separated list of DNS search domains

Enter the following to set the hostname:

# hostnamectl set-hostname hostname ↵

where hostname is the hostname to assign

Enter the following to reboot the deployer host VM:

# systemctl reboot ↵

Close the console session by pressing Ctrl+] (right bracket).

Install NSP Kubernetes registry

Log in as the root or NSP admin user on the NSP deployer host.

Enter the following:

# mkdir /opt/nsp ↵

Transfer the following downloaded file to the /opt/nsp directory on the NSP deployer host:

NSP_K8S_DEPLOYER_R_r.tar.gz

Enter the following:

# cd /opt/nsp ↵

Enter the following:

# tar xvf NSP_K8S_DEPLOYER_R_r.tar.gz ↵

where R_r is the NSP release ID, in the form Major_minor

The file is expanded, and the following directories are created:

• /opt/nsp/nsp-k8s-deployer-release-ID

• /opt/nsp/nsp-registry-release-ID

After the file expansion completes successfully, enter the following to remove the bundle file, which is no longer required:

# rm -f NSP_K8S_DEPLOYER_R_r.tar.gz ↵

The file is deleted.

Enter the following:

# cd nsp-registry-release-ID/bin ↵

Enter the following:

# ./nspregistryctl install ↵

The following prompt is displayed.

Enter a registry admin password:

Create a registry administrator password; the password must:

• be a minimum of 10 characters

• include at least one:

  • uppercase character

  • lowercase character

  • digit

  • special character in the following list:

    ! # $ % & ( ) * + , - . / : ; = ? @ \ ^ _ { | }

Enter the password.

The following prompt is displayed.

Confirm the registry admin password:

Re-enter the password.

The registry installation begins, and messages like the following are displayed.

✔ New installation detected.

✔ Initialize system.

date time Copy container images ...

date time Install/update package [container-selinux] ...

✔ Installation of container-selinux has completed.

date time Install/update package [k3s-selinux] ...

✔ Installation of k3s-selinux has completed.

date time Setup required tools ...

✔ Initialization has completed.

date time Install k3s ...

date time Waiting for up to 10 minutes for k3s initialization ...

..............................................

✔ Installation of k3s has completed.

➜ Generate self-signed key and cert.

date time Registry TLS key file: /opt/nsp/nsp-registry/tls/nokia-nsp-registry.key

date time Registry TLS cert file: /opt/nsp/nsp-registry/tls/nokia-nsp-registry.crt

date time Install registry apps ...

date time Waiting for up to 10 minutes for registry services to be ready ...

..........

✔ Registry apps installation is completed.

date time Generate artifacts ...

date time Apply artifacts ...

date time Setup registry.nsp.nokia.local certs ...

date time Setup a default project [nsp] ...

date time Setup a cron to regenerate the k3s certificate [nsp] ...

✔ Post configuration is completed.

✔ Installation has completed.

Migrate legacy cluster parameters

If you are upgrading from Release 22.6, go to Step 44.

Enter the following:

# cd /opt/nsp/nsp-k8s-deployer-release-ID/tools ↵

Copy the hosts.yml file saved in Step 8 to the current directory.

Enter the following:

# ./extracthosts hosts.yml ↵

The current NSP cluster node entries are displayed, as shown in the following example for a three-node cluster:

hosts:

- nodeName: node1

nodeIp: 192.168.96.11

accessIp: 203.0.113.11

- nodeName: node2

nodeIp: 192.168.96.12

accessIp: 203.0.113.12

- nodeName: node3

nodeIp: 192.168.96.13

accessIp: 203.0.113.13

Review the output to ensure that each node entry is correct.

Open the following new deployer configuration file using a plain-text editor such as vi:

/opt/nsp/nsp-k8s-deployer-release-ID/config/k8s-deployer.yml

Perform one of the following to configure the cluster node entries for the deployment.

1. If you are upgrading from Release 22.3, configure the cluster node entries in the new configuration file using the extracted hosts.yml file output. Table 8-1, hosts.yml and k8s-deployer.yml parameters lists the former and new parameter names, and the required values.

   Note: The nodeName value:

   • can include only ASCII alphanumeric and hyphen characters

   • cannot include an upper-case character

   • cannot begin or end with a hyphen

   • cannot begin with a number

   • cannot include an underscore

   • must end with a number

   Note: The node order in the k8s-deployer.yml file must match the order in the hosts.yml file.

   Table 8-1: hosts.yml and k8s-deployer.yml parameters

   hosts.yml parameter	k8s-deployer.yml parameter	Value
   node_entry_header	nodeName	node hostname
   ansible_host	—	same IP address used for access_ip
   ip	nodeIp	IP address; private, if NAT is used
   access_ip	accessIp	IP address; public, if NAT is used
   —	isIngress	whether the node acts as a load-balancer endpoint

2. If you are upgrading from Release 22.6, merge the current k8s-deployer.yml settings into the new file.

   1. Open the old k8s-deployer.yml file saved in Step 8.

   2. Apply the settings in the old file to the same parameters in the new file.

   3. Close the old k8s-deployer.yml file.

Edit the following line in the cluster section of the new file to read:

  hosts: "/opt/nsp/nsp-k8s-deployer-release-ID/config/hosts.yml"

If you have disabled remote root access to the NSP cluster VMs, configure the following parameters in the cluster section, sshAccess subsection:

  sshAccess:

    userName: "user"

    privateKey: "path"

where

user is the designated NSP ansible user

path is the SSH key path, for example, /home/user/.ssh/id_rsa

In the following section, specify the virtual IP addresses for the NSP to use as the internal load-balancer endpoints.

Note: A single-node NSP cluster requires at least the client_IP address.

The addresses are the virtualIP values for NSP client, internal, and mediation access that you intend to specify in Step 102 and Step 103 in the nsp-config.yml file.

loadBalancerExternalIps:

    - client_IP

    - internal_IP

    - trapV4_mediation_IP

    - trapV6_mediation_IP

    - flowV4_mediation_IP

    - flowV6_mediation_IP

Configure the following parameter, which specifies whether dual-stack NE management is enabled:

Note: Dual-stack NE management can function only when the network environment is appropriately configured, for example:

• Only valid, non-link-local static or DHCPv6-assigned addresses are used.

• A physical or virtual IPv6 subnet is configured for IPv6 communication with the NEs.

  enable_dual_stack_networks: value

where value must be set to true if the cluster VMs support both IPv4 and IPv6 addressing

If the existing deployment includes an RPM-based IPRC, add a node entry for the IPRC after the existing node entries.

If the deployment includes dedicated MDM cluster VMs, as identified in Step 24 of To prepare for an NSP system upgrade from Release 22.6 or earlier, add an entry for each identified VM.

Note: If the deployment includes the IPRC, you must add the MDM node entries after the IPRC entry.

Save and close the new k8s-deployer.yml file.

Create NSP cluster VMs

For each required NSP cluster VM, perform one of the following to create the VM.

Note: Each NSP cluster VM requires a hostname; you must change the default of ‘localhost’ to an actual hostname.

1. Deploy the downloaded NSP_K8S_PLATFORM_RHEL8_yy_mm.qcow2 disk image; perform Step 6 to Step 16 of To deploy an NSP RHEL qcow2 disk image.

2. Deploy the NSP_K8S_PLATFORM_RHEL8_yy_mm.ova disk image; see the documentation for your virtualization environment for information.

   Note: For OVA-image deployment, it is strongly recommended that you mount the /opt directory on a separate hard disk that has sufficient capacity to allow for future expansion.

3. Manually install the RHEL OS and configure the disk partitions, as described in Manual NSP RHEL OS installation and Chapter 2, NSP disk setup and partitioning.

Perform Step 55 to Step 72 for each NSP cluster VM to configure the required interfaces.

Configure NSP cluster interfaces

Enter the following on the NSP deployer host to open a console session on the VM:

# virsh console NSP_cluster_VM ↵

where NSP_cluster_VM is the VM name

You are prompted for credentials.

Enter the following credentials:

• username—root

• password—available from technical support

A virtual serial console session opens on the NSP cluster VM.

Enter the following:

# ip a ↵

The available network interfaces are listed; information like the following is displayed for each:

if_n: if_name: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

    link/ether MAC_address

    inet IPv4_address/v4_netmask brd broadcast_address scope global noprefixroute if_name

       valid_lft forever preferred_lft forever

    inet6 IPv6_address/v6_netmask scope link

       valid_lft forever preferred_lft forever

Record the if_name and MAC_address values of the interfaces that you intend to use.

Enter the following for each interface:

# nmcli con add con-name con_name ifname if_name type ethernet mac MAC_address ↵

where

con_name is a connection name that you assign to the interface for ease of identification; for example, ClientInterface or MediationInterface

if_name is the interface name recorded in Step 58

MAC_address is the MAC address recorded in Step 58

Enter the following for each interface:

# nmcli con mod con_name ipv4.addresses IP_address/netmask ↵

where

con_name is the connection name assigned in Step 59

IP_address is the IP address to assign to the interface

netmask is the subnet mask to assign

Enter the following for each interface:

# nmcli con mod con_name ipv4.method static ↵

Enter the following for each interface:

# nmcli con mod con_name ipv4.gateway gateway_IP ↵

gateway_IP is the gateway IP address to assign

Note: This command sets the default gateway on the primary interface and the gateways for all secondary interfaces.

Enter the following for all secondary interfaces:

# nmcli con mod con_name ipv4.never-default yes ↵

Enter the following for each interface:

Note: You must specify a DNS name server. If DNS is not deployed, you must use a non-routable IP address as a nameserver entry.

Note: Any hostnames used in an NSP deployment must be resolved by a DNS server.

Note: An NSP deployment that uses IPv6 networking for client communication must use a hostname configuration.

# nmcli con mod con_name ipv4.dns nameserver_1,nameserver_2...nameserver_n ↵

where nameserver_1 to nameserver_n are the available DNS name servers

To optionally specify one or more DNS search domains, enter the following for each interface:

# nmcli con mod con_name ipv4.dns-search search_domains ↵

where search_domains is a comma-separated list of DNS search domains

Open the following file with a plain-text editor such as vi:

/etc/sysctl.conf

Locate the following line:

vm.max_map_count=value

Edit the line to read as follows; if the line is not present, add the line to the end of the file:

vm.max_map_count=262144

Save and close the file.

If you are installing in a KVM environment, enter the following:

# mkdir /opt/nsp ↵

Enter the following to reboot the NSP cluster VM:

# systemctl reboot ↵

Close the console session by pressing Ctrl+] (right bracket).

Deploy container environment

Log in as the root or NSP admin user on the NSP deployer host.

Open a console window.

For password-free NSP deployer host access to the NSP cluster VMs, you require an SSH key.

To generate and distribute the SSH key, perform the following steps.

1. Enter the following:

   # ssh-keygen -N "" -f path -t rsa ↵

   where path is the SSH key file path, for example, /home/user/.ssh/id_rsa

   An SSH key is generated.

2. Enter the following for each NSP cluster VM to distribute the key to the VM.

   # ssh-copy-id -i key_file user@address ↵

   where

   user is the designated NSP ansible user configured in Step 47, if root-user access is restricted; otherwise, user@ is not required

   key_file is the SSH key file, for example, /home/user/.ssh/id_rsa.pub

   address is the NSP cluster VM IP address

Enter the following:

# cd /opt/nsp/nsp-k8s-deployer-release-ID/bin ↵

Enter the following to create the new hosts.yml file:

# ./nspk8sctl config -c ↵

Enter the following to list the node entries in the new hosts.yml file:

# ./nspk8sctl config -l ↵

Output like the following example for a four-node cluster is displayed:

Note: If NAT is used in the cluster:

• The ip value is the private IP address of the cluster node.

• The access_ip value is the public address of the cluster node.

Otherwise:

• The ip value is the private IP address of the cluster node.

• The access_ip value matches the ip value.

Note: The ansible_host value must match the access_ip value.

Existing cluster hosts configuration is:

node_1_name:

  ansible_host: 203.0.113.11

  ip: private_IP

  access_ip: public_IP

  node_labels:

    isIngress: "true"

node_2_name:

  ansible_host: 203.0.113.12

  ip: private_IP

  access_ip: public_IP

  node_labels:

    isIngress: "true"

node_3_name:

  ansible_host: 203.0.113.13

  ip: private_IP

  access_ip: public_IP

  node_labels:

    isIngress: "true"

node_4_name:

  ansible_host: 203.0.113.14

  ip: private_IP

  access_ip: public_IP

  node_labels:

    isIngress: "false"

Verify the IP addresses.

Enter the following to import the Kubernetes images to the repository:

.# ./nspk8sctl import ↵

Enter the following:

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspk8sctl --ask-pass install

.# ./nspk8sctl install ↵

The NSP Kubernetes environment is deployed.

The NSP cluster member named node1 is designated the NSP cluster host for future configuration activities; record the NSP cluster host IP address for future reference.

Open a console window on the NSP cluster host.

Enter the following periodically to display the status of the Kubernetes system pods:

Note: You must not proceed to the next step until each pod STATUS reads Running or Completed.

# kubectl get pods -A ↵

The pods are listed.

Enter the following periodically to display the status of the NSP cluster nodes:

Note: You must not proceed to the next step until each node STATUS reads Ready.

# kubectl get nodes -o wide ↵

The NSP cluster nodes are listed, as shown in the following three-node cluster example:

NAME    STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   

node1   Ready    master   nd    version   int_IP        ext_IP

node2   Ready    master   nd    version   int_IP        ext_IP

node3   Ready    <none>   nd    version   int_IP        ext_IP

Restore NSP system files

Transfer the following downloaded file to the /opt/nsp directory on the NSP deployer host:

NSP_DEPLOYER_R_r.tar.gz

Enter the following on the NSP deployer host:

# cd /opt/nsp ↵

Enter the following:

# tar xvf NSP_DEPLOYER_R_r.tar.gz ↵

where R_r is the NSP release ID, in the form Major_minor

The bundle file is expanded, and the following directory of NSP installation files is created:

/opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID

Enter the following:

# rm -f NSP_DEPLOYER_R_r.tar.gz ↵

The bundle file is deleted.

Restore the required NSP configuration files.

1. Enter the following:

   # mkdir /tmp/appliedConfig ↵

2. Enter the following:

   # cd /tmp/appliedConfig ↵

3. Transfer the following configuration backup file saved in To prepare for an NSP system upgrade from Release 22.6 or earlier to the /tmp/appliedConfig directory:

   nspConfiguratorConfigs.zip

4. Enter the following:

   # unzip nspConfiguratorConfigs.zip ↵

   The configuration files are extracted to the current directory, and include some or all of the following, depending on the previous deployment:

   • license file

   • nsp-config.yml file

   • TLS files; may include subdirectories

   • SSH key files

   • nsp-configurator/generated directory content

Label NSP cluster nodes

Merge the current nsp-deployer.yml settings into the new nsp-deployer.yml file.

1. Open the following new cluster configuration file and the old nsp-deployer.yml file saved in Step 9 using a plain-text editor such as vi:

   /opt/nsp/NSP-CN-DEP-release-ID/config/nsp-deployer.yml

2. Apply the settings in the old file to the same parameters in the new file.

3. Close the old k8s-deployer.yml file.

If you have disabled remote root access to the NSP cluster VMs, configure the following parameters in the cluster section, sshAccess subsection:

  sshAccess:

    userName: "user"

    privateKey: "path"

where

user is the designated NSP ansible user

path is the SSH key path, for example, /home/user/.ssh/id_rsa

Save and close the new nsp-deployer.yml file.

Enter the following to apply the node labels to the NSP cluster:

# ./nspdeployerctl config ↵

Enter the following to import the NSP images and Helm charts to the NSP Kubernetes registry:

Note: The import operation may take 20 minutes or longer.

# ./nspdeployerctl import ↵

Open a console window on the NSP cluster host.

Enter the following to display the node labels:

# kubectl get nodes --show-labels ↵

Cluster node information is displayed.

View the information to ensure that all NSP labels are added to the cluster VMs.

Configure NSP software

You must merge the nsp-config.yml file content from the existing deployment into the new nsp-config.yml file.

Open the following files using a plain-text editor such as vi:

• former configuration file—/tmp/appliedConfig/nsp-config.yml file extracted in Step 90

• new configuration file—/opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/config/nsp-config.yml

Note: See nsp-config.yml file format for configuration information.

Copy each configured parameter line from the previous nsp-config.yml file and use the line to overwrite the same line in the new file, if applicable. The following parameters are not present in the new nsp-config.yml file, and are not to be merged:

• platform section—address configuration now in ingressApplications section

  • advertisedAddress

  • mediationAdvertisedAddress

  • mediationAdvertisedAddressIpv6

  • internalAdvertisedAddress

• elb section—all parameters; section removed

• tls section—information now held in secrets:

  • truststorePass

  • keystorePass

  • customKey

  • customCert

• mtls section—information now held in secrets:

  • mtlsCACert

  • mtlsClientCert

  • mtlsKey

• sso section—authMode parameter obsolete

Note: You must maintain the structure of the new file, as any new configuration options for the new release must remain.

Note: You must replace each configuration line entirely, and must preserve the leading spaces in each line.

Note: If NSP application-log forwarding to NSP Elasticsearch is enabled, special configuration is required.

• OpenSearch replaces Elasticsearch as the local log-viewing utility. Consequently, the Elasticsearch configuration cannot be directly copied from the current NSP configuration to the new configuration. Instead, you must configure the parameters in the logging—forwarding—applicationLogs—opensearch section of the new NSP configuration file.

• Elasticsearch is introduced as a remote log-forwarding option. You can enable NSP application-log forwarding to a remote Elasticsearch server in the logging—forwarding—applicationLogs—elasticsearch section of the new NSP configuration file.

  See Centralized logging for more information about configuring NSP logging options.

Configure the following parameter in the platform section as shown below:

Note: You must preserve the lead spacing of the line.

  clusterHost: "cluster_host_address"

where cluster_host_address is the address of NSP cluster member node1, which is subsequently used for cluster management operations

You must apply the address values from the former configuration file to the new parameters.

Configure the following parameters in the platform section, ingressApplications subsection as shown below.

Each address is an address from the ingressApplications section of the k8s-deployer.yml file described in Step 48.

Note: The client_IP value is mandatory; the address is used for interfaces that remain unconfigured, such as in a single-interface deployment.

Note: If the client network uses IPv6, you must specify the NSP cluster hostname as the client_IP value.

Note: The trapForwarder addresses that you specify must differ from the client_IP value, even in a single-interface deployment.

  ingressApplications:

    ingressController:

      clientAddresses:

        virtualIp: "client_IP"

        advertised: "client_public_address"

      internalAddresses:

        virtualIp: "internal_IP"

        advertised: "internal_public_address"

    trapForwarder:

      mediationAddresses:

        virtualIpV4: "trapV4_mediation_IP"

        advertisedV4: "trapV4_mediation_public_address"

        virtualIpV6: "trapV6_mediation_IP"

        advertisedV6: "trapV6_mediation_public_address"

where

client_IP is the address for external client access

internal_IP is the address for internal communication

trapV4_mediation_IP is the address for IPv4 network mediation

trapV6_mediation_IP is the address for IPv6 network mediation

each public_address value is an optional address to advertise instead of the associated _IP value, for example, in a NAT environment

If flow collection is enabled, configure the following parameters in the platform section, ingressApplications subsection as shown below.

Each address is an address from the ingressApplications section of the k8s-deployer.yml file described in Step 48.

    flowForwarder:

      mediationAddresses:

        virtualIpV4: "flowV4_mediation_IP"

        advertisedV4: "flowV4_mediation_public_address"

        virtualIpV6: "flowV6_mediation_IP"

        advertisedV6: "flowV6_mediation_public_address"

where

flowV4_mediation_IP is the address for IPv4 flow collection

flowV6_mediation_IP is the address for IPv6 flow collection

each _public_address value is an optional address to advertise instead of the associated mediation_IP value, for example, in a NAT environment

Configure the type parameter in the deployment section as shown below:

deployment:

    type: "deployment_type"

where deployment_type is one of the parameter options listed in the section

If the NSP system currently performs model-driven telemetry or classic telemetry statistics collection, perform the following steps.

1. Enable the following installation option:

   id: networkInfrastructureManagement-gnmiTelemetry

2. Configure the throughputFactor parameter in the nsp—modules—telemetry—gnmi section; see the parameter description in the nsp-config.yml for the required value, which is based on the management scale:

              throughputFactor: n

   where n is the required throughput factor for your deployment

If required, configure NSP collection of accounting statistics. Perform the following steps:

1. Enable the following installation option:

   id: networkInfrastructureManagement-accountingTelemetry

2. Configure the collectFromClassicNes parameter in the nsp—modules—telemetry—accounting section. The default is false; toggle the flag to true to enable the NSP to process accounting files from classic NEs.

CAUTION: If the collectFromClassicNes flag is true: to prevent duplicate file collection you must disable file rollover traps on the NE and disable the polling policy for tmnxLogFileIdEntry from NFM-P. See the NSP Classic Management User Guide and the NE CLI documentation.

After the rollover trap is disabled on the NE, the NFM-P and all other third-party systems that depend on the trap to trigger file collection will no longer collect accounting and event files.

If all of the following are true, configure the following parameters in the integrations section:

• The NSP system includes the NFM-P.

• You want the NFM-P to forward system metrics to the NSP cluster.

• The NFM-P main server and main database are on separate stations:

    nfmpDB:

      primaryIp: ""

      standbyIp: ""

If both of the following are true, configure the following parameters in the integrations section:

• The NSP system includes the NFM-P.

• You want the NFM-P to forward system metrics to the NSP cluster.

• The NFM-P system includes one or more auxiliary servers:

    auxServer:

      primaryIpList: ""

      standbyIpList: ""

If the NSP system includes one or more Release 22 analytics servers that are not being upgraded as part of the current NSP system upgrade, you must enable NSP and analytics compatibility; otherwise, you can skip this step.

Set the legacyPortEnabled parameter in the analyticsServer subsection of the integrations section to true as shown below:

  analyticsServer:

    legacyPortEnabled: true

If required, configure the user authentication parameters in the sso section, as shown below; see NSP SSO configuration parameters for configuration information.

If you have an updated license, ensure that the location of your license.zip file, as indicated in the nsp-config.yml file, is in the correct location on the NSP deployer host.

Save and close the new nsp-config.yml file.

Close the previous nsp-config.yml file.

If you are upgrading the new primary (former standby) cluster in a DR deployment, stop here and return to Workflow for DR NSP system upgrade from Release 22.6 or earlier.

Restore dedicated MDM node labels

If you are not including any dedicated MDM nodes in addition to the number of member nodes in a standard or enhanced NSP cluster, go to Step 130.

Log in as the root or NSP admin user on the NSP cluster host.

Open a console window.

Perform the following steps for each additional MDM node.

1. Enter the following to open an SSH session on the MDM node.

   Note: The root password for a VM created using the Nokia qcow2 image is available from technical support.

   # ssh MDM_node ↵

   where MDM_node is the node IP address

2. Enter the following:

   # mkdir -p /opt/nsp/volumes/mdm-server ↵

3. Enter the following:

   # chown -R 1000:1000 /opt/nsp/volumes ↵

4. Enter the following:

   # exit ↵

Enter the following:

# kubectl get nodes -o wide ↵

A list of nodes like the following is displayed.

NAME    STATUS   ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   

node1   Ready    master   nd   version   int_IP   ext_IP

node2   Ready    master   nd   version   int_IP   ext_IP

node3   Ready    <none>   nd   version   int_IP   ext_IP

Record the NAME value of each node whose INTERNAL-IP value is the IP address of a node that has been added to host an additional MDM instance.

For each node, enter the following sequence of commands:

# kubectl label node node mdm=true ↵

where node is the recorded NAME value of the MDM node

Install Kubernetes secrets

If you are configuring a standalone NSP cluster, or the new primary cluster in a DR deployment, enter the following; otherwise, go to Step 129:

# cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

Enter the following:

Note: To install the Kubernetes secrets, you require the backed-up TLS artifacts extracted in Step 90.

# ./nspdeployerctl secret install ↵

The following prompt is displayed:

Would you like to use your own CA key pair for the NSP Internal Issuer? [yes,no]

Provide the internal CA artifacts from the appliedConfigs location from Step 90.

1. Enter yes ↵.

   The following messages and prompt are displayed:

2. Building secret 'ca-key-pair-internal-nspdeployer'

   The CA key pair used to sign certificates generated by the NSP Internal Issuer.

   Please enter the internal CA private key:

3. Enter the full path of the internal private key (/tmp/appliedConfig/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/tls/ca/ca_internal.key).

   The following prompt is displayed:

   Please enter the internal CA certificate:

4. Enter the full path of the internal certificate (/tmp/appliedConfig/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/tls/ca/ca_internal.pem).

   The following messages are displayed for each Kubernetes namespace:

   Adding secret ca-key-pair-internal-nspdeployer to namespace namespace...

   secret/ca-key-pair-internal-nspdeployer created

The following prompt is displayed:

Would you like to use your own CA key pair for the NSP External Issuer? [yes,no]

Provide your own certificate to secure the external network.

1. Enter yes ↵.

   The following messages and prompt are displayed:

   Building secret 'ca-key-pair-external-nspdeployer'

   The CA key pair used to sign certificates generated by the NSP External Issuer.

   Please enter the external CA private key:

2. Enter the full path of the external private key (/tmp/appliedConfig/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/tls/ca/ca.key).

   The following prompt is displayed:

   Please enter the external CA certificate:

3. Enter the full path of the external certificate (/tmp/appliedConfig/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/tls/ca/ca.pem):

   The following messages are displayed for each Kubernetes namespace:

   Adding secret ca-key-pair-external-nspdeployer to namespace namespace...

   secret/ca-key-pair-external-nspdeployer created

Would you like to provide a custom private key and certificate for use by NSP endpoints when securing TLS connections over the client network? [yes,no]

If the original installation specified custom certificates in the tls section of nsp-config,yml, enter yes ↵ .

1. Enter yes ↵.

   The following messages and prompt are displayed:

   Building secret 'nginx-nb-tls-nsp'

   TLS certificate for securing the ingress gateway.

   Please enter the ingress gateway private key:

2. Enter the full path of the private key file for client access (/tmp/appliedConfig/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/tls/custom/customKey)

   The following prompt is displayed:

   Please enter the ingress gateway public certificate:

3. Enter the full path of the public certificate file for client access (/tmp/appliedConfig/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/tls/custom/customCert)..

   The following prompt is displayed:

   Please enter the ingress gateway public trusted CA certificate bundle:

4. Enter the full path of the public trusted CA certificate bundle file (/tmp/appliedConfig/nsp/NSP-CN-DEP-old-release-ID/NSP-CN-old-release-ID/tls/custom/customCaCert).

   The following message is displayed:

     Adding secret nginx-nb-tls-nsp to namespace namespace...

If the deployment includes MDM and mTLS is enabled in nsp-config.yml, the following prompt is displayed:

Would you like to provide mTLS certificates for the NSP mediation interface for two-way TLS authentication? [yes,no]

Perform one of the following.

1. Enter no ↵ if you are not using mTLS or have no certificate to provide for mTLS.

2. Provide your own certificate to secure MDM and gNMI telemetry.

   1. Enter yes ↵.

   2. The following messages and prompt are displayed:

      Building secret 'mediation-mtls-key'

      mTLS artifacts use to secure MDM communications with nodes.

      Please enter the mediation private key:

   3. Enter the full path of the mediation private key.

      The following prompt is displayed:

      Please enter the mediation CA certificate:

   4. Enter the full path of the mediation CA certificate.

      The following messages are displayed:

        Adding secret mediation-mtls-key to namespace namespace...

      secret/mediation-mtls-key created

        Adding secret mediation-mtls-key to namespace namespace...

      secret/mediation-mtls-key created

Back up the Kubernetes secrets.

1. Enter the following:

   # ./nspdeployerctl secret -o backup_file backup ↵

   where backup_file is the absolute path and name of the backup file to create

   As the secrets are backed up, messages like the following are displayed for each Kubernetes namespace:

   Backing up secrets to /opt/backupfile...

     Including secret namespace:ca-key-pair-external

     Including secret namespace:ca-key-pair-internal

     Including secret namespace:nsp-tls-store-pass

   When the backup is complete, the following prompt is displayed:

   Please provide an encryption password for backup_file

   enter aes-256-ctr encryption password:

2. Enter a password.

   The following prompt is displayed:

   Verifying - enter aes-256-ctr encryption password:

3. Re-enter the password.

   The backup file is encrypted using the password.

4. Record the password for use when restoring the backup.

5. Record the name of the data center associated with the backup.

6. Transfer the backup file to a secure location in a separate facility for safekeeping.

Restore standby Kubernetes secrets

WARNING Deployment failure

The install script will advise you to delete customCaCert, stating that the file is no longer required by NSP. This message is not correct for installations that use custom certificates.

Do not delete this file if you are using custom certificates, as the deployment will fail.

If you are configuring the new standby (former primary) cluster in a DR deployment, obtain the secrets backup file from the NSP cluster in the new primary data center.

1. Enter the following:

   # scp address:path/backup_file /tmp/ ↵

   where

   address is the address of the NSP deployer host in the primary cluster

   path is the absolute file path of the backup file created in Step 128

   backup_file is the secrets backup file name

   The backup file is transferred to the local /tmp directory.

2. Enter the following:

   # cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

3. Enter the following:

   ./nspdeployerctl secret -i /tmp/backup_file restore ↵

   The following prompt is displayed:

   Please provide the encryption password for /opt/backupfile

   enter aes-256-ctr decryption password:

4. Enter the password recorded in Step 128.

   As the secrets are restored, messages like the following are displayed for each Kubernetes namespace:

   Restoring secrets from backup_file...

   secret/ca-key-pair-external created

     Restored secret namespace:ca-key-pair-external

   secret/ca-key-pair-internal created

     Restored secret namespace:ca-key-pair-internal

   secret/nsp-tls-store-pass created

     Restored secret namespace:nsp-tls-store-pass

5. If you answer yes to the Step 126 prompt for client access during the primary NSP cluster configuration, you must update the standby server secret for client access using the custom certificate and key files that are specific to the standby cluster.

   # ./nspdeployerctl secret -s nginx-nb-tls-nsp -n psaRestricted -f tls.key=customKey -f tls.crt=customCert -f ca.crt=customCaCert update ↵

   where

   customKey is the full path of the private server key

   customCert is the full path of the server public certificate

   customCaCert is the full path of the CA public certificate

   Custom certificate and key files are created by performing To generate custom TLS certificate files for the NSP.

   Messages like the following are displayed as the server secret is updated. Do not delete customCaCert if you are using custom certificates, as the deployment will fail.

   secret/nginx-nb-tls-nsp patched

   The following files may contain sensitive information. They are no longer required by NSP and may be removed.

     customKey

     customCert

     customCaCert

Deploy NSP cluster

Enter the following on the NSP deployer host:

# cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

If you are upgrading the new standby (former primary) cluster in a DR deployment, go to Step 134.

If you are creating the new standalone NSP cluster, or the new primary NSP cluster in a DR deployment, you must start the cluster in restore mode; enter the following:

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass install --config --restore

# ./nspdeployerctl install --config --restore ↵

Restore NSP data

If you are creating the new standalone NSP cluster, or the new primary NSP cluster in a DR deployment, you must restore the NSP databases, file service data, and Kubernetes secrets; perform “How do I restore the NSP cluster databases?” in the NSP System Administrator Guide.

Start NSP

If you are creating the new standby cluster in a DR deployment, enter the following on the NSP deployer host:

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass install --config --deploy

# ./nspdeployerctl install --config --deploy ↵

The NSP starts.

Monitor NSP initialization

Open a console window on the NSP cluster host.

Monitor and validate the NSP cluster initialization.

Note: You must not proceed to the next step until each NSP pod is operational.

Note: If you are upgrading a standalone NSP system, or the primary NSP cluster in a DR deployment, the completed NSP cluster initialization marks the end of the network management outage associated with the upgrade.

1. Enter the following every few minutes:

   # kubectl get pods -A ↵

   The status of each NSP cluster pod is displayed; the NSP cluster is operational when the status of each pod is Running or Completed.

2. If the Network Operations Analytics - Baseline Analytics installation option is enabled, ensure that the following pods are listed; otherwise, see the NSP Troubleshooting Guide for information about troubleshooting an errored pod:

   Note: The output for a non-HA deployment is shown below; an HA cluster has three sets of three baseline pods, three rta-ignite pods, and two spark-operator pods.

   • analytics-rtanalytics-tomcat

   • baseline-anomaly-detector-n-exec-1

   • baseline-trainer-n-exec-1

   • baseline-window-evaluator-n-exec-1

   • rta-anomaly-detector-app-driver

   • rta-ignite-0

   • rta-trainer-app-driver

   • rta-windower-app-driver

   • spark-operator-m-n

3. If any pod fails to enter the Running or Completed state, see the NSP Troubleshooting Guide for information about troubleshooting an errored pod.

4. Enter the following to display the status of the NSP cluster members:

   # kubectl get nodes ↵

   The NSP Kubernetes deployer log file is /var/log/nspk8sctl.log.

Enter the following on the NSP cluster host to ensure that all pods are running:

# kubectl get pods -A ↵

The status of each pod is listed; all pods are running when the displayed STATUS value is Running or Completed.

The NSP deployer log file is /var/log/nspdeployerctl.log.

Verify upgraded NSP cluster operation

Use a browser to open the NSP cluster URL.

Verify the following.

• The NSP sign-in page opens.

• The NSP UI opens after you sign in.

• In a DR deployment, if the standby cluster is operational and you specify the standby cluster address, the browser is redirected to the primary cluster address.

Note: If the UI fails to open, perform “How do I remove the stale NSP allowlist entries?” in the NSP System Administrator Guide to ensure that unresolvable host entries from the previous deployment do not prevent NSP access.

Import NFM-P users and groups

If you need to import NFM-P users to the NSP local user database as you transition to OAUTH2 user authentication, perform “How do I import users and groups from NFM-P?” in the NSP System Administrator Guide.

Upgrade MDM adaptors

If the NSP system currently performs model-driven telemetry or classic telemetry statistics collection, you must upgrade your MDM adaptors to the latest in the adaptor suite delivered as part of the new NSP release, and install the required Custom Resources, or CRs.

Perform the following steps.

Note: Upgrading the adaptors to the latest version is mandatory in order for gNMI telemetry collection to function.

1. Upgrade the adaptor suites; see “How do I install adaptor artifacts that are not supported in the Artifacts view?” in the NSP System Administrator Guide for information.

2. When the adaptor suites are upgraded successfully, use NSP Artifacts to install the required telemetry artifact bundles that are packaged with the adaptor suites.

   • nsp-telemetry-cr-nodeType-version.rel.release.ct.zip

   • nsp-telemetry-cr-nodeType-version.rel.release-va.zip

3. View the messages displayed during the installation to verify that the artifact installation is successful.

Restore classic telemetry collection

Telemetry data collection for classically mediated NEs does not automatically resume after an upgrade from NSP Release 23.11 or earlier. Manual action is required to restore the data collection.

If your NSP system collects telemetry data from :classically mediated NEs, restore the telemetry data collection.

1. The upgrade changes IPV6 and Dual Stack NE IDs. Using the updated NE ID from Device Management, edit your subscriptions and update the NE ID in the Object Filter. See “How do I manage subscriptions?” in the NSP Data Collection and Analysis Guide.

2. If you have subscriptions with telemetry type telemetry:/base/interfaces/utilization, update the control files:

   1. Disable the subscriptions with telemetry type telemetry:/base/interfaces/utilization.

   2. Obtain the latest nsp-telemetry-cr-va-sros artifact bundle from the Nokia Support portal and install it; see “How do I install an artifact bundle?” in the NSP Network Automation Guide.

   3. Enable the subscriptions.

3. Add the classically mediated devices to a unified discovery rule that uses GRPC mediation; see “How do I stitch a classic device to a unified discovery rule?” in the NSP Device Management Guide.

   Collection resumes for Classic NEs.

Note: The subscription processing begins after the execution of a discovery rule, and may take 15 minutes or more.

Synchronize LSP paths

After upgrading NSP to Release 24.8 or later from any release earlier than 24.8, path properties for existing LSP paths require updating using the sync API.

The following API call ensures that the tunnel-id and signaling-type properties for LSPs that existed prior to the upgrade to NSP 24.8 or later are updated with the correct device mappings. This step is required for MDM NEs only, and is not required for classically managed NEs.

https://server_IP/restconf/operations/nsp-admin-resync:trigger-resync

Method: POST

Body:

{

  "nsp-admin-resync:input": {

    "plugin-id": "mdm",

    "network-element": [

      {

        "ne-id": {{ne_ipaddress}},

        "sbi-classes": [

          {

            "class-id": "nokia-state:/state/router/mpls/lsp/primary"

          },

          {

            "class-id": "nokia-state:/state/router/mpls/lsp/secondary"

          }

        ]

      }

    ]

  }

}

where ne_ipaddress is the IP address of the NE

Confirm the completion of the resync operation for the affected MDM NEs by performing the following steps:

1. On the nspos-resync-fw-0 pod, enter the following command:

   # kubectl exec -it -n nsp-psa-restricted nspos-resync-fw-0 -- /bin/bash ↵

2. Run a tail on mdResync.log:

   # tail -f /opt/nsp/os/resync-fw/logs/mdResync.log ↵

3. The log contains messages similar to the following after completing sync on an NE:

   done resync neId=3.3.3.3, resync task id=48, isFullResync=false, isScheduleResync=false, device classes=[nokia-state:/state/router/mpls/lsp/primary, nokia-state:/state/router/mpls/lsp/secondary]

4. Verify that the sync is completed for all required MDM NEs.

The following API call updates path control UUIDs and properties in existing IETF TE tunnel primary-paths and secondary-paths. This API syncs values from the path control database to the YANG database. The synchronization process occurs in the background when the API call is executed.

https://server_IP/lspcore/api/v1/syncLspPaths

Method: GET

Sample result:

HTTP Status OK/40X

{    response: {

        data: null,

        status:  40x,

        errors: {

            errorMessage: ""

        },

        startRow: 0,

        endRow: 0,

        totalRows: {}

    }

}

HTTP Status OK

{

    response: {

        data: null,

        status:  200,

        startRow: 0,

        endRow: 0,

        totalRows: {}

    }

}

Perform post-upgrade tasks

Modify your UAC configuration to maintain user access to the Service Configuration Health dashlet in the Network Map and Health view.

At a minimum, to maintain user access to the Service Configuration Health dashlet, you must open all role objects that control access to the Network Map and Health view, make at least one change to each role (even if only to the Description field), and save your change(s). This action is essential to maintain user access to the Service Configuration Health dashlet after an upgrade.

See “How do I configure a role?” in the NSP System Administrator Guide for information on modifying roles.

To remove the sensitive NSP security information from the local file system enter the following:

# rm -rf /tmp/appliedConfigs ↵

The /tmp/appliedConfigs directory is deleted.

If you uninstalled any NSP logical inventory adaptor suites in Step 25 of To prepare for an NSP system upgrade from Release 22.6 or earlier, perform the following steps.

1. Perform “How do I install adaptor artifacts that are not supported in the Artifacts view?” in the NSP System Administrator Guide to re-install the adaptor suites.

2. Enable logical inventory polling policies in the NSP.

Use the NSP to monitor device discovery and to check network management functions.

Back up the NSP databases; perform “How do I back up the NSP cluster databases?” in the NSP System Administrator Guide.

Close the open console windows.

End of steps