Obtain software

Download the following installation files:

• nspos-auxdb-R.r.p-rel.v.rpm

• VerticaSw_PreInstall.sh

• nspos-jre-R.r.p-rel.v.rpm

• vertica-R.r.p-rel.tar

where

R.r.p is the NSP release identifier, in the form MAJOR.minor.patch

v is a version number

For each auxiliary database cluster, transfer the downloaded files to a station that is reachable by each station in the cluster.

Verify auxiliary database synchronization

Before upgrading the auxiliary database, you must verify the success of the most recent copy-cluster operation, which synchronizes the database data between the clusters.

Note: You must not proceed to the next step until the operation is complete and successful.

Issue the following RESTCONF API call periodically to check the copy-cluster status:

Note: In order to issue a RESTCONF API call, you require a token; see the My First NSP API Client tutorial on the Network Developer Portal for information.

GET https://address/restconf/data/auxdb:auxdb-agent

where address is the advertised address of the primary NSP cluster

The call returns a status of SUCCESS, as shown below, for a successfully completed copy-cluster operation:

{

    "auxdb-agent": {

        "name": "nspos-auxdb-agent",

        "application-mode": "ACTIVE",

        "copy-cluster": {

            "source-cluster": "cluster_2",

            "target-cluster": "cluster_1",

            "time-started": "timestamp",

            "status": "SUCCESS"

        }

    }

}

Back up database

CAUTION Data Loss

If you specify a backup location on the database data partition, data loss or corruption may occur.

The auxiliary database backup location must be an absolute path on a partition other than the database data partition.

Before upgrading the first auxiliary database cluster (typically the standby cluster), back up the auxiliary database as described in the NSP System Administrator Guide for the installed release.

Note: The backup location requires 20% more space than the database data consumes.

Note: If the backup location is remote, a 1 Gb/s link to the location is required; if achievable, a higher-capacity link is recommended.

Enable maintenance mode on auxiliary database agent

Before upgrading the auxiliary database, perform the following steps.

1. Log in as the root or NSP admin user on the NSP deployer host in the primary data center and enter the following:

   # export KUBECONFIG=/opt/nsp/nsp-configurator/kubeconfig/nsp_kubeconfig ↵

2. Enter the following:

   # kubectl patch configmap/nspos-auxdb-agent-overrides -n namespace --type=merge -p '{"data":{"nspos-auxdb-agent-overrides.json":"{\"auxDbAgent\":{\"config\":{\"maintenance-mode\":true}}}"}}' ↵

   where namespace is the nspos-auxdb-agent namespace

3. Repeat substep 2 as the root or NSP admin user on the NSP deployer host in the standby data center.

4. On the NSP deployer host in the primary data center, enter the following to restart the nspos-auxdb-agent pod:

   # kubectl delete -n namespace pod `kubectl describe -n namespace pods | grep -P ^^Name: | grep -oP nspos-auxdb-agent[-a-zA-Z0-9]+` ↵

5. Issue the following RESTCONF API call against the primary NSP cluster to verify that the agent is in maintenance mode:

   Note: In order to issue a RESTCONF API call, you require a token; see the My First NSP API Client tutorial on the Network Developer Portal for information.

   GET https://address/restconf/data/auxdb:auxdb-agent

   where address is the advertised address of the primary NSP cluster

   The call returns information like the following:

   {

       "auxdb-agent": {

           "name": "nspos-auxdb-agent",

           "application-mode": "MAINTENANCE",

           "copy-cluster": {

               "source-cluster": "cluster_2",

               "target-cluster": "cluster_1",

               "time-started": "timestamp",

               "status": "SUCCESS"

           }

       }

   }

   The agent is in maintenance mode if the application-mode is MAINTENANCE, as shown in the example.

Stop database proxies

Perform the following to stop the auxiliary database proxy services.

1. Log in as the root user on each auxiliary database station in the cluster.

2. Enter the following:

   # systemctl stop nspos-auxdbproxy.service ↵

Stop auxiliary database

Log in as the root user on a station in the auxiliary database cluster that you are upgrading.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following to block external access to the auxiliary database ports:

# ./auxdbAdmin.sh shieldsUp ↵

Enter the following to stop the auxiliary database:

# ./auxdbAdmin.sh stop ↵

The following prompt is displayed:

Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:

Enter the dba password.

Information like the following is displayed:

Database samdb stopped successfully

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1     | internal_IP_1 | STATE | version | db_name

 node_2     | internal_IP_2 | STATE | version | db_name

...

 node_n     | internal_IP_n | STATE | version | db_name

  Output captured in log_file

Note: If the inital stop does not succeed, force_stop is attempted automatically until the auxiliary database is stopped. If the database does not stop, do not proceed to the next step and contact Nokia support for further assistance.

Prepare each station for upgrade

Perform the following steps on each existing auxiliary database station in the cluster.

Log in as the root user on the station.

Open a console window.

Transfer the downloaded installation files to an empty directory on the station.

Note: You must ensure that the directory is empty.

Note: In subsequent steps, the directory is called the NSP software directory.

Navigate to the NSP software directory.

Note: The directory must contain only the installation files.

Enter the following:

# chmod +x * ↵

Enter the following:

# ./VerticaSw_PreInstall.sh ↵

Information like the following is displayed:

Logging Vertica pre install checks to log_file

INFO: About to remove proxy parameters set by a previous run of 

            this script from    /etc/profile.d/proxy.sh

INFO: Completed removing proxy parameters set by a previous run of 

            this script from    /etc/profile.d/proxy.sh

INFO: About to set proxy parameters in /etc/profile.d/proxy.sh...

INFO: Completed setting proxy parameters in /etc/profile.d/proxy.sh...

INFO: Found pre-existing [/opt/nsp/nfmp/auxdb/install/config/install.config]

      file on this machine. This file will be validated.

INFO: About to remove kernel parameters set by a previous run of 

      this script from /etc/sysctl.conf

INFO: Completed removing kernel parameters set by a previous run of 

      this script from /etc/sysctl.conf

INFO: About to set kernel parameters in /etc/sysctl.conf...

INFO: Completed setting kernel parameters in /etc/sysctl.conf...

INFO: About to change the current values of the kernel parameters

INFO: Completed changing the current values of the kernel parameters

INFO: About to remove ulimit parameters set by a previous run of 

      this script from  /etc/security/limits.conf

INFO: Completed removing ulimit parameters set by a previous run of 

      this script from  /etc/security/limits.conf

INFO: About to set ulimit parameters in /etc/security/limits.conf...

INFO: Completed setting ulimit parameters in /etc/security/limits.conf...

INFO: Removing /var/log/wtmp entry from /etc/logrotate.conf

INFO: Adding /etc/logrotate.d/wtmp

Checking user group nsp... 

WARNING: user group with the specified name already exists locally.

Checking user nsp... 

INFO: user with the specified name already exists locally. Setting shell to /usr/sbin/nologin.

Checking Linux user group samauxdb... 

WARNING: user group with the specified name already exists locally.

Checking user samauxdb... 

WARNING: user with the specified name already exists locally.

Moving logfile from /tmp ...

     ... to /opt/nsp/nfmp/auxdb/install/log

Changing ownership of the directory /opt/nsp/nfmp/auxdb/install to samauxdb:samauxdb.

Removing group write and world permissions from the directory /opt/nsp/nfmp/auxdb/install.

Changing ownership of /opt/nsp/nfmp/auxdb files.

Set virtio device queue scheduler for /dev/vda: [none] mq-deadline kyber bfq 

INFO: Updating auxiliary database prep script.

Stopping and disabling systemd service [nspos-auxdbproxy.service]

Stopping and disabling systemd service [nspos-auxdb.service]

************************************************************************ Changes were made that require a restart.                            * Please restart this host before proceeding with Vertica installation.***********************************************************************

Enter the following to reboot the station:

# systemctl reboot ↵

The station reboots.

When the reboot is complete, log in as the root user on the station.

Open a console window.

Navigate to the NSP software directory.

Enter the following:

# dnf install nspos-*.rpm ↵

The dnf utility resolves any package dependencies, and displays the following prompt for each package:

Total size: nn G

Is this ok [y/N]: 

Enter y. The following and the installation status are displayed as each package is installed:

Downloading Packages:

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

The package installation is complete when the following is displayed:

Complete!

Upgrade database

Log in as the root user on a station in the auxiliary database cluster that you are upgrading.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following to start the upgrade:

# ./auxdbAdmin.sh upgrade tar_file ↵

where tar_file is the absolute path and filename of the vertica-R.r.p-rel.tar file in the NSP software directory.

The following prompt is displayed:

Updating Vertica - Please perform a backup before proceeding with this option

Do you want to proceed (YES/NO)?

Enter YES ↵.

The following prompt is displayed:

Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:

Enter the dba password.

The following prompt is displayed:

Please confirm auxiliary database dba password:

Enter the dba password again.

The upgrade begins, and operational messages are displayed.

The upgrade is complete when the following is displayed:

# Post-upgrade tasks completed.

Enter the following to display the auxiliary database status:

# ./auxdbAdmin.sh status ↵

Information like the following is displayed:

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1     | internal_IP_1 | STATE | version | db_name

 node_2     | internal_IP_2 | STATE | version | db_name

...

 node_n     | internal_IP_n | STATE | version | db_name

   ################################################################################

   WARNING: Shields are up on one or more auxiliary database hosts.

            External database access will be blocked until the shields are dropped.

   Shields up on host internal_IP_1

   Shields up on host internal_IP_2

...

   Shields up on host internal_IP_n

   ################################################################################

  Output captured in log_file

The cluster is running when each STATE entry reads UP.

Note: You must not proceed to the next step until the cluster is running.

Back up database for migration to new OS version

Perform the following steps.

1. Open the /opt/nsp/nfmp/auxdb/install/config/install.config file using a plain-text editor such as vi.

2. Replace the lines in the TLS settings section with the following:

   # TLS settings.

   # A PKI Server IP address or hostname must be specified in order to generate certificates.

   pki_server=address

   pki_server_port=80

   where address is one of the following in the platform—ingressApplications—ingressController section of the nsp-config.yml file on the local NSP deployer host:

   In the internalAddresses subsection, if configured, otherwise, in the clientAddresses subsection:

   • if configured, the advertised value

   • otherwise, the virtualIp value

   If an external pki-server is used for generation of NSP certificate artifacts, use the address and port of the external pki-server instead.

3. Save and close the install.config file.

4. Copy the install.config file to a secure location on a separate station that is unaffected by the upgrade activity.

Enter the following to regenerate the TLS certificates:

# ./auxdbAdmin.sh configureTLS ↵

You must back up the auxiliary database data and configuration information to prepare for the migration to the new OS version.

Note: The operation may take considerable time.

Perform the following steps.

1. Enter the following:

   # ./auxdbAdmin.sh backup /path/samAuxDbBackup_restore.conf ↵

   where path is the backup file path specified in Step 4

   The following prompt is displayed:

   Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:

2. Enter the dba password.

   The backup operation begins, and messages like the following are displayed.

   Copying backup config file to /tmp/auxdbadmin-backup-ID

   Backup snapshot name - AuxDbBackUpID_auxdbAdmin

   Starting auxiliary database backup...

   The backup is complete when the following is displayed:

   Output captured in 

   /opt/nsp/nfmp/auxdb/install/log/auxdbAdmin.sh.timestamp.log

3. Transfer the database backup file to a secure location on a separate station that is unaffected by the upgrade activity.

Perform the following steps on each auxiliary database station to preserve the upgrade logs.

1. Enter the following:

   # tar czf /opt/nsp/auxdb-node_N-upgrade-log.tar.gz /opt/vertica/log /opt/nsp/nfmp/auxdb/install/log /opt/nsp/nfmp/auxdb/install/proxy/log /opt/nsp/nfmp/auxdb/catalog/samdb/v_samdb_node*/vertica.log ↵

   where N is the node number, as shown in the Step 32 output example

   An auxdb-node_N-upgrade-log.tar.gz file is created in the current working directory.

2. Transfer the file to a secure location on a separate station that is unaffected by the upgrade activity.

Recommission stations, if required

If you are reusing any auxiliary database stations, recommission each station according to the platform specifications in this guide and in the NSP Planning Guide.

For information about deploying the RHEL 9 OS using an NSP OEM disk image, see NSP disk-image deployment.

Note: You must reformat the following disk partitions:

• root

• /opt/nsp/nfmp/auxdb/data

Install new software

Perform To prepare a station for NSP auxiliary database installation on each auxiliary database station.

Copy each auxdb-node_N-upgrade-log.tar.gz created in Step 36 to the /opt/nsp directory on the associated auxiliary database station.

Log in as the root user on an auxiliary database station.

Copy the install.config file saved in Step 33, substep 3 to the following directory:

/opt/nsp/nfmp/auxdb/install/config

Log in to any auxiliary database station as the root user.

Note: The software is installed on one station, and then automatically propagated to the other stations in the cluster.

Open a console window.

Perform Step 10 to Step 14 from To install the NSP auxiliary database.

Prepare to restore auxiliary database

Stop the auxiliary database.

1. Log in as the root user on any station in the auxiliary database cluster.

2. Enter the following:

   # /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh stop ↵

   You are prompted to enter the database user password.

3. Enter the password. The auxiliary database stops.

If the backup files to restore are not in the original backup location on each auxiliary database station, perform the following steps.

1. If you know the original backup location, go to substep 6.

2. Open the following file for viewing:

   path/AuxDbBackUp/samAuxDbBackup_restore.conf

   where path is the current location of the backup file set

3. Locate the [Mapping] section, which contains one line like the following for each auxiliary server station:

   v_name_node0001 = IP_address:path/AuxDbBackUp

   The path is the original backup location.

4. Record the original backup location.

5. Close the samAuxDbBackup_restore.conf file.

6. Copy the AuxDbBackUp directory contents from the current backup location to the AuxDbBackUp directory in the original backup location on each auxiliary database station.

7. As the root user, enter the following command on each auxiliary database station:

   # chown -R samauxdb path ↵

   where path is the absolute path of the original backup location

If the backup is being restored on any stations that have different internal IP addresses, perform the following steps on each auxiliary database station.

1. Open the following file using a plain-text editor such as vi:

   path/AuxDbBackUp/samAuxDbBackup_restore.conf

   where path is the location of the backup file set

2. Add the following internal mapping section to the end of the file; the example below is for an auxiliary database of three stations:

   [NodeMapping]

   v_name_node0001 = IP_address_1

   v_name_node0002 = IP_address_2

   v_name_node0003 = IP_address_3

   where

   v_name_node000n is the station name shown in the [Mapping] section of the file

   IP_address_n is the new internal IP address of the station

3. Save and close the samAuxDbBackup_restore.conf file.

Restore auxiliary database

Enter the following on one auxiliary database station:

# /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh restore path/AuxDbBackUp/samAuxDbBackup_restore.conf ↵

where path is the absolute path of the database backup created in Step 33

The restore operation begins. The following messages and progress indicators are displayed:

Starting full restore of database db_name.

Participating nodes: node_1, node_2, ... node_n.

Restoring from restore point: AuxDbBackUpID_datestamp_timestamp

Determining what data to restore from backup.

[==============================================] 100%

Approximate bytes to copy: nnnnnnnn of nnnnnnnnn total.

Syncing data from backup to cluster nodes.

When the restore is complete, the second progress indicator reaches 100%, and the following message is displayed:

[==============================================] 100%

Restoring catalog. Restore complete!

Restore database user passwords

If a database user password has changed since the creation of the database backup, the NSP components cannot authenticate with the restored auxiliary database.

In such a scenario, you must perform the following steps to update each affected database user password to the value that the NSP components recognize.

1. Enter the following on the current auxiliary database station to connect to the auxiliary database:

   # vsql ↵

   You are prompted for credentials.

2. Enter the following:

   • user—samauxdb

   • password—database administrator password recorded during creation of backup

3. Enter the following once for each user whose password has changed since the backup creation:

   ALTER USER user IDENTIFIED BY 'password';

   where

   user is the database user name

   password is the newer password recognized by the NSP

4. Verify the success of the password change by:

   • attempting to connect to the auxiliary database as each affected user

   • ensuring that the NSP components are able to connect to the auxiliary database

Enable required database services and proxies

Enter the following to enable the database services:

# ./auxdbAdmin.sh startAndEnableServices ↵

Activate upgraded auxiliary database cluster

If the first auxiliary database cluster has been upgraded (typically the standby cluster when the upgrade began), the cluster may now be activated.

Note: This step is only performed once during a geo-redundant auxiliary database upgrade, after the first cluster has been upgraded and before the second cluster is upgraded.

Issue the following RESTCONF API call to activate the upgraded auxiliary database cluster, after which the cluster assumes the primary role:

Note: In order to issue a RESTCONF API call, you require a token; see the My First NSP API Client tutorial on the Network Developer Portal for information.

POST https://address/restconf/data/auxdb:clusters/cluster=cluster_N/activate

where

address is the advertised address of the primary NSP cluster

N is the auxiliary database cluster number

The following is the request body:

{

  "auxdb:input" : {

    "force": true

  }

}

Disable maintenance mode for auxiliary database agents

If both geo-redundant auxiliary database clusters have been upgraded maintenance mode may be disabled for the auxiliary database agent. Perform the following steps.

1. Log in as the root or NSP admin user on the NSP deployer host in the primary data center and enter the following:

   # export KUBECONFIG=/opt/nsp/nsp-configurator/kubeconfig/nsp_kubeconfig ↵

2. Enter the following:

   # kubectl patch configmap/nspos-auxdb-agent-overrides -n namespace --type=merge -p '{"data":{"nspos-auxdb-agent-overrides.json":"{\"auxDbAgent\":{\"config\":{\"maintenance-mode\":false}}}"}}' ↵

   where namespace is the nspos-auxdb-agent namespace

3. Repeat substep 2 as the root or NSP admin user on the NSP deployer host in the standby data center.

4. On the NSP deployer host in the primary data center, enter the following to restart the nspos-auxdb-agent pod:

   # kubectl delete -n namespace pod `kubectl describe -n namespace pods | grep -P ^^Name: | grep -oP nspos-auxdb-agent[-a-zA-Z0-9]+` ↵

Close the open console windows.

End of steps