Obtain software

Download the following installation files:

• nspos-auxdb-R.r.p-rel.v.rpm

• VerticaSw_PreInstall.sh

• nspos-jre-R.r.p-rel.v.rpm

• vertica-R.r.p-rel.tar

where

R.r.p is the NSP release identifier, in the form MAJOR.minor.patch

v is a version number

For each auxiliary database cluster, transfer the downloaded files to a station that is reachable by each station in the cluster.

Verify auxiliary database synchronization

If you are upgrading the standby auxiliary database cluster, you must verify the success of the most recent copy-cluster operation, which synchronizes the database data between the clusters.

Note: You must not proceed to the next step until the operation is complete and successful.

Issue the following RESTCONF API call periodically to check the copy-cluster status:

Note: In order to issue a RESTCONF API call, you require a token; see this tutorial on the Network Developer Portal for information.

GET https://address/restconf/data/auxdb:/auxdb-agent

where address is the advertised address of the primary NSP cluster

The call returns a status of SUCCESS, as shown below, for a successfully completed copy-cluster operation:

<HashMap>

      <auxdb-agent>

         <name>nspos-auxdb-agent</name>

         <application-mode>ACTIVE</application-mode>

         <copy-cluster>

            <source-cluster>cluster_M</source-cluster>

            <target-cluster>cluster_N</target-cluster>

            <time-started>timestamp</time-started>

            <status>SUCCESS</status>

         </copy-cluster>

      </auxdb-agent>

</HashMap>

Back up database

CAUTION Data Loss

If you specify a backup location on the database data partition, data loss or corruption may occur.

The auxiliary database backup location must be an absolute path on a partition other than the database data partition.

If you are upgrading the standby cluster, back up the auxiliary database as described in the NSP System Administrator Guide for the installed release.

Note: The backup location requires 20% more space than the database data consumes.

Note: If the backup location is remote, a 1 Gb/s link to the location is required; if achievable, a higher-capacity link is recommended.

Stop and disable standby database proxy

If you are upgrading the standby auxiliary database cluster, perform one of the following to stop the auxiliary database proxy:

1. If you are upgrading from Release 23.11 or later, enter the following:

   # systemctl disable nspos-auxdbproxy.service ↵

2. If you are upgrading from Release 23.8 or earlier, enter the following:

   # systemctl disable nfmp-auxdbproxy.service ↵

Enable auxiliary database maintenance mode

If you are upgrading the standby auxiliary database cluster, perform the following steps.

1. Log in as the root or NSP admin user on the NSP cluster host in the primary data center.

2. Enter the following:

   # kubectl patch configmap/nspos-auxdb-agent-overrides -n namespace --type=merge -p '{"data":{"nspos-auxdb-agent-overrides.json":"{\"auxDbAgent\":{\"config\":{\"maintenance-mode\":true}}}"}}' ↵

   where namespace is the nspos-auxdb-agent namespace

3. Enter the following to restart the nspos-auxdb-agent pod:

   # kubectl delete -n namespace pod `kubectl describe -n namespace pods | grep -P ^^Name: | grep -oP nspos-auxdb-agent[-a-zA-Z0-9]+` ↵

4. Issue the following RESTCONF API call to verify that the agent is in maintenance mode:

   NOTE: In order to issue a RESTCONF API call, you require a token; see this tutorial on the Network Developer Portal for information.

   GET https://address/restconf/data/auxdb:/auxdb-agent

   where address is the advertised address of the primary NSP cluster

   The call returns information like the following:

   {

       "auxdb-agent": {

           "name": "nspos-auxdb-agent",

           "application-mode": "MAINTENANCE",

           "copy-cluster": {

               "source-cluster": "cluster_2",

               "target-cluster": "cluster_1",

               "time-started": "timestamp",

               "status": "SUCCESS"

           }

       }

   }

   The agent is in maintenance mode if the application-mode is MAINTENANCE, as shown in the example.

Stop standby auxiliary database services

If you are upgrading the standby auxiliary database cluster, perform one of the following to disable the database services:

1. If you are upgrading from Release 23.11 or later, enter the following sequence of commands

   systemctl disable nspos-auxdb.service

   systemctl disable vertica_agent.service

   systemctl disable verticad.service

2. If you are upgrading from Release 23.8 or earlier, enter the following sequence of commands:

   systemctl disable nfmp-auxdb.service

   systemctl disable vertica_agent.service

   systemctl disable verticad.service

Stop database

Log in as the root user on a station in the auxiliary database cluster that you are upgrading.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following to stop the auxiliary database:

# ./auxdbAdmin.sh stop ↵

Enter the following to display the auxiliary database status:

# ./auxdbAdmin.sh status ↵

Information like the following is displayed:

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1 | internal_IP_1 | STATE | version | db_name

 node_2 | internal_IP_2 | STATE | version | db_name

.

.

.

 node_n | internal_IP_n | STATE | version | db_name

      Output captured in log_file

The cluster is stopped when each STATE entry reads DOWN.

Repeat Step 12 periodically until the cluster is stopped.

Note: You must not proceed to the next step until the cluster is stopped.

Prepare all stations for upgrade

Perform Step 16 to Step 33 on each station in the cluster that you are upgrading.

Go to Step 34.

Prepare individual station for upgrade

If the auxiliary database station is deployed in a VM created using an NSP RHEL OS disk image, perform To apply a RHEL update to an NSP image-based OS.

Log in as the root user on the station.

Open a console window.

Perform one of the following to stop the auxiliary database services:

1. If you are upgrading from Release 23.11 or later, enter the following sequence of commands:

   systemctl stop nspos-auxdb.service

   systemctl stop vertica_agent.service

   systemctl stop verticad.service

2. If you are upgrading from Release 23.8 or earlier, enter the following sequence of commands:

   systemctl stop nfmp-auxdb.service

   systemctl stop vertica_agent.service

   systemctl stop verticad.service

Perform one of the following to disable the database services:

1. If you are upgrading from Release 23.11 or later, enter the following sequence of commands

   systemctl disable nspos-auxdb.service

   systemctl disable nspos-auxdbproxy.service

   systemctl disable vertica_agent.service

   systemctl disable verticad.service

2. If you are upgrading from Release 23.8 or earlier, enter the following sequence of commands:

   systemctl disable nfmp-auxdb.service

   systemctl disable nfmp-auxdbproxy.service

   systemctl disable vertica_agent.service

   systemctl disable verticad.service

Transfer the downloaded installation files to an empty directory on the station.

Note: You must ensure that the directory is empty.

Note: In subsequent steps, the directory is called the NSP software directory.

Navigate to the NSP software directory.

Note: The directory must contain only the installation files.

Enter the following:

# chmod +x * ↵

Enter the following:

# ./VerticaSw_PreInstall.sh ↵

Information like the following is displayed:

Logging Vertica pre install checks to log_file

INFO: About to remove proxy parameters set by a previous run of this script from /etc/profile.d/proxy.sh

INFO: Completed removing proxy parameters set by a previous run of this script from /etc/profile.d/proxy.sh

INFO: About to set proxy parameters in /etc/profile.d/proxy.sh...

INFO: Completed setting proxy parameters in /etc/profile.d/proxy.sh...

INFO: About to remove kernel parameters set by a previous run of this script from /etc/sysctl.conf

INFO: Completed removing kernel parameters set by a previous run of this script from /etc/sysctl.conf

INFO: About to set kernel parameters in /etc/sysctl.conf...

INFO: Completed setting kernel parameters in /etc/sysctl.conf...

INFO: About to change the current values of the kernel parameters

INFO: Completed changing the current values of the kernel parameters

INFO: About to remove ulimit parameters set by a previous run of this script from /etc/security/limits.conf

INFO: Completed removing ulimit parameters set by a previous run of this script from /etc/security/limits.conf

INFO: About to set ulimit parameters in /etc/security/limits.conf...

INFO: Completed setting ulimit parameters in /etc/security/limits.conf...

Checking Vertica DBA group samauxdb...

WARNING: Vertica DBA group with the specified name already exists locally.

Checking Vertica user samauxdb...

WARNING: Vertica user with the specified name already exists locally.

Changing ownership of the directory /opt/nsp/nfmp/auxdb/install to samauxdb:samauxdb.

Adding samauxdb to sudoers file.

Changing ownership of /opt/nsp/nfmp/auxdb files.

INFO: About to remove commands set by a previous run of this script from /etc/rc.d/rc.local

INFO: Completed removing commands set by a previous run of this script from /etc/rc.d/rc.local

INFO: About to add setting to /etc/rc.d/rc.local...

INFO: Completed adding setting to /etc/rc.d/rc.local...

Enter the following to reboot the station:

# systemctl reboot ↵

The station reboots.

When the reboot is complete, log in as the root user on the station.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following to display the auxiliary database status:

# ./auxdbAdmin.sh status ↵

Information like the following is displayed:

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1 | internal_IP_1 | STATE | version | db_name

 node_2 | internal_IP_2 | STATE | version | db_name

.

.

.

 node_n | internal_IP_n | STATE | version | db_name

      Output captured in log_file

if any STATE entry is not DOWN, perform the following steps.

1. Enter the following to stop the auxiliary database:

   # ./auxdbAdmin.sh stop ↵

2. Repeat Step 29 periodically until each STATE entry reads DOWN.

   Note: You must not proceed to the next step until each STATE entry reads DOWN.

Navigate to the NSP software directory.

Enter the following:

# dnf install nspos-*.rpm ↵

The dnf utility resolves any package dependencies, and displays the following prompt for each package:

Total size: nn G

Installed size: nn G 

Is this ok [y/d/N]: 

Enter y. The following and the installation status are displayed as each package is installed:

Downloading Packages:

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

The package installation is complete when the following is displayed:

Complete!

Upgrade database

Log in as the root user on a station in the auxiliary database cluster that you are upgrading.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Block external access to the auxiliary database ports:

1. Enter the following:

   # su - samauxdb ↵

2. Enter the following:

   bash$ ssh-keygen -t rsa ↵

3. Enter the following for each other station in the auxiliary database cluster:

   bash$ ssh-copy-id internal_IP ↵

   where internal_IP is the internal IP address of another station in the cluster

4. Enter the following:

   bash$ exit ↵

5. Enter the following:

   # ./auxdbAdmin.sh shieldsUp ↵

Start the upgrade.

1. Enter the following:

   # ./auxdbAdmin.sh upgrade tar_file --ask-pass ↵

   where tar_file is the absolute path and filename of the vertica-R.r.p-rel.tar file in the NSP software directory

   The following prompt is displayed:

   Please enter the current password for samauxdb:

2. Enter the samauxdb password.

   The following prompt is displayed:

   Please re-enter the current password for samauxdb:

3. Enter the samauxdb password again.

   The following prompt is displayed:

   Please enter the current password for samuser:

4. Enter the samuser password.

   The following prompt is displayed:

   Please re-enter the current password for samuser:

5. Enter the samuser password again.

   The following messages and prompt are displayed:

   Populating auxiliary database user passwords in the vault

   Updating Vertica - Please perform a backup before proceeding with this option

   Do you want to proceed (YES/NO)?

Enter YES ↵.

The following prompt is displayed:

Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:

Enter the dba password.

The following prompt is displayed:

Please verify auxiliary database dba password:

Enter the dba password again.

The upgrade begins, and operational messages are displayed.

The upgrade is complete when the following is displayed:

Database database_name started successfully

  Output captured in log_file

Enter the following to display the auxiliary database status:

# ./auxdbAdmin.sh status ↵

Information like the following is displayed:

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1 | internal_IP_1 | STATE | version | db_name

 node_2 | internal_IP_2 | STATE | version | db_name

.

.

.

 node_n | internal_IP_n | STATE | version | db_name

      Output captured in log_file

The cluster is running when each STATE entry reads UP.

Enter the following to regenerate the TLS certificates:

# ./auxdbAdmin.sh configureTLS force-gen ↵

Repeat Step 42 periodically until the cluster is running.

Note: You must not proceed to the next step until the cluster is running.

Enter the following to regenerate the TLS certificates:

# ./auxdbAdmin.sh configureTLS force-gen ↵

Enter the following:

# ./auxdbAdmin.sh shieldsDown ↵

Update database schema

If the NSP deployment includes the NFM-P, update the NFM-P database schema.

Note: The schema update may take considerable time.

1. Log in as the nsp user on the standalone or primary NFM-P main server.

2. Open a console window.

3. Enter the following:

   bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

4. Enter the following:

   bash$ ./nmsserver.bash upgradeAuxDbSchema ↵

   The following prompt is displayed:

   Auxiliary database clusters:

   1: IP_a,IP_b,IP_c

   2: IP_x,IP_y,IP_z

   Select auxiliary database to upgrade:

5. Enter the number that corresponds to the cluster you are upgrading.

   The following messages and prompt are displayed:

   WARNING: About to upgrade samdb schema on the auxiliary database cluster [IP_a,IP_b,IP_c].

   It is recommended that a database backup is performed before proceeding.

   Type "YES" to continue

6. Enter YES.

   The following prompt is displayed:

   Please enter the auxiliary database port [5433]:

7. Enter the auxiliary database port number; press Enter to accept the default of 5433.

   The following prompt is displayed:

   Please enter the auxiliary database user password:

8. Enter the required password.

   The following messages are displayed as the upgrade begins:

   INFO: Database upgrade can take a very long time on large databases.

   INFO: logs are stored under /opt/nsp/nfmp/server/nms/log/auxdb. Check the logs for progress.

   INFO: Node Name[v_samdb_node0001]->IP[IP_address]->Status[UP]

   INFO: About to perform upgrade

Configure TLS

Open the /opt/nsp/nfmp/auxdb/install/config/install.config file using a plain-text editor such as vi.

Edit the following lines in the file to read as shown below:

secure=true

pki_server=address

pki_server_port=80

where address is one of the following in the platform—ingressApplications—ingressController section of the nsp-config.yml file on the local NSP deployer host:

In the internalAddresses subsection, if configured, otherwise, in the clientAddresses subsection:

• if configured, the advertised value

• otherwise, the virtualIp value

Save and close the install.config file.

Enable database services

Perform the following steps on each station in the auxiliary database cluster.

1. Log in as the root user.

2. Open a console window.

3. Enter the following sequence of commands to enable the database services:

   systemctl enable nspos-auxdb.service

   systemctl enable nspos-auxdbproxy.service

   systemctl enable vertica_agent.service

   systemctl enable verticad.service

If you are upgrading the primary auxiliary database cluster, enter the following on each station in the cluster:

# systemctl start nspos-auxdbproxy.service ↵

The auxiliary database proxy starts.

Disable auxiliary database maintenance mode

Perform the following steps.

1. Log in as the root or NSP admin user on the NSP cluster host.

2. Enter the following:

   # kubectl patch configmap/nspos-auxdb-agent-overrides -n namespace --type=merge -p '{"data":{"nspos-auxdb-agent-overrides.json":"{\"auxDbAgent\":{\"config\":{\"maintenance-mode\":false}}}"}}' ↵

   where namespace is the nspos-auxdb-agent namespace

3. Enter the following to restart the nspos-auxdb-agent pod:

   # kubectl delete -n namespace pod `kubectl describe -n namespace pods | grep -P ^^Name: | grep -oP nspos-auxdb-agent[-a-zA-Z0-9]+` ↵

Verify auxiliary database status

Issue the following RESTCONF API call to verify that the primary auxiliary database cluster is in active mode:

Note: In order to issue a RESTCONF API call, you require a token; see this tutorial on the Network Developer Portal for information.

GET /data/auxdb:/auxdb-agent HTTP/1.1

Request body:

    Host: address

    Content-Type: application/json

    Authorization: bearer_and_token_from_session_manager

where address is the advertised address of the primary NSP cluster

The cluster is in active mode if the response includes ACTIVE.

Issue the following RESTCONF API call to verify the auxiliary database operation:

Note: In order to issue a RESTCONF API call, you require a token; see this tutorial on the Network Developer Portal for information.

GET https://address/restconf/data/auxdb:/clusters

where address is the advertised address of the primary NSP cluster

The call returns auxiliary database cluster status information like the following; if each mode and status value are not as shown below, contact technical support.

<HashMap>

    <clusters>

        <cluster>

            <name>cluster_M</name>

            <mode>ACTIVE</mode>

            <status>UP</status>

            <nodes>

                <external-ip>203.0.113.101</external-ip>

                <internal-ip>10.1.2.101</internal-ip>

                <status>UP</status>

            </nodes>

            <nodes>

                <external-ip>203.0.113.102</external-ip>

                <internal-ip>10.1.2.102</internal-ip>

                <status>UP</status>

            </nodes>

            <nodes>

                <external-ip>203.0.113.103</external-ip>

                <internal-ip>10.1.2.103</internal-ip>

                <status>UP</status>

            </nodes>

        </cluster>

        <cluster>

            <name>cluster_N</name>

            <mode>STANDBY</mode>

            <status>ON_STANDBY</status>

            <nodes>

                <external-ip>203.0.113.104</external-ip>

                <internal-ip>10.1.2.104</internal-ip>

                <status>READY</status>

            </nodes>

            <nodes>

                <external-ip>203.0.113.105</external-ip>

                <internal-ip>10.1.2.105</internal-ip>

                <status>READY</status>

            </nodes>

            <nodes>

                <external-ip>203.0.113.106</external-ip>

                <internal-ip>10.1.2.106</internal-ip>

                <status>READY</status>

            </nodes>

        </cluster>

    </clusters>

</HashMap>

Close the open console windows.

End of steps