CAUTION

Service Disruption

Install software

Add a hostname entry for the new station to the /etc/hosts file on each existing auxiliary database station.

Note: The hostname must be the fully qualified hostname, and not the short hostname.

Note: Hostnames are case-sensitive.

Log in as the root user on the new auxiliary database station.

Add a hostname entry for the new station to the /etc/hosts file on the new station using the following criteria.

• The first entry for the station hostname in the file must be the station IP address that is reachable by each main server and NSP cluster.

• The hostname must be the fully qualified hostname, and not the short hostname.

• The hostname must:

  • contain only ASCII alphanumeric and hyphen characters.

  • not begin or end with a hyphen.

  • not begin with a number.

  • comply with the format defined in IETF RFC 1034.

  • use period characters delimit the FQDN components.

  • not exceed 63 characters.

Note: Hostnames are case-sensitive.

Perform “To apply the RHEL 8 swappiness workaround” in the NSP Installation and Upgrade Guide on the station.

Download the following installation files to an empty local directory:

Note: You must ensure that the directory is empty.

Note: The software release must match the software release of the existing auxiliary database.

• nspos-auxdb-R.r.p-rel.v.rpm

• VerticaSw_PreInstall.sh

• nspos-jre-R.r.p-rel.v.rpm

• vertica-R.r.p-rel.tar

where

R.r.p is the NSP release identifier, in the form MAJOR.minor.patch

v is a version number

Note: In subsequent steps, the directory is called the software directory.

Open a console window.

Navigate to the software directory.

Note: The directory must contain only the installation files.

Enter the following:

# chmod +x * ↵

Enter the following:

# ./VerticaSw_PreInstall.sh ↵

The script displays configuration messages like the following, and a prompt:

INFO: About to set kernel parameters in /etc/sysctl.conf...

INFO: Completed setting kernel parameters in /etc/sysctl.conf...

INFO: About to change the current values of the kernel parameters INFO:

Completed changing the current values of the kernel parameters

INFO: About to set ulimit parameters in /etc/security/limits.conf...

INFO: Completed setting ulimit parameters in /etc/security/limits.conf...

Checking Vertica DBA group samauxdb... 

Adding Vertica DBA group samauxdb... 

Checking Vertica user samauxdb... 

Adding samauxdb... 

Set password for samauxdb...

New password:

Enter a password that conforms to the RHEL password criteria.

The following prompt is displayed:

Retype new password:

Re-enter the password.

Messages like the following are displayed:

Changing password for user samauxdb.

passwd: all authentication tokens updated successfully.

Changing ownership of the directory /opt/nsp/nfmp/auxdb to samauxdb:samauxdb.

Adding samauxdb to sudoers file.

Changing ownership of /opt/nsp/nfmp/auxdb files.

INFO: About to add setting to /etc/rc.d/rc.local...

INFO: Completed adding setting to /etc/rc.d/rc.local...

If the script instructs you to perform a restart, perform the following steps.

1. Enter the following:

   # systemctl reboot ↵

   The station restarts.

2. Log in to the station as the root user.

3. Open a console window.

4. Navigate to the software directory.

Enter the following:

# tar xvf vertica-R.r.p-rel.tar $(tar tf vertica-R.r.p-rel.tar | sort -V | tail -1) ↵

Enter the following:

# dnf install *.rpm ↵

The dnf utility resolves any package dependencies, and displays the following prompt:

Total size: nn G

Installed size: nn G 

Is this ok [y/d/N]: 

Enter y. The following and the installation status are displayed as each package is installed:

Downloading Packages:

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

The package installation is complete when the following is displayed:

Complete!

Add new station to auxiliary database configuration

When the package installation is complete, perform the following steps on each auxiliary database station to stop the database proxy.

Note: If the auxiliary database is geo-redundant, you must stop the database proxy on each station in each auxiliary database cluster.

1. Log in to the station as the root user.

2. Open a console window.

3. Enter the following:

   # systemctl stop nspos-auxdbproxy.service ↵

4. Verify that the proxy is stopped; enter the following:

   # systemctl status nspos-auxdbproxy ↵

Log in to an existing auxiliary database station as the root user.

Note: If the auxiliary database is geo-redundant, the station must be in the primary auxiliary database cluster.

Open a console window.

Enter the following:

# /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh addNode internal_IP external_IP ↵

where

internal_IP is the IP address from which the station communicates with the other auxiliary database stations

external_IP is the IP address from which the station communicates with other components in the deployment

The script displays the following:

   Adding hosts(s) hostname to auxiliary database cluster ...

You are prompted to enter the auxiliary database dba password.

Enter the samauxdb database administrator password.

You are prompted to enter the root user password for the new station.

Enter the password of the root user account on the new station.

CAUTION Installation Failure

If the addition of the station to the auxiliary database is interrupted, the operation fails and support intervention may be required.

You must answer yes to the prompt described in this step.

If the auxiliary database contains a large amount of data, the addition of the station may take considerable time. In such a scenario, the following prompt is displayed:

Do you want to continue waiting? (yes/no) [yes]

Press ↵ to accept the default of yes.

Note: The prompt may be displayed several times during the operation.

When the script execution is complete, open the /opt/nsp/nfmp/auxdb/install/config/install.config file using a plain-text editor such as vi.

CAUTION Service Disruption

Changing a parameter in the auxiliary database install.config file can have serious consequences that include service disruption.

Do not change any parameter in the install.config file, other than the parameters described in the step, without guidance from technical support.

Edit the following line in the file to read:

hosts=internal_IP1,internal_IP2...internal_IPn,new_internal_IP

where

internal_IP1,internal_IP2...internal_IPn are the IP addresses of the existing auxiliary database stations

new_internal_IP is the IP address from which the new station communicates with the other auxiliary database stations

Edit the following line in the file to read:

export_hosts=internal_IP1[export_IP1],internal_IP2[export_IP2]...internal_IPn[export_IPn],new_internal_IP[new_export_IP]

where

internal_IP1[export_IP1],internal_IP2[export_IP2],internal_IPn[export_IPn] are the IP addresses of the existing auxiliary database stations

new_internal_IP is the IP address from which the new station communicates with the other auxiliary database stations

new_export_IP is the IP address from which the new station communicates with the NFM-P servers

Save and close the install.config file.

Enter the following to generate nodeexporter certificates:

# /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh configureTLS force-gen ↵

Initialize auxiliary database

Enter the following:

# ./auxdbAdmin.sh distributeConfig ↵

The updated auxiliary database configuration is distributed to each auxiliary database station.

Rebalance cluster

Enter the following:

# ./auxdbAdmin.sh rebalance ↵

The following prompt is displayed:

Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:

Enter the database user password.

The following messages and prompt are displayed:

KSAFE factor of 1 detected

Rebalance was started in the background. Please don't reboot system until rebalance is completed.

You can track the rebalance process by typing 'ps -fe | grep rebalance'. Rebalance can take a very long time on large databases. Please be patient.

Type YES to continue

CAUTION Data corruption

Interrupting the rebalance operation can cause data corruption.

Do not interrupt the operation by, for example, rebooting the station.

Enter YES.

The rebalance operation begins, and the following is displayed:

Output captured in /opt/nsp/nfmp/auxdb/install/log/auxdbAdmin.sh.timestamp.log

where timestamp is the start time of the rebalance operation

Monitor the rebalance operation; do not proceed to the next step until the operation is complete.

Note: A rebalance operation on a large database may take several hours.

Enter the following periodically to identify whether the rebalance process is active:

#  ps -ef | grep rebalance ↵

If the rebalance process is active, two lines of output are displayed. If the rebalance is complete, only one line like the following is displayed:

root    nnnnn  nnnn 0 hh:mm pts/0   00:00:00 grep --color=auto rebalance

When the rebalance is complete, if the auxiliary database is not geo-redundant, go to Step 39.

Update standby auxiliary database configuration

Log in to an existing standby auxiliary database station as the root user.

Open a console window.

Perform Step 19 to Step 32.

Stop the standby auxiliary database; enter the following on an existing standby auxiliary database station:

# ./auxdbAdmin.sh stop ↵

If the NSP deployment does not include the NFM-P, go to Step 42.

Configure NFM-P main servers

CAUTION Misconfiguration Risk

If you alter the original IP-address assignments, or the order of the IP addresses, the station addition fails.

Do not change any of the original auxiliary database station IP address assignments, or the address order, in the main server configuration.

In a geo-redundant auxiliary database deployment, the order of the IP addresses must match in each main server configuration in each data center.

Perform the following steps on each main server station.

Note: In a geo-redundant system, you must perform the steps on the main servers in each data center.

Note: In a redundant system, you must perform the steps on the standby main server station first.

1. Log in to the main server station as the nsp user.

2. Open a console window.

3. Enter the following:

   bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

4. Enter the following:

   bash$ ./nmsserver.bash stop ↵

5. Enter the following:

   bash$ ./nmsserver.bash appserver_status ↵

   The server status is displayed; the server is fully stopped if the status is the following:

   Application Server is stopped

   If the server is not fully stopped, wait five minutes and then repeat this step. Do not perform the next step until the server is fully stopped.

6. Enter the following to switch to the root user:

   bash$ su ↵

7. Enter the following:

   # samconfig -m main ↵

   The following is displayed:

   Start processing command line inputs...

   <main>

8. Enter the following:

   Note: For a geo-redundant auxiliary database, the order of the IP addresses must be the same on each main server in each data center.

   <main> configure auxdb ip-list IP_list exit ↵

   where

   IP_list is a list of the IP addresses in the following format:

   cluster_1_IP1,cluster_1_IP2,cluster_1_IPn,cluster_1_new_IP;cluster_2_IP1,cluster_2_IP2,cluster_2_IPn,cluster_2_new_IP

   where new_IP is the external IP address of the new station

   Note: The format example shows the new address added to cluster_1; cluster_2 addresses are present only for a geo-redundant auxiliary database. Ensure that you add new_IP to the address list for the appropriate cluster.

9. Enter the following:

   <main> apply ↵

   The configuration is applied.

10. Enter the following:

    <main> exit ↵

    The samconfig utility closes.

11. Enter the following to switch back to the nsp user:

    # exit ↵

Start NFM-P main servers

On the standalone or primary main server station, enter the following to start the main server:

bash$ ./nmsserver.bash start ↵

The main server starts, and the station is added to the auxiliary database.

If the NFM-P system is redundant, enter the following on the standby main server station to start the main server:

bash$ ./nmsserver.bash start ↵

The main server starts.

Configure NSP clusters

Log in as the root or NSP admin user on the NSP deployer host.

Open the following file using a plain-text editor such as vi:

/opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/nsp-config.yml

Locate the following section:

    auxDb:

      secure: "value"

      ipList: ""

      standbyIpList: ""

Edit the section to include the new auxiliary database station address, and to specify whether TLS is enabled, as shown below:

Note: You must preserve the leading spaces in each line.

    auxDb:

      secure: "value"

      ipList: "cluster_1_IP1,cluster_1_IP2 ... cluster_1_IPn"

      standbyIpList: "cluster_2_IP1,cluster_2_IP2 ... cluster_2_IPn"

where

cluster_1_IP1, cluster_1_IP2...cluster_1_IPn are the external IP addresses of the stations in the local cluster, including the new station address

cluster_2_IP1, cluster_2_IP2,cluster_2_IPn are the external IP addresses of the stations in the peer cluster, including the new station address; required only for geo-redundant deployment

Save and close the nsp-config.yml file.

Enter the following to put the changes into effect:

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the nspdeployerctl command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass install --config –-deploy

# /opt/nsp/NSP-CN-DEP-release-ID/bin/nspdeployerctl install --config --deploy ↵

The NSP configuration is updated.

Enter the following:

# kubectl get pods -A ↵

The pods are listed.

Locate the auxiliary database pod entry, which resembles the following:

namespace     nspos-auxdb-agent-pod_ID     1/1     Running     0          uptime

Enter the following:

# kubectl delete pod nspos-auxdb-agent-pod_ID -n namespace ↵

where

namespace is the namespace of the pod entry in Step 49

pod_ID is the pod ID of the auxiliary database pod entry in Step 49

The pod is deleted and recreated to include the new station.

Start auxiliary database proxies

Perform the following steps on the existing and new stations in each auxiliary database cluster.

1. Log in to the station as the root user.

2. Open a console window.

3. Enter the following:

   # systemctl start nspos-auxdbproxy.service ↵

   The auxiliary database proxy starts.

Close the open console windows.

End of steps