Obtain software

Download the following installation files to an empty local directory on a station that is reachable by each station in the auxiliary database cluster:

• nspos-auxdb-R.r.p-rel.v.rpm

• VerticaSw_PreInstall.sh

• nspos-jre-R.r.p-rel.v.rpm

• vertica-R.r.p-rel.tar

where

R.r.p is the NSP release identifier, in the form MAJOR.minor.patch

v is a version number

Commission new stations, if required

If you are deploying the auxiliary database on one or more new stations, perform the following steps.

For information about deploying the RHEL OS using an NSP OEM disk image, see NSP disk-image deployment.

Note: The IP addresses on the interfaces of a new auxiliary database station must match the addresses on the station that it replaces.

1. Commission each station according to the platform specifications in this guide and in the NSP Planning Guide.

2. Perform To apply the RHEL 8 swappiness workaround on the station.

Back up database

CAUTION Data Loss

If you specify a backup location on the database data partition, data loss or corruption may occur.

The auxiliary database backup location must be an absolute path on a partition other than the database data partition.

Back up the auxiliary database as described in the NSP System Administrator Guide for the installed release.

Note: The backup location requires 20% more space than the database data consumes.

Note: If the backup location is remote, a 1 Gb/s link to the location is required; if achievable, a higher-capacity link is recommended.

Stop and disable auxiliary database proxies

Stop and disable the database proxy on each station.

1. Enter the following sequence of commands as the root user on each station in the auxiliary database cluster:

   systemctl stop nfmp-auxdbproxy.service

   systemctl disable nfmp-auxdbproxy.service

   The proxy stops, and is disabled.

2. Enter the following sequence of commands as the root user on each station in the auxiliary database cluster:

   systemctl stop nfmp-auxdbproxy.service

   systemctl disable nfmp-auxdbproxy.service

   The proxy stops, and is disabled.

Enable auxiliary database maintenance mode

Enable nspos-auxdb-agent maintenance mode.

1. Log in as the root or NSP admin user on the NSP cluster host .

2. Enter the following to set the nspos-auxdb-agent mode to maintenance:

   # kubectl patch configmap/nspos-auxdb-agent-overrides -n namespace --type=merge -p '{"data":{"nspos-auxdb-agent-overrides.json":"{\"auxDbAgent\":{\"config\":{\"maintenance-mode\":true}}}"}}' ↵

   where namespace is the nspos-auxdb-agent namespace

3. Enter the following to restart the nspos-auxdb-agent pod:

   # kubectl delete -n namespace pod `kubectl describe -n namespace pods | grep -P ^^Name: | grep -oP nspos-auxdb-agent[-a-zA-Z0-9]+` ↵

4. Issue the following RESTCONF API call to verify that the agent is in maintenance mode:

   NOTE: In order to issue a RESTCONF API call, you require a token; see this tutorial on the Network Developer Portal for information.

   GET https://address/restconf/data/auxdb:/auxdb-agent

   where address is the advertised address of the NSP cluster

   The call returns information like the following:

   {

       "auxdb-agent": {

           "name": "nspos-auxdb-agent",

           "application-mode": "MAINTENANCE",

           "copy-cluster": {

               "source-cluster": "cluster_2",

               "target-cluster": "cluster_1",

               "time-started": "timestamp",

               "status": "SUCCESS"

           }

       }

   }

   The agent is in maintenance mode if the application-mode is MAINTENANCE, as shown in the example.

5. Log in as the root or NSP admin user on the NSP cluster host.

6. Enter the following to set the nspos-auxdb-agent mode to maintenance:

   # kubectl patch configmap/nspos-auxdb-agent-overrides -n namespace --type=merge -p '{"data":{"nspos-auxdb-agent-overrides.json":"{\"auxDbAgent\":{\"config\":{\"maintenance-mode\":true}}}"}}' ↵

Stop auxiliary database

Log in as the root user on an auxiliary database station.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following to stop the auxiliary database:

# ./auxdbAdmin.sh stop ↵

Enter the following to display the auxiliary database status:

# ./auxdbAdmin.sh status ↵

Information like the following is displayed:

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1 | internal_IP_1 | STATE | version | db_name

 node_2 | internal_IP_2 | STATE | version | db_name

.

.

.

 node_n | internal_IP_n | STATE | version | db_name

      Output captured in log_file

The cluster is stopped when each STATE entry reads DOWN.

Repeat Step 10 periodically until the cluster is stopped.

Note: You must not proceed to the next step until the cluster is stopped.

Prepare all stations for upgrade

Perform Step 14 to Step 30 on each auxiliary database station.

Go to Step 31.

Prepare individual station for upgrade

Log in as the root user on the auxiliary database station.

Open a console window.

Enter the following sequence of commands to stop the auxiliary database services:

systemctl stop nfmp-auxdbproxy.service

systemctl stop nfmp-auxdb.service

systemctl stop vertica_agent.service

systemctl stop verticad.service

Enter the following sequence of commands to disable the database services:

systemctl disable nfmp-auxdb.service

systemctl disable nfmp-auxdbproxy.service

systemctl disable vertica_agent.service

systemctl disable verticad.service

Transfer the downloaded installation files to an empty directory on the station.

Note: You must ensure that the directory is empty.

Note: In subsequent steps, the directory is called the NSP software directory.

Navigate to the NSP software directory.

Note: The directory must contain only the installation files.

Enter the following:

# chmod +x * ↵

Enter the following:

# ./VerticaSw_PreInstall.sh ↵

Messages like the following are displayed:

Logging Vertica pre install checks to log_file

INFO: About to remove proxy parameters set by a previous run of this script from /etc/profile.d/proxy.sh

INFO: Completed removing proxy parameters set by a previous run of this script from /etc/profile.d/proxy.sh

INFO: About to set proxy parameters in /etc/profile.d/proxy.sh...

INFO: Completed setting proxy parameters in /etc/profile.d/proxy.sh...

INFO: About to remove kernel parameters set by a previous run of this script from /etc/sysctl.conf

INFO: Completed removing kernel parameters set by a previous run of this script from /etc/sysctl.conf

INFO: About to set kernel parameters in /etc/sysctl.conf...

INFO: Completed setting kernel parameters in /etc/sysctl.conf...

INFO: About to change the current values of the kernel parameters

INFO: Completed changing the current values of the kernel parameters

INFO: About to remove ulimit parameters set by a previous run of this script from /etc/security/limits.conf

INFO: Completed removing ulimit parameters set by a previous run of this script from /etc/security/limits.conf

INFO: About to set ulimit parameters in /etc/security/limits.conf...

INFO: Completed setting ulimit parameters in /etc/security/limits.conf...

Checking Vertica DBA group samauxdb...

WARNING: Vertica DBA group with the specified name already exists locally.

Checking Vertica user samauxdb...

WARNING: Vertica user with the specified name already exists locally.

Changing ownership of the directory /opt/nsp/nfmp/auxdb/install to samauxdb:samauxdb.

Adding samauxdb to sudoers file.

Changing ownership of /opt/nsp/nfmp/auxdb files.

INFO: About to remove commands set by a previous run of this script from /etc/rc.d/rc.local

INFO: Completed removing commands set by a previous run of this script from /etc/rc.d/rc.local

INFO: About to add setting to /etc/rc.d/rc.local...

INFO: Completed adding setting to /etc/rc.d/rc.local...

Enter the following to reboot the station:

# systemctl reboot ↵

The station reboots.

When the reboot is complete, log in as the root user on the station.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following to display the auxiliary database status:

# ./auxdbAdmin.sh status ↵

Information like the following is displayed:

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1 | internal_IP_1 | STATE | version | db_name

 node_2 | internal_IP_2 | STATE | version | db_name

.

.

.

 node_n | internal_IP_n | STATE | version | db_name

      Output captured in log_file

if any STATE entry is not DOWN, perform the following steps.

1. Enter the following to stop the auxiliary database:

   # ./auxdbAdmin.sh stop ↵

2. Repeat Step 26 periodically until each STATE entry reads DOWN.

   Note: You must not proceed to the next step until each STATE entry reads DOWN.

Navigate to the NSP software directory.

Enter the following:

# yum install nspos-*.rpm ↵

The yum utility resolves any package dependencies, and displays the following prompt for each package:

Total size: nn G

Installed size: nn G 

Is this ok [y/d/N]: 

Enter y. The following and the installation status are displayed as each package is installed:

Downloading Packages:

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

The package installation is complete when the following is displayed:

Complete!

Upgrade database

Log in as the root user on an auxiliary database station.

Open a console window.

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following:

# ./auxdbAdmin.sh upgrade tar_file ↵

where tar_file is the absolute path and filename of the vertica-R.r.p-rel.tar file in the NSP software directory

The following prompt is displayed:

Updating Vertica - Please perform a backup before proceeding with this option

Do you want to proceed (YES/NO)?

Enter YES ↵.

The following prompt is displayed:

Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:

Enter the dba password.

The following prompt is displayed:

Please verify auxiliary database dba password:

Enter the dba password again.

The upgrade begins, and operational messages are displayed.

The upgrade is complete when the following is displayed:

Database database_name started successfully

  Output captured in log_file

Enter the following to display the auxiliary database status:

# ./auxdbAdmin.sh status ↵

Information like the following is displayed:

Database status

 Node       | Host          | State | Version | DB

------------+---------------+-------+---------+-------

 node_1 | internal_IP_1 | STATE | version | db_name

 node_2 | internal_IP_2 | STATE | version | db_name

.

.

.

 node_n | internal_IP_n | STATE | version | db_name

      Output captured in log_file

The cluster is running when each STATE entry reads UP.

Repeat Step 38 periodically until the cluster is running.

Note: You must not proceed to the next step until the cluster is running.

Back up database for migration to new OS version

You must back up the auxiliary database data and configuration information to prepare for the migration to the new OS version.

Note: The operation may take considerable time.

Perform the following steps.

1. Enter the following:

   # ./auxdbAdmin.sh backup /path/samAuxDbBackup_restore.conf ↵

   where path is the backup file path specified in Step 3

   The following prompt is displayed:

   Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:

2. Enter the dba password.

   The backup operation begins, and messages like the following are displayed.

   Copying backup config file to /tmp/auxdbadmin-backup-ID

   Backup snapshot name - AuxDbBackUpID_auxdbAdmin

   Starting auxiliary database backup...

   The backup is complete when the following is displayed:

   Output captured in 

   /opt/nsp/nfmp/auxdb/install/log/auxdbAdmin.sh.timestamp.log

3. Transfer the database backup file to a secure location on a separate station that is unaffected by the upgrade activity.

4. Open the following file using a plain-text editor such as vi:

   /opt/nsp/nfmp/auxdb/install/config/install.config

5. Add the following lines directly above the line that reads, “# Do not modify.....”

   # TLS settings.

   # Set secure=false (default is enabled) to disable secure JDBC and

   # proxy connections.

   # If secure=true then a PKI Server IP address or hostname must be

   # specified in order to generate certificates.

   secure=true

   pki_server=address

   pki_server_port=80

   where address is one of the following in the platform—ingressApplications—ingressController section of the nsp-config.yml file on the local NSP deployer host:

   In the internalAddresses subsection, if configured, otherwise, in the clientAddresses subsection:

   • if configured, the advertised value

   • otherwise, the virtualIp value

6. Save and close the install.config file.

7. Copy the install.config file to a secure location on a separate station that is unaffected by the upgrade activity.

Perform the following steps on each auxiliary database station to preserve the upgrade logs.

1. Enter the following:

   # tar czf /opt/nsp/auxdb-node_N-upgrade-log.tar.gz /opt/vertica/log /opt/nsp/nfmp/auxdb/install/log /opt/nsp/nfmp/auxdb/install/proxy/log /opt/nsp/nfmp/auxdb/catalog/samdb/v_samdb_node*/vertica.log ↵

   where N is the node number, as shown in the Step 38 output example

   An auxdb-node_N-upgrade-log.tar.gz file is created in the current working directory.

2. Transfer the file to a secure location on a separate station that is unaffected by the upgrade activity.

Recommission stations, if required

If you are reusing any auxiliary database stations, recommission each station according to the platform specifications in this guide and in the NSP Planning Guide.

For information about deploying the RHEL OS using an NSP OEM disk image, see NSP disk-image deployment.

Note: You must reformat the following disk partitions:

• root

• /opt/nsp/nfmp/auxdb/data

Install new software

Perform To prepare a station for NSP auxiliary database installation on each auxiliary database station.

Copy each auxdb-node_N-upgrade-log.tar.gz created in Step 41 to the /opt/nsp directory on the associated auxiliary database station.

Log in as the root user on an auxiliary database station.

Copy the install.config file saved in Step 40, substep 7 to the following directory:

/opt/nsp/nfmp/auxdb/install/config

Enter the following:

# cd /opt/nsp/nfmp/auxdb/install/bin ↵

Enter the following to block external access to the auxiliary database ports:

# ./auxdbAdmin.sh shieldsUp ↵

Enter the following to regenerate the TLS certificates:

# ./auxdbAdmin.sh configureTLS force-gen ↵

Enter the following:

# ./auxdbAdmin.sh install ↵

The script sequentially prompts you to enter and re-enter new passwords for the following user accounts:

• samauxdb

• samuser

• samanalytic

• samanalytic_ano

At each prompt, enter or re-enter a password, as required.

The script then sequentially prompts for the root user password of each auxiliary database station.

Enter the required password at each prompt.

Messages like the following are displayed as the software is installed on each station and the database is created:

Populating auxiliary database user passwords in the vault

Installing auxiliary database on IP_address ....

Cleaning auxiliary database host(s) in ssh known_hosts for root and samauxdb users.

Creating auxiliary database cluster

   Successfully created auxiliary database cluster

   Creating auxiliary database

Distributing changes to cluster.

        Creating database samdb

        Starting bootstrap node node_name (IP_address)

        Starting bootstrap node node_name (IP_address)

        .

        .

        .

        Starting nodes:

                node_name (IP_address

                node_name (IP_address

                .

                .

                .

        Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.

Installing OS_package package

        Success: package OS_package installed

Database creation SQL tasks completed successfully.

Database samdb created successfully.

  Successfully created auxiliary database

  Performing post install configuration tasks

Creating public interface for host node_name (IP_address

Creating public interface for host node_name (IP_address

.

.

.

CREATE NETWORK INTERFACE

ALTER NODE

Setting DB samdb restart policy to never and replicating to cluster...

Database samdb policy set to never

Installing user defined extension libraries and functions.

Unzipping Python libraries.

Setting sticky bit on all nodes.

  INFO: About to configure TLS ....

  Generating TLS certificates

  INFO: About to validate key and certificate

        Make sure the certificate has not expired and that the specified date range is current and valid.

            Not Before: date

            Not After : date

  INFO: Complete validating key and certificate

  INFO: Adding certificate to AuxDB

Distributing configuration to all nodes

   Post install configuration tasks completed successfully.

   Successfully installed auxiliary database.

  Output captured in /opt/nsp/nfmp/auxdb/install/log/auxdbAdmin.sh.timestamp.log

When the software installation on each station is complete, stop the auxiliary database.

1. Enter the following:

   # ./auxdbAdmin.sh stop ↵

2. Enter the following to display the auxiliary database status:

   # ./auxdbAdmin.sh status ↵

   Information like the following is displayed:

   Database status

    Node       | Host          | State | Version | DB

   ------------+---------------+-------+---------+-------

    node_1 | internal_IP_1 | STATE | version | db_name

    node_2 | internal_IP_2 | STATE | version | db_name

   .

   .

   .

    node_n | internal_IP_n | STATE | version | db_name

         Output captured in log_file

   The cluster is stopped when each STATE entry reads DOWN.

3. Repeat substep 2 periodically until the cluster is stopped.

   Note: You must not proceed to the next step until the cluster is stopped.

Stop NFM-P main server

If the NSP deployment includes the NFM-P, and the NFM-P is running, stop the NFM-P main server.

1. Log in to the main server station as the nsp user.

2. Open a console window.

3. Enter the following:

   bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

4. Enter the following:

   bash$ ./nmsserver.bash stop ↵

5. Enter the following:

   bash$ ./nmsserver.bash appserver_status ↵

   The server status is displayed; the server is fully stopped if the status is the following:

   Application Server is stopped

   If the server is not fully stopped, wait five minutes and then repeat this step. Do not perform the next step until the server is fully stopped.

6. Enter the following to switch to the root user:

   bash$ su ↵

Prepare to restore auxiliary database

Stop the auxiliary database.

1. Log in as the root user on any station in the auxiliary database cluster.

2. Enter the following:

   # /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh stop ↵

   You are prompted to enter the database user password.

3. Enter the password. The auxiliary database stops.

If the backup files to restore are not in the original backup location on each auxiliary database station, perform the following steps.

1. If you know the original backup location, go to substep 6.

2. Open the following file for viewing:

   path/AuxDbBackUp/samAuxDbBackup_restore.conf

   where path is the current location of the backup file set

3. Locate the [Mapping] section, which contains one line like the following for each auxiliary server station:

   v_name_node0001 = IP_address:path/AuxDbBackUp

   The path is the original backup location.

4. Record the original backup location.

5. Close the samAuxDbBackup_restore.conf file.

6. Copy the AuxDbBackUp directory contents from the current backup location to the AuxDbBackUp directory in the original backup location on each auxiliary database station.

7. As the root user, enter the following command on each auxiliary database station:

   # chown -R samauxdb path ↵

   where path is the absolute path of the original backup location

If the backup is being restored on any stations that have different internal IP addresses, perform the following steps on each auxiliary database station.

1. Open the following file using a plain-text editor such as vi:

   path/AuxDbBackUp/samAuxDbBackup_restore.conf

   where path is the location of the backup file set

2. Add the following internal mapping section to the end of the file; the example below is for an auxiliary database of three stations:

   [NodeMapping]

   v_name_node0001 = IP_address_1

   v_name_node0002 = IP_address_2

   v_name_node0003 = IP_address_3

   where

   v_name_node000n is the station name shown in the [Mapping] section of the file

   IP_address_n is the new internal IP address of the station

3. Save and close the samAuxDbBackup_restore.conf file.

Restore auxiliary database

Enter the following on one auxiliary database station:

# /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh restore path/AuxDbBackUp/samAuxDbBackup_restore.conf ↵

where path is the absolute path of the database backup created in Step 40

The restore operation begins. The following messages and progress indicators are displayed:

Starting full restore of database db_name.

Participating nodes: node_1, node_2, ... node_n.

Restoring from restore point: AuxDbBackUpID_datestamp_timestamp

Determining what data to restore from backup.

[==============================================] 100%

Approximate bytes to copy: nnnnnnnn of nnnnnnnnn total.

Syncing data from backup to cluster nodes.

When the restore is complete, the second progress indicator reaches 100%, and the following message is displayed:

[==============================================] 100%

Restoring catalog. Restore complete!

Enter the following:

# /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh uninstallUDxLibraries ↵

Enter the following:

# /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh installUDxLibraries ↵

Restore database user passwords

If a database user password has changed since the creation of the database backup, the NSP components cannot authenticate with the restored auxiliary database.

In such a scenario, you must perform the following steps to update each affected database user password to the value that the NSP components recognize.

1. Enter the following on the current auxiliary database station to connect to the auxiliary database:

   # vsql ↵

   You are prompted for credentials.

2. Enter the following:

   • user—samauxdb

   • password—database administrator password recorded during creation of backup

3. Enter the following once for each user whose password has changed since the backup creation:

   ALTER USER user IDENTIFIED BY 'password';

   where

   user is the database user name

   password is the newer password recognized by the NSP

4. Verify the success of the password change by:

   • attempting to connect to the auxiliary database as each affected user

   • ensuring that the NSP components are able to connect to the auxiliary database

Update database schema

Enter the following to allow external access to the auxiliary database ports:

# /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh shieldsDown ↵

If the NSP deployment includes the NFM-P, update the NFM-P database schema.

Note: The schema update may take considerable time.

1. Log in as the nsp user on the NFM-P main server.

2. Open a console window.

3. Enter the following:

   bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

4. Enter the following:

   bash$ ./nmsserver.bash upgradeAuxDbSchema ↵

   The following prompt is displayed:

   Auxiliary database clusters:

   1: IP_a,IP_b,IP_c

   Select auxiliary database to upgrade:

5. Enter 1.

   The following messages and prompt are displayed:

   WARNING: About to upgrade samdb schema on the auxiliary database cluster [IP_a,IP_b,IP_c].

   It is recommended that a database backup is performed before proceeding.

   Type "YES" to continue

6. Enter YES.

   The following prompt is displayed:

   Please enter the auxiliary database port [5433]:

7. Enter the auxiliary database port number; press Enter to accept the default of 5433.

   The following prompt is displayed:

   Please enter the auxiliary database user password:

8. Enter the required password.

   The following messages are displayed as the schema update begins:

   INFO: Database upgrade can take a very long time on large databases.

   INFO: logs are stored under /opt/nsp/nfmp/server/nms/log/auxdb. Check the logs for progress.

   INFO: Node Name[v_samdb_node0001]->IP[IP_address]->Status[UP]

   INFO: About to perform upgrade

Enable required database services and proxies

Perform the following steps on each auxiliary database station.

1. Log in as the root user.

2. Open a console window.

3. Enter the following sequence of commands to enable the database services:

   systemctl enable nspos-auxdb.service

   systemctl enable nspos-auxdbproxy.service

   systemctl enable vertica_agent.service

   systemctl enable verticad.service

Enter the following on each auxiliary database station:

# systemctl start nspos-auxdbproxy.service ↵

The auxiliary database proxy starts.

Disable maintenance mode for auxiliary database agents

Activate the nspos-auxdb-agent.

1. Log in as the root or NSP admin user on the NSP cluster host.

2. Enter the following to set the nspos-auxdb-agent mode to active:

   # kubectl patch configmap/nspos-auxdb-agent-overrides -n namespace --type=merge -p '{"data":{"nspos-auxdb-agent-overrides.json":"{\"auxDbAgent\":{\"config\":{\"maintenance-mode\":false}}}"}}' ↵

   where namespace is the nspos-auxdb-agent namespace

3. Enter the following to restart the nspos-auxdb-agent pod:

   # kubectl delete -n namespace pod `kubectl describe -n namespace pods | grep -P ^^Name: | grep -oP nspos-auxdb-agent[-a-zA-Z0-9]+` ↵

The cluster enters active mode within approximately one minute.

Verify auxiliary database status

Issue the following RESTCONF API call to verify that the auxiliary database cluster is in active mode:

Note: In order to issue a RESTCONF API call, you require a token; see this tutorial on the Network Developer Portal for information.

GET /data/auxdb:/auxdb-agent HTTP/1.1

Request body:

    Host: address

    Content-Type: application/json

    Authorization: bearer_and_token_from_session_manager

where address is the advertised address of the NSP cluster

The cluster is in active mode if the response includes ACTIVE.

Issue the following RESTCONF API call to verify the auxiliary database operation:

Note: In order to issue a RESTCONF API call, you require a token; see this tutorial on the Network Developer Portal for information.

GET https://address/restconf/data/auxdb:/clusters

where address is the advertised address of the NSP cluster

The call returns auxiliary database cluster status information, as shown below for a three-node cluster; if the mode and status values are not as shown below, contact technical support. Otherwise, at this point, the auxiliary database is fully upgraded and prepared for the NFM-P main server startup.

<HashMap>

    <clusters>

        <cluster>

            <name>cluster_A</name>

            <mode>ACTIVE</mode>

            <status>UP</status>

            <nodes>

                <external-ip>203.0.113.101</external-ip>

                <internal-ip>10.1.2.101</internal-ip>

                <status>UP</status>

            </nodes>

            <nodes>

                <external-ip>203.0.113.102</external-ip>

                <internal-ip>10.1.2.102</internal-ip>

                <status>UP</status>

            </nodes>

            <nodes>

                <external-ip>203.0.113.103</external-ip>

                <internal-ip>10.1.2.103</internal-ip>

                <status>UP</status>

            </nodes>

        </cluster>

    </clusters>

</HashMap>

Close the open console windows.

End of steps