How do I restore an auxiliary database?

Purpose

Perform this procedure to restore an auxiliary database from an auxiliary database backup file set.

CAUTION 

CAUTION

Service Disruption

Restoring an auxiliary database in a deployment that includes the NFM-P requires a shutdown of the NFM-P system and causes a network management outage.

Ensure that you perform this procedure only during a scheduled maintenance period.

Steps
Stop auxiliary database proxies
 

Stop the database proxy on each auxiliary database station.

Note: In a geo-redundant auxiliary database, you must stop the proxy on each station in both the primary and standby clusters, starting with the standby cluster's auxiliary database stations.

  1. Log in as the root user on the station.

  2. Enter the following:

    # systemctl stop nspos-auxdbproxy.service ↵

  3. Verify that the proxy is stopped; enter the following:

    systemctl status nspos-auxdbproxy.service ↵


If the NSP deployment does not include the NFM-P, go to Step 5.


Stop NFM-P main and auxiliary servers
 

If the NFM-P system is redundant, stop the standby main server and the associated auxiliary servers.

  1. Perform How do I stop an auxiliary server? to stop each Preferred and Reserved auxiliary server of the standby main server.

  2. Log in to the standby main server as the nsp user.

  3. Open a console window.

  4. Enter the following:

    bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

  5. Enter the following to stop the main server:

    bash$ ./nmsserver.bash stop ↵

  6. Enter the following to display the main server status:

    bash$ ./nmsserver.bash appserver_status ↵

    The server status is displayed; the server is fully stopped if the status is the following:

    Application Server is stopped

    If the server is not fully stopped, wait five minutes and then repeat this step. Do not perform the next step until the server is fully stopped.


Stop the standalone or primary main server and the associated auxiliary servers.

  1. Perform How do I stop an auxiliary server? to stop each Preferred and Reserved auxiliary server of the primary or standalone main server.

  2. Log in to the main server station as the nsp user.

  3. Open a console window.

  4. Enter the following:

    bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

  5. Enter the following to stop the main server:

    bash$ ./nmsserver.bash stop ↵

  6. Enter the following to display the main server status:

    bash$ ./nmsserver.bash appserver_status ↵

    The server status is displayed; the server is fully stopped if the status is the following:

    Application Server is stopped

    If the server is not fully stopped, wait five minutes and then repeat this step. Do not perform the next step until the server is fully stopped.


Prepare auxiliary database stations
 

If the auxiliary database is geo-redundant, you must identify which auxiliary database cluster is currently primary.

If you do not know which cluster is currently the primary cluster, perform How do I check the auxiliary database status?.


Stop the standalone or primary auxiliary database.

Note: In a geo-redundant auxiliary database, you must ensure that each auxiliary database cluster is stopped before you attempt to perform a restore operation.

  1. Log in as the root user on any station in the standalone or primary auxiliary database cluster.

  2. Enter the following to block external access to the auxiliary database ports:

    /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh shieldsUp ↵

  3. Enter the following to stop the auxiliary database:

    /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh stop ↵

    You are prompted to enter the database administrator user's password.

  4. Enter the password. The auxiliary database stops.

  5. Enter the following to display the auxiliary database status:

    /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh status ↵

    Information like the following is displayed:

    Database status

     Node       | Host          | State | Version | DB

    ------------+---------------+-------+---------+-------

     node_1     | internal_IP_1 | STATE | version | db_name

     node_2     | internal_IP_2 | STATE | version | db_name

    .

    .

    .

     node_n     | internal_IP_n | STATE | version | db_name

          Output captured in log_file

    The cluster is stopped when each STATE entry reads DOWN.

  6. Repeat substep 5 periodically until the cluster is stopped.

    Note: If the cluster is not stopped, enter the following to force the auxiliary database to stop.

    /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh force_stop ↵

    Note: You must not proceed to the next step until the cluster is stopped.


If the backup files to restore are not in the original backup location on each auxiliary database station, perform the following steps.

  1. If you know the original backup location, go to substep 6.

  2. Open the following file for viewing:

    path/AuxDbBackUp/samAuxDbBackup_restore.conf

    where path is the current location of the backup file set

  3. Locate the [Mapping] section, which contains one line like the following for each auxiliary server station:

    v_name_node0001 = IP_address:path/AuxDbBackUp

    The path is the original backup location.

  4. Record the original backup location.

  5. Close the samAuxDbBackup_restore.conf file.

  6. Copy the AuxDbBackUp directory contents from the current backup location to the AuxDbBackUp directory in the original backup location on each auxiliary database station.

  7. As the root user, enter the following command on each auxiliary database station:

    chown -R samauxdb path

    where path is the absolute path of the original backup location


If the backup is being restored on any stations that have different internal IP addresses, perform the following steps on each auxiliary database station.

  1. Open the following file using a plain-text editor such as vi:

    path/AuxDbBackUp/samAuxDbBackup_restore.conf

    where path is the location of the backup file set

  2. Add the following internal mapping section to the end of the file; the example below is for an auxiliary database of three stations:

    [NodeMapping]

    v_name_node0001 = IP_address_1

    v_name_node0002 = IP_address_2

    v_name_node0003 = IP_address_3

    where

    v_name_node000n is the station name shown in the [Mapping] section of the file

    IP_address_n is the new internal IP address of the station

  3. Save and close the samAuxDbBackup_restore.conf file.


Perform auxiliary database restore operation
 

Perform one of the following on one auxiliary database station.

  1. Restore the latest backup; enter the following:

    /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh restore path/AuxDbBackUp/samAuxDbBackup_restore.conf ↵

    where path is the original backup directory

  2. Restore a backup other than the latest; perform the following steps.

    1. Enter the following:

      ls path/AuxDbBackUp/.auxdb_backup_history ↵

      where path is the original backup directory

      The directory contents are listed; the following files are present for each previous backup:

      • AuxDbBackUpID_datestamp_timestamp_samAuxDbBackup_info.txt

      • AuxDbBackUpID_datestamp_timestamp_samAuxDbBackup_restore.txt

      where

      datestamp is the backup date, in the form YYYYMMDD

      timestamp is the backup time, in the form hhmmss

      ID is a unique numerical identifier

    2. Based on the date and time of the backup that you want to restore, identify and record the ID, datestamp, and timestamp values.

    3. Enter the following:

      /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh restore path/AuxDbBackUp/.auxdb_backup_history/AuxDbBackUpID_datestamp_timestamp_samAuxDbBackup_restore.conf ↵

      where

      path is the original backup directory

      ID is the recorded ID value

      datestamp is the recorded datestamp value

      timestamp is the recorded timestamp value

The restore operation begins. The following messages and progress indicators are displayed:

Starting auxiliary database restore...

Starting full restore of database db_name.

Participating nodes: node_1node_2... node_n.

Restoring from restore point: AuxDbBackUpID_datestamp_timestamp

Determining what data to restore from backup.

[==============================================] 100%

Approximate bytes to copy: nnnnnnnn of nnnnnnnnn total.

Syncing data from backup to cluster nodes.

When the restore is complete, the second progress indicator reaches 100%, and the following message is displayed:

[==============================================] 100%

Restoring catalog.

Restore complete!

Once the restore is complete, the database will start and post-restore configuration will be performed.

Vertica restore completed. Starting the database.

Info: no password specified, using none

Going with traditional slower startup

 Starting nodes:

  v_samdb_node0001 (IP_a)

  .

  .

  .

Database samdb: Startup Succeeded.  All Nodes are UP

The following prompt is displayed:

Please enter auxiliary database dba password [if you are doing initial setup for auxiliary database, press enter]:


10 

Enter the database administrator password recorded during creation of the backup.

Post-restore configuration output appears on the console, followed by:

  Output captured in /opt/nsp/nfmp/auxdb/install/log/auxdbAdmin.sh.timestamp.log


11 

Enter the following to configure TLS settings:

/opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh configureTLS ↵


12 

Enter the following to allow external access to the auxiliary database ports:

/opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh shieldsDown ↵


Restore database user passwords
 
13 

If a database user password has changed since the creation of the database backup, the NSP components cannot authenticate with the restored auxiliary database.

In such a scenario, you must perform the following steps to update each affected database user password to the value that the NSP components recognize.

  1. Enter the following on the current auxiliary database station to connect to the auxiliary database:

    su - samauxdb ↵

    vsql ↵

    You are prompted for credentials.

  2. Enter the following:

    • user—samauxdb

    • password—database administrator password recorded during creation of backup

  3. Enter the following once for each user whose password has changed since the backup creation:

    ALTER USER user IDENTIFIED BY 'password'; ↵

    where

    user is the database user name

    password is the newer password recognized by the NSP

  4. Exit vsql:

    \q ↵

  5. Enter the following with each affected user to verify the success of the password change:

    vsql -U user -c '\echo success' ↵

    Enter the newer password for the user.

    Confirm that the word "success" appears on the console.

  6. Ensure that the NSP components are able to connect to the auxiliary database.


Start auxiliary database proxies
 
14 

Start the auxiliary database proxy on each auxiliary database station.

Note: In a geo-redundant auxiliary database, you must start the proxy on the stations of the restored auxiliary database cluster first.

  1. Log in as the root user on the station.

  2. Enter the following:

    # systemctl start nspos-auxdbproxy.service ↵


15 

If the NSP deployment does not include the NFM-P, go to Step 18.


Start NFM-P servers
 
16 

Start the standalone or primary main server and associated auxiliary servers.

  1. Log in to the main server station as the nsp user.

  2. Open a console window.

  3. Enter the following:

    bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

  4. Enter the following to start the main server:

    bash$ ./nmsserver.bash start ↵

  5. Enter the following:

    bash$ ./nmsserver.bash appserver_status ↵

    The server status is displayed; the server is fully initialized if the status is the following:

    Application Server process is running.  See nms_status for more detail.

    If the server is not fully initialized, wait five minutes and then repeat this step. Do not perform the next step until the server is fully initialized.

  6. Perform How do I start an auxiliary server? to start each Preferred and Reserved auxiliary server of the primary or standalone main server.


17 

If the NFM-P system is redundant, start the standby main server and associated auxiliary servers.

  1. Log in to the standby main server as the nsp user.

  2. Open a console window.

  3. Enter the following:

    bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

  4. Enter the following to start the main server:

    bash$ ./nmsserver.bash start ↵

  5. Enter the following to display the main server status:

    bash$ ./nmsserver.bash appserver_status ↵

    The server status is displayed; the server is fully initialized if the status is the following:

    Application Server process is running.  See nms_status for more detail.

    If the server is not fully initialized, wait five minutes and then repeat this step. Do not perform the next step until the server is fully initialized.

  6. Perform How do I start an auxiliary server? to start each Preferred and Reserved auxiliary server of the standby main server.


18 

Close the open console windows.

End of steps