How do I recreate an auxiliary database?

Purpose

If an NSP auxiliary database is not functioning and cannot return to operational status, you may need to recreate the database. Depending on the severity of the failure, however, Nokia support may be able to repair the database.

Perform this procedure only if all attempts to repair the auxiliary database fail, there is no database backup from which the database may be restored, and Nokia support directs you to recreate the database.

Note: If you are performing this procedure to recover from a failed upgrade, uninstall and install the auxiliary database, then restore the backup taken prior to upgrade as described in the NSP Installation and Upgrade Guide instead of executing this procedure.

Steps
 

If possible, back up the auxiliary database to preserve it for potential data recovery. If data recovery is not required, you can skip this step.

Note: The auxiliary database must be running in order for a backup to be successful. If the auxiliary database cannot be started, you cannot create a database backup.


In order to back up an auxiliary database, all members of the local auxiliary database cluster must be operational and available, or the backup fails. Perform How do I check the auxiliary database status? to verify that each local cluster member is operational.


Perform the following steps on each auxiliary database station to collect the auxiliary database log files.

Note: You cannot specify /tmp, or any directory below /tmp, as the output directory.

  1. Log in to the station as the root user.

  2. Enter the following:

    /opt/nsp/nfmp/auxdb/install/bin/auxdbAdmin.sh getDebugFiles output_dir days

    where

    output_dir is a local directory that is to contain the output files

    days is the optional number of days for which to collect log files; if not specified, all logs are collected


On one auxiliary database station, open the /opt/nsp/nfmp/auxdb/install/config/install.config file using a plain-text editor such as vi.


Ensure that the internal and external IP addresses in the following lines are correctly assigned to the auxiliary database stations:

hosts=internal_IP1,internal_IP2...internal_IPn

export_hosts=internal_IP1[export_IP1],internal_IP2[export_IP2]...internal_IPn[export_IPn]


If you intend to reuse existing auxiliary database stations, uninstall the auxiliary database as described in the NSP Installation and Upgrade Guide.


Install the auxiliary database as described in the NSP Installation and Upgrade Guide.

Note: When prompted to enter passwords for auxiliary database users, you must use the same passwords that were previously used for each user.


Reconfigure NFM-P auxiliary database tables
 

If the NSP deployment does not include the NFM-P, go to Step 15.


When the auxiliary database is fully initialized, perform the following steps on each NFM-P main server station to start the main server, if the server is not started.

Note: In a redundant system, you must start the primary main server first.

  1. Log in as the root user on the main server station.

  2. Open a console window.

  3. Enter the following:

    bash$ ./nmsserver.bash start ↵

  4. Enter the following:

    bash$ ./nmsserver.bash appserver_status ↵

    The server status is displayed; the server is fully initialized if the status is the following:

    Application Server process is running.  See nms_status for more detail.

    If the server is not fully initialized, wait five minutes and then repeat this step. Do not perform the next step until the server is fully initialized.

The main server detects the auxiliary database reinstallation, raises a Critical alarm against the auxiliary database, and begins recreating the auxiliary database schema.

Note: If the main server fails to recreate the schema, the server retries periodically until all schema elements are created.

When the schema recreation is complete, the reinstallation alarm clears, and the NFM-P raises a Major alarm that you must clear manually.


10 

If the auxiliary database includes custom data tables, perform the following steps to recreate and repopulate the tables.

  1. Log on to the standalone or primary main server station as the nsp user.

  2. Enter the following:

    bash$ cd /opt/nsp/nfmp/server/nms/bin ↵

  3. Create each custom table, as described in Step 6 of How do I create and manage custom auxiliary database table attributes in NFM-P?, using the saved XML file that defines the table.

  4. Add the required data to each table, as described in Step 7 of How do I create and manage custom auxiliary database table attributes in NFM-P?, using the saved CSV file for the table.


11 

If you use the Analytics dynamic inventory reporting function, you must recreate the custom inventory tables and dynamic Analytics managed-object definitions.

  1. Remove the existing dynamic Analytics managed-object definitions; enter the following on the main server station:

    bash$ ./nmsserver.bash dynamic_analyticmo remove class mo_name ↵

    where

    class is the object class

    mo_name is the name of the dynamic Analytics managed object

  2. Create the custom inventory tables; enter the following:

    bash$ ./customData.bash --createTables XML_file

    where XML_file is the saved XML file that defines the custom inventory tables

  3. Create the dynamic Analytics managed-object definitions; enter the following:

    bash$ ./nmsserver.bash dynamic_analyticmo create dyn_inv_XML_file

    where dyn_inv_XML_file is the absolute path and name of the saved XML file that defines the Analytics dynamic inventory configuration


12 

If you have created dynamic aggregations, recreate the aggregations.

Note: Any previous aggregation data is lost during the auxiliary database schema recreation earlier in the procedure.

  1. Enter the following on the main server station for each dynamic aggregation to remove the aggregation:

    bash$ ./nmsserver.bash dynamic_aggregation remove aggregation ↵

    where aggregation is the name of the aggregation

  2. Enter the following to create the dynamic aggregation:

    bash$ ./nmsserver.bash dynamic_aggregation create agg_XML_file

    where agg_XML_file is the absolute path and name of the saved XML file that defines the aggregation configuration

  3. Re-enable the dynamic aggregations using an NFM-P XML API or GUI client.


13 

If you use periodic accounting tables, enter the following on the main server station to create the tables:

Note: Any previous periodic accounting data is lost during the auxiliary database schema recreation earlier in the procedure.

bash$ ./nmsserver.bash accountingPeriodic create per_acc_XML_file

where per_acc_XML_file is the absolute path and name of the saved XML file that defines the periodic accounting configuration


14 

Close the open console windows.


Reset the collection schema
 
15 

Log in as the root or NSP admin user on the NSP deployer host.


16 

Enter the following:

cd /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/tools/mdm/bin ↵

./yang-files.bash --user user --pass password --reset collection ↵


Undeploy the NSP
 
17 

Log in as the root or NSP admin user on the NSP deployer host.


18 

Perform Step 19 to Step 22 on each NSP cluster, and then go to Step 23.

Note: In a DR deployment, you must perform the steps first on the standby cluster.


19 

Perform the following steps to preserve the existing deployment configuration.

  1. Open the following file using a plain-text editor such as vi:

    /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/config/nsp-config.yml

  2. Edit the following line in the platform section, kubernetes subsection to read as shown below:

      deleteOnUndeploy:false

  3. Save and close the file.


20 

Enter the following:

cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵


21 

Enter the following to undeploy the NSP:

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass uninstall --undeploy

./nspdeployerctl uninstall --undeploy ↵


22 

On the NSP cluster host, enter the following periodically to display the status of the Kubernetes system pods:

Note: You must not proceed to the next step until the output lists only the following:

  • pods in kube-system namespace

  • nsp-backup-storage pod

kubectl get pods -A ↵

The pods are listed.


Redeploy the NSP
 
23 

Enter the following to redeploy the NSP:

Note: In a DR deployment, you must perform the steps first on the primary cluster.

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass install --deploy

./nspdeployerctl install --deploy ↵

The re-creation of the auxiliary database collection tables begins.

Do not perform the next step until the auxiliary database collection tables are fully initialized.


24 

Close the open console windows.


Reconfigure NSP auxiliary database tables
 
25 

If auxiliary database custom data tables were present before the reinstallation, reconfigure the custom data tables as described in procedure: How do I create and manage custom auxiliary database table attributes in NSP?.


26 

If auxiliary database anonymization was enabled before the reinstallation, re-enable anonymization for Analytics reports as described in the NSP Analytics Report Catalog.

End of steps