How do I recover a failed nsp-tomcat database in a DR NSP deployment?
Purpose
Perform this procedure if you experience nsp-tomcat database inconsistencies and need to restore the database.
Steps
1 |
If the inconsistent database instance is in the primary data center, check the standby instance status.
|
2 |
If either command in Step 1 fails due to inconsistencies, perform How do I recover both failed nsp-tomcat databases in a DR NSP deployment using a database backup?. |
3 |
If both commands in Step 1 succeed, perform a switchover.
|
4 |
Enter the following to perform a manual backup on the new primary NSP cluster host: # kubectl create job nsp-tomcat-backup-manual --from cronjob/nsp-tomcat-backup ↵ |
5 |
If the backup fails due to inconsistencies, both database instances are inconsistent; perform How do I recover both failed nsp-tomcat databases in a DR NSP deployment using a database backup?. |
6 |
Enter the following: # kubectl exec nsp-backup-storage-0 -- ls /tmp/backups/nsp-tomcat/ ↵ The backup files are listed. |
7 |
Record the name of the most recent backup. |
8 |
Enter the following: # kubectl cp nsp-backup-storage-0:/tmp/backups/nsp-tomcat/backup_file /tmp/backup_file ↵ where backup_file is the recorded backup file name The backup file is copied to the temp directory on the NSP cluster host. |
9 |
Enter the following: # scp /tmp/backup_file address:/tmp ↵ where address is the address of the new standby NSP cluster host The backup file is copied to the /tmp directory on the new standby NSP cluster host. |
Restore database on new standby cluster | |
10 |
Log in as the root or NSP admin user on the new standby NSP cluster host. |
11 |
Open a console window. |
12 |
Enter the following: # helm list | grep nsp-tomcat | awk -F' ' '{ print $10 }' ↵ The nsp-tomcat version is displayed. |
13 |
Record the nsp-tomcat version. |
14 |
Enter the following to uninstall nsp-tomcat: # helm uninstall -n nsp-psa-restricted nsp-tomcat ↵ The nsp-tomcat uninstallation begins. |
15 |
Enter the following command block to monitor the uninstallation: while [ 0 -lt `kubectl get pvc --all-namespaces | grep nsp-tomcat | wc -l` ] || [ 0 -lt `kubectl get pv --all-namespaces | grep nsp-tomcat | wc -l` ] do sleep 2 echo "Still there..." done The uninstallation is complete when the command prompt is displayed. |
16 |
When the uninstallation is complete, enter the following to install nsp-tomcat: # helm upgrade nsp-tomcat --install oci://registry.nsp.nokia.local/nsp/charts/cn-nsp-tomcat --namespace nsp-psa-restricted --version version --timeout 300s -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/values.yaml -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/restore.yaml ↵ where version is the nsp-tomcat version recorded in Step 13 nsp-tomcat is installed. |
17 |
Enter the following: # kubectl get pods -A ↵ The pods are listed. |
18 |
If the nsp-tomcat pod is not running, repeat the command in Step 17. Note: You must not proceed to the next step until nsp-tomcat is running. |
19 |
Enter the following: # /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/tools/database/nspos- db-restore-k8s.sh nsp-tomcat /tmp/backup_file ↵ You are prompted to restore the database. |
20 |
Respond to the prompt. The nsp-tomcat database is restored from the backup file. |
21 |
Enter the following to uninstall nsp-tomcat: # helm uninstall -n nsp-psa-restricted nsp-tomcat ↵ The nsp-tomcat uninstallation begins. Note: You must not proceed to the next step until the uninstallation is complete. |
22 |
The nsp-tomcat uninstallation begins. Enter the following command block to monitor the uninstallation: while [ 0 -lt `kubectl get pvc --all-namespaces | grep nsp-tomcat | wc -l` ] || [ 0 -lt `kubectl get pv --all-namespaces | grep nsp-tomcat | wc -l` ] do sleep 2 echo "Still there..." done The uninstallation is complete when the command prompt is displayed. When the uninstallation is complete, enter the following: |
23 |
When the uninstallation is complete, enter the following to install nsp-tomcat: # helm upgrade nsp-tomcat --install oci://registry.nsp.nokia.local/nsp/charts/cn-nsp-tomcat --namespace nsp-psa-restricted --version version --timeout 300s -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/values.yaml ↵ where version is the nsp-tomcat version recorded in Step 13 nsp-tomcat is installed. |
24 |
When the nsp-tomcat pod installation is complete, monitor the following log file in the nsp-tomcat pod to ensure that the database starts successfully: /opt/nsp/server/replication/logs/debug.log |
Back up database | |
25 |
Optionally, to confirm that the database restore is successful, perform a switchover and perform a backup to ensure that no inconsistencies remain. |
26 |
Close the open console windows. End of steps |