If the inconsistent database instance is in the primary data center, check the standby instance status.

1. Log in as the root or NSP admin user on the standby NSP cluster host.

2. Open a console window.

3. Enter the following:

   # kubectl exec -it -n $(kubectl get pods -A | grep nsp-tomcat | tail -1 | awk '{print $1}') $(kubectl get pods -A | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $2}') -c nsp-tomcat -- /opt/nsp/server/replication/bin/neo4j-admin backup --backup-dir=/tmp/restoreData --from=nsp-tomcat-headless-svc:6363 --database=graph.db ↵

4. Enter the following:

   # kubectl exec -it -n $(kubectl get pods -A | grep nsp-tomcat | tail -1 | awk '{print $1}') $(kubectl get pods -A | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $2}') -c nsp-tomcat -- /opt/nsp/server/replication/bin/neo4j-admin backup --backup-dir=/tmp/restoreData --from=nsp-tomcat-headless-svc:6363 --database=system ↵

If either command in Step 1 fails due to inconsistencies, perform How do I recover both failed nsp-tomcat databases in a DR NSP deployment using a database backup?.

If both commands in Step 1 succeed, perform a switchover.

1. Log in as the root or NSP admin user on the standby NSP cluster host.

2. Enter the following:

   # kubectl exec -it -n $(kubectl get pods -A | awk '/nsp-role-manager/ {print $1}') $(kubectl get pods -A | awk '/nsp-role-manager/ {print $2}') -c role-manager --  /opt/nsp/os/rolemgr/bin/rmgrctl toActive ↵

   When the switchover is complete, the standby cluster is the new primary cluster.

3. When the switchover is complete, sign in to the NSP as the admin user.

4. Verify that you can open Path Control.

   Note: You must not proceed to the next step until Path Control is available.

Enter the following to perform a manual backup on the new primary NSP cluster host:

# kubectl create job nsp-tomcat-backup-manual --from cronjob/nsp-tomcat-backup -n $(kubectl get cronjob -A | grep nsp-tomcat-backup | awk '{print $1}') ↵

If the backup fails due to inconsistencies, both database instances are inconsistent; perform How do I recover both failed nsp-tomcat databases in a DR NSP deployment using a database backup?.

Enter the following:

# kubectl exec nsp-backup-storage-0 -n $(kubectl get pods -A | grep nsp-backup-storage | awk '{print $1}') -- ls /tmp/backups/nsp-tomcat/ ↵

The backup files are listed.

Record the name of the most recent backup.

Enter the following:

# kubectl cp -n $(kubectl get pods -A | grep nsp-backup-storage | awk '{print $1}') nsp-backup-storage-0:/tmp/backups/nsp-tomcat/backup_file /tmp/backup_file ↵

where backup_file is the recorded backup file name

The backup file is copied to the temp directory on the NSP cluster host.

Enter the following:

# scp /tmp/backup_file address:/tmp ↵

where address is the address of the new standby NSP cluster host

The backup file is copied to the /tmp directory on the new standby NSP cluster host.

Restore database on new standby cluster

Log in as the root or NSP admin user on the new standby NSP cluster host.

Open a console window.

Enter the following:

# helm list -A | grep nsp-tomcat | awk -F' ' '{ print $10 }' ↵

The nsp-tomcat version is displayed.

Record the nsp-tomcat version.

Enter the following to uninstall nsp-tomcat:

# helm uninstall -n nsp-psa-restricted nsp-tomcat ↵

The nsp-tomcat uninstallation begins.

Enter the following command block to monitor the uninstallation:

while [ 0 -lt `kubectl get pods -n nsp-psa-restricted | grep nsp-tomcat | egrep -v backup | wc -l` ]

do

sleep 2

echo "Still there..."

done

The uninstallation is complete when the command prompt is displayed.

When the uninstallation is complete, enter the following to install nsp-tomcat:

# helm upgrade nsp-tomcat --install oci://registry.nsp.nokia.local/nsp/charts/cn-nsp-tomcat --namespace nsp-psa-restricted --version version --timeout 300s -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/values.yaml -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/restore.yaml ↵

where version is the nsp-tomcat version recorded in Step 13

nsp-tomcat is installed.

Enter the following:

# kubectl get pods -A ↵

The pods are listed.

If the nsp-tomcat pod is not running, repeat the command in Step 17.

Note: You must not proceed to the next step until nsp-tomcat is running.

Enter the following:

# /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/tools/database/nspos-

db-restore-k8s.sh nsp-tomcat /tmp/backup_file ↵

You are prompted to restore the database.

Respond to the prompt.

The nsp-tomcat database is restored from the backup file.

Enter the following to uninstall nsp-tomcat:

# helm uninstall -n nsp-psa-restricted nsp-tomcat ↵

The nsp-tomcat uninstallation begins.

Note: You must not proceed to the next step until the uninstallation is complete.

The nsp-tomcat uninstallation begins.

Enter the following command block to monitor the uninstallation:

while [ 0 -lt `kubectl get pods -n nsp-psa-restricted | grep nsp-tomcat | egrep -v backup | wc -l` ]

do

sleep 2

echo "Still there..."

done

The uninstallation is complete when the command prompt is displayed.

When the uninstallation is complete, enter the following:

When the uninstallation is complete, enter the following to install nsp-tomcat:

# helm upgrade nsp-tomcat --install oci://registry.nsp.nokia.local/nsp/charts/cn-nsp-tomcat --namespace nsp-psa-restricted --version version --timeout 300s -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/values.yaml ↵

where version is the nsp-tomcat version recorded in Step 13

nsp-tomcat is installed.

When the nsp-tomcat pod installation is complete, monitor the following log file in the nsp-tomcat pod to ensure that the database starts successfully:

/opt/nsp/server/replication/logs/debug.log

Back up database

Optionally, to confirm that the database restore is successful, perform a switchover and perform a backup to ensure that no inconsistencies remain.

Close the open console windows.

End of steps