Disable the auto-switchover function to ensure that no switchover occurs during the database restoration.

Enter the following as the root or NSP admin user on the NSP cluster host in each data center:

# kubectl exec -it $(kubectl get pods | awk '/nspos-asm/ {print $1;exit}') -c nspos-asm-app -- /opt/nsp/os/asm/bin/asmctl disableAutoFailover ↵

Enter the following as the root or NSP admin user on the NSP cluster host in either data center:

# helm list | grep nsp-tomcat | awk -F' ' '{ print $10 }' ↵

The nsp-tomcat version is displayed.

Record the nsp-tomcat version.

Enter the following on the NSP cluster host in each data center to uninstall nsp-tomcat:

Note: You must perform the step on the standby NSP cluster host first.

# helm uninstall -n nsp-psa-restricted nsp-tomcat ↵

The nsp-tomcat uninstallation begins.

Enter the following command block to monitor the uninstallation:

while [ 0 -lt `kubectl get pvc --all-namespaces | grep nsp-tomcat | wc -l` ] || [ 0 -lt `kubectl get pv --all-namespaces | grep nsp-tomcat | wc -l` ]

do

sleep 2

echo "Still there..."

done

The uninstallation is complete when the command prompt is displayed.

When the uninstallation is complete, enter the following on the NSP cluster host in each data center to install nsp-tomcat:

Note: You must perform the step on the primary NSP cluster host first.

# helm upgrade nsp-tomcat --install oci://registry.nsp.nokia.local/nsp/charts/cn-nsp-tomcat --namespace nsp-psa-restricted --version version --timeout 300s -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/values.yaml -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/restore.yaml ↵

where version is the nsp-tomcat version recorded in Step 3

nsp-tomcat is installed.

Enter the following sequence of commands to back up the nsp-tomcat database in the primary data center:

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- /opt/nsp/server/replication/bin/neo4j-admin copy --from path=/opt/nsp/server/tomcat/work/data/databases/graph.db --to-database=fixed.db --from-path-tx=/opt/nsp/server/tomcat/work/data/transactions/graph.db ↵

# mkdir -p /opt/fixed-db-backup/database ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- mkdir /tmp/restoreData/database ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- bash -c "cd /opt/nsp/server/tomcat/work/data/databases/; tar -czvf /tmp/restoreData/database/fixed.db.tar.gz ./fixed.db" ↵

# kubectl cp $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}'):/tmp/restoreData/database/fixed.db.tar.gz /opt/fixed-db-backup/database/fixed.db.tar.gz ↵

# cd /opt/fixed-db-backup/database/ ↵

# tar -xvf ./fixed.db.tar.gz ↵

# mv ./fixed.db ./graph.db ↵

# mkdir -p /opt/fixed-db-backup/transactions/ ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- mkdir /tmp/restoreData/transactions ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- bash -c "cd /opt/nsp/server/tomcat/work/data/transactions/; tar -czvf /tmp/restoreData/transactions/fixed.db.tar.gz ./fixed.db" ↵

# kubectl cp $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}'):/tmp/restoreData/transactions/fixed.db.tar.gz /opt/fixed-db-backup/transactions/fixed.db.tar.gz ↵

# cd /opt/fixed-db-backup/transactions/ ↵

# tar -xvf ./fixed.db.tar.gz ↵

# mv ./fixed.db ./graph.db ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- rm -rfd /tmp/restoreData/database ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- rm -rfd /tmp/restoreData/transactions ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- rm -rfd /opt/nsp/server/tomcat/work/data/databases/fixed.db ↵

# kubectl exec $(kubectl get pods | grep nsp-tomcat | egrep -v "haproxy|backup" | tail -1 | awk '{print $1}') -- rm -rfd /opt/nsp/server/tomcat/work/data/transactions/fixed.db ↵

Enter the following sequence of commands on the NSP cluster host in the primary data center for each deployed nsp-tomcat pod:

# cd /opt/fixed-db-backup/database ↵

# kubectl exec nsp-tomcat-DC_name-pod_ID -- mkdir /tmp/restoreData/database ↵

# kubectl cp ./graph.db nsp-tomcat-DC_name-pod_ID:/tmp/restoreData/database/graph.db ↵

# kubectl exec nsp-tomcat-DC_name-pod_ID -- rm -rf /opt/nsp/server/tomcat/work/data/databases/graph.db ↵

# kubectl exec nsp-tomcat-DC_name-pod_ID -- /opt/nsp/server/replication/bin/neo4j-admin restore --force --database=graph.db --from=/tmp/restoreData/database/graph.db ↵

# cd /opt/fixed-db-backup/transactions ↵

# kubectl exec nsp-tomcat-DC_name-pod_ID -- rm -rfd /opt/nsp/server/tomcat/work/data/transactions/graph.db ↵

# kubectl cp ./graph.db nsp-tomcat-DC_name-pod_ID:/opt/nsp/server/tomcat/work/data/transactions/graph.db ↵

# kubectl exec nsp-tomcat-DC_name-pod_ID -- rm -rfd /tmp/restoreData/database ↵

DC_name is the primary data center name

pod_ID is the nsp-tomcat pod ID

Enter the following on the primary NSP cluster host to uninstall nsp-tomcat:

# helm uninstall -n nsp-psa-restricted nsp-tomcat ↵

The nsp-tomcat uninstallation begins.

Note: You must not proceed to the next step until the uninstallation is complete.

Enter the following command block to monitor the uninstallation:

while [ 0 -lt `kubectl get pvc --all-namespaces | grep nsp-tomcat | wc -l` ] || [ 0 -lt `kubectl get pv --all-namespaces | grep nsp-tomcat | wc -l` ]

do

sleep 2

echo "Still there..."

done

The uninstallation is complete when the command prompt is displayed.

When the uninstallation is complete, enter the following on the NSP cluster host in the primary data center to install nsp-tomcat:

# helm upgrade nsp-tomcat --install oci://registry.nsp.nokia.local/nsp/charts/cn-nsp-tomcat --namespace nsp-psa-restricted --version version --timeout 300s -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/values.yaml ↵

where version is the nsp-tomcat version recorded in Step 3

nsp-tomcat is installed.

Enter the following:

# kubectl get pods -A ↵

The pods are listed.

If the nsp-tomcat pod is not running, repeat the command in Step 12.

Note: You must not proceed to the next step until nsp-tomcat is running.

Sign in to the NSP as the admin user.

Verify that you can open Path Control.

You must not proceed to the next step until Path Control is available.

Enter the following on the standby NSP cluster host for each nsp-tomcat pod:

# kubectl exec nsp-tomcat-DC_name-pod_ID -- rm -rf /opt/nsp/server/tomcat/work/data/* ↵

DC_name is the primary data center name

pod_ID is the nsp-tomcat pod ID

Enter the following on the standby NSP cluster host to uninstall nsp-tomcat:

# helm uninstall -n nsp-psa-restricted nsp-tomcat ↵

The nsp-tomcat uninstallation begins.

Note: You must not proceed to the next step until the uninstallation is complete.

Enter the following command block to monitor the uninstallation:

while [ 0 -lt `kubectl get pvc --all-namespaces | grep nsp-tomcat | wc -l` ] || [ 0 -lt `kubectl get pv --all-namespaces | grep nsp-tomcat | wc -l` ]

do

sleep 2

echo "Still there..."

done

The uninstallation is complete when the command prompt is displayed.

When the uninstallation is complete, enter the following on the NSP cluster host in the standby data center to install nsp-tomcat:

# helm upgrade nsp-tomcat --install oci://registry.nsp.nokia.local/nsp/charts/cn-nsp-tomcat --namespace nsp-psa-restricted --version version --timeout 300s -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/values.yaml -f /opt/nsp/config/helm/values/sdn/nsp-tomcat/restore.yaml ↵

where version is the nsp-tomcat version recorded in Step 3

nsp-tomcat is installed.

Enable the auto-switchover function; enter the following as the root or NSP admin user on the NSP cluster host in each data center:

# kubectl exec -it $(kubectl get pods | awk '/nspos-asm/ {print $1;exit}') -c nspos-asm-app -- /opt/nsp/os/asm/bin/asmctl enableAutoFailover ↵

Optionally, when both NSP clusters are fully operational, back up the nsp-tomcat database; enter the following on the primary NSP cluster host:

kubectl create job nsp-tomcat-backup-manual --from cronjob/nsp-tomcat-backup ↵

The nsp-tomcat database is backed up.

Close the open console windows.

End of steps