How do I restore the NSP cluster databases?

Purpose

Perform this procedure to restore one or more of the following in each NSP cluster:

  • NSP Kubernetes secrets

  • NSP file service data

  • Neo4j database

  • PostgreSQL database

  • nspos-solr database

  • nsp-tomcat database

  • nrcx-tomcat database

  • OpenSearch database

Note: The Neo4j, nsp-tomcat, and nrcx-tomcat database backup files are each named graph.db. You must ensure that you are using the correct graph.db backup file when you restore the Neo4j, nsp-tomcat, or nrcx-tomcat database.

Note: If you are performing the procedure as part of a system conversion, migration, or upgrade procedure in a DR deployment, you must perform the procedure only in the new primary NSP cluster.

Note: You can specify a local backup file path, or a remote path, if the remote server is reachable from the NSP deployer VM and from the NSP cluster host.

To specify a remote path, use the following format for the backup_file parameter in the command, where user has access to backup_file at the server address:

user@server:/backup_file

Note: If root access for remote operations is disabled in the NSP configuration, remote operations such as SSH and SCP as the root user are not permitted within an NSP cluster. Steps that describe such an operation as the root user must be performed as the designated non-root user with sudoer privileges.

For simplicity, such steps describe only root-user access.

Note: release-ID in a file path has the following format:

R.r.p-rel.version

where

R.r.p is the NSP release, in the form MAJOR.minor.patch

version is a numeric value

Steps
Prepare to restore databases
 
1 

Open a terminal session to the NSP deployer VM.

2 

Log in as the root or NSP admin user.

3 

If you are restoring the data on new NSP cluster VMs, create and distribute an SSH key for password-free NSP deployer VM access to each NSP cluster VM.

  1. Enter the following:

    # ssh-keygen -N "" -f path -t rsa ↵

    where path is the SSH key file path, for example, /home/user/.ssh/id_rsa

    An SSH key is generated.

  2. Enter the following for each NSP cluster VM to distribute the key to the VM.

    # ssh-copy-id -i key_file user@address ↵

    where

    user is the designated NSP ansible user, if root-user access is restricted; otherwise, user@ is not required

    key_file is the SSH key file, for example, /home/user/.ssh/id_rsa.pub

    address is the NSP cluster VM IP address

4 

Perform one of the following.

Note: You must not proceed to the next step until the cluster is ready.

  1. If both of the following are true, you must stop each cluster and remove all existing cluster data:

    • You are restoring the data in an existing NSP cluster, rather than on new NSP cluster VMs.

    • You are restoring all NSP databases.

    Perform the following steps on the NSP deployer VM in each NSP cluster.

    Note: In a DR deployment, you must perform the steps first on the standby cluster.

    1. Log in as the root or NSP admin user.

    2. Open the following file with a plain-text editor such as vi:

      /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/config/nsp-config.yml

    3. Edit the following line in the platform section, kubernetes subsection to read as shown below:

        deleteOnUndeploy:true

    4. Save and close the file.

    5. Enter the following:

      # cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

    6. Enter the following:

      # ./nspdeployerctl uninstall --undeploy ↵

      The NSP cluster is undeployed, and the existing data is removed.

  2. If you are not restoring all databases, you must delete only the existing data in each database that you are restoring.

    Perform the following steps on the NSP deployer VM in each NSP cluster.

    Note: In a DR deployment, you must perform the steps first on the NSP cluster that you want to start as the standby cluster.

    1. Log in as the root or NSP admin user.

    2. Open the following file using a plain-text editor such as vi:

      /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/config/nsp-config.yml

    3. Edit the following line in the platform section, kubernetes subsection to read as shown below:

        deleteOnUndeploy:false

    4. Save and close the file.

    5. On the NSP cluster host, enter the following to determine which node has the database files are:

      # kubectl get pods -o wide -A | grep pod | awk '{print $8}' ↵

      where pod is the pod name

    6. Log in on the NSP cluster node where the pod is scheduled.

    7. On the node, enter the following for each database that you are restoring:

      # rm -rf /opt/nsp/volumes/local-provisioner/directory/* ↵

      where directory is the directory name that includes the PVC name associated with the pod. The directory contains the database for respective pod and is one of the following:

      • nspos-neo4j

      • nspos-postgresql

      • nspos-solr

      • nsp-tomcat

      • nrcx-tomcat

      • opensearch-cluster-master

      For example, if the pod name is nspos-solr-statefulset-0:

      • The corresponding pvc name is solr-pvc-nspos-solr-statefulset-0

      • The corresponding directory is pvc-af7c8e88-1a5a-47ac-ac9e-5457049f416a_nsp-psa-restricted_solr-pvc-nspos-solr-statefulset-0

      • The PV name is pvc-af7c8e88-1a5a-47ac-ac9e-5457049f416a

      To delete the nsp-file-service database:

      1. Log in to the nfs-server pod.

      2. Enter the following command from node 1 to log in to the nfs-server pod:

        # kubectl exec -it nfs_server_pod -n nsp-psa-privileged -- bash ↵

        where nfs_server_pod is the name of the nfs-server pod.

      3. Enter the following command to delete the nsp-file-service database:

        # rm -rf /exports/nsp-psa-restricted-nsp-file-service-app-data/fileservice/*

Enable NSP restore mode
 
5 

Enter the following on the NSP deployer VM:

# cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

6 

Enter the following to enter restore mode:

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the nspdeployerctl command, as shown in the example below, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl install --config --restore --ask-pass

# ./nspdeployerctl install --config --restore↵

7 

Note: The nspos-solr-statefulset-n and opensearch-cluster-master-n pods are only supported starting with Release 25.8. The pods are excluded from the restore operation if the system has been upgraded from a release earlier than Release 25.8.

The following NSP cluster pods must be operational before the restore begins:

  • nsp-backup-storage-n

  • nspos-neo4j-core-default-n

  • nspos-postgresql-primary-n

  • nsp-file-service-app-n

  • nspos-solr-statefulset-n

  • present only if the NSP deployment includes Path Control functions:

    • nsp-tomcat-dc_name-n

    • nrcx-tomcat-dc_name-n

    where dc_name is the dcName value in the cluster configuration file

  • present only if the NSP Deployment enables the loggingMonitoring feature:

    • opensearch-cluster-master-n

Enter the following periodically to list the pods; the cluster is ready for the restore when each required pod is in the Running state:

# kubectl get pods -A ↵

8 

If any required pod is not Running, return to Step 7.

Note: A restore attempt fails unless each required pod is Running.

Restore data
 
9 

Enter the following on the NSP deployer VM:

# cd /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/tools/database ↵

10 

Enter one or more of the following, as required, to restore system data and databases:

Note: In a DR deployment, you must perform the steps first in the data center that you want to start as the primary data center.

  1. To restore the NSP file service data:

    # ./nspos-db-restore-k8s.sh nsp-file-service backup_dir/backup_file ↵

  2. To restore the NSP Neo4j database:

    # ./nspos-db-restore-k8s.sh nspos-neo4j backup_dir/backup_file ↵

  3. To restore the NSP PostgreSQL database:

    # ./nspos-db-restore-k8s.sh nspos-postgresql backup_dir/backup_file ↵

  4. To restore the NSP Solr database:

    # ./nspos-db-restore-k8s.sh nspos-solr backup_dir/backup_file ↵

    Note: This step applies only starting with Release 25.8. The NSP Solr database is excluded from backup and restore operations if the system has been upgraded from a release earlier than Release 25.8.

  5. To restore the NSP Tomcat database:

    # ./nspos-db-restore-k8s.sh nsp-tomcat backup_dir/backup_file ↵

  6. To restore the cross-domain Tomcat database:

    # ./nspos-db-restore-k8s.sh nrcx-tomcat backup_dir/backup_file ↵

  7. To restore the OpenSearch database:

    # ./nspos-db-restore-k8s.sh opensearch backup_dir/backup_file ↵

Note: This step applies only in Release 25.8 or later. The OpenSearch database is excluded from backup and restore operations if the system has been upgraded from a release earlier than Release 25.8.

where

backup_dir is the directory that contains the backup file

backup_file is the backup file name, for example:

  • for PostgreSQL, the name is nspos-postgresql_backup_timestamp.tar.gz

  • for OpenSearch, the name is opensearch-cluster-master-n

Start NSP clusters
 
11 

Perform the following steps in each data center.

Note: In a DR deployment, you must perform the steps first in the data center that you want to start as the primary data center.

  1. Open a terminal session to the NSP deployer VM.

  2. Log in as the root or NSP admin user.

  3. Open the following file with a plain-text editor such as vi:

    /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/config/nsp-config.yml

  4. Edit the following line in the platform section, kubernetes subsection to read as shown below:

      deleteOnUndeploy:false

  5. Save and close the file.

  6. Enter the following:

    # cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

  7. On the NSP deployer VM, enter the following:

    # ./nspdeployerctl install --deploy ↵

    The NSP initializes using the restored data.

  8. Enter the following periodically on the NSP cluster host to display the cluster status:

    # kubectl get pods -A ↵

    The cluster is operational when the status of each pod is Running.

12 

Close the open console windows.

End of steps