How do I recreate the NSP Kubernetes secrets?

Purpose
CAUTION 

CAUTION

Potential Service Disruption

Recreating the NSP Kubernetes secrets requires a shutdown and restart of each NSP cluster, which is service-affecting.

Ensure that you perform the procedure only during a scheduled maintenance window under the guidance of technical support.

The following scenarios require the recreation of the Kubernetes secrets in an NSP cluster:

  • After a catastrophic NSP cluster failure

  • After a Kubernetes uninstallation, which removes the secrets, in the event that you need to reinstall Kubernetes

Perform the procedure on each NSP cluster that requires recreated secrets.

Steps
 
1 

Log in as the root or NSP admin user on the NSP deployer host.

2 

Stop the NSP cluster.

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the nspdeployerctl command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass uninstall –-undeploy

  1. Open the following file using a plain-text editor such as vi:

    /opt/nsp/NSP-CN-DEP-release-ID/NSP-CN-release-ID/config/nsp-config.yml

  2. Edit the following line in the platform section, kubernetes subsection to read as shown below:

      deleteOnUndeploy:false

  3. Save and close the file.

  4. Enter the following:

    # cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

  5. Enter the following:

    # ./nspdeployerctl uninstall –-undeploy ↵

    The NSP cluster stops.

3 

Enter the following to uninstall the current secrets:

# ./nspdeployerctl secret uninstall ↵

Messages like the following are displayed as each secret is uninstalled.

Removing secret secret...

  Removing from namespace nsp-psa-baseline

secret "secret" deleted

  Removing from namespace nsp-psa-privileged

secret "secret" deleted

  Removing from namespace nsp-psa-restricted

secret "secret" deleted

4 

Enter the following:

# ./nspdeployerctl secret install ↵

The following prompt is displayed:

Would you like to use your own CA key pair for the NSP Internal Issuer? [yes,no]

5 

Perform one of the following.

  1. Enter no ↵.

    The NSP generates the internal key and certificate files.

  2. Provide your own certificate to secure the internal network.

    1. Enter yes ↵.

      The following messages and prompt are displayed:

    2. Building secret 'ca-key-pair-internal-nspdeployer'

      The CA key pair used to sign certificates generated by the NSP Internal Issuer.

      Please enter the internal CA private key:

    3. Enter the full path of the internal private key.

      The following prompt is displayed:

      Please enter the internal CA certificate:

    4. Enter the full path of the internal certificate:

      The following messages are displayed for each Kubernetes namespace:

      Adding secret ca-key-pair-internal-nspdeployer to namespace namespace...

      secret/ca-key-pair-internal-nspdeployer created

The following prompt is displayed:

Would you like to use your own CA key pair for the NSP External Issuer? [yes,no]

6 

Perform one of the following.

  1. Enter no ↵.

    The NSP generates the external key and certificate files.

  2. Provide your own certificate to secure the external network.

    1. Enter yes ↵.

      The following messages and prompt are displayed:

      Building secret 'ca-key-pair-external-nspdeployer'

      The CA key pair used to sign certificates generated by the NSP External Issuer.

      Please enter the external CA private key:

    2. Enter the full path of the external private key.

      The following prompt is displayed:

      Please enter the external CA certificate:

    3. Enter the full path of the external certificate:

      The following messages are displayed for each Kubernetes namespace:

      Adding secret ca-key-pair-external-nspdeployer to namespace namespace...

      secret/ca-key-pair-external-nspdeployer created

Would you like to provide a custom private key and certificate for use by NSP endpoints when securing TLS connections over the client network? [yes,no]

7 

Perform one of the following.

  1. Enter no ↵.

    The NSP generates the client key and certificate files.

  2. Provide your own certificate for the client network.

    1. Enter yes ↵

      The following messages and prompt are displayed:

      Building secret 'nginx-nb-tls-nsp'

      TLS certificate for securing the ingress gateway.

      Please enter the ingress gateway private key:

    2. Enter the full path of the private key file for client access.

      The following prompt is displayed:

      Please enter the ingress gateway public certificate:

    3. Enter the full path of the public certificate file for client access.

      The following prompt is displayed:

      Please enter the ingress gateway public trusted CA certificate bundle:

    4. Enter the full path of the public trusted CA certificate bundle file.

      The following message is displayed:

        Adding secret nginx-nb-tls-nsp to namespace namespace...

8 

If the deployment includes MDM, the following prompt is displayed:

Would you like to provide mTLS certificates for the NSP mediation interface for two-way TLS authentication? [yes,no]

Perform one of the following.

  1. Enter no ↵ if you are not using mTLS or have no certificate to provide for mTLS.

  2. Provide your own certificate to secure MDM and gNMI telemetry.

    1. Enter yes ↵.

    2. The following messages and prompt are displayed:

      Building secret 'mediation-mtls-key'

      mTLS artifacts use to secure MDM communications with nodes.

      Please enter the mediation private key:

    3. Enter the full path of the mediation private key.

      The following prompt is displayed:

      Please enter the mediation CA certificate:

    4. Enter the full path of the mediation CA certificate.

      The following messages are displayed:

        Adding secret mediation-mtls-key to namespace namespace...

      secret/mediation-mtls-key created

        Adding secret mediation-mtls-key to namespace namespace...

      secret/mediation-mtls-key created

9 

Back up the secrets.

  1. Enter the following:

    # ./nspdeployerctl secret -o backup_file backup ↵

    where backup_file is the full path and name of the backup file to create

    As the secrets are backed up, messages like the following are displayed for each Kubernetes namespace:

    Backing up secrets to /opt/backupfile...

      Including secret namespace:ca-key-pair-external

      Including secret namespace:ca-key-pair-internal

      Including secret namespace:nsp-tls-store-pass

    When the backup is complete, the following prompt is displayed:

    Please provide an encryption password for backup_file

    enter aes-256-ctr encryption password:

  2. Enter a password.

    The following prompt is displayed:

    Verifying - enter aes-256-ctr encryption password:

  3. Re-enter the password.

    The backup file is encrypted using the password.

  4. Record the password for use when restoring the backup.

  5. Record the name of the data center associated with the backup.

  6. Transfer the backup file to a secure location in a separate facility for safekeeping.

10 

If the NSP is a DR deployment, obtain and restore the NSP secrets backup file from the NSP cluster in the primary data center.

  1. Enter the following on the standby NSP deployer host:

    # scp address:path/backup_file /tmp/ ↵

    where

    address is the address of the NSP deployer host in the primary cluster

    path is the full path of the backup file created in Step 9

    backup_file is the secrets backup file name

    The backup file is transferred to the local /tmp directory.

  2. Enter the following:

    # cd /opt/nsp/NSP-CN-DEP-release-ID/bin ↵

  3. Enter the following:

    ./nspdeployerctl secret -i /tmp/backup_file restore ↵

    The following prompt is displayed:

    Please provide the encryption password for /opt/backupfile

    enter aes-256-ctr decryption password:

  4. Enter the password recorded in Step 9.

    As the secrets are restored, messages like the following are displayed for each Kubernetes namespace:

    Restoring secrets from backup_file...

    secret/ca-key-pair-external created

      Restored secret namespace:ca-key-pair-external

    secret/ca-key-pair-internal created

      Restored secret namespace:ca-key-pair-internal

    secret/nsp-tls-store-pass created

      Restored secret namespace:nsp-tls-store-pass

  5. If you answer yes to the Step 7 prompt for client access during the primary NSP cluster configuration, you must update the standby server secret for client access using the custom certificate and key files that are specific to the standby cluster.

    Enter the following:

    # ./nspdeployerctl secret -s nginx-nb-tls-nsp -n psaRestricted -f tls.key=customKey -f tls.crt=customCert -f ca.crt=customCaCert update ↵

    where

    customKey is the full path of the private server key file

    customCert is the full path of the server public certificate file

    customCaCert is the full path of the CA public certificate file

    Messages like the following are displayed as the server secret is updated:

    secret/nginx-nb-tls-nsp patched

    The following files may contain sensitive information. They are no longer required by NSP and may be removed.

      customKey

      customCert

      customCaCert

11 

Enter the following to start the NSP cluster:

Note: If the NSP cluster VMs do not have the required SSH key, you must include the --ask-pass argument in the nspdeployerctl command, as shown in the following example, and are subsequently prompted for the root password of each cluster member:

nspdeployerctl --ask-pass install --config –-deploy

# ./nspdeployerctl install --config –-deploy ↵

The NSP cluster starts, and the configuration update is put into effect.

12 

Close the console window.

End of steps