Performing a software upgrade

You use the fss-upgrade.sh script to perform the software upgrade steps. The command syntax is as follows:
[root@fss-deployer ~]# /root/bin/fss-upgrade.sh
Usage: /root/bin/fss-upgrade.sh configure <config-file> - Reads config file and configures accordingly.
 ./bin/fss-upgrade.sh discover - Discovers cluster.
 ./bin/fss-upgrade.sh upgrade - Upgrades FSS application.
 ./bin/fss-upgrade.sh status - Prints status of cluster and FSS Chart information.
 ./bin/fss-upgrade.sh clean - Cleans temporary files.
 ./bin/fss-upgrade.sh [help | -h] - Prints usage.

This script updates the entire Fabric Services System application using the new Helm charts.

Note: The Fabric Services System 23.4 release significantly improves the infrastructure of several components. Consequently, the upgrade process requires that you update these component in several steps; the high-level steps are as follows:
  1. Using the new Deployer VM, upgrade the Fabric Services System application in place on the existing Kubernetes environment.
  2. Take a backup of the new release using the Deployer VM, as described in Backup and restore in the Fabric Services System User Guide.
  3. Shut down and remove the Fabric Services System virtual machine nodes.
  4. Use the latest base image to deploy new Fabric Services System virtual machine nodes, as described in Preparing for software upgrade.
  5. Restore the application from the back taken in Step 2, as described in Backup and restore in the Fabric Services System User Guide.

Ensure that you have completed the steps described in Preparing for software upgrade. Execute the following steps while logged in to the new deployer VM that you created in Preparing for software upgrade.
  1. Update the configuration.
    In the following example, the input configuration file is customer-input.json.
    [root@fss-deployer ~]# /root/bin/fss-upgrade.sh configure customer-input.json
        Timesync service is running on 10.254.106.74  Time difference is 0 seconds
        Timesync service is running on 10.254.106.75  Time difference is 0 seconds
        Timesync service is running on 10.254.106.76  Time difference is 0 seconds
        Timesync service is running on 10.254.106.77  Time difference is 0 seconds
        Timesync service is running on 10.254.106.78  Time difference is 0 seconds
        Timesync service is running on 10.254.106.79  Time difference is 0 seconds
      Maximum time difference between nodes 0 seconds
      Successfully configured. Please run /root/bin/fss-upgrade.sh discover
  2. Discover the cluster.
    [root@fss-deployer ~]# /root/bin/fss-upgrade.sh discover
    NAME                   STATUS   ROLES                  AGE   VERSION
    fss-node01             Ready    control-plane,master   72m   v1.23.1
    fss-node02             Ready    control-plane,master   71m   v1.23.1
    fss-node03             Ready    control-plane,master   71m   v1.23.1
    fss-node04             Ready    <none>                 70m   v1.23.1
    fss-node05             Ready    <none>                 70m   v1.23.1
    fss-node06             Ready    <none>                 70m   v1.23.1
    FSS release discovered fss-FSS_22_12_B1-charts-v22.12.2-11
    
  3. Upgrade the Fabric Services System application.
    [root@fss-deployer ~]# /root/bin/fss-upgrade.sh upgrade
    NAME                   STATUS   ROLES                  AGE   VERSION
    fss-node01             Ready    control-plane,master   75m   v1.23.1
    fss-node02             Ready    control-plane,master   74m   v1.23.1
    fss-node03             Ready    control-plane,master   74m   v1.23.1
    fss-node04             Ready    <none>                 73m   v1.23.1
    fss-node05             Ready    <none>                 73m   v1.23.1
    fss-node06             Ready    <none>                 73m   v1.23.1
    FSS will be upgraded from fss-FSS_22_12_B1-charts-v22.12.2-11 to fss-FSS_23_4_B1-charts v23.4.1-109 : Are you sure [YyNn]? y
    Upgrade in progress...
    Upgrading fss-logs
    Upgrading traefik and ingress routes
    traefik release discovered: traefik-10.14.2 ; Deployer packages traefik release:  traefik 21.0.0
    Upgrading kafka and kafkaop if required
    KAFKA not present.  Installing fresh
    Error from server (NotFound): secrets "kafka-fss-cluster-ca-cert" not found
    Error from server (NotFound): secrets "kafka-fss-clients-ca-cert" not found
    W0529 07:14:29.567723    7676 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
    W0529 07:14:29.575490    7676 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
    Release "prod" has been upgraded. Happy Helming!
    NAME: prod
    LAST DEPLOYED: Mon May 29 07:14:18 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    NOTES:
    
       Checking for FSS pods
           Waiting for  default  fss-auth   1   Running
          Waiting for FSS pods to comeup
           Waiting for  default  fss-auth   1   Running
          Waiting for FSS pods to comeup
           Waiting for  default  fss-cfgsync   1   Running
          Waiting for FSS pods to comeup
       All FSS pods are running
       Checking for FSS digitalsandbox pods
       FSS digital sandbox pods are running
       Checking for digitalsandbox pods
       Digital sandbox pods are running
    
       FSS is ready, you can access FSS using https://prod.k8s.nokia.com
    
  4. Verify that the upgrade is successful.
    1. Check the status of the compute nodes.
      The STATUS field should show Ready.
      [root@fss-deployer ~]# /root/bin/fss-upgrade.sh status
      NAME             STATUS   ROLES                  AGE   VERSION
      fss-node0#       Ready    control-plane,master   27m   v1.23.1
      fss-node02       Ready    control-plane,master   27m   v1.23.1
      fss-node03       Ready    control-plane,master   27m   v1.23.1
      fss-node04       Ready    <none>                 26m   v1.23.1
      fss-node05       Ready    <none>                 26m   v1.23.1 
      fss-node06       Ready    <none>                 26m   v1.23.1 
      FSS release discovered: fss-FSS_23_4_B1-charts-v23.4.1-yyy ; Deployer packages FSS release:  fss-FSS_23_4_B1-charts v23.4.1-yyy
         Checking for FSS pods
         All FSS pods are running
         Checking for FSS digitalsandbox pods
         FSS digital sandbox pods are running
         Checking for digitalsandbox pods
         Digital sandbox pods are running
       
         FSS is ready, you can access FSS using https://myhost.mydomain.com
    2. Display the Helm chart version.
      [root@fss-deployer ~]# export KUBECONFIG=/var/lib/fss/config.fss
      [root@fss-deployer ~]# helm list
      NAME    	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART                             	APP VERSION
      fss-logs	default  	1       	2023-05-25 19:49:37.968802894 +0000 UTC	deployed	fluent-bit-0.20.9                 	1.9.9
      kafka   	default  	1       	2023-05-25 20:00:16.161414151 +0000 UTC	deployed	fss-strimzi-kafka-0.1.8           	1.16.0
      prod    	default  	1       	2023-05-25 20:02:21.883150466 +0000 UTC	deployed	fss-FSS_23_4_B1-charts-v23.4.1-xxx	v23.4.1
    3. Check the status of system services and processes.
      [root@fss-deployer ~]# kubectl get pods -A
      NAMESPACE           NAME                                                       READY   STATUS      RESTARTS       AGE
      cert-manager               cert-manager-78759ccb7c-4pssc                              1/1     Running     0              12h
      cert-manager               cert-manager-cainjector-7d487f784-zkr55                    1/1     Running     0              12h
      cert-manager               cert-manager-webhook-6dbcf856cc-62l4w                      1/1     Running     0              12h
      default                    fss-logs-fluent-bit-6sn2k                                  1/1     Running     0              12h
      default                    fss-logs-fluent-bit-8h6wb                                  1/1     Running     0              12h
      default                    fss-logs-fluent-bit-9wmtr                                  1/1     Running     0              12h
      default                    fss-logs-fluent-bit-ltnc8                                  1/1     Running     0              12h
      default                    fss-logs-fluent-bit-rlzts                                  1/1     Running     0              12h
      default                    fss-logs-fluent-bit-w8fgk                                  1/1     Running     0              12h
      default                    prod-ds-apiserver-67cd5696dc-6vdhp                         1/1     Running     0              12h
      default                    prod-ds-cli-d98576cc8-pbzxm                                1/1     Running     0              12h
      default                    prod-ds-docker-registry-5b467bbf67-29bb2                   1/1     Running     0              12h
      default                    prod-ds-imgsvc-deploy-958b4bb6f-f9cgg                      1/1     Running     0              12h
      default                    prod-fss-alarmmgr-79669d987c-4m26p                         1/1     Running     0              12h
      default                    prod-fss-auth-68d7797465-dvbqb                             1/1     Running     0              12h
      default                    prod-fss-catalog-d586f956-tl8z7                            1/1     Running     0              12h
      default                    prod-fss-cfggen-7654fd9679-g9qn7                           1/1     Running     0              12h
      default                    prod-fss-cfgsync-5b7487bcfd-757n9                          1/1     Running     0              12h
      default                    prod-fss-connect-5786db898c-jlnnr                          1/1     Running     0              12h
      default                    prod-fss-da-0                                              1/1     Running     0              12h
      default                    prod-fss-da-1                                              1/1     Running     0              12h
      default                    prod-fss-da-2                                              1/1     Running     0              12h
      default                    prod-fss-da-3                                              1/1     Running     0              12h
      default                    prod-fss-da-4                                              1/1     Running     0              12h
      default                    prod-fss-da-5                                              1/1     Running     0              12h
      default                    prod-fss-da-6                                              1/1     Running     0              12h
      default                    prod-fss-da-7                                              1/1     Running     0              12h
      default                    prod-fss-deviationmgr-acl-5c5cfc76c-fwpz8                  1/1     Running     0              12h
      default                    prod-fss-deviationmgr-bfd-6d7b554f9f-xml4n                 1/1     Running     0              12h
      default                    prod-fss-deviationmgr-interface-6c79fcc578-4748w           1/1     Running     0              12h
      default                    prod-fss-deviationmgr-netinst-57d885f474-dzbsc             1/1     Running     0              12h
      default                    prod-fss-deviationmgr-platform-f8cb8c5f6-f5dw9             1/1     Running     0              12h
      default                    prod-fss-deviationmgr-qos-78986d7f56-77jbp                 1/1     Running     0              12h
      default                    prod-fss-deviationmgr-routingpolicy-568bc7b5c5-nt4b7       1/1     Running     0              12h
      default                    prod-fss-deviationmgr-system-86d9459cc4-q85n8              1/1     Running     0              12h
      default                    prod-fss-dhcp-5f4bd96ffb-475rv                             1/1     Running     0              12h
      default                    prod-fss-dhcp6-b65449855-mqf5v                             1/1     Running     0              12h
      default                    prod-fss-digitalsandbox-8fd7c5b55-2zf5v                    1/1     Running     0              12h
      default                    prod-fss-filemgr-75c6dfc75-sm77t                           1/1     Running     0              12h
      default                    prod-fss-imagemgr-69d997488c-ph9p2                         1/1     Running     0              12h
      default                    prod-fss-intentmgr-c644856c8-x82x5                         1/1     Running     0              12h
      default                    prod-fss-inventory-7ff597c8dc-66ql9                        1/1     Running     0              12h
      default                    prod-fss-labelmgr-85cd5d646c-cwvzk                         1/1     Running     0              12h
      default                    prod-fss-maintmgr-7874767b7c-rg5qv                         1/1     Running     0              12h
      default                    prod-fss-mgmtstack-659c7755b4-5mw68                        1/1     Running     0              12h
      default                    prod-fss-oper-da-0                                         1/1     Running     0              12h
      default                    prod-fss-oper-da-1                                         1/1     Running     0              12h
      default                    prod-fss-oper-da-2                                         1/1     Running     0              12h
      default                    prod-fss-oper-da-3                                         1/1     Running     0              12h
      default                    prod-fss-oper-da-4                                         1/1     Running     0              12h
      default                    prod-fss-oper-da-5                                         1/1     Running     0              12h
      default                    prod-fss-oper-da-6                                         1/1     Running     0              12h
      default                    prod-fss-oper-da-7                                         1/1     Running     0              12h
      default                    prod-fss-oper-topomgr-7b7f9fc7fd-dcqfr                     1/1     Running     0              12h
      default                    prod-fss-protocolmgr-5c877747bb-r5b7z                      1/1     Running     0              12h
      default                    prod-fss-topomgr-d8b5c899-zxcsl                            1/1     Running     0              12h
      default                    prod-fss-transaction-65f595d6df-ghfhx                      1/1     Running     0              12h
      default                    prod-fss-version-6d87bf6cc5-9fvbz                          1/1     Running     0              12h
      default                    prod-fss-web-7dcfdcbb5d-r85cp                              1/1     Running     0              12h
      default                    prod-fss-workloadmgr-55df864cc-d8spn                       1/1     Running     0              12h
      default                    prod-fss-ztp-75f44cc674-4p6lv                              1/1     Running     0              12h
      default                    prod-keycloak-0                                            1/1     Running     0              12h
      default                    prod-mongodb-arbiter-0                                     1/1     Running     0              12h
      default                    prod-mongodb-primary-0                                     1/1     Running     0              12h
      default                    prod-mongodb-secondary-0                                   1/1     Running     0              12h
      default                    prod-neo4j-core-0                                          1/1     Running     0              12h
      default                    prod-postgresql-0                                          1/1     Running     0              12h
      default                    prod-sftpserver-77cd8696d5-8j689                           1/1     Running     0              12h
      digital-sandbox-system     prod-netsegmd-5dvk6                                        1/1     Running     0              12h
      digital-sandbox-system     prod-netsegmd-72hzp                                        1/1     Running     1 (12h ago)    12h
      digital-sandbox-system     prod-netsegmd-c7bh4                                        1/1     Running     0              12h
      digital-sandbox-system     prod-netsegmd-gzztk                                        1/1     Running     1 (12h ago)    12h
      digital-sandbox-system     prod-netsegmd-hrmw8                                        1/1     Running     1 (12h ago)    12h
      digital-sandbox-system     prod-netsegmd-jr9zf                                        1/1     Running     0              12h
      digital-sandbox-system     srlfabrics-controller-manager-579ccbfb8b-v77gk             2/2     Running     0              12h
      kafka                      kafka-fss-entity-operator-b6757b664-kshcl                  3/3     Running     0              12h
      kafka                      kafka-fss-kafka-0                                          1/1     Running     0              12h
      kafka                      kafka-fss-kafka-1                                          1/1     Running     0              12h
      kafka                      kafka-fss-kafka-2                                          1/1     Running     0              12h
      kafka                      kafka-fss-zookeeper-0                                      1/1     Running     0              12h
      kafka                      kafka-fss-zookeeper-1                                      1/1     Running     0              12h
      kafka                      kafka-fss-zookeeper-2                                      1/1     Running     0              12h
      kafka                      strimzi-cluster-operator-5bc66cb4f9-w6nkd                  1/1     Running     0              12h
      kube-system                coredns-7947bbfb58-27pgt                                   1/1     Running     0              12h
      kube-system                coredns-7947bbfb58-snjff                                   1/1     Running     0              12h
      kube-system                dns-autoscaler-74d7b6c6c-l4f2h                             1/1     Running     0              12h
      kube-system                kube-apiserver-fss-node01                                  1/1     Running     1              12h
      kube-system                kube-apiserver-fss-node02                                  1/1     Running     1              12h
      kube-system                kube-controller-manager-fss-node03                         1/1     Running     1              12h
      kube-system                kube-controller-manager-fss-node04                         1/1     Running     1              12h
      kube-system                kube-flannel-g8tr2                                         1/1     Running     0              12h
      kube-system                kube-flannel-hvm7p                                         1/1     Running     0              12h
      kube-system                kube-flannel-n6fpp                                         1/1     Running     0              12h
      kube-system                kube-flannel-p86hz                                         1/1     Running     0              12h
      kube-system                kube-flannel-rjw8z                                         1/1     Running     0              12h
      kube-system                kube-flannel-rn8wr                                         1/1     Running     0              12h
      kube-system                kube-proxy-b646r                                           1/1     Running     0              12h
      kube-system                kube-proxy-dv6dx                                           1/1     Running     0              12h
      kube-system                kube-proxy-kzm7m                                           1/1     Running     0              12h
      kube-system                kube-proxy-qk5lp                                           1/1     Running     0              12h
      kube-system                kube-proxy-x74mh                                           1/1     Running     0              12h
      kube-system                kube-proxy-zh5hd                                           1/1     Running     0              12h
      kube-system                kube-scheduler-fss-node01                                  1/1     Running     1              12h
      kube-system                kube-scheduler-fss-node02                                  1/1     Running     1              12h
      kube-system                nginx-proxy-fss-node03                                     1/1     Running     0              12h
      kube-system                nginx-proxy-fss-node04                                     1/1     Running     0              12h
      kube-system                nginx-proxy-fss-node05                                     1/1     Running     0              12h
      kube-system                nginx-proxy-fss-node06                                     1/1     Running     0              12h
      kube-system                snapshot-controller-55dbff48d-28ptd                        1/1     Running     0              12h
      kube-system                snapshot-controller-55dbff48d-jzvtw                        1/1     Running     0              12h
      kube-system                traefik-2f8d9                                              1/1     Running     0              12h
      kube-system                traefik-2j7hr                                              1/1     Running     0              12h
      kube-system                traefik-dwf27                                              1/1     Running     0              12h
      kube-system                traefik-q545f                                              1/1     Running     0              12h
      kube-system                traefik-qgp74                                              1/1     Running     0              12h
      kube-system                traefik-vfj8m                                              1/1     Running     0              12h
      rook-ceph                  csi-cephfsplugin-6pfsm                                     3/3     Running     0              12h
      rook-ceph                  csi-cephfsplugin-95bm4                                     3/3     Running     0              12h
      rook-ceph                  csi-cephfsplugin-ddgw2                                     3/3     Running     0              12h
      rook-ceph                  csi-cephfsplugin-fc6sn                                     3/3     Running     0              12h
      rook-ceph                  csi-cephfsplugin-nj776                                     3/3     Running     0              12h
      rook-ceph                  csi-cephfsplugin-provisioner-59dd67b6c-c2b4f               6/6     Running     0              12h
      rook-ceph                  csi-cephfsplugin-provisioner-59dd67b6c-x2cdh               6/6     Running     0              12h
      rook-ceph                  csi-cephfsplugin-w5wdn                                     3/3     Running     0              12h
      rook-ceph                  rook-ceph-mds-ceph-filesystem-a-86bbf4ff-kdt24             1/1     Running     0              12h
      rook-ceph                  rook-ceph-mds-ceph-filesystem-b-77df4fbfcc-thc26           1/1     Running     0              12h
      rook-ceph                  rook-ceph-mgr-a-7554d5cdbc-qpq6w                           2/2     Running     0              12h
      rook-ceph                  rook-ceph-mgr-b-7f6ccbc99f-zxzj5                           2/2     Running     0              12h
      rook-ceph                  rook-ceph-mon-a-6666946d94-c86ck                           1/1     Running     0              12h
      rook-ceph                  rook-ceph-mon-b-744574b89f-hjvfw                           1/1     Running     0              12h
      rook-ceph                  rook-ceph-mon-c-7585f86975-jqqb5                           1/1     Running     0              12h
      rook-ceph                  rook-ceph-operator-77cf896f-7slvw                          1/1     Running     0              12h
      rook-ceph                  rook-ceph-osd-0-775f95d9f5-shrpl                           1/1     Running     0              12h
      rook-ceph                  rook-ceph-osd-1-59d94965b8-hg75j                           1/1     Running     0              12h
      rook-ceph                  rook-ceph-osd-2-84bb4966db-nv6mb                           1/1     Running     0              12h
      rook-ceph                  rook-ceph-osd-prepare-fss-node01                           0/1     Completed   0              12h
      rook-ceph                  rook-ceph-osd-prepare-fss-node02                           0/1     Completed   0              12h
      rook-ceph                  rook-ceph-osd-prepare-fss-node03                           0/1     Completed   0              12h
      rook-ceph                  rook-ceph-tools-c859b5678-hnpsb                            1/1     Running     0              12h
  5. Restore previously added software catalog images.
    The upgrade procedure replaces the software catalog. Changes that you made to the software catalog are lost after the upgrade. If you previously added new SR Linux images to the software catalog, you need to modify the software catalog and then upload the SR Linux images. For instructions, see Adding a network operating system version to the software catalog in the Fabric Services System User Guide. You must complete this step before you can access the Fabric Services System to make any change.
Upgrading the Fabric Services System cluster