Performing a software upgrade

Use the fss-upgrade.sh script to perform the software upgrade steps. The command syntax is as follows:
[root@fss-deployer ~]# /root/bin/fss-upgrade.sh
Usage: /root/bin/fss-upgrade.sh configure <config-file> - Reads config file and configures accordingly.
 ./bin/fss-upgrade.sh discover - Discovers cluster.
 ./bin/fss-upgrade.sh upgrade - Upgrades FSS application.
 ./bin/fss-upgrade.sh status - Prints status of cluster and FSS Chart information.
 ./bin/fss-upgrade.sh clean - Cleans temporary files.
 ./bin/fss-upgrade.sh [help | -h] - Prints usage.

This script updates the entire Fabric Services System application using the new Helm charts.

Using the new deployer VM, upgrade the Fabric Services System application in place on the existing Fabric Services System cluster.
Ensure that you have completed the steps described in Preparing for software upgrade. Execute the following steps while logged in to the new deployer VM that you created in Preparing for software upgrade.
  1. Update the configuration.
    In the following example, the input configuration file is customer-input.json.
    [root@fss-deployer ~]# /root/bin/fss-upgrade.sh configure customer-input.json
        Timesync service is running on 192.0.2.74  Time difference is 0 seconds
        Timesync service is running on 192.0.2.75  Time difference is 0 seconds
        Timesync service is running on 192.0.2.76  Time difference is 0 seconds
        Timesync service is running on 192.0.2.77  Time difference is 0 seconds
        Timesync service is running on 192.0.2.78  Time difference is 0 seconds
        Timesync service is running on 192.0.2.79  Time difference is 0 seconds
      Maximum time difference between nodes 0 seconds
      Successfully configured. Please run /root/bin/fss-upgrade.sh discover
  2. Discover the cluster.
    [root@fss-deployer ~]# /root/bin/fss-upgrade.sh discover
    Trying to discover fss-node01 192.0.2.74
    Trying to discover fss-node02 192.0.2.75
    Trying to discover fss-node03 192.0.2.76
    NAME STATUS ROLES AGE VERSION
    fss-node01 Ready control-plane,master 14h v1.23.1
    fss-node02 Ready control-plane,master 14h v1.23.1
    fss-node03 Ready control-plane,master 14h v1.23.1
    fss-node04 Ready <none> 14h v1.23.1
    fss-node05 Ready <none> 14h v1.23.1
    fss-node06 Ready <none> 14h v1.23.1
    FSS release discovered fss-FSS_23_8_B1-charts-v23.8.4-35
  3. Upgrade the Fabric Services System application.
    [root@fss-deployer ~]# /root/bin/fss-upgrade.sh upgrade
    NAME                   STATUS   ROLES                  AGE   VERSION
    fss-node01             Ready    control-plane,master   75m   v1.23.1
    fss-node02             Ready    control-plane,master   74m   v1.23.1
    fss-node03             Ready    control-plane,master   74m   v1.23.1
    fss-node04             Ready    <none>                 73m   v1.23.1
    fss-node05             Ready    <none>                 73m   v1.23.1
    fss-node06             Ready    <none>                 73m   v1.23.1
    FSS will be upgraded from fss-FSS_23_8_B1-charts-v23.8.4-12 to fss-FSS_24_5_B1-charts-v24.5.1-269 : Are you sure [YyNn]? y
    
    Upgrade in progress...
    Upgrading fss-logs
    fss-logs release discovered: fluent-bit-0.20.9 ; Deployer packages fss-logs release: fluent-bit 0.39.0
    Upgrading cert-manager
    cert-manager release discovered: cert-manager-v1.12.0 ; Deployer packages cert-manager release: cert-manager v1.13.3
    cert-manager upgrade started
    Upgrading metallb
    metallb release discovered: metallb-0.13.7 ; Deployer packages metallb release: metallb 0.13.7
    metallb upgrade not required
    Upgrading traefik and ingress routes
    traefik release discovered: traefik-21.0.0 ; Deployer packages traefik release: traefik 26.0.0
    Upgrading kafka and kafkaop if required
    kafkaop release discovered: strimzi-kafka-operator-0.31.0 ; Deployer packages kafkaop release: strimzi-kafka-operator 0.37.0
    NEW KAFKA operator, upgrade chart and update kafka
    Waiting for fss services to stop
    KAFKA will be deleted and reinstalled.
    Upgrading / Installing new FSS Chart
    Release "prod" has been upgraded. Happy Helming!
    NAME: prod
    LAST DEPLOYED: Mon Mar 18 06:42:09 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 3
    NOTES:
    KAFKA being reinstalled.
    Checking for FSS pods
    Waiting for default fss-catalog 1 Running
    Waiting for FSS pods to comeup
    Waiting for default fss-version 1 Running
    Waiting for FSS pods to comeup
    All FSS pods are running
    Checking for FSS digitalsandbox pods
    FSS digital sandbox pods are running
    Checking for digitalsandbox pods
    Digital sandbox pods are running
    
    FSS is started, it can take upto 10 mins to be ready for use
    GUI can be accessed at https://192.0.2.74
  4. Verify that the upgrade is successful.
    1. Check the status of the compute nodes.
      The STATUS field should show Ready.
      [root@fss-deployer ~]# /root/bin/fss-upgrade.sh status
      NAME             STATUS   ROLES                  AGE   VERSION
      fss-node01       Ready    control-plane,master   27m   v1.23.1
      fss-node02       Ready    control-plane,master   27m   v1.23.1
      fss-node03       Ready    control-plane,master   27m   v1.23.1
      fss-node04       Ready    <none>                 26m   v1.23.1
      fss-node05       Ready    <none>                 26m   v1.23.1 
      fss-node06       Ready    <none>                 26m   v1.23.1 
      Installed FSS release: fss-FSS_24_5_B1-charts-v24.5.1-XXX ; Available FSS release: fss-FSS_24_5_B1-charts-v24.5.1-XXX
      Checking for FSS pods
      All FSS pods are running
      Checking for FSS digitalsandbox pods
      FSS digital sandbox pods are running
      Checking for digitalsandbox pods
      Digital sandbox pods are running
      
      FSS is started, it can take upto 10 mins to be ready for use
      GUI can be accessed at https://10.254.107.75
    2. Display the Helm chart version.
      [root@fss-deployer ~]# export KUBECONFIG=/var/lib/fss/config.fss
      [root@fss-deployer ~]# helm listNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
      fss-logs default 2 2024-03-18 06:35:35.469592459 +0000 UTC deployed fluent-bit-0.39.0 2.1.10 
      kafka default 1 2024-03-18 06:42:22.087340711 +0000 UTC deployed fss-strimzi-kafka-0.1.11 3.5.1 
      prod default 3 2024-03-18 06:42:09.940429218 +0000 UTC deployed fss-FSS_24_5_B1-charts-v24.5.1-269 v24.5.1 	
    3. Check the status of system services and processes.
      [root@fss-deployer ~]# kubectl get pods -ANAMESPACE NAME READY STATUS RESTARTS AGE
      cert-manager cert-manager-7bff5cd474-hs5sf 1/1 Running 0 19m
      cert-manager cert-manager-cainjector-c79788c78-r7tsg 1/1 Running 0 19m
      cert-manager cert-manager-webhook-65f55dfd68-jv92l 1/1 Running 0 19m
      default fss-logs-fluent-bit-485hn 1/1 Running 0 19m
      default fss-logs-fluent-bit-8mw8p 1/1 Running 0 19m
      default fss-logs-fluent-bit-8zvmb 1/1 Running 0 18m
      default fss-logs-fluent-bit-hkxql 1/1 Running 0 18m
      default fss-logs-fluent-bit-jp87k 1/1 Running 0 17m
      default fss-logs-fluent-bit-mctvg 1/1 Running 0 18m
      default prod-ds-apiserver-7db4cb486c-jjd72 1/1 Running 0 12m
      default prod-ds-cli-6f9d474fd5-7vl2b 1/1 Running 0 12m
      default prod-ds-docker-registry-5b467bbf67-w4cps 1/1 Running 0 11h
      default prod-ds-imgsvc-deploy-88b6f5d74-c7kxz 1/1 Running 0 12m
      default prod-fss-alarmmgr-75fbfcff7-jhvh7 1/1 Running 0 12m
      default prod-fss-alertmgr-webhook-2spqg 1/1 Running 0 12m
      default prod-fss-alertmgr-webhook-7m8xc 1/1 Running 0 11m
      default prod-fss-alertmgr-webhook-cbckz 1/1 Running 0 12m
      default prod-fss-alertmgr-webhook-gcxsp 1/1 Running 0 12m
      default prod-fss-alertmgr-webhook-l7j6d 1/1 Running 0 12m
      default prod-fss-alertmgr-webhook-r76d5 1/1 Running 0 12m
      default prod-fss-auth-587bcdc997-sfg9w 1/1 Running 0 12m
      default prod-fss-catalog-779bc9b857-t6jdf 1/1 Running 0 12m
      default prod-fss-cfggen-5d74d7444c-7st2r 1/1 Running 0 12m
      default prod-fss-cfgsync-7dc4fdf6bc-psxx7 1/1 Running 0 11m
      default prod-fss-connect-6cf889b4fc-c2bg8 1/1 Running 0 12m
      default prod-fss-da-0 1/1 Running 0 12m
      default prod-fss-da-1 1/1 Running 0 12m
      default prod-fss-da-2 1/1 Running 0 12m
      default prod-fss-da-3 1/1 Running 0 12m
      default prod-fss-da-4 1/1 Running 0 12m
      default prod-fss-da-5 1/1 Running 0 12m
      default prod-fss-da-6 1/1 Running 0 12m
      default prod-fss-da-7 1/1 Running 0 12m
      default prod-fss-deviationmgr-acl-c4f9fd588-nh2d4 1/1 Running 0 11m
      default prod-fss-deviationmgr-bfd-778dd6f456-fxgn2 1/1 Running 0 11m
      default prod-fss-deviationmgr-interface-8db5b55f5-cqgqz 1/1 Running 0 11m
      default prod-fss-deviationmgr-netinst-56fb5b7887-rgjzg 1/1 Running 0 11m
      default prod-fss-deviationmgr-platform-7f6d4f47d9-464jd 1/1 Running 0 11m
      default prod-fss-deviationmgr-qos-75bd65f489-4n5nw 1/1 Running 0 12m
      default prod-fss-deviationmgr-routingpolicy-5d74cf65cb-bhb5r 1/1 Running 0 11m
      default prod-fss-deviationmgr-system-8f8887c67-glrzg 1/1 Running 0 11m
      default prod-fss-dhcp-5d66876465-z8xf4 1/1 Running 0 12m
      default prod-fss-dhcp6-65779bf496-d69qz 1/1 Running 0 12m
      default prod-fss-digitalsandbox-68576766d5-tvvt5 1/1 Running 0 11m
      default prod-fss-filemgr-74965868d9-mbfl5 1/1 Running 0 11m
      default prod-fss-imagemgr-55445fbbb4-pdbm2 1/1 Running 0 11m
      default prod-fss-intentmgr-565fc64855-hn8kh 1/1 Running 0 11m
      default prod-fss-inventory-6bf79df9d5-z2jcj 1/1 Running 0 11m
      default prod-fss-labelmgr-998d95bd-sln88 1/1 Running 0 11m
      default prod-fss-maintmgr-5f4b4ff745-ndhd2 1/1 Running 0 11m
      default prod-fss-mgmtstack-f959df45d-2nsj8 1/1 Running 0 11m
      default prod-fss-oper-da-0 1/1 Running 0 12m
      default prod-fss-oper-da-1 1/1 Running 0 12m
      default prod-fss-oper-da-2 1/1 Running 0 12m
      default prod-fss-oper-da-3 1/1 Running 0 12m
      default prod-fss-oper-da-4 1/1 Running 0 12m
      default prod-fss-oper-da-5 1/1 Running 0 12m
      default prod-fss-oper-da-6 1/1 Running 0 12m
      default prod-fss-oper-da-7 1/1 Running 0 12m
      default prod-fss-oper-topomgr-56cbcb7586-xp8fx 1/1 Running 0 11m
      default prod-fss-protocolmgr-75f866f5d-tjtdv 1/1 Running 0 11m
      default prod-fss-syncmgr-f5784f8df-q9f2l 1/1 Running 0 12m
      default prod-fss-topomgr-75d4656f45-wlnn2 1/1 Running 0 11m
      default prod-fss-transaction-55ccfbd46f-9nvwv 1/1 Running 0 11m
      default prod-fss-version-7dd5c7877-wrq5z 1/1 Running 0 6m53s
      default prod-fss-web-6674c4d54c-4hgvb 1/1 Running 0 11m
      default prod-fss-workloadmgr-7bc56d8556-dx4nw 1/1 Running 0 11m
      default prod-fss-ztp-6687c57b75-695zd 1/1 Running 0 11m
      default prod-keycloak-0 1/1 Running 0 12m
      default prod-mongodb-arbiter-0 1/1 Running 0 12m
      default prod-mongodb-primary-0 1/1 Running 0 12m
      default prod-mongodb-secondary-0 1/1 Running 0 12m
      default prod-neo4j-core-0 1/1 Running 0 11h
      default prod-postgresql-0 1/1 Running 0 12m
      default prod-sftpserver-77cd8696d5-hxwwg 1/1 Running 0 11h
      digital-sandbox-system prod-netsegmd-2fdk7 1/1 Running 0 12m
      digital-sandbox-system prod-netsegmd-cj7xz 1/1 Running 0 12m
      digital-sandbox-system prod-netsegmd-f64wm 1/1 Running 0 11m
      digital-sandbox-system prod-netsegmd-gdc9t 1/1 Running 0 9m1s
      digital-sandbox-system prod-netsegmd-pl7mn 1/1 Running 0 12m
      digital-sandbox-system prod-netsegmd-wwwqs 1/1 Running 0 12m
      digital-sandbox-system srlfabrics-controller-manager-5dd4fc9555-br5wl 2/2 Running 0 12m
      kafka kafka-fss-entity-operator-8fc655656-lnkvp 3/3 Running 0 8m31s
      kafka kafka-fss-kafka-0 1/1 Running 0 10m
      kafka kafka-fss-kafka-1 1/1 Running 0 10m
      kafka kafka-fss-kafka-2 1/1 Running 0 10m
      kafka kafka-fss-zookeeper-0 1/1 Running 0 12m
      kafka kafka-fss-zookeeper-1 1/1 Running 0 12m
      kafka kafka-fss-zookeeper-2 1/1 Running 0 12m
      kafka strimzi-cluster-operator-5dbd754b4d-57dhs 1/1 Running 0 19m
      kube-system coredns-7947bbfb58-nk9xn 1/1 Running 0 2d14h
      kube-system coredns-7947bbfb58-sbdlb 1/1 Running 0 2d14h
      kube-system dns-autoscaler-74d7b6c6c-cw4cw 1/1 Running 0 2d14h
      kube-system kube-apiserver-blrsrlfsphw01-compute1 1/1 Running 1 2d14h
      kube-system kube-apiserver-blrsrlfsphw01-compute2 1/1 Running 1 2d14h
      kube-system kube-apiserver-blrsrlfsphw01-compute3 1/1 Running 1 2d14h
      kube-system kube-controller-manager-blrsrlfsphw01-compute1 1/1 Running 1 2d14h
      kube-system kube-controller-manager-blrsrlfsphw01-compute2 1/1 Running 1 2d14h
      kube-system kube-controller-manager-blrsrlfsphw01-compute3 1/1 Running 1 2d14h
      kube-system kube-flannel-b6s2c 1/1 Running 0 2d14h
      kube-system kube-flannel-bktxx 1/1 Running 0 2d14h
      kube-system kube-flannel-d7n6h 1/1 Running 0 2d14h
      kube-system kube-flannel-gbvdn 1/1 Running 0 2d14h
      kube-system kube-flannel-hmb56 1/1 Running 0 2d14h
      kube-system kube-flannel-wcsmg 1/1 Running 0 2d14h
      kube-system kube-proxy-6czd6 1/1 Running 0 2d14h
      kube-system kube-proxy-g8zgl 1/1 Running 0 2d14h
      kube-system kube-proxy-l7zzt 1/1 Running 0 2d14h
      kube-system kube-proxy-m6h9g 1/1 Running 0 2d14h
      kube-system kube-proxy-ntt57 1/1 Running 0 2d14h
      kube-system kube-proxy-xjd68 1/1 Running 0 2d14h
      kube-system kube-scheduler-blrsrlfsphw01-compute1 1/1 Running 1 2d14h
      kube-system kube-scheduler-blrsrlfsphw01-compute2 1/1 Running 1 2d14h
      kube-system kube-scheduler-blrsrlfsphw01-compute3 1/1 Running 1 2d14h
      kube-system nginx-proxy-blrsrlfsphw01-compute4 1/1 Running 0 2d14h
      kube-system nginx-proxy-blrsrlfsphw01-compute5 1/1 Running 0 2d14h
      kube-system nginx-proxy-blrsrlfsphw01-compute6 1/1 Running 0 2d14h
      kube-system snapshot-controller-55dbff48d-jxp4f 1/1 Running 0 2d14h
      kube-system snapshot-controller-55dbff48d-zshwb 1/1 Running 0 2d14h
      kube-system traefik-2zqqf 1/1 Running 0 19m
      kube-system traefik-dkmbv 1/1 Running 0 19m
      kube-system traefik-jhsxw 1/1 Running 0 19m
      kube-system traefik-mdwcx 1/1 Running 0 19m
      kube-system traefik-smgvx 1/1 Running 0 19m
      kube-system traefik-tcjgd 1/1 Running 0 19m
      metallb-system metallb-controller-b99bbcfff-vnz96 1/1 Running 0 2d14h
      metallb-system metallb-speaker-drgcj 1/1 Running 0 2d14h
      metallb-system metallb-speaker-fw2ng 1/1 Running 0 2d14h
      metallb-system metallb-speaker-ghsv7 1/1 Running 0 2d14h
      metallb-system metallb-speaker-nj5hj 1/1 Running 0 2d14h
      metallb-system metallb-speaker-q4bbs 1/1 Running 0 2d14h
      metallb-system metallb-speaker-zk7mg 1/1 Running 0 2d14h
      rook-ceph csi-cephfsplugin-5jrj6 3/3 Running 0 2d14h
      rook-ceph csi-cephfsplugin-6mdf9 3/3 Running 0 2d14h
      rook-ceph csi-cephfsplugin-p2g7j 3/3 Running 0 2d14h
      rook-ceph csi-cephfsplugin-provisioner-59dd67b6c-8gwzv 6/6 Running 0 2d14h
      rook-ceph csi-cephfsplugin-provisioner-59dd67b6c-9j5h6 6/6 Running 0 2d14h
      rook-ceph csi-cephfsplugin-ps8wz 3/3 Running 0 2d14h
      rook-ceph csi-cephfsplugin-rbbr5 3/3 Running 0 2d14h
      rook-ceph csi-cephfsplugin-smpxx 3/3 Running 0 2d14h
      rook-ceph rook-ceph-mds-ceph-filesystem-a-55cfd884fc-2m2ds 1/1 Running 0 2d14h
      rook-ceph rook-ceph-mds-ceph-filesystem-b-7568464675-ng266 1/1 Running 0 2d14h
      rook-ceph rook-ceph-mgr-a-5888bc4d88-qpvbr 2/2 Running 0 2d14h
      rook-ceph rook-ceph-mgr-b-58d9f87688-km8dl 2/2 Running 0 2d14h
      rook-ceph rook-ceph-mon-a-5d45554b5c-5dbgh 1/1 Running 0 2d14h
      rook-ceph rook-ceph-mon-b-b6d489fc8-ssfkh 1/1 Running 0 2d14h
      rook-ceph rook-ceph-mon-c-5869bf7b69-k7lr8 1/1 Running 0 2d14h
      rook-ceph rook-ceph-operator-77cf896f-tw6zd 1/1 Running 0 2d14h
      rook-ceph rook-ceph-osd-0-6dbfd94778-5ttjz 1/1 Running 0 2d14h
      rook-ceph rook-ceph-osd-1-6d7679c466-76x2c 1/1 Running 0 2d14h
      rook-ceph rook-ceph-osd-2-78949bfc94-zbt9n 1/1 Running 0 2d14h
      rook-ceph rook-ceph-osd-prepare-blrsrlfsphw01-compute1-86f4f 0/1 Completed 0 8h
      rook-ceph rook-ceph-osd-prepare-blrsrlfsphw01-compute2-47jb6 0/1 Completed 0 8h
      rook-ceph rook-ceph-osd-prepare-blrsrlfsphw01-compute3-58xmq 0/1 Completed 0 8h
      rook-ceph rook-ceph-tools-c859b5678-2qhzs 1/1 Running 0 2d14h
  5. Restore previously added software catalog images.
    The upgrade procedure replaces the software catalog. Changes that you made to the software catalog are lost after the upgrade. If you previously added new SR Linux images to the software catalog, you need to modify the software catalog and then upload the SR Linux images. For instructions, see Adding a network operating system version to the software catalog in the Fabric Services System User Guide. You must complete this step before you can access the Fabric Services System to make any change.
Upgrading the Fabric Services System cluster