Software upgrade and rollback
The Fabric Services System supports software upgrades from one release to a newer one in a KVM or VMware vSphere environment running in a supported configuration (with one deployer VM and one, three, or six Fabric Services System virtual machine nodes).
The procedures in this section describe how to prepare for the software upgrade and execute the software upgrade. The rollback procedure describes how to roll back after an upgrade, if needed.
The supported upgrade path is from Release 24.12.x to Release 25.8.y.
Note: In the software upgrade and rollback procedures,
Release
24.12.x
is referred to as the old release and Release
25.1.y
is the new release.
Preparing for software upgrade
-
Display the system information and status.
Log in to the old deployer VM using SSH.
-
Back up the necessary files from the deployer VM.
Copy the following files to a directory that is outside of the deployer VM:
-
SSH files
/root/.ssh/id_rsa
/root/.ssh/id_rsa.pub
/root/.ssh/known_hosts
-
the input JSON configuration and user values YAML file (if any) that were used to install the Fabric Services System, for example:
/root/customer-input.json
/root/user_values.yaml
-
optional files; for example, SRL image tar files that were added to the old deployer VM
-
-
Back up the Fabric Services System.
For instructions, see
Backup and restore
in the Fabric Services System User Guide. -
Shut down the old deployer VM.
Note: Ensure that the old deployer VM is not configured to automatically restart.
-
Create a new deployer VM using the image for the new release.
For instructions, see Fabric Services System deployer VM creation. The new deployer should have the same network configuration and the same IP address of old deployer VM.
- Copy the files that you saved in Step 2 to the new deployer VM in the same locations.
Performing a software upgrade
Use the fss-upgrade.sh script to perform the software upgrade
steps. The command syntax is as
follows:
[root@fss-deployer ~]# /root/bin/fss-upgrade.sh
Usage: /root/bin/fss-upgrade.sh configure <config-file> - Reads config file and configures accordingly.
./bin/fss-upgrade.sh discover - Discovers cluster.
./bin/fss-upgrade.sh upgrade - Upgrades FSS application.
./bin/fss-upgrade.sh status - Prints status of cluster and FSS Chart information.
./bin/fss-upgrade.sh clean - Cleans temporary files.
./bin/fss-upgrade.sh [help | -h] - Prints usage.This script updates the entire Fabric Services System application using the new Helm charts.
Using the new deployer VM, upgrade the Fabric Services System application in place on the existing Fabric Services System cluster.-
Update the configuration.
In the following example, the input configuration file is customer-input.json.
[root@fss-deployer ~]# /root/bin/fss-upgrade.sh configure customer-input.json Timesync service is running on 192.0.2.74 Time difference is 0 seconds Timesync service is running on 192.0.2.75 Time difference is 0 seconds Timesync service is running on 192.0.2.76 Time difference is 0 seconds Timesync service is running on 192.0.2.77 Time difference is 0 seconds Timesync service is running on 192.0.2.78 Time difference is 0 seconds Timesync service is running on 192.0.2.79 Time difference is 0 seconds Maximum time difference between nodes 0 seconds Successfully configured. Please run /root/bin/fss-upgrade.sh discover -
Discover the cluster.
[root@fss-deployer ~]# /root/bin/fss-upgrade.sh discover Trying to discover fss-node01 192.0.2.74 Trying to discover fss-node02 192.0.2.75 Trying to discover fss-node03 192.0.2.76 NAME STATUS ROLES AGE VERSION fss-node01 Ready control-plane,master 14h v1.23.1 fss-node02 Ready control-plane,master 14h v1.23.1 fss-node03 Ready control-plane,master 14h v1.23.1 fss-node04 Ready <none> 14h v1.23.1 fss-node05 Ready <none> 14h v1.23.1 fss-node06 Ready <none> 14h v1.23.1 FSS release discovered fss-FSS_24_5_B1-charts-v24.5.2-57 -
Upgrade the Fabric Services System application.
[root@fss-deployer ~]# /root/bin/fss-upgrade.sh upgrade NAME STATUS ROLES AGE VERSION fss-node01 Ready control-plane,master 75m v1.23.1 fss-node02 Ready control-plane,master 74m v1.23.1 fss-node03 Ready control-plane,master 74m v1.23.1 fss-node04 Ready <none> 73m v1.23.1 fss-node05 Ready <none> 73m v1.23.1 fss-node06 Ready <none> 73m v1.23.1 FSS will be upgraded from fss-FSS_24_5_B1-charts-v24.5.2-57 to fss-FSS_24_8_B1-charts-v24.8.1-XX : Are you sure [YyNn]? y Upgrade in progress... Upgrading fss-logs fss-logs release discovered: fluent-bit-0.20.9 ; Deployer packages fss-logs release: fluent-bit 0.39.0 Upgrading cert-manager cert-manager release discovered: cert-manager-v1.12.0 ; Deployer packages cert-manager release: cert-manager v1.13.3 cert-manager upgrade started Upgrading metallb metallb release discovered: metallb-0.13.7 ; Deployer packages metallb release: metallb 0.13.7 metallb upgrade not required Upgrading traefik and ingress routes traefik release discovered: traefik-21.0.0 ; Deployer packages traefik release: traefik 26.0.0 Upgrading kafka and kafkaop if required kafkaop release discovered: strimzi-kafka-operator-0.31.0 ; Deployer packages kafkaop release: strimzi-kafka-operator 0.37.0 NEW KAFKA operator, upgrade chart and update kafka Waiting for fss services to stop KAFKA will be deleted and reinstalled. Upgrading / Installing new FSS Chart Release "prod" has been upgraded. Happy Helming! NAME: prod LAST DEPLOYED: Tue Aug 20 06:42:09 2024 NAMESPACE: default STATUS: deployed REVISION: 3 NOTES: KAFKA being reinstalled. Checking for FSS pods Waiting for default fss-catalog 1 Running Waiting for FSS pods to comeup Waiting for default fss-version 1 Running Waiting for FSS pods to comeup All FSS pods are running Checking for FSS digitalsandbox pods FSS digital sandbox pods are running Checking for digitalsandbox pods Digital sandbox pods are running FSS is started, it can take upto 10 mins to be ready for use GUI can be accessed at https://192.0.2.74 -
Verify that the upgrade is successful.
-
Restore previously added software catalog images.
The upgrade procedure replaces the software catalog. Changes that you made to the software catalog are lost after the upgrade. If you previously added new SR Linux images to the software catalog, you need to modify the software catalog and then upload the SR Linux images. For instructions, see
Adding a network operating system version to the software catalog
in the Fabric Services System User Guide. You must complete this step before you can access the Fabric Services System to make any change.
Upgrading the Fabric Services System cluster
After you complete the procedure Performing a software upgrade, the Fabric Services System software is upgraded to the new release.
-
Back up the Fabric Services System.
For instructions, see
Backup and restore
in the Fabric Services System User Guide. - Make a copy of the customer-input.json file, certificates, SR Linux images and other user-created files from the old deployer VM to a safe location.
- Delete all the compute VMs.
-
Recreate the compute VM with the Release 25.8 base image.
The number of compute VMs, hostname and IP address details should be same as the previous Fabric Services System cluster.
- Copy the customer-input.json file, certificates, SR Linux images and other user-created files to the new deployer VM.
-
Reset the old cluster configurations.
Log in to the deployer VM running the new release and execute the /root/bin/fss-install.sh reset command.
-
Set up key-based authentication from the deployer VM to all the compute
VMs.
If password authentication is enabled on the node for SSH, enter the following command from the deployer VM:
# ssh-copy-id root@<node IP/FQDN> -
Restore the backup that you created in Step 1.
For instructions, see
Backup and restore
in the Fabric Services System User Guide.
To configure the LDAP settings or remote syslog capabilities, complete the following
procedures in the Fabric Services System User Guide:
To bring-up the system in geo-redundant configuration, see Deploying a geo-redundant system.Configuring LDAP after software upgrade
Configuring a remote syslog server for user audit logs
Deploying a geo-redundant system
If you want to bring up geo-redundant system after the software upgrade, complete the following steps. The newly upgraded system is considered the active system.
-
Ensure that the upgraded system is up and running.
Check the Fabric Services System pod status and the fabric status.
-
Install another cluster, the standby cluster.
Follow the instructions described in this document, starting with Installation overview.Note: The standby cluster must of the same size and the same software version as the active system.
- Verify the network connectivity between the active and standby clusters and the connectivity with the SR Linux network.
-
Follow the instructions in
Geo-redundancy
in the Fabric Services System User Guide.
Performing a software rollback
- Shut down and delete the deployer VM and Fabric Services System virtual machine nodes for the upgraded deployment.
-
Deploy and configure the deployer VM with the old release.
For instructions, see the procedures in The Fabric Services System deployer VM.
-
Deploy new base virtual machine nodes.
For instructions, see Preparing the Fabric Services System virtual machine nodes.
-
Restore the saved backup of the old release.
For instructions, see
Backup and restore
in the Fabric Services System User Guide.