Installation

The installation of the Fabric Services System integration for OpenShift is performed using the Helm charts that are provided as part of each release of the Fabric Services System.

Package information

The Fabric Services System integration for OpenShift is provided as a tar ball (for example: fsc-v23.4.0-13.tar.gz) which contains the following files:

  • fsc-*-<release_tag>-images.tar: The container images for the FSC Version $src_tag.
  • fsc-charts-<release_tag>.tgz: A generic Helm package for FSC installation.
  • fsc-installer.sh: A utility to store the container images and the charts in registries.

where <release-tag> represents the release version (such as v23.4.0-13).

Using the installer script to push the container images and charts

You can use the fsc-installer.sh script to push the container images to a container image registry and, optionally, push the Helm charts to a Helm repository.

Note: Currently, only an upload to a container image registry that requires authentication is supported.

The script accepts the following information:

  • User ID: The user name to connect to the container image registry. (required)
  • Registry URL: Path to the container image registry. (required)
  • Helm repository URL: URL to the Helm repository to where the charts need to be uploaded. (optional).

The script can be run with the following command:

# ./fsc-installer.sh -u <user-id> -r <registry-url> -e <helm-repo>

For example:

# ./fsc-installer.sh -u imageuploader -r registry.domain.tld/fsc -e http://helm-repo.domain.tld/fsc

Installing the OpenShift and Kubernetes Plugin

This procedure describes the installation and configuration of the OpenShift and Kubernetes plugin as might be performed on a baremetal OKD cluster version 4.10.

Some steps in this procedure are further illustrated with sample configurations in OpenShift supporting objects and examples

  1. Create a local values file as described in Sample local.yaml file. At minimum this file must contain:
    • the "dockerConfig" secret to access the container registry
    • the Fabric Services System server information
  2. On the Top Of Rack node ports connected to the Kubernetes nodes, enable the following:
    • VLAN tagging
    • LLDP at the port and system levels
  3. On the Kubenetes Linux servers, do the following:
    • enable LLDP
    • configure LLDP to advertise interface name
  4. Pre-install the following on all nodes in the Kubernetes cluster:
    • CNI for Multus
    • IPVLAN
    • MACVLAN
    • IPAM
    • SR-IOV
    • SR-IOV device plugin
  5. When using IPVLAN or MACVLAN, configure VLAN interfaces on the Linux system.
    Note: This VLAN interface is used as the master interface in the Network Attachment Definition (NAD). A common name for VLAN interfaces can be configured on all worker nodes so that Pods scheduled on that worker node can be use that interface. However, Pods can be scheduled on specific worker nodes using Node selector and master interfaces referred to in the Network Attachment Definition must be present on those worker nodes for Pods to come up correctly.
  6. For SR-IOV, while using Virtual Function (VF), VLANs should not be present on the VF to be used by the Network Attachment Definition. These will be automatically configured when the NAD is deployed in a Pod. While using SR-IOV, only one Pod should be configured per VF.
  7. For SR-IOV, while using the Port function (PF) for IP VLAN, configure VLAN interfaces before using them in the NAD and Pod deployment.
  1. Add and update the helm repo, then run the installation script with the following commands: 
    # helm repo add <repoid> <repo URL> --username <username> --password <password>
# helm repo update
# helm install <RELEASE NAME> [-f <overrides file name>] <complete path to the chart> [--dry-run]
    For example:
    # helm repo add fsprepo https://artifacts.gitlabsr.nuq.ion.nokia.net/repository/fsp-charts/ --username "fsp-charts-ro" --password  "******"
  "fsprepo" has been added to your repositories
# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nfs-subdir-external-provisioner" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "fsprepo" chart repository
Update Complete. ??Happy Helming!??

# helm install prod fsprepo/fsc-charts --version v23.4.0-13 -f local.yaml
  2. Apply a Network Attachment Definition.
    Note: In the local values file, the value of the injectCni parameter can be either true or false.
    • the default value is true, which enbles automatic CNI injection.
    • if the value is false, you must specify FSC-CNI plugin information in the NetworkAttachmentDefinitions that are referenced by WCI and require automatic fabric connectivity.

    • For examples of NAD files, see REFERENCE.

    For examples of NAD files, see Network Attachment Definitions (NADs),

  3. Apply a WorkloadConnectivityIntent (WCI) using the command kubectl apply -f <file-name>

    Applying WCI results in the following:

    • it creates tenants and subnets in the Fabric Services System
    • it automatically injects fsc-cni plugin information in the NAD
    • it creates a HostPortLabel for each NAD
    Note: All NADs referenced by the WCI must be applied before WCI can be applied. Also, NADs referenced by the WCI should be unique across subnets and tenants.

    NADs should not be deleted before WCI deletion. Only one WCI can be created per tenant, and the tenant name must be unique.

    For examples of WCI files, see WorkloadConnectivityIntent (WCI) examples

  4. Verify status in the Fabric Services System. Confirm that:
    • a workload VPN intent has been created in the Fabric Services System for the WCI with tenant description as WCI metadata.name
    • subnets with the name spec.subnets.name have been configured for this workload VPN intent (tenant)
  5. Configure Pod deployment with NADs in Annotations. Apply Pod deployment using the command kubectl apply -f <file-name>.
    Note: Single or Multiple Annotations can be specified per Pod/NAD name

    For examples of Pod deployments, see Pod configuration

  6. Verify the creation of objects related to the Workload Interface Verification (WLI) using kubectl get workloadinterfaces.fsc.fss.nokia.com -n fsc-system.

    Upon Pod deployment, a WorkloadInterface (WLI) is created in the namespace “fsc-system” per worker node, per NAD on which Pods are deployed.

    For examples of WLI verification, see Workload Interface (WLI)

  7. In the Fabric Services System, verify that sub-interfaces have been created with the specified VLAN (as in the NAD master-interface for NAD configured with Pod as Annotation) per NAD, per worker node..
  8. Validate data paths.
    1. Log into the shell of one of the Pods and ping the other pod. THe ping should succeed.
    2. Check the statistics of Top of Rack node using show interface ethernet-1/<port-id> detail and verify that the Tx and Rx statistics for sub-interface increments.
Logging information is available for the FSC controller and FSC-CNI.
  • FSC-Controller Logs are available at /var/log/fsc-data/logs/ fsc-controller-manager.log on master nodes
    [root@master3 logs]# pwd
/var/log/fsc-data/logs
root@master3 logs]# ls -lrt
-rw-r--r--. 1 root root 6387577 Jul 28 06:46 fsc-controller-manager.log
  • FSC-CNI Logs are available at /var/log/fsc-cni.log on worker nodes
    [root@worker1 log]# ls -lrt
-rw-r--r--.  1 root        root            30881095 Jul 28 06:46 fsc-cni.log
[root@worker1 log]#
[root@worker1 log]# pwd
/var/log

OpenShift supporting objects and examples

Samples provided

This topic includes descriptions and examples of the following:

Helm chart override and default values

This section describes:
  • Helm chart values
  • Helm chart default values

The Helm charts for the Fabric Services System integration for OpenShift includes the following overridable properties.

fss-fsc:
  image:
    repository: <image repository>
    pullPolicy: 
    tag: <image tag>
    mgrImageName: <name of fsc pod controller image>
    cniImageName: <name of fsc cni image>
    certImageName: <name of fsc certmgr image>

  global:  
    openShift: <boolean - whether the cluster is openShift based, For Rel 23.4 this is the only option>

  fscInfo:
    dockerConfig: <base64 encoded secret for accessing the container registry>

  cniInfo: <Specify values for FSC CNI operation>
    log:
      level: <trace, debug, info, warning, error, fatal, panic>
      genFile: <boolean - true, false>
      maxAge: <integer - Duration to persist the log files in days>
      maxBackup: <integer - Number of log files to be persisted>
      maxSize: <integer - Size of log file in MB>
      path: <The file name and path for the logs on the container>  
    injectCni: <boolean - Specify true, false, whether the fsc-cni is to be added into the NAD definition automatically or not. Default is true > 
    maxUnavailable: <integer - specify the maximum number of FSC CNI  DaemonSet pods that can be unavailable during an update> 
  fssInfo:
    hostName: <hostname of the machine where FSS is running>
    ipAddr: <IP address of the host where FSS is running>
    userId: <User id to connect to the FSS>
    password: <Password to connect to the FSS>
    tlsEnable: <boolean - Whether the connection to FSS uses TLS>
    pluginId: <The Unique id across FSC plugins talking to the same FSS>
    pluginName: <User identifiable name for the plugin>
    deploymentName: < User identifiable name for the deployment, Max length is 79 characters>
    deploymentDescription: <Deployment description>
    tlsSkipVerify: <boolean - true/false - Whether to skip the verification of TLS certificates - valid only if tlsEnable is true)>
    tlsCertData: <TLS Certificate data – valid only if tlsEnable is true>
    heartbeatInterval: <The value in seconds for the keepalives between FSC plugin and FSS, recommended range 3 – 10 secs>
    supportsHeartbeat: <boolean - Supports generating alarms on FSS on plugin reachability when enabled, takes true/false>
    actionables: <FSS notifications that needs to be acted upon by FSC plugin. Allowed values are “DEPLOYMENT_UPDATED,AUDIT_REQUESTED”. DEPLOYMENT_UPDATED actionable support enables FSC to be aware of Deployment admin state (Admin Up or Admin Down) in Fabric Services System. AUDIT_REQUESTED actionable support enabled on-demand audit for the plugin in the Fabric Services System.
  mgrInfo:
    log:
      level: <trace, debug, info, warning, error, fatal, panic>
      genFile: <boolean - true, false>
      maxAge: <integer - Duration to persist the log files in days>
      maxBackup: <integer - Number of log files to be persisted>
      maxSize: <integer - Size of log file in MB>
      path: <The file name and path for the logs on the container>  
    connMapSize: <integer - Optimize memory utilization by providing the peak value of the number of concurrent pod interfaces created>

The default values for the helm chart properties, where <helm release name> release name given during helm install, are:

fss-fsc:
  global:
    openShift: true

  fscInfo:
    dockerConfig: ""

  cniInfo:
    log:
      level: info
      genFile: true
      maxAge: 7
      maxBackup: 3
      maxSize: 100
      path: /var/log/fsc-cni.log
    injectCni: true
    maxUnavailable: 3

  fssInfo:
    hostName: "fss.nokia.com"
    ipAddr: 127.0.0.1
    userId: ""
    password: ""
    tlsEnable: true
    tlsSkipVerify: true
    pluginId: "k8s-plugin-id"
    pluginName: "k8s-plugin-name"
    deploymentName: "k8s-deployment-name"
    deploymentDescription: "k8s connect deployment"
    tlsCertData: ""
    heartbeatInterval: 3
    supportsHeartbeat: true
    actionables: |
      DEPLOYMENT_UPDATED
      AUDIT_REQUESTED

  mgrInfo:
    log:
      level: info
      genFile: true
      maxAge: 7
      maxBackup: 3
      maxSize: 100
      path: /fss/data/fsc-data/logs/fsc-controller-manager.log
    connMapSize: 750

Sample local.yaml file

In the following sample local.yaml file, the dockerConfig is the base64 encoded pull secret for downloading the FSC container images from the registry.

fss-fsc:
  image:
    repository: registry.gitlabsr.nuq.ion.nokia.net/sr/linux/fsp/fsc
    pullPolicy: IfNotPresent
    tag: v23.4.0-47
    mgrImageName: fsc-manager
    cniImageName: fsc-cni
    certImageName: fsc-cert
  global:
    openShift: true
  fscInfo:
    dockerConfig: ewoJImF1dGhzIjogewoJCSJyZWdpc3RyeS5naXRs************************************mxjanA0YlRKaFVHbGlkMWR4VlVKWWEzRklRbkp4WXc9PSIKCQl9Cgl9Cn0KCg==
  cniInfo:
    log:
      level: info
      genFile: true
      maxAge: 7
      maxBackup: 3
      maxSize: 100
      path: /var/log/fsc-cni.log
    injectCni: true
    maxUnavailable: 3
  fssInfo:
    hostName: "fss.nokia.com"
    ipAddr: 127.0.0.1
    userId: ""
    password: ""
    tlsEnable: true
    tlsSkipVerify: true
    pluginId: "k8s-plugin-id"
    pluginName: "k8s-plugin-name"
    deploymentName: "k8s-deployment-name"
    deploymentDescription: "k8s connect deployment"
    heartbeatInterval: 3
    supportsHeartbeat: true
    actionables: |
      DEPLOYMENT_UPDATED
      AUDIT_REQUESTED
  mgrInfo:
    log:
      level: info
      genFile: true
      maxAge: 7
      maxBackup: 3
      maxSize: 100
      path: /fss/data/fsc-data/logs/fsc-controller-manager.log
    connMapSize: 750

Network Attachment Definitions (NADs)

A Network Attachment Definition (NAD)

The following NAD includes the FSC-CNI plugin information.

[fsc-helper@blrfsctb01-fsc-helper default]$ cat defnad9port2_cni.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: def-nad9-port2
spec:
  config: '{
     "cniVersion": "0.3.1",
     "name": "def-nad9-port2",
     "plugins": [
       {
         "type": "ipvlan",
         "master": "fscintf2.2709",
         "mode": "l2",    
         "ipam": {
           "type": "whereabouts",
           "range": "29.1.1.1/24",
           "gateway": "29.1.1.254"
         }
       },
       {
         "type": "fsc-cni",
         "args": {
           "parent": "default/def-nad9-port2",  
           "cnicache": "/var/lib/cni/fsc-cni"
         }
       }
     ]
    }'
Note:
  • “type” should always be “fsc-cni” for FSC plugin.
  • “parent” in the fsc-cni follows the format “namespace/<network-attachment-definition-name>”. For the default namespace, it should be “default/ <network-attachment-definition-name>”
  • <network-attachment-definition-name> must match the metadata.name of the NetworkAttachmentDefinition
  • cnicache is the location to store the prevresult plugin config provided by multus. The configuration from this location is retrieved later for deletion purpose

The following is an example of an SR-IOV NAD:

[fsc-helper@blrfsctb02-fsc-helper NAD]$ cat nad-sriov_vf0_516.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: blrone-sriov-vf0-516
  namespace: blrone
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_netdevice
spec:
  config: '{
  "cniVersion": "0.3.1",
  "name": "blrone-sriovnet-vf0-516",
  "plugins": [
    {
      "type": "sriov",
      "cniVersion": "0.3.1",
      "name": "blrone-sriovnet-vf0-516",
      "vlan": 516,
      "deviceID": "0000:01:03.6",
      "ipam": {
        "type": "whereabouts",
        "range": "40.9.1.1/24",
        "gateway": "40.9.1.254"
      }
    }
  ]
}'

The following is an example of an SR-IOV NAD that is untagged, and does not speicfy VLAN:

[fsc-helper@blrfsctb02-fsc-helper NAD]$ cat nad-sriov_vf0_517_untagged.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: blrone-sriov-vf0-517-untagged
  namespace: blrone
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_netdevice
spec:
  config: '{
  "cniVersion": "0.3.1",
  "name": "blrone-sriovnet-vf0-517-untagged",
  "plugins": [
    {
      "type": "sriov",
      "cniVersion": "0.3.1",
      "name": "blrone-sriovnet-vf0-517-untagged",
      "deviceID": "0000:01:03.7",
      "ipam": {
        "type": "whereabouts",
        "range": "40.9.1.1/24",
        "gateway": "40.9.1.254"
      }
    }
  ]
}'

The following is an example of an SR-IOV NAD that specifies VLAN as 0:

[fsc-helper@blrfsctb02-fsc-helper NAD]$ cat nad-sriov_vf0_513_untagged.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: blrone-sriov-vf0-513-untagged
  namespace: blrone
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_netdevice
spec:
  config: '{
  "cniVersion": "0.3.1",
  "name": "blrone-sriovnet-vf0-513-untagged",
  "plugins": [
    {
      "type": "sriov",
      "cniVersion": "0.3.1",
      "name": "blrone-sriovnet-vf0-513-untagged",
      "vlan": 0,
      "deviceID": "0000:01:03.3",
      "ipam": {
        "type": "whereabouts",
        "range": "40.7.1.1/24",
        "gateway": "40.7.1.254"
      }
    }
  ]
}'

After FSC-CNI injection on a WCI Deployment, FSC-CNI plugin information is available in the Network Attachment Definition file after automatic injection, as shown in the example below.

[fsc-helper@blrfsctb01-fsc-helper default]$ kubectl describe network-attachment-definitions.k8s.cni.cncf.io def-nad11-port2 -n blrtwo
Name:         def-nad11-port2
Namespace:    blrtwo
Labels:       <none>
Annotations:  <none>
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2022-07-19T13:55:40Z
  Generation:          50
  Managed Fields:
    API Version:  k8s.cni.cncf.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:

      f:spec:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-27T08:59:23Z
    API Version:  k8s.cni.cncf.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:config:
    Manager:         fsc-manager
    Operation:       Update
    Time:            2022-07-28T06:09:20Z
  Resource Version:  18598415
  UID:               ac661ad5-44e3-47ac-a658-e099b509d2b8
Spec:
  Config:  {"cniVersion":"0.3.1","name":"def-nad11-port2","plugins":[{"ipam":{"gateway":"27.1.1.254","range":"27.1.1.1/24","type":"whereabouts"},"master":"fscintf2.2706","mode":"l2","type":"ipvlan"},{"args":{"cnicache":"/var/lib/cni/fsc-cni","parent":"blrtwo/def-nad11-port2"},"type":"fsc-cni"}]}
Events:    <none>
Note:
  • After a WorkloadConnectivityIntent (WCI) is deployed, if you reapply a NAD definition that is referenced by th WCI and that NAD definition does not include fsc-cni plugin information, it NOT injected again.
  • When modifying a NAD referenced by a WCI, you must add fsc-cni plugin information in the Network Attachment Definition file and the fsc-cni plugin should be the last one. To obtain plugin information, use Kubectl get network-attachment-definitions <NAD-Name> -o yaml” of NAD before modification.
  • The “Parent name” in the fsc-cni should match the NAD name and must be in the format <namespace/NAD-name> when you specify fsc-cni plugin information in the Network Attachment Definition file during modification.

The following sample shows a NAD configuration for IPVLAN, for modification after WCI deployment.

[fsc-helper@blrfsctb01-fsc-helper default]$ cat blrtwodefnad11port2_cni.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: def-nad11-port2
  namespace: blrtwo
spec:
  config: '{
       "cniVersion": "0.3.1",
       "name": "def-nad11-port2",
       "plugins": [
         {
           "type": "ipvlan",
           "master": "fscintf2.2706",
           "mode": "l2",    
           "ipam": {
             "type": "whereabouts",
             "range": "27.1.1.1/24",
             "gateway": "27.1.1.254"
           }
         },
         {
           "type": "fsc-cni",
           "args": {
             "parent": "blrtwo/def-nad11-port2", 
             "cnicache": "/var/lib/cni/fsc-cni"
           }
         }
       ]
      }'

WorkloadConnectivityIntent (WCI) examples

A WorkloadConnectivityIntent file includes the following information:

  • Metadata.name = name of tenant to be created
  • Metadata.namespace is always fsc-system
  • Spec.namespace = this is not mandatory. However, if specified the all NADs must be in same namespace as spec.namespace. If this is not specified, subnets can reference NADs from any namespace
  • Spec.subnets: This is list of subnets to be created for this tenant.
  • Spec.subnets.cni : This provides a list of NADs that would be part of this subnet
  • NAD can be specified as <NAD-name> or <namespace/NAD-name>.
    • When spec.namespace is “Not configured”, specify the NAD as <NAD_name> is considered as “default/<NAD-name>”.
    • When spec.namespace is “configured”, specify the NAD as <NAD_name> is considered as “spec.namespace/<NAD-name>”.
  • Spec.type is “IRB”, for Integrated Routing and Bridging
  • Spec.subnets.type: only "bridged" is currently supported

The following WCI uses the spec.namespqace "Configured".

[fsc-helper@blrfsctb01-fsc-helper default]$ cat copy_blrtwointentNS.yaml
apiVersion: fsc.fss.nokia.com/v1
kind: WorkloadConnectivityIntent
metadata:
  name: blrtwotenant10
  namespace: fsc-system
spec:
  namespace: blrtwo
  type: "IRB"
  subnets:
   - name: "blrtwot1sub1"
     type: "bridged"
     cni:
       - "def-nad10"
       - "blrtwo/def-nad9"
   - name: "blrtwot1sub2"
     type: "bridged"
     cni:
       - "blrtwo/def-nad9-port2"
       - "def-nad10-port2"
   - name: "blrtwot1sub4"
     type: "bridged"
      cni:
       - "blrtwo/def-nad11-port2"
        - "def-nad11"

The following WCI uses the spec.namespqace "Not Configured".

[fsc-helper@blrfsctb01-fsc-helper default]$ cat defintent_multiNamespace.yaml
apiVersion: fsc.fss.nokia.com/v1
kind: WorkloadConnectivityIntent
metadata:
  name: deftenant1
  namespace: fsc-system
spec:
  type: "IRB"
  subnets:
   - name: "deft1sub1"
     type: "bridged"
     cni:
       - "def-nad1"
       - "def-nad2"
       - "def-nad3"
       - "def-nad4"
       - "def-nad5"
       - "def-nad9"
       - "default/def-nad6"
       - "blrone/blrone-nad2"
       - "blrone/blrone-nad1"
       - "def-nad9-port2"
   - name: "deft1sub2"
     type: "bridged"
     cni:
       - "def-nad1-port2"
       - "def-nad2-port2"
       - "def-nad3-port2"
       - "def-nad4-port2"
       - "def-nad5-port2"
       - "default/def-nad6-port2"
       - "default/def-nad7-port2"
       - "blrone/blrone-nad1-port2"
       - "blrone/blrone-nad2-port2"
       - "blrtwo/blrtwo-nad6"
       - "default/def-nad7"
   - name: "deft1sub3"
     type: "bridged"
     cni:
       - "blrtwo/blrtwo-nad6-port2"
       - "blrtwo/blrtwo-nad7-port2"
       - "blrtwo/blrtwo-nad7"
       - "default/def-nad10-port2"
       - "default/def-nad10"
   - name: "deft1sub4"
     type: "bridged"
     cni:
       - "default/def-nad8-port2"
        - "default/def-nad8"

You can view the workload connectivity intent contents using kubectl describe workloadconnectivityintents.fsc.fss.nokia.com blrtwotenant10 -n fsc-system

WCI output after WCI deployment includes the following fields:

  • Connectstatus: Sync-Done in this field indicates whether WCI is deployed on FSS through connect.
  • Cnistatus: Connectedpods is “true” when there are Pods are available using this NAD are available. If there are no Pods using this, status is “false”.
  • Crdstatus:
    • "CNI-Validation-Failed" - Failed to validate NAD presence.
    • "Queued-update-FSS" - Queued Update to FSS. FSS status will be monitored by Connect-Status.
    • "Queued-delete-FSS" - Queued delete to FSS. FSS status will be monitored by Connect Status.
  • Connectstatus:
    • “Sync-Done” in this field indicates whether NAD is deployed in FSS through connect.
    • "Reg-Failed" in this field indicates FSS registration had failed. There is no retries to connect to FSS.
    • "Sync-Pending" in this field indicates request has been send to FSS to create resources however response if yet to be received. Retry will be done to achieve Sync-done with FSS.
    • "Sync-Deleted" in this field indicates request to delete resources has been sent to FSS and successful response is received.
    • "Sync-Mark-Delete " in this field indicates request to delete resources has been sent to FSS but response is yet to be received
    • "Sync-Add-Failed " in this field indicates request to add or create resources has been sent to FSS and failure is received from FSS for some reason. No retries are done in this case.
    • "Sync-Del-Failed " in this field indicates request to delete resources has been sent to FSS and failure is received from FSS for some reason. No retries are done in this case.

The sample below shows the output of a workload connectivity intent query after WCI deployment.

[fsc-helper@blrfsctb01-fsc-helper default]$ kubectl describe workloadconnectivityintents.fsc.fss.nokia.com blrtwotenant10 -n fsc-system
Name:         blrtwotenant10
Namespace:    fsc-system
Labels:       <none>
Annotations:  <none>
API Version:  fsc.fss.nokia.com/v1
Kind:         WorkloadConnectivityIntent
Metadata:
  Creation Timestamp:  2022-07-28T06:09:19Z
  Finalizers:
    fsc.io/gWCFinalizer
  Generation:  2
  Managed Fields:
    API Version:  fsc.fss.nokia.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"fsc.io/gWCFinalizer":
      f:spec:
        f:managedid:
        f:subnets:
    Manager:      fsc-manager
    Operation:    Update
    Time:         2022-07-28T06:09:19Z
    API Version:  fsc.fss.nokia.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:namespace:
        f:type:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-07-28T06:09:19Z
    API Version:  fsc.fss.nokia.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:connectstatus:
        f:crdstatus:
        f:subnetstatus:
    Manager:         fsc-manager
    Operation:       Update
    Subresource:     status
    Time:            2022-07-28T06:09:20Z
  Resource Version:  18598421
  UID:               cfdf73bb-ff85-45e2-aca0-0b1f3d7643e1
Spec:
  Managedid: 
  Namespace:  blrtwo
  Subnets:
    Cni:
      def-nad10
    Managedid: 
    Name:       blrtwot1sub1
    Type:       bridged
    Cni:
      blrtwo/def-nad9-port2
    Managedid: 
    Name:       blrtwot1sub2
    Type:       bridged
    Cni:
      blrtwo/def-nad11-port2
      def-nad11
      def-nad10-port2
    Managedid: 
    Name:       blrtwot1sub4
    Type:       bridged
    Cni:
      blrtwo/def-nad9
    Managedid: 
    Name:       blrtwot1sub5
    Type:       bridged
  Type:         IRB
Status:
  Connectstatus:  Sync-Done
  Crdstatus:      Queued-update-FSS
  Subnetstatus:
    Cnistatus:
      Connectedpods:  false
      Connectstatus:  Sync-Done
      Name:           def-nad10
      Namespace:      blrtwo
    Connectstatus:    Sync-Done
    Name:             blrtwot1sub1
    Cnistatus:
      Connectedpods:  false
      Connectstatus:  Sync-Done
      Name:           blrtwo/def-nad9-port2
      Namespace:      blrtwo
    Connectstatus:    Sync-Done
    Name:             blrtwot1sub2
    Cnistatus:
      Connectedpods:  false
      Connectstatus:  Sync-Done
      Name:           blrtwo/def-nad11-port2
      Namespace:      blrtwo
      Connectedpods:  false
      Connectstatus:  Sync-Done
      Name:           def-nad11
      Namespace:      blrtwo
      Connectedpods:  false
      Connectstatus:  Sync-Done
      Name:           def-nad10-port2
      Namespace:      blrtwo
    Connectstatus:    Sync-Done
    Name:             blrtwot1sub4
    Cnistatus:
      Connectedpods:  false
      Connectstatus:  Sync-Done
      Name:           blrtwo/def-nad9
      Namespace:      blrtwo
    Connectstatus:    Sync-Done
    Name:             blrtwot1sub5
Events:               <none>

Pod configuration

Single or Multiple Annotations can be specified per POD/NAD name

Apply a Pod/Deployment using kubectl apply -f <file-name>

The following is an example of a Pod with a single NAD annotation.

[fsc-helper@blrfsctb01-fsc-helper default]$ cat blrtwodefdepPod10_port2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blrtwo-def-nad10-port2-dep
  namespace: blrtwo
  labels:
    project: fsc
spec:
  replicas: 7
  selector:
    matchLabels:
      project: fsc
  template:
    metadata:
      labels:
        project: fsc
      annotations:
        k8s.v1.cni.cncf.io/networks: blrtwo/def-nad10-port2
    spec:
      imagePullSecrets:
      - name: regcred
      containers:
      - name: centos1
        imagePullPolicy: IfNotPresent
        image: centos/tools
        command: ["/bin/bash"]
        args: ["-c", "while true; do echo hello; sleep 10;done"]     
        ports:
        - containerPort: 8080

The following is a sample Pod configuration with multiple NAD annotations

[fsc-helper@blrfsctb01-fsc-helper default]$ cat defdepPod7_select.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: def-nad7-8-9-10-dep-select
  labels:
    project: fsc
spec:
  replicas: 5
  selector:
    matchLabels:
      project: fsc
  template:
    metadata:
      labels:
        project: fsc
      annotations:
        k8s.v1.cni.cncf.io/networks: def-nad7, def-nad8, def-nad9, def-nad10 
    spec:
      nodeSelector:
        sriovnic: present
      imagePullSecrets:
      - name: regcred
      containers:
      - name: centos1
        imagePullPolicy: IfNotPresent
        image: centos/tools
        command: ["/bin/bash"]
        args: ["-c", "while true; do echo hello; sleep 10;done"]
        ports:
        - containerPort: 8080

Workload Interface (WLI)

Upon Pod deployment, a WorkloadInterface (WLI) is created in the namespace “fsc-system” per worker node, per NAD on which Pods are deployed. The WLI data can be obtained using the command kubectl get workloadinterfaces.fsc.fss.nokia.com -n fsc-system

For example:

[fsc-helper@blrfsctb01-fsc-helper default]$ kubectl get workloadinterfaces.fsc.fss.nokia.com -n fsc-system
NAME                                  AGE
Worker1.lab.fsc.io-blrtwo-def-nad10   17m
Worker2.lab.fsc.io-blrtwo-def-nad10   17m

A sub-interface is created on deployment of the first Pod on a worker node for each NAD. However, when deploying multiple Pods using the same NAD on the same worker node, the same WLI keeps track of all Pods sharing the sub-interface. Upon removal of a Pod from a worker node, an entry is removed from the WLI. Upon removal of the last Pod sharing a specific WLI, the WLI itself is deleted for that worker node and a deletion message to delete a sub-interface is sent to the Fabric Services System.

To obtain details about the pods using a WLI, use the command kubectl describe workloadinterfaces.fsc.fss.nokia.com <wli-name> -n fsc-system

In the resulting output:

  • “Sync-Done” in this field indicates successful creation of sub-interfaces through Connect.
  • " Sync-Reg-Failed " in this field indicates that Fabric Services System registration has failed. No further attempts are made to connect to the Fabric Services System.
  • "Sync-Pending" in this field indicates the request has been send to the Fabric Services System to create resources however response if yet to be received. Another attempt will be made to achieve Sync-done with the Fabric Services System.
  • "Sync-Deleted" in this field indicates request to delete resources has been sent to the Fabric Services System and a successful response was received.
  • "Sync-Mark-Delete " in this field indicates request to delete resources has been sent to the Fabric Services System but no response has been received.
  • "Sync-Add-Failed " in this field indicates request to add or create resources has been sent to the Fabric Services System and failure is received from the Fabric Services System for some reason. No further attempts are made in this case.
  • "Sync-Del-Failed " in this field indicates request to delete resources has been sent to the Fabric Services System and failure is received from the Fabric Services System for some reason. No further attempts are made in this case.

The following is a sample output from a workload description command.

[fsc-helper@blrfsctb01-fsc-helper default]$ kubectl describe workloadinterfaces.fsc.fss.nokia.com worker1.lab.fsc.io-blrtwo-def-nad10 -n fsc-system
Name:         worker1.lab.fsc.io-blrtwo-def-nad10
Namespace:    fsc-system
Labels:       <none>
Annotations:  fsc/metadata: {"uid":"58542c1f-4a3c-4987-9140-8d3029a05e37","creationTimestamp":"2022-07-28T06:46:09Z"}
API Version:  fsc.fss.nokia.com/v1
Kind:         WorkloadInterface
Metadata:
  Creation Timestamp:  2022-07-28T06:46:09Z
  Generation:          1
  Managed Fields:
    API Version:  fsc.fss.nokia.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:cni:
        f:server-interface:
          .:
          f:interface:
          f:node:
          f:vlan-end:
          f:vlan-start:
          f:vlan-type:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2022-07-28T06:46:09Z
    API Version:  fsc.fss.nokia.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:connectstatus:
        f:pending-podkeys:
        f:synced-podkeys:
          .:
          f:worker1.lab.fsc.io:37721f66056e9e87038cf39a73e023b27046dabb7ec1b0a5d59dcfe6d53081a0-net1:
          f:worker1.lab.fsc.io:4df27e16cc53a660f3841a10522c02e254f722907c7be9f193965107750cca4f-net1:
          f:worker1.lab.fsc.io:c7acdb797dae90fa2318b47063493dbd24ca46c300ca7560cebdb0448c5dc74d-net1:
          f:worker1.lab.fsc.io:e21de0ba145d2576f8f1423b9d5dfece6fe399ec9db8996c05afbce837a36533-net1:
    Manager:         fsc-manager
    Operation:       Update
    Subresource:     status
    Time:            2022-07-28T06:46:09Z
  Resource Version:  18611818
  UID:               21fbed90-43fb-465f-bbfb-e5de7ee9fe9d
Spec:
  Cni:  blrtwo/def-nad10
  Server - Interface:
    Interface:     enp6s0
    Node:          worker1.lab.fsc.io
    Vlan - End:    2006
    Vlan - Start:  2006
    Vlan - Type:   VLANTYPE_VALUE
Status:
  Connectstatus:  Sync-Done
  Pending - Podkeys:
  Synced - Podkeys:
    Worker1.lab.fsc.io:37721f66056e9e87038cf39a73e023b27046dabb7ec1b0a5d59dcfe6d53081a0-net1: blrtwo/blrtwo-def-nad10-dep-7bd5f9f9c4-ptmnx
    worker1.lab.fsc.io:4df27e16cc53a660f3841a10522c02e254f722907c7be9f193965107750cca4f-net1: blrtwo/blrtwo-def-nad10-dep-7bd5f9f9c4-lzjm7
    worker1.lab.fsc.io:c7acdb797dae90fa2318b47063493dbd24ca46c300ca7560cebdb0448c5dc74d-net1: blrtwo/blrtwo-def-nad10-dep-7bd5f9f9c4-dg6kt
    worker1.lab.fsc.io:e21de0ba145d2576f8f1423b9d5dfece6fe399ec9db8996c05afbce837a36533-net1: blrtwo/blrtwo-def-nad10-dep-7bd5f9f9c4-2xvnq
Events:     
<none>