Dynamic-VF support for SRIOV

The Fabric Services Controller (FSC) supports SRIOV and Dynamic-VF SRIOV, including:

Support for OpenShift sriovnetwork for creating SRIOV Network Attachment Definitions (NADs)
Support for the dynamic allocation of Virtual functions across different worker nodes and master nodes from the defined resource pools
Support for multiple replica-sets of pods on the same sriovnetwork or NAD, as a VF PCI address is not required in the NAD

Static-VFs are also supported; however either Static-VF or Dynamic-VF can be used in a cluster, but not both at the same time. When using Dynamic-VF in an OpenShift cluster, the sriovoperatorconfig spec.enableInjector must be set to True. This allows the automatic addition of SRIOV VF resource limits to Pods. While using Static-VF, the spec.enableInjector parameter must be set to False on the cluster.

With Dynamic-VF support, you can now allocate resource pools across different worker nodes and master nodes (and across multiple ports of the same worker or master node) using the OpenShift sriovnetworknodepolicy. Pods come up by dynamic allocation of SRIOV virtual function resources from the pool.

If any node fails, Pods can switch to other nodes by selecting available resources on the other nodes from the same resource pool. As Pods move across worker nodes, the Fabric Services Controller application manages the removal of interfaces from one port and their re-creation on another port, providing seamless connectivity in the fabric.

Dynamic-VF feature supports both RDMA and non-RDMA for Mellanox card.

Note:

In SriovNetworkNodePolicy spec.linkType option is not supported.
Dynamic modification of SriovNetwork that is already in use by applied workloadconnectivityintent (WCI) on cluster, is not supported. Instead, you must:
- first delete the pods using the network-attachment-definition (NAD) created from the SriovNetwork
- then delete the NAD from WCI
- then update the SriovNetwork
- then add the NAD in WCI and re-apply WCI
FSC supports both Static-VF (using PCI address directly in network-attachment-definition (NAD) without using sriovnetwork) and Dynamic-VF allocation. Static-VF requires spec.enableInjector field in SriovOperatorConfig to be set to false on cluster whereas it should be set to true for Dynamic-VF. Hence both Static-VF and dynamic-VF feature cannot be used together on the cluster.

Sample configurations

SRIOV Operator configuration

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovOperatorConfig
metadata:
  name: default
  namespace: openshift-sriov-network-operator
spec:
  configDaemonNodeSelector:
    kubernetes.io/os: linux
  enableInjector: true

SRIOV Network Node Policy – RDMA False (Mellanox)


apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: sriov-vf-create-worker5-respool1
  namespace: openshift-sriov-network-operator
spec:
  resourceName: sriov_fab1_nics
  nodeSelector:
    kubernetes.io/hostname: worker1.nokia.com
  priority: 50
  mtu: 1500
  needVhostNet: false
  numVfs: 64
  nicSelector:
    vendor: "15b3"
    deviceID: "101f"
    pfNames: ["enp1s0f0#0-15", "enp1s0f1#16-31"]
    rootDevices: ["0000:01:00.0", "0000:01:00.1"]
  isRdma: false

SRIOV Network Node Policy – RDMA True (Mellanox)


apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: sriov-vf-create-worker5-respool3-rdma
  namespace: openshift-sriov-network-operator
spec:
  resourceName: sriov_fab3_nics_rdma
  nodeSelector:
    kubernetes.io/hostname: worker1.nokia.com
  priority: 50
  mtu: 1500
  needVhostNet: false
  numVfs: 64
  nicSelector:
    vendor: "15b3"
    deviceID: "101f"
    pfNames: ["enp1s0f0#32-47", "enp1s0f1#48-63"]
    rootDevices: ["0000:01:00.0", "0000:01:00.1"]
  isRdma: true

SRIOV Network (Single-plugin format – for Automatic Injection)

apiVersion: "sriovnetwork.openshift.io/v1"
kind: SriovNetwork
metadata:
  name: default-500-sriov-network-nad
  namespace: openshift-sriov-network-operator
spec:
    resourceName: sriov_fab1_nics
    networkNamespace: default
    vlan: 500
    spoofchk: "off"
    trust: "off"
    link_state: "enable"
    ipam: |
      {
        "type": "whereabouts",
        "range": "40.2.1.1/24",
        "gateway": "40.2.1.254"
      }

SRIOV Network (Multiple-plugin format – for Automatic Injection)


apiVersion: "sriovnetwork.openshift.io/v1"
kind: SriovNetwork
metadata:
  name: blrone-806-sriov-network-nad-mplugin
  namespace: openshift-sriov-network-operator
spec:
    resourceName: sriov_fab2_nics
    networkNamespace: blrone
    vlan: 806
    spoofchk: "off"
    trust: "off"
    link_state: "enable"
    ipam: |
      {
        "type": "whereabouts",
        "range": "41.5.1.1/24",
        "gateway": "41.5.1.254"
      }
    metaPlugins : |
     {
       "type": "tuning",
       "sysctl": {
         "net.ipv4.conf.IFNAME.accept_redirects": "1"
       }
     },
     {
       "type": "vrf",
       "vrfname": "red"
     }

SRIOV Network (Multiple-plugin format – for Manual Injection)


apiVersion: "sriovnetwork.openshift.io/v1"
kind: SriovNetwork
metadata:
  name: default-sriov-network-nad-mplugin-501
  namespace: openshift-sriov-network-operator
spec:
    resourceName: sriov_fab1_nics
    networkNamespace: default
    vlan: 501
    spoofchk: "off"
    trust: "off"
    link_state: "enable"
    ipam: |
      {
        "type": "whereabouts",
        "range": "40.2.1.1/24",
        "gateway": "40.2.1.254"
      }
    metaPlugins : |
      {
        "type": "tuning",
        "sysctl": {
          "net.ipv4.conf.IFNAME.accept_redirects": "1"
        }
      },
      {
        "type": "vrf",
        "vrfname": "blue"
      },
      {
        "type": "fsc-cni",
        "args": {
          "parent": "default/default-sriov-network-nad-mplugin-501",
          "cnicache": "/var/lib/cni/fsc-cni/"
        }
      }