OpenShift Connect plugin

EDA Cloud Connect integrates with OpenShift to provide fabric-level application networks for OpenShift pods and services. The Connect integration leverages the OpenShift Multus CNI solution to support managing the fabric directly from OpenShift and making the fabric dynamically respond to the networking needs of the application.

OpenShift provides the following advantages and capabilities:

  • Direct integration into the network management workflow of OpenShift.
  • Use of the common CNIs used by enterprise applications and CNFs like IPVLAN and SR-IOV.
  • Automatic provisioning of the fabric based on where the application pods need the connectivity.
  • Support for advanced workflows.

Supported versions

The following versions of OpenShift are supported:

  • Red Hat OpenShift 4.16

  • Red Hat OpenShift 4.18

Prerequisites

Before using the Connect OpenShift plugin, ensure the following prerequisites are met:
  • Openshift cluster is up and running.
  • NMState-Operator and Multus are installed on the OpenShift cluster.
    Note: For more information about the NMState Operator, see the Red Hat OpenShift documentation About the Kubernetes NMState Operator.
  • EDA cluster is up and running.
  • EDA Cloud Connect app is installed.
  • You have access to the controller container image named ghcr.io/nokia-eda/eda-connect-k8s-controller:3.0.0.
  • NMState Operator is configured to listen for LLDP TLVs. Create the following resource in your OpenShift cluster and make sure to include all interfaces in the list that are connected to leaf switches managed by EDA:
    apiVersion: nmstate.io/v1 
kind: NodeNetworkConfigurationPolicy 
metadata: 
  name: enable-receive-lldp 
spec: 
  desiredState: 
    interfaces: 
    - name: <interface-name> 
      lldp: 
        enabled: true

Architecture

The OpenShift plugin consists of a controller which monitors the following resources in OpenShift:
  • The physical NIC configuration and correlation to NetworkAttachmentDefinitions through the NMState Operator.
  • NetworkAttachmentDefinitions and their master interfaces.
  • ConnectNetworkDefinitions for the configuration of Layer 2 and Layer 3 services configuration.

Operational modes

The OpenShift plugin supports three methods of associating NetworkAttachmentDefinitions (NADs) to EDA BridgeDomains, called Operational modes:

Table 1. Operational modes
Operational mode Description
Transparent association

This association does not require any information from EDA in OpenShift. For every unique master interface defined in NADs in the OpenShift Cluster, the plugin will create a unique BridgeDomain. This association only supports OpenShift Managed Mode.
Annotation based association

In this association, annotations are added to the NAD that reference an existing EDA BridgeDomain. This association only supports EDA-managed mode.
ConnectNetworkDefinition association

The ConnectNetworkDefinition is a Custom Resource Definition that gets added to the OpenShift Cluster and is used to describe the relationship between the different services and NetworkAttachmentDefinitions, and how the services relate to each other. This association supports both OpenShift-managed and EDA-managed modes.
Note: It is possible to use a mix of these modes in a single OpenShift cluster, as well as switch and migrate between them.

Supported features

The following are some of the supported OpenShift features:

  • CMS-managed integration mode

  • EDA-managed integration mode

  • Network Attachment Definition Transparent operational mode (CMS-managed only)

  • Connect Network Definition operational mode (CMS-managed and EDA Managed)

  • Network Attachment Definition Annotation operational mode (EDA-managed only)

  • Optimally configure subinterfaces to minimize configuration and security footprint of network services

  • LAG/LACP interfaces

  • VLAN Trunking

  • Audits

The following are supported CNIs:

  • MACVLAN

  • IPVLAN

  • SRIOV

  • Dynamic SRIOV

Deployment

This section describes how to prepare for the deployment of the OpenShift plugin and how to deploy the plugin.

EDA Kubernetes preparation

Creating a service account and token

Creating a service account

The EDA Connect OpenShift plugin uses a ServiceAccount in the EDA Kubernetes cluster to create the necessary resources in the EDA cluster for the integration to work.

To create a service account in the EDA Kubernetes cluster, use the following resource.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-controller-plugin
  namespace: eda-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: k8s-controller-plugin
subjects:
- kind: ServiceAccount
  name: k8s-controller-plugin
  namespace: eda-system
roleRef:
  kind: ClusterRole
  # This cluster role is assumed to be already installed by connect app.
  name: eda-connect-plugin-cluster-role
  apiGroup: rbac.authorization.k8s.io
Note: This service account must be created in the eda-system namespace.
Creating a service account token
A service account token is used by the plugin to connect to the EDA Kubernetes cluster. You can create a service account token by using the following manifest, which should be applied on the EDA Kubernetes cluster.
--- 
apiVersion: v1 
kind: Secret 
type: kubernetes.io/service-account-token 
metadata: 
  name: k8s-controller-plugin
   
  annotations: 
    kubernetes.io/service-account.name: k8s-controller-plugin
After creating the service account token, you can retrieve the actual token itself using the following command. This token will need to be provided to the plugin during deployment.
kubectl get secrets/k8s-controller-plugin \ 
  --template={{.data.token}} | base64 --decode

Fetching the EDA Connect OpenShift plugin Helm charts

There are two ways to obtain the Helm charts to deploy the EDA Connect OpenShift plugin:

  1. Using the EDA Playground, which was used to install EDA, you can clone the Github repository: 
    make download-connect-k8s-helm-charts
  2. Downloading the release tarball and unpacking it: 
    curl -sLO https://github.com/nokia-eda/connect-k8s-helm-charts/archive/refs/tags/3.0.0.tar.gz 
tar zxf 3.0.0.tar.gz

Deploying the plugin in OpenShift

The following procedures are used on the OpenShift cluster in which the plugin needs to be deployed.

Creating a namespace for the OpenShift plugin

The OpenShift plugin uses its own namespace to separate it from other resources in the OpenShift cluster. You can create the correct namespace using the following command:
kubectl create namespace eda-connect-k8s-controller

Configuring a Pull Secret for the Controller Image

If the EDA Connect OpenShift plugin Controller image is hosted in a registry that requires authentication, a Kubernetes Secret needs to be created for OpenShift to be able to pull the image.

The following command does so for the officially hosted image with a secure read-only token to the registry.
export PULL_TOKEN=<PULL_TOKEN> 
kubectl create secret docker-registry eda-k8s-image-secret \ 
  --docker-server=ghcr.io/nokia-eda/eda-connect-k8s-controller \ 
  --docker-username=nokia-eda-bot \ 
  --docker-password=${PULL_TOKEN} \ 
  -n eda-connect-k8s-controller

Setting up the local Helm values

Create a helm-values.yaml file with the following content and update the fields as appropriate:
controllerEnvConfig: 
  connectpluginname: eda-openshift-controller-plugin 
  heartbeat: "10" 
  namespace: <eda-namespace> # The namespace of the Service Account in EDA 
  skiptlsverify: False 
  loglevel: info 
  tlscertificatedata: <EDA K8s certificate data can be extracted from kubeconfig from cluster `certificate-authority-data`> 
  tlsenabled: True 
controllerEnvSecret: 
  connectHost: https://<EDA-k8s-cluster-ip or hostname>:<port> # (Caution - Do not use EDA API values, use EDA k8s API values) 
  connectPassword: <Secret long-lived token of the service account created before> 
  connectUsername: <Name of the service account based on which secret token was created>

The following table describes possible Helm values:

Table 2. Possible Helm values
Helm value Description
connectpluginname

A name for the plugin. Ensure this is a unique name within your EDA environment.
heartbeat

The interval in seconds at which the plugin should send heartbeats. Good values are 10-30, lower values can cause extra unnecessary load on the system.
namespace

A name of a namespace, where EDA Connect service is deployed.
skiptlsverify

This value can be enabled to disable server TLS certificate verification when connecting to the EDA Kubernetes cluster.
tlscertificatedata

When certificate validation is enabled, this property can contain the certificate information of the EDA Kubernetes cluster, similar to what a kubeconfig would contain. This is only needed if certificate validation is enabled and if the EDA Kubernetes certificate has not been signed by a trusted authority.

tlsenabled

This value should always be set to true to ensure TLS is used to secure the communication with the EDA Kubernetes cluster.
connectHost

The URL used to reach the EDA Kubernetes cluster API.
connectPassword

The long-lived token created in Creating a service account token in Creating a service account and token.
connectUsername

The service account name for the account created in Creating a service account in Creating a service account and token.

Deploying the OpenShift plugin using its Helm charts

You can now deploy the EDA Connect OpenShift plugin using its Helm charts with the following command:
helm install eda-k8s connect-k8s-helm-charts/ \
  -n eda-connect-k8s-controller \
  -f helm-values.yaml \
  --set controller.imagePullSecretName=eda-k8s-image-secret

Deployment verification

You can verify if the plugin was deployed by checking if the controller is running in the OpenShift Cluster.
$ kubectl get pods -n connect-k8s-controller
NAME                                            READY   STATUS  RESTARTS AGE
connect-k8s-controller-manager-c8d4875bc-bpzrx  2/2     Running 0        66m
On the EDA Kubernetes environment you can verify the plugin has been registered.
$ kubectl get connectplugins
NAME                                   PROVIDED NAME           PLUGIN TYPE   AGE
470e9af1-b85b-439b-b81a-ab71a7166bb0   k8s-controller-plugin   KUBERNETES    2h

Operational modes

The OpenShift Connect plugin operates in the following operational modes:
  • Transparent operational mode
  • Connect Network Definition operational mode
  • NAD annotation operational mode

Using the Transparent operational mode

To use the Transparent operational mode, create network attachment definitions (NADs) in OpenShift without any EDA Connect annotations or any reference to the NAD in a Connect Network Definition (CND).

By doing so, the plugin creates a new EDA BridgeDomain for each NAD with a unique master (+VLAN) interface. If a NAD is created for which a NAD with the same master interface and VLAN already exists, it is associated with the existing BridgeDomain.

When you remove the NAD, the EDA bridge domain is also removed.

Using the Connect Network Definition operational mode

Connect Network Definition (CND) is a custom resource definition (CRD) that is added to the OpenShift cluster on deployment of the plugin.

A CND contains a design of all the network services and configuration an application may need. It can be used to define EDA Routers and EDA BridgeDomain resources. For each BridgeDomain, it is possible to associate one or more NADs with it. By doing so, the plugin knows how to connect applications into different network services.

A preProvisionHostGroupSelector can be used to pre-provision the Connect interface for a NAD interface on a set of selected hosts, regardless of whether they are consumed by any pod.
Note:

The preProvisionHostGroupSelector attribute is only applicable to IPVLAN and MACVLAN types of NetworkAttachmentDefinitions.

You can use the CMS-managed integration mode, the EDA-managed integration mode, or a combination of both.

Multiple CNDs can exist, for instance one per application.

Using the ConnectNetworkDefinition

The CND identifies how NADs are associated with different BridgeDomain resources. Multiple NADs can be associated with the same BridgeDomain.

Multiple NADs in one BridgeDomain

The following is a sample configuration of the CND usage for multiple NetworkAttachmentDefinitions to be residing in a single subnet, when they belong to different master interfaces. This is an example of OpenShift Managed Mode.

apiVersion: connect.eda.nokia.com/v1
kind: ConnectNetworkDefinition
metadata:
  name: cnd1
spec:
  subnets:
    - name: "subnet1"
      networkAttachmentDefinitions:
        - name: ipvlan-ns1/ipvlan-nad1
        - name: ipvlan-ns1/ipvlan-nad2

Multiple NADs in one BridgeDomain with VLAN Trunking

The following is a sample configuration of the CND usage for multiple NetworkAttachmentDefinitions to be residing in a single subnet, while using trunk VLANs.

apiVersion: connect.eda.nokia.com/v1
kind: ConnectNetworkDefinition
metadata:
  name: cnd2
spec:
  subnets:
    - name: "trunked-subnet1"
      networkAttachmentDefinitions:
        - name: ipvlan-ns1/ipvlan-nad1-untagged
          trunkVlans:
            - 10
        - name: sriov-ns1/sriov-nad1-untagged
          trunkVlans:
            - 20

Using EDA-managed

The following is a sample configuration of the CND usage for multiple NetworkAttachmentDefinitions to be part of a single BridgeDomain that was pre-created in EDA.

apiVersion: connect.eda.nokia.com/v1
kind: ConnectNetworkDefinition
metadata:
  name: cnd3
spec:
  subnets:
    - name: "eda-managed-subnet1"
      linkedBridgeDomain: eda_bridgedomain_1
      networkAttachmentDefinitions:
        - name: ipvlan-ns1/ipvlan-nad1
        - name: sriov-ns1/sriov-nad1

Using preProvisionHostGroupSelector to pre-provision Connect interface with a label selector

To use the preProvisionHostGroupSelector attribute in the CND, ensure that the OpenShift/K8s nodes are labeled correctly and the same value is used in the CND. The following example shows the labeling of a node.
kubectl label nodes node-1 connect.eda.nokia.com/hostGroup=net-group-1
kubectl label nodes node-2 connect.eda.nokia.com/hostGroup=net-group-2

The following sample configuration shows the pre-configuration of the Connect interface in the CND.

apiVersion: connect.eda.nokia.com/v1
kind: ConnectNetworkDefinition
metadata:
  name: cnd4
spec:
  subnets:
    - name: "subnet-1"
      networkAttachmentDefinitions:
        - name: ipvlan-ns1/ipvlan-nad1
          preProvisionHostGroupSelector: 'net-group-1'
        - name: macvlan-ns1/macvlan-nad1
          preProvisionHostGroupSelector: 'net-group-2'

Using the NAD Annotation operational mode

The NAD Annotation operational mode only works for the EDA-managed integration mode because it relies on an annotation on the NAD that identifies the pre-existing EDA BridgeDomain resource to which the NAD needs to be associated with.

To use this operational mode, when creating or updating a NAD, add the following annotation to it:
'connect.eda.nokia.com/bridgedomain': <eda-bridge-domain-name>
In case of VLAN trunking, a more complex annotation can be used, for example:
'connect.eda.nokia.com/bridgedomain': <eda-bridge-domain-name>:<vlan-id>, <eda-bridge-domain-name-2>:<vlan-id>

Audit

The EDA Cloud Connect Plugins are listening to or directly interacting with the cloud platforms they manage. This connection can be broken temporarily or be out of sync. To automatically fix these out of sync events, Audit can be run on the Plugin.

An Audit can be launched through the UI, by navigating to System Administration > Connect > Audit.

Figure 1. Launch an audit
Alternatively, you can create an Audit resource in the Kubernetes cluster of EDA with the following content:
apiVersion: connect.eda.nokia.com/v1
kind: ConnectAudit
metadata:
  name: test-audit
  namespace: eda
spec:
  connectPluginName: <name of the plugin>
  scope: PLUGIN

Audit result

An Audit runs out of band inside the Plugin. This process can be followed using the .status.state field, which progresses from Scheduled to InProgress to Finished. When the Audit is finished, the spec.finished field will be true.

The following shows an example outout.
status:
  endTime: "2024-12-16T13:37:56Z"
  enqueueTime: "2024-12-16T13:37:56Z"
  outcome: Success
  results:
  - auditType: ConnectPluginAudit
    foundDiscrepancies:
    - connectResourceKind: BridgeDomain
      connectResourceName: 4030b313-60c5-4256-8135-57833913ce67
      outcome: Success
      pluginResourceKind: vlan on Distributed Virtual Portgroup
      type: Missing
    - connectResourceKind: Vlan
      connectResourceName: 33b01228-b522-434c-b099-26ff69ec57c4
      outcome: Success
      pluginResourceKind: vlan on Distributed Virtual Portgroup
      type: Dangling
    - connectResourceKind: Vlan
      connectResourceName: 0c7f00d6-5d0a-469f-85a7-6733e457df8c
      outcome: Success
      pluginResourceKind: vlan on Distributed Virtual Portgroup
      type: Missing
    outcome: Success
    state: Finished
  state: Finished
  totalNumberOfDiscrepancies: 3
  totalNumberOfSuccessfulDiscrepancies: 3

An Audit Status consists of one or more results, each one referencing a different part of the Audit. In the case of PLUGINAudit, the only stage is the Plugin auditing against EDA, so typically there will be only one Result. The Result has an AuditType to indicate the stage of the Audit, as well as an outcome. The outcome is Success if the Audit was able to correct any found discrepancies.

If there are any discrepancies found, they are listed in the foundDiscrepancies list, detailing what resources were involved and what the taken action is. Dangling resources are resources left in EDA that are not available in the Plugin environment, while missing resources are the opposite. Finally, Misconfigured resources are available in both environments, but have one or more misconfigured fields.

A count is provided of the totalNumberOfDiscrepancies as well as successfully and failed fixes.

Troubleshooting

Controller plugin not running

Verify the following items:

  • Service Account Token configuration is correct.
  • Connectivity between controller pod in OpenShift and the EDA cluster.
  • Heartbeat interval is a non-negative integer.
  • Plugin name must comply with this regex check: '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]'. It may only contain alphanumerical characters and '.', '_', '-' (dot, underscore, and dash) and must start and end with an alphanumerical character.

Nothing is created in EDA

Verify the following items:

  • Plugin controller can access the NMstate API and NetworkAttachmentDefinition API on the OpenShift cluster.
  • Plugin can reach EDA cluster correctly.

Plugin is not configuring the correct state

Verify the following items:

  • Inspect EDA resources, such as VLAN, BridgeDomain and ConnectInterface.
  • Check the logs of the plugin pod.