Following on the Using Kcli to prepare for OCM testing, we’re going to prepare KMM testing in Hub-Spoke approach.

First we need to prepare our .docker/config.json with the contents of our OpenShift pull secret used with Kcli.

mkdir -p ~/.docker/
cp openshift_pull.json ~/.docker/config.json

Warning advisories

Note

Semi-scripted version available at automate.sh

Warning

We’re using pre-release bits of the software, that’s why we need to define a custom catalog for both the Hub and the Spokes. Once KMM is released it will be available from the official one and just the Policy will be needed.

Danger

Ensure that podman login quay.io is working correctly so that we can push the built images.

Using `podman` instead of `docker`

If you’re using podman like we’ll do, we need to create a symbolic link for the command to the docker name, as the Makefile does use docker.

ln -s /usr/bin/podman /usr/bin/docker

Testing KMM Hub&Spoke on OpenShift

KMM allows OpenShift to build required Kernel modules for the nodes, for example to take advantage of certain network cards or hardware that is required for operations.

Dependencies

quay.io account repository accessibility

if the repositories are not created beforehand, please make sure to convert them to public ones after pushing to them

Setup the Hub

  1. Build and push the KMM-Hub container image, the KMM-Hub OLM bundle and catalog from the midstream repository:

    MYUSER=iranzo
    
    git clone [email protected]:rh-ecosystem-edge/kernel-module-management.git
    cd kernel-module-management
    
     # Build KMM-Hub container image
    
     HUB_IMG=quay.io/${MYUSER}/kernel-module-management-hub:latest make docker-build-hub
    
     # Push KMM-Hub container image
    
     HUB_IMG=quay.io/${MYUSER}/kernel-module-management-hub:latest make docker-push-hub
    
     # Generate KMM-Hub OLM bundle
    
     HUB_IMG=quay.io/${MYUSER}/kernel-module-management-hub:latest BUNDLE_IMG=quay.io/${MYUSER}/kernel-module-management-bundle-hub make bundle-hub
    
     # Build KMM-Hub OLM bundle container
    
     BUNDLE_IMG=quay.io/${MYUSER}/kernel-module-management-bundle-hub make bundle-build-hub
    
     # Push KMM-Hub OLM bundle container
    
     BUNDLE_IMG=quay.io/${MYUSER}/kernel-module-management-bundle-hub make bundle-push
    
     # Build KMM-Hub OLM catalog
    
     CATALOG_IMG=quay.io/${MYUSER}/kernel-module-management-catalog-hub:latest BUNDLE_IMGS=quay.io/${MYUSER}/kernel-module-management-bundle-hub make catalog-build
    
     # Push KMM-Hub OLM catalog
    
     CATALOG_IMG=quay.io/${MYUSER}/kernel-module-management-catalog-hub:latest make catalog-push
    
  2. Deploy the KMM-Hub OLM catalog on the Hub:

    As mentioned earlier we need to perform this step as we’re consuming pre-release bits, so the images are not available in the regular OpenShift catalog.

    cat <<EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: kmm-hub-catalog
      namespace: openshift-marketplace
    spec:
      sourceType: grpc
      image: quay.io/${MYUSER}/kernel-module-management-catalog-hub:latest
      displayName: KMM Hub Catalog
      publisher: ${MYUSER}
      updateStrategy:
        registryPoll:
          interval: 5m
    EOF
    
  3. Deploy the KMM-Hub operator by creating an OLM Subscription:

    cat <<EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: kmm-operator
      namespace: openshift-operators
    spec:
      channel: alpha
      installPlanApproval: Automatic
      name: kernel-module-management-hub
      source: kmm-hub-catalog
      sourceNamespace: openshift-marketplace
    EOF
    
  4. Deploy the ACM Policy that adds the required permissions to the Spoke klusterlet, in order for the latter to be able to CRUD KMM Module CRs:

    cat <<EOF | oc apply -f -
    ---
    apiVersion: policy.open-cluster-management.io/v1
    kind: Policy
    metadata:
      name: allow-klusterlet-deploy-kmm-modules
    spec:
      remediationAction: enforce
      disabled: false
      policy-templates:
        - objectDefinition:
            apiVersion: policy.open-cluster-management.io/v1
            kind: ConfigurationPolicy
            metadata:
              name: klusterlet-deploy-modules
            spec:
              severity: high
              object-templates:
              - complianceType: mustonlyhave
                objectDefinition:
                  apiVersion: rbac.authorization.k8s.io/v1
                  kind: ClusterRole
                  metadata:
                    name: kmm-module-manager
                  rules:
                    - apiGroups: [kmm.sigs.x-k8s.io]
                      resources: [modules]
                      verbs: [create, delete, get, list, patch, update, watch]
              - complianceType: mustonlyhave
                objectDefinition:
                  apiVersion: rbac.authorization.k8s.io/v1
                  kind: RoleBinding
                  metadata:
                    name: klusterlet-kmm
                    namespace: open-cluster-management-agent
                  subjects:
                  - kind: ServiceAccount
                    name: klusterlet-work-sa
                    namespace: open-cluster-management-agent
                  roleRef:
                    kind: ClusterRole
                    name: kmm-module-manager
                    apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: apps.open-cluster-management.io/v1
    kind: PlacementRule
    metadata:
      name: all-clusters-except-local
    spec:
      clusterSelector:
        matchExpressions:
          - key: name
            operator: NotIn
            values:
              - local-cluster
    ---
    apiVersion: policy.open-cluster-management.io/v1
    kind: PlacementBinding
    metadata:
      name: bind-klusterlet-kmm-all-clusters
    placementRef:
      apiGroup: apps.open-cluster-management.io
      kind: PlacementRule
      name: all-clusters-except-local
    subjects:
      - apiGroup: policy.open-cluster-management.io
        kind: Policy
        name: allow-klusterlet-deploy-kmm-modules
    EOF
    
  5. Create a namespace on the Hub, in which the Build and Sign jobs will be created:

    oc create ns kmm-jobs-test
    

Setup the Spoke

  1. Build and push the KMM container image, the KMM OLM bundle and catalog from midstream repository:

    Again, we need to perform this step as we’re consuming pre-release bits, so the images are not available in the regular OpenShift catalog.

      # Build KMM container image
      IMG=quay.io/${MYUSER}/kernel-module-management:latest make docker-build
    
      # Push KMM container image
      IMG=quay.io/${MYUSER}/kernel-module-management:latest make docker-push
    
      # Generate KMM OLM bundle
      IMG=quay.io/${MYUSER}/kernel-module-management:latest BUNDLE_IMG=quay.io/${MYUSER}/kernel-module-management-bundle make bundle
    
      # Build KMM OLM bundle container
      BUNDLE_IMG=quay.io/${MYUSER}/kernel-module-management-bundle make bundle-build
    
      # Push KMM OLM bundle container
      BUNDLE_IMG=quay.io/${MYUSER}/kernel-module-management-bundle make bundle-push
    
      # Build KMM OLM catalog
      CATALOG_IMG=quay.io/${MYUSER}/kernel-module-management-catalog:latest BUNDLE_IMGS=quay.io/${MYUSER}/kernel-module-management-bundle make catalog-build
    
      # Push KMM OLM catalog
      CATALOG_IMG=quay.io/${MYUSER}/kernel-module-management-catalog:latest make catalog-push
    
  2. Deploy the KMM catalog on the Spoke:

    cat <<EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: kmm-catalog
      namespace: openshift-marketplace
    spec:
      sourceType: grpc
      image: quay.io/${MYUSER}/kernel-module-management-catalog:latest
      displayName: KMM Catalog
      publisher: ${MYUSER}
      updateStrategy:
        registryPoll:
          interval: 5m
    EOF
    
  3. Deploy the KMM operator by creating an OLM Subscription:

    cat <<EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: kernel-module-management
      namespace: openshift-operators
    spec:
      channel: alpha
      config:
        env:
        - name: KMM_MANAGED
          value: "1"
      installPlanApproval: Automatic
      name: kernel-module-management
      source: kmm-catalog
      sourceNamespace: openshift-marketplace
    EOF
    
  4. Create a namespace on the Spoke, in which the Module will be created

    oc create ns kmm-tests
    

Create a ManagedClusterModule on the Hub

The following ManagedClusteModule includes a selector for the default clusterset.

Warning

In this example, we’ve created manually a docker secret in order to allow KMM-Hub to push the newly generated module container image to my quay.io repository. For example:

cat <<EOF | oc apply -f -
apiVersion: v1
data:
  .dockerconfigjson: THISISWHEREMYSECRET GOES
kind: Secret
metadata:
  name: ${MYUSER}-quay-cred
  namespace: kmm-jobs-test
type: kubernetes.io/dockerconfigjson
EOF

Note

Pay attention to the cluster selector in the example below:

selector:
  cluster.open-cluster-management.io/clusterset: default

So that it targets one of your clusters.

  1. Deploy a ManagedClusterModule to test the setup e2e:

    And now, the manifest itself:

    cat <<EOF | oc apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: mod-example
      namespace: kmm-jobs-test
    data:
      dockerfile: |
        FROM image-registry.openshift-image-registry.svc:5000/openshift/driver-toolkit as builder
        ARG KERNEL_VERSION
        ARG MY_MODULE
        WORKDIR /build
        RUN git clone https://github.com/cdvultur/kmm-kmod.git
        WORKDIR /build/kmm-kmod
        RUN cp kmm_ci_a.c mod-example.c
        RUN make
    
        FROM registry.redhat.io/ubi8/ubi-minimal
        ARG KERNEL_VERSION
        ARG MY_MODULE
        RUN microdnf -y install kmod
        COPY --from=builder /usr/bin/kmod /usr/bin/
        RUN for link in /usr/bin/modprobe /usr/bin/rmmod; do \
            ln -s /usr/bin/kmod "\$link"; done
        COPY --from=builder /etc/driver-toolkit-release.json /etc/
        COPY --from=builder /build/kmm-kmod/*.ko /opt/lib/modules/\${KERNEL_VERSION}/
        RUN depmod -b /opt \${KERNEL_VERSION}
    ---
    apiVersion: hub.kmm.sigs.x-k8s.io/v1beta1
    kind: ManagedClusterModule
    metadata:
      name: mod-example
    spec:
      spokeNamespace: kmm-tests
      jobNamespace: kmm-jobs-test
      selector:
        cluster.open-cluster-management.io/clusterset: default
      moduleSpec:
        imageRepoSecret:
          name: ${MYUSER}-quay-cred
        moduleLoader:
          container:
            modprobe:
              moduleName: mod-example
            imagePullPolicy: Always
            kernelMappings:
            - regexp: '^.+$'
              containerImage: quay.io/${MYUSER}/module:\${KERNEL_FULL_VERSION}
              pull:
                insecure: true
                insecureSkipTLSVerify: true
              build:
                buildArgs:
                  - name: MY_MODULE
                    value: mod-example.o
                dockerfileConfigMap:
                  name: mod-example
        selector:
          node-role.kubernetes.io/worker: ""
    EOF
    
  2. Check the respective build on the Hub:

    oc get builds -n kmm-jobs-test
    
  3. After the build is completed, the respective ManifestWork will be created:

    oc get manifestworks -n <spoke-cluster-name>
    
  4. Check whether the KMM Module is created on the Spoke:

    oc get modules -n kmm-tests
    
  5. Check the KMM ModuleLoader DaemonSet on the Spoke:

    oc get ds -n kmm-tests