[![Build Status][ci-img]][ci] [![Go Report Card][goreport-img]][goreport] [![Docker Image][docker-img]][docker]

KubeMod

KubeMod is a universal Kubernetes mutating operator.

It introduces ModRule - a custom Kubernetes resource that can intercept the deployment of any Kubernetes object and apply targeted modifications to it, or reject it before it is deployed to the cluster.

Use KubeMod to:

• Customize opaque Helm charts and Kubernetes operators.
• Build a system of policy rules to reject misbehaving resources.
• Develop your own sidecar container injections - no coding required.
• Derive metadata from external manifests.
• ...exercise your imagination :)

Table of contents

• Installation
• Deploying our first ModRule
• Common use cases
  • Modification of behavior
  • Modification of metadata
  • Sidecar injection
  • Resource rejection
• Anatomy of a ModRule
  • Match section
  • Patch section
• Miscellaneous
  • Operation type
  • Execution tiers
  • Namespaced and cluster-wide resources
  • Synthetic references
  • Target resources
  • Note on idempotency
  • Debugging ModRules
  • KubeMod's version of JSONPath
  • Declarative kubectl apply

---

Installation

KubeMod requires Kubernetes 1.21 or later, architecture AMD64 or ARM64.

As a Kubernetes operator, KubeMod is deployed into its own namespace — kubemod-system.

Run the following commands to deploy KubeMod.

# Make KubeMod ignore Kubernetes' system namespace.
kubectl label namespace kube-system admission.kubemod.io/ignore=true --overwrite
# Deploy KubeMod.
kubectl apply -f https://raw.githubusercontent.com/kubemod/kubemod/v0.20.0/bundle.yaml

By default KubeMod allows you to target a limited set of high-level resource types, such as deployments and services.

See target resources for the full list as well as instructions on how to expand or limit it.

Upgrade

If you are upgrading from a previous version of KubeMod, run the following:

# Delete the KubeMod certificate generation job in case KubeMod has already been installed.
kubectl delete job kubemod-crt-job -n kubemod-system
# Make KubeMod ignore Kubernetes' system namespace.
kubectl label namespace kube-system admission.kubemod.io/ignore=true --overwrite
# Upgrade KubeMod operator.
kubectl apply -f https://raw.githubusercontent.com/kubemod/kubemod/v0.20.0/bundle.yaml

Uninstall

To uninstall KubeMod and all its resources, run:

kubectl delete -f https://raw.githubusercontent.com/kubemod/kubemod/v0.20.0/bundle.yaml

Note: Uninstalling KubeMod will also remove all your ModRules deployed to all Kubernetes namespaces.

Deploying our first ModRule

Once KubeMod is installed, you can deploy ModRules to intercept the creation and update of specific resources and perform modifications on them.

For example, here's a ModRule which intercepts the creation of Deployment resources whose app labels equal nginx and include at least one container of nginx version 1.14.*.

The ModRule patches the matching Deployments on-the-fly to enforce a specific securityContext and add annotation my-annotation.

Since KubeMod intercepts and patches resources before they are deployed to Kubernetes, we are able to patch read-only fields such as securityContext without the need to drop and recreate existing resources.

apiVersion: api.kubemod.io/v1beta1
kind: ModRule
metadata:
  name: my-modrule
spec:
  type: Patch

  match:
    # Match deployments ...
    - select: '$.kind'
      matchValue: Deployment

    # ... with label app = nginx ...
    - select: '$.metadata.labels.app'
      matchValue: 'nginx'

    # ... and at least one container whose image matches nginx:1.14.* ...
    - select: '$.spec.template.spec.containers[*].image'
      matchRegex: 'nginx:1\.14\..*'

    # ... but has no explicit runAsNonRoot security context.
    # Note: "negate: true" flips the meaning of the match.
    - select: '$.spec.template.spec.securityContext.runAsNonRoot == true'
      negate: true

  patch:
    # Add custom annotation.
    - op: add
      path: /metadata/annotations/my-annotation
      value: whatever

    # Enforce non-root securityContext and make nginx run as user 101.
    - op: add
      path: /spec/template/spec/securityContext
      value: |-
        fsGroup: 101
        runAsGroup: 101
        runAsUser: 101
        runAsNonRoot: true

Save the above ModRule to file my-modrule.yaml and deploy it to the default namespace of your Kubernetes cluster:

kubectl apply -f my-modrule.yaml

After the ModRule is created, the creation of any nginx Kubernetes Deployment resource in the same namespace will be intercepted by KubeMod, and if the Deployment resource matches the ModRule's match section, the resource will be patched with the collection of patch operations.

To list all ModRules deployed to a namespace, run the following:

kubectl get modrules

Common use cases

The development of KubeMod was motivated by the proliferation of Kubernetes Operators and Helm charts which are sometimes opaque to customizations and lead to runtime issues.

For example, consider these issues:

• https://github.com/elastic/cloud-on-k8s/issues/2328
• https://github.com/jaegertracing/jaeger-operator/issues/1096

Oftentimes these issues are showstoppers that render the chart/operator impossible to use for certain use cases.

With the help of KubeMod we can make those charts and operators work for us. Just deploy a ModRule which targets the problematic primitive resource and patch it on the fly at the time it is created.

See the following sections for a number of typical use cases for KubeMod.

Modification of behavior

Here's a typical black-box operator issue which can be fixed with KubeMod: https://github.com/elastic/cloud-on-k8s/issues/2328.

The issue is that when the Elastic Search operator creates Persistent Volume Claims, it attaches an ownerReference to them such that they are garbage-collected after the operator removes the Elastic Search stack of resources.

This makes sense when we plan to dynamically scale Elastic Search up and down, but it doesn't make sense if we don't plan to scale dynamically, but we do want to keep the Elastic Search indexes during Elastic Search reinstallation (see comments here and here).

A solution to this issue would be the following ModRule which simply removes the ownerReference from PVCs created by the Elastic Search operator at the time they are deployed, thus excluding those resources from Kubernetes garbage collection:

apiVersion: api.kubemod.io/v1beta1
kind: ModRule
metadata:
  name: my-modrule
spec:
  type: Patch

  matches:
    # Match persistent volume claims ...
    - select: '$.kind'
      matchValue: PersistentVolumeClaim

    # ... created by the elasticsearch operator.
    - select: '$.metadata.labels["common.k8s.elastic.co/type"]'
      matchValue: elasticsearch

  patch:
    # Remove the ownerReference if it exists, thus excluding the resource from Kubernetes garbage collection.
    - op: remove
      path: /metadata/ownerReferences/0

Modification of metadata

With the help of ModRules, one can dynamically modify the resources generated by one operator such that another operator can detect those resources.

For example, Istio's sidecar injection can be controlled by pod annotation sidecar.istio.io/inject. If another operator creates a deployment which we want to explicitly exclude from Istio's injection mechanism, we can create a ModRule which modifies that deployment by adding this annotation with value "false".

The following ModRule explicitly excludes the Jaeger collector deployment created by the Jaeger Operator from Istio sidecar injection:

apiVersion: api.kubemod.io/v1beta1
kind: ModRule
metadata:
  name: my-modrule
spec:
  type: Patch

  match:
    # Match deployments ...
    - select: '$.kind'
      matchValue: Deployment

    # ... with label app = jaeger ...
    - select: '$.metadata.labels.app'
      matchValue: jaeger

    # ... and label app.kubernetes.io/component = collector ...
    - select: '$.metadata.labels["app.kubernetes.io/component"]'
      matchValue: collector

    # ... but with and no annotation sidecar.istio.io/inject.
    - select: '$.metadata.annotations["sidecar.istio.io/inject"]'
      negate: true

  patch:
    # Add Istio annotation sidecar.istio.io/inject=false to exclude this deployment from Istio injection.
    - op: add
      path: /metadata/annotations/sidecar.istio.io~1inject
      value: '"false"'

Sidecar injection

With the help of ModRules, one can dynamically inject arbitrary sidecar containers into Deployment and StatefulSet resources. The patch part of the ModRule is a Golang template which takes the tar