ExtendedDaemonSet

ExtendedDaemonSet aims to provide a new implementation of the Kubernetes DaemonSet resource with key features:

• Canary Deployment: Deploy a new DaemonSet version with only a few nodes.
• Custom Rolling Update: Improve the default rolling update logic available in Kubernetes batch/v1 Daemonset.

How to use it

Deployment

To use the ExtendedDaemonSet controller in your Kubernetes cluster, only two commands are required:

First, deploy the Custom Resources Definitions:

$ make install

Then deploy default manifest (uses Kustomize)

$ make deploy

By default, the controller only watches the ExtendedDaemonSet resources that are present in its own namespace. If you want to deploy the controller cluster wide, add a Kustomization to the config/manager

            env:
            - name: WATCH_NAMESPACE
              value: ""

Alternatively, you can use this helm chart to deploy:

helm repo add datadog https://helm.datadoghq.com
helm repo update
helm install eds datadog/extendeddaemonset

Demo application

If you want to test and compare the advantages of the ExtendedDaemonSet over the the standard DaemonSet, you can use the demo application available in the /example folder. Follow the below scenario:

First, you need a Kubernetes cluster with several nodes; we recommend using three nodes. If you want, you can use kind.sigs.k8s.io to create a three node cluster with the following command: kind create cluster --config examples/kind-cluster-configuration.yaml.

This creates a three node cluster with one control-plane and two worker nodes:

$ kind create cluster --config examples/kind-cluster-configuration.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 ✓ Preparing nodes 📦📦📦
 ✓ Creating kubeadm config 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Cluster creation complete. You can now use the cluster with:

ExtendedDaemonSet controller deployment

# deploy the controller needed crds
$ make install

# deploy the controller pod
$ make deploy

# you should see the extendeddaemonset controller pod running
$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
extendeddaemonset-855cd7c679-gpmql   1/1     Running   0          2m11s

foo ExtendedDaemonSet deployment

Create the foo app with the ExtendedDaemonSet. For demo purposes, we'll use the registry.k8s.io/pause Docker image, which is only awaiting a terminating signal. You can look at the foo application definition in the file examples/foo-eds_v1.yaml.

$ kubectl apply -f examples/foo-eds_v1.yaml
extendeddaemonset.datadoghq.com/foo created

You can see the state of the ExtendedDaemonSet foo with:

$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS    ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Running   foo-8z7lr               44s

# Also the `extendeddaemonsetreplicaset` resource generated by the controller from the `foo` EDS instance:
$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-8z7lr   active   3         3         3       3                           61s

foo ExtendedDaemonSet deployment update with canary strategy

Now we can try to update the ExtendedDaemonSet foo. The only difference between the two versions is the Docker image used in the pod template.

$ diff examples/foo-eds_v1.yaml examples/foo-eds_v2.yaml
17c17
<         image: registry.k8s.io/pause:3.0
---
>         image: registry.k8s.io/pause:3.1

$ kubectl apply -f examples/foo-eds_v2.yaml
extendeddaemonset.datadoghq.com/foo configured

As you can see with the following command, a canary ReplicaSet is now configured for the foo ExtendedDaemonSet. Additionally, a new ExtendedReplicaSet has been created to handle the new foo pod template version.

$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS   ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Canary   foo-8z7lr   foo-xdj4b   85s

$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-8z7lr   active   2         2         2       2                           2m
foo-xdj4b   canary   1         1         1       1                           40s

$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME                                 READY   STATUS    RESTARTS   AGE
foo-8z7lr-bp9w8                      1/1     Running   0          108s
foo-8z7lr-jlvrq                      1/1     Running   0          88s
foo-xdj4b-zvss2                      1/1     Running   0          8s

Only one pod is running with the ExtendedReplicaSet foo-xdj4b pod template version. This corresponds to the setting spec.canary.replicas in the ExtendedDaemonSet foo.

Rolling update after the canary deployment validation period ended

After 5 minutes, which corresponds to spec.canary.duration, the controller will set as valid and activate the foo-xdj4b ExtendedReplicaSet. It will trigger the full foo-xdj4b ExtendedReplicaSet deployment.

$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS    ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Running   foo-xdj4b               9m21s

$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-xdj4b   active   3         3         3       3                           8m21s

$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME              READY   STATUS    RESTARTS   AGE
foo-xdj4b-hh6d8   1/1     Running   0          5m11s
foo-xdj4b-rgtk9   1/1     Running   0          5m31s
foo-xdj4b-zvss2   1/1     Running   0          10m

Overwrite container's Pod resources for a specific Node

The ExtendedDaemonset controller allows to overwrite the container's pod managed by an ExtendedDaemonset for a specific Node, thanks to an annotation that you can set on the Node: resources.extendeddaemonset.datadoghq.com/<eds-namespace>.<eds-name>.<container-name>={...}. the value corresponds to the Resources definition in JSON.

For example, for the ExtendedDaemonset named foo in the bar namespace. The container myapp resources specification can be overwriten by adding the following annotation on a Node:

$ kubectl annotate node <node-name> `resources.extendeddaemonset.datadoghq.com/bar.foo.myapp={"requests":{"cpu":"2.0","memory":"2G"}}`
node/<node-name> annotated

Overwrite container's Pod resources for a set of Nodes with ExtendedDaemonsetSettings

In some cases (for example with different nodes type), it can be useful to have different resource configurations for a Daemonset to handle the Node's workload specificity.

To do so you can create an instance of ExtendedDaemonsetSetting resource that aims to overwrite the resources definition of the container(s) present in ExtendedDaemonset Pods.

the information needed is:

• spec.nodeSelector: a NodeLabels selector that matches with the nodes where it must trigger the usage of this resource.
• spec.reference: contains enough information to let you identify the referred resource.
• spec.containers: contains a list of Container spec overwrites.

apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonsetSetting
metadata:
  name: foo-xxl-node
spec:
  nodeSelector:
    matchLabels:
      node-type: xxl
  reference:
    kind: ExtendedDaemonset
    name: foo
  containers:
   - name: daemon
    resources:
      requests:
        cpu: "0.5"
        memory: "300m"

Remove a pod on a given node using nodeAffinity

In some cases, it could be useful to remove a daemon pod on a given node. This can be done using the podTemplate.spec.affinity.nodeAffinity field.

First set a new requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms field

apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonSet
metadata:
  name: foo
spec:
  template:
    spec:
      //...
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: extendeddaemonset.datadoghq.com/exclude
                operator: NotIn
                values:
                - foo

Then add the label extendeddaemonset.datadoghq.com/exclude=foo to the node in question

kubectl label nodes <your-node-name> extendeddaemonset.datadoghq.com/exclude=foo

Canary settings

The Canary deployment can be customized in a few ways.

• replicas: The number of replica pods to participate in the Canary deployment
• duration: The duration of the Canary deployment, after which the Canary deployment will end and the active ExtendedReplicaSet will update
• autoPause.enabled: Activation of the Canary deployment auto pausing feature (default is true)
• autoPause.maxRestarts: The maximum number of restarts tolerable before the Canary deployment is automatically paused (default is 2)
• validationMode: Used to configure how a canary deployment is validated. Possible values are auto (default) and manual. In manual mode canary w