Kubernetes Reference Deployment

<!-- markdown-link-check-disable-next-line -->

<!-- markdown-link-check-disable-next-line -->

Summary

KRD (Kubernetes Reference Deployment) is a framework for easily deploying Kubernetes clusters on bare-metal or virtual machines using Ansible. It provides a configurable and extensible platform for validating Cloud Native technologies across various Linux distributions.

A central concept in KRD is the use of the POD Descriptor File (PDF)—a YAML file that defines your cluster configuration. You can start with the default configuration or explore the sample setups for advanced scenarios.

Supported Linux Distributions

| Distribution | Version | | :----------- | :---------: | | Ubuntu | 20.04/22.04 | | openSUSE | Leap | | Fedora | 37/38/39/40 |

Included Components

KRD integrates a wide range of Kubernetes-related components for testing, benchmarking, or production experimentation:

| Component | Description | Source | Status | | :-------------------: | :---------------------------------- | :--------------------- | :------------: | | Kubernetes | Base Kubernetes deployment | [kubespray][1] | ✅ Done | | Kubevirt | VM workloads on Kubernetes | | ✅ Tested | | Virtink | VM workloads on Kubernetes | | ✅ Tested | | Multus | Multi-network pod support | | ✅ Tested | | NFD | Node Feature Discovery | [nfd role][4] | ✅ Tested | | Istio | Service Mesh | | ✅ Tested | | PMEM | Persistent Memory CSI | [pmem role][6] | 🚧 Implemented | | QAT | Intel QuickAssist support | [qat_plugin role][8] | ✅ Tested | | SR-IOV | SR-IOV Plugin | [sriov_plugin role][9] | 🚧 Implemented | | Knative | Serverless platform | | ✅ Tested | | Kong | Ingress Controller | | ✅ Tested | | HAProxy | Ingress Controller | | ✅ Tested | | MetalLB | Network load balancer | | ✅ Tested | | Falco | Security runtime monitoring | | ✅ Tested | | Rook | Storage Operator | | ✅ Tested | | OPA Gatekeeper | Policy Enforcement | | ✅ Tested | | Kyverno | Policy Enforcement | | ✅ Tested | | Kubewarden | Policy Enforcement | | ✅ Tested | | kube-monkey | Chaos engineering for Kubernetes | | 🚧 Implemented | | Nephio | Telco-native orchestration | | 🚧 Implemented | | LocalAI | Local LLM inference support | | 🚧 Implemented | | K8SGPT | Kubernetes diagnostics | | 🚧 Implemented | | ArgoCD | GitOps & CI/CD workflows | | 🚧 Implemented | | Tekton | GitOps & CI/CD workflows | | 🚧 Implemented | | Longhorn | Storage Operator | | ✅ Tested | | TopoLVM | CSI storage with topology awareness | | ✅ Tested | | Fluent | Logging agent | | ✅ Tested | | CloudNativePG | PostgreSQL operator | | ✅ Tested | | LiteLLM | Local LLM inference support | | 🚧 Implemented | | kagent | KRM-native agentic AI platform | | 🚧 Implemented | | kube-prometheus-stack | Prometheus stack | | 🚧 Implemented |

Quick Deployment

Use the All-in-One script for a streamlined, unattended deployment:

curl -fsSL http://bit.ly/KRDaio | bash

This will clone the repository, install dependencies, and provision a default cluster setup.

Customizing your Cluster

KRD supports custom cluster definitions using the PDF (YAML) format. Vagrant uses the default configuration, but this can be overridden via a config/pdf.yml file.

You can also visualize your cluster topology using:

tox -e diagram

Installing Add-Ons

Use KRD command script to install optional components on an existing cluster.

./krd_command.sh -h

To install Kubernetes add-ons:

KRD_ADDONS_LIST=nfd ./krd_command.sh -a install_k8s_addons

Note: Some add-ons include validation scripts in the tests directory.

Day-2 Operations

KRD supports full lifecycle operations, including upgrades, node addition, and service removal.

Example: Upgrade your Kubernetes cluster to version v1.18.10 using Kubespray v2.14.2:

KRD_KUBE_VERSION=v1.18.10 KRD_KUBESPRAY_VERSION=v2.14.2 ./krd_command.sh -a upgrade_k8s

Environment Configuration

KRD relies on environment variables to control behavior and customization. Below are some of the most relevant variables.

Global Variables

| Name | Default | Description | | :------------------------------------ | :--------------------------------------------- | :------------------------------------------------------------------------------ | | KRD_DEBUG | false | Enables verbose execution | | KRD_ANSIBLE_DEBUG | false | Enables Ansible verbose execution | | PKG_DEBUG | false | Enables cURL package installer verbose execution | | KRD_KUBE_VERSION | v1.33.7 | Specifies the Kubernetes version to be upgraded | | KRD_KUBESPRAY_VERSION | v2.30.0 | Specifies the Kubespray version to be used during the upgrade process | | KRD_KUBESPRAY_REPO | https://github.com/kubernetes-sigs/kubespray | Specifies the Git repository to fetch the Kubespray's source code | | KRD_ENABLE_TESTS | false | Enables the functional tests during the deployment process | | KRD_HELM_CHART | | Specifies the Helm chart to be installed | | KRD_FOLDER | /opt/krd | KRD source code destination folder | | KRD_ACTIONS_LIST | install_k8s | Specifies a list of KRD actions to be installed during the All-in-One execution | | KRD_ADDONS_LIST | addons | Specifies the Kubernetes AddOns to be installed by install_k8s_addons | | KRD_HUGEPAGES_ENABLED | true | Enable/Disable [HugeTLB Pages][17] on cluster nodes | | KRD_QAT_PLUGIN_MODE | dpdk | Specifies the deployment mode for Intel QuickAssist Plugin | | KRD_KUBESPHERE_DEVOPS_ENABLED | true | Enable/Disable KubeSphere DevOps System | | KRD_KUBESPHERE_METRICS_SERVER_ENABLED | false | Enable/Disable [Horizontal Pod Autoscaler][15] on KubeSphere | | KRD_KUBESPHERE_SERVICEMESH_ENABLED | false | Enable/Disable KubeSphere Service Mesh | | KRD_ENABLE_ISTIO_ADDONS | false | Enable/Disable Istio AddOns(Grafana, Kiali and Prometheus) | | KRD_METALLB_ADDRESS_POOLS_LIST |