⛵ flatops

A GitOps-managed Kubernetes homelab cluster running on Talos Linux.

📋 Overview

This repository contains the declarative configuration for kantai, a bare-metal Kubernetes cluster. The cluster is designed for home infrastructure workloads with a focus on:

GitOps-driven operations via FluxCD
Advanced networking with Cilium, Envoy Gateway, external-dns, Cloudflare, and cert-manager
Distributed storage using Rook-Ceph
GPU workloads with NVIDIA GPU Operator
Comprehensive observability using VictoriaMetrics and Grafana
Continuous integration via Renovate

🏗️ Cluster Architecture

Nodes

| Node | Role | Hardware | |------|------|----------| | kantai1 | Hyper-converged control plane and workloads | <ul><li>AMD EPYC 7443P, 256 GiB</li><li>NVIDIA RTX 4000 Ada Generation, 24 GB</li><li>Micron 9300 PRO, 4 TB, x7</li><li>Seagate Exos X20, 18 TB, x15</li><li>NVIDIA ConnectX-5</li><li>LSI 9500-8e</li><li>45Drives HL-15</li></ul> | | kantai2 | Virtual arm64 control plane and workloads | <ul><li>Apple M2 Mac Mini, 16 GB (mem), 500 GB (block)</li><li>UTM + QEMU hypervisor</li></ul> | | kantai3 | Hyper-converged control plane and workloads | <ul><li>AMD Ryzen Embedded V1500B, 32 GB</li><li>NVIDIA T400, 4 GB</li><li>Seagate Exos X18, 18 TB, x6</li><li>NVIDIA ConnectX-3</li><li>QNAP TS-673A</li></ul> |

Network

kantai is connected to an all-Ubiquiti network, with a Hi-Capacity Aggregation as the TOR and a Dream Machine Pro as the gateway/router/firewall. Recent versions of Unifi Network and Unifi OS support BGP, which is used to advertise load balancer addresses and thus provide node-balanced services to the network. The cluster's virtual network is dual-stack IPv4 and IPv6.

The cluster uses kantai.xyz as its public domain. It is registered at Cloudflare which also acts as the DNS authority. Cloudflare also proxies requests for services available from the public internet and tunnels them to the cluster for DDOS and privacy protection.

The cluster integrates with a Tailscale tailnet for private secure global access.

IPv4

Cluster nodes are connected to the main Ubiquiti network which uses 10.1.0.0/16.
Cilium advertises routes to load-balanced services using BGP.
A Unifi network matching the load balancer CIDR is programmed to prevent unnecessary NAT hairpinning and allow flows through the firewall.
Cilium masquerades pod addresses to node addresses.

| Role | CIDR | |-----------------|----------------| | Pod | 10.11.0.0/16 | | Service | 10.11.0.0/16 | | Cilium LB IPAM | 10.11.0.0/16 |

IPv6

For IPv6 networking, I decided to use globally routable addresses for pods, services, and LB IPAM. This means no masquerading is necessary, which is more in the spirit of IPv6. Routes and firewalls must still be programmed for traffic to flow.

Cluster nodes are connected to the main Ubiquiti network which receives an IPv6 /64 prefix via prefix delegation and assigns addresses to clients via SLAAC.
3 additional /64 prefixes are manually reserved for pods, services, and Cilium LB IPAM.
Cilium advertises routes to load-balanced services using BGP (same as IPv4).
A Unifi network matching the load balancer CIDR is programmed to prevent unnecessary NAT hairpinning and allow flows through the firewall (same as IPv4).
IPv6 masquerading is disabled.

🔧 Core Components

GitOps & Cluster Management

FluxCD

The cluster is managed entirely through GitOps using FluxCD. All resources are declared in this repository and automatically reconciled to the cluster. The Flux Operator manages the FluxCD instance.

Kustomizations define the desired state of each application
HelmReleases manage Helm chart deployments
OCIRepositories pull charts from OCI registries
Drift detection ensures cluster state matches Git

tuppr

Automated Talos and Kubernetes upgrades are managed by tuppr. Upgrade CRDs (TalosUpgrade, KubernetesUpgrade) define version targets with health checks that ensure VolSync backups complete and Ceph cluster health is OK before proceeding.

Renovate

The repository is constantly updated using Renovate and flux-local. Minor and patch updates are applied automatically while major releases require human approval.

Networking

Cilium

Cilium serves as the CNI in kube-proxy replacement mode, providing:

eBPF-based networking with native routing
BGP control plane for advertising load-balanced services to the Unifi gateway
LoadBalancer IP Address Management to assign routable addresses to load-balanced services
Network policies for pod-level traffic control
Bandwidth Manager with BBR for bandwith and congestion control

Envoy Gateway

Envoy Gateway provides a complete and up-to-date implemenmtation of the Kubernetes Gateway API with advanced extensions.

An external Gateway is used for routes that should be available from the public internet (via a Cloudflare Tunnel), while an internal Gateway is used for routes that should only be accessible on the local network or on my tailnet.

external-dns

external-dns automatically manages DNS records for services:

Cloudflare for external Gateway routes
UniFi for internal Gateway routes using @kashalls's excellent Unifi provider

Tailscale

The Tailscale Operator integrates the cluster with my tailnet.

API Server Proxy - The Kubernetes API server is accessible over the tailnet via Tailscale's API server proxy in auth mode, enabling API server access with tailnet authn/authz.
Split-Horizon DNS - A k8s-gateway deployment serves as a kantai.xyz split-horizon DNS server on the tailnet for all HTTPRoute resources with a kantai.xyz hostname, making them resolvable on the tailnet (but not reachable since the Envoy Gateway services use the Cilium BGP LoadBalancer class; see next). The k8s-gateway service itself is exposed to the tailnet using a Tailscale load balancer service.
The Unifi gateway is connected to the tailnet and programmed as a subnet router for the Cilium BGP LoadBalancer's IPv4 CIDR, making all such services reachable over the tailnet.

Multus

Multus CNI enables attaching multiple network interfaces to pods. Used for workloads requiring direct LAN access via macvlan interfaces with dual-stack networking support.

Secrets Management

external-secrets + 1Password

external-secrets synchronizes secrets from 1Password into Kubernetes using the 1Password Connect server. A ClusterSecretStore provides cluster-wide access to secrets.

Certificate Management

cert-manager + trust-manager

cert-manager automates certificate lifecycle management:

Maintains a wildcard certificate for kantai.xyz using Let's Encrypt DNS challenge (Cloudflare API)
trust-manager distributes CA bundles across namespaces

Identity & Authentication

Pocket ID

Pocket ID serves as the in-cluster OIDC provider, enabling:

Kubernetes API server OIDC authentication
OIDC authentication for apps that do not natively support it via Envoy Gateway's SecurityPolicy extension
Centralized identity management for applications

Flatops

Install / Use

README