<div align="center">

Kashall's Infrastructure

        

                       

</div>

What is this?

This is the repository I use to version control my kubernetes cluster I deploy and maintain at home. I currently use Talos to provide a secure, minimal and immutable environment for Kubernetes. Previous iterations of this repository relied on Debian-based Operating Systems which can lead unwanted changes in the base system.

How did you do this?

Thanks to onedr0p, there is the cluster template that allows you to easily get started with your own kubernetes cluster at home. You don't need to have multiple computers or a fancy setup to get one working.

If you're interested, you can also join the community Home Operations. Several people are involved daily and it makes for some interesting conversations.

Directory Helper

This repository uses the following layout for Kubernetes.

📁 bootstrap
├── 📝 helmfile.yaml # Helmreleases required to run bootstrap flux.
└── 📝 secrets.yaml.tpl # Secrets required to bootstrap flux.
📁 kubernetes
├── 📁 apps # Per-cluster application-specific configurations.
├── 📁 components # Flux & Talos configurations for setting up the cluster.
└── 📁 flux # Flux configuration, application repositories and more.
📁 talos
├── 📁 nodes # Override configurations for each individual node.
├── 📝 machineconfig.yaml.j2 # Base configuration for all nodes.
└── 📝 talos.env # Kubernetes and Talos Version Variables
📁 unifi # Configuration files for UniFi
📝 kubeconfig
📝 talosconfig

☁️ Cloud Dependencies

While most of my infrastructure and workloads are self-hosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not.

| Service | Use | Cost | |---------------------------------------------------------|----------------------------------------------------------------|----------------| | 1Password | Secrets with External Secrets | ~$55/yr | | Cloudflare | Domains, Workers, Pages, and R2 | ~$240/yr | | Backblaze B2 | Backups | $1/m | | GitHub | Hosting this repository and continuous integration/deployments | Free | | Let's Encrypt | Issuing SSL Certificates with Cert Manager | Free | | Migadu | Email Hosting | ~$20/yr | | Pushover | Kubernetes Alerts and application notifications | Free | | UniFi Site Manager | UniFi External Access Management | Free | | | | Total: ~$10/mo |

---

💻 Networking

Networking Diagram

flowchart LR
    classDef gateway fill:#163a1e,stroke:#27ae60,color:#fff
    classDef switch fill:#1e2a4a,stroke:#3498db,color:#fff
    classDef compute fill:#4a1e3a,stroke:#e74c3c,color:#fff
    classDef storage fill:#3a2a1e,stroke:#f39c12,color:#fff
    classDef ap fill:#1e3a2a,stroke:#2ecc71,color:#fff

    Internet(["The Internet"])

    Internet -- "2 Gbps ↓ / 350 Mbps ↑" --> UCG["UCG Fiber"]:::gateway
    UCG -- 2.5G --> FLEX["USW Flex 2.5G 8 PoE"]:::switch
    UCG -- 10G --> AGG["USW Pro Aggregation"]:::switch

    FLEX -- 2.5G --> U7XG(["U7 Pro XG (Office)"]):::ap
    FLEX -- 1G --> UAPAC(["UAP-AC-Pro (Dining Room)"]):::ap

    AGG -- 10G --> MAX["USW Pro Max 16"]:::switch
    AGG -- 20G LACP --> MS01["3x MS-01 (Talos)"]:::compute
    AGG -- 10G --> MSR1["MS-R1 (Talos)"]:::compute
    AGG -- 20G LACP --> TN["TrueNAS"]:::storage

    MAX -- 1G --> U6LR(["U6-LR (Garage)"]):::ap
    MAX -- 1G --> U7PRO(["U7-Pro (Lab)"]):::ap

Networks & Vlans

| Name | VLAN | Description | |---------------------|------|-------------------------------------------------------------------------------------| | Management | 1 | Servers + Network Management | | Devices | 2 | Wireless Devices and Workstations | | IoT | 3 | Small devices that have the potential to be compromised, so they don't get to talk to each other. | | Services | 4 | No DHCP, Simply a network for Cluster BGP | | "I Don't Trust You" | 86 | Non-affiliated organization issued devices (school or work devices) |

🌐 DNS

UniFi released a new feature update with UniFi routers that allow you to create custom dns records to be served to the whole network. I wrote External DNS Unifi Webhook to allow External DNS to gather service and ingress hosts from my clusters and deploy the records to my routers local dns server without any extra local resolvers or moving parts.

---

🔧 Hardware

<details> <summary>Click to see the rack!</summary> Updated 05/25/2024 <img src="https://owo.whats-th.is/2drDDRN.jpg" align="center" width="200px" alt="rack"/> </details>

| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose | |---------------------|-------|--------------|----------------|------|------------------|------------------| | UCG Fiber | 1 | - | 1TiB NVMe | - | UniFi OS | Router | | USW Flex 2.5G 8 PoE | 1 | - | - | - | UniFi OS | Switching | | USW Pro Max 16 PoE | 1 | - | - | - | UniFi OS | Switching | | USW Pro Aggregation | 1 | - | - | - | UniFi OS | Aggregation | | U7 Pro XG | 1 | - | - | - | - | Office AP | | U6 LR | 1 | - | - | - | - | Garage AP | | UDB Switch | 1 | - | - | - | UniFi OS | Garage Workbench | | USP-PDU-Pro | 1 | - | - | - | - | Rack PDU | | MS-01 | 3 | 1TB NVMe | 2TB PM9A3 U.2 | 96GB | Talos | Control Plane | | MS R1 | 1 | 1TB NVMe | 1TB NVMe | 64GB | Talos | Worker | | Fran | 1 | 2x1TB SSD | 5x8TB (raidz2) | 64GB | Debian | Storage | | JetKVM | 1 | 16GB (Flash) | - | - | JetKVM | Network KVM | | Eaton 5PX1500RT | 1 | -