ProxLB - (Re)Balance VM Workloads in Proxmox Clusters

<img align="left" src="https://cdn.gyptazy.com/img/ProxLB.jpg"/> <br> <p float="center"><img src="https://img.shields.io/github/license/gyptazy/ProxLB"/><img src="https://img.shields.io/github/contributors/gyptazy/ProxLB"/><img src="https://img.shields.io/github/last-commit/gyptazy/ProxLB/main"/><img src="https://img.shields.io/github/issues-raw/gyptazy/ProxLB"/><img src="https://img.shields.io/github/issues-pr/gyptazy/ProxLB"/></p>

Table of Contents

1. Introduction
2. Features
3. How does it work?
4. Documentation
5. Installation
   1. Requirements / Dependencies
   2. Debian Package
   3. Container / Docker
   4. Source
6. Usage / Configuration
   1. Proxmox HA Integration
   2. Options
7. Affinity & Anti-Affinity Rules
   1. Affinity Rules
   2. Anti-Affinity Rules
   3. Ignore VMs
   4. Pin VMs to Hypervisor Nodes
8. Maintenance
9. Misc
   1. Bugs
   2. Contributing
   3. Support
10. Enterprise-Support
11. Prox-Tools Collection
12. Author(s)

Introduction

ProxLB is an advanced load balancing solution (initially written by @gyptazy) specifically designed for Proxmox clusters, addressing the absence of an intelligent and more advanced resource scheduler. As a third-party solution, ProxLB enhances the management and efficiency of Proxmox clusters by intelligently distributing workloads across available nodes. Workloads can be balanced by different times like the guest's memory, CPU or disk usage or their assignment to avoid overprovisioning and ensuring resources.

One of the key advantages of ProxLB is that it is fully open-source and free, making it accessible for anyone to use, modify, and contribute to. This ensures transparency and fosters community-driven improvements. ProxLB supports filtering and ignoring specific nodes and guests through configuration files and API calls, providing administrators with the flexibility to tailor the load balancing behavior to their specific needs.

A standout feature of ProxLB is its maintenance mode. When enabled, all guest workloads are automatically moved to other nodes within the cluster, ensuring that a node can be safely updated, rebooted, or undergo hardware maintenance without disrupting the overall cluster operation. Additionally, ProxLB supports both affinity and anti-affinity rules, allowing operators to group multiple guests to run together on the same node or ensure that certain guests do not run on the same node, depending on the cluster's node count. This feature is crucial for optimizing performance and maintaining high availability.

ProxLB can also return the best next node for guest placement, which can be integrated into CI/CD pipelines using tools like Ansible or Terraform. This capability streamlines the deployment process and ensures efficient resource utilization. Furthermore, ProxLB leverages the Proxmox API, including the entire ACL (Access Control List) system, for secure and efficient operation. Unlike some solutions, it does not require SSH access, enhancing security and simplifying configuration.

Overall, ProxLB significantly enhances resource management by intelligently distributing workloads, reducing downtime through its maintenance mode, and providing improved flexibility with affinity and anti-affinity rules. Its seamless integration with CI/CD tools and reliance on the Proxmox API make it a robust and secure solution for optimizing Proxmox cluster performance.

Video of Migration

<img src="https://cdn.gyptazy.com/img/proxlb-rebalancing-demo.gif"/>

Features

ProxLB's key features are by enabling automatic rebalancing of VMs and CTs across a Proxmox cluster based on memory, CPU, and local disk usage while identifying optimal nodes for automation. It supports maintenance mode, affinity rules, and seamless Proxmox API integration with ACL support, offering flexible usage as a one-time operation, a daemon, or through the Proxmox Web GUI.

Features

• Rebalance VMs/CTs in the cluster by:
  • Memory
  • Disk (only local storage)
  • CPU
• Rebalance by different modes:
  • Used resources
  • Assigned resources
  • PSI (Pressure) of resources
• Get best nodes for further automation
• Supported Guest Types
  • VMs
  • CTs
• Maintenance Mode
  • Set node(s) into maintenance
  • Move all workloads to different nodes
• Affinity / Anti-Affinity Rules
• Fully based on Proxmox API
  • Fully integrated into the Proxmox ACL
  • No SSH required
• Usage
  • One-Time
  • Daemon
  • Proxmox Web GUI Integration

How does it work?

ProxLB is a load-balancing system designed to optimize the distribution of virtual machines (VMs) and containers (CTs) across a cluster. It works by first gathering resource usage metrics from all nodes in the cluster through the Proxmox API. This includes detailed resource metrics for each VM and CT on every node. ProxLB then evaluates the difference between the maximum and minimum resource usage of the nodes, referred to as "Balanciness." If this difference exceeds a predefined threshold (which is configurable), the system initiates the rebalancing process.

Before starting any migrations, ProxLB validates that rebalancing actions are necessary and beneficial. Depending on the selected balancing mode — such as CPU, memory, or disk — it creates a balancing matrix. This matrix sorts the VMs by their maximum used or assigned resources, identifying the VM with the highest usage. ProxLB then places this VM on the node with the most free resources in the selected balancing type. This process runs recursively until the operator-defined Balanciness is achieved. Balancing can be defined for the used or max. assigned resources of VMs/CTs.

Documentation

This README.md doesn't contain all information and only highlights the most important facts. Extended information, such like API permissions, creating dedicated user, best-practices in running ProxLB and much more can be found in the docs/ directory. Please consult the documentation before creating issues.

Installation

ProxLB can be installed as a Debian package or by a container image. With the provided Helm charts it can also easily be deployed into Kubernetes stacks.

Note: The Debian repository and container image registry are currently operated by @gyptazy and will be migrated to a new location in the future.

Requirements / Dependencies

• Proxmox
  • Proxmox 7.x
  • Proxmox 8.x
  • Proxmox 9.x
• Python3.x
• proxmoxer
• requests
• urllib3
• pyyaml

The dependencies can simply be installed with pip by running the following command:

pip install -r requirements.txt

Note: Distribution packages, such like the provided .deb package will automatically resolve and install all required dependencies by using already packaged version from the distribution's repository. By using the Docker (container) image or Debian packages, you do not need to take any care of the requirements listed here.

Debian Package

ProxLB is a powerful and flexible load balancer designed to work across various architectures, including amd64, arm64, rv64 and many other ones that support Python. It runs independently of the underlying hardware, making it a versatile choice for different environments. This chapter covers the step-by-step process to install ProxLB on Debian-based systems, including Debian clones like Ubuntu.

Quick-Start

You can simply use this snippet to install the repository and to install ProxLB on your system.

# Add GPG key
curl -fsSL https://packages.credativ.com/public/proxtools/public.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/proxtools-archive-keyring.gpg

# Add repository
echo "deb [signed-by=/etc/apt/keyrings/proxtools-archive-keyring.gpg] \
https://packages.credativ.com/public/proxtools stable main" \
| sudo tee /etc/apt/sources.list.d/proxlb.list

# Update & install
sudo apt-get update
sudo apt-get -y install proxlb

# Copy example config
sudo cp /etc/proxlb/proxlb_example.yaml /etc/proxlb/proxlb.yaml

# Adjust the config to your needs
sudo vi /etc/proxlb/proxlb.yaml

# Start service
sudo systemctl start proxlb

# Adjust the config to your needs
sudo vi /etc/proxlb/proxlb.yaml
sudo systemctl start proxlb

Afterwards, ProxLB is running in the background and balances your cluster by your defined balancing method (default: memory).

Details

ProxLB provides two different repositories:

• https://packages.credativ.com/public/proxtools stable main
• https://packages.credativ.com/public/proxtools snapshots main

The repository is signed and the GPG key can be found at:

• https://packages.credativ.com/public/proxtools/archive-keyring.gpg

You can also simply import it by running:

# KeyID:  34C5B9642CD591E5D090A03B062A8A3A410B831D
# UID:    Proxtools Repository Signer <info@credativ.de>
# SHA256: 4cb4a74b25f775616709eb0596eeeac61d8d28717f4872fef2d68fb558434ed3  public.key

wget -O /etc/apt/keyrings/proxtools-archive-keyring.gpg https://packages.credativ.com/public/proxtools/public.key

Container Images / Docker

Using the ProxLB container images is straight forward and only requires you to mount the config file.

Available images can be found at the GitHub packages page or Docker Hub.

# Pull the image from GHCR
docker pull ghcr.io/credativ/proxlb:latest
# or Docker Hub
docker