llm-inference-scaling
Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.
Install / Use
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-inference-scalingInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of llm-inference-scaling
llm-inference-scaling scores 87/100 on our quality scale, 509th of 950 AI & Machine Learning skills we index.
Its SKILL.md is 7.7 KB long, well organised into 25 sections with 7 code examples: a thorough specification that gives an agent plenty to work with.
With 1,113 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
- Our last check on 2026-10-11 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 98/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-07. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
llm-inference-scaling compared with similar skills
All 4 of these similar skills score higher than llm-inference-scaling; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| llm-inference-scaling (this skill)by BagelHole | 87 | 1.1k | 6mo ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 92.8k | 1d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
Frequently asked questions
- How do I install llm-inference-scaling?
- Run
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-inference-scaling. The install tabs above show the steps for each supported agent. - Which AI agents does llm-inference-scaling work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is llm-inference-scaling safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 98/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is llm-inference-scaling still maintained?
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
Skill content
View source on GitHubname: llm-inference-scaling description: Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads. license: MIT metadata: author: devops-skills version: "1.0"
LLM Inference Scaling
Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
When to Use This Skill
Use this skill when:
- LLM API traffic is unpredictable and you need to scale up/down automatically
- Managing a fleet of vLLM or TGI inference pods on Kubernetes
- Reducing inference costs with spot/preemptible GPU instances
- Implementing queue-based autoscaling for batch inference jobs
- Building a multi-model serving platform that shares GPU resources
Prerequisites
- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
- KEDA (Kubernetes Event-Driven Autoscaler) installed
- Prometheus with GPU metrics (
dcgm-exporterorgpu-operator) - Helm 3+ for chart deployments
GPU Node Setup
# Install NVIDIA GPU Operator (handles drivers, container toolkit, DCGM)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
--namespace gpu-operator \
--create-namespace \
--set driver.enabled=true \
--set dcgm.enabled=true \
--set devicePlugin.enabled=true
# Verify GPU nodes are recognized
kubectl get nodes -l nvidia.com/gpu.present=true
kubectl describe node <gpu-node> | grep nvidia
vLLM Deployment with GPU Resources
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama-8b
labels:
app: vllm
model: llama-3.1-8b
spec:
replicas: 1
selector:
matchLabels:
app: vllm
model: llama-3.1-8b
template:
metadata:
labels:
app: vllm
model: llama-3.1-8b
spec:
nodeSelector:
nvidia.com/gpu.present: "true"
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: vllm
image: vllm/vllm-openai:latest
args:
- "--model"
- "meta-llama/Llama-3.1-8B-Instruct"
- "--tensor-parallel-size"
- "1"
- "--gpu-memory-utilization"
- "0.90"
- "--max-num-seqs"
- "128"
resources:
requests:
nvidia.com/gpu: "1"
memory: "20Gi"
cpu: "4"
limits:
nvidia.com/gpu: "1"
memory: "24Gi"
cpu: "8"
ports:
- containerPort: 8000
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: hf-token
key: token
KEDA Autoscaling on Prometheus Metrics
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-scaledobject
spec:
scaleTargetRef:
name: vllm-llama-8b
minReplicaCount: 1
maxReplicaCount: 8
cooldownPeriod: 300 # 5 min before scale-down
pollingInterval: 15
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus-server.monitoring:9090
metricName: vllm_num_requests_waiting
threshold: "10" # scale up if >10 requests waiting
query: |
sum(vllm:num_requests_waiting{deployment="vllm-llama-8b"})
- type: prometheus
metadata:
serverAddress: http://prometheus-server.monitoring:9090
metricName: vllm_gpu_cache_usage
threshold: "0.8" # scale up if KV cache >80% full
query: |
avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama-8b"})
Queue-Based Scaling (Redis + KEDA)
# ScaledJob for async batch inference
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: llm-batch-inference
spec:
jobTargetRef:
template:
spec:
containers:
- name: inference-worker
image: myapp/inference-worker:latest
env:
- name: REDIS_URL
value: redis://redis:6379
- name: QUEUE_NAME
value: inference-jobs
restartPolicy: OnFailure
minReplicaCount: 0
maxReplicaCount: 20
pollingInterval: 5
successfulJobsHistoryLimit: 3
triggers:
- type: redis
metadata:
address: redis:6379
listName: inference-jobs
listLength: "5" # 1 worker per 5 queued jobs
Spot Instance Strategy
# Mixed node pool: on-demand + spot GPUs
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-autoscaler-priority-config
data:
priorities: |
10: # low priority = prefer
- .*spot.*
50:
- .*on-demand.*
---
# Node affinity for spot with on-demand fallback
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 80
preference:
matchExpressions:
- key: node.kubernetes.io/lifecycle
operator: In
values: [spot]
- weight: 20
preference:
matchExpressions:
- key: node.kubernetes.io/lifecycle
operator: In
values: [on-demand]
Cluster Autoscaler for GPU Nodes
# AWS EKS — enable cluster autoscaler for GPU node group
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
--namespace kube-system \
--set autoDiscovery.clusterName=my-cluster \
--set awsRegion=us-east-1 \
--set rbac.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::ACCOUNT:role/ClusterAutoscalerRole \
--set extraArgs.skip-nodes-with-local-storage=false \
--set extraArgs.expander=least-waste
# Annotate GPU node group for autoscaler
kubectl annotate node <node> \
cluster-autoscaler.kubernetes.io/safe-to-evict="false"
Scaling Metrics to Monitor
# Prometheus queries for scaling decisions
# Requests waiting in vLLM queue
sum(vllm:num_requests_waiting) by (model)
# GPU KV cache utilization (>80% = bottleneck)
avg(vllm:gpu_cache_usage_perc) by (pod)
# Tokens per second throughput
sum(rate(vllm:generation_tokens_total[5m])) by (model)
# P99 time-to-first-token
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
Common Issues
| Issue | Cause | Fix |
|-------|-------|-----|
| Pods stuck in Pending | No GPU nodes available | Check cluster autoscaler logs; verify node group limits |
| Scale-up too slow | Cluster autoscaler delay + model load time | Pre-warm replicas; increase minReplicaCount |
| GPU fragmentation | Multiple small models on large GPUs | Use MIG partitioning or consolidate model sizes |
| Spot eviction causes errors | Spot instance reclamation | Add PodDisruptionBudget; use graceful shutdown |
| KEDA not scaling | Prometheus query returns no data | Test query in Prometheus UI first |
Best Practices
- Set
minReplicaCount: 1to avoid cold starts; scale to 0 only for batch jobs. - Use
PodDisruptionBudgetwithminAvailable: 1to survive spot evictions. - Pre-pull model weights into a shared PVC to speed up pod startup by 5–10×.
- Separate model families across node pools (A10G for 7B, A100 for 70B).
- Use Kubernetes VPA for CPU/memory right-sizing alongside KEDA for replica count.
Related Skills
- vllm-server - vLLM configuration and tuning
- gpu-server-management - GPU node setup
- model-serving-kubernetes - KServe
- kubernetes-ops - Core Kubernetes
- llm-cost-optimization - Cost strategies
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Understand-Anything
92.8kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
