llm-gateway
Deploy an API gateway for LLM traffic with load balancing, rate limiting, key management, semantic caching, fallback routing, and cost tracking. Covers LiteLLM Proxy, OpenRouter-compatible setup, and custom Nginx/Traefik patterns.
Install / Use
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-gatewayInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of llm-gateway
llm-gateway scores 87/100 on our quality scale, 511th of 947 AI & Machine Learning skills we index.
Its SKILL.md is 7.8 KB long, well organised into 21 sections with 6 code examples: a thorough specification that gives an agent plenty to work with.
With 1,113 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
- Our last check on 2026-11-02 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 98/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-07. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
llm-gateway compared with similar skills
All 4 of these similar skills score higher than llm-gateway; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| llm-gateway (this skill)by BagelHole | 87 | 1.1k | 6mo ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 92.8k | 1d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
Frequently asked questions
- How do I install llm-gateway?
- Run
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-gateway. The install tabs above show the steps for each supported agent. - Which AI agents does llm-gateway work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is llm-gateway safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 98/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is llm-gateway still maintained?
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
Skill content
View source on GitHubname: llm-gateway description: Deploy an API gateway for LLM traffic with load balancing, rate limiting, key management, semantic caching, fallback routing, and cost tracking. Covers LiteLLM Proxy, OpenRouter-compatible setup, and custom Nginx/Traefik patterns. license: MIT metadata: author: devops-skills version: "1.0"
LLM Gateway
A unified API gateway that routes LLM requests across providers and self-hosted models — with rate limiting, cost tracking, caching, and failover.
When to Use This Skill
Use this skill when:
- Running multiple LLM backends (OpenAI, Anthropic, vLLM, Ollama) behind a single endpoint
- Enforcing per-team or per-user rate limits and spend budgets
- Implementing automatic fallback when a provider is down
- Adding semantic caching to reduce API costs by 20–50%
- Centralizing API key management instead of distributing keys to every app
Prerequisites
- Docker and Docker Compose
- A PostgreSQL or SQLite database (for LiteLLM state)
- LLM API keys (OpenAI, Anthropic, etc.) or self-hosted vLLM endpoints
- Optional: Redis for caching and rate limiting
LiteLLM Proxy — Quick Start
LiteLLM is the de facto open-source LLM gateway with OpenAI-compatible API.
# Run with Docker
docker run -d \
--name litellm-proxy \
-p 4000:4000 \
-e OPENAI_API_KEY=$OPENAI_API_KEY \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-v $(pwd)/litellm-config.yaml:/app/config.yaml \
ghcr.io/berriai/litellm:main-latest \
--config /app/config.yaml \
--detailed_debug
LiteLLM Configuration
# litellm-config.yaml
model_list:
# OpenAI models
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.e…[redacted]
rpm: 10000
tpm: 2000000
- model_name: gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_key: os.e…[redacted]
# Anthropic
- model_name: claude-sonnet-4-6
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.e…[redacted]
# Self-hosted vLLM instances (load balanced)
- model_name: llama-3.1-8b
litellm_params:
model: openai/meta-llama/Llama-3.1-8B-Instruct
api_base: http://vllm-1:8000/v1
api_key: fake # vLLM key
- model_name: llama-3.1-8b
litellm_params:
model: openai/meta-llama/Llama-3.1-8B-Instruct
api_base: http://vllm-2:8000/v1 # second replica — auto load balanced
api_key: fake
# Fallback: cheap model if primary fails
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o-mini # fallback to cheaper model
api_key: os.e…[redacted]
router_settings:
routing_strategy: least-busy # or: latency-based, simple-shuffle
num_retries: 3
retry_after: 5
allowed_fails: 2
cooldown_time: 60
# Fallback configuration
fallbacks:
- gpt-4o: [claude-sonnet-4-6]
- claude-sonnet-4-6: [gpt-4o]
litellm_settings:
# Semantic caching
cache: true
cache_params:
type: redis
host: redis
port: 6379
similarity_threshold: 0.90 # cache if >90% semantic similarity
# Logging
success_callback: ["langfuse"]
failure_callback: ["langfuse"]
langfuse_public_key: os.environ/LANGFUSE_PUBLIC_KEY
langfuse_secret_key: os.e…[redacted]
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: postgresql://litellm:password@postgres:5432/litellm
store_model_in_db: true
Docker Compose: Full Gateway Stack
services:
litellm:
image: ghcr.io/berriai/litellm:main-latest
command: ["--config", "/app/config.yaml", "--port", "4000"]
volumes:
- ./litellm-config.yaml:/app/config.yaml
ports:
- "4000:4000"
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
- DATABASE_URL=postgresql://litellm:password@postgres:5432/litellm
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_started
restart: unless-stopped
postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: litellm
POSTGRES_USER: litellm
POSTGRES_PASSWORD: password
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm"]
interval: 5s
retries: 5
restart: unless-stopped
redis:
image: redis:7-alpine
command: redis-server --maxmemory 2gb --maxmemory-policy allkeys-lru
volumes:
- redis-data:/data
restart: unless-stopped
volumes:
postgres-data:
redis-data:
Virtual Keys & Rate Limiting
# Create a virtual API key for a team (via LiteLLM API)
curl -X POST http://localhost:4000/key/generate \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"team_id": "team-backend",
"key_alias": "backend-team-key",
"models": ["gpt-4o-mini", "llama-3.1-8b"],
"max_budget": 100, # USD limit
"budget_duration": "monthly",
"rpm_limit": 100, # requests per minute
"tpm_limit": 500000 # tokens per minute
}'
# View spend
curl http://localhost:4000/spend/keys \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
Nginx Load Balancer (Alternative/Complement)
# nginx.conf — round-robin across vLLM replicas
upstream vllm_backends {
least_conn;
server vllm-1:8000 max_fails=3 fail_timeout=30s;
server vllm-2:8000 max_fails=3 fail_timeout=30s;
server vllm-3:8000 max_fails=3 fail_timeout=30s;
keepalive 32;
}
server {
listen 80;
server_name llm-api.internal;
# Rate limiting
limit_req_zone $http_authorization zone=per_key:10m rate=100r/m;
limit_req zone=per_key burst=20 nodelay;
location /v1/ {
proxy_pass http://vllm_backends;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_read_timeout 300s; # long timeout for streaming
proxy_buffering off; # required for SSE streaming
proxy_cache_bypass 1;
}
}
Monitoring Gateway Health
# Check LiteLLM health
curl http://localhost:4000/health
# Model-level health
curl http://localhost:4000/health/liveliness
# Spend by model
curl http://localhost:4000/spend/models \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
# Active virtual keys
curl http://localhost:4000/key/list \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
Common Issues
| Issue | Cause | Fix |
|-------|-------|-----|
| ConnectionRefusedError to backend | Backend not reachable | Check api_base URL; verify backend is healthy |
| Rate limit errors (429) | Budget/RPM exceeded | Increase limits or rotate to fallback model |
| Slow streaming responses | proxy_buffering enabled | Set proxy_buffering off in Nginx |
| Cache miss rate high | Threshold too strict | Lower similarity_threshold to 0.85 |
| Postgres connection errors | DB not ready | Add depends_on with condition: service_healthy |
Best Practices
- Use virtual keys per team/app — never expose raw provider API keys.
- Enable
cache: truewith Redis for repeated or similar queries; can cut costs 30–50%. - Set
num_retries: 3with fallbacks to handle provider outages gracefully. - Log all requests to Langfuse or OpenTelemetry for cost attribution and debugging.
- Use
least-busyrouting strategy for self-hosted models to avoid GPU saturation.
Related Skills
- vllm-server - Backend inference server
- llm-inference-scaling - Auto-scaling backends
- llm-caching - Semantic cache patterns
- llm-cost-optimization - Cost management
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Understand-Anything
92.8kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
