vllm-server
Deploy and manage vLLM for high-throughput LLM inference. Configure continuous batching, tensor parallelism, quantization, and OpenAI-compatible API endpoints for production LLM serving.
Install / Use
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill vllm-serverInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of vllm-server
vllm-server scores 87/100 on our quality scale, 510th of 947 AI & Machine Learning skills we index.
Its SKILL.md is 6.1 KB long, well organised into 36 sections with 10 code examples: a thorough specification that gives an agent plenty to work with.
With 1,113 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
- Our last check on 2026-11-02 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 98/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-07. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
vllm-server compared with similar skills
All 4 of these similar skills score higher than vllm-server; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| vllm-server (this skill)by BagelHole | 87 | 1.1k | 6mo ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 92.8k | 1d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
Frequently asked questions
- How do I install vllm-server?
- Run
npx skills add BagelHole/DevOps-Security-Agent-Skills --skill vllm-server. The install tabs above show the steps for each supported agent. - Which AI agents does vllm-server work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is vllm-server safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 98/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is vllm-server still maintained?
- The repository was last updated about 6 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
Skill content
View source on GitHubname: vllm-server description: Deploy and manage vLLM for high-throughput LLM inference. Configure continuous batching, tensor parallelism, quantization, and OpenAI-compatible API endpoints for production LLM serving. license: MIT metadata: author: devops-skills version: "1.0"
vLLM Server Management
Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching.
When to Use This Skill
Use this skill when:
- Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale
- Building an OpenAI-compatible API endpoint for self-hosted models
- Optimizing LLM throughput and latency for production traffic
- Running multi-GPU inference with tensor or pipeline parallelism
- Deploying quantized models to reduce GPU memory requirements
Prerequisites
- NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production)
- Docker or Python 3.9+ with pip
- 40GB+ VRAM for 70B models; 8GB+ for 7B models
nvidia-container-toolkitfor Docker GPU passthrough
Quick Start
# Install vLLM
pip install vllm
# Serve a model (OpenAI-compatible API)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--api-key your-secret-key
# Test the endpoint
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-secret-key" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Docker Deployment
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.1-8B-Instruct \
--api-key your-secret-key
Docker Compose (Production)
services:
vllm:
image: vllm/vllm-openai:latest
runtime: nvidia
environment:
- NVIDIA_VISIBLE_DEVICES=all
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
volumes:
- model-cache:/root/.cache/huggingface
ports:
- "8000:8000"
ipc: host
command: >
--model meta-llama/Llama-3.1-70B-Instruct
--tensor-parallel-size 2
--max-model-len 32768
--gpu-memory-utilization 0.90
--api-key ${VLLM_API_KEY}
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
volumes:
model-cache:
Key Configuration Options
Multi-GPU Tensor Parallelism
# Split one model across 4 GPUs
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.90
Quantization (Lower VRAM)
# AWQ quantization (70B on 2x A100 40GB)
vllm serve casperhansen/llama-3-70b-instruct-awq \
--quantization awq \
--tensor-parallel-size 2
# GPTQ quantization
vllm serve TheBloke/Llama-2-70B-Chat-GPTQ \
--quantization gptq
# FP8 (H100 NVL native)
vllm serve meta-llama/Llama-3.1-405B-Instruct \
--quantization fp8 \
--tensor-parallel-size 8
Structured Output & Tools
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--enable-auto-tool-choice \
--tool-call-parser llama3_json \
--guided-decoding-backend outlines
LoRA Adapters
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--enable-lora \
--lora-modules sql-lora=/path/to/sql-lora \
code-lora=/path/to/code-lora \
--max-lora-rank 64
Performance Tuning
# Maximize throughput for batch workloads
vllm serve <model> \
--max-num-seqs 256 \ # max concurrent sequences
--max-num-batched-tokens 8192 \ # tokens per batch
--gpu-memory-utilization 0.95 \ # use 95% VRAM
--swap-space 4 # CPU swap (GiB)
# Minimize latency for interactive use
vllm serve <model> \
--max-num-seqs 32 \
--enforce-eager # disable CUDA graph capture
Benchmarking
# Install benchmark tool
pip install vllm
# Run throughput benchmark
python -m vllm.entrypoints.openai.run_batch \
--model meta-llama/Llama-3.1-8B-Instruct \
--input-file prompts.jsonl \
--output-file results.jsonl
# Benchmark with vllm bench
vllm bench throughput \
--model meta-llama/Llama-3.1-8B-Instruct \
--num-prompts 1000 \
--input-len 512 \
--output-len 128
Monitoring
# Check running server stats
curl http://localhost:8000/metrics # Prometheus metrics
# Key metrics to watch:
# vllm:num_requests_running - active requests
# vllm:gpu_cache_usage_perc - KV cache utilization
# vllm:generation_tokens_per_s - throughput
# vllm:time_to_first_token_ms - TTFT latency
# vllm:e2e_request_latency_seconds - end-to-end latency
Common Issues
| Issue | Cause | Fix |
|-------|-------|-----|
| CUDA out of memory | Model too large for VRAM | Add --quantization awq or reduce --gpu-memory-utilization |
| Slow cold start | Model not cached | Pre-pull with huggingface-cli download <model> |
| Low throughput | Too few concurrent requests | Increase --max-num-seqs |
| KV cache full errors | Context length too long | Set --max-model-len lower |
| tokenizer error | Tokenizer mismatch | Use --tokenizer to specify correct tokenizer |
Best Practices
- Use
--gpu-memory-utilization 0.90to leave headroom for CUDA kernels. - Pin model versions with
--revisionfor reproducible deployments. - Set
HF_HUB_OFFLINE=1in production to prevent unexpected downloads. - Use AWQ or GPTQ quantization before tensor parallelism — lower VRAM first.
- Enable
--enable-chunked-prefillfor long-context workloads. - Monitor
gpu_cache_usage_perc— above 95% causes queuing.
Related Skills
- llm-inference-scaling - Auto-scaling vLLM deployments
- gpu-server-management - GPU driver setup
- llm-gateway - Load balancing across vLLM instances
- llm-cost-optimization - Cost management
- model-serving-kubernetes - K8s deployment
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Understand-Anything
92.8kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
