SkillAgentSearch skills...

llama-cpp

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

Install / Use

npx skills add Orchestra-Research/AI-Research-SKILLs --skill llama-cpp

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

93/100

Supported Platforms

Universal

Our assessment of llama-cpp

llama-cpp scores 93/100 on our quality scale, 226th of 947 AI & Machine Learning skills we index (top 24%).

Its SKILL.md is 5.8 KB long, well organised into 52 sections with 11 code examples: a solid amount of guidance for an agent.

With 14,543 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
26/30
Structure
20/20
Description
15/15
Adoption
18/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated 36 days ago, so llama-cpp is actively maintained.
  • Our last check on 2026-10-25 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-09-26. Automated pattern scan on 2026-09-26. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

llama-cpp compared with similar skills

All 4 of these similar skills score higher than llama-cpp; compare them before choosing.

SkillScoreStarsUpdatedFormat
llama-cpp (this skill)by Orchestra-Research9314.5k36d agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
Understand-Anythingby Egonex-AI10092.8k1d agoCLAUDE.md
headroomby headroomlabs-ai10079.8ktodayCLAUDE.md

Frequently asked questions

How do I install llama-cpp?
Run npx skills add Orchestra-Research/AI-Research-SKILLs --skill llama-cpp. The install tabs above show the steps for each supported agent.
Which AI agents does llama-cpp work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is llama-cpp safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is llama-cpp still maintained?
The repository was last updated 36 days ago, so llama-cpp is actively maintained.

name: llama-cpp description: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU. version: 1.0.0 author: Orchestra Research license: MIT tags: [Inference Serving, Llama.cpp, CPU Inference, Apple Silicon, Edge Deployment, GGUF, Quantization, Non-NVIDIA, AMD GPUs, Intel GPUs, Embedded] dependencies: [llama-cpp-python]

llama.cpp

Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.

When to use llama.cpp

Use llama.cpp when:

  • Running on CPU-only machines
  • Deploying on Apple Silicon (M1/M2/M3/M4)
  • Using AMD or Intel GPUs (no CUDA)
  • Edge deployment (Raspberry Pi, embedded systems)
  • Need simple deployment without Docker/Python

Use TensorRT-LLM instead when:

  • Have NVIDIA GPUs (A100/H100)
  • Need maximum throughput (100K+ tok/s)
  • Running in datacenter with CUDA

Use vLLM instead when:

  • Have NVIDIA GPUs
  • Need Python-first API
  • Want PagedAttention

Quick start

Installation

# macOS/Linux
brew install llama.cpp

# Or build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# With Metal (Apple Silicon)
make LLAMA_METAL=1

# With CUDA (NVIDIA)
make LLAMA_CUDA=1

# With ROCm (AMD)
make LLAMA_HIP=1

Download model

# Download from HuggingFace (GGUF format)
huggingface-cli download \
    TheBloke/Llama-2-7B-Chat-GGUF \
    llama-2-7b-chat.Q4_K_M.gguf \
    --local-dir models/

# Or convert from HuggingFace
python convert_hf_to_gguf.py models/llama-2-7b-chat/

Run inference

# Simple chat
./llama-cli \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    -p "Explain quantum computing" \
    -n 256  # Max tokens

# Interactive chat
./llama-cli \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --interactive

Server mode

# Start OpenAI-compatible server
./llama-server \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -ngl 32  # Offload 32 layers to GPU

# Client request
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-2-7b-chat",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.7,
    "max_tokens": 100
  }'

Quantization formats

GGUF format overview

| Format | Bits | Size (7B) | Speed | Quality | Use Case | |--------|------|-----------|-------|---------|----------| | Q4_K_M | 4.5 | 4.1 GB | Fast | Good | Recommended default | | Q4_K_S | 4.3 | 3.9 GB | Faster | Lower | Speed critical | | Q5_K_M | 5.5 | 4.8 GB | Medium | Better | Quality critical | | Q6_K | 6.5 | 5.5 GB | Slower | Best | Maximum quality | | Q8_0 | 8.0 | 7.0 GB | Slow | Excellent | Minimal degradation | | Q2_K | 2.5 | 2.7 GB | Fastest | Poor | Testing only |

Choosing quantization

# General use (balanced)
Q4_K_M  # 4-bit, medium quality

# Maximum speed (more degradation)
Q2_K or Q3_K_M

# Maximum quality (slower)
Q6_K or Q8_0

# Very large models (70B, 405B)
Q3_K_M or Q4_K_S  # Lower bits to fit in memory

Hardware acceleration

Apple Silicon (Metal)

# Build with Metal
make LLAMA_METAL=1

# Run with GPU acceleration (automatic)
./llama-cli -m model.gguf -ngl 999  # Offload all layers

# Performance: M3 Max 40-60 tokens/sec (Llama 2-7B Q4_K_M)

NVIDIA GPUs (CUDA)

# Build with CUDA
make LLAMA_CUDA=1

# Offload layers to GPU
./llama-cli -m model.gguf -ngl 35  # Offload 35/40 layers

# Hybrid CPU+GPU for large models
./llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20  # GPU: 20 layers, CPU: rest

AMD GPUs (ROCm)

# Build with ROCm
make LLAMA_HIP=1

# Run with AMD GPU
./llama-cli -m model.gguf -ngl 999

Common patterns

Batch processing

# Process multiple prompts from file
cat prompts.txt | ./llama-cli \
    -m model.gguf \
    --batch-size 512 \
    -n 100

Constrained generation

# JSON output with grammar
./llama-cli \
    -m model.gguf \
    -p "Generate a person: " \
    --grammar-file grammars/json.gbnf

# Outputs valid JSON only

Context size

# Increase context (default 512)
./llama-cli \
    -m model.gguf \
    -c 4096  # 4K context window

# Very long context (if model supports)
./llama-cli -m model.gguf -c 32768  # 32K context

Performance benchmarks

CPU performance (Llama 2-7B Q4_K_M)

| CPU | Threads | Speed | Cost | |-----|---------|-------|------| | Apple M3 Max | 16 | 50 tok/s | $0 (local) | | AMD Ryzen 9 7950X | 32 | 35 tok/s | $0.50/hour | | Intel i9-13900K | 32 | 30 tok/s | $0.40/hour | | AWS c7i.16xlarge | 64 | 40 tok/s | $2.88/hour |

GPU acceleration (Llama 2-7B Q4_K_M)

| GPU | Speed | vs CPU | Cost | |-----|-------|--------|------| | NVIDIA RTX 4090 | 120 tok/s | 3-4× | $0 (local) | | NVIDIA A10 | 80 tok/s | 2-3× | $1.00/hour | | AMD MI250 | 70 tok/s | 2× | $2.00/hour | | Apple M3 Max (Metal) | 50 tok/s | ~Same | $0 (local) |

Supported models

LLaMA family:

  • Llama 2 (7B, 13B, 70B)
  • Llama 3 (8B, 70B, 405B)
  • Code Llama

Mistral family:

  • Mistral 7B
  • Mixtral 8x7B, 8x22B

Other:

  • Falcon, BLOOM, GPT-J
  • Phi-3, Gemma, Qwen
  • LLaVA (vision), Whisper (audio)

Find models: https://huggingface.co/models?library=gguf

References

Resources

  • GitHub: https://github.com/ggerganov/llama.cpp
  • Models: https://huggingface.co/models?library=gguf
  • Discord: https://discord.gg/llama-cpp

Related Skills

View on GitHub
GitHub Stars14.5k
CategoryAI
Updated1mo ago
Forks1.0k

Languages

TeX

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions