Llama Vulkan Strix

Docker Compose for llama.cpp GGUF servers on AMD Strix Halo: Qwen, Gemma, and Laguna packages (abliterated and quantized), stock Vulkan plus ROCmFP4/MTP and ROCmFPX, parallel slots, with prefill/decode and quality metrics measured on this rig.

Generate Convert Improve

Install / Use

npx skills add hec-ovi/llama-vulkan-strix

Installs into whichever agent you are using.

About this skill

Quality Score

0/100

README

<h1 align="center">llama-vulkan-strix</h1> <p align="center"> <strong>Docker Compose for llama.cpp on AMD Strix Halo: Qwen, Gemma, and Laguna GGUFs (abliterated and quantized), with stock Vulkan, ROCmFP4 + MTP, and ROCmFPX. Parallel slots and context are per package in .env. Prefill, decode, and quality metrics below are measured on this rig so you can pick what fits.</strong> </p> <p align="center"> <img src="https://img.shields.io/badge/AMD-Strix_Halo-ED1C24?logo=amd&logoColor=white" alt="AMD Strix Halo" /> <img src="https://img.shields.io/badge/backend-Vulkan-AC162C?logo=vulkan&logoColor=white" alt="Vulkan" /> <img src="https://img.shields.io/badge/llama.cpp-server-000000" alt="llama.cpp" /> <img src="https://img.shields.io/badge/License-MIT-blue" alt="License" /> </p>

What this is

Docker Compose packages for llama.cpp on gfx1151 (Strix Halo). Each model block in .env wires path, template, max context, and parallel concurrency. Stock stack: docker compose up -d pulls ghcr.io/ggml-org/llama.cpp:server-vulkan and serves the GGUF you pick on :8080. Packages cover abliterated/heretic and quantized GGUFs (Qwen3.6, Gemma 4 26B-A4B, Laguna S 2.1).

Speed vs quality is your call. This README has prefill and decode on this box, plus quality vs original (KL, refusals) and capability scores from the model cards (SWE, Terminal, tools, STEM). Use those to choose a package.

Custom formats the stock image cannot load: docker-compose.rocmfp4.yml (plunderstruck Qwen3.6 ROCmFP4 + MTP, long first build) and docker-compose.laguna-rocmfpx.yml (Laguna Runtime V2 / ROCmFPX). One stack at a time on port 8080.

Supported models

| Stack | Compose | Models | |---|---|---| | Stock Vulkan | docker-compose.yml + compose/models/<package>.yml | Packages in .env.example: Qwen3.6-27B heretic-v2 Q8_0 (MTP), Qwen3.6-35B-A3B heretic Q8_0, Laguna S 2.1 IQ4_XS, Gemma 4 26B-A4B abliterated Q5_K_M. Any other standard GGUF works if you set paths yourself. | | ROCmFP4 + MTP | docker-compose.rocmfp4.yml | plunderstruck Qwen3.6 ROCmFP4 GGUFs only (27B, 27B-OBLITERATED, 35B-A3B-MTP). Custom Q4_0_ROCMFP4 tensors. | | Laguna ROCmFPX | docker-compose.laguna-rocmfpx.yml | Chadrock Laguna S 2.1 ROCmFP4 V4 GGUF only (pinned Ciru Runtime V2). |

Quick start

Prerequisites: AMD Strix Halo (Ryzen AI Max+, gfx1151), Docker + Compose, GGUFs on disk.

cd ~/workspace/llama-vulkan-strix
cp .env.example .env
# edit .env: MODELS_DIR, RENDER_GID / VIDEO_GID, then uncomment ONE model package
# (COMPOSE_FILE + LLM_MODEL + LLM_CHAT_TEMPLATE + alias + ctx + parallel)

docker compose up -d
docker compose logs -f llm

COMPOSE_FILE is what wires the per-model package (MTP, flash-attn, Laguna sampling, q8 KV). Example for Gemma:

COMPOSE_FILE=docker-compose.yml:compose/models/gemma-4-26b-a4b.yml
LLM_MODEL=gemma-4-26b-a4b-abliterated/Gemma-4-26B-A4B-It-Abliterated-Q5_K_M.gguf
LLM_CHAT_TEMPLATE=gemma-4-26b-a4b-abliterated/chat_template.jinja
LLM_ALIAS=gemma-4-26b-a4b-abliterated-q5
LLM_CTX_PER_SLOT=131072
LLM_PARALLEL=5
LLM_CTX_TOTAL=655360

Download Gemma (use HF_TOKEN from .env for authenticated speed):

set -a && source .env && set +a
HF_TOKEN="$HF_TOKEN" hf download \
  SevenOfNine/Gemma-4-26B-A4B-It-Abliterated-GGUF \
  Gemma-4-26B-A4B-It-Abliterated-Q5_K_M.gguf \
  --local-dir "$MODELS_DIR/gemma-4-26b-a4b-abliterated"
# drop chat_template.jinja next to the GGUF if it is not already there

Call it:

curl http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"llm","messages":[{"role":"user","content":"hi"}]}'

Model packages (comment / uncomment)

Each package is a full block in .env.example: model path, chat template, alias, per-slot context, parallel slots, total context, and COMPOSE_FILE pointing at compose/models/<name>.yml.

| Package | GGUF (under MODELS_DIR) | Active / total | Quant | MTP | Native max ctx | Package file | |---|---|---|---|:--:|---:|---| | A | Qwen3.6-27B-heretic-v2/...-Q8_0.gguf | 27B dense | Q8_0 | yes | 262144 | compose/models/qwen3.6-27b-heretic.yml | | B | Qwen3.6-35B-A3B-heretic/...-Q8_0.gguf | 3B / 35B MoE | Q8_0 | no* | 262144 | compose/models/qwen3.6-35b-heretic.yml | | C | laguna-s-2.1/Laguna-S-2.1-IQ4_XS-00001-of-00002.gguf | 8B / 118B MoE | IQ4_XS | no | 262144 | compose/models/laguna-s-2.1.yml | | D | gemma-4-26b-a4b-abliterated/...-Q5_K_M.gguf | ~4B / 26B MoE | Q5_K_M | no | 262144 | compose/models/gemma-4-26b-a4b.yml |

* Package B is the non-MTP-preserved heretic quant. For MTP on 35B, use the ROCmFP4 stack or a Native-MTP-Preserved GGUF with a matching package.

Default profile for all four: 5 parallel slots x 131072 = 655360 total KV. That is half the native 256k per slot, chosen so five concurrent requests fit Strix Halo GTT after weights. You can raise LLM_CTX_PER_SLOT toward 262144 if you drop parallel or free GTT.

Context and parallel slots

llama-server treats --ctx-size as the total KV cache shared by its slots; --parallel is the slot count. See the server option reference.

LLM_CTX_PER_SLOT: context available to one request
LLM_PARALLEL: concurrent slots
LLM_CTX_TOTAL must equal their product

python3 scripts/check_context_config.py .env --noob-context 131072
docker compose config -q
curl -fsS http://localhost:${LLM_PORT:-8080}/slots |
  python3 scripts/check_context_config.py .env --noob-context 131072 --slots-json -

Max context with 5 parallel slots

| Model | Architecture max | Package default (5 slots) | 5 x full native max? | |---|---:|---:|---| | Qwen3.6-27B heretic Q8 | 262144 | 131072 per slot (total 655360) | Possible in principle (total 1310720); heavy on GTT with ~28 GB weights | | Qwen3.6-35B-A3B heretic Q8 | 262144 | 131072 per slot | Same math; ~35 GB Q8 weights | | Laguna S 2.1 IQ4_XS | 262144 (YaRN in GGUF) | 131072 per slot, q8 KV | 5 x 256k f16 KV alone is ~80+ GiB class; package uses q8 KV | | Gemma 4 26B-A4B Q5 | 262144 (medium Gemma 4) | 131072 per slot, q8 KV | Same; E2B/E4B are 128k, 26B A4B is 256k |

Sources: Qwen3.6-27B / 35B-A3B config (262144), Gemma 4 medium models 256k, Laguna GGUF YaRN to 262144. All four packages use the same 5 x 131072 default on this repo.

GTT, not VRAM

On Strix Halo the dedicated "VRAM" is a small BIOS carve-out; unified RAM is GTT. Weights should land in GTT.

The default compose sets GGML_VK_PREFER_HOST_MEMORY=1. Prove it after load:

scripts/verify-gtt.sh --min-gtt-mib 16000

Raise the GTT pool once in GRUB

amdgpu sizes GTT from ttm.pages_limit (default ~half of RAM). Large multi-slot loads need more. Edit GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off amdgpu.gttsize=114688 ttm.pages_limit=29360128"

Then sudo update-grub and reboot. ttm.pages_limit=29360128 is 112 GiB of GTT. Check:

cat /sys/module/ttm/parameters/pages_limit   # want 29360128, not ~16182224
python3 scripts/gpu_mem.py

ROCmFP4 + MTP (optional, slow first install)

plunderstruck's Qwen3.6 GGUFs use custom Q4_0_ROCMFP4 types. Stock server-vulkan cannot load them. docker-compose.rocmfp4.yml builds charlie12345/rocmfp4-llama (mtp-rocmfp4-strix).

First install cost: empty Docker store means pulling ubuntu:26.04, downloading the TheRock ROCm 7.13 dist tarball, and compiling Vulkan + HIP for gfx1151. That is multi-GB of downloads and a long compile (often tens of minutes). Later starts reuse the image. The container needs /dev/kfd and /dev/dri even though compute runs on Vulkan.

docker compose -f docker-compose.rocmfp4.yml up -d --build
docker compose -f docker-compose.rocmfp4.yml logs -f llm

MTP (--spec-type draft-mtp) is on by default in that file. Details and measured ROCmFP4 throughput: docs/qwen3.6-35b-a3b-mtp-rocmfp4.md, docs/qwen3.6-27b-mtp-rocmfp4.md.

Laguna ROCmFPX (optional)

Separate stack for the Chadrock ROCmFP4 V4 GGUF. First build also compiles a pinned Ciru Runtime V2 commit.

docker compose -f docker-compose.laguna-rocmfpx.yml up -d --build

Benchmarks

Use this section to pick a package. Nothing is ranked "best"; each table answers a different question.

Quality vs original (ablation / quant): how close the uncensored or quantized weights stay to the full-precision base (KL, refusals, MMLU). Low KL means the brain is mostly intact.
Capability (model cards): what the original models score on agentic coding (SWE, Terminal), tools (MCPMark), and hard STEM (LiveCodeBench, AIME). Approximate ceiling for the GGUF you serve.
Served speed on this box: prefill = tokens/s while chewing the prompt (matters for long context and tools); decode = tokens/s while streaming the reply (what you feel in chat). Measured here on stock Vulkan and, separately, on the ROCmFP4 stack.

Quality: ablation and quant vs original

KL and refusal counts are from the heretic/abliteration authors. MMLU is their re-run of original vs ablated at full precision before quant. Quant KL against BF16 for these exact files was not re-measured on this box; Q8_0 is near-lossless in community tables, IQ4_XS / Q5_K_M trade more.

| Model (served quant) | Ablation KL vs original | Refusals ablated / original | MMLU ablated / original | Quant note | |---|---:|---|---|---| | Qwen3.6-27B heretic-v2 Q8_0 | 0.0021 | **6 / 9