llm-eval-harness
Tests/benchmarks a third-party LLM endpoint (OpenAI- or Anthropic-compatible): availability, fidelity, speed, concurrency, protocol compliance, quality regression. Use before hand-rolling a curl loop, when onboarding a provider, or debugging "system prompt 不生效" / a tok/s claim — 测评/压测一个模型或渠道.
Install / Use
npx skills add daymade/claude-code-skills --skill llm-eval-harnessInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of llm-eval-harness
llm-eval-harness scores 93/100 on our quality scale, 249th of 947 AI & Machine Learning skills we index (top 27%).
Its SKILL.md is 18 KB long, well organised into 15 sections with 9 code examples: a thorough specification that gives an agent plenty to work with.
With 1,426 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so llm-eval-harness is actively maintained.
- Our last check on 2026-10-26 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-04. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
llm-eval-harness compared with similar skills
All 4 of these similar skills score higher than llm-eval-harness; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| llm-eval-harness (this skill)by daymade | 93 | 1.4k | 53d ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 92.8k | 1d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
Frequently asked questions
- How do I install llm-eval-harness?
- Run
npx skills add daymade/claude-code-skills --skill llm-eval-harness. The install tabs above show the steps for each supported agent. - Which AI agents does llm-eval-harness work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is llm-eval-harness safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is llm-eval-harness still maintained?
- The repository was last updated about 2 months ago, so llm-eval-harness is actively maintained.
Skill content
View source on GitHubname: llm-eval-harness description: >- Tests/benchmarks a third-party LLM endpoint (OpenAI- or Anthropic-compatible): availability, fidelity, speed, concurrency, protocol compliance, quality regression. Use before hand-rolling a curl loop, when onboarding a provider, or debugging "system prompt 不生效" / a tok/s claim — 测评/压测一个模型或渠道. Not for TTS/voice-clone supplier eval (use the audio skill).
LLM Eval Harness
Overview
Give this skill an endpoint (base_url + model + an API key in an env var) and it
measures whether the endpoint actually works and whether the model is fast, stable,
protocol-correct, and good enough — instead of trusting the vendor's headline numbers.
Six dimensions, usually scattered across ad-hoc scripts that get rewritten (with the
same bugs) every time:
| Dimension | Script | Answers |
|---|---|---|
| Availability | scripts/availability_probe.py | which model IDs work here, with 3-state error classification |
| Request fidelity | scripts/fidelity_probe.py | do system prompt / tools / history actually REACH the model? |
| Speed | scripts/speed_probe.py | TTFT + sustained decode tok/s, thinking-aware |
| Concurrency / stability | scripts/concurrency_probe.py | success rate, p50/p90 latency, where it breaks |
| Protocol compliance | scripts/protocol_probe.py | does the Anthropic thinking block actually fire when requested, AND does the endpoint accept one already sitting in history on a later turn (N≥10, both are separate checks — see Dimension 3)? |
| Quality / use-case regression | scripts/usecase_runner.py + blind judges | does it pass your accumulated cases? |
Which dimensions to run
Route from what the user is actually doing:
- "接入新模型/新网关先测一下" / onboarding a provider → Availability → Fidelity, then speed/concurrency if adoption looks likely
- "这个 ID 能不能用 / 写个支持列表" → Availability (and read disciplines §9 before writing any "unavailable" verdict)
- "模型不听 system prompt / agent 行为怪但没报错" → Fidelity (system canary)
- "快不快 / tok/s 是真的吗" → Speed; "扛不扛得住" → Concurrency
- "这个兼容端点是真兼容吗" → Protocol + Fidelity's tools/auth checks
- "这个模型/渠道用着用着就 400 了 / session 断了 / continue 不下去了" (extended-thinking client,
multi-turn session) → Protocol
--check history-replayspecifically — this symptom is the generation check passing while history-replay silently doesn't; don't stop at generation - "换模型质量会不会掉" → Quality regression
- "要给厂商报 bug" → run the relevant probe with
--output, then follow references/vendor_evidence_protocol.md - "部署闸门 / 常驻监控 / 告警老误报" → references/production_testing_patterns.md
Don't run all six ritually — pick what answers the user's question.
Key handling (non-negotiable): every script takes the API key by env-var name
(--key-env MY_KEY), never the key value on the command line — so it stays out of ps,
shell history, and any saved report. Never hardcode a key into a use-case file or a
wrapper. Read references/evaluation_disciplines.md
for the full reasoning behind this and the other disciplines.
Your private data lives outside this bundle. Use-case libraries, model rosters, and
keys belong in ~/.llm-eval/ (or wherever you keep secrets), NOT in this skill directory —
the skill is generic and public; your test suite is yours. See "Use-case library" below.
Quick start
Detect what you have, then run the dimensions that apply. For an OpenAI-compatible model:
export MY_KEY=sk-... # the key never appears in a command below
# Speed: real-task throughput + sustained decode ceiling
uv run --with openai python scripts/speed_probe.py \
--base-url https://api.example.com/v1 --model some-model --key-env MY_KEY --mode both
# Concurrency: ramp until it breaks
uv run --with aiohttp python scripts/concurrency_probe.py \
--url https://api.example.com/v1/chat/completions --model some-model --key-env MY_KEY \
--format openai --concurrency 10 20 40 60
If the endpoint is Anthropic-Messages-shaped (/v1/messages), also run the protocol probe
(below). Pick dimensions by what the user actually asked — don't run all six if they only
asked "is it fast?".
Dimension 0 — Availability (which model IDs actually work)
uv run --with aiohttp python scripts/availability_probe.py \
--base-url <base> --key-env <ENV> --format both \
--models model-a vendor/model-b @models.txt --output /tmp/avail.json
- Verdicts are 3-state per failure, not pass/fail:
no-channel(no route for this ID),upstream-error(route exists, upstream failing — retry later),empty-content(usually a max_tokens artifact on reasoning models, NOT a broken model). The probe keeps--max-tokensat 8192 by default precisely so thinking models don't read as dead — do not "optimize" it downward. - Before writing any "unavailable" list: verify each failing ID exists in official
docs. Current-generation model names postdate your training data — WebSearch the
vendor's / model developer's model page instead of enumerating guesses; suffixes
(
-preview, dated variants, tier names) decide routability. Full traps: disciplines §9. - Never probe a client-side marker as if it were a real model ID. Bracket-suffixed
strings like
some-model[1m]are frequently a CLI client's own local convention (Claude Code parses and strips[1m]before the request is ever built — see claude-switch-models-setup's "Configuring Context Window Size" section for the full mechanism) — they never appear on the wire. The probe warns when it sees one, but the discipline is yours: probe the bare model ID the vendor actually documents, not a string copied out of a Claude CodeANTHROPIC_MODELenv var. - The gateway's
/v1/modelslisting is one input, never the verdict — real gateways route models the listing omits.
Dimension 0.5 — Request fidelity (does your payload reach the model?)
uv run --with aiohttp python scripts/fidelity_probe.py \
--base-url <base> --model <model> --key-env <ENV> \
--format anthropic --check all --repeat 10 --output /tmp/fidelity.json
- Four checks:
system(canary code planted in the system prompt — can the model echo it back?),tools(full round-trip including returning a tool result and verifying the final answer uses it),multiturn(plant facts across turns, recall them all),auth(x-api-key vs Bearer on both protocol endpoints — gateways commonly accept both on one path and only one on the other, and the wrong-header 401 reads exactly like a dead key). - This is the dimension that catches the nastiest gateway failure: everything returns 200 and chats fluently, but the system prompt never reached the model — so any tool whose rules live in the system prompt silently misbehaves with no error anywhere. On one real gateway, one model alias never delivered the system prompt in 100+ samples across every legal way of sending it, while sibling aliases delivered intermittently with rates that changed by the hour.
- Delivery through routed gateways is probabilistic and time-varying — hence
--repeat(default 10) and a three-state verdict (delivered / intermittent k-of-N / not-delivered). A clean single-window result is still just that window: re-sample in another time slice before publishing a number (disciplines §12). - The probe reports token-accounting corroboration but never trusts it alone — behavioral evidence rules; see disciplines §10 (calibrate the meter) and §11 (why the canary is a neutral external fact, not an identity or obedience test).
Dimension 1 — Speed (thinking-aware)
uv run --with openai python scripts/speed_probe.py \
--base-url <…/v1> --model <model> --key-env <ENV> --mode both --output /tmp/speed.json
mixedruns representative tasks (what real usage feels like);decodeforces one long output to find the sustained ceiling (the number to compare against a vendor's claim);bothdoes both.- The trap this script exists to avoid: reasoning models stream thinking in a separate
reasoning_contentfield, butcompletion_tokenscounts it. Collecting onlycontentwhile dividing bycompletion_tokensproduces wildly inflated numbers — a real ~750 tok/s model once measured as 4700 tok/s this way. The script captures both, takes TTFT as the first token of either kind, and reportscompletion_tokens / (total − TTFT). - Read the output correctly: real-task throughput is lower than the decode ceiling because short outputs never reach steady state — that's expected, not a bug. Report both numbers, and note when the model emits thinking (its end-to-end latency includes reasoning time, not just typing).
Dimension 2 — Concurrency / stability
uv run --with aiohttp python scripts/concurrency_probe.py \
--url <full endpoint URL> --model <model> --key-env <ENV> \
--format openai|anthropic --concurrency 10 20 40 60 --output /tmp/conc.json
- Pass several
--concurrencylevels to ramp and find the ceiling — the level where success rate drops or latency explodes. A model that's fast single-threaded can still collapse at modest concurrency (real example: one provider held 50 concurrent at 0.4s while another dropped requests at just 5 concurrent). - The script isolates from any ambient proxy (
trust_env=False) and disables keep-alive pooling (force_close) — otherwise you measure the proxy's limit or one pinned upstream replica, not the model. It prints a "concurrency proof" (overlapping request pairs) so you can confirm requests really ran in parallel. - Distinguish failure modes from the output: HTTP 429 (clean throttle, retriable) vs a TCP drop that hangs to timeout (much worse for UX) vs 5xx. They imply very different fixes.
Dimension 3 — Protocol compliance (Anthropic thinking block: generation AND history-replay)
uv run python scripts/protocol_probe.py \
--url <…/v1/messages> --model <model> --key-env <ENV> --check all --repeat 10 --output /tmp/proto.json
- Only relevant for endpoints claiming Anthropic
/v1/messagescompatibility.--check all(default) runs BOTH sub-checks — they test different code paths and a vendor can pass one while hard-failing the other:- generation: does
thinking: {type: enabled}actually producethinking_delta/signature_deltaSSE events when you request it? (the original check) - history-replay: does the endpoint ACCEPT a
type: "thinking"block that's already sitting in a prior assistant turn, when that history is replayed back on a later turn — exactly what Claude Code and every other agentic client does on every continuation? (--check history-replayto run just this one)
- generation: does
- Real incident (2026-07-21) that motivated the history-replay check: a Kimi/Moonshot model via a China reseller passed generation fine (emits thinking correctly when asked) but hard-rejected history-replay — 400 "invalid part type: thinking" the instant a prior thinking block came back as input, killing the session on every subsequent turn. Passing generation told us nothing about this; they're orthogonal failure modes (response generation vs. request validation).
- Don't conclude "vendor/reseller X is broken" from one cross-axis comparison — and don't
stop at the first single-axis comparison that confirms a vendor-documented parameter either.
This took three rounds to get right, kept in disciplines §§17-19 as the canonical cautionary
tale: round 1 compared a different model AND a different reseller at once and (wrongly)
blamed the reseller. Round 2 fixed that — same reseller, only the model varied — and the
result matched a documented, named vendor parameter (Moonshot's own
preserve_thinking), whic
Truncated for display — read the full file on GitHub.
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Understand-Anything
92.8kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
