batch-api-orchestrator
Decides batch vs realtime for bulk LLM work, prices both, and plans chunking, idempotency keys and partial-failure retries
Install / Use
npx skills add borghei/Claude-Skills --skill batch-api-orchestratorInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of batch-api-orchestrator
batch-api-orchestrator scores 87/100 on our quality scale, 518th of 950 AI & Machine Learning skills we index.
Its SKILL.md is 5.9 KB long, well organised into 12 sections with 1 code example: a thorough specification that gives an agent plenty to work with.
It has 817 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so batch-api-orchestrator is actively maintained.
- Our last check on 2026-11-02 found the source still online.
- No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
- Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-07. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
batch-api-orchestrator compared with similar skills
All 4 of these similar skills score higher than batch-api-orchestrator; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| batch-api-orchestrator (this skill)by borghei | 87 | 817 | 57d ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.9k | today | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.7k | 9d ago | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 93.0k | 2d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.9k | today | CLAUDE.md |
Frequently asked questions
- How do I install batch-api-orchestrator?
- Run
npx skills add borghei/Claude-Skills --skill batch-api-orchestrator. The install tabs above show the steps for each supported agent. - Which AI agents does batch-api-orchestrator work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is batch-api-orchestrator safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is batch-api-orchestrator still maintained?
- The repository was last updated about 2 months ago, so batch-api-orchestrator is actively maintained.
Skill content
View source on GitHubname: batch-api-orchestrator description: > This skill should be used when the user asks to "batch LLM requests", "should I use the batch API", "estimate batch vs realtime cost", "design a bulk LLM job", or "process thousands of prompts cheaply". license: MIT + Commons Clause metadata: version: 1.0.0 author: borghei category: engineering domain: ai-engineering updated: 2026-06-29 tags: [batch-api, llm, cost-optimization, throughput, async]
Batch API Orchestrator
Category: Engineering Domain: AI Engineering
Overview
Decide when to run LLM work through an asynchronous batch API versus realtime/streaming, then design the job so it is cheap, idempotent, and resilient to partial failure. Batch APIs typically cost roughly half of realtime in exchange for higher latency (results arrive over minutes to hours, not milliseconds), which makes them ideal for evals, backfills, embeddings, and bulk classification/extraction — and wrong for anything a human is waiting on. This skill is model- and vendor-agnostic: it reasons about the batch pattern, not any one provider's API.
Clarify First
Before recommending or designing a batch job, confirm these inputs. If any is unknown or vague, ASK — do not assume:
- [ ] Latency tolerance — is a human waiting (interactive), or can results land in minutes/hours? (sets
--latency-toleranceand the batch-vs-realtime verdict) - [ ] Volume & token shape — how many requests, and the average input/output tokens each? (sets
--requests,--avg-input-tokens,--avg-output-tokensfor the cost estimate) - [ ] Pricing & discount — your realtime per-token prices and the batch discount your vendor offers (sets
--realtime-input-price,--realtime-output-price,--batch-discount; defaults are neutral placeholders, not real prices)
Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions.
Quick Start
cd engineering/batch-api-orchestrator
# 1. Should this be batch or realtime, and what does it cost?
python scripts/batch_cost_estimator.py \
--requests 50000 --avg-input-tokens 800 --avg-output-tokens 200 \
--realtime-input-price 3.0 --realtime-output-price 15.0 \
--batch-discount 0.5 --latency-tolerance hours
# 2. Plan the chunking / idempotency / retry strategy for the job
python scripts/batch_job_planner.py \
--total-items 50000 --max-batch-size 10000 --retry-policy exponential --json
Tools Overview
| Tool | Purpose | Key Flags |
|------|---------|-----------|
| scripts/batch_cost_estimator.py | Compare realtime vs batch cost, show savings, and recommend batch or realtime given latency tolerance | --requests, --avg-input-tokens, --avg-output-tokens, --realtime-input-price, --realtime-output-price, --batch-discount, --latency-tolerance, --json |
| scripts/batch_job_planner.py | Produce a chunking + idempotency + partial-failure plan for a bulk job | --total-items, --max-batch-size, --retry-policy, --max-retries, --json |
Both scripts: Python 3 standard library only, argparse CLI, --json and human-readable output. Run --help for full usage.
Workflows
1. Decide batch vs realtime, then size the cost
- Gather volume and token shape (
--requests,--avg-input-tokens,--avg-output-tokens). - Plug in your vendor prices and batch discount — never assume them.
- Run
batch_cost_estimator.pywith the real--latency-tolerance(realtime,minutes, orhours). - Read the verdict: if work is interactive, the tool recommends realtime regardless of savings; otherwise it quantifies the batch savings.
- Sanity-check against the decision tree in
references/batch-patterns-and-decision-tree.md.
2. Design a resilient bulk job
- Run
batch_job_planner.pywith--total-items,--max-batch-size, and a--retry-policy. - Adopt the generated idempotency-key scheme so re-submitting a chunk never double-charges or double-writes.
- Wire result reconciliation: match every output back to its request id, and collect the unmatched into a dead-letter set.
- Apply the partial-failure handling (retry only failed items, never the whole batch) from the reference.
- Choose polling vs callback for completion, per the reference guidance.
Reference Documentation
- references/batch-patterns-and-decision-tree.md — when-to-batch decision tree; job design (idempotency keys, partial failures, reconciliation, polling vs callback); fitting use cases (evals, backfills, embeddings, bulk classification/extraction); and anti-patterns such as batching interactive requests.
- references/cost-and-throughput-economics.md — the cost/throughput tradeoff in depth: the ~half-cost rule of thumb, throughput vs latency, queueing, chunk sizing, and how to model the break-even between a faster realtime path and a cheaper batch path.
Common Patterns
- Batch the patient, stream the impatient — if no human is blocked on the result, default to batch for the cost win; reserve realtime/streaming for interactive UX.
- Idempotency key per item — derive a stable key (e.g. hash of input + job version) so retries and re-submissions are safe and never double-billed.
- Retry the item, not the batch — on partial failure, re-enqueue only the failed request ids; resubmitting the whole chunk wastes money and re-runs successes.
- Reconcile by request id — never rely on output ordering; join results back to inputs by id and route the unmatched to a dead-letter queue for inspection.
- Right-size chunks — split by the vendor's max-batch limit and by your own blast-radius tolerance, not into one giant job whose failure is all-or-nothing.
- Embeddings and evals are the sweet spot — large, latency-insensitive, embarrassingly parallel workloads capture the full batch discount with the least risk.
Related Skills
claude-mem
117.9kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Agent-Reach
112.7kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Understand-Anything
93.0kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
79.9kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
