training-check
Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs
Install / Use
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill training-checkInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AutomationSupported Platforms
Our assessment of training-check
training-check scores 94/100 on our quality scale, 361st of 2,790 Automation skills we index (top 13%).
Its SKILL.md is 5.6 KB long, well organised into 12 sections with 4 code examples: a solid amount of guidance for an agent.
With 19,197 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated 2 days ago, so training-check is actively maintained.
- Our last check on 2026-11-16 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
ReviewOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review judged it risky: The skill tells the agent to create recurring CronCreate jobs without asking the user: "do not ask the user whether to set it up. Just set it.".
AI review: risky
- The skill tells the agent to create recurring CronCreate jobs without asking the user: "do not ask the user whether to set it up. Just set it."
- It authorizes the agent to autonomously kill training sessions based on its own metric judgments.
- It grants Bash(*) and includes an SSH fallback, giving broad remote execution ability if arguments or paths are manipulated.
AI review by kimi-k2.7-code on 2026-09-28. Automated pattern scan on 2026-09-28. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
training-check compared with similar skills
All 4 of these similar skills score higher than training-check; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| training-check (this skill)by wanshuiyin | 94 | 19.2k | 2d ago | SKILL.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 81.9k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
| CowAgentby zhayujie | 100 | 48.0k | today | CLAUDE.md |
Frequently asked questions
- How do I install training-check?
- Run
npx skills add wanshuiyin/Auto-claude-code-research-in-sleep --skill training-check. The install tabs above show the steps for each supported agent. - Which AI agents does training-check work with?
- It is written for Claude Code and OpenAI Codex, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is training-check safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review judged it risky: The skill tells the agent to create recurring CronCreate jobs without asking the user: "do not ask the user whether to set it up. Just set it.". It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is training-check still maintained?
- The repository was last updated 2 days ago, so training-check is actively maintained.
Skill content
View source on GitHubname: training-check description: Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks. argument-hint: "[wandb-run-path]" allowed-tools: Bash(*), Read, Grep, Glob, Write, Edit, mcp__codex__codex, mcp__codex__codex-reply
Training Check
Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.
⏱ This skill is correctly cron-wired (see below): it polls machine-checkable training health (NaN / divergence / idle GPU) — the additive external-wait shape in
shared-references/external-cadence.md. The occasional Codex call for an ambiguous metric is a one-shot check per tick, not a multi-round verdict loop, so it stays additive — it never grows into a wrapped verdict skill.
Context: $ARGUMENTS
Constants
- WANDB_ENTITY and WANDB_PROJECT: read from CLAUDE.md or passed as argument (format:
entity/project/run_id) - CHECK_INTERVAL: starts at 10 minutes, then gradually increases if consistently healthy: 10 min → 20 min → 30 min → 60 min (cap)
- REVIEWER_MODEL =
gpt-6-astra— used via Codex MCP for ambiguous cases only
When to Use
- After training is confirmed running (session alive, loss decreasing for first few steps)
- Set up via CronCreate to fire periodically during training
- This skill checks training QUALITY, not process HEALTH. Process health (session alive, GPU utilization) is watchdog.py's job.
Workflow
Step 1: Read WandB Metrics
import wandb
api = wandb.Api()
run = api.run("<entity>/<project>/<run_id>")
history = run.history()
If WandB is unreachable (API error, network issue), fall back to reading the log file directly via SSH:
ssh server "tail -100 /path/to/training.log"
Check these signals:
- Loss trend: Is training loss decreasing over the last N steps?
- Eval metrics: Are evaluation metrics improving (or at least not degrading)?
- NaN / Inf: Any NaN or Inf values in loss or gradients?
- Spikes: Sudden large jumps in loss (>10x normal variance)?
- Learning rate: Is the schedule behaving as expected?
- Gradient norm: Exploding or vanishing?
Step 2: Judgment
| Signal | Judgment | Action | |--------|----------|--------| | NaN/Inf in loss | Clearly bad | Stop training, investigate | | Loss diverging (increasing for >N steps) | Clearly bad | Stop training, investigate | | Eval metrics significantly worse than baseline | Clearly bad | Stop training, investigate | | Loss decreasing, metrics improving | Clearly fine | Continue, increase check interval | | Loss flat but not diverging | Unsure | → Step 3 (Codex judgment) | | Metrics noisy, can't tell trend | Unsure | → Step 3 (Codex judgment) | | Slightly worse than baseline but still early | Unsure | → Step 3 (Codex judgment) |
Step 3: Codex Judgment (only when unsure)
Only escalate to Codex when the signal is ambiguous. For clearly good or clearly bad signals, act directly.
mcp__codex__codex:
model: gpt-6-astra
config: {"model_reasoning_effort": "xhigh"}
prompt: |
TRAINING HEALTH CHECK — need your judgment on ambiguous metrics.
Run: <entity>/<project>/<run_id>
Current epoch/step: X / Y total
Training loss (last 10 checkpoints): [values]
Eval metrics (last 3 evals): [values]
Baseline reference: [numbers from paper/reproduction]
What I'm unsure about: [specific concern]
Please respond with exactly one of:
- STOP: clearly problematic, should kill training
- CONTINUE: looks fine, check again next interval
- WAIT: not enough data to judge, check again sooner
Step 4: Act
| Decision | Action | |----------|--------| | Stop | Kill the training session. Save the WandB run URL, key metrics, and reason for stopping. Log to project notes for debugging. | | Continue | Do nothing. Will be invoked again at next interval (increase interval if consistently healthy). | | Wait | Do nothing but keep the current short interval (don't increase). |
Integration with Watchdog
Training-check and watchdog.py operate at different levels:
| Layer | Tool | What it checks | Frequency | |-------|------|----------------|-----------| | Process health | watchdog.py | Session alive? GPU active? | Every 60s (continuous) | | Training quality | training-check | Loss trend? Metrics improving? | Every 10-60 min (periodic) |
Use both together:
- Watchdog catches crashes and idle GPUs immediately
- Training-check catches subtle quality issues (loss plateau, metric degradation)
Rules
- Do not stop training on first sign of noise — some loss spikes are normal. Look at trends over multiple checkpoints.
- When stopping training, always save the WandB run URL and key metrics as evidence.
- If both WandB and log files are unreachable, report the connectivity issue and try again next interval. Do not assume training is broken.
- Gradually increase check interval when healthy (10 → 20 → 30 → 60 min). Reset to 10 min after any anomaly.
- This skill is meant to be automated via CronCreate — do not ask the user whether to set it up. Just set it.
CronCreate Setup Example
After training is confirmed stable:
CronCreate (recurring, every 10 minutes initially):
"Run /training-check for wandb run <entity>/<project>/<run_id>"
As the check interval increases, delete the old CronCreate job and create a new one with the longer interval.
Related Skills
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
ruflo
81.9k🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
CowAgent
48.0kOpen-source personal AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, multi-channel. Lightweight, extensible, one-line install.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
