compare
Same-epoch comparison and analysis of training runs across wandb, neptune, tensorboard, mlflow, or local result files. Aligns runs at the student's current step (never current-vs-final-of-baseline) and separates proxy metrics from downstream targets.
Install / Use
npx skills add fcakyon/phd-skills --skill compareInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
OperationsSupported Platforms
Tags
Our assessment of compare
compare scores 85/100 on our quality scale, 548th of 763 Operations skills we index.
Its SKILL.md is 4.7 KB long, well organised into 12 sections with 3 code examples: a solid amount of guidance for an agent.
It has 406 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so compare is actively maintained.
- Our last check on 2026-11-05 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-05. Automated pattern scan on 2026-10-05. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
compare compared with similar skills
All 4 of these similar skills score higher than compare; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| compare (this skill)by fcakyon | 85 | 406 | 2mo ago | SKILL.md |
| algorithmic-artby anthropics | 100 | 177.9k | 58d ago | SKILL.md |
| pptxby anthropics | 100 | 177.9k | 58d ago | SKILL.md |
| prospectingby coreyhaines31 | 100 | 60.1k | 8d ago | SKILL.md |
| ad-creativeby coreyhaines31 | 100 | 60.1k | 8d ago | SKILL.md |
Frequently asked questions
- How do I install compare?
- Run
npx skills add fcakyon/phd-skills --skill compare. The install tabs above show the steps for each supported agent. - Which AI agents does compare work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is compare safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is compare still maintained?
- The repository was last updated about 2 months ago, so compare is actively maintained.
Skill content
View source on GitHubname: compare description: Same-epoch comparison of training runs across wandb, neptune, tensorboard, or mlflow. Aligns runs at the student's current step (never current-vs-final-of-baseline) and separates proxy metrics from downstream targets. Use when the user asks to compare runs, check if a run is improving, track lag against a baseline, rank experiments, or evaluate run-vs-run performance.
Compare: same-epoch run comparison across trackers
The most common comparison error is reporting "run A is 4 percentage points behind baseline" when run A is at epoch 11 of 100 and the baseline number is from epoch 100. The student is still training; the comparison is meaningless. This skill enforces same-epoch alignment.
The agentic Stop hook routes here from reason when an assistant reports a delta without aligning the runs.
When to run
The user just said any of:
- "compare run A to baseline / to run B"
- "is my run improving / catching up / falling behind"
- "rank these experiments"
- "X vs Y wandb / neptune"
- "track lag against baseline"
Auto-detect the tracker
Check in this order:
WANDB_API_KEYenv var set, orwandbimports in the project → wandbNEPTUNE_API_TOKENenv var set → neptuneMLFLOW_TRACKING_URIenv var set, ormlruns/dir present → mlflowruns/orlightning_logs/dir present → tensorboard*results*.json/*meta*.jsonfiles in run dirs → local file format
If none, ask the user where metrics live before guessing.
The protocol
1. Identify the runs
Get full names (no shortcodes). If the user says "fvs-fm vs the baseline", clarify:
- which
fvs-fmrun (project + entity + run-id) - which baseline (full run name; baselines often have several variants)
2. Fetch metric history (not just final value)
You need the full curve, not the last reported value. Final-value-only comparisons hide convergence dynamics.
For wandb:
import wandb
api = wandb.Api()
run = api.run("entity/project/run-id")
history = run.history(samples=10000) # full history, not just summary
For tensorboard, parse the event files (tensorboard.backend.event_processing.event_accumulator.EventAccumulator).
For neptune / mlflow, use their respective APIs.
3. Find the student's current step
The student is the run still in progress (or the one being evaluated). Get its current epoch / step from the latest history row.
4. Slice the baseline at the same step
This is the critical step. The baseline went all the way to (say) epoch 100. The student is at epoch 11. Pull the baseline's metrics at epoch 11, not at epoch 100.
student_step = student_history['epoch'].max()
baseline_at_same_step = baseline_history[baseline_history['epoch'] == student_step]
If the baseline doesn't have an exactly-matching step, interpolate or pick the nearest. State which.
5. Separate proxy metrics from downstream
Most ML pipelines have a proxy metric (cheap, computed during training, kNN accuracy on features, loss, perplexity) and a target downstream metric (expensive, computed periodically or only at the end, finetuned linear probe accuracy, downstream task F1).
The proxy is for tracking convergence; the target is what the project is actually optimizing. Reporting only the proxy can mislead, a run that lags on kNN may close the gap on downstream finetune. Report both, separately:
| student (ep 11) | baseline (ep 11) | delta |
| proxy (kNN top-1) | 36.4% | 38.9% | -2.5 |
| downstream (linear) | not yet | 42.1% | n/a |
If the user only has proxy data, say so explicitly. Never declare a winner from proxy alone.
6. Run names in output
In every line of the report, use full run names. Never cs-ad vs fvs-fm; always phase1-7src-conv-s-adaptor-mlp vs phase1-7src-fastvit-s-featmap-mlp. Future-you reading this will not remember the shortcode.
Anti-patterns
- "X is behind baseline by 4pp": without saying at what step. Almost always wrong.
- "X has converged": without showing the last 5 epochs of the curve.
- "Best run is Y": based on a metric that was logged differently across runs (different reduction, different eval set).
- Single-seed comparison treated as definitive. Note variance if known; otherwise label as single-seed.
Output
Compact comparison table per metric pair (proxy + downstream). Each row aligned at the student's current step. Each cell traceable to a specific tracker run-id and step. End with one or two sentences interpreting the comparison, student is on track to catch up at step N, projected from current slope is a useful framing; student is winning / losing is rarely warranted before convergence.
Related Skills
algorithmic-art
177.9kCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems.
pptx
177.9kUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an em…
prospecting
60.1kWhen the user wants to find and qualify prospects for outbound, or asks who to target first, which signal to prioritize, or where to spot buyers with the problem: buying signals, enrichment, verification, account research.
ad-creative
60.1kWhen the user wants to write or judge ad creative, or asks which ad, hook, or angle to run or kill: headlines, primary text, and variations for Google, Meta, LinkedIn, and TikTok, fresh angles when creative fatigues. Test hypotheses name the metric, comparison, and expected direction.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
