experiment-design
A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked.
Install / Use
npx skills add rampstackco/claude-skills --skill experiment-designInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
Content & MediaSupported Platforms
Our assessment of experiment-design
experiment-design scores 86/100 on our quality scale, 740th of 1,155 Content & Media skills we index.
Its SKILL.md is 31 KB long, well organised into 18 sections and no code examples: a thorough specification that gives an agent plenty to work with.
It has 902 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so experiment-design is actively maintained.
- Our last check on 2026-11-03 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-04. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
experiment-design compared with similar skills
All 4 of these similar skills score higher than experiment-design; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| experiment-design (this skill)by rampstackco | 86 | 902 | 2mo ago | SKILL.md |
| taste-skillby Leonxlnx | 100 | 116.8k | 8d ago | CLAUDE.md |
| Claude-Code-Game-Studiosby Donchitos | 100 | 39.7k | 7d ago | CLAUDE.md |
| siyuanby siyuan-note | 100 | 47.3k | today | MCP Server |
| x402by browser-use | 100 | 121.0k | 3d ago | SKILL.md |
Frequently asked questions
- How do I install experiment-design?
- Run
npx skills add rampstackco/claude-skills --skill experiment-design. The install tabs above show the steps for each supported agent. - Which AI agents does experiment-design work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is experiment-design safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is experiment-design still maintained?
- The repository was last updated about 2 months ago, so experiment-design is actively maintained.
Skill content
View source on GitHubname: experiment-design
description: "A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked. Hypothesis writing, sample size, duration, segment analysis, running discipline, matching a result to a pre-committed decision rule, and the common failure modes that produce confidently wrong shipping decisions. Use this skill whenever the user is planning a test that has not run yet: framing a hypothesis, sizing the sample, setting duration, choosing guardrails, or deciding whether something is worth testing at all. Triggers on design an experiment, experiment plan, A/B test, split test, multivariate test, holdout, experiment hypothesis, sample size, minimum detectable effect, MDE, test duration, guardrail metric, no peeking, pre-committed decision rule, is this worth testing. Use experimentation-analytics instead when the test has already run and the question is how to read the result panel."
category: product
catalog_summary: "Hypothesis to decision: sample size, duration, segment analysis, interpretation, and the failure modes that produce wrong shipping calls"
display_order: 4
Experiment Design
A senior product manager's playbook for running experiments that produce trustworthy decisions.
The default state of experimentation in most companies is sloppy. PMs run tests against vague hypotheses, look at results too early, ignore guardrails, stratify into noise, and ship features whose lift is mostly measurement error. The cost is real: ship the wrong thing, kill the right thing, learn the wrong lesson, repeat.
This skill is the discipline that prevents most of those mistakes. It assumes you have a working experimentation platform (Statsig, PostHog, GrowthBook, Optimizely, Amplitude, Eppo, Kameleoon; the platform does not matter for the principles). It assumes you have product-design and engineering pipelines that can deliver real treatment changes. The hard part is the thinking, and that is what is here.
When to use this skill: any time you are about to design or interpret an experiment. Read the relevant section before you start, not after the test is running.
What this skill covers
The skill spans the full experiment lifecycle. Pre-experiment readiness (is this thing even worth testing). Hypothesis design (cause, effect, magnitude, mechanism). Sample size and minimum detectable effect (do you have enough traffic to learn anything). Duration (how long is long enough, when does the cycle bias the result). Running discipline (no peeking, guardrails, sequential testing). Interpretation (the three buckets and the inconclusive case). Decision-making (matching the result to a pre-committed rule).
The skill does not cover feature flag operational mechanics; those live in the feature-flagging skill, which handles flag taxonomy, environment management, and stale-flag cleanup as a separate discipline. The skill does not cover statistical analysis depth; for delta methods, variance reduction techniques like CUPED, and Bayesian alternatives, see the experimentation-analytics skill. The skill does not cover platform-specific tooling; for MCP commands, auth models, and platform-specific configuration, consult the chosen platform's official documentation. This skill produces the experiment design; the platform implements it.
For the orchestration layer above (which experiments to run, in what order, with what cadence), see the forthcoming experimentation-platform-orchestrator skill. That skill schedules; this skill designs.
The framework: 12 considerations for trustworthy experiment results
A defensible experiment design sits at the intersection of twelve considerations. Each is covered in detail in its own section below.
- Hypothesis discipline. Cause, effect, magnitude, and mechanism. The hypothesis names what is being tested, what should move, by how much, and why.
- Sample size and minimum detectable effect (MDE). Whether the test has enough traffic to detect the effect at the chosen power. Refuse to run underpowered tests.
- Test duration. Longer of the sample-size-hit duration and a full weekly cycle. UI/UX changes need at least 14 days regardless.
- What NOT to A/B test. UX bugs, legal-required changes, brand-philosophy questions, decisions already made, designs whose randomization cannot be clean.
- Segment analysis. Pre-registered segments are evidence; post-hoc segments are noise mining. The multiple comparisons problem is real.
- Interaction effects. Concurrent tests on the same surface can interfere. Mutex enforcement or coordination required.
- Ratio metrics and variance estimation. Naive variance estimators on ratios understate uncertainty. Confirm the platform uses a ratio-aware estimator.
- Network effects and two-sided markets. Treatment can leak into control via interference. Cluster randomization, switchback, or geographic isolation when needed.
- Sequential testing and the peeking problem. Daily peeking inflates false positive rates. Use sequential testing methods when available; pre-commit otherwise.
- Pre-commitment vs p-hacking. Write down the primary metric, MDE, duration, segments, and decision rule before launch. Apply mechanically when results come in.
- Reading results and making the call. Three buckets: clear win, clear loss, inconclusive. The inconclusive bucket exists for a reason; resist the pull to ship anyway.
- Common failures and fixes. A short rapid-fire pattern catalog, expanded in
references/common-failures.md.
The sections below cover each consideration in turn. Read the relevant section before running the experiment, not after.
Hypothesis discipline
The most important section in the skill. Most experiment failures trace back to a vague hypothesis.
A real hypothesis has four parts: cause, effect, magnitude, mechanism. Cause is the change you are making. Effect is the metric you expect to move. Magnitude is how much you expect it to move and from what baseline. Mechanism is why you expect this change to produce this effect.
Bad hypothesis, common shape: "We think the new pricing page will increase conversions." What is wrong with it: no magnitude (how much), no mechanism (why), and the metric is "conversions" rather than a specific event with a clear definition. The team will run this test, look at the result, and argue about what counts as a win. Pre-commitment is impossible because nothing was committed.
Good hypothesis, same domain: "Replacing the three-tier pricing comparison with a single recommended tier will increase signup-to-paid conversion by 8 percent (currently 12 percent, target 13 percent) by reducing decision friction for users who already know they want to subscribe." Cause is the tier replacement. Effect is signup-to-paid conversion, defined as the user reaches the paywall and completes payment within seven days. Magnitude is 8 percent relative lift, taking the rate from 12 to 13 percent absolute. Mechanism is decision friction reduction. Now the team has something to test, a number to hit, and a story to falsify.
Primary metric vs guardrails. The primary metric is the thing you are trying to move. Guardrails are the things that must not break: revenue, retention, support ticket volume, page load time, error rates. Pick exactly one primary metric. Pick three to five guardrails. Multiple primary metrics destroy the discipline because they let you cherry-pick the favorable one when results come in.
Falsifiability test. Before launching the experiment, write down what would make you NOT ship this. If the answer is "nothing, we are committed to the change regardless," the hypothesis is not real and the experiment is theater. Skip the test, save the engineering time, and just ship the change.
Directional vs magnitude distinction. Knowing the change moves the needle is different from knowing it moves the needle enough to matter. A 0.3 percent absolute lift on signup conversion may be statistically significant with enough traffic and still not justify the engineering cost of maintaining the change. Magnitude matters as much as direction; the hypothesis names the magnitude that would justify shipping.
For templates and worked examples across common metric types, see references/hypothesis-templates.md.
Sample size and minimum detectable effect
Sample size grows with the inverse square of the effect you want to detect. Detecting a 1 percent lift requires roughly one hundred times the sample needed to detect a 10 percent lift. Most PMs underestimate this.
The basic decision rule: if your minimum detectable effect (MDE) at current traffic and a reasonable test duration is greater than 5 percent absolute lift, you probably need a bigger MDE. Tiny changes that need huge samples to detect are usually not worth shipping anyway. The change is small either because the underlying mechanism is weak or because the implementation is timid. A weak mechanism is not worth a launch. A timid implementation should be made bolder before testing.
The "we do not have enough traffic" trap. Real for very small products. Lazy for everyone else. If you have ten thousand users a week and you are trying to detect a half-percent absolute lift, you are not running an experiment, you are sampling noise. Pick changes whose expected effect is large enough to detect at your traffic level. If the change is genuinely small, ship it without a test (small upside, small downside, low cost) or do not ship it at all.
Power. The test's ability to detect an effect that is actually present. The conventional floor is 80 percent. Below that, you are rolling dice; the test will frequently miss real effects. Higher power costs more sample. Most platforms default to 80; if you change it, document why.
One-sided vs two-sided. Most PM tests are two-sided despite the temptation to claim otherwise. A one-sided test says "I only care about the positive direction; if the change makes things worse, I do not need to detect it." That is rarely true. If the new pricing page tanks conversion, you want to know. Default to two-sided. If you genuinely want one-sided, document the asymmetry before running.
For pre-calculated sample size tables across common conversion rate baselines and MDEs, see references/sample-size-tables.md. The tables are starting points, not substitutes for running the math against your specific traffic and metric.
Test duration
Minimum duration is the longer of two constraints. Constraint one: the sample size hits the calculated requirement. Constraint two: the test runs at least one full weekly cycle. Testing only Monday through Wednesday misses weekend behavior, which on most consumer products differs meaningfully from weekday behavior.
Novelty effects. New things attract attention. The first few days of a winning test often overstate the lift. Users notice the change, click it, and produce a temporary effect that fades as the novelty wears off. Run long enough to see if the lift survives the novelty period. Two weeks is the conventional minimum for any UI/UX experiment, even if the sample size hits faster.
Primacy effects. The opposite problem. Existing users may resist the change in week one and adapt by week three. Common in UI rearrangement tests. Killing the test in week one because the result looks negative misses the point that primacy is bigger than the underlying effect at that timescale.
Holdout periods. If the experiment changes a permanent feature (notification frequency, default settings, search ranking), keep a holdout group OFF the new behavior for at least a month after launch. The holdout measures long-term effect, not just the day-1 lift. Long-term effects are often different from short-term effects: a notification change tha
Truncated for display — read the full file on GitHub.
Related Skills
taste-skill
116.8kAgent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages
Claude-Code-Game-Studios
39.7kTurn Claude Code into a full game dev studio — 49 AI agents, 72 workflow skills, and a complete coordination system mirroring real studio hierarchy.
siyuan
47.3kAn open-source, privacy-first, self-hosted knowledge workspace where humans and AI agents work together 开源、隐私优先、自托管的知识工作空间,让人与智能体在此协作
x402
121.0kSet up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
