SkillAgentSearch skills...

experimentation-analytics

How to read experiment results without fooling yourself. Confidence intervals, p-values, multiple testing, sequential testing, CUPED, heterogeneous treatment effects, ratio metrics, network effects, dashboard reconciliation, and the interpretation failures that produce confidently wrong shipping dec…

Install / Use

npx skills add rampstackco/claude-skills --skill experimentation-analytics

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

86/100

Supported Platforms

Universal

Our assessment of experimentation-analytics

experimentation-analytics scores 86/100 on our quality scale, 439th of 604 Data & Analytics skills we index.

Its SKILL.md is 35 KB long, well organised into 18 sections and no code examples: a thorough specification that gives an agent plenty to work with.

It has 902 GitHub stars, a meaningful sign that others use it.

Substance
30/30
Structure
13/20
Description
15/15
Adoption
13/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so experimentation-analytics is actively maintained.
  • Our last check on 2026-11-03 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-04. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

experimentation-analytics compared with similar skills

All 4 of these similar skills score higher than experimentation-analytics; compare them before choosing.

SkillScoreStarsUpdatedFormat
experimentation-analytics (this skill)by rampstackco869022mo agoSKILL.md
x402by browser-use100121.0k3d agoSKILL.md
algorithmic-artby anthropics100177.9k55d agoSKILL.md
pptxby anthropics100177.9k55d agoSKILL.md
designby nextlevelbuilder100149.0k6d agoSKILL.md

Frequently asked questions

How do I install experimentation-analytics?
Run npx skills add rampstackco/claude-skills --skill experimentation-analytics. The install tabs above show the steps for each supported agent.
Which AI agents does experimentation-analytics work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is experimentation-analytics safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is experimentation-analytics still maintained?
The repository was last updated about 2 months ago, so experimentation-analytics is actively maintained.

name: experimentation-analytics description: "How to read experiment results without fooling yourself. Confidence intervals, p-values, multiple testing, sequential testing, CUPED, heterogeneous treatment effects, ratio metrics, network effects, dashboard reconciliation, and the interpretation failures that produce confidently wrong shipping decisions. Use this skill whenever the user is reading a finished experiment result panel and about to make a ship, kill, or iterate decision, or when an experiment number does not match the dashboard number. Triggers on read experiment results, result panel, ship or kill decision, p-value, confidence interval, statistical significance, multiple testing, peeking, sequential testing, CUPED, variance reduction, heterogeneous treatment effects, ratio metric, network effects, inconclusive test, experiment versus dashboard mismatch. Use experiment-design instead when the test has not run yet and the question is hypothesis, sample size, duration, or what to test." category: product catalog_summary: "Read result panels without fooling yourself: confidence intervals, p-values, multiple testing, sequential testing, CUPED, ratio metrics, network effects, dashboard reconciliation" display_order: 6

Experimentation Analytics

A data-team-mentor's playbook for interpreting experiment results without fooling yourself.

The result panel is the moment-of-truth for an experiment. The numbers on it determine whether you ship, kill, or iterate. They also expose every shortcut taken in the design phase: an underpowered test produces wide confidence intervals; a peeked test produces a too-narrow p-value; a ratio metric without delta-method correction produces overconfident lift estimates. Most ship-the-wrong-thing decisions trace back to misreading the result panel.

This skill is the discipline that prevents misreading. It assumes the experiment was designed well (see the experiment-design skill). It assumes the platform's results panel is technically correct (most modern platforms are; some older ones are not). It assumes you can read a number off a screen. The hard part is knowing what each number actually means and what it does not, and that is what is here.

When to use this skill: any time you are reading an experiment result panel and about to make a ship, kill, or iterate decision.


What this skill is for

This skill covers result interpretation, the statistical concepts that make the numbers trustworthy, and the dashboard reconciliation work that prevents executive-level confusion when the experiment number does not match the BI number. The audience is product managers and data analysts who read experiment results together and need a shared vocabulary that does not paper over the dangerous parts of statistics.

Companion skills cover the adjacent territory. The experiment-design skill covers pre-experiment thinking: hypothesis, sample size, MDE, segments, what NOT to test. Read it before designing the test; read this skill when reading the result. The feature-flagging skill covers the operational mechanics of flag management, environment promotion, and stale-flag cleanup. Together the three skills span the experimentation lifecycle from intent through interpretation. For platform-specific MCP commands, consult the chosen platform's docs; Statsig, PostHog, Optimizely, GrowthBook, Eppo, Amplitude, and Kameleoon all expose rich analytics surfaces that this skill informs how to read.


The result panel: what every modern platform should expose

A result panel that omits any of the following is a black box. Treat results from black-box platforms with extra skepticism, and consider exporting raw assignment and event data into a notebook where you can compute the missing pieces yourself.

What a competent platform exposes:

  • Variants and traffic allocation (e.g., 50/50, 33/33/33). Allocation drift across the test window indicates assignment bugs.
  • Per-variant primary metric: point estimate, confidence interval (or credible interval for Bayesian), sample size at the variant level.
  • Lift: variant minus control, expressed as both absolute change and relative percent. Both numbers matter; relative is intuitive, absolute is what shows up in revenue calculations.
  • Statistical significance: p-value (frequentist) or probability of being best (Bayesian). The methodology should be labeled clearly so you know which interpretation rules apply.
  • Variance reduction technique applied: CUPED, post-stratification, regression adjustment. If the platform applies these silently, ask which.
  • Guardrail metric statuses: each guardrail labeled green, amber, or red against its tolerance. The tolerance was set at design time; the panel just enforces it.
  • Per-segment results for pre-registered segments only. Post-hoc segment slicers are tempting and dangerous.
  • Test status: running, ended, decision filed.
  • A time series of the lift across the test window. This is where novelty effects, primacy effects, and assignment bugs become visible.

If you are looking at a result panel that hides any of these, the first move is to surface them, not to ship.


Confidence intervals: the most important number

The single most important number on the result panel is the confidence interval (CI) on the lift. More important than the point estimate. More important than the p-value. The CI tells you what you actually know.

What a 95% CI of [+2%, +6%] means: under repeated sampling, the true effect would fall in this range 95% of the time. The true effect is most likely somewhere near the middle, but the extremes are entirely consistent with the data.

What it does not mean: it does not literally mean "there is a 95% chance the true effect is between +2% and +6%." That is the Bayesian credible interval, which often gives similar numerical answers but is conceptually different. PMs can usually live with the loose intuition; analysts should know the precise version when defending a number to a skeptic.

The width of the CI matters more than the center for most ship decisions. A wide CI means you do not know much yet. A narrow CI means you know with precision. The point estimate is your best guess; the width is your humility.

Practical decision rules, in order of importance:

  1. If the CI includes zero AND a meaningful positive number (say [-1%, +5%]), you do not have enough data to ship. Period. The point estimate may look favorable, but the data is consistent with no effect and consistent with a meaningful win. You cannot tell which.
  2. If the CI is all-positive (lower bound greater than zero, e.g., [+1%, +4%]), there is a real effect. Now evaluate magnitude: is the lower bound large enough to be worth the implementation cost?
  3. If the CI is all-negative (upper bound less than zero, e.g., [-5%, -1%]), there is real harm. Kill the test.
  4. If the CI straddles zero but is narrow (e.g., [-0.5%, +0.5%]), this is a real null result. The effect is small enough to call essentially zero. Useful information; do not ship the change for "lift" reasons (you found none) but do not panic about harm either.
  5. If the CI straddles zero and is wide (e.g., [-5%, +8%]), the test is inconclusive. The data is consistent with a moderate win, no effect, or a moderate loss. Run longer, run bigger, or accept that the question cannot be answered at the available traffic.

For a worked-example cheatsheet, see references/confidence-interval-cheatsheet.md.


P-values: what they mean and what they do not

The p-value is the probability of observing the lift you saw (or a larger one) IF the true effect were zero. A p-value of 0.04 means: under the null hypothesis of no effect, you would see this much lift purely by chance about 4% of the time.

What the p-value does not mean, despite frequent abuse:

  • It does not mean "there is a 96% chance the treatment works." That sentence has no defensible interpretation; the p-value is computed under the assumption that the treatment does NOT work, so it cannot tell you the probability that it does.
  • It does not mean "the effect is large." A tiny effect tested against a huge sample can produce a vanishingly small p-value. The p-value is about the strength of evidence against the null, not the size of the effect.
  • It does not mean "the result will replicate." A p-value of 0.04 is associated with replication rates well below 50% in most published research; statistical significance is not reproducibility.

The 0.05 cutoff is convention, not law. If you pre-committed to alpha equals 0.05, follow it; the discipline of pre-commitment is more valuable than the specific threshold. If you did not pre-commit, p equals 0.06 is not categorically different from p equals 0.04, and treating it as such is theater.

Always read the CI alongside the p-value. The p-value tells you about the null hypothesis; the CI tells you about the magnitude. Both matter; neither is sufficient alone. A p-value of 0.001 with a CI of [+0.1%, +0.3%] is a real but practically tiny effect; a p-value of 0.08 with a CI of [-1%, +12%] is a noisy estimate that could be huge or zero. The former is technically significant and not worth shipping; the latter is technically not significant and you might still want to dig deeper.

The peeking problem applies to p-values directly. Standard p-values assume one analysis at the end of the test. Multiple analyses inflate the false positive rate. Modern platforms with sequential testing report "always-valid" or "anytime-valid" p-values that survive peeking; older platforms do not. Know which you are looking at. See references/p-value-interpretation-guide.md for deeper coverage.


Multiple testing corrections

The problem in one sentence: with twenty independent comparisons at alpha equals 0.05, you expect one false positive purely by chance. With fifty comparisons, two or three. With a hundred, five.

Where multiple testing creeps in unintentionally:

  • Multiple variants: A vs B vs C vs D is three pairwise comparisons against control, not one.
  • Multiple metrics: tracking primary plus six guardrails plus three secondary metrics is ten chances to find significance somewhere.
  • Multiple segments: ten segments times three metrics is thirty chances.
  • Multiple time windows: looking at week 1, week 2, week 3, and the full test is four chances per metric per segment.

Two correction methods you should know:

  • Bonferroni correction. Divide alpha by the number of comparisons. Conservative; controls the family-wise error rate (the probability of even one false positive across the family of tests). Use when false positives are catastrophic.
  • Benjamini-Hochberg (BH) correction. Less conservative; controls the false discovery rate (the expected proportion of false positives among the things called significant). Better for exploratory analysis.

Most modern platforms support these natively as a configuration option. Configure them, or know which the platform applies by default. Some platforms apply corrections silently to the displayed p-values; others report uncorrected numbers and expect you to do the math.

The PM-friendly heuristic: pre-register your primary metric and primary segment. Treat everything else as exploratory. Findings in non-primary metrics or non-primary segments require larger effects, replication in a follow-up test, or both before they justify shipping. The discipline of designating primary up front protects you from the multiple-testing trap better than any correction formula.


Sequential testing math

The peeking problem revisited from the analytics side. Classical t-tests, z-tests, and proportion tests assume one analysis at the end of a test. If you analyze the data five times during the test and stop the moment you see significance, your false positive rate is much higher than the nominal 5%. With daily peek

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars902
CategoryData
Updated2mo ago
Forks124

Languages

JavaScript

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions