SkillAgentSearch skills...

ab-test-analyzer

Statistical significance calculator for A/B test results with sample size requirements, segment breakdowns, and hypothesis generation

Install / Use

npx skills add irinabuht12-oss/marketing-skills --skill ab-test-analyzer

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

92/100

Supported Platforms

Universal

Tags

Our assessment of ab-test-analyzer

ab-test-analyzer scores 92/100 on our quality scale, 820th of 4,705 Development & Engineering skills we index (top 18%).

Its SKILL.md is 5.4 KB long, well organised into 29 sections with 4 code examples: a solid amount of guidance for an agent.

With 7,761 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
26/30
Structure
20/20
Description
15/15
Adoption
17/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated 26 days ago, so ab-test-analyzer is actively maintained.
  • Our last check on 2026-10-28 found the source still online.
  • No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
  • Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

Review

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review judged it risky: The skill instructs the agent to add a third-party MCP connector (`claude mcp add ryze --transport http https://connector.get-ryze.ai/mcp`) that proxies access to Google Ads, Meta Ads, GA4, and Search Console.

AI review: risky

  • The skill instructs the agent to add a third-party MCP connector (`claude mcp add ryze --transport http https://connector.get-ryze.ai/mcp`) that proxies access to Google Ads, Meta Ads, GA4, and Search Console.
  • It directs live, sensitive advertising/analytics account data through an unofficial external service, increasing exfiltration and over-permissioning risk.
  • The A/B test methodology and output templates themselves are benign and contain no destructive or credential-stealing commands.

AI review by kimi-k2.7-code on 2026-10-06. Automated pattern scan on 2026-10-06. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

ab-test-analyzer compared with similar skills

All 4 of these similar skills score higher than ab-test-analyzer; compare them before choosing.

SkillScoreStarsUpdatedFormat
ab-test-analyzer (this skill)by irinabuht12-oss927.8k26d agoSKILL.md
taste-skillby Leonxlnx100120.0ktodayCLAUDE.md
hyperframesby heygen-com10088.2ktodayCLAUDE.md
ai-job-searchby ai-job-search-dev10073.0ktodayCLAUDE.md
claude-howtoby luongnv8910047.5k1d agoCLAUDE.md

Frequently asked questions

How do I install ab-test-analyzer?
Run npx skills add irinabuht12-oss/marketing-skills --skill ab-test-analyzer. The install tabs above show the steps for each supported agent.
Which AI agents does ab-test-analyzer work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is ab-test-analyzer safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review judged it risky: The skill instructs the agent to add a third-party MCP connector (claude mcp add ryze --transport http https://connector.get-ryze.ai/mcp) that proxies access to Google Ads, Meta Ads, GA4, and Search Console. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is ab-test-analyzer still maintained?
The repository was last updated 26 days ago, so ab-test-analyzer is actively maintained.

name: ab-test-analyzer description: Statistical significance calculator for A/B test results with sample size requirements, segment breakdowns, and hypothesis generation. Use when feeding test results, checking statistical significance, calculating sample sizes, analyzing experiment outcomes, or generating next test ideas based on results. Platform: Google and Meta. metadata: platform: Google and Meta

A/B Test Analyzer

Evaluate A/B test results with statistical rigor and generate actionable next steps.

Process

  1. Collect test data - Variations, sample sizes, conversions, time period
  2. Check validity - Runtime, sample size, peeking issues
  3. Calculate significance - Z-score, p-value, confidence interval
  4. Segment analysis - Device, source, new vs returning
  5. Interpret results - Statistical vs practical significance
  6. Generate hypotheses - "Why it worked" and next test ideas

Sample Size Calculator

n = 2 × (Zα/2 + Zβ)² × p(1-p) / δ²

Where:
- n = sample size per variation
- Zα/2 = 1.96 (95% confidence) or 2.58 (99%)
- Zβ = 0.84 (80% power) or 1.28 (90%)
- p = baseline conversion rate
- δ = minimum detectable effect (absolute)

Quick Reference (95% confidence, 80% power): | Baseline CR | 10% Relative MDE | 20% Relative MDE | |-------------|------------------|------------------| | 2% | 78,000/var | 19,500/var | | 5% | 30,000/var | 7,500/var | | 10% | 14,300/var | 3,600/var |

Significance Calculation

Z-score = (pB - pA) / √(SE²_A + SE²_B)
SE = √(p × (1-p) / n)

If |Z| > 1.96 → Significant at 95%
If |Z| > 2.58 → Significant at 99%

Validity Checklist

  • [ ] Ran for minimum 14 days (day-of-week effects)
  • [ ] Met required sample size before concluding
  • [ ] No "peeking" at results during test (inflates false positives to 40%+)
  • [ ] Traffic split was random and consistent
  • [ ] No external factors (holidays, outages, campaigns)

Bayesian vs Frequentist

| Use Case | Approach | |----------|----------| | Large samples (>10K), yes/no decision | Frequentist | | Continuous monitoring, early stopping | Bayesian | | Regulatory/legal requirements | Frequentist | | Want probability statements | Bayesian |

Output Format

## A/B Test Analysis

### Test Summary
- **Test Name**: [Name]
- **Hypothesis**: [What you expected]
- **Runtime**: [X days]
- **Sample Size**: Control [n] / Variant [n]

### Results
| Metric | Control | Variant | Lift | Significant? |
|--------|---------|---------|------|--------------|
| Conversion Rate | X% | Y% | +Z% | ✅/❌ |

### Statistical Details
- Z-score: [X]
- P-value: [X]
- Confidence Interval: [X% to Y%]
- Power achieved: [X]%

### Validity Assessment
- ✅/❌ Minimum runtime met
- ✅/❌ Sample size requirement met
- ✅/❌ No external confounds identified

### Segment Breakdown
| Segment | Control CR | Variant CR | Lift | Notes |
|---------|-----------|------------|------|-------|
| Mobile | | | | |
| Desktop | | | | |

### Interpretation
[Statistical significance vs practical significance discussion]

### "Why It Worked" Hypotheses
1. [Hypothesis about mechanism]
2. [Alternative explanation]

### Next Test Ideas
1. [Follow-up test based on learnings]
2. [Adjacent hypothesis to explore]

### Confidence Level: [HIGH/MEDIUM/LOW]

Example

Input: "Tested new headline. Control: 10,000 visitors, 250 conversions. Variant: 10,000 visitors, 300 conversions. Ran for 7 days."

Output:

## A/B Test Analysis

### Test Summary
- **Hypothesis**: New headline increases conversions
- **Runtime**: 7 days ⚠️
- **Sample Size**: Control 10,000 / Variant 10,000

### Results
| Metric | Control | Variant | Lift | Significant? |
|--------|---------|---------|------|--------------|
| Conversion Rate | 2.5% | 3.0% | +20% | ✅ Yes (95%) |

### Statistical Details
- Z-score: 2.28
- P-value: 0.023
- Confidence Interval: +2.3% to +37.7%
- Power achieved: 62% ⚠️

### Validity Assessment
- ❌ Minimum runtime NOT met (7 days < 14 days recommended)
- ⚠️ Sample size marginal for 20% MDE
- ❓ Cannot assess external confounds without more context

### Interpretation
Result is **statistically significant** but validity concerns exist:
1. 7-day runtime may miss day-of-week patterns
2. Wide confidence interval (+2% to +38%) indicates uncertainty
3. Recommend extending test 7 more days to confirm

### "Why It Worked" Hypotheses
1. New headline more clearly communicates value proposition
2. Specificity/numbers in headline increased credibility

### Next Test Ideas
1. Test headline variations that emphasize the winning element
2. Apply same messaging pattern to subheadline

### Confidence Level: MEDIUM
Statistical significance achieved, but short runtime reduces confidence.

Guidelines

  • Never declare a winner without checking validity
  • Distinguish statistical significance from practical significance
  • If test ran <7 days, always recommend extending
  • If sample size insufficient, calculate required runtime to reach it
  • Ask for segment data if not provided - results often differ by device/source

Data access (Ryze MCP)

This skill works best with live account data. Connect the free Ryze MCP once and Claude reads your Google Ads, Meta Ads, GA4 and Search Console directly:

  • claude.ai / Claude Desktop: Settings → Connectors → Add custom connector → https://connector.get-ryze.ai/mcp
  • Claude Code: claude mcp add ryze --transport http https://connector.get-ryze.ai/mcp
  • Cursor: Settings → MCP → add the same URL

Setup guide: https://www.get-ryze.ai/how-to-connect-claude-to-google-meta-ads-mcp

Related Skills

View on GitHub
GitHub Stars7.8k
CategoryDevelopment
Updated26d ago
Forks579

Trust signals

88/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 medium