SkillAgentSearch skills...

data-warehouse-experimentation

Running experiments out of the data warehouse instead of via dedicated experiment platforms. SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs…

Install / Use

npx skills add rampstackco/claude-skills --skill data-warehouse-experimentation

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

93/100

Supported Platforms

Universal

Our assessment of data-warehouse-experimentation

data-warehouse-experimentation scores 93/100 on our quality scale, 149th of 606 Data & Analytics skills we index (top 25%).

Its SKILL.md is 28 KB long, well organised into 25 sections with 8 code examples: a thorough specification that gives an agent plenty to work with.

It has 902 GitHub stars, a meaningful sign that others use it.

Substance
30/30
Structure
20/20
Description
15/15
Adoption
13/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so data-warehouse-experimentation is actively maintained.
  • Our last check on 2026-10-27 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-04. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

data-warehouse-experimentation compared with similar skills

All 4 of these similar skills score higher than data-warehouse-experimentation; compare them before choosing.

SkillScoreStarsUpdatedFormat
data-warehouse-experimentation (this skill)by rampstackco939022mo agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
Agent-Reachby Panniantong100112.3k9d agoCLAUDE.md
headroomby headroomlabs-ai10079.8ktodayCLAUDE.md
Scraplingby D4Vinci100103.6k2d agoMCP Server

Frequently asked questions

How do I install data-warehouse-experimentation?
Run npx skills add rampstackco/claude-skills --skill data-warehouse-experimentation. The install tabs above show the steps for each supported agent.
Which AI agents does data-warehouse-experimentation work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is data-warehouse-experimentation safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is data-warehouse-experimentation still maintained?
The repository was last updated about 2 months ago, so data-warehouse-experimentation is actively maintained.

name: data-warehouse-experimentation description: "Running experiments out of the data warehouse instead of via dedicated experiment platforms. SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs platforms like Statsig and Optimizely. Triggers on warehouse-native experimentation, run experiments in BigQuery, run experiments in Snowflake, dbt experiments, SQL t-test, CUPED variance reduction, exposure log, sample ratio mismatch, sequential testing, mSPRT, doubly robust estimation, build vs buy experimentation. Also triggers when the team is choosing between platform and warehouse, building warehouse-native experiment infrastructure, auditing one, or running an experiment with a custom metric the platform cannot handle." category: product catalog_summary: "Run experiments out of the warehouse: SQL assignment, exposure logs, dbt metric definitions, statistical analysis, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs platforms" display_order: 9

Data Warehouse Experimentation

A senior data scientist's playbook for running experiments natively out of BigQuery, Snowflake, or any modern data warehouse, with metric definitions in dbt and statistical analysis in SQL or Python.

Most companies that run experiments at scale use a dedicated platform. Statsig, Optimizely, LaunchDarkly with experimentation, PostHog, Amplitude Experiment. The platforms are good. They handle assignment, instrumentation, and analysis in one product, and the SQL-savvy data team does not have to reinvent the variance reduction wheel.

There is a different operational model that mature data teams increasingly choose: warehouse-native experimentation. Assignment happens in code or via feature flags. Exposure events fire to the warehouse like any other event. Metrics are defined as dbt models. Statistical analysis runs as SQL or in a Python notebook against warehouse data. The "experiment platform" is just your existing data stack.

This skill covers when warehouse-native is the right call, the architecture, and the specific techniques that make it work: assignment patterns, exposure logging discipline, metric definitions in dbt, t-tests and CUPED in SQL, sequential testing, and the pitfalls that take down homegrown setups.

When to use this skill: deciding between platform vs warehouse-native, building a warehouse-native experiment infrastructure, auditing an existing one, or running a specific experiment when the platform of record cannot handle a custom metric or segmentation.


What this skill is for

This skill spans the operational execution model for warehouse-native experimentation. It does not replace the methodology and interpretation skills; it composes with them.

  • experiment-design covers methodology: hypotheses, sample size, randomization unit, primary metric. Tool-agnostic. Read it first to design the experiment correctly regardless of where it runs.
  • experimentation-analytics covers interpretation: confidence intervals, p-values, effect size, decision frameworks. Tool-agnostic. Read it when results land.
  • experimentation-platform-orchestrator covers the platform-vs-warehouse decision in detail. Read it to decide whether to use a platform or this skill.
  • feature-flagging covers assignment infrastructure when not running through a platform. Read it for the flag-management discipline that this skill assumes.
  • This skill (data-warehouse-experimentation) covers the operational execution: SQL-based assignment, exposure logging, metric definitions in dbt, statistical analysis in SQL or Python, variance reduction, sequential testing.

The distinction is between "what to do" (the methodology and interpretation skills) and "how to do it without a vendor platform" (this skill). Read this skill after you have decided warehouse-native is the right call. If you are still deciding, start with experimentation-platform-orchestrator.


When warehouse-native is the right call

Six factors push the decision toward warehouse-native.

  1. Cost at volume. Platforms charge per MAU or per event. At 10K MAU the platform is cheap; at 1M MAU the bill becomes a real budget item. Warehouse-native runs on infrastructure you already pay for.
  2. Custom metrics. If your primary metric is a complex business metric (revenue with refund-aware logic, cohort LTV, retention bracket, multi-event composites), platforms can struggle. Warehouse-native expresses any metric you can write in SQL.
  3. Custom segmentation. Enterprise customers, account-tier crosscuts, complex behavioral segments. Platforms have segmentation features; the depth varies. dbt models compose without limit.
  4. Trust requirements. Regulated industries (healthcare, finance, government) need full transparency into the math. Warehouse-native gives you every step of the calculation auditable in SQL.
  5. Existing data team strength. If you have data engineers and data scientists, you have most of the infrastructure. Adding experimentation discipline on top costs less than adopting a new platform.
  6. Iteration on metric definitions. Platforms ship metric updates on their own cadence. Warehouse-native iterates as fast as your dbt deployments.

Five factors push toward platform.

  1. Frontend visual experiments. Optimizely's bread and butter. Variant code injected via a script tag, with WYSIWYG editing.
  2. Sub-week iteration speed. Some platforms set up an experiment in 30 minutes; warehouse-native often takes a day or more for the first run of a new metric pattern.
  3. Teams without strong data infrastructure. If you do not have a warehouse, dbt, and analysts, do not start with warehouse-native. The platform is the right call.
  4. Mobile experimentation. SDK-based assignment with offline support is the platform's job, not the warehouse's.
  5. Out-of-the-box sequential testing with strict guarantees. Statsig and Eppo ship mSPRT with calibrated alpha-spending. Building this in-house is real work.

Detail and a decision tree in references/warehouse-vs-platform-decision.md. Many mature teams use both; warehouse-native for the hard cases, platform for fast iteration on standard experiments.


The architecture

Four components, in order of data flow.

  1. Assignment. How users get bucketed into variants. Hash function, feature flag, or randomized assignment table.
  2. Exposure logging. A discrete event fired the first time a user is exposed to the experiment, written to the warehouse like any other event.
  3. Metric definitions. SQL queries (or dbt models) that compute the primary and secondary metrics from warehouse events.
  4. Analysis. Statistical computation in SQL or Python that joins exposure to metrics and produces effect estimates with confidence intervals.

The flow. User visits the product. Assignment determines the bucket (control or treatment). If the user is exposed to the variant (sees the treatment-specific behavior), an exposure event fires to the warehouse. The user takes actions, generating metric events to the same warehouse. At analysis time, exposure joins to metrics on the assignment unit (typically user_id); the analysis computes lift and produces a decision.

The exposure-event pattern is critical. Without it you can compute only an "intent-to-treat" analysis (everyone assigned, regardless of whether they saw the variant). With it you compute the "exposed" analysis on the population that actually experienced the variant. The latter is usually what you want, especially when the variant only affects a subset of the assigned users (e.g., users who reached a specific page).


Assignment patterns

Three approaches.

Deterministic hash assignment. The default for warehouse-native.

MOD(ABS(FARM_FINGERPRINT(CONCAT(user_id, 'exp_button_color_v1'))), 100) < 50

The salt ('exp_button_color_v1') ensures different experiments produce uncorrelated assignments for the same user. Reproducible (same input always produces the same bucket), no service dependency, salt isolation across experiments. The assignment can be computed inline in any SQL query.

Feature flag assignment. Rely on a feature flag service (LaunchDarkly, Statsig flags, Unleash, internal) to do bucketing; the warehouse just records the assignment that the flag service chose.

-- Read assignment from the flag service's logs
SELECT user_id, variant_id, assigned_at
FROM flag_service.assignments
WHERE flag_key = 'exp_button_color_v1'

This works when the flag service is the source of truth for assignment and the warehouse mirrors the assignment table. Useful when assignment must respect flag-service rules (e.g., percentage rollouts, targeting rules) that are inconvenient to replicate in SQL.

Randomized assignment table. Pre-randomize users into a table at experiment start.

CREATE TABLE exp_button_color_v1_assignments AS
SELECT
  user_id,
  CASE WHEN RAND() < 0.5 THEN 'control' ELSE 'treatment' END AS variant_id
FROM dim_users
WHERE eligible = true;

Less common; useful when the eligibility set is fixed at experiment start and you want assignment to be deterministic and explicit (e.g., for compliance audit). The downside: new users joining mid-experiment are not in the table; either skip them or fall back to hash assignment.

The deterministic hash approach is the default for warehouse-native because it requires no service dependency and produces stable, auditable assignments. Detail in references/assignment-and-exposure-patterns.md.


The exposure log

The single most important discipline in warehouse-native experimentation.

Required exposure event schema:

| Field | Type | Notes | |---|---|---| | experiment_id | string | Unique identifier per experiment. | | variant_id | string | The variant the user was bucketed into. | | user_id | string | The assignment unit. | | exposed_at | timestamp | ISO 8601 UTC. The moment exposure fired. | | context_* | various | Optional context properties: device, page, account_id. |

Fire exposure exactly when the user has seen the variant-specific behavior. Not at page load. Not at session start. Not at app open.

The "delayed exposure" trap. If the variant only matters at button click and you fire exposure at page load, every page-load user enters the analysis whether or not they ever saw the variant. The control group includes users who never reached the button; the treatment group does too. The analysis dilutes the real effect.

Worked example. The treatment shows a new pricing page; the control shows the old one. Fire exposure when the pricing page loads, not when the user lands on the homepage. Users who never reach the pricing page are not exposed to either variant; they should not be in the analysis.

The "always-fire" trap. Some implementations fire exposure on every variant-specific interaction. The user clicks the button five times; exposure fires five times. The exposure log is now five times larger than it should be, and analysis tools that count distinct user_ids in exposure handle this correctly while tools that count rows do not.

The discipline. Fire exactly one exposure event per user per experiment, at the moment of first variant-specific exposure. Use a deterministic flag in the client (or a server-side cache) to enforce single-fire. Detail in references/assignment-and-exposure-patterns.md.


Metric definitions in dbt models

Defining metrics as dbt models gives you four things.

  1. Version control on metric definitions. Every change to a metric is a git commit. The history is queryable.
  2. Testability. dbt tests on the metric outp

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars902
CategoryData
Updated2mo ago
Forks124

Languages

JavaScript

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions