ml-data-leakage-guard
Detects and prevents data leakage in machine learning and mathematical modeling. Use after ML tasks involving data cleaning, feature engineering, data augmentation, algorithm development, normalization, missing value imputation, dimensionality reduction, feature selection, or time series modeling.
Install / Use
npx skills add foryourhealth111-pixel/Vibe-Skills --skill ml-data-leakage-guardInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AutomationSupported Platforms
Tags
Our assessment of ml-data-leakage-guard
ml-data-leakage-guard scores 95/100 on our quality scale, 277th of 2,780 Automation skills we index (top 10%).
Its SKILL.md is 13 KB long, well organised into 45 sections with 21 code examples: a thorough specification that gives an agent plenty to work with.
With 3,433 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 3 months ago, so ml-data-leakage-guard is actively maintained.
- Our last check on 2026-10-24 found the source still online.
- It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit β read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-02. Automated pattern scan on 2026-10-02. It catches known dangerous patterns, not every risk β read a skill before letting an agent act on it.
ml-data-leakage-guard compared with similar skills
All 4 of these similar skills score higher than ml-data-leakage-guard; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| ml-data-leakage-guard (this skill)by foryourhealth111-pixel | 95 | 3.4k | 3mo ago | SKILL.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 81.9k | today | CLAUDE.md |
| Claude-Code-Game-Studiosby Donchitos | 100 | 39.7k | 7d ago | CLAUDE.md |
| Scraplingby D4Vinci | 100 | 103.6k | 2d ago | MCP Server |
Frequently asked questions
- How do I install ml-data-leakage-guard?
- Run
npx skills add foryourhealth111-pixel/Vibe-Skills --skill ml-data-leakage-guard. The install tabs above show the steps for each supported agent. - Which AI agents does ml-data-leakage-guard work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is ml-data-leakage-guard safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is ml-data-leakage-guard still maintained?
- The repository was last updated about 3 months ago, so ml-data-leakage-guard is actively maintained.
Skill content
View source on GitHubname: ml-data-leakage-guard description: "Detects and prevents data leakage in machine learning and mathematical modeling. Use after ML tasks involving data cleaning, feature engineering, data augmentation, algorithm development, normalization, missing value imputation, dimensionality reduction, feature selection, or time series modeling. Checks if features/statistics would be available at prediction time."
ML Data Leakage Guard Skill
Automatically detects and prevents data leakage in machine learning workflows by verifying that all preprocessing steps, feature engineering, and statistical computations would be available at prediction time.
When to Use This Skill
Use this skill after work involving:
- Data preprocessing (normalization, standardization, scaling)
- Missing value imputation
- Feature engineering and feature selection
- Dimensionality reduction (PCA, SVD, t-SNE)
- Target encoding or label encoding
- Time series feature construction
- Data augmentation strategies
- Algorithm development and optimization
- Train-test split procedures
- Cross-validation setup
Not For / Boundaries
- Pure theoretical ML discussions without implementation
- Model architecture design (without data preprocessing)
- Hyperparameter tuning (unless it involves data-dependent operations)
Core Principle
The Golden Rule: At the exact moment of prediction in production, can I access this value from the database or compute it using only information available up to that point?
If the answer is "no" or "not completely", then data leakage exists.
Quick Reference
Critical Leakage Patterns
Pattern 1: Preprocessing Before Split
# β WRONG: Leakage - fit on entire dataset
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # Uses test set statistics
X_train, X_test = train_test_split(X_scaled, y)
# β
CORRECT: Fit only on training data
X_train, X_test, y_train, y_test = train_test_split(X, y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Fit on train only
X_test_scaled = scaler.transform(X_test) # Transform test using train statistics
Pattern 2: Global Missing Value Imputation
# β WRONG: Uses global statistics including test set
df['age'].fillna(df['age'].mean(), inplace=True) # Global mean includes test data
X_train, X_test = train_test_split(df, y)
# β
CORRECT: Compute statistics on training set only
X_train, X_test, y_train, y_test = train_test_split(df, y)
train_mean = X_train['age'].mean() # Only from training data
X_train['age'].fillna(train_mean, inplace=True)
X_test['age'].fillna(train_mean, inplace=True) # Use train mean for test
Pattern 3: PCA/Dimensionality Reduction on Full Dataset
# β WRONG: PCA learns variance structure from test set
pca = PCA(n_components=10)
X_reduced = pca.fit_transform(X) # Includes test set variance
X_train, X_test = train_test_split(X_reduced, y)
# β
CORRECT: Fit PCA only on training data
X_train, X_test, y_train, y_test = train_test_split(X, y)
pca = PCA(n_components=10)
X_train_reduced = pca.fit_transform(X_train) # Learn from train only
X_test_reduced = pca.transform(X_test) # Apply train-learned transformation
Pattern 4: Target Encoding with Full Dataset
# β WRONG: Uses target values from test set
category_means = df.groupby('category')['target'].mean() # Includes test targets
df['category_encoded'] = df['category'].map(category_means)
X_train, X_test = train_test_split(df, y)
# β
CORRECT: Compute encoding only from training targets
X_train, X_test, y_train, y_test = train_test_split(df, y)
category_means = X_train.groupby('category')['target'].mean() # Train only
X_train['category_encoded'] = X_train['category'].map(category_means)
X_test['category_encoded'] = X_test['category'].map(category_means)
Pattern 5: Feature Selection on Full Dataset
# β WRONG: Feature selection sees test set
from sklearn.feature_selection import SelectKBest
selector = SelectKBest(k=10)
X_selected = selector.fit_transform(X, y) # Uses test set for selection
X_train, X_test = train_test_split(X_selected, y)
# β
CORRECT: Select features using training data only
X_train, X_test, y_train, y_test = train_test_split(X, y)
selector = SelectKBest(k=10)
X_train_selected = selector.fit_transform(X_train, y_train) # Train only
X_test_selected = selector.transform(X_test) # Apply train-learned selection
Pattern 6: Random Split on Temporal Data
# β WRONG: Random split on time series (uses future to predict past)
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
# β
CORRECT: Time-based split for temporal data
split_date = '2024-01-01'
X_train = df[df['date'] < split_date]
X_test = df[df['date'] >= split_date]
Pattern 7: Future Function in Time Series Features
# β WRONG: Uses future data to compute current features
df['daily_avg'] = df.groupby('date')['value'].transform('mean') # Includes all day's data
# β
CORRECT: Use only past data (expanding window)
df = df.sort_values('timestamp')
df['cumulative_avg'] = df.groupby('user_id')['value'].expanding().mean().reset_index(0, drop=True)
Pattern 8: Post-Event Features
# β WRONG: Feature only exists after the outcome
# Predicting loan default using "number of collection calls" as feature
# Collection calls only happen AFTER default occurs
# β
CORRECT: Use only pre-event features
# Use features available BEFORE the outcome: credit score, income, debt ratio, etc.
Pattern 9: Leakage in Cross-Validation
# β WRONG: Preprocessing before CV split
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
scores = cross_val_score(model, X_scaled, y, cv=5) # Each fold sees other folds' statistics
# β
CORRECT: Preprocessing inside CV pipeline
from sklearn.pipeline import Pipeline
pipeline = Pipeline([
('scaler', StandardScaler()),
('model', LogisticRegression())
])
scores = cross_val_score(pipeline, X, y, cv=5) # Scaling happens per fold
Pattern 10: Data Augmentation Leakage
# β WRONG: Augment before split (test set influenced by augmented train data)
X_augmented = augment_data(X) # Augmentation sees all data
X_train, X_test = train_test_split(X_augmented, y)
# β
CORRECT: Augment only training data after split
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train_augmented = augment_data(X_train) # Augment train only
# X_test remains unchanged
Leakage Detection Checklist
After any ML preprocessing or feature engineering, verify:
- Fit-Transform Order: Did I fit any transformer (scaler, encoder, imputer, PCA) on the full dataset before splitting?
- Global Statistics: Did I compute any statistics (mean, std, median, mode) using the entire dataset?
- Feature Selection: Did I select features using information from the test set?
- Target Information: Did any feature encoding use target values from the test set?
- Temporal Order: For time series, did I use random splits or include future information?
- Post-Event Features: Are any features only available after the outcome occurs?
- Cross-Validation: Did I preprocess before setting up CV folds?
- Production Reality: At prediction time, can I actually compute this feature with available data?
The "Prediction Time" Test
For every feature and preprocessing step, ask:
When the model is deployed and receives a new data point at time T:
- Can I query this value from the database?
- Can I compute this statistic using only data available before time T?
- Does this feature require knowing the outcome I'm trying to predict?
If any answer is "NO", you have data leakage.
Examples
Example 1: Detecting Normalization Leakage
Input: Code that normalizes data before train-test split
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_normalized = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_normalized, y, test_size=0.2)
Leakage Detection:
- β LEAKAGE DETECTED:
fit_transformon entire dataset X - The scaler learned mean and std from the full dataset, including test set
- At prediction time, you won't have access to test set statistics
- Test set performance is artificially inflated
Corrected Code:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
scaler = StandardScaler()
X_train_normalized = scaler.fit_transform(X_train) # Fit on train only
X_test_normalized = scaler.transform(X_test) # Transform using train statistics
Example 2: Detecting Missing Value Imputation Leakage
Input: Code that fills missing values with global mean
df = pd.read_csv('data.csv')
df['income'].fillna(df['income'].mean(), inplace=True)
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2)
Leakage Detection:
- β LEAKAGE DETECTED: Global mean computed on entire dataset
- The mean includes test set values
- At prediction time, you won't know the test set mean
- This inflates test performance
Corrected Code:
df = pd.read_csv('data.csv')
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2)
train_mean = X_train['income'].mean() # Compute mean from training data only
X_train['income'].fillna(train_mean, inplace=True)
X_test['income'].fillna(train_mean, inplace=True) # Use training mean for test
Example 3: Detecting Time Series Leakage
Input: Stock price prediction with random split and rolling average feature
df['rolling_avg_7d'] = df.groupby('stock')['price'].rolling(7, center=True).mean()
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
Leakage Detection:
- β LEAKAGE DETECTED - Multiple Issues:
center=Truein rolling window uses future prices (t+3 days) to compute feature at time t- Random split on temporal data means using "tomorrow's data" to predict "yesterday"
- At prediction time, you can't access future prices
Corrected Code:
# Fix 1: Use backward-looking rolling window (no center=True)
df = df.sort_values(['stock', 'date'])
df['rolling_avg_7d'] = df.groupby('stock')['price'].rolling(7, min_periods=1).mean().reset_index(0, drop=True)
# Fix 2: Time-based split instead of random
split_date = '2024-01-01'
X_train = df[df['date'] < split_date]
X_test = df[df['date'] >= split_date]
Example 4: Detecting Target Encoding Leakage
Input: Category encoding using target mean from full dataset
category_target_mean = df.groupby('category')['target'].mean()
df['category_encoded'] = df['category'].map(category_target_mean)
X_train, X_test, y_train, y_test = train_test_split(df.drop('target', axis=1), df['target'])
Leakage Detection:
- β LEAKAGE DETECTED: Target encoding uses test set labels
- The encoding directly uses the answer (target) from test set
- This is severe leakage - you're literally using test labels as features
- At prediction time, you don't know the target value
Corrected Code:
X_train, X_test, y_train, y_test = train_test_split(df.drop('target', axis=1), df['target'])
# Compute target mean only from training data
train_df = X_train.copy()
train_df['target'] = y_train
category_target_mean = train_df.groupby('category')['target'].mean()
X_train['category_encoded'] = X_train['category'].map(category_target_mean)
X_test['category_encoded'] = X_test['category'].map(category_target_mean)
Example 5: Detecting Post-Event Feature Leakage
Input: Predicting customer churn using "number of retention calls" as feature
features = ['account_age', 'monthly_spend', 'support_tickets', 'retention_calls_count']
X = df[features]
y = df['churned']
Leakage Detection:
- β LEAKAGE DETECTED: Post-event feature
- "retention_calls_count" only exists AFTER customer shows churn signals
Truncated for display β read the full file on GitHub.
Related Skills
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu β one CLI, zero API fees.
ruflo
81.9kπ The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
Claude-Code-Game-Studios
39.7kTurn Claude Code into a full game dev studio β 49 AI agents, 72 workflow skills, and a complete coordination system mirroring real studio hierarchy.
Scrapling
103.6kπ·οΈ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit β see the Safety scan above for what the skill file itself contains.