SkillAgentSearch skills...

senior-data-engineer

Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka

Install / Use

npx skills add borghei/Claude-Skills --skill senior-data-engineer

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

81/100

Category

Automation

Supported Platforms

Universal

Our assessment of senior-data-engineer

senior-data-engineer scores 81/100 on our quality scale, 2418th of 2,769 Automation skills we index.

Its SKILL.md is 5.4 KB long, well organised into 13 sections with 1 code example: a solid amount of guidance for an agent.

It has 817 GitHub stars, a meaningful sign that others use it.

Substance
26/30
Structure
17/20
Description
12/15
Adoption
12/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so senior-data-engineer is actively maintained.
  • Our last check on 2026-11-08 found the source still online.
  • No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
  • Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-08. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

senior-data-engineer compared with similar skills

All 4 of these similar skills score higher than senior-data-engineer; compare them before choosing.

SkillScoreStarsUpdatedFormat
senior-data-engineer (this skill)by borghei8181757d agoSKILL.md
taste-skillby Leonxlnx100117.6k9d agoCLAUDE.md
Agent-Reachby Panniantong100112.7k9d agoCLAUDE.md
rufloby ruvnet10082.0ktodayCLAUDE.md
headroomby headroomlabs-ai10079.9ktodayCLAUDE.md

Frequently asked questions

How do I install senior-data-engineer?
Run npx skills add borghei/Claude-Skills --skill senior-data-engineer. The install tabs above show the steps for each supported agent.
Which AI agents does senior-data-engineer work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is senior-data-engineer safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is senior-data-engineer still maintained?
The repository was last updated about 2 months ago, so senior-data-engineer is actively maintained.

name: senior-data-engineer description: > Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures. license: MIT + Commons Clause metadata: version: 1.2.0 author: borghei category: engineering domain: data-engineering updated: 2026-06-17 tags: [airflow, spark, data-pipelines, warehousing, etl] python-tools: pipeline_orchestrator.py, data_quality_validator.py, etl_performance_optimizer.py tech-stack: python, sql, spark, airflow, dbt, kafka

Senior Data Engineer

Generate pipeline configurations (Airflow, Prefect, Dagster), validate data quality with profiling and anomaly detection, and optimize SQL/Spark performance with actionable recommendations.

Core Capabilities

  • Pipeline generation — Airflow/Prefect/Dagster DAG code for batch and incremental loads, with DAG validation.
  • Data quality — schema validation, profiling, anomaly detection, data contracts, and Great Expectations suite generation.
  • ETL/ELT optimization — SQL and Spark analysis, partition strategy, and query cost estimation per warehouse.
  • Architecture decisions — batch vs streaming and warehouse vs lakehouse trade-off frameworks.
  • Reliability patterns — incremental watermarks, dead letter queues, freshness checks, and schema-drift detection.

When to Use

  • Designing a data architecture or choosing batch vs streaming / warehouse vs lakehouse.
  • Building or generating Airflow/Spark/dbt pipelines.
  • Adding data-quality checks or data contracts.
  • Optimizing slow ETL/ELT queries or troubleshooting pipeline failures.

Clarify First

Before generating pipelines, confirm these inputs. If any is unknown or vague, ASK — do not assume:

  • [ ] Orchestrator — Airflow / Prefect / Dagster (--type; changes the generated DAG code)
  • [ ] Source, destination & load mode — systems involved and batch vs incremental (--source/--destination/--mode; shapes the pipeline)
  • [ ] Data-quality expectations — the schema and contracts to enforce (drives the Great Expectations suite generation)

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.

Quick Start

# Generate an Airflow DAG for incremental PostgreSQL -> Snowflake
python scripts/pipeline_orchestrator.py generate \
  --type airflow --source postgres --destination snowflake \
  --tables orders,customers --mode incremental --schedule "0 5 * * *"

# Validate data quality against a schema
python scripts/data_quality_validator.py validate data.csv \
  --schema schema.json --detect-anomalies --json

# Profile a dataset
python scripts/data_quality_validator.py profile data.csv --json

# Optimize a slow SQL query
python scripts/etl_performance_optimizer.py analyze-sql query.sql \
  --warehouse snowflake --json

# Estimate query cost
python scripts/etl_performance_optimizer.py estimate-cost query.sql \
  --warehouse bigquery --stats data_stats.json --json

Tools

| Tool | Subcommands | Purpose | |------|-------------|---------| | pipeline_orchestrator.py | generate, validate, template | Generate Airflow/Prefect/Dagster pipeline code, validate DAGs | | data_quality_validator.py | validate, profile, generate-suite, contract, schema | Schema validation, profiling, anomaly detection, Great Expectations | | etl_performance_optimizer.py | analyze-sql, analyze-spark, optimize-partition, estimate-cost, template | SQL/Spark optimization, partition strategy, cost estimation |

All subcommands support --json for machine-readable output and --output for file writing.

References

Load the reference that matches the task — keep this file lean and pull detail on demand:

Integration Points

| Skill | Integration | |-------|-------------| | senior-data-scientist | Feature engineering consumes curated mart data | | senior-ml-engineer | ML pipelines depend on feature store tables | | senior-devops | CI/CD for dbt, Airflow deployment, container orchestration | | senior-architect | Architecture reviews for lakehouse vs warehouse decisions | | code-reviewer | Pipeline code reviews for DAGs, dbt models, Spark jobs |

Related Skills

View on GitHub
GitHub Stars817
CategoryAutomation
Updated1mo ago
Forks141

Languages

HTML

Trust signals

88/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 medium