senior-data-engineer
Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka
Install / Use
npx skills add borghei/Claude-Skills --skill senior-data-engineerInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AutomationSupported Platforms
Our assessment of senior-data-engineer
senior-data-engineer scores 81/100 on our quality scale, 2418th of 2,769 Automation skills we index.
Its SKILL.md is 5.4 KB long, well organised into 13 sections with 1 code example: a solid amount of guidance for an agent.
It has 817 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so senior-data-engineer is actively maintained.
- Our last check on 2026-11-08 found the source still online.
- No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
- Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-08. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
senior-data-engineer compared with similar skills
All 4 of these similar skills score higher than senior-data-engineer; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| senior-data-engineer (this skill)by borghei | 81 | 817 | 57d ago | SKILL.md |
| taste-skillby Leonxlnx | 100 | 117.6k | 9d ago | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.7k | 9d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 82.0k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.9k | today | CLAUDE.md |
Frequently asked questions
- How do I install senior-data-engineer?
- Run
npx skills add borghei/Claude-Skills --skill senior-data-engineer. The install tabs above show the steps for each supported agent. - Which AI agents does senior-data-engineer work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is senior-data-engineer safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is senior-data-engineer still maintained?
- The repository was last updated about 2 months ago, so senior-data-engineer is actively maintained.
Skill content
View source on GitHubname: senior-data-engineer description: > Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures. license: MIT + Commons Clause metadata: version: 1.2.0 author: borghei category: engineering domain: data-engineering updated: 2026-06-17 tags: [airflow, spark, data-pipelines, warehousing, etl] python-tools: pipeline_orchestrator.py, data_quality_validator.py, etl_performance_optimizer.py tech-stack: python, sql, spark, airflow, dbt, kafka
Senior Data Engineer
Generate pipeline configurations (Airflow, Prefect, Dagster), validate data quality with profiling and anomaly detection, and optimize SQL/Spark performance with actionable recommendations.
Core Capabilities
- Pipeline generation — Airflow/Prefect/Dagster DAG code for batch and incremental loads, with DAG validation.
- Data quality — schema validation, profiling, anomaly detection, data contracts, and Great Expectations suite generation.
- ETL/ELT optimization — SQL and Spark analysis, partition strategy, and query cost estimation per warehouse.
- Architecture decisions — batch vs streaming and warehouse vs lakehouse trade-off frameworks.
- Reliability patterns — incremental watermarks, dead letter queues, freshness checks, and schema-drift detection.
When to Use
- Designing a data architecture or choosing batch vs streaming / warehouse vs lakehouse.
- Building or generating Airflow/Spark/dbt pipelines.
- Adding data-quality checks or data contracts.
- Optimizing slow ETL/ELT queries or troubleshooting pipeline failures.
Clarify First
Before generating pipelines, confirm these inputs. If any is unknown or vague, ASK — do not assume:
- [ ] Orchestrator — Airflow / Prefect / Dagster (
--type; changes the generated DAG code) - [ ] Source, destination & load mode — systems involved and batch vs incremental (
--source/--destination/--mode; shapes the pipeline) - [ ] Data-quality expectations — the schema and contracts to enforce (drives the Great Expectations suite generation)
Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.
Quick Start
# Generate an Airflow DAG for incremental PostgreSQL -> Snowflake
python scripts/pipeline_orchestrator.py generate \
--type airflow --source postgres --destination snowflake \
--tables orders,customers --mode incremental --schedule "0 5 * * *"
# Validate data quality against a schema
python scripts/data_quality_validator.py validate data.csv \
--schema schema.json --detect-anomalies --json
# Profile a dataset
python scripts/data_quality_validator.py profile data.csv --json
# Optimize a slow SQL query
python scripts/etl_performance_optimizer.py analyze-sql query.sql \
--warehouse snowflake --json
# Estimate query cost
python scripts/etl_performance_optimizer.py estimate-cost query.sql \
--warehouse bigquery --stats data_stats.json --json
Tools
| Tool | Subcommands | Purpose |
|------|-------------|---------|
| pipeline_orchestrator.py | generate, validate, template | Generate Airflow/Prefect/Dagster pipeline code, validate DAGs |
| data_quality_validator.py | validate, profile, generate-suite, contract, schema | Schema validation, profiling, anomaly detection, Great Expectations |
| etl_performance_optimizer.py | analyze-sql, analyze-spark, optimize-partition, estimate-cost, template | SQL/Spark optimization, partition strategy, cost estimation |
All subcommands support --json for machine-readable output and --output for file writing.
References
Load the reference that matches the task — keep this file lean and pull detail on demand:
- references/pipeline-workflows.md — the three end-to-end worked pipelines with code: batch ETL (PostgreSQL → dbt → Snowflake), real-time streaming (Kafka → Spark → Delta Lake), and the data-quality framework. Read when building a concrete pipeline.
- references/decisions-and-troubleshooting.md — the batch-vs-streaming and warehouse-vs-lakehouse decision frameworks, anti-patterns, and the troubleshooting table. Read when choosing an architecture or diagnosing a failure.
- references/data_pipeline_architecture.md — deep reference on pipeline architecture patterns. Read for architecture design depth.
- references/data_modeling_patterns.md — dimensional modeling and data-modeling patterns. Read when modeling marts and dimensions.
- references/dataops_best_practices.md — DataOps practices for CI/CD, testing, and operating pipelines. Read when operationalizing pipelines.
Integration Points
| Skill | Integration |
|-------|-------------|
| senior-data-scientist | Feature engineering consumes curated mart data |
| senior-ml-engineer | ML pipelines depend on feature store tables |
| senior-devops | CI/CD for dbt, Airflow deployment, container orchestration |
| senior-architect | Architecture reviews for lakehouse vs warehouse decisions |
| code-reviewer | Pipeline code reviews for DAGs, dbt models, Spark jobs |
Related Skills
taste-skill
117.6kAgent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages
Agent-Reach
112.7kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
ruflo
82.0k🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
headroom
79.9kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
