bigquery-pipeline-audit
Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.
Install / Use
npx skills add github/awesome-copilot --skill bigquery-pipeline-auditInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AutomationSupported Platforms
Our assessment of bigquery-pipeline-audit
bigquery-pipeline-audit scores 88/100 on our quality scale, 1479th of 2,790 Automation skills we index.
Its SKILL.md is 4.9 KB long, well organised into 8 sections and no code examples: a solid amount of guidance for an agent.
With 39,348 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so bigquery-pipeline-audit is actively maintained.
- Our last check on 2026-10-31 found the source still online.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-09-28. Automated pattern scan on 2026-09-28. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
bigquery-pipeline-audit compared with similar skills
All 4 of these similar skills score higher than bigquery-pipeline-audit; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| bigquery-pipeline-audit (this skill)by github | 88 | 39.3k | 53d ago | SKILL.md |
| Agent-Reachby Panniantong | 100 | 112.3k | 8d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 81.9k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.8k | today | CLAUDE.md |
| Claude-Code-Game-Studiosby Donchitos | 100 | 39.7k | 7d ago | CLAUDE.md |
Frequently asked questions
- How do I install bigquery-pipeline-audit?
- Run
npx skills add github/awesome-copilot --skill bigquery-pipeline-audit. The install tabs above show the steps for each supported agent. - Which AI agents does bigquery-pipeline-audit work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is bigquery-pipeline-audit safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is bigquery-pipeline-audit still maintained?
- The repository was last updated about 2 months ago, so bigquery-pipeline-audit is actively maintained.
Skill content
View source on GitHubname: bigquery-pipeline-audit description: 'Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness. Returns a structured report with exact patch locations.'
BigQuery Pipeline Audit: Cost, Safety and Production Readiness
You are a senior data engineer reviewing a Python + BigQuery pipeline script. Your goals: catch runaway costs before they happen, ensure reruns do not corrupt data, and make sure failures are visible.
Analyze the codebase and respond in the structure below (A to F + Final). Reference exact function names and line locations. Suggest minimal fixes, not rewrites.
A) COST EXPOSURE: What will actually get billed?
Locate every BigQuery job trigger (client.query, load_table_from_*,
extract_table, copy_table, DDL/DML via query) and every external call
(APIs, LLM calls, storage writes).
For each, answer:
- Is this inside a loop, retry block, or async gather?
- What is the realistic worst-case call count?
- For each
client.query, isQueryJobConfig.maximum_bytes_billedset? For load, extract, and copy jobs, is the scope bounded and counted against MAX_JOBS? - Is the same SQL and params being executed more than once in a single run? Flag repeated identical queries and suggest query hashing plus temp table caching.
Flag immediately if:
- Any BQ query runs once per date or once per entity in a loop
- Worst-case BQ job count exceeds 20
maximum_bytes_billedis missing on anyclient.querycall
B) DRY RUN AND EXECUTION MODES
Verify a --mode flag exists with at least dry_run and execute options.
dry_runmust print the plan and estimated scope with zero billed BQ execution (BigQuery dry-run estimation via job config is allowed) and zero external API or LLM callsexecuterequires explicit confirmation for prod (--env=prod --confirm)- Prod must not be the default environment
If missing, propose a minimal argparse patch with safe defaults.
C) BACKFILL AND LOOP DESIGN
Hard fail if: the script runs one BQ query per date or per entity in a loop.
Check that date-range backfills use one of:
- A single set-based query with
GENERATE_DATE_ARRAY - A staging table loaded with all dates then one join query
- Explicit chunks with a hard
MAX_CHUNKScap
Also check:
- Is the date range bounded by default (suggest 14 days max without
--override)? - If the script crashes mid-run, is it safe to re-run without double-writing?
- For backdated simulations, verify data is read from time-consistent snapshots
(
FOR SYSTEM_TIME AS OF, partitioned as-of tables, or dated snapshot tables). Flag any read from a "latest" or unversioned table when running in backdated mode.
Suggest a concrete rewrite if the current approach is row-by-row.
D) QUERY SAFETY AND SCAN SIZE
For each query, check:
- Partition filter is on the raw column, not
DATE(ts),CAST(...), or any function that prevents pruning - No
SELECT *: only columns actually used downstream - Joins will not explode: verify join keys are unique or appropriately scoped and flag any potential many-to-many
- Expensive operations (
REGEXP,JSON_EXTRACT, UDFs) only run after partition filtering, not on full table scans
Provide a specific SQL fix for any query that fails these checks.
E) SAFE WRITES AND IDEMPOTENCY
Identify every write operation. Flag plain INSERT/append with no dedup logic.
Each write should use one of:
MERGEon a deterministic key (e.g.,entity_id + date + model_version)- Write to a staging table scoped to the run, then swap or merge into final
- Append-only with a dedupe view:
QUALIFY ROW_NUMBER() OVER (PARTITION BY <key>) = 1
Also check:
- Will a re-run create duplicate rows?
- Is the write disposition (
WRITE_TRUNCATEvsWRITE_APPEND) intentional and documented? - Is
run_idbeing used as part of the merge or dedupe key? If so, flag it.run_idshould be stored as a metadata column, not as part of the uniqueness key, unless you explicitly want multi-run history.
State the recommended approach and the exact dedup key for this codebase.
F) OBSERVABILITY: Can you debug a failure?
Verify:
- Failures raise exceptions and abort with no silent
except: passor warn-only - Each BQ job logs: job ID, bytes processed or billed when available, slot milliseconds, and duration
- A run summary is logged or written at the end containing:
run_id, env, mode, date_range, tables written, total BQ jobs, total bytes run_idis present and consistent across all log lines
If run_id is missing, propose a one-line fix:
run_id = run_id or datetime.utcnow().strftime('%Y%m%dT%H%M%S')
Final
1. PASS / FAIL with specific reasons per section (A to F). 2. Patch list ordered by risk, referencing exact functions to change. 3. If FAIL: Top 3 cost risks with a rough worst-case estimate (e.g., "loop over 90 dates x 3 retries = 270 BQ jobs").
Related Skills
Agent-Reach
112.3kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
ruflo
81.9k🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
headroom
79.8kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Claude-Code-Game-Studios
39.7kTurn Claude Code into a full game dev studio — 49 AI agents, 72 workflow skills, and a complete coordination system mirroring real studio hierarchy.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
