bigquery-migration-validator
Validates that tables migrated to BigQuery match their source (Teradata, Snowflake, Oracle, Redshift, Databricks, PostgreSQL, SQL Server, MySQL, MariaDB, Hive, Impala, Vertica) with the BigQuery Migration Service (BQMS) validation tool: schema and data type comparison, column aggregate comparison, a…
Install / Use
npx skills add google/skills --skill bigquery-migration-validatorInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
Data & AnalyticsSupported Platforms
Our assessment of bigquery-migration-validator
bigquery-migration-validator scores 95/100 on our quality scale, 70th of 604 Data & Analytics skills we index (top 12%).
Its SKILL.md is 17 KB long, well organised into 16 sections with 1 code example: a thorough specification that gives an agent plenty to work with.
With 20,340 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so bigquery-migration-validator is actively maintained.
- Our last check on 2026-11-15 found the source still online.
- It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-11-15. Automated pattern scan on 2026-11-15. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
bigquery-migration-validator compared with similar skills
All 4 of these similar skills score higher than bigquery-migration-validator; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| bigquery-migration-validator (this skill)by google | 95 | 20.3k | 54d ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| x402by browser-use | 100 | 121.0k | 3d ago | SKILL.md |
| algorithmic-artby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
| pptxby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
Frequently asked questions
- How do I install bigquery-migration-validator?
- Run
npx skills add google/skills --skill bigquery-migration-validator. The install tabs above show the steps for each supported agent. - Which AI agents does bigquery-migration-validator work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is bigquery-migration-validator safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is bigquery-migration-validator still maintained?
- The repository was last updated about 2 months ago, so bigquery-migration-validator is actively maintained.
Skill content
View source on GitHubname: bigquery-migration-validator metadata: version: "1.0.0" category: BigDataAndAnalytics publisher: google description: >- Validates that tables migrated to BigQuery match their source (Teradata, Snowflake, Oracle, Redshift, Databricks, PostgreSQL, SQL Server, MySQL, MariaDB, Hive, Impala, Vertica) with the BigQuery Migration Service (BQMS) validation tool: schema and data type comparison, column aggregate comparison, and row-level data comparison. Use when generating or editing a validation config, running a validation, choosing sampling or partitioning, fixing connection or driver errors, or investigating validation results in BigQuery and Cloud Storage to decide which mismatches to fix. Don't use for SQL translation (use bigquery-migration-sql-translator), extracting source database catalog metadata or query logs, generating SQL scripts (such as COPY INTO) or exporting or unloading source tables to Parquet on Cloud Storage, loading staged files into BigQuery (such as bq load), or general BigQuery query failures and tuning (use bigquery-troubleshooting or bigquery-optimization).
BigQuery Migration Validator
The validation tool compares a source table with its migrated BigQuery table. It runs in two halves:
- Client (this machine): Reads the config, queries the source database, and stages a row sample, source aggregates, and run metadata in Cloud Storage. It then starts a BigQuery Migration workflow for each table and exits.
- Backend (Google Cloud, asynchronous): Loads the staged sample into BigQuery, compares it with the target table, and writes the results to BigQuery and Cloud Storage.
Results do not exist when the command exits. Wait for the workflow to finish before reading them (Section 4).
Usage attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
Before you begin
-
Google Cloud CLI and project: Ensure the [Google Cloud CLI][gcloud_install] is installed, verify that the Google Cloud project has an active Cloud Billing account, and configure the target project (check regional service availability at [Google Cloud locations][cloud_locations] when choosing
--locationfor validation workflows):gcloud config set project PROJECT_ID -
Enable the APIs (ask the user before enabling APIs): BigQuery (
bigquery.googleapis.com), BigQuery Migration (bigquerymigration.googleapis.com), Cloud Storage (storage.googleapis.com), and Secret Manager (secretmanager.googleapis.com, when reading source passwords from Secret Manager):gcloud services enable \ bigquery.googleapis.com \ bigquerymigration.googleapis.com \ storage.googleapis.com # Only when source passwords come from Secret Manager: gcloud services enable secretmanager.googleapis.com -
Local runtime, authentication, and IAM permissions (ask before installing software):
- Runtime and drivers: Java 11 or later (JRE 11+) first on
PATH, thebqmsgcloud component (installed on first use or withgcloud components install bqms), local network access to the source database, and vendor JDBC driver JARs for Teradata (terajdbc4.jar), Vertica (vertica-jdbc.jar), Impala, and DuckDB in./driversorBQMS_DRIVER_DIRwhen required (other JDBC drivers download on first use; see connections and drivers). - Google Cloud credentials and IAM: Authenticate the Google Cloud CLI
and Application Default Credentials with
gcloud auth loginandgcloud auth application-default login(or setGOOGLE_APPLICATION_CREDENTIALSfor a service account). The identity needs write access to the staging Cloud Storage bucket (roles/storage.objectAdmin), permission to create BigQuery Migration workflows (roles/bigquerymigration.editor), and permission to run BigQuery jobs and write validation results in the target project and, if set, theresults.projectIdproject (roles/bigquery.dataEditorandroles/bigquery.jobUser). With Secret Manager passwords, it also needsroles/secretmanager.secretAccessoron the secret, orroles/iam.serviceAccountTokenCreatoron the service account named insecretImpersonateServiceAccount(which then needs the accessor role).
- Runtime and drivers: Java 11 or later (JRE 11+) first on
-
Cost and cleanup: Review Google Cloud Pricing for BigQuery, Cloud Storage, and Secret Manager. Staged samples in Cloud Storage contain source rows. When validation is complete, ask the user before deleting the staged run directory (a
gcs_pathsentry inrun_state.json) and any per-run tables they no longer need (list them as in the results reference, Section 5). Deleting the run directory also deletes the backend reports stored there (results.json,validation_summary.txt,column_details.txt,queries.sql) andsample_metadata.json, sorun_history.gcs_urino longer resolves; say so and offer to copy the reports locally first:gcloud storage cp GCS_RUN_PATH/results.json \ GCS_RUN_PATH/validation_summary.txt GCS_RUN_PATH/column_details.txt \ GCS_RUN_PATH/queries.sql LOCAL_DIR/ gcloud storage rm --recursive GCS_RUN_PATH bq rm -f -t RESULTS_PROJECT:DATASET.TABLE_NAME
1. What gets compared
Pillar | --validation-types | Compares | Coverage
:------------ | :------------------- | :-------------------------------------------------------------------------------------------------------------------------- | :-------
Schema | schema | Column presence, mapping, and data type compatibility (one of 9 compatibility notes) | All columns
Aggregate | aggregate | Column statistics per type (COUNT, SUM, AVG, MIN, MAX, SUM_LENGTH, ...), computed per partition, then rolled up | 100% of rows (pushed down)
Data | data | Cell-by-cell values joined on comparison keys, with a distance per mismatched cell | Sampled or full rows
With --validation-types omitted, all three run. With --validation-types set
(for example --validation-types=schema,aggregate), only the listed types run.
Query (code) validation is not available in this tool.
Each pillar catches what the others miss. Aggregates cover every row cheaply but cannot see offsetting errors (one row +5, another -5). Row sampling sees those but only in the sample. Run both for sign-off.
2. Quick start
Both validation commands run in the alpha release track through the Google
Cloud CLI, which calls the bqms component (Java 11+ required). Arguments after
-- are forwarded unchanged to the validation client.
[!IMPORTANT] The commands are in the alpha track. If gcloud reports an invalid choice or unknown command, ask the user before running
gcloud components update(orgcloud components install alpha bqms). If the Google Cloud CLI was installed with a system package manager such asaptoryum, the component manager is disabled; have the user upgradegoogle-cloud-clithrough that package manager instead. If the command is still unavailable, stop and point the user to the CLI reference and the [installation guide][gcloud_install] rather than retrying.
# 1. Generate per-table configs offline from extractor metadata or BQMS
# translation output (writes ./validation_config/).
gcloud alpha bq migration validation configure \
--metadata=dwh-migration-oracle-metadata.zip \
--target-dataset=warehouse \
--output=validation_config
# 2. Edit validation_config/connections.yaml (check the source dialect;
# set the target dataset) and settings.yaml (googleCloudStorage.gcsPath),
# then run schema and aggregate checks first.
gcloud alpha bq migration validation run validation_config/ \
--location=us --validation-types=schema,aggregate
# 3. Run row-level comparison, optionally on a subset of tables.
gcloud alpha bq migration validation run validation_config/ \
--location=us --validation-types=data --tables=orders
Then wait for the workflows (Section 4) and investigate (Section 5).
Every answer that runs or recommends validation run must tell the user that
the comparison finishes later in a BigQuery Migration workflow, that exit code 0
does not mean the data matched, and how to check the workflow before reading
results. Users who skip this read an empty or stale results dataset as a pass.
3. Configure
Settings come from YAML (modular validation_config/ directory or one
validation-config.yaml), then CLI flags. The essentials:
connections: asourceand a BigQuerytarget. The target needsprojectIdanddataset; withoutdatasetthe run fails, even when everytargetTable.nameis fully qualified.configurecan scaffold the wrong sourcedialect(for examplesnowflakefor PostgreSQL metadata), so check it and its properties before the first run. Connection names are letters and digits only; a name with_or-fails config validation. If the user asks for such a name, drop the separators (for examplesales-dbbecomessalesdb) and tell them why, because the name also forms the environment variable prefix. Keep secrets out of YAML: useBQMS_VALIDATION_<CONNECTION>_PASSWORDor a Secret Manager reference (see connections and drivers). Never ask the user to type a password into the conversation and never echo one. For commands you run yourself, use a Secret Manager reference (passwordSecretorBQMS_VALIDATION_<CONNECTION>_PASSWORD_SECRET); give plaintextexportlines only for the user to run in their own terminal.googleCloudStorage.gcsPath: required staging bucket (gs://bucket/path, no trailing slash). Results artifacts also land here.- Per table:
sourceTableandtargetTable(each withnameand, undersourceTable,comparisonKeys: the columns that uniquely identify a row for the validation join when the table has no primary key — explain thatcomparisonKeysonly control how validation matches rows, so no database primary key or schema change is needed;configurefills them in only when the metadata includes primary keys),columnSettings(exclusions, transforms, tolerances),sampleSize, andpartitioning. results.datasetand optionalresults.projectId: BigQuery dataset and project for results (defaultsvalidation_resultsand the target connection's project). Query results where these point.
Details: configuration reference.
4. Run and wait
- Run
gcloud alpha bq migration validation run. Exit code 1 means a config or source-side failure; read the error and fix it. Exit code 0 means the samples were staged (or that usage was printed for a bad forwarded flag; check the output). Mismatches never change the exit code. - Confirm a workflow started. The client prints a workflow name per table and
writes
run_state.json(run ID, Cloud Storage paths, workflow names, results dataset) to--output-dir(default/tmp/validation_outputs, passed after--). If it instead prints "Could not initiate BigQuery Migration validation workflow", fix the cause it reports (usually permissions or the API not enabled) and re-run
Truncated for display — read the full file on GitHub.
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
x402
121.0kSet up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key.
algorithmic-art
177.9kCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems.
pptx
177.9kUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an em…
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
