bigquery-migration-data-loader
Provisions BigQuery datasets, ingests staged Parquet files directly into native BigQuery tables by using bq load, and deploys already-translated views from local or Cloud Storage BigQuery Migration Service (BQMS) translation outputs
Install / Use
npx skills add google/skills --skill bigquery-migration-data-loaderInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
OperationsSupported Platforms
Tags
Our assessment of bigquery-migration-data-loader
bigquery-migration-data-loader scores 98/100 on our quality scale, 31st of 745 Operations skills we index (top 5%).
Its SKILL.md is 16 KB long, well organised into 12 sections with 5 code examples: a thorough specification that gives an agent plenty to work with.
With 20,340 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so bigquery-migration-data-loader is actively maintained.
- It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-11-15. Automated pattern scan on 2026-11-15. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
bigquery-migration-data-loader compared with similar skills
All 4 of these similar skills score higher than bigquery-migration-data-loader; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| bigquery-migration-data-loader (this skill)by google | 98 | 20.3k | 54d ago | SKILL.md |
| x402by browser-use | 100 | 121.0k | 3d ago | SKILL.md |
| algorithmic-artby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
| pptxby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
| designby nextlevelbuilder | 100 | 149.0k | 6d ago | SKILL.md |
Frequently asked questions
- How do I install bigquery-migration-data-loader?
- Run
npx skills add google/skills --skill bigquery-migration-data-loader. The install tabs above show the steps for each supported agent. - Which AI agents does bigquery-migration-data-loader work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is bigquery-migration-data-loader safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is bigquery-migration-data-loader still maintained?
- The repository was last updated about 2 months ago, so bigquery-migration-data-loader is actively maintained.
Skill content
View source on GitHubname: bigquery-migration-data-loader metadata: version: "1.0.0" category: BigDataAndAnalytics publisher: google description: >- Provisions BigQuery datasets, ingests staged Parquet files directly into native BigQuery tables by using bq load, and deploys already-translated views from local or Cloud Storage BigQuery Migration Service (BQMS) translation outputs. Use when provisioning target BigQuery datasets for warehouse migrations, batch loading staged Parquet files into native BigQuery tables by using bq load, auditing loaded table row counts, or deploying already-translated BigQuery views into a target dataset. Don't use for extracting source database catalog metadata, translating SQL queries, views, stored procedures, or DDL schemas to GoogleSQL (use bigquery-migration-sql-translator), exporting or unloading source database tables (such as Snowflake, Databricks, Teradata, or Oracle) to Parquet on Cloud Storage, or running post-migration cross-database data validation.
BigQuery Migration Data Loader
This skill coordinates the automated provisioning of BigQuery datasets, direct
ingestion of staged Parquet files into native BigQuery tables by using the
bq load command, and automated view deployment directly from BigQuery
Migration Service (BQMS) translation output directories (local or Cloud
Storage).
Usage attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
Before you begin
-
Google Cloud CLI, project, and location setup: Install the Google Cloud CLI (including the
gcloudandbqcommand-line tools), confirm that your Google Cloud project has an active Cloud Billing account attached, and configure the target project and default Compute Engine region:gcloud config set project PROJECT_ID gcloud config set compute/region REGIONgcloud config set compute/regionsets the default Compute Engine region; thebqCLI does not readcompute/regionand defaults toUSunless--locationis passed onbqcommands or set in~/.bigqueryrc(or configured through a project or organizationdefault_location). BigQuery is retiring the multi-region designation—includingUSandEUin the single-regions list colocated withus-central1andeurope-west4, with no change to existing data residency (see BigQuery locations)—so always pass an explicit--locationonbqcommands that matches your staged Cloud Storage bucket, as described in step 5 (check regional service availability at Google Cloud locations). -
Enable the APIs: Ask the user before enabling APIs, then enable the BigQuery and Cloud Storage APIs (
bigquery.googleapis.comandstorage.googleapis.com):gcloud services enable bigquery.googleapis.com storage.googleapis.com -
Authentication and required IAM roles: Authenticate with
gcloud auth loginandgcloud auth application-default login(or setGOOGLE_APPLICATION_CREDENTIALSfor service account authentication). Verify that your identity or service account has the required IAM roles on each resource:- Grant BigQuery Job User (
roles/bigquery.jobUser) on the target Google Cloud project. - Grant BigQuery Data Editor (
roles/bigquery.dataEditor) on the target Google Cloud project (to create new datasets withbq mk) or on an existing target dataset (to create or overwrite tables and views). - Grant Storage Object Viewer (
roles/storage.objectViewer) on the Cloud Storage bucket holding staged Parquet files and translation metadata.
- Grant BigQuery Job User (
-
Python environment: Ask the user for confirmation before installing Python packages, then install PyYAML in a virtual environment:
pip install pyyaml==6.0.2 -
Location, cost, and cleanup: Use the location of your staged Cloud Storage bucket for the dataset and every
bqcommand: the bucket's region for a regional bucket, orUSorEUfor aUSorEUmulti-region bucket. Default tous-central1only when you also choose the bucket location. Loading from a bucket in a different location incurs data transfer charges (see BigQuery locations). Review Google Cloud pricing before loading large datasets or running audit queries. Ask the user for confirmation before deleting temporary staged Parquet files (gcloud storage rm -r gs://BUCKET/EXPORT_PREFIX) or test datasets (bq rm -r -f PROJECT_ID:DATASET_ID) when no longer needed.
1. Loader workflow overview
The following diagram illustrates the four-stage ingestion and view deployment pipeline:
flowchart TD
subgraph 1. Inputs
A1["Translation Output Directory / Cloud Storage URI<br>gs://bucket/translation_output/metadata/"]
A2["Staged Parquet Files<br>gs://bucket/data_unload/TABLE/*.parquet"]
end
subgraph 2. Ingestion Pipeline
B1["1. Ensure Dataset Exists<br>bq mk --dataset --location=LOCATION DATASET"]
B2["2. Concurrent Parquet Ingestion<br>bq load --source_format=PARQUET --replace=true<br>Auto-creates native tables"]
B3["3. Row Count Audit<br>Print BigQuery Row Counts for Comparison"]
B4["4. Deploy Consumption Views<br>Deploys views directly from consolidated views.sql or relations/"]
end
A1 & A2 --> B1 --> B2 --> B3 --> B4
2. Direct translation output ingestion
[!TIP] Migration workflow context: Before running the loader, translate source schemas and views into GoogleSQL by using the
bigquery-migration-sql-translatorskill and stage exported source table Parquet files in Cloud Storage. After loading tables and deploying views, compare row counts and data aggregates against the source database.
The data loader directly consumes BQMS translation output directories (the
--translation-metadata-dir flag):
- Zero-DDL native ingestion: Table schemas are inferred and created
automatically from staged Parquet files during
bq load. Inferred types follow the Parquet types, not the translated DDL: naive source timestamps (no time zone) load asTIMESTAMPinstead ofDATETIME, andSTRING(n)andNUMERIC(p, s)parameters are dropped (see Step 2 in Section 3). - Automated object discovery: Base tables and views are auto-discovered
directly from the translation output metadata. Pass the BQMS
metadata/directory (or its parent), which containsconfig/DATABASE/SCHEMA/*.yamlfiles whosesourceNamenames each object (table configs also setexternalTableNameorexternalTableDdlPath; the others are views) and view DDL inrelations/db_DATABASE/schema_SCHEMA/VIEW.sql(or a singleviews.sql). - Automated view deployment: Translated views are deployed directly into
the target dataset with source references rewritten to the target BigQuery
project and dataset. Plain
CREATE VIEWstatements are deployed asCREATE OR REPLACE VIEWso that reruns don't fail on existing views. - Flexible Parquet path resolution: Under the
--parquet-base-uri(BASE_URI) flag, the loader automatically checks schema-scoped subfolders (BASE_URI/SCHEMA/TABLE/*.parquet) first if present and falls back to flat table paths (BASE_URI/TABLE/*.parquet). For example,gcloud storage cp -r LOCAL_DIR/SCHEMA gs://BUCKET/data_unload/stages a local schema folder of table subfolders underdata_unload/SCHEMA/. - Single target dataset: All discovered schemas load into one
--datasetby table or view name. If tables or views from different source databases or schemas share a name, the loader stops before creating the dataset and lists the conflicting objects; load each schema into its own dataset instead.
3. CLI execution interface (bq load)
Step 1: Provision the target dataset
Set the shell variables for your environment, and then create the target dataset if it doesn't already exist:
PROJECT_ID="PROJECT_ID"
DATASET_ID="DATASET_ID"
LOCATION="LOCATION"
TABLE_NAME="TABLE_NAME"
BUCKET="BUCKET"
PREFIX="data_unload"
bq show --dataset --project_id=${PROJECT_ID} ${DATASET_ID} >/dev/null 2>&1 || \
bq mk --dataset --project_id=${PROJECT_ID} --location=${LOCATION} ${DATASET_ID}
Replace the following:
PROJECT_ID: the target Google Cloud project ID.DATASET_ID: the target BigQuery dataset ID.LOCATION: the BigQuery location for the dataset and load jobs (for example,us-central1).TABLE_NAME: the name of the table to load.BUCKET: the name of the Cloud Storage bucket that holds the staged Parquet files.
Step 2: Ingest Parquet tables (auto-creates native table schema)
Pass --decimal_target_types=NUMERIC --decimal_target_types=BIGNUMERIC --decimal_target_types=STRING so BigQuery selects the smallest compatible type
(NUMERIC → BIGNUMERIC → STRING) based on each Parquet DECIMAL(p, s)
column's precision and scale: columns with p - s <= 29, s <= 9 map to
NUMERIC, columns with p - s <= 38, s <= 38 (such as DECIMAL(38, 10)) widen
to BIGNUMERIC, and larger DECIMAL256 precisions or scales fall back to
STRING:
bq load \
--project_id=${PROJECT_ID} \
--location=${LOCATION} \
--source_format=PARQUET \
--replace=true \
--decimal_target_types=NUMERIC \
--decimal_target_types=BIGNUMERIC \
--decimal_target_types=STRING \
${DATASET_ID}.${TABLE_NAME} \
"gs://${BUCKET}/${PREFIX}/${TABLE_NAME}/*.parquet"
The --replace=true flag overwrites the destination table, including its
schema, with the schema inferred from the Parquet files. If you pre-created
tables from translated BQMS DDL statements to preserve target types such as
JSON, INTERVAL, or RANGE, either run TRUNCATE TABLE and then load with
--replace=false (so that reruns stay idempotent and don't duplicate rows), or
keep --replace=true and pass an explicit --schema. If you use the loader
template with native JSON columns, also set
_WRAP_JSON_QUERY_WITH_PARSE_JSON = False in the copied script: by default it
wraps JSON_QUERY() calls in SAFE.PARSE_JSON() for JSON source columns that
Parquet loads as STRING, which fails on native JSON columns.
Parquet TIMESTAMP columns always load as BigQuery TIMESTAMP, including
naive timestamps (isAdjustedToUTC=false, such as source TIMESTAMP WITHOUT TIME ZONE columns) that the translated DDL maps to DATETIME; the wall-clock
values are kept and read as UTC. To restore DATETIME after loading, rewrite
those columns with DATETIME(column), which converts at UTC by default:
CREATE OR REPLACE TABLE `PROJECT_ID.DATASET_ID.TABLE_NAME` AS
SELECT * REPLACE (DATETIME(created_at) AS created_at)
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`;
Step 3: Run a row-count audit
Count the rows in the loaded table so that you can compare the result with the source table:
bq query \
--project_id=${PROJECT_ID} \
--location=${LOCATION} \
--use_legacy_sql=false \
"SELECT '${TABLE_NAME}' AS table_name, count(*) AS row_count FROM \`${PROJECT_ID}\`.\`${DATASET_ID}\`.\`${TABLE_NAME}\`;"
Truncated for display — read the full file on GitHub.
Related Skills
x402
121.0kSet up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key.
algorithmic-art
177.9kCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems.
pptx
177.9kUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an em…
design
149.0kComprehensive design skill: brand identity, design tokens, UI styling, logo generation (55 styles, Gemini, Atlas Cloud, or MuAPI AI), corporate identity program (50 deliverables, CIP mockups), HTML presentations (Chart.js), banner design (22 styles, social/ads/web/print), icon design (15 styles, SVG…
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
