bigquery-migration-metadata-extractor
Extracts database catalog metadata and query logs from data warehouses by running the DWH migration extractor locally or on a temporary Google Cloud VM through the BigQuery Migration Cloud Extractor (credentials held in Secret Manager), and inspects extracted `.zip` archives to report object and vol…
Install / Use
npx skills add google/skills --skill bigquery-migration-metadata-extractorInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
Data & AnalyticsSupported Platforms
Our assessment of bigquery-migration-metadata-extractor
bigquery-migration-metadata-extractor scores 98/100 on our quality scale, 28th of 604 Data & Analytics skills we index (top 5%).
Its SKILL.md is 30 KB long, well organised into 11 sections with 7 code examples: a thorough specification that gives an agent plenty to work with.
With 20,340 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so bigquery-migration-metadata-extractor is actively maintained.
- It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-11-15. Automated pattern scan on 2026-11-15. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
bigquery-migration-metadata-extractor compared with similar skills
All 4 of these similar skills score higher than bigquery-migration-metadata-extractor; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| bigquery-migration-metadata-extractor (this skill)by google | 98 | 20.3k | 54d ago | SKILL.md |
| claude-memby thedotmack | 100 | 117.6k | today | CLAUDE.md |
| x402by browser-use | 100 | 121.0k | 3d ago | SKILL.md |
| algorithmic-artby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
| pptxby anthropics | 100 | 177.9k | 55d ago | SKILL.md |
Frequently asked questions
- How do I install bigquery-migration-metadata-extractor?
- Run
npx skills add google/skills --skill bigquery-migration-metadata-extractor. The install tabs above show the steps for each supported agent. - Which AI agents does bigquery-migration-metadata-extractor work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is bigquery-migration-metadata-extractor safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is bigquery-migration-metadata-extractor still maintained?
- The repository was last updated about 2 months ago, so bigquery-migration-metadata-extractor is actively maintained.
Skill content
View source on GitHubname: bigquery-migration-metadata-extractor
metadata:
version: "1.0.0"
category: BigDataAndAnalytics
publisher: google
description: >-
Extracts database catalog metadata and query logs from data warehouses by
running the DWH migration extractor locally or on a temporary Google Cloud VM
through the BigQuery Migration Cloud Extractor (credentials held in Secret
Manager), and inspects extracted .zip archives to report object and volume
statistics. Use when extracting database metadata or query logs from
Snowflake, Redshift, Teradata, Oracle, PostgreSQL, SQL Server, Hive, etc.,
deciding between local and cloud (Cloud Extractor) extraction, or inspecting
extracted dwh-migration-*.zip archives to print post-extraction statistics.
Don't use for SQL dialect translation (use bigquery-migration-sql-translator),
exporting table data to Parquet or CSV files (such as Snowflake COPY INTO
unloads) or loading it into BigQuery (use the BigQuery Data Transfer Service
or bigquery-basics), or validating migrated data.
BigQuery Migration Metadata Extraction
Extraction runs the DWH migration extractor (dwh-migration-dumper) through two
gcloud commands to collect catalog DDL and query history—not table row data. If
the user wants to migrate table rows (such as Snowflake COPY INTO unloads),
ask whether they prefer the [BigQuery Data Transfer Service][bq_dts] (including
the [Snowflake connector][snowflake_dts]) or exporting to Cloud Storage and
loading with bq load. Both extract commands are in the alpha release
track:
Command | Use for
:--------------------------------------------- | :------
gcloud alpha bq migration metadata extract | Catalog metadata (metadata connectors)
gcloud alpha bq migration query-logs extract | Query history (*-logs connectors)
Both take the same arguments. gcloud flags come first; the extractor's own flags
come after -- and are passed to the extractor unchanged:
gcloud alpha bq migration {metadata|query-logs} extract \
--connector=CONNECTOR --output=gs://BUCKET/PATH/ \
[--run-mode=local|cloud] [cloud flags] \
-- [EXTRACTOR FLAGS] [PASSWORD]
--connectorand--outputare required gcloud flags.--outputis a Cloud Storage path (gs://...); local mode also accepts a local file or directory path. This skill uses Cloud Storage in both run modes.--run-mode=local(the default) runs the extractor on this machine.--run-mode=cloudruns it on a temporary Compute Engine VM (the BigQuery Migration Cloud Extractor).- Database flags (
--host,--port,--user,--database,--schema,--warehouse,--role,--driver,--assessment,--query-log-*, ...) go after--. Putting them before--is an error. - How the password is passed depends on the run mode (Section 4): an extractor
response file (
@FILE) in local mode, and--passwordread from a file in cloud mode. Never type a literal password.
[!IMPORTANT] The commands are in the alpha track. If gcloud reports an invalid choice or unknown command, ask the user before updating the Google Cloud CLI, then retry. Update it with
gcloud components update, or, if gcloud was installed with a system package manager such asaptoryum(which disables the component manager), upgrade thegoogle-cloud-clipackage with that package manager.
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
Before you begin
-
Google Cloud CLI, authentication, project, region, and zone: Ensure the [Google Cloud CLI][gcloud_install] is installed, verify that the Google Cloud project has an active Cloud Billing account, authenticate with user and Application Default Credentials, and configure the target project, a default region such as
us-central1(check regional service availability at [Google Cloud locations][cloud_locations]), and a zone in that region. Cloud mode takes the VM's zone from--zone, thencompute/zone, thenus-central1-a; it does not readcompute/region:gcloud auth login gcloud auth application-default login gcloud config set project {project_id} gcloud config set compute/region {region} gcloud config set compute/zone {zone} -
Output bucket: This skill writes the ZIP archives to a Cloud Storage bucket (
--output) that the project owns and that is not public, in both run modes. In both modes the command checks the bucket before it starts the extractor, so the caller needsstorage.buckets.liston the bucket's project andstorage.buckets.getIamPolicyon the bucket unless public access prevention is enforced on it. In local mode the Application Default Credentials also need to create objects in the bucket. -
Run-mode prerequisites (ask the user before installing software or enabling APIs):
- Local mode (
--run-mode=local): Java 11 or later (JRE 11+) onPATH, thebqmsgcloud component, vendor JDBC driver JARs (--driver) when required, and local network access to the database; see the local extraction guide. Without a terminal the extract command cannot prompt to installbqmsand fails, so once the user agrees, install it first withgcloud components install bqms --quiet(for a gcloud installed withaptoryum, install thegoogle-cloud-cli-bqmspackage instead). - Cloud mode (
--run-mode=cloud):- APIs: Compute Engine (
compute.googleapis.com), Secret Manager (secretmanager.googleapis.com), Cloud Storage (storage.googleapis.com), and Cloud Logging (logging.googleapis.com). - Network and subnet: A VPC
--networkand--subnet(defaultdefault) with routing to the database, and the VM's egress IP range allowlisted by the database. The VM also needs internet access to download the extractor and reach Google APIs: keep the default external IP, or with--no-addressadd Cloud NAT in the VM's region. A Cloud VPN or Cloud Interconnect route to the database does not replace it. - IAM permissions: The caller needs roles to create the VM and
the secret, for example Compute Admin (
roles/compute.admin) and Secret Manager Admin (roles/secretmanager.admin), plus the bucket permissions in item 2. With a prepared--gce-service-account, the caller also needs Service Account User (roles/iam.serviceAccountUser) on that account, and the account itself needs the roles listed in the cloud extraction guide. - Default service account: Without
--gce-service-account, every run creates or reusesbqms-cloud-extractor@{project_id}.iam.gserviceaccount.comand grants it nine project roles, includingroles/editor,roles/compute.admin, androles/cloudkms.cryptoKeyDecrypter. This needs Service Account Admin (roles/iam.serviceAccountAdmin) and Project IAM Admin (roles/resourcemanager.projectIamAdmin). Without a terminal, gcloud accepts its own "create it?" prompt, so get the user's explicit approval before you omit--gce-service-account; see the cloud extraction guide.
- APIs: Compute Engine (
- Local mode (
-
Cost and cleanup: Review Google Cloud Pricing for Compute Engine, Secret Manager, and Cloud Storage. After a cloud run, delete the run's Secret Manager secret (Section 5).
1. Choose the Run Mode
| --run-mode=local (default) | --run-mode=cloud (Cloud Extractor)
:----------------------- | :------------------------------------------------------- | :-----------------------------------
Where the extractor runs | This machine (needs a Java runtime) | Temporary Compute Engine VM
Network path to the DB | This machine's network (VPN, on-prem, bastion) | The VM's VPC; the DB must accept connections from it
JDBC drivers | Any, including vendor JARs passed with --driver | Only drivers bundled with the extractor; the VM has no vendor JARs and cannot read a local --driver path
Credentials | @FILE response file read by the local extractor | --password (read from a file) stored with the other extractor arguments in a Secret Manager secret that the VM reads
Failed extractor tasks | Partial ZIP still lands in --output | No ZIP is uploaded; the run fails
Read next | local extraction guide | cloud extraction guide
Pick local when any of these is true:
- The connector needs a JDBC driver that the cloud VM does not have:
teradata,teradata-logs,teradata14-logs,oracle,oracle-logs,oracle-stats,sqlserver,mysql,netezza,vertica,vertica-logs. Locally, thebqmslauncher downloads the Oracle, SQL Server, and MySQL drivers itself (the local extractor ships without JDBC drivers, so the launcher also downloads the Snowflake, Redshift, PostgreSQL, and Greenplum drivers); Teradata, Netezza, and Vertica need--driver(Section 3). - The database is only reachable from the user's network (on-prem, VPN, private endpoint with no route from Google Cloud).
- The user cannot create Compute Engine VMs, service accounts, or secrets in the project.
--hostislocalhost,127.0.0.1,::1, or a private address with no route from a Google Cloud VPC. Never use--run-mode=cloudfor these; the VM cannot reach them (on the VM,127.0.0.1is the VM itself, even if the user says the address is routed). Ask for a routable address, or, if the user already accepted a fallback, run locally with the same--outputand drop the cloud-only flags.
Pick cloud when the database is reachable from Google Cloud (Snowflake, Redshift, BigQuery, cloud-hosted PostgreSQL/Greenplum, Hive metastore) and the user wants nothing installed locally.
Tell the user which run mode you chose and why, for example that cloud mode cannot load vendor JDBC drivers or reach a database that only the user's network can reach.
2. Connectors and Commands
Each connector extracts either catalog metadata or query logs, so it belongs to exactly one command. Run both commands to get both.
Source | metadata extract connector | query-logs extract connector
:-------------------- | :---------------------------------------------------------------------------------------------------- | :-----------------------------
Snowflake | snowflake (or snowflake-account-usage-metadata, snowflake-information-schema-metadata) | snowflake-logs (or snowflake-account-usage-logs, snowflake-information-schema-logs)
Amazon Redshift | redshift | redshift-raw-logs (or redshift-logs, redshift-serverless-logs)
Teradata | teradata | teradata-logs (version 15+), teradata14-logs
Oracle | oracle (DDL), oracle-stats (workload statistics for assessment; window set by --query-log-days) |
Truncated for display — read the full file on GitHub.
Related Skills
claude-mem
117.6kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
x402
121.0kSet up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key.
algorithmic-art
177.9kCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems.
pptx
177.9kUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an em…
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
