SkillAgentSearch skills...

bigquery-migration-metadata-extractor

Extracts database catalog metadata and query logs from data warehouses by running the DWH migration extractor locally or on a temporary Google Cloud VM through the BigQuery Migration Cloud Extractor (credentials held in Secret Manager), and inspects extracted `.zip` archives to report object and vol…

Install / Use

npx skills add google/skills --skill bigquery-migration-metadata-extractor

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

98/100

Supported Platforms

Universal

Our assessment of bigquery-migration-metadata-extractor

bigquery-migration-metadata-extractor scores 98/100 on our quality scale, 28th of 604 Data & Analytics skills we index (top 5%).

Its SKILL.md is 30 KB long, well organised into 11 sections with 7 code examples: a thorough specification that gives an agent plenty to work with.

With 20,340 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
20/20
Description
15/15
Adoption
18/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so bigquery-migration-metadata-extractor is actively maintained.
  • It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-11-15. Automated pattern scan on 2026-11-15. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

bigquery-migration-metadata-extractor compared with similar skills

All 4 of these similar skills score higher than bigquery-migration-metadata-extractor; compare them before choosing.

SkillScoreStarsUpdatedFormat
bigquery-migration-metadata-extractor (this skill)by google9820.3k54d agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
x402by browser-use100121.0k3d agoSKILL.md
algorithmic-artby anthropics100177.9k55d agoSKILL.md
pptxby anthropics100177.9k55d agoSKILL.md

Frequently asked questions

How do I install bigquery-migration-metadata-extractor?
Run npx skills add google/skills --skill bigquery-migration-metadata-extractor. The install tabs above show the steps for each supported agent.
Which AI agents does bigquery-migration-metadata-extractor work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is bigquery-migration-metadata-extractor safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is bigquery-migration-metadata-extractor still maintained?
The repository was last updated about 2 months ago, so bigquery-migration-metadata-extractor is actively maintained.

name: bigquery-migration-metadata-extractor metadata: version: "1.0.0" category: BigDataAndAnalytics publisher: google description: >- Extracts database catalog metadata and query logs from data warehouses by running the DWH migration extractor locally or on a temporary Google Cloud VM through the BigQuery Migration Cloud Extractor (credentials held in Secret Manager), and inspects extracted .zip archives to report object and volume statistics. Use when extracting database metadata or query logs from Snowflake, Redshift, Teradata, Oracle, PostgreSQL, SQL Server, Hive, etc., deciding between local and cloud (Cloud Extractor) extraction, or inspecting extracted dwh-migration-*.zip archives to print post-extraction statistics. Don't use for SQL dialect translation (use bigquery-migration-sql-translator), exporting table data to Parquet or CSV files (such as Snowflake COPY INTO unloads) or loading it into BigQuery (use the BigQuery Data Transfer Service or bigquery-basics), or validating migrated data.

BigQuery Migration Metadata Extraction

Extraction runs the DWH migration extractor (dwh-migration-dumper) through two gcloud commands to collect catalog DDL and query history—not table row data. If the user wants to migrate table rows (such as Snowflake COPY INTO unloads), ask whether they prefer the [BigQuery Data Transfer Service][bq_dts] (including the [Snowflake connector][snowflake_dts]) or exporting to Cloud Storage and loading with bq load. Both extract commands are in the alpha release track:

Command | Use for :--------------------------------------------- | :------ gcloud alpha bq migration metadata extract | Catalog metadata (metadata connectors) gcloud alpha bq migration query-logs extract | Query history (*-logs connectors)

Both take the same arguments. gcloud flags come first; the extractor's own flags come after -- and are passed to the extractor unchanged:

gcloud alpha bq migration {metadata|query-logs} extract \
    --connector=CONNECTOR --output=gs://BUCKET/PATH/ \
    [--run-mode=local|cloud] [cloud flags] \
    -- [EXTRACTOR FLAGS] [PASSWORD]
  • --connector and --output are required gcloud flags. --output is a Cloud Storage path (gs://...); local mode also accepts a local file or directory path. This skill uses Cloud Storage in both run modes.
  • --run-mode=local (the default) runs the extractor on this machine. --run-mode=cloud runs it on a temporary Compute Engine VM (the BigQuery Migration Cloud Extractor).
  • Database flags (--host, --port, --user, --database, --schema, --warehouse, --role, --driver, --assessment, --query-log-*, ...) go after --. Putting them before -- is an error.
  • How the password is passed depends on the run mode (Section 4): an extractor response file (@FILE) in local mode, and --password read from a file in cloud mode. Never type a literal password.

[!IMPORTANT] The commands are in the alpha track. If gcloud reports an invalid choice or unknown command, ask the user before updating the Google Cloud CLI, then retry. Update it with gcloud components update, or, if gcloud was installed with a system package manager such as apt or yum (which disables the component manager), upgrade the google-cloud-cli package with that package manager.

Usage Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read and follow the instructions in Usage Attribution.

Before you begin

  1. Google Cloud CLI, authentication, project, region, and zone: Ensure the [Google Cloud CLI][gcloud_install] is installed, verify that the Google Cloud project has an active Cloud Billing account, authenticate with user and Application Default Credentials, and configure the target project, a default region such as us-central1 (check regional service availability at [Google Cloud locations][cloud_locations]), and a zone in that region. Cloud mode takes the VM's zone from --zone, then compute/zone, then us-central1-a; it does not read compute/region:

    gcloud auth login
    gcloud auth application-default login
    gcloud config set project {project_id}
    gcloud config set compute/region {region}
    gcloud config set compute/zone {zone}
    
  2. Output bucket: This skill writes the ZIP archives to a Cloud Storage bucket (--output) that the project owns and that is not public, in both run modes. In both modes the command checks the bucket before it starts the extractor, so the caller needs storage.buckets.list on the bucket's project and storage.buckets.getIamPolicy on the bucket unless public access prevention is enforced on it. In local mode the Application Default Credentials also need to create objects in the bucket.

  3. Run-mode prerequisites (ask the user before installing software or enabling APIs):

    • Local mode (--run-mode=local): Java 11 or later (JRE 11+) on PATH, the bqms gcloud component, vendor JDBC driver JARs (--driver) when required, and local network access to the database; see the local extraction guide. Without a terminal the extract command cannot prompt to install bqms and fails, so once the user agrees, install it first with gcloud components install bqms --quiet (for a gcloud installed with apt or yum, install the google-cloud-cli-bqms package instead).
    • Cloud mode (--run-mode=cloud):
      • APIs: Compute Engine (compute.googleapis.com), Secret Manager (secretmanager.googleapis.com), Cloud Storage (storage.googleapis.com), and Cloud Logging (logging.googleapis.com).
      • Network and subnet: A VPC --network and --subnet (default default) with routing to the database, and the VM's egress IP range allowlisted by the database. The VM also needs internet access to download the extractor and reach Google APIs: keep the default external IP, or with --no-address add Cloud NAT in the VM's region. A Cloud VPN or Cloud Interconnect route to the database does not replace it.
      • IAM permissions: The caller needs roles to create the VM and the secret, for example Compute Admin (roles/compute.admin) and Secret Manager Admin (roles/secretmanager.admin), plus the bucket permissions in item 2. With a prepared --gce-service-account, the caller also needs Service Account User (roles/iam.serviceAccountUser) on that account, and the account itself needs the roles listed in the cloud extraction guide.
      • Default service account: Without --gce-service-account, every run creates or reuses bqms-cloud-extractor@{project_id}.iam.gserviceaccount.com and grants it nine project roles, including roles/editor, roles/compute.admin, and roles/cloudkms.cryptoKeyDecrypter. This needs Service Account Admin (roles/iam.serviceAccountAdmin) and Project IAM Admin (roles/resourcemanager.projectIamAdmin). Without a terminal, gcloud accepts its own "create it?" prompt, so get the user's explicit approval before you omit --gce-service-account; see the cloud extraction guide.
  4. Cost and cleanup: Review Google Cloud Pricing for Compute Engine, Secret Manager, and Cloud Storage. After a cloud run, delete the run's Secret Manager secret (Section 5).

1. Choose the Run Mode

  | --run-mode=local (default) | --run-mode=cloud (Cloud Extractor) :----------------------- | :------------------------------------------------------- | :----------------------------------- Where the extractor runs | This machine (needs a Java runtime) | Temporary Compute Engine VM Network path to the DB | This machine's network (VPN, on-prem, bastion) | The VM's VPC; the DB must accept connections from it JDBC drivers | Any, including vendor JARs passed with --driver | Only drivers bundled with the extractor; the VM has no vendor JARs and cannot read a local --driver path Credentials | @FILE response file read by the local extractor | --password (read from a file) stored with the other extractor arguments in a Secret Manager secret that the VM reads Failed extractor tasks | Partial ZIP still lands in --output | No ZIP is uploaded; the run fails Read next | local extraction guide | cloud extraction guide

Pick local when any of these is true:

  • The connector needs a JDBC driver that the cloud VM does not have: teradata, teradata-logs, teradata14-logs, oracle, oracle-logs, oracle-stats, sqlserver, mysql, netezza, vertica, vertica-logs. Locally, the bqms launcher downloads the Oracle, SQL Server, and MySQL drivers itself (the local extractor ships without JDBC drivers, so the launcher also downloads the Snowflake, Redshift, PostgreSQL, and Greenplum drivers); Teradata, Netezza, and Vertica need --driver (Section 3).
  • The database is only reachable from the user's network (on-prem, VPN, private endpoint with no route from Google Cloud).
  • The user cannot create Compute Engine VMs, service accounts, or secrets in the project.
  • --host is localhost, 127.0.0.1, ::1, or a private address with no route from a Google Cloud VPC. Never use --run-mode=cloud for these; the VM cannot reach them (on the VM, 127.0.0.1 is the VM itself, even if the user says the address is routed). Ask for a routable address, or, if the user already accepted a fallback, run locally with the same --output and drop the cloud-only flags.

Pick cloud when the database is reachable from Google Cloud (Snowflake, Redshift, BigQuery, cloud-hosted PostgreSQL/Greenplum, Hive metastore) and the user wants nothing installed locally.

Tell the user which run mode you chose and why, for example that cloud mode cannot load vendor JDBC drivers or reach a database that only the user's network can reach.

2. Connectors and Commands

Each connector extracts either catalog metadata or query logs, so it belongs to exactly one command. Run both commands to get both.

Source | metadata extract connector | query-logs extract connector :-------------------- | :---------------------------------------------------------------------------------------------------- | :----------------------------- Snowflake | snowflake (or snowflake-account-usage-metadata, snowflake-information-schema-metadata) | snowflake-logs (or snowflake-account-usage-logs, snowflake-information-schema-logs) Amazon Redshift | redshift | redshift-raw-logs (or redshift-logs, redshift-serverless-logs) Teradata | teradata | teradata-logs (version 15+), teradata14-logs Oracle | oracle (DDL), oracle-stats (workload statistics for assessment; window set by --query-log-days) |

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars20.3k
CategoryData
Updated1mo ago
Forks1.7k

Languages

Python

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions