---

name: gke-ai-troubleshooting-tpu-vm-repair-stuck description: >- Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state) after host errors, maintenance events, or faulty host reports in single-host, multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when TPU instances in a slice node pool remain in REPAIRING, host error events appear in audit logs, or TPU slices are unschedulable due to a VM in repair. Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are Ready (use gke-workload-troubleshooting), JobSet preemption events (use gke-ai-troubleshooting-jobset-interruption), or Slice custom resource lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring). metadata: version: "1.0.1" category: Containers

Troubleshoot GKE TPU VMs stuck in Compute Engine repair (REPAIRING)

This skill checks the VM status, the host events, and the GKE repair status, and routes recovery through GKE instead of direct Compute Engine changes.

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vm-repair-stuck) to User-Agent headers.

Prerequisites

• Tools:
  • For installing and initializing gcloud (which covers authentication and project selection), follow Before you begin.
  • For installing kubectl and required plugins, follow Install kubectl and Install required plugins.
• Cloud Billing & Project Configuration: Ensure the project has an active Cloud Billing account. Follow Before you begin to initialize gcloud and enable the Google Kubernetes Engine API (container.googleapis.com), ensure the Compute Engine API (compute.googleapis.com) and Cloud Logging API (logging.googleapis.com) are enabled, and follow Store cluster information for kubectl to configure kubectl cluster credentials.
• Required IAM Roles:
  • Kubernetes Engine Viewer (roles/container.viewer)
  • Compute Viewer (roles/compute.viewer)
  • Logs Viewer (roles/logging.viewer)
  • For the user who enables node auto-repair in Step 5, item 1: Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
• Documentation:
  • View information about MIGs and managed instances (Sections: "List MIGs", "Check the status of managed instances")
  • Compute Engine instance lifecycle (Sections: "REPAIRING state")
  • Troubleshooting VM shutdowns and reboots (Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")
  • Configure cluster access for kubectl (Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")
  • Add and manage node pools (Sections: "View node pools in a Standard cluster", "Deploy a Pod to a specific node pool")
  • Troubleshoot NodeNotReady state (Sections: "Check the node's status and conditions", "Confirm node preemption")
  • About TPUs in GKE (Sections: "Multi-host TPU slice node pools")
  • Use dynamic slicing with a custom scheduler (Sections: "Requirements", "Verify the status of the nodes and the partitions", "Monitor the status of the Slice custom resource", "Hardware maintenance and failures")
  • About GKE dynamic slicing (Sections: "Key characteristics of dynamic slicing configurations")
  • Deploy TPUs in All Capacity mode in GKE (Sections: "Manage node failures", "Node failure and recovery")
  • Report and repair faulty TPU hosts in All Capacity mode (Sections: "Understand the faulty host report process", "Review report faulty host operations")
  • Auto-repair nodes (Sections: "Settings for Autopilot and Standard", "Repair criteria", "Node repair process", "Verify node auto-repair is enabled for a Standard node pool", "Enable auto-repair for an existing Standard node pool", "Get information about recent automated repair events", "Node auto repair in TPU slice nodes")
  • Deploy TPU workloads in GKE Standard (Sections: "Before you begin", "Configure auto repair for TPU slice nodes")
  • Get support (Sections: "Before you contact support", "Recommendations for efficient support")
  • REST Resource: projects.locations.clusters.nodePools (Sections: "Resource: NodePool")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

Diagnostic workflow

Step 0: Collect context [Low Risk]

Collect the target parameters:

• {project_id}: Google Cloud project ID
• {cluster_name}: GKE cluster name
• {location}: Cluster region or zone
• {cluster_mode}: Cluster mode (Autopilot or Standard)
• {tpu_version}: TPU version (such as Ironwood tpu7x, TPU v6e, v5p, v5e, or v4) and whether the node pool is a single-host or multi-host TPU slice node pool
• {nodepool_name}: TPU node pool name
• {group_name} and {zone}: Name and zone of the managed instance group (MIG) that backs the node pool:
  • In a Standard cluster, follow View node pools in a Standard cluster to describe {nodepool_name} and read its instanceGroupUrls field (see Resource: NodePool).
  • In an Autopilot cluster (or if {nodepool_name} isn't known), follow List MIGs to list the managed instance groups in {project_id} and find {group_name} and {zone}.
• {instance_name}: Affected TPU VM name, if known. If the user doesn't know {instance_name}, Step 1 finds the instances in REPAIRING.
• {issue_time}: Incident timestamp in RFC3339 UTC, if known

---

Step 1: Check the VM status in the managed instance group [Low Risk]

To list the instances in {group_name} with their status (STATUS / instanceStatus) and current action (ACTION / currentAction), follow the instructions in Check the status of managed instances.

• Analysis:
  • STATUS (instanceStatus) is REPAIRING: Compute Engine is repairing the VM because it encountered an internal error or its host server is unavailable due to maintenance. The VM can't be used during the repair. See REPAIRING state. Record {instance_name} and proceed to Step 2.
  • No instance is in REPAIRING (every instance reports RUNNING with ACTION NONE, or an instance reports another status or action such as ACTION RECREATING, which means "The instance is being replaced" per Check the status of managed instances): Check each node's status and conditions following Check the node's status and conditions and hand off to the matching sibling skill:
    • If a node's Ready condition is Unknown (NodeStatusUnknown), hand off to gke-ai-troubleshooting-node-unresponsive-timeout.
    • If a node isn't Ready for another reason, hand off to gke-node-notready.
    • If every node is Ready (per Repair criteria, "A node reporting a Ready status is considered healthy") but the workload's Pods are still Pending, hand off to gke-workload-troubleshooting.

---

Step 2: Check Compute Engine system events [Low Risk]

To list the system events for {instance_name} around {issue_time}, follow the instructions in the section Querying Cloud Audit Logs. Compare the method field with the table in the Reviewing Cloud Audit Logs section of the same document:

• compute.instances.hostError, compute.instances.terminateOnHostMaintenance, or compute.instances.automaticRestart: Record the matching system event log entries and timestamps, and continue to Steps 3 and 4.
• compute.instances.preempted: Compute Engine preempted a Spot VM or preemptible VM. If the affected workload is a JobSet, hand off to gke-ai-troubleshooting-jobset-interruption. Otherwise, follow Confirm node preemption.
• No method from the table, or another method: Per the note in Reviewing Cloud Audit Logs, if a VM rebooted and none of the listed methods appears in the query results, the reboot likely happened because the VM's operating system initiated it. If the VM is still in REPAIRING in Step 1 (or a method such as compute.instances.reportHostAsFaulty appears), continue to Steps 3 and 4.

---

Step 3: Check the node status and slice impact [Low Risk]

Follow [Check the node's status and conditions](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notrea