---

name: gke-ai-troubleshooting-tpu-reservation-stockout description: >- Diagnose and resolve cluster-wide GKE TPU node pool creation or scale-up failures caused by Compute Engine reservation stockouts (GCE_STOCKOUT, ZONE_RESOURCE_POOL_EXHAUSTED) when existing unhealthy or NotReady TPU node pools in the cluster have node auto-repair disabled and hold reserved TPU capacity. Use when TPU node pool creation or scale-up fails with GCE_STOCKOUT, ZONE_RESOURCE_POOL_EXHAUSTED, or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS despite having a specific Compute Engine TPU reservation. Don't use for general pod scheduling without a TPU reservation or workload-level container crashes. metadata: version: "1.0.1" category: Containers

Troubleshoot GKE TPU reservation stockouts caused by leaked node pool capacity

Use this skill to diagnose and resolve GKE TPU node pool creation or scale-up failures that fail with GCE_STOCKOUT, ZONE_RESOURCE_POOL_EXHAUSTED, or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS (or queue due to lack of capacity) even though they target a specific Compute Engine TPU reservation.

If node auto-repair is disabled on an existing TPU node pool, GKE doesn't repair its nodes when they become NotReady. While their VMs are still running, they keep consuming slots in the reservation (inUseCount is the number of running instances that consume the reservation). Per Ensure reservation availability, the reservation must have sufficient available TPU chips at the time of node pool creation, and per Multi-host TPU slice node pools, if GKE is unable to deploy one node in a slice, no nodes in the TPU slice node pool are deployed. When the free slots in the reservation (specificReservation.count - specificReservation.inUseCount; see Consumable instances in a reservation) are fewer than the number of nodes that the new TPU node pool needs, creation fails with GCE_STOCKOUT or ZONE_RESOURCE_POOL_EXHAUSTED (or queues due to lack of capacity).

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-reservation-stockout)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-reservation-stockout) to User-Agent headers.

Prerequisites

• Required IAM Roles:
  • Logs Viewer (roles/logging.viewer): Query Cloud Audit Logs in Cloud Logging.
  • Monitoring Viewer (roles/monitoring.viewer): Query GKE node readiness metrics in Cloud Monitoring.
  • Kubernetes Engine Viewer (roles/container.viewer): Inspect GKE node pools (management.autoRepair), cluster operations, and node conditions.
  • Compute Viewer (roles/compute.viewer): Inspect Compute Engine reservation utilization (specificReservation.inUseCount vs specificReservation.count), consuming instances, and VM labels (including in the reservation's owner project if using a shared reservation).
  • For remediation ([High Risk] steps): Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
• API Enablement & Billing:
  • Ensure the target project has an active Google Cloud billing account attached: gcloud billing projects describe {project_id}.
  • Enable required APIs: container.googleapis.com, compute.googleapis.com, logging.googleapis.com, and monitoring.googleapis.com.
• CLI & Authentication Setup:
  • Install Google Cloud SDK (gcloud) and kubectl: https://cloud.google.com/sdk/docs/install.
• Cluster mode: This skill applies to Standard clusters. Autopilot clusters always auto-repair nodes, and you can't turn this off (see Settings for Autopilot and Standard).
• Reference Documentation:
  • Auto-repair nodes (Sections: "Settings for Autopilot and Standard", "Repair criteria", "Node repair process", "Verify node auto-repair is enabled for a Standard node pool", "Node auto repair in TPU slice nodes", "Enable auto-repair for an existing Standard node pool", "Get information about recent automated repair events")
  • Troubleshoot GKE Standard node pools (Sections: "Issue: Node pool creation fails due to insufficient resources")
  • Troubleshoot TPUs in GKE (Sections: "How GKE handles capacity issues", "Insufficient TPU resources to satisfy the TPU request")
  • About TPUs in GKE (Sections: "Multi-host TPU slice node pools")
  • Consuming reserved zonal resources (Sections: "Consuming a specific shared reservation")
  • About reservations (Sections: "How reservations work")
  • View reservations or future reservation requests (Sections: "Limitations", "View reservations", "Consumable instances in a reservation", "View a list of instances that are consuming a reservation")
  • Organize resources using labels (Sections: "View labels")
  • Create and manage cluster and node pool labels (Sections: "Automatically applied labels")
  • Deploy TPU workloads in GKE Standard (Sections: "Ensure reservation availability", "Monitor health metrics for TPU nodes and node pools", "Configure auto repair for TPU slice nodes")
  • Add and manage node pools (Sections: "Delete a node pool")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

Diagnostic workflow

Step 0: Collect context and set the investigation window

Identify the following environment parameters from the user prompt or cluster context:

• {project_id}, {cluster_name}, {location}, {failing_nodepool_name}, {reservation_name}, {reservation_project_id} (the project that owns {reservation_name}, which defaults to {project_id} for a single-project reservation or is the owner project for a shared reservation; see Consuming a specific shared reservation), {zone} (the zone of the reservation), and {issue_time}.
• Time window: By default, query a 60-minute window [{issue_time} - 30m, {issue_time} + 30m] ({start_time} to {end_time}).

---

Step 1: Confirm the stockout error and reservation utilization [Low Risk]

1. Check Cloud Audit Logs and GKE operations for stockout or capacity errors:
   • Run the query in Issue: Node pool creation fails due to insufficient resources across [{start_time}, {end_time}] and look for the documented error messages, such as ZONE_RESOURCE_POOL_EXHAUSTED or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS.
   • Per Insufficient TPU resources to satisfy the TPU request, TPU errors can also contain GCE_STOCKOUT.
   • Per How GKE handles capacity issues, GKE sometimes queues a TPU node pool creation request instead of failing it immediately, surfacing a ... cannot be created ... due to lack of capacity message instead of GCE_STOCKOUT. List the node pool creation operation with gcloud container operations list (--filter="operationType=CREATE_NODE_POOL").
2. Check Compute Engine reservation utilization:
   • Per Limitations, query shared reservations in the owner project where they were created ({reservation_project_id}).
   • Follow the View reservations section to run gcloud compute reservations describe {reservation_name} --zone={zone} --project={reservation_project_id} or gcloud compute reservations list --project={reservation_project_id}, and compare specificReservation.inUseCount against specificReservation.count.

Interpret the results

• Don't suggest deleting any node pool until you have identified which existing VMs and node pools hold the reserved capacity (Step 2).
• Stockout confirmed: Per Ensure reservation availability and Multi-host TPU slice node pools, if GKE can't deploy one node in a multi-host slice, no nodes in the slice are deployed. If a stockout or lack-of-capacity message is present and the free slots (specificReservation.count - specificReservation.inUseCount; see Consumable instances in a reservation) are fewer than the number of nodes that {failing_nodepool_name} needs, proceed to Step 2.

---

Step 2: Find which VMs and node pools hold the reservation [Low Risk]

1. Check which VMs are consuming {reservation_name}:
   • Per How reservations work, a specifically targeted reservation can be used by VMs in the reservation's project ({reservation_project_id}) or, for a shared reservation, in any project that it's shared with.
   • Follow View a list of instances that are consuming a reservation to list the VMs that use {reservation_name}.
   • Follow View labels to view each VM's labels, and check the GKE labels in Automatically applied labels (goog-k8s-cluster-name, goog-k8s-cluster-location, and goog-k8s-node-pool-name) to find each VM's cluster and node pool.
2. Check node readiness in {cluster_name}:
   • Query the Cloud Monitoring metric kubernetes_io:node_status_condition (monitored_resource="k8s_node", cluster_name="{cluster_name}", condition="Ready") described in Monitor health metrics for TPU nodes and node pools. The status la