---

name: gke-ai-troubleshooting-tpu-partition-overlap description: >- Diagnose and resolve GKE TPU slice creation failures (SliceCreationFailed) caused by partition overlap conflicts where requested TPU partitions or their nodes are already used by another slice or JobSet. Use when diagnosing GKE TPU slice creation failures with SliceCreationFailed partition or node conflicts. Don't use for general slice lifecycle states or slice provisioning failures without node overlap (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring), quota exhaustion, or unresponsive TPU nodes. metadata: version: "1.0.1" category: Containers

Troubleshoot TPU slice creation failures caused by partition overlap

A dynamic slice fails to form with the status reason SliceCreationFailed when, among other documented causes, any of its nodes are already used by another slice. This skill confirms the failure, rules out the other documented causes, finds the slice that uses the same nodes, and routes the fix through GKE.

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-partition-overlap) to User-Agent headers.

Prerequisites

• Tools:
  • For installing and initializing gcloud (which covers authentication and project selection), follow Before you begin.
  • For installing kubectl and required plugins, follow Install kubectl and Install required plugins.
• Cloud Billing & Project Configuration: Ensure the project has an active Cloud Billing account. Follow Before you begin to initialize gcloud and enable the Google Kubernetes Engine API (container.googleapis.com), ensure the Cloud Monitoring API (monitoring.googleapis.com) is enabled, and follow Store cluster information for kubectl to configure kubectl cluster credentials.
• Required IAM Roles:
  • Kubernetes Engine Viewer (roles/container.viewer)
  • Monitoring Viewer (roles/monitoring.viewer)
• Documentation:
  • Use dynamic slicing with a custom scheduler (Sections: "Before you begin", "Target your node pool to belong to a block or a sub-block", "Verify the status of the nodes and the partitions", "Create a Slice custom resource", "Monitor the status of the Slice custom resource", "Delete the slice")
  • Configure cluster access for kubectl (Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")
  • Slice (Sections: "Slice status", "Slice specification")
  • Schedule dynamic slices with Kueue and TAS (Sections: "Target your node pool to belong to a block or a sub-block", "Create a dynamic slice with Kueue and TAS", "Define the partition health selection", "Optimize workload scheduling with slice reuse", "Enable Slice reuse", "Enforce slice deletion per workload", "Monitor the status of the slice", "Monitor the metrics of the slice", "Clean up")
  • GKE system metrics (Sections: "Kubernetes metrics")
  • Monitored resource types (Sections: "k8s_node_pool")
  • About GKE dynamic slicing (Sections: "Dynamic slicing configurations")
  • Get support (Sections: "Before you contact support", "Recommendations for efficient support")

Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run any other command that changes the cluster. Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

Diagnostic workflow

Step 0: Collect context [Low Risk]

Collect the target parameters:

• {project_id}: Google Cloud project ID
• {cluster_name}: GKE cluster name
• {location}: Cluster region or zone
• {slice_name}: Name of the Slice custom resource that failed to form
• {node_pool_name}: Name of the TPU node pool (or node pools) backing the partitions in {slice_name}, if known. If the user doesn't know {node_pool_name}, after Step 1 run the Cloud Monitoring query in Step 3 for the partition IDs in {partition_ids} and read the node_pool_name label on the k8s_node_pool monitored resource.
• {issue_time}: Incident timestamp in RFC3339 UTC, if known. If the user doesn't know {issue_time}, use the Last Transition Time (lastTransitionTime, "A timestamp of the most recent status change" in Slice status) of the Ready condition from Step 1. By default, query Cloud Monitoring over a 60-minute window around {issue_time}.

---

Step 1: Confirm the SliceCreationFailed status [Low Risk]

To describe {slice_name}, follow the instructions in the section Monitor the status of the Slice custom resource.

• In Status > Conditions, check that the Reason of the Ready condition is SliceCreationFailed, and record its Last Transition Time (to use as {issue_time} if needed).
• Record the IDs in Spec > Partition Ids as {partition_ids}, the Topology, and any Labels and Annotations on {slice_name}.
• If the Ready reason isn't SliceCreationFailed, this skill doesn't apply. See Slice status for what each reason means, and hand off to gke-ai-troubleshooting-tpu-dynamic-slices-monitoring.

---

Step 2: Rule out the other documented causes [Low Risk]

Per Create a Slice custom resource, slice creation also fails with SliceCreationFailed when:

• The selected nodes don't exist.
• The topology doesn't match the number of partitions. A dynamic sub-slice lists exactly one partition ID. A dynamic super-slice lists (A × B × C) / 64 partition IDs.

If either of those causes applies, hand off to gke-ai-troubleshooting-tpu-dynamic-slices-monitoring.

Otherwise, follow Verify the status of the nodes and the partitions on {node_pool_name}:

• Verify that the nodes exist and carry each ID in {partition_ids}.
• On the nodes that carry {partition_ids}, record every partition ID of every shape (4x4x4, 2x4x4, 2x2x4, 2x2x2, and 2x2x1) from the cloud.google.com/gke-tpu-partition-.*-id labels. Per Dynamic slicing configurations, dynamic sub-slicing partitions a 4x4x4 sub-block into smaller topologies on the same physical node pool, so partitions of different shapes share the same nodes.
• Read each partition's cloud.google.com/gke-tpu-partition-.*-state label. Per Verify the status of the nodes and the partitions and Define the partition health selection, count HEALTHY partitions and DEGRADED 4x4x4 partitions ("still usable for dynamic slice formation"; "Smaller topologies don't have a degraded state") as available. Partitions in UNHEALTHY ("can't be used to form a slice"), UNSET, or INCOMPLETE aren't available. If a requested partition is in UNHEALTHY, UNSET, or INCOMPLETE, suggest an available partition whose nodes no other slice uses (Step 4, item 2), hand off to gke-ai-troubleshooting-tpu-dynamic-slices-monitoring, or escalate to Cloud Customer Care (Step 4, item 4).

---

Step 3: Find the slice that uses the same nodes [Low Risk]

Per Create a Slice custom resource, slice creation fails with SliceCreationFailed when any nodes on the custom resource are used by another slice. Because partitions of different shapes on the same sub-block share nodes (see Dynamic slicing configurations), another slice conflicts with {slice_name} if it lists any of the partition IDs (of any shape) recorded on {slice_name}'s nodes in Step 2.

1. List the Slice custom resources in the cluster (as shown in Delete the slice) and describe each one following Monitor the status of the Slice custom resource to read Spec > Partition Ids, Metadata > Labels, and Status > Conditions (Ready reason):
   • Treat any slice other than {slice_name} whose Spec > Partition Ids includes any partition ID recorded on {slice_name}'s nodes in Step 2 (whether the exact ID in {partition_ids} or another shape's partition ID on the same nodes) as conflicting. Record its name, its Ready reason (such as ACTIVE), and whether Kueue manages it (Step 4, item 1).
2. To check partition-to-slice associations in Cloud Monitoring (including around {issue_time}), query the kubernetes.io/accelerator/partition/state metric listed in Monitor the metrics of the slice. Per Kubernetes metrics, this metric "is 1 if the partition state is detected, 0 otherwise", and its labels include partition_id, slice_id ("ID of the associated slice"), and state ("Health of the partition (e.g., HEALTHY, UNHEALTHY)"). Run the query for each {partition_id} recorded on {slice_name}'s nodes in Step 2:

kubernetes_io:accelerator_partition_state{
  monitored_resource="k8s_node_pool",
  cluster_name="{cluster_name}",
  partition_id="{partition_id}"
}

• Read only time series whose value is 1 (where state="HEALTHY" means the partition is healthy and functional, per Verify the status of the nodes and the partitions).