GKE Dynamic Slicing: TPU Requirements and Scheduling Checks

GKE dynamic slicing lets a workload request a TPU topology without creating a separate fixed-topology node pool for every job. For engineers trying to use it, the first checks are specific: Ironwood TPU7x hardware, a supported Standard cluster in the Rapid channel, and All Capacity mode reservations.

Google highlighted the capability in its November 9 infrastructure announcements. This guide focuses on the configuration and failure states that matter when a job cannot form the slice it requested, rather than treating every part of dynamic slicing as newly available that day.

Super-slicing and sub-slicing solve different problems

The GKE concepts guide describes pre-provisioned TPU7x sub-blocks with a 4x4x4 topology and 16 TPU VM nodes. The slice controller forms the workload's logical topology from that capacity at scheduling time.

Super-slicing joins sub-blocks into a larger slice. Two 4x4x4 blocks can form a 4x4x8 topology for a job that needs more connected capacity. Sub-slicing divides a block into smaller isolated topologies so different jobs can use the same physical node pool.

That gives a platform team more flexibility when large training runs and smaller inference or experimentation jobs share a fleet. It does not remove the need to acquire the underlying TPU capacity, and it does not mean any arbitrary topology can be requested.

Check these prerequisites before changing the workload

The current documentation lists these requirements:

Requirement	What to verify
Cluster mode and channel	GKE Standard, Rapid channel
Accelerator	Ironwood, or TPU7x
Node operating system	Container-Optimized OS
Capacity	All Capacity mode reservations
Super-slicing version	GKE 1.35.2-gke.1842000 or later
Sub-slicing version	GKE 1.36.0-gke.3712000 or later

A newer GKE version alone does not satisfy the reservation or hardware requirements. Check the actual cluster and node pool, not just a Terraform variable that was intended to configure them.

There is a historical maintenance caveat in the current docs: nodes with a pending maintenance event ending between September 18 and September 30, 2026 need that host maintenance performed before sub-slicing. If you operate older reserved capacity, inspect those events and plan the maintenance rather than assuming a software upgrade alone completes the prerequisites.

Choose one scheduling route deliberately

Google supports direct management of Slice custom resources for custom schedulers, or integration with Kueue and Topology Aware Scheduling. The Kueue and TAS guide is the appropriate starting point when you want the scheduler to create slices from queued workloads.

A custom scheduler takes on more coordination work. The direct Slice guide separates enabling the controller, provisioning capacity, creating the logical slice, monitoring its health and deleting it when finished. Treat those as distinct states rather than declaring the infrastructure ready when the node pool exists.

For super-slicing, topology dimensions must be multiples of four and sorted in non-decreasing order. The documentation gives 4x4x8 as valid and 4x8x4 as invalid. For sub-slicing, the supported shapes are 2x2x1, 2x2x2, 2x2x4 and 2x4x4. Validate the requested shape before investigating model code.

Diagnose the partition before the model

The node label cloud.google.com/gke-tpu-partition-[shape]-state reports whether a partition is usable for the corresponding shape. The documented read-only inspection is:

kubectl describe node NODE_NAME | grep -E "cloud.google.com/gke-tpu-partition-.*-state"

Replace NODE_NAME with an actual node. HEALTHY means the partition is usable. UNHEALTHY means it cannot form a slice. UNSET points to unsuccessful controller initialization, while INCOMPLETE means some nodes have not been provisioned.

DEGRADED is a separate condition supported for the top-level 4x4x4 partition. It should not be interpreted as a healthy smaller sub-slice. A slice being usable and a slice delivering normal performance are different operational claims.

Then inspect the Slice resource itself. SliceCreationFailed indicates unmet prerequisites or failed preflight checks; ACTIVATING means formation is still in progress; ACTIVE means the slice is ready. ACTIVE_DEGRADED permits work but may affect performance. FAILED means it is no longer ready to execute workloads.

This order saves time. If the partition is incomplete or the slice has not become active, changing the model's batch size is unlikely to address the underlying scheduling failure.

Recovery still needs an application plan

The documentation describes replacing faulty partitions instead of recreating an entire fixed node pool. That can reduce infrastructure recovery work, but the training or serving application still needs a tested response to interruption.

For manual maintenance, the custom-scheduler guide requires deleting active super-slices first. Sub-slicing follows a different recovery sequence: the slice can enter FAILED and the scheduler attempts reformation on healthy nodes. Build the runbook around the configuration you use instead of assuming one maintenance procedure fits both.

Google's November 9 blog reports startup improvements of up to five times and recovery improvements of up to 4.5 times compared with static node pools. These are infrastructure comparisons, not a claim that model training steps or token generation become five times faster.

There is also a date distinction: GKE release notes already list TPU sub-slicing GA on August 3. The November announcement is useful context for evaluating the capability, but it is not evidence that every feature first became available then.

For a mixed TPU fleet, dynamic slicing is most interesting when fixed topology is stranding capacity or forcing repeated provisioning. Measure queue time, slice-formation time and end-to-end recovery separately. That is how you find out whether the operational flexibility is improving useful work rather than merely moving the waiting time elsewhere.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help infrastructure teams explain technical choices and operational trade-offs in clear developer content.

Book a 30-minute call.