PARAS MADAN
All writing

Latest in AI

SageMaker Capacity Reservations: Check Which Accounts Use Your GPUs

Use the new SageMaker Capacity Reservation APIs to separate reserved inventory, account-level allocations, and actual GPU utilization.

A SageMaker HyperPod cluster can occupy four reserved instances even when it isn't running a job. If a team reads “four in use” as “four instances busy,” it can miss idle GPUs and overestimate how much capacity it needs next month.

The November 21 AWS SDK changelog adds SageMaker APIs for viewing capacity reservations and usage. They help answer a specific question: of the capacity we reserved, which accounts are occupying it, and how much remains unallocated? I would start with that inventory before trying to use the numbers for scheduling or cost decisions.

Start with the reservation ARN

The SageMaker Capacity Reservations guide covers capacity reserved for enterprise customers through an On-Demand Capacity Reservation. From the AWS account and Region that own the reservation, use ListCapacityReservations to find its ARN. Look for the sm-capacity-reservation/ resource path and an ID starting with smcr-. The similarly named DescribeReservedCapacity expects a different ARN containing reserved-capacity/, so check the ARN before selecting the command.

Then use DescribeCapacityReservation for instance type, instance count, Availability Zone, dates, and reservation status. A reservation can be scheduled and visible before its start time. Use its status and start and end times, rather than the presence of an ARN, to decide whether it belongs in today's inventory.

Separate eligibility from allocation

DescribeResourceAssociations shows the associated SageMaker resources in the caller's account and the status of each association. An active reservation and an active association tell you different things: the former is in its availability window; the latter lets the associated service use it. Neither proves that an endpoint or cluster is occupying every instance. Also, a list of associations in one account is not a list of every account using that reservation.

For allocation, call GetCapacityReservationUsage with the same SMCapacityReservationArn. Save TotalInstanceCount, AvailableInstanceCount, and LastRefreshTime once per snapshot. Read every page of InstanceUsageSet; each account appears at most once across pages, with counts broken down by resource type. Don't sum the top-level capacity fields across pages. They describe the reservation, not the accounts on that page.

For a reporting role, use the read-only permissions in AWS's IAM guide. Listing does not support resource-level scoping, but the three detail and usage actions can be scoped to the intended reservation ARN.

Walk through a small reconciliation

Say a reservation has eight instances. The usage snapshot shows four allocated to a HyperPod cluster in one account and two allocated to inference in another, with two available. The inventory reconciliation is six allocated plus two available, which matches the eight reserved. It does not say the six allocated instances are running at 100% GPU utilization. The four HyperPod instances can be occupied by the cluster even when no workload is running.

If your dashboard instead shows four allocated and two available, look for an unconsumed page before investigating the workloads. Check LastRefreshTime as well: a fresh API response may contain a snapshot from before the last scale event. Record that timestamp next to the numbers, and don't present a stale or partial snapshot as proof of missing capacity.

Keep two questions on the dashboard

Use one panel for reserved, allocated, and available instances, with owner Region, Availability Zone, and refresh time. Use a separate workload panel for GPU activity over the same period. If a cluster has four instances allocated and little GPU activity, you've found something to investigate with its owner. If it has no allocation in the snapshot, you're asking a different question about placement or association.

That separation makes a small capacity review useful: you can tell which team is occupying the reservation, how much room remains in the snapshot, and where you still need workload evidence before changing capacity.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI infrastructure teams explain what their products actually measure through clear technical writing and practical storytelling.

Book a 30-minute call.