PARAS MADAN
All writing

Latest in AI

EC2 GPU Performance Checks: Diagnose a Slow AI Endpoint Before Replacing It

Separate EC2's free GPU performance checks from paid Detailed Monitoring, check metric support, and plan a controlled response to slow inference.

A model endpoint can keep answering requests while getting noticeably slower. That leaves an awkward operational question: did the workload change, is a GPU underused, or does the hardware need attention?

AWS's November 19 EC2 release adds two different ways to investigate that question. Performance status checks identify certain recoverable hardware conditions. Expanded Detailed Monitoring provides more utilization data. Treating those as one feature would miss both the troubleshooting distinction and the billing difference.

Here is how I would introduce them into an inference runbook, without turning every slow response into a reboot.

Separate the free signal from the paid investigation

The launch announcement says performance status checks have no additional charge. The new Detailed Monitoring metrics are opt-in and incur standard CloudWatch charges for EC2 Detailed Monitoring. Availability covers commercial AWS Regions and AWS GovCloud (US), but individual instance and metric support still matters.

The release includes one-minute GPU metrics and a ten-second CPU utilization metric. They complement existing in-guest tools rather than requiring monitoring software inside the instance.

My starting point would be to surface the free hardware signal in the existing incident view, then enable a small paid metric set for a representative serving pool. Decide what question each metric answers before applying it across the fleet. Do not read “no additional charge” as a promise that the entire dashboard, alarms and telemetry pipeline cost nothing.

Check hardware status before changing the model

According to the EC2 status-check documentation, performance checks detect hardware problems that reduce accelerated-instance performance without necessarily making the instance unreachable. One example is a GPU memory condition that needs a reboot to complete row remapping.

In DescribeInstanceStatus, inspect PerformanceStatus, including Status, ImpairedSince and Remediation. A failed check increments StatusCheckFailed_Performance and also the combined StatusCheckFailed metric.

That combined signal deserves attention if your runbook previously treated every status failure as a reachability problem. Record the specific failed check rather than jumping straight to replacement.

For a hypothetical slow endpoint, I would capture the impairment time beside the serving release, request rate and latency change. An overlapping timestamp is a useful lead, not proof that the hardware explains every delayed request. A passing check is not a certificate that batching, queues or model execution are efficient either.

Verify which GPU metrics actually exist

The metric support table is more useful than assuming every GPU family exposes the same dashboard. For example, GPUMemoryUtilization includes P5 variants and specified P6 families, while GPUUtilization has a narrower listed set. GPUPowerUtilization is available through basic monitoring at five-minute intervals; the other accelerator metrics require Detailed Monitoring and an appropriate metric category.

Maintenance or older GPU firmware can cause missing data. AWS documents firmware requirements for particular P5 and P6-B200 metrics. Missing telemetry therefore must not be silently interpreted as zero utilization.

Keep an inventory with instance family, expected metrics and observed reporting. Before investigating a supposedly idle GPU, establish that the chart represents a supported, reporting device. Otherwise a dashboard gap can become a completely unnecessary capacity decision.

Compare individual GPUs, not just the instance average

The AWS Compute walkthrough describes instance-level and per-GPU series. The latter use InstanceId and GpuId. Metric categories select what is collected, and updating a shared category affects its associated instances with eventual consistency.

I would first compare devices within the same serving pool, using equivalent traffic windows. If one GPU looks different, check workload placement before declaring the entire instance oversized. If every device looks lightly used while requests queue, investigate the path feeding the model rather than buying more accelerators immediately.

Keep customer-facing latency and throughput beside these infrastructure charts. The goal is not maximum utilization at any cost. A change that packs work more tightly but makes interactive responses slower may be the wrong tradeoff.

Make remediation a controlled serving change

AWS's walkthrough recommends confirming the reported action, protecting active work, draining the affected node and then rebooting when appropriate. It also recommends checking status after recovery. That sequence matters more than copying a reboot command into an alarm handler.

For an inference service, my proposed checklist would add three gates: confirm spare capacity can accept traffic, stop assigning new requests to the affected instance, and verify the model is loaded and serving correctly before restoring traffic. Set an owner for the decision if capacity is too tight to drain safely.

Start with notification rather than unconditional reboot automation. A hardware recommendation tells you what may fix the instance; your service runbook still needs to protect the people using it.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI infrastructure teams explain operational tradeoffs through clear technical writing and practical storytelling.

Book a 30-minute call.