Latest in AI
SageMaker HyperPod hung-job detection: start with notify, then prove recovery
Pilot HyperPod Slurm hang detection without premature cancellation, distinguish EKS monitoring from recovery, and budget for training validation checks.
A training job can remain “running” while producing nothing useful. The GPUs stay allocated, the scheduler sees an active job, and the team discovers the stall hours later.
AWS's November 18 HyperPod update addresses that gap with Slurm hung-job detection, alongside expanded EKS infrastructure checks. These are different capabilities for different orchestrators, not one switch that repairs every training failure. AWS announcement.
I would introduce detection before automatic cancellation. The first question is whether the signal describes a genuinely stalled job. The second is whether restarting that job actually recovers useful work.
Check the cluster before changing the job
AWS says Slurm hung-job detection requires a HyperPod Slurm AMI released on or after November 10. Its suggested read-only check is srun --help: look for --no-hang-detection. If a software update is needed, plan it separately. The update replaces instance root volumes, so preserve anything needed from those volumes first.
Do not bundle a cluster upgrade, new detection rules and a production training launch into one change. I would record the current image, entrypoint, checkpoint location and job owner before the pilot. That gives the team something concrete to compare if behavior changes.
Enable a notification-only pilot
Detection is off by default. A monitored job sets HYPERPOD_HANG_RULES to a rules-file path readable from every compute node. This minimal JSON enables default detection:
{"enabled": true}
Reference the saved file in the batch script:
#SBATCH --export=ALL,HYPERPOD_HANG_RULES=/fsx/my-project/rules.json
The default action, notify, records an event without stopping the job. Custom log-pattern rules replace default detection rather than supplementing it. If a rules file is missing or invalid, the job can run unmonitored; inspect CloudWatch for CONFIG_ERROR, not just the scheduler output. Hung-job detection guide.
For the first pilot, pick a short, representative workload with a clear owner. Route the resulting events into the team's existing alerting process. A record sitting unread in CloudWatch does not shorten an overnight incident.
Define progress in training terms
A useful progress signal should distinguish completed work from repeated status messages. I would prefer a completed-step or completed-epoch record over a heartbeat that merely proves a process is alive.
Build the timeout from observed startup, checkpoint and evaluation pauses. Keep a sample of normal logs beside the rule so another engineer can explain why the chosen gap is unusual. Test both a healthy run with a long legitimate pause and a controlled stalled run on disposable capacity.
Use explicit acceptance criteria: no false alarm during the known pause, an alert during the intended stall, and enough evidence in that alert for the owner to identify the job. Faster detection is not automatically better if it repeatedly interrupts valid training.
Cancellation is not proof of successful recovery
With cancel, Slurm normally requeues the job. A job submitted with --no-requeue instead stays cancelled. Confirm the cluster's retry limits before enabling this behavior. Detection actions.
HyperPod's existing auto-resume mechanism is separate: it handles failures and can restart after hardware replacement. Its documentation requires a reproducible entrypoint and consistent environment setup on replacement nodes. Auto-resume documentation.
My rollout gate would be a demonstrated return to the expected checkpoint, followed by new training progress. Check optimizer state, data position and output naming as part of that exercise. A successful resubmission that starts from zero, overwrites artifacts or repeats an unrecoverable failure is not the outcome the team wanted.
Keep the EKS capabilities separate
The new Kubernetes Resource Monitoring Agent reports affected infrastructure, pods and jobs; it does not itself reboot nodes or restart workloads. AWS's example validation run added roughly eight minutes for the training checks, within a full deep-health-check run lasting 88 minutes. Those are AWS's measurements, not a provisioning-time guarantee. AWS setup and measurements.
MultiNodeTrainingValidation applies to supported EFA-enabled NVIDIA GPU types on EKS. It runs a small reference workload per instance and across instances of the same type. Check results appear in the cluster's DeepHealthCheckResults log streams. Deep health checks.
Reserve both time and compute budget for validation. A passing reference workload is useful evidence about the capacity, but I would still run the team's own small training smoke test before committing to a long run. Hardware validation and application recovery answer different questions.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help AI infrastructure teams explain complex systems through clear technical writing and practical product stories.