# NVIDIA Recovery Explained: Slurm Training Recovery and Its Current Limits

New deployment details show how NVIDIA coordinates failed training jobs, but recovery-action speed is not the same as getting useful training back.

By Paras Madan | 2026-11-03


NVIDIA published new deployment details and operational measurements for NVIDIA Recovery on November 2. The open-source service coordinates recovery across distributed training jobs. Its current public scope is Slurm, including Slurm on Kubernetes, and its repository still labels it experimental rather than production-ready.

This is a technical update about an existing project, not a newly released model or a claim that every NVIDIA training cluster can now recover automatically.

## What NVIDIA reported on November 2

The [technical post](https://developer.nvidia.com/blog/keep-distributed-training-running-with-automated-workload-recovery/) describes monitoring one training cluster from July 9 through October 9, covering nearly 200 jobs a day. NVIDIA reports approximately 8,550 anomaly detections, with average detection and recovery-action latencies of 69 and 94 seconds.

These are vendor observations from one cluster, not an independent comparison. Crucially, the recovery-action measurement ends when the automated action happens. Scheduling and checkpoint restoration still affect when useful training resumes.

NVIDIA also describes starting deployments in monitor mode. Operators can inspect recommendations before enabling scheduler actions. That is a sensible evaluation boundary: a recovery system needs to earn permission to interrupt and restart expensive work.

## How NVIDIA Recovery handles a failed training job

The [project repository](https://github.com/dsx-ai-factory/NVIDIA-Recovery) describes detection of crashes, hangs and unusually slow workers. A failed attempt becomes a recorded recovery decision, which the scheduler integration carries out. The restarted application then loads a checkpoint through its existing restart logic.

This separates deciding what should happen from checking that the action actually happened. The distinction matters if the controller itself restarts midway through recovery, or if the scheduler has already completed an action.

It also places a clear boundary around the product. **Restarting a job does not recover work that was never checkpointed.** Teams still need useful progress signals, valid checkpoints and an application that can resume from them.

## Slurm support is not native Kubernetes recovery

The public repository lists Slurm and Slurm-on-Kubernetes through Slinky, but says Kubernetes-native workload recovery is not implemented. Running a Slurm scheduler inside Kubernetes does not make arbitrary Kubernetes jobs supported.

This should also be distinguished from [Mission Control 2.5](https://www.parasmadan.in/blog/nvidia-mission-control-2-5-kubernetes-recovery), whose supported Run:ai and training-operator integrations are a separate product boundary. Similar recovery terminology is not evidence that the two packages have identical support.

The repository’s [support policy](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/SUPPORT.md) explicitly warns that APIs, configuration and behaviour can change. Community response goals are not service-level agreements or around-the-clock incident support.

## What to check before evaluating it

There is another practical dependency to inspect. The [Attribution documentation](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/docs/public-api/optional-attribution.md) says the diagnostic provider is enabled by default but requires authorized source access. An explicitly degraded mode can run without that provider; it should not be mistaken for a test of provider-assisted diagnosis.

For a platform team, an evaluation should therefore follow the whole failure path: detection, recommendation, scheduler action, checkpoint restoration and the next successful checkpoint. Recording only the restart event makes the system look better without establishing that the research run is progressing again.

The useful question is how much productive training survives an interruption. Faster recovery actions help, but the next completed checkpoint is stronger evidence than a green controller dashboard.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams explain infrastructure releases and operational trade-offs in content developers and founders can actually use.

[Book a 30-minute call](https://cal.com/paras-madan/30min).