Latest in AI
Together GPU Clusters Auto Repair: Controls, Data Loss and Preview Features
Together's latest technical breakdown explains which GPU repairs can run unattended and where an operator still needs to make the decision.
Together AI published a detailed explanation of GPU Clusters auto repair on October 27, including per-fault approval controls and preview integrations for external detectors. This is an update on an existing system, not a first launch. The useful question is what happens to running work when a repair starts.
What is available, and what remains in preview?
Together's technical breakdown lists passive health checks, repair recommendations, approval policies, wait-for-idle controls and API and console repairs as generally available. Node-label remediation contracts, which let external detectors request repairs, remain in preview. Host power cycling is also described as a preview capability.
The company had already announced auto node repair in July. The new post is useful because it explains how the platform chooses an action, escalates recurring faults and limits the damage from a burst of correlated alerts.
That distinction matters for evaluation. A new architecture explanation can make an existing service easier to assess without implying that every capability in it just became available.
A fault does not always mean replacing hardware
Together's health-check documentation covers GPU memory errors, thermal throttling, PCIe issues and InfiniBand faults, among other signals. An Xid is an NVIDIA driver or hardware fault reported in the kernel log. It is evidence to interpret, rather than a universal instruction to replace a machine.
Repair can escalate from a VM reboot to reprovisioning and then migration onto different hardware. These are distinct responses, not interchangeable ways of restarting a job.
This separation is useful for distributed workloads. A software problem and a broken GPU may both stop a training run, but they do not require the same recovery path. Automatically choosing the most disruptive response to every failure can waste capacity as well as work.
Approval and checkpointing are separate decisions
The node-repair guide offers approval-first or automatic policies, with consent configurable by fault. Individual nodes can be excluded for investigation.
Wait-for-idle can defer repair until work finishes or a deadline expires. Removing the deadline avoids interrupting a job but can leave faulty capacity unrepaired indefinitely.
Automatic repair does not remove the need for recoverable workloads. Teams still need checkpoints stored somewhere a replacement machine can access. A system that restores capacity quickly is not necessarily a system that preserves every minute of a training run.
Wait behavior also differs for clusters without saved settings. Inspect the configured policy rather than assume a universal default.
Which repairs preserve local data?
A reboot preserves local scratch and temporary data. Reprovisioning and migration lose local VM data. Recovery time is only part of that tradeoff.
For a long training job, the relevant recovery cost includes the work since the last usable checkpoint, not only the time spent repairing the machine. For inference, it includes whether other replicas can serve requests while one disappears. The right policy therefore depends on the workload, not just the error code.
Not every documented signal generates an automatic recommendation; some still receive internal review. Detection coverage is not identical to unattended repair coverage.
Why the safeguards matter
The new explanation describes limits on simultaneous repairs and a circuit breaker for correlated failures. That is an important design choice: many nodes becoming unhealthy together can point to shared infrastructure rather than many independent hardware failures.
The practical test for this kind of automation is whether it restores useful service without turning a shared outage into a wave of unnecessary rebuilds. Faster repair is valuable, but controlled repair is what makes it safe to leave unattended.
I’m Paras. I work at the intersection of AI, engineering and distribution.
I help technical teams explain complex AI infrastructure through clear technical writing and developer storytelling.