Latest in AI
BrokenML Benchmark: How Access Changes AI Agents' ML Repair Results
A new benchmark compares code-only debugging with logs and live clusters, but its results need to be read alongside the permissions and test environment.
BrokenML tests whether AI agents can repair failed machine-learning jobs when they have different levels of access. Alex Zhebrak shared the benchmark on October 12. Its most useful question for an engineering team is straightforward: how much does a debugging agent need to see and control before it can make a reliable repair?
The public project contains 46 faulty jobs and tests nine models under three access conditions. It is a small, deliberately constructed benchmark, rather than evidence that an agent is ready to manage a production training cluster unattended.
What BrokenML tests
The tasks run PyTorch training or a llama.cpp inference server in Docker Compose. Each contains an injected fault. The agent changes the job's configuration or code, and a separate verifier runs the proposed repair on a fresh cluster with fresh data.
That fresh run matters. An agent can make its current session look healthy through a temporary intervention. A repair that survives a new environment provides a stronger signal that the underlying change is useful.
The benchmark compares files-only access, live read-only logs and access to the running cluster. In the last condition, the agent has root access within the test environment. These are materially different operating conditions, so the result should always travel with the access setting.
What the reported results show
The author's results report gives these pooled pass rates across the nine models:
| Access condition | Reported pass rate |
|---|---|
| Files only | 43% |
| Live read-only logs | 71% |
| Live cluster | 84% |
These percentages describe this task set and one run per model, task and condition. They are not general success probabilities for arbitrary ML incidents.
The report places GPT-6.1 Sol at 46 of 46 with live access, compared with 35 of 46 from files alone. But the top seven models were not statistically separated from each other in the live-cluster comparison after the report's correction for multiple comparisons. A perfect observed score on this set should not become a claim of perfect reliability.
For someone choosing an agent, the more actionable comparison may be the same model under different access settings. Before paying for a stronger model, check whether the current agent is missing the logs, state or test feedback a human would need to diagnose the problem.
Where the benchmark is limited
The main study ran on CPUs using Gloo, a distributed communication backend. It did not test agents repairing GPU clusters. Its faults were injected, and its single-run design cannot measure how often the same agent would produce a different outcome when retried.
The scoring rules also make clear that files-only access removes the bug report and runtime evidence along with cluster access. The experiment therefore changes several things together. It cannot isolate the value of interactive control from the value of simply receiving more information. The study's raw runs are not public, another limit on external inspection.
There are implementation effects too. The project's known-issues document records terminal wrapping that caused configuration failures and infrastructure cases regraded with a patched runtime. Its cost accounting missed some truncated responses, including a substantial part of Sonnet's bill. Those caveats are worth reading before using the leaderboard for purchasing decisions.
More evidence does not require unrestricted production access
In the launch discussion, a reader asked whether failures made the system worse. The author reported that 13 of 414 live-cluster fixes did so. That is an author-reported result, but it is a useful reminder that a repair metric and a damage metric answer different questions.
A sensible evaluation for your own team could start with recorded incidents in an isolated environment. Give the agent the same initial evidence each time, vary one permission at a time and record both successful repairs and unwanted changes. Keep production credentials and customer data outside that experiment.
BrokenML offers a concrete starting point for that kind of evaluation. The next question is whether a team can preserve the benefit of runtime evidence while narrowing the changes an agent is allowed to make.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help technical founders turn research, product decisions and engineering work into clear writing that people can understand and share.