Latest in AI
IBM CUGA vs Hermes and Deep Agents: What the AppWorld Benchmark Shows
IBM's same-model comparison makes a useful case for choosing an agent harness carefully, with important limits on what the scores establish.
IBM's CUGA team published a November 3 comparison in which three agent harnesses used GLM-5.3-flash but completed very different shares of the same tasks. CUGA finished 81.1%, Hermes 46.7% and Deep Agents 30%. The result puts attention on the software around the model, although the configurations were different.
CUGA benchmark results
The IBM team's report covers 90 AppWorld tasks grouped into 30 scenarios, with one attempt per task. These are the team's reported results, not an independent replication.
| Harness | Tasks completed | Scenarios with all three variants completed |
|---|---|---|
| CUGA | 73/90, or 81.1% | 20/30, or 66.7% |
| Hermes | 42/90, or 46.7% | 8/30, or 26.7% |
| Deep Agents | 27/90, or 30% | 2/30, or 6.7% |
CUGA used an AppWorld-specific configuration with decomposition, checks and reflection. Hermes used Harbor's default setup over MCP. Deep Agents used MCP with additional middleware and fixes. IBM explicitly says the runs did not test policy enforcement.
That makes the result useful evidence about these configurations on this workload. It does not establish a universal ranking of the three frameworks, or show that replacing a harness will reproduce the same gain elsewhere.
What an agent harness actually controls
A harness is the surrounding software that gives a model tools, manages its execution and decides what happens after each response. Two agents can use identical model weights and still give the model substantially different problems to solve.
The published AppWorld adapter makes some of those differences concrete. CUGA uses its own registry over OpenAPI specifications, while Hermes and Deep Agents access the simulated apps through MCP. The repository also distinguishes between agent-managed login and a runner that logs in first. Reproductions need to specify that choice rather than treating every CUGA run as equivalent.
The adapter evaluates the final application state and submitted answer. Full task credit requires every AppWorld test to pass. It separates grading access from the running agent and describes removing ground-truth material from the runtime image.
Those details matter because an agent can produce a convincing answer after making the wrong change. Checking the resulting state asks a harder and more useful question than whether the final response sounds right.
Why tool configuration deserves its own evaluation
The same repository's Deep Agents integration notes describe pinning login tools so a tool selector cannot hide a necessary authentication step. The integration also adds retry handling and a tool-call limit to contain loops.
These are ordinary engineering decisions with direct consequences for task completion. If an agent cannot see the operation needed to log in, a better reasoning model may spend more time searching without fixing the underlying problem.
For a team choosing an agent stack, a useful local comparison would hold the business task and permissions steady, then record where each configuration fails. Missing tools, failed pagination and incorrect calculations need different fixes. A single completion percentage hides that distinction.
Completion and permission enforcement are separate tests
CUGA's policy documentation describes runtime controls including human approval before selected tools execute and ToolGuard checks that can block calls. It also includes playbooks and tool guidance, which steer behaviour through instructions.
A high completion score cannot substitute for testing those controls. A system that completes a payment correctly but skips the required approval has still failed the business workflow. Conversely, refusing a prohibited action may be the correct outcome even when a benchmark rewards completing it.
The practical next step is to evaluate both: whether the agent can finish the work and whether it respects the conditions under which it is allowed to act. IBM's comparison is a reason to inspect that surrounding software more closely before paying for a larger model.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help technical teams explain complex AI products and research clearly, so developers and founders understand what changed and why it matters.