NVIDIA NeMo Compass: Turning Agent Failures Into Useful Evaluations

NVIDIA's November 4 tutorial demonstrates a practical agent-improvement loop: investigate recorded failures with NeMo Compass, build evaluations with NeMo Eval Author, change the agent's instructions and measure the result. The example uses a workplace assistant at a fictional company.

This is useful for teams whose agents perform well on general benchmarks but still mishandle ordinary requests. The question becomes specific: which behavior failed, and what evidence would show that a proposed fix actually helps?

Compass makes traces easier to investigate

The NeMo Compass repository describes several ways to inspect agent activity. It can flag unusual runs, recurring tool errors, complaints in user messages, behavior that conflicts with the agent's stated goals and patterns associated with poor evaluation scores.

Supported sources include Braintrust, LangSmith, Langfuse, MLflow and local trace files. That gives teams a way to investigate evidence they already collect rather than starting with an entirely new logging system.

The repository explicitly calls Compass an early research preview that is not intended for production use. Its interfaces may change, and findings need human review against the underlying traces. Treat it as an experimental analysis tool, with the same access controls you would apply to any system reading customer conversations or internal records.

A draft request becomes a test case

In NVIDIA's example, the assistant was asked to draft a project update but also sent it without confirmation. Compass examined 42 traces. The team then created four Harbor evaluation tasks and changed the assistant's instructions to distinguish preparing a message from permission to send it.

NeMo Eval Author supplies skills for coding assistants to build and review custom evaluations. It supports Harbor and NeMo Gym formats, using the agent's intended behavior, repository and available traces. A first suite can be created without production traces, although auditing observed coverage requires completed evaluation runs.

The difficult part remains deciding what success means. A test should inspect the action the agent took, not merely accept a final answer that says it followed the rules. It also needs cases where sending really is authorized, so an overly cautious fix does not make the assistant useless.

Read the results at their actual scale

NVIDIA reports these results after the instruction change:

Evaluation group	Before	After
Development	3 passes in 6 runs	6 passes in 6 runs
Held out	4 passes in 6 runs	6 passes in 6 runs

Each group contained two tasks run three times. The held-out tasks were hidden from the coding agents proposing changes.

Those results show improvement on a narrow demonstration. Repeated runs of two tasks do not establish reliable behavior across a company's full workload. A broader evaluation needs different wording, ambiguous instructions, changed recipients, tool failures and legitimate approvals.

Keep that distinction visible when reporting progress. A perfect score on a small test set is a reason to expand testing, not a substitute for it.

Review the tests as carefully as the fix

Eval Author is also a research preview. Its documentation warns that generated cases can be too easy and that graders and coverage findings need review. A passing test can say more about a weak check than a strong agent.

For an initial experiment, choose one recurring failure with a clear expected outcome. Inspect the raw trace, build both failure and success cases, and record a baseline before changing anything. Keep some scenarios hidden from whoever proposes the fix, then check unrelated behaviors for regressions.

For consequential actions, pair that process with enforcement outside the model. A clearer prompt helps communicate intent, while an approval check in the tool layer can prevent execution when authorization is missing. Evaluations then test both whether the agent understands the boundary and whether the system enforces it.

The value of this workflow is the connection between an observed problem and a repeatable check. It gives engineering teams something more useful to discuss than a broad claim that the agent has become better.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams explain technical products through clear writing, storytelling and distribution.

Book a 30-minute call.