Latest in AI
Foundry Human Feedback: Turn Agent Traces Into Regression Tests
Build task-level evaluation rubrics from reviewed Foundry traces, validate expected outputs and avoid misleading agent quality scores.
An agent can earn a good overall score while repeatedly making the mistake your customers care about. A support answer might sound helpful, follow the requested format and still promise something the company cannot deliver.
Microsoft Foundry's new human-in-the-loop evaluation workflow gives teams a way to turn those specific failures into repeatable tests. The useful question is not whether a reviewer liked the old answer. It is whether the next version avoids the same mistake.
What the preview actually adds
Microsoft's November 18 announcement describes a public preview introduced at Ignite on November 17. Reviewers annotate traces with ratings, tags, comments and expected outputs. Those reviewed interactions can become datasets with scoring criteria tailored to individual cases.
This is different from collecting thumbs-down feedback in a dashboard. The feedback becomes something that can assess a future response, even when its wording changes.
The prerequisites include a Foundry project with agent traces, connected Application Insights, appropriate permissions, a supported region and a model deployment for rubric generation. Before copying traces into a dataset, review their contents and who will be able to access them.
The preview documentation says there is no service-level agreement and Microsoft does not recommend the preview for production workloads. Treat this as an evaluation pilot, not a production release gate you can assume is dependable.
Write feedback that describes a failure
I would start with a small set of interactions that someone on the team can explain without consulting another model. Include successful cases as well as failures so the dataset does not reward an agent that simply refuses everything.
Consider a fictional subscription-support agent. A customer asks to cancel after the next renewal. The agent cancels immediately. “Bad answer” is too vague to become a useful test. “The customer requested cancellation after the next renewal; the agent must preserve access until that date” gives the reviewer and evaluator something concrete.
Record the relevant policy version alongside the example. Otherwise, a legitimate policy change can look like a model regression, or an obsolete answer can remain the supposed ground truth.
I would also separate a wrong explanation from a wrong action. A fluent apology should not compensate for changing the subscription at the wrong time.
Generate the dataset, then review it
The dataset creation guide documents the portal route: open Traces, select annotated traces or conversations, choose Add to dataset, enable rubric generation if needed and select a model deployment. Open the resulting rows before evaluating.
For SDK users, rubric generation requires azure-ai-projects 2.11.0 or later. The documentation distinguishes creating a new dataset with OVERWRITE from adding rows with MERGE; merging fails when the dataset does not already exist.
My review would ask three questions. Does the row preserve the customer's actual request? Does the criterion test the failure we observed? Could an answer technically satisfy the wording while still giving the customer the wrong result?
For the cancellation example, a criterion that only checks for the phrase “next renewal” is insufficient. The test needs to preserve the requested timing, not reward a keyword.
Check the expected-output schema
There is an important failure mode in the task-level rubric evaluator documentation. If any entry in expected is malformed, the evaluator ignores the entire field and grades only the criteria.
Each expected entry needs an annotationId. Query-and-response rows use output; message-based rows use outcome. Conversation-level evaluation also requires test_case_description.
That deserves a validation check before trusting a score. A run that completes successfully does not necessarily prove that every intended expected answer was considered.
The evaluator, builtin.task_level_rubric, uses each row's rubric_dimensions and optional expected output. Generated criteria have no weights. Shared quality or safety rubrics remain separate checks rather than automatically becoming part of this task-specific score.
Compare versions without changing the exam
I would keep the initial dataset version fixed while comparing two agent versions. Save the model, instructions, tool configuration and evaluator setup with each result. Otherwise, several moving parts can make an apparent improvement impossible to explain.
Read individual failures before accepting the aggregate. In the subscription example, an improvement on routine billing questions should not hide a new cancellation-timing error.
Use a separate set of examples that were not used to tune the agent. Repeatedly fixing the same small collection is useful engineering work, but it is not evidence that unfamiliar requests will behave equally well.
The practical outcome is a review process that produces tests the team understands. Generated rubrics can reduce the work of writing those tests. They do not remove the need to decide what a correct result means.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help AI teams explain evaluation workflows and developer products through clear technical writing and practical storytelling.