# AgentCore Evaluations: Choose Metrics for a Support Agent A worked refund example shows how to separate completion, policy compliance and privacy instead of hiding failures inside one score. By Paras Madan | 2026-11-11 Amazon Bedrock AgentCore Evaluations can assess an agent’s answers, tool use and overall session outcome. For a support agent, start by defining a successful resolution and the failures that must block release. Choosing an evaluator because its name sounds useful is an easy way to measure the wrong thing. AWS’s November 10 [evaluation playbook](https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/) describes built-in, managed DeepEval and AutoEval evaluators, plus custom evaluators. The important implementation choice is which evidence each judge needs. A polished final reply cannot establish whether a refund actually happened. ## Define the outcome before choosing the metric Consider this illustrative support request: “Refund the duplicate charge, keep my subscription active and send confirmation to my existing email.” The agent needs to understand three requirements and complete the permitted actions without exposing another customer’s data. Write those requirements as separate acceptance conditions. The duplicate charge must be identified, the refund must have a successful backend result, the subscription must remain active and the confirmation must use the verified destination. A friendly message saying everything is done satisfies none of those conditions on its own. This exercise also reveals missing instrumentation. If the trace contains a refund request but not the tool’s result, the evaluator lacks the evidence to distinguish success from an attempted action. Fix that visibility gap before changing the judge prompt. Otherwise, improving the evaluation may mean teaching it to make more confident assumptions. ## Match the scope to the failure The [built-in prompt templates](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/prompt-templates-builtin.html) distinguish session, trace and tool evidence. GoalSuccessRate examines the conversation’s goals across the session. Tool evaluators inspect the action under review, while response evaluators assess properties of the assistant’s answer. For the refund example, begin with session completion, the correctness of the refund tool arguments and whether the final response matches the observed tool result. Add a privacy check for the customer information the agent returns. These checks answer different questions, so keep their results separate. A wrong charge identifier is a tool-argument failure even if the API returns success. A correct refund followed by “your subscription has been canceled” is a response failure even if no cancellation occurred. An accurate response that includes another customer’s email is a privacy failure. A single average obscures the distinction that tells an engineer what to repair. ## Use managed evaluators where their definitions fit AWS’s catalog includes GoalSuccessRate and tool-selection and parameter checks. Its managed DeepEval options include ConversationCompleteness and PIILeakage; AutoEval includes specialized checks such as Security. Managed execution means AWS hosts the evaluator code and selects the judge model. Custom derived evaluators provide control over the model and rubric. Read the definition rather than treating the name as a promise. For example, the documented Helpfulness rubric focuses on whether a response advances the user’s goal, explicitly separating that from factual accuracy and backend operations. It should not be your proof that money was refunded. Similarly, the documented Faithfulness check looks for conflict with conversation history. That can identify a reply contradicting a failed tool result, but it does not independently verify the truth of an external system. The evaluator can only inspect the evidence you provide. ## Build a small regression set with distinct failure cases Start with a deliberately designed set of sessions rather than a large pile of unlabeled conversations. Include a complete refund, a failed refund reported honestly, a failed refund reported as successful, a wrong-charge refund, an accidentally canceled subscription and a reply exposing another customer’s information. Have a person label each condition for each session. Then compare the evaluator’s explanation with those labels. A disagreement may reveal a bad judge, an ambiguous policy or an incomplete trace. Those require different fixes, and retaining the explanation helps tell them apart. Add a valid refusal case too. If the customer is not authorized to request the refund, completion should not mean issuing it anyway. Define the acceptable resolution as explaining the limitation and following the approved escalation path. Otherwise, your benchmark rewards an unsafe shortcut. These are proposed test cases, not results from a tested implementation. Run them against the exact evaluator and agent versions you plan to release, then preserve that configuration with the results so a later comparison is meaningful. ## Separate release gates from production sampling AWS describes on-demand evaluation for individual sessions, batch evaluation for fixed regression datasets and online evaluation for sampled production traffic. Checks requiring reference answers or expected trajectories belong in batch evaluation rather than ordinary live traffic. Use the regression set to decide whether a change is acceptable. Use production sampling to discover situations the set missed. Neither replaces the other: a fixed set can become too familiar, while sampled traffic may miss rare but costly failures. Multi-dimension custom evaluators can score several criteria from one judge call when they share the same evidence, scope and model. Keep the dimension results independent. For this example, clearer wording should never compensate for refunding the wrong charge. **Treat a release gate as a decision rule, not a leaderboard.** Specify which failures block release, which require review and which are quality trends. The point of evaluation is to make the next engineering decision more precise, not to produce a reassuring number. I’m Paras. I work at the intersection of AI, engineering and distribution. I help AI teams explain technical decisions, evaluation methods and product tradeoffs in content developers can understand and use. [Book a 30-minute call](https://cal.com/paras-madan/30min).