PARAS MADAN
All writing

Latest in AI

How to catch flaky agents with Google ADK 2.15

Use Google ADK's pass@k and pass^k metrics to separate occasional success from repeatable behavior, then inspect the tool arguments behind failures.

An inventory agent finds the right item in three trials and checks the wrong warehouse in the fourth. The failure might disappear on a rerun, but that does not make the agent ready. You need a test that tells you how often it follows the right process, not just whether it ever gets there.

Google's ADK Python 2.15.0 release adds two repeated-run metrics and a more granular way to score tool arguments. Together they can tell you whether a case is flaky and help narrow down why.

Make one task repeatable

Start with a read-only inventory lookup: the user asks whether a specific item is available at the Pune warehouse. Give the eval both the expected tool call and the expected answer, and keep the inventory data fixed. If the underlying record changes between runs, a failed comparison may tell you more about the fixture than the agent.

Include a second case with the same item in a different warehouse. That makes a wrong warehouse argument visible even when the agent has chosen the right tool. Start with a handful of these cases and spend more runs on the ones that matter rather than repeating the entire suite at once.

Measure capability and reliability separately

The versioned eval configuration guide describes two estimates. Pass@k asks whether at least one of k runs passes. Pass^k asks whether all k runs pass. A run must pass every configured metric to count as a pass for these estimates.

For a hypothetical case with three passes and one failure, every subset of three runs includes at least one pass, but only one of the four possible subsets contains three passes. That gives pass@3 of 1 and pass^3 of 0.25 from the same four runs. The gap is useful: the agent can solve the task, but a single successful demo has hidden a failure.

Add a gate for the critical cases

Start by reporting both values without thresholds. Once you have a baseline, set a threshold for the behavior your team wants to enforce. Here is a small, deliberately strict example for a stable read-only fixture:

{
  "criteria": {
    "tool_trajectory_avg_score": 1.0
  },
  "multi_run_config": {
    "num_runs": 4,
    "cross_run_criteria": [
      {"method": "pass_at_k", "k": 3},
      {"method": "pass_hat_k", "k": 3, "threshold": 1.0}
    ]
  }
}

This config gates the tool trajectory; add a separate response criterion if the wording or facts in the answer also matter. A threshold of 1.0 is a rule for this test, not proof that future production runs always pass.

Pass the file explicitly to the CLI. The tagged command implementation takes an agent folder and an eval set path, and returns a nonzero status for failed tests or an empty summary:

adk eval ./inventory_agent ./evals/inventory.evalset.json \
  --config_file_path ./evals/reliability.json --print_detailed_results

Use partial credit to diagnose, not excuse, errors

The new tool parameter metric can award credit for individual arguments. If the agent sends the right SKU but the wrong warehouse, that is different from calling the wrong tool entirely. Keep the exact trajectory check for the critical fields and use parameter scores to investigate failures.

Think about retries before choosing the pairing rule. With IN_ORDER, the parameter metric pairs by tool name in sequence, so an incorrect first lookup can be the one scored even if the next attempt succeeds. Use that deliberately if the first lookup matters. Do not loosen the comparison just to make a flaky case pass.

Keep the run evidence

ADK now saves per-case multi-run summaries in .evalset_result.json. Save that file for each candidate release and inspect failed, crashed and unevaluated runs alongside the score. A pass metric with too few scored runs can be unavailable; do not treat a missing value as evidence the agent passed.

Review the Pune case before the overall mean. A strong aggregate does not make the wrong-warehouse failure harmless. The point of repeating the eval is to catch the failure your user would notice even when most other runs look good.

Make your developer docs easier to use

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams turn evaluation features and SDK changes into technical writing and product stories that developers can put to work.

Book a 30-minute call.