PARAS MADAN
All writing

Latest in AI

How to Test a Data Agent With a Synthetic Warehouse and an Answer Key

Build a small data-agent evaluation with duplicate events, conflicting metric definitions, a computed answer key and a question the agent should decline.

A data agent can query the right table and still give the wrong answer. Maybe it counts a retry twice. Maybe it follows a metric definition from an old chat message even though a current policy document says something different. A clean table and five well-phrased queries will not expose either mistake.

On November 20, Google Cloud described an internal data simulator that creates synthetic organizations with databases, documents and conversations. Google describes seeded generation, verified answers and read-only evaluation. It is an engineering write-up, not a release of the simulator for customers.

You can apply the idea without building an entire simulated company. Start with one business metric, a few rows and two documents that disagree for a reason.

Create a small fixture with a real ambiguity

Suppose a support lead wants to know how many tickets missed the first-response SLA on November 10. Make up a support event table with ticket ID, date, elapsed minutes and priority. Use four synthetic event rows:

TicketDateMinutesPriority
T101Nov 1075Standard
T101Nov 1075Standard
T102Nov 1055Standard
T103Nov 1020Urgent

The duplicate T101 row represents a retry. Give the agent a current policy that says standard tickets breach after 60 minutes, urgent tickets after 15 minutes, and repeated events do not add another breach. Include a clearly older chat message that describes the retired 45-minute threshold for standard tickets. Give both sources dates and make the policy's authority explicit.

Keep the rows and documents fixed for the first run. For larger fixtures, generate them from a recorded seed and check them into your evaluation workspace. A comparison is hard to interpret if the test data changes with the prompt.

Compute a correct answer and a few wrong ones

The correct count here is two: T101 and T103. The SQLite aggregate reference explains that COUNT(DISTINCT ticket_id) filters duplicate ticket IDs before counting. Apply it after filtering for the date and the thresholds by priority. Store the expected ticket IDs as well as the count.

Now compute the mistakes you intended to catch. Counting rows after the correct filter yields three; using the retired standard threshold with deduplication also yields three. The totals alone cannot tell those two failures apart. Their evidence differs: the first repeats T101, while the second incorrectly includes T102.

Keep a separate question asking for a metric the fixture does not define, such as “handoff recovery score.” The expected result is a clear statement that the measure cannot be established from the available evidence, not zero and not a metric the agent makes up.

Give the agent the sources, not the solutions

Expose the synthetic table and dated documents through read-only tools. Keep the expected answers, mistake cases and rubric outside the agent's accessible data. Otherwise the test measures whether the agent found the answer key.

For each response, record the query or retrieved evidence where available, the IDs used and the claimed definition. Score the count and explanation separately. An answer that gets two by chance is not as useful as one that identifies both tickets and cites the current rule.

If a run fails, change one variable: retrieval order, policy naming, prompt or model. Repeat the same fixture before moving on. That lets you see whether a change fixed the problem you meant to fix.

Use the fixture as a regression test, not a deployment guarantee

Four rows cannot tell you whether a warehouse agent handles slow queries, sparse permissions or every equivalent way people ask about an SLA. This test answers a narrower question: can the agent apply a current policy, avoid double counting and say when it lacks evidence?

Once that works, add one complication at a time. Try a policy transition that falls inside the reporting window or a join that introduces another duplicate. Make sure each new case has a defensible reference answer before adding it to the pass rate. The point is not to make the fixture large. It is to make a failure explainable.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI teams explain practical engineering decisions and evaluation workflows through clear technical writing and storytelling.

Book a 30-minute call.