PARAS MADAN
All writing

Latest in AI

Tether 0.4 Agent Evals: Replay One Recorded Run Before Changing a Policy

Use a synthetic trace to check what a new rule would block, then separate that result from a live test of what the agent would do next.

Tether 0.4 adds a way to run a suite of checks against recorded AI agent runs. The versioned changelog lists assertions for tool calls, file contents visible in the trace, and step or cost limits. If you are changing a rule that governs what an agent can do, you can start with one recording and test the policy decision without asking a model to generate the same run again.

Pick one decision you need to keep

Use a small synthetic task: the agent should write a report to a scratch folder while leaving a separate reference note untouched. Give it a fake model response or a recorded response from a test run. Write down what a pass means before seeing the replay. It should produce the report, never write to the reference note, and stop within a small number of steps.

Start by replaying the untouched run and comparing the result to the recording. The Tether replay guide explains that matched model and tool calls get their recorded responses, and policy decisions are evaluated again. Use the original trace and its accompanying blobs. The project documents tether pack for keeping them together. Do not use a customer trace for this exercise: the docs warn that prompts, model responses and tool output can be stored in full and unencrypted.

Replay with the candidate policy and inspect the first difference

For a recording you already made with Tether and a reviewed candidate policy, the documented command shape is:

tether replay run.jsonl --policy strict.toml --out candidate.jsonl
tether diff run.jsonl candidate.jsonl

If the first difference is an allowed scratch-folder write becoming a denial, you now know which policy decision to inspect. Check whether the intended restriction was limited to the reference note. A broad deny might protect the note and block the only write the agent needed to finish.

If the decision looks right, use a small suite rather than treating one replay as decisive. Tether's 0.4 release adds tether eval for replaying the tasks in a directory and checking their eval.toml assertions. Keep one expected success and one expected denial in that suite. Only assert on effects the trace records: a shell command that happens to create a file doesn't necessarily record the file's contents, while a recorded file-writing tool can expose those contents as part of the call.

Know when replay stops answering the question

A new prompt or model can produce a different request. A previously denied tool call can become allowed under a changed policy, but the recording has no result for a tool that never ran. By default, the documented replay stops when it cannot answer from the recording. That stop is useful evidence. It marks the point past which you are no longer evaluating the old run.

Leave live continuation off for the replay check. If you need to see what the agent does next, run a separate fresh test in a properly isolated environment with fake tools and test data. Tether's documentation warns its default process sandbox is not a security boundary. The important distinction is between evidence that a rule made the expected decision on known input, and evidence that a new run can still complete the task safely. Keep both before promoting the rule.

I'm Paras. I work across AI, engineering and distribution.

I help AI teams turn agent behavior, evaluation and deployment changes into technical guidance developers can use.

Book a 30-minute call.