PARAS MADAN
All writing

Latest in AI

How to Compare Coding-Agent Harnesses When You Change Models

A small comparison matrix can separate model gains from harness behavior and expose costs hidden by a leaderboard's winner.

Changing a coding agent's model can change which harness works best for it. The harness controls the tools, prompts and execution loop around the model, so keeping that layer fixed is a choice worth testing rather than an assumption that the new model will behave like the old one.

RUNTA's November 17 FrontierHarness v1.1 report compares model-harness combinations on terminal tasks served through Baseten. Its preferred harness changes with the model. Those results are a reason to test combinations, not a universal recommendation to replace your current agent.

Read the evaluation's boundaries before choosing a winner

The report uses open models from one provider, with inference costs estimated at list prices. That excludes a complete account of your team's operating costs. Its “best value” selection also includes a quality-versus-cost rule, rather than choosing the smallest bill. The reported uncertainty leaves several model-specific winners close to alternatives.

Treat the November report as a dated snapshot. The live leaderboard can change independently, so save the model ID, harness version and scoring definition behind any result you plan to compare. Do not combine a current rank with a price or configuration from an older post.

Start with four combinations, not a long tool shortlist

Suppose your team uses Harness A with Model X and wants to move to Model Y. A useful first experiment compares A/X, A/Y, B/X and B/Y, where B is one alternative that supports both models. This is an experiment design, not a benchmark I have run.

Keeping A/X establishes the current baseline. Comparing A/Y with A/X isolates the proposed model change within the same harness. Comparing B/Y with A/Y checks whether the alternative helps the new model. B/X tells you whether B was simply better for this workload all along.

Use the same repository snapshots and owner-approved task instructions. Include a bug fix that requires reading existing code, a small feature with a hidden edge case and a maintenance task representative of your actual work. Preserve an independent acceptance check for each. Do not let the candidate agent rewrite the check that determines whether its attempt passed.

Record the settings that can invalidate the comparison

For every attempt, save the exact model and provider, harness revision, reasoning setting, tool permissions, task budget and starting commit. Record whether previous conversation state or persistent memory was present. A fresh attempt and a resumed session answer different questions.

Keep network and filesystem access consistent unless the purpose of the experiment is to compare those capabilities. Never give a candidate production credentials merely because another agent already has them. Use disposable test environments with approved access, and separate an infrastructure startup failure from a model that starts successfully but cannot solve the task.

Harbor's repository documents support for evaluating agents, defining benchmarks and running isolated environments locally or through providers. It is one way to organize repeated trials, not a requirement for this four-cell pilot. Review its current adapter support before assuming your chosen harness-model pair is available.

Calculate cost per accepted result

Consider these invented totals for 20 attempts on the same task mix:

CombinationTotal inference costAccepted resultsCost per accepted result
A/X$1212$1.00
A/Y$1010$1.00
B/X$1515$1.00
B/Y$1216$0.75

A/Y has the lowest bill but also fewer accepted results. B/Y costs more in total while producing the lowest cost per accepted result. These figures demonstrate the arithmetic only; they are not current model prices, measured outcomes or evidence that B/Y will win your evaluation.

Include failed attempts in the spending total. Report runtime charges and reviewer effort separately so an inference saving does not hide a larger bill elsewhere. Also retain elapsed time: an inexpensive combination that regularly misses a delivery deadline may not fit the job.

Inspect disagreements before switching the team

Review tasks where one combination passed and another failed. Look for recurring causes such as losing context, choosing the wrong tool or producing plausible code without finishing the acceptance checks. A small aggregate difference can come from a few unusual tasks, so repeat promising comparisons rather than treating one pilot as a permanent ranking.

Choose the combination that meets the team's quality and access requirements at an acceptable cost. Save the experiment configuration with the decision. When the next model, harness or provider changes, you will have a baseline to rerun instead of another leaderboard argument.

I'm Paras. I write about AI engineering and how technical choices affect real workflows.

I help developer-tool teams explain evaluation results and product trade-offs through clear technical writing.

Book a 30-minute call.