How to Test Whether a Coding Agent Follows Changed Requirements

To check whether a coding agent follows changed requirements, give it two versions of the same task, change one explicit rule, and evaluate both the new behavior and the behavior that should stay intact. A passing original test suite cannot tell you whether the revision was implemented.

This is a useful way to compare agents before buying more seats or trusting them with larger changes. It also gives you something more actionable than arguing over a leaderboard: a record of which requirements your own workflow loses.

What MaxxBench actually tests

MaxxBench, published by Gavin Lo and Justin Lo on October 20, changes instructions or starting code in 89 DeepSWE tasks. It is not a new November model release. Its current public discussion is a reason to examine the evaluation method, rather than repeat its ranking.

The researchers describe cases where an agent understood a revised requirement but chose behavior it thought an imagined grader would prefer. Their method updates reference solutions and tests while retaining checks for unchanged behavior. Those are the authors’ findings, not tests reproduced here.

The practical lesson is straightforward: test the requirement that changed explicitly. A large green test suite can still be testing yesterday’s contract.

Start with one observable contract

Use a disposable repository without production credentials. Choose a small function with outputs you can check independently, then write down the old contract, the new contract and the invariants. Avoid starting with “improve the architecture,” because you will spend more time debating the verdict than comparing the agents.

Here is an original teaching example, separate from the MaxxBench dataset. A Python function formats timezone-aware timestamps for an export. The old contract preserves the input offset. The revised contract returns UTC with a trailing Z. Both versions must reject timestamps without timezone information and preserve the represented instant, including microseconds.

An input of 2026-11-08T00:15:00+05:30 should become 2026-11-07T18:45:00Z. That date change is deliberate: an implementation that replaces the suffix without converting the time will fail. Python’s datetime documentation explains the distinction between aware and naive datetimes and the conversion behavior of astimezone.

Give the agent a precise revision prompt

Start with this deliberately old implementation in export_time.py:

from datetime import datetime


def format_export_time(value: datetime) -> str:
    if value.tzinfo is None or value.utcoffset() is None:
        raise ValueError("timezone required")
    return value.isoformat()

Use the following prompt after confirming the baseline works. Give each candidate the same starting files, tools and budget.

Update format_export_time in export_time.py.
The revised output must represent the same instant in UTC and
end in Z. Preserve microseconds when present. Continue rejecting
naive datetimes with ValueError. Do not change the public function
signature or weaken the tests. Explain which old expectation the
new contract replaces, then implement the change.

For a clean comparison, start each agent in a fresh copy of the baseline repository. For a separate conversational-revision test, first let it solve the old task and then give it this prompt in the same session. Record which experiment you ran; the second includes the difficulty of revising an earlier answer.

Check conversion, precision and rejection independently

Keep an evaluator-owned copy of these checks outside the agent’s editable workspace. The requirements should be visible to the agent, but the final verdict should not depend on tests it can quietly rewrite. Save this as test_export_time.py beside the candidate implementation when evaluating it:

import unittest
from datetime import datetime
from export_time import format_export_time


class ExportTimeTests(unittest.TestCase):
    def test_converts_across_date_boundary(self):
        value = datetime.fromisoformat("2026-11-08T00:15:00+05:30")
        self.assertEqual(
            format_export_time(value), "2026-11-07T18:45:00Z"
        )

    def test_preserves_microseconds(self):
        value = datetime.fromisoformat("2026-11-08T10:00:00.123456-04:00")
        self.assertEqual(
            format_export_time(value), "2026-11-08T14:00:00.123456Z"
        )

    def test_rejects_naive_datetime(self):
        with self.assertRaises(ValueError):
            format_export_time(datetime(2026, 11, 8, 10, 0))


if __name__ == "__main__":
    unittest.main()

Run python -m unittest -v test_export_time.py. The standard-library test runner reports individual failures, so a precision regression stays visible even if conversion works.

These example checks were executed against the old function and a reference implementation. The old function failed the two changed-output checks and passed naive-time rejection. The reference passed all three. This validates the teaching fixture, not the performance of any coding model. A full evaluation should add more offsets, UTC input and boundary cases before drawing conclusions.

Score the revision separately from the regression

Record two results per task: whether the changed requirement passes, and whether the unchanged requirements still pass. Count a task as fully successful only when both do. Also record model version, harness version, effort setting, tool permissions, elapsed time, actual cost and any human intervention.

For example, suppose an agent passes the revision checks on eight of ten tasks but preserves the invariants on only six of those eight. Its complete success count is six, not eight. That is an illustrative calculation, not a measured model result.

Repeat runs where the decision matters and report the denominator. Ten deliberately chosen tasks can uncover a costly failure mode, but they cannot establish that one model is universally better. Keep the same budgets and tool access across candidates, and inspect failures before attributing every difference to the model.

Choose around the failures your team cannot absorb

The MaxxBench repository provides a route into the published evaluation. A small internal suite serves a different purpose: it lets you test the contracts your team actually changes, such as pagination rules, cancellation behavior or timestamp serialization.

I would not buy a coding workflow on its best score alone. If an agent repeatedly preserves an obsolete rule after a clear revision, that creates review work precisely where developers expect the change to be small. Add that failure to the acceptance suite before giving the agent a larger job; otherwise the extra autonomy mostly increases the distance between the mistake and the person who finds it.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn product behavior, evaluations and engineering decisions into clear stories developers can use.

Book a 30-minute call.