# How to Measure AI Coding Productivity Without Counting More Code

A practical measurement plan for separating faster implementation from faster delivery, with a worked example of where the savings disappear.

By Paras Madan | 2026-11-07


To measure AI coding productivity, compare accepted work of similar difficulty and count implementation, review and rework together. A developer finishing a draft faster is useful, but it does not tell you whether the team ships reliable changes sooner. Keep elapsed delivery time separate from engineering hours because they answer different questions.

The distinction matters in the discussion around [METR’s November 5 assessment of AI R&D at Anthropic](https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/). METR estimated that model capability progress on well-specified tasks had accelerated to roughly twice its historical rate, with substantial uncertainty. It also reported employee estimates of individual speedups reaching 12 times, based on informal polls that METR explicitly treated as loose evidence. Those are different measurements, not competing estimates of the same thing.

A product team should not copy either number into its staffing plan. Anthropic’s research environment and internal models are not a controlled trial of your repository, and much of the assessment’s informative evidence was redacted. The useful question is narrower: where does AI remove work from your delivery process, and where does it move that work to someone else?

## Choose the unit of finished work first

Start with a recurring task that has a clear acceptance test: fixing a reproducible bug, adding a documented endpoint or migrating one dependency. Avoid combining an unfamiliar architecture change with a routine UI adjustment and calling both one ticket. A ticket count becomes misleading as soon as the size of tickets changes.

Before a task starts, record its category, approximate scope and acceptance criteria. For an endpoint change, acceptance might include the expected response, authorization behavior, regression coverage and a reviewer’s approval. These criteria should apply whether the implementation uses AI or not. If the AI group is allowed to deliver a narrower version, the comparison has already failed.

**Measure the same outcome at the same quality bar.** More generated code, more prompts and more completed agent runs can explain how people work, but none is a substitute for that outcome.

## Keep an effort clock and a delivery clock

The effort clock adds up human time spent understanding the task, preparing context, implementing, checking, reviewing and correcting the change. Include the reviewer’s time, not just the author’s. Keep model and tool charges alongside it rather than pretending they are free.

The delivery clock measures elapsed time from a defined starting point to accepted delivery. This includes waiting for review, CI and deployment. Record those waits separately so that a slow review queue does not get mistaken for slow coding, or a faster CI system for a better model.

This gives you two useful ratios: baseline human hours divided by assisted human hours, and baseline elapsed delivery time divided by assisted elapsed delivery time. They can move in different directions. An agent that runs overnight may reduce human effort without changing a release date. A developer who spends more time preparing a precise task may get an earlier result while saving fewer working hours.

## A worked example of savings that move downstream

Consider this illustrative comparison, not a measured result from a team or a test of a particular tool. Both changes satisfy the same acceptance criteria, and the categories include all recorded human work.

| Human effort | Baseline workflow | AI-assisted workflow |
| --- | --- | --- |
| Understanding and implementation | 6 hours | 2 hours |
| Review and verification | 1 hour | 2 hours |
| Corrections and rework | 1 hour | 2 hours |
| Total | 8 hours | 6 hours |

Looking only at implementation produces a three-times speedup. Counting the whole job produces a 1.33-times speedup and a 25% reduction in human effort. That is still valuable, but it is a much smaller basis for a budget or hiring decision.

If both changes then wait two days for the same release window, customers may see no faster delivery at all. The next useful investment could be better task specifications or a smaller review queue, rather than more agent capacity. Buying additional generation while verification is the constraint can make the queue longer.

## Compare similar tasks without pretending they are identical

For a small pilot, choose a task category and observation period in advance, then allocate comparable tasks between the current workflow and an AI-assisted workflow where that is practical. Spread both conditions across developers so that one experienced engineer does not become the entire AI result. Record the model, tool version and relevant workflow settings, because changing them halfway through changes what you are evaluating.

Do not ask someone to solve the identical issue twice and treat the second attempt as independent. They already know the code and the answer. Instead, use matched task categories, record meaningful differences and show the distribution rather than only an average. A few difficult failures can consume the savings from many easy tasks.

Record skipped tasks and participants too. In its [February 2026 experiment-design update](https://metr.org/blog/2026-02-24-uplift-update/), METR described selection effects when developers avoided work they did not want to attempt without AI, alongside difficulties timing concurrent agents. A pilot that excludes those cases may misrepresent where the tool helps most.

A before-and-after comparison is easier to run, but weaker as causal evidence. Repository familiarity, staffing, scope and CI changes may explain part of the difference. Write those changes next to the result instead of quietly attributing everything to AI.

## Follow the change beyond the merge

Choose a consistent observation period for rollbacks, reopened issues and escaped defects. A change that passes review but creates a production incident has not become efficient because the incident appears in another team’s spreadsheet. Record those costs against the original work when you can establish the connection.

Keep a short failure review too. Was the specification incomplete? Did the agent misunderstand an interface? Did the reviewer miss a generated edge case? This makes the pilot useful even when the headline productivity number is inconclusive, because each failure points to a different intervention.

The result should be a decision about a workflow: which tasks benefit, what supervision they need and what limits wider adoption. If you can only show that code appeared faster, you have measured generation. The business still needs evidence that less work remained after it appeared.

I’m Paras. I work at the intersection of AI, engineering and distribution.

I help technical teams turn engineering decisions and product evidence into clear stories developers and founders can use.

[Book a 30-minute call](https://cal.com/paras-madan/30min)