Latest in AI
FrontierCode 2.0: Coding Benchmark Results and What Mergeability Measures
Cognition's new benchmark tests whether an agent notices the requirements a developer did not spell out, which changes how its scores should be read.
Cognition introduced FrontierCode 2.0 on October 27, a coding benchmark built around whether a maintainer would accept an agent's changes. The new suite has 132 tasks and puts more weight on requirements implied by the codebase. Its results are useful for teams deciding how much review autonomous coding still needs.
What FrontierCode 2.0 measures
The live leaderboard describes a mix of tests, rubrics and other checks covering correctness, test quality, scope and repository conventions. Maintainers create the tasks, and Cognition reviews them. Internet research is allowed, but consulting sources that contain the solution can invalidate a run.
A weighted score is calculated only after blocking requirements are cleared. Failing a blocker produces zero. The score is not simply the percentage of tasks completed, and it should not be read as a percentage of an engineer's job that can be automated.
That distinction matters when a solution appears to work. A patch may fix the reported symptom while changing a public interface that another application depends on. The engineering team still has to absorb the cost of finding that problem, reviewing another attempt and checking the final change.
FrontierCode 2.0 benchmark results
Cognition's release comparison reports these scores at each model's best reasoning effort:
| Model | FrontierCode 1.1 | FrontierCode 2.0 |
|---|---|---|
| Claude Opus 5.5 | 54.6% | 45.5% |
| GPT-6 Astra | 53.3% | 42.8% |
| Grok 4.7 | 47.6% | 28.6% |
| Gemini 4 Argon | 47.5% | 26.3% |
| SWE-2 | 50.0% | 23.9% |
| DeepSeek V4.1 Flash | 43.9% | 19.3% |
| Kimi K3 | 44.2% | 17.7% |
These are vendor-reported results on a changed evaluation. Lower numbers across versions do not establish that the models became worse. Cognition expanded the work involved and increased the demands placed on a passing solution.
The reported cost measure is average API spend per rollout at list prices, including public discounts. It excludes the broader economics of adopting a coding agent, such as review time and integration work. A low-cost attempt can still become expensive if someone has to reconstruct what it changed.
Why thoroughness changes the result
Compared with 1.1, reference solutions in 2.0 modify 67% more files and 89% more lines on average. Cognition also loosened some scope restrictions while strengthening checks for implied correctness requirements.
The useful idea here is to test what happens when the prompt leaves something unsaid. In the leaderboard's Mock Service Worker example, an agent resets a generator but fails to run its cleanup logic. The visible bug is addressed while a related expectation remains broken.
This is a better discussion to have inside a team than asking whether an agent can produce a patch at all. Reviewers usually need to know whether the patch fits the surrounding system, including behaviour that was never copied into the issue description.
How to use the benchmark without overreading it
Cognition's original methodology distinguishes passing all blockers from accumulating a rubric score. It also explains why maintainer judgment requires calibration and manual review. Those design choices make the evaluation informative, but they do not remove judgment from the process.
Cognition builds coding agents itself, so this remains a vendor-run evaluation rather than an independent procurement verdict. The launch materials provide methodology and examples, but a public leaderboard is not the same as a complete, independently reproducible test environment.
For an engineering team, the practical next step is to use a similar review standard on a handful of its own tasks. Pick changes involving existing interfaces, cleanup behaviour and several related code paths. Measure how many corrections a reviewer needs, along with the model bill.
The model that looks cheapest on a chart may still consume the most engineering attention. What matters for autonomous coding is how much trustworthy work reaches the end of the review process.
I'm Paras. I work at the intersection of AI, engineering and distribution.
I help technical teams explain their engineering work through clear articles, launch stories and developer-focused content.