Latest in AI
CWE-bench v2: What the AI Security Benchmark Scores Actually Measure
The new public audit-and-patch tasks make security agents easier to compare, provided you keep the task sets, retries and costs separate.
Collinear AI’s November 3 CWE-bench v2 announcement introduces 100 public tasks testing whether AI agents can find and fix security weaknesses without breaking working software. The release makes defensive capability easier to inspect, but its different leaderboards answer different questions.
CWE stands for Common Weakness Enumeration: a way of naming categories of software weaknesses. The public set covers 91 such categories. Breadth is useful here because an agent that handles one familiar bug pattern can still miss a different failure in the same codebase.
What changed in CWE-bench v2?
The project’s version comparison shows v1 had 120 held-out tasks covering 73 weakness types. V2 combines 100 public tasks with 150 held-out tasks, covering 185 distinct types across the two sets. The public and held-out category counts overlap, so they should not simply be added together.
V2 also moves to programmatic verifiers curated using judge panels and manual review. The authors say the checks test whether the security weakness is closed while legitimate behavior still works, rather than demanding one particular patch structure.
That is an important constraint. A patch that disables an entire feature might stop the bad input, but it has not necessarily repaired the product. A useful defensive evaluation must reject that shortcut.
Gemini leads the public set, Claude leads the combined set
The public-set leaderboard, updated November 3, reports the following selected results. These are Collinear’s evaluations of model-and-harness combinations, not an independent comparison of model weights alone.
| Model | Harness | Public pass@1 | Public pass@4 | Average cost per rollout |
|---|---|---|---|---|
| Gemini 4 Argon | Antigravity SDK | 67.8% | 81.0% | $3.92 |
| Claude Opus 5.5 | Claude Code | 67.3% | 78.0% | $1.83 |
| GPT-6 Astra | Codex | 63.8% | 70.0% | $3.86 |
| DeepSeek V4.1 Flash (1017) | OpenCode | 57.0% | 72.0% | $0.04 |
The main website also shows a combined 250-task table, dated November 2. There, Claude Opus 5.5 leads at 67.1% pass@1, followed by Muse Wonder at 66.1% and Gemini 4 Argon at 65.0%.
A model’s rank only makes sense alongside the task set and harness. Mixing the public winner with the combined score would create a comparison neither leaderboard makes. Different harnesses also mean these results reflect tool orchestration and runtime behavior as well as the underlying model.
Pass@1 and pass@4 are different measurements
The public leaderboard defines pass@1 as the mean reward across 400 rollouts: four attempts on each of 100 tasks. Pass@4 is the share of tasks solved at least once across those attempts. The latter is not the average of four scores.
Each public rollout has a one-hour agent limit. The dataset card defines a successful reward as passing every security and legitimate-behavior check. If the verifier cannot evaluate a trial, it is void rather than automatically scored as a model failure.
The retry distinction matters operationally. A higher pass@4 shows that additional attempts can produce a correct patch. It does not show that a production system, without the benchmark’s verifier, will reliably identify which candidate patch is correct.
The cost column is narrower than a deployment budget
For Claude, GPT and Qwen, the public leaderboard uses harness-recorded costs. For other entries, it calculates trajectory tokens against OpenRouter list prices as of November 3, including cached-input discounts.
DeepSeek’s reported $0.04 per rollout is therefore useful evidence for an inexpensive first pass under this setup. It is not the total cost of securing a repository. Running environments, reviewing patches, investigating misses and validating production behavior still require resources.
Compare the cost of an accepted, reviewed fix, including failed attempts and escalation, before deciding that a cheaper rollout produces a cheaper security workflow.
Public tasks still require access approval
The dataset card is readable publicly, but downloading its files requires accepting Hugging Face access conditions. The tasks use Harbor environments and executable verifiers. The reference solutions are included for package verification and must stay hidden from an evaluated agent.
Programmatic grading makes a result easier to reproduce; it does not prove a patched system has no other weaknesses. Public tasks can also become familiar to future models, which is one reason the separate held-out set matters.
The practical value of this release is a more inspectable way to test defensive agents. For teams choosing one, the next useful evidence is whether it catches their own recurring failure classes and produces patches their reviewers can safely accept.
I’m Paras. I work at the intersection of AI, engineering and distribution.
I help technical teams explain research, benchmarks and product launches in language developers can use without losing the important caveats.