Latest in AI
DAYJOB Tax Benchmark: Results, Public Samples and What the Scores Mean
Surge AI tests complete tax assignments, and the low pass rates need a more careful reading than a model ranking alone provides.
Surge AI released DAYJOB: Tax on October 27, a benchmark for agents completing professional tax assignments. Its leading reported result is a 10.8% pass rate for Claude Sonnet 5.5. The useful question is what counts as passing, because this evaluation measures much more than getting a calculation right.
What DAYJOB Tax evaluates
The release announcement describes 50 assignments built around realistic professional work. Agents must inspect records, resolve inconsistencies and produce a finished deliverable, rather than answer an isolated tax question. The collection spans 1,789 files and eight areas of practice.
Surge says tax specialists created the tasks and reviewed them in three layers. One important failure pattern is answering the client's stated question while missing a consequential issue in the supporting records. That is a useful distinction for enterprise agents: completing the requested document and understanding the actual problem are separate capabilities.
DAYJOB Tax results and the pass threshold
The published leaderboard reports these selected results:
| Model and reported configuration | Pass rate |
|---|---|
| Claude Sonnet 5.5, Adaptive/Max | 10.8% |
| Claude Fable 5.1, Adaptive/Max | 9.6% |
| Claude Opus 5.5, Adaptive/Max | 9.2% |
| Muse Wonder 1.0, Max reasoning | 6% |
| GPT-6 Astra, Max reasoning | 4.8% |
These are pass@1 results with a 95% rubric threshold. An attempt must satisfy at least 95% of its task's criteria to pass. A 10.8% pass rate is not 10.8% accuracy across every calculation or question. It is the proportion of attempts clearing a demanding completion standard.
The distinction matters when comparing this with a multiple-choice exam or a benchmark that awards partial credit. A system can get much of an assignment right and still fail because the remaining error is important.
What developers can actually download
The Hugging Face dataset card makes an important availability distinction: only two sample tasks are public. Access to the full 50-task evaluation requires contacting Surge. Scores on the samples should not be presented as full-benchmark results.
Tasks use Harbor format and a Linux working environment. The card says browsing and package downloads are blocked during execution, with network access restricted to allowed model API hosts. It also specifies five attempts per task for comparisons with the published configuration. The materials include synthetic documents based on professional records, not simply a collection of live client files.
Those conditions belong alongside the scores. A different harness, access to online research or different tools would be a different experiment.
What this changes for enterprise evaluations
These are Surge's reported results, not an independent replication. A small public sample also limits what outsiders can inspect about the entire task set. The leaderboard should not be treated as a certification that any model can independently handle tax work.
For an AI product team, the more useful design lesson is to evaluate the handoff. Does the agent surface unresolved assumptions before producing a polished final document? Does the reviewer see the missing evidence, or only a confident answer?
An evaluation that records those failures can guide a safer workflow even before the model gets better. The next improvement worth watching is whether agents become better at knowing when the assignment is not ready to finish.
I’m Paras. I work at the intersection of AI, engineering and distribution.
I help technical teams turn AI research and product updates into clear stories that developers and founders can use.