Agent4Sci: What Scientific AI Agents Gain From Better Skills

Researchers affiliated with Anthropic and its Fellows Program introduced Agent4Sci in a paper submitted on October 20, 2026. The scientific ML testbed examines whether coding agents improve existing models, and whether those gains come from research insight or routine engineering. Its clearest practical result concerns how agents are instructed.

What Agent4Sci measures

The Agent4Sci repository packages 20 benchmarks, 25 tasks and 11 scientific domains. An agent receives a paper, codebase, data and a target metric, then tries to improve the starting model within a compute budget.

The runner separates optimization, evaluation and auditing. Training and validation data are available while the agent works; held-out test data is staged only for evaluation. A separate auditor reviews the resulting trajectory. The code uses Apache 2.0, while datasets and underlying benchmarks retain their own terms.

For a team evaluating research agents, that separation is useful. A higher score becomes more credible when the system proposing changes does not also control the final test. It also makes the submitted artifacts worth inspecting, rather than treating a confident final message as the result.

What the benchmark results actually say

The paper tested Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4 through Codex, and Gemini 3.1 Pro. Each run received eight H200 GPUs and 12 hours. Most gains came from familiar engineering changes rather than domain-specific insight.

Adding skills changed the reported mean relative improvement over reproduced baselines:

Agent	Default setup	With skills
Claude Opus 4.6	10.4%	20.1%
Codex with GPT-5.4	4.4%	8.1%

These are relative improvements across benchmark metrics, not task success rates or percentage-point increases in accuracy.

There is an important timing caveat. In his author comment, Ming Zhong says the project was completed by May 2026 for NeurIPS. This is newly published research using earlier models, not a comparison of October's latest Claude, GPT and Gemini releases.

What the skills change in the workflow

The repository's skills ask agents to ground ideas in literature, manage experiments more deliberately, and periodically reconsider their approach. Claude receives an enforced checklist through a tool-call hook; Codex can invoke its checklist on demand. That implementation difference matters when trying to reproduce the setup.

The practical question is whether your agent spends its budget exploring useful alternatives or repeatedly polishing the first plausible approach. In an engineering workflow, those can look similar from the outside: both produce code, logs and plenty of activity. The distinction becomes clearer when you inspect which hypotheses were tested and why the next experiment was chosen.

How to try Agent4Sci, and what remains unproven

The runner requires Docker, NVIDIA Container Toolkit and at least one GPU. Its dry-run option previews commands without API keys. Real runs need model-provider credentials and access to the benchmark data. Those requirements are separate from the much larger hardware budget used in the paper.

The paper warns about run-to-run variance, possible skill overfitting and the absence of a direct human-researcher baseline. Its numbers should not be treated as a reliable agent ranking or evidence that scientific researchers have been replaced.

For teams building long-running agents, the next experiment is a controlled comparison on their own workload: same model, same budget, different research guidance. Track final quality alongside resource use and the diversity of attempted ideas. A better agent workflow should change the decisions being made, not simply generate a longer record of activity.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help technical founders turn AI research and engineering updates into clear stories that developers can understand and share.

Book a 30-minute call.