Benchmarks for "AI scientists" ask an agent to run an experiment from a real paper, then grade the conclusion it writes. On EXP-Bench, conclusions written by models that ran nothing were graded correct on 67% to 88% of the tasks, depending on the model and the judge. On 16 tasks, an agent that actually ran the experiments earned less credit than the same model answering with no tools at all.
An AI-scientist agent is a language model that does research steps on its own: it reads a research question, edits code, runs experiments and reports what it found. EXP-Bench tests this with 461 tasks taken from 51 NeurIPS and ICLR 2024 papers. Each task gives the agent a research question, instructions and the paper's code with key parts removed. The agent should rebuild and run the experiment and write a conclusion. An LLM judge, a second language model with a fixed grading prompt, then compares that conclusion with the gold conclusion written from the paper and answers "correct" or "incorrect".
Here is one task, from a paper on generating molecular-dynamics trajectories, and three answers that ran nothing:
The highlighted numbers are invented; no experiment was run. Both answers that agree with the paper were graded correct, and the one that disagrees was not. A correct grade here tells you the answer points the same way as the paper, not that anyone ran anything. DryRun measures how much of the benchmark's conclusion credit can be earned this way, and what happens when agents can, or must, run the experiment.
Two open models that saw only the question and the agent instructions, and could run nothing, were credited on 70.5% (gpt-oss-120b) and 66.6% (Qwen3-32B) of all 461 tasks by the main judge. The other judges were more generous: 88.3% and 84.8% from Llama-3.3-70B, 72.2% and 72.7% from o3. A template with no model at all earned 50.8% from the main judge and 67.9% from o3. The judges are not broken: they credit the task's own gold conclusion (100% and 91.5%) and never credit another paper's. EXP-Bench also multiplies the conclusion grade by a design score to filter answers without a valid design, but the template still keeps 45.1% under that combined score, because its design field, the instructions copied word for word, gets an average design score of 89.4 out of 100.
| Answer (nothing was run in any row) | Tasks | Main judge | Llama judge | o3 |
|---|---|---|---|---|
| Checks on the judges | ||||
| Another paper's gold conclusion | 461 | 0.0 | 0.0 | – |
| The task's own gold conclusion | 461 | 100.0 | 91.5 | – |
| Model sees the question only | ||||
| gpt-oss-120b | 461 | 45.1 | 79.6 | – |
| Qwen3-32B | 461 | 38.0 | 70.9 | – |
| Model sees the question and the agent instructions | ||||
| gpt-oss-120b | 461 | 70.5 | 88.3 | 72.2 |
| Qwen3-32B | 461 | 66.6 | 84.8 | 72.7 |
| … plus the source paper | ||||
| gpt-oss-120b | 451 | 81.4 | 94.2 | – |
| Qwen3-32B | 451 | 72.7 | 89.6 | – |
| No model | ||||
| Template for every task | 461 | 50.8 | 74.2 | 67.9 |
| "Yes" + the question restated | 273 | 55.7 | 79.5 | 74.4 |
| "No" + the question restated | 273 | 1.8 | 1.5 | 2.2 |
Conclusion credit (%). "Template for every task" uses the "yes" sentence on yes/no questions, "The implementation works as specified and produces the requested result" on implementation tasks, and "the method under study outperforms the alternatives" on the rest. o3 was run on the main answer sets only (–: not run).
273 of the 461 tasks are yes/no questions. Two labeller models read the gold conclusion as "yes" on 84% and 93% of them. That is why answering "yes" with the question restated earns 55.7%, while answering "no" the same way earns 1.8%. The judges are not simply accepting anything: when we built answers from the gold conclusion and changed one property, they rejected reversed and null results and mostly rejected answers with claims or numbers removed. But the main judge credited 68.7% of gold answers padded with invented numbers. To test whether these "yes" answers were judge mistakes, the author labelled 112 hard cases against the rubric without seeing the verdicts, and accepted all 15 generic template answers that a judge had credited. So this credit is consistent with the rubric itself, not only with judge error.
| Answer built from the gold conclusion | Tasks | Main judge | Llama judge |
|---|---|---|---|
| Gold conclusion, unchanged | 461 | 100.0 | 91.5 |
| Direction reversed | 377 | 1.1 | 0.3 |
| Turned into a null result | 377 | 5.0 | 2.4 |
| Only the first claim kept | 270 | 9.3 | 22.6 |
| Gold numbers removed | 170 | 14.1 | 28.8 |
| Invented numbers added | 99 | 68.7 | 39.4 |
Conclusion credit (%) when one property of the gold conclusion is changed.
We picked 18 tasks from two papers whose code runs on one GPU (CycleNet, a time-series forecaster, and BaDExpert, a backdoor defence) and ran gpt-oss-120b as an agent in mini-swe-agent, a small open-source agent loop, with 60 minutes and 100 model calls per task. In 22 of its 36 episodes with execution it executed no code, and in 30 it never had a clean code run lasting 10 seconds. Some answers reported numbers from random draws or from an echo statement. The main judge credited it on 4 and 7 of the 18 tasks in the two execution conditions, and on 6 when it could only read and write files, against 10 for the prompt-only model. EXP-Bench's log monitor flagged every one of those credited conclusions as not produced by an experiment. In these runs the monitor, not the conclusion score, kept credit away from them.
When we told the agent outright to run the experiment and report only what the runs produced, its logs showed it running the repository's code on the task's data in only 2 of 36 episodes, both on one task and for one epoch. The judge credited both, on a task where the prompt-only answer was not credited, so the score can reward a real run. How the score treats real runs in general needs an agent that runs experiments reliably, which is Finding 4.
We repeated the check on 16 of those tasks with Claude Code (Claude Sonnet 5.5), once with the benchmark's released prompt (60 minutes) and once with a prompt that requires running the experiment (90 minutes). An independent LLM adjudicator read every log and answer and decided whether the experiment was completed (the required metric measured for every required method and dataset) and whether the conclusion was supported by the agent's own results. With the must-run prompt, Claude Code completed the experiment on 11 of 16 tasks, and on 7 its conclusion was also supported by its runs. The earlier gpt-oss-120b agent completed 1 or 2. Yet the main judge credited the must-run answers on only 4 of 16 tasks, while the same Claude model answering from the prompt with no tools was credited on 12. Of the 11 episodes, across both prompts, that completed the experiment and reported a supported conclusion, the main judge credited 4.
| Answer (same 16 tasks) | Completed | Completed + supported | Main judge | Llama judge | o3 |
|---|---|---|---|---|---|
| Claude Code, must-run prompt | 11 | 7 | 4 | 8 | 7 |
| Claude Code, released prompt | 9 | 4 | 5 | 7 | 6 |
| Same model, prompt only, no tools | – | – | 12 | 13 | 13 |
| Template, no model | – | – | 8 | – | – |
Number of tasks out of 16. One episode per task and prompt. –: nothing was run, or that judge was not applied. The must-run answers versus the prompt-only answers on the main judge: exact McNemar test p = 0.039 (released prompt: p = 0.092).
An agent's conclusion and its experiment can come apart, in both directions. A conclusion can be credited when no experiment was run, because the gold answer is usually the hypothesis the question already states. And a conclusion drawn from an experiment that really ran can go uncredited. A score built only from conclusions therefore measures how well an answer matches the paper's stated result, which is not the same as doing the research.
The repository holds the code for every experiment and the per-task answers and verdicts behind the tables above. One script recomputes the headline numbers from those files on a laptop, no GPU needed.
git clone https://github.com/EvenEureka/DryRun.git
cd DryRun
pip install -r requirements.txt
python reproduce/headline_numbers.py
Rerunning the judges and agents needs GPUs to serve the open models and an API key for o3; see the README.
These notes are numbered by finding. They say where each result holds and what it does not show.
download=True or pip install).