Evaluating AI scientists · Research project · 2026

DryRun: Measuring How Much AI-Scientist Benchmark Credit Needs No Experiment

Yiwen Lu · University of Notre Dame

Benchmarks for "AI scientists" ask an agent to run an experiment from a real paper, then grade the conclusion it writes. On EXP-Bench, conclusions written by models that ran nothing were graded correct on 67% to 88% of the tasks, depending on the model and the judge. On 16 tasks, an agent that actually ran the experiments earned less credit than the same model answering with no tools at all.

The problem, with one real example

An AI-scientist agent is a language model that does research steps on its own: it reads a research question, edits code, runs experiments and reports what it found. EXP-Bench tests this with 461 tasks taken from 51 NeurIPS and ICLR 2024 papers. Each task gives the agent a research question, instructions and the paper's code with key parts removed. The agent should rebuild and run the experiment and write a conclusion. An LLM judge, a second language model with a fixed grading prompt, then compares that conclusion with the gold conclusion written from the paper and answers "correct" or "incorrect".

Here is one task, from a paper on generating molecular-dynamics trajectories, and three answers that ran nothing:

Research question
Does replacing the time attention mechanism with a non-causal Hyena operator in the SiT architecture allow the model to generate extremely long, consistent trajectories that capture dynamics over multiple timescales?
A fixed template, no model, no experiment
Yes: replacing the time attention mechanism with a non-causal Hyena operator in the SiT architecture allow the model to generate extremely long, consistent trajectories …
both judges: correct
The same template, flipped
No: it is not the case that replacing the time attention mechanism …
both judges: incorrect
gpt-oss-120b, shown only the question and instructions, unable to run code
… In our experiments the Hyena-SiT model reduced the RMS error of torsional autocorrelation functions from 0.12 (attention) to 0.04 (Hyena) … These improvements are statistically significant (p < 0.01). …
both judges: correct

The highlighted numbers are invented; no experiment was run. Both answers that agree with the paper were graded correct, and the one that disagrees was not. A correct grade here tells you the answer points the same way as the paper, not that anyone ran anything. DryRun measures how much of the benchmark's conclusion credit can be earned this way, and what happens when agents can, or must, run the experiment.

What the score is meant to reward Task question, instructions, paper's code (masked) Run the experiment rebuild code, train, measure read the results Conclusion a few sentences of text LLM judge compares with the paper's gold conclusion Credit correct / incorrect What we sent through the same judge ① Prompt-only model sees the question, no code, cannot run anything 461 tasks ② Template, no model "Yes" + question restated, and similar fixed sentences 461 tasks ③ Agent with tools gpt-oss-120b can run commands in the repository 18 tasks ④ Strong agent, must run Claude Code, told to run it; we check that it really did 16 tasks Orange: answers that by design ran nothing. White: agents that could run the experiment. All four kinds of answer are graded by the benchmark's own judge prompt, used word for word.
Top: the pipeline whose last step, the conclusion score, we audit. Bottom: the four kinds of answer we fed into that last step. If the score tracked experiments, ① and ② should earn little credit and ④ should earn the most.

Words used on this page

Gold conclusionThe reference answer stored with each task, written from the source paper and its code.
CreditThe share of tasks whose conclusion the judge marks "correct". EXP-Bench calls this the conclusion score.
JudgesMain judge: gpt-oss-120b. Second judge: Llama-3.3-70B. Check: OpenAI o3. All use EXP-Bench's released grading prompt word for word.
Prompt-only modelA model that answers from the task text alone, with no code and no way to run anything. It is a baseline, not an agent.
TemplateA fixed sentence built from the question by a short program, with no model and no knowledge of the gold answer. For a yes/no question it writes "Yes:" and then the question turned into a statement.
Log monitorA separate EXP-Bench check that reads the agent's command log and flags conclusions that were not produced by an experiment.

What we found

67–88%
of the 461 tasks credited to models that ran nothing and saw only the question and instructions, depending on the model and judge.
56% vs 2%
Credit for "yes" plus the question restated, versus the same with "no", on the 273 yes/no questions. No model involved.
4 vs 12
Tasks credited (of 16) when Claude Code was told to run the experiment, versus the same model answering with no tools.
Finding 1 · The floor

A credited conclusion does not require the experiment.

Two open models that saw only the question and the agent instructions, and could run nothing, were credited on 70.5% (gpt-oss-120b) and 66.6% (Qwen3-32B) of all 461 tasks by the main judge. The other judges were more generous: 88.3% and 84.8% from Llama-3.3-70B, 72.2% and 72.7% from o3. A template with no model at all earned 50.8% from the main judge and 67.9% from o3. The judges are not broken: they credit the task's own gold conclusion (100% and 91.5%) and never credit another paper's. EXP-Bench also multiplies the conclusion grade by a design score to filter answers without a valid design, but the template still keeps 45.1% under that combined score, because its design field, the instructions copied word for word, gets an average design score of 89.4 out of 100.

Answer (nothing was run in any row)TasksMain judgeLlama judgeo3
Checks on the judges
Another paper's gold conclusion4610.00.0–
The task's own gold conclusion461100.091.5–
Model sees the question only
gpt-oss-120b46145.179.6–
Qwen3-32B46138.070.9–
Model sees the question and the agent instructions
gpt-oss-120b46170.588.372.2
Qwen3-32B46166.684.872.7
… plus the source paper
gpt-oss-120b45181.494.2–
Qwen3-32B45172.789.6–
No model
Template for every task46150.874.267.9
"Yes" + the question restated27355.779.574.4
"No" + the question restated2731.81.52.2

Conclusion credit (%). "Template for every task" uses the "yes" sentence on yes/no questions, "The implementation works as specified and produces the requested result" on implementation tasks, and "the method under study outperforms the alternatives" on the rest. o3 was run on the main answer sets only (–: not run).

Finding 2 · Why it works

Most gold conclusions say "yes" to the question, and the grading rubric accepts answers that do the same.

273 of the 461 tasks are yes/no questions. Two labeller models read the gold conclusion as "yes" on 84% and 93% of them. That is why answering "yes" with the question restated earns 55.7%, while answering "no" the same way earns 1.8%. The judges are not simply accepting anything: when we built answers from the gold conclusion and changed one property, they rejected reversed and null results and mostly rejected answers with claims or numbers removed. But the main judge credited 68.7% of gold answers padded with invented numbers. To test whether these "yes" answers were judge mistakes, the author labelled 112 hard cases against the rubric without seeing the verdicts, and accepted all 15 generic template answers that a judge had credited. So this credit is consistent with the rubric itself, not only with judge error.

Answer built from the gold conclusionTasksMain judgeLlama judge
Gold conclusion, unchanged461100.091.5
Direction reversed3771.10.3
Turned into a null result3775.02.4
Only the first claim kept2709.322.6
Gold numbers removed17014.128.8
Invented numbers added9968.739.4

Conclusion credit (%) when one property of the gold conclusion is changed.

Finding 3 · An agent with tools

Having execution tools did not mean running the experiment.

We picked 18 tasks from two papers whose code runs on one GPU (CycleNet, a time-series forecaster, and BaDExpert, a backdoor defence) and ran gpt-oss-120b as an agent in mini-swe-agent, a small open-source agent loop, with 60 minutes and 100 model calls per task. In 22 of its 36 episodes with execution it executed no code, and in 30 it never had a clean code run lasting 10 seconds. Some answers reported numbers from random draws or from an echo statement. The main judge credited it on 4 and 7 of the 18 tasks in the two execution conditions, and on 6 when it could only read and write files, against 10 for the prompt-only model. EXP-Bench's log monitor flagged every one of those credited conclusions as not produced by an experiment. In these runs the monitor, not the conclusion score, kept credit away from them.

When we told the agent outright to run the experiment and report only what the runs produced, its logs showed it running the repository's code on the task's data in only 2 of 36 episodes, both on one task and for one epoch. The judge credited both, on a task where the prompt-only answer was not credited, so the score can reward a real run. How the score treats real runs in general needs an agent that runs experiments reliably, which is Finding 4.

Finding 4 · A strong agent that runs

When the experiments were actually run, the score did not reward it.

We repeated the check on 16 of those tasks with Claude Code (Claude Sonnet 5.5), once with the benchmark's released prompt (60 minutes) and once with a prompt that requires running the experiment (90 minutes). An independent LLM adjudicator read every log and answer and decided whether the experiment was completed (the required metric measured for every required method and dataset) and whether the conclusion was supported by the agent's own results. With the must-run prompt, Claude Code completed the experiment on 11 of 16 tasks, and on 7 its conclusion was also supported by its runs. The earlier gpt-oss-120b agent completed 1 or 2. Yet the main judge credited the must-run answers on only 4 of 16 tasks, while the same Claude model answering from the prompt with no tools was credited on 12. Of the 11 episodes, across both prompts, that completed the experiment and reported a supported conclusion, the main judge credited 4.

Answer (same 16 tasks)CompletedCompleted + supportedMain judgeLlama judgeo3
Claude Code, must-run prompt117487
Claude Code, released prompt94576
Same model, prompt only, no tools––121313
Template, no model––8––

Number of tasks out of 16. One episode per task and prompt. –: nothing was run, or that judge was not applied. The must-run answers versus the prompt-only answers on the main judge: exact McNemar test p = 0.039 (released prompt: p = 0.092).

What this tells us about agents

An agent's conclusion and its experiment can come apart, in both directions. A conclusion can be credited when no experiment was run, because the gold answer is usually the hypothesis the question already states. And a conclusion drawn from an experiment that really ran can go uncredited. A score built only from conclusions therefore measures how well an answer matches the paper's stated result, which is not the same as doing the research.

For people who build these benchmarks

How we ran it

Run the numbers yourself

The repository holds the code for every experiment and the per-task answers and verdicts behind the tables above. One script recomputes the headline numbers from those files on a laptop, no GPU needed.

git clone https://github.com/EvenEureka/DryRun.git
cd DryRun
pip install -r requirements.txt
python reproduce/headline_numbers.py

Rerunning the judges and agents needs GPUs to serve the open models and an API key for o3; see the README.

Details and limits

These notes are numbered by finding. They say where each result holds and what it does not show.

Finding 1 · The floor

  • We measure one component, the conclusion score. The full EXP-Bench pipeline also runs a log monitor before scoring and checks implementation and execution; we used the monitor only in the agent checks and did not rerun the full pipeline.
  • Every rate depends on the judge. EXP-Bench's released o3-mini judge could not be run; o3 is from the same model series but differs in model and serving. The prompt text is the released one, but the two open judges received it as a user message, while the released code (and our o3 call) sends it as a system message.
  • We had read all 461 tasks and gold conclusions before these analyses, so every result is exploratory.
  • EXP-Bench names its combined conclusion-and-design metric without giving a formula; the 45.1% uses the per-task product, our reading of it.
  • Credit stays high on the tasks where an auditor model finds the gold conclusion not implied by the prompt (65.0% and 61.4%), and replacing method, model and dataset names with placeholders changes it by only −2.6 points. These checks do not rule out that the models know the papers.

Finding 2 · Why it works

  • The "yes" shares (84% and 93%) come from model labels, not human labels.
  • Rewriting the questions to remove leading wording lowered prompt-only credit by only about 8 points (from 61.8% to 53.6% and from 59.3% to 50.6%), but the rewrites are not validated: four auditor models agree on which prompts lean one way only at chance level (Fleiss' κ = 0.05). So we cannot say how much credit would remain if all direction cues were removed, or where the remaining predictability comes from.
  • The human labels come from one person, the author, who knew the hypotheses. The 112 items were chosen to over-represent hard cases, so they show that rubric-consistent template credit exists, not how often it occurs. On them, the main judge agreed with the author on 85 items (κ = 0.53) and the Llama judge on 68 (κ = 0.14).

Finding 3 · Agent with tools

  • 18 tasks from two papers and one agent configuration. 83 of its 391 model calls broke the agent's action format, and 5 episodes failed because the default Python lacked the repository's packages.
  • The log monitor's clean result holds for the three standard conditions. When the agent was told to run the experiment, the monitor did not flag 3 of the 7 episodes the main judge credited.
  • The prompt-only model and the agent differ in budget and sampling. With o3 and the Llama judge, execution was also not distinguishable from no execution (all paired p ≥ 0.45); on 18 tasks this does not show that they are equal.
  • In the must-run condition an automatic rule counted 7 of 36 episodes as grounded in the agent's own runs, and 11 episodes ended with no answer; reading the logs, only 2 of the 7 ran the repository's code on the task's data.

Finding 4 · Strong agent

  • 16 tasks from the same two papers, one episode per task and prompt. For 10 of the 16 tasks the repository copy lacks methods the agent has to implement itself.
  • "Completed" and "supported" are an LLM adjudicator's classifications. In the author's audit of 8 random episodes, the two agreed on "completed" in 7 and on "supported" in 4; in all five disagreements the adjudicator was the stricter one.
  • The gpt-oss-120b comparison (completed on 1 or 2 of 16) comes from the earlier runs, re-classified by the same adjudicator; budgets were not matched.
  • We did not establish why runs that were completed and supported lost credit. On the development task, the gold conclusion contained a claim that the run could not test.
  • EXP-Bench's log monitor flagged none of the 32 episodes. It reads only the command log, so it cannot catch numbers that appear only in the written answer.
  • The compute nodes had internet access; 4 must-run and 2 released-prompt episodes show a download step in their logs (for example download=True or pip install).