Research project · in progress · updated 18 November 2026

Adaptive Graph Harness Optimization for LLM Agents

Yiwen Lu · University of Notre Dame

A coding agent is a language model plus the program around it, the harness: what the model is asked to do first, when its work is checked, and what happens after a failed check. This project writes the harness as a graph, compares hand-designed graphs on bug-fixing tasks, and is now searching a space of 3,600 graphs for better ones.

75% → 98%
bugs repaired by gpt-6.1-terra when the harness checks the patch and sends failures back, tests visible
63% → 77%
the same change of harness for the smaller gpt-6-luna
3,600
distinct single-agent harness graphs generated from nine design variables, all passing the static checks

1 · A harness as a graph

A node is one unit of execution: a stretch of agent conversation, or a deterministic program such as a test run. It reads the shared state, writes to it, and returns an outcome label such as done, pass or fail.

An edge is a control rule: node X returned outcome r, so run node Y next. Edges carry no data. Everything a later node needs is in the shared state: the conversation, the repository working copy, and the latest feedback.

A · Plain loop: 1 node, 1 edge Edit code solve Submit done B · Check and retry: 2 nodes, 3 edges Edit code solve Run tests check Submit done pass fail (up to 2)
Figure 1. The two graphs behind the headline numbers. In A the agent edits until it says it is done. B adds one program node and one edge that leads back: a failed test run returns the agent to editing with the failure output in the conversation. solid: the node calls the modeldashed: a program, no model call

The engine has six node types. Before a graph runs, static checks confirm that every outcome of every node has an edge, every node can be reached, loops are bounded by visit limits, and each node's inputs are written on every path that leads to it.

Node typeWhat it doesOutcomes it can return
agentThe model issues shell commands one at a time until it says it is done. Its role sets the instruction: solve, reproduce the bug, fix after reproducing, or verify before submitting.done · step limit · context too long · stalled
planThe model reads the file list and writes a short plan.next
checkA program runs the project's tests on the current patch.pass · fail
reflectThe model reads the failure output and writes down the cause.next
resetA program restores the code and clears the conversation, keeping only the note on the cause.next
compactThe model replaces a long conversation with a summary.next

2 · What the hand-designed graphs showed

Seventeen graphs were written by hand for short bug-fixing tasks from SWE-smith (real Python repositories with an injected bug) and run with two models. After a pilot on 27 development tasks, nine graphs were run on 81 confirmation tasks in two conditions: with the failing tests visible in the repository, so a check node can run them, and with those tests hidden. Two more graphs were added for the comparison in Table 1.

Plain loop (graph A)Check and retry (graph B)
0%25%50%75%100%difference [95% interval]share of bugs repairedTests visible to the agentgpt-6.1-terragpt-6.1-terra, tests visible to the agent: plain loop 74.9% of 81 tasksgpt-6.1-terra, tests visible to the agent: check and retry 98.4% of 81 tasks75%98%+23.5 [14.8, 32.5]gpt-6-lunagpt-6-luna, tests visible to the agent: plain loop 63.4% of 81 tasksgpt-6-luna, tests visible to the agent: check and retry 77.0% of 81 tasks63%77%+13.6 [7.0, 20.6]Tests hiddengpt-6.1-terragpt-6.1-terra, tests hidden: plain loop 49.4% of 77 tasksgpt-6.1-terra, tests hidden: check and retry 46.8% of 77 tasks47%49%−2.6 [−6.5, 0.0]gpt-6-lunagpt-6-luna, tests hidden: plain loop 39.7% of 78 tasksgpt-6-luna, tests hidden: check and retry 39.7% of 78 tasks40%40%0.0 [−5.1, 5.1]
Figure 2. Share of bugs repaired on the 81 confirmation tasks. Right column: check-and-retry minus plain loop in percentage points, with a 95% interval from a paired bootstrap over tasks. Visible-test rows pool three runs per task; hidden-test rows are one run on the 77 and 78 tasks completed by every graph.

With tests visible, checking the patch and returning failures raises the repair rate.From 75% to 98% for gpt-6.1-terra (+23.5 points, 95% interval 14.8 to 32.5; better on 22 tasks, worse on 0) and from 63% to 77% for gpt-6-luna (+13.6 points, 7.0 to 20.6; better on 19, worse on 2). The price is 12% and 27% more model calls.

With tests hidden, no graph showed an improvement over the plain loop.The check node then has nothing informative to run. Across the eight other graphs measured in this condition, differences to the plain loop ranged from −3.9 to +2.6 points, every interval including zero.

Enforcing the check does more than suggesting it.Asking the model to review its own work before submitting changed the repair rate by +1.3 points for gpt-6.1-terra and +9.0 for gpt-6-luna, less than the enforced check for both (Figure 3). Handing the model the same test checker as a tool it may call helped the larger model; making the check a node the patch must pass added a further +7.4 points for gpt-6.1-terra and +7.4 for gpt-6-luna (Table 1).

graph contains an enforced check nodegraph does not
gpt-6.1-terra−100+10+20+30Check, retry (graph B)gpt-6.1-terra: Check, retry (graph B) minus plain loop +20.8 points [11.7, 31.2], 77 tasks+20.8Check, find cause, retrygpt-6.1-terra: Check, find cause, retry minus plain loop +20.8 points [11.7, 29.9], 77 tasks+20.8Check, find cause, start overgpt-6.1-terra: Check, find cause, start over minus plain loop +18.2 points [10.4, 27.3], 77 tasks+18.2Self-reviewgpt-6.1-terra: Self-review minus plain loop +1.3 points [0.0, 3.9], 77 tasksSelf-review after summarizinggpt-6.1-terra: Self-review after summarizing minus plain loop +1.3 points [0.0, 3.9], 77 tasksPlan firstgpt-6.1-terra: Plan first minus plain loop 0.0 points [−3.9, 3.9], 77 tasksReproduce the bug firstgpt-6.1-terra: Reproduce the bug first minus plain loop −1.3 points [−6.5, 3.9], 77 tasksSummarize long conversationsgpt-6.1-terra: Summarize long conversations minus plain loop −1.3 points [−5.2, 2.6], 77 tasksgpt-6-luna−100+10+20+30gpt-6-luna: Check, retry (graph B) minus plain loop +19.2 points [10.3, 28.2], 78 tasks+19.2gpt-6-luna: Check, find cause, retry minus plain loop +12.8 points [5.1, 20.5], 78 tasks+12.8gpt-6-luna: Check, find cause, start over minus plain loop +7.7 points [1.3, 14.1], 78 tasks+7.7gpt-6-luna: Self-review minus plain loop +9.0 points [3.8, 15.4], 78 tasksgpt-6-luna: Self-review after summarizing minus plain loop +6.4 points [0.0, 12.8], 78 tasksgpt-6-luna: Plan first minus plain loop +1.3 points [−5.1, 7.7], 78 tasksgpt-6-luna: Reproduce the bug first minus plain loop +2.6 points [−5.1, 10.3], 78 tasksgpt-6-luna: Summarize long conversations minus plain loop +3.8 points [−3.8, 12.8], 78 taskspercentage points relative to the plain loop
Figure 3. Tests visible: repair rate of eight hand-designed graphs minus the plain loop, in percentage points, with 95% paired-bootstrap intervals (one run per task). Only the three graphs with an enforced check move clearly away from zero for both models.
Tests visible, 81 tasks, two runs per taskgpt-6.1-terragpt-6-luna
Plain loop74.1%65.4%
Plain loop + checker offered as a tool, with an instruction to use it90.1%68.5%
Checker offered as a tool + enforced check node with retries97.5%75.9%
Enforced minus offered, 95% interval+7.4 [1.9, 13.6]+7.4 [2.5, 13.0]

Table 1. Offering the checker versus enforcing it. The design of this comparison was fixed before these runs.

3 · From 17 hand-drawn graphs to a space of 3,600

The hand-designed graphs differ in a small number of decisions. Writing those decisions down as nine design variables turns the collection into a space: choose a value for each variable and a compiler assembles the graph. Set the variables below and the graph is redrawn.

How a setting becomes a graph

  1. Main line. The compiler lays out the nodes that are switched on in a fixed order: plan, reproduce, edit, summarize, self-review, run tests, submit. Consecutive nodes are joined by the edge for the normal outcome (done, next, pass).
  2. Side loops. Three outcomes lead off the main line and back to the editing node: fail from the test check (variables ⑥ and ⑦ decide what lies on the way back and how often it may be taken), context too long (variable ⑧), and stalled (variable ⑨).
  3. Limits. Every node on a loop gets a visit limit computed from the retry settings, so each graph is guaranteed to end. The static checks of section 1 are then run on the result.
  4. Counting. Variables ⑥ and ⑦ do nothing unless ⑤ is on, so those settings are merged:
①②③④: 2×2×2×3 = 24    ⑤⑥⑦: 1 without the check + 3×3 with it = 10    ⑧⑨: 3×5 = 15    24 × 10 × 15 = 3,600

Of the 6,480 raw settings, 3,600 graphs remain. All are structurally different from one another, all pass the static checks, and they range from 1 to 11 nodes. The 17 hand-designed graphs are 17 points of this space. The compiler is harness_graph/grammar.py; the drawing on this page is a JavaScript port of it, checked against the Python version on every setting.

4 · In progress: search and task-conditioned selection

With a space of graphs, harness design becomes a search problem: given a fixed number of agent runs, which graph should be tried on which task next, and which graph should be returned at the end? The returned graph is then scored on tasks the search never saw.

Candidate graphs from the 3,600 Search procedure picks the next graph and task Run one episode this graph on this bug Return one graph scored on held-out tasks graph, task repaired or not, and how the run ended budget used up
Figure 4. The search loop. Each box is a program; each arrow says what is passed. The feedback arrow carries the grade and the way the run ended (wrong patch submitted, out of steps, no edit made), which one of the procedures uses to choose its next change.

Search. Five procedures share one interface and one episode budget: uniform allocation, successive halving, Thompson sampling, Thompson sampling with a linear model over the design variables, and local search that changes one variable at a time, either at random or chosen from the way the last runs failed.

Selection before the run. A second question is whether different tasks should get different graphs. Three policies are compared: one fixed graph, a rule on whether executable tests are available, and a model on features of the task text. All are fitted on search tasks, frozen, and scored on other tasks.

PartStatus
Execution engine, six node types, static checksbuilt, tested
Grammar of 3,600 graphs and its compilerbuilt, tested
Five search procedures; offline replay on stored runsbuilt, tested
Three selection policies; fit, freeze, score on other tasksbuilt, tested
Task splits: 188 for search, 56 for selection, 232 held out for confirmation (120 of them from three repositories used nowhere else)built
Table of 32 graphs × 188 search tasks on an open-weight model (Qwen3.8-27B)measured
Search over the whole grammar with live agent runsrunning
Selection of one searched graph and confirmation on the 232 held-out tasksnext
Graphs with several agents (separate conversations, parallel branches)not started

First measurement. On the open-weight model with tests visible, check-and-retry again leads the plain loop: 82.4% against 73.3% on 187 tasks, one run each. The best of 30 graphs drawn at random from the grammar reached 82.9%. Whether search returns a graph that beats the hand-designed one on held-out tasks is what the running experiment tests.

5 · Scope and limits