A coding agent is a language model plus the program around it, the harness: what the model is asked to do first, when its work is checked, and what happens after a failed check. This project writes the harness as a graph, compares hand-designed graphs on bug-fixing tasks, and is now searching a space of 3,600 graphs for better ones.
A node is one unit of execution: a stretch of agent conversation, or a deterministic program such as a test run. It reads the shared state, writes to it, and returns an outcome label such as done, pass or fail.
An edge is a control rule: node X returned outcome r, so run node Y next. Edges carry no data. Everything a later node needs is in the shared state: the conversation, the repository working copy, and the latest feedback.
The engine has six node types. Before a graph runs, static checks confirm that every outcome of every node has an edge, every node can be reached, loops are bounded by visit limits, and each node's inputs are written on every path that leads to it.
| Node type | What it does | Outcomes it can return |
|---|---|---|
agent | The model issues shell commands one at a time until it says it is done. Its role sets the instruction: solve, reproduce the bug, fix after reproducing, or verify before submitting. | done · step limit · context too long · stalled |
plan | The model reads the file list and writes a short plan. | next |
check | A program runs the project's tests on the current patch. | pass · fail |
reflect | The model reads the failure output and writes down the cause. | next |
reset | A program restores the code and clears the conversation, keeping only the note on the cause. | next |
compact | The model replaces a long conversation with a summary. | next |
Seventeen graphs were written by hand for short bug-fixing tasks from SWE-smith (real Python repositories with an injected bug) and run with two models. After a pilot on 27 development tasks, nine graphs were run on 81 confirmation tasks in two conditions: with the failing tests visible in the repository, so a check node can run them, and with those tests hidden. Two more graphs were added for the comparison in Table 1.
With tests visible, checking the patch and returning failures raises the repair rate.From 75% to 98% for gpt-6.1-terra (+23.5 points, 95% interval 14.8 to 32.5; better on 22 tasks, worse on 0) and from 63% to 77% for gpt-6-luna (+13.6 points, 7.0 to 20.6; better on 19, worse on 2). The price is 12% and 27% more model calls.
With tests hidden, no graph showed an improvement over the plain loop.The check node then has nothing informative to run. Across the eight other graphs measured in this condition, differences to the plain loop ranged from −3.9 to +2.6 points, every interval including zero.
Enforcing the check does more than suggesting it.Asking the model to review its own work before submitting changed the repair rate by +1.3 points for gpt-6.1-terra and +9.0 for gpt-6-luna, less than the enforced check for both (Figure 3). Handing the model the same test checker as a tool it may call helped the larger model; making the check a node the patch must pass added a further +7.4 points for gpt-6.1-terra and +7.4 for gpt-6-luna (Table 1).
check nodegraph does not| Tests visible, 81 tasks, two runs per task | gpt-6.1-terra | gpt-6-luna |
|---|---|---|
| Plain loop | 74.1% | 65.4% |
| Plain loop + checker offered as a tool, with an instruction to use it | 90.1% | 68.5% |
Checker offered as a tool + enforced check node with retries | 97.5% | 75.9% |
| Enforced minus offered, 95% interval | +7.4 [1.9, 13.6] | +7.4 [2.5, 13.0] |
Table 1. Offering the checker versus enforcing it. The design of this comparison was fixed before these runs.
The hand-designed graphs differ in a small number of decisions. Writing those decisions down as nine design variables turns the collection into a space: choose a value for each variable and a compiler assembles the graph. Set the variables below and the graph is redrawn.
done, next, pass).fail from the test check (variables ⑥ and ⑦ decide what lies on the way back and how often it may be taken), context too long (variable ⑧), and stalled (variable ⑨).Of the 6,480 raw settings, 3,600 graphs remain. All are structurally different from one another, all pass the static checks, and they range from 1 to 11 nodes. The 17 hand-designed graphs are 17 points of this space. The compiler is harness_graph/grammar.py; the drawing on this page is a JavaScript port of it, checked against the Python version on every setting.
With a space of graphs, harness design becomes a search problem: given a fixed number of agent runs, which graph should be tried on which task next, and which graph should be returned at the end? The returned graph is then scored on tasks the search never saw.
Search. Five procedures share one interface and one episode budget: uniform allocation, successive halving, Thompson sampling, Thompson sampling with a linear model over the design variables, and local search that changes one variable at a time, either at random or chosen from the way the last runs failed.
Selection before the run. A second question is whether different tasks should get different graphs. Three policies are compared: one fixed graph, a rule on whether executable tests are available, and a model on features of the task text. All are fitted on search tasks, frozen, and scored on other tasks.
| Part | Status |
|---|---|
| Execution engine, six node types, static checks | built, tested |
| Grammar of 3,600 graphs and its compiler | built, tested |
| Five search procedures; offline replay on stored runs | built, tested |
| Three selection policies; fit, freeze, score on other tasks | built, tested |
| Task splits: 188 for search, 56 for selection, 232 held out for confirmation (120 of them from three repositories used nowhere else) | built |
| Table of 32 graphs × 188 search tasks on an open-weight model (Qwen3.8-27B) | measured |
| Search over the whole grammar with live agent runs | running |
| Selection of one searched graph and confirmation on the 232 held-out tasks | next |
| Graphs with several agents (separate conversations, parallel branches) | not started |
First measurement. On the open-weight model with tests visible, check-and-retry again leads the plain loop: 82.4% against 73.3% on 187 tasks, one run each. The best of 30 graphs drawn at random from the grammar reached 82.9%. Whether search returns a graph that beats the hand-designed one on held-out tasks is what the running experiment tests.