UQO Working Papers · WP29 · v1.0
Protocol-Guided Autonomous Research with Claude Code
Scientific Verification, Reproducibility, and the Cost of Delegation: A Controlled Study
- Year
- 2026
- First release
- October 2026
- Latest revision
- October 2026
- Publication status
- Working paper
Protocol structure buys auditability, not correctness, for bounded research tasks
Across 288 live Claude Code runs on 24 tasks, verified completion moves from 0.903 to 0.917 between the open instruction and the full method (+0.014, interval [−0.028, +0.056]) while inference cost per run rises two- to three-fold.
Abstract
Instruction files that steer coding agents through research tasks range from one sentence to several thousand words of phases, gates and integrity rules, but whether that structure changes what the agent delivers has not been measured with the agent held fixed. The paper reports 288 live Claude Code runs (362 sessions; client 2.1.314, model claude-opus-5-5) on 24 bounded scientific tasks in six families, three fresh sessions per task under each of four regimes: an open instruction; a structured protocol with phases, ledgers, stopping and recovery rules; the protocol plus a controller-enforced executable gate; and the same plus a separate review-and-repair session. Executors ran in bare mode on isolated hosts; deterministic hidden oracles, validated on seeded artefacts, scored every run on a separate host; design, tasks and analysis were frozen with hashes before the confirmatory runs.
Intervention-free verified completion was 0.903 under the open instruction and 0.917 under the full method; the paired task-level difference is +0.014 (task-cluster bootstrap 95% interval [−0.028, +0.056]), the mechanism contrasts are equally null, and critical integrity violations were rare in every arm. The protocol, gate and reviewer bought full compliance with the deliverable contract at two to three times the inference cost per run. Every non-success of the frozen scoring was, on inspection, a defect of the oracles rather than of the executors (an undefined feasibility label, a generator artefact that twelve sessions correctly excluded, a rounding tolerance, a regular-expression false positive); under a corrected rubric every arm completes every task and the null is unchanged. Arm D's review-repair loop, though available, was never triggered.
Interruption-and-resumption, stale-input, duplicate-job and missing-fixture experiments, a length-matched control and three held-out end-to-end mini-projects complete the evidence, together with an instrumented account of how the paper was itself produced under a mandate of the same kind. For bounded tasks with known answers, the marginal value of protocol structure lies in auditability, not correctness, and the scarce resource is a trustworthy oracle.
Key findings
288
live Claude Code runs, 362 sessions, 24 tasks
0.903 vs 0.917
verified completion, open instruction vs full method
+0.014
D−A contrast, 95% interval [−0.028, +0.056]
0.24 → 0.64 USD
mean inference cost per run, arm A to D
0
arm C or D deliverables failing the contract check
4
oracle defects behind every frozen non-success
Scientific contributions
- First controlled comparison of instruction regimes
- Four arms (open, protocol, protocol plus executable gate, plus review-and-repair) with the agent, tools, inputs and budget held fixed, oracles protected from the executor, and the design preregistered and frozen with hashes before the confirmatory runs.
- A precise null on correctness
- Completion 0.903 (A) against 0.917 (D), paired difference +0.014 with a task-cluster bootstrap interval of [−0.028, +0.056]; the B−A, C−B and D−C contrasts are equally flat and integrity violations are rare in every arm.
- Structure buys contract compliance at a price
- No arm C or D deliverable failed the post-hoc contract check, against 7 in arm A and 9 in arm B; mean inference cost per run rose from 0.24 USD to 0.42, 0.45 and 0.64 USD and median wall time from 68 to 176 s, a flat quality–cost frontier.
- The failures were the oracles'
- Every non-success of the frozen scoring traced to an evaluator defect: an undefined feasibility label, a generator artefact that twelve independent sessions correctly excluded, a rounding tolerance and a regular-expression false positive; under a corrected rubric every arm completes every task.
- A delegation-contract reference implementation
- Controller with durable SQLite state, isolated bare-mode executors, oracle-free runtime gate, separate reviewer, deterministic hidden evaluators and authentic capture, with every claimed invariant enforced by code; 24 confirmatory and 7 development tasks plus three transfer mini-projects.
- A reflexive, instrumented account
- The paper documents its own production from a 6,900-word mandate by a Claude Code builder session, kept separate from the confirmatory evidence, with an intervention ledger recording only the mandate and the launch instruction.
Methodology
Methodology (5)
Four-arm randomized design
Three fresh sessions per task per arm; primary endpoint intervention-free verified completion within a shared resource envelope; primary contrast D−A with task-cluster bootstrap intervals and permutation tests over the 24 task clusters.
Isolation and hidden oracles
Executors run on hosts holding no copy of the reference answers; evaluators are deterministic Python programs on a separate host, never an LLM judging itself, and the executor cannot read them.
Controller-enforced executable gate
Arm C adds a runtime gate that checks the deliverable contract and reproducibility and returns failures to the session; arm D adds a separate review-and-repair session of the same model.
Fault and recovery experiments
Interruption-and-resumption, stale-input, duplicate-job and missing-fixture faults, a length-matched unstructured instruction as control, and three held-out end-to-end mini-projects.
Full resource accounting
Cost and time read from logged inference usage and wall clock, never self-report; cost per verified completion rises from 0.27 to 0.70 USD across arms.
Research data
- Benchmark of bounded scientific tasks
- 24 confirmatory tasks in six families (data audit, estimation and inference, forecasting under information constraints, source-grounded synthesis, theory and numerical validation, integrated mini-projects) plus 7 development tasks, with provenance for 14 real-data and 5 source-document tasks and hidden deterministic oracles validated on seeded correct and incorrect artefacts.
- Run records
- 288 confirmatory runs (362 sessions) of Claude Code 2.1.314 with claude-opus-5-5 in bare mode on cluster nodes, under a 4.00 USD / 1,500 s envelope; all splits together (smoke, pilot, main, rerun, transfer, recovery, ablation) total 171.76 USD of inference, with every number regenerated from runs/state.sqlite.
- Corpus of researcher-written mandates
- Eleven hashed source mandates coded to motivate the protocol arms, together with a verified literature matrix and BibTeX.
Reproducibility
- Every number in the paper regenerated from runs/state.sqlite by analysis/ scripts (build_analysis_dataset, primary_analysis, robustness, figures, tables) via make analyze && make paper
- Preregistration, deliverable contract, treatments, evaluation plan and amendments ledger in protocol/; design, tasks and analysis frozen with hashes before the confirmatory runs
- Infrastructure test suite (state machine, gate, evaluator, bundle hygiene) runs without provider access; evaluator validation on seeded artefacts in evaluators/run_validations.py
- Full LaTeX source of the 110-page paper in the UQO template; the 6,900-word research mandate preserved verbatim with its hash
Research artifacts
Version history
| Version | Date | Description | File |
|---|---|---|---|
| v1.0current | October 2026 | Current manuscript as published on www.spboucher.ai (revised edition after the 2026-10-17 scientific review for papers 1–17). | PDF · 110 p. |