An Empirical Study of Harness Design for Coding Agents
Three September 17 preprints take on agent scaffolding, overclaiming and troubleshooting, while small repos ship checks that an agent's claims match its diff.
TL;DR
- Three September 17 arXiv preprints study coding-agent harness design, overclaiming propensity in frontier agents, and stateful retrieval for troubleshooting agents.
- Tooling tracks the same concern: Emetgate is a verification gate between an LLM and a source tree, NiceTryGPT targets CTF shortcuts, and Bailout bills itself as a coding agent meant to be deleted.
- A separate post argues misaligned citations open a path to LLM-generated scientific fraud.
"An Empirical Study of Harness Design for Coding Agents" treats the scaffolding around a model, not the weights, as the variable under study. It appeared in the same September 17 arXiv batch as "Quantifying Overclaiming Propensity in Frontier LLM Agents," which frames overclaiming as something to measure directly in agent behavior. [1] [2]
A third preprint, RAFT, describes a stateful retrieval-augmented framework for troubleshooting agents, adding persistent state to retrieval. The three titles point at operational reliability rather than raw capability. [3]
Small repositories are shipping the same idea as tooling. Emetgate is described as a verification gate between an LLM and your source tree; NiceTryGPT targets shortcuts in capture-the-flag work; Bailout is framed as a coding agent meant to be deleted; is-gpt-nerfed watches for silent model changes; and i-dont-believe-you is a skill meant to stop an agent claiming success when the diff says otherwise. [4] [5] [6] [7] [8]
Evaluation integrity is part of the argument. A September 20 post, "Misaligned citations and LLM-generated scientific fraud," ties citation drift to fabricated research, a risk that applies directly to any agent whose outputs are not checked against sources. [9]
Why it matters
Coding agents are being deployed on the strength of benchmark scores. If the harness and the claim-checking are what determine reliability, then evaluation and verification research is the more actionable half of the field.
Editor's note
The arXiv items include titles and authors but not abstracts in the collection, so each paper's scope is described from its title and no results are claimed.