HarnessDev finds most agent-harness changes fail to generalize
Researchers from ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, M-A-P, and TokenWave.AI introduced HarnessDev, a benchmark that evaluates the runnable software surrounding a language model rather than only its answers. The harness includes execution, tools, context, state, recovery, and verification. Six models started from the same minimal seed, which scored 0 without a loop, planner, verifier, retry, or stopping rule, and built harnesses for five benchmarks covering four domains.
In the Creation stage, Opus 4.8 achieved the highest average Self-Eval score, 67.8, compared with 86.2 for a human-engineered reference. The largest weakness was search: GPT-5.5 reached 52.6 on BrowseComp against a 92.2 reference. Models matched or exceeded references in writing and machine-learning experimentation, while code and search remained weaker. Harness size and self-test counts showed little relationship with quality, and substantial amounts of generated state, memory, and writing code were never executed.
The Evolution stage produced 73 official versions and 64 adjacent changes. All five self-runtime lineages improved on held-out tasks, but gains averaged only 3.11 points and were not consistent. Feedback and held-out results moved in the same direction 34 times out of 64, while only two changes showed clear positive evidence. Harness performance also depended heavily on the executor: Opus 4.8 fell from 69.3 to 33.0 on SWE-bench Pro when its executor was replaced with Gemini 3.1 Pro.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.