Do Guardrails Make AI Coding Agents Better? What 270 Runs Showed

An experiment of 270 unattended coding-agent runs across three models and seven tasks found that instruction files and pre-commit gates did not raise first-pass rates, with 267 of 270 runs passing hidden tests at first "done." Guardrails showed up instead in what runs left behind: on four later tasks, 9 of 60 gated runs left a defect versus 15 of 60 ungated, a gap too small to call. The author notes limits including five runs per cell, one small codebase, and self-written tasks and harness, and publishes the run records and a reproduction script in a companion repository.