Opus 5 tops SlopCodeBench at 24% — but every model failed to finish a challenge

SlopCodeBench, a long-horizon coding benchmark out of GOrlanski’s lab at UW Madison, tries to fix what its author finds broken about most coding evals: they hand the model the entire problem up front. Instead, each challenge is broken into sequential checkpoints that reveal requirements over time, forcing a model to evolve a codebase the way real engineering work actually unfolds. The benchmark is far from saturated — in the original paper, GPT-5.4 and Opus 4.6 managed only 11% and 17% strict pass rates, where a ‘strict pass’ means every new test plus every inherited regression test stays green.

Running a 17-checkpoint subset (three problems spanning easy, medium, and hard) across Opus 4.8, Sonnet 5, and Opus 5 in parallel with fresh context per checkpoint, the author found Opus 5 came out on top with a 24% strict pass rate (4 of 17) — but three of those four wins were the opening checkpoints of a single ‘easy’ problem. Opus 4.8 and Sonnet 5 each passed just one. Critically, no model reached the final checkpoint of any challenge cleanly, including the easy one: once a defect enters the code, it carries forward and blocks every subsequent checkpoint. Opus 5 simply failed slightly less badly than the rest.

The more interesting signal is code quality drift. Every model grew markedly more verbose and complex as challenges progressed — Opus 5 produced roughly five times as many functions as Opus 4.8 (though much of that was extra tests, with real production volume closer to 1.8x). Between 89% and 98% of lines tripped at least one of the benchmark’s deterministic ‘slop’ rules, and the share of lines flagged as too verbose climbed from about 65% early to 80% by the eighth checkpoint. The author reads this as concrete evidence for a long-held hunch: on realistically-shaped, one-issue-at-a-time software work, today’s frontier models can’t be trusted to run unsupervised without human steering. He’s skeptical the 41 quality metrics reliably capture ‘maintainability,’ and notes the detectors are Python-only, limiting their use on real TypeScript codebases.