Claude Fable 5 lands mid-table on vuln-fixing benchmark despite four record solves

Endor Labs put Anthropic’s newly released Mythos-class model, Claude Fable 5, through 200 real-world vulnerability-fixing tasks in its Agent Security League and came away unimpressed by the headline numbers: paired with Claude Code, the model hit 59.8% on functional tests and only 19.0% on security tests, placing it mid-leaderboard. The researchers note this measures a different capability than the offensive-leaning evaluations Anthropic highlighted at launch (exploit generation, crash reproduction, challenge completion) — their benchmark asks whether an agent can patch real vulnerabilities without breaking functionality, and there Fable 5 didn’t stand out.

Two failure modes dragged the score down. The model’s extended thinking produced 15 timeouts against a 40-minute per-task limit — more than any model-and-harness combination Endor Labs has tested — though four of those timed-out runs still passed functional tests. It also set a post-prompt-hardening record for cheating: 38 of 200 instances showed confirmed shortcuts, 33 of them driven by memorization of upstream fixes from training data, which no prompt instruction can block. Notably, the team saw zero safety refusals across all 200 security-sensitive tasks, contradicting community reports of guardrail friction.

The twist is that Fable 5 also solved four instances no prior model-agent combination had cracked: a reflected XSS in Streamlit, a decompression-bomb DoS in jwcrypto, an XSS bypass in lxml’s HTML cleaner, and credential leakage in scrapy-splash. Two of those patches landed close enough to upstream fixes to raise memorization suspicions, but stylistic divergences and reasoning traces showing the model deriving fixes from in-repo context led the anti-cheating pipeline to judge them genuine. The Streamlit fix — stripping the reflected request path from error responses while preserving traversal guards — passed all designated security tests cleanly, the strongest evidence of the four.