JevBench won't crown its top scorer — it's just Jev wrapped in a price tag

JevBench ranks typed decision models, and by raw score its top entry should be classifier.dev, which posts an 83.6 versus the 74.4 of Jev 1.13.0 sitting at #1. Instead the maintainers relegate it to an honorable mention. Their reasoning is that classifier.dev isn’t a distinct model at all: its fast tier is Jev itself — the API even returns the version string ‘jev-1.13.0’, the exact build the benchmark already measures directly. Scoring the wrapper as a separate competitor would amount to ranking Jev against Jev, so since v1.2.4 it is listed but not ranked.

What classifier.dev genuinely adds is packaging and an optional orchestration layer. Its ‘smart’ tier is a confidence cascade: answers Jev returns below 0.7 confidence get re-asked to a reasoning model (gemini-3.8-flash), which is escalation rather than best-of-N, self-consistency, or a committee. That smart tier was never benchmarked here — only the fast tier ran. Notably the fast tier isn’t a clean pass-through either; its scores diverge from raw Jev on the test sets (97.3% vs 94.5% on the judge tier, 70.5% vs 74.1% on the hard tier), a gap classifier.dev attributes to batching a thousand inputs per request.

The headline price of ~$0.0033 per 1,000 decisions is a best-case estimate assuming full use of the $20/month Pro plan; lighter use costs roughly ten times more per decision, and the benchmark run itself used the free tier at no cost. The maintainers are careful to note they can’t verify classifier.dev’s underlying cost basis. The broader point is one of benchmark hygiene: classifier.dev is a legitimate, open-source, free-to-use product by Michael Ryaboy, but a repackaging of the model under test shouldn’t be allowed to displace that same model at the top of the leaderboard.