Skip to content

Verify: corrected v5 confirmation and historical comparison ​

Primary v5 result ​

The verify candidate's primary result uses the 536 confirmation items whose normalized question text appears in neither practice nor DEV. The original 544-item split leaked 8 practice questions and reused 32 practice call references. The frozen manifest and decisions are preserved; the full result is a sensitivity analysis. Scores are percent correct.

Verify v5 analysisdriftbench-v5promptfooDeepEvalInspectLangfuseLangWatch
Primary, 536 items without practice/DEV question overlap78.7376.8760.2676.4975.7575.19
Sensitivity, original 544 items including overlap78.8677.2160.8576.8475.9275.18
LJB TEST, 221 items76.0275.5758.8273.3075.1173.30

Each row gives v5 one significant win over DeepEval and four ties after exact paired tests with Holm over five peers. These are separate analyses from the historical 25-comparison table below. The exclusion is a post-review correction using recorded decisions, with no new model calls. Full results and exclusions.

Historical LongJudgeBench comparison (PR 351) ​

Model: claude-haiku-5-5 for every contestant. Unit: item. driftbench and promptfoo rows are the recorded ledger cells; peer rows are new runs on the same sealed items. Rows marked "pointwise grader, higher score wins" score each reply alone with that tool's pointwise grader; the tool has no pairwise grader.

ContestantGraderScoredriftbench minus thisdriftbench wins / lossesExact pHolm pMissingGrader errorsUSD
driftbenchrecorded cell74.21% (164/221)0
promptfoorecorded cell71.95% (159/221)+2.26 pts19 / 140.48710
deepevalGEval, threshold 0.5 (DeepEval 4.2.8)58.82% (130/221)+15.38 pts44 / 103.39e-065.08e-050300.2641
inspectmodel_graded_qa (inspect_ai 0.3.278)73.30% (162/221)+0.90 pts17 / 150.861000.3456
langfuseAnswer Critic (Langfuse 64ba74f9)75.11% (166/221)-0.90 pts13 / 150.8511000.2605
langwatchLLM-as-a-Judge Boolean (LangWatch e683dd9e)73.30% (162/221)+0.90 pts16 / 140.8561000.2842

MIT. driftbench is made by Nexus.