Skip to content

MM-Eval: driftbench against every peer ​

Model: claude-haiku-5-5 for every contestant. Unit: item, macro over five subsets. driftbench and promptfoo rows are the recorded ledger cells; peer rows are new runs on the same sealed items. Rows marked "pointwise grader, higher score wins" score each reply alone with that tool's pointwise grader; the tool has no pairwise grader.

Note: a few items stay undecided after three passes because the model refused them (an acceptable-use refusal on biology content). They count as wrong for the contestant, as the recorded driftbench cell does.

Cohort: primary ​

ContestantGraderScoredriftbench minus thisdriftbench wins / lossesExact pHolm pMissingGrader errorsUSD
driftbenchrecorded cell88.86% (n=591)0
promptfoorecorded cell86.45% (n=591)+2.41 pts71 / 350.0006060.006060
deepevalArenaGEval (DeepEval 4.2.8)84.09% (n=591)+4.77 pts83 / 306.31e-071.01e-05301.0437
inspectmodel_graded_qa (inspect_ai 0.3.278), pointwise grader, higher score wins75.36% (n=591)+13.51 pts202 / 474.48e-248.96e-23700.8229
langfuseAnswer Critic (Langfuse 64ba74f9), pointwise grader, higher score wins73.45% (n=591)+15.41 pts218 / 432.34e-294.92e-28620.4622
langwatchComparison, swap and reconcile (LangWatch e683dd9e)83.87% (n=591)+4.99 pts47 / 155.78e-050.000693720.6708

Cohort: all ​

ContestantGraderScoredriftbench minus thisdriftbench wins / lossesExact pHolm pMissingGrader errorsUSD
driftbenchrecorded cell87.67% (n=600)0
promptfoorecorded cell85.50% (n=600)+2.17 pts71 / 370.001380
deepevalArenaGEval (DeepEval 4.2.8)83.92% (n=600)+3.75 pts83 / 361.96e-05301.0437
inspectmodel_graded_qa (inspect_ai 0.3.278), pointwise grader, higher score wins74.67% (n=600)+13.00 pts202 / 507.68e-23700.8229
langfuseAnswer Critic (Langfuse 64ba74f9), pointwise grader, higher score wins72.75% (n=600)+14.92 pts218 / 465.94e-28620.4622
langwatchComparison, swap and reconcile (LangWatch e683dd9e)83.00% (n=600)+4.67 pts47 / 170.000227720.6708

MIT. driftbench is made by Nexus.