Skip to content

Public judge benchmarks ​

The benchmarks page shows one chart per benchmark: driftbench beside every competitor on one model. This page holds the rest: one chart and one table per benchmark with every model that was run, and a "not run" where a contestant was not. Bars run from the highest score to the lowest, on one scale of 0 to 100. A refused item counts as wrong, and the table says how many. The sets, the rules and the limits of each result are in how the README numbers were measured. A correction across the whole family of cells is pending, so no cell is called a win or a loss.

DeepEval, Inspect, Langfuse and LangWatch were run on Claude Haiku 5.5 only. promptfoo and driftbench were also run on GPT-6 Luna and Gemini 3.5 Flash-Lite. Paired tests, wins, losses and spend per benchmark are in the all-competitors tables.

JudgeBench ​

Three models, 270 sealed pairs each.

JudgeBench: one panel per model, driftbench beside every recorded competitor, highest to lowest. The numbers are in the table below.

ModelSetdriftbenchpromptfooDeepEvalInspectLangfuseLangWatch
Claude Haiku 5.5270 pairs88.983.086.380.074.883.7
GPT-6 Luna270 pairs87.084.8not runnot runnot runnot run
Gemini 3.5 Flash-Lite270 pairs79.679.6not runnot runnot runnot run

LongJudgeBench ​

Verify is run on GPT-6 Luna for promptfoo and driftbench, and on Claude Haiku 5.5 for all contestants. Writing preference has 173 pairs per model, and a rule that picks the longer reply is drawn as a baseline.

LongJudgeBench verify: one panel per model, driftbench beside every recorded competitor, highest to lowest. The numbers are in the table below.

ModelSetdriftbenchpromptfooDeepEvalInspectLangfuseLangWatchalways fail
Claude Haiku 5.5221 items74.271.958.873.375.173.378.7
GPT-6 Luna221 items74.772.4not runnot runnot runnot run78.7
Gemini 3.5 Flash-Litenot runnot runnot runnot runnot runnot runnot run78.7

LongJudgeBench writing preference: one panel per model, driftbench beside every recorded competitor, highest to lowest. The numbers are in the table below.

ModelSetdriftbenchpromptfooDeepEvalInspectLangfuseLangWatchlonger reply
Claude Haiku 5.5173 pairs61.356.958.411.614.552.667.6
GPT-6 Luna173 pairs53.856.9not runnot runnot runnot run67.6
Gemini 3.5 Flash-Lite173 pairs59.060.4not runnot runnot runnot run67.6

MM-Eval ​

Up to 600 sealed pairs per model. Claude Haiku 5.5 scores 591 pairs; the nine missing are safety items.

MM-Eval: one panel per model, driftbench beside every recorded competitor, highest to lowest. The numbers are in the table below.

ModelSetdriftbenchpromptfooDeepEvalInspectLangfuseLangWatch
GPT-6 Luna600 pairs88.189.0not runnot runnot runnot run
Claude Haiku 5.5591 pairs88.986.584.1 (3 refused)75.4 (7 refused)73.5 (6 refused)83.9 (7 refused)
Gemini 3.5 Flash-Lite600 pairs81.483.8not runnot runnot runnot run

RubricBench ​

Three models, 790 pairs each.

RubricBench: one panel per model, driftbench beside every recorded competitor, highest to lowest. The numbers are in the table below.

ModelSetdriftbenchpromptfooDeepEvalInspectLangfuseLangWatch
Claude Haiku 5.5790 pairs82.775.677.6 (1 refused)26.625.1 (1 refused)74.9
GPT-6 Luna790 pairs82.381.5not runnot runnot runnot run
Gemini 3.5 Flash-Lite790 pairs80.880.8not runnot runnot runnot run

RubricEval ​

Data not public. No chart is planned.

MIT. driftbench is made by Nexus.