Skip to content

What it does ​

Every feature of driftbench, grouped by what you do with it. The README shows each group with an example you can copy.

<a href="../.github/readme/areas-light.png"> <picture> <source media="(prefers-color-scheme: dark)" srcset="../.github/readme/areas-dark.png"> <img alt="Four things driftbench tests, on four cards. Test conversations: replay a real conversation, let a simulated user talk, or throw in a surprise at any step. Rules and judges: rules check what code can check, and a judge decides the rest once it has proved it can. Going off track: where the agent first went wrong, and how many steps it took to get back. Numbers you can trust: every rate with its range, and a yes or no your CI can act on." src="../.github/readme/areas-light.png" width="100%"> </picture> </a>

Write and generate tests

  • Specs: one conversation or task as blocks. Each carries a starting point (Given), a user turn (When) and an expected result (Then).
  • Generated tests: code writes the blocks from an approved map of situations, and reports the pairs of values they cover.
  • Tests from production: draft specs from OpenTelemetry, Nexus or Claude Code traces, or from a spreadsheet.

Run agents

Check and judge

  • 37 machine checks: tool calls and their arguments, text, language, personal data, end state, files, spreadsheets, cost.
  • Judges: a model judges what code cannot, must quote its evidence, and a panel votes when the first judge is unsure.
  • Hand labels: label blocks in a local viewer and measure each judge against them.
  • Block rules and fault finding: trace a verdict to its rule file and a failure to its fix.

<sub>Fault finding ships as rule files and a runner. driftbench report and driftbench view list the fixes of a campaign from its stored diagnoses (see Fixing the block that failed).</sub>

Measure variance

  • Intervals on every rate, repeats, and the share of runs that pass every time.
  • Where results vary: how much of the spread comes from the agent, the wording, the persona or the judge.
  • Drift and recovery: the first block that went wrong, and how many blocks the agent needs to come back.

Compare runs

Record and replay

Validate the evaluator

  • Planted-truth suites: code builds cases with a known answer and measures what driftbench reports.
  • A validity gate for tests: a test that a planted fault does not fail, or that a constant reply passes, is rejected.

Only here, in full. We read seven other evaluation tools against 30 capabilities. Six are found in full in none of them:

Only here, in fullWhat you getProof
Recovery metricsInject a deviation at a chosen block and count the blocks until the agent is back, the failures after it, and whether its actions stayed saferecovery.test.ts
Drift across turnsThe first block that went wrong, and which of five kinds of drift it wasdrift.test.ts
Judge calibrationA judge owns a criterion only after it separates a pass from a fail 3 times out of 3 and agrees with your hand labelscalibration.test.ts
Intervals and significance testsAn interval on every rate, paired tests between two versions, and a correction across criteriaintervals.test.ts
End state assertionsCheck what the agent actually left behind: the order placed, the file written, the cells of the workbookend_state.test.ts
Multi-language handlingCheck the language of a reply and that it stays stable through the conversationlang.test.ts

<a href="../.github/readme/capability-matrix-light.png"> <picture> <source media="(prefers-color-scheme: dark)" srcset="../.github/readme/capability-matrix-dark.png"> <img alt="Of 30 evaluation capabilities: driftbench 30. In full, in part, not found: deepeval 12, 14, 4. langwatch 12, 11, 7. promptfoo 10, 15, 5. langfuse 10, 7, 13. inspect_ai 9, 14, 7. opik 9, 10, 11. braintrust 2, 11, 17." src="../.github/readme/capability-matrix-light.png" width="100%"> </picture> </a> <p align="center"><sub>Every cell with its source: <a href="comparison.md">comparison page</a>. <a href="comparison/methods.md#the-capability-matrix">How we measured</a>.</sub></p>

How it compares &ZeroWidthSpace;

The capability matrix is above. One page per tool, row by row:

ToolPage
promptfoopromptfoo
InspectInspect
DeepEvalDeepEval
LangfuseLangfuse
LangWatchLangWatch

What driftbench does not do: retrieval metrics for RAG, promptfoo's catalogue of red-team plugins, a hosted platform with a team UI. → how each row was checked

MIT. driftbench is made by Nexus.