Skip to content

Comparison ​

This page sets driftbench beside seven evaluation tools, capability by capability. Every cell of a reference tool carries one source and the date it was read. driftbench cells say what is built today, link to the page that describes it, and end with a proof: the test that fails when the capability breaks.

Tools compared ​

How to read a cell ​

  • yes: the cited page shows the capability.
  • partial: the cited page shows part of it.
  • not found: the cited page, where the capability would be documented, does not show it. This is not proof that the tool lacks it.
  • not documented in its docs: no page read for the tool documents it.

All pages were read on 2026-10-11. Tools change; a cell is true of the page on that date.

Capabilities ​

#Capabilitylangwatchpromptfoodeepevalinspect_aiopiklangfusebraintrustdriftbench
1Unit of evaluationyes, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11block: built in the spec format, phase 0 (block) Proof: schema.test.ts.
2Test case formatyes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11YAML or JSON spec: built, phase 0 (spec reference) Proof: loader.test.ts.
3Scripted user turnsyes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11blocks hold the user turns, and the scripted driver replays them: built, phase 1 (running) Proof: scripted.test.ts.
4Simulated useryes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11a persona on the model port or a recording, facts given only when asked (in code), a stop token, refused on the agent's or a judge's model family: driftbench simulate, campaign run --driver simulated, built (simulated user) Proof: simulated.test.ts, simulate.test.ts.
5Injected deviation in a conversationyes, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11twelve deviation classes: the spec's own perturbations, deviations drawn by the perturbing driver from an approved red-team pack, and attacks the adaptive attacker picks from the agent's last replies: built (recovery, attacker) Proof: perturb.test.ts, attack.test.ts.
6Start from a prior conversation statepartial, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11given.prefix and teacher-forced runs, every block from its own Given: built, phases 1 and 3 (run modes) Proof: teacher.test.ts.
7Assertions on textpartial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11must, must_not, lang: built, phase 1 (checks); similarity to a reference text with a pluggable embedder, local by default: built (checks); length, word and character bounds on the reply: built (checks) json, the reply is valid JSON with an optional strict schema and path assertions: built (checks); refusal, refusal phrases in seven languages: built (checks). Proof: must.test.ts, similarity.test.ts, length.test.ts, reply_json.test.ts, refusal.test.ts.
8Tool call assertionsyes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11tools, tool_order, skills, guards: built, phase 1; tools_offered, the tools the agent was offered in the block: built (checks); tool_f1, precision, recall and F1 of the calls with argument matching: built (checks). Proof: tools.test.ts, tools_offered.test.ts, tool_f1.test.ts.
9End state assertionsnot found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11end_state, workspace, artifacts: built, phase 1 (checks) Proof: end_state.test.ts, workspace.test.ts, artifacts.test.ts.
10LLM judge designyes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11criteria with pass and fail pictures, a fixture pair per criterion, chat and system 1 judges: built, phase 2 (judges) Proof: criteria.test.ts.
11Judge model choice and ensemblespartial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11judge council, two of three agreeing votes, and a cascade: built, phase 2 (judges) Proof: council.test.ts.
12Judge calibrationnot found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11fixture pairs and kappa against hand labels, which judge owns each criterion: built, phase 2 (judges); hand labels per block and criterion from a local viewer, written to the labels folder calibration reads: built (viewer); real judges measured on the claude command, 2026-10-12: haiku separates 12 of 12 Acme pairs 3 of 3, kappa 0.95 over 40 labels, and a haiku, sonnet and opus council assigns 6 of the 7 criteria it covers, marked single-family (judges) Proof: calibration.test.ts, view.test.ts.
13Turn and conversation metricspartial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11per-block verdicts, min over blocks: built, phase 1 (drift); turn relevancy, a machine floor with a judged criterion: built (checks) Proof: relevancy.test.ts.
14Drift across turnsnot documented in its docs, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11first drift block, drift rate: built, phase 1; the five drift classes, the ledger check and the stage classifier: built, phase 3 (drift); knowledge_retention, a fact given earlier not asked again and used, from the ledger and the trace, with a judge fallback: built (checks) Proof: drift.test.ts, knowledge_retention.test.ts.
15Recovery and robustness metricspartial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11turns to recover, recovery label, collateral, safe action per perturbed run: built, phase 3; their shares across runs, overall and per deviation class, with intervals: built in driftbench report (recovery, scorecards) Proof: recovery.test.ts, perturb.test.ts.
16Repeats, pass@k, pass^knot found, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11pass@k, pass^k, unreliability: built in the statistics library; per golden in the scorecard, with intervals: built, phase 4 (statistics, scorecards) Proof: reliability.test.ts.
17Intervals and significance testsnot found, source, read 2026-10-11not documented in its docs, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11Wilson, clustered bootstrap, McNemar, Wilcoxon, Benjamini-Hochberg: built in the statistics library; on every rate of the scorecard and in driftbench compare: built, phase 4 (statistics, scorecards) Proof: intervals.test.ts, comparison.test.ts.
18Baseline comparison, regression gatepartial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11non-inferiority verdict: built in the statistics library; campaign comparison (driftbench compare), the CI gate (driftbench ci) and a GitHub Action: built, phase 4 (statistics, CLI) Proof: ci.test.ts, compare.test.ts.
19Dataset managementnot found, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11content hash per spec, behaviour inventory and coverage matrices: built, phase 0 (CLI); immutable content-addressed versions of spec sets and prompts with push, list, diff and byte-for-byte restore, recorded by the campaign that ran them: built (datasets) Proof: registry.test.ts, registry.test.ts.
20Production traces into test casespartial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes: driftbench import traces drafts one spec per traced conversation from OpenTelemetry GenAI spans (two SDK shapes), Nexus traces or Claude Code sessions, the observed state as each block's Given, no expect copied, approval required (traces, tests in packages/adapters/test/traces and packages/cli/test/import.test.ts); driftbench monitor --source traces watches the same traces for drift; spec import from a CSV file or a spreadsheet through a column mapping: built (datasets) Proof: import-traces.test.ts, monitor-traces.test.ts.
21Red teamingyes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11twelve deviation classes and a red-team pack of approved phrasings, drawn by seed: built, phase 3 (recovery, CLI); an adaptive attacker that picks each attack from the agent's last replies, seeded and replayable, scored by the recovery record (driftbench attack): built (attacker) Proof: redteam.test.ts, attack.test.ts.
22Caching and determinismyes, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11cache by block and Given, determinism smoke: built, phase 1 (running) Proof: cache.test.ts, smoke.test.ts.
23Parallelism, retries, budget capspartial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11not found, source, read 2026-10-11worker pool and budget stop: built, phase 1 (running); one retry policy for every model call (408, 429, 5xx and network errors, backoff with jitter, Retry-After, set in the spec target and on the command line, each retry recorded): built, release phase (retries) Proof: retry.test.ts, campaign.test.ts.
24Result storagepartial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11SQLite store and JSON export of a campaign: built, phase 1 (running); a local viewer over the store: built (viewer) Proof: store.test.ts, bundle.test.ts.
25Reports and run diffspartial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11HTML scorecard with intervals, paired campaign comparison, per-block drilldown with evidence and judge reasoning, JUnit XML: built, phase 4 (scorecards); the same pages served live from the store, with labelling: built (viewer) Proof: scorecard.test.ts, view.test.ts.
26CI integrationyes, source, read 2026-10-11partial, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11exit codes of the spec and campaign commands: built, phases 0 and 1; the CI gate driftbench ci and a GitHub Action that fails the job unless ci shows parity: built, phase 4 (CLI); a pytest plugin, one test per spec, and a vitest describeSpec, one test per block: built (Python, vitest) Proof: ci.test.ts, test_plugin.py, describe-spec.test.ts.
27Custom checks, judges and adaptersyes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11function, http, nexus and process adapters and custom checks: built, phase 1; OpenAI Agents, Anthropic, Google, LangGraph, CrewAI, Vercel AI SDK, Pydantic AI and LiteLLM adapters, and the Claude Code estate adapter with a live runner: built, 13 adapters in all, each through the same contract suite (adapters) Proof: process.contract.test.ts, custom.test.ts.
28Multi-language handlingnot documented in its docs, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11lang, lang_stable: built, phase 1 (checks) Proof: lang.test.ts, relevancy-languages.test.ts.
29Cost and latency per turnpartial, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11partial, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11not found, source, read 2026-10-11budget per block, cost and latency per segment: built, phase 1 (checks) Proof: budget.test.ts.
30Docs and maintenanceyes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11yes, source, read 2026-10-11partial, source, read 2026-10-11these docs, with links and the generated reference checked in CI: built; a docs-site workflow: built, release phase (index) Proof: docs.test.ts.

Where driftbench is ahead ​

Four places, each with its evidence. What the other tools do comes from the rows above, read on 2026-10-11. The command output below is what each command printed on the bundled examples, with no model call.

The block is the unit. Every exchange is a unit test with its own Given, When and Then. In teacher-forced mode a block runs from its own Given, so it gets a pass rate of its own, free of the blocks before it. The other tools score a scenario run, a test row, a conversation test case, a sample, a dataset item or a trace (row 1). Proof: schema.test.ts, teacher.test.ts.

driftbench checks its own numbers. Validator ladders prove that a part of driftbench works before its numbers are trusted (validators). Each ladder scores against a planted truth: faults, recoveries and effects put in at known places, so every result has an exact answer. A ladder runs a bank of at least 300 items in families and tiers. A hardness gate holds a bank back until its pilot forecasts at least 100 iterations to clear it, with at least 100 distinct failing causes. An independent audit then checks four things: a blind pilot, a cheater, a review of the causes and planted bugs. No score is quoted here until a ladder of the current version passes its audit. None of the seven tools shows judge calibration against hand labels in full (row 12).

Proof: bank.test.ts, hardness.test.ts, audit.test.ts.

A code agent is tested as an estate. Its instructions, skills, subagents, guards, tools and settings are read into atoms, approved by a person and turned into tests; recorded sessions replay offline; a mutant deletes one rule so the tests must notice (testing an estate). Of the six code-agent benchmarks below, none grades office files in the pages read.

text
$ pnpm driftbench estate mutate examples/estates/fixture/approved.yaml --atom instruction.standing.r3-read-a-file-before-you --delete --to <dir>
Mutant: delete instruction.standing.r3-read-a-file-before-you in CLAUDE.md.
  before: - **R3.** Read a file before you edit it.
  after:  (the line is gone)
  map be2e7a0919cb -> 046d83f242ed

Proof: mutate.test.ts, replay.test.ts.

Recovery is measured. A deviation lands at a chosen block, and driftbench counts the blocks the agent needs to come back, the failures after it and whether the action stayed safe. Among the seven tools, the pages read show part of this for one, through red-team robustness, and none of it for the other six (row 15).

text
$ pnpm driftbench perturb examples/acme/specs/conversations/plan-upgrade.yaml --class D2_goal_shift --at 2
✓ Recovery: immediate, 0 blocks to come back (the spec allows 1 block), collateral 0, safe action yes, asked first yes.

Proof: recovery.test.ts, perturb.test.ts.

Tool approval limit for runtimes that run their own tools &ZeroWidthSpace;

Driftbench refuses a denied tool call before it runs wherever the harness runs the tool: the Vercel AI, function and process adapters, and the HTTP chat adapter, where the harness answers each requested tool from the spec. For a runtime that runs its own tools (Nexus, the Claude Code runtime, traces, and SDK loops that own their tool objects), the harness receives the call only after the runtime has run it. The gate then marks the call as denied and fails the block, but the call has already happened. Gating before the call would need each runtime to ask the harness before every call, which none does today.

Known limits of driftbench's sandbox cleanup &ZeroWidthSpace;

  • A process killed with SIGKILL cannot run its cleanup. A workspace folder and a command it started are left behind. SIGTERM, SIGINT and a normal exit are handled.
  • A docker container is not stopped when the run ends unclean: docker containers are not part of the exit cleanup. Measured only with a mocked docker command line, since the host this was built on has no docker.

Where no tool reads yes &ZeroWidthSpace;

In six rows, none of the seven tools shows the capability in full in the pages read: end state assertions (row 9), judge calibration (row 12), drift across turns (row 14), recovery metrics (row 15), intervals and significance tests (row 17), and multi-language handling (row 28). driftbench covers all six. End state assertions and multi-language handling are built, phase 1: see checks. Judge calibration (phase 2), the five drift classes and the recovery metrics of each perturbed run (phase 3) are built: see judges, drift and recovery. Intervals and significance tests are built, phase 4, in the statistics library, the scorecard and driftbench compare: see statistics and scorecards.

Code-agent benchmarks &ZeroWidthSpace;

driftbench evaluates a code-interpreter agent with the same blocks as a conversation. These benchmarks grade code agents. The question for each: what decides a pass, and does it grade office files such as workbooks, documents and decks.

BenchmarkWhat decides a passOffice files gradedSource
SWE-benchthe model's patch is applied in a fresh container, then the required tests must passno: only code tests judge the resultsource, read 2026-10-11
Terminal-Bencheach task has a test script; the pass rule is not stated on the page readnot documented in its docs, read 2026-10-11source, read 2026-10-11
OpenHands evaluationnot found: the evaluation folder returned 404, and the README was not read to the endnot documented in its docs, read 2026-10-11source, read 2026-10-11
GAIAone question, one answer; the scoring rule is not in the abstract readno: answers are textsource, read 2026-10-11
AgentBenchone episode per task, in 8 environments; the success rule is not in the README or the abstractnot documented in its docs, read 2026-10-11source, read 2026-10-11
inspect_ai sandboxesa sandbox per sample; scorers read the end state with exec and read_file; no built-in file diff or snapshotno file-content scorer on the pages readsource, read 2026-10-11

None of the six grades office files in the pages read. driftbench reads workbooks, documents, decks and PDFs with its artifact readers, and diffs the workspace of each block, phase 1; judges read the produced files themselves through the same readers, phase 2 (judges). See checks.

MIT. driftbench is made by Nexus.