Comparison ​
This page sets driftbench beside seven evaluation tools, capability by capability. Every cell of a reference tool carries one source and the date it was read. driftbench cells say what is built today, link to the page that describes it, and end with a proof: the test that fails when the capability breaks.
Tools compared ​
- langwatch: the scenario library, github.com/langwatch/scenario, and its docs at langwatch.ai/scenario.
- promptfoo: github.com/promptfoo/promptfoo, docs at promptfoo.dev/docs.
- deepeval: github.com/confident-ai/deepeval, docs at deepeval.com/docs.
- inspect_ai: github.com/UKGovernmentBEIS/inspect_ai, docs at inspect.aisi.org.uk.
- opik: github.com/comet-ml/opik.
- langfuse: github.com/langfuse/langfuse.
- braintrust: docs at braintrust.dev/docs/evaluate, scorers at github.com/braintrustdata/autoevals.
How to read a cell ​
yes: the cited page shows the capability.partial: the cited page shows part of it.not found: the cited page, where the capability would be documented, does not show it. This is not proof that the tool lacks it.not documented in its docs: no page read for the tool documents it.
All pages were read on 2026-10-11. Tools change; a cell is true of the page on that date.
Capabilities ​
| # | Capability | langwatch | promptfoo | deepeval | inspect_ai | opik | langfuse | braintrust | driftbench |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Unit of evaluation | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | block: built in the spec format, phase 0 (block) Proof: schema.test.ts. |
| 2 | Test case format | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | YAML or JSON spec: built, phase 0 (spec reference) Proof: loader.test.ts. |
| 3 | Scripted user turns | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | blocks hold the user turns, and the scripted driver replays them: built, phase 1 (running) Proof: scripted.test.ts. |
| 4 | Simulated user | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | a persona on the model port or a recording, facts given only when asked (in code), a stop token, refused on the agent's or a judge's model family: driftbench simulate, campaign run --driver simulated, built (simulated user) Proof: simulated.test.ts, simulate.test.ts. |
| 5 | Injected deviation in a conversation | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | twelve deviation classes: the spec's own perturbations, deviations drawn by the perturbing driver from an approved red-team pack, and attacks the adaptive attacker picks from the agent's last replies: built (recovery, attacker) Proof: perturb.test.ts, attack.test.ts. |
| 6 | Start from a prior conversation state | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | given.prefix and teacher-forced runs, every block from its own Given: built, phases 1 and 3 (run modes) Proof: teacher.test.ts. |
| 7 | Assertions on text | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | must, must_not, lang: built, phase 1 (checks); similarity to a reference text with a pluggable embedder, local by default: built (checks); length, word and character bounds on the reply: built (checks) json, the reply is valid JSON with an optional strict schema and path assertions: built (checks); refusal, refusal phrases in seven languages: built (checks). Proof: must.test.ts, similarity.test.ts, length.test.ts, reply_json.test.ts, refusal.test.ts. |
| 8 | Tool call assertions | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | tools, tool_order, skills, guards: built, phase 1; tools_offered, the tools the agent was offered in the block: built (checks); tool_f1, precision, recall and F1 of the calls with argument matching: built (checks). Proof: tools.test.ts, tools_offered.test.ts, tool_f1.test.ts. |
| 9 | End state assertions | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | end_state, workspace, artifacts: built, phase 1 (checks) Proof: end_state.test.ts, workspace.test.ts, artifacts.test.ts. |
| 10 | LLM judge design | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | criteria with pass and fail pictures, a fixture pair per criterion, chat and system 1 judges: built, phase 2 (judges) Proof: criteria.test.ts. |
| 11 | Judge model choice and ensembles | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | judge council, two of three agreeing votes, and a cascade: built, phase 2 (judges) Proof: council.test.ts. |
| 12 | Judge calibration | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | fixture pairs and kappa against hand labels, which judge owns each criterion: built, phase 2 (judges); hand labels per block and criterion from a local viewer, written to the labels folder calibration reads: built (viewer); real judges measured on the claude command, 2026-10-12: haiku separates 12 of 12 Acme pairs 3 of 3, kappa 0.95 over 40 labels, and a haiku, sonnet and opus council assigns 6 of the 7 criteria it covers, marked single-family (judges) Proof: calibration.test.ts, view.test.ts. |
| 13 | Turn and conversation metrics | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | per-block verdicts, min over blocks: built, phase 1 (drift); turn relevancy, a machine floor with a judged criterion: built (checks) Proof: relevancy.test.ts. |
| 14 | Drift across turns | not documented in its docs, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | first drift block, drift rate: built, phase 1; the five drift classes, the ledger check and the stage classifier: built, phase 3 (drift); knowledge_retention, a fact given earlier not asked again and used, from the ledger and the trace, with a judge fallback: built (checks) Proof: drift.test.ts, knowledge_retention.test.ts. |
| 15 | Recovery and robustness metrics | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | turns to recover, recovery label, collateral, safe action per perturbed run: built, phase 3; their shares across runs, overall and per deviation class, with intervals: built in driftbench report (recovery, scorecards) Proof: recovery.test.ts, perturb.test.ts. |
| 16 | Repeats, pass@k, pass^k | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | pass@k, pass^k, unreliability: built in the statistics library; per golden in the scorecard, with intervals: built, phase 4 (statistics, scorecards) Proof: reliability.test.ts. |
| 17 | Intervals and significance tests | not found, source, read 2026-10-11 | not documented in its docs, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | Wilson, clustered bootstrap, McNemar, Wilcoxon, Benjamini-Hochberg: built in the statistics library; on every rate of the scorecard and in driftbench compare: built, phase 4 (statistics, scorecards) Proof: intervals.test.ts, comparison.test.ts. |
| 18 | Baseline comparison, regression gate | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | non-inferiority verdict: built in the statistics library; campaign comparison (driftbench compare), the CI gate (driftbench ci) and a GitHub Action: built, phase 4 (statistics, CLI) Proof: ci.test.ts, compare.test.ts. |
| 19 | Dataset management | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | content hash per spec, behaviour inventory and coverage matrices: built, phase 0 (CLI); immutable content-addressed versions of spec sets and prompts with push, list, diff and byte-for-byte restore, recorded by the campaign that ran them: built (datasets) Proof: registry.test.ts, registry.test.ts. |
| 20 | Production traces into test cases | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes: driftbench import traces drafts one spec per traced conversation from OpenTelemetry GenAI spans (two SDK shapes), Nexus traces or Claude Code sessions, the observed state as each block's Given, no expect copied, approval required (traces, tests in packages/adapters/test/traces and packages/cli/test/import.test.ts); driftbench monitor --source traces watches the same traces for drift; spec import from a CSV file or a spreadsheet through a column mapping: built (datasets) Proof: import-traces.test.ts, monitor-traces.test.ts. |
| 21 | Red teaming | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | twelve deviation classes and a red-team pack of approved phrasings, drawn by seed: built, phase 3 (recovery, CLI); an adaptive attacker that picks each attack from the agent's last replies, seeded and replayable, scored by the recovery record (driftbench attack): built (attacker) Proof: redteam.test.ts, attack.test.ts. |
| 22 | Caching and determinism | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | cache by block and Given, determinism smoke: built, phase 1 (running) Proof: cache.test.ts, smoke.test.ts. |
| 23 | Parallelism, retries, budget caps | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | not found, source, read 2026-10-11 | worker pool and budget stop: built, phase 1 (running); one retry policy for every model call (408, 429, 5xx and network errors, backoff with jitter, Retry-After, set in the spec target and on the command line, each retry recorded): built, release phase (retries) Proof: retry.test.ts, campaign.test.ts. |
| 24 | Result storage | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | SQLite store and JSON export of a campaign: built, phase 1 (running); a local viewer over the store: built (viewer) Proof: store.test.ts, bundle.test.ts. |
| 25 | Reports and run diffs | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | HTML scorecard with intervals, paired campaign comparison, per-block drilldown with evidence and judge reasoning, JUnit XML: built, phase 4 (scorecards); the same pages served live from the store, with labelling: built (viewer) Proof: scorecard.test.ts, view.test.ts. |
| 26 | CI integration | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | exit codes of the spec and campaign commands: built, phases 0 and 1; the CI gate driftbench ci and a GitHub Action that fails the job unless ci shows parity: built, phase 4 (CLI); a pytest plugin, one test per spec, and a vitest describeSpec, one test per block: built (Python, vitest) Proof: ci.test.ts, test_plugin.py, describe-spec.test.ts. |
| 27 | Custom checks, judges and adapters | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | function, http, nexus and process adapters and custom checks: built, phase 1; OpenAI Agents, Anthropic, Google, LangGraph, CrewAI, Vercel AI SDK, Pydantic AI and LiteLLM adapters, and the Claude Code estate adapter with a live runner: built, 13 adapters in all, each through the same contract suite (adapters) Proof: process.contract.test.ts, custom.test.ts. |
| 28 | Multi-language handling | not documented in its docs, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | lang, lang_stable: built, phase 1 (checks) Proof: lang.test.ts, relevancy-languages.test.ts. |
| 29 | Cost and latency per turn | partial, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | partial, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | not found, source, read 2026-10-11 | budget per block, cost and latency per segment: built, phase 1 (checks) Proof: budget.test.ts. |
| 30 | Docs and maintenance | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | yes, source, read 2026-10-11 | partial, source, read 2026-10-11 | these docs, with links and the generated reference checked in CI: built; a docs-site workflow: built, release phase (index) Proof: docs.test.ts. |
Where driftbench is ahead ​
Four places, each with its evidence. What the other tools do comes from the rows above, read on 2026-10-11. The command output below is what each command printed on the bundled examples, with no model call.
The block is the unit. Every exchange is a unit test with its own Given, When and Then. In teacher-forced mode a block runs from its own Given, so it gets a pass rate of its own, free of the blocks before it. The other tools score a scenario run, a test row, a conversation test case, a sample, a dataset item or a trace (row 1). Proof: schema.test.ts, teacher.test.ts.
driftbench checks its own numbers. Validator ladders prove that a part of driftbench works before its numbers are trusted (validators). Each ladder scores against a planted truth: faults, recoveries and effects put in at known places, so every result has an exact answer. A ladder runs a bank of at least 300 items in families and tiers. A hardness gate holds a bank back until its pilot forecasts at least 100 iterations to clear it, with at least 100 distinct failing causes. An independent audit then checks four things: a blind pilot, a cheater, a review of the causes and planted bugs. No score is quoted here until a ladder of the current version passes its audit. None of the seven tools shows judge calibration against hand labels in full (row 12).
Proof: bank.test.ts, hardness.test.ts, audit.test.ts.
A code agent is tested as an estate. Its instructions, skills, subagents, guards, tools and settings are read into atoms, approved by a person and turned into tests; recorded sessions replay offline; a mutant deletes one rule so the tests must notice (testing an estate). Of the six code-agent benchmarks below, none grades office files in the pages read.
$ pnpm driftbench estate mutate examples/estates/fixture/approved.yaml --atom instruction.standing.r3-read-a-file-before-you --delete --to <dir>
Mutant: delete instruction.standing.r3-read-a-file-before-you in CLAUDE.md.
before: - **R3.** Read a file before you edit it.
after: (the line is gone)
map be2e7a0919cb -> 046d83f242edProof: mutate.test.ts, replay.test.ts.
Recovery is measured. A deviation lands at a chosen block, and driftbench counts the blocks the agent needs to come back, the failures after it and whether the action stayed safe. Among the seven tools, the pages read show part of this for one, through red-team robustness, and none of it for the other six (row 15).
$ pnpm driftbench perturb examples/acme/specs/conversations/plan-upgrade.yaml --class D2_goal_shift --at 2
✓ Recovery: immediate, 0 blocks to come back (the spec allows 1 block), collateral 0, safe action yes, asked first yes.Proof: recovery.test.ts, perturb.test.ts.
Tool approval limit for runtimes that run their own tools ​
Driftbench refuses a denied tool call before it runs wherever the harness runs the tool: the Vercel AI, function and process adapters, and the HTTP chat adapter, where the harness answers each requested tool from the spec. For a runtime that runs its own tools (Nexus, the Claude Code runtime, traces, and SDK loops that own their tool objects), the harness receives the call only after the runtime has run it. The gate then marks the call as denied and fails the block, but the call has already happened. Gating before the call would need each runtime to ask the harness before every call, which none does today.
Known limits of driftbench's sandbox cleanup ​
- A process killed with SIGKILL cannot run its cleanup. A workspace folder and a command it started are left behind. SIGTERM, SIGINT and a normal exit are handled.
- A docker container is not stopped when the run ends unclean: docker containers are not part of the exit cleanup. Measured only with a mocked docker command line, since the host this was built on has no docker.
Where no tool reads yes ​
In six rows, none of the seven tools shows the capability in full in the pages read: end state assertions (row 9), judge calibration (row 12), drift across turns (row 14), recovery metrics (row 15), intervals and significance tests (row 17), and multi-language handling (row 28). driftbench covers all six. End state assertions and multi-language handling are built, phase 1: see checks. Judge calibration (phase 2), the five drift classes and the recovery metrics of each perturbed run (phase 3) are built: see judges, drift and recovery. Intervals and significance tests are built, phase 4, in the statistics library, the scorecard and driftbench compare: see statistics and scorecards.
Code-agent benchmarks ​
driftbench evaluates a code-interpreter agent with the same blocks as a conversation. These benchmarks grade code agents. The question for each: what decides a pass, and does it grade office files such as workbooks, documents and decks.
| Benchmark | What decides a pass | Office files graded | Source |
|---|---|---|---|
| SWE-bench | the model's patch is applied in a fresh container, then the required tests must pass | no: only code tests judge the result | source, read 2026-10-11 |
| Terminal-Bench | each task has a test script; the pass rule is not stated on the page read | not documented in its docs, read 2026-10-11 | source, read 2026-10-11 |
| OpenHands evaluation | not found: the evaluation folder returned 404, and the README was not read to the end | not documented in its docs, read 2026-10-11 | source, read 2026-10-11 |
| GAIA | one question, one answer; the scoring rule is not in the abstract read | no: answers are text | source, read 2026-10-11 |
| AgentBench | one episode per task, in 8 environments; the success rule is not in the README or the abstract | not documented in its docs, read 2026-10-11 | source, read 2026-10-11 |
| inspect_ai sandboxes | a sandbox per sample; scorers read the end state with exec and read_file; no built-in file diff or snapshot | no file-content scorer on the pages read | source, read 2026-10-11 |
None of the six grades office files in the pages read. driftbench reads workbooks, documents, decks and PDFs with its artifact readers, and diffs the workspace of each block, phase 1; judges read the produced files themselves through the same readers, phase 2 (judges). See checks.