driftbench docs ​

driftbench evaluates AI agents block by block. A block is one exchange with a conversational agent, or one step of a task done by a code-interpreter agent. Each block is a unit test with a Given, a When and a Then, and a spec is a succession of blocks. Both kinds of block share one model: the same runner, checks, judges and statistics.
Every page says what exists today and what is not built yet, with the phase that delivers it when one is assigned.
Start here ​
- The guide: learn driftbench by doing. Quickstart tests a small agent with no key. A walkthrough on your own agent goes from your agent's files to tests, runs one, changes a rule and reruns only what it affects. Eight more parts cover writing, generating, running, judging, reading the numbers, diagnosing, rerunning less and proving the evaluator.
- Recipes: evaluate your own agent, add a judge, gate a pull request.
- Innovations: the ten ideas that lead, then six more, each with its measured impact, its sources, its limits and the files to copy.
- How it works: the loop of generating tests, running the agent, judging each block, measuring, fixing and rerunning the blast radius.
- What it does: every feature, grouped, with the capability matrix.
- The command line: a tour of the commands with a runnable example each.
- Adapters and extension points: how to reach your agent, and how to add a check, a judge, a criterion, a file reader or a store.
- Benchmarks: the public judge benchmarks, the hard cases, the cost of a judgement and how to rebuild the sealed sets.
Concepts ​
- Block: the unit of evaluation, the four runtime concepts (tools, skills, guards, stages), one conversation spec and one task spec.
- Given, When, Then: the initial conditions, the stimulus and the expected result of a block.
- Run modes: teacher-forced, free-running and perturbed runs of the same spec.
- Blast radius: which blocks a change to the agent can affect, so a fix reruns its radius and not the whole suite.
- Drift: the five drift classes, and the metrics across a succession of blocks.
- Recovery: the twelve deviation classes, and how fast the agent comes back.
- Script: writing a conversation spec in code, with the steps
user,agent,judgeandinjection, and the steps that have no spec form. - Judges: judged criteria, the judge council and calibration.
- Statistics: repeats, intervals, paired tests and the parity verdict.
Running ​
- Retries: the one retry policy of every model call (408, 429, 5xx and network errors, backoff with jitter, Retry-After), how to set it in a spec and on the command line, and where each retry is recorded.
- Running:
runSpec, campaigns in modes T, F and P (campaign plan,campaign run,campaign status,campaign resume), the determinism smoke test, the cache, the budget and the run record. - Simulated user: a persona played through the model port or a recording,
driftbench simulateandcampaign run --driver simulated. - The default judge:
createDefaultJudge, the one task-based policy behindestate run,replay, livejudge calibrateandcompare --pairwise, its versions, its cost and its audit. - CLI plugins: load your own adapters, checks and judges into the command line, from
driftbench.yamlor--plugin. - Red-team plugins: 40 attack plugins, each with a generation template and a grader rubric.
- Adaptive attacker: red-team attacks picked from the agent's last replies, scored by the recovery record,
driftbench attack. - Multi-turn red-team user: an attacker with one objective over many turns, Crescendo or GOAT, scored each turn, stopping when the agent gives in.
- Generating tests: the map of situations and its relations,
extractand its approval,generate, the validity gate and its funnel, task families, the variance pilot and intake. - From a rule to its tests: the ledger id of a prompt rule, the blocks that carry it, the proof of each link by deleting the rule, and the rerun set of an edit, on the Northwind estate.
- Testing an estate: how driftbench tests a code-interpreter agent as a whole estate (instructions, skills, subagents, guards, tools, settings): atoms, the estate graph, the plan, generated tests, replay over recorded sessions, maintenance.
- Statistics capabilities to build: the statistics driftbench computes inline and the package also exports, in seven groups (descriptive, many comparisons, an upper bound on a rate, classification and agreement, distances, design and allocation, exact tests), each defined with its formula, result and refused inputs.
- What you can change in an estate: every key of
driftbench.estate.yaml, with a worked example that runs in CI. - The estate chain: the flow from a prompt to its ids and tests, proved by one test on the Northwind fixture.
- The Northwind campaign: one campaign on a code agent with parents, simulated users, the value check, blame, the fix list and the scorecard; a free rehearsal and a capped live run.
- The estate pointer reader: how prose files are cut into units, how each unit keeps an id across edits, how a recorded labeller gives units labels, and the stack of two labeller classes; the contract the pointer validator bank rests on.
- The variable store: the one catalogue of variables, values, weights, boundaries and impossible combinations that the prompt-agent map and the estate map share, with the provenance of every entry (a quote of the agent's files, the counts of past conversations, or the owner's words), the loader that checks it, the conversions from and to the space map, and three worked examples.
- Datasets and prompts:
importspecs from a CSV file or a spreadsheet through a mapping, and the registry of immutable dataset and prompt versions (dataset,prompt) that a campaign records. - Python: run specs from Python, typed results, a pytest plugin (
pytest --driftbench specs/), and a Python function as the agent. - vitest:
describeSpec(path, target), one vitest test per block. - Campaign-level criteria: one threshold over every run of a campaign, for example a pass rate; a library API, no command flag yet.
- Production traces:
driftbench import tracesdrafts specs from OpenTelemetry, Nexus or Claude Code traces;driftbench monitor --source traceswatches production for drift in time windows, with the measured delay and false alarms. - Releasing: the 0.1.0 packages, the release check and the tag-only release workflow.
- Validators: code that proves driftbench works: seven ladders (map, generator, judges, system, estate, statistics, pointer) and the executor loop that drives each to 100 percent.
- Benchmark floors: the shipped default judge on immutable fictional slices, recorded responses, free replay, and regression floors in CI.
Reference ​
- Spec reference: every field of a spec, generated from the zod schemas.
- Campaign configuration reference: every field of the
campaign:key indriftbench.yaml. - Task family reference: every field of a task family file and its review file, generated from the zod schemas.
- Map reference: every field of a map proposal and of an approved map.
- Interview reference: every field of the intake questionnaire.
- Criteria and labels reference: every field of a criterion file and of a label file.
- Personas and red-team reference: every field of a persona, a user recording, a red-team pack file and a red-team plugin.
- Change manifest reference: every field of a change manifest for
driftbench radius. - Estate files reference: every field of an estate map, an estate plan, a phrase bank and a session index.
- Blame and placement reference: every evidence name a blame or placement rule reads, every item of the shipped blame and placement files, and the steps of the shipped diagnostics.
- CLI: every
driftbenchcommand, its options, its output and its exit codes. - Public API: what a package's public API is, the frozen list of its names, and how a name is added or removed.
- Checks: every field of
expect, the check that reads it, and its verdict ids.
Adapters ​
- Adapters: the contract between driftbench and the agent under test, and the suite every adapter passes.
- function: an async function in the same process.
- http: an endpoint that speaks JSON, answering with an Observed or an OpenAI-compatible chat completion.
- nexus: an agent on the Nexus platform.
- process: a code-interpreter agent run as a command in a sandbox.
- openai-agents: an agent built with the OpenAI Agents SDK.
- anthropic: an agent loop on the Anthropic Messages API.
- google: an agent on the Gemini API, through
@google/genai. - langgraph: a LangGraph graph.
- crewai: a CrewAI crew, in a Python process or deployed on CrewAI AMP.
- vercel-ai: an agent run by the Vercel AI SDK's
generateText. - pydantic-ai: a Pydantic AI agent in a Python process.
- litellm: a model behind a LiteLLM proxy or any OpenAI-compatible endpoint.
- Claude Code estates: a Claude Code estate and its recorded sessions, read for the estate commands and
driftbench replay.
Example scorecards ​
- Example scorecards: the HTML scorecards
driftbench reportwrote from the Acme campaigns. - Viewer and hand labels:
driftbench view, a local website over a store, where a person labels blocks for judge calibration.
Comparison ​
- Comparison: driftbench beside seven evaluation tools and six code-agent benchmarks, with a source and a read date for every cell.
- Comparison pages: one page per evaluation tool, with the result, the basis and a reproduce line for every row.
Research ​
- Budget-bounded test generation for AI agent evaluation: a research draft describing the budget selector and its evaluation.
- Block phrasing experiments: four pre-registered experiments on how a customer's wording changes a block's result.
- Block parents experiments: three pre-registered experiments on parent links, root counts, clustered intervals and isolation runs.
- Block given experiments: three pre-registered experiments on the Given of a block: tool call records, context size and a Given verifier.
- Simulated human experiments for blocks: three pre-registered experiments on simulated users, an agenda engine and uncovered production clusters.
- Block links experiments: two pre-registered experiments on section links and per-root counting, with prompt mutation as ground truth.
- Block framework core experiments: four pre-registered experiments on backward compatibility, defaults, overrides and re-judging in a block framework prototype.
- Block framework persona experiments: three pre-registered experiments on the persona grid, prompt unit routing and ownership refusals.
- No fresh JudgeBench or RubricBench pool remains: an audit showing that no fresh JudgeBench or RubricBench item remains, with planning power numbers.
- Block framework value, blame and diagnosis experiments: five pre-registered experiments on the value-check ladder, blame rules and diagnosis.
What exists today ​
Phases 0, 1 and 2 are built, and most of phases 3 and G, with parts of phase 4 and the release:
- The spec format: zod schemas in
packages/core/src/schema, with the TypeScript types inferred from them. - The loader: YAML and JSON files, authoring shorthand, schema errors with their path, semantic checks, and lint advice.
driftbench spec validate,driftbench spec lint, anddriftbench spec coveragewith the behaviour inventory it reads. See CLI.- The runner in modes T, F and P, with the scripted, perturbing and simulated drivers and the adaptive attacker, campaigns, a worker pool, a budget stop, the cache, resume, and the determinism smoke test, from code and from the command line. See running and run modes.
- Drift and recovery: the five drift classes, the ledger check, the stage classifier, the compounding index, and the recovery metrics of every perturbed run. See drift and recovery.
- The 37 machine checks, with their verdict ids, the judge check behind a council, and the artifact readers for workbooks, documents, decks, PDFs, CSV, JSON, text and code. See checks.
- Judges in
@agent-nexus/driftbench-judges: the judge clients, the council and its cascade, cassettes, calibration on fixture pairs and hand labels, anddriftbench judge. See judges. - The function, http, nexus and process adapters, the OpenAI Agents, Anthropic, Google and LangGraph adapters, the CrewAI, Vercel AI SDK, Pydantic AI and LiteLLM adapters, their contract suite, and the local and docker sandboxes. See adapters.
- One retry policy for every model call, set in the spec target or on the command line, with every retry recorded. See retries.
- The SQLite store: campaigns, runs, the cache, cost units and a JSON export.
- The statistics library,
@agent-nexus/driftbench-stats: intervals, the bootstrap, paired tests, false discovery control, power, the non-inferiority verdict, change detection, judge agreement, variance components and coverage estimators, each checked against Python reference values. See statistics. - Test generation: the map of situations with its relations,
driftbench extractand its approval, thespacecommands,driftbench generate, the validity gate (driftbench gate), task families (driftbench family), the variance pilot (driftbench pilot) and intake (driftbench intake). See generating tests. - Estates:
driftbench estate scan,lint,approve,map,plan,generate,diffandimpact, anddriftbench replayover recorded sessions. See testing an estate. - The validator frame,
driftbench validate, with the judge and statistics ladders registered. See validators. - A leak guard that refuses personal data in the repository, and CI.
- These docs, the generator of the spec reference, and a link checker.
- The Acme Telecom example in
examples/acme: a deterministic agent with tools and no model, its conversation and task specs, a behaviour inventory, the map, the generated and gated suites, the criteria and labels, and a task family.
The command line reaches the Acme example agents (function:acme-sales, process:acme-ops, and the pilot's control agent function:acme-random) and recorded sessions; the other adapters are used from code.
Every command driftbench --help lists is built. Not built yet: the runtime battery, and the other items the CLI reference lists.
Roadmap ​
Each row names a command that shows the phase working on the bundled examples, with no model call. Run it from the repository root.
| Phase | Delivers | Status | Proof |
|---|---|---|---|
| 0, foundations | spec format, loader, spec validate, spec lint, spec coverage, leak guard, CI, synthetic example agent, docs | built | pnpm driftbench spec validate examples/acme/specs |
| 1, runner | runner in free-running mode with a scripted user; machine checks for text, language, personal data, tools, skills, guards, cards, end state, budget, execution, artifacts, workspace and safety; store, cache and resume; cost and latency per segment; sandbox workspace; artifact readers; function, http, nexus and process adapters | built | pnpm driftbench campaign run examples/acme/specs --store <db> |
| 2, judges | judge clients, the council and its cascade, calibration on fixture pairs and hand labels, judges for task artifacts | built | pnpm driftbench judge calibrate --criteria examples/acme/criteria --labels examples/acme/labels --cassette examples/acme/criteria/calibration/cassette.json --out <file> |
| 3, perturbation, drift and recovery | simulated user, perturbing driver with the twelve deviation classes, stage classifier and ledger check, drift classes, recovery metrics, teacher-forced mode; the same deviations on code-agent tasks | built | pnpm driftbench perturb examples/acme/specs/conversations/plan-upgrade.yaml --class D2_goal_shift --at 2 |
| 4, statistics and gates | statistics functions with reference fixtures, the parity verdict, scorecard, run diff, JUnit output, CI gate, drift monitor, viewer, runtime battery | built but the runtime battery, which is not built | pnpm driftbench ci --specs examples/acme/specs/conversations --target fault:x3-skips-recap --faults examples/acme/faults --baseline <campaign> --store <db> --margin 50 |
| G, the space of situations | the map of situations, its coverage and completeness, the map extractor, generation, the gate, task families, the variance pilot, intake, selection and the holdout | built | pnpm driftbench select --cover examples/acme/map/space.yaml --specs examples/acme/gated --out <dir> --k 10 |
| E, estates | estate scan, lint, approval and map; the plan; test generation; replay; maintenance; the live runner and estate mutants | built; the unit runner is not built | pnpm driftbench estate mutate examples/estates/fixture/approved.yaml --atom instruction.standing.r3-read-a-file-before-you --delete --to <dir> |
| V, validators | the validator frame and six ladders | in progress: the frame, the judge ladder and the statistics ladder are built | pnpm driftbench validate stats |
| Release | SDK agent adapters, docs site, version 0.1.0 on npm | in progress: eight SDK adapters and the docs site are built; version 0.1.0 is prepared (release workflow and check) and not published | pnpm build && node tools/release/check.mjs |
There is no separate phase for code agents. A task spec runs through each phase with the conversation specs.
Working on the docs ​
pnpm docs:referencewrites the spec reference and the reference page of every other file people write (family, campaign configuration, map, interview, criteria and labels, personas and red-team, change manifest, estate files) from the zod schemas.pnpm docs:reference --checkexits 1 when a committed page is stale.pnpm docs:linkschecks that every relative link indocs/resolves and that every page is reachable from this one.pnpm testruns the tests of both tools, and checks the pages: every YAML example loads, no page has an em dash or an en dash, every issue and warning code is on the CLI page, names that belong to one runtime appear only underdocs/adapters, and every comparison cell has a source and a read date.- Pages use present tense and short sentences. A claim about a part that is not built says "planned, phase N". A number has a source, or it is left out. Command output on a page is what the command printed.