Skip to content

driftbench docs ​

driftbench, an agent evaluation framework by Nexus: every exchange is a unit test, every number carries an interval, drift and recovery measured block by block.

driftbench evaluates AI agents block by block. A block is one exchange with a conversational agent, or one step of a task done by a code-interpreter agent. Each block is a unit test with a Given, a When and a Then, and a spec is a succession of blocks. Both kinds of block share one model: the same runner, checks, judges and statistics.

Every page says what exists today and what is not built yet, with the phase that delivers it when one is assigned.

Start here ​

  • The guide: learn driftbench by doing. Quickstart tests a small agent with no key. A walkthrough on your own agent goes from your agent's files to tests, runs one, changes a rule and reruns only what it affects. Eight more parts cover writing, generating, running, judging, reading the numbers, diagnosing, rerunning less and proving the evaluator.
  • Recipes: evaluate your own agent, add a judge, gate a pull request.
  • Innovations: the ten ideas that lead, then six more, each with its measured impact, its sources, its limits and the files to copy.
  • How it works: the loop of generating tests, running the agent, judging each block, measuring, fixing and rerunning the blast radius.
  • What it does: every feature, grouped, with the capability matrix.
  • The command line: a tour of the commands with a runnable example each.
  • Adapters and extension points: how to reach your agent, and how to add a check, a judge, a criterion, a file reader or a store.
  • Benchmarks: the public judge benchmarks, the hard cases, the cost of a judgement and how to rebuild the sealed sets.

Concepts ​

  • Block: the unit of evaluation, the four runtime concepts (tools, skills, guards, stages), one conversation spec and one task spec.
  • Given, When, Then: the initial conditions, the stimulus and the expected result of a block.
  • Run modes: teacher-forced, free-running and perturbed runs of the same spec.
  • Blast radius: which blocks a change to the agent can affect, so a fix reruns its radius and not the whole suite.
  • Drift: the five drift classes, and the metrics across a succession of blocks.
  • Recovery: the twelve deviation classes, and how fast the agent comes back.
  • Script: writing a conversation spec in code, with the steps user, agent, judge and injection, and the steps that have no spec form.
  • Judges: judged criteria, the judge council and calibration.
  • Statistics: repeats, intervals, paired tests and the parity verdict.

Running ​

  • Retries: the one retry policy of every model call (408, 429, 5xx and network errors, backoff with jitter, Retry-After), how to set it in a spec and on the command line, and where each retry is recorded.
  • Running: runSpec, campaigns in modes T, F and P (campaign plan, campaign run, campaign status, campaign resume), the determinism smoke test, the cache, the budget and the run record.
  • Simulated user: a persona played through the model port or a recording, driftbench simulate and campaign run --driver simulated.
  • The default judge: createDefaultJudge, the one task-based policy behind estate run, replay, live judge calibrate and compare --pairwise, its versions, its cost and its audit.
  • CLI plugins: load your own adapters, checks and judges into the command line, from driftbench.yaml or --plugin.
  • Red-team plugins: 40 attack plugins, each with a generation template and a grader rubric.
  • Adaptive attacker: red-team attacks picked from the agent's last replies, scored by the recovery record, driftbench attack.
  • Multi-turn red-team user: an attacker with one objective over many turns, Crescendo or GOAT, scored each turn, stopping when the agent gives in.
  • Generating tests: the map of situations and its relations, extract and its approval, generate, the validity gate and its funnel, task families, the variance pilot and intake.
  • From a rule to its tests: the ledger id of a prompt rule, the blocks that carry it, the proof of each link by deleting the rule, and the rerun set of an edit, on the Northwind estate.
  • Testing an estate: how driftbench tests a code-interpreter agent as a whole estate (instructions, skills, subagents, guards, tools, settings): atoms, the estate graph, the plan, generated tests, replay over recorded sessions, maintenance.
  • Statistics capabilities to build: the statistics driftbench computes inline and the package also exports, in seven groups (descriptive, many comparisons, an upper bound on a rate, classification and agreement, distances, design and allocation, exact tests), each defined with its formula, result and refused inputs.
  • What you can change in an estate: every key of driftbench.estate.yaml, with a worked example that runs in CI.
  • The estate chain: the flow from a prompt to its ids and tests, proved by one test on the Northwind fixture.
  • The Northwind campaign: one campaign on a code agent with parents, simulated users, the value check, blame, the fix list and the scorecard; a free rehearsal and a capped live run.
  • The estate pointer reader: how prose files are cut into units, how each unit keeps an id across edits, how a recorded labeller gives units labels, and the stack of two labeller classes; the contract the pointer validator bank rests on.
  • The variable store: the one catalogue of variables, values, weights, boundaries and impossible combinations that the prompt-agent map and the estate map share, with the provenance of every entry (a quote of the agent's files, the counts of past conversations, or the owner's words), the loader that checks it, the conversions from and to the space map, and three worked examples.
  • Datasets and prompts: import specs from a CSV file or a spreadsheet through a mapping, and the registry of immutable dataset and prompt versions (dataset, prompt) that a campaign records.
  • Python: run specs from Python, typed results, a pytest plugin (pytest --driftbench specs/), and a Python function as the agent.
  • vitest: describeSpec(path, target), one vitest test per block.
  • Campaign-level criteria: one threshold over every run of a campaign, for example a pass rate; a library API, no command flag yet.
  • Production traces: driftbench import traces drafts specs from OpenTelemetry, Nexus or Claude Code traces; driftbench monitor --source traces watches production for drift in time windows, with the measured delay and false alarms.
  • Releasing: the 0.1.0 packages, the release check and the tag-only release workflow.
  • Validators: code that proves driftbench works: seven ladders (map, generator, judges, system, estate, statistics, pointer) and the executor loop that drives each to 100 percent.
  • Benchmark floors: the shipped default judge on immutable fictional slices, recorded responses, free replay, and regression floors in CI.

Reference ​

Adapters ​

  • Adapters: the contract between driftbench and the agent under test, and the suite every adapter passes.
  • function: an async function in the same process.
  • http: an endpoint that speaks JSON, answering with an Observed or an OpenAI-compatible chat completion.
  • nexus: an agent on the Nexus platform.
  • process: a code-interpreter agent run as a command in a sandbox.
  • openai-agents: an agent built with the OpenAI Agents SDK.
  • anthropic: an agent loop on the Anthropic Messages API.
  • google: an agent on the Gemini API, through @google/genai.
  • langgraph: a LangGraph graph.
  • crewai: a CrewAI crew, in a Python process or deployed on CrewAI AMP.
  • vercel-ai: an agent run by the Vercel AI SDK's generateText.
  • pydantic-ai: a Pydantic AI agent in a Python process.
  • litellm: a model behind a LiteLLM proxy or any OpenAI-compatible endpoint.
  • Claude Code estates: a Claude Code estate and its recorded sessions, read for the estate commands and driftbench replay.

Example scorecards ​

  • Example scorecards: the HTML scorecards driftbench report wrote from the Acme campaigns.
  • Viewer and hand labels: driftbench view, a local website over a store, where a person labels blocks for judge calibration.

Comparison ​

  • Comparison: driftbench beside seven evaluation tools and six code-agent benchmarks, with a source and a read date for every cell.
  • Comparison pages: one page per evaluation tool, with the result, the basis and a reproduce line for every row.

Research ​

What exists today ​

Phases 0, 1 and 2 are built, and most of phases 3 and G, with parts of phase 4 and the release:

  • The spec format: zod schemas in packages/core/src/schema, with the TypeScript types inferred from them.
  • The loader: YAML and JSON files, authoring shorthand, schema errors with their path, semantic checks, and lint advice.
  • driftbench spec validate, driftbench spec lint, and driftbench spec coverage with the behaviour inventory it reads. See CLI.
  • The runner in modes T, F and P, with the scripted, perturbing and simulated drivers and the adaptive attacker, campaigns, a worker pool, a budget stop, the cache, resume, and the determinism smoke test, from code and from the command line. See running and run modes.
  • Drift and recovery: the five drift classes, the ledger check, the stage classifier, the compounding index, and the recovery metrics of every perturbed run. See drift and recovery.
  • The 37 machine checks, with their verdict ids, the judge check behind a council, and the artifact readers for workbooks, documents, decks, PDFs, CSV, JSON, text and code. See checks.
  • Judges in @agent-nexus/driftbench-judges: the judge clients, the council and its cascade, cassettes, calibration on fixture pairs and hand labels, and driftbench judge. See judges.
  • The function, http, nexus and process adapters, the OpenAI Agents, Anthropic, Google and LangGraph adapters, the CrewAI, Vercel AI SDK, Pydantic AI and LiteLLM adapters, their contract suite, and the local and docker sandboxes. See adapters.
  • One retry policy for every model call, set in the spec target or on the command line, with every retry recorded. See retries.
  • The SQLite store: campaigns, runs, the cache, cost units and a JSON export.
  • The statistics library, @agent-nexus/driftbench-stats: intervals, the bootstrap, paired tests, false discovery control, power, the non-inferiority verdict, change detection, judge agreement, variance components and coverage estimators, each checked against Python reference values. See statistics.
  • Test generation: the map of situations with its relations, driftbench extract and its approval, the space commands, driftbench generate, the validity gate (driftbench gate), task families (driftbench family), the variance pilot (driftbench pilot) and intake (driftbench intake). See generating tests.
  • Estates: driftbench estate scan, lint, approve, map, plan, generate, diff and impact, and driftbench replay over recorded sessions. See testing an estate.
  • The validator frame, driftbench validate, with the judge and statistics ladders registered. See validators.
  • A leak guard that refuses personal data in the repository, and CI.
  • These docs, the generator of the spec reference, and a link checker.
  • The Acme Telecom example in examples/acme: a deterministic agent with tools and no model, its conversation and task specs, a behaviour inventory, the map, the generated and gated suites, the criteria and labels, and a task family.

The command line reaches the Acme example agents (function:acme-sales, process:acme-ops, and the pilot's control agent function:acme-random) and recorded sessions; the other adapters are used from code.

Every command driftbench --help lists is built. Not built yet: the runtime battery, and the other items the CLI reference lists.

Roadmap ​

Each row names a command that shows the phase working on the bundled examples, with no model call. Run it from the repository root.

PhaseDeliversStatusProof
0, foundationsspec format, loader, spec validate, spec lint, spec coverage, leak guard, CI, synthetic example agent, docsbuiltpnpm driftbench spec validate examples/acme/specs
1, runnerrunner in free-running mode with a scripted user; machine checks for text, language, personal data, tools, skills, guards, cards, end state, budget, execution, artifacts, workspace and safety; store, cache and resume; cost and latency per segment; sandbox workspace; artifact readers; function, http, nexus and process adaptersbuiltpnpm driftbench campaign run examples/acme/specs --store <db>
2, judgesjudge clients, the council and its cascade, calibration on fixture pairs and hand labels, judges for task artifactsbuiltpnpm driftbench judge calibrate --criteria examples/acme/criteria --labels examples/acme/labels --cassette examples/acme/criteria/calibration/cassette.json --out <file>
3, perturbation, drift and recoverysimulated user, perturbing driver with the twelve deviation classes, stage classifier and ledger check, drift classes, recovery metrics, teacher-forced mode; the same deviations on code-agent tasksbuiltpnpm driftbench perturb examples/acme/specs/conversations/plan-upgrade.yaml --class D2_goal_shift --at 2
4, statistics and gatesstatistics functions with reference fixtures, the parity verdict, scorecard, run diff, JUnit output, CI gate, drift monitor, viewer, runtime batterybuilt but the runtime battery, which is not builtpnpm driftbench ci --specs examples/acme/specs/conversations --target fault:x3-skips-recap --faults examples/acme/faults --baseline <campaign> --store <db> --margin 50
G, the space of situationsthe map of situations, its coverage and completeness, the map extractor, generation, the gate, task families, the variance pilot, intake, selection and the holdoutbuiltpnpm driftbench select --cover examples/acme/map/space.yaml --specs examples/acme/gated --out <dir> --k 10
E, estatesestate scan, lint, approval and map; the plan; test generation; replay; maintenance; the live runner and estate mutantsbuilt; the unit runner is not builtpnpm driftbench estate mutate examples/estates/fixture/approved.yaml --atom instruction.standing.r3-read-a-file-before-you --delete --to <dir>
V, validatorsthe validator frame and six laddersin progress: the frame, the judge ladder and the statistics ladder are builtpnpm driftbench validate stats
ReleaseSDK agent adapters, docs site, version 0.1.0 on npmin progress: eight SDK adapters and the docs site are built; version 0.1.0 is prepared (release workflow and check) and not publishedpnpm build && node tools/release/check.mjs

There is no separate phase for code agents. A task spec runs through each phase with the conversation specs.

Working on the docs &ZeroWidthSpace;

  • pnpm docs:reference writes the spec reference and the reference page of every other file people write (family, campaign configuration, map, interview, criteria and labels, personas and red-team, change manifest, estate files) from the zod schemas. pnpm docs:reference --check exits 1 when a committed page is stale.
  • pnpm docs:links checks that every relative link in docs/ resolves and that every page is reachable from this one.
  • pnpm test runs the tests of both tools, and checks the pages: every YAML example loads, no page has an em dash or an en dash, every issue and warning code is on the CLI page, names that belong to one runtime appear only under docs/adapters, and every comparison cell has a source and a read date.
  • Pages use present tense and short sentences. A claim about a part that is not built says "planned, phase N". A number has a source, or it is left out. Command output on a page is what the command printed.

MIT. driftbench is made by Nexus.