Skip to content

The driftbench guide ​

This guide teaches driftbench by doing. Every part follows the same shape, so you always know where to look:

Part of a stepWhat it tells you
WhyWhy you do this step, in one or two sentences.
You needWhat must exist before you start, with an example.
Relevant commandsOne bullet per command: what it does, then each option and what you pass, then what it produces.
You typeThe full command you can copy.
You seeReal output of that command, copied from a run.
How to read itWhat each line means, and what a good result looks like.
If it goes wrongThe message you may see, and the exact fix.

Every command in the guide is run by a test on GitHub, and every output shown is what the command printed. A few outputs are marked as recorded on the page, with how they were made. They are the commands that call a billed model (run once by hand), and the validate stats run (a GitHub runner, with its run id). A command that never ends, such as a server, is shown without output and marked as not run. So is a command that takes minutes, such as a full ladder.

Install ​

You need Node 22.13 or newer, git and pnpm 10 (corepack enable turns pnpm on).

bash
git clone https://github.com/NexusGPT/driftbench.git
cd driftbench
pnpm install
pnpm build
cd packages/cli && npm link && cd ../..
driftbench --version
  • pnpm install fetches the code the tool needs.
  • pnpm build compiles it. It takes a few minutes the first time.
  • npm link makes the command driftbench available in every folder. The last line prints its version.

Every command in the guide starts with driftbench, and you type it exactly so. Run the commands that read examples/ from the root of the clone. The npm package @agent-nexus/driftbench catches up at the next release. Until then, use the clone.

If it goes wrong.

  • pnpm: command not found: run corepack enable, then open a new terminal.
  • driftbench: command not found after npm link: the folder of global npm commands is not on your path. Run npm prefix --global. Add its bin folder to your PATH.
  • Cannot find module when you run driftbench: the build did not finish. Run pnpm build again in the root of the clone.

Where to start ​

You want toRead
Test a small agent in two minutes, with no keyQuickstart
Test your own agent from its files, run a test, change a rule, rerun only what it affectsA walkthrough on your own agent
Write a test by hand, check it, run it, and read a failureWrite tests by hand
Let driftbench write tests from a map of situations, see what they cover, and keep the useful onesGenerate tests
Connect your agent, run it in three modes, repeat runs, play a simulated user and an attackerRun your agent
Hand a question to a judge, and decide whether the judge deserves your trustJudge answers
Read the pass rate with its interval, compare two versions of an agent, and gate a pull requestRead the numbers
Find where a failure comes from, change a rule, and score the stored runs againDiagnose and fix
After an edit, rerun only the tests it can change, and carry the restRerun less
Check that driftbench itself reports the truthProve the evaluator
Build a spec one field at a time, from the smallest valid file to a rich oneSpec fields
See each kind of check pass and fail on a tiny agent, and read how a failing check names itselfChecks
Turn a table of rows into many specs, keep them as a versioned dataset, and plan the rows from a variable storeVariables and datasets
Know every command family, find help, pass options and read exit codesThe command line
Set how driftbench reads your agent's folder, key by key, and check the fileEstate settings
Plan, run, read and resume a campaign, with repeats, budgets and retriesCampaigns
Connect a function, a web endpoint or a command to driftbenchConnect your agent
Hand an object built with an SDK to driftbenchAdapters for SDKs
Run a Claude Code folder or a Nexus agent, and cap what the run spendsClaude Code and Nexus agents
Load your own check or adapter, and see the shipped red-team pluginsPlugins
Run specs from a Python script and from pytest, with a Python function or script as the agentPython
Run a spec inside a vitest suite, one test per block, and compare it with campaign runVitest
Read what an agent did, turn a trace into a draft test, replay a session against a new ruleTraces
Write a scorecard, browse runs in the local viewer, label a block by handThe viewer and scorecards

The reference pages below hold the detail of each area. Each guide part above links to them.

AreaPage
Write tests by handSpec reference, blocks
Generate testsGenerating tests
Run your agentRunning, adapters
Judge answersJudges
Read the numbersStatistics, scorecards
Rerun lessBlast radius
Check driftbench itselfValidators

Words used in the guide ​

Each word is explained again where it first appears. This list is for looking one up.

WordPlain meaningExample
AgentA program that answers people or does a task, using a model.A refund helper that answers "I want a refund".
Test, or specOne file that holds a conversation and what you expect.refund.yaml.
BlockOne message from the user and what must be true after the agent's reply.The user says "It is A-1042", and the agent must start the refund.
Given, When, ThenWhere a block starts, what the user says, and what must be true after.Given a fresh session, when the user asks for a refund, then the reply asks for an order number.
CheckA rule that code can test on a reply.The reply must contain "order number".
ExpectWhat must be true after the agent replies to a block.must: ['order number'].
SkillA folder of instructions that an agent loads when a task needs it..claude/skills/weekly-report/SKILL.md.
CriterionA question in plain words that a judge answers with pass or fail."The reply asks at most one question."
JudgeA model that reads an answer when code cannot check it."Does the reply ask before it acts?"
EstateThe folder of files that define an agent: prompt, skills, helper agents, settings.The Northwind folder.
AtomOne thing your files tell the agent to do or to use, kept with a quote of its line."Never run git push --force."
MapThe list of all atoms of an estate.build/map.yaml.
PlanThe list of situations to test, chosen so that every pair of situations meets once.build/plan.yaml.
Ledger idAn id of a sentence of your prompt that stays the same when you edit the sentence.wrl4c7wlvb.
Blast radiusThe tests that an edit can change.17 of 1254 blocks.
IntervalThe range a rate could really lie in, given how many tests you ran.50% with an interval of 23% to 77%.
AdapterThe piece that connects driftbench to your agent.A function, a web address, a command.
StageA step of the conversation that your agent goes through, named in the spec.address, then offer, then recap.
Session stateWhere the conversation stands when a test starts.resumed, or after_compaction.
State libraryA file that tells driftbench how to put the agent in each session state.northwind/live/states/library.json.
PrefixThe earlier messages that a block starts after.Two messages before the block.
must and must_notPhrases the reply has to contain, and phrases it must not contain.must: ['order number'].
CampaignOne run of a set of tests, kept in a store under an id.estate-run-d0c1fe2d38a8.
PerturbationA surprise that driftbench injects at one step of a conversation.D2_goal_shift, a user who changes the goal.
Stand-inA script that plays recorded answers in place of the agent. It costs nothing.northwind/live/fake/claude.mjs.
LabelerA model that reads your prose and says which sentences are rules.system2:anthropic.
SeederA method that writes one family of tests from a map.twise, which covers every pair of values.
CellOne test file run once.17 of 1251 cells.
Mode FThe agent runs the whole conversation on its own replies.Mode F, free-running.

MIT. driftbench is made by Nexus.