Quickstart ​
About two minutes once driftbench is installed. The install needs Node 22.13 or newer, git, pnpm 10 and a clone of the repository, as the guide index shows. The build takes a few minutes. No API key is needed: the agent below is a short script that stands in for yours.
This page uses campaign run, which runs test files you wrote against any target. A campaign is one run of all your tests, kept under an id. If your agent is a Claude Code folder and you want driftbench to write the tests, use estate run in the walkthrough instead.
1. Make a folder and install.
Why. You need the driftbench command, and a folder for your tests. Install driftbench from a clone of the repository. The package on npm is older than this guide and lacks things the quickstart uses, so npm install gives errors here. The npm package catches up at the next release. Until then, use the clone.
Relevant commands.
git clone https://github.com/NexusGPT/driftbench.git: copies the repository to your machine · the address is the repository · it produces the folderdriftbench.cd driftbench: goes into the clone · no options · the next commands run there.pnpm install: fetches the code the tool needs · no options · it fillsnode_modules. Ifpnpmis missing, runcorepack enablefirst.pnpm build: compiles the tool · no options · it takes a few minutes the first time.cd packages/cli && npm link && cd ../..: makes the commanddriftbenchavailable in every folder ·npm linkis the step that does it · it produces the command, and you return to the clone root.driftbench --version: checks the install · no options · it prints the version number.mkdir my-evals && cd my-evals: makes a folder for your tests and goes into it. Make it where you like. The later steps need only this folder · no options · it produces the folder.mkdir specs: makes the folder that will hold your tests · no options · it produces the folder.
git clone https://github.com/NexusGPT/driftbench.git
cd driftbench
pnpm install
pnpm build
cd packages/cli && npm link && cd ../..
driftbench --version
mkdir my-evals && cd my-evals
mkdir specsIf it goes wrong. pnpm: command not found: run corepack enable, then open a new terminal. driftbench: command not found after the link: run npm prefix --global and add its bin folder to your PATH. Cannot find module: the build did not finish, so run pnpm build again in the clone.
2. Create agent.mjs. It reads the user message on standard input, looks the order up in orders.json in its working folder, and prints the reply. Replace it with a call to your own agent later.
Why. driftbench tests an agent, so it needs one to talk to. This short script stands in for yours, so the quickstart needs no key and costs nothing.
// A stand-in for your agent: reads the user message on stdin, prints the reply.
import { existsSync, readFileSync } from 'node:fs';
const message = readFileSync(0, 'utf8').trim();
const history = JSON.parse(readFileSync(process.env.DRIFTBENCH_HISTORY_FILE, 'utf8'));
// The orders the spec seeds in the agent's working folder.
const orders = existsSync('orders.json') ? JSON.parse(readFileSync('orders.json', 'utf8')) : {};
const order = message.match(/\bA-\d+\b/)?.[0];
if (/refund/i.test(message)) {
console.log('I can help with a refund. What is your order number?');
} else if (order && orders[order]) {
console.log(`Refund started for order ${order}. You will get an email within 2 days.`);
} else if (order) {
console.log(`I cannot find order ${order}.`);
} else {
console.log(`Sorry, I did not get that. (${history.length} messages so far)`);
}If it goes wrong (step 2). The script fails when you run it by hand. It reads DRIFTBENCH_HISTORY_FILE, which driftbench sets during a run. Create a file that holds [] with echo '[]' > h.json, then pipe a message in: echo refund | DRIFTBENCH_HISTORY_FILE=h.json node agent.mjs.
3. Create specs/refund.yaml. Each block is one test, with its starting conditions (given), the user's message and the expected reply. The spec-level given seeds order A-1042 in the agent's working folder and sets the channel. The second block starts from a prefix. A prefix is the earlier messages of the conversation. It lets the block run without replaying the first block.
Why. A spec is the test: what the user says and what the agent must answer. Everything driftbench reports comes from specs.
id: refund-happy-path
title: A customer asks for a refund and gives an order number
target: { adapter: process, agent: my-agent }
given:
world:
workspace:
files: { orders.json: '{"A-1042": {"status": "paid", "total": 49.90}}' }
vars: { channel: web-chat }
blocks:
- user: Hi, I would like a refund please.
expect:
must: ['order number']
must_not: ['Refund started']
- given:
prefix:
- { role: user, content: 'Hi, I would like a refund please.' }
- { role: assistant, content: 'I can help with a refund. What is your order number?' }
user: It is A-1042.
expect:
must: ['Refund started', 'A-1042']
must_not: ['Sorry']If it goes wrong (step 3). The spec file is not valid YAML, or a key is misspelt. Step 4 checks it with spec validate, which names the file and the key. Fix the indentation first: the blocks of blocks: start with a dash at the same column.
4. Check the spec, then run it. The output is trimmed to its verdict lines.
Why. Checking first catches a typo before anything runs. Running then shows the verdict of each block.
Relevant commands.
driftbench spec validate specs: checks that every test file in a folder is well formed, before anything runs ·specsis the folder to check · it prints one line per file and a count, and exits 0 when every file is valid.driftbench campaign run specs --target process:my-agent --process-command ...: runs every block of every test against your agent and checks each reply ·specsis the folder of tests.--target process:my-agentsays how to reach the agent.processis the adapter that starts a command.my-agentis its name in the spec.--process-commandis a JSON list, the program and its arguments, that starts your agent. Here it isnodeand the path ofagent.mjs.$PWDis the current folder · it prints one verdict line per test and a count, and exits 1 when a test fails.
$ driftbench spec validate specs
✓ specs/refund.yaml refund-happy-path · 2 blocks · hash 7cfacf8c527b
Checked 1 spec: 1 valid, 0 invalid, 0 warnings.
$ driftbench campaign run specs --target process:my-agent --process-command "[\"node\",\"$PWD/agent.mjs\"]"
✓ refund-happy-path · process:my-agent · F · repeat 0 pass 2 blocks
1 cell: 1 pass, 0 fail, 0 failed. Spent $0.00.The hash after the spec is a short fingerprint of its content. It changes when you edit the file. A cell is one test run once. Here there is one test, so there is one cell.
If it goes wrong (step 4). spec validate exits 1 and prints a code: fix the key it names. campaign run exits 2 and prints Fix the invalid specs first: run spec validate and fix its message. The run prints a failure on the first try: the text of the agent changed, so compare agent.mjs with step 2.
5. Break the agent and run again. Change Refund started for order to Started a return for in agent.mjs. The run now exits 1 and names the block and the check. The output counts blocks from 0, so block 1 is the second block:
Why. A test that cannot fail proves nothing. Breaking the agent on purpose shows you what a failure looks like, and that the report names the block and the check.
$ driftbench campaign run specs --target process:my-agent --process-command "[\"node\",\"$PWD/agent.mjs\"]" # exit 1
✗ refund-happy-path · process:my-agent · F · repeat 0 fail block 1 must.0
1 cell: 0 pass, 1 fail, 0 failed. Spent $0.00.If it goes wrong. The run still passes: the text you changed is not the text the agent prints. Search agent.mjs for Refund started for order and change exactly that.
If it goes wrong (step 5). The run still passes. The text you changed is not the text the agent prints. Search agent.mjs for Refund started for order and change exactly that.
6. Keep the runs and open the scorecard. Run step 4 again with --repeats 3 --store runs.db, then write the scorecard.
Why. One run proves little. Repeats and a kept store give every rate a range, and the scorecard puts it all on one page.
Relevant commands.
campaign run ... --repeats 3 --store runs.db: the same run as step 4, kept ·--repeats 3runs each test 3 times, because one run proves little;--store runs.dbis the file where the runs are kept · it adds the runs toruns.db.driftbench report --store runs.db --out scorecard.html: turns the kept runs into a page you can open ·--store runs.dbis the file to read;--out scorecard.htmlis the page to write · it produces one HTML file with every rate, its interval and every conversation.
$ driftbench campaign run specs --target process:my-agent --process-command "[\"node\",\"$PWD/agent.mjs\"]" --repeats 3 --store runs.db
$ driftbench report --store runs.db --out scorecard.html<p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="../assets/scorecard-dark.png"> <img src="../assets/scorecard-light.png" alt="A driftbench scorecard, one HTML file, dark by default with a light theme. At the top, the candidate campaign against its baseline. The candidate has 12 of 26 cases passing against 26 of 26 for the baseline. It has 36 of 78 runs passing. It has 14 of 26 cases regressed and none improved, McNemar p = 0.0001. Below, the two versions side by side, then three charts. The charts show cases passing by test, runs passing by mode and failed checks by type. Runs passing by mode: teacher-forced 57.1%, free-running 57.1%, perturbed 33.3%, each with its interval. Failed checks by type are led by tools (192) and must (162). Then the tests list. Each test shows its status, its score and one pass or fail pill per check family. The first one fails 0 of 4 cases." width="820" /> </picture> </p> <p align="center"><sub>The scorecard of the example agents with a bug planted on purpose in each. Numbers and charts first, then every test, then every run: a click opens its conversation and its checks.</sub></p>
If it goes wrong (step 6). report exits 2 and prints No store at runs.db. You ran step 4 without --store runs.db, so add the option and run it again. It prints No campaign <name> in the store: pass a campaign name that the store holds.
Next: evaluate your own agent, add a judge, or gate a pull request. The full spec format is in the spec reference.
If it goes wrong ​
npm: command not found, or an engine warning: Node is missing or older than 22. Install a current Node from nodejs.org and run step 1 again.spec validateexits 1 and prints a file, a key and a code: the spec is not well formed. Fix the key it names. Every code is listed in the CLI reference.campaign runcannot start your agent: the path in--process-commandis wrong. Runnode agent.mjsby hand in the folder, with a message on standard input, to see the error.- The run still passes after you broke the agent: the text you changed is not the text the agent prints. Search
agent.mjsforRefund started for orderand change exactly that. The runs are not kept: pass --store <file> to keep them.: you ran without--store. Add--store runs.db, then write the scorecard from it.