Write tests by hand ​
This part teaches you to write a test yourself, in a file, and to run it. A test is called a spec: one file that holds one conversation with your agent and what you expect from it. You write three specs: a refund helper, the same helper with a starting state, and the Acme sales assistant that driftbench ships. Every command below is run by a test on GitHub, and every output is real. Nothing calls a model and nothing is paid for.
Before you start ​
Why you do this. A spec needs an agent to talk to. This step makes a folder and a tiny stand-in agent, a script that plays a refund helper, so you need no key.
You need.
- Node 22.13 or newer, and driftbench installed, as in the guide index. In a clone of this repository, write
pnpm driftbenchwhere this page saysdriftbench. - A folder for your tests. Nothing else.
Relevant commands.
mkdir -p evals/specs: makes the folderevalsand, inside it, the folderspecsthat will hold your tests ·-pmakes every missing folder on the way · it produces both folders.cd evals: goes into the folder · no options · every later command runs from there.
$ mkdir -p evals/specs
$ cd evalsThe stand-in agent is the file agent.mjs. It reads the user message, looks the order up in orders.json in its working folder, and prints one reply. You replace it with a call to your own agent later. This is the same script as in the quickstart.
// file: agent.mjs
// A stand-in for your agent: reads the user message on stdin, prints the reply.
import { existsSync, readFileSync } from 'node:fs';
const message = readFileSync(0, 'utf8').trim();
const history = JSON.parse(readFileSync(process.env.DRIFTBENCH_HISTORY_FILE, 'utf8'));
// The orders the spec seeds in the agent's working folder.
const orders = existsSync('orders.json') ? JSON.parse(readFileSync('orders.json', 'utf8')) : {};
const order = message.match(/\bA-\d+\b/)?.[0];
if (/refund/i.test(message)) {
console.log('I can help with a refund. What is your order number?');
} else if (order && orders[order]) {
console.log(`Refund started for order ${order}. You will get an email within 2 days.`);
} else if (order) {
console.log(`I cannot find order ${order}.`);
} else {
console.log(`Sorry, I did not get that. (${history.length} messages so far)`);
}1. What a spec is ​
Why you do this. Before you type a line, you should know the parts of the file and the words that name them.
You need. Nothing yet. Read this table once. The next steps use every row.
| Word | Plain meaning | Example |
|---|---|---|
| Spec | One YAML file: one conversation and what you expect. | specs/refund-ask.yaml |
id | A short name for the spec. Letters, digits, dots, underscores and hyphens. | refund-ask |
title | One line a person reads. | A customer asks for a refund |
target | Which agent to run, and how to reach it. | { adapter: process, agent: my-agent } |
| Block | One message from the user, and what must be true after the agent answers. A spec holds one or more blocks, in order. | The user says "It is A-1042". |
given | The starting state of a block: earlier messages, files, facts, settings. | The folder holds orders.json. |
when | What reaches the agent. You write it as user: <message>. | user: Hi, I would like a refund please. |
expect | What must be true after the agent answers. Each line of it is one check. | The reply must contain "order number". |
The words Given, When and Then name the three parts of a block: the starting state, the message, and what you expect. The full format is in the spec reference, the block page and the Given, When, Then page.
A check is a rule that code can test on the reply, such as "the reply contains this text". Code gives the same answer every time and costs nothing. This is why this page uses checks only.
Relevant commands. None in this step.
How to read it. A spec is a list of small tests (the blocks) inside one file. The agent runs once per block. A block passes when every check in its expect passes. A spec passes when every block passes.
If it goes wrong. Nothing to run yet, so nothing can go wrong. The next step checks your file.
2. Write the first test: one block ​
Why you do this. One block is the smallest useful test. It shows the three parts and the two commands you will use all the time: check, then run.
You need. The folder evals/specs and the stand-in agent from above. This spec has one block. The user asks for a refund. The agent must ask for an order number and must not start a refund yet.
# file: specs/refund-ask.yaml
id: refund-ask
title: A customer asks for a refund and the agent asks for the order number
target: { adapter: process, agent: my-agent }
blocks:
- user: Hi, I would like a refund please.
expect:
must: ['order number']
must_not: ['Refund started']idnames the spec in every report.targetsays the adapter isprocess, which starts a command, and the agent is calledmy-agent. An adapter is the piece that connects driftbench to your agent:processstarts a command,functioncalls a function,httpcalls a web address. See adapters.useris the message the agent receives.mustis a list of patterns that must appear in the reply. Each pattern is a regular expression, a short text pattern. A plain word matches itself. Case does not matter.must_notis a list of patterns that must not appear.
Relevant commands.
driftbench spec validate: checks that a spec file is well formed, before anything runs ·specs/refund-ask.yamlis the file to check; you can also pass a folder · it prints one line per spec, with itsid, its number of blocks and a hash, then a count. It exits 0 when every spec is valid. The hash is a short code of the spec's content: the same content gives the same hash, and any edit changes it.driftbench campaign run: runs every block of every spec on the agent and checks each reply ·specs/refund-ask.yamlis the spec to run.--process-commandis the command that starts your agent, as a list of words in JSON. The list holdsnodeand the full path ofagent.mjs.$PWDstands for the current folder · it prints one line per run and a count, and exits 1 when a run fails. A campaign is one such run of a set of specs.
You type.
$ driftbench spec validate specs/refund-ask.yaml
✓ specs/refund-ask.yaml refund-ask · 1 block · hash ad5a44cf474a
Checked 1 spec: 1 valid, 0 invalid, 0 warnings.
$ driftbench campaign run specs/refund-ask.yaml --process-command "[\"node\",\"$PWD/agent.mjs\"]"
✓ refund-ask · process:my-agent · F · repeat 0 pass 1 block
1 cell: 1 pass, 0 fail, 0 failed. Spent $0.00.How to read it.
✓ specs/refund-ask.yaml ...means the file is valid. The count at the end must say0 invalid.✓ refund-ask · process:my-agent · F · repeat 0 pass 1 blockis one cell: one spec, run once on one agent.Fis the run mode, the free-running mode, where the agent answers from its own earlier replies.passis the verdict.Spent $0.00.is the cost. It is zero because the stand-in agent calls no model.
If it goes wrong.
No spec files found.(exit 1): the path holds no*.yamlfile. Check the path and the file name.(file) SPEC_PARSE cannot read the file: ENOENT: no such file or directory(exit 1): the path does not exist. Relative paths are read from the folder where you type the command.campaign runcannot start the agent: the path in--process-commandis wrong. Runnode agent.mjsby hand with a message on standard input, to see the error.
3. Give a block a starting state: given ​
Why you do this. Most agents need facts to answer: an order that exists, a customer, a setting. The given part puts those facts in place before the message is sent, so the test does not depend on luck.
You need. Three things to set, and the spec below sets all three.
world.workspace.filesseeds files in the agent's working folder. Here it writesorders.json, which holds order A-1042.varssets runtime variables the adapter hands to the agent, such as the channel the conversation runs on.prefixgives a block the messages that came before it. The block starts in the middle of a conversation, and the blocks above it do not run again. Each message has arole(userorassistant) and itscontent.
given can sit on the spec, where it holds for every block, or on one block. This spec has three blocks. Block 1 asks. Block 2 gives an order that exists. Block 3 gives one that does not.
# file: specs/refund-flow.yaml
id: refund-flow
title: A customer asks for a refund, gives an order, then gives a wrong order
target: { adapter: process, agent: my-agent }
given:
world:
workspace:
files: { orders.json: '{"A-1042": {"status": "paid", "total": 49.90}}' }
vars: { channel: web-chat }
blocks:
- user: Hi, I would like a refund please.
expect:
must: ['order number']
must_not: ['Refund started']
- given:
prefix:
- { role: user, content: 'Hi, I would like a refund please.' }
- { role: assistant, content: 'I can help with a refund. What is your order number?' }
user: It is A-1042.
expect:
starts_with: Refund started
must: ['Refund started', 'A-1042']
must_not: ['Sorry']
regex: ['within \d days']
length: { words: { max: 20 } }
- given:
prefix:
- { role: user, content: 'Hi, I would like a refund please.' }
- { role: assistant, content: 'I can help with a refund. What is your order number?' }
user: It is A-9999.
expect:
must: ['cannot find order A-9999']
must_not: ['Refund started']The checks of block 2 are new:
starts_withsays the reply begins with this text.regexis a list of patterns that must match, and here the case must match too.\dstands for one digit.lengthbounds the reply. Here the reply has at most 20 words. You can also boundchars.
Relevant commands.
driftbench spec lint: does whatspec validatedoes, then adds advice about specs that are valid but weak, such as a block that checks nothing ·specsis the folder to check;--strict(not used here) turns each warning into a failure, which suits a pipeline that guards pull requests · it prints one line per spec and a count, and exits 0 when no spec is invalid.driftbench campaign run: as in step 2 · this timespecs/refund-flow.yamlis the spec, and the same--process-commandstarts the agent · it prints one line per run and a count.
You type.
$ driftbench spec lint specs
✓ specs/refund-ask.yaml refund-ask · 1 block · hash ad5a44cf474a
✓ specs/refund-flow.yaml refund-flow · 3 blocks · hash d934aabfcf75
Checked 2 specs: 2 valid, 0 invalid, 0 warnings.
$ driftbench campaign run specs/refund-flow.yaml --process-command "[\"node\",\"$PWD/agent.mjs\"]"
✓ refund-flow · process:my-agent · F · repeat 0 pass 3 blocks
1 cell: 1 pass, 0 fail, 0 failed. Spent $0.00.How to read it.
0 warningsmeans the lint found nothing to advise on.pass 3 blocksmeans all three blocks passed. Block 2 passed because the fileorders.jsonexisted in the agent's folder. Without theworldpart, the agent would have answered "I cannot find order A-1042."- Block 3 proves the opposite case: an order that is not in the file must not start a refund.
If it goes wrong.
- A block fails on block 2 with
must.0: the fileorders.jsonis not where the agent reads it. Thefilesmap writes in the agent's working folder, so read from the current folder in your own agent. GIVEN_UNMETin the lint output (a warning): the Given of a block is wrong, and that block would not run. The message names the check. See the Given checks.- A YAML error such as
SPEC_PARSE: quote any value that holds:or{, and keep the indentation at two spaces.
4. Check what the agent calls: tools, stage and rules ​
Why you do this. A good reply is not enough. An agent also acts: it calls tools, such as a lookup or an order. You check that the right tool is called once, with the right input, and that a forbidden tool is never called.
You need. An agent that calls tools. Acme Telecom is a made-up phone company. driftbench ships its sales assistant as the target function:acme-sales. The assistant is an agent that runs on your machine with no model. It has five tools. This step uses two: coverage.check looks up an address, and catalog.search lists plans.
The spec below follows one customer. In block 1 the customer says hello, and the assistant must ask for an address before any lookup. In block 2 the customer gives an address, and the assistant must check the coverage once and then search the home plans of that address.
# file: specs/acme-coverage.yaml
id: acme-coverage
title: A new customer gets a coverage check before any offer
target: { adapter: function, agent: acme-sales }
stages: [address, offer]
blocks:
- user: Hello, I'm new here and looking for home internet.
expect:
stage: address
tools:
- { label: coverage.check, count: 0 }
- { label: catalog.search, count: 0 }
must: ['Acme Telecom', 'street address']
- user: 3 Orchard Way, Acmeville
expect:
stage: offer
tool_order: in_order
tools:
- label: coverage.check
count: 1
input: { address: { regex: '^3 Orchard Way' } }
- label: catalog.search
count: 1
input: { category: home, address_id: addr-102 }
must: ['Good news', '300 Mbps', 'Acme Fiber 300']
must_not: ['Fiber is not available', 'Fiber 1000']
rules:
forbidden_tools: [order.create, notify_sales]
max_calls: { coverage.check: 1, catalog.search: 1 }The new keys:
stageslists the stages of the conversation: the named points the agent moves through. The agent reports the stage it reached, andstagein a block checks it. Every stage you check must be in this list.toolslists the tool calls you expect in the block.labelis the tool name.countis how many times it is called;0means it must not be called. A count of 0 is a prohibition. It also covers the helpers the agent hands work to. To limit it to the main agent, addactor: main.inputsays what the call must hold, argument by argument. An argument can be a plain value, orregex,contains,one_of,nonemptyorabsent.tool_order: in_ordersays the tools are called in the order listed. Other calls may sit between them.rulesholds rules for the whole conversation.forbidden_toolsnames tools that must never be called.max_callscaps the calls of a tool.
Relevant commands.
driftbench campaign run: as before ·specs/acme-coverage.yamlis the spec; there is no--targetand no--process-command, because the spec names its own target, which driftbench reaches by name · it prints the run line and the count.
You type.
$ driftbench campaign run specs/acme-coverage.yaml
✓ acme-coverage · function:acme-sales · F · repeat 0 pass 2 blocks
1 cell: 1 pass, 0 fail, 0 failed. Spent $0.00.How to read it.
function:acme-salesis the target written as<adapter>:<agent>.pass 2 blocksmeans the stage, the two tools with their inputs, the text and the rules all held in both blocks.- The spec checks what the assistant does as well as what it says. If the assistant offered a plan before it looked up the address, the
tool_orderandcountchecks would fail.
If it goes wrong.
No adapter is set up for function:acme-sales. ...(exit 2): the Acme example is not part of the installed command. It lives in theexamples/acmefolder of the repository. Run this step from a clone of the repository.STAGE_UNKNOWNfromspec validate: astagein a block is not in thestageslist. Add it to the list or fix the spelling.TOOL_COUNT_CONTRADICTION: you gavecount: 0and also aninput. A tool that must not run has no input to check. Remove theinput.MUST_CONFLICT: the same pattern is inmustand inmust_not. Remove one.
5. Read a failure ​
Why you do this. A test that cannot fail proves nothing, and a failure you cannot read costs you time. Here you break a test on purpose and learn to find the cause in three reads.
You need. A spec that expects something the agent does not say. This one expects "Refund approved", but the agent says "Refund started". Keep it in its own folder so the folder specs stays green.
# file: broken/refund-wrong-word.yaml
id: refund-wrong-word
title: This test expects a word the agent never says
target: { adapter: process, agent: my-agent }
given:
world:
workspace:
files: { orders.json: '{"A-1042": {"status": "paid", "total": 49.90}}' }
blocks:
- user: It is A-1042.
expect:
must: ['Refund approved']Relevant commands.
driftbench campaign run: as before ·--store runs.dbis the file where the run is kept, so you can read it again · it adds the run toruns.db, and exits 1 because a cell fails.driftbench campaign status: shows a kept run again ·--store runs.dbis the file to read; with no campaign id it reads the latest campaign;--jsonprints every verdict with its evidence, as JSON · it prints one line per cell, or the JSON, and exits 1 while a cell fails.
You type. The JSON output is trimmed to its key lines.
$ driftbench campaign run broken/refund-wrong-word.yaml --process-command "[\"node\",\"$PWD/agent.mjs\"]" --store runs.db # exit 1
✗ refund-wrong-word · process:my-agent · F · repeat 0 fail block 0 must.0
1 cell: 0 pass, 1 fail, 0 failed. Spent $0.00.
$ driftbench campaign status --store runs.db --json # exit 1
"specId": "refund-wrong-word",
"state": "fail",
"failing": [
"block 0 must.0"
],
"id": "must.0",
"outcome": "fail",
"gating": true,
"evidence": "/Refund approved/ is missing; closest span \"Refund started\", differs by 5 characters; visible text \"Refund started for order A-1042. You will get an email withi...\""How to read it.
✗ ... fail block 0 must.0says three things. The cell failed. The failing block is block 0, because blocks count from 0. The failing check ismust.0, the first pattern ofmust. A check name is the key inexpect, then a dot, then an index or a name.- The JSON is long, so the block above keeps its key lines.
stateis the verdict of the cell.failinglists the failed checks. Each check inverdictshas anid, anoutcome(passorfail),gating(true means a fail fails the block) and theevidence. - Read the evidence first. It names the pattern that is missing. It quotes the closest text in the reply,
Refund started, which is 5 characters away. It also shows the start of the reply it searched. - The
piichecks you did not write also appear in the JSON, withpass. They run on every block that has text. They look in the reply for an IBAN, a national number, a phone number or an email address. - In free-running mode (
F), a later block starts from the agent's own earlier replies. So read the lowest block number first. A later failure often follows from an earlier one.
A table of the check names you will meet:
| Check name | Comes from | Fails when |
|---|---|---|
must.<n> | expect.must | the pattern number n is missing from the reply |
must_not.<n> | expect.must_not | the pattern number n is in the reply |
tools.<tool>.count | expect.tools | the tool was called a different number of times |
tools.<tool>.input.<arg> | expect.tools[].input | an argument of a call differs from what you wrote |
forbidden.<tool> | rules.forbidden_tools | a forbidden tool was called |
stage.reached | expect.stage | the agent ended the block at another stage |
The full list of checks and their evidence is in the checks reference.
If it goes wrong.
The runs are not kept: pass --store <file> to keep them.: you ran without--store. Run again with--store runs.db, then read it.- The cell shows
!and not✗: the run ended with an error, which means the agent crashed or the adapter could not reach it. That is not a wrong answer. Run the agent by hand to see the error. - A check you expect is not in the list: it is not in
expect. A check runs only when the block asks for it.
6. Run the whole folder ​
Why you do this. You now have three specs that pass. One command runs them all, so you can run the folder after every change to your agent.
You need. The folder specs from the steps above.
Relevant commands.
driftbench campaign run: as before ·specsis the folder. driftbench finds every*.yamlfile in it.--process-commandstarts the agent for the specs that targetprocess:my-agent. The Acme spec reaches its own agent · it prints one line per spec and a count.
You type.
$ driftbench campaign run specs --process-command "[\"node\",\"$PWD/agent.mjs\"]"
✓ acme-coverage · function:acme-sales · F · repeat 0 pass 2 blocks
✓ refund-ask · process:my-agent · F · repeat 0 pass 1 block
✓ refund-flow · process:my-agent · F · repeat 0 pass 3 blocks
3 cells: 3 pass, 0 fail, 0 failed. Spent $0.00.How to read it. Three cells, three passes, no cost. The exit code is 0, so a script or a pipeline can rely on it.
If it goes wrong.
- One cell fails: read it with step 5. Fix the agent, or fix the expectation if the expectation was wrong.
Campaign <id> already exists. Resume it with: driftbench campaign resume <id>: you gave--campaignan id the store already holds. Pick another id, or drop the option.
What a judge adds ​
Some qualities no pattern can test, such as "the reply is polite" or "the reply asks only one question". For these you write a criterion: a question in plain words that a model, called a judge, answers with pass or fail. You list criteria at the top of the spec and name them in a block.
# file: specs/judged.yaml
id: judged-one-question
title: The reply asks the customer at most one question
target: { adapter: process, agent: my-agent }
criteria:
- name: one_question
rubric: The reply asks the customer at most one question.
blocks:
- user: Hello, I'm new here and looking for home internet.
expect:
criteria: [one_question@gating]Check that the file is a valid spec:
$ driftbench spec validate specs/judged.yaml
✓ specs/judged.yaml judged-one-question · 1 block · hash b0da038221dd
Checked 1 spec: 1 valid, 0 invalid, 0 warnings.@gating says that a failing answer fails the block. Without it, the verdict is reported and the block still passes. A campaign with no judge prints Not judged: and gives these criteria no verdict. Prefer a check whenever code can decide, because a check is free and always gives the same answer. Judges are the subject of the judges page.
Where to go next ​
- Let driftbench write hundreds of specs for you: Generate tests.
- Every key of a spec: the spec reference. Every check and its evidence: the checks reference.
- Run the same specs several times, keep them and open a scorecard: step 6 of the quickstart.