The command line ​
This is a hands-on tour of the driftbench command. It walks through the command families in the order a person meets them. Each family gets one real run on the Acme Telecom sample agent. The guide also shows how to find help, how to pass options, and how to read an exit code. Every command below is run by a test on GitHub, and every excerpt is real output. Nothing calls a model and nothing is paid for. The full list of commands and options is in the CLI reference, and a shorter table is in the command tour.
Before you start ​
Why you do this. The tour uses the sample files of the repository, so you need the tool and a copy of those files.
You need.
- Node 22.13 or newer, and driftbench installed from a clone, as the guide index shows. That gives you the command
driftbench. - The folder
examples/acmefrom a clone of this repository. In a clone, runpnpm installand putpnpmin front of everydriftbenchbelow. - Acme is a made-up phone company. Its sales assistant is a small program, so no model runs. A spec is one test file. This is the start of one,
examples/acme/specs/conversations/tool-failure.yaml:
id: acme-tool-failure
title: The coverage service is down, and the assistant falls back to a sales callback
target: { adapter: function, agent: acme-sales }
blocks:
- user: Hi, I want home internet at 7 Quarry Lane, Acmeville.How a command is built. Every command has the same shape:
driftbench, then a family (spec,campaign,estateand so on).- Then, for some families, a subcommand (
spec validate,campaign run). - Then arguments, which are the files or folders the command works on.
- Then options, which start with
--and change how the command works.
1. Find help ​
Why you do this. You will meet a command you do not know. The help is inside the tool, it matches your version, and it needs no network.
You need. Nothing but the driftbench command.
Relevant commands.
driftbench help: prints an overview of driftbench, every command grouped by task, and the index of help topics · no arguments · it prints text and exits 0.driftbench help <topic>: prints one topic, a page about one capability ·<topic>is a topic id such ascore-exit-codes, or its last part,exit-codes, when that part is unique · it prints the page.driftbench help <area>: lists the topics of one area, such asrecipes,authoring,running,judging,results,validateorredteam·recipesis the area here · it prints one line per topic.driftbench help search <words>: finds the topics that hold every word ·exit codeis the two words to find · it prints each topic with its first matching line.driftbench help --list: lists every command and subcommand with one line each and the topic about it · no arguments · it prints one line per command.driftbench <command> --help: prints the usage of one command ·spec validateis the command here, and--helpasks for its usage · it prints the arguments, the options with their defaults, and the topics about the command.
You type and you see.
$ driftbench help core-exit-codes
Exit codes
What each driftbench exit code means, 0 pass, 1 check failed, 2 bad input, and the commands that use other codes.
## What it is
## When to use it
## Commands and every option
$ driftbench help recipes
Help topics of the area recipes
recipes
$ driftbench help --list
Every driftbench command, with one line each and the help topic about it.
$ driftbench spec validate --help
Usage: driftbench spec validate [options] <paths...>
Parse and validate specs. Exit 1 when any spec is invalid.
paths spec files or folders
--json print one JSON document
-h, --help display help for commandHow to read it.
- Every topic has the same sections. They are: what it is, when to use it, every option, inputs, outputs, a worked example and common errors.
- In a usage,
<x>is a required argument,[x]an optional one, andx...means one or more. driftbench helpis the best start when you do not know the command.<command> --helpis the best start when you know it.
If it goes wrong.
driftbench: no help topic or command named "<name>"with exit 2: the name is not a topic. Rundriftbench helpand copy an id, or rundriftbench help search <word>.driftbench help campaign runprints the usage ofcampaignonly. For a subcommand, typedriftbench campaign run --help.
$ driftbench help no-such-topic # exit 2
driftbench: no help topic or command named "no-such-topic". List the topics: driftbench help. Search them: driftbench help search no-such-topic2. Pass options ​
Why you do this. Options change what a command does. They follow a few rules, and the rules are the same for every command.
You need. The Acme specs folder, examples/acme/specs.
Relevant commands.
driftbench campaign plan: lists the runs a campaign would make and runs nothing ·examples/acme/specs/conversations/plan-upgrade.yamlis the spec to plan.--target function:acmeis the agent to plan it on.--mode T,Fnames two run modes in one value.--repeats 2runs each test twice.--jsonprints one JSON document · it prints the plan, with one row per run.driftbench spec validate: checks that spec files are well formed ·--jsonprints one JSON document · it exits 0 when every spec is valid.
These are the rules for options.
- A value follows its option:
--repeats 2. The form--repeats=2works too. - An option that takes a list can be repeated:
--target function:acme --target function:acme-sales. Some also take one value with commas:--mode T,F. - Options of a subcommand go after the subcommand.
driftbench campaign --repeats 2 planis an error. --jsonprints one JSON document on standard output. Exit codes do not change.--store <file>names the file that keeps runs.--config <file>names the project config,driftbench.yaml.--out <path>names where a command writes files.- A relative path is read from the folder where you type the command.
--plugin <module>is the one option of the program itself. It loads your own code before the command runs, and it may stand anywhere on the line.
You type and you see.
$ driftbench campaign plan examples/acme/specs/conversations/plan-upgrade.yaml --target function:acme --mode T,F --repeats 2
Plan: 1 spec, 1 target, modes T, F, 2 repeats: 4 cells, 20 blocks.
Projected cost: $0.00 (function: no unit measured).
$ driftbench campaign plan examples/acme/specs/conversations/plan-upgrade.yaml --target=function:acme --json
"blocks": 5,How to read it.
4 cellsis two modes times two repeats. A cell is one test run once, in one mode.Projected cost: $0.00is right: the Acme agent is a program, and no model runs.- The second command prints JSON, so a script can read it. Here the line
"blocks": 5,is one field of that document.
If it goes wrong.
error: unknown option '--x'with exit 2: the option does not belong to that command. Readdriftbench <command> --help, and check that the option stands after the subcommand.--repeats takes a whole number of at least 1, not "zero".with exit 2: give a whole number.error: unknown command 'x'with exit 2: rundriftbench --helpfor the list of families.
3. Spec: check your tests ​
Why you do this. A spec with a typo fails late and confusingly. Checking first costs a second.
You need. A folder of specs. Acme has seven, in examples/acme/specs. A spec file says what the user says and what the agent must answer.
Relevant commands.
driftbench spec validate: parses each spec and reports whether it is valid ·examples/acme/specsis a folder, searched for*.yamlfiles · it prints one line per file and a count, and exits 1 when any spec is invalid.driftbench spec lint: does whatvalidatedoes, then prints advice (warnings) ·--strictwould turn warnings into failures · it prints the same lines and a count of warnings.driftbench spec coverage: shows which behaviours and deviations the specs cover ·--inventory examples/acme/inventory.yamlis the list of behaviours the agent must show · it prints a table of covered and missing cells, and exits 1 when something is missing.
You type and you see.
$ driftbench spec validate examples/acme/specs
✓ examples/acme/specs/conversations/tool-failure.yaml acme-tool-failure · 3 blocks · 1 perturbation · hash e5bc5c9ba9d2
Checked 7 specs: 7 valid, 0 invalid, 0 warnings.
$ driftbench spec lint examples/acme/specs
Checked 7 specs: 7 valid, 0 invalid, 0 warnings.
$ driftbench spec coverage examples/acme/specs --inventory examples/acme/inventory.yaml # exit 1
Behaviours by channel
Coverage: 4 of 14 behaviours covered, 7 of 10 flow deviations covered.How to read it.
- A
✓line names the file, the spec id, its blocks (one exchange each) and its hash, a short fingerprint of the content. A changed spec has a new hash. spec coverageexits 1 on Acme by design. The inventory keeps known gaps so you can see what a missing cell looks like. Its last line counts what is covered.- Exit 0 means the specs are valid. Coverage shows whether they test enough.
If it goes wrong.
- A
✗line with a code such asSPEC_PARSE: the spec is not well formed. Fix the key the line names. Every code is in the CLI reference. - A path that does not exist is reported as an invalid spec and exits 1:
$ driftbench spec validate examples/acme/specs/no-such-folder # exit 1
✗ examples/acme/specs/no-such-folder
Checked 1 spec: 0 valid, 1 invalid, 0 warnings.4. Campaign: run the tests ​
Why you do this. A campaign runs your specs on an agent and keeps the results. It is the command that tests the agent. The campaigns page covers it in depth.
You need. Specs, and a target, which is the way driftbench reaches your agent. Acme's sales assistant is the target function:acme-sales.
Relevant commands.
driftbench campaign run: sends each block of each spec to the agent and checks each answer ·examples/acme/specs/conversationsis the folder of specs.--target function:acme-salesis the agent.--store demo.dbis the file that keeps the runs.--campaign demois the name of this campaign · it prints one line per cell and a total, and writesdemo.db.
You type and you see.
$ driftbench campaign run examples/acme/specs/conversations --target function:acme-sales --store demo.db --campaign demo
✓ acme-plan-upgrade · function:acme-sales · F · repeat 0 pass 5 blocks
✓ acme-tool-failure · function:acme-sales · F · repeat 0 pass 3 blocks
6 cells: 6 pass, 0 fail, 0 failed. Spent $0.00.How to read it.
✓is a pass,✗a failure. The letterFis the run mode: the agent answers on its own replies. Modes areT,FandP.Spent $0.00is the money spent on the agent's model. Acme spends none.- Without
--store, the runs vanish when the command ends. Keep the store, becausereport,compareandviewread it.
If it goes wrong.
Campaign demo already exists. Resume it with: driftbench campaign resume demo: the store holds that name. Pick another--campaign, or drop the option.No adapter is set up for <target>.: the target is wrong. The message lists the targets this install reaches.
5. Report and view: read the results ​
Why you do this. A stored campaign is a file of numbers. These commands turn it into pages a person reads.
You need. The store demo.db from step 4.
Relevant commands.
driftbench report: writes the scorecard of a campaign as one HTML page ·--store demo.dbis the file to read.--out scorecard.htmlis the page to write · it writes the page and prints the pass rate with its interval. It exits 0 when written.driftbench compare: compares a candidate campaign with a baseline and says whether the candidate is no worse ·--store,--baselineand--candidatename the file and the two campaigns · it exits 0 when no worse and 1 when worse. The read the numbers page runs it.driftbench view: serves a local web page of a store, with hand labels ·--store demo.dbis the file to show · it runs until you stop it. Our tests do not run it.
$ driftbench report --store demo.db --out scorecard.html
Campaign demo: 6 runs, 6 pass, 0 fail, 0 did not complete.
Wrote scorecard.html.driftbench view --store demo.dbHow to read it.
reportprints the counts first, then where it wrote the page. Openscorecard.htmlin a browser.viewprintsdriftbench view atand a local address on your own machine. Open that address in a browser, and press Ctrl+C in the terminal to stop.
If it goes wrong.
No store at demo.db: this command reads a store and does not make one.with exit 2: the path is wrong, or no campaign ran with--storeyet.
6. Generate and extract: write tests by code ​
Why you do this. Writing hundreds of specs by hand is slow. These commands write them from a map, a list of the situations the agent can face. See generate tests.
You need. A map. Acme has one, examples/acme/map/approved.yaml, and a proposal a person has not approved yet.
Relevant commands.
driftbench extract approve: lists the decisions a person has to make on a proposed map ·examples/acme/map/proposal.yamlis the proposal · it prints how many decisions are open.--allapproves every entry.driftbench generate: writes specs from an approved map, by code, with no model ·--mapis the approved map.--out generatedis the folder for the specs.--t 2makes every pair of situations meet at least once · it writes the specs and prints a table of what each seeder wrote. It exits 1 while some specs are incomplete.
You type and you see.
$ driftbench extract approve examples/acme/map/proposal.yaml
examples/acme/map/proposal.yaml: 291 of 291 decisions are open (51 dimensions, 111 constraints, 55 phrases, 74 relations).
$ driftbench generate --map examples/acme/map/approved.yaml --out generated --t 2 # exit 1
Seed 1, t = 2.
Coverage on the axis map at t = 2: complete specs 708 of 708 (100.0%), all specs 708 (100.0%).How to read it.
291 of 291 decisions are openmeans a person has approved nothing in the proposal yet.generateexits 1 on Acme because some generated specs are incomplete. They wait for an approved phrase. The report lists which.- The commands
gateandselectcome next in this family. They reject tests that cannot fail and keep the smallest useful set. Generate tests runs them.
If it goes wrong.
extractwithout--no-modelor--replaycalls a model and needs a key. Run it only on your own agent, as the CLI reference describes.
7. Estate: read an agent folder ​
Why you do this. A coding agent is a folder of files: a system prompt, skills and settings. The estate family reads the folder into rules and tests. The walkthrough uses it in full.
You need. An estate, which is the agent's folder. The sample is examples/estates/fixture, the Northwind ops assistant.
Relevant commands.
driftbench estate scan: reads the folder into a map of atoms, where an atom is one thing the files tell the agent to do ·examples/estates/fixtureis the folder.--no-modelreads only the structure, so no model is called.--out estate-map.yamlis the map file · it writes the map and prints the counts.driftbench estate lint: checks the folder against its map for orphan files, unreachable files and missing references ·estate-map.yamlis the map · it prints one line per finding and exits 1 when there are findings.
You type and you see.
$ driftbench estate scan examples/estates/fixture --no-model --out estate-map.yaml
Wrote estate-map.yaml: 47 atoms (47 read by code and approved, 0 split from prose by the model and waiting for approval), 34 edges.
$ driftbench estate lint estate-map.yaml # exit 1
estate-map.yaml: estate at examples/estates/fixture, 47 atoms.
ORPHAN .claude/steps.json: nothing in the estate names this file
19 findings.How to read it.
- An orphan is a file that nothing names. An unreachable file is one that the standing instructions never lead to.
- The fixture holds 19 findings planted on purpose, so the lint has something to find.
- The other estate commands (
build,plan,generate,run,unit) are in the CLI reference. Settings are on the estate settings page.
If it goes wrong.
Cannot find the estate folder <folder>.with exit 2: the folder does not exist. Check the path.
8. Radius: rerun only what a change can affect ​
Why you do this. After you edit the agent, most tests cannot change. The radius lists the ones that can, and runs nothing. See rerun less.
You need. A suite of specs, and a change manifest: a short file that says what changed. Acme has examples/acme/radius/stage-recap.yaml, which says the recap stage changed.
Relevant commands.
driftbench radius: lists the blocks a change can affect, against the whole suite ·examples/acme/gated/specsis the suite.--changeis the manifest.--mode F,Tasks for the radius in both modes · it prints counts and a table, and runs nothing.
You type and you see.
$ driftbench radius examples/acme/gated/specs --change examples/acme/radius/stage-recap.yaml --mode F,T
Blast radius of acme-v1 to acme-v2 (manifest edeee14e9937): stage:recap.
195 specs, 861 blocks scored (6 more run with their spec and are not scored: a gate rejected them).
Mode F, free-running: 298 of 861 blocks in the radius (233 direct, 65 forward), 563 out. 102 of 195 cells.
Mode T, teacher-forced: 233 of 861 blocks in the radius (233 direct, 0 forward), 628 out. 102 of 195 cells.How to read it.
- In mode F, 298 of 861 blocks can change. The other 563 cannot, so a rerun may skip them.
- In mode T, each block runs alone, so only the 233 direct blocks need a rerun.
If it goes wrong.
radius needs spec paths and --change <manifest>.with exit 2: give the suite and a manifest. For an estate,radius ledgerderives a manifest from the files.
9. Judge: score what code cannot check ​
Why you do this. Some answers need judgement, such as "was the reply polite". A judge is a model that scores an answer against a written criterion. This family checks the criteria before you trust them. See judge answers.
You need. A folder of criteria. Acme has twelve in examples/acme/criteria, each with a passing and a failing sample.
Relevant commands.
driftbench judge check: validates each criterion and its sample transcripts, and calls no model ·examples/acme/criteriais the folder · it prints one line per criterion and a count, and exits 1 on an issue.driftbench judge run: replays a panel of judges on recorded answers ·--criteriais the folder and--cassetteis the recording · it needs no key when you give a recording. A live judge is billed, so our tests do not run one.
You type and you see.
$ driftbench judge check examples/acme/criteria
ok care_redirect [system1, gating]: pass fixture 2 line(s), fail fixture 2 line(s)
12 valid criteria, 0 issue(s).How to read it.
system1is a quick, cheap judge that answers yes or no.system2is a careful one.gatingmeans a failure of that criterion fails the block.okmeans the criterion parses and its two samples are present.
If it goes wrong.
- A problem line starts with the file name, then the key and the reason. The summary ends with
0 valid criteria, 3 issue(s).or similar, and the exit code is 1. Fix the key and run the check again.
10. Validate: check driftbench itself ​
Why you do this. driftbench measures your agent, so it must measure correctly. The validate family proves this. Each ladder of levels builds an input with a known answer, and checks that driftbench finds it.
You need. A clone of the repository. A ladder reads files from the repository, so run it from the repository root.
Relevant commands.
driftbench validate floors: prints the level each ladder must reach in CI, read fromvalidators/floors.json· no arguments · it prints one line, and it printsnonewhen the folder holds no such file.driftbench validate <component>: runs one ladder ·<component>isestate,generator,judges,map,stats,systemorvariables, orall· it prints a table of levels, and the exit code is the level reached.
$ driftbench validate floors
Floors (validators/floors.json): none.driftbench validate statsThe second command is not run by our tests, because a ladder takes minutes.
How to read it.
validate floorsprintsnonehere because the test folder is not the repository root. In a clone it prints the floors.- A ladder stops at the first failing level. The exit code is the level reached, from 0 to 63. Level 0 means level 1 failed.
If it goes wrong.
- Exit 64 is an input error, such as an unknown component. The message reads
No validator is registered for "x".and lists the registered names.
11. Read an exit code ​
Why you do this. A script or a CI job reads the exit code. The code says whether what the command checks holds.
You need. A shell. In sh or bash, echo $? prints the code of the last command.
The codes.
| Code | Meaning |
|---|---|
| 0 | What the command checks holds, or what it writes was written. |
| 1 | What the command checks fails: an invalid spec, a failing cell, a regression, a coverage gap. |
| 2 | Usage or input error: an unknown option, a missing file, a bad number, an unreadable store. |
| 3 | campaign run, campaign resume, ci, campaign rejudge and family verify only: no test failed, but a block did not run. |
A few commands give the codes another meaning, and say so in their own help. validate exits with the level it reached. view runs until you stop it. The topic is driftbench help core-exit-codes.
You type and you see.
$ driftbench campaign plan examples/acme/specs --mode X # exit 2
Unknown mode "X". The modes are T, F and P, alone or separated by commas.
$ driftbench campaign status --store no-such-store.db # exit 2
No store at no-such-store.db: this command reads a store and does not make one.
$ driftbench spec validate --no-such-option examples/acme/specs # exit 2
error: unknown option '--no-such-option'How to read it.
- Exit 1 means driftbench measured and found a problem with the agent or the files. Exit 2 means it never measured, because the command could not start.
- The message that goes with a nonzero code is on standard error. The results are on standard output.
- Piping output into
headcan cut the command short before it sets its code. Write to a file when the code matters.
If it goes wrong.
- A CI job fails and you cannot tell why: read the last lines on standard error. Exit 2 is almost always a path or an option.