Skip to content

The command line ​

A tour of the commands with a runnable example each. CLI is the full reference.

driftbench <command> --help prints every option. The examples below run from a clone (git clone https://github.com/NexusGPT/driftbench && cd driftbench && pnpm install), on the fictional Acme Telecom agent, with no model call. Put pnpm in front of each example in a clone, or npx in your own project. campaign run writes the ci.db the later rows read.

Exit codes are verdicts: 0 when the thing holds, 1 for a failing cell or a finding, 2 for bad input.

Write and check specs

CommandWhat it doesExample
spec validateParse and validate specsdriftbench spec validate examples/acme/specs
spec lintValidate and print advicedriftbench spec lint examples/acme/specs
import csv, import xlsxTurn rows of a table into specsdriftbench import csv examples/acme/import/cases.csv --mapping examples/acme/import/mapping.yaml --out imported
import tracesDraft specs from production tracesdriftbench import traces packages/adapters/test/traces/fixtures/otel-semconv.json --from otel --out drafts

Run an agent

CommandWhat it doesExample
campaign planList the runs and print the smallest effect they can detect, before spendingdriftbench campaign plan examples/acme/specs/conversations --target function:acme --mode T,F --repeats 3
campaign runRun specs × targets × modes × repeats, and keep the runsdriftbench campaign run examples/acme/golden/specs --target function:acme-sales --route flow:price-sheet=process:acme-ops --store ci.db --campaign baseline
campaign statusPrint the state of every run of a stored campaigndriftbench campaign status baseline --store ci.db
smokeRun one spec several times and say whether the transcripts differdriftbench smoke examples/acme/specs/conversations/plan-upgrade.yaml --target function:acme
perturbInject one deviation and print the recoverydriftbench perturb examples/acme/specs/conversations/plan-upgrade.yaml --class D2_goal_shift --at 2
simulateLet a simulated user play a personadriftbench simulate examples/acme/specs/conversations/plan-upgrade.yaml --persona examples/acme/personas/sam-upgrader.yaml --recording examples/acme/personas/recordings/sam-upgrader.yaml --target function:acme
attackAttack adaptively, each move picked from the agent's last repliesdriftbench attack examples/acme/specs/conversations/plan-upgrade.yaml --target function:acme --rounds 3 --seed 1
replayRun a spec suite over recorded sessions, offlinedriftbench replay examples/estates/sessions/price-sheet-build.jsonl --suite examples/estates/sessions/specs/price-sheet-build.yaml

Judge

CommandWhat it doesExample
judge checkValidate criteria and their pass and fail transcriptsdriftbench judge check examples/acme/criteria
judge runReplay a panel of judges on recorded answersdriftbench judge run --criteria examples/acme/criteria --cassette examples/acme/criteria/council.cassette.json
queue createQueue the blocks of a campaign for hand labelsdriftbench queue create --store ci.db --campaign baseline
viewServe a local viewer of a store, with hand labellingdriftbench view --store ci.db

Change the agent, or the rules, and rerun little

CommandWhat it doesExample
radiusList the blocks a change to the agent can affect, and run nothingdriftbench radius examples/acme/gated/specs --change examples/acme/radius/stage-recap.yaml --mode F,T
isolationCheck which children of a parent link need their parentdriftbench isolation <spec> --target function:acme-sales (a template: the spec needs parent links and a target that passes it)
objectList, show, explain and check the rule filesdriftbench object list
campaign rejudgeScore stored runs again under a changed rule, with no agent calldriftbench campaign rejudge <specs> --from baseline --campaign rescored --store ci.db --use check-set/<id> (a template: put the specs and the object you wrote)

Report and compare

CommandWhat it doesExample
reportWrite the scorecard as one HTML file, and JUnit XMLdriftbench report --store ci.db --out scorecard.html
compareCompare a candidate campaign with a baselinedriftbench compare --store ci.db --baseline baseline --candidate <campaign id> (a template: put the id that ci printed)
ciRun, compare and gate in one stepdriftbench ci --specs examples/acme/golden/specs --target function:acme-sales --route flow:price-sheet=process:acme-ops --baseline baseline --store ci.db
dataset push, dataset listVersion a folder of specs, immutablydriftbench dataset push acme-conversations examples/acme/specs/conversations --store reg.db

<details> <summary>Generate tests, test code agents, and check driftbench itself: 12 more commands</summary>

<br/>

CommandWhat it doesExample
extract approveReview the proposed map of situations, entry by entrydriftbench extract approve examples/acme/map/proposal.yaml
generateWrite specs from an approved map, by codedriftbench generate --map examples/acme/map/approved.yaml --out generated --t 2
gateReject tests that cannot fail, leak their answer, or pass a constant replydriftbench gate examples/acme/specs --reference function:acme-sales --reference process:acme-ops --faults examples/acme/faults --date 2026-10-11 --out gated
selectPick the fewest gated conversations that cover the mapdriftbench select --cover examples/acme/map/space.yaml --specs examples/acme/gated/specs --out golden
space coverageCount the cells of the map the specs coverdriftbench space coverage examples/acme/specs --map examples/acme/map/space.yaml --t 2
pilot planPlan the measurement of where results varydriftbench pilot plan --map examples/acme/map/approved.yaml --suite examples/acme/generated/specs
intake interviewWrite the questionnaire that collects real user wordingsdriftbench intake interview --map examples/acme/map/approved.yaml --out questions.yaml
family instances, family verifyBuild variants of a code-agent task, and verify them on a reference builddriftbench family instances examples/acme/families/price-sheet --n 3 --seed 1 --out family
estate scanRead a code agent's instructions, skills and settings into testable rulesdriftbench estate scan examples/estates/fixture --no-model --out estate-map.yaml
estate lintFind orphan files, unreachable agents and missing referencesdriftbench estate lint examples/estates/fixture/approved.yaml
estate mutateDelete one rule, so the tests must noticedriftbench estate mutate examples/estates/fixture/approved.yaml --atom instruction.standing.r3-read-a-file-before-you --delete --to mutant
validate floorsPrint the level each self-test suite must hold in CIdriftbench validate floors

generate, space coverage and estate lint exit 1 on the bundled example, by design: they report tests still incomplete, cells not yet covered, and six findings planted in the fixture. monitor, holdout, prompt and the other subcommands are in the CLI reference.

</details>

MIT. driftbench is made by Nexus.