Variables and datasets ​
This part shows how values get into a test, and how one table of rows becomes many tests. A variable is a value that changes from one test to the next, such as an order number. A dataset is a versioned set of test files that driftbench keeps in a store. Every command below is run by a test on GitHub, and every output is real. Nothing calls a model and nothing is paid for. The detail is in the variable store and datasets and prompts.
Before you start ​
Why you do this. The examples need a tiny agent to talk to. This step makes a folder and a stand-in agent, a script that looks an order up, so you need no key.
You need.
- Node 22.13 or newer, and driftbench installed, as in the guide index. In a clone of this repository, write
pnpm driftbenchwhere this page saysdriftbench. - A folder for your files, with a folder
introfor a first spec. Make it inside your clone of this repository, so that../examplesis the examples folder of the clone.
Relevant commands.
mkdir -p evals/intro: makes the folderevalsand, inside it, the folderintro·-pmakes every missing folder on the way · it produces both folders.cd evals: goes into the folder · no options · every later command runs from there.
$ mkdir -p evals/intro
$ cd evalsThe stand-in agent is the file agent.mjs. It looks an order number up in orders.json in its working folder. Without an order number it repeats the message.
// file: agent.mjs
// A stand-in for your agent: reads the user message on stdin, prints one reply.
import { existsSync, readFileSync } from 'node:fs';
const message = readFileSync(0, 'utf8').trim();
const orders = existsSync('orders.json') ? JSON.parse(readFileSync('orders.json', 'utf8')) : {};
const order = message.match(/\bA-\d+\b/)?.[0];
if (order && orders[order]) {
console.log(`Order ${order} is ${orders[order].status}.`);
} else if (order) {
console.log(`I cannot find order ${order}.`);
} else {
console.log(`You wrote: ${message}`);
}1. How values enter a spec ​
Why you do this. Before you build a table of tests, you should know the doors a value can use. Some doors exist and some do not. This step tests both.
You need. Nothing new. The table lists the doors.
| Door | What it does | Example |
|---|---|---|
| The message | The text of user goes to the agent as written. | user: Where is order A-1042? |
given.world.workspace.files | Seeds files in the agent working folder before the block. | orders.json holds order A-1042. |
given.ledger | Facts the agent must hold, which the checks read. | customer: new |
given.vars | Runtime variables, such as the clock or the channel, that an adapter hands to the agent. | channel: chat |
| A table import | Turns each row of a CSV file or a spreadsheet into a spec. | Step 2. |
driftbench has no placeholder inside a spec. The spec below proves it. Block 1 sends and the agent must receive those characters unchanged. Block 2 reads a value seeded in a file.
# file: intro/literal.yaml
id: literal-braces
title: Braces reach the agent as written, and a seeded file reaches it as a file
target: { adapter: process, agent: my-agent }
given:
vars: { name: Sam }
world:
workspace:
files: { orders.json: '{"A-1042": {"status": "paid"}}' }
blocks:
- user: 'Hello {{name}}'
expect:
must: ['\{\{name\}\}']
- user: Where is order A-1042?
expect:
must: ['Order A-1042 is paid']varssetsname: Sam. The process adapter keeps the variables in the run record and in the hash of the block. It does not hand them to your command. The function adapter does: its agent function receives them as the third argument.fileswritesorders.jsoninto the working folder of the agent. This is how a value reaches a command.
Relevant commands.
driftbench campaign run: runs every block of every spec on the agent and checks each reply ·intro/literal.yamlis the spec to run.--process-commandis the command that starts your agent, as a list of words in JSON. The list holdsnodeand the full path ofagent.mjs.$PWDstands for the current folder · it prints one line per run and a count, and exits 1 when a run fails.
You type.
$ driftbench campaign run intro/literal.yaml --process-command "[\"node\",\"$PWD/agent.mjs\"]"
✓ literal-braces · process:my-agent · F · repeat 0 pass 2 blocks
1 cell: 1 pass, 0 fail, 0 failed. Spent $0.00.How to read it. pass 2 blocks means the agent received as plain text and found order A-1042 through the seeded file. A value in a spec is the text you write. To make the text differ from test to test, write a table and import it.
If it goes wrong.
- Block 2 fails with
must.0: the fileorders.jsonis not where the agent reads it. Thefilesmap writes in the agent working folder, so read from the current folder in your own agent. campaign runcannot start your agent: the path in--process-commandis wrong. Runnode agent.mjsby hand with a message on standard input to see the error.
2. One table of rows becomes many specs ​
Why you do this. Ten orders mean ten near-identical tests. You write one table of rows and one mapping instead. driftbench turns each row into a spec.
You need. A table orders.csv with a header row and one row per order, and a mapping mapping.yaml that says which column fills which field.
# file: orders.csv
order,status,expected
A-1042,paid,A-1042|paid
A-2001,shipped,A-2001|shipped
A-3003,refunded,A-3003|refunded# file: mapping.yaml
spec:
id: "order-{order}"
title: "Where is order {order}?"
target: { adapter: process, agent: my-agent }
given:
world:
workspace:
files: { orders.json: '{{"{order}": {{"status": "{status}"}}' }
block:
user: "Where is order {order}?"
expect:
must: { $column: expected, $split: "|" }{order}takes the cell of the columnorder.{row}is the number of the source row.- Two opening braces in a row write one opening brace. That is how the JSON of
orders.jsonis written in the template. $columnreads one column with options.$split: "|"turns the cell into a list.$list: truemakes a list of the whole cell.$asconverts totext,number,booleanorjson.group_by(not used here) names a column: rows that share its value become the blocks of one spec. Without it, each row is one spec with one block.sheetpicks the sheet of a workbook, andheader_rowsays which row names the columns.
Relevant commands.
driftbench import csv: turns the rows of a table into spec files ·orders.csvis the table.--mapping mapping.yamlsays which column fills which field.--out specsis the folder that receives one<id>.yamlper spec · it prints one line per spec and a count, and exits 0 when every spec is written.driftbench spec validate: checks that every spec file in a folder is well formed ·specsis the folder · it prints one line per spec and a count.driftbench campaign run: as in step 1 ·specsis the folder; the same--process-commandstarts the agent · it prints one line per run and a count.
You type.
$ driftbench import csv orders.csv --mapping mapping.yaml --out specs
✓ specs/order-A-1042.yaml order-A-1042 · 1 block · row 2
✓ specs/order-A-2001.yaml order-A-2001 · 1 block · row 3
✓ specs/order-A-3003.yaml order-A-3003 · 1 block · row 4
Imported 3 specs from 3 rows of orders.csv into specs.
$ driftbench spec validate specs
Checked 3 specs: 3 valid, 0 invalid, 0 warnings.
$ driftbench campaign run specs --process-command "[\"node\",\"$PWD/agent.mjs\"]"
✓ order-A-1042 · process:my-agent · F · repeat 0 pass 1 block
✓ order-A-2001 · process:my-agent · F · repeat 0 pass 1 block
✓ order-A-3003 · process:my-agent · F · repeat 0 pass 1 block
3 cells: 3 pass, 0 fail, 0 failed. Spent $0.00.How to read it.
row 2and so on in the import lines are the source rows. Row 1 is the header.- Every written spec starts with a comment that names its file and rows, such as
# Imported from orders.csv, rows 2. 3 cells: 3 passmeans each order spec ran once and passed. The table is the dataset of the tests; the specs are generated from it.- Add a row to the table and import again with
--forceto add a test.
If it goes wrong.
the mapping names column "nope", which orders.csv does not have; its columns are order, status, expected(exit 2): the mapping uses a column the table lacks. Use a name from the header row.import csv reads a .csv or .tsv file; ... is not one.(exit 2): the file is a workbook. Useimport xlsxfor a.xlsxfile.✗ row 2: spec ...(exit 1) and nothing is written: the mapping makes an invalid spec. Fix the mapping or the cell. Every spec is checked before anything is written.- A spec file that exists with other content stops the import (exit 2). Add
--forceto overwrite it.
The two wrong values, run on purpose:
# file: broken/missing-column.yaml
spec:
id: "order-{nope}"
block:
user: "Where is order {order}?"$ driftbench import csv orders.csv --mapping broken/missing-column.yaml --out broken/out # exit 2
the mapping names column "nope", which orders.csv does not have; its columns are order, status, expected
$ driftbench import csv ../examples/acme/import/cases.xlsx --mapping ../examples/acme/import/mapping.yaml --out broken/out # exit 2
import csv reads a .csv or .tsv file; ../examples/acme/import/cases.xlsx is not one.3. What each loader accepts ​
Why you do this. Each way of bringing values in reads one kind of file. Knowing what it accepts saves a failed import.
You need. Nothing new. The table lists the loaders.
| Loader | It reads | Typed values | Note |
|---|---|---|---|
import csv | A .csv or .tsv file. | Every field is text. | Use $as: number to convert. |
import xlsx | One sheet of a .xlsx workbook. | Numbers and booleans stay typed. A formula gives its cached result. | sheet picks the sheet. |
import traces | OpenTelemetry spans, Nexus traces or a Claude Code session. | Not applicable. | Writes drafts with no expect; a person writes it. |
dataset push | A folder of spec files. | Not applicable. | Refuses the push when a spec is invalid. |
| Variable store | A YAML or JSON file, variables.yaml. | Weights are numbers. | Step 5. |
The Acme example ships a workbook and a mapping. The same rows give the same specs from CSV and from the workbook.
Relevant commands.
driftbench import xlsx: turns one sheet of a workbook into spec files ·../examples/acme/import/cases.xlsxis the workbook.--mappingis the mapping, which names the sheetCases.--outis the folder that receives the specs · it prints one line per spec and a count.
You type.
$ driftbench import xlsx ../examples/acme/import/cases.xlsx --mapping ../examples/acme/import/mapping.yaml --out acme-specs
✓ acme-specs/acme-imported-upgrade.yaml acme-imported-upgrade · 2 blocks · rows 2, 3
✓ acme-specs/acme-imported-porting.yaml acme-imported-porting · 1 block · row 4
Imported 2 specs from 3 rows of cases.xlsx into acme-specs.How to read it. Rows 2 and 3 share a conversation value, so the mapping groups them into one spec of two blocks. Row 4 is a spec of its own.
If it goes wrong.
has no sheet "<name>"; its sheets are ...(exit 2): the mapping names a sheet the workbook lacks. Fixsheet.
4. Keep the specs as a versioned dataset ​
Why you do this. You change the table, import again, and later wonder which tests produced which result. A dataset keeps each version of a folder of specs, so a campaign can name the exact version it ran.
You need. The folder specs from step 2, and a store file. Add a fourth order to the table first.
# file: orders.csv
order,status,expected
A-1042,paid,A-1042|paid
A-2001,shipped,A-2001|shipped
A-3003,refunded,A-3003|refunded
A-4004,paid,A-4004|paidRelevant commands.
driftbench dataset push: stores a folder of specs as the next version of a named dataset ·order-specsis the dataset name.specsis the folder.--store runs.dbis the store file, made when missing · it prints the version, a hash and the number of files.driftbench dataset diff: shows what changed between a version and the one before it ·order-specs@2is the version · it prints one line per file, markedAadded,Dremoved orMchanged, and exits 1 when they differ.driftbench dataset restore: writes a version back to a folder, byte for byte ·order-specs@1is the version.--out restoredis the folder · it prints how many files it wrote.driftbench import csv: as in step 2 ·--forceoverwrites spec files that exist.
You type.
$ driftbench dataset push order-specs specs --store runs.db --note 'three orders'
Pushed dataset order-specs@1 (hash 0b20174121e0): 3 files from specs.
$ driftbench import csv orders.csv --mapping mapping.yaml --out specs --force
Imported 4 specs from 4 rows of orders.csv into specs.
$ driftbench dataset push order-specs specs --store runs.db --note 'four orders'
Pushed dataset order-specs@2 (hash cc0c5a9e18b8): 4 files from specs.
$ driftbench dataset diff order-specs@2 --store runs.db # exit 1
A order-A-4004.yaml
1 file differs between order-specs@1 and order-specs@2.
$ driftbench dataset restore order-specs@1 --store runs.db --out restored
Restored dataset order-specs@1 into restored: 3 written, 0 unchanged.
$ driftbench dataset list order-specs --store runs.dbHow to read it.
order-specs@1is the name and the version number. A version never changes. Pushing the same content again gives back the same version.- The hash is a code of the file list. The same files always give the same hash.
dataset list --store runs.dbprints every version with its time and note. The time changes on every run, so this page does not show it.restorestops when a file already exists with other bytes. Add--forceto write over it.
If it goes wrong.
No dataset <name> in <store>.(exit 2): the name is not in the store. Rundataset list --store runs.dbto see what the store holds.Fix the invalid specs first: nothing was pushed.(exit 1): a spec of the folder is invalid. Runspec validateon the folder first.✗ <file> differs from <ref>(exit 1) onrestore: the target holds edited files. Pass--outfor another folder, or--force.
5. Run a campaign on a dataset and read what it records ​
Why you do this. A campaign that names its dataset can be reproduced. The campaign record keeps the name, the version and the hash of everything it ran.
You need. The dataset order-specs in runs.db, with two versions.
Relevant commands.
driftbench campaign run: as in step 1 ·--dataset order-specsruns the specs of the latest version of the dataset, restored byte for byte.--campaign c-ordersis the id of the campaign.--store runs.dbkeeps it.--process-commandstarts the agent · it prints one line per run and a count.driftbench campaign status: shows a kept campaign again, and calls no agent ·c-ordersis the campaign ·--store runs.dbis the store to read · it prints one line per cell, and exits 0 when every cell passes.driftbench campaign export: writes a campaign and everything it ran to a folder, to carry to another store ·c-ordersis the campaign.--storeis the store.--out bundlesis the folder · it writesbundles/c-orderswith the runs, the specs and the dataset files.
You type.
$ driftbench campaign run --dataset order-specs --campaign c-orders --store runs.db --process-command "[\"node\",\"$PWD/agent.mjs\"]"
✓ order-A-1042 · process:my-agent · F · repeat 0 pass 1 block
✓ order-A-2001 · process:my-agent · F · repeat 0 pass 1 block
✓ order-A-3003 · process:my-agent · F · repeat 0 pass 1 block
✓ order-A-4004 · process:my-agent · F · repeat 0 pass 1 block
4 cells: 4 pass, 0 fail, 0 failed. Spent $0.00.
$ driftbench campaign status c-orders --store runs.db
✓ order-A-1042 · process:my-agent · F · repeat 0 pass 1 block
✓ order-A-4004 · process:my-agent · F · repeat 0 pass 1 block
4 cells: 4 pass, 0 fail, 0 failed. Spent $0.00.
$ driftbench campaign export c-orders --store runs.db --out bundles
Exported campaign c-orders to bundles/c-orders: 4 runs, 4 specs, 1 sources (dataset order-specs@2).
$ grep -A6 '"sources"' bundles/c-orders/campaign.json
"sources": [
{
"kind": "dataset",
"name": "order-specs",
"version": 2,
"hash": "cc0c5a9e18b89d5e2cc0f3e241ed2ae9bbb3c553635e267d32be329b942f7996"
}How to read it.
- Four cells ran, one for each row of the table. Each cell is one spec run once.
- The
sourcesentry of the campaign names the dataset, its version and its hash. A reader a year later knows which tests produced the runs. campaign statusprints the stored result again. It also prints a line with the time the campaign was created. That line changes on every run, so this page does not show it. The output above is trimmed to the first and last cell.- Import the bundle on another machine with
driftbench campaign importand run the same dataset again there.
If it goes wrong.
Campaign c-orders already exists. Resume it with: driftbench campaign resume c-orders(exit 2): the store holds a campaign with this id. Pick another id.--dataset and --prompt read the registry of a store: add --store <file>.: a dataset lives in a store. Add--store runs.db.No dataset <ref> in <store>. Push it with: driftbench dataset push <name> <path>: the name or version is not in the store.
6. Plan the rows from a variable store ​
Why you do this. A table of rows needs the right rows. Too few and a case is missed. Too many and you pay for repeats. A variable store lists the things a situation varies on, and a planner chooses rows that cover every pair of values.
You need. A file variables.yaml with two variables, and a short script that loads it and plans six rows. The store is a YAML or JSON file with format, agent, version and a list of variables. Each variable has values, and a value may carry a weight, the share of users it stands for.
# file: variables.yaml
format: 1
agent: northwind-orders
version: '1'
variables:
- id: channel
title: The channel a customer writes on
kind: channel
role: axis
approved: true
values:
- { id: web_chat, title: On the web chat, weight: 0.7 }
- { id: email, title: By email, weight: 0.3 }
- id: order_state
title: What the order system says about the order
kind: data
role: axis
approved: true
values:
- { id: paid, title: The order is paid }
- { id: shipped, title: The order is shipped }
- { id: refunded, title: The order is refunded }# file: broken-variables.yaml
format: 1
agent: northwind-orders
version: '1'
variables:
- id: order_state
title: What the order system says about the order
kind: data
role: axis
approved: true
values:
- { id: paid, title: The order is paid, weight: 1 }
- { id: shipped, title: The order is shipped }// file: plan.ts
import { planV2 } from '@agent-nexus/driftbench-generate';
import { loadVariableStore } from '@agent-nexus/driftbench-space';
async function main(): Promise<void> {
for (const file of ['variables.yaml', 'broken-variables.yaml']) {
const loaded = await loadVariableStore(file);
if (!loaded.ok) {
for (const issue of loaded.errors) console.log(`${file}: ${issue.code} ${issue.path}`);
continue;
}
console.log(`${file}: ${loaded.store.variables.length} variables`);
const plan = planV2(loaded.store, { seed: 1, budget: 6 });
for (const row of plan.rows) console.log(`${row.id} ${JSON.stringify(row.values)}`);
}
}
main();kindsays how a spec is placed on the variable:stage,behaviour,deviation,persona,channel,dataortask.role: axismeans the variable is crossed with the other axes. Adetailstays out of the cross and is varied inside a cell.approved: truesays a person approved the variable. The planner uses only what a person approved.- A weight is relative. Give a weight to every value of a variable or to none.
Relevant commands.
npx tsx plan.ts: runs the script ·plan.tsis the script above · it prints one line per file and the planned rows. The guide test runs it with the driftbench packages of this clone standing in for the ones you install.
You type.
$ npx tsx plan.ts
variables.yaml: 2 variables
row.001 {"channel":"web_chat","order_state":"paid"}
row.002 {"channel":"web_chat","order_state":"shipped"}
row.003 {"channel":"web_chat","order_state":"refunded"}
row.004 {"channel":"email","order_state":"shipped"}
row.005 {"channel":"email","order_state":"paid"}
row.006 {"channel":"email","order_state":"refunded"}
broken-variables.yaml: VARIABLES_WEIGHT_INCOMPLETE variables.0.values.1.weightHow to read it.
- Each planned row gives one value to every approved variable. The six rows hold every pair of a channel and an order state.
- The broken store names its problem by code and by path: a variable that weighs some of its values and not all. Fix it by weighing every value or none.
- No command reads a variable store yet. You copy the rows into a table, then import the table as in step 2.
If it goes wrong.
VARIABLES_PARSE: the file is not valid YAML or JSON. Check the indentation.SCHEMA_codes: a field is missing or has the wrong form. The path names it.- Every code is in the variable store page.
Where to go next ​
- Every field of a spec: Spec fields.
- The detail of imports, the registry and exports: datasets and prompts.
- How to read the campaign you get: step 6 of the quickstart and the read the numbers part.