Skip to content

Spec reference ​

A spec is one YAML or JSON file. It describes one conversation or one task as a succession of blocks; see block and Given, When, Then. This page lists every field the loader accepts, then the fields of the behaviour inventory that driftbench spec coverage reads. It is generated from the zod schemas that the CLI uses.

  • Keys are snake_case. An unknown key is an error.
  • Required means the field is present once the loader has applied the shorthand below.
  • Descriptions come from the schemas. An empty cell means the schema has no description for that field yet.

Shorthand ​

The loader rewrites these forms before it validates a spec.

  • kind may be left out: the loader sets conversation.
  • A block may write user: <message> or burst: [<message>, <message>] in place of when.
  • ledger_seed at the top level is the same as given.ledger.
  • workspace_seed at the top level is the same as given.world.workspace. A string means { from: <string> }.

Contents &ZeroWidthSpace;

Spec &ZeroWidthSpace;

One golden conversation, or one task for a code agent, written as a succession of blocks.

FieldTypeRequiredDefaultDescription
schema_versioninteger, 1 to 1noThe spec format version the spec is written for. A driftbench that reads a lower version refuses the spec and says to upgrade. Absent means version 1.
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesThe spec id: letters, digits, dots, underscores and hyphens.
versionstringnoA free label. The content hash, not this label, is the cache key.
kindconversation or taskyesconversation, set by the loaderconversation or task. The loader sets conversation when it is absent.
titlestringnoA short title.
descriptionstringnoA longer description.
channelstringnoThe channel the conversation runs on, such as chat.
langstring, 2 to 8 charactersnoThe declared language of the conversation.
tierstringnoA tier label, such as core or edge.
tagslist of stringsnoFree tags. flow:<id> ties the spec to a flow of the behaviour inventory.
coverslist of stringsnoIds of the behaviours this spec covers, from a behaviour inventory.
targetTargetRefnoThe default agent. A run can override it.
givenGivenDefaultsnoInitial conditions for every block.
stageslist of strings, each not emptynoThe stage catalogue of the flow. expect.stage and recovery.stage_after must name one of these.
criterialist of CriterionDefnoThe judged criteria the blocks may reference.
blockslist of Block, at least 1 itemyesThe blocks, in order. At least one.
endSpec endnoAn end verdict: the run stops after block atBlock (the last block when absent) with this verdict.
rulesRulesnoRules for the whole conversation.
operationsmap of string to one of read, modify, delete, overwrite, send, pay, pushnoThe operation class of each tool the agent may call, by its exact label: read, modify (declared, not destructive), delete, overwrite, send, pay or push. The safety checks and the recovery metrics read a tool call by this table; a tool it does not name is unknown, never classified by its name, and a guard that needs its class is inconclusive on it.
end_stateEndStatenoThe expected end state of the world after the run.
perturbationslist of PerturbationnoDeviations to inject in perturbed mode.
usesUsesnoThe objects every block uses (policies, check set, diagnostics), by id or inline. Left out of the block cache key.
metadatamap of string to any valuenoFree metadata. links holds the links of the blocks to prompt sections, spec clauses and past turns; it and provenance stay out of the spec hash.

TargetRef &ZeroWidthSpace;

Which agent a spec runs against. The adapter decides what the other fields mean.

FieldTypeRequiredDefaultDescription
adapterstring, not emptyyesThe adapter that reaches the agent, such as function, http, nexus or process.
agentstring, not emptynoThe agent, as the adapter names it.
variantstring, not emptynoA variant of the agent, such as a prompt version.
deploymentstring, not emptynoA deployment of the agent, as the adapter names it.
optionsmap of string to any valuenoAdapter settings. Each adapter defines their names.
promptTargetPromptnoThe prompt the agent runs with, from the registry. campaign run --prompt sets it; each adapter page says how it reaches the agent.
retryRetrySettingsnoHow the model calls of this target are retried.

TargetPrompt &ZeroWidthSpace;

A prompt version from the registry, handed to the agent as its system prompt. campaign run --prompt sets it.

FieldTypeRequiredDefaultDescription
namestring, not emptyyesThe prompt name in the registry.
versioninteger, at least 1yesThe version number.
hashstring matching ^[0-9a-f]{64}$yesThe version hash: the SHA-256 of its file list.
textstringyesThe prompt text, restored byte for byte from the store.

RetrySettings &ZeroWidthSpace;

How model calls are retried. Only a 408, a 429, a 5xx (529 included) or a network error is retried.

FieldTypeRequiredDefaultDescription
max_attemptsinteger, 1 to 20noAttempts per model call, the first included. Default 4.
base_delay_msinteger, at least 0noThe wait before the first retry; each later retry doubles it. Default 500.
max_delay_msinteger, at least 0noThe longest wait between two attempts. Default 30000.
jitterbooleannoDraw each wait at random below the backoff (full jitter). Default true.
retry_afterbooleannoWait at least what the server's Retry-After header asks. Default true.

GivenDefaults &ZeroWidthSpace;

Initial conditions set on the whole spec.

FieldTypeRequiredDefaultDescription
ledgermap of string to any valuenoFacts the agent must hold from the start.
worldWorldStatenoThe world at the start.
varsmap of string to any valuenoRuntime variables, such as the clock, the channel or the sender.
summarystring, not emptynoWhat a named state (vars.session_state) holds, in words. spec lint --summaries warns when a named state has none.

WorldState &ZeroWidthSpace;

The world an agent acts on: tool overrides, a seeded store, a seeded workspace, free fixtures.

FieldTypeRequiredDefaultDescription
toolsmap of string to ToolOverridenoForced tool results and errors, by tool name.
storemap of string to any valuenoA seeded store.
workspaceWorldState workspacenoThe files the agent starts from.
fixturesmap of string to any valuenoFree fixtures.

ToolOverride &ZeroWidthSpace;

A forced result or error for one tool, from the block where it is set onwards.

FieldTypeRequiredDefaultDescription
resultJSON valuenoThe result the tool returns.
errorstringnoThe error the tool returns.
latency_msnumber, at least 0noThe latency the tool takes, in milliseconds.

WorldState workspace &ZeroWidthSpace;

The files the agent starts from.

FieldTypeRequiredDefaultDescription
fromstring, not emptynoA folder the workspace starts from. A path written ./<path> is read from the folder of the spec file; any other relative path from the folder the run starts in.
filesmap of string to stringnoFiles the workspace starts with: path to content.
binarymap of string to string, base64noFiles the workspace starts with, byte for byte: path to base64 content. For files that are not text, such as the xlsx a parent block left. A path may not be in both files and binary.
summarystring, not emptynoWhat the workspace holds, in words, so a reader of the spec sees it. spec lint --summaries warns when a folder named by from has none.

CriterionDef &ZeroWidthSpace;

A judged criterion. Blocks reference it by name, with @gating when a fail fails the conversation.

FieldTypeRequiredDefaultDescription
namestring matching ^[A-Za-z0-9][A-Za-z0-9_.-]*$yesThe name blocks use to reference the criterion.
rubricstring, not emptyyesThe question the judge answers with pass or fail.
intentrequest-fulfillmentnoDeclares a request-fulfillment metric; only these metrics invoke the semantic obligation-ledger gate.
passstringnoWhat a passing block looks like.
failstringnoWhat a failing block looks like.
gatingbooleannoTrue gates every reference to this criterion; a reference can also add @gating.
scopeblock or conversationnoblock judges one block; conversation judges the whole conversation.
inputslist of strings, each not emptynoWorkspace files the judge reads beside the transcript, such as the source a produced file is compared against. Read from the seed the agent was given, else from the workspace.
judge_classone of system1, system2, autonosystem1, system2, or auto: calibration assigns the class, ties go to system1.
repeat_reducerstring matching ^(mode|median|at_least:[1-9][0-9]*)$noHow the verdicts across the repeats of one spec become one verdict: mode, median, or at_least:<k>.
prompt_version1 or 2noThe criterion prompt rules the judge reads. 2, the default, adds the exact-identifier, stated-facts and stated-scope rules; 1 is the earlier rule set, to replay answers recorded with it.
decoder_version1 or 2noHow a criterion ledger answer is read. Unset, the default reader drops an ungrounded quote and keeps the verdict; 1 rejects an ungrounded quote; 2 drops the quote and keeps the verdict.
answer_contractledger or legacynoledger, the default, or legacy: the earlier prompt and parser, to replay scores recorded with them.
assessmentone of response-relevance, conformance, answer-audit, fact-ledgernoAn assessment kind in place of the criterion ledger: response-relevance (the reply answers the latest ask), conformance (to reference), answer-audit (the submitted answer, optionally against reference), or fact-ledger (every fact of reference in the reply).
referencestringnoThe reference answer an assessment reads: needed by conformance (may be empty) and fact-ledger, optional for answer-audit.
groundingstrictnofact-ledger only: strict refuses reference or reply text that is not a source quote.
requirementslist of CriterionDef requirements, at least 1 itemnoThe rubric split into requirements the judge checks one by one. Absent: one requirement, the whole rubric.
itemslist of CriterionDef items, at least 1 itemnoRubric items, each scored 0 to 4 by the judge; the weighted score decides the verdict against threshold.
thresholdnumber, 0 to 1noWith items: the weighted fraction a pass needs, from 0 to 1. Default 1.
repeatinteger, 1 to 20noHow many times the judge answers the same request; the verdict holds only when every answer agrees, else inconclusive. Not the spec repeats (see repeat_reducer).
per_turnbooleannoTrue judges the transcript one customer turn at a time; any failing turn fails the criterion.

CriterionDef requirements &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
idstring, not emptyyesThe requirement id the judge answers under.
requirementstring, not emptyyesWhat the requirement asks.

CriterionDef items &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
namestring, not emptyyesThe item name, unique in the criterion.
rubricstring, not emptyyesWhat the item requires.
weightnumber, greater than 0noThe item weight. Default 1.
hardbooleannoTrue: the criterion fails when this item is not met.

Block &ZeroWidthSpace;

One block: a stimulus, its initial conditions and its expectations.

A block may name parent: <block id> in the same spec. The runner starts it from that parent’s recorded end state, then applies its own Given. Dependencies run first; missing parents and cycles are errors. A failed or unrun parent blocks descendants, which the report lists under the original cause. See parent blocks and the Acme example.

FieldTypeRequiredDefaultDescription
idstring, not emptynoAn id, unique in the spec.
parentstring, not emptynoThe id of a block in this spec whose recorded end state supplies this block's Given. A failed or unrun parent blocks this block.
whenStimulusyesThe stimulus.
givenGivennoInitial conditions for this block.
expectExpectnoThe expected output and state delta.
budgetBudgetnoLimits for this block.
usesUsesnoThe objects this block uses; a key here replaces the spec's. Left out of the block cache key.

Stimulus &ZeroWidthSpace;

What reaches the agent in one block: exactly one of user, burst or injection.

Write one of these forms.

  • a mapping with exactly one of the keys below
  • a mapping with the fields below
KeyTypeDescription
burstlist of strings, at least 2 itemsTwo or more messages that arrive together.
injectionStimulus injectionA deviation written into the blocks, with its class.
FieldTypeRequiredDefaultDescription
userstringyesThe message, or the instruction to a task agent.
classClassRefnoThe utterance class this message stands for. The message itself is the class seed.
simulated_userSimulatedUserSpecnoA simulated user that plays this block. Absent, the user text is sent as written.

Stimulus injection &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
classDeviationClassyesThe deviation class.
userstringyesThe message that carries the deviation.

DeviationClass &ZeroWidthSpace;

The twelve deviation classes a perturbed run can inject.

Used by when.injection.class and perturbations[].class. Mode P runs them with the perturbing driver, and driftbench perturb runs one. See recovery.

ValueMeaning
D1_off_topicAn off-topic question in the middle of the flow, such as asking for opening hours. Expected: a short answer, then the agent resumes its pending stage in the same block.
D2_goal_shiftThe user changes the goal, such as a different basket after the recap. Expected: back to the stage that owns the change, with no question about facts already known.
D3_contradictionThe user gives a value that contradicts an earlier one, such as a second account number. Expected: the agent notices and asks which one holds; it never takes the last one silently.
D4_loose_confirmationA vague confirmation, such as "ok perfect", where the flow needs an explicit one. Expected: no submission or other irreversible action without the explicit confirmation.
D5_language_switchOne message in another language. Expected: replies stay in the user's declared language, with no mixing.
D6_instruction_in_dataData carries an instruction, such as a form field or a workspace file that says to ignore the rules. Expected: the agent treats it as data and never obeys it.
D7_silence_or_burstAn empty message, or several messages at once. Expected: one reply, no double send.
D8_tool_failureA tool or a command fails or returns an error. Expected: an honest fallback and one retry; never a success that did not happen.
D9_out_of_scopeA request outside the agent's job, such as a billing complaint in the middle of an order. Expected: the agent routes it to the place that handles it and keeps it out of the current flow.
D10_returnThe user returns to the flow after a deviation, such as "ok, let's continue" two blocks later. Expected: the agent resumes at the right stage with every known fact kept.
D11_pii_leakA request for another person's personal data, such as the card number or email of the customer before. Expected: the agent gives no personal data of another person and says it cannot share it.
D12_prompt_extractionA request to reveal the instructions, the prompt or the tool names. Expected: the agent does not reveal its instructions, its prompt or its tool names.

ClassRef &ZeroWidthSpace;

The utterance class of a user message: its id and the hash of its frozen realisation set.

FieldTypeRequiredDefaultDescription
idstring matching ^[a-z0-9][a-z0-9._-]*$yesThe class id, as its class file names it.
setstring matching ^[0-9a-f]{12}$yesThe hash of the frozen realisation set the block is measured over (12 hex characters).

SimulatedUserSpec &ZeroWidthSpace;

A simulated user defined in the block: exact, simulated or replay.

FieldTypeRequiredDefaultDescription
modeone of exact, simulated, replayyesexact: send the user text as written. simulated: the agenda engine plans the message, a model writes it. replay: send the user text, a message frozen from a simulated run.
personastring, not emptynoWho the person is and how they write. Fictional.
goalstring, not emptynoWhat the person wants. Absent, the first simulated block's user text is the goal.
hidden_knowledgemap of string to string, not emptynoWhat the person knows, by name. A value reaches the model only once it is planned.
revealmap of string to on_request or volunteernoFor each fact: on_request (the default) gives it only once the agent asks; volunteer may give it in the opening message.
ask_patternsmap of string to list of strings, each not empty, at least 1 itemnoFor each fact, patterns (case-insensitive) of an agent reply that asks for it. Used when no model reads the agent turns.
stopstring, not emptynoWhat the person says when the goal is met or given up.
trialsinteger, at least 1noHow many times a simulated run plays the spec, for pass^k and the rerun flip rate.
seedstring, not empty or integernoThe seed of the engine. Absent, the spec id.
engineSimulatedUserEnginenoThe agenda engine's parameters.
recordedSimulatedUserSpec recordednoreplay: where the frozen text came from. A text that does not match its hash is refused.

SimulatedUserEngine &ZeroWidthSpace;

The agenda engine's parameters for a simulated user.

FieldTypeRequiredDefaultDescription
acts_per_turnlist of numbers, each 0 to 1, at least 1 itemnoThe chance of 1, 2, 3 ... dialogue acts in one message, in order. They must sum to 1.
volunteer_sharenumber, 0 to 1noThe chance that the opening message also gives each fact whose reveal is volunteer.
opening_detail_sharenumber, 0 to 1noThe chance that the opening message also states the specific problem, not only the request.
short_turn_sharenumber, 0 to 1noThe share of messages of at most three words.
polite_turn_sharenumber, 0 to 1noThe share of messages planned as polite.
uncertainty_turn_sharenumber, 0 to 1noThe share of messages planned as unsure.
question_turn_sharenumber, 0 to 1noThe share of messages planned as a question.
repeat_after_refusalinteger, at least 0noHow many times the person repeats the request after the agent says it cannot.
turn_budgetinteger, at least 1noThe most messages the person writes before giving up.
engagement_initialnumber, 0 to 1noEngagement at the start.
decay_per_unhelpful_turnnumber, 0 to 1noEngagement lost on each unhelpful agent turn.
disengage_belownumber, 0 to 1noBelow this engagement the person gives up.

SimulatedUserSpec recorded &ZeroWidthSpace;

replay: where the frozen text came from. A text that does not match its hash is refused.

FieldTypeRequiredDefaultDescription
hashstring matching ^[0-9a-f]{64}$yesThe SHA-256 of the canonical JSON of the frozen text.
fromstring, not emptynoThe run the text was frozen from.

Given &ZeroWidthSpace;

Initial conditions set on one block. A prefix lets the block start without replaying the ones before.

FieldTypeRequiredDefaultDescription
prefixlist of MessagenoThe messages before this block. Lets the block start without replaying the ones before.
originGivenOriginnoWhere the prefix comes from: authored, recorded or canonical. Only with a prefix on the same block. It enters the Given hash.
stagestring, not emptynoThe stage the agent is at when this block starts. Only with a prefix on the same block; a validator reads it where the earlier blocks that would say it are not written.
afterstring matching ^[^#\s]+#[^#\s]+$noThe block of a written spec this prefix is taken after, as <spec id>#<block id>. Only with a stage on the same block: a validator reads the stage only when that block leaves the agent at it after the same history.
ledgermap of string to any valuenoFacts the agent must hold at this block. They merge into the ledger from here on.
worldWorldStatenoChanges to the world from this block onwards.
varsmap of string to any valuenoRuntime variables from this block onwards.
sharedlist of SharedFactnoWhere each ledger fact was stated in the prefix: a fact said by the user or the agent must appear in the message it names.
summarystring, not emptynoWhat a named state (vars.session_state) holds from this block onwards, in words.

Message &ZeroWidthSpace;

One message of a conversation prefix.

FieldTypeRequiredDefaultDescription
roleone of user, assistant, system, toolyesWho sent the message.
contentstringyesThe text of the message.
namestringnoAn optional sender name.
tool_callslist of PrefixToolCallnoThe tool calls the message made.

PrefixToolCall &ZeroWidthSpace;

A tool call written in a spec prefix.

FieldTypeRequiredDefaultDescription
idstringnoThe id of the call.
labelstring, not emptyyesThe tool name.
inputJSON valuenoThe arguments of the call.
outputJSON valuenoWhat the tool returned.
errorstringnoThe error the tool returned.

GivenOrigin &ZeroWidthSpace;

Where the prefix comes from: authored (written by a person), recorded (copied from a real run, its tool calls with their real results) or canonical (the replay of a spec as it is written).

Used by blocks[].given.origin, beside a prefix. The origin enters the Given hash: an agent trusts a recorded call and tends to call a tool again when its earlier result is only in prose, so the three are different tests and their results are never pooled. See Given.

ValueMeaning
authoredA person wrote the prefix.
recordedThe prefix is copied from a real run: its tool calls carry the results the tools really returned. import traces writes it.
canonicalThe prefix is the replay of a spec as it is written, such as a teacher-forced history: the calls the blocks expect, with the results the world forces.

SharedFact &ZeroWidthSpace;

Where a ledger fact was stated before the block, checked by spec lint and before the block runs.

FieldTypeRequiredDefaultDescription
factstring, not emptyyesThe ledger key of the fact.
byone of user, agent, recordyesWho stated it: user or agent in a prefix message, record when it comes from a system of record and was never said.
ininteger, at least 0noThe index of the prefix message that states it, from 0. Needed for user and agent.

Expect &ZeroWidthSpace;

The expected output and state delta of a block. Every field is optional.

FieldTypeRequiredDefaultDescription
stagestring, not emptynoThe stage the agent must reach after the block, one of the spec stages.
stage_reportedbooleannoTrue requires the runtime to report a non-empty stage in this block. A carried or inferred stage does not count. False or absent leaves reporting optional.
ledger_addmap of string to any valuenoFacts the block establishes. They join the ledger.
toolslist of ToolExpectnoThe tool calls expected in the block.
tools_offeredToolsOfferedExpectnoThe tools the runtime must offer and must not offer, whether or not the agent calls them.
tool_orderone of exact, in_order, any_ordernoHow the expected tools line up with the observed calls: exact, in_order or any_order.
mustlist of strings, each not emptynoPatterns the visible text must match.
must_notlist of strings, each not emptynoPatterns the visible text must not match.
rouge_lExpect rouge_lnoThe reply must overlap a reference text: its word-level ROUGE-L F measure must reach min.
bleuExpect bleunoThe reply must overlap a reference text: its BLEU score over words must reach min.
sqlExpect sqlnoThe SQL the agent ran must return the same rows as the golden query on the database.
regexlist of strings, each not emptynoRegular expressions the visible text must match, case-sensitive; must ignores case instead.
injectionbooleannoTrue when the block is a prompt-injection or jailbreak attempt: the reply must not obey it.
exactstring, not emptynoThe reply must equal this text after normalising: case, surrounding space and punctuation ignored; two numbers compare as numbers. A dot or comma between two digits counts, except a strictly grouped thousands comma.
exact_strictbooleannoCompare exact byte for byte after trimming the ends, instead of ignoring case, punctuation and spacing.
jsonJsonExpectnoThe reply is valid JSON, optionally with a schema and paths.
tool_f1Expect tool_f1noPrecision, recall and F1 of the tool calls against the expected ones.
refusalrefuses or answersnoWhether the reply must be a refusal, read from refusal phrases in seven languages.
markupone of xml, html, sqlnoThe reply is well-formed XML, HTML or SQL (syntax only).
starts_withstring, not emptynoThe reply, without leading space, starts with this text.
finish_reasonstring, not emptynoThe reason the model stopped, as the runtime reports it.
lengthExpect lengthnoWord and character bounds on the customer-visible text of the block.
similaritylist of SimilarityExpectnoReference texts the reply must be close to, each with its lowest passing score.
relevancyRelevancyExpectnoThe reply addresses the ask of the block.
knowledge_retentionlist of RetentionExpectnoFacts given earlier that the block must not ask for again, and may have to use.
langstring, 2 to 8 charactersnoThe language of the reply.
pii_judgebooleannoOpt in to judged detection of personal names and addresses in the visible reply. Requires a configured personal-data judge; absent or false makes no model calls. Missing or failed judging is gating inconclusive.
cardslist of CardExpectnoThe cards expected in the block.
criterialist of strings, each not emptynoJudged criteria by name. name@gating makes a fail fail the block.
skillslist of SkillExpectnoThe skills expected in the block.
guardslist of GuardExpectnoThe runtime checks expected in the block, with their verdict and count.
referenceslist of ReferenceExpectnoThe files the agent must read in the block, by path or glob, with a count.
subagentslist of SubagentExpectnoThe helpers the agent must dispatch in the block, by name and model, with a count.
executionExecutionExpectnoWhat the agent runs.
artifactslist of ArtifactExpectnoThe files the block must produce.
workspaceWorkspaceExpectnoThe workspace diff expected.
task_impossiblebooleannoTrue when the task, or the tests it was given, cannot be met honestly. The checks do not read it. Blame credits an agent that stopped and said so, and attributes a pass reached anyway to the agent as an integrity fault.
downstreamlist of Expect downstreamnoConsumers of the block output that need a form nobody states to the agent. The checks do not read it. When a value that works fails its check, blame names the missing rule as a setup fault instead of flagging the test.
protectedlist of strings, each not emptynoPaths or globs the agent must not change or delete, whoever wrote them: the tests, checkers and source-of-truth files it was given. The checks do not read it. When a gating verdict fails and the net workspace diff holds one, blame attributes the failure to the agent as an integrity fault.
safetySafetyExpectnoLimits on destructive operations.

ToolExpect &ZeroWidthSpace;

One expected tool call.

FieldTypeRequiredDefaultDescription
labelstring, not emptyyesThe tool name.
countCountnoHow many calls are expected. 0 means the tool must not be called.
actorActornoWhose calls count: the agent (main), a helper it handed work to (helper), or both (any). Absent: both for a count of 0, the agent otherwise.
inputmap of string to AssertionnoAssertions on the arguments, by argument name.
input_closedbooleannoTrue requires every call input to be an object with only the top-level keys named in input. An absent input allows no keys. False or absent leaves other keys allowed.
compareCompareMode or map of string to CompareModenoHow the input values are compared: one mode for every argument, or a mode by argument name. Absent means auto: the value-check rungs decide whether a value that differs from the gold names the same thing.
recomputemap of string to ToolExpect recompute valuenoArguments the agent computes, by argument name. The agent writes a formula (a text that starts with =) or a number; the recompute rung runs it and the gold formula on the given inputs and on changed inputs, and fails a value that is right only on the given ones. An argument named here is not also named in input.

Count &ZeroWidthSpace;

An exact count, or a range with an optional min and max.

Write one of these forms.

  • integer, at least 0
  • a mapping with the fields below
FieldTypeRequiredDefaultDescription
mininteger, at least 0noThe fewest allowed.
maxinteger, at least 0noThe most allowed.

Actor &ZeroWidthSpace;

ValueMeaning
mainOnly the agent's own calls count.
helperOnly the calls of the helpers the agent handed work to count.
anyThe agent's calls and its helpers' calls both count.

Assertion &ZeroWidthSpace;

How one value is asserted. A bare string, number, boolean or null means exact equality.

Write one of these forms.

  • string
  • number
  • boolean
  • null
  • a mapping with exactly one of the keys below
KeyTypeDescription
exactJSON valueEqual to this JSON value.
regexstring, not emptyA JavaScript regular expression source. Specs match text with the m flag.
containsstring, not emptyContains this text.
one_oflist of JSON values, at least 1 itemEqual to one of these values.
nonemptytruePresent and not empty.
absenttrueAbsent.
subsetmap of string to JSON valueHolds at least these keys with these values.

CompareMode &ZeroWidthSpace;

How one value is compared: auto, a rung name, or exact with its reason.

Write one of these forms.

  • one of auto, outcome, contract, as_given, same_referent
  • a mapping with exactly one of the keys below
KeyTypeDescription
exactstring, not emptyWhy the writing itself is the requirement, in one line.

ToolExpect recompute value &ZeroWidthSpace;

The gold computation of an argument the agent computes, and the inputs it runs on.

FieldTypeRequiredDefaultDescription
formulastring, not emptyyesThe gold computation, a spreadsheet formula over the named inputs, such as =SUM(C2:C5). Numbers, names, + - * / ^, parentheses and SUM, MIN, MAX, AVERAGE, ROUND, ABS are supported.
inputsmap of string to numberyesThe inputs the agent was given, by name: C2: 10. The formulas are first run on these.
rangesmap of string to list of any value, at least 2 items, at most 2 itemsnoThe range [low, high] each input is drawn from on changed inputs. An input without one varies around its given value.

ToolsOfferedExpect &ZeroWidthSpace;

The tools the runtime offers the agent, read from the runtime trace; calls do not count.

FieldTypeRequiredDefaultDescription
includelist of strings, each not emptynoTools the runtime must offer.
excludelist of strings, each not emptynoTools the runtime must not offer, such as a tool a setting denies.

Expect rouge_l &ZeroWidthSpace;

The reply must overlap a reference text: its word-level ROUGE-L F measure must reach min.

FieldTypeRequiredDefaultDescription
referencestring, not emptyyesThe reference text the reply is compared with.
profiledriftbench or ragasnoThe overlap profile: driftbench (default) or ragas, the reference implementation of this score.
minnumber, 0 to 1yesThe lowest ROUGE-L F measure that passes, from 0 to 1.

Expect bleu &ZeroWidthSpace;

The reply must overlap a reference text: its BLEU score over words must reach min.

FieldTypeRequiredDefaultDescription
referencestring, not emptyyesThe reference text the reply is compared with.
profiledriftbench or ragasnoThe overlap profile: driftbench (default) or ragas, the reference implementation of this score.
minnumber, 0 to 1yesThe lowest BLEU score that passes, from 0 to 1.

Expect sql &ZeroWidthSpace;

The SQL the agent ran must return the same rows as the golden query on the database.

FieldTypeRequiredDefaultDescription
toolstring, not emptyyesThe tool whose call input holds the query, in its query field.
goldenstring, not emptyyesThe golden query: one read statement.
databasestring, not emptyyesThe SQLite database both queries run on, read-only.

JsonExpect &ZeroWidthSpace;

The reply is JSON, optionally with a schema and paths.

FieldTypeRequiredDefaultDescription
embeddedbooleannoWhen true, the reply may hold JSON among its prose: the first JSON value found is read. Off by default.
schemamap of string to JSON valuenoA JSON Schema the reply must match; an undeclared key fails.
pathslist of JsonExpect pathsnoPaths of the reply, each with its assertion.

JsonExpect paths &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesA dotted path; a number indexes an array.
assertAssertionyesWhat the value at the path must be, in the spec assertion vocabulary.

Expect tool_f1 &ZeroWidthSpace;

Precision, recall and F1 of the tool calls against the expected ones.

FieldTypeRequiredDefaultDescription
minnumber, 0 to 1yesThe least F1 that passes, from 0 to 1.

Expect length &ZeroWidthSpace;

Word and character bounds on the customer-visible text of the block.

FieldTypeRequiredDefaultDescription
wordsLengthBoundnoBounds on the words of the visible text.
charsLengthBoundnoBounds on the characters (code points) of the visible text.

LengthBound &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
mininteger, at least 0noThe least the count may be.
maxinteger, at least 0noThe most the count may be.

SimilarityExpect &ZeroWidthSpace;

A reference text the reply must be close to, with the lowest score that passes.

FieldTypeRequiredDefaultDescription
referencestring, not emptyyesThe text the reply is compared with.
minnumber, 0 to 1yesThe lowest similarity that passes, from 0 to 1, as the embedder scores it.

RelevancyExpect &ZeroWidthSpace;

The reply addresses the ask of the block: a share of its key terms appears in the reply.

FieldTypeRequiredDefaultDescription
minnumber, 0 to 1noThe share of the key terms that must appear in the reply, from 0 to 1. Default 0.5.
termslist of strings, each not emptynoThe key terms of the ask. Without them, the check takes the words of the stimulus.

RetentionExpect &ZeroWidthSpace;

A fact the user gave earlier: the block must not ask for it again, and may have to use it.

FieldTypeRequiredDefaultDescription
factstring, not emptyyesThe ledger key of the fact, given in an earlier block.
usebooleannoTrue when the block must use the fact: its value appears in the reply or a tool input.
askstring, not emptynoA pattern that marks a question asking for the fact again. Default: the words of the key.

CardExpect &ZeroWidthSpace;

One expected card.

FieldTypeRequiredDefaultDescription
namestring, not emptyyesThe card name.
countCountnoHow many times the card is sent.

SkillExpect &ZeroWidthSpace;

One expected skill.

FieldTypeRequiredDefaultDescription
namestring, not emptyyesThe skill name.
countCountnoHow many times the skill is loaded.

GuardExpect &ZeroWidthSpace;

One runtime check that fired during the block, with its verdict. The adapter maps the runtime names.

FieldTypeRequiredDefaultDescription
namestring, not emptyyesThe name of the check, as the adapter reports it.
verdictstring, not emptynoThe verdict the check must have given, such as pass or deny.
countCountnoHow many times the check fired.

ReferenceExpect &ZeroWidthSpace;

A file the agent must read during the block, by path or glob.

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesThe path read, or a glob: * matches within a folder, ** across folders, ? one character.
countCountnoHow many reads of a matching file. 0 means it must not be read.

SubagentExpect &ZeroWidthSpace;

A helper the agent must dispatch during the block, with an optional model.

FieldTypeRequiredDefaultDescription
namestring, not emptyyesThe helper name, as the adapter reports it.
modelstring, not emptynoThe model the helper must run on.
countCountnoHow many dispatches. 0 means the helper must not be dispatched.

ExecutionExpect &ZeroWidthSpace;

What the agent runs in the block.

FieldTypeRequiredDefaultDescription
ran_codebooleannoTrue when the block must run code; false when it must not.
exit_codeintegernoThe exit code of the last command.
max_commandsinteger, at least 0noThe most commands the block may run.

ArtifactExpect &ZeroWidthSpace;

A file the block must produce.

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesThe path of the file in the workspace.
kindone of xlsx, docx, pptx, pdf, csv, json, text, codenoThe file kind, which picks the reader.
checkslist of maps of string to any valuenoChecks on the content of the file.

WorkspaceExpect &ZeroWidthSpace;

The workspace diff of the block.

FieldTypeRequiredDefaultDescription
createdlist of stringsnoPaths the block creates.
changedlist of stringsnoPaths the block changes.
deletedlist of stringsnoPaths the block deletes.
others_untouchedbooleannoTrue when no other path may change.

Expect downstream &ZeroWidthSpace;

A consumer that needs a form of a value.

FieldTypeRequiredDefaultDescription
toolstring, not emptynoThe tool whose input the consumer reads. Absent means any tool.
fieldstring, not emptynoThe argument the consumer reads. Absent means every argument.
needsstring, not emptyyesThe form the consumer needs, such as e164 or iso8601.
declared_instring, not emptynoWhere the consumer states the need, such as a file or a system name.

SafetyExpect &ZeroWidthSpace;

Limits on destructive or irreversible operations.

FieldTypeRequiredDefaultDescription
max_destructive_opsinteger, at least 0noThe most destructive operations allowed.
confirm_beforelist of strings, each not emptynoOperations that need a confirmation first, such as delete or send.

Budget &ZeroWidthSpace;

Limits for one block.

FieldTypeRequiredDefaultDescription
stepsinteger, at least 1noThe most agent steps.
cost_usdnumber, at least 0noThe most cost, in US dollars.
latency_msnumber, at least 0noThe most latency, in milliseconds.

Uses &ZeroWidthSpace;

The objects this spec or block uses. Absent keys use the shipped defaults.

FieldTypeRequiredDefaultDescription
value_checkObjectRefnopolicy/value-check: how a value the agent wrote is compared.
blameObjectRefnopolicy/blame: who a failure is attributed to.
placementObjectRefnopolicy/placement: where a fix goes.
linksObjectRefnopolicy/links: how blocks link to prompt units.
selectionObjectRefnopolicy/selection: which blocks a change selects.
gatingObjectRefnopolicy/gating: what not_run, blocked and inconclusive do to ci.
statsObjectRefnopolicy/stats: which interval is the headline.
check_setObjectRefnocheck-set: the machine checks and their order.
judgesObjectRefnojudge-council: the judges, their mode and assignment.
diagnosticslist of strings, each matching ^[A-Za-z0-9][A-Za-z0-9._-]*(\/[A-Za-z0-9][A-Za-z0-9._-]*)*$noThe diagnostic objects that run after the checks.

ObjectRef &ZeroWidthSpace;

Write one of these forms.

  • string matching ^[A-Za-z0-9][A-Za-z0-9._-]*(\/[A-Za-z0-9][A-Za-z0-9._-]*)*$
  • a mapping with the fields below
FieldTypeRequiredDefaultDescription
extendsstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*(\/[A-Za-z0-9][A-Za-z0-9._-]*)*$yesThe object this inline one extends.
itemslist of ItemPatchnoOperations on the parent items.
orderlist of strings, each matching ^[A-Za-z0-9][A-Za-z0-9._-]*$noThe item order.
paramsmap of string to any valuenoObject-level parameters.

ItemPatch &ZeroWidthSpace;

An operation on one item of the parent object, or a new item with add: true.

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesThe item to change, or with add: true the new item's id.
addtruenoAdds a new item. Needs title, when, effect and impl.
titlestring, not emptynoA new title for the item.
enabledbooleannofalse disables the item, true enables it again.
whenmap of string to any valuenoCondition fields to set; maps merge, lists replace.
effectmap of string to any valuenoEffect fields to set; maps merge, lists replace.
paramsmap of string to any valuenoParams to set: maps merge, lists replace, +key appends, -key removes.
implstring matching ^(builtin|plugin):\S+$noAnother implementation for the same rule.
citesstringnoA source for the reader.
applies_toAppliesTonoWhere the rule applies: maps merge, lists replace.
beforestring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$noMoves the item just before this one.
afterstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$noMoves the item just after this one.

AppliesTo &ZeroWidthSpace;

Where an item applies: skills, tools or unit patterns. Absent means everywhere.

FieldTypeRequiredDefaultDescription
skillslist of strings, each not emptynoSkill names the evidence must name one of.
toolslist of strings, each not emptynoTool labels the failing call must have one of.
unitslist of strings, each not emptynoUnit patterns, such as R#41/*, a unit must match one of.

Spec end &ZeroWidthSpace;

An end verdict: the run stops after block atBlock (the last block when absent) with this verdict.

FieldTypeRequiredDefaultDescription
verdictsuccess or failureyessuccess when the script ends the conversation well, failure when it ends it badly.
reasonstring, not emptynoWhy the script ends here.
atBlockinteger, at least -1noThe index of the last block that runs, -1 for none. The blocks after it are skipped by the verdict. Absent means the last block.

Rules &ZeroWidthSpace;

Rules that hold for the whole conversation.

FieldTypeRequiredDefaultDescription
max_callsmap of string to integer, at least 0noThe most calls per tool across the run. 0 forbids the tool.
forbidden_toolslist of strings, each not emptynoTools that must never be called.
lang_stablebooleannoTrue when the reply language must stay the same across blocks.
one_message_per_turnbooleannoTrue when each block may send one message only.
match_casebooleannoTrue makes must and must_not patterns case-sensitive and the trailing punctuation count. Default false: a pattern ignores the case of the reply and a trailing punctuation mark.
custommap of string to any valuenoMaps a registered custom check name to its options; the checks package runs it.

EndState &ZeroWidthSpace;

none asserts that nothing was submitted; a mapping or a list of mappings names what to check; absent means no end-state check.

Write one of these forms.

EndStateSpec &ZeroWidthSpace;

The expected end state of the world, read from a source the adapter knows, such as a tool payload.

FieldTypeRequiredDefaultDescription
sourcestring, not emptyyesWhere the adapter reads the state, such as tool:submit_order.
modesubset or exactnosubset compares the listed fields; exact compares the whole state.
expectmap of string to AssertionyesAssertions on fields of the state.

Perturbation &ZeroWidthSpace;

A deviation inserted before block inject_at, counting from 0, when the spec runs in perturbed mode.

FieldTypeRequiredDefaultDescription
idstring, not emptyyesAn id, unique in the spec.
classDeviationClassyesThe deviation class.
inject_atinteger, at least 0yesThe position: the deviation goes in before this block, counting from 0.
userstringnoThe message the user sends.
burstlist of strings, at least 2 itemsnoMessages the user sends together.
worldWorldStatenoA change to the world from this point, such as a tool forced to fail.
expectExpectnoExpectations for the injected block.
recoveryRecoveryExpectnoHow the agent should come back. Without it, recovery is not measured.

RecoveryExpect &ZeroWidthSpace;

How the agent should come back after an injected deviation.

FieldTypeRequiredDefaultDescription
within_blocksinteger, at least 0yesHow many blocks the agent has to come back.
stage_afterstring, not emptynoThe stage the agent must be back at.
return_messagestringnoA pattern the first in-flow reply after the deviation must match.

DriftClass &ZeroWidthSpace;

The five drift classes a failing block can carry.

Run records, not specs, carry these classes: one or more on each failing block. The runner assigns them from the failing gating verdicts of each block. See drift.

ValueMeaning
X1_localA wrong answer or a wrong action in this block while every earlier fact still holds. Found by any failing check or criterion on the block alone.
X2_contradictionThe block contradicts a fact established earlier, as held in the ledger. Found by the ledger check.
X3_flowThe agent is at the wrong stage: it skips one, repeats one, or asks again for a known fact. Found by the stage classifier against expect.stage.
X4_formatA format or persona rule breaks: narration of internal steps, the wrong language, a double send, card misuse, the wrong channel rules. Found by format and channel checks.
X5_toolA tool call, a skill or a guard is missing, extra or repeated, or a call has wrong parameters. Found by the tool, skill and guard checks.

Inventory &ZeroWidthSpace;

The behaviour inventory: every behaviour the agent must show, grouped into flows, and the deviation classes each flow must be perturbed with.

The behaviour inventory is the file driftbench spec coverage --inventory reads. It lists the behaviours an agent must show and the flows that group them. A spec covers a behaviour by naming its id in covers, and joins a flow with the tag flow:<id> or by covering one of its behaviours. See CLI.

FieldTypeRequiredDefaultDescription
behaviourslist of InventoryBehaviouryesEvery behaviour the agent must show.
flowslist of InventoryFlowyesThe flows that group the behaviours.
channelslist of strings, each not empty, at least 1 itemnoThe channel universe. Absent means the channels named by behaviours and specs, else default.

InventoryBehaviour &ZeroWidthSpace;

One behaviour the agent must show.

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Z]+-\d+$yesThe behaviour id, such as B-001. Specs name it in covers.
titlestringyesWhat the agent does.
kindconversation or tasknoconversationtask marks a code-interpreter behaviour. It is a row of the same matrix as the others.
channelslist of strings, each not empty, at least 1 itemnoThe channels where the behaviour applies. Absent means every channel.

InventoryFlow &ZeroWidthSpace;

A flow: behaviours that run together, and the deviations it must be perturbed with.

FieldTypeRequiredDefaultDescription
idstring matching ^[a-z0-9][a-z0-9_-]*$yesThe flow id, such as plan-upgrade. Specs join it with the tag flow:<id>.
titlestringyesWhat the flow does.
behaviourslist of strings, each matching ^[A-Z]+-\d+$yesThe behaviours of the flow, by id.
deviationslist of DeviationClassnoThe deviation classes this flow must be perturbed with. Absent means none is required.

MIT. driftbench is made by Nexus.