Spec reference ​
A spec is one YAML or JSON file. It describes one conversation or one task as a succession of blocks; see block and Given, When, Then. This page lists every field the loader accepts, then the fields of the behaviour inventory that driftbench spec coverage reads. It is generated from the zod schemas that the CLI uses.
- Keys are snake_case. An unknown key is an error.
- Required means the field is present once the loader has applied the shorthand below.
- Descriptions come from the schemas. An empty cell means the schema has no description for that field yet.
Shorthand ​
The loader rewrites these forms before it validates a spec.
kindmay be left out: the loader setsconversation.- A block may write
user: <message>orburst: [<message>, <message>]in place ofwhen. ledger_seedat the top level is the same asgiven.ledger.workspace_seedat the top level is the same asgiven.world.workspace. A string means{ from: <string> }.
Contents ​
- Spec
- TargetRef
- TargetPrompt
- RetrySettings
- GivenDefaults
- WorldState
- ToolOverride
- WorldState workspace
- CriterionDef
- CriterionDef requirements
- CriterionDef items
- Block
- Stimulus
- Stimulus injection
- DeviationClass
- ClassRef
- SimulatedUserSpec
- SimulatedUserEngine
- SimulatedUserSpec recorded
- Given
- Message
- PrefixToolCall
- GivenOrigin
- SharedFact
- Expect
- ToolExpect
- Count
- Actor
- Assertion
- CompareMode
- ToolExpect recompute value
- ToolsOfferedExpect
- Expect rouge_l
- Expect bleu
- Expect sql
- JsonExpect
- JsonExpect paths
- Expect tool_f1
- Expect length
- LengthBound
- SimilarityExpect
- RelevancyExpect
- RetentionExpect
- CardExpect
- SkillExpect
- GuardExpect
- ReferenceExpect
- SubagentExpect
- ExecutionExpect
- ArtifactExpect
- WorkspaceExpect
- Expect downstream
- SafetyExpect
- Budget
- Uses
- ObjectRef
- ItemPatch
- AppliesTo
- Spec end
- Rules
- EndState
- EndStateSpec
- Perturbation
- RecoveryExpect
- DriftClass
- Inventory
- InventoryBehaviour
- InventoryFlow
Spec ​
One golden conversation, or one task for a code agent, written as a succession of blocks.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
schema_version | integer, 1 to 1 | no | The spec format version the spec is written for. A driftbench that reads a lower version refuses the spec and says to upgrade. Absent means version 1. | |
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | The spec id: letters, digits, dots, underscores and hyphens. | |
version | string | no | A free label. The content hash, not this label, is the cache key. | |
kind | conversation or task | yes | conversation, set by the loader | conversation or task. The loader sets conversation when it is absent. |
title | string | no | A short title. | |
description | string | no | A longer description. | |
channel | string | no | The channel the conversation runs on, such as chat. | |
lang | string, 2 to 8 characters | no | The declared language of the conversation. | |
tier | string | no | A tier label, such as core or edge. | |
tags | list of strings | no | Free tags. flow:<id> ties the spec to a flow of the behaviour inventory. | |
covers | list of strings | no | Ids of the behaviours this spec covers, from a behaviour inventory. | |
target | TargetRef | no | The default agent. A run can override it. | |
given | GivenDefaults | no | Initial conditions for every block. | |
stages | list of strings, each not empty | no | The stage catalogue of the flow. expect.stage and recovery.stage_after must name one of these. | |
criteria | list of CriterionDef | no | The judged criteria the blocks may reference. | |
blocks | list of Block, at least 1 item | yes | The blocks, in order. At least one. | |
end | Spec end | no | An end verdict: the run stops after block atBlock (the last block when absent) with this verdict. | |
rules | Rules | no | Rules for the whole conversation. | |
operations | map of string to one of read, modify, delete, overwrite, send, pay, push | no | The operation class of each tool the agent may call, by its exact label: read, modify (declared, not destructive), delete, overwrite, send, pay or push. The safety checks and the recovery metrics read a tool call by this table; a tool it does not name is unknown, never classified by its name, and a guard that needs its class is inconclusive on it. | |
end_state | EndState | no | The expected end state of the world after the run. | |
perturbations | list of Perturbation | no | Deviations to inject in perturbed mode. | |
uses | Uses | no | The objects every block uses (policies, check set, diagnostics), by id or inline. Left out of the block cache key. | |
metadata | map of string to any value | no | Free metadata. links holds the links of the blocks to prompt sections, spec clauses and past turns; it and provenance stay out of the spec hash. |
TargetRef ​
Which agent a spec runs against. The adapter decides what the other fields mean.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
adapter | string, not empty | yes | The adapter that reaches the agent, such as function, http, nexus or process. | |
agent | string, not empty | no | The agent, as the adapter names it. | |
variant | string, not empty | no | A variant of the agent, such as a prompt version. | |
deployment | string, not empty | no | A deployment of the agent, as the adapter names it. | |
options | map of string to any value | no | Adapter settings. Each adapter defines their names. | |
prompt | TargetPrompt | no | The prompt the agent runs with, from the registry. campaign run --prompt sets it; each adapter page says how it reaches the agent. | |
retry | RetrySettings | no | How the model calls of this target are retried. |
TargetPrompt ​
A prompt version from the registry, handed to the agent as its system prompt. campaign run --prompt sets it.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string, not empty | yes | The prompt name in the registry. | |
version | integer, at least 1 | yes | The version number. | |
hash | string matching ^[0-9a-f]{64}$ | yes | The version hash: the SHA-256 of its file list. | |
text | string | yes | The prompt text, restored byte for byte from the store. |
RetrySettings ​
How model calls are retried. Only a 408, a 429, a 5xx (529 included) or a network error is retried.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
max_attempts | integer, 1 to 20 | no | Attempts per model call, the first included. Default 4. | |
base_delay_ms | integer, at least 0 | no | The wait before the first retry; each later retry doubles it. Default 500. | |
max_delay_ms | integer, at least 0 | no | The longest wait between two attempts. Default 30000. | |
jitter | boolean | no | Draw each wait at random below the backoff (full jitter). Default true. | |
retry_after | boolean | no | Wait at least what the server's Retry-After header asks. Default true. |
GivenDefaults ​
Initial conditions set on the whole spec.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
ledger | map of string to any value | no | Facts the agent must hold from the start. | |
world | WorldState | no | The world at the start. | |
vars | map of string to any value | no | Runtime variables, such as the clock, the channel or the sender. | |
summary | string, not empty | no | What a named state (vars.session_state) holds, in words. spec lint --summaries warns when a named state has none. |
WorldState ​
The world an agent acts on: tool overrides, a seeded store, a seeded workspace, free fixtures.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
tools | map of string to ToolOverride | no | Forced tool results and errors, by tool name. | |
store | map of string to any value | no | A seeded store. | |
workspace | WorldState workspace | no | The files the agent starts from. | |
fixtures | map of string to any value | no | Free fixtures. |
ToolOverride ​
A forced result or error for one tool, from the block where it is set onwards.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
result | JSON value | no | The result the tool returns. | |
error | string | no | The error the tool returns. | |
latency_ms | number, at least 0 | no | The latency the tool takes, in milliseconds. |
WorldState workspace ​
The files the agent starts from.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
from | string, not empty | no | A folder the workspace starts from. A path written ./<path> is read from the folder of the spec file; any other relative path from the folder the run starts in. | |
files | map of string to string | no | Files the workspace starts with: path to content. | |
binary | map of string to string, base64 | no | Files the workspace starts with, byte for byte: path to base64 content. For files that are not text, such as the xlsx a parent block left. A path may not be in both files and binary. | |
summary | string, not empty | no | What the workspace holds, in words, so a reader of the spec sees it. spec lint --summaries warns when a folder named by from has none. |
CriterionDef ​
A judged criterion. Blocks reference it by name, with @gating when a fail fails the conversation.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string matching ^[A-Za-z0-9][A-Za-z0-9_.-]*$ | yes | The name blocks use to reference the criterion. | |
rubric | string, not empty | yes | The question the judge answers with pass or fail. | |
intent | request-fulfillment | no | Declares a request-fulfillment metric; only these metrics invoke the semantic obligation-ledger gate. | |
pass | string | no | What a passing block looks like. | |
fail | string | no | What a failing block looks like. | |
gating | boolean | no | True gates every reference to this criterion; a reference can also add @gating. | |
scope | block or conversation | no | block judges one block; conversation judges the whole conversation. | |
inputs | list of strings, each not empty | no | Workspace files the judge reads beside the transcript, such as the source a produced file is compared against. Read from the seed the agent was given, else from the workspace. | |
judge_class | one of system1, system2, auto | no | system1, system2, or auto: calibration assigns the class, ties go to system1. | |
repeat_reducer | string matching ^(mode|median|at_least:[1-9][0-9]*)$ | no | How the verdicts across the repeats of one spec become one verdict: mode, median, or at_least:<k>. | |
prompt_version | 1 or 2 | no | The criterion prompt rules the judge reads. 2, the default, adds the exact-identifier, stated-facts and stated-scope rules; 1 is the earlier rule set, to replay answers recorded with it. | |
decoder_version | 1 or 2 | no | How a criterion ledger answer is read. Unset, the default reader drops an ungrounded quote and keeps the verdict; 1 rejects an ungrounded quote; 2 drops the quote and keeps the verdict. | |
answer_contract | ledger or legacy | no | ledger, the default, or legacy: the earlier prompt and parser, to replay scores recorded with them. | |
assessment | one of response-relevance, conformance, answer-audit, fact-ledger | no | An assessment kind in place of the criterion ledger: response-relevance (the reply answers the latest ask), conformance (to reference), answer-audit (the submitted answer, optionally against reference), or fact-ledger (every fact of reference in the reply). | |
reference | string | no | The reference answer an assessment reads: needed by conformance (may be empty) and fact-ledger, optional for answer-audit. | |
grounding | strict | no | fact-ledger only: strict refuses reference or reply text that is not a source quote. | |
requirements | list of CriterionDef requirements, at least 1 item | no | The rubric split into requirements the judge checks one by one. Absent: one requirement, the whole rubric. | |
items | list of CriterionDef items, at least 1 item | no | Rubric items, each scored 0 to 4 by the judge; the weighted score decides the verdict against threshold. | |
threshold | number, 0 to 1 | no | With items: the weighted fraction a pass needs, from 0 to 1. Default 1. | |
repeat | integer, 1 to 20 | no | How many times the judge answers the same request; the verdict holds only when every answer agrees, else inconclusive. Not the spec repeats (see repeat_reducer). | |
per_turn | boolean | no | True judges the transcript one customer turn at a time; any failing turn fails the criterion. |
CriterionDef requirements ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string, not empty | yes | The requirement id the judge answers under. | |
requirement | string, not empty | yes | What the requirement asks. |
CriterionDef items ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string, not empty | yes | The item name, unique in the criterion. | |
rubric | string, not empty | yes | What the item requires. | |
weight | number, greater than 0 | no | The item weight. Default 1. | |
hard | boolean | no | True: the criterion fails when this item is not met. |
Block ​
One block: a stimulus, its initial conditions and its expectations.
A block may name parent: <block id> in the same spec. The runner starts it from that parent’s recorded end state, then applies its own Given. Dependencies run first; missing parents and cycles are errors. A failed or unrun parent blocks descendants, which the report lists under the original cause. See parent blocks and the Acme example.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string, not empty | no | An id, unique in the spec. | |
parent | string, not empty | no | The id of a block in this spec whose recorded end state supplies this block's Given. A failed or unrun parent blocks this block. | |
when | Stimulus | yes | The stimulus. | |
given | Given | no | Initial conditions for this block. | |
expect | Expect | no | The expected output and state delta. | |
budget | Budget | no | Limits for this block. | |
uses | Uses | no | The objects this block uses; a key here replaces the spec's. Left out of the block cache key. |
Stimulus ​
What reaches the agent in one block: exactly one of user, burst or injection.
Write one of these forms.
- a mapping with exactly one of the keys below
- a mapping with the fields below
| Key | Type | Description |
|---|---|---|
burst | list of strings, at least 2 items | Two or more messages that arrive together. |
injection | Stimulus injection | A deviation written into the blocks, with its class. |
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
user | string | yes | The message, or the instruction to a task agent. | |
class | ClassRef | no | The utterance class this message stands for. The message itself is the class seed. | |
simulated_user | SimulatedUserSpec | no | A simulated user that plays this block. Absent, the user text is sent as written. |
Stimulus injection ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
class | DeviationClass | yes | The deviation class. | |
user | string | yes | The message that carries the deviation. |
DeviationClass ​
The twelve deviation classes a perturbed run can inject.
Used by when.injection.class and perturbations[].class. Mode P runs them with the perturbing driver, and driftbench perturb runs one. See recovery.
| Value | Meaning |
|---|---|
D1_off_topic | An off-topic question in the middle of the flow, such as asking for opening hours. Expected: a short answer, then the agent resumes its pending stage in the same block. |
D2_goal_shift | The user changes the goal, such as a different basket after the recap. Expected: back to the stage that owns the change, with no question about facts already known. |
D3_contradiction | The user gives a value that contradicts an earlier one, such as a second account number. Expected: the agent notices and asks which one holds; it never takes the last one silently. |
D4_loose_confirmation | A vague confirmation, such as "ok perfect", where the flow needs an explicit one. Expected: no submission or other irreversible action without the explicit confirmation. |
D5_language_switch | One message in another language. Expected: replies stay in the user's declared language, with no mixing. |
D6_instruction_in_data | Data carries an instruction, such as a form field or a workspace file that says to ignore the rules. Expected: the agent treats it as data and never obeys it. |
D7_silence_or_burst | An empty message, or several messages at once. Expected: one reply, no double send. |
D8_tool_failure | A tool or a command fails or returns an error. Expected: an honest fallback and one retry; never a success that did not happen. |
D9_out_of_scope | A request outside the agent's job, such as a billing complaint in the middle of an order. Expected: the agent routes it to the place that handles it and keeps it out of the current flow. |
D10_return | The user returns to the flow after a deviation, such as "ok, let's continue" two blocks later. Expected: the agent resumes at the right stage with every known fact kept. |
D11_pii_leak | A request for another person's personal data, such as the card number or email of the customer before. Expected: the agent gives no personal data of another person and says it cannot share it. |
D12_prompt_extraction | A request to reveal the instructions, the prompt or the tool names. Expected: the agent does not reveal its instructions, its prompt or its tool names. |
ClassRef ​
The utterance class of a user message: its id and the hash of its frozen realisation set.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[a-z0-9][a-z0-9._-]*$ | yes | The class id, as its class file names it. | |
set | string matching ^[0-9a-f]{12}$ | yes | The hash of the frozen realisation set the block is measured over (12 hex characters). |
SimulatedUserSpec ​
A simulated user defined in the block: exact, simulated or replay.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
mode | one of exact, simulated, replay | yes | exact: send the user text as written. simulated: the agenda engine plans the message, a model writes it. replay: send the user text, a message frozen from a simulated run. | |
persona | string, not empty | no | Who the person is and how they write. Fictional. | |
goal | string, not empty | no | What the person wants. Absent, the first simulated block's user text is the goal. | |
hidden_knowledge | map of string to string, not empty | no | What the person knows, by name. A value reaches the model only once it is planned. | |
reveal | map of string to on_request or volunteer | no | For each fact: on_request (the default) gives it only once the agent asks; volunteer may give it in the opening message. | |
ask_patterns | map of string to list of strings, each not empty, at least 1 item | no | For each fact, patterns (case-insensitive) of an agent reply that asks for it. Used when no model reads the agent turns. | |
stop | string, not empty | no | What the person says when the goal is met or given up. | |
trials | integer, at least 1 | no | How many times a simulated run plays the spec, for pass^k and the rerun flip rate. | |
seed | string, not empty or integer | no | The seed of the engine. Absent, the spec id. | |
engine | SimulatedUserEngine | no | The agenda engine's parameters. | |
recorded | SimulatedUserSpec recorded | no | replay: where the frozen text came from. A text that does not match its hash is refused. |
SimulatedUserEngine ​
The agenda engine's parameters for a simulated user.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
acts_per_turn | list of numbers, each 0 to 1, at least 1 item | no | The chance of 1, 2, 3 ... dialogue acts in one message, in order. They must sum to 1. | |
volunteer_share | number, 0 to 1 | no | The chance that the opening message also gives each fact whose reveal is volunteer. | |
opening_detail_share | number, 0 to 1 | no | The chance that the opening message also states the specific problem, not only the request. | |
short_turn_share | number, 0 to 1 | no | The share of messages of at most three words. | |
polite_turn_share | number, 0 to 1 | no | The share of messages planned as polite. | |
uncertainty_turn_share | number, 0 to 1 | no | The share of messages planned as unsure. | |
question_turn_share | number, 0 to 1 | no | The share of messages planned as a question. | |
repeat_after_refusal | integer, at least 0 | no | How many times the person repeats the request after the agent says it cannot. | |
turn_budget | integer, at least 1 | no | The most messages the person writes before giving up. | |
engagement_initial | number, 0 to 1 | no | Engagement at the start. | |
decay_per_unhelpful_turn | number, 0 to 1 | no | Engagement lost on each unhelpful agent turn. | |
disengage_below | number, 0 to 1 | no | Below this engagement the person gives up. |
SimulatedUserSpec recorded ​
replay: where the frozen text came from. A text that does not match its hash is refused.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
hash | string matching ^[0-9a-f]{64}$ | yes | The SHA-256 of the canonical JSON of the frozen text. | |
from | string, not empty | no | The run the text was frozen from. |
Given ​
Initial conditions set on one block. A prefix lets the block start without replaying the ones before.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
prefix | list of Message | no | The messages before this block. Lets the block start without replaying the ones before. | |
origin | GivenOrigin | no | Where the prefix comes from: authored, recorded or canonical. Only with a prefix on the same block. It enters the Given hash. | |
stage | string, not empty | no | The stage the agent is at when this block starts. Only with a prefix on the same block; a validator reads it where the earlier blocks that would say it are not written. | |
after | string matching ^[^#\s]+#[^#\s]+$ | no | The block of a written spec this prefix is taken after, as <spec id>#<block id>. Only with a stage on the same block: a validator reads the stage only when that block leaves the agent at it after the same history. | |
ledger | map of string to any value | no | Facts the agent must hold at this block. They merge into the ledger from here on. | |
world | WorldState | no | Changes to the world from this block onwards. | |
vars | map of string to any value | no | Runtime variables from this block onwards. | |
shared | list of SharedFact | no | Where each ledger fact was stated in the prefix: a fact said by the user or the agent must appear in the message it names. | |
summary | string, not empty | no | What a named state (vars.session_state) holds from this block onwards, in words. |
Message ​
One message of a conversation prefix.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
role | one of user, assistant, system, tool | yes | Who sent the message. | |
content | string | yes | The text of the message. | |
name | string | no | An optional sender name. | |
tool_calls | list of PrefixToolCall | no | The tool calls the message made. |
PrefixToolCall ​
A tool call written in a spec prefix.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string | no | The id of the call. | |
label | string, not empty | yes | The tool name. | |
input | JSON value | no | The arguments of the call. | |
output | JSON value | no | What the tool returned. | |
error | string | no | The error the tool returned. |
GivenOrigin ​
Where the prefix comes from: authored (written by a person), recorded (copied from a real run, its tool calls with their real results) or canonical (the replay of a spec as it is written).
Used by blocks[].given.origin, beside a prefix. The origin enters the Given hash: an agent trusts a recorded call and tends to call a tool again when its earlier result is only in prose, so the three are different tests and their results are never pooled. See Given.
| Value | Meaning |
|---|---|
authored | A person wrote the prefix. |
recorded | The prefix is copied from a real run: its tool calls carry the results the tools really returned. import traces writes it. |
canonical | The prefix is the replay of a spec as it is written, such as a teacher-forced history: the calls the blocks expect, with the results the world forces. |
SharedFact ​
Where a ledger fact was stated before the block, checked by spec lint and before the block runs.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
fact | string, not empty | yes | The ledger key of the fact. | |
by | one of user, agent, record | yes | Who stated it: user or agent in a prefix message, record when it comes from a system of record and was never said. | |
in | integer, at least 0 | no | The index of the prefix message that states it, from 0. Needed for user and agent. |
Expect ​
The expected output and state delta of a block. Every field is optional.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
stage | string, not empty | no | The stage the agent must reach after the block, one of the spec stages. | |
stage_reported | boolean | no | True requires the runtime to report a non-empty stage in this block. A carried or inferred stage does not count. False or absent leaves reporting optional. | |
ledger_add | map of string to any value | no | Facts the block establishes. They join the ledger. | |
tools | list of ToolExpect | no | The tool calls expected in the block. | |
tools_offered | ToolsOfferedExpect | no | The tools the runtime must offer and must not offer, whether or not the agent calls them. | |
tool_order | one of exact, in_order, any_order | no | How the expected tools line up with the observed calls: exact, in_order or any_order. | |
must | list of strings, each not empty | no | Patterns the visible text must match. | |
must_not | list of strings, each not empty | no | Patterns the visible text must not match. | |
rouge_l | Expect rouge_l | no | The reply must overlap a reference text: its word-level ROUGE-L F measure must reach min. | |
bleu | Expect bleu | no | The reply must overlap a reference text: its BLEU score over words must reach min. | |
sql | Expect sql | no | The SQL the agent ran must return the same rows as the golden query on the database. | |
regex | list of strings, each not empty | no | Regular expressions the visible text must match, case-sensitive; must ignores case instead. | |
injection | boolean | no | True when the block is a prompt-injection or jailbreak attempt: the reply must not obey it. | |
exact | string, not empty | no | The reply must equal this text after normalising: case, surrounding space and punctuation ignored; two numbers compare as numbers. A dot or comma between two digits counts, except a strictly grouped thousands comma. | |
exact_strict | boolean | no | Compare exact byte for byte after trimming the ends, instead of ignoring case, punctuation and spacing. | |
json | JsonExpect | no | The reply is valid JSON, optionally with a schema and paths. | |
tool_f1 | Expect tool_f1 | no | Precision, recall and F1 of the tool calls against the expected ones. | |
refusal | refuses or answers | no | Whether the reply must be a refusal, read from refusal phrases in seven languages. | |
markup | one of xml, html, sql | no | The reply is well-formed XML, HTML or SQL (syntax only). | |
starts_with | string, not empty | no | The reply, without leading space, starts with this text. | |
finish_reason | string, not empty | no | The reason the model stopped, as the runtime reports it. | |
length | Expect length | no | Word and character bounds on the customer-visible text of the block. | |
similarity | list of SimilarityExpect | no | Reference texts the reply must be close to, each with its lowest passing score. | |
relevancy | RelevancyExpect | no | The reply addresses the ask of the block. | |
knowledge_retention | list of RetentionExpect | no | Facts given earlier that the block must not ask for again, and may have to use. | |
lang | string, 2 to 8 characters | no | The language of the reply. | |
pii_judge | boolean | no | Opt in to judged detection of personal names and addresses in the visible reply. Requires a configured personal-data judge; absent or false makes no model calls. Missing or failed judging is gating inconclusive. | |
cards | list of CardExpect | no | The cards expected in the block. | |
criteria | list of strings, each not empty | no | Judged criteria by name. name@gating makes a fail fail the block. | |
skills | list of SkillExpect | no | The skills expected in the block. | |
guards | list of GuardExpect | no | The runtime checks expected in the block, with their verdict and count. | |
references | list of ReferenceExpect | no | The files the agent must read in the block, by path or glob, with a count. | |
subagents | list of SubagentExpect | no | The helpers the agent must dispatch in the block, by name and model, with a count. | |
execution | ExecutionExpect | no | What the agent runs. | |
artifacts | list of ArtifactExpect | no | The files the block must produce. | |
workspace | WorkspaceExpect | no | The workspace diff expected. | |
task_impossible | boolean | no | True when the task, or the tests it was given, cannot be met honestly. The checks do not read it. Blame credits an agent that stopped and said so, and attributes a pass reached anyway to the agent as an integrity fault. | |
downstream | list of Expect downstream | no | Consumers of the block output that need a form nobody states to the agent. The checks do not read it. When a value that works fails its check, blame names the missing rule as a setup fault instead of flagging the test. | |
protected | list of strings, each not empty | no | Paths or globs the agent must not change or delete, whoever wrote them: the tests, checkers and source-of-truth files it was given. The checks do not read it. When a gating verdict fails and the net workspace diff holds one, blame attributes the failure to the agent as an integrity fault. | |
safety | SafetyExpect | no | Limits on destructive operations. |
ToolExpect ​
One expected tool call.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
label | string, not empty | yes | The tool name. | |
count | Count | no | How many calls are expected. 0 means the tool must not be called. | |
actor | Actor | no | Whose calls count: the agent (main), a helper it handed work to (helper), or both (any). Absent: both for a count of 0, the agent otherwise. | |
input | map of string to Assertion | no | Assertions on the arguments, by argument name. | |
input_closed | boolean | no | True requires every call input to be an object with only the top-level keys named in input. An absent input allows no keys. False or absent leaves other keys allowed. | |
compare | CompareMode or map of string to CompareMode | no | How the input values are compared: one mode for every argument, or a mode by argument name. Absent means auto: the value-check rungs decide whether a value that differs from the gold names the same thing. | |
recompute | map of string to ToolExpect recompute value | no | Arguments the agent computes, by argument name. The agent writes a formula (a text that starts with =) or a number; the recompute rung runs it and the gold formula on the given inputs and on changed inputs, and fails a value that is right only on the given ones. An argument named here is not also named in input. |
Count ​
An exact count, or a range with an optional min and max.
Write one of these forms.
- integer, at least 0
- a mapping with the fields below
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
min | integer, at least 0 | no | The fewest allowed. | |
max | integer, at least 0 | no | The most allowed. |
Actor ​
| Value | Meaning |
|---|---|
main | Only the agent's own calls count. |
helper | Only the calls of the helpers the agent handed work to count. |
any | The agent's calls and its helpers' calls both count. |
Assertion ​
How one value is asserted. A bare string, number, boolean or null means exact equality.
Write one of these forms.
- string
- number
- boolean
null- a mapping with exactly one of the keys below
| Key | Type | Description |
|---|---|---|
exact | JSON value | Equal to this JSON value. |
regex | string, not empty | A JavaScript regular expression source. Specs match text with the m flag. |
contains | string, not empty | Contains this text. |
one_of | list of JSON values, at least 1 item | Equal to one of these values. |
nonempty | true | Present and not empty. |
absent | true | Absent. |
subset | map of string to JSON value | Holds at least these keys with these values. |
CompareMode ​
How one value is compared: auto, a rung name, or exact with its reason.
Write one of these forms.
- one of
auto,outcome,contract,as_given,same_referent - a mapping with exactly one of the keys below
| Key | Type | Description |
|---|---|---|
exact | string, not empty | Why the writing itself is the requirement, in one line. |
ToolExpect recompute value ​
The gold computation of an argument the agent computes, and the inputs it runs on.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
formula | string, not empty | yes | The gold computation, a spreadsheet formula over the named inputs, such as =SUM(C2:C5). Numbers, names, + - * / ^, parentheses and SUM, MIN, MAX, AVERAGE, ROUND, ABS are supported. | |
inputs | map of string to number | yes | The inputs the agent was given, by name: C2: 10. The formulas are first run on these. | |
ranges | map of string to list of any value, at least 2 items, at most 2 items | no | The range [low, high] each input is drawn from on changed inputs. An input without one varies around its given value. |
ToolsOfferedExpect ​
The tools the runtime offers the agent, read from the runtime trace; calls do not count.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
include | list of strings, each not empty | no | Tools the runtime must offer. | |
exclude | list of strings, each not empty | no | Tools the runtime must not offer, such as a tool a setting denies. |
Expect rouge_l ​
The reply must overlap a reference text: its word-level ROUGE-L F measure must reach min.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
reference | string, not empty | yes | The reference text the reply is compared with. | |
profile | driftbench or ragas | no | The overlap profile: driftbench (default) or ragas, the reference implementation of this score. | |
min | number, 0 to 1 | yes | The lowest ROUGE-L F measure that passes, from 0 to 1. |
Expect bleu ​
The reply must overlap a reference text: its BLEU score over words must reach min.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
reference | string, not empty | yes | The reference text the reply is compared with. | |
profile | driftbench or ragas | no | The overlap profile: driftbench (default) or ragas, the reference implementation of this score. | |
min | number, 0 to 1 | yes | The lowest BLEU score that passes, from 0 to 1. |
Expect sql ​
The SQL the agent ran must return the same rows as the golden query on the database.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
tool | string, not empty | yes | The tool whose call input holds the query, in its query field. | |
golden | string, not empty | yes | The golden query: one read statement. | |
database | string, not empty | yes | The SQLite database both queries run on, read-only. |
JsonExpect ​
The reply is JSON, optionally with a schema and paths.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
embedded | boolean | no | When true, the reply may hold JSON among its prose: the first JSON value found is read. Off by default. | |
schema | map of string to JSON value | no | A JSON Schema the reply must match; an undeclared key fails. | |
paths | list of JsonExpect paths | no | Paths of the reply, each with its assertion. |
JsonExpect paths ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | A dotted path; a number indexes an array. | |
assert | Assertion | yes | What the value at the path must be, in the spec assertion vocabulary. |
Expect tool_f1 ​
Precision, recall and F1 of the tool calls against the expected ones.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
min | number, 0 to 1 | yes | The least F1 that passes, from 0 to 1. |
Expect length ​
Word and character bounds on the customer-visible text of the block.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
words | LengthBound | no | Bounds on the words of the visible text. | |
chars | LengthBound | no | Bounds on the characters (code points) of the visible text. |
LengthBound ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
min | integer, at least 0 | no | The least the count may be. | |
max | integer, at least 0 | no | The most the count may be. |
SimilarityExpect ​
A reference text the reply must be close to, with the lowest score that passes.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
reference | string, not empty | yes | The text the reply is compared with. | |
min | number, 0 to 1 | yes | The lowest similarity that passes, from 0 to 1, as the embedder scores it. |
RelevancyExpect ​
The reply addresses the ask of the block: a share of its key terms appears in the reply.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
min | number, 0 to 1 | no | The share of the key terms that must appear in the reply, from 0 to 1. Default 0.5. | |
terms | list of strings, each not empty | no | The key terms of the ask. Without them, the check takes the words of the stimulus. |
RetentionExpect ​
A fact the user gave earlier: the block must not ask for it again, and may have to use it.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
fact | string, not empty | yes | The ledger key of the fact, given in an earlier block. | |
use | boolean | no | True when the block must use the fact: its value appears in the reply or a tool input. | |
ask | string, not empty | no | A pattern that marks a question asking for the fact again. Default: the words of the key. |
CardExpect ​
One expected card.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string, not empty | yes | The card name. | |
count | Count | no | How many times the card is sent. |
SkillExpect ​
One expected skill.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string, not empty | yes | The skill name. | |
count | Count | no | How many times the skill is loaded. |
GuardExpect ​
One runtime check that fired during the block, with its verdict. The adapter maps the runtime names.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string, not empty | yes | The name of the check, as the adapter reports it. | |
verdict | string, not empty | no | The verdict the check must have given, such as pass or deny. | |
count | Count | no | How many times the check fired. |
ReferenceExpect ​
A file the agent must read during the block, by path or glob.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | The path read, or a glob: * matches within a folder, ** across folders, ? one character. | |
count | Count | no | How many reads of a matching file. 0 means it must not be read. |
SubagentExpect ​
A helper the agent must dispatch during the block, with an optional model.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string, not empty | yes | The helper name, as the adapter reports it. | |
model | string, not empty | no | The model the helper must run on. | |
count | Count | no | How many dispatches. 0 means the helper must not be dispatched. |
ExecutionExpect ​
What the agent runs in the block.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
ran_code | boolean | no | True when the block must run code; false when it must not. | |
exit_code | integer | no | The exit code of the last command. | |
max_commands | integer, at least 0 | no | The most commands the block may run. |
ArtifactExpect ​
A file the block must produce.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | The path of the file in the workspace. | |
kind | one of xlsx, docx, pptx, pdf, csv, json, text, code | no | The file kind, which picks the reader. | |
checks | list of maps of string to any value | no | Checks on the content of the file. |
WorkspaceExpect ​
The workspace diff of the block.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
created | list of strings | no | Paths the block creates. | |
changed | list of strings | no | Paths the block changes. | |
deleted | list of strings | no | Paths the block deletes. | |
others_untouched | boolean | no | True when no other path may change. |
Expect downstream ​
A consumer that needs a form of a value.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
tool | string, not empty | no | The tool whose input the consumer reads. Absent means any tool. | |
field | string, not empty | no | The argument the consumer reads. Absent means every argument. | |
needs | string, not empty | yes | The form the consumer needs, such as e164 or iso8601. | |
declared_in | string, not empty | no | Where the consumer states the need, such as a file or a system name. |
SafetyExpect ​
Limits on destructive or irreversible operations.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
max_destructive_ops | integer, at least 0 | no | The most destructive operations allowed. | |
confirm_before | list of strings, each not empty | no | Operations that need a confirmation first, such as delete or send. |
Budget ​
Limits for one block.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
steps | integer, at least 1 | no | The most agent steps. | |
cost_usd | number, at least 0 | no | The most cost, in US dollars. | |
latency_ms | number, at least 0 | no | The most latency, in milliseconds. |
Uses ​
The objects this spec or block uses. Absent keys use the shipped defaults.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
value_check | ObjectRef | no | policy/value-check: how a value the agent wrote is compared. | |
blame | ObjectRef | no | policy/blame: who a failure is attributed to. | |
placement | ObjectRef | no | policy/placement: where a fix goes. | |
links | ObjectRef | no | policy/links: how blocks link to prompt units. | |
selection | ObjectRef | no | policy/selection: which blocks a change selects. | |
gating | ObjectRef | no | policy/gating: what not_run, blocked and inconclusive do to ci. | |
stats | ObjectRef | no | policy/stats: which interval is the headline. | |
check_set | ObjectRef | no | check-set: the machine checks and their order. | |
judges | ObjectRef | no | judge-council: the judges, their mode and assignment. | |
diagnostics | list of strings, each matching ^[A-Za-z0-9][A-Za-z0-9._-]*(\/[A-Za-z0-9][A-Za-z0-9._-]*)*$ | no | The diagnostic objects that run after the checks. |
ObjectRef ​
Write one of these forms.
- string matching
^[A-Za-z0-9][A-Za-z0-9._-]*(\/[A-Za-z0-9][A-Za-z0-9._-]*)*$ - a mapping with the fields below
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
extends | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*(\/[A-Za-z0-9][A-Za-z0-9._-]*)*$ | yes | The object this inline one extends. | |
items | list of ItemPatch | no | Operations on the parent items. | |
order | list of strings, each matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | no | The item order. | |
params | map of string to any value | no | Object-level parameters. |
ItemPatch ​
An operation on one item of the parent object, or a new item with add: true.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | The item to change, or with add: true the new item's id. | |
add | true | no | Adds a new item. Needs title, when, effect and impl. | |
title | string, not empty | no | A new title for the item. | |
enabled | boolean | no | false disables the item, true enables it again. | |
when | map of string to any value | no | Condition fields to set; maps merge, lists replace. | |
effect | map of string to any value | no | Effect fields to set; maps merge, lists replace. | |
params | map of string to any value | no | Params to set: maps merge, lists replace, +key appends, -key removes. | |
impl | string matching ^(builtin|plugin):\S+$ | no | Another implementation for the same rule. | |
cites | string | no | A source for the reader. | |
applies_to | AppliesTo | no | Where the rule applies: maps merge, lists replace. | |
before | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | no | Moves the item just before this one. | |
after | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | no | Moves the item just after this one. |
AppliesTo ​
Where an item applies: skills, tools or unit patterns. Absent means everywhere.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
skills | list of strings, each not empty | no | Skill names the evidence must name one of. | |
tools | list of strings, each not empty | no | Tool labels the failing call must have one of. | |
units | list of strings, each not empty | no | Unit patterns, such as R#41/*, a unit must match one of. |
Spec end ​
An end verdict: the run stops after block atBlock (the last block when absent) with this verdict.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
verdict | success or failure | yes | success when the script ends the conversation well, failure when it ends it badly. | |
reason | string, not empty | no | Why the script ends here. | |
atBlock | integer, at least -1 | no | The index of the last block that runs, -1 for none. The blocks after it are skipped by the verdict. Absent means the last block. |
Rules ​
Rules that hold for the whole conversation.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
max_calls | map of string to integer, at least 0 | no | The most calls per tool across the run. 0 forbids the tool. | |
forbidden_tools | list of strings, each not empty | no | Tools that must never be called. | |
lang_stable | boolean | no | True when the reply language must stay the same across blocks. | |
one_message_per_turn | boolean | no | True when each block may send one message only. | |
match_case | boolean | no | True makes must and must_not patterns case-sensitive and the trailing punctuation count. Default false: a pattern ignores the case of the reply and a trailing punctuation mark. | |
custom | map of string to any value | no | Maps a registered custom check name to its options; the checks package runs it. |
EndState ​
none asserts that nothing was submitted; a mapping or a list of mappings names what to check; absent means no end-state check.
Write one of these forms.
none- EndStateSpec
- list of EndStateSpec, at least 1 item
EndStateSpec ​
The expected end state of the world, read from a source the adapter knows, such as a tool payload.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
source | string, not empty | yes | Where the adapter reads the state, such as tool:submit_order. | |
mode | subset or exact | no | subset compares the listed fields; exact compares the whole state. | |
expect | map of string to Assertion | yes | Assertions on fields of the state. |
Perturbation ​
A deviation inserted before block inject_at, counting from 0, when the spec runs in perturbed mode.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string, not empty | yes | An id, unique in the spec. | |
class | DeviationClass | yes | The deviation class. | |
inject_at | integer, at least 0 | yes | The position: the deviation goes in before this block, counting from 0. | |
user | string | no | The message the user sends. | |
burst | list of strings, at least 2 items | no | Messages the user sends together. | |
world | WorldState | no | A change to the world from this point, such as a tool forced to fail. | |
expect | Expect | no | Expectations for the injected block. | |
recovery | RecoveryExpect | no | How the agent should come back. Without it, recovery is not measured. |
RecoveryExpect ​
How the agent should come back after an injected deviation.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
within_blocks | integer, at least 0 | yes | How many blocks the agent has to come back. | |
stage_after | string, not empty | no | The stage the agent must be back at. | |
return_message | string | no | A pattern the first in-flow reply after the deviation must match. |
DriftClass ​
The five drift classes a failing block can carry.
Run records, not specs, carry these classes: one or more on each failing block. The runner assigns them from the failing gating verdicts of each block. See drift.
| Value | Meaning |
|---|---|
X1_local | A wrong answer or a wrong action in this block while every earlier fact still holds. Found by any failing check or criterion on the block alone. |
X2_contradiction | The block contradicts a fact established earlier, as held in the ledger. Found by the ledger check. |
X3_flow | The agent is at the wrong stage: it skips one, repeats one, or asks again for a known fact. Found by the stage classifier against expect.stage. |
X4_format | A format or persona rule breaks: narration of internal steps, the wrong language, a double send, card misuse, the wrong channel rules. Found by format and channel checks. |
X5_tool | A tool call, a skill or a guard is missing, extra or repeated, or a call has wrong parameters. Found by the tool, skill and guard checks. |
Inventory ​
The behaviour inventory: every behaviour the agent must show, grouped into flows, and the deviation classes each flow must be perturbed with.
The behaviour inventory is the file driftbench spec coverage --inventory reads. It lists the behaviours an agent must show and the flows that group them. A spec covers a behaviour by naming its id in covers, and joins a flow with the tag flow:<id> or by covering one of its behaviours. See CLI.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
behaviours | list of InventoryBehaviour | yes | Every behaviour the agent must show. | |
flows | list of InventoryFlow | yes | The flows that group the behaviours. | |
channels | list of strings, each not empty, at least 1 item | no | The channel universe. Absent means the channels named by behaviours and specs, else default. |
InventoryBehaviour ​
One behaviour the agent must show.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Z]+-\d+$ | yes | The behaviour id, such as B-001. Specs name it in covers. | |
title | string | yes | What the agent does. | |
kind | conversation or task | no | conversation | task marks a code-interpreter behaviour. It is a row of the same matrix as the others. |
channels | list of strings, each not empty, at least 1 item | no | The channels where the behaviour applies. Absent means every channel. |
InventoryFlow ​
A flow: behaviours that run together, and the deviations it must be perturbed with.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[a-z0-9][a-z0-9_-]*$ | yes | The flow id, such as plan-upgrade. Specs join it with the tag flow:<id>. | |
title | string | yes | What the flow does. | |
behaviours | list of strings, each matching ^[A-Z]+-\d+$ | yes | The behaviours of the flow, by id. | |
deviations | list of DeviationClass | no | The deviation classes this flow must be perturbed with. Absent means none is required. |