Criteria and labels reference ​
A criteria folder holds one file per criterion with its fixture pair, read by driftbench judge check and judge run; a labels folder holds the hand labels judge calibrate reads and the viewer writes. See judges. The page lists every field, generated from the zod schemas the CLI uses.
- Keys are as written below. An unknown key is an error.
- Descriptions come from the schemas. An empty cell means the schema has no description for that field yet.
Contents ​
- CriterionFile
- CriterionDef
- CriterionDef requirements
- CriterionDef items
- CriterionFile fixtures
- CriterionFixture
- TranscriptEntry
- CriterionFixture given
- LabelFile
- LabelledBlock
- LabelledBlock given
- AnnotationAnchor
CriterionFile ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
criterion | CriterionDef | yes | A judged criterion. Blocks reference it by name, with @gating when a fail fails the conversation. | |
fixtures | CriterionFile fixtures | yes | The fixture pair: the owning judge must separate the two every time before it is trusted. |
CriterionDef ​
A judged criterion. Blocks reference it by name, with @gating when a fail fails the conversation.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string matching ^[A-Za-z0-9][A-Za-z0-9_.-]*$ | yes | The name blocks use to reference the criterion. | |
rubric | string, not empty | yes | The question the judge answers with pass or fail. | |
intent | request-fulfillment | no | Declares a request-fulfillment metric; only these metrics invoke the semantic obligation-ledger gate. | |
pass | string | no | What a passing block looks like. | |
fail | string | no | What a failing block looks like. | |
gating | boolean | no | True gates every reference to this criterion; a reference can also add @gating. | |
scope | block or conversation | no | block judges one block; conversation judges the whole conversation. | |
inputs | list of strings, each not empty | no | Workspace files the judge reads beside the transcript, such as the source a produced file is compared against. Read from the seed the agent was given, else from the workspace. | |
judge_class | one of system1, system2, auto | no | system1, system2, or auto: calibration assigns the class, ties go to system1. | |
repeat_reducer | string matching ^(mode|median|at_least:[1-9][0-9]*)$ | no | How the verdicts across the repeats of one spec become one verdict: mode, median, or at_least:<k>. | |
prompt_version | 1 or 2 | no | The criterion prompt rules the judge reads. 2, the default, adds the exact-identifier, stated-facts and stated-scope rules; 1 is the earlier rule set, to replay answers recorded with it. | |
decoder_version | 1 or 2 | no | How a criterion ledger answer is read. Unset, the default reader drops an ungrounded quote and keeps the verdict; 1 rejects an ungrounded quote; 2 drops the quote and keeps the verdict. | |
answer_contract | ledger or legacy | no | ledger, the default, or legacy: the earlier prompt and parser, to replay scores recorded with them. | |
assessment | one of response-relevance, conformance, answer-audit, fact-ledger | no | An assessment kind in place of the criterion ledger: response-relevance (the reply answers the latest ask), conformance (to reference), answer-audit (the submitted answer, optionally against reference), or fact-ledger (every fact of reference in the reply). | |
reference | string | no | The reference answer an assessment reads: needed by conformance (may be empty) and fact-ledger, optional for answer-audit. | |
grounding | strict | no | fact-ledger only: strict refuses reference or reply text that is not a source quote. | |
requirements | list of CriterionDef requirements, at least 1 item | no | The rubric split into requirements the judge checks one by one. Absent: one requirement, the whole rubric. | |
items | list of CriterionDef items, at least 1 item | no | Rubric items, each scored 0 to 4 by the judge; the weighted score decides the verdict against threshold. | |
threshold | number, 0 to 1 | no | With items: the weighted fraction a pass needs, from 0 to 1. Default 1. | |
repeat | integer, 1 to 20 | no | How many times the judge answers the same request; the verdict holds only when every answer agrees, else inconclusive. Not the spec repeats (see repeat_reducer). | |
per_turn | boolean | no | True judges the transcript one customer turn at a time; any failing turn fails the criterion. |
CriterionDef requirements ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string, not empty | yes | The requirement id the judge answers under. | |
requirement | string, not empty | yes | What the requirement asks. |
CriterionDef items ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string, not empty | yes | The item name, unique in the criterion. | |
rubric | string, not empty | yes | What the item requires. | |
weight | number, greater than 0 | no | The item weight. Default 1. | |
hard | boolean | no | True: the criterion fails when this item is not met. |
CriterionFile fixtures ​
The fixture pair: the owning judge must separate the two every time before it is trusted.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
pass | CriterionFixture | yes | A slice the criterion must pass. | |
fail | CriterionFixture | yes | A slice the criterion must fail. |
CriterionFixture ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
note | string | no | What the slice shows, for a reader. | |
transcript | list of TranscriptEntry, at least 1 item | yes | The slice the judge reads: what the user sent, what the agent did and said. | |
artifact | string | no | The text of a produced file, for a task criterion. | |
inputs | map of string to string | no | The workspace inputs the criterion names, by path: the file text the judge compares against. | |
given | CriterionFixture given | no | A summary of the block's Given, as the judge reads it. |
TranscriptEntry ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
role | one of user, agent, tool_call, tool_result, card, context | yes | Who or what the line comes from: user, agent, tool_call, tool_result, card or context. | |
text | string | yes | The text of the line. | |
tool | string | no | The tool name, on a tool_call or tool_result line. |
CriterionFixture given ​
A summary of the block's Given, as the judge reads it.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
stage | string | no | The stage the block expects. | |
ledger | map of string to any value | no | The facts the agent holds at the block. |
LabelFile ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
criterion | string, not empty | yes | The name of the criterion the labels are for. | |
labelled_by | string | no | Who labelled, for a reader. | |
blocks | list of LabelledBlock, at least 1 item | yes | The labelled blocks. |
LabelledBlock ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
note | string | no | What the slice shows, for a reader. | |
transcript | list of TranscriptEntry, at least 1 item | yes | The slice the judge reads: what the user sent, what the agent did and said. | |
artifact | string | no | The text of a produced file, for a task criterion. | |
inputs | map of string to string | no | The workspace inputs the criterion names, by path: the file text the judge compares against. | |
given | LabelledBlock given | no | A summary of the block's Given, as the judge reads it. | |
id | string, not empty | yes | The id of the block; a new label for the same id replaces the old one. | |
label | pass or fail | yes | The human's verdict on the criterion: pass or fail. | |
flags | list of sent text not stored | no | Labels with unavailable sent text are excluded from calibration. | |
anchor | AnnotationAnchor | no | The element of the block the label is about; absent: the whole block. |
LabelledBlock given ​
A summary of the block's Given, as the judge reads it.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
stage | string | no | The stage the block expects. | |
ledger | map of string to any value | no | The facts the agent holds at the block. |
AnnotationAnchor ​
Write one of these forms.
- a mapping with
kind: message, with the fields below - a mapping with
kind: tool_call, with the fields below - a mapping with
kind: field, with the fields below - a mapping with
kind: span, with the fields below
With kind: message:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
kind | message | yes | A transcript line. | |
index | integer, at least 0 | yes | The index of the line in the transcript, from 0. |
With kind: tool_call:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
kind | tool_call | yes | A tool the transcript calls. | |
tool | string, not empty | yes | The tool name. | |
occurrence | integer, at least 1 | no | Which call of that tool, from 1; 1 when absent. |
With kind: field:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
kind | field | yes | A field of the ledger or of a tool output. | |
path | string, not empty | yes | The dot path of the field. |
With kind: span:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
kind | span | yes | A trace span, which the transcript cannot show. | |
span_id | string, not empty | yes | The span id the run recorded. |