Skip to content

Criteria and labels reference ​

A criteria folder holds one file per criterion with its fixture pair, read by driftbench judge check and judge run; a labels folder holds the hand labels judge calibrate reads and the viewer writes. See judges. The page lists every field, generated from the zod schemas the CLI uses.

  • Keys are as written below. An unknown key is an error.
  • Descriptions come from the schemas. An empty cell means the schema has no description for that field yet.

Contents ​

CriterionFile ​

FieldTypeRequiredDefaultDescription
criterionCriterionDefyesA judged criterion. Blocks reference it by name, with @gating when a fail fails the conversation.
fixturesCriterionFile fixturesyesThe fixture pair: the owning judge must separate the two every time before it is trusted.

CriterionDef ​

A judged criterion. Blocks reference it by name, with @gating when a fail fails the conversation.

FieldTypeRequiredDefaultDescription
namestring matching ^[A-Za-z0-9][A-Za-z0-9_.-]*$yesThe name blocks use to reference the criterion.
rubricstring, not emptyyesThe question the judge answers with pass or fail.
intentrequest-fulfillmentnoDeclares a request-fulfillment metric; only these metrics invoke the semantic obligation-ledger gate.
passstringnoWhat a passing block looks like.
failstringnoWhat a failing block looks like.
gatingbooleannoTrue gates every reference to this criterion; a reference can also add @gating.
scopeblock or conversationnoblock judges one block; conversation judges the whole conversation.
inputslist of strings, each not emptynoWorkspace files the judge reads beside the transcript, such as the source a produced file is compared against. Read from the seed the agent was given, else from the workspace.
judge_classone of system1, system2, autonosystem1, system2, or auto: calibration assigns the class, ties go to system1.
repeat_reducerstring matching ^(mode|median|at_least:[1-9][0-9]*)$noHow the verdicts across the repeats of one spec become one verdict: mode, median, or at_least:<k>.
prompt_version1 or 2noThe criterion prompt rules the judge reads. 2, the default, adds the exact-identifier, stated-facts and stated-scope rules; 1 is the earlier rule set, to replay answers recorded with it.
decoder_version1 or 2noHow a criterion ledger answer is read. Unset, the default reader drops an ungrounded quote and keeps the verdict; 1 rejects an ungrounded quote; 2 drops the quote and keeps the verdict.
answer_contractledger or legacynoledger, the default, or legacy: the earlier prompt and parser, to replay scores recorded with them.
assessmentone of response-relevance, conformance, answer-audit, fact-ledgernoAn assessment kind in place of the criterion ledger: response-relevance (the reply answers the latest ask), conformance (to reference), answer-audit (the submitted answer, optionally against reference), or fact-ledger (every fact of reference in the reply).
referencestringnoThe reference answer an assessment reads: needed by conformance (may be empty) and fact-ledger, optional for answer-audit.
groundingstrictnofact-ledger only: strict refuses reference or reply text that is not a source quote.
requirementslist of CriterionDef requirements, at least 1 itemnoThe rubric split into requirements the judge checks one by one. Absent: one requirement, the whole rubric.
itemslist of CriterionDef items, at least 1 itemnoRubric items, each scored 0 to 4 by the judge; the weighted score decides the verdict against threshold.
thresholdnumber, 0 to 1noWith items: the weighted fraction a pass needs, from 0 to 1. Default 1.
repeatinteger, 1 to 20noHow many times the judge answers the same request; the verdict holds only when every answer agrees, else inconclusive. Not the spec repeats (see repeat_reducer).
per_turnbooleannoTrue judges the transcript one customer turn at a time; any failing turn fails the criterion.

CriterionDef requirements &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
idstring, not emptyyesThe requirement id the judge answers under.
requirementstring, not emptyyesWhat the requirement asks.

CriterionDef items &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
namestring, not emptyyesThe item name, unique in the criterion.
rubricstring, not emptyyesWhat the item requires.
weightnumber, greater than 0noThe item weight. Default 1.
hardbooleannoTrue: the criterion fails when this item is not met.

CriterionFile fixtures &ZeroWidthSpace;

The fixture pair: the owning judge must separate the two every time before it is trusted.

FieldTypeRequiredDefaultDescription
passCriterionFixtureyesA slice the criterion must pass.
failCriterionFixtureyesA slice the criterion must fail.

CriterionFixture &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
notestringnoWhat the slice shows, for a reader.
transcriptlist of TranscriptEntry, at least 1 itemyesThe slice the judge reads: what the user sent, what the agent did and said.
artifactstringnoThe text of a produced file, for a task criterion.
inputsmap of string to stringnoThe workspace inputs the criterion names, by path: the file text the judge compares against.
givenCriterionFixture givennoA summary of the block's Given, as the judge reads it.

TranscriptEntry &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
roleone of user, agent, tool_call, tool_result, card, contextyesWho or what the line comes from: user, agent, tool_call, tool_result, card or context.
textstringyesThe text of the line.
toolstringnoThe tool name, on a tool_call or tool_result line.

CriterionFixture given &ZeroWidthSpace;

A summary of the block's Given, as the judge reads it.

FieldTypeRequiredDefaultDescription
stagestringnoThe stage the block expects.
ledgermap of string to any valuenoThe facts the agent holds at the block.

LabelFile &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
criterionstring, not emptyyesThe name of the criterion the labels are for.
labelled_bystringnoWho labelled, for a reader.
blockslist of LabelledBlock, at least 1 itemyesThe labelled blocks.

LabelledBlock &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
notestringnoWhat the slice shows, for a reader.
transcriptlist of TranscriptEntry, at least 1 itemyesThe slice the judge reads: what the user sent, what the agent did and said.
artifactstringnoThe text of a produced file, for a task criterion.
inputsmap of string to stringnoThe workspace inputs the criterion names, by path: the file text the judge compares against.
givenLabelledBlock givennoA summary of the block's Given, as the judge reads it.
idstring, not emptyyesThe id of the block; a new label for the same id replaces the old one.
labelpass or failyesThe human's verdict on the criterion: pass or fail.
flagslist of sent text not storednoLabels with unavailable sent text are excluded from calibration.
anchorAnnotationAnchornoThe element of the block the label is about; absent: the whole block.

LabelledBlock given &ZeroWidthSpace;

A summary of the block's Given, as the judge reads it.

FieldTypeRequiredDefaultDescription
stagestringnoThe stage the block expects.
ledgermap of string to any valuenoThe facts the agent holds at the block.

AnnotationAnchor &ZeroWidthSpace;

Write one of these forms.

  • a mapping with kind: message, with the fields below
  • a mapping with kind: tool_call, with the fields below
  • a mapping with kind: field, with the fields below
  • a mapping with kind: span, with the fields below

With kind: message:

FieldTypeRequiredDefaultDescription
kindmessageyesA transcript line.
indexinteger, at least 0yesThe index of the line in the transcript, from 0.

With kind: tool_call:

FieldTypeRequiredDefaultDescription
kindtool_callyesA tool the transcript calls.
toolstring, not emptyyesThe tool name.
occurrenceinteger, at least 1noWhich call of that tool, from 1; 1 when absent.

With kind: field:

FieldTypeRequiredDefaultDescription
kindfieldyesA field of the ledger or of a tool output.
pathstring, not emptyyesThe dot path of the field.

With kind: span:

FieldTypeRequiredDefaultDescription
kindspanyesA trace span, which the transcript cannot show.
span_idstring, not emptyyesThe span id the run recorded.

MIT. driftbench is made by Nexus.