Skip to content

Task family reference ​

A task family is one code-agent task written once with knobs, in family.yaml, beside its oracle and the review a person signs; see task families. driftbench family instances, verify and review read these files. The family file comes first, then the review file. The page lists every field, generated from the zod schemas the CLI uses.

  • Keys are snake_case. An unknown key is an error.
  • Descriptions come from the schemas. An empty cell means the schema has no description for that field yet.

Contents ​

FamilyFile ​

A code-agent task family.

FieldTypeRequiredDefaultDescription
version1yesThe format version of the family file: 1.
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
titlestring, not emptyyesA short title.
descriptionstring, not emptyyesWhat the task asks of the agent, for the reviewer.
targetFamilyFile targetyesThe agent every instance runs against.
specFamilyFile specyesFields copied into every instance spec.
oraclestring, not emptyyesThe oracle module beside the family file: it exports expected(input).
reviewstring, not emptyyesThe review file beside the family file, signed once by a person.
knobsmap of string to KnobyesThe knobs, by name: what varies from one instance to the next.
constraintslist of Constraintno[]Rules every drawn set of knobs must meet; a draw that breaks one is drawn again.
wording_knobstring matching ^[A-Za-z][A-Za-z0-9_]*$yesThe style knob whose value picks the phrase of every part.
tablesmap of string to TableyesThe tables the seed files render, by name.
fileslist of SeedFile, at least 1 itemyesThe files of the workspace seed.
cornersFamilyFile cornersyesThe corner-case variants of each instance, chosen by the instance seed.
phraseslist of Phrase, at least 1 itemyesAn approved phrase: the only text an instance can hold.
blockslist of BlockDef, at least 1 itemyesThe blocks of every instance, in order.
checksCheckVocabularyyesPer file kind, the reader checks the expected files compile to; each must be a check of that reader.
others_untouchedbooleannotrueTrue: every block also expects that no path other than those the oracle names changes.
plantedPlantednoA variant whose workspace holds an instruction planted in a file: the expected outcome stays the same.

FamilyFile target ​

The agent every instance runs against.

FieldTypeRequiredDefaultDescription
adapterstring, not emptyyesThe adapter that reaches the agent, as in a spec target.
agentstring, not emptynoThe agent, as the adapter names it.

FamilyFile spec ​

Fields copied into every instance spec.

FieldTypeRequiredDefaultDescription
channelstringnoThe channel of every instance spec.
langstringnoThe declared language of every instance spec.
tierstringnoThe tier label of every instance spec.
tagslist of stringsnoTags every instance spec carries.
coverslist of stringsnoBehaviour ids every instance spec covers.
stageslist of strings, each not emptynoThe stage catalogue of every instance spec.

Knob ​

Write one of these forms.

RangeKnob ​

A whole number between min and max, both included.

FieldTypeRequiredDefaultDescription
classKnobClassyesdifficulty, structure or style: what kind of change the knob makes.
descriptionstringnoWhat the knob changes, for the reviewer.
minintegeryesThe smallest value, included.
maxintegeryesThe largest value, included.

KnobClass ​

ValueMeaning
difficultyA count: rows, files or sheets.
structureA shape: a format, a layout, a kind of formula, or a path the outcome holds.
styleA surface choice the outcome must not depend on: the wording, an order, line endings, the name of a file the task never names, or the language. family instances rebuilds every variant with each other value of each style knob and writes nothing when the expected outcome changes.

EnumKnob ​

One value of a list.

FieldTypeRequiredDefaultDescription
classKnobClassyesdifficulty, structure or style: what kind of change the knob makes.
descriptionstringnoWhat the knob changes, for the reviewer.
valueslist of strings, each not empty, at least 1 itemyesThe values, in the order the reviewer reads them.

Constraint ​

Write one of these forms.

  • a mapping with compare, with the fields below
  • a mapping with implies, with the fields below

With compare:

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
compareConstraint compareyesEvery drawn instance holds left op right.

With implies:

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
impliesConstraint impliesyesWhen the knobs meet when, they must meet require.

Constraint compare ​

Every drawn instance holds left op right.

FieldTypeRequiredDefaultDescription
leftstring matching ^[A-Za-z][A-Za-z0-9_]*$yesA whole-number knob.
opone of <, <=, ==, !=, >=, >yesHow left compares with right.
rightstring matching ^[A-Za-z][A-Za-z0-9_]*$ or numberyesA whole-number knob or a number.

Constraint implies &ZeroWidthSpace;

When the knobs meet when, they must meet require.

FieldTypeRequiredDefaultDescription
whenmap of string to string or number or list of strings or numbers, at least 1 itemyesEvery named knob holds the value given, or one of the values listed.
requiremap of string to string or number or list of strings or numbers, at least 1 itemyesEvery named knob holds the value given, or one of the values listed.

Table &ZeroWidthSpace;

A table the seed files render, one row per item.

FieldTypeRequiredDefaultDescription
rowsCountyesHow many rows the table holds.
columnslist of Column, at least 1 itemyesGenerated in this order; a column reads only earlier ones.

Count &ZeroWidthSpace;

Write one of these forms.

  • integer, at least 0
  • a mapping with exactly one of the keys below
KeyTypeDescription
knobstring matching ^[A-Za-z][A-Za-z0-9_]*$A name: a letter, then letters, digits or underscores.

Column &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
namestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.
genGeneratoryesHow the value of each row is drawn.

Generator &ZeroWidthSpace;

Write one of these forms.

  • a mapping with exactly one of the keys below
  • a mapping with the fields below
KeyTypeDescription
groupsGenerator groupsPicks count distinct values; every row gets one, every picked value at least one row.
choicelist of strings, at least 1 itemOne value of the list per row.
choice_byGenerator choice_byOne value of the list the earlier column value names.
intGenerator intA whole number between min and max, both included; a number in JSON, digits in CSV.
FieldTypeRequiredDefaultDescription
templatestring, not emptyyesThe text of each cell, with its placeholders.
seq_bystring matching ^[A-Za-z][A-Za-z0-9_]*$noThe column whose value restarts {seq}; without it {seq} counts every row.

Generator groups &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
valueslist of strings, each not empty, at least 1 itemyesThe values to pick from.
countCountyesHow many distinct values the table uses; each one holds at least one row.

Generator choice_by &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
columnstring matching ^[A-Za-z][A-Za-z0-9_]*$yesAn earlier column of the table.
valuesmap of string to list of strings, at least 1 itemyesPer value of that column, the list the row picks from.

Generator int &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
minintegeryesThe smallest value, included.
maxintegeryesThe largest value, included.

SeedFile &ZeroWidthSpace;

Write one of these forms.

  • a mapping with csv, with the fields below
  • a mapping with json, with the fields below
  • a mapping with text, with the fields below
  • a mapping with xlsx, with the fields below
  • a mapping with copy, with the fields below

With csv:

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesThe workspace path; {knob} fills in a knob value, so a style knob can name a file.
whenmap of string to string or number or list of strings or numbers, at least 1 itemnoThe file exists only in instances whose knobs meet this.
csvSeedFile csvyesThe rows of a table, written as CSV with a header.

With json:

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesThe workspace path; {knob} fills in a knob value, so a style knob can name a file.
whenmap of string to string or number or list of strings or numbers, at least 1 itemnoThe file exists only in instances whose knobs meet this.
jsonSeedFile jsonyesThe rows of a table, written as a JSON list of records.

With text:

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesThe workspace path; {knob} fills in a knob value, so a style knob can name a file.
whenmap of string to string or number or list of strings or numbers, at least 1 itemnoThe file exists only in instances whose knobs meet this.
textSeedFile textyesAn approved phrase, written as the whole file.

With xlsx:

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesThe workspace path; {knob} fills in a knob value, so a style knob can name a file.
whenmap of string to string or number or list of strings or numbers, at least 1 itemnoThe file exists only in instances whose knobs meet this.
xlsxSeedFile xlsxyesA workbook already in the workspace: rows of a table, one sheet per value of a column.

With copy:

FieldTypeRequiredDefaultDescription
pathstring, not emptyyesThe workspace path; {knob} fills in a knob value, so a style knob can name a file.
whenmap of string to string or number or list of strings or numbers, at least 1 itemnoThe file exists only in instances whose knobs meet this.
copySeedFile copyyesAn existing file, copied as it is.

SeedFile csv &ZeroWidthSpace;

The rows of a table, written as CSV with a header.

FieldTypeRequiredDefaultDescription
tablestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.
columnslist of strings, each matching ^[A-Za-z][A-Za-z0-9_]*$, at least 1 itemyesThe header, in the order written.
column_order_knobstring matching ^[A-Za-z][A-Za-z0-9_]*$noColumns are shuffled when this knob holds "shuffled".
line_endings_knobstring matching ^[A-Za-z][A-Za-z0-9_]*$noLines end with CR LF when this knob holds "crlf".

SeedFile json &ZeroWidthSpace;

The rows of a table, written as a JSON list of records.

FieldTypeRequiredDefaultDescription
tablestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.
columnslist of strings, each matching ^[A-Za-z][A-Za-z0-9_]*$, at least 1 itemyesThe keys of each record, in the order written.
rootstring, not emptyyesThe key of the list of records in the top-level object.
column_order_knobstring matching ^[A-Za-z][A-Za-z0-9_]*$noKeys are shuffled when this knob holds "shuffled".

SeedFile text &ZeroWidthSpace;

An approved phrase, written as the whole file.

FieldTypeRequiredDefaultDescription
phrasestring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.

SeedFile xlsx &ZeroWidthSpace;

A workbook already in the workspace: rows of a table, one sheet per value of a column.

FieldTypeRequiredDefaultDescription
tablestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.
sheet_bystring matching ^[A-Za-z][A-Za-z0-9_]*$yesThe column whose values name the sheets: one sheet per value, in the order values first appear.
sheet_nameas-is or titlenoas-isThe sheet name: the value as written, or with a capital first letter.
columnslist of SeedFile xlsx columns, at least 1 itemyesThe columns of every sheet, in the order written.

SeedFile xlsx columns &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
headerstring, not emptyyesThe header cell of the column.
columnstring matching ^[A-Za-z][A-Za-z0-9_]*$yesThe table column the cells come from, in table order.
timesnumbernoA number cell multiplied by this factor.

SeedFile copy &ZeroWidthSpace;

An existing file, copied as it is.

FieldTypeRequiredDefaultDescription
fromstring, not emptyyesA file beside the family file, such as a workbook.

FamilyFile corners &ZeroWidthSpace;

The corner-case variants of each instance, chosen by the instance seed.

FieldTypeRequiredDefaultDescription
per_instanceinteger, at least 0yesHow many corner-case variants each instance gets, at most the number of kinds.
kindslist of CornerKind, at least 1 itemyesThe corner cases a variant can apply.

CornerKind &ZeroWidthSpace;

Write one of these forms.

  • a mapping with empty_cell, with the fields below
  • a mapping with duplicate_row, with the fields below
  • a mapping with column_order, with the fields below
  • a mapping with unit_change, with the fields below

With empty_cell:

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
empty_cellCornerKind empty_cellyesOne filled cell of the column, chosen by the seed, is emptied.

With duplicate_row:

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
duplicate_rowCornerKind duplicate_rowyesOne row, chosen by the seed, is written twice.

With column_order:

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
column_orderCornerKind column_orderyesThe columns of every file that renders the table are rotated by one.

With unit_change:

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
unit_changeCornerKind unit_changeyesEvery value of the column a rule matches is converted to another unit.

CornerKind empty_cell &ZeroWidthSpace;

One filled cell of the column, chosen by the seed, is emptied.

FieldTypeRequiredDefaultDescription
tablestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.
columnstring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.

CornerKind duplicate_row &ZeroWidthSpace;

One row, chosen by the seed, is written twice.

FieldTypeRequiredDefaultDescription
tablestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.

CornerKind column_order &ZeroWidthSpace;

The columns of every file that renders the table are rotated by one.

FieldTypeRequiredDefaultDescription
tablestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.

CornerKind unit_change &ZeroWidthSpace;

Every value of the column a rule matches is converted to another unit.

FieldTypeRequiredDefaultDescription
tablestring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.
columnstring matching ^[A-Za-z][A-Za-z0-9_]*$yesA name: a letter, then letters, digits or underscores.
ruleslist of CornerKind unit_change rules, at least 1 itemyesThe conversions, tried in order on each value.

CornerKind unit_change rules &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
matchstring, not emptyyesA pattern whose first group is the number, such as ^(\d+) GB$.
unitstring, not emptyyesThe unit written after the converted number.
factornumber, greater than 0yesWhat the number is multiplied by.

Phrase &ZeroWidthSpace;

An approved phrase: the only text an instance can hold.

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
textstring, not emptyyesThe text, with {slot} placeholders.
approvedbooleanyesTrue once a person approved the text; an unapproved phrase never reaches an instance.

BlockDef &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
idstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
whenmap of string to string or number or list of strings or numbers, at least 1 itemnoThe block exists only in instances whose knobs meet this.
stagestring, not emptynoThe stage the block expects, copied into the instance spec.
partslist of BlockPart, at least 1 itemyesThe instruction, part by part; each part whose condition holds adds one phrase.
expectmap of string to any valuenoExpectations that hold in every instance (execution, skills, guards, safety); the oracle adds files, workspace and must.
budgetmap of string to any valuenoLimits for the block, copied into every instance spec as its budget.

BlockPart &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
whenmap of string to string or number or list of strings or numbers, at least 1 itemnoEvery named knob holds the value given, or one of the values listed.
phraseslist of strings, each matching ^[A-Za-z0-9][A-Za-z0-9._-]*$, at least 1 itemyesOne per wording; the wording knob picks one.

CheckVocabulary &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
xlsxlist of strings, each not empty, at least 1 itemnoChecks of the workbook reader.
csvlist of strings, each not empty, at least 1 itemnoChecks of the CSV reader.
jsonlist of strings, each not empty, at least 1 itemnoChecks of the JSON reader.
textlist of strings, each not empty, at least 1 itemnoChecks of the text reader.

Planted &ZeroWidthSpace;

A variant whose workspace holds an instruction planted in a file: the expected outcome stays the same.

FieldTypeRequiredDefaultDescription
knobstring matching ^[A-Za-z][A-Za-z0-9_]*$yesThe variant exists when this knob holds "with".
filestring, not emptyyesThe file the instruction is planted in.
phrasestring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesThe planted text, an approved phrase.
blockstring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesThe block whose expectations gain must_not and safety.
must_notlist of strings, each not empty, at least 1 itemyesPatterns the reply must not hold.
safetymap of string to any valuenoSafety expectations added to the block in the planted variant.

ReviewFile &ZeroWidthSpace;

FieldTypeRequiredDefaultDescription
familystring matching ^[A-Za-z0-9][A-Za-z0-9._-]*$yesAn id: letters, digits, dots, underscores and hyphens.
family_sha256string matching ^[0-9a-f]{64}$yesThe SHA-256 of the family file the signature holds for.
oracle_sha256string matching ^[0-9a-f]{64}$yesThe SHA-256 of the oracle the signature holds for.
questionslist of strings, each not empty, at least 1 itemyesWhat the reviewer answers before signing.
signed_bystring, not empty or nullyesThe person who signed, null while the review is pending.
signed_onstring matching ^\d{4}-\d{2}-\d{2}$ or nullyesThe date of the signature, YYYY-MM-DD; null while the review is pending.
notesstringnoThe reviewer's notes.

MIT. driftbench is made by Nexus.