Task family reference ​
A task family is one code-agent task written once with knobs, in family.yaml, beside its oracle and the review a person signs; see task families. driftbench family instances, verify and review read these files. The family file comes first, then the review file. The page lists every field, generated from the zod schemas the CLI uses.
- Keys are snake_case. An unknown key is an error.
- Descriptions come from the schemas. An empty cell means the schema has no description for that field yet.
Contents ​
- FamilyFile
- FamilyFile target
- FamilyFile spec
- Knob
- RangeKnob
- KnobClass
- EnumKnob
- Constraint
- Constraint compare
- Constraint implies
- Table
- Count
- Column
- Generator
- Generator groups
- Generator choice_by
- Generator int
- SeedFile
- SeedFile csv
- SeedFile json
- SeedFile text
- SeedFile xlsx
- SeedFile xlsx columns
- SeedFile copy
- FamilyFile corners
- CornerKind
- CornerKind empty_cell
- CornerKind duplicate_row
- CornerKind column_order
- CornerKind unit_change
- CornerKind unit_change rules
- Phrase
- BlockDef
- BlockPart
- CheckVocabulary
- Planted
- ReviewFile
FamilyFile ​
A code-agent task family.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
version | 1 | yes | The format version of the family file: 1. | |
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
title | string, not empty | yes | A short title. | |
description | string, not empty | yes | What the task asks of the agent, for the reviewer. | |
target | FamilyFile target | yes | The agent every instance runs against. | |
spec | FamilyFile spec | yes | Fields copied into every instance spec. | |
oracle | string, not empty | yes | The oracle module beside the family file: it exports expected(input). | |
review | string, not empty | yes | The review file beside the family file, signed once by a person. | |
knobs | map of string to Knob | yes | The knobs, by name: what varies from one instance to the next. | |
constraints | list of Constraint | no | [] | Rules every drawn set of knobs must meet; a draw that breaks one is drawn again. |
wording_knob | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | The style knob whose value picks the phrase of every part. | |
tables | map of string to Table | yes | The tables the seed files render, by name. | |
files | list of SeedFile, at least 1 item | yes | The files of the workspace seed. | |
corners | FamilyFile corners | yes | The corner-case variants of each instance, chosen by the instance seed. | |
phrases | list of Phrase, at least 1 item | yes | An approved phrase: the only text an instance can hold. | |
blocks | list of BlockDef, at least 1 item | yes | The blocks of every instance, in order. | |
checks | CheckVocabulary | yes | Per file kind, the reader checks the expected files compile to; each must be a check of that reader. | |
others_untouched | boolean | no | true | True: every block also expects that no path other than those the oracle names changes. |
planted | Planted | no | A variant whose workspace holds an instruction planted in a file: the expected outcome stays the same. |
FamilyFile target ​
The agent every instance runs against.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
adapter | string, not empty | yes | The adapter that reaches the agent, as in a spec target. | |
agent | string, not empty | no | The agent, as the adapter names it. |
FamilyFile spec ​
Fields copied into every instance spec.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
channel | string | no | The channel of every instance spec. | |
lang | string | no | The declared language of every instance spec. | |
tier | string | no | The tier label of every instance spec. | |
tags | list of strings | no | Tags every instance spec carries. | |
covers | list of strings | no | Behaviour ids every instance spec covers. | |
stages | list of strings, each not empty | no | The stage catalogue of every instance spec. |
Knob ​
Write one of these forms.
RangeKnob ​
A whole number between min and max, both included.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
class | KnobClass | yes | difficulty, structure or style: what kind of change the knob makes. | |
description | string | no | What the knob changes, for the reviewer. | |
min | integer | yes | The smallest value, included. | |
max | integer | yes | The largest value, included. |
KnobClass ​
| Value | Meaning |
|---|---|
difficulty | A count: rows, files or sheets. |
structure | A shape: a format, a layout, a kind of formula, or a path the outcome holds. |
style | A surface choice the outcome must not depend on: the wording, an order, line endings, the name of a file the task never names, or the language. family instances rebuilds every variant with each other value of each style knob and writes nothing when the expected outcome changes. |
EnumKnob ​
One value of a list.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
class | KnobClass | yes | difficulty, structure or style: what kind of change the knob makes. | |
description | string | no | What the knob changes, for the reviewer. | |
values | list of strings, each not empty, at least 1 item | yes | The values, in the order the reviewer reads them. |
Constraint ​
Write one of these forms.
- a mapping with
compare, with the fields below - a mapping with
implies, with the fields below
With compare:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
compare | Constraint compare | yes | Every drawn instance holds left op right. |
With implies:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
implies | Constraint implies | yes | When the knobs meet when, they must meet require. |
Constraint compare ​
Every drawn instance holds left op right.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
left | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A whole-number knob. | |
op | one of <, <=, ==, !=, >=, > | yes | How left compares with right. | |
right | string matching ^[A-Za-z][A-Za-z0-9_]*$ or number | yes | A whole-number knob or a number. |
Constraint implies ​
When the knobs meet when, they must meet require.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
when | map of string to string or number or list of strings or numbers, at least 1 item | yes | Every named knob holds the value given, or one of the values listed. | |
require | map of string to string or number or list of strings or numbers, at least 1 item | yes | Every named knob holds the value given, or one of the values listed. |
Table ​
A table the seed files render, one row per item.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
rows | Count | yes | How many rows the table holds. | |
columns | list of Column, at least 1 item | yes | Generated in this order; a column reads only earlier ones. |
Count ​
Write one of these forms.
- integer, at least 0
- a mapping with exactly one of the keys below
| Key | Type | Description |
|---|---|---|
knob | string matching ^[A-Za-z][A-Za-z0-9_]*$ | A name: a letter, then letters, digits or underscores. |
Column ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
name | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. | |
gen | Generator | yes | How the value of each row is drawn. |
Generator ​
Write one of these forms.
- a mapping with exactly one of the keys below
- a mapping with the fields below
| Key | Type | Description |
|---|---|---|
groups | Generator groups | Picks count distinct values; every row gets one, every picked value at least one row. |
choice | list of strings, at least 1 item | One value of the list per row. |
choice_by | Generator choice_by | One value of the list the earlier column value names. |
int | Generator int | A whole number between min and max, both included; a number in JSON, digits in CSV. |
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
template | string, not empty | yes | The text of each cell, with its placeholders. | |
seq_by | string matching ^[A-Za-z][A-Za-z0-9_]*$ | no | The column whose value restarts {seq}; without it {seq} counts every row. |
Generator groups ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
values | list of strings, each not empty, at least 1 item | yes | The values to pick from. | |
count | Count | yes | How many distinct values the table uses; each one holds at least one row. |
Generator choice_by ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
column | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | An earlier column of the table. | |
values | map of string to list of strings, at least 1 item | yes | Per value of that column, the list the row picks from. |
Generator int ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
min | integer | yes | The smallest value, included. | |
max | integer | yes | The largest value, included. |
SeedFile ​
Write one of these forms.
- a mapping with
csv, with the fields below - a mapping with
json, with the fields below - a mapping with
text, with the fields below - a mapping with
xlsx, with the fields below - a mapping with
copy, with the fields below
With csv:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | The workspace path; {knob} fills in a knob value, so a style knob can name a file. | |
when | map of string to string or number or list of strings or numbers, at least 1 item | no | The file exists only in instances whose knobs meet this. | |
csv | SeedFile csv | yes | The rows of a table, written as CSV with a header. |
With json:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | The workspace path; {knob} fills in a knob value, so a style knob can name a file. | |
when | map of string to string or number or list of strings or numbers, at least 1 item | no | The file exists only in instances whose knobs meet this. | |
json | SeedFile json | yes | The rows of a table, written as a JSON list of records. |
With text:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | The workspace path; {knob} fills in a knob value, so a style knob can name a file. | |
when | map of string to string or number or list of strings or numbers, at least 1 item | no | The file exists only in instances whose knobs meet this. | |
text | SeedFile text | yes | An approved phrase, written as the whole file. |
With xlsx:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | The workspace path; {knob} fills in a knob value, so a style knob can name a file. | |
when | map of string to string or number or list of strings or numbers, at least 1 item | no | The file exists only in instances whose knobs meet this. | |
xlsx | SeedFile xlsx | yes | A workbook already in the workspace: rows of a table, one sheet per value of a column. |
With copy:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
path | string, not empty | yes | The workspace path; {knob} fills in a knob value, so a style knob can name a file. | |
when | map of string to string or number or list of strings or numbers, at least 1 item | no | The file exists only in instances whose knobs meet this. | |
copy | SeedFile copy | yes | An existing file, copied as it is. |
SeedFile csv ​
The rows of a table, written as CSV with a header.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
table | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. | |
columns | list of strings, each matching ^[A-Za-z][A-Za-z0-9_]*$, at least 1 item | yes | The header, in the order written. | |
column_order_knob | string matching ^[A-Za-z][A-Za-z0-9_]*$ | no | Columns are shuffled when this knob holds "shuffled". | |
line_endings_knob | string matching ^[A-Za-z][A-Za-z0-9_]*$ | no | Lines end with CR LF when this knob holds "crlf". |
SeedFile json ​
The rows of a table, written as a JSON list of records.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
table | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. | |
columns | list of strings, each matching ^[A-Za-z][A-Za-z0-9_]*$, at least 1 item | yes | The keys of each record, in the order written. | |
root | string, not empty | yes | The key of the list of records in the top-level object. | |
column_order_knob | string matching ^[A-Za-z][A-Za-z0-9_]*$ | no | Keys are shuffled when this knob holds "shuffled". |
SeedFile text ​
An approved phrase, written as the whole file.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
phrase | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. |
SeedFile xlsx ​
A workbook already in the workspace: rows of a table, one sheet per value of a column.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
table | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. | |
sheet_by | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | The column whose values name the sheets: one sheet per value, in the order values first appear. | |
sheet_name | as-is or title | no | as-is | The sheet name: the value as written, or with a capital first letter. |
columns | list of SeedFile xlsx columns, at least 1 item | yes | The columns of every sheet, in the order written. |
SeedFile xlsx columns ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
header | string, not empty | yes | The header cell of the column. | |
column | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | The table column the cells come from, in table order. | |
times | number | no | A number cell multiplied by this factor. |
SeedFile copy ​
An existing file, copied as it is.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
from | string, not empty | yes | A file beside the family file, such as a workbook. |
FamilyFile corners ​
The corner-case variants of each instance, chosen by the instance seed.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
per_instance | integer, at least 0 | yes | How many corner-case variants each instance gets, at most the number of kinds. | |
kinds | list of CornerKind, at least 1 item | yes | The corner cases a variant can apply. |
CornerKind ​
Write one of these forms.
- a mapping with
empty_cell, with the fields below - a mapping with
duplicate_row, with the fields below - a mapping with
column_order, with the fields below - a mapping with
unit_change, with the fields below
With empty_cell:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
empty_cell | CornerKind empty_cell | yes | One filled cell of the column, chosen by the seed, is emptied. |
With duplicate_row:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
duplicate_row | CornerKind duplicate_row | yes | One row, chosen by the seed, is written twice. |
With column_order:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
column_order | CornerKind column_order | yes | The columns of every file that renders the table are rotated by one. |
With unit_change:
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
unit_change | CornerKind unit_change | yes | Every value of the column a rule matches is converted to another unit. |
CornerKind empty_cell ​
One filled cell of the column, chosen by the seed, is emptied.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
table | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. | |
column | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. |
CornerKind duplicate_row ​
One row, chosen by the seed, is written twice.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
table | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. |
CornerKind column_order ​
The columns of every file that renders the table are rotated by one.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
table | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. |
CornerKind unit_change ​
Every value of the column a rule matches is converted to another unit.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
table | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. | |
column | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | A name: a letter, then letters, digits or underscores. | |
rules | list of CornerKind unit_change rules, at least 1 item | yes | The conversions, tried in order on each value. |
CornerKind unit_change rules ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
match | string, not empty | yes | A pattern whose first group is the number, such as ^(\d+) GB$. | |
unit | string, not empty | yes | The unit written after the converted number. | |
factor | number, greater than 0 | yes | What the number is multiplied by. |
Phrase ​
An approved phrase: the only text an instance can hold.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
text | string, not empty | yes | The text, with {slot} placeholders. | |
approved | boolean | yes | True once a person approved the text; an unapproved phrase never reaches an instance. |
BlockDef ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
id | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
when | map of string to string or number or list of strings or numbers, at least 1 item | no | The block exists only in instances whose knobs meet this. | |
stage | string, not empty | no | The stage the block expects, copied into the instance spec. | |
parts | list of BlockPart, at least 1 item | yes | The instruction, part by part; each part whose condition holds adds one phrase. | |
expect | map of string to any value | no | Expectations that hold in every instance (execution, skills, guards, safety); the oracle adds files, workspace and must. | |
budget | map of string to any value | no | Limits for the block, copied into every instance spec as its budget. |
BlockPart ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
when | map of string to string or number or list of strings or numbers, at least 1 item | no | Every named knob holds the value given, or one of the values listed. | |
phrases | list of strings, each matching ^[A-Za-z0-9][A-Za-z0-9._-]*$, at least 1 item | yes | One per wording; the wording knob picks one. |
CheckVocabulary ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
xlsx | list of strings, each not empty, at least 1 item | no | Checks of the workbook reader. | |
csv | list of strings, each not empty, at least 1 item | no | Checks of the CSV reader. | |
json | list of strings, each not empty, at least 1 item | no | Checks of the JSON reader. | |
text | list of strings, each not empty, at least 1 item | no | Checks of the text reader. |
Planted ​
A variant whose workspace holds an instruction planted in a file: the expected outcome stays the same.
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
knob | string matching ^[A-Za-z][A-Za-z0-9_]*$ | yes | The variant exists when this knob holds "with". | |
file | string, not empty | yes | The file the instruction is planted in. | |
phrase | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | The planted text, an approved phrase. | |
block | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | The block whose expectations gain must_not and safety. | |
must_not | list of strings, each not empty, at least 1 item | yes | Patterns the reply must not hold. | |
safety | map of string to any value | no | Safety expectations added to the block in the planted variant. |
ReviewFile ​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
family | string matching ^[A-Za-z0-9][A-Za-z0-9._-]*$ | yes | An id: letters, digits, dots, underscores and hyphens. | |
family_sha256 | string matching ^[0-9a-f]{64}$ | yes | The SHA-256 of the family file the signature holds for. | |
oracle_sha256 | string matching ^[0-9a-f]{64}$ | yes | The SHA-256 of the oracle the signature holds for. | |
questions | list of strings, each not empty, at least 1 item | yes | What the reviewer answers before signing. | |
signed_by | string, not empty or null | yes | The person who signed, null while the review is pending. | |
signed_on | string matching ^\d{4}-\d{2}-\d{2}$ or null | yes | The date of the signature, YYYY-MM-DD; null while the review is pending. | |
notes | string | no | The reviewer's notes. |