Skip to content

Campaign configuration reference ​

The campaign: key of driftbench.yaml, or the file named by --config, holds campaign settings. Command-line flags override the file. See the campaign key for how campaign run, campaign plan and ci use these settings. The page lists every field, generated from the zod schemas the CLI uses.

  • Keys are snake_case. An unknown key is an error.
  • Descriptions come from the schemas. An empty cell means the schema has no description for that field yet.

Contents ​

CampaignConfig ​

The settings of a campaign, each the flag of campaign run it names. A flag on the command line wins over the file.

FieldTypeRequiredDefaultDescription
specslist of strings, each not empty, at least 1 itemnoSpec files or folders, read from the working folder. Used by campaign run when the command line names none.
datasetstring, not emptynoA dataset version of the store to run, name or name@version: --dataset.
promptslist of strings, each not emptynoPrompt versions of the store to record, each name or name@version: --prompt, once per item.
targetslist of strings, each not emptynoThe targets every spec runs on, each adapter or adapter:agent: --target, once per item.
routeslist of strings, each not emptynoTargets by spec tag, each tag=target: --route, once per item.
modeslist of one of T, F, P, at least 1 itemnoThe run modes, such as [T, F]: --mode.
repeatsinteger, at least 1noRuns per spec, target and mode: --repeats.
agent_versionstring, not emptynoThe version of the agent under test: --agent-version.
storestring, not emptynoThe store file the campaign is kept in, read from the working folder: --store.
cachebooleannoFalse neither replays runs from the cache nor writes to it: --no-cache. campaign run only.
cache_suitestring, not emptynoThe cache suite of the campaign: --cache-suite.
concurrencyinteger, at least 1noCells that run at the same time: --concurrency.
budget_usdnumber, at least 0noA hard stop on the campaign's spend: --budget-usd.
process_commandstring, not emptynoThe command that runs the agent of a process target: --process-command.
driverscripted or simulatednoWho writes the user turns, the spec's own script or a persona: --driver.
personaslist of strings, each not emptynoWith the simulated driver, persona files: --persona, once per item.
recordingslist of strings, each not emptynoWith the simulated driver, recordings to replay: --recording, once per item.
modelbooleannoWith the simulated driver, true speaks through the model port, a paid call: --model.
agent_familystring, not emptynoThe model family of the agent under test: --agent-family.
judge_familieslist of strings, each not emptynoThe model families of the judges: --judge-family, once per item.
retryCampaignRetryConfignoHow model calls are retried, over the retry settings of the spec target.

CampaignRetryConfig ​

How model calls are retried, over the retry settings of the spec target.

FieldTypeRequiredDefaultDescription
max_attemptsinteger, at least 1noAttempts per model call, the first included: --retry-attempts.
base_delay_msinteger, at least 0noThe wait before the first retry, doubled for each later one: --retry-base-ms.
max_delay_msinteger, at least 0noThe longest wait between two attempts: --retry-max-ms.

MIT. driftbench is made by Nexus.