Skip to content

Inspect: comparison ​

Inspect (inspect_ai) is a framework for evaluating language model applications. This page compares driftbench with inspect_ai version 0.3.278, read in its source at commit a1dbea8.

Results come from the comparison table. Limit means a row is not being worked, with its reason stated.

CapabilityResultBasisExampleReproduce
Model-graded QA prompt (closed question, GRADE letter)ahead on accuracy; held-out ahead on accuracymeasured hard and held-out comparison; compact ledger with bounded retry is not yet the shipped default2026-10-13, Haiku 5.5. Hard: ours 111/120, audit 120/120, theirs 115/120. Held-out: ours 52/60, audit 60/60, theirs 57/60. Hard USD per 100: ours 0.04450, audit 0.05695, theirs 0.05867. Fresh seed: 61326. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results. Original baseline stays 111/120 hard and 48/60 held-out; the original under-review 1270e1cb run is 114/120 and 44/60 on those identical inputs.CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts qa ours --compact --replay; repeat with theirs and add --heldout for the fresh set. Then CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts qa audit --compact --replay; add --heldout for the fresh set.
Model-graded fact template (contains the expert content)equal accuracy; held-out ahead on accuracymeasured hard and held-out comparison; compact ledger with bounded retry is not yet the shipped default2026-10-13, Haiku 5.5. Hard: ours 111/120, audit 120/120, theirs 120/120. Held-out: ours 52/60, audit 60/60, theirs 59/60. Hard USD per 100: ours 0.04450, audit 0.05695, theirs 0.05557. Fresh seed: 61326. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results. Original baseline stays 111/120 hard and 48/60 held-out; the original under-review 1270e1cb run is 114/120 and 44/60 on those identical inputs. The earlier 20/20 cost claim uses the QA template and cannot establish fact-template performance.CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts fact ours --compact --replay; repeat with theirs and add --heldout for the fresh set. Then CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts fact audit --compact --replay; add --heldout for the fresh set.
GRADE verdict parse (last GRADE token)limitHistorical improvement claim has no recorded example or reproducible earlier-version baseline.The hard harness mirrors the pinned default last-GRADE parser; an invalid final token is unscored. This establishes parser fidelity, not an earlier-version advantage.pnpm exec vitest run --config comparisons/inspect/hard/vitest.config.ts --maxWorkers=2
Grader panel majority, clean councilequal accuracy, ahead on cost; held-out ahead on accuracymeasured hard and held-out comparison; adaptive ledger prefix is not yet the shipped default2026-10-13, Haiku 5.5. Hard: ours 115/120, audit 120/120, theirs 120/120. Held-out: ours 60/60, audit 60/60, theirs 59/60. Hard USD per 100: ours 0.07094, audit 0.06863, theirs 0.13238. Fresh seed: 40103. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results. Fresh legacy replay: 37/40 plus three unscored for both reducers. The explicit_confirmation rubric changes after recording; none of its three current request hashes matches any judge recording.CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts panel ours --scoped --replay; repeat with theirs and add --heldout for the fresh set. Then node --import tsx comparisons/inspect/hard/adaptive.mts.
Grader panel majority, borderline blocksequal accuracy, ahead on cost; held-out equal accuracy, ahead on costmeasured hard and held-out comparison; adaptive ledger prefix is not yet the shipped default2026-10-13, Haiku 5.5. Hard: ours 113/120, audit 120/120, theirs 120/120. Held-out: ours 56/60, audit 60/60, theirs 60/60. Hard USD per 100: ours 0.06856, audit 0.06722, theirs 0.12110. Fresh seed: 52130. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results.CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts borderline ours --scoped --replay; repeat with theirs and add --heldout for the fresh set. Then node --import tsx comparisons/inspect/hard/adaptive.mts.
mode reduceraheadmeasured head-to-head[1, unscored, unscored] gives 1.0 under Inspect; ours gives undecided, since a verdict needs more than half the repeats scored.npx vitest run packages/stats/test/repeat-reducers.test.ts. The Inspect side is comparisons/inspect/reducers/mode-extra.py.
median reducer (tie at one half)aheadmeasured head-to-headRepeats pass and fail under median: Inspect gives 0.5, which a gate of 0.5 or more reads as a pass; ours is undecided.Run comparisons/inspect/reducers/reducer-measure.py (Inspect side) and comparisons/inspect/reducers/reducer-measure-ours.ts with tsx (ours). Test: packages/stats/test/repeat-reducers.test.ts, "a median tie is undecided, never a half score".
at_least reducer (unscored repeats)aheadmeasured against our earlier versionRepeats [pass, inconclusive, fail] under at_least:2: Inspect returns fail; ours now returns undecided.npx vitest run packages/stats/test/repeat-reducers.test.ts, test "at_least is undecided when an unscored repeat could still make k passes". The Inspect side is comparisons/inspect/reducers/reducer-measure.py.
pass^k reducer (value)aheadmeasured head-to-head[pass, pass, unscored] at k = 2: Inspect gives 1.0 because it drops the unscored repeat; ours gives 0.333, with range [0.333, 1.000].npx tsx comparisons/inspect/intervals/passk-incomplete-ours.mts. The Inspect side is comparisons/inspect/intervals/passk-incomplete.py.
pass@k reducer, incomplete runsaheadmeasured against our earlier versionThree runs, two complete and passing, one incomplete, k = 2: value 1.0, with the range widened to cover both outcomes of the incomplete run and flagged incomplete. Before the change there was no range and no flag.npx vitest run packages/report/test/incomplete-runs.test.ts
Pass^k and pass@k intervals for every kaheadcompared against their docs or codeInspect's reducer returns one number with no interval. Ours gives a value and bounds for each k; the share of draws with the value inside its interval is 1.000 in every cell after the fix (0.250 before).npx vitest run packages/report/test/at-k.test.ts. Simulation: npx tsx comparisons/inspect/intervals/coverage-sim.mts raw (before the fix) and widened (current); compare the last line of each.
Built-in scorers (includes, match, exact)aheadmeasured against our earlier version"fiber" against "Fiber is here": failed before, passes now. "Mbps$" against "fiber at 300 Mbps.": failed before, passes now. "\b300\b" against "3000 Mbps plan" still fails, by design.npx vitest run packages/checks/test/must-match.test.ts. The Inspect side is comparisons/inspect/scorers/scorers-measure.py.
Sandbox exec output bound (10 MiB)aheadmeasured head-to-headA 20,000,000-character write to stdout: Inspect returns 10,431,744 bytes with no flag; ours returns 10,485,760 bytes with truncated set to true.The Inspect side is comparisons/inspect/sandbox/sandbox-measure.py (mock model, no spend; needs an environment with inspect_ai installed). Ours: npx tsx comparisons/inspect/sandbox/sandbox-measure-ours.mts, also tested in packages/sandbox/test/local.test.ts.
Sandbox file read bound (100 MiB)aheadmeasured head-to-headA FIFO that streams 150 MB: Inspect returns all 157,286,400 bytes; ours refuses a non-regular file at once. Equal on regular files of any size.The Inspect side is comparisons/inspect/sandbox/fifo-read.py (needs an environment with inspect_ai installed). Ours: npx tsx comparisons/inspect/sandbox/read-bound-stream-ours.mts.
Read path confinement (host and outside-workspace reads)aheadmeasured head-to-headread_file("/etc/hosts") returns 231 bytes in Inspect; ours raises SandboxPathError on it and on "../secret-outside.txt".npx tsx comparisons/inspect/sandbox/read-ours.mts. The Inspect side is comparisons/inspect/sandbox/sandbox-isolation.py.
Workspace walk bounds (files, depth, bytes hashed)aheadcompared against their docs or codeInspect's sandbox code, as read, has no workspace walk bound. Ours stops a snapshot past 20,000 files, 32 folder levels or 1 GiB hashed.Not reproducible from the repository yet. The limits are in packages/sandbox/src/limits.ts; no test file is named in the evidence.
Per-sample lifecycle: cleanup when a run is killed mid-sampleaheadmeasured against our earlier versionA command still running after SIGTERM to its parent: one process survived before the change and zero after, since it is now killed with its process group. Open: SIGKILL still leaves folders and commands behind.npx vitest run packages/sandbox/test/cleanup.test.ts
Container network default (none)limitcompared against their docs or code. Limit: no docker on this host.No example recorded in the evidence.npx vitest run packages/sandbox/test/docker.test.ts. The two real-docker tests skip without a docker host.
Tool approval for harness-run toolsaheadcompared against their docs or codeA deny on coverage.check: the vercel-ai tool's execute never runs, the model gets the refusal, and the recorded call carries "denied by tool policy rule coverage.check".npx vitest run packages/adapters/test/vercel-ai.test.ts packages/adapters/test/function.test.ts packages/adapters/test/process.test.ts, and npx vitest run packages/core/test/runner/tool-policy-run.test.ts.
Tool approval for remote runtimes (Nexus, claude-code, traces, SDK loops that own tool objects)limitcompared against their docs or code. Limit: self-running runtimes, structural.For Nexus, claude-code and traces, the harness receives a call only after the runtime has run it, so a denial is recorded but cannot stop the call.npx vitest run packages/adapters/test/http.test.ts (the pre-call test) and npx vitest run packages/core/test/runner/tool-policy-run.test.ts (the after-the-fact gate).
Eval log: per-run record and drilldownaheadmeasured against our earlier versionThe run-log test fails on the old drilldown (2 red) and passes on the new one. Fields shown per block and run: 14 before, 19 after.npx vitest run packages/report/test/run-log.test.ts
Eval log: event timeline as it happens, kept through a killaheadmeasured against our earlier versionBefore the change no events were emitted during a run, so a killed run kept nothing. Now two events are read back before any later write, and a line cut short by the kill is skipped.npx vitest run packages/core/test/runner/events.test.ts packages/core/test/runner/event-log.test.ts
Eval log: resume an interrupted sample from its last finished blockaheadmeasured against our earlier versionA run killed after two of four blocks resumes with two adapter sends, not four; before the change the same test fails with four sends.npx vitest run packages/core/test/runner/resume.test.ts

Not compared ​

  • Eval log format field for field. Inspect has per-model-call start times and a viewer, which ours does not have yet.
  • A killed run's log. Inspect was not run on killed runs, so no killed-run comparison exists.
  • Container network enforcement. This host has no docker, so the flag was checked, not the enforcement.
  • Approval before the call for self-running runtimes. Inspect's gate covers only the tools it defines, so there is no like-for-like run.
  • Workspace walk bounds. Read from the source only; no snapshot was run on either side.

MIT. driftbench is made by Nexus.