Inspect: comparison ​
Inspect (inspect_ai) is a framework for evaluating language model applications. This page compares driftbench with inspect_ai version 0.3.278, read in its source at commit a1dbea8.
Results come from the comparison table. Limit means a row is not being worked, with its reason stated.
| Capability | Result | Basis | Example | Reproduce |
|---|---|---|---|---|
| Model-graded QA prompt (closed question, GRADE letter) | ahead on accuracy; held-out ahead on accuracy | measured hard and held-out comparison; compact ledger with bounded retry is not yet the shipped default | 2026-10-13, Haiku 5.5. Hard: ours 111/120, audit 120/120, theirs 115/120. Held-out: ours 52/60, audit 60/60, theirs 57/60. Hard USD per 100: ours 0.04450, audit 0.05695, theirs 0.05867. Fresh seed: 61326. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results. Original baseline stays 111/120 hard and 48/60 held-out; the original under-review 1270e1cb run is 114/120 and 44/60 on those identical inputs. | CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts qa ours --compact --replay; repeat with theirs and add --heldout for the fresh set. Then CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts qa audit --compact --replay; add --heldout for the fresh set. |
| Model-graded fact template (contains the expert content) | equal accuracy; held-out ahead on accuracy | measured hard and held-out comparison; compact ledger with bounded retry is not yet the shipped default | 2026-10-13, Haiku 5.5. Hard: ours 111/120, audit 120/120, theirs 120/120. Held-out: ours 52/60, audit 60/60, theirs 59/60. Hard USD per 100: ours 0.04450, audit 0.05695, theirs 0.05557. Fresh seed: 61326. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results. Original baseline stays 111/120 hard and 48/60 held-out; the original under-review 1270e1cb run is 114/120 and 44/60 on those identical inputs. The earlier 20/20 cost claim uses the QA template and cannot establish fact-template performance. | CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts fact ours --compact --replay; repeat with theirs and add --heldout for the fresh set. Then CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts fact audit --compact --replay; add --heldout for the fresh set. |
| GRADE verdict parse (last GRADE token) | limit | Historical improvement claim has no recorded example or reproducible earlier-version baseline. | The hard harness mirrors the pinned default last-GRADE parser; an invalid final token is unscored. This establishes parser fidelity, not an earlier-version advantage. | pnpm exec vitest run --config comparisons/inspect/hard/vitest.config.ts --maxWorkers=2 |
| Grader panel majority, clean council | equal accuracy, ahead on cost; held-out ahead on accuracy | measured hard and held-out comparison; adaptive ledger prefix is not yet the shipped default | 2026-10-13, Haiku 5.5. Hard: ours 115/120, audit 120/120, theirs 120/120. Held-out: ours 60/60, audit 60/60, theirs 59/60. Hard USD per 100: ours 0.07094, audit 0.06863, theirs 0.13238. Fresh seed: 40103. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results. Fresh legacy replay: 37/40 plus three unscored for both reducers. The explicit_confirmation rubric changes after recording; none of its three current request hashes matches any judge recording. | CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts panel ours --scoped --replay; repeat with theirs and add --heldout for the fresh set. Then node --import tsx comparisons/inspect/hard/adaptive.mts. |
| Grader panel majority, borderline blocks | equal accuracy, ahead on cost; held-out equal accuracy, ahead on cost | measured hard and held-out comparison; adaptive ledger prefix is not yet the shipped default | 2026-10-13, Haiku 5.5. Hard: ours 113/120, audit 120/120, theirs 120/120. Held-out: ours 56/60, audit 60/60, theirs 60/60. Hard USD per 100: ours 0.06856, audit 0.06722, theirs 0.12110. Fresh seed: 52130. Every hard row requests 120 cases; held-out requests 60. Unscored counts and Wilson intervals are in the results. | CAP_USD=0 node --import tsx comparisons/inspect/hard/run.mts borderline ours --scoped --replay; repeat with theirs and add --heldout for the fresh set. Then node --import tsx comparisons/inspect/hard/adaptive.mts. |
| mode reducer | ahead | measured head-to-head | [1, unscored, unscored] gives 1.0 under Inspect; ours gives undecided, since a verdict needs more than half the repeats scored. | npx vitest run packages/stats/test/repeat-reducers.test.ts. The Inspect side is comparisons/inspect/reducers/mode-extra.py. |
| median reducer (tie at one half) | ahead | measured head-to-head | Repeats pass and fail under median: Inspect gives 0.5, which a gate of 0.5 or more reads as a pass; ours is undecided. | Run comparisons/inspect/reducers/reducer-measure.py (Inspect side) and comparisons/inspect/reducers/reducer-measure-ours.ts with tsx (ours). Test: packages/stats/test/repeat-reducers.test.ts, "a median tie is undecided, never a half score". |
| at_least reducer (unscored repeats) | ahead | measured against our earlier version | Repeats [pass, inconclusive, fail] under at_least:2: Inspect returns fail; ours now returns undecided. | npx vitest run packages/stats/test/repeat-reducers.test.ts, test "at_least is undecided when an unscored repeat could still make k passes". The Inspect side is comparisons/inspect/reducers/reducer-measure.py. |
| pass^k reducer (value) | ahead | measured head-to-head | [pass, pass, unscored] at k = 2: Inspect gives 1.0 because it drops the unscored repeat; ours gives 0.333, with range [0.333, 1.000]. | npx tsx comparisons/inspect/intervals/passk-incomplete-ours.mts. The Inspect side is comparisons/inspect/intervals/passk-incomplete.py. |
| pass@k reducer, incomplete runs | ahead | measured against our earlier version | Three runs, two complete and passing, one incomplete, k = 2: value 1.0, with the range widened to cover both outcomes of the incomplete run and flagged incomplete. Before the change there was no range and no flag. | npx vitest run packages/report/test/incomplete-runs.test.ts |
| Pass^k and pass@k intervals for every k | ahead | compared against their docs or code | Inspect's reducer returns one number with no interval. Ours gives a value and bounds for each k; the share of draws with the value inside its interval is 1.000 in every cell after the fix (0.250 before). | npx vitest run packages/report/test/at-k.test.ts. Simulation: npx tsx comparisons/inspect/intervals/coverage-sim.mts raw (before the fix) and widened (current); compare the last line of each. |
| Built-in scorers (includes, match, exact) | ahead | measured against our earlier version | "fiber" against "Fiber is here": failed before, passes now. "Mbps$" against "fiber at 300 Mbps.": failed before, passes now. "\b300\b" against "3000 Mbps plan" still fails, by design. | npx vitest run packages/checks/test/must-match.test.ts. The Inspect side is comparisons/inspect/scorers/scorers-measure.py. |
| Sandbox exec output bound (10 MiB) | ahead | measured head-to-head | A 20,000,000-character write to stdout: Inspect returns 10,431,744 bytes with no flag; ours returns 10,485,760 bytes with truncated set to true. | The Inspect side is comparisons/inspect/sandbox/sandbox-measure.py (mock model, no spend; needs an environment with inspect_ai installed). Ours: npx tsx comparisons/inspect/sandbox/sandbox-measure-ours.mts, also tested in packages/sandbox/test/local.test.ts. |
| Sandbox file read bound (100 MiB) | ahead | measured head-to-head | A FIFO that streams 150 MB: Inspect returns all 157,286,400 bytes; ours refuses a non-regular file at once. Equal on regular files of any size. | The Inspect side is comparisons/inspect/sandbox/fifo-read.py (needs an environment with inspect_ai installed). Ours: npx tsx comparisons/inspect/sandbox/read-bound-stream-ours.mts. |
| Read path confinement (host and outside-workspace reads) | ahead | measured head-to-head | read_file("/etc/hosts") returns 231 bytes in Inspect; ours raises SandboxPathError on it and on "../secret-outside.txt". | npx tsx comparisons/inspect/sandbox/read-ours.mts. The Inspect side is comparisons/inspect/sandbox/sandbox-isolation.py. |
| Workspace walk bounds (files, depth, bytes hashed) | ahead | compared against their docs or code | Inspect's sandbox code, as read, has no workspace walk bound. Ours stops a snapshot past 20,000 files, 32 folder levels or 1 GiB hashed. | Not reproducible from the repository yet. The limits are in packages/sandbox/src/limits.ts; no test file is named in the evidence. |
| Per-sample lifecycle: cleanup when a run is killed mid-sample | ahead | measured against our earlier version | A command still running after SIGTERM to its parent: one process survived before the change and zero after, since it is now killed with its process group. Open: SIGKILL still leaves folders and commands behind. | npx vitest run packages/sandbox/test/cleanup.test.ts |
| Container network default (none) | limit | compared against their docs or code. Limit: no docker on this host. | No example recorded in the evidence. | npx vitest run packages/sandbox/test/docker.test.ts. The two real-docker tests skip without a docker host. |
| Tool approval for harness-run tools | ahead | compared against their docs or code | A deny on coverage.check: the vercel-ai tool's execute never runs, the model gets the refusal, and the recorded call carries "denied by tool policy rule coverage.check". | npx vitest run packages/adapters/test/vercel-ai.test.ts packages/adapters/test/function.test.ts packages/adapters/test/process.test.ts, and npx vitest run packages/core/test/runner/tool-policy-run.test.ts. |
| Tool approval for remote runtimes (Nexus, claude-code, traces, SDK loops that own tool objects) | limit | compared against their docs or code. Limit: self-running runtimes, structural. | For Nexus, claude-code and traces, the harness receives a call only after the runtime has run it, so a denial is recorded but cannot stop the call. | npx vitest run packages/adapters/test/http.test.ts (the pre-call test) and npx vitest run packages/core/test/runner/tool-policy-run.test.ts (the after-the-fact gate). |
| Eval log: per-run record and drilldown | ahead | measured against our earlier version | The run-log test fails on the old drilldown (2 red) and passes on the new one. Fields shown per block and run: 14 before, 19 after. | npx vitest run packages/report/test/run-log.test.ts |
| Eval log: event timeline as it happens, kept through a kill | ahead | measured against our earlier version | Before the change no events were emitted during a run, so a killed run kept nothing. Now two events are read back before any later write, and a line cut short by the kill is skipped. | npx vitest run packages/core/test/runner/events.test.ts packages/core/test/runner/event-log.test.ts |
| Eval log: resume an interrupted sample from its last finished block | ahead | measured against our earlier version | A run killed after two of four blocks resumes with two adapter sends, not four; before the change the same test fails with four sends. | npx vitest run packages/core/test/runner/resume.test.ts |
Not compared ​
- Eval log format field for field. Inspect has per-model-call start times and a viewer, which ours does not have yet.
- A killed run's log. Inspect was not run on killed runs, so no killed-run comparison exists.
- Container network enforcement. This host has no docker, so the flag was checked, not the enforcement.
- Approval before the call for self-running runtimes. Inspect's gate covers only the tools it defines, so there is no like-for-like run.
- Workspace walk bounds. Read from the source only; no snapshot was run on either side.