Skip to content

Chart data ​

tools/readme-charts.mjs reads the README chart numbers from result files of this repository. This folder holds the files that are not result files of a comparison.

chart-facts.csv ​

Written by the script on every run. It lists the title sentence of each chart and every number drawn, with the file the number was read from. The rows impact-blocks, impact-radius and impact-moj are the numbers the README quotes in prose for blocks, blast radius and mixture of judges. node tools/readme-charts.mjs --check exits 1 when a chart or this file no longer matches its sources.

mm-eval-sealed-test.json ​

A copy of research/astra-mm/data/test-paired.json, with the same values and this repository's JSON formatting, at commit a7800af8d82459d2850f6478b0d3e5e5fde6edfb of the branch research/astra-mm-replicate. That branch is not merged into main, so the file is copied here for the chart to read.

  • What it measures: the MM-Eval benchmark (arXiv 2410.17578), five core subsets, scored with the benchmark's own rule. The first judge is Claude Haiku 5.5. When its two readings of a pair do not agree decisively, a panel of three other model families votes.
  • Items: 1,212, every remaining test item that no earlier run had scored. The list was sealed at commit 2154225b, before the first call, and scored once.
  • Result: 89.49 alone, 91.54 with the panel, a paired gain of 2.05 points, 95 percent interval 1.02 to 3.29 (paired bootstrap, 20,000 draws).
  • Limits stated by the source: three of the five subset intervals touch zero. Three safety items were provider refusals that the panel recovered; without them the gain is 1.75, interval 0.77 to 2.96. Both arms are driftbench; no other tool is run.

The full write-up, the seal and every recorded call are in RESULT.md on that commit. The development-set replay that ships on main is in examples/escalation.

sealed/ ​

Copies of result files from research branches that are not merged into main, with the same values and this repository's JSON formatting. In sealed/verify.json the field key is renamed to cellId in the published copy; its 10 values, which are cache cell hashes, are unchanged.

FileBranch, commitPath on that branch
sealed/judgebench.jsonresearch/ledger-jb, ed4412072de2f7904879cbb397ecd2306673d7f6examples/ledger-jb/results/cells.json
sealed/mm-eval.jsonresearch/ledger-mm, c61e2b24a6be30758ea776edf9185babc138bba2research/ledger-mm/test-results.json
sealed/rubricbench-luna.jsonresearch/ledger-rb, d36a01002409fd23f91a234078d5c95559741e86examples/judge-benchmarks/experiments/ledger-rb/luna-complete-check.json
sealed/rubricbench-flash-lite.jsonresearch/ledger-rb, d36a01002409fd23f91a234078d5c95559741e86examples/judge-benchmarks/experiments/ledger-rb/flash-lite/live-summary.json
sealed/verify.jsonresearch/ledger-verify, 1f94d87ed373790819fa2565e4f63a65ebff76f1examples/ledger-verify/evidence/cells.json
sealed/writing.jsonresearch/ledger-wp, 5db7e7adaae3b2c7d669630e12c0c9eb1548f6d7examples/ledger-wp/results/cells.json
sealed/rubricbench-haiku.jsonresearch/ledger-rb, fe76cd48b9078924d4404e5af94ff5f0cd494ca8examples/judge-benchmarks/experiments/ledger-rb/live-summary.json

What they measure and their limits: how the README numbers were measured.

MIT. driftbench is made by Nexus.