Skip to content

Reader-first judge (length-aware) ​

An explicit opt-in configuration favors the longer reply unless machine checks and the five-criterion model rubric justify a shorter-reply exception. It remains separate from the default judge.

Frozen DEV ​

Report timestamp: 2026-10-13T04:53:41.183Z. TEST checkpoint: 2026-10-13T04:48:25.497Z.

All 90 DEV pairs are covered, with 180 scored games. Each pair has two model presentations; code pools their scores, so the two reported games share one decision. Both presentations retain separate quotes and scores in the raw report.

VariantCorrect gamesGames gained vs lengthGames lost vs lengthUSD per 100 pairs
Length alone78.89% (142/180)000
Features plus length78.89% (142/180)000
Reader-first judge (length-aware): features plus judge plus length80.00% (144/180)420.81568
Frozen holistic, historical64.44% (116/180)n/an/a0.57181

Selected rule: judgeMargin=4, minFailures=1, featureMargin=5. The final full DEV replay runs after the selection commit 7797431e936dfcc7555aac6b91ac845f98a94e3e. Freeze record cites that DEV result and the exact source hashes. The committed frozen judge is c50b2dc3f0c6c3e9bc3b82d848fb9bc1a98451c4. Current evidence commit before this report: 8e6d2cc2959d1500fec8dfcef8b777921c682779.

The shorter reply wins only when both ballots are valid, its pooled sum advantage reaches judgeMargin, and minFailures criteria have longer<=2, shorter>=3 and a machine flag present only in the longer reply. An invalid ballot retains the length default with an explicit invalid flag. Equal lengths tie. No holistic or competitor ballot enters this calculation.

Machine feature firing on DEV &ZeroWidthSpace;

Feature familyReply flagsPairs with flagOne-sided pairsQualifying supported failures
structure0/1800/9000
economy57/18041/90256
directness0/1800/9000
context85/18052/90192
simplicity4/1804/9042
  • Length: English whitespace units, Chinese Han characters. It exactly reproduces the supplied 142/180 baseline.
  • Structure: length>500, no heading/list, and one paragraph or mean paragraph length>250 English / 400 Chinese.
  • Economy: filler+hedge phrases>6 per 1,000 units, duplicate sentences>0.08, or redundant four-grams>0.15.
  • Directness: litotes>1 per 1,000 units.
  • Context: undefined acronyms>3, unexplained dictionary jargon>3, or context-reference phrases>1 per 1,000 units.
  • Simplicity: mean sentence length>35 English / 60 Chinese, or estimated English reading grade>14. Chinese has no English reading-grade estimate.

All raw feature values, including headings, lists, paragraph counts, means and densities, are retained for every reply. These are heuristic flags, not semantic proof. Requested repetition and appropriate literary forms remain legitimate under the model rubric. Invalid-evidence fallbacks: 22/90 pairs.

DEV changes versus longer-wins &ZeroWidthSpace;

PairLanguageA / B lengthLongerReader-firstGoldEffectSupported failures
d3f7457d-11ad-4f02-a849-31518f5a729d:c8879d119baezh1180 / 2816BAAgaineconomy
8a981764-d2ff-44a0-a3bb-78090ec8107f:301c17a7b13bzh1701 / 2753BAAgaincontext
43b647a6-2c1f-4f0b-95b3-da61f793802c:f227cf6be572en5187 / 813ABAlosscontext, simplicity

Machine-only changes versus longer-wins &ZeroWidthSpace;

PairLanguageA / B lengthLongerFeaturesGoldEffectSupport

No decisions differ from length alone.

Every DEV threshold variant &ZeroWidthSpace;

The grid is declared before completion of the exploratory DEV run. Selection maximizes combined DEV accuracy, then minimizes overrides, then prefers a larger judge margin and failure count. The feature-only margin is selected independently, preferring a larger margin on ties. Final DEV replay confirms the chosen rule without new calls. This is tuning on DEV, not a fresh DEV estimate.

Judge marginMinimum failuresDEV games correctOverride pairsGain pairsLoss pairs
41144/180321
103142/180000
102142/180000
101142/180000
83142/180000
82142/180000
81142/180000
63142/180000
53142/180000
43142/180000
33142/180000
23142/180000
31142/180422
21142/180422
62140/180101
61140/180101
52140/180101
51140/180101
42140/180101
32140/180101
22140/180101
Feature marginDEV games correctOverride pairsGain pairsLoss pairs
5142/180000
4142/180000
3142/180000
2142/180211
1114/18018216

TEST confirmation &ZeroWidthSpace;

One frozen TEST run, continued once after a session-limit interruption, covers all 173 pairs and 346 games. Successful prompts replay from cache without repeat calls. The source hashes match the committed freeze. TEST never selects a threshold. Earlier studies already evaluated this split; this is confirmation on a previously evaluated split.

VariantCorrect gamesGames gained vs lengthGames lost vs lengthUSD per 100 pairs
Length alone69.94% (242/346)000
Features plus length69.94% (242/346)000
Reader-first judge (length-aware): features plus judge plus length71.68% (248/346)600.79618
Frozen holistic, historical58.96% (204/346)n/an/a0.54818
promptfoo, historical62.72% (217/346)n/an/a0.40649

Combined minus holistic: 44 games. Combined minus promptfoo: 31 games. Combined minus length: 6 games. These are observed counts, not significance or general-superiority claims.

Feature familyReply flagsPairs with flagOne-sided pairsQualifying supported failures
structure0/3460/17300
economy83/34655/173273
directness2/3462/17320
context147/34692/173373
simplicity8/3467/17362

TEST changes versus longer-wins &ZeroWidthSpace;

PairLanguageA / B lengthLongerReader-firstGoldEffectSupported failures
c8156bb9-f3c1-492c-84fe-1466333c128b:374f6aace872zh746 / 3549BAAgaincontext
ab7b3fb5-a0f1-4c7b-a04f-68e90a175261:9ef6b2a28174zh1408 / 2826BAAgaincontext
65fca3a9-6621-4162-a477-33083c83b1f4:bdb0d59fd0f1zh396 / 14334BAAgaineconomy, simplicity

Invalid-evidence fallbacks: 74/173 pairs.

Cost, safety and default preservation &ZeroWidthSpace;

Returned spend: USD 2.11150276, 526 returned calls. Interrupted-call uncertainty reserve: USD 0.20. Total accounted exposure: USD 2.31150276 of USD 20.00. The original two-worker process is stopped at the owner's concurrency update; its returned calls replay for free. Subsequent concurrency is at most six. The initial new-call cutoff is 2026-10-13 06:39 Brussels. The owner then authorizes immediate continuation after the 06:40 session reset, until the network drops, and another continuation after the outage with the same freeze. The continuation has a 07:40 Brussels safety deadline before the original 07:45 completion target. Recipe costs count both required orders of completed pairs, including replayed calls, and are quoted per 100 pairs (200 presentations). Unfinished orders are excluded from that recipe rate but included in returned spend.

All 42 pre-existing judge source files except the export index are byte-identical to base d702d90b. The index adds named exports. The full judges suite passes 602 tests with two test workers and a 60-second per-test timeout; the focused golden and configuration run passes 41 tests, including 1,028 legacy cassette entries. The new reader-first fixtures pass 20 tests. Root and research TypeScript checks pass. The package and its workspace dependencies build successfully, including type declarations; the built package export smoke check passes. A first package-only build fails because dependency declarations are not built, then the dependency-inclusive build resolves it. Two runs with the default five-second timeout time out during a dynamic import in an unchanged pairwise test; the run with a 60-second timeout passes without a code change. All nine caller cutoff and cost tests pass using fake processes, with no paid calls. After TEST completes, the single formatter correction is applied. The measured source remains at its frozen commit; format-only-proof.json verifies an identical emitted JavaScript syntax tree. All 20 reader-first fixture tests pass again, and the targeted Biome check exits successfully with warnings only.

The package API is createReaderFirstJudge(client, { configuration: 'reader-first' }). It is never selected automatically. The doc section is docs/concepts/judges.md. Default source hashes are in results/default-proof.json. Rules and source provenance are in README.md; each criterion's source quotes, scores, machine measurements and decisions are in results/final-dev.json and results/test.json when present. Raw responses are in recordings/haiku.jsonl. One approved benchmark contact is consistently replaced in loaded candidates, recordings and exports; see comparisons/reader-first/recordings/REDACTION.md. Other public benchmark contacts retain their scoped guard annotations. Model-generated punctuation in raw evidence is preserved.

Historical baselines are supplied by the prior holistic and protocol reports. Length bias is deliberately used here. A better score on these labels does not establish better writing evaluation elsewhere. No default row is merged with this configuration.

MIT. driftbench is made by Nexus.