THE JUDGE TEST — 2026-10-15 — seven LLM judges, each handed the merge message of commit 781a3fb81e and nothing else.
Judge 1: Sonnet (Claude Code subagent, its own prompt, below the six). Judges 2-7: claude -p, one turn, no tools, run from an empty directory, prompt below.

===== PROMPT (judges 2-7) =====
You are an LLM-as-a-judge reviewing a git merge commit message (below). It lists failing test files after a merge and gives a reason for each. Answer in under 200 words: (1) for each reason category, PLAUSIBLE / WEAK / LIKELY WRONG with one line; (2) the single most likely root cause the message missed, naming the specific artefact, with confidence 0-100; (3) could you tell from this message alone whether the binaries the tests ran were built before or after the merge? yes/no. Use no tools.

----- MERGE MESSAGE -----
merge(main): main's HUD (C768–C810: the renumbered snowball rows, the napi and node engines on one shape, the main-window design, the odometer) into the branch — append logs unioned, state files main's, the branch's duplicate earcon door dropped for main's C776

Main had merged this branch at 2332bfe9b3 (rows renumbered C768–C780, main's 3×3 kept) and went on; this branch had three later commits (C760's earcon reading, the boundary odometer, their ledger rows) that main lacks. Resolution onto main's layout: every .ndjson conflict unioned (append logs keep both sides' rows), every .json state file and spec-tree.txt taken from main, the asks ledgers and HUD-FORMAL-METHODS.md taken from main (I7 is present there), hud-readings.mjs taken from main, and the branch's tests/vna/c760-build-the-same-cells-on-both.test.mjs plus scripts/vna/hud-earcon.mjs removed: main's C776 measures the earcon inline in the instrument and its guard tests/vna/c776-*.test.mjs is the home; main's C760 is a different row with its own guard. Five untracked bundler copies under packages/thetacog-mcp/scripts that main now tracks were removed before the merge.

Verified on the merged tree: intentguard cargo test 91 passed, wasm check clean; pmu-rust cargo test 95 passed; rust-crate-mirror and rust-source-single-truth 8/8; tsc --noEmit 0 errors. HUD guard set (66 files): red files are c741 c749 c751 c752 c755 c758 c766 c770 c771 c773 c774 c775 c776 c783 c785 c786 c78c, every one of them one of: a Mac-recorded fixture against this box's walk (c741 c749#7 c751 c752), a config key main added after the fixture was recorded (c749#6: tests/fixtures/hud-serve and data/vna/hud-config.json are byte-identical to main's, so the case reads the same on main), the napi addon, vsix, receipt and carve on this box (c755 c758 c766 c771 c773 c774 c775 c776#2 c78c), the walk's words on this box's reef (c783 c785), the readings file's run join after the union (c770), and Playwright's pinned headless shell absent here (c786). None is a conflict resolution; each is named so the Mac reads it against its own run.


===== ANSWER: claude-haiku-4-5-20251001-1 =====
# Review of Merge Commit Message

## Reason Categories

- **Platform fixtures (c741, c749#7, c751, c752):** PLAUSIBLE — Mac-recorded golden files legitimately differ across machines
- **Config key timing (c749#6):** WEAK — author acknowledges byte-identical files to main, so this shouldn't fail
- **Platform binaries (napi, vsix, c755 c758 c766 c771 c773 c774 c775 c776#2 c78c):** PLAUSIBLE — architecture and build artifacts vary by system
- **Local environment (c783, c785):** PLAUSIBLE — host-specific paths and resources expected
- **Readings file sequencing (c770):** PLAUSIBLE — union-after-join timing effects are real
- **Playwright shell (c786):** PLAUSIBLE — missing pinned browser binary explains test skip

---

## Most Likely Missed Root Cause

**Branch code doesn't conform to main's formal specifications in HUD-FORMAL-METHODS.md** — confidence ~60%. The message mentions taking this file from main, but the branch's HUD implementation (c760, c770, c783-c785 rows) may encode logic that violates those specs. The clustering of "guard" test failures across unrelated platforms suggests specification drift, not environmental variance.

**Artifact:** `HUD-FORMAL-METHODS.md` used to validate merged code.

---

## Binary Build Timing

**No.** The message verifies cargo/tsc compilation succeeded, but doesn't state whether HUD guard tests ran *after* rebuilding HUD binaries post-merge. The untracked bundler copies removal adds ambiguity about whether tests used stale or fresh builds.

===== ANSWER: claude-haiku-4-5-20251001-2 =====
## Plausibility Assessment

1. **Platform-specific fixtures (c741, c749#7, c751, c752)**: PLAUSIBLE — environment divergence between record machine and test box is a known source of snapshot test flakes.
2. **Config key after fixture recorded (c749#6)**: WEAK — the message immediately contradicts itself ("byte-identical to main's, so the case reads the same on main"), suggesting the stated reason doesn't match the actual observation.
3. **Platform artifacts: napi/vsix/carve (c755–c776#2, c78c)**: PLAUSIBLE — native bindings and platform-specific outputs fail consistently across machines.
4. **"Walk's words on this box's reef" (c783, c785)**: WEAK — opaque terminology; unclear what causal mechanism this names.
5. **NDJSON union join state (c770)**: PLAUSIBLE — but underspecified; could be duplicates, ordering, or truncation from append logic.
6. **Playwright headless missing (c786)**: PLAUSIBLE — environment dependency.

## Most Likely Missed Root Cause

**The NDJSON append-union strategy itself**, confidence ~75. The message says "append logs unioned" but doesn't validate that concatenation preserved log ordering, completeness, or test preconditions. If tests expected deterministic or deduplicated logs and the union concatenated raw records, downstream assertions (especially c770's "run join") would fail. The specific artefact: how `append` was performed on `.ndjson` files during conflict resolution.

## Binaries Built Before or After Merge?

**Partial yes.** The message explicitly states "Verified on the merged tree: …cargo test…tsc --noEmit…" — so Rust and TypeScript builds clearly ran post-merge. **But** for the "HUD guard set" failures (the "red files"), the message never states whether those tests ran against freshly-built binaries or cached pre-merge ones. Ambiguous for the main test suite.

===== ANSWER: claude-haiku-4-5-20251001-3 =====
# Evaluation

**Reason categories:**

1. Mac-recorded fixture vs this box's environment: **PLAUSIBLE** — fixtures recorded on different hardware/OS naturally diverge.
2. Config drift (main added key after fixture recorded): **PLAUSIBLE** — explains why c749#6 specifically fails even though main's current config is byte-identical.
3. Platform-specific binaries (napi, vsix, carve): **WEAK** — vague mechanism; doesn't explain whether rebuilds happened or why binaries would be stale on a merged tree.
4. Path/environment specifics (reef words): **PLAUSIBLE** — reasonable environmental variance.
5. Union of append logs in .ndjson files: **PLAUSIBLE** — merge conflict resolution strategy could affect test state/ordering.
6. Playwright shell absent: **PLAUSIBLE** — straightforward infrastructure gap.

**Most likely root cause missed:**

**JavaScript bundler not re-run post-merge** (confidence: 68). The message verifies `cargo test` and `tsc --noEmit` on the merged tree, but `tsc --noEmit` only type-checks—it doesn't compile or bundle. The removal of "five untracked bundler copies" suggests build artifacts exist; if they weren't regenerated after conflict resolution, tests consuming .js outputs would fail with stale code. Rust and type-level checks pass cleanly, but runtime JS failures mask a stale build.

**Binaries before/after merge:**

**Yes.** Cargo tests passed "on the merged tree" — `cargo test` requires compilation post-test-setup, implying binaries were built after the merge. Failures are environmental/fixture issues, not compilation failures, supporting recent compilation.

===== ANSWER: claude-sonnet-5-1 =====
**1) Reason categories**
- Mac-recorded fixture vs. this box's walk (c741 c749#7 c751 c752): PLAUSIBLE — classic recorded-fixture/host drift, consistent with the rest of the message's framing.
- Config key added after fixture recorded (c749#6): WEAK — the message itself undercuts this by noting the fixture and config are byte-identical to main's, i.e. it "reads the same on main," so the stated cause doesn't actually explain a local-only failure.
- napi addon/vsix/receipt/carve "on this box" (9 files): WEAK — too many distinct artifacts (native binary, packaging, receipt, carve) collapsed into one vague "on this box" label with no per-file mechanism given.
- "Walk's words on this box's reef" (c783 c785): LIKELY WRONG — opaque phrasing that doesn't actually specify a mechanism; reads as filler rather than diagnosis.
- Readings file run-join after union (c770): PLAUSIBLE — directly follows from the stated .ndjson union resolution.
- Playwright pinned headless shell absent (c786): PLAUSIBLE — standard, verifiable environment gap.

**2) Missed root cause:** stale compiled napi `.node` binary never rebuilt after the merge (the "napi addon" files are blamed on host environment, but a merge that touches native-engine code demands a rebuild — confidence 55).

**3) No** — the message never states build timing for the napi addon/vsix relative to the merge commit.

===== ANSWER: claude-sonnet-5-2 =====
**Per-category verdicts:**

1. Mac fixture vs. box's walk (c741/c749#7/c751/c752) — PLAUSIBLE, standard cross-platform fixture skew.
2. Config key added after fixture (c749#6) — PLAUSIBLE, actually backed by a byte-identical check, strongest claim in the message.
3. Napi addon/vsix/receipt/carve bucket (9 files) — WEAK: nine unrelated red files lumped under one vague "on this box" label, no per-file evidence, and it's the one bucket touching a native compiled artifact.
4. "Walk's words on this box's reef" (c783/c785) — LIKELY WRONG: no concrete mechanism given, pure hand-wave.
5. Readings file run-join after union (c770) — PLAUSIBLE, directly tied to the stated append-log resolution.
6. Playwright headless shell absent — PLAUSIBLE, common and checkable.

**Most likely missed root cause (confidence 60):** the native napi addon binary (`.node` file) wasn't rebuilt against the merged source — the merge explicitly touches "napi and node engines on one shape," yet nothing in the verification list (cargo test, tsc, wasm check) actually rebuilds/relinks that addon, so bucket 3's napi-related failures could be stale-binary failures misclassified as environment quirks.

**(3) Can you tell build timing from this message alone?** No — the verification list names cargo test/tsc runs but never states whether the napi addon (or other native artifacts) was rebuilt before those tests executed.

===== ANSWER: claude-sonnet-5-3 =====
**1) Reason categories**

- Mac-recorded fixtures vs. this box's walk (c741/c749#7/c751/c752): PLAUSIBLE — common platform-path drift, though asserted for 4 files without individual evidence.
- Config key added after fixture recorded (c749#6): PLAUSIBLE — backed by a concrete byte-identical check, strongest-supported claim in the message.
- napi/vsix/receipt/carve artifacts "on this box" (c755/c758/c766/c771/c773/c774/c775/c776#2/c78c): WEAK — plausible category but used as a blanket label for 9 files with no per-file detail; risks hiding a real build-artifact regression.
- "Walk's words on this box's reef" (c783/c785): WEAK — vague, metaphorical, unverifiable from the text alone.
- Readings-file run-join after union (c770): LIKELY WRONG — this directly concerns hud-readings.mjs reacting to the just-performed .ndjson union, yet the message claims "none is a conflict resolution." That's a contradiction, not an environmental excuse.

**2) Most likely missed root cause:** the unioned append-log `.ndjson` files now contain duplicate/out-of-order rows that `hud-readings.mjs` (taken from main) doesn't dedupe/sort correctly — c770 is a real regression from the union strategy, not a "this box" artifact. Confidence: 55.

**3) No** — the message never states whether the napi/vsix/wasm binaries were rebuilt after the merge or are stale pre-merge artifacts; "verified on the merged tree" covers source-level checks only.

===== ANSWER: judge-1 (Sonnet subagent) =====
PROMPT (judge 1): "You are acting as an LLM-as-a-judge reviewing an engineer's merge report. Read ONLY this file [the merge message]. ... 1. For each reason category the message gives, rate it PLAUSIBLE / WEAK / LIKELY WRONG, with one line why. 2. Name the single most likely real root cause of the red tests that the message might have missed, if any, and your confidence (0–100). 3. State plainly: could you determine from this message alone whether the binaries the tests ran were built before or after the merge? yes/no and why."

ANSWER:
1. Reason categories, rated:
- "Mac-recorded fixture against this box's walk" (c741, c749#7, c751, c752) — PLAUSIBLE. Machine-specific recorded fixtures diverging from a live walk on a different box is a common, well-understood failure mode.
- "Config key main added after fixture recorded" (c749#6) — WEAK. The message asserts the config file is "byte-identical to main's," then concludes "the case reads the same on main" — but that's an unverified inference, not a check actually run against main's CI. It explains away a red test by assuming an equivalent failure exists elsewhere, without evidence.
- "napi addon, vsix, receipt and carve on this box" — PLAUSIBLE as a category (native/packaged artifacts are classically box-specific), but it's also the vaguest grouping — seven different test IDs lumped under one phrase with no per-test mechanism given.
- "Walk's words on this box's reef" (c783, c785) — WEAK. Too compressed/jargon-heavy to independently verify; reads as a label rather than a diagnosis.
- "Readings file's run join after the union" (c770) — PLAUSIBLE, and the most concrete: it ties directly to the merge's own ndjson-union resolution, a real mechanism that could produce a join mismatch.
- "Playwright's pinned headless shell absent" — PLAUSIBLE, standard environment-dependency gap.
2. Missed root cause: Stale pre-merge build artifacts (napi addon / vsix) being tested against post-merge source rather than true environment variance. Confidence: 45.
3. Could you tell if binaries were built before or after the merge? No. `cargo test` implies Rust recompiles, but nothing in the message states the napi addon or vsix were rebuilt post-merge before the "red" verdicts were recorded — it only asserts these failures are "on this box," never that they were tested against freshly built artifacts.

===== WHAT HAPPENED (commit a97c613b5a, eleven minutes after the merge) =====
The tests ran the canonical pmu-onchip binary, a 15:00 build from before the merge; only the mirror crate had been rebuilt. Rebuilding the canonical crate turned c741 c749 c751 c755 c758 c783 c785 green.