FABRIC-2.md: ACL-RWT-style 3x3 Latin square re-run on Stadium substrate (item 5.1/F.3)

Found and fixed a real bug before any campaign work could start: EXEC-DOE's
own CSV output was almost entirely lost to console interleaving with the
routine per-tick heartbeat export -- same bug class as Section L's PLOT
case. Fix: HB-OFF immediately before EXEC-DOE, HB-ON after DOE: complete.
Confirmed HB-ON-first (the reverse order) does NOT fix it -- tested
directly, row loss recurred identically.

Also found: L8-DOE/WL-HI/WL-LO (the mechanism bare_metal/README.md
describes as auto-run) don't exist anywhere in capsules/, and Makefile.
starkernel's DOE_SEED variable is declared but never referenced -- both
vestigial, matching Section K's earlier staleness finding.

Built QEMU-serial-socket injection tooling (socat) to drive EXEC-DOE
interactively after boot, since it requires live REPL input, not just
observation. Two real defects found and fixed in that tooling itself: a
log-discovery race (self-excluding the very log it needed to find,
causing two separate stuck-injector incidents, one overnight) and an
unredirected background launch that deadlocked socat on a full stdout
pipe. Both fixed by having the orchestrator pass exact log/socket paths
directly and always launching through the harness's tracked-background
mechanism.

First full campaign attempt ran all 9 cells as three ISA-blocked loops,
reusing one build per architecture -- caught mid-run: this confounds ISA
with time/session-order, invalidating the Latin square design. Discarded
(logs kept as audit artifacts, not treated as valid data) and re-run
clean: all 9 (arch, seed) cells in fully randomized order, fresh clean
rebuild before every single cell, one continuous sitting. Result:
4,320/4,320 rows captured, zero VM errors anywhere.

This validates the campaign mechanism runs cleanly and reproducibly under
the post-4.6 Stadium substrate -- satisfies item 5.1's own concern that a
green POST suite isn't evidence determinism holds post-migration. It does
NOT produce an ACL-RWT overhead number: ACL.4th is not self-activated in
this repo's default init.4th, so these 9 cells ran with ACL inactive.
Reproducing the original +0.0054%-+0.0088% measurement needs a paired
ACL-enabled/disabled run using this now-validated mechanism -- scoped,
not attempted here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

FABRIC-2.md

@@ -1890,3 +1890,106 @@ two obstacles/workarounds, replacing their earlier "not yet visually confirmed"
 
 This closes the last open item from SDK v1.9.0 scoping's own list. Only the version-string bump
 remains, and it's a release decision, not implementation work.
+
+## M. ACL-RWT-style 3x3 Latin square re-run on the Stadium substrate — 2026-08-19/20
+
+Captain Bob asked to start the deferred F.3 item ("5.1 DoE re-run + ACL-RWT re-measurement")
+now that item 4.6 (Section H) closed. What follows covers the mechanism work and the campaign
+itself; the ACL overhead *number* is explicitly NOT produced here (see caveat at the end).
+
+**A real, previously-unknown bug found before any campaign work could start: `EXEC-DOE`'s own
+CSV output is almost entirely lost to console interleaving with the routine per-tick
+`[HADES][DOE]` heartbeat export, the same bug class as Section L's framebuffer `PLOT` case.**
+First pilot cell (amd64, seed 12345, `12345 30 EXEC-DOE`) ran cleanly to `DOE: complete` with
+zero VM errors, but only 1 of the expected 480 `EMIT-ROW` lines survived in the log — the very
+last one (`run_id=479`). Root cause identical in shape to Section L: the heartbeat CSV logger
+(`doe_log.c`) and `EXEC-DOE`'s own `EMIT-ROW` prints share the same serial console, and the
+heartbeat's continuous per-tick output clobbers `EXEC-DOE`'s rows almost every time. Fix:
+`HB-OFF` immediately before `EXEC-DOE`, `HB-ON` once `DOE: complete` appears — confirmed this
+recovers all 480/480 rows cleanly (verified repeatedly, see below). `HB-ON` before `EXEC-DOE`
+(the reverse order) does **not** fix it — tested directly, heartbeat rows kept streaming
+throughout and the same near-total row loss recurred, confirming `HB-OFF` (not `HB-ON`) is the
+side that actually silences the per-tick export.
+
+**A second, separate finding: `L8-DOE`/`WL-HI`/`WL-LO` (the mechanism `experiments/bare_metal/
+README.md` describes as "the DoE," auto-run from `init.4th`) do not exist anywhere in
+`capsules/` — confirmed by grep, zero matches, and `init.4th` calls no DoE mechanism at all.**
+This matches Section K's own earlier finding but is re-confirmed here since it directly
+determined what "re-run the DoE" had to mean in practice: the only live, invokable mechanism is
+`doe.4th`'s `EXEC-DOE ( seed n-reps -- )`, loaded via `S" doe.4th" EXEC` (not auto-run either).
+README's staleness on this point remains flagged, not fixed (same as Section K).
+
+**Third finding: `DOE_SEED` (`Makefile.starkernel:233`, default `12345`) is declared but never
+referenced anywhere else in the Makefile — vestigial, same class of staleness as `L8-DOE`.**
+No build-time seed injection exists; `EXEC-DOE`'s seed must be typed at the live `ok>` prompt.
+
+**Tooling built to drive this: QEMU serial socket injection via `socat`, since these campaigns
+require typing FORTH commands into a running kernel after boot, not just observing serial
+output.** `Makefile.starkernel`'s `qemu` target already exposes a Unix-domain socket per
+architecture (`-chardev socket,...,path=$SOCK`); `printf '<command>\r\n' | socat -
+UNIX-CONNECT:$SOCK` reliably injects a line into the live REPL. Two real defects found and
+fixed in this tooling itself, both session-scoped (scratchpad scripts, not committed):
+
+1. **Log-discovery race.** An early injector script auto-discovered "the new log file" by
+   snapshotting existing logs (`OLD_LOGS`) then polling for one not in that set. If the
+   snapshot happened *after* the target boot's own `touch $LOG` (which happens early in the
+   `qemu` rule, before qemu itself launches) — a real possibility any time two separate tool
+   calls are used to start the boot and then the injector — the target log gets permanently
+   self-excluded and the script spins forever. Root cause of two separate stuck-injector
+   incidents (one for ~15 hours, overnight; one caught within minutes on retry). **Fix:**
+   stopped having the injector discover its own target; the orchestrating side determines the
+   exact log/socket path once (via a short, explicitly-bounded wait) and passes it in directly
+   — no discovery, no race.
+2. **Unredirected background launch.** One injector was launched via a bare `&`/`disown`
+   instead of the harness's own tracked-background mechanism. Its first `socat` call began
+   relaying the live (heartbeat-heavy, pre-`HB-OFF`) serial stream back through its own stdout;
+   with nothing reading that pipe, the buffer filled and `socat` blocked permanently inside a
+   `write()` syscall, deadlocking the whole script before it ever delivered a single command.
+   Every properly-tracked background launch (which redirects output to a drained file)
+   avoided this. **Fix:** never launch these injectors outside the tracked-background
+   mechanism.
+
+**A related process-discipline lesson, not a code defect: the first attempt at the full
+campaign ran all 9 cells as three strict ISA-blocked loops (amd64 x3, then aarch64 x3, then
+riscv64 x3), reusing one build per architecture rather than rebuilding per cell.** Caught by
+Captain Bob mid-run: this confounds ISA with time/session-order — exactly what a Latin square
+is supposed to control against — and reusing one build across seeds doesn't match "every
+ISA-seed pairing occupies a unique session" (the original campaign's own stated design
+principle, per `bare_metal_doe_report.tex`). The 3 amd64 cells completed under the flawed
+protocol were discarded (not deleted — logs remain as audit artifacts, just not treated as
+part of the valid campaign).
+
+**Final campaign, run clean in one continuous sitting 2026-08-20, randomized order, fresh
+`clean` + full rebuild before every single cell:**
+
+| Order | Arch    | Seed  | Rows    | Errors |
+|-------|---------|-------|---------|--------|
+| 1     | riscv64 | 13579 | 480/480 | 0      |
+| 2 | aarch64 | 12345 | 480/480 | 0 |
+| 3 | amd64 | 13579 | 480/480 | 0 |
+| 4 | amd64 | 12345 | 480/480 | 0 |
+| 5 | riscv64 | 67890 | 480/480 | 0 |
+| 6 | amd64 | 67890 | 480/480 | 0 |
+| 7 | aarch64 | 67890 | 480/480 | 0 |
+| 8 | aarch64 | 13579 | 480/480 | 0 |
+| 9 | riscv64 | 12345 | 480/480 | 0 |
+
+**4,320/4,320 total rows captured across all 9 cells, zero VM errors, zero `UNKNOWN WORD`
+faults anywhere in any cell.** Order was randomized up front (Python `random.shuffle` over the
+9 (arch, seed) pairs), not the sequential/nested order shown by architecture above. Raw serial
+logs (audit artifacts, committed): `logs/20260820-0{82610,83018,83710,84147,84627,85030,85509,
+90205,90903}/`. Extracted per-cell CSVs (note: `scripts/extract_doe.sh` targets a *different*,
+unrelated CSV format — the routine `[HADES][DOE]` heartbeat export, not `EXEC-DOE`'s own
+output — so a separate extraction was written for this data):
+`experiments/bare_metal/runs/acl-rwt-20260820/`.
+
+**What this closes, and what it explicitly does NOT close.** This satisfies item 5.1's own
+concern — "a green POST suite is not evidence that determinism holds under the Stadium
+migration, needs its own campaign" — the campaign mechanism itself runs cleanly, reproducibly,
+and completely under the post-4.6 Stadium substrate, across all three architectures, under
+proper randomized-order/fresh-build discipline. **It does NOT produce an ACL-RWT overhead
+number.** `capsules/ACL.4th` is not self-activated in this repo's default `init.4th`
+(`\ S" ACL.4th" EXEC`, commented out) — these 9 cells ran with ACL inactive, same as every
+other boot in this document. Reproducing the original `+0.0054%–+0.0088%` measurement would
+need a paired run (ACL enabled vs. disabled) using this now-validated mechanism and tooling —
+scoped but not attempted here.