FABRIC.md: item 4.5d Finding 4 -- deep-traced, not yet root-caused

Computed the loader's true runtime relocation delta to correctly correlate
fault addresses (the running image is starkernel_loader.efi, a relocated
PE, not the separately-linked starkernel_kernel.elf assumed at first).
Fault RIP decodes to log_message()'s entry -- coincidental, not causal,
since it's called on nearly every HADES dispatch during word registration.

Used QEMU's monitor for -d exec,int tracing. Late-start tracing (stop right
before the danger zone to keep trace size down) failed twice -- the window
between a detectable checkpoint and the crash is shorter than host-side
reaction latency. Fell back to full-boot tracing from -S (~2.7GB per
attempt, not committed). That trace shows an unremarkable, normal-looking
repeating three-block loop immediately before the fault, then "Servicing
hardware INT=0x20" (APIC_TIMER_VECTOR) with IDT already showing limit=0 at
that instant.

Ruled out a second illegitimate lidt call (only one call site exists
anywhere, one-time M4 boot setup; searched the trace for any later
execution of that address range and found none). Not yet established: the
actual corrupting write. Documented two remaining explanations (earlier
silent corruption vs. a genuine TCG artifact) and that pinpointing the
exact instruction needs GDB-level single-stepping, a bigger tooling step
than attempted this session.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-08-11 12:40:30 -04:00
co-authored by Claude Sonnet 5
parent 1c5e926c3f
commit 065ab50240
6 changed files with 18345 additions and 38 deletions
+58 -1
View File
@@ -5692,7 +5692,64 @@ document and committing that amendment as its own item.*
> Same cascade shape as before (`#GP``#GP` → double fault → triple fault → silent QEMU
> exit), so this was very likely mis-diagnosed as "part of the same hang" before tracing
> distinguished the two. `-O2` reverted; `muldiv64()`'s fix is kept (real, independently
> verified). 4.5d cannot complete until Finding 4 is resolved.
> verified).
>
> **Follow-up, 2026-08-11 — deeper tracing, not yet root-caused.** Computed the loader's
> true runtime-vs-link-time relocation delta (printed `&arch_interrupts_init` at runtime,
> diffed against its `nm` link-time address: delta `0x3d5b9000`) to correlate fault
> addresses against the actual binary — necessary because the running image is
> `starkernel_loader.efi` (a PE, `MONOLITHIC_BUILD`-embedded, genuinely relocated at UEFI
> load time), not the separately-linked `starkernel_kernel.elf` first assumed; that
> separate `starkernel_loader.elf` debug-symbol target (`Makefile.starkernel:688`) fails to
> link (`R_X86_64_32S` relocation error) and isn't part of `all` — a pre-existing gap, not
> touched.
>
> With the delta in hand, the fault RIP (`0x3d5cc0e0`/`0x3d5cc0e0`-ish across runs) decodes
> to link address `0x130e0` — the very entry of `log_message()` (`vm/host/shim.c`). This is
> almost certainly coincidental, not causal: `log_message()` is called on essentially every
> HADES/ECW dispatch during word registration, so an async 100 Hz timer tick landing exactly
> at its entry is unsurprising and doesn't implicate the function itself.
>
> Used QEMU's monitor to enable `-d exec,int` execution+exception tracing (the earlier
> `-d int`-only session had already established the fault; this pass wanted the
> instruction-by-instruction lead-up). Two attempts to start tracing late (right before the
> danger zone, via the monitor's `stop`/`log`/`cont` sequence, to keep the trace small)
> **both failed** — QEMU reached the crash and exited before the host-side script could
> connect, meaning the window between a detectable serial checkpoint and the actual fault is
> shorter than the host's reaction latency even under `-S` synchronization races. Fell back
> to tracing from cold boot with `-S` + immediate `log exec,int` before any `cont` (reliable,
> but expensive: ~2.7 GB / ~39M lines per attempt; both trace files deleted after use, not
> committed).
>
> That full trace shows the CPU executing a small, tight, **three-block repeating loop**
> (`0x3d5bd120``0x3d5bcad9``0x3d5bcd05`) many times immediately before the fault — a
> completely unremarkable pattern consistent with normal per-word dictionary-registration
> work, not a wild jump in progress. Immediately after: `Stopped execution of TB chain`,
> `Servicing hardware INT=0x20` (`APIC_TIMER_VECTOR`), and then the `#GP` — with `IDT=` already
> showing `limit=0` at that exact instant. Re-examined the error code (`e=0102`) in this
> light: `IDT=1` (bit 1), index `0x102>>3=32` — consistent with *either* "descriptor 32
> specifically is bad" *or* "the table's limit is exceeded for any index," and an
> x86-64 IDT-limit violation reports the attempted vector number in the error code either
> way, so this doesn't distinguish between "one corrupted descriptor" and "the whole table
> became limit=0" — both remain live explanations.
>
> **Ruled out:** a second, illegitimate `lidt` call — confirmed only one call site in the
> entire codebase (`interrupts.c:476`, one-time M4 boot setup), and searched the exec trace
> for any *later* execution of `arch_interrupts_init()`'s address range — found none; the
> only occurrences are the single legitimate boot-time cluster.
> **Not yet established:** the actual corrupting write/instruction. The repeating loop
> immediately preceding the fault looks unremarkable in the trace, which means either (a)
> the corruption happened earlier still and IDTR/the IDT table sat silently wrong for a
> while before the next timer tick exposed it (most likely, given nothing in the visible
> lead-up looks like a wild jump), or (b) it's a genuine TCG emulation artifact rather than a
> guest-code bug at all (not tested — would need a different accelerator or QEMU version to
> rule in/out, and this project's methodology is TCG-only by design).
>
> **Where this leaves it:** pinpointing the exact corrupting instruction from here needs
> either GDB-level single-stepping (QEMU's `-s -S` gdb stub + a matching cross-gdb, watching
> the `idt[]` array and IDTR directly across the whole pre-fault window) or a fundamentally
> different narrowing strategy — a bigger tooling step than the tracing done so far, and one
> this session did not attempt. 4.5d cannot complete until Finding 4 is resolved.
- [ ] **4.5e — Three-arch acceptance boot with optimization enabled.** Depends on 4.5d. This
is the actual gate, per CLAUDE.md's non-negotiable acceptance criteria — a kernel that has