FABRIC.md: item 4.5d -- Finding 3 (muldiv64) root-caused and fixed via QEMU tracing, Finding 4 blocks

Used QEMU-side instrumentation (-d int tracing, plus a working chardev-based
monitor -- the older bareword -monitor syntax silently fails on QEMU
10.2.1) as directed. The PM Timer "stall" was never a hang: it was a #DE
divide error cascading to a triple fault, which -no-reboot converts into a
silent clean QEMU exit -- indistinguishable from a hang without tracing,
and why arch_relax() (solving a hang that didn't exist) had no effect.
Root cause and fix documented in full (also see commit b43e51a).

With the fix, amd64 boot at -O2 now proceeds far past the original stall --
through capsule birth and into Hermes's word registration -- before hitting
a second, different fault (Finding 4): a direct #GP at IDT index 32
(APIC_TIMER_VECTOR), not yet root-caused. Same silent-triple-fault-exit
shape, so this was very likely bundled into "the hang" before tracing
distinguished the two separate bugs.

-O2 reverted again (uncommitted); muldiv64's fix is kept, real and
independently verified at unchanged -O0 on all three architectures.
Routine three-arch artifacts from this session's verification runs
included.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-08-11 12:17:23 -04:00
co-authored by Claude Sonnet 5
parent b43e51a95d
commit 1c5e926c3f
18 changed files with 104344 additions and 77 deletions
+59 -1
View File
@@ -5634,7 +5634,65 @@ document and committing that amendment as its own item.*
> a bigger step than another guess-and-check pass. Notable stakes: per CLAUDE.md, QEMU/TCG
> is not *a* target for this project, it's *the* acceptance target — this bug blocks the
> actual thing that matters, not an edge case.
> 4.5d cannot complete until this is resolved.
>
> **Root-caused and fixed, 2026-08-11 — it was never a hang.** Used QEMU-side
> instrumentation as Captain Bob directed: `-d int` execution-exception tracing plus a
> chardev-based monitor socket (the older bareword `-monitor unix:...,server,nowait` syntax
> silently failed to create a socket on QEMU 10.2.1; the modern
> `-chardev socket,...` + `-mon chardev=...` form works). The trace showed the CPU never
> looping at all: a genuine **`#DE` (divide error) at the second `muldiv64()` call site**,
> cascading through a double fault into a **triple fault** — which `-no-reboot` converts
> into a silent, clean QEMU exit (`exit code 0`, empty stdout), indistinguishable from an
> infinite hang from the *guest* side without tracing. This is also why the earlier
> `arch_relax()` attempt did nothing: it was solving a hang that didn't exist.
>
> Register state at the fault: `RAX=0xe8d4a51000` — exactly `1000 × 1,000,000,000`,
> confirming the values feeding `muldiv64(elapsed_ticks, 1000000000ull, PMTIMER_FREQ_HZ)`
> (`timer.c:597`, `elapsed_ticks` at loop exit == `target_ticks` == 1000). That product fits
> entirely in the low 64 bits, so `mulq`'s high-word output (`RDX`) is 0. `muldiv64()`'s
> inline asm declared `RDX` as a plain output (`"=d"(hi)`) — telling GCC only "I want to
> read RDX's value after this block," with nothing indicating that `mulq` writes RDX
> *before* `divq` needs a *different* value (the divisor `c`) out of it. Nothing stopped the
> register allocator from placing `c` itself in `RDX`, which `mulq` then overwrote with 0
> before `divq` ever read it — dividing by a corrupted 0 instead of the intended
> `PMTIMER_FREQ_HZ` (3,579,545). Worked by accident at `-O0`'s more conservative
> allocation; `-O2` actually hit it.
>
> A `unsigned __int128` rewrite was tried first (cleanest fix in principle) but needs
> libgcc's `__udivti3` for the general 128÷64 case, undefined in this freestanding,
> `-nostdlib` build — not viable, the same class of problem as the `putc`/`getc` finding
> earlier in this item. Fixed instead by declaring `rdx` a pure **clobber** rather than an
> output — the same pattern the Linux kernel's own `mul_u64_u64_div_u64` uses. A clobber
> tells GCC the register is used internally for the whole asm block and must never be
> allocated to any operand, which is exactly the guarantee the previous constraint list was
> missing. Also added a defensive `end_tsc < start_tsc` guard in the caller — this file's
> own comments already flag TSC non-monotonicity as a real risk under TCG hypervisor mode,
> and an underflowed `delta_tsc` would hit the same class of quotient-overflow `#DE`; not
> the bug that was actually found, but a real latent risk given what this function's own
> documentation already says about the environment. Fix committed (`b43e51a`), three-arch
> acceptance boot clean at unchanged `-O0`.
>
> **Verified this specific stall is gone:** with `-O2` re-enabled (uncommitted), amd64 boot
> now proceeds far past this point — through capsule birth, Mama birth, ACL pinning, and
> into Hermes's word registration — before hitting a **second, different, not-yet-fixed**
> fault (below). Given that, `-O2` was **not** left enabled; reverted to `-O0` again.
>
> **Finding 4, NOT fixed, blocking this item: a second, distinct fault during Hermes word
> registration at `-O2`.** Same tracing technique, same deterministic reproduction (stuck at
> the exact same serial line, `Registering FORTH-79 arithmetic words...`, across repeated
> runs). Different signature this time: `check_exception old: 0xffffffff new 0xd` — a
> **`#GP` (General Protection, vector `0xd`) directly**, not a `#DE`, with error code
> `0x102`. Decoded per the x86-64 selector error-code format (bit 0 = external, bit 1 = IDT
> table indicator, bit 2 = TI, bits 315 = selector index): `IDT=1`, index = `0x102 >> 3` =
> **32** — exactly `APIC_TIMER_VECTOR`. This fires during otherwise-unrelated Hermes
> word-registration work, which is consistent with the periodic 100 Hz timer interrupt (that
> fires continuously in the background regardless of what else is running) hitting a
> problem with its own IDT descriptor — not yet traced to a specific cause the way Finding 3
> was; no register-state correlation to a specific call site has been done yet for this one.
> Same cascade shape as before (`#GP``#GP` → double fault → triple fault → silent QEMU
> exit), so this was very likely mis-diagnosed as "part of the same hang" before tracing
> distinguished the two. `-O2` reverted; `muldiv64()`'s fix is kept (real, independently
> verified). 4.5d cannot complete until Finding 4 is resolved.
- [ ] **4.5e — Three-arch acceptance boot with optimization enabled.** Depends on 4.5d. This
is the actual gate, per CLAUDE.md's non-negotiable acceptance criteria — a kernel that has