aarch64: fix BYE cold-restart crash — PSCI SYSTEM_RESET via HVC, not SMC

Root cause of the aarch64 BYE cold-restart exception (present since at
least 2026-08-08, ESR_EL1=0x02000000/EC=0 "Unknown reason"), found via
live gdb single-stepping through the actual crash: arch_cold_reset()
issued PSCI SYSTEM_RESET via `smc #0`, but QEMU's aarch64 virt machine
booted with AAVMF (UEFI firmware, no genuine EL3/TrustZone secure
monitor) serves PSCI via HVC, not SMC -- nothing exists to answer an
SMC call, so it trapped as an illegal instruction straight into the
kernel's own exception handler. Not memory corruption, not a race --
a wrong conduit for this boot configuration.

Fix: smc #0 -> hvc #0. Function ID and calling convention unchanged.

Getting to this required first discovering that starkernel_kernel.elf
is not the binary that actually runs -- MONOLITHIC_BUILD links
kernel_main() directly into starkernel_loader.efi, a completely
separate, differently-linked build artifact. Every earlier gdb
breakpoint attempt this session failed because it used addresses from
the wrong file. Real addresses (UEFI-chosen ImageBase + linker-map
RVA) let gdb catch the crash live for the first time.

Verified: full aarch64 acceptance pass, 30/30 stress-campaign reps
PASS (unaffected -- this bug only manifested on BYE), and BYE now
exits cleanly with no exception for the first time in this
investigation.

Full writeup in FABRIC-2.md Section I.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-08-18 18:24:26 -04:00
co-authored by Claude Sonnet 5
parent bc31916461
commit b24a5a6e25
12 changed files with 368452 additions and 78539 deletions
+101 -23
View File
@@ -1008,7 +1008,7 @@ something this item introduced.
---
## I. aarch64 `BYE` cold-restart exception — real PSCI bug fixed, but not the actual crash cause; crash is a pre-existing, unresolved bug class — 2026-08-18
## I. aarch64 `BYE` cold-restart exception — RESOLVED 2026-08-18 (SMC→HVC PSCI conduit)
Surfaced during the item 4.6 acceptance pass (Section H): on aarch64 only, typing `BYE` at
the `ok>` prompt reaps children and prints `BYE: cold restart` successfully, then faults
@@ -1150,27 +1150,105 @@ unremarkable: `mama_word_bye` ends with `bl 40ef60 <arch_cold_reset>`, a plain d
to the right address — ruling out a bad relocation or corrupted call instruction as the
reason the breakpoints didn't fire.
**Sanity check, decisive:** set `hbreak console_println` — a function called thousands of
times from the very first moment of kernel boot — on a fresh boot. After the serial log had
already accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
**Sanity check:** set `hbreak console_println` — a function called thousands of times from
the very first moment of kernel boot — on a fresh boot. After the serial log had already
accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
necessarily called thousands of times to produce that output), gdb still reported only
`Continuing.` — zero breakpoint hits, ever. **This means gdb breakpoints (software and
hardware alike) do not function at all against this QEMU aarch64 (`cortex-a57`, TCG) target
in this environment** — not an icache-coherency property of our kernel, a limitation of the
debugging setup itself. The icache-corruption hypothesis this session was pursuing is
therefore neither confirmed nor refuted; it's simply unreachable with this tooling as
configured. All gdb sessions and their QEMU instances were killed; no code changes came out
of this thread beyond the (kept) heap-address prints and the `-m 4096` bump, both harmless
diagnostics worth keeping regardless.
`Continuing.` — zero breakpoint hits, ever, against our kernel.
**Where this stands:** root cause narrowed but not found. Confirmed heap-internal (not
kernel code, not firmware, not truly random/unmapped memory — see the decisive finding
above). Live single-stepping is not currently viable against this target; before attempting
it again, the QEMU aarch64 gdbstub setup itself needs to be validated independently (try a
different QEMU version, `-accel tcg,thread=single`, or confirm hardware breakpoint support
against a trivial known-working aarch64 QEMU target first, outside this kernel entirely) —
don't repeat this exact approach expecting a different result. Two competing crash
signatures still unreconciled: `ESR_EL1=0x02000000`/EC=0 "Unknown reason" (this crash, and
the 2026-08-08 keyboard-input crash) vs. `ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with
a poisoned-looking `FAR_EL1=0x000055bd` (June/July 2026 crashes) — may or may not be the same
bug. Genuinely open.
**Refined by a follow-up control test — this is NOT a blanket "gdbstub is broken" finding.**
A trivial standalone aarch64 program (hand-assembled: `mov`/`add`/`subs`/`b.ne` loop, no
UEFI, no MMU, linked at `0x40080000`, booted directly via QEMU's own `-kernel` flag,
bypassing our loader entirely) hit a hardware breakpoint (`hbreak loop`) on the very first
try — `Breakpoint 1, 0x...4008000c in loop ()`, exactly as expected. **The gdbstub mechanism
itself works fine in this QEMU build/environment.** The failure is specific to *our* kernel.
**Narrowed further, decisively: it fails from our kernel's very first instruction, not
something that develops later in boot.** Set `hbreak kernel_main` (the ELF entry point
itself, `0x400da0`, confirmed by `readelf -h`) — after the serial log had already printed PMM
statistics (`Used pages: 1411`, `Total MB: 4062` — output that can only come from well inside
`kernel_main`/`kernel_main_deep`, several stages past entry), gdb still reported only
`Continuing.`, never a hit. **Breakpoints fail on our kernel from the first instruction
onward, not partway through boot** — ruling out anything `kernel_main_deep` or later does
(MMU reconfiguration, EL switches deep in boot, etc.) as the cause, since the very first
instruction is already unreachable to gdb.
**Where this leaves it:** the standalone test (works) differs from our kernel (doesn't work)
in exactly one respect that matters here — the standalone test boots directly via QEMU's
`-kernel` flag (no firmware involved), while ours boots through the full UEFI path (AAVMF
firmware → `starkernel_loader.efi``ExitBootServices()` → jump to `kernel_main`). The likely
culprit is something about that handoff — UEFI's own MMU/paging state, exception-level
context, or how our loader transitions into the kernel — that QEMU's gdbstub can't track
across, not anything in our kernel's own C code. Not yet tested directly (would need a
minimal UEFI-booted test kernel to isolate whether the loader phase or the UEFI-to-kernel
jump specifically is where it breaks) — a good next concrete step, more promising than
retrying breakpoints against the real kernel again. All gdb sessions and QEMU instances from
this thread were killed; no code changes came out of it beyond the (kept) heap-address prints
and the `-m 4096` bump, both harmless diagnostics worth keeping regardless.
**The actual blocker, found and fixed: `starkernel_kernel.elf` is the wrong symbol file.**
`MONOLITHIC_BUILD` (`Makefile.starkernel`) compiles `kernel_main()` and everything it calls
directly into `starkernel_loader.efi` — the PE32+ binary UEFI actually loads and executes.
`starkernel_kernel.elf` is a *completely separate build artifact*: different object list
(`KERNEL_OBJS` vs. `LOADER_OBJS`), different linker (`$(LD)` vs. `lld-link`), different link
step entirely, sharing nothing but overlapping source files. Every gdb breakpoint address in
this investigation — `nm`/`objdump` addresses like `arch_cold_reset`'s `0x40ef60` — came from
that wrong file and had no relationship to where the code actually lives at runtime. That's
why breakpoints never fired even at `kernel_main`'s own first instruction: the address was
simply wrong, on every attempt.
**How the real address was found.** `starkernel_loader.efi` links with `/base:0` but is a
PE image UEFI relocates to a runtime-chosen `ImageBase`, so real address = `ImageBase + RVA`.
Got the RVA for any symbol by re-running the exact `lld-link` command `make -n` prints, with
`/map:out.map` appended (a linker map, since the final stripped `.efi` carries no symbol
table `llvm-nm`/`llvm-objdump-18` can read — confirmed via `file format coff-arm64` showing
"no symbols"). Got the runtime `ImageBase` with a one-line temporary probe in `efi_main()`
(`src/starkernel/boot/uefi_loader.c`, right after the first `debug_checkpoint`): fetch
`EFI_LOADED_IMAGE_PROTOCOL` for the running image, print `->ImageBase` via `ConOut`.
Confirmed deterministic across boots of the same build (`0x13e019000`, identical on repeat).
**Reverted after capture** (kept out of the tree — this was a one-shot diagnostic, not a
permanent addition, per this project's usual probe-then-revert practice).
**With the correct address, gdb worked immediately.** `hbreak *<ImageBase+RVA>` for
`kernel_main` fired on the very first try — confirming the gdbstub/tooling was never broken,
only every address fed to it in this investigation. Re-armed the same way at
`arch_cold_reset`'s real address and ran the full ~30-minute boot to `BYE` with gdb attached;
the breakpoint fired exactly where expected, right after "BYE: cold restart" printed.
**Single-stepped from there — the real mechanism.** `x/8i $pc` confirmed the disassembly
matched source exactly: `bl` to `arch_disable_interrupts()`, `mov`/`movk` building the PSCI
function ID, `smc #0x0`, then the wfi loop. Stepping instruction-by-instruction through to
and past the `smc` showed **PC does not fall through to the wfi loop** — it jumps to another
address entirely, which itself branches into what is unmistakably a normal function prologue
(`sub sp, sp, #0x40`). This is our own kernel's exception vector correctly catching an
illegal instruction and dispatching to the handler that prints `*** EXCEPTION (aarch64) ***`
— exactly the behavior observed in every prior serial log. **The SMC instruction genuinely
traps; nothing is corrupted, nothing jumps into garbage heap memory.** (The earlier
"fault address is inside the kmalloc heap" finding, Section I above this one, was real
arithmetic on real numbers, but those numbers came from the same wrong symbol file — a
coincidental match, not evidence about the actual mechanism. Left in place above as an
honest record of the investigation's path, not retracted.)
**Root cause: wrong PSCI conduit.** QEMU's aarch64 `virt` machine booted with AAVMF (UEFI
firmware, no genuine EL3/TrustZone secure monitor) serves PSCI via **HVC** (hypervisor call,
EL2), not **SMC** (secure monitor call, EL3) — there is no real secure-world firmware present
to answer an SMC. `arch_cold_reset()` used `smc #0`, which traps as an illegal instruction
with nothing to service it — a textbook "wrong conduit for this boot configuration" bug, not
memory corruption, not a race, not an icache issue.
**Fix:** `src/starkernel/arch/aarch64/arch.c`, `smc #0``hvc #0`. Function ID and calling
convention (the `mov`/`movk` building `0x84000009`) unchanged — only the conduit instruction.
**Verification:** rebuilt aarch64 clean, ran the full acceptance pass (30/30 stress-campaign
reps PASS, unaffected — this bug only manifests on `BYE`) through to `BYE`. Result: `BYE:
reaping children``BYE: cold restart` → clean process exit, code 0. **No exception.** First
clean aarch64 `BYE` exit recorded in this document.
**Loose end, deliberately not chased further this session:** two possibly-different crash
signatures existed historically — this bug's `ESR_EL1=0x02000000`/EC=0 "Unknown reason"
signature (also seen in an unrelated 2026-08-08 keyboard-input log), versus an older
`ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with a poisoned-looking `FAR_EL1=0x000055bd`
from June/July 2026 crashes. This fix explains and resolves the former. Whether the latter
was the same underlying SMC/HVC conduit issue (plausible — an illegal-instruction trap can
manifest with different ESR encodings depending on exact CPU/QEMU state) or a genuinely
separate bug was not re-investigated; nothing currently reproduces it to check against.