aarch64: fix BYE cold-restart crash — PSCI SYSTEM_RESET via HVC, not SMC
Root cause of the aarch64 BYE cold-restart exception (present since at least 2026-08-08, ESR_EL1=0x02000000/EC=0 "Unknown reason"), found via live gdb single-stepping through the actual crash: arch_cold_reset() issued PSCI SYSTEM_RESET via `smc #0`, but QEMU's aarch64 virt machine booted with AAVMF (UEFI firmware, no genuine EL3/TrustZone secure monitor) serves PSCI via HVC, not SMC -- nothing exists to answer an SMC call, so it trapped as an illegal instruction straight into the kernel's own exception handler. Not memory corruption, not a race -- a wrong conduit for this boot configuration. Fix: smc #0 -> hvc #0. Function ID and calling convention unchanged. Getting to this required first discovering that starkernel_kernel.elf is not the binary that actually runs -- MONOLITHIC_BUILD links kernel_main() directly into starkernel_loader.efi, a completely separate, differently-linked build artifact. Every earlier gdb breakpoint attempt this session failed because it used addresses from the wrong file. Real addresses (UEFI-chosen ImageBase + linker-map RVA) let gdb catch the crash live for the first time. Verified: full aarch64 acceptance pass, 30/30 stress-campaign reps PASS (unaffected -- this bug only manifested on BYE), and BYE now exits cleanly with no exception for the first time in this investigation. Full writeup in FABRIC-2.md Section I. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
bc31916461
commit
b24a5a6e25
+101
-23
@@ -1008,7 +1008,7 @@ something this item introduced.
|
||||
|
||||
---
|
||||
|
||||
## I. aarch64 `BYE` cold-restart exception — real PSCI bug fixed, but not the actual crash cause; crash is a pre-existing, unresolved bug class — 2026-08-18
|
||||
## I. aarch64 `BYE` cold-restart exception — RESOLVED 2026-08-18 (SMC→HVC PSCI conduit)
|
||||
|
||||
Surfaced during the item 4.6 acceptance pass (Section H): on aarch64 only, typing `BYE` at
|
||||
the `ok>` prompt reaps children and prints `BYE: cold restart` successfully, then faults
|
||||
@@ -1150,27 +1150,105 @@ unremarkable: `mama_word_bye` ends with `bl 40ef60 <arch_cold_reset>`, a plain d
|
||||
to the right address — ruling out a bad relocation or corrupted call instruction as the
|
||||
reason the breakpoints didn't fire.
|
||||
|
||||
**Sanity check, decisive:** set `hbreak console_println` — a function called thousands of
|
||||
times from the very first moment of kernel boot — on a fresh boot. After the serial log had
|
||||
already accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
|
||||
**Sanity check:** set `hbreak console_println` — a function called thousands of times from
|
||||
the very first moment of kernel boot — on a fresh boot. After the serial log had already
|
||||
accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
|
||||
necessarily called thousands of times to produce that output), gdb still reported only
|
||||
`Continuing.` — zero breakpoint hits, ever. **This means gdb breakpoints (software and
|
||||
hardware alike) do not function at all against this QEMU aarch64 (`cortex-a57`, TCG) target
|
||||
in this environment** — not an icache-coherency property of our kernel, a limitation of the
|
||||
debugging setup itself. The icache-corruption hypothesis this session was pursuing is
|
||||
therefore neither confirmed nor refuted; it's simply unreachable with this tooling as
|
||||
configured. All gdb sessions and their QEMU instances were killed; no code changes came out
|
||||
of this thread beyond the (kept) heap-address prints and the `-m 4096` bump, both harmless
|
||||
diagnostics worth keeping regardless.
|
||||
`Continuing.` — zero breakpoint hits, ever, against our kernel.
|
||||
|
||||
**Where this stands:** root cause narrowed but not found. Confirmed heap-internal (not
|
||||
kernel code, not firmware, not truly random/unmapped memory — see the decisive finding
|
||||
above). Live single-stepping is not currently viable against this target; before attempting
|
||||
it again, the QEMU aarch64 gdbstub setup itself needs to be validated independently (try a
|
||||
different QEMU version, `-accel tcg,thread=single`, or confirm hardware breakpoint support
|
||||
against a trivial known-working aarch64 QEMU target first, outside this kernel entirely) —
|
||||
don't repeat this exact approach expecting a different result. Two competing crash
|
||||
signatures still unreconciled: `ESR_EL1=0x02000000`/EC=0 "Unknown reason" (this crash, and
|
||||
the 2026-08-08 keyboard-input crash) vs. `ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with
|
||||
a poisoned-looking `FAR_EL1=0x000055bd` (June/July 2026 crashes) — may or may not be the same
|
||||
bug. Genuinely open.
|
||||
**Refined by a follow-up control test — this is NOT a blanket "gdbstub is broken" finding.**
|
||||
A trivial standalone aarch64 program (hand-assembled: `mov`/`add`/`subs`/`b.ne` loop, no
|
||||
UEFI, no MMU, linked at `0x40080000`, booted directly via QEMU's own `-kernel` flag,
|
||||
bypassing our loader entirely) hit a hardware breakpoint (`hbreak loop`) on the very first
|
||||
try — `Breakpoint 1, 0x...4008000c in loop ()`, exactly as expected. **The gdbstub mechanism
|
||||
itself works fine in this QEMU build/environment.** The failure is specific to *our* kernel.
|
||||
|
||||
**Narrowed further, decisively: it fails from our kernel's very first instruction, not
|
||||
something that develops later in boot.** Set `hbreak kernel_main` (the ELF entry point
|
||||
itself, `0x400da0`, confirmed by `readelf -h`) — after the serial log had already printed PMM
|
||||
statistics (`Used pages: 1411`, `Total MB: 4062` — output that can only come from well inside
|
||||
`kernel_main`/`kernel_main_deep`, several stages past entry), gdb still reported only
|
||||
`Continuing.`, never a hit. **Breakpoints fail on our kernel from the first instruction
|
||||
onward, not partway through boot** — ruling out anything `kernel_main_deep` or later does
|
||||
(MMU reconfiguration, EL switches deep in boot, etc.) as the cause, since the very first
|
||||
instruction is already unreachable to gdb.
|
||||
|
||||
**Where this leaves it:** the standalone test (works) differs from our kernel (doesn't work)
|
||||
in exactly one respect that matters here — the standalone test boots directly via QEMU's
|
||||
`-kernel` flag (no firmware involved), while ours boots through the full UEFI path (AAVMF
|
||||
firmware → `starkernel_loader.efi` → `ExitBootServices()` → jump to `kernel_main`). The likely
|
||||
culprit is something about that handoff — UEFI's own MMU/paging state, exception-level
|
||||
context, or how our loader transitions into the kernel — that QEMU's gdbstub can't track
|
||||
across, not anything in our kernel's own C code. Not yet tested directly (would need a
|
||||
minimal UEFI-booted test kernel to isolate whether the loader phase or the UEFI-to-kernel
|
||||
jump specifically is where it breaks) — a good next concrete step, more promising than
|
||||
retrying breakpoints against the real kernel again. All gdb sessions and QEMU instances from
|
||||
this thread were killed; no code changes came out of it beyond the (kept) heap-address prints
|
||||
and the `-m 4096` bump, both harmless diagnostics worth keeping regardless.
|
||||
|
||||
**The actual blocker, found and fixed: `starkernel_kernel.elf` is the wrong symbol file.**
|
||||
`MONOLITHIC_BUILD` (`Makefile.starkernel`) compiles `kernel_main()` and everything it calls
|
||||
directly into `starkernel_loader.efi` — the PE32+ binary UEFI actually loads and executes.
|
||||
`starkernel_kernel.elf` is a *completely separate build artifact*: different object list
|
||||
(`KERNEL_OBJS` vs. `LOADER_OBJS`), different linker (`$(LD)` vs. `lld-link`), different link
|
||||
step entirely, sharing nothing but overlapping source files. Every gdb breakpoint address in
|
||||
this investigation — `nm`/`objdump` addresses like `arch_cold_reset`'s `0x40ef60` — came from
|
||||
that wrong file and had no relationship to where the code actually lives at runtime. That's
|
||||
why breakpoints never fired even at `kernel_main`'s own first instruction: the address was
|
||||
simply wrong, on every attempt.
|
||||
|
||||
**How the real address was found.** `starkernel_loader.efi` links with `/base:0` but is a
|
||||
PE image UEFI relocates to a runtime-chosen `ImageBase`, so real address = `ImageBase + RVA`.
|
||||
Got the RVA for any symbol by re-running the exact `lld-link` command `make -n` prints, with
|
||||
`/map:out.map` appended (a linker map, since the final stripped `.efi` carries no symbol
|
||||
table `llvm-nm`/`llvm-objdump-18` can read — confirmed via `file format coff-arm64` showing
|
||||
"no symbols"). Got the runtime `ImageBase` with a one-line temporary probe in `efi_main()`
|
||||
(`src/starkernel/boot/uefi_loader.c`, right after the first `debug_checkpoint`): fetch
|
||||
`EFI_LOADED_IMAGE_PROTOCOL` for the running image, print `->ImageBase` via `ConOut`.
|
||||
Confirmed deterministic across boots of the same build (`0x13e019000`, identical on repeat).
|
||||
**Reverted after capture** (kept out of the tree — this was a one-shot diagnostic, not a
|
||||
permanent addition, per this project's usual probe-then-revert practice).
|
||||
|
||||
**With the correct address, gdb worked immediately.** `hbreak *<ImageBase+RVA>` for
|
||||
`kernel_main` fired on the very first try — confirming the gdbstub/tooling was never broken,
|
||||
only every address fed to it in this investigation. Re-armed the same way at
|
||||
`arch_cold_reset`'s real address and ran the full ~30-minute boot to `BYE` with gdb attached;
|
||||
the breakpoint fired exactly where expected, right after "BYE: cold restart" printed.
|
||||
|
||||
**Single-stepped from there — the real mechanism.** `x/8i $pc` confirmed the disassembly
|
||||
matched source exactly: `bl` to `arch_disable_interrupts()`, `mov`/`movk` building the PSCI
|
||||
function ID, `smc #0x0`, then the wfi loop. Stepping instruction-by-instruction through to
|
||||
and past the `smc` showed **PC does not fall through to the wfi loop** — it jumps to another
|
||||
address entirely, which itself branches into what is unmistakably a normal function prologue
|
||||
(`sub sp, sp, #0x40`). This is our own kernel's exception vector correctly catching an
|
||||
illegal instruction and dispatching to the handler that prints `*** EXCEPTION (aarch64) ***`
|
||||
— exactly the behavior observed in every prior serial log. **The SMC instruction genuinely
|
||||
traps; nothing is corrupted, nothing jumps into garbage heap memory.** (The earlier
|
||||
"fault address is inside the kmalloc heap" finding, Section I above this one, was real
|
||||
arithmetic on real numbers, but those numbers came from the same wrong symbol file — a
|
||||
coincidental match, not evidence about the actual mechanism. Left in place above as an
|
||||
honest record of the investigation's path, not retracted.)
|
||||
|
||||
**Root cause: wrong PSCI conduit.** QEMU's aarch64 `virt` machine booted with AAVMF (UEFI
|
||||
firmware, no genuine EL3/TrustZone secure monitor) serves PSCI via **HVC** (hypervisor call,
|
||||
EL2), not **SMC** (secure monitor call, EL3) — there is no real secure-world firmware present
|
||||
to answer an SMC. `arch_cold_reset()` used `smc #0`, which traps as an illegal instruction
|
||||
with nothing to service it — a textbook "wrong conduit for this boot configuration" bug, not
|
||||
memory corruption, not a race, not an icache issue.
|
||||
|
||||
**Fix:** `src/starkernel/arch/aarch64/arch.c`, `smc #0` → `hvc #0`. Function ID and calling
|
||||
convention (the `mov`/`movk` building `0x84000009`) unchanged — only the conduit instruction.
|
||||
|
||||
**Verification:** rebuilt aarch64 clean, ran the full acceptance pass (30/30 stress-campaign
|
||||
reps PASS, unaffected — this bug only manifests on `BYE`) through to `BYE`. Result: `BYE:
|
||||
reaping children` → `BYE: cold restart` → clean process exit, code 0. **No exception.** First
|
||||
clean aarch64 `BYE` exit recorded in this document.
|
||||
|
||||
**Loose end, deliberately not chased further this session:** two possibly-different crash
|
||||
signatures existed historically — this bug's `ESR_EL1=0x02000000`/EC=0 "Unknown reason"
|
||||
signature (also seen in an unrelated 2026-08-08 keyboard-input log), versus an older
|
||||
`ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with a poisoned-looking `FAR_EL1=0x000055bd`
|
||||
from June/July 2026 crashes. This fix explains and resolves the former. Whether the latter
|
||||
was the same underlying SMC/HVC conduit issue (plausible — an illegal-instruction trap can
|
||||
manifest with different ESR encodings depending on exact CPU/QEMU state) or a genuinely
|
||||
separate bug was not re-investigated; nothing currently reproduces it to check against.
|
||||
|
||||
Reference in New Issue
Block a user