aarch64 BYE crash: add heap-address diagnostics, rule out gdb debugging on this target

Continued investigating the aarch64 BYE cold-restart exception (FABRIC-2.md
Section I). Added permanent boot diagnostics: kmalloc_heap_base_addr()/
kmalloc_heap_end_addr() now print in print_heap_stats(), confirming the
fault address is provably inside the kmalloc heap (not kernel code, not
firmware). Bumped aarch64 QEMU RAM to 4096MB to test heap-placement
sensitivity (no effect -- heap size is a fixed 2GiB default, independent
of total RAM once "enough" exists).

Three separate live gdb debugging attempts (software breakpoint, hardware
breakpoint on arch_cold_reset, hardware breakpoint on mama_word_bye's
entry) all silently failed to fire despite disassembly-confirmed-correct
addresses and confirmed execution reaching those points. A sanity check
(hbreak on console_println, called thousands of times per boot) also never
fired even 8802 lines into a serial log -- conclusively a gdbstub/QEMU
tooling limitation for this aarch64 target, not a kernel-side finding.
Live single-stepping is not currently viable here; documented so it isn't
re-attempted the same way.

Root cause still open. Full trail in FABRIC-2.md Section I.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-08-18 17:06:58 -04:00
co-authored by Claude Sonnet 5
parent 8d90538801
commit bc31916461
11 changed files with 308066 additions and 32 deletions
+77 -30
View File
@@ -1096,34 +1096,81 @@ now-confirmed-ineffective interrupt mask was removed from `mama_word_bye()`, res
its pre-hypothesis form) rather than carry forward a change that both doesn't work and
correctly bothered Captain Bob on design grounds.
**Correction: the "nowhere near `arch_cold_reset`" argument above does not actually hold.**
That 4-byte shift, tracking a code-size change exactly, is what exposed the flaw: it means
the fault address is coupled to kernel binary layout, which sent us back to check how the
kernel is actually loaded. `src/starkernel/boot/uefi_loader.c` calls UEFI's
`AllocatePages(AllocateAnyPages, ...)` — the kernel is loaded at an address UEFI's own page
allocator chooses at boot, not a fixed base — and `elf_loader.c`'s `elf_apply_relocations()`
then applies real PIE-style relocations against that chosen base. `nm`'s addresses (like
`arch_cold_reset`'s `0x40ef40`) are link-time addresses, assuming the ELF's own default base;
they say nothing about where the code actually lands at runtime, which is wherever
`AllocateAnyPages` happened to place it — plausibly right in the `0xbe0xxxxx` neighborhood
this crash keeps landing in. Comparing the two earlier builds (with/without the PSCI fix)
and seeing the address stay bit-for-bit identical proved nothing either, in hindsight — that
fix only changed an immediate value, not instruction count, so nothing in the image's size
or layout changed between those two builds regardless of where the real fault was. **Net
effect: the original "this must be garbage heap memory, not real code" conclusion is
unproven, not confirmed.** It remains plausible, but so does "this is legitimate relocated
kernel code that a proper address-translation would identify," and nothing done in this pass
distinguishes the two. No load-base address is printed anywhere in the current boot log, so
there was no data available to settle it further.
**False alarm, then corrected back: the kernel does NOT relocate to a UEFI-chosen address.**
The 4-byte shift briefly looked like it disproved "nowhere near `arch_cold_reset`," and led
to a (wrong) detour: `AllocatePages(AllocateAnyPages, ...)` in `uefi_loader.c` was mistaken
for the address the *executable code* loads at. Checked directly against
`src/starkernel/boot/elf_loader.c`'s `elf_load_kernel()`: that `AllocateAnyPages` call only
allocates a scratch buffer to hold the raw ELF *file bytes* before parsing. The actual
segment-load address (`load_base`) is either `0` for `ET_EXEC` or a fixed `0x400000` for
`ET_DYN` — never UEFI-chosen. `readelf -h` on the built kernel confirms `Type: EXEC`, so
`load_base=0`: the kernel really does run at exactly the addresses its own link-time ELF
symbol table (and `nm`) report. The original "`0xbe03e81c` is nowhere near
`arch_cold_reset`'s real code" conclusion was correct after all; the mid-session "correction"
above was itself the mistake, now itself corrected. (The 4-byte shift is still real and still
needs an explanation — see the heap finding below, which supplies one.)
**Where this stands:** root cause not found. Two competing signatures observed across
history (`ESR_EL1=0x02000000`/EC=0 "Unknown reason" here and in the 2026-08-08 keyboard
test; `ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with a poisoned-looking
`FAR_EL1=0x000055bd` in the June/July 2026 crashes) may or may not be the same underlying
bug. The interrupt-race hypothesis is refuted. The "wild jump into garbage RAM" framing is
unproven, undermined by not accounting for `AllocateAnyPages`-based relocation. Stopped here
deliberately, per Captain Bob's direction, rather than continuing to dig live. Next session
should start by printing the actual UEFI-chosen load base and kernel entry point at boot
(nothing currently does), so `ELR_EL1` values can be translated back to real source
locations instead of link-time guesses — that is the missing piece every path above kept
running into. Genuinely open, unlike everything else this document tracks as closed.
**Decisive finding: the fault address is provably inside the kmalloc heap.** Added console
prints of `kmalloc_heap_base_addr()`/`kmalloc_heap_end_addr()` (both already existed as
accessors, just never logged) right after `kmalloc_init()`, and bumped the aarch64 QEMU
`-m` from 2048 to 4096 (`Makefile.starkernel`) to see whether more physical RAM shifts
anything. Rebuilt and booted: `Heap base addr: 1207959552` (`0x48000000`), `Heap end addr:
3355443200` (`0xc8000000`). Both recorded fault addresses, `0xbe03e81c` and `0xbe03e820`,
fall squarely inside that range (`0x48000000 ≤ 0xbe03e81c ≤ 0xc8000000`, about 1.97 GiB into
the 2 GiB heap, ~167 MB short of its end). **This settles the "garbage RAM vs. real code"
question the earlier back-and-forth couldn't: it is heap-internal, not kernel code, not
firmware.** It also explains the 4-byte shift cleanly: the fault isn't landing at a fixed
absolute address, it's the result of whatever computation goes wrong reading a value that
depends on the heap's or kernel's own layout — a computation that necessarily moves by a few
bytes when the kernel binary's size changes by a few bytes. The bump to 4096 MB RAM did not
change heap placement (heap size is a fixed 2 GiB default via `KARGS_DEFAULT_HEAP_SIZE`,
independent of total RAM once "enough" exists), so that specific change didn't add further
data, but confirms heap placement isn't RAM-size-sensitive at this configuration.
**Live gdb debugging attempted three times against the full acceptance-harness device set —
conclusively ruled out as a viable path in this environment, for a reason unrelated to the
kernel itself.** Bumped aarch64 QEMU to `-m 4096` (`Makefile.starkernel`) and added
`kmalloc_heap_base_addr()`/`kmalloc_heap_end_addr()` console prints (`kernel_main.c`,
`print_heap_stats()`) to get the heap-bracketing evidence above. Then, with the same full
device set (`artdisk`, keyboard, `ramfb`) plus `-s -S` and `gdb-multiarch` attached via a
long-lived FIFO-fed session (so gdb could sit waiting through the full ~30-minute real-time
boot without a `-batch` timeout cutting it off):
1. Software breakpoint (`break arch_cold_reset`) at its correctly-identified address
(`0x40ef60`, confirmed against `nm` for that exact build) — did not fire. Crash occurred
normally, gdb reported `[Inferior 1 (process 1) exited normally]`.
2. Hardware breakpoint (`hbreak arch_cold_reset`) at the same address, confirmed "Hardware
assisted breakpoint 1 at 0x40ef60" by gdb — also did not fire. Same outcome.
3. Hardware breakpoint at `mama_word_bye`'s own entry (`0x40bf80`) — unconditionally reached
(confirmed by "BYE: reaping children" printing, the function's first statement) — also
did not fire.
Disassembly (`aarch64-linux-gnu-objdump`) confirmed the call site itself is correct and
unremarkable: `mama_word_bye` ends with `bl 40ef60 <arch_cold_reset>`, a plain direct branch
to the right address — ruling out a bad relocation or corrupted call instruction as the
reason the breakpoints didn't fire.
**Sanity check, decisive:** set `hbreak console_println` — a function called thousands of
times from the very first moment of kernel boot — on a fresh boot. After the serial log had
already accumulated **8,802 lines** (deep into Artemis's stress campaign, `console_println`
necessarily called thousands of times to produce that output), gdb still reported only
`Continuing.` — zero breakpoint hits, ever. **This means gdb breakpoints (software and
hardware alike) do not function at all against this QEMU aarch64 (`cortex-a57`, TCG) target
in this environment** — not an icache-coherency property of our kernel, a limitation of the
debugging setup itself. The icache-corruption hypothesis this session was pursuing is
therefore neither confirmed nor refuted; it's simply unreachable with this tooling as
configured. All gdb sessions and their QEMU instances were killed; no code changes came out
of this thread beyond the (kept) heap-address prints and the `-m 4096` bump, both harmless
diagnostics worth keeping regardless.
**Where this stands:** root cause narrowed but not found. Confirmed heap-internal (not
kernel code, not firmware, not truly random/unmapped memory — see the decisive finding
above). Live single-stepping is not currently viable against this target; before attempting
it again, the QEMU aarch64 gdbstub setup itself needs to be validated independently (try a
different QEMU version, `-accel tcg,thread=single`, or confirm hardware breakpoint support
against a trivial known-working aarch64 QEMU target first, outside this kernel entirely) —
don't repeat this exact approach expecting a different result. Two competing crash
signatures still unreconciled: `ESR_EL1=0x02000000`/EC=0 "Unknown reason" (this crash, and
the 2026-08-08 keyboard-input crash) vs. `ESR_EL1=0x9600004f`/EC=0x25 genuine data-abort with
a poisoned-looking `FAR_EL1=0x000055bd` (June/July 2026 crashes) — may or may not be the same
bug. Genuinely open.