Document the riscv64 kernel stack trampoline fix
Write-up of the 2026-08-02 riscv64 boot crash investigation and fix
(commit 7366275), in the same style as docs/lithosananke/amd64-isr-fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
757ce97dc1
commit
0fa9bd730e
@@ -0,0 +1,183 @@
|
|||||||
|
# Kernel Stack Trampoline Fix — riscv64
|
||||||
|
|
||||||
|
**Date**: 2026-08-02
|
||||||
|
**Branch**: `master`
|
||||||
|
**Commit**: `7366275`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
The riscv64 kernel crashed partway through VM bootstrap during the 2026-08-01
|
||||||
|
DoE campaign run, well after amd64 and aarch64 had both booted cleanly to
|
||||||
|
`ok>` in the same session. The crash was a wild jump: execution landed at an
|
||||||
|
address nowhere inside the kernel's own linked image, then almost immediately
|
||||||
|
faulted trying to load from a small, near-null address.
|
||||||
|
|
||||||
|
```
|
||||||
|
Registering FORTH-79 Standard word set...
|
||||||
|
Registering FORTH-79 arithmetic words...
|
||||||
|
FORTH-79 Standard word set registration complete
|
||||||
|
*** EXCEPTION (riscv64) ***
|
||||||
|
scause = 0x000000000000000d
|
||||||
|
sepc = 0x00000000be27c3c2
|
||||||
|
stval = 0x00000000000074d8
|
||||||
|
```
|
||||||
|
|
||||||
|
`scause = 0xd` is a supervisor-mode load page fault. `sepc` (~0xbe27c3c2) sat
|
||||||
|
near the top of QEMU's 1 GB RAM window — nowhere close to the kernel's own
|
||||||
|
`.text`/`.data`/`.bss` footprint (`0x400000`–`0xcdf9a0`, confirmed via `nm`/
|
||||||
|
`readelf` on the exact build that produced the log). `stval` (0x74d8, ~30 KB)
|
||||||
|
stayed fixed across rebuilds even as `sepc` drifted — the signature of a
|
||||||
|
corrupted return address rather than a fixed logic bug at one source line.
|
||||||
|
|
||||||
|
## Investigation
|
||||||
|
|
||||||
|
Two dead ends were ruled out before the real cause surfaced:
|
||||||
|
|
||||||
|
1. **ELF relocation / buffer overlap in the split-build loader.** The
|
||||||
|
codebase has a `!MONOLITHIC_BUILD` path (`elf_load_kernel()` in
|
||||||
|
`elf_loader.c`) that loads `kernel.elf` from the ESP into a UEFI
|
||||||
|
`AllocateAnyPages` buffer. This looked like a plausible overlap candidate
|
||||||
|
at first, but `Makefile.starkernel` defaults to `MONOLITHIC=1`, meaning
|
||||||
|
`kernel_main` is linked directly into the loader and that code path never
|
||||||
|
runs. Confirmed by checking the built binary for the investigation's own
|
||||||
|
diagnostic strings — absent, because the compilation unit was built with
|
||||||
|
`-DMONOLITHIC_BUILD` and the code was preprocessed out.
|
||||||
|
2. **`RAW_LOG()` silently no-ops on every non-amd64 arch.** All of
|
||||||
|
`load_kernel_from_esp()`'s narration (and an early ConOut-based
|
||||||
|
diagnostic) never reached the captured serial log, on riscv64, in any
|
||||||
|
build. `RAW_LOG` was defined only for `ARCH_AMD64`; everywhere else it
|
||||||
|
expanded to `((void)0)`. This made straightforward diagnosis impossible
|
||||||
|
until a real raw-UART writer was added for riscv64 (see Secondary Fix).
|
||||||
|
|
||||||
|
With real output available, `console_println()` checkpoints were bisected
|
||||||
|
through `vm_init_with_host()` and `sk_vm_bootstrap_parity()`. Every checkpoint
|
||||||
|
inside `vm_init_with_host()` printed successfully, including the very last
|
||||||
|
line of the function — but the checkpoint immediately after its *return*
|
||||||
|
never appeared. Re-running with a trivial, unrelated local-variable addition
|
||||||
|
shifted the crash earlier and changed the exception type entirely (from a
|
||||||
|
load fault to an instruction-fetch fault at `sepc == stval`, i.e. the CPU
|
||||||
|
tried to *execute* an unmapped address). A crash whose location and even
|
||||||
|
category shift under unrelated code changes is the standard signature of
|
||||||
|
stack corruption, not a fixed bug at a specific call site.
|
||||||
|
|
||||||
|
## Root Cause
|
||||||
|
|
||||||
|
`kernel_main.c` documented the gap directly, once read closely:
|
||||||
|
|
||||||
|
> On amd64, `kernel_entry.S` switches the stack from UEFI's default to a
|
||||||
|
> 2 MiB zero-initialised BSS stack and tail-calls this function as
|
||||||
|
> `kernel_main_impl`. On aarch64 and riscv64 the assembly trampoline is not
|
||||||
|
> yet implemented and the UEFI loader calls `kernel_main` directly.
|
||||||
|
|
||||||
|
amd64 has carried a dedicated stack-switch trampoline
|
||||||
|
(`src/starkernel/arch/amd64/kernel_entry.S`) since early on, with its own
|
||||||
|
comment explaining why: *"UEFI hands control to kernel_main() running on a
|
||||||
|
small UEFI-provided stack (typically 128 KB–256 KB). The FORTH interpreter +
|
||||||
|
DOE experiment loop can easily exceed that depth."* riscv64 never had the
|
||||||
|
equivalent. It ran the entire VM bootstrap — 27 chained word-registration
|
||||||
|
modules, physics/SSM init, Tripod capsule birth — directly on whatever small
|
||||||
|
stack EDK2's riscv64 firmware handed the loader, overflowing it and
|
||||||
|
corrupting a return address. aarch64 has the same theoretical exposure but
|
||||||
|
happens not to trip it, apparently because its firmware's default boot stack
|
||||||
|
is larger — incidental, not a guarantee.
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
|
||||||
|
**`src/starkernel/arch/riscv64/kernel_entry.S`** (new) — RISC-V stack-switch
|
||||||
|
trampoline mirroring amd64's:
|
||||||
|
|
||||||
|
```asm
|
||||||
|
.section .bss
|
||||||
|
.align 4
|
||||||
|
.global g_kernel_stack
|
||||||
|
g_kernel_stack:
|
||||||
|
.space 0x200000
|
||||||
|
g_kernel_stack_top:
|
||||||
|
.global g_kernel_stack_top
|
||||||
|
|
||||||
|
.section .text
|
||||||
|
.extern kernel_main_impl
|
||||||
|
.global kernel_main
|
||||||
|
kernel_main:
|
||||||
|
ld t0, BOOT_INFO_KERNEL_STACK_BASE_OFFSET(a0)
|
||||||
|
bnez t0, .Ldynamic_stack
|
||||||
|
la t0, g_kernel_stack_top
|
||||||
|
j .Lstack_ready
|
||||||
|
.Ldynamic_stack:
|
||||||
|
ld t1, BOOT_INFO_KERNEL_STACK_SIZE_OFFSET(a0)
|
||||||
|
add t0, t0, t1
|
||||||
|
.Lstack_ready:
|
||||||
|
andi t0, t0, -16
|
||||||
|
mv sp, t0
|
||||||
|
mv s0, zero
|
||||||
|
tail kernel_main_impl
|
||||||
|
```
|
||||||
|
|
||||||
|
**`src/starkernel/kernel_main.c`** — riscv64 now builds `kernel_main_impl`
|
||||||
|
(invoked via the trampoline) instead of `kernel_main` directly, same pattern
|
||||||
|
as amd64:
|
||||||
|
|
||||||
|
```c
|
||||||
|
#if defined(__x86_64__) || defined(__riscv)
|
||||||
|
void kernel_main_impl(BootInfo *boot_info) {
|
||||||
|
#else
|
||||||
|
void kernel_main(BootInfo *boot_info) {
|
||||||
|
#endif
|
||||||
|
```
|
||||||
|
|
||||||
|
**`Makefile.starkernel`** — wires the new file into the riscv64 loader build:
|
||||||
|
|
||||||
|
```makefile
|
||||||
|
ifeq ($(ARCH),riscv64)
|
||||||
|
LOADER_ASM += $(KERNEL_SRC)/arch/$(ARCH)/kernel_entry.S
|
||||||
|
endif
|
||||||
|
```
|
||||||
|
|
||||||
|
## Secondary Fix
|
||||||
|
|
||||||
|
`RAW_LOG()` in `src/starkernel/boot/uefi_loader.c` was a no-op on every arch
|
||||||
|
except amd64, silencing all of the loader's own diagnostic messages on
|
||||||
|
riscv64 (and aarch64). Added a real raw-UART writer for riscv64, targeting
|
||||||
|
QEMU virt's 16550-compatible `uart8250` at MMIO `0x10000000` (matches
|
||||||
|
`Domain0 Region03` in the OpenSBI boot banner) — the same byte-polling
|
||||||
|
protocol as the existing amd64 port-I/O version, just MMIO instead of `outb`/
|
||||||
|
`inb`. Existing and future `RAW_LOG()` call sites in the loader now actually
|
||||||
|
produce output on riscv64.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
| Criterion | Before | After |
|
||||||
|
|---|---|---|
|
||||||
|
| `sepc` after word registration | Wild address outside kernel image (`0xbe27c3c2`-ish, drifting per build) | N/A — no longer occurs |
|
||||||
|
| Exception | Load or instruction-fetch page fault, non-deterministic between builds | None |
|
||||||
|
| `[Hera] ok>` (riscv64) | Never reached | ✅ reached, Tripod fleet (Hera/Hermes/Artemis) runs to completion |
|
||||||
|
| `PASS: E2E msg flow` | Never logged | ✅ |
|
||||||
|
| `PARITY:MAMA_INIT` / `Init: Mama birth OK` | Never logged | ✅ |
|
||||||
|
| Loader `RAW_LOG()` output on riscv64 | Silent no-op | ✅ visible in serial log |
|
||||||
|
|
||||||
|
## Acceptance Testing
|
||||||
|
|
||||||
|
All three architectures must boot to `ok>`, run the Tripod fleet, and reach
|
||||||
|
`[Hera] ok>`. Per repo convention, run in this exact order, one QEMU instance
|
||||||
|
at a time, always `clean` before `qemu`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make -f Makefile.starkernel ARCH=amd64 clean qemu
|
||||||
|
make -f Makefile.starkernel ARCH=aarch64 clean qemu
|
||||||
|
make -f Makefile.starkernel ARCH=riscv64 clean qemu
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify each log:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep "ok>" logs/<session>/<arch>/qemu-<arch>-*.log
|
||||||
|
grep "PASS: E2E msg flow" logs/<session>/<arch>/qemu-<arch>-*.log
|
||||||
|
grep "PARITY:MAMA_INIT" logs/<session>/<arch>/qemu-<arch>-*.log
|
||||||
|
grep "EXCEPTION" logs/<session>/<arch>/qemu-<arch>-*.log # must be empty
|
||||||
|
```
|
||||||
|
|
||||||
|
All three architectures confirmed clean on 2026-08-02 (see
|
||||||
|
`logs/20260802-064050/riscv64/`, `logs/20260802-064424/amd64/`,
|
||||||
|
`logs/20260802-064656/aarch64/`, committed alongside this fix).
|
||||||
Reference in New Issue
Block a user