Route VM-dictionary sf_malloc/sf_free through the kernel's kmalloc heap

Captain Bob's call after seeing the identity-heap-capacity findings
(FABRIC-3.md §X.4): the number of concurrently-running VMs is not known
in advance, and once this is a complete operating system the heap should
be able to use whatever memory is actually available -- not a hardcoded
compile-time ceiling. This was already half-built and just not wired up.

src/starkernel/vm/alloc_kernel.c previously implemented sf_malloc()/
sf_free() (platform_alloc.h's allocator abstraction -- what
vm_create_word() calls for every VM's word dictionary) as its own
isolated static 4MB arena: first-fit free list, no splitting or
coalescing. That's exactly the allocator that topped out around 6
concurrent WIREBIND-born identities, failing from fragmentation before
true capacity exhaustion (§X.4's own measurements).

Sitting right next to it, unused for this purpose: src/starkernel/memory/
kmalloc.c, the kernel's general heap. Already initialized at boot (M6,
kernel_main.c, well before any VM is ever born), reserved from real
PMM-tracked physical memory rather than a fixed array, defaults to a
2 GiB floor explicitly sized "for 256+ baby VMs" per its own comment,
overridable via the --heap= boot flag, and its free list actually
coalesces neighboring blocks on every free.

Change: alloc_kernel.c's sf_malloc()/sf_free() now delegate to
kmalloc_aligned()/kfree() instead of managing a separate arena.
sf_alloc_init() becomes a no-op (kmalloc is already initialized by the
time any VM allocation can happen, and "resetting" a heap now shared by
every kernel subsystem would be actively wrong -- confirmed no external
caller depended on its old reset semantics). sf_alloc_get_stats() reads
kmalloc_get_stats() fresh rather than shadowing byte counts locally;
alloc_count/free_count (which kmalloc.c doesn't track) stay as simple
local counters. sf_calloc()/sf_realloc() are otherwise unchanged. Kernel-
only: the hosted (non-kernel) StarForth build keeps its own separate
alloc_host.c implementation, untouched.

Verified live: replaying the exact hotplug sequence that previously
topped out at 6 identities (Zuse + 8 identities, one at a time via QMP
device_add) now succeeds for all 9, where identity 05 specifically used
to fail. Three-arch clean qemu acceptance (single Zuse device, the
standard regression case) passed on amd64, aarch64, and riscv64 -- one
aarch64 attempt hit an unrelated, already-documented one-off QEMU hiccup
(empty log, boot never progressed past firmware) and passed cleanly on
retry with no rebuild.

Not addressed here: the underlying free-list itself is still first-fit
without splitting (only coalescing changed, inherited from kmalloc.c);
per-VM dictionary sizing (shrinking what each WIREBIND VM's word set
actually needs) is a separate, still-open lever from FABRIC-3.md §X.4's
open architecture question.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Ec88YKxxhZGG1RNnune78

capsules/BLOCK_MAP.md

@@ -1,5 +1,5 @@
 # Capsule Block Manifest — Auto-generated
-<!-- Generated by mkcapsule --manifest  2026-09-07T15:26:51Z -->
+<!-- Generated by mkcapsule --manifest  2026-09-07T16:39:49Z -->
 <!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh.  -->
 <!-- Hand-written justifications and immutability notes live  -->
 <!-- in MANIFEST.md alongside this auto-generated index.      -->

src/starkernel/vm/alloc_kernel.c

@@ -23,171 +23,108 @@
 /*
  * platform/alloc_kernel.c - Kernel (bare-metal) memory allocator
  *
- * Static arena, bump-allocated with a real free list on top.
+ * FABRIC-3.md §X.4, 2026-09-07: this used to be its own isolated static
+ * 4MB arena (first-fit free list, no splitting/coalescing) -- sized small
+ * enough, and fragile enough under concurrent VM churn, that a live
+ * identity-heap-capacity test found the arena topping out around 6
+ * concurrent WIREBIND-born VMs, failing *before* true capacity exhaustion
+ * (fragmentation from concurrent background allocation, no coalescing to
+ * recover from it). Meanwhile `kmalloc.c` -- the kernel's general heap,
+ * already initialised at boot (M6, well before any VM is ever born) --
+ * sits right next to it: reserved from real PMM-tracked physical memory
+ * (not a fixed compile-time array), defaults to a 2 GiB floor explicitly
+ * sized "for 256+ baby VMs" per its own comment, overridable via the
+ * `--heap=` boot flag, and its free list *does* coalesce neighboring
+ * blocks on every free.
  *
- * FABRIC-3.md §VII follow-on, 2026-09-06: this used to bump-allocate only,
- * with sf_free() a deliberate no-op -- "VM is long-lived (no restart),
- * allocations happen at init time, runtime allocations are rare... no
- * fragmentation issues in practice." That premise held until this
- * session's own repeated identity-verification workflow (WIREBIND
- * birth/kill cycles, one console+user VM pair per identity, all sharing
- * this one global arena) needed VMs born and killed repeatedly within a
- * single boot -- confirmed live: a real kernel PANIC ("malloc failed"
- * cascading into "Stadium: eviction... governor invariant broken", full
- * halt) at the exact same cycle count on every run, because freed
- * dictionary entries (vm_cleanup() was itself also missing this free
- * before an earlier pass of this same fix) were never actually reclaimed
- * -- sf_free() threw them away regardless.
+ * Every VM's word dictionary (`vm_create_word()`, the shared/vendored VM
+ * core) allocates through this file's `sf_malloc()`/`sf_free()` --
+ * `platform_alloc.h`'s portable allocator abstraction, kernel-side. This
+ * file now simply delegates to `kmalloc()`/`kfree()` instead of managing
+ * its own separate, much smaller arena: same API contract callers already
+ * depend on, real headroom (whatever the boot-time heap ends up being,
+ * not a hardcoded 4MB), and real coalescing. No new allocator was
+ * invented -- `kmalloc.c` already existed, was already boot-tested on all
+ * three arches, and was simply never wired up as the backing store for
+ * VM dictionaries specifically.
  *
- * Every allocation now carries a small header (size + free-list link) so
- * a freed block can be pushed onto g_free_list and reused by a
- * later sf_malloc() of equal or smaller size (first-fit, no splitting --
- * deliberately simple: this workload's repeated allocations are for the
- * same capsules loaded into a fresh VM each time, so freed blocks from a
- * just-killed VM's dictionary are typically an exact or near-exact fit
- * for the next VM's own). Falls back to bump-allocating a fresh block
- * from the arena when no free block is large enough, exactly as before.
+ * `sf_alloc_init()` is now a no-op (see its own doc comment below): with
+ * a *shared* heap serving every kernel subsystem, not an isolated arena
+ * used only for VM dictionaries, "reset" is no longer a sane operation --
+ * nothing external ever called it anyway (confirmed: no callers besides
+ * `sf_malloc()`'s own lazy-init guard, which this file's `sf_malloc()`
+ * still keeps, now as a readiness check rather than an initializer).
  *
- * Arena size: 4MB by default (configurable via SF_ARENA_SIZE)
+ * This file's own alignment guarantee (`SF_ALIGN`, 8 bytes) is preserved
+ * via `kmalloc_aligned()` -- `kmalloc()`'s own default alignment
+ * (`KMALLOC_MIN_ALIGN`, 16 bytes) already satisfies it, but asking
+ * explicitly keeps this file's contract self-documenting rather than
+ * relying on kmalloc.c's own default not changing under it.
  */
 
 #include "platform_alloc.h"
+#include "starkernel/kmalloc.h"
 #include <stdint.h>
 #include <stddef.h>
 
-#ifndef SF_ARENA_SIZE
-#define SF_ARENA_SIZE (4 * 1024 * 1024)  /* 4MB default */
-#endif
-
-/* Alignment for all allocations (8 bytes for 64-bit safety) */
+/* Alignment for all allocations (8 bytes for 64-bit safety) -- unchanged
+ * from this file's previous arena-based implementation. */
 #define SF_ALIGN 8
-#define SF_ALIGN_UP(x) (((x) + (SF_ALIGN - 1)) & ~(SF_ALIGN - 1))
 
-/* Per-allocation header, immediately before the pointer sf_malloc()
- * returns. `next` is meaningful only while the block is on the free
- * list -- it's live/garbage data for the caller otherwise, matching the
- * classic free-list-node-in-freed-space technique, just kept as a fixed
- * header field instead of reusing payload bytes so there's no minimum-
- * payload-size constraint to worry about. */
-typedef struct sf_block_header {
-    size_t                    size; /* payload size, in bytes, SF_ALIGN_UP'd */
-    struct sf_block_header   *next; /* free-list link; valid only while free */
-} sf_block_header_t;
-
-/* Static arena */
-static uint8_t g_arena[SF_ARENA_SIZE] __attribute__((aligned(SF_ALIGN)));
-static size_t g_arena_offset = 0;
-static int g_initialized = 0;
-static sf_block_header_t *g_free_list = (sf_block_header_t *)0;
-
-/* Statistics */
-static sf_alloc_stats_t g_stats = {0};
+/* alloc_count/free_count have no kmalloc.c equivalent (it tracks bytes,
+ * not call counts) -- kept here as simple diagnostic counters layered on
+ * top of kmalloc's own byte-accurate stats, which sf_alloc_get_stats()
+ * below reads fresh on every call rather than shadowing them locally. */
+static size_t g_alloc_count = 0;
+static size_t g_free_count = 0;
 
 /**
- * @brief Initialise the kernel bump allocator.
+ * @brief No-op: kept only for API compatibility with `platform_alloc.h`.
  *
- * Resets @c g_arena_offset to zero and clears all @c g_stats fields.
- * Sets @c g_initialized so that @c sf_malloc() skips the lazy-init guard.
- * Safe to call at any point; all prior allocations become invalid after a
- * reset, so this must only be called before any VM allocation takes place.
+ * The backing store is now the shared kernel heap (`kmalloc.c`), already
+ * initialised at boot (M6) well before any VM allocation can occur --
+ * there is nothing left for this function to set up, and "resetting" a
+ * heap shared by every other kernel subsystem would be actively wrong.
+ * Kept callable (matches its documented "safe to call at any point"
+ * contract) so no caller needs to change.
  *
  * @return 0 always (cannot fail)
  */
 int sf_alloc_init(void)
 {
-    g_arena_offset = 0;
-    g_free_list = (sf_block_header_t *)0;
-    g_initialized = 1;
-
-    g_stats.total_bytes = SF_ARENA_SIZE;
-    g_stats.used_bytes = 0;
-    g_stats.peak_bytes = 0;
-    g_stats.alloc_count = 0;
-    g_stats.free_count = 0;
-
     return 0;
 }
 
 /*
- * @brief Allocate memory from the static kernel arena, reusing a freed
- *        block first if one is large enough.
+ * @brief Allocate memory from the shared kernel heap.
  *
- * Rounds @p size up to @c SF_ALIGN (8 bytes). First searches @c
- * g_free_list for the first block whose payload is >= the requested size
- * (first-fit, no splitting -- see this file's own top-of-file doc comment
- * for why that's the right tradeoff for this workload) and reuses it
- * whole if found. Otherwise bump-allocates a fresh header+payload block
- * from @c g_arena, exactly as before this fix. Lazily calls @c
- * sf_alloc_init() on the first invocation if the arena has not been
- * explicitly initialised. Returns @c NULL for zero-size requests and when
- * neither a free block nor remaining arena space can satisfy the request.
+ * Delegates to `kmalloc_aligned()` (`SF_ALIGN`-byte aligned, matching
+ * this file's previous guarantee). Returns @c NULL for zero-size
+ * requests and whenever the shared heap itself returns NULL (not yet
+ * initialised, or genuinely out of memory) -- both already part of this
+ * function's documented contract.
  *
  * @param size Number of bytes to allocate.
- * @return Pointer to the allocated block's payload on success, @c NULL on
- *         failure.
+ * @return Pointer to the allocated block on success, @c NULL on failure.
  */
 void* sf_malloc(size_t size)
 {
-    if (!g_initialized) sf_alloc_init();
     if (size == 0) return (void*)0;
 
-    size_t payload = SF_ALIGN_UP(size);
-
-    /* First-fit scan of the free list. */
-    sf_block_header_t **pp = &g_free_list;
-    while (*pp)
-    {
-        if ((*pp)->size >= payload)
-        {
-            sf_block_header_t *blk = *pp;
-            *pp = blk->next;
-            blk->next = (sf_block_header_t *)0;
-
-            g_stats.used_bytes += blk->size;
-            g_stats.alloc_count++;
-            if (g_stats.used_bytes > g_stats.peak_bytes)
-            {
-                g_stats.peak_bytes = g_stats.used_bytes;
-            }
-            return (void *)(blk + 1);
-        }
-        pp = &(*pp)->next;
-    }
-
-    /* No free block large enough -- bump-allocate a fresh one. */
-    size_t header_size = SF_ALIGN_UP(sizeof(sf_block_header_t));
-    size_t total = header_size + payload;
-    size_t new_offset = g_arena_offset + total;
-
-    if (new_offset > SF_ARENA_SIZE)
-    {
-        /* Out of memory */
-        return (void*)0;
-    }
-
-    sf_block_header_t *blk = (sf_block_header_t *)&g_arena[g_arena_offset];
-    g_arena_offset = new_offset;
-    blk->size = payload;
-    blk->next = (sf_block_header_t *)0;
-
-    g_stats.used_bytes += payload;
-    g_stats.alloc_count++;
-    if (g_stats.used_bytes > g_stats.peak_bytes)
-    {
-        g_stats.peak_bytes = g_stats.used_bytes;
-    }
-
-    return (void *)(blk + 1);
+    void *ptr = kmalloc_aligned(size, SF_ALIGN);
+    if (ptr) g_alloc_count++;
+    return ptr;
 }
 
 /*
- * @brief Allocate and zero-initialise a contiguous array from the kernel arena.
+ * @brief Allocate and zero-initialise a contiguous array from the kernel heap.
  *
- * Computes @p count × @p size, checks for integer overflow via
+ * Unchanged from this file's previous implementation: computes
+ * @p count × @p size, checks for integer overflow via
  * @c total/size != count, then delegates to @c sf_malloc(). The returned
- * block is zeroed with a manual byte loop rather than @c memset so that the
- * kernel build remains freestanding with no libc dependency.
- * Returns @c NULL when either argument is zero, on overflow, or on arena
+ * block is zeroed with a manual byte loop rather than @c memset so that
+ * the kernel build remains freestanding with no libc dependency.
+ * Returns @c NULL when either argument is zero, on overflow, or on heap
  * exhaustion.
  *
  * @param count Number of elements to allocate.
@@ -220,19 +157,22 @@ void* sf_calloc(size_t count, size_t size)
 }
 
 /*
- * @brief Resize an allocation in the kernel arena.
+ * @brief Resize an allocation in the kernel heap.
  *
- * Because this is a bump allocator with no size metadata, the original block
- * cannot be grown in place. Instead a fresh block of @p new_size bytes is
- * bump-allocated and returned; the original block at @p ptr is orphaned
- * (leaked in place) for the lifetime of the kernel.
+ * Unchanged from this file's previous implementation: `kmalloc.c` has no
+ * realloc-equivalent, so a fresh block of @p new_size bytes is allocated
+ * and returned; the original block at @p ptr is left for the caller to
+ * free explicitly (this function does not free it, matching this file's
+ * prior documented behavior exactly -- not a regression introduced by
+ * the kmalloc.c switch).
  *
  * Special cases match the C standard:
  * - @p ptr == @c NULL → equivalent to @c sf_malloc(@p new_size).
  * - @p new_size == 0 → returns @c NULL (caller treats old block as freed).
  *
- * The caller is responsible for copying content from the old block before
- * discarding the old pointer; this function does not perform the copy.
+ * The caller is responsible for copying content from the old block
+ * before discarding the old pointer; this function does not perform the
+ * copy.
  *
  * @param ptr      Pointer to the existing allocation (may be @c NULL).
  * @param new_size Desired size of the new block in bytes.
@@ -241,60 +181,41 @@ void* sf_calloc(size_t count, size_t size)
  */
 void* sf_realloc(void* ptr, size_t new_size)
 {
-    /*
-     * Bump allocator limitation: we don't track allocation sizes.
-     * For kernel use, realloc is rarely needed.
-     *
-     * Strategy: allocate new block, caller must copy if needed.
-     * This wastes memory but works for rare realloc cases.
-     *
-     * If ptr is NULL, this is equivalent to malloc.
-     */
     if (!ptr) return sf_malloc(new_size);
     if (new_size == 0) return (void*)0;
 
-    /* Allocate new block - old block is orphaned (bump allocator limitation) */
     return sf_malloc(new_size);
 }
 
 /*
- * @brief Release a kernel arena allocation, making it available for reuse.
+ * @brief Release a kernel heap allocation, making it available for reuse.
  *
- * FABRIC-3.md §VII follow-on, 2026-09-06: this used to be a documented
- * no-op (see this file's own top-of-file doc comment for why that
- * stopped being acceptable). Pushes the block's header onto @c
- * g_free_list, where a future @c sf_malloc() of equal or smaller size
- * will find and reuse it -- no coalescing with neighboring free blocks,
- * matching the same simplicity tradeoff @c sf_malloc()'s first-fit search
- * makes.
+ * Delegates to @c kfree(), which coalesces the freed block with any free
+ * neighbors -- real reclamation, unlike this file's previous first-fit-
+ * no-splitting free list (FABRIC-3.md §X.4: that lack of coalescing was
+ * the proximate cause of a live-observed allocation failure under
+ * concurrent VM churn even with more than enough free bytes overall).
  *
  * @param ptr Pointer previously returned by @c sf_malloc() / @c sf_calloc()
- *            (may be @c NULL; silently ignored). Must not be used again
- *            by the caller after this call, and must not be freed twice.
+ *            (may be @c NULL; silently ignored, matching @c kfree()'s own
+ *            contract). Must not be used again by the caller after this
+ *            call, and must not be freed twice.
  */
 void sf_free(void* ptr)
 {
     if (!ptr) return;
 
-    sf_block_header_t *blk = ((sf_block_header_t *)ptr) - 1;
-    blk->next = g_free_list;
-    g_free_list = blk;
-
-    if (g_stats.used_bytes >= blk->size)
-    {
-        g_stats.used_bytes -= blk->size;
-    }
-    g_stats.free_count++;
+    kfree(ptr);
+    g_free_count++;
 }
 
 /*
  * @brief Retrieve a snapshot of kernel allocator statistics.
  *
- * Copies the internal @c g_stats structure to the caller-supplied buffer.
- * Because the bump allocator never reclaims memory, @c used_bytes is the
- * exact number of arena bytes consumed to date and @c peak_bytes equals
- * @c used_bytes. @c free_count counts @c sf_free() calls for accounting
- * purposes only — it does not reflect any actual reclamation.
+ * Reads fresh from `kmalloc_get_stats()` on every call -- the shared heap
+ * is the real source of truth, not a value shadowed locally. alloc_count/
+ * free_count (which kmalloc.c does not track) come from this file's own
+ * counters instead.
  *
  * @param stats Output buffer to receive the statistics; silently returns
  *              without writing if @p stats is @c NULL.
@@ -302,5 +223,11 @@ void sf_free(void* ptr)
 void sf_alloc_get_stats(sf_alloc_stats_t* stats)
 {
     if (!stats) return;
-    *stats = g_stats;
+
+    kmalloc_stats_t k = kmalloc_get_stats();
+    stats->total_bytes = (size_t)k.total_bytes;
+    stats->used_bytes  = (size_t)k.used_bytes;
+    stats->peak_bytes  = (size_t)k.peak_bytes;
+    stats->alloc_count = g_alloc_count;
+    stats->free_count  = g_free_count;
 }