FABRIC-3.md §X: document the §IX.5 follow-on — xHCI root-cause/fix, live hotplug walkthrough, identity heap capacity findings

Documents three commits' worth of live-tested work in one continuous
arc, picking up exactly where §IX.5 left off:

- X.1: the "4th-device enumeration failure" was a QEMU test-harness
  port-topology artifact, not a driver bug (commit 30c26ad).
- X.2: the real driver bug it uncovered once ports were fixed --
  Configure Endpoint completions silently dropped under concurrent
  multi-device enumeration, root-caused via temporary reverted probes
  and fixed with the same single-in-flight discipline the rest of the
  driver already uses (commit e10fb76).
- X.3: a live 9-device hotplug walkthrough (Zuse, bob/rajames, 00-06,
  one at a time via QMP) verifying the dynamic attach path and
  per-device identity data integrity, distinct from the boot-time scan
  path X.1/X.2 exercised.
- X.4: exact, live-measured kernel-heap-arena numbers per identity VM
  (457,392 bytes, zero variance across 6 identities), a real leak found
  and fixed (orphaned console VM never torn down on user-VM birth
  failure, commit d8a195b), and the open architecture question this
  surfaces -- identity capacity is fragmentation-limited around 6
  concurrent identities today, not yet decided how to address.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Ec88YKxxhZGG1RNnune78

FABRIC-3.md

@@ -1495,3 +1495,154 @@ failure-path logging itself hasn't been exercised live yet, only compiled. Three
 failure that motivated §IX.5 (`xhci: not a Mass Storage/SCSI/BOT device` on a 4th concurrent
 attach) remains unexplained and unfixed — tracked as the next item, separate from the four
 closures above.
+
+## X. §IX.5 follow-on: xHCI concurrent enumeration, root-caused and CLOSED; identity VM heap
+capacity measured, one real leak found and fixed, capacity question still OPEN — 2026-09-06/07
+
+Picks up exactly where §IX.5 left off. Two separate root causes, both real, found by actually
+attaching multiple devices live rather than reasoning about the driver in the abstract — same
+lesson §VII and §IX already taught this project twice over.
+
+### X.1 — The "4th-device enumeration failure" was never a driver bug — CLOSED, commit `30c26ad`
+
+Root cause, confirmed live via QEMU's own `info usb` monitor command *before touching any
+kernel code*: the default `qemu-xhci` controller (`p2=4,p3=4`) exposes only **4 real dual-role
+ports**, not 8 as the parameter names suggest (p2/p3 count the *same* physical ports from the
+USB2/USB3 register sets, not additive). Every device attach in this project used
+`-device usb-storage,bus=xhci0.0,drive=...` with no explicit `port=`; past the 4th device, QEMU
+silently auto-inserts an internal USB2 hub and everything past that is invisible to a driver
+with no hub-descent logic. The driver's `"not a Mass Storage/SCSI/BOT device"` warning was
+**correct** — the device really was a hub. The per-slot xHCI/BOT machinery from §VII was
+inspected and found structurally sound for real top-level-port devices; no live corruption bug
+was in it.
+
+**Fix, entirely in the QEMU test harness, not the kernel:** new `XHCI_PORTS` Make variable
+(`Makefile.starkernel`, default 16, overridable) sizes `p2`/`p3` on all three arches with real
+headroom above the current 9-device identity roster — sized deliberately, not exactly matched
+to today's count, per the standing ruling against hardcoding a bound to today's scale (§VII.4).
+`ZUSEDISK_QEMU_ARGS` now gives Zuse's drive an explicit `port=1`; `QEMU_EXTRA`'s own doc comment
+shows the pattern for attaching more devices.
+
+**Found and fixed in passing:** `ZUSEDISK`'s Makefile default (`disk/zuse.img`) was stale —
+deleted from git at `c3db963`, superseded by `disk/thumbdrives/zuse-thumb-ident.img`, but the
+default was never updated, so a plain `make qemu` silently booted with no Zuse device attached
+at all. Corrected to the real minted image.
+
+**Flagged, not touched (not asked):** `scripts/bleach_zuse_img.sh` and `disk/README.md` still
+reference the deleted `disk/zuse.img` path.
+
+Three-arch `clean qemu` acceptance (single Zuse device, port 1) passed on amd64/aarch64/riscv64
+with this change.
+
+### X.2 — The real driver bug: Configure Endpoint completions dropped under concurrency — CLOSED, commit `e10fb76`
+
+With X.1's port fix in place, actually booting the real kernel with Zuse + all 8 identity
+drives attached concurrently (9 devices, each its own real port) surfaced a second, genuine bug:
+only **1 of 9** devices ever reached `blkio_usb: MSC device ready`. No error and no success for
+the other 8 — silence, not a diagnosable failure.
+
+Root-caused live via temporary WARN-level diagnostic probes (written, captured, fully reverted
+— the committed fix contains no probe code) at four points in `xhci.c`: the initial port scan,
+the connect handler, the Command Completion Event handler, and the deferred per-slot dispatch
+loop. The probes disproved the first hypothesis (a stalled `pending_events[]` connect queue —
+all 9 devices actually completed Enable Slot + Address Device correctly) and found the real one:
+`xhci_poll_events()`'s deferred dispatch loop submitted `XHCI_NEXT_ACTION_CONFIGURE_ENDPOINT` (a
+Command Ring op) **unconditionally** for every slot with that action pending in a single pass —
+unlike Enable Slot/Address Device/Disable Slot, which are correctly gated behind
+`dev->connect_state == XHCI_CONN_IDLE` before ever submitting. With 2+ devices enumerating at
+once, this let multiple Configure Endpoint commands sit outstanding on the Command Ring
+simultaneously. Their completion is correlated purely via the single shared `dev->connect_state`
+field, not the completion event's own Slot ID — so whichever slot's completion happened to land
+while `connect_state` still read `AWAIT_CONFIGURE_ENDPOINT` got correctly chained into
+`SET_CONFIG`; every other slot's completion arrived after `connect_state` had already moved on
+and was silently swallowed by the handler's generic "unrelated command completion" catch-all —
+no error path exists for that catch-all, which is exactly why it produced silence rather than a
+diagnosable failure.
+
+**Fix:** the same single-in-flight discipline this driver already uses for every other Command
+Ring op. If the Ring isn't free when a slot's Configure Endpoint action comes due, put the
+action back on that slot instead of racing a second command onto a busy Ring; the next tick's
+dispatch pass retries it.
+
+**Verified live:** the same 9-device concurrent boot now produces 9 `MSC device ready` lines and
+zero xHCI errors (was 1 of 9). Three-arch single-device regression (the standard acceptance
+case) still passed on all three arches.
+
+### X.3 — Live 9-device hotplug walkthrough, on request: dynamic attach path also verified, not just boot-time scan
+
+Separate from X.1/X.2's boot-time-attach testing, Captain Bob directed a live walkthrough: one
+QEMU instance up with nothing attached, then Zuse, bob/rajames, and 00 through 06 hotplugged one
+at a time via QMP `drive_add`+`device_add` on distinct `XHCI_PORTS`, watching the kernel's live
+reaction after each. This exercises `xhci_poll_events()`'s Port Status Change *event* path
+(true hotplug), not `xhci_scan_ports_for_already_connected()`'s boot-time initial-scan path
+(X.1/X.2's scenario) — a genuinely different code path through the same driver.
+
+Every hotplug through 04 succeeded cleanly: `blkio_usb: MSC device ready`, correct distinct
+per-device identity readback (a temporary probe read each drive's own `full_name`/`username`
+straight off its own devblock via `blkio_read()`, confirming e.g. slot 3 really said `"00"`, not
+Zuse's or bob's data — no cross-contamination), and a live `WIREBIND` birth per identity. Six
+identities (including Zuse, who costs no extra VM) were live simultaneously with zero errors.
+The 05 hotplug is where X.4 below picks up — its USB/xHCI attach itself succeeded fine (further
+confirming X.2's fix generalizes to the hotplug path, not just boot-time scan); what failed next
+was a different layer entirely.
+
+### X.4 — Identity VM kernel-heap capacity: exact numbers measured live, one real leak found and FIXED (commit `d8a195b`), the capacity/fragmentation question itself is OPEN
+
+X.3's 05 hotplug attached at the USB layer but then failed at `WIREBIND`'s VM-birth step with a
+kernel allocator exhaustion error (`vm_create_word: malloc failed`, `vm_init: rolling window
+malloc failed`, `PARITY:BIRTH_FAILED`) — graceful, no crash, boot continued to `(zuse) ok>`, but
+a real scalability wall. Measured precisely (not estimated) via a temporary allocator-stats
+probe in `capsule_run.c`'s `capsule_parity_log_birth`/`capsule_parity_log_birth_failed` (written,
+captured, fully reverted — no probe code committed), replaying the same hotplug sequence:
+
+- Kernel heap arena (`alloc_kernel.c`, `SF_ARENA_SIZE`) is a **fixed 4 MiB (4,194,304 bytes)** —
+  does not grow.
+- Boot baseline (Hera + Hermes + Artemis, the 3 system VMs): **781,904 bytes**.
+- Zuse costs **nothing extra** — she authenticates into the existing Hera session rather than
+  birthing a new VM.
+- Every other identity (`WIREBIND`) costs **exactly 457,392 bytes, with zero variance**,
+  confirmed identical across bob, 00, 01, 02, 03, 04. This is two VM births per identity: a
+  "default personality" console VM (228,576 bytes) then the real user session (228,816 bytes).
+- **6 identities fit** (bob + 00 + 01 + 02 + 03 + 04): arena at 3,526,256 / 4,194,304 used,
+  668,048 free.
+- **The 7th identity (05) failed — not from clean capacity exhaustion.** 668,048 bytes were
+  free, more than the usual 457,392 needed, but the attempt died partway through (peak
+  3,876,624, only 350,368 bytes into itself), with individual 128-136 byte word-registration
+  allocations failing. 662 `sf_free()` calls happened *during* that single failed attempt —
+  background heartbeat/physics activity from the 6 already-live VMs churning the same arena
+  concurrently. The allocator's free-list is documented as first-fit with **no splitting or
+  coalescing**, so fragmentation from that concurrent churn is the likely proximate cause, not
+  raw capacity.
+
+**The bug found and FIXED here:** the failure was *leaking* memory, permanently, on top of the
+fragmentation problem. `capsule_wirebind_try_attach()` births its two VMs (console, then user)
+in sequence; when the second failed, the console VM that had *already succeeded* was never torn
+down — an orphan nothing ever points a real user at, since `WIREBIND` only ever hands the caller
+the user VM's id. Confirmed live: after the failed attempt, `used_bytes` settled at 3,754,832 —
+228,576 bytes *above* the pre-attempt baseline, i.e. one whole console VM's worth, permanently
+gone. Every failed attach was making the arena strictly worse for the next attempt, not neutral.
+
+Fix: `capsule_vm_kill(username)` on the now-orphaned console VM before returning from the
+failure path in `capsule_wirebind.c` — the same teardown `capsule_wirebind_eject()`/
+`capsule_wirebind_unclean_detach()` already use elsewhere in that file (`vm_cleanup()` +
+`sf_free()`, confirmed live in §IX.2/§IX.3 to actually reclaim per-word dictionary allocations).
+Verified live with the same probe, replaying the identical failure: `used_bytes` after the fix
+returns to **exactly** the pre-attempt baseline (3,526,256, matching precisely) instead of
+leaking 228,576 bytes. Three-arch acceptance passed with the fix included.
+
+**Still open — the actual architecture question, not yet decided (Captain Bob, 2026-09-07):**
+"identity is everything in this OS." The leak fix makes retries neutral instead of compounding,
+but the underlying ~6-identity practical ceiling (itself a fragmentation artifact, not a hard
+capacity number) is unaddressed. Options on the table, none chosen yet:
+- Grow the heap / make the arena size dynamic instead of a fixed 4 MiB compile-time constant.
+- Shrink per-VM dictionaries — every `WIREBIND` user VM currently carries the full word set,
+  including things like `TTF-TEXT`/`SCROLL-BACK`/`KBD-SCAN` a non-console user session may not
+  need.
+- Address the fragmentation contributor directly (splitting/coalescing in the free-list, or
+  serializing VM-birth allocation against concurrent background allocation) — this is *why*
+  identity 05 failed even with more free bytes than it needed.
+
+**Also flagged, not touched (not asked):** `Makefile.starkernel`'s
+`printf '$(KERNEL_ARGS)\n' > starforth.cfg` breaks when `KERNEL_ARGS` starts with `--` (dash's
+`printf` misreads it as an option). Worked around live during X.2's debug-level testing with a
+harmless prefix token (`KERNEL_ARGS="x --log-level=debug"`); not fixed in the Makefile itself.