LithosAnanake/docs/working/architecture/HERMES-MESSAGE-BLOCK-STORAGE-DESIGN-20260708.md

Hermes Message/Channel Block-Backed Storage — Design Doc

Date: 2026-07-08 (rev 2026-07-09d) Branch: lithosananke Status: Design only. Not implemented. This is the dependency flagged in HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md's Status header — that doc's heat-window sizing is provisional until this one lands. Author: Captain Bob / Claude Code

Revision history (kept honest per this repo's convention — see VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md for the pattern):

• rev a (original): proposed Hermes duplicating Artemis's free-map allocator shape verbatim, in isolation, as a standalone Hermes-only change. Wrong framing — see rev b.

• rev b: corrected rev a toward a two-VM "shared Physical BAM / Logical BAM effort" — Hermes and Artemis jointly building toward a common logical layer. Also wrong. Captain Bob's direct correction: "no direct coupling between Hermes and Artemis... Hermes knows NOTHING about Artemis or a bam of any kind." Hermes's only inter-VM capability is messaging (send/receive on the common channel, request → ACK/NACK) — never reaching into another VM's storage, and never a BAM concept shared across VMs.

• rev c (this revision): the actual confirmed architecture, reached after grounding in how Artemis concretely lays out its own disk today. Every VM manages its own storage independently — own RAM blocks, then own ramdrive blocks, then its own dedicated appended virtio-blk disk, exactly mirroring Artemis's already-proven pattern but with zero knowledge of or coupling to Artemis (or to any other VM). No shared allocator, no coordinator VM. "Physical BAM" and "Logical BAM" are per-VM concepts — each VM eventually has its own of each — not a cross-VM split. Confirmed verbatim: "each VM uses its ram blocks and ramdisk blocks then appends the virtual drive" / "yes, that's correct" / "it's going to look like PBAM and LBAM are the same right now. this will change when we start figuring out how to use and shape the tripod."

• rev d (this revision, 2026-07-09): rev c is corrected. A full Tripod-docs audit (reading .claude/TRIPOD.md/HERMES.md/ARTEMIS.md in full, which should have happened before rev a was ever written) found that rev c's "every VM independently, no coordinator" picture is not how the Tripod is actually specified. TRIPOD.md's dispatch table is explicit: Block I/O → Artemis, reached via VM-EXEC naming the VM by capability, not heat or convenience. ARTEMIS.md is direct: "Artemis is memory. She owns the external attached virtual disk image and everything that lives on it." Hermes does not grow its own disk. Corrected architecture: Artemis is sole storage authority for the fleet. Hermes's message-block persistence need is satisfied by asking Artemis, not by building parallel storage machinery — this supersedes rev c's "own dedicated appended virtio-blk disk" tier for Hermes entirely (see "Storage tiering," corrected below).

  This does not reopen rev b's rejection of direct coupling. "There is to be no tight coupling between ANY VM" — reaffirmed by Captain Bob in the same breath as naming Artemis storage authority — and it isn't in tension with that at all: TRIPOD.md's capability-based dispatch (VM-EXEC by name, request/ACK-NACK) is already the Tripod's normal, loose, message-mediated interaction model. Hermes asking Artemis to store a message's payload is the same shape as any other cross-VM request — it was only ever a contradiction under the mistaken assumption that "Artemis as authority" meant Hermes reaching into Artemis's BAM or address space directly. It doesn't. Hermes never learns a physical LBN, never touches FM-*/BLK-ALLOC, never knows a zone exists. See "The logical-reference gap, resolved" below for the concrete mechanism.

• rev e (this revision, 2026-07-09): rev d overcorrected. Captain Bob's direct correction: "ALL VMs regardless have ram backed blocks zero to 2047, ramdrive from 2048 to its end. that's a VMs state and identity hard requirement." Rev c's tiers 1–2 (own RAM 0–2047, own ramdrive 2048+, per VM) were right — rev d was wrong to fold them into "Artemis owns everything." They're not storage Hermes borrows or requests; they're part of what makes a VM a VM at all, same category as its own dictionary and stacks. Corroborating evidence, not invented: capsule_birth_baby()'s own comment — "PERSONALITY: per-VM block storage is M9 scope; no-op until then" — names this as a real, planned milestone, just not built yet.

  What actually changes under Artemis-as-storage-authority is narrower than rev d claimed: only the tier beyond a VM's own ramdrive. Rev c's tier 3 ("own dedicated appended virtio-blk disk") is the one thing that doesn't survive — no VM other than Artemis appends a persistent disk. Once Hermes's own ramdrive is exhausted, that's when it asks Artemis, not before. Sections below are corrected to reflect three tiers again: own RAM, own ramdrive, then (not own disk, but) ask Artemis.

---

Execution rules

Binding for all implementation work under this doc, matching the discipline VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md established and Captain Bob's explicit intent for this specific piece of work — this is as much a Tripod discovery exercise as it is a storage redesign:

1. Small units. One new word, one call-site migration, one allocator primitive at a time — not the whole redesign in one pass.
2. Clean + make at checkpoints. make -f Makefile.starkernel clean then build after each unit that touches buildable code.
3. QEMU when applicable, one ISA at a time (amd64 first, fastest under TCG), reserving the full three-arch pass for milestones — new primitives wired end-to-end, deletions landed, before declaring a phase implemented. This redesign is expected to surface real Tripod gaps the same way the LOAD-DOE bug did during the doe-campaign.4th migration — that's a feature of doing this incrementally with real boots, not a risk to design around.
4. Never modify FM-TEST/FM-SET/FM-CLR/FM-FIND-FREE/BLK-ALLOC/ BLK-FREE in artemis/init.4th. They're registered, tested, HARD LOCKED (blocks 4110–4113, capsules/MANIFEST.md) — Artemis's own disk-tier allocator. Per rev e, Hermes never touches these, and never builds an equivalent for Artemis's disk; Hermes's own RAM/ramdrive tiers (own M9 allocator, once that lands) are a separate, independent implementation over Hermes's own blocks, not a modification, extension, or generalization of Artemis's disk-tier primitives.

---

Problem

hermes/init.4th's message and channel storage is a fixed in-memory pool: CREATE MSG-ARENA MSG-MAX MSG-CELLS * CELLS ALLOT (Block 4101), a 32-slot, 9-cell-per-message (MSG-CELLS) free-list array; CH-ARENA is the same shape at 16 slots. A fixed compile-time array can't grow without recompiling, and Hermes's own comments already flag real exhaustion risk (FLEET-DOE-WORKLOAD's block 2058 comment: BROADCAST-TEST... would exhaust Hermes's fixed member/message arenas after ~20 runs").

Separately, the heat-physics sibling doc's window-sizing already depends on knowing the real capacity messages will be observed at (see that doc's Status header) — an arena bound that can silently change relative to MSG-MAX/CH-MAX compile-time constants makes that reasoning fragile.

Goal

Give Hermes's messages and channels a way to outgrow the fixed 32/16 in-memory arenas — first into Hermes's own per-VM RAM/ramdrive blocks (M9 scope), and only past that, by asking Artemis to hold what Hermes's own storage can't — matching TRIPOD.md's dispatch table exactly: "Block I/O → Artemis... VM-EXEC takes a VM name. The caller names the correct VM for the work type. There is no routing algorithm." Hermes does grow its own block-addressed storage (own RAM, own ramdrive — that's its per-VM identity, not optional); what it never grows is a disk of its own. Past its own ramdrive, it sends a message; Artemis does the storing.

Explicitly not the goal: Hermes appending its own persistent disk (rev c's tier 3 — superseded, see revision history). Also not the goal: designing Hermes's own per-VM RAM/ramdrive block layout (M9 scope, separate work) or Artemis's own storage internals (zones, BAM cell layout, migration policy) — the latter already specified, in more detail than this doc would add, in .claude/ARTEMIS.md's Future Material section, and explicitly not build-authorized yet ("Do not implement any of this without Captain Bob explicitly reopening it").

The C/FORTH boundary question

Tiers 1–2 (Hermes's own RAM/ramdrive) are M9 scope — genuinely deferred, not this doc's to design, so this question doesn't apply to them here. For tier 3 (asking Artemis), the question doesn't arise for Hermes at all — Hermes only ever calls VM-EXEC (or the equivalent message-send word) naming Artemis, the same capability-based dispatch every other Tripod interaction uses. Artemis's own block-storage subsystem is already proven pure FORTH (artemis/init.4th's free-map bitmap, allocator, LE32!/LE64! arch-neutral I/O, all composed over BLOCK/BUFFER/ C@/C!, zero new C) — that precedent matters for whoever eventually builds out Artemis's Future Material, not for anything in this doc.

Storage model: three tiers — own RAM, own ramdrive, then ask Artemis

Every VM — Hermes included — has its own RAM-backed blocks 0–2047 and its own ramdrive from 2048 to its end, as a hard requirement of what a VM is (capsule_birth_baby()'s own "PERSONALITY: per-VM block storage is M9 scope" comment names this as planned, not yet built). ARTEMIS.md's "Artemis is memory. She owns the external attached virtual disk image and everything that lives on it" is about the external persistent disk specifically — it doesn't reach into a VM's own RAM or ramdrive, which are that VM's own identity, not storage Artemis administers.

1. Own RAM blocks first (0–2047). Hermes's own private RAM-backed block range, part of its M9 per-VM identity, once that milestone lands. Today's MSG-ARENA/CH-ARENA (CREATE...ALLOT) are the pre-M9 stand-in for this tier — an in-VM-memory working set, not yet block-addressed. Fast, synchronously touchable, no round trip — the right home for a message actively being delivered, ACKed, or NACKed.
2. Own ramdrive next (2048 to its end), once tier 1 isn't enough. Also part of Hermes's own per-VM identity under M9 — Hermes's own ramdrive, not a shared or Artemis-administered one. This is where MSG-ARENA/CH-ARENA graduate to once they outgrow the fixed 32/16-slot compile-time arrays, block-addressed within Hermes's own range, still entirely within Hermes.
3. Ask Artemis, only once tier 2 is exhausted. This is the one place rev c was wrong and rev d's correction actually applies: Hermes does not append its own persistent disk. Once its own ramdrive runs out (or a message needs to survive past Hermes's own reap), it sends Artemis a store request and gets back a handle — see "The logical-reference gap, resolved," below.

Escalation between tiers is compudynamics-driven, not a size threshold. ARTEMIS.md's own Migration Policy already establishes the pattern for exactly this kind of question: "No zone is special... when a zone has pressure... the coldest block migrates first because it is coldest... no magic constants... everything else is self-organizing." That's not specific to Artemis's disk zones — it's the same heat/cool/ reap model already governing word heat, VM heat, and (once msg_decay_slope_q48 lands) message heat. Tier 1→2 pressure and tier 2→3 pressure work the same way: the coldest message under RAM-tier pressure is what migrates to Hermes's own ramdrive, and the coldest ramdrive-tier message under pressure is what gets sent to Artemis — not an arbitrary byte-count cutoff invented for this doc. The concrete numbers (window depth, pressure threshold) are still M9/Future-Material scope, unspecified — but the mechanism isn't an open question, it's already-established compudynamics, applied one level up.

capsules/MANIFEST.md's existing Hermes ranges (4100–4153) stay what they've always been — capsule source organization for hermes/init.4th's word definitions, not a claim on tiers 1–2's future per-VM block ranges (those are a distinct, not-yet-designed M9 concern, out of scope here — see Open questions).

The logical-reference gap, resolved

The open question flagged when rev c was superseded: ARTEMIS.md's already-specified Logical BAM (ARTEMIS.md, "Two BAMs" — 4 cells: physical LBN, heat, XXHash64 content identity, flags) is keyed by content. That's the right key for what it's explicitly scoped for — "Content-addressing, ACL records, and cold capsule storage live here... Identity is derived from block content — same content, same identity... Mutation is detectable: hash changes if content changes." A live Hermes message is exactly the case that breaks that key: its content can change during its lifetime (composed, delivered, ACKed), so a content hash cannot be the stable handle Hermes holds onto across a request/fetch round trip — the hash itself would move whenever the content did.

The resolution doesn't require inventing a new mechanism — HERMES.md already has the right shape for a stable, VM-minted reference, used today for exactly this purpose: "channel_id = owner VM identity + monotonic sequence number. Unforgeable by non-owners. Hermes mints it. No VM constructs its own channel_id." A message or channel that needs Artemis-backed storage gets stored under that same kind of handle — Hermes's own minted ID, opaque to Artemis, stable for the object's lifetime regardless of what its content is doing or where Artemis physically keeps it. Artemis's storage-request protocol accepts that opaque ID as the key for this class of entry — necessarily a different internal bookkeeping path from the content-addressed Logical BAM entries ARTEMIS.md already specifies for cold/immutable data, since the two have genuinely different identity semantics (stable-opaque-handle vs. content-hash). Designing that internal path is Artemis's own future work, not this doc's — flagged here only so the gap has a named, grounded answer instead of sitting open.

Physical BAM / Logical BAM — per-VM, both Hermes's and Artemis's own

Rev c had this right and rev d wrongly discarded it: Physical BAM (free-map + block-heat allocator) and Logical BAM (normalizing that same VM's own heat for fleet-conservation participation) are per-VM concepts, not something only Artemis has. Under M9, Hermes eventually needs its own Physical/Logical BAM over its own RAM/ramdrive blocks (tiers 1–2) — a new, independent implementation, out of scope for this doc (M9 is separate work), but real and not something this doc should keep implying doesn't exist.

What rev d got right and stays true under rev e: Hermes's own BAM (once built) has nothing to do with, and never touches, Artemis's disk-tier BAM. ARTEMIS.md's Future Material section already specifies Artemis's own version at a level of detail this doc doesn't need to restate: Physical BAM (2 cells — physical LBN, free flag; "the raw substrate... no heat, no identity"), Logical BAM (4 cells — physical LBN, heat, content identity, flags), three thermal zones with a self-organizing migration policy. That whole section is explicitly marked Future Material in ARTEMIS.md, not build-authorized without Captain Bob reopening it — unrelated to whatever shape Hermes's own future per-VM BAM eventually takes, and artemis/init.4th's FM-*/BLK-ALLOC/BLK-FREE (blocks 4110–4113, ★ HARD LOCKED) remain untouched by Hermes either way.

The mechanism (sketch — tier 3 protocol shape only)

Tiers 1–2 (Hermes's own RAM/ramdrive) are M9 scope, not sketched here — that's per-VM block storage work shared by every Tripod VM, not specific to Hermes's messages, and belongs in its own doc when M9 is reopened. What follows is tier 3 only: what happens once Hermes's own ramdrive is exhausted and it has to ask Artemis.

1. A store/fetch request, dispatched the same way any Tripod work is

TRIPOD.md's dispatch model already covers this — Hermes names Artemis by capability, same as any Block I/O work: something like ART-STORE ( addr len logical-id -- ack/nack ) and ART-FETCH ( logical-id -- addr len ), sent via VM-EXEC (or the message-passing equivalent once Hermes's async model, HERMES.md's G1/G8 items, supports a return value). logical-id is Hermes's own minted ID — see "The logical-reference gap, resolved," above.

2. What Hermes sends is a payload, not a block

Hermes doesn't decide block geometry, packing, or free-map granularity — that's Artemis's Future Material to design (ARTEMIS.md's "Physical BAM Block Geometry" section already sketches a 3-data+1-metadata packing scheme for variable-length records generally, which a Hermes message would be one instance of, once that chapter is reopened). This doc stops at "Hermes hands Artemis a payload and an ID" — how Artemis packs, allocates, and migrates it is out of scope here by design, not by gap.

3. Synchronous limitation, already named in HERMES.md

HERMES.md's G8 gap applies directly: "Wiring [a cross-VM value] into K-FLEET in Hera requires a cross-VM return value — something the current synchronous VM-EXEC model cannot deliver." ART-FETCH returning a payload has the same shape problem. Until the async inter-VM model lands, a Hermes→Artemis fetch request is bounded by whatever VM-EXEC's synchronous call/return can actually carry — not redesigned here, just inherited as a precondition.

Consumer code migration

Deferred to implementation, for the same reason rev c deferred it: the concrete request/response word names, and everything on Artemis's side of the boundary, depend on Future Material that isn't build-authorized yet (ARTEMIS.md: "Do not begin implementation without explicit instruction from Captain Bob"). A follow-up pass once Artemis's flat-pool Immediate Goal is done and this chapter is explicitly reopened should add this section for real.

Verification approach

• Three-arch acceptance as usual once any unit reaches boot-behavior- affecting completeness, per the Execution Rules above.
• Real message-storm test: repeat BROADCAST-TEST-style traffic well past the ~20-run point that exhausts today's 32-slot arena (per the comment already in hermes/init.4th), confirm messages that overflow to Artemis are still retrievable and don't silently drop — the concrete, observable proof this redesign did something, not just moved the same limit elsewhere.
• TRIPOD-TEST's existing PASS: broadcast reached all 3 / PASS: msg queued / PASS: E2E msg flow checks should continue passing unchanged throughout — this redesign changes what happens on arena overflow, not Hermes's external message-delivery behavior.

Explicitly out of scope

• The heat-decay mechanism (msg_decay_slope_q48) — sibling doc, HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md, independent of this doc's rev d/e corrections.
• Hermes's own per-VM RAM/ramdrive BAM (tiers 1–2, M9 scope). Real, confirmed (capsule_birth_baby()'s own comment names it), and not designed in this doc — a separate, larger piece of work shared in shape by every Tripod VM, not specific to Hermes's messages.
• Channel storage (CH-ARENA) — this doc focuses on messages, the higher-volume and more clearly arena-exhaustion-prone of the two per the existing hermes/init.4th comment. Whether channels need the same overflow-to-Artemis treatment is a question for after messages are done, not decided here.
• Artemis's own storage internals (zones, BAM cell layout, packing, migration policy). Fully specified as Future Material in ARTEMIS.md already; explicitly not build-authorized without Captain Bob reopening that chapter. This doc's protocol sketch (section 1-3 above) doesn't depend on those internals being resolved to be a correct shape — it just can't be implemented before they are.
• Finishing Artemis's own K-FLEET wiring gap. Tracked in ARTEMIS-BAM-ACCEPTANCE-20260703.md's Deferred/Next table and ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md. Not this doc's to close.
• Artemis block heat generally — sibling doc, ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md.

Open questions (flagged, not guessed at)

1. The concrete word names and wire shape for the store/fetch protocol — sketched at the level of "a request with a logical ID," not specified further, since it depends on Artemis's own not-yet-authorized storage internals.
2. Target message capacity — "bigger than 32" is not a number. Needs a real target, ideally informed by the message-storm test's observed exhaustion behavior at various loads.
3. How Artemis's internal store distinguishes opaque-handle entries (Hermes messages) from content-addressed entries (cold capsules, ACL certs) — flagged in "The logical-reference gap, resolved," above, as Artemis's own design work.
4. The synchronous-call-return limitation (section 3 above) — whether this doc's protocol waits for HERMES.md's async model, or an interim shape is worth designing sooner.
5. Whether anything here genuinely needs a new C primitive — no evidence either way yet, since Artemis's side of this isn't designed. Precedent (artemis/init.4th's existing block subsystem is pure FORTH) suggests no, but that's Artemis's question to answer when this chapter reopens, not this doc's to assume.
6. The concrete pressure threshold for tier escalation — the mechanism is settled (compudynamics heat-pressure, matching ARTEMIS.md's Migration Policy — see "Storage model," above); the numbers (how much ramdrive pressure triggers a tier-2→3 request) depend on M9's own design, which doesn't exist yet.
7. ARTEMIS-FREEMAP-CAPSULE-RESERVATION-GAP-20260710.md applies here too. Artemis's live BLK-ALLOC doesn't yet reserve the LBN range her own capsule-loaded content occupies — a gap in Artemis's current allocator, not something this doc introduces. But a future ART-STORE request from Hermes (section 1 above) would hand out blocks from that same unprotected pool, so this doc's tier-3 implementation inherits that gap and shouldn't be built before (or without accounting for) it being closed.