LithosAnanake/docs/working/architecture/HERMES-MESSAGE-BLOCK-STORAGE-DESIGN-20260708.md

# Hermes Message/Channel Block-Backed Storage — Design Doc
**Date:** 2026-07-08 (rev 2026-07-09d)
**Branch:** `lithosananke`
**Status:** Design only. Not implemented. This is the dependency flagged
in `HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md`'s Status header —
that doc's heat-window sizing is provisional until this one lands.
**Author:** Captain Bob / Claude Code

**Revision history** (kept honest per this repo's convention — see
`VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md` for the pattern):

- **rev a** (original): proposed Hermes duplicating Artemis's free-map
  allocator shape verbatim, in isolation, as a standalone Hermes-only
  change. Wrong framing — see rev b.
- **rev b**: corrected rev a toward a two-VM "shared Physical BAM /
  Logical BAM effort" — Hermes and Artemis jointly building toward a
  common logical layer. **Also wrong.** Captain Bob's direct correction:
  *"no direct coupling between Hermes and Artemis... Hermes knows NOTHING
  about Artemis or a bam of any kind."* Hermes's only inter-VM capability
  is messaging (send/receive on the common channel, request → ACK/NACK)
  — never reaching into another VM's storage, and never a BAM concept
  shared across VMs.
- **rev c** (this revision): the actual confirmed architecture, reached
  after grounding in how Artemis concretely lays out its own disk today.
  **Every VM manages its own storage independently** — own RAM blocks,
  then own ramdrive blocks, then its own dedicated appended virtio-blk
  disk, exactly mirroring Artemis's already-proven pattern but with zero
  knowledge of or coupling to Artemis (or to any other VM). No shared
  allocator, no coordinator VM. "Physical BAM" and "Logical BAM" are
  per-VM concepts — each VM eventually has its own of each — not a
  cross-VM split. Confirmed verbatim: *"each VM uses its ram blocks and
  ramdisk blocks then appends the virtual drive"* / *"yes, that's
  correct"* / *"it's going to look like PBAM and LBAM are the same right
  now. this will change when we start figuring out how to use and shape
  the tripod."*
- **rev d** (this revision, 2026-07-09): rev c is corrected. A full
  Tripod-docs audit (reading `.claude/TRIPOD.md`/`HERMES.md`/`ARTEMIS.md`
  in full, which should have happened before rev a was ever written)
  found that rev c's "every VM independently, no coordinator" picture is
  not how the Tripod is actually specified. `TRIPOD.md`'s dispatch table
  is explicit: `Block I/O → Artemis`, reached via `VM-EXEC` naming the
  VM by capability, not heat or convenience. `ARTEMIS.md` is direct:
  *"Artemis is memory. She owns the external attached virtual disk image
  and everything that lives on it."* Hermes does not grow its own disk.
  **Corrected architecture: Artemis is sole storage authority for the
  fleet.** Hermes's message-block persistence need is satisfied by
  *asking* Artemis, not by building parallel storage machinery — this
  supersedes rev c's "own dedicated appended virtio-blk disk" tier for
  Hermes entirely (see "Storage tiering," corrected below).

  This does **not** reopen rev b's rejection of direct coupling. *"There
  is to be no tight coupling between ANY VM"* — reaffirmed by Captain Bob
  in the same breath as naming Artemis storage authority — and it isn't
  in tension with that at all: `TRIPOD.md`'s capability-based dispatch
  (`VM-EXEC` by name, request/ACK-NACK) is already the Tripod's normal,
  loose, message-mediated interaction model. Hermes asking Artemis to
  store a message's payload is the same shape as any other cross-VM
  request — it was only ever a contradiction under the mistaken
  assumption that "Artemis as authority" meant Hermes reaching into
  Artemis's BAM or address space directly. It doesn't. Hermes never
  learns a physical LBN, never touches `FM-*`/`BLK-ALLOC`, never knows a
  zone exists. See "The logical-reference gap, resolved" below for the
  concrete mechanism.
- **rev e** (this revision, 2026-07-09): rev d overcorrected. Captain
  Bob's direct correction: *"ALL VMs regardless have ram backed blocks
  zero to 2047, ramdrive from 2048 to its end. that's a VMs state and
  identity hard requirement."* Rev c's tiers 1–2 (own RAM 0–2047, own
  ramdrive 2048+, per VM) were **right** — rev d was wrong to fold them
  into "Artemis owns everything." They're not storage Hermes borrows or
  requests; they're part of what makes a VM a VM at all, same category as
  its own dictionary and stacks. Corroborating evidence, not invented:
  `capsule_birth_baby()`'s own comment — *"PERSONALITY: per-VM block
  storage is M9 scope; no-op until then"* — names this as a real, planned
  milestone, just not built yet.

  **What actually changes under Artemis-as-storage-authority is narrower
  than rev d claimed: only the tier beyond a VM's own ramdrive.** Rev c's
  tier 3 ("own dedicated appended virtio-blk disk") is the one thing that
  doesn't survive — no VM other than Artemis appends a persistent disk.
  Once Hermes's own ramdrive is exhausted, *that's* when it asks Artemis,
  not before. Sections below are corrected to reflect three tiers again:
  own RAM, own ramdrive, then (not own disk, but) ask Artemis.

---

## Execution rules

Binding for all implementation work under this doc, matching the
discipline `VM-PHYSICS-DYNAMIC-FLEET-DESIGN-20260705.md` established and
Captain Bob's explicit intent for this specific piece of work — this is
as much a Tripod discovery exercise as it is a storage redesign:

1. **Small units.** One new word, one call-site migration, one allocator
   primitive at a time — not the whole redesign in one pass.
2. **Clean + make at checkpoints.** `make -f Makefile.starkernel clean`
   then build after each unit that touches buildable code.
3. **QEMU when applicable**, one ISA at a time (amd64 first, fastest under
   TCG), reserving the full three-arch pass for milestones — new
   primitives wired end-to-end, deletions landed, before declaring a
   phase implemented. This redesign is expected to surface real Tripod
   gaps the same way the `LOAD-DOE` bug did during the `doe-campaign.4th`
   migration — that's a feature of doing this incrementally with real
   boots, not a risk to design around.
4. **Never modify `FM-TEST`/`FM-SET`/`FM-CLR`/`FM-FIND-FREE`/`BLK-ALLOC`/
   `BLK-FREE` in `artemis/init.4th`.** They're registered, tested,
   HARD LOCKED (blocks 4110–4113, `capsules/MANIFEST.md`) — Artemis's own
   disk-tier allocator. Per rev e, Hermes never touches these, and never
   builds an equivalent for Artemis's disk; Hermes's own RAM/ramdrive
   tiers (own M9 allocator, once that lands) are a separate, independent
   implementation over Hermes's own blocks, not a modification, extension,
   or generalization of Artemis's disk-tier primitives.

---

## Problem

`hermes/init.4th`'s message and channel storage is a fixed in-memory pool:
`CREATE MSG-ARENA MSG-MAX MSG-CELLS * CELLS ALLOT` (Block 4101), a
32-slot, 9-cell-per-message (`MSG-CELLS`) free-list array; `CH-ARENA` is
the same shape at 16 slots. A fixed compile-time array can't grow without
recompiling, and Hermes's own comments already flag real exhaustion risk
(`FLEET-DOE-WORKLOAD`'s block 2058 comment: `BROADCAST-TEST`... would
exhaust Hermes's fixed member/message arenas after ~20 runs").

Separately, the heat-physics sibling doc's window-sizing already depends
on knowing the real capacity messages will be observed at (see that doc's
Status header) — an arena bound that can silently change relative to
`MSG-MAX`/`CH-MAX` compile-time constants makes that reasoning fragile.

## Goal

Give Hermes's messages and channels a way to outgrow the fixed 32/16
in-memory arenas — first into Hermes's own per-VM RAM/ramdrive blocks
(M9 scope), and only past that, **by asking Artemis to hold what Hermes's
own storage can't** — matching `TRIPOD.md`'s dispatch table exactly:
*"Block I/O → Artemis... `VM-EXEC` takes a VM name. The caller names the
correct VM for the work type. There is no routing algorithm."* Hermes
does grow its own block-addressed storage (own RAM, own ramdrive — that's
its per-VM identity, not optional); what it never grows is a *disk* of
its own. Past its own ramdrive, it sends a message; Artemis does the
storing.

**Explicitly not the goal:** Hermes appending its own persistent disk
(rev c's tier 3 — superseded, see revision history). Also not the goal:
designing Hermes's own per-VM RAM/ramdrive block layout (M9 scope,
separate work) or Artemis's own storage internals (zones, BAM cell
layout, migration policy) — the latter already specified, in more detail
than this doc would add, in `.claude/ARTEMIS.md`'s Future Material
section, and explicitly not build-authorized yet ("Do not implement any
of this without Captain Bob explicitly reopening it").

## The C/FORTH boundary question

Tiers 1–2 (Hermes's own RAM/ramdrive) are M9 scope — genuinely deferred,
not this doc's to design, so this question doesn't apply to them here.
For tier 3 (asking Artemis), the question doesn't arise for Hermes at
all — Hermes only ever calls `VM-EXEC` (or the equivalent message-send
word) naming Artemis, the same capability-based dispatch every other
Tripod interaction uses. Artemis's own block-storage subsystem is already
proven pure FORTH (`artemis/init.4th`'s free-map bitmap, allocator,
`LE32!`/`LE64!` arch-neutral I/O, all composed over `BLOCK`/`BUFFER`/
`C@`/`C!`, zero new C) — that precedent matters for whoever eventually
builds out Artemis's Future Material, not for anything in this doc.

## Storage model: three tiers — own RAM, own ramdrive, then ask Artemis

Every VM — Hermes included — has its own RAM-backed blocks 0–2047 and its
own ramdrive from 2048 to its end, as a hard requirement of what a VM is
(`capsule_birth_baby()`'s own "PERSONALITY: per-VM block storage is M9
scope" comment names this as planned, not yet built). `ARTEMIS.md`'s
*"Artemis is memory. She owns the external attached virtual disk image
and everything that lives on it"* is about the **external persistent
disk** specifically — it doesn't reach into a VM's own RAM or ramdrive,
which are that VM's own identity, not storage Artemis administers.

1. **Own RAM blocks first (0–2047).** Hermes's own private RAM-backed
   block range, part of its M9 per-VM identity, once that milestone
   lands. Today's `MSG-ARENA`/`CH-ARENA` (`CREATE...ALLOT`) are the
   pre-M9 stand-in for this tier — an in-VM-memory working set, not yet
   block-addressed. Fast, synchronously touchable, no round trip — the
   right home for a message actively being delivered, ACKed, or NACKed.
2. **Own ramdrive next (2048 to its end)**, once tier 1 isn't enough.
   Also part of Hermes's own per-VM identity under M9 — Hermes's own
   ramdrive, not a shared or Artemis-administered one. This is where
   `MSG-ARENA`/`CH-ARENA` graduate to once they outgrow the fixed
   32/16-slot compile-time arrays, block-addressed within Hermes's own
   range, still entirely within Hermes.
3. **Ask Artemis, only once tier 2 is exhausted.** This is the one place
   rev c was wrong and rev d's correction actually applies: Hermes does
   not append its own persistent disk. Once its own ramdrive runs out (or
   a message needs to survive past Hermes's own reap), it sends Artemis a
   store request and gets back a handle — see "The logical-reference gap,
   resolved," below.

**Escalation between tiers is compudynamics-driven, not a size
threshold.** `ARTEMIS.md`'s own Migration Policy already establishes the
pattern for exactly this kind of question: *"No zone is special... when a
zone has pressure... the coldest block migrates first because it is
coldest... no magic constants... everything else is self-organizing."*
That's not specific to Artemis's disk zones — it's the same heat/cool/
reap model already governing word heat, VM heat, and (once
`msg_decay_slope_q48` lands) message heat. Tier 1→2 pressure and tier
2→3 pressure work the same way: the coldest message under RAM-tier
pressure is what migrates to Hermes's own ramdrive, and the coldest
ramdrive-tier message under pressure is what gets sent to Artemis — not
an arbitrary byte-count cutoff invented for this doc. The concrete
numbers (window depth, pressure threshold) are still M9/Future-Material
scope, unspecified — but the *mechanism* isn't an open question, it's
already-established compudynamics, applied one level up.

`capsules/MANIFEST.md`'s existing Hermes ranges (4100–4153) stay what
they've always been — capsule source organization for `hermes/init.4th`'s
word definitions, not a claim on tiers 1–2's future per-VM block ranges
(those are a distinct, not-yet-designed M9 concern, out of scope here —
see Open questions).

## The logical-reference gap, resolved

The open question flagged when rev c was superseded: `ARTEMIS.md`'s
already-specified Logical BAM (`ARTEMIS.md`, "Two BAMs" — 4 cells:
physical LBN, heat, XXHash64 content identity, flags) is keyed by
content. That's the right key for what it's explicitly scoped for —
*"Content-addressing, ACL records, and cold capsule storage live here...
Identity is derived from block content — same content, same identity...
Mutation is detectable: hash changes if content changes."* A live Hermes
message is exactly the case that breaks that key: its content can change
during its lifetime (composed, delivered, ACKed), so a content hash
cannot be the stable handle Hermes holds onto across a request/fetch
round trip — the hash itself would move whenever the content did.

The resolution doesn't require inventing a new mechanism — `HERMES.md`
already has the right shape for a stable, VM-minted reference, used today
for exactly this purpose: *"`channel_id` = owner VM identity + monotonic
sequence number. Unforgeable by non-owners. Hermes mints it. No VM
constructs its own channel_id."* A message or channel that needs
Artemis-backed storage gets stored under that same kind of handle —
Hermes's own minted ID, opaque to Artemis, stable for the object's
lifetime regardless of what its content is doing or where Artemis
physically keeps it. Artemis's storage-request protocol accepts that
opaque ID as the key for this class of entry — necessarily a *different*
internal bookkeeping path from the content-addressed Logical BAM entries
`ARTEMIS.md` already specifies for cold/immutable data, since the two
have genuinely different identity semantics (stable-opaque-handle vs.
content-hash). Designing that internal path is Artemis's own future
work, not this doc's — flagged here only so the gap has a named, grounded
answer instead of sitting open.

## Physical BAM / Logical BAM — per-VM, both Hermes's and Artemis's own

Rev c had this right and rev d wrongly discarded it: Physical BAM
(free-map + block-heat allocator) and Logical BAM (normalizing that same
VM's own heat for fleet-conservation participation) are **per-VM
concepts**, not something only Artemis has. Under M9, Hermes eventually
needs its own Physical/Logical BAM over its own RAM/ramdrive blocks
(tiers 1–2) — a new, independent implementation, out of scope for this
doc (M9 is separate work), but real and not something this doc should
keep implying doesn't exist.

What rev d got right and stays true under rev e: Hermes's own BAM (once
built) has nothing to do with, and never touches, Artemis's disk-tier
BAM. `ARTEMIS.md`'s Future Material section already specifies Artemis's
own version at a level of detail this doc doesn't need to restate:
Physical BAM (2 cells — physical LBN, free flag; "the raw substrate... no
heat, no identity"), Logical BAM (4 cells — physical LBN, heat, content
identity, flags), three thermal zones with a self-organizing migration
policy. That whole section is explicitly marked Future Material in
`ARTEMIS.md`, not build-authorized without Captain Bob reopening it —
unrelated to whatever shape Hermes's own future per-VM BAM eventually
takes, and `artemis/init.4th`'s `FM-*`/`BLK-ALLOC`/`BLK-FREE` (blocks
4110–4113, ★ HARD LOCKED) remain untouched by Hermes either way.

## The mechanism (sketch — tier 3 protocol shape only)

Tiers 1–2 (Hermes's own RAM/ramdrive) are M9 scope, not sketched here —
that's per-VM block storage work shared by every Tripod VM, not specific
to Hermes's messages, and belongs in its own doc when M9 is reopened.
What follows is tier 3 only: what happens once Hermes's own ramdrive is
exhausted and it has to ask Artemis.

### 1. A store/fetch request, dispatched the same way any Tripod work is

`TRIPOD.md`'s dispatch model already covers this — Hermes names Artemis
by capability, same as any Block I/O work: something like `ART-STORE
( addr len logical-id -- ack/nack )` and `ART-FETCH ( logical-id --
addr len )`, sent via `VM-EXEC` (or the message-passing equivalent once
Hermes's async model, `HERMES.md`'s G1/G8 items, supports a return
value). `logical-id` is Hermes's own minted ID — see "The
logical-reference gap, resolved," above.

### 2. What Hermes sends is a payload, not a block

Hermes doesn't decide block geometry, packing, or free-map granularity —
that's Artemis's Future Material to design (`ARTEMIS.md`'s "Physical BAM
Block Geometry" section already sketches a 3-data+1-metadata packing
scheme for variable-length records generally, which a Hermes message
would be one instance of, once that chapter is reopened). This doc stops
at "Hermes hands Artemis a payload and an ID" — how Artemis packs,
allocates, and migrates it is out of scope here by design, not by gap.

### 3. Synchronous limitation, already named in `HERMES.md`

`HERMES.md`'s G8 gap applies directly: *"Wiring [a cross-VM value] into
`K-FLEET` in Hera requires a cross-VM return value — something the
current synchronous `VM-EXEC` model cannot deliver."* `ART-FETCH`
returning a payload has the same shape problem. Until the async
inter-VM model lands, a Hermes→Artemis fetch request is bounded by
whatever `VM-EXEC`'s synchronous call/return can actually carry — not
redesigned here, just inherited as a precondition.

## Consumer code migration

Deferred to implementation, for the same reason rev c deferred it: the
concrete request/response word names, and everything on Artemis's side
of the boundary, depend on Future Material that isn't build-authorized
yet (`ARTEMIS.md`: *"Do not begin implementation without explicit
instruction from Captain Bob"*). A follow-up pass once Artemis's flat-pool
Immediate Goal is done and this chapter is explicitly reopened should add
this section for real.

## Verification approach

- Three-arch acceptance as usual once any unit reaches boot-behavior-
  affecting completeness, per the Execution Rules above.
- Real message-storm test: repeat `BROADCAST-TEST`-style traffic well
  past the ~20-run point that exhausts today's 32-slot arena (per the
  comment already in `hermes/init.4th`), confirm messages that overflow
  to Artemis are still retrievable and don't silently drop — the
  concrete, observable proof this redesign did something, not just moved
  the same limit elsewhere.
- `TRIPOD-TEST`'s existing `PASS: broadcast reached all 3` /
  `PASS: msg queued` / `PASS: E2E msg flow` checks should continue
  passing unchanged throughout — this redesign changes what happens on
  arena overflow, not Hermes's external message-delivery behavior.

## Explicitly out of scope

- **The heat-decay mechanism** (`msg_decay_slope_q48`) — sibling doc,
  `HERMES-MESSAGE-CHANNEL-PHYSICS-DESIGN-20260708.md`, independent of
  this doc's rev d/e corrections.
- **Hermes's own per-VM RAM/ramdrive BAM (tiers 1–2, M9 scope).** Real,
  confirmed (`capsule_birth_baby()`'s own comment names it), and not
  designed in this doc — a separate, larger piece of work shared in
  shape by every Tripod VM, not specific to Hermes's messages.
- **Channel storage** (`CH-ARENA`) — this doc focuses on messages, the
  higher-volume and more clearly arena-exhaustion-prone of the two per
  the existing `hermes/init.4th` comment. Whether channels need the same
  overflow-to-Artemis treatment is a question for after messages are
  done, not decided here.
- **Artemis's own storage internals** (zones, BAM cell layout, packing,
  migration policy). Fully specified as Future Material in `ARTEMIS.md`
  already; explicitly not build-authorized without Captain Bob reopening
  that chapter. This doc's protocol sketch (section 1-3 above) doesn't
  depend on those internals being resolved to be a correct *shape* — it
  just can't be implemented before they are.
- **Finishing Artemis's own `K-FLEET` wiring gap.** Tracked in
  `ARTEMIS-BAM-ACCEPTANCE-20260703.md`'s Deferred/Next table and
  `ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md`. Not this doc's to close.
- **Artemis block heat generally** — sibling doc,
  `ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md`.

## Open questions (flagged, not guessed at)

1. **The concrete word names and wire shape** for the store/fetch
   protocol — sketched at the level of "a request with a logical ID,"
   not specified further, since it depends on Artemis's own
   not-yet-authorized storage internals.
2. **Target message capacity** — "bigger than 32" is not a number.
   Needs a real target, ideally informed by the message-storm test's
   observed exhaustion behavior at various loads.
3. **How Artemis's internal store distinguishes opaque-handle entries
   (Hermes messages) from content-addressed entries (cold capsules, ACL
   certs)** — flagged in "The logical-reference gap, resolved," above,
   as Artemis's own design work.
4. **The synchronous-call-return limitation** (section 3 above) — whether
   this doc's protocol waits for `HERMES.md`'s async model, or an interim
   shape is worth designing sooner.
5. **Whether anything here genuinely needs a new C primitive** — no
   evidence either way yet, since Artemis's side of this isn't designed.
   Precedent (`artemis/init.4th`'s existing block subsystem is pure
   FORTH) suggests no, but that's Artemis's question to answer when this
   chapter reopens, not this doc's to assume.
6. **The concrete pressure threshold for tier escalation** — the
   *mechanism* is settled (compudynamics heat-pressure, matching
   `ARTEMIS.md`'s Migration Policy — see "Storage model," above); the
   *numbers* (how much ramdrive pressure triggers a tier-2→3 request)
   depend on M9's own design, which doesn't exist yet.
7. **`ARTEMIS-FREEMAP-CAPSULE-RESERVATION-GAP-20260710.md` applies
   here too.** Artemis's live `BLK-ALLOC` doesn't yet reserve the LBN
   range her own capsule-loaded content occupies — a gap in Artemis's
   current allocator, not something this doc introduces. But a future
   `ART-STORE` request from Hermes (section 1 above) would hand out
   blocks from that same unprotected pool, so this doc's tier-3
   implementation inherits that gap and shouldn't be built before (or
   without accounting for) it being closed.