Add FENCE word (SDK v1.9.0 scoping); fix severe pre-existing FORGET use-after-free

FENCE ( -- ) exposes the dict_fence_latest/dict_fence_here state FORGET
already honored internally, letting callers (e.g. a future SDK capsule)
raise the boundary after loading their own content -- no new VM fields,
no policy logic beyond exposing existing state.

Writing a direct test for it surfaced a real, severe, pre-existing bug in
FORGET's relink logic, unrelated to FENCE itself and reproducible with the
original boot-time fence alone:

- Forgetting the single newest word incorrectly destroyed every other word
  back to the fence too, not just the target.
- Forgetting an older word (correctly cascading to remove newer words too,
  per FORTH-79 semantics) crashed with SIGSEGV.

Root cause: the relink code's target_prev pointer was, by construction,
always inside the range the preceding loop had just freed whenever target
wasn't vm->latest -- so writing through it was a use-after-free every time
that branch executed. Fixed by removing the target_prev tracking and both
branches entirely; vm->latest unconditionally becomes target_next (target's
own captured, still-valid link) after the free loop, correct in every case.

Added a FENCE test suite to dictionary_manipulation_words_test.c (Module 14)
including the exact regression case (forgetting the newest word must not
disturb an older one). Verified zero warnings and identical POST/dict_hash
results across all three kernel architectures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-08-19 05:57:18 -04:00
co-authored by Claude Sonnet 5
parent 6d8b48f100
commit 4e7dcdf889
11 changed files with 113606 additions and 20706 deletions
+50
View File
@@ -1752,3 +1752,53 @@ Block range for `capsules/sdk.4th`, whenever it's written: `5200+` (clear of `tu
once scoped: three-arch boot to `zuse)ok>`, POST green, `mkcapsule --lint` clean, HOWTOs
present for everything shipped — matching every other acceptance bar in this document, nothing
new invented for this specific release.
**Decisions made 2026-08-19 (user's calls, per the open questions above):** SDK vocabulary
re-exports the cookbook (`turtle.4th` + `doe.4th`) plus `FENCE`; load model is REPL-only
(`S" sdk.4th" EXEC`, matching `turtle.4th`'s own precedent, not wired into `init.4th`);
kernel-only is explicitly accepted (the SDK is a kernel-side developer surface, hosted builds
remain compile-sanity-only). Version string and `turtle.4th`'s unconfirmed rendering remain
open, not decided here.
**`FENCE ( -- )` implemented, and a real, severe pre-existing bug in `FORGET` found and fixed
in the process — not a design bug in the new word, a bug in code this session didn't touch
until now.** Added `dictionary_word_fence()` right next to `FORGET` in `defining_words.c`
(registered alongside it) — it does exactly what the scoping above proposed: raises
`dict_fence_latest`/`dict_fence_here` to the current dictionary top, nothing more. While
writing a direct reproduction test for it (three plain `CREATE`d words, forget the newest,
confirm the older one survives), the older one did *not* survive — `UNKNOWN WORD` — with no
`FENCE` call involved at all, using only the original boot-time fence. Traced to
`defining_words.c`'s `FORGET` relink logic (`~653-660`, pre-fix): when the target being
forgotten is `vm->latest` itself (the single newest word), the code branches on
`target_prev == NULL` and — reading that as "nothing exists between `vm->latest` and the
fence" — resets `vm->latest = vm->dict_fence_latest` directly, discarding *everything* back to
the fence instead of just the one targeted word. That reasoning was simply wrong:
`target_prev == NULL` only means target is newest, not that it's adjacent to the fence.
**Escalated on a second reproduction, past the first (narrower) framing — this is a
use-after-free, not just an off-by-one.** Testing the *other* direction (forget the *oldest*
of three words, which per FORTH-79 "target and newer" semantics should legitimately remove all
three) produced a **SIGSEGV**, not wrong output. Root cause: the preceding free loop already
frees every entry from `vm->latest` down through target inclusive — which includes
`target_prev` (by construction, the search loop's `prev` pointer is always the entry
immediately newer than target, and thus always inside that just-freed range whenever target
isn't `vm->latest` itself). The `if (target_prev) { target_prev->link = ...; vm->latest =
target_prev; }` branch was therefore *always* operating on already-freed memory when it ran at
all — the `target_prev == NULL` branch (discussed above) merely hid this by taking the *other*
path instead, in the one case where it happened not to matter. Fix: deleted the `target_prev`
tracking and both branches entirely; `vm->latest` unconditionally becomes `target_next`
(target's own captured `->link`, valid and outside the freed range) after the free loop —
correct whether target was latest, oldest, or in the middle, and correct whether it was the
only survivor-adjacent entry or the fence itself. Verified this collapses cleanly to the
already-correct fence-boundary-rejection case too (`FORGET DUP` against the original boot
fence still correctly refuses, no crash, dictionary intact afterward).
Added a `FENCE` `WordTestSuite` entry to `dictionary_manipulation_words_test.c` right after
`FORGET`'s own (Module 14) — three cases: forgetting a post-`FENCE` word succeeds, forgetting a
pre-`FENCE` word is refused, and `forget_latest_keeps_predecessor` — the exact regression this
fix addresses, encoded as a permanent POST case so it can't silently regress again. Verified:
zero build warnings hosted and on all three kernel architectures; hosted POST 1009→1012 (+3,
matching the new test count exactly), 0 failed, 0 errors; three-arch kernel boot identical
(1012/0/0 on amd64/aarch64/riscv64), `dict_hash` matches exactly across all three (changed from
the pre-`FENCE` baseline as expected — a new C word legitimately changes the dictionary hash;
cross-arch agreement is what's being checked, not stability against the prior baseline).
+1 -1
View File
@@ -1,5 +1,5 @@
# Capsule Block Manifest — Auto-generated
<!-- Generated by mkcapsule --manifest 2026-08-19T05:50:05Z -->
<!-- Generated by mkcapsule --manifest 2026-08-19T09:54:15Z -->
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
<!-- Hand-written justifications and immutability notes live -->
<!-- in MANIFEST.md alongside this auto-generated index. -->
BIN
View File
Binary file not shown.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
Binary file not shown.
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -70,6 +70,34 @@ static WordTestSuite dict_manip_word_suites[] = {
3, {0}
},
{
/* FENCE ( -- ): raises the FORGET boundary to the current dictionary
* top. Added alongside a real bug fix in FORGET itself, caught while
* building this word -- forgetting the single newest word used to
* incorrectly destroy every other word back to the fence too (a
* use-after-free in the relink logic, not just an off-by-one), a
* SIGSEGV in the worst case. "forget_latest_keeps_predecessor" below
* is exactly the regression case that fix addresses. */
"FENCE", {
{
"basic", "CREATE fnc1 FENCE CREATE fnc2 FORGET fnc2",
"Should forget a word defined after FENCE", TEST_NORMAL, 0, 1, {0}
},
{
"protects_older", "CREATE fnc3 CREATE fnc4 FENCE FORGET fnc3",
"Should refuse to forget a word defined before FENCE", TEST_ERROR_CASE, 1, 1, {0}
},
{
"forget_latest_keeps_predecessor",
"CREATE fnc5 FENCE CREATE fnc6 CREATE fnc7 FORGET fnc7 fnc6 DROP",
"Forgetting the newest word must not disturb an older one still above FENCE",
TEST_NORMAL, 0, 1, {0}
},
{NULL, NULL, NULL, TEST_NORMAL, 0, 0, {0}}
},
3, {0}
},
{
"IMMEDIATE", {
{"basic", ": test3 42 ; IMMEDIATE test3 . CR", "Should execute immediately", TEST_NORMAL, 0, 1, {0}},
+31 -13
View File
@@ -76,6 +76,7 @@ static void defining_word_literal(VM * vm);
static void defining_word_does(VM * vm);
static void defining_word_immediate(VM * vm);
static void dictionary_word_forget(VM * vm);
static void dictionary_word_fence(VM * vm);
/* ───────────────────────────── Runtimes ───────────────────────────── */
@@ -579,18 +580,16 @@ static void dictionary_word_forget(VM *vm) {
int forget_failed = 0;
sf_mutex_lock(&vm->dict_lock);
/* Find target and its previous entry */
DictEntry *prev = NULL, *e = vm->latest;
DictEntry *target = NULL, *target_prev = NULL;
/* Find target */
DictEntry *e = vm->latest;
DictEntry *target = NULL;
while (e) {
if (!(e->flags & WORD_HIDDEN) &&
e->name_len == (uint8_t) nlen &&
memcmp(e->name, namebuf, (size_t) nlen) == 0) {
target = e;
target_prev = prev;
break;
}
prev = e;
e = e->link;
}
if (!target) {
@@ -649,14 +648,17 @@ static void dictionary_word_forget(VM *vm) {
e = next;
}
/* Relink: if target_prev exists, point it past the freed chain */
if (target_prev) {
target_prev->link = target_next; /* Skip freed entries */
vm->latest = target_prev; /* Latest is now the entry before target */
} else {
/* No prev means target was latest - reset to fence */
vm->latest = vm->dict_fence_latest;
}
/* Relink: FORGET removes target and everything newer (closer to
* vm->latest) -- the free loop above just freed every entry from
* vm->latest through target inclusive, which by definition includes
* any entry between them. There is nothing left to "skip past" --
* target_next (target's own surviving link) is always the correct
* new head, whether or not target was vm->latest itself. (Previously
* this branched on whether an entry existed between vm->latest and
* target, and relinked through it -- but that entry was always inside
* the just-freed range, making it a use-after-free on every FORGET of
* a non-latest word.) */
vm->latest = target_next;
vm->here = (size_t) new_here;
@@ -669,6 +671,21 @@ forget_exit:
return;
}
/* FENCE ( -- ) raise the FORGET boundary to the current dictionary top.
* dict_fence_latest/dict_fence_here already exist and are already honored
* by FORGET (set once at bootstrap, protecting the base wordset); this is
* the first word exposing that state so callers can advance it further --
* e.g. an SDK capsule calling FENCE after loading protects its own
* definitions the same way the base wordset is already protected. */
static void dictionary_word_fence(VM *vm) {
if (!vm) return;
sf_mutex_lock(&vm->dict_lock);
vm->dict_fence_latest = vm->latest;
vm->dict_fence_here = vm->here;
sf_mutex_unlock(&vm->dict_lock);
log_message(LOG_DEBUG, "FENCE: raised to HERE=%ld", (long) vm->dict_fence_here);
}
/* DOES> — IMMEDIATE: finalize the defining words create-part and compile DOES>-body */
static void defining_word_does(VM *vm) {
if (!vm) return;
@@ -755,6 +772,7 @@ void register_defining_words(VM *vm) {
/* Dictionary management */
register_word(vm, "FORGET", dictionary_word_forget);
register_word(vm, "FENCE", dictionary_word_fence);
/* Compile helpers — immediate */
register_word(vm, "COMPILE", defining_word_compile);