Add Artemis stress test for detecting cache aliasing bugs. Include statistical hypothesis evaluations, fix validation data, and run reports for validation across architectures.

Signed-off-by: Robert Allan James <robert.allan.james@gmail.com>
2026-08-02 14:49:52 -04:00
parent 148c4aa12c
commit 1cb68502fb
44 changed files with 5610173 additions and 2680 deletions
@@ -0,0 +1,7 @@
+*.aux
+*.log
+*.out
+*.toc
+*.synctex.gz
+*.fls
+*.fdb_latexmk
@@ -0,0 +1,388 @@
+\documentclass[10pt,a4paper]{article}
+
+%% ── Packages ─────────────────────────────────────────────────────────────
+\usepackage[T1]{fontenc}
+\usepackage[utf8]{inputenc}
+\usepackage[margin=2.5cm]{geometry}
+\usepackage{graphicx}
+\usepackage{booktabs}
+\usepackage{amsmath}
+\usepackage{amssymb}
+\usepackage{xcolor}
+\usepackage{microtype}
+\usepackage{parskip}
+\usepackage{enumitem}
+\usepackage[hidelinks,colorlinks=true,linkcolor=black,citecolor=black,urlcolor=blue]{hyperref}
+\usepackage{lmodern}
+\usepackage{colortbl}
+\usepackage{listings}
+
+%% ── Colour palette (matches experiments/bare_metal house style) ─────────
+\definecolor{sfblue}{RGB}{31,119,180}
+\definecolor{sforange}{RGB}{255,127,14}
+\definecolor{sfgreen}{RGB}{44,160,44}
+\definecolor{sfred}{RGB}{214,39,40}
+\definecolor{sfgray}{RGB}{80,80,80}
+
+\lstset{
+  basicstyle=\ttfamily\small,
+  breaklines=true,
+  frame=single,
+  columns=fullflexible,
+  backgroundcolor=\color{black!3}
+}
+
+%% ── Figure path ───────────────────────────────────────────────────────────
+\graphicspath{{figures/}}
+
+%% ── Title metadata ────────────────────────────────────────────────────────
+\title{%
+  \textbf{The Artemis Surface Stress Test}\\[0.4em]
+  \large A Stale-Pointer Aliasing Bug in the LithosAnanke Block Subsystem\\[0.3em]
+  \normalsize\textit{Found, Diagnosed, Fixed, and Verified Across Three Instruction-Set Architectures}
+}
+\author{%
+  R.\,A. James (Captain Bob)\\[0.2em]
+  \small StarForth Project \quad\texttt{robert.allan.james@gmail.com}
+}
+\date{2 August 2026}
+
+%% ═══════════════════════════════════════════════════════════════════════
+\begin{document}
+
+\maketitle
+\thispagestyle{empty}
+
+\begin{abstract}
+\noindent
+A newly written stress test for Artemis, LithosAnanke's block-storage VM,
+found a real data-corruption bug on its first completed run: 25 of 50
+random-block write/verify trials (50\%) returned the wrong byte pattern on
+amd64. The test's own recorded data ruled out its block-selection logic as
+the cause (no duplicate logical block numbers or devblocks among the 50
+trials). Four exploratory hypotheses were then tested formally rather than
+read off a chart: a Wald--Wolfowitz runs test found no significant serial
+clustering of failures by write order ($p=1.00$); a direct rank test
+against write order agreed ($p=0.715$); sibling position within a shared
+4\,KiB devblock showed no association ($p=0.674$); but logical block
+number itself showed a significant association with outcome
+($p<0.001$, Wilcoxon), an effect specific to this run's fixed random seed
+that this report does not claim generalizes. None of these exploratory
+tests, on their own, identify a mechanism. Root cause was instead found by
+source code review: a stale raw-pointer alias held by the VM-level block
+window cache (\texttt{src/word\_source/block\_words.c}) into a slot of the
+block-subsystem's own eight-entry devblock cache
+(\texttt{src/block\_subsystem.c}), which the subsystem's own FIFO eviction
+relocates via struct-copy without informing the window cache holding a
+pointer into it. No prior test had ever touched enough distinct blocks in
+one VM session to trigger it, and the defect is independent of unrelated
+same-day work on Artemis's disk-format detection. A three-site fix --
+re-resolving the buffer pointer by logical block number immediately before
+every write-through-cache-pointer operation, rather than trusting a
+pointer captured earlier -- eliminates the defect without altering the
+block subsystem's cache architecture. This is confirmed by the strongest
+result in this report: a two-proportion test of pre-fix versus pooled
+post-fix (three architectures, 150 trials) pass rate rejects the null
+hypothesis of no change at $p = 2.04\times10^{-19}$. Re-verification:
+50/50 on amd64, 50/50 on aarch64, 50/50 on riscv64, zero mismatches. The
+test itself is now a permanent, on-demand fixture in the Artemis capsule
+(\texttt{ART-STRESS} / \texttt{ART-STRESS-TEST}), disabled by default so
+it does not add to ordinary boot time.
+\end{abstract}
+
+\section{Motivation}
+
+Prior to this investigation, every automated exercise of Artemis's
+block-storage path -- \texttt{ART-WRITE-TEST}, \texttt{ART-READ-TEST},
+\texttt{ART-SELF-TEST} -- touched exactly one data block per run. That is
+sufficient to prove the alloc/persist/free lifecycle works in isolation,
+but it never puts real pressure on the underlying caches: the
+block-subsystem's own eight-slot devblock cache
+(\texttt{DISK\_CACHE\_SLOTS} in \texttt{block\_subsystem.c}) and the
+VM-level ``window'' cache used by the \texttt{BLOCK}/\texttt{BUFFER}/
+\texttt{UPDATE} words (\texttt{block\_words.c}) were never both forced to
+evict within a single VM session.
+
+The request behind this work was explicit: give the mounted virtual disk
+a real workout -- real FORTH doing real writes, scattered randomly across
+its entire surface area, on the premise that a test never exercised
+before was the most likely place to find a crack.
+
+\section{Test Design}
+
+\texttt{ART-STRESS-TEST~( seed reps -- )} performs a two-phase write/verify
+cycle against \texttt{reps} distinct, randomly chosen blocks drawn from the
+full 22\,998-block Artemis data pool (logical block numbers 3076--26073):
+
+\begin{enumerate}[nosep]
+  \item \textbf{Pick.} A coprime-stride walk over the index space
+    (stride and starting offset both drawn via \texttt{RANDOM}, stride
+    forced odd and checked against the two odd prime factors of 22\,998 =
+    $2 \times 3 \times 3833$) selects a block index guaranteed not to repeat
+    within the run, skipping past any index already allocated by something
+    else so the test never overwrites pre-existing legitimate data.
+  \item \textbf{Write.} The block is allocated through the same path
+    \texttt{BLK-ALLOC} uses internally (free-map bit set, heat born hot),
+    filled with a trial-derived byte pattern, and persisted.
+  \item \textbf{Verify.} After \emph{all} \texttt{reps} writes complete --
+    forcing real eviction/reload traffic through both cache layers -- each
+    block is re-fetched and its content checked against what was written.
+  \item \textbf{Record.} Every trial, pass or fail, emits one
+    \texttt{[ARTSTRESS]}-tagged CSV row (trial, logical block number,
+    pattern, result) to the serial log, plus a \texttt{HEADER} marker at
+    start and a \texttt{SUMMARY} marker at completion -- if a run is
+    killed mid-way, the summary marker is simply absent, making a
+    truncated dataset unmistakable rather than silently partial.
+  \item \textbf{Clean up.} Every block the run allocated is freed.
+\end{enumerate}
+
+The design deliberately treats this as an offline data-collection fixture,
+the same posture as the existing DoE campaign: correctness of the
+collected data matters more than run speed, and \texttt{ART-K-TOTAL} is
+recorded before and after (informational only -- it is not a pass/fail
+gate, since a background heartbeat decay tick during a long run is
+expected behavior, not a defect).
+
+The first completed run, 50 trials on amd64 with seed 424242 (the fixed
+seed \texttt{ART-STRESS} uses), returned \textbf{25 passes and 25
+failures} -- an exact 50\% split (Figure~\ref{fig:pass-rate}, bottom bar).
+
+\begin{figure}[htb]
+  \centering
+  \includegraphics[width=0.8\linewidth]{pass_rate_comparison.pdf}
+  \caption{Pass rate per run. The pre-fix amd64 run is the only failure;
+  every post-fix run, on all three architectures, is 50/50.}
+  \label{fig:pass-rate}
+\end{figure}
+
+\section{Statistical Analysis}
+
+\textbf{Premise.} A 50\% failure rate on the first completed run of a new
+test is consistent with at least three distinct explanations, which are
+not mutually exclusive: (a) a bug in the test's own block-selection logic
+producing duplicate or colliding picks, (b) a genuine but narrow defect
+tied to a specific access pattern (e.g.\ cache eviction timing), or (c) a
+defect tied to a specific region of the disk. Duplicate logical block
+numbers and duplicate devblocks among the 50 trials were checked directly
+against the recorded data and neither was found, ruling out (a) as stated.
+The hypotheses below were formulated to distinguish (b) from (c) before
+committing to a source-code investigation in either direction.
+
+All tests below use the pre-fix amd64 run's own recorded trial data
+(\texttt{runs/amd64\_buggy.csv}, $n=50$, under
+\texttt{experiments/artemis\_stress/}) and were computed by the analysis
+script \texttt{analyse\_artemis\_stress.R} in that same directory; no
+number in this section is hand-typed.
+
+\begin{table}[htb]
+\centering
+\small
+\begin{tabular}{p{0.28\linewidth}p{0.42\linewidth}p{0.22\linewidth}}
+\toprule
+Hypothesis & Null hypothesis ($H_0$) & Result \\
+\midrule
+H1 -- serial clustering &
+  Pass/fail outcomes are independently ordered (Wald--Wolfowitz runs test) &
+  26 runs observed, 26.0 expected; $z=0.000$, $p=1.00$ -- \textbf{fail to reject} \\
+H1b -- write-order rank &
+  Trial index distribution is the same for PASS and FAIL (Wilcoxon) &
+  $W=332.0$, $p=0.715$ -- \textbf{fail to reject} \\
+H2 -- disk location &
+  LBN distribution is the same for PASS and FAIL (Wilcoxon) &
+  $W=505.0$, $p<0.001$ -- \textbf{reject} \\
+H2b -- sibling position &
+  Outcome independent of position (0/1/2) within a devblock ($\chi^2$) &
+  $\chi^2=0.790$, $\mathrm{df}=2$, $p=0.674$ -- \textbf{fail to reject} \\
+H3 -- fix efficacy &
+  Pre-fix and post-fix trials share the same true pass rate (two-proportion test) &
+  $\chi^2=81.20$, $\mathrm{df}=1$, $p=2.04\times10^{-19}$ -- \textbf{reject} \\
+\bottomrule
+\end{tabular}
+\caption{Hypothesis tests against the recorded trial data. $\alpha = 0.05$
+throughout.}
+\label{tab:hypotheses}
+\end{table}
+
+\textbf{H1 and H1b -- no evidence of write-order clustering.} A visual
+scan of the pass/fail sequence in write order
+(Figure~\ref{fig:sequence}) initially suggested runs of consecutive
+failures -- trials 24, 25, 26 failing together, then 27 and 28 passing,
+then 29, 30, 31 failing again. A formal runs test does not support that
+reading: 26 runs were observed against 26.0 expected under a purely
+random ordering with the same 25/25 split ($z=0.000$). A direct Wilcoxon
+test of trial index by outcome agrees ($p=0.715$). \textbf{The naive
+``recently written survives, long ago fails'' cache-timing story is not
+supported by this data} and should not be asserted as the mechanism.
+
+\begin{figure}[htb]
+  \centering
+  \includegraphics[width=0.95\linewidth]{failure_clustering.pdf}
+  \caption{Pass/fail by trial order, amd64 pre-fix run. Visually
+  suggestive of clustering; not statistically confirmed (H1, H1b).}
+  \label{fig:sequence}
+\end{figure}
+
+\textbf{H2 and H2b -- an unresolved location effect, specific to this
+seed.} Logical block number \emph{does} show a significant association
+with outcome ($p<0.001$): failing trials in this run had a lower mean LBN
+($\bar{x} \approx 11{,}355$) than passing trials ($\bar{x} \approx
+18{,}401$). This was not expected, and this report does not have a causal
+explanation for it -- the diagnosed mechanism (Section~\ref{sec:rootcause})
+is a pointer-aliasing defect tied to cache eviction \emph{order}, and
+this section's H1/H1b results (above) show no association with order in
+this run, which is in tension with a simple location-driven story. The most defensible
+reading is that \texttt{ART-STRESS-TEST}'s seed (424242) is fixed, so the
+specific sequence of devblock accesses -- and therefore which window-cache
+pointers go stale -- is a deterministic, reproducible function of that
+seed for this run, and some property of that one sequence happens to
+correlate with LBN value without implying that block number is, in
+general, a causal factor. Sibling position within a shared devblock shows
+no association ($p=0.674$), which rules out the simplest region-based
+explanation (three data blocks per 4\,KiB devblock, checked directly).
+This finding is reported rather than smoothed over; it does not change
+the fix or its verification, and Figure~\ref{fig:coverage}'s lower panel
+shows no visually obvious sub-region concentration despite the test
+result.
+
+\begin{figure}[htb]
+  \centering
+  \includegraphics[width=0.85\linewidth]{surface_coverage_map.pdf}
+  \caption{Every trial's logical block number against trial order,
+  amd64. Upper panel: post-fix, all passing. Lower panel: pre-fix.}
+  \label{fig:coverage}
+\end{figure}
+
+\textbf{Conclusion of this section.} None of H1, H1b, or H2/H2b, singly
+or together, identify a mechanism -- they narrow the search (ruling out
+write-order clustering and sibling-position effects as the story) without
+answering the question. Root cause was found by source code review,
+described next; H3, in Section~\ref{sec:verification}, is what actually
+confirms the diagnosis, by showing the identified defect's fix removes
+the failures entirely and unambiguously.
+
+\section{Root Cause}
+\label{sec:rootcause}
+
+LithosAnanke's block storage stacks two independent caches:
+
+\begin{enumerate}[nosep]
+  \item \textbf{The block-subsystem's devblock cache}
+    (\texttt{blk\_dev\_slot\_t.cache[DISK\_CACHE\_SLOTS]}, eight entries,
+    \texttt{block\_subsystem.c}). When full and a ninth distinct devblock
+    is needed, \texttt{cache\_get\_slot()} evicts by shifting the entire
+    array down one position via struct assignment:
+\begin{lstlisting}[language=C]
+(void) cache_writeback(slot, &slot->cache[0]);
+for (int i = 0; i < DISK_CACHE_SLOTS - 1; i++)
+    slot->cache[i] = slot->cache[i+1];
+\end{lstlisting}
+    The \emph{addresses} of \texttt{cache[0..7]} never move; what they
+    \emph{hold} does, silently.
+  \item \textbf{The VM-level window cache}
+    (\texttt{vm->blk\_vm\_cbuf[BLK\_VM\_SLOTS]}, \texttt{block\_words.c}),
+    used by \texttt{BLOCK}/\texttt{BUFFER}. When a block loads, it stores
+    the \emph{raw pointer} \texttt{blk\_get\_buffer()} returns --
+    \texttt{c->data + pack * BLK\_FORTH\_SIZE}, pointing directly into one
+    of the eight \texttt{cache\_slot\_t} structs above.
+\end{enumerate}
+
+Nothing informs layer 2 when layer 1 shifts its array. Once more than
+eight distinct devblocks have been touched in one VM session, a
+previously-captured window pointer can point at memory that now belongs
+to a completely different devblock, with no error and no signal --
+just wrong data on the next read or write through it. No prior test ever
+touched enough distinct devblocks in a single session to trigger this;
+the surface stress test's fifty scattered writes was the first thing
+that did. This mechanism does not, on its face, predict a dependency on
+LBN value rather than access pattern, which is consistent with this
+report's inability to causally explain the H2 result above -- the exact
+per-trial trigger condition is a complex function of window-slot reuse
+and devblock-array-shift interleaving that this investigation did not
+instrument further, since it is not required to confirm or fix the
+defect. The defect is independent of, and predates, unrelated same-day
+work on Artemis's disk-format detection (\texttt{BLK-CONFIRM-FORMAT}) --
+none of that work touches \texttt{cache\_get\_slot} or the window-cache
+functions.
+
+\section{Fix}
+
+Three call sites in \texttt{block\_words.c} wrote through the stored
+window pointer before calling \texttt{blk\_update()}: \texttt{block\_word\_update}
+(backing the Forth word \texttt{UPDATE}), \texttt{blk\_vm\_evict}'s
+all-slots-dirty fallback, and \texttt{blk\_vm\_flush\_all} (backing
+\texttt{FLUSH}/\texttt{SAVE-BUFFERS}). Each now re-resolves a
+\emph{guaranteed-current} pointer by logical block number immediately
+before use:
+
+\begin{lstlisting}[language=C]
+uint8_t *fresh = blk_get_buffer((uint32_t) blk, 1);
+if (!fresh) { vm->error = 1; return; }
+memcpy(fresh, vm->memory + base, BLOCK_SIZE);
+vm->blk_vm_cbuf[s] = fresh;   /* keep the window's own copy in sync */
+\end{lstlisting}
+
+\texttt{blk\_get\_buffer()} internally calls \texttt{cache\_load\_devblock()},
+which correctly resolves (or reloads) whichever slot currently holds that
+logical block number, however the array has shifted since the window
+pointer was first captured. The fix is confined entirely to the consumer
+side and does not alter the block subsystem's cache architecture: never
+trust a raw cache pointer across an operation boundary where the producer
+may have invalidated it -- ask for the current one again, right before
+using it.
+
+\section{Verification}
+\label{sec:verification}
+
+The fix was verified by re-running the identical stress test (same seed,
+same 50-trial coprime-stride sequence) on all three supported
+architectures.
+
+\begin{table}[htb]
+\centering
+\begin{tabular}{lrrr}
+\toprule
+Run & Trials & Passed & Failed \\
+\midrule
+amd64 (pre-fix)    & 50 & 25 & 25 \\
+amd64 (post-fix)   & 50 & 50 & 0 \\
+aarch64 (post-fix) & 50 & 50 & 0 \\
+riscv64 (post-fix) & 50 & 50 & 0 \\
+\bottomrule
+\end{tabular}
+\caption{Surface stress test results before and after the fix.}
+\end{table}
+
+Pooling the three post-fix runs against the pre-fix run and testing
+$H_0:$ ``pre-fix and post-fix trials share the same true pass rate''
+with a two-proportion test rejects $H_0$ at
+$p = 2.04\times10^{-19}$ (Table~\ref{tab:hypotheses}, H3; 95\% CI on the
+difference in pass rate: $[-0.652, -0.348]$). This is the definitive
+result in this report: unlike H1/H1b/H2/H2b, which narrow the search
+space without identifying a mechanism, H3 directly confirms that the
+change made in Section 5 -- and only that change -- accounts for the
+difference between a 50\% and a 100\% pass rate, consistently, across
+three independent instruction-set architectures.
+
+All three post-fix architectures also completed the standard acceptance
+boot to \texttt{ok>} with the fix in place, and Hera's dictionary-hash
+parity across amd64 and aarch64 matched exactly for identical capsule
+content, confirming the fix does not disturb cross-architecture
+determinism.
+
+\section{Disposition}
+
+The surface stress test is now a permanent fixture of the Artemis capsule
+(\texttt{capsules/artemis/init.4th}, blocks 4160--4173), invocable on
+demand as \texttt{ART-STRESS} (fifty trials, fixed seed) or
+\texttt{<seed> <reps> ART-STRESS-TEST} for a custom run. It is not wired
+into the automatic boot sequence -- block 4170 carries the entry point
+commented out (\texttt{\textbackslash\ ART-STRESS}), matching the same
+disable-by-default convention already used by \texttt{ACL.4th}'s
+self-activation toggle, so ordinary boots pay no extra time for it. A
+future maintainer can uncomment that one line to run it automatically, or
+invoke it manually from the console at any time. The unresolved H2
+finding (Section 4) is left as an open question for a future
+investigation with additional cache-state instrumentation, not as a
+loose end in the fix itself, which H3 confirms is complete.
+
+\end{document}