Add Artemis stress test for detecting cache aliasing bugs. Include statistical hypothesis evaluations, fix validation data, and run reports for validation across architectures.

Signed-off-by: Robert Allan James <robert.allan.james@gmail.com>
2026-08-02 14:49:52 -04:00
parent 148c4aa12c
commit 1cb68502fb
44 changed files with 5610173 additions and 2680 deletions
@@ -0,0 +1,20 @@
+"key","value"
+"h1_runs_obs",26
+"h1_runs_exp",26
+"h1_z",0
+"h1_p",1
+"h1b_W",332
+"h1b_p",0.714773975516392
+"h2_W",505
+"h2_p",0.000107733285139378
+"h2b_chisq",0.790350877192982
+"h2b_df",2
+"h2b_p",0.67356186013519
+"h3_pre_n",50
+"h3_pre_pass",25
+"h3_post_n",150
+"h3_post_pass",150
+"h3_chisq",81.2038095238095
+"h3_p",2.03599124928093e-19
+"h3_ci_lo",-0.651923715768301
+"h3_ci_hi",-0.348076284231699
@@ -0,0 +1,32 @@
+Artemis surface stress test -- hypothesis tests
+=================================================
+
+H1 -- pre-fix failure clustering (Wald-Wolfowitz runs test)
+  H0: pass/fail outcomes are independently ordered (no clustering).
+  Observed runs = 26 (expected under H0 = 26.00, sd = 3.50)
+  z = 0.000, p = 1.00e+00
+  -> fail to reject H0 at alpha=0.05: failures are not serially clustered.
+
+H1b -- pre-fix outcome vs. trial/write order directly (Wilcoxon)
+  H0: trial index is drawn from the same distribution for PASS/FAIL
+      (complements H1: the runs test only catches consecutive-value
+      clustering, not a general rank-order trend).
+  W = 332.0, p = 0.715
+  -> fail to reject H0 at alpha=0.05: outcome is not associated with write order.
+
+H2 -- pre-fix failure vs. disk location (Wilcoxon rank-sum test)
+  H0: logical block number is drawn from the same distribution
+      for PASS and FAIL trials (failure does not depend on location).
+  W = 505.0, p = 0.000
+  -> REJECT H0 at alpha=0.05: failure IS associated with LBN.
+
+  Secondary check -- sibling position (0/1/2) vs. outcome (chi-squared)
+  chi-sq = 0.790, df = 2, p = 0.674
+  -> fail to reject H0: outcome is not associated with sibling position.
+
+H3 -- did the fix change the pass rate (two-proportion test)
+  H0: pre-fix and post-fix trials share the same true pass rate.
+  Pre-fix: 25/50 = 50.0%   Post-fix (pooled, 3 arches): 150/150 = 100.0%
+  chi-sq = 81.20, df = 1, p = 2.04e-19
+  95% CI on the difference in pass rate: [-0.652, -0.348]
+  -> REJECT H0 at alpha=0.05: the fix DID CHANGE the pass rate.