426 lines
15 KiB
Markdown
426 lines
15 KiB
Markdown
<!-- Moved from docs/02-experiments/heartbeat-doe/design.md to docs/working/experiments/02-experiments/heartbeat-doe/design.md on 2026-06-16 (docs reorg Phase 2) -->
|
||
# StarForth Factorial DoE with Heartbeat Integration
|
||
|
||
**Date:** November 20, 2025
|
||
**Status:** Design Phase
|
||
**Experiment:** 2^6 Factorial with Heartbeat-Aware Metrics Collection
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
This document describes the **second-generation factorial DoE** that incorporates heartbeat observability into metric collection. Building on the baseline DOE (Stage 1: 3200 runs, 6 factors, no heartbeat), this experiment captures **per-tick heartbeat dynamics** to identify configurations with optimal stability characteristics.
|
||
|
||
### Key Innovation
|
||
|
||
Instead of measuring static performance, this experiment measures **temporal stability**:
|
||
- How quickly does the system converge under load changes?
|
||
- How much does heartbeat interval modulate in response to workload?
|
||
- What is the jitter envelope after settling?
|
||
- Which configuration achieves smoothest load→heartbeat coupling?
|
||
|
||
---
|
||
|
||
## Design Points (Top 5 Configurations from Stage 1)
|
||
|
||
Based on Stage 1 analysis, the elite configurations are:
|
||
|
||
1. **Config 1:** `1_0_1_1_1_0` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=OFF)
|
||
2. **Config 2:** `1_0_1_1_1_1` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=ON)
|
||
3. **Config 3:** `1_1_0_1_1_1` (F1=ON, F2=ON, F3=OFF, F4=ON, F5=ON, F6=ON)
|
||
4. **Config 4:** `1_0_1_0_1_0` (F1=ON, F2=OFF, F3=ON, F4=OFF, F5=ON, F6=OFF)
|
||
5. **Config 5:** `0_1_1_0_1_1` (F1=OFF, F2=ON, F3=ON, F4=OFF, F5=ON, F6=ON)
|
||
|
||
**Factors Mapping:**
|
||
- F1 = ENABLE_LOOP_1_HEAT_TRACKING
|
||
- F2 = ENABLE_LOOP_2_ROLLING_WINDOW
|
||
- F3 = ENABLE_LOOP_3_LINEAR_DECAY
|
||
- F4 = ENABLE_LOOP_4_PIPELINING_METRICS
|
||
- F5 = ENABLE_LOOP_5_WINDOW_INFERENCE
|
||
- F6 = ENABLE_LOOP_6_DECAY_INFERENCE
|
||
|
||
---
|
||
|
||
## New Heartbeat Metrics (Per-Tick Collection)
|
||
|
||
### 1. Heartbeat Interval Dynamics
|
||
|
||
```c
|
||
struct HeartbeatTickMetrics {
|
||
uint64_t tick_count; /* Absolute tick number */
|
||
uint64_t tick_interval_ns; /* Wall-clock duration of this tick */
|
||
uint64_t instantaneous_workload_ops; /* Dictionary lookups in this tick */
|
||
double instantaneous_load_ratio; /* (ops this tick) / (avg ops/tick) */
|
||
};
|
||
```
|
||
|
||
**Purpose:** Capture how heartbeat frequency modulates in response to workload intensity.
|
||
|
||
### 2. Cache Performance (Per-Tick Window)
|
||
|
||
```c
|
||
struct CacheWindowMetrics {
|
||
uint64_t tick_start;
|
||
uint64_t cache_hits_window; /* Hits in last 100 ticks */
|
||
uint64_t cache_attempts_window; /* Attempts in last 100 ticks */
|
||
double cache_hit_percent_window; /* Rolling window %, 0-100 */
|
||
};
|
||
```
|
||
|
||
**Purpose:** Measure cache stability, not just final hit rate.
|
||
|
||
### 3. Rolling Window Diagnostics
|
||
|
||
```c
|
||
struct WindowDiagnostics {
|
||
uint32_t effective_window_size; /* Current adaptive size */
|
||
uint64_t pattern_diversity; /* Unique word IDs observed */
|
||
double diversity_growth_rate; /* % new patterns per 1000 executions */
|
||
int is_warm; /* 1 if window is representative */
|
||
};
|
||
```
|
||
|
||
**Purpose:** Track window convergence and saturation.
|
||
|
||
### 4. Jitter Envelope
|
||
|
||
```c
|
||
struct JitterMetrics {
|
||
double tick_interval_mean_ns; /* Average heartbeat interval */
|
||
double tick_interval_stddev_ns; /* StdDev of interval */
|
||
double tick_interval_cv; /* Coefficient of variation (σ/μ) */
|
||
uint64_t outlier_count; /* Ticks >3σ from mean */
|
||
double outlier_ratio; /* Outliers / total ticks */
|
||
};
|
||
```
|
||
|
||
**Purpose:** Measure temporal smoothness and jitter control.
|
||
|
||
### 5. Decay Slope Tracking
|
||
|
||
```c
|
||
struct DecayTrajectory {
|
||
uint64_t tick_sample_count; /* How many ticks sampled */
|
||
double decay_slope_observed; /* Fitted slope from observations */
|
||
double slope_convergence_rate; /* Speed of slope convergence */
|
||
int slope_direction; /* -1=decreasing, 0=stable, +1=increasing */
|
||
};
|
||
```
|
||
|
||
**Purpose:** Monitor whether decay slope inference is tracking correctly.
|
||
|
||
### 6. Load-Response Coupling
|
||
|
||
```c
|
||
struct LoadResponseCoupling {
|
||
double correlation_load_to_interval; /* Correlation(workload, heartbeat_interval) */
|
||
double response_latency_ticks; /* Ticks until interval reacts to load change */
|
||
double overshoot_ratio; /* Peak response / steady-state response */
|
||
double settling_time_ticks; /* Ticks to settle within 10% of target */
|
||
};
|
||
```
|
||
|
||
**Purpose:** Measure how quickly and smoothly heartbeat responds to workload shifts.
|
||
|
||
---
|
||
|
||
## Extended DoeMetrics Struct
|
||
|
||
Add to `include/doe_metrics.h`:
|
||
|
||
```c
|
||
typedef struct {
|
||
/* === Original Metrics (preserved) === */
|
||
/* ... existing 35+ metrics ... */
|
||
|
||
/* === NEW: Heartbeat Stability Metrics === */
|
||
|
||
/* Tick dynamics */
|
||
uint64_t total_heartbeat_ticks; /* Total ticks during run */
|
||
double tick_interval_mean_ns; /* Average heartbeat interval */
|
||
double tick_interval_stddev_ns; /* StdDev of heartbeat interval */
|
||
double tick_interval_cv; /* Coefficient of variation */
|
||
uint64_t tick_outlier_count_3sigma; /* Ticks >3σ from mean */
|
||
double tick_outlier_ratio; /* Outliers / total ticks */
|
||
|
||
/* Cache stability window */
|
||
double cache_hit_percent_rolling_mean; /* Rolling avg, last 100 ticks */
|
||
double cache_hit_percent_rolling_stddev; /* Rolling stddev */
|
||
double cache_stability_index; /* How stable is cache hit %? */
|
||
|
||
/* Window convergence */
|
||
uint32_t window_final_effective_size; /* Final adaptive window size */
|
||
double pattern_diversity_final; /* Final unique patterns observed */
|
||
double diversity_growth_final; /* Growth rate at end */
|
||
int window_warmth_achieved; /* 1 = warm, 0 = still cold */
|
||
|
||
/* Decay slope convergence */
|
||
double decay_slope_fitted_final; /* Final fitted decay slope */
|
||
double decay_slope_convergence_rate; /* How fast did slope converge? */
|
||
int decay_slope_stable; /* 1 = stable direction, 0 = oscillating */
|
||
|
||
/* Load-response */
|
||
double load_interval_correlation; /* Correlation(load, heartbeat_interval) */
|
||
double response_latency_ticks; /* Ticks until response to workload change */
|
||
double overshoot_ratio; /* Peak response / steady-state */
|
||
double settling_time_ticks; /* Time to 10% settling band */
|
||
|
||
/* Aggregate stability score */
|
||
double overall_stability_score; /* Composite metric: (low jitter + fast convergence + tight correlation) */
|
||
|
||
} DoeMetrics; /* Now extends to ~65+ columns */
|
||
```
|
||
|
||
---
|
||
|
||
## Heartbeat Data Collection Architecture
|
||
|
||
### Per-Tick Logging (Lightweight)
|
||
|
||
During VM execution, capture minimal per-tick data:
|
||
|
||
```c
|
||
/* In vm.c vm_tick_heartbeat_worker() */
|
||
void vm_tick_heartbeat_worker(VM *vm) {
|
||
HeartbeatSnapshot snapshot = {
|
||
.published_tick = vm->heartbeat.tick_count,
|
||
.published_ns = platform_monotonic_ns(),
|
||
.window_width = vm->rolling_window.effective_window_size,
|
||
.decay_slope_q48 = vm->decay_slope_q48,
|
||
.hot_word_count = vm->hot_word_count_at_check,
|
||
.stale_word_count = vm->stale_word_count_at_check,
|
||
.total_heat = vm->total_heat_at_last_check,
|
||
};
|
||
|
||
/* Publish snapshot (double-buffered, lock-free read) */
|
||
int next_slot = 1 - vm->heartbeat.snapshot_index;
|
||
vm->heartbeat.snapshots[next_slot] = snapshot;
|
||
vm->heartbeat.snapshot_index = next_slot;
|
||
}
|
||
```
|
||
|
||
### Post-Run Analysis (Batch Processing)
|
||
|
||
After test execution completes, analyze collected tick data:
|
||
|
||
```c
|
||
/* In src/doe_metrics.c metrics_from_vm() */
|
||
void analyze_heartbeat_trajectory(VM *vm, DoeMetrics *metrics) {
|
||
HeartbeatSnapshot *snapshots = vm->heartbeat.snapshots;
|
||
uint64_t total_ticks = vm->heartbeat.tick_count;
|
||
|
||
/* 1. Compute tick interval statistics */
|
||
double intervals[total_ticks];
|
||
for (size_t i = 1; i < total_ticks; i++) {
|
||
intervals[i] = snapshots[i].published_ns - snapshots[i-1].published_ns;
|
||
}
|
||
metrics->tick_interval_mean_ns = compute_mean(intervals, total_ticks);
|
||
metrics->tick_interval_stddev_ns = compute_stddev(intervals, total_ticks);
|
||
metrics->tick_interval_cv = metrics->tick_interval_stddev_ns / metrics->tick_interval_mean_ns;
|
||
|
||
/* 2. Compute outlier ratio (ticks >3σ from mean) */
|
||
double threshold_3sigma = metrics->tick_interval_mean_ns + 3.0 * metrics->tick_interval_stddev_ns;
|
||
uint64_t outliers = 0;
|
||
for (size_t i = 0; i < total_ticks; i++) {
|
||
if (intervals[i] > threshold_3sigma) outliers++;
|
||
}
|
||
metrics->tick_outlier_ratio = (double)outliers / total_ticks;
|
||
|
||
/* 3. Compute decay slope convergence */
|
||
double slopes[total_ticks];
|
||
for (size_t i = 100; i < total_ticks; i++) { /* Skip warm-up */
|
||
slopes[i] = snapshots[i].decay_slope_q48 / 65536.0;
|
||
}
|
||
metrics->decay_slope_fitted_final = slopes[total_ticks - 1];
|
||
metrics->decay_slope_convergence_rate = compute_convergence_rate(slopes, total_ticks);
|
||
|
||
/* 4. Compute overall stability score */
|
||
double stability = 0.0;
|
||
stability += (100.0 - metrics->tick_interval_cv); /* Lower CV = higher score */
|
||
stability += (100.0 - metrics->tick_outlier_ratio * 100.0); /* Fewer outliers = higher */
|
||
stability += metrics->decay_slope_convergence_rate; /* Fast convergence = higher */
|
||
metrics->overall_stability_score = stability / 3.0;
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## CSV Output Schema (Extended)
|
||
|
||
**New columns (per heartbeat experiment run):**
|
||
|
||
```
|
||
timestamp,configuration,run_number,
|
||
[... original 38 columns ...],
|
||
total_heartbeat_ticks,
|
||
tick_interval_mean_ns,tick_interval_stddev_ns,tick_interval_cv,
|
||
tick_outlier_count_3sigma,tick_outlier_ratio,
|
||
cache_hit_percent_rolling_mean,cache_hit_percent_rolling_stddev,cache_stability_index,
|
||
window_final_effective_size,pattern_diversity_final,diversity_growth_final,window_warmth_achieved,
|
||
decay_slope_fitted_final,decay_slope_convergence_rate,decay_slope_stable,
|
||
load_interval_correlation,response_latency_ticks,overshoot_ratio,settling_time_ticks,
|
||
overall_stability_score
|
||
```
|
||
|
||
**Total:** ~60 columns per run
|
||
|
||
---
|
||
|
||
## R Analysis Template
|
||
|
||
```r
|
||
library(tidyverse)
|
||
library(ggplot2)
|
||
library(gridExtra)
|
||
|
||
# Load heartbeat DoE data
|
||
doe_data <- read.csv("experiment_results_with_heartbeat.csv") %>%
|
||
mutate(
|
||
configuration = as.factor(configuration),
|
||
timestamp = as.POSIXct(timestamp)
|
||
)
|
||
|
||
# ========== MAIN ANALYSIS ==========
|
||
|
||
# 1. Stability Ranking
|
||
stability_rank <- doe_data %>%
|
||
group_by(configuration) %>%
|
||
summarize(
|
||
mean_stability_score = mean(overall_stability_score),
|
||
mean_jitter_cv = mean(tick_interval_cv),
|
||
mean_convergence_rate = mean(decay_slope_convergence_rate),
|
||
mean_settling_time = mean(settling_time_ticks),
|
||
n_runs = n()
|
||
) %>%
|
||
mutate(
|
||
rank = rank(-mean_stability_score, ties.method = "min")
|
||
) %>%
|
||
arrange(rank)
|
||
|
||
print("=== STABILITY RANKINGS ===")
|
||
print(stability_rank)
|
||
|
||
# 2. Visualization: Stability Scores
|
||
ggplot(doe_data, aes(x = reorder(configuration, overall_stability_score, FUN=median),
|
||
y = overall_stability_score,
|
||
fill = configuration)) +
|
||
geom_boxplot(alpha = 0.7) +
|
||
geom_jitter(width = 0.2, alpha = 0.4) +
|
||
coord_flip() +
|
||
theme_minimal() +
|
||
labs(title = "Overall Stability Scores by Configuration",
|
||
x = "Configuration",
|
||
y = "Stability Score (0-100)")
|
||
ggsave("01_stability_scores.png", width = 10, height = 6)
|
||
|
||
# 3. Visualization: Jitter Control (CV)
|
||
ggplot(doe_data, aes(x = reorder(configuration, tick_interval_cv, FUN=median),
|
||
y = tick_interval_cv,
|
||
fill = configuration)) +
|
||
geom_boxplot(alpha = 0.7) +
|
||
coord_flip() +
|
||
theme_minimal() +
|
||
labs(title = "Heartbeat Jitter (Coefficient of Variation)",
|
||
x = "Configuration",
|
||
y = "CV (lower = less jitter)")
|
||
ggsave("02_jitter_control.png", width = 10, height = 6)
|
||
|
||
# 4. Visualization: Convergence Speed
|
||
ggplot(doe_data, aes(x = reorder(configuration, decay_slope_convergence_rate, FUN=median),
|
||
y = decay_slope_convergence_rate,
|
||
fill = configuration)) +
|
||
geom_boxplot(alpha = 0.7) +
|
||
coord_flip() +
|
||
theme_minimal() +
|
||
labs(title = "Decay Slope Convergence Speed",
|
||
x = "Configuration",
|
||
y = "Convergence Rate (ticks)")
|
||
ggsave("03_convergence_speed.png", width = 10, height = 6)
|
||
|
||
# 5. Correlation: Load Response Coupling
|
||
ggplot(doe_data, aes(x = load_interval_correlation,
|
||
y = overall_stability_score,
|
||
color = configuration)) +
|
||
geom_point(alpha = 0.6, size = 2) +
|
||
facet_wrap(~configuration) +
|
||
theme_minimal() +
|
||
labs(title = "Load-Response Coupling vs Stability",
|
||
x = "Correlation(load, interval)",
|
||
y = "Overall Stability Score")
|
||
ggsave("04_load_response_coupling.png", width = 12, height = 8)
|
||
|
||
# 6. Statistical Test: Is top config significantly better?
|
||
top_config <- stability_rank$configuration[1]
|
||
other_configs <- stability_rank$configuration[-1]
|
||
|
||
top_data <- doe_data %>% filter(configuration == top_config) %>% pull(overall_stability_score)
|
||
other_data <- doe_data %>% filter(configuration %in% other_configs) %>% pull(overall_stability_score)
|
||
|
||
t_test <- t.test(top_data, other_data)
|
||
print(sprintf("\nT-test: %s vs others\np-value: %.4f", top_config, t_test$p.value))
|
||
|
||
# 7. Golden Configuration Summary
|
||
cat("\n=== GOLDEN CONFIGURATION CANDIDATE ===\n")
|
||
golden <- stability_rank %>% slice(1)
|
||
print(golden)
|
||
|
||
cat("\nInterpretation:")
|
||
cat(sprintf("\n- Lowest jitter (CV): %.4f", golden$mean_jitter_cv))
|
||
cat(sprintf("\n- Fastest convergence: %.1f ticks", golden$mean_convergence_rate))
|
||
cat(sprintf("\n- Settling time: %.1f ticks", golden$mean_settling_time))
|
||
cat(sprintf("\n- Recommendation: Use '%s' as golden configuration\n", golden$configuration))
|
||
```
|
||
|
||
---
|
||
|
||
## Execution Plan
|
||
|
||
### Phase 1: Implementation (3-4 hours)
|
||
1. Extend `DoeMetrics` struct with heartbeat columns
|
||
2. Implement `analyze_heartbeat_trajectory()` in `doe_metrics.c`
|
||
3. Capture snapshot data during heartbeat ticks
|
||
4. Update CSV schema
|
||
|
||
### Phase 2: Experiment Execution (2-4 hours)
|
||
1. Create `run_factorial_doe_with_heartbeat.sh` script
|
||
2. Run 5 elite configurations × 50 runs each = 250 total runs
|
||
3. Collect heartbeat + performance data
|
||
|
||
### Phase 3: Analysis (2-3 hours)
|
||
1. Load CSV in R
|
||
2. Compute stability rankings
|
||
3. Generate visualizations
|
||
4. Identify golden configuration
|
||
|
||
---
|
||
|
||
## Expected Outcomes
|
||
|
||
**Primary Question:** Which of the 5 elite configurations exhibits:
|
||
- Lowest heartbeat jitter (CV)?
|
||
- Fastest convergence to steady state?
|
||
- Strongest load-response correlation?
|
||
- Cleanest temporal signature?
|
||
|
||
**Expected Winner:** Configuration with balanced F-factors, likely:
|
||
- F1=1 (heat tracking required)
|
||
- F3=1 or F5=1 (decay or inference, but not both)
|
||
- F6=0 or F6=1 (decay inference optional)
|
||
|
||
**Golden Configuration:** Becomes baseline physics for MamaForth & PapaForth downstream work.
|
||
|
||
---
|
||
|
||
## Success Criteria
|
||
|
||
✓ All 5 configurations build and run without segfault
|
||
✓ Heartbeat data collected for all 250 runs
|
||
✓ Stability rankings computed and visualized
|
||
✓ Golden configuration identified with p < 0.05 significance
|
||
✓ R analysis report generated with 5+ plots
|
||
|
||
---
|
||
|
||
**Version:** 1.0
|
||
**Status:** Ready for implementation
|
||
**Next Step:** Implement heartbeat metrics capture and extended DoE script |