LithosAnanake/docs/working/experiments/02-experiments/heartbeat-doe/design.md

StarForth Factorial DoE with Heartbeat Integration

Date: November 20, 2025 Status: Design Phase Experiment: 2^6 Factorial with Heartbeat-Aware Metrics Collection

---

Overview

This document describes the second-generation factorial DoE that incorporates heartbeat observability into metric collection. Building on the baseline DOE (Stage 1: 3200 runs, 6 factors, no heartbeat), this experiment captures per-tick heartbeat dynamics to identify configurations with optimal stability characteristics.

Key Innovation

Instead of measuring static performance, this experiment measures temporal stability:

• How quickly does the system converge under load changes?
• How much does heartbeat interval modulate in response to workload?
• What is the jitter envelope after settling?
• Which configuration achieves smoothest load→heartbeat coupling?

---

Design Points (Top 5 Configurations from Stage 1)

Based on Stage 1 analysis, the elite configurations are:

1. Config 1: 1_0_1_1_1_0 (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=OFF)
2. Config 2: 1_0_1_1_1_1 (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=ON)
3. Config 3: 1_1_0_1_1_1 (F1=ON, F2=ON, F3=OFF, F4=ON, F5=ON, F6=ON)
4. Config 4: 1_0_1_0_1_0 (F1=ON, F2=OFF, F3=ON, F4=OFF, F5=ON, F6=OFF)
5. Config 5: 0_1_1_0_1_1 (F1=OFF, F2=ON, F3=ON, F4=OFF, F5=ON, F6=ON)

Factors Mapping:

• F1 = ENABLE_LOOP_1_HEAT_TRACKING
• F2 = ENABLE_LOOP_2_ROLLING_WINDOW
• F3 = ENABLE_LOOP_3_LINEAR_DECAY
• F4 = ENABLE_LOOP_4_PIPELINING_METRICS
• F5 = ENABLE_LOOP_5_WINDOW_INFERENCE
• F6 = ENABLE_LOOP_6_DECAY_INFERENCE

---

New Heartbeat Metrics (Per-Tick Collection)

1. Heartbeat Interval Dynamics

struct HeartbeatTickMetrics {
    uint64_t tick_count;                    /* Absolute tick number */
    uint64_t tick_interval_ns;              /* Wall-clock duration of this tick */
    uint64_t instantaneous_workload_ops;    /* Dictionary lookups in this tick */
    double   instantaneous_load_ratio;      /* (ops this tick) / (avg ops/tick) */
};

Purpose: Capture how heartbeat frequency modulates in response to workload intensity.

2. Cache Performance (Per-Tick Window)

struct CacheWindowMetrics {
    uint64_t tick_start;
    uint64_t cache_hits_window;             /* Hits in last 100 ticks */
    uint64_t cache_attempts_window;         /* Attempts in last 100 ticks */
    double   cache_hit_percent_window;      /* Rolling window %, 0-100 */
};

Purpose: Measure cache stability, not just final hit rate.

3. Rolling Window Diagnostics

struct WindowDiagnostics {
    uint32_t effective_window_size;         /* Current adaptive size */
    uint64_t pattern_diversity;             /* Unique word IDs observed */
    double   diversity_growth_rate;         /* % new patterns per 1000 executions */
    int      is_warm;                       /* 1 if window is representative */
};

Purpose: Track window convergence and saturation.

4. Jitter Envelope

struct JitterMetrics {
    double   tick_interval_mean_ns;         /* Average heartbeat interval */
    double   tick_interval_stddev_ns;       /* StdDev of interval */
    double   tick_interval_cv;              /* Coefficient of variation (σ/μ) */
    uint64_t outlier_count;                 /* Ticks >3σ from mean */
    double   outlier_ratio;                 /* Outliers / total ticks */
};

Purpose: Measure temporal smoothness and jitter control.

5. Decay Slope Tracking

struct DecayTrajectory {
    uint64_t tick_sample_count;             /* How many ticks sampled */
    double   decay_slope_observed;          /* Fitted slope from observations */
    double   slope_convergence_rate;        /* Speed of slope convergence */
    int      slope_direction;               /* -1=decreasing, 0=stable, +1=increasing */
};

Purpose: Monitor whether decay slope inference is tracking correctly.

6. Load-Response Coupling

struct LoadResponseCoupling {
    double   correlation_load_to_interval;  /* Correlation(workload, heartbeat_interval) */
    double   response_latency_ticks;        /* Ticks until interval reacts to load change */
    double   overshoot_ratio;               /* Peak response / steady-state response */
    double   settling_time_ticks;           /* Ticks to settle within 10% of target */
};

Purpose: Measure how quickly and smoothly heartbeat responds to workload shifts.

---

Extended DoeMetrics Struct

Add to include/doe_metrics.h:

typedef struct {
    /* === Original Metrics (preserved) === */
    /* ... existing 35+ metrics ... */

    /* === NEW: Heartbeat Stability Metrics === */

    /* Tick dynamics */
    uint64_t total_heartbeat_ticks;         /* Total ticks during run */
    double   tick_interval_mean_ns;         /* Average heartbeat interval */
    double   tick_interval_stddev_ns;       /* StdDev of heartbeat interval */
    double   tick_interval_cv;              /* Coefficient of variation */
    uint64_t tick_outlier_count_3sigma;     /* Ticks >3σ from mean */
    double   tick_outlier_ratio;            /* Outliers / total ticks */

    /* Cache stability window */
    double   cache_hit_percent_rolling_mean;     /* Rolling avg, last 100 ticks */
    double   cache_hit_percent_rolling_stddev;   /* Rolling stddev */
    double   cache_stability_index;              /* How stable is cache hit %? */

    /* Window convergence */
    uint32_t window_final_effective_size;   /* Final adaptive window size */
    double   pattern_diversity_final;       /* Final unique patterns observed */
    double   diversity_growth_final;        /* Growth rate at end */
    int      window_warmth_achieved;        /* 1 = warm, 0 = still cold */

    /* Decay slope convergence */
    double   decay_slope_fitted_final;      /* Final fitted decay slope */
    double   decay_slope_convergence_rate;  /* How fast did slope converge? */
    int      decay_slope_stable;            /* 1 = stable direction, 0 = oscillating */

    /* Load-response */
    double   load_interval_correlation;     /* Correlation(load, heartbeat_interval) */
    double   response_latency_ticks;        /* Ticks until response to workload change */
    double   overshoot_ratio;               /* Peak response / steady-state */
    double   settling_time_ticks;           /* Time to 10% settling band */

    /* Aggregate stability score */
    double   overall_stability_score;       /* Composite metric: (low jitter + fast convergence + tight correlation) */

} DoeMetrics;  /* Now extends to ~65+ columns */

---

Heartbeat Data Collection Architecture

Per-Tick Logging (Lightweight)

During VM execution, capture minimal per-tick data:

/* In vm.c vm_tick_heartbeat_worker() */
void vm_tick_heartbeat_worker(VM *vm) {
    HeartbeatSnapshot snapshot = {
        .published_tick = vm->heartbeat.tick_count,
        .published_ns = platform_monotonic_ns(),
        .window_width = vm->rolling_window.effective_window_size,
        .decay_slope_q48 = vm->decay_slope_q48,
        .hot_word_count = vm->hot_word_count_at_check,
        .stale_word_count = vm->stale_word_count_at_check,
        .total_heat = vm->total_heat_at_last_check,
    };

    /* Publish snapshot (double-buffered, lock-free read) */
    int next_slot = 1 - vm->heartbeat.snapshot_index;
    vm->heartbeat.snapshots[next_slot] = snapshot;
    vm->heartbeat.snapshot_index = next_slot;
}

Post-Run Analysis (Batch Processing)

After test execution completes, analyze collected tick data:

/* In src/doe_metrics.c metrics_from_vm() */
void analyze_heartbeat_trajectory(VM *vm, DoeMetrics *metrics) {
    HeartbeatSnapshot *snapshots = vm->heartbeat.snapshots;
    uint64_t total_ticks = vm->heartbeat.tick_count;

    /* 1. Compute tick interval statistics */
    double intervals[total_ticks];
    for (size_t i = 1; i < total_ticks; i++) {
        intervals[i] = snapshots[i].published_ns - snapshots[i-1].published_ns;
    }
    metrics->tick_interval_mean_ns = compute_mean(intervals, total_ticks);
    metrics->tick_interval_stddev_ns = compute_stddev(intervals, total_ticks);
    metrics->tick_interval_cv = metrics->tick_interval_stddev_ns / metrics->tick_interval_mean_ns;

    /* 2. Compute outlier ratio (ticks >3σ from mean) */
    double threshold_3sigma = metrics->tick_interval_mean_ns + 3.0 * metrics->tick_interval_stddev_ns;
    uint64_t outliers = 0;
    for (size_t i = 0; i < total_ticks; i++) {
        if (intervals[i] > threshold_3sigma) outliers++;
    }
    metrics->tick_outlier_ratio = (double)outliers / total_ticks;

    /* 3. Compute decay slope convergence */
    double slopes[total_ticks];
    for (size_t i = 100; i < total_ticks; i++) {  /* Skip warm-up */
        slopes[i] = snapshots[i].decay_slope_q48 / 65536.0;
    }
    metrics->decay_slope_fitted_final = slopes[total_ticks - 1];
    metrics->decay_slope_convergence_rate = compute_convergence_rate(slopes, total_ticks);

    /* 4. Compute overall stability score */
    double stability = 0.0;
    stability += (100.0 - metrics->tick_interval_cv);           /* Lower CV = higher score */
    stability += (100.0 - metrics->tick_outlier_ratio * 100.0); /* Fewer outliers = higher */
    stability += metrics->decay_slope_convergence_rate;         /* Fast convergence = higher */
    metrics->overall_stability_score = stability / 3.0;
}

---

CSV Output Schema (Extended)

New columns (per heartbeat experiment run):

timestamp,configuration,run_number,
[... original 38 columns ...],
total_heartbeat_ticks,
tick_interval_mean_ns,tick_interval_stddev_ns,tick_interval_cv,
tick_outlier_count_3sigma,tick_outlier_ratio,
cache_hit_percent_rolling_mean,cache_hit_percent_rolling_stddev,cache_stability_index,
window_final_effective_size,pattern_diversity_final,diversity_growth_final,window_warmth_achieved,
decay_slope_fitted_final,decay_slope_convergence_rate,decay_slope_stable,
load_interval_correlation,response_latency_ticks,overshoot_ratio,settling_time_ticks,
overall_stability_score

Total: ~60 columns per run

---

R Analysis Template

library(tidyverse)
library(ggplot2)
library(gridExtra)

# Load heartbeat DoE data
doe_data <- read.csv("experiment_results_with_heartbeat.csv") %>%
  mutate(
    configuration = as.factor(configuration),
    timestamp = as.POSIXct(timestamp)
  )

# ========== MAIN ANALYSIS ==========

# 1. Stability Ranking
stability_rank <- doe_data %>%
  group_by(configuration) %>%
  summarize(
    mean_stability_score = mean(overall_stability_score),
    mean_jitter_cv = mean(tick_interval_cv),
    mean_convergence_rate = mean(decay_slope_convergence_rate),
    mean_settling_time = mean(settling_time_ticks),
    n_runs = n()
  ) %>%
  mutate(
    rank = rank(-mean_stability_score, ties.method = "min")
  ) %>%
  arrange(rank)

print("=== STABILITY RANKINGS ===")
print(stability_rank)

# 2. Visualization: Stability Scores
ggplot(doe_data, aes(x = reorder(configuration, overall_stability_score, FUN=median),
                      y = overall_stability_score,
                      fill = configuration)) +
  geom_boxplot(alpha = 0.7) +
  geom_jitter(width = 0.2, alpha = 0.4) +
  coord_flip() +
  theme_minimal() +
  labs(title = "Overall Stability Scores by Configuration",
       x = "Configuration",
       y = "Stability Score (0-100)")
ggsave("01_stability_scores.png", width = 10, height = 6)

# 3. Visualization: Jitter Control (CV)
ggplot(doe_data, aes(x = reorder(configuration, tick_interval_cv, FUN=median),
                      y = tick_interval_cv,
                      fill = configuration)) +
  geom_boxplot(alpha = 0.7) +
  coord_flip() +
  theme_minimal() +
  labs(title = "Heartbeat Jitter (Coefficient of Variation)",
       x = "Configuration",
       y = "CV (lower = less jitter)")
ggsave("02_jitter_control.png", width = 10, height = 6)

# 4. Visualization: Convergence Speed
ggplot(doe_data, aes(x = reorder(configuration, decay_slope_convergence_rate, FUN=median),
                      y = decay_slope_convergence_rate,
                      fill = configuration)) +
  geom_boxplot(alpha = 0.7) +
  coord_flip() +
  theme_minimal() +
  labs(title = "Decay Slope Convergence Speed",
       x = "Configuration",
       y = "Convergence Rate (ticks)")
ggsave("03_convergence_speed.png", width = 10, height = 6)

# 5. Correlation: Load Response Coupling
ggplot(doe_data, aes(x = load_interval_correlation,
                      y = overall_stability_score,
                      color = configuration)) +
  geom_point(alpha = 0.6, size = 2) +
  facet_wrap(~configuration) +
  theme_minimal() +
  labs(title = "Load-Response Coupling vs Stability",
       x = "Correlation(load, interval)",
       y = "Overall Stability Score")
ggsave("04_load_response_coupling.png", width = 12, height = 8)

# 6. Statistical Test: Is top config significantly better?
top_config <- stability_rank$configuration[1]
other_configs <- stability_rank$configuration[-1]

top_data <- doe_data %>% filter(configuration == top_config) %>% pull(overall_stability_score)
other_data <- doe_data %>% filter(configuration %in% other_configs) %>% pull(overall_stability_score)

t_test <- t.test(top_data, other_data)
print(sprintf("\nT-test: %s vs others\np-value: %.4f", top_config, t_test$p.value))

# 7. Golden Configuration Summary
cat("\n=== GOLDEN CONFIGURATION CANDIDATE ===\n")
golden <- stability_rank %>% slice(1)
print(golden)

cat("\nInterpretation:")
cat(sprintf("\n- Lowest jitter (CV): %.4f", golden$mean_jitter_cv))
cat(sprintf("\n- Fastest convergence: %.1f ticks", golden$mean_convergence_rate))
cat(sprintf("\n- Settling time: %.1f ticks", golden$mean_settling_time))
cat(sprintf("\n- Recommendation: Use '%s' as golden configuration\n", golden$configuration))

---

Execution Plan

Phase 1: Implementation (3-4 hours)

1. Extend DoeMetrics struct with heartbeat columns
2. Implement analyze_heartbeat_trajectory() in doe_metrics.c
3. Capture snapshot data during heartbeat ticks
4. Update CSV schema

Phase 2: Experiment Execution (2-4 hours)

1. Create run_factorial_doe_with_heartbeat.sh script
2. Run 5 elite configurations × 50 runs each = 250 total runs
3. Collect heartbeat + performance data

Phase 3: Analysis (2-3 hours)

1. Load CSV in R
2. Compute stability rankings
3. Generate visualizations
4. Identify golden configuration

---

Expected Outcomes

Primary Question: Which of the 5 elite configurations exhibits:

• Lowest heartbeat jitter (CV)?
• Fastest convergence to steady state?
• Strongest load-response correlation?
• Cleanest temporal signature?

Expected Winner: Configuration with balanced F-factors, likely:

• F1=1 (heat tracking required)
• F3=1 or F5=1 (decay or inference, but not both)
• F6=0 or F6=1 (decay inference optional)

Golden Configuration: Becomes baseline physics for MamaForth & PapaForth downstream work.

---

Success Criteria

✓ All 5 configurations build and run without segfault ✓ Heartbeat data collected for all 250 runs ✓ Stability rankings computed and visualized ✓ Golden configuration identified with p < 0.05 significance ✓ R analysis report generated with 5+ plots

---

Version: 1.0 Status: Ready for implementation Next Step: Implement heartbeat metrics capture and extended DoE script