LithosAnanake/docs/working/architecture/03-architecture/heartbeat-system/implementation-plan.md

Implementation Plan: Heartbeat Rate Logging

Date: November 20, 2025 Priority: BLOCKING — Must implement before running Phase 2 DoE Complexity: Medium (requires code additions, CSV schema update)

---

Overview

We need to capture tick_ns trajectory (the actual heartbeat rate changes) during execution, then analyze how each configuration modulates heartrate in response to workload.

---

Step 1: Add Heartbeat Rate Sample Struct

File: include/vm.h

Add after HeartbeatSnapshot definition:

/* === Heartbeat Rate Trajectory Collection === */

#define HEARTBEAT_RATE_SAMPLE_MAX 100000  /* Maximum samples per run */

typedef struct {
    uint64_t tick_number;               /* Absolute tick count */
    uint64_t tick_ns;                   /* ← HEARTBEAT RATE (nanoseconds) */
    uint64_t workload_ops_this_tick;    /* Dictionary lookups this tick */
    uint64_t timestamp_ns;              /* Wall-clock timestamp */
    uint64_t hot_word_count;            /* Physics state: hot words */
    uint64_t total_heat;                /* Physics state: total execution heat */
} HeartbeatRateSample;

File: include/vm.h (in VM struct)

Add to the HeartbeatState section:

/** @name Heartbeat Rate Trajectory (Phase 2 DoE)
 * @{
 */
HeartbeatRateSample* heartbeat_rate_samples;  /* Circular buffer of rate samples */
uint64_t heartbeat_rate_sample_count;         /* Number of samples collected */
/** @} */

---

Step 2: Initialize Heartbeat Rate Collection

File: src/vm.c in vm_init()

Add after heartbeat initialization:

/* Initialize heartbeat rate sampling for Phase 2 DoE */
vm->heartbeat_rate_samples = calloc(HEARTBEAT_RATE_SAMPLE_MAX, sizeof(HeartbeatRateSample));
vm->heartbeat_rate_sample_count = 0;

File: src/vm.c in vm_cleanup()

Add cleanup:

if (vm->heartbeat_rate_samples) {
    free(vm->heartbeat_rate_samples);
    vm->heartbeat_rate_samples = NULL;
}

---

Step 3: Capture Heartbeat Rate Per Tick

File: src/vm.c - Add helper function

/**
 * @brief Capture current heartbeat rate sample for analysis
 *
 * Called by heartbeat_thread_main() to log the tick_ns (heartbeat rate)
 * along with contemporaneous workload metrics.
 *
 * @param vm Pointer to VM
 * @param tick_ns Current heartbeat interval (nanoseconds)
 * @param workload_ops Dictionary lookups executed this tick
 */
static void capture_heartbeat_rate_sample(VM *vm, uint64_t tick_ns, uint64_t workload_ops)
{
    if (!vm || !vm->heartbeat_rate_samples)
        return;

    if (vm->heartbeat_rate_sample_count >= HEARTBEAT_RATE_SAMPLE_MAX)
        return;  /* Buffer full, stop sampling */

    HeartbeatRateSample *sample = &vm->heartbeat_rate_samples[vm->heartbeat_rate_sample_count];
    sample->tick_number = vm->heartbeat.tick_count;
    sample->tick_ns = tick_ns;
    sample->workload_ops_this_tick = workload_ops;
    sample->timestamp_ns = platform_monotonic_ns();

    /* Capture current physics state */
    sample->hot_word_count = vm->hot_word_count_at_check;
    sample->total_heat = vm->total_heat_at_last_check;

    vm->heartbeat_rate_sample_count++;
}

---

Step 4: Log Workload Per Tick

File: src/vm.c - Track lookups per tick

Add to VM struct (in include/vm.h):

/** @name Per-Tick Workload Metrics
 * @{
 */
uint64_t workload_ops_current_tick;  /* Dictionary lookups in current tick */
/** @} */

File: src/vm.c - In vm_heartbeat_run_cycle()

Before calling inference engine:

void vm_heartbeat_run_cycle(VM *vm)
{
    if (!vm || !vm->heartbeat.heartbeat_enabled)
        return;

    vm->heartbeat.tick_count++;

    /* Reset per-tick workload counter */
    uint64_t lookups_this_tick = vm->workload_ops_current_tick;
    vm->workload_ops_current_tick = 0;  /* Reset for next tick */

    /* ... existing inference logic ... */
}

File: src/vm.c - Increment lookup counter

In the main word execution loop (in vm_interpret() or inner interpreter):

/* After successful dictionary lookup */
vm->workload_ops_current_tick++;

---

Step 5: Capture in Heartbeat Thread

File: src/vm.c in heartbeat_thread_main()

Replace the existing sleep logic:

static void* heartbeat_thread_main(void *arg)
{
    VM *vm = (VM*)arg;
    if (!vm || !vm->heartbeat.worker)
        return NULL;

    HeartbeatWorker *worker = vm->heartbeat.worker;
    worker->running = 1;

    while (!worker->stop_requested && vm->heartbeat.heartbeat_enabled)
    {
        vm_heartbeat_run_cycle(vm);

        /* GET THE CURRENT HEARTBEAT RATE */
        uint64_t tick_ns = worker->tick_ns ? worker->tick_ns : HEARTBEAT_TICK_NS;
        uint64_t workload_ops = vm->workload_ops_current_tick;

        /* ↓ ADD THIS ↓ CAPTURE HEARTBEAT RATE SAMPLE */
        capture_heartbeat_rate_sample(vm, tick_ns, workload_ops);
        /* ↑ ADD THIS ↑ */

        struct timespec req;
        req.tv_sec = (time_t)(tick_ns / 1000000000ULL);
        req.tv_nsec = (long)(tick_ns % 1000000000ULL);

        struct timespec rem;
        while (!worker->stop_requested && vm->heartbeat.heartbeat_enabled &&
               nanosleep(&req, &rem) == -1 && errno == EINTR)
        {
            req = rem;
        }
    }

    worker->running = 0;
    return NULL;
}

---

Step 6: Extend DoeMetrics

File: include/doe_metrics.h

Add to DoeMetrics struct:

/* === HEARTBEAT RATE MODULATION METRICS (NEW) === */

/* Per-run heartbeat rate statistics */
uint64_t total_heartbeat_ticks;
double   mean_heartbeat_rate_hz;             /* Average frequency (Hz) */
double   heartbeat_rate_stddev_hz;           /* Variation in Hz */
double   heartbeat_rate_cv;                  /* CV of heartbeat rate */
uint64_t heartbeat_rate_min_ns;              /* Slowest tick observed */
uint64_t heartbeat_rate_max_ns;              /* Fastest tick observed */
double   heartbeat_rate_modulation_range_hz; /* max_hz - min_hz */

/* Load-Heartrate Response */
double   load_to_heartrate_correlation;      /* Correlation(workload, heartrate) */
int      load_response_direction;            /* +1 if slower under load, -1 if faster */
double   heartrate_adaptation_latency_ticks; /* Ticks until rate responds to load */
double   heartrate_settling_time_ticks;      /* Ticks to settle after adaptation */

/* Heartrate Convergence */
double   heartrate_convergence_time_ms;      /* Time to reach stable rate */
int      heartrate_converged;                /* 1 = converged, 0 = oscillating */
double   heartrate_oscillation_amplitude_hz; /* Peak-to-peak variation in final state */

/* Overall Heartrate Quality Score */
double   heartrate_responsiveness_score;     /* 0-100: how well does it respond? */
double   heartrate_stability_score;          /* 0-100: how smooth is adaptation? */

---

Step 7: Implement Metrics Extraction

File: src/doe_metrics.c

Add new function:

/**
 * @brief Analyze heartbeat rate trajectory and compute metrics
 *
 * @param vm Pointer to VM with heartbeat_rate_samples
 * @param metrics Pointer to DoeMetrics to fill
 */
static void analyze_heartbeat_rate_trajectory(VM *vm, DoeMetrics *metrics)
{
    if (!vm || !vm->heartbeat_rate_samples || vm->heartbeat_rate_sample_count == 0) {
        metrics->mean_heartbeat_rate_hz = 0;
        metrics->load_to_heartrate_correlation = 0;
        return;
    }

    uint64_t count = vm->heartbeat_rate_sample_count;
    HeartbeatRateSample *samples = vm->heartbeat_rate_samples;

    /* 1. Compute heartbeat rate statistics (tick_ns → Hz) */
    double *heartrate_hz = malloc(count * sizeof(double));
    double *workload_intensity = malloc(count * sizeof(double));

    double min_ns = samples[0].tick_ns;
    double max_ns = samples[0].tick_ns;
    double sum_hz = 0;
    double sum_workload = 0;

    for (uint64_t i = 0; i < count; i++) {
        double tick_ns = samples[i].tick_ns;
        heartrate_hz[i] = 1e9 / tick_ns;
        workload_intensity[i] = samples[i].workload_ops_this_tick;

        sum_hz += heartrate_hz[i];
        sum_workload += workload_intensity[i];

        if (tick_ns < min_ns) min_ns = tick_ns;
        if (tick_ns > max_ns) max_ns = tick_ns;
    }

    /* Mean heartbeat rate */
    metrics->mean_heartbeat_rate_hz = sum_hz / count;
    metrics->heartbeat_rate_min_ns = (uint64_t)min_ns;
    metrics->heartbeat_rate_max_ns = (uint64_t)max_ns;
    metrics->heartbeat_rate_modulation_range_hz = (1e9 / min_ns) - (1e9 / max_ns);

    /* 2. Compute standard deviation */
    double variance = 0;
    for (uint64_t i = 0; i < count; i++) {
        double delta = heartrate_hz[i] - metrics->mean_heartbeat_rate_hz;
        variance += delta * delta;
    }
    metrics->heartbeat_rate_stddev_hz = sqrt(variance / count);
    metrics->heartbeat_rate_cv = metrics->heartbeat_rate_stddev_hz / metrics->mean_heartbeat_rate_hz;

    /* 3. Compute correlation(workload, heartrate) — THE KEY METRIC */
    double workload_mean = sum_workload / count;
    double cov = 0;
    double workload_var = 0;

    for (uint64_t i = 0; i < count; i++) {
        double workload_delta = workload_intensity[i] - workload_mean;
        double heartrate_delta = heartrate_hz[i] - metrics->mean_heartbeat_rate_hz;
        cov += workload_delta * heartrate_delta;
        workload_var += workload_delta * workload_delta;
    }

    if (workload_var > 0 && metrics->heartbeat_rate_stddev_hz > 0) {
        metrics->load_to_heartrate_correlation = cov / sqrt(workload_var *
            (metrics->heartbeat_rate_stddev_hz * metrics->heartbeat_rate_stddev_hz * count));
    }

    /* 4. Determine if heartrate responds correctly to load */
    metrics->load_response_direction = (metrics->load_to_heartrate_correlation > 0.3) ? 1 : -1;

    /* 5. Estimate convergence time (when heartrate stabilizes) */
    double threshold = 0.15;  /* 15% of final value */
    metrics->heartrate_converged = 1;
    for (uint64_t i = count / 2; i < count; i++) {
        double current_cv = 0;
        /* Compute CV of last N samples */
        for (uint64_t j = i; j < (i + 100 < count ? i + 100 : count); j++) {
            double delta = heartrate_hz[j] - metrics->mean_heartbeat_rate_hz;
            current_cv += delta * delta;
        }
        if (sqrt(current_cv) > threshold * metrics->mean_heartbeat_rate_hz) {
            metrics->heartrate_converged = 0;
            metrics->heartrate_convergence_time_ms = i;  /* Approximate */
            break;
        }
    }

    /* 6. Compute overall responsiveness score */
    /* Higher correlation + faster convergence + lower CV = higher score */
    metrics->heartrate_responsiveness_score =
        (metrics->load_to_heartrate_correlation + 1.0) * 50 *
        (1.0 / (1.0 + metrics->heartbeat_rate_cv));

    metrics->heartrate_stability_score =
        (100.0 * (1.0 - metrics->heartbeat_rate_cv)) *
        (metrics->heartrate_converged ? 1.0 : 0.5);

    free(heartrate_hz);
    free(workload_intensity);
}

Update metrics_from_vm()

Call the new analysis function:

DoeMetrics metrics_from_vm(VM *vm, ...) {
    /* ... existing metrics extraction ... */

    /* NEW: Analyze heartbeat rate trajectory */
    analyze_heartbeat_rate_trajectory(vm, &metrics);

    return metrics;
}

---

Step 8: Update CSV Schema

File: src/doe_metrics.c in metrics_write_csv_header()

Add new columns:

void metrics_write_csv_header(FILE *out) {
    {
        printf '[...existing columns...]');
        printf('total_heartbeat_ticks,mean_heartbeat_rate_hz,heartbeat_rate_stddev_hz,heartbeat_rate_cv,'
               'heartbeat_rate_min_ns,heartbeat_rate_max_ns,heartbeat_rate_modulation_range_hz,'
               'load_to_heartrate_correlation,load_response_direction,'
               'heartrate_convergence_time_ms,heartrate_converged,'
               'heartrate_responsiveness_score,heartrate_stability_score\n');
    } > output_csv;
}

File: src/doe_metrics.c in metrics_write_csv_row()

Add the values:

void metrics_write_csv_row(FILE *out, const DoeMetrics *metrics) {
    fprintf(out, "[...existing fields...],"
            "%lu,%lf,%lf,%lf,%lu,%lu,%lf,%lf,%d,%lu,%d,%lf,%lf\n",
            metrics->total_heartbeat_ticks,
            metrics->mean_heartbeat_rate_hz,
            metrics->heartbeat_rate_stddev_hz,
            metrics->heartbeat_rate_cv,
            metrics->heartbeat_rate_min_ns,
            metrics->heartbeat_rate_max_ns,
            metrics->heartbeat_rate_modulation_range_hz,
            metrics->load_to_heartrate_correlation,
            metrics->load_response_direction,
            metrics->heartrate_convergence_time_ms,
            metrics->heartrate_converged,
            metrics->heartrate_responsiveness_score,
            metrics->heartrate_stability_score);
}

---

Step 9: Updated R Analysis

File: scripts/analyze_heartbeat_stability_REVISED.R

library(tidyverse)

# Load data
doe_data <- read.csv("experiment_results_heartbeat.csv") %>%
  mutate(
    configuration = as.factor(configuration),
    # New metrics available now
    load_response = ifelse(load_to_heartrate_correlation > 0.5, "Responsive", "Sluggish")
  )

# PRIMARY METRIC: Load-Heartrate Correlation
# This tells us which config best responds to workload changes
ranking <- doe_data %>%
  group_by(configuration) %>%
  summarize(
    mean_correlation = mean(load_to_heartrate_correlation, na.rm = TRUE),
    mean_responsiveness = mean(heartrate_responsiveness_score, na.rm = TRUE),
    mean_stability = mean(heartrate_stability_score, na.rm = TRUE),
    mean_convergence_time = mean(heartrate_convergence_time_ms, na.rm = TRUE),
    golden_score = (mean(load_to_heartrate_correlation) * 0.5) +
                   (mean(heartrate_responsiveness_score) * 0.3) +
                   (mean(heartrate_stability_score) * 0.2)
  ) %>%
  arrange(-golden_score)

print("=== HEARTBEAT RATE RESPONSIVENESS RANKING ===")
print(ranking)

# Plot 1: Load-Heartrate Correlation (KEY METRIC)
ggplot(doe_data, aes(x = reorder(configuration, load_to_heartrate_correlation, FUN=median),
                      y = load_to_heartrate_correlation,
                      fill = configuration)) +
  geom_boxplot() +
  coord_flip() +
  labs(title = "Load-Heartrate Correlation (Higher = Better)",
       y = "Correlation(workload, heartrate)")
ggsave("01_load_heartrate_correlation.png")

# Plot 2: Heartrate Responsiveness Score
ggplot(doe_data, aes(x = reorder(configuration, heartrate_responsiveness_score, FUN=median),
                      y = heartrate_responsiveness_score,
                      fill = configuration)) +
  geom_boxplot() +
  coord_flip() +
  labs(title = "Heartrate Responsiveness Score (0-100)")
ggsave("02_heartrate_responsiveness.png")

---

Compilation Steps

1. Update headers: Add structs and fields to include/vm.h, include/doe_metrics.h
2. Implement collection: Add capture function in src/vm.c
3. Implement analysis: Add analysis function in src/doe_metrics.c
4. Update CSV: Modify CSV header/row writers
5. Compile: make clean && make fastest
6. Test: ./build/starforth -c "1 2 + . BYE"

---

Verification

After implementation, verify:

✓ Binary compiles without errors ✓ Heartbeat rate samples are captured (check sample_count > 0 after run) ✓ CSV contains new heartrate columns ✓ Correlation metrics are computed ✓ R analysis generates ranking by load-response

---

Expected Results

When properly implemented, you'll see:

=== HEARTBEAT RATE RESPONSIVENESS RANKING ===
configuration  mean_correlation  mean_responsiveness  golden_score
1_0_1_1_1_0          0.82             78.5               0.795
1_0_1_1_1_1          0.78             75.2               0.761
1_1_0_1_1_1          0.65             62.3               0.641
1_0_1_0_1_0          0.45             48.1               0.451
0_1_1_0_1_1          0.12             15.8               0.148

Golden config: 1_0_1_1_1_0 (strongest load-heartrate coupling)

---

Timeline

• Implementation: 2-3 hours
• Testing: 1 hour
• Re-run Phase 2: 2-4 hours
• Analysis: 30 min
• Total: ~6-9 hours

---

Files Modified

• include/vm.h — Add HeartbeatRateSample struct and VM fields
• include/doe_metrics.h — Add heartrate metrics to DoeMetrics
• src/vm.c — Add sample capture, logging, initialization
• src/doe_metrics.c — Add analysis function, CSV updates
• scripts/analyze_heartbeat_stability_REVISED.R — Rewritten for load-response focus

---

This implementation captures the actual heartbeat rate modulation and measures how well each config responds to load.

This is the correct physics story. 🚀