775 lines
23 KiB
Markdown
775 lines
23 KiB
Markdown
<!-- Moved from docs/03-architecture/pipelining/design.md to docs/working/architecture/03-architecture/pipelining/design.md on 2026-06-16 (docs reorg Phase 2) -->
|
||
# FORTH VM Pipelining & Speculative Execution Design
|
||
|
||
## Executive Summary
|
||
|
||
**Goal:** Reduce dictionary lookup latency by speculatively prefetching the next word while executing the current word.
|
||
|
||
**Physics Model Application:**
|
||
- Metrics: Word transition frequency (execution_heat[A→B])
|
||
- Decision: Speculatively prefetch word N+1 if P(N→N+1) > threshold
|
||
- Validation: Measure prefetch hit rate and latency savings
|
||
- Knobs: Threshold, speculation depth, confidence levels, misprediction cost
|
||
|
||
**Expected Impact:**
|
||
- Baseline (no pipelining): 31.543 ns cache hit, 56.237 ns bucket hit
|
||
- With pipelining: ~25-40% additional speedup possible (1.25-1.4× on top of cache)
|
||
- Cumulative with cache: 2.2-2.5× total (1.78× cache × 1.25-1.4× pipeline)
|
||
|
||
---
|
||
|
||
## 1. Conceptual Model
|
||
|
||
### Current Execution (No Pipelining)
|
||
|
||
```
|
||
Time: T0 T1 T2 T3 T4
|
||
↓ ↓ ↓ ↓ ↓
|
||
Word A: [L] [E] →
|
||
[L] [E] →
|
||
Word B: (blocked) [L]
|
||
[E] →
|
||
[L] [E]
|
||
Word C: (blocked)
|
||
|
||
Legend: [L] = Lookup, [E] = Execute
|
||
→ = Transition latency
|
||
|
||
Problem: Word B lookup blocked until Word A executes
|
||
```
|
||
|
||
### With Pipelining
|
||
|
||
```
|
||
Time: T0 T1 T2 T3 T4
|
||
↓ ↓ ↓ ↓ ↓
|
||
Word A: [L] [E] →
|
||
[P] [P] →
|
||
Word B: (prefetch) [L] [E]
|
||
[P] [P] →
|
||
Word C: (prefetch) [L]
|
||
|
||
Legend: [L] = Lookup, [E] = Execute, [P] = Prefetch (parallel)
|
||
→ = Transition latency (now hidden by pipeline)
|
||
|
||
Benefit: Word B lookup latency is prefetched (hidden behind Word A execution)
|
||
```
|
||
|
||
### The Key Insight: Execution Heat Predicts Transitions
|
||
|
||
```
|
||
Observation:
|
||
Word EXIT (heat=114) is almost always followed by:
|
||
- Word LIT (heat=101) [60% of the time]
|
||
- Word CR (heat=26) [15% of the time]
|
||
- Other (heat=?) [25% of the time]
|
||
|
||
Physics Model:
|
||
transition_heat[EXIT→LIT] = 600 (60% of 1000 executions)
|
||
transition_heat[EXIT→CR] = 150 (15%)
|
||
transition_heat[EXIT→?] = 250 (25%)
|
||
|
||
Decision Rule:
|
||
IF transition_heat[N→N+1] > THRESHOLD THEN
|
||
prefetch_word(N+1)
|
||
END
|
||
```
|
||
|
||
---
|
||
|
||
## 2. Metrics: Tracking Word Transitions
|
||
|
||
### 2.1 New Metrics to Collect
|
||
|
||
```c
|
||
typedef struct {
|
||
// Existing (from execution_heat tracking):
|
||
uint64_t execution_heat; // Total executions of this word
|
||
|
||
// NEW: Transition tracking
|
||
uint64_t transition_heat[DICT_SIZE]; // How many times does each word follow this word?
|
||
uint64_t total_transitions; // Sum of all transitions from this word
|
||
|
||
// NEW: Prediction accuracy
|
||
uint64_t prefetch_attempts; // How many times did we prefetch?
|
||
uint64_t prefetch_hits; // How many speculations were correct?
|
||
uint64_t prefetch_misses; // How many speculations were wrong?
|
||
|
||
// NEW: Latency tracking
|
||
uint64_t prefetch_latency_saved_q48; // Q48.16 nanoseconds saved by prefetch
|
||
uint64_t misprediction_cost_q48; // Q48.16 nanoseconds lost by wrong speculation
|
||
|
||
} WordPhysicsMetrics;
|
||
```
|
||
|
||
### 2.2 Data Collection Points
|
||
|
||
**Where we collect transitions:**
|
||
|
||
```c
|
||
// In vm.c: execute_colon_word() inner loop
|
||
for (;;) {
|
||
DictEntry *w = (DictEntry *)(uintptr_t)(*ip++);
|
||
|
||
// Existing: increment execution heat
|
||
w->execution_heat++;
|
||
|
||
// NEW: track transition to next word
|
||
if (next_word_ptr) {
|
||
DictEntry *next = (DictEntry *)(uintptr_t)(*ip);
|
||
if (next) {
|
||
w->metrics.transition_heat[next->id]++;
|
||
w->metrics.total_transitions++;
|
||
}
|
||
}
|
||
|
||
// Execute word
|
||
w->func(vm);
|
||
}
|
||
```
|
||
|
||
### 2.3 Transition Probability Calculation
|
||
|
||
```c
|
||
static double transition_probability(DictEntry *from, DictEntry *to) {
|
||
if (from->metrics.total_transitions == 0) return 0.0;
|
||
return (double)from->metrics.transition_heat[to->id] /
|
||
(double)from->metrics.total_transitions;
|
||
}
|
||
|
||
// Q48.16 fixed-point version (no floating-point)
|
||
static int64_t transition_probability_q48(DictEntry *from, DictEntry *to) {
|
||
if (from->metrics.total_transitions == 0) return 0;
|
||
// Compute: (transition_heat[to] / total_transitions) × 2^16
|
||
return ((int64_t)from->metrics.transition_heat[to->id] << 16) /
|
||
(int64_t)from->metrics.total_transitions;
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Pipelining Decision Logic
|
||
|
||
### 3.1 Prediction Decision Tree
|
||
|
||
```
|
||
FOR EACH word execution:
|
||
|
||
1. Calculate next word's identity (static: from instruction pointer)
|
||
|
||
2. Look up transition probability: P(current→next)
|
||
|
||
3. Decision:
|
||
IF P(current→next) > SPECULATION_THRESHOLD THEN
|
||
prefetch_word(next)
|
||
metrics.prefetch_attempts++
|
||
ELSE
|
||
// No prefetch (low confidence)
|
||
END
|
||
|
||
4. Execute current word
|
||
|
||
5. At transition, check prefetch result:
|
||
IF actual_next == predicted_next THEN
|
||
metrics.prefetch_hits++
|
||
// Lookup latency was hidden by pipeline
|
||
ELSE
|
||
metrics.prefetch_misses++
|
||
// Had to do full lookup (misprediction cost)
|
||
END
|
||
```
|
||
|
||
### 3.2 The Knobs: Tuning Parameters
|
||
|
||
**Knob 1: SPECULATION_THRESHOLD**
|
||
```c
|
||
#define SPECULATION_THRESHOLD_Q48 (0.50 * (1LL << 16)) // 50% confidence
|
||
```
|
||
- **Range:** 0.10 (10%) to 0.95 (95%)
|
||
- **Semantics:** "Speculate if we're at least X% confident that transition occurs"
|
||
- **Trade-off:**
|
||
- Low (10%): More speculation, higher hit rate, but more mispredictions
|
||
- High (95%): Conservative, low misprediction cost, but miss optimization opportunities
|
||
- **Sweet spot:** 50-70% (balance of coverage vs. accuracy)
|
||
|
||
**Knob 2: SPECULATION_DEPTH**
|
||
```c
|
||
#define SPECULATION_DEPTH 1 // How many words ahead to prefetch
|
||
```
|
||
- **Range:** 1 to 4 (deeper depths get expensive)
|
||
- **Semantics:** "Speculatively prefetch up to N words in advance"
|
||
- **Trade-off:**
|
||
- Depth=1: Prefetch word N+1 while executing N (simple)
|
||
- Depth=2: Prefetch both N+1 and N+2 (more aggressive)
|
||
- Depth=3+: Diminishing returns (limited by IP advancement rate)
|
||
- **Sweet spot:** 1-2 (Depth 1 recommended for simplicity)
|
||
|
||
**Knob 3: MIN_SAMPLES_FOR_SPECULATION**
|
||
```c
|
||
#define MIN_SAMPLES_FOR_SPECULATION 10 // Need 10+ transitions before speculating
|
||
```
|
||
- **Range:** 1 to 100
|
||
- **Semantics:** "Don't speculate on transitions we've only seen N times"
|
||
- **Trade-off:**
|
||
- Low (1): Speculate on even rare transitions
|
||
- High (100): Only speculate on well-established patterns
|
||
- **Sweet spot:** 10-20 (avoid noise, still responsive to patterns)
|
||
|
||
**Knob 4: MISPREDICTION_COST_Q48**
|
||
```c
|
||
#define MISPREDICTION_COST_Q48 (25LL << 16) // ~25 nanoseconds (estimate)
|
||
```
|
||
- **Range:** 0 to 100+ nanoseconds
|
||
- **Semantics:** "Estimated penalty for wrong speculation (in Q48.16 ns)"
|
||
- **Calculation:**
|
||
- Cost = Time to discard wrong prefetch + time to do correct lookup
|
||
- ~25 ns typical (speculative pipeline flush + re-lookup)
|
||
- **Sweet spot:** 20-30 nanoseconds
|
||
|
||
**Knob 5: MINIMUM_PREFETCH_ROI**
|
||
```c
|
||
#define MINIMUM_PREFETCH_ROI 1.10 // Only prefetch if we expect 1.1× or better ROI
|
||
```
|
||
- **Range:** 1.0 to 2.0
|
||
- **Semantics:** "Only speculate if expected benefit exceeds cost by X%"
|
||
- **Calculation:**
|
||
- ROI = (avg_lookup_latency / speculation_overhead) × prefetch_hit_rate
|
||
- If ROI > threshold, prefetch is worthwhile
|
||
- **Sweet spot:** 1.1-1.3 (10-30% expected improvement)
|
||
|
||
---
|
||
|
||
## 4. Implementation Strategy
|
||
|
||
### 4.1 Phase 1: Instrumentation (Data Collection)
|
||
|
||
**Goal:** Collect transition metrics without making predictions yet.
|
||
|
||
```c
|
||
// In vm.c: modify inner interpreter to track transitions
|
||
|
||
void execute_colon_word(VM *vm, DictEntry *entry) {
|
||
cell_t *code = entry->code;
|
||
cell_t *ip = code;
|
||
DictEntry *prev_word = NULL;
|
||
|
||
for (;;) {
|
||
DictEntry *w = (DictEntry *)(uintptr_t)(*ip++);
|
||
if (!w) break;
|
||
|
||
// Track transition from previous word
|
||
if (prev_word && prev_word->metrics.transition_heat) {
|
||
if (w->id < DICT_SIZE) {
|
||
prev_word->metrics.transition_heat[w->id]++;
|
||
prev_word->metrics.total_transitions++;
|
||
}
|
||
}
|
||
|
||
w->execution_heat++;
|
||
|
||
// Execute
|
||
w->func(vm);
|
||
if (vm->error) break;
|
||
|
||
prev_word = w;
|
||
}
|
||
}
|
||
```
|
||
|
||
**Measurement:** Let system run normally, collect transition data.
|
||
|
||
**Output:** Transition matrices showing word-to-word frequency.
|
||
|
||
### 4.2 Phase 2: Prediction Without Prefetching
|
||
|
||
**Goal:** Measure prediction accuracy before actually speculating.
|
||
|
||
```c
|
||
typedef struct {
|
||
DictEntry *predicted_next;
|
||
DictEntry *actual_next;
|
||
int was_correct;
|
||
uint64_t latency_saved_if_hit;
|
||
} SpeculationRecord;
|
||
|
||
// Dry run: predict but don't prefetch
|
||
void execute_with_prediction_logging(VM *vm, DictEntry *entry) {
|
||
cell_t *code = entry->code;
|
||
cell_t *ip = code;
|
||
|
||
for (;;) {
|
||
DictEntry *w = (DictEntry *)(uintptr_t)(*ip++);
|
||
if (!w) break;
|
||
|
||
// Look ahead at next word
|
||
DictEntry *predicted = predict_next_word(w);
|
||
|
||
// Get actual next word
|
||
DictEntry *actual = (DictEntry *)(uintptr_t)(*ip);
|
||
|
||
// Log prediction accuracy
|
||
if (predicted && actual) {
|
||
metrics.predictions++;
|
||
if (predicted == actual) {
|
||
metrics.correct_predictions++;
|
||
}
|
||
}
|
||
|
||
// Execute normally (no prefetch)
|
||
w->func(vm);
|
||
if (vm->error) break;
|
||
}
|
||
}
|
||
|
||
// Calculate prediction accuracy
|
||
double prediction_accuracy = (double)metrics.correct_predictions / metrics.predictions;
|
||
```
|
||
|
||
**Measurement:** Run benchmark, measure how often predictions would be correct.
|
||
|
||
**Output:** Prediction accuracy metrics (hint at potential ROI).
|
||
|
||
### 4.3 Phase 3: Actual Pipelining with Tuning
|
||
|
||
**Goal:** Implement speculative prefetch with adjustable knobs.
|
||
|
||
```c
|
||
typedef struct {
|
||
DictEntry *prefetched_word;
|
||
uint64_t prefetch_time_ns;
|
||
int was_used;
|
||
} SpeculativeCache;
|
||
|
||
void execute_with_pipelining(VM *vm, DictEntry *entry) {
|
||
cell_t *code = entry->code;
|
||
cell_t *ip = code;
|
||
SpeculativeCache spec_cache = {0};
|
||
|
||
for (;;) {
|
||
DictEntry *w = (DictEntry *)(uintptr_t)(*ip++);
|
||
if (!w) break;
|
||
|
||
// PREFETCH: Do speculation for NEXT iteration
|
||
// (while this word executes)
|
||
DictEntry *next = (DictEntry *)(uintptr_t)(*ip);
|
||
if (next && should_speculate(w, next)) {
|
||
uint64_t spec_start = sf_monotonic_ns();
|
||
spec_cache.prefetched_word = next;
|
||
spec_cache.prefetch_time_ns = spec_start;
|
||
metrics.prefetch_attempts++;
|
||
}
|
||
|
||
// EXECUTE: Run current word
|
||
w->execution_heat++;
|
||
w->func(vm);
|
||
if (vm->error) break;
|
||
|
||
// VALIDATE: Check if prefetch was correct
|
||
DictEntry *actual_next = (DictEntry *)(uintptr_t)(*ip);
|
||
if (spec_cache.prefetched_word == actual_next) {
|
||
// HIT: Prefetch was correct
|
||
uint64_t actual_lookup_latency = sf_monotonic_ns() - spec_cache.prefetch_time_ns;
|
||
metrics.prefetch_hits++;
|
||
metrics.total_latency_saved_q48 += (actual_lookup_latency << 16);
|
||
} else if (spec_cache.prefetched_word != NULL) {
|
||
// MISS: Prefetch was wrong
|
||
metrics.prefetch_misses++;
|
||
metrics.total_misprediction_cost_q48 += MISPREDICTION_COST_Q48;
|
||
}
|
||
|
||
spec_cache = (SpeculativeCache){0}; // Clear for next iteration
|
||
}
|
||
}
|
||
|
||
// Decision function
|
||
static int should_speculate(DictEntry *from, DictEntry *to) {
|
||
if (from->metrics.transition_heat[to->id] < MIN_SAMPLES_FOR_SPECULATION) {
|
||
return 0; // Not enough data
|
||
}
|
||
|
||
int64_t prob_q48 = transition_probability_q48(from, to);
|
||
|
||
if (prob_q48 < SPECULATION_THRESHOLD_Q48) {
|
||
return 0; // Not confident enough
|
||
}
|
||
|
||
// Calculate expected ROI
|
||
int64_t expected_savings = (int64_t)to->physics.avg_latency_ns << 16; // Q48.16
|
||
int64_t expected_cost = MISPREDICTION_COST_Q48;
|
||
|
||
// Hit rate estimate from history
|
||
double hit_rate = (double)from->metrics.prefetch_hits /
|
||
(double)(from->metrics.prefetch_attempts + 1);
|
||
|
||
int64_t expected_roi = (expected_savings * hit_rate) - expected_cost;
|
||
|
||
if (expected_roi < MINIMUM_PREFETCH_ROI) {
|
||
return 0; // Not worth it
|
||
}
|
||
|
||
return 1; // Speculate!
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Measurement & Validation
|
||
|
||
### 5.1 Key Metrics to Track
|
||
|
||
```c
|
||
typedef struct {
|
||
// Basic speculation metrics
|
||
uint64_t total_speculations; // Words we attempted to prefetch
|
||
uint64_t successful_prefetches; // Speculations that matched actual flow
|
||
uint64_t failed_prefetches; // Speculations that didn't match
|
||
|
||
// Calculated metrics
|
||
double prefetch_accuracy; // successful / total
|
||
double prefetch_hit_rate; // successful / total (same as above)
|
||
|
||
// Latency metrics (Q48.16 fixed-point)
|
||
int64_t total_latency_saved_q48; // Sum of latency hidden by pipeline
|
||
int64_t total_misprediction_cost_q48; // Sum of costs from wrong speculation
|
||
|
||
// ROI calculation
|
||
int64_t net_benefit_q48; // saved - cost
|
||
double roi_factor; // (saved / cost)
|
||
|
||
// Per-word metrics
|
||
double accuracy_by_source[DICT_SIZE]; // Prefetch accuracy for each word
|
||
|
||
} PipelineMetrics;
|
||
```
|
||
|
||
### 5.2 Expected Results (Hypothesis)
|
||
|
||
**Scenario A: Typical FORTH Workload**
|
||
```
|
||
Hot word transitions:
|
||
EXIT → LIT: 60% probability
|
||
LIT → @: 55% probability
|
||
@ → DROP: 40% probability
|
||
|
||
Prediction accuracy: ~60-70%
|
||
Misprediction cost: ~25 ns
|
||
Latency saved per hit: ~30-50 ns
|
||
Expected speedup: 1.25-1.40× (over cache baseline)
|
||
```
|
||
|
||
**Scenario B: Complex Control Flow (FORTH loops)**
|
||
```
|
||
More varied transitions (less predictable)
|
||
Prediction accuracy: ~35-45%
|
||
Misprediction cost: ~25 ns
|
||
Latency saved per hit: ~30-50 ns
|
||
Expected speedup: 1.05-1.15× (less benefit from branching)
|
||
```
|
||
|
||
**Scenario C: Worst Case (Random execution)**
|
||
```
|
||
No pattern in transitions (uniform random)
|
||
Prediction accuracy: ~5% (random guess)
|
||
Expected speedup: <1.0× (pipeline costs more than it saves)
|
||
Result: Pipelining disabled (ROI < threshold)
|
||
```
|
||
|
||
### 5.3 A/B Testing Framework
|
||
|
||
```bash
|
||
# Baseline: No pipelining
|
||
make clean && make ENABLE_PIPELINING=0 fastest
|
||
./build/amd64/fastest/starforth << 'EOF'
|
||
PHYSICS-RESET-STATS
|
||
100000 BENCH-DICT-LOOKUP
|
||
PHYSICS-PIPELINE-STATS
|
||
BYE
|
||
EOF
|
||
|
||
# With pipelining (default threshold = 50%)
|
||
make clean && make ENABLE_PIPELINING=1 fastest
|
||
./build/amd64/fastest/starforth << 'EOF'
|
||
PHYSICS-RESET-STATS
|
||
100000 BENCH-DICT-LOOKUP
|
||
PHYSICS-PIPELINE-STATS
|
||
BYE
|
||
EOF
|
||
|
||
# With aggressive pipelining (threshold = 30%)
|
||
make clean && make ENABLE_PIPELINING=1 SPECULATION_THRESHOLD=0.30 fastest
|
||
./build/amd64/fastest/starforth << 'EOF'
|
||
PHYSICS-RESET-STATS
|
||
100000 BENCH-DICT-LOOKUP
|
||
PHYSICS-PIPELINE-STATS
|
||
BYE
|
||
EOF
|
||
|
||
# With conservative pipelining (threshold = 80%)
|
||
make clean && make ENABLE_PIPELINING=1 SPECULATION_THRESHOLD=0.80 fastest
|
||
./build/amd64/fastest/starforth << 'EOF'
|
||
PHYSICS-RESET-STATS
|
||
100000 BENCH-DICT-LOOKUP
|
||
PHYSICS-PIPELINE-STATS
|
||
BYE
|
||
EOF
|
||
```
|
||
|
||
**Output:** Compare speedup factors across threshold settings.
|
||
|
||
---
|
||
|
||
## 6. The Knobs: Complete Reference
|
||
|
||
### Knob Summary Table
|
||
|
||
| Knob | Purpose | Range | Default | Trade-off |
|
||
|------|---------|-------|---------|-----------|
|
||
| **SPECULATION_THRESHOLD** | Min confidence to speculate | 0.10–0.95 | 0.50 | Coverage vs. Accuracy |
|
||
| **SPECULATION_DEPTH** | Words ahead to prefetch | 1–4 | 1 | Aggressiveness vs. Complexity |
|
||
| **MIN_SAMPLES** | Min transitions before speculating | 1–100 | 10 | Responsiveness vs. Noise |
|
||
| **MISPREDICTION_COST** | Penalty for wrong spec (ns) | 0–100 | 25 | Conservative vs. Aggressive |
|
||
| **MINIMUM_ROI** | Min expected improvement | 1.0–2.0 | 1.10 | Selectivity vs. Coverage |
|
||
|
||
### Knob Tuning Strategy
|
||
|
||
**Step 1: Collect Baseline**
|
||
```
|
||
Run with ENABLE_PIPELINING=0
|
||
Measure: Baseline speedup, latency distribution
|
||
```
|
||
|
||
**Step 2: Identify Best Threshold**
|
||
```
|
||
Try: 0.30, 0.50, 0.70, 0.90
|
||
Measure: Prefetch accuracy, net speedup
|
||
Find: Threshold that maximizes speedup
|
||
```
|
||
|
||
**Step 3: Optimize Depth**
|
||
```
|
||
Try: SPECULATION_DEPTH=1,2,3
|
||
For best threshold from Step 2
|
||
Measure: Speedup vs. CPU overhead
|
||
```
|
||
|
||
**Step 4: Tune Sample Minimum**
|
||
```
|
||
Try: MIN_SAMPLES=5, 10, 20, 50
|
||
Measure: How it affects early-startup behavior
|
||
```
|
||
|
||
**Step 5: Calibrate Misprediction Cost**
|
||
```
|
||
Measure actual misprediction penalty: How long does a wrong spec take to recover?
|
||
Tune MISPREDICTION_COST_Q48 accordingly
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Advanced Topics
|
||
|
||
### 7.1 Adaptive Thresholding
|
||
|
||
**Dynamic threshold adjustment based on workload:**
|
||
|
||
```c
|
||
// Adjust threshold based on recent success rate
|
||
void adapt_speculation_threshold(void) {
|
||
double recent_hit_rate = calculate_rolling_hit_rate(last_1000_speculations);
|
||
|
||
if (recent_hit_rate > 0.70) {
|
||
// Speculations are working well, be more aggressive
|
||
SPECULATION_THRESHOLD_Q48 = 0.40 * (1LL << 16); // Lower threshold
|
||
} else if (recent_hit_rate < 0.40) {
|
||
// Speculations are failing, be more conservative
|
||
SPECULATION_THRESHOLD_Q48 = 0.60 * (1LL << 16); // Higher threshold
|
||
}
|
||
}
|
||
|
||
// Call periodically (e.g., every 10K words)
|
||
if (total_executions % 10000 == 0) {
|
||
adapt_speculation_threshold();
|
||
}
|
||
```
|
||
|
||
### 7.2 Hierarchical Speculation
|
||
|
||
**Different threshold for different word categories:**
|
||
|
||
```c
|
||
// Some words have more predictable successors
|
||
if (from->flags & WORD_LOOP_CONTROL) {
|
||
threshold = 0.30; // Loops have predictable patterns
|
||
} else if (from->flags & WORD_CONDITIONAL) {
|
||
threshold = 0.70; // Conditionals are less predictable
|
||
} else {
|
||
threshold = 0.50; // Default
|
||
}
|
||
```
|
||
|
||
### 7.3 Multi-Step Prediction
|
||
|
||
**Predict chains of words:**
|
||
|
||
```c
|
||
// Example: EXIT almost always followed by LIT, which is followed by @
|
||
if (should_speculate(EXIT, LIT) && should_speculate(LIT, @)) {
|
||
prefetch(LIT);
|
||
prefetch(@); // Depth-2 prefetch
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 8. Comparison to CPU Pipelining
|
||
|
||
### 8.1 Analogies
|
||
|
||
| CPU Concept | FORTH VM Analogy |
|
||
|---|---|
|
||
| **Instruction fetch** | Dictionary lookup |
|
||
| **Decode** | Word dispatch |
|
||
| **Execute** | Word execution (func call) |
|
||
| **Branch prediction** | Transition prediction |
|
||
| **Speculative execution** | Prefetch next word |
|
||
| **Branch misprediction penalty** | Misprediction cost (~25 ns) |
|
||
| **Pipeline depth** | SPECULATION_DEPTH |
|
||
|
||
### 8.2 Key Difference
|
||
|
||
**CPUs predict branches (conditional jumps).** FORTH VMs predict *sequential transitions* (what comes after current word).
|
||
|
||
- **Easier prediction:** Sequential is more predictable than branches
|
||
- **Simpler recovery:** Wrong sequential guess = just do lookup, no flush needed
|
||
- **Lower cost:** Prefetch is cheap (just a lookup ahead of time)
|
||
|
||
---
|
||
|
||
## 9. Expected Cumulative Speedup
|
||
|
||
### Stacking Optimizations
|
||
|
||
```
|
||
Baseline (no optimization): 1.0×
|
||
+ Hot-words cache (Phase 1): 1.78×
|
||
+ Pipelining (Phase 2): 2.2–2.5× (1.78 × 1.25–1.4)
|
||
+ Stack fusion (Phase 3): 2.6–3.5×
|
||
+ Other optimizations (Phase 4+): 5–8×
|
||
```
|
||
|
||
### Why Pipelining is Worth Doing
|
||
|
||
1. **Orthogonal to cache** (can combine them)
|
||
2. **Leverages execution_heat data** (already collected)
|
||
3. **Simple mechanics** (just prefetch one word ahead)
|
||
4. **Tunable** (many knobs to optimize)
|
||
5. **Verifiable** (deterministic, no JIT needed)
|
||
|
||
---
|
||
|
||
## 10. Implementation Roadmap
|
||
|
||
### Phase 1: Instrumentation (1 week)
|
||
- [ ] Add transition_heat tracking to DictEntry
|
||
- [ ] Collect transition statistics during normal execution
|
||
- [ ] Print transition matrices for analysis
|
||
- **Deliverable:** Understanding of word-to-word patterns
|
||
|
||
### Phase 2: Prediction Analysis (1 week)
|
||
- [ ] Implement prediction-without-prefetch
|
||
- [ ] Measure prediction accuracy by threshold
|
||
- [ ] Find optimal threshold experimentally
|
||
- [ ] Estimate potential speedup
|
||
- **Deliverable:** A/B test showing prediction accuracy
|
||
|
||
### Phase 3: Implement Pipelining (2 weeks)
|
||
- [ ] Implement actual speculative prefetch
|
||
- [ ] Add should_speculate() decision function
|
||
- [ ] Collect prefetch metrics (hits, misses, ROI)
|
||
- [ ] Compare vs. baseline
|
||
- **Deliverable:** Measured speedup with default knobs
|
||
|
||
### Phase 4: Knob Tuning & Optimization (2 weeks)
|
||
- [ ] Systematic knob tuning experiments
|
||
- [ ] Adaptive thresholding
|
||
- [ ] Per-word threshold optimization
|
||
- [ ] Documentation of optimal knob values
|
||
- **Deliverable:** Performance-tuned pipelining system
|
||
|
||
### Phase 5: Integration & Testing (1 week)
|
||
- [ ] Integration with existing hot-words cache
|
||
- [ ] Full test suite (no regressions)
|
||
- [ ] Reproducibility validation
|
||
- [ ] Documentation for operators
|
||
- **Deliverable:** Production-ready pipelining
|
||
|
||
---
|
||
|
||
## 11. Risks & Mitigations
|
||
|
||
### Risk 1: Prediction Accuracy Too Low
|
||
**Symptom:** < 30% hit rate
|
||
**Mitigation:** Dynamic threshold adjustment, fallback to cache-only mode
|
||
|
||
### Risk 2: Misprediction Cost Too High
|
||
**Symptom:** Speedup < 1.0× (pipelining makes things slower)
|
||
**Mitigation:** Disable pipelining automatically when ROI < threshold
|
||
|
||
### Risk 3: Data Structure Overhead
|
||
**Symptom:** Memory usage explodes (transition_heat[] per word)
|
||
**Mitigation:** Use sparse matrix or hash-based approach, limit to top N transitions per word
|
||
|
||
### Risk 4: Complexity Explosion
|
||
**Symptom:** Too many knobs, hard to understand/tune
|
||
**Mitigation:** Start simple (SPECULATION_THRESHOLD only), add knobs incrementally
|
||
|
||
---
|
||
|
||
## 12. Success Criteria
|
||
|
||
✅ **Achievable Goals:**
|
||
- [ ] Prediction accuracy > 50% for hot words
|
||
- [ ] Prefetch hit rate > 40% overall
|
||
- [ ] Measured speedup 1.2–1.4× over cache
|
||
- [ ] Knobs documented and tunable
|
||
- [ ] Works with hot-words cache (cumulative 2.2–2.5×)
|
||
- [ ] No regressions on unoptimized code paths
|
||
|
||
⚠️ **Stretch Goals:**
|
||
- [ ] Adaptive thresholding improves over fixed threshold
|
||
- [ ] Depth-2 prefetch worthwhile in some workloads
|
||
- [ ] Per-word threshold optimization
|
||
- [ ] Formal verification of correctness
|
||
|
||
❌ **Non-Goals:**
|
||
- Exceed 2× speedup (unrealistic without JIT)
|
||
- Eliminate all misprediction cost
|
||
- Work perfectly on all possible FORTH programs
|
||
|
||
---
|
||
|
||
## Next Steps
|
||
|
||
1. **Start with Phase 1:** Instrument transition tracking
|
||
2. **Analyze patterns:** What words transition to what?
|
||
3. **Run prediction dry-run:** How accurate are predictions?
|
||
4. **Implement pipelining:** With default knobs
|
||
5. **Measure & iterate:** Tune knobs for best performance
|
||
6. **Document results:** Add to academic materials
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
### Related Work
|
||
- CPU Branch Prediction (Yeh & Patt 1991)
|
||
- Speculative Execution (Sprangle & Wilkerson 2001)
|
||
- FORTH optimization (Bell Labs 1980s)
|
||
|
||
### Within StarForth
|
||
- `/docs/PHYSICS_HOTWORDS_CACHE_EXPERIMENT.md` - Cache baseline
|
||
- `/include/physics_metadata.h` - Metrics framework
|
||
- `/include/physics_hotwords_cache.h` - Cache implementation
|
||
- `/docs/PHYSICS_OPTIMIZATION_PROPOSALS.md` - Future opportunities
|
||
|