// Moved from docs/src/INTEGRATION_NOTES.adoc to docs/working/scratch/src/INTEGRATION_NOTES.adoc on 2026-06-16 (docs reorg Phase 2)
== Integration Scripts Documentation
:toc: left
:toc-title: Contents
:toclevels: 3
xref:./README.adoc[← Back to Documentation Index]



=== Overview

Two complementary sync scripts for bidirectional integration between
StarForth and StarshipOS.

=== Scripts

==== 1. `+integrator.sh+` (StarForth → StarshipOS) ✅ WORKING

* *Location*: `+StarForth/maint/integrator.sh+`
* *Direction*: StarForth → StarshipOS
* *Source*: StarForth root
* *Destination*: `+$STARSHIPOS_ROOT/l4/pkg/starforth/server+`
* *Quarantine*: `+StarForth/maint/quarantine+`
* *Path stripping*: Strips `+server/+` prefix from files
* *Status*: Fully debugged and tested (spent all day on this)

==== 2. `+integrator.sh+` (StarshipOS → StarForth) 🆕 NEW

* *Final Location*: `+StarshipOS/maint/integrator.sh+`
* *Direction*: StarshipOS → StarForth (reverse)
* *Source*: StarshipOS root
* *Destination*: `+$STARFORTH_ROOT+`
* *Quarantine*: `+StarshipOS/maint/quarantine+`
* *Path stripping*: Strips `+l4/pkg/starforth/server/+` prefix from
files
* *Status*: Generated based on working integrator.sh, needs testing

=== The Rules (Cap’t Bob Sez)

[arabic]
. *Never mkdir into target* — both repos must already exist
. *Blacklist = Law* — anything matching it is ignored
. *If file exists at same relative path in destination* → overwrite
(green)
. *Otherwise* → quarantine for review (orange)

=== How They Work

Both scripts: 1. Read files from `+maint/mergefiles.txt+` (or generate
from `+git diff --name-only HEAD~1+`) 2. Check each file against
`+maint/blacklist.txt+` 3. Strip appropriate path prefix to compute
destination path 4. If destination exists → overwrite (safe) 5. If
destination doesn’t exist → quarantine for Captain’s review 6. Refresh
git index when done

=== Next Steps

* [ ] Move `+integrator.sh+` to `+StarshipOS/maint/+`
* [ ] Test `+integrator.sh+` in StarshipOS
* [ ] Verify blacklist.txt and mergefiles.txt exist in StarshipOS/maint/
* [ ] Test round-trip sync (both directions)

=== Notes

* Both scripts are meant to be run from their respective repository
roots
* Quarantine prevents accidental file creation in unexpected locations
* The path stripping logic is the key difference between the two scripts

=== ⚠️ Special Cases

==== Documentation Directory (`+docs/+`)

*Issue*: Documentation needs bidirectional sync but has different paths:
- StarForth: `+docs/+` - StarshipOS: `+l4/pkg/starforth/docs/+`

*Current Status*: ⏳ *NOT YET INTEGRATED* - Manual handling required
*Captain’s Note*: Will be integrated into integrator.sh later with
custom path mapping logic

*Paths*: - `+/home/rajames/CLionProjects/StarForth/docs+` -
`+/home/rajames/CLionProjects/StarshipOS/l4/pkg/starforth/docs+`

*Future Enhancement*: Add special case handling in both integrators for
`+docs/+` directory with custom path transformations

'''''

=== Test & Repair Cycle (StarshipOS → StarForth)

==== Date: 2025-10-19

==== Status: ✅ PRODUCTION READY

This section documents the complete debugging process for
`+StarshipOS/maint/integrator.sh+`. Use this methodology to test and
debug the reverse script (`+StarForth/maint/integrator.sh+`).

'''''

==== 🐛 Bugs Found & Fixed

===== Bug #1: Blacklist Regex Matching (Line 43) - CRITICAL

*Problem*: Used `+grep -Fqx+` which treats patterns as *fixed strings*,
but `+blacklist.txt+` contains *regex patterns* like
`+(^|/)?Makefile($|[.].*)+`.

*Symptom*: Makefiles and blacklisted files were NOT being blocked.

*Original Code*:

[source,bash]
----
if grep -Fqx -f "$BLACKLIST" <<< "$FILE"; then
----

*Fix*:

[source,bash]
----
if grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' "$BLACKLIST") <<< "$FILE"; then
----

*Changes*: - `+-F+` removed (enables regex) - `+-x+` removed (allows
partial line matching) - `+-E+` added (extended regex) - `+sed+` filter
added (see Bug #2)

'''''

===== Bug #2: Empty Lines in Blacklist - CRITICAL

*Problem*: Line 5 in `+blacklist.txt+` is empty. Empty patterns in
`+grep -f+` match *everything*.

*Symptom*: ALL files were being blocked when blacklist was enabled.

*Test*:

[source,bash]
----
# This blocks EVERYTHING because of empty line:
echo "any_file.txt" | grep -Eq -f blacklist.txt && echo "BLOCKED"

# This works correctly:
echo "any_file.txt" | grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' blacklist.txt)
----

*Fix*: Filter out empty lines and comments using
`+sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d'+`

'''''

===== Bug #3: Case Sensitivity in mergefiles.txt - MINOR

*Problem*: `+mergefiles.txt+` listed `+testing.md+` but actual file is
`+TESTING.md+`.

*Symptom*: File skipped by line 40 (`+[[ -f "$SRC" ]] || continue+`).

*Fix*: Corrected filename in `+mergefiles.txt+`.

'''''

==== 📋 Testing Methodology

===== Phase 1: Basic Execution Test

[source,bash]
----
# Run with existing mergefiles.txt
bash maint/integrator.sh
----

*Expected*: Script runs without errors, processes files listed in
`+mergefiles.txt+`.

'''''

===== Phase 2: Blacklist Enforcement Test

Create comprehensive test file:

[source,bash]
----
cat > maint/mergefiles_blacklist_test.txt << 'EOF'
l4/pkg/starforth/server/Makefile
l4/pkg/starforth/server/Makefile.inc
l4/pkg/starforth/server/README.md
l4/pkg/starforth/server/src/panic.c
l4/pkg/starforth/server/src/core.c
l4/pkg/starforth/server/Control
maint/integrator.sh
EOF

# Create test file that should be quarantined
touch l4/pkg/starforth/server/src/core.c

# Clear quarantine
rm -rf maint/quarantine/*

# Swap mergefiles and run test
cp maint/mergefiles.txt maint/mergefiles.txt.backup
cp maint/mergefiles_blacklist_test.txt maint/mergefiles.txt
bash maint/integrator.sh
----

*Expected Results*: | File | Expected Action | Reason | |——|—————-|———|
| `+Makefile+` | 🔴 *BLOCKED* | Matches `+(^|/)?Makefile($|[.].*)+` | |
`+Makefile.inc+` | 🔴 *BLOCKED* | Matches `+(^|/)?Makefile($|[.].*)+` |
| `+README.md+` | 🟢 *OVERWRITE* | Not blacklisted, exists in StarForth
| | `+panic.c+` | 🔴 *BLOCKED* | Matches `+server/src/panic\.c$+` | |
`+core.c+` | 🟠 *QUARANTINE* | Not blacklisted, doesn’t exist in
StarForth | | `+Control+` | 🔴 *BLOCKED* | Matches
`+(^|/)?Control($|/)+` | | `+integrator.sh+` | 🟢 *OVERWRITE* | Not
blacklisted, exists in StarForth |

*Verify Blacklist*:

[source,bash]
----
# Test individual patterns
echo "l4/pkg/starforth/server/Makefile" | grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' maint/blacklist.txt) && echo "BLOCKED" || echo "NOT BLOCKED"

echo "l4/pkg/starforth/server/src/panic.c" | grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' maint/blacklist.txt) && echo "BLOCKED" || echo "NOT BLOCKED"
----

'''''

===== Phase 3: Path Stripping Verification

*Test*: Verify that `+l4/pkg/starforth/server/+` prefix is correctly
stripped.

[source,bash]
----
# Check where files land in destination
FILE="l4/pkg/starforth/server/README.md"
RELATIVE="${FILE#l4/pkg/starforth/server/}"
echo "Source: $FILE"
echo "Destination: $STARFORTH_ROOT/$RELATIVE"
----

*Expected*: - Source: `+l4/pkg/starforth/server/README.md+` -
Destination: `+/home/rajames/CLionProjects/StarForth/README.md+`

*Verify*:

[source,bash]
----
diff l4/pkg/starforth/server/README.md $STARFORTH_ROOT/README.md && echo "✓ Synced correctly"
----

'''''

===== Phase 4: Overwrite vs Quarantine Logic

*Rule 3*: If file exists at destination → *overwrite* (green) *Rule 4*:
If file doesn’t exist → *quarantine* (orange)

*Test Overwrite*:

[source,bash]
----
# README.md exists in StarForth root
test -f $STARFORTH_ROOT/README.md && echo "Exists → should OVERWRITE"
----

*Test Quarantine*:

[source,bash]
----
# core.c doesn't exist in StarForth
test -f $STARFORTH_ROOT/src/core.c || echo "Doesn't exist → should QUARANTINE"

# Verify it's quarantined
test -f maint/quarantine/l4/pkg/starforth/server/src/core.c && echo "✓ Correctly quarantined"
----

'''''

===== Phase 5: Auto-Generation from Git (Webhook Mode)

*Test*: Script generates `+mergefiles.txt+` from
`+git diff --name-only HEAD~1+` when file doesn’t exist.

[source,bash]
----
# Remove mergefiles.txt to trigger auto-generation
mv maint/mergefiles.txt maint/mergefiles.txt.manual

# Run script - should auto-generate
bash maint/integrator.sh

# Check what was generated
cat maint/mergefiles.txt
----

*Expected Output*:

....
[INFO] No mergefiles list found — generating diff from last commit.
....

*Verify*:

[source,bash]
----
git diff --name-only HEAD~1  # Should match mergefiles.txt content
----

'''''

==== ✅ Final Production Test

[source,bash]
----
# 1. Restore original mergefiles.txt
cp maint/mergefiles.txt.backup maint/mergefiles.txt

# 2. Clear quarantine
rm -rf maint/quarantine/*

# 3. Run production test
bash maint/integrator.sh

# 4. Verify sync
diff maint/integrator.sh $STARFORTH_ROOT/maint/integrator.sh && echo "✓ Synced"
----

*Expected Output*:

....
🛰️  Quark webhook engaged — StarshipOS → StarForth (Reverse Sync)

[Quarantining] l4/pkg/starforth/server/TESTING.md
[Overwriting] maint/integrator.sh

🧾 First Officer's Reverse Report:
  • Destination base: /home/rajames/CLionProjects/StarForth
  • Quarantine:       /home/rajames/CLionProjects/StarshipOS/maint/quarantine
  • Merge list:       2 files processed

Awaiting Cappy's signature: Captain Bob ✍️
....

'''''

==== 🔧 Debugging Commands Reference

===== Check Blacklist Patterns

[source,bash]
----
# View blacklist with line numbers
cat -n maint/blacklist.txt

# Test specific file against blacklist
FILE="l4/pkg/starforth/server/Makefile"
grep -Eq -f <(sed '/^[[:space:]]*$/d; /^[[:space:]]*#/d' maint/blacklist.txt) <<< "$FILE" && echo "BLOCKED" || echo "NOT BLOCKED"
----

===== Verify Path Stripping

[source,bash]
----
# For StarshipOS → StarForth
FILE="l4/pkg/starforth/server/src/vm.c"
RELATIVE="${FILE#l4/pkg/starforth/server/}"
echo "$RELATIVE"  # Should output: src/vm.c

# For StarForth → StarshipOS (reverse)
FILE="server/src/vm.c"
RELATIVE="${FILE#server/}"
echo "$RELATIVE"  # Should output: src/vm.c
----

===== Check File Existence

[source,bash]
----
# Check source
ls -la l4/pkg/starforth/server/TESTING.md

# Check destination
ls -la $STARFORTH_ROOT/TESTING.md

# Check quarantine
ls -la maint/quarantine/l4/pkg/starforth/server/TESTING.md
----

===== Monitor Git Index Refresh

[source,bash]
----
# Manual refresh (what script does)
cd $STARFORTH_ROOT && git update-index --really-refresh
----

'''''

==== 🎯 Apply to StarForth → StarshipOS Script

Use the same methodology to test `+StarForth/maint/integrator.sh+`:

*Key Differences*: 1. *Path stripping*: `+server/+` →
`+l4/pkg/starforth/server/+` 2. *Direction*: StarForth → StarshipOS 3.
*Blacklist patterns*: Different (StarForth-specific patterns)

*Same Fixes Needed*: - Change `+grep -Fqx+` → `+grep -Eq+` - Add `+sed+`
filter for empty lines and comments - Verify case sensitivity in
mergefiles.txt

*Test Pattern*:

[source,bash]
----
cd /home/rajames/CLionProjects/StarForth

# Phase 1: Basic test
bash maint/integrator.sh

# Phase 2: Blacklist test (create test mergefiles)
# Phase 3: Path stripping test
# Phase 4: Overwrite vs quarantine test
# Phase 5: Auto-generation test
----

'''''

==== 📊 Test Results Summary

[width="100%",cols="51%,26%,23%",options="header",]
|===
|Test Category |Status |Notes
|Basic Execution |✅ PASS |Script runs without errors

|Blacklist Enforcement |✅ PASS |Regex patterns work, empty lines
filtered

|Path Stripping |✅ PASS |`+l4/pkg/starforth/server/+` removed correctly

|Overwrite Logic |✅ PASS |Existing files overwritten (README.md, vm.h)

|Quarantine Logic |✅ PASS |New files quarantined (TESTING.md, core.c)

|Auto-Generation |✅ PASS |Generates from `+git diff HEAD~1+`

|Git Index Refresh |✅ PASS |StarForth index updated
|===

*Final Status*: 🚀 *PRODUCTION READY* for git push webhook

'''''

_Debugged: 2025-10-19_ _Captain Bob ✍️_
