- Add Sprint 3 plan (37 steps, global 39-75) with canonical memory annotations, test quality requirements, global step numbering, and Sprint 2 overlap notes - Refactor all docs to use canonical annotation families (@Deallocate, @Lifetime, @Reclaim, @Owner, @Allocate) replacing simplified @deref 4-strategy system - Replace @perf with canonical @Hot/@Cold, @Inline, @Pure from annotations/6 optimization - Replace @memory-footprint, @execution-mode, @deref-explicit with canonical equivalents - Update REQUIREMENTS_OVERVIEW, SPRINT_1_REQUIREMENTS, SPRINT_2_PLAN, SPRINT_2_VISION, C++ Implementation Roadmap, example files, and progress report - Remove duplicate bonus steps 41-42, consolidate Phase 3h from 7 to 4 steps Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
425 lines
26 KiB
Markdown
425 lines
26 KiB
Markdown
# Sprint 3 Plan: Whetstone Editor - Core Functionality & Classical Mode
|
||
|
||
37 steps (global steps 39–75). Build one thing, test it, move on. If something breaks, the problem is in that step.
|
||
|
||
> **Step numbering:** Sprint 2 ended at Step 38. Sprint 3 steps are numbered 39–75 to continue the global sequence. Phase-local references (e.g., "Phase 3a Step 1") map to global Step 39, etc.
|
||
|
||
For detailed architecture and design rationale, see:
|
||
- `REQUIREMENTS_OVERVIEW.md` — Core vision, SemAnno schema, memory strategies
|
||
- `annotations/Memory strategy.md` — **Canonical memory annotation reference** (10 strategies across 20 languages)
|
||
- `annotations/6 optimization and intent.md` — **Canonical optimization annotation reference** (`@Hot`/`@Cold`, `@Inline`, `@Pure`, etc.)
|
||
- `annotations/8 strategy choice and policy.md` — Policy annotations and strategy menus
|
||
- `annotations/C++ Implementation Roadmap.md` — C++ generator and memory management details
|
||
|
||
> **Note on annotation naming:** Sprint 2 used a simplified 4-strategy `@deref` system from `REQUIREMENTS_OVERVIEW.md`. Sprint 3 adopts the **canonical annotation system** defined in `annotations/Memory strategy.md`, which is the authoritative source. The canonical system uses distinct annotation families — `@Deallocate`, `@Lifetime`, `@Reclaim`, `@Owner`, `@Allocate` — each targeting a specific memory paradigm. The old `DerefStrategy` class will be refactored into these granular annotation types. See the Migration Notes at the bottom for the full mapping.
|
||
|
||
> **Test quality requirement:** All step tests MUST contain real assertions that exercise the code under test. Sprint 2 tests were placeholder stubs that print "PASS" without verifying behavior. Sprint 3 tests must `#include` the relevant headers, construct real AST objects, call real functions, and assert expected outputs. A test that just prints "PASS" is not a test.
|
||
|
||
---
|
||
|
||
## Canonical Memory Annotation Reference
|
||
|
||
From `annotations/Memory strategy.md` — these are the annotations Sprint 3 implements:
|
||
|
||
| Annotation | Paradigm | Description | Key Languages |
|
||
|---|---|---|---|
|
||
| `@Deallocate(Explicit)` | Manual (MM) | Explicit allocation/deallocation, direct `free()`/`delete` | C, C++ (manual mode), Zig |
|
||
| `@Lifetime(RAII)` | RAII | Destructor-based cleanup at scope end | C++ |
|
||
| `@Reclaim(Tracing)` | GC | Automatic runtime reclamation via tracing/mark-sweep | Python, Java, JS, Go, C#, Ruby |
|
||
| `@Reclaim(Cycle)` | GC | Cycle-detection garbage collection | PHP |
|
||
| `@Reclaim(Escape)` | GC | Escape-analysis-based heap vs stack decision | Go |
|
||
| `@Owner(Single)` | Ownership | Compile-time single-owner lifetime tracking (Rust-like) | Rust |
|
||
| `@Owner(Shared_ARC)` | ARC | Deterministic reference counting | Swift, Objective-C |
|
||
| `@Allocate(Static)` | Static | Fixed memory buffers, no dynamic allocation | Fortran |
|
||
| `@Allocate(Register)` | Register | Direct register mapping or stack spill | Assembly |
|
||
| `@Allocate(Allocator)` | Allocator | Explicit allocator parameter threading | Zig |
|
||
|
||
From `annotations/6 optimization and intent.md` — optimization annotations:
|
||
|
||
| Annotation | Purpose |
|
||
|---|---|
|
||
| `@Hot` / `@Cold` | Profile-guided branch prediction hints → C++ `[[likely]]`/`[[unlikely]]` |
|
||
| `@Inline(Always\|Never\|Hint)` | Function inlining control |
|
||
| `@Pure` | No side effects (referential transparency) |
|
||
| `@TailCall` | Tail call optimization eligible |
|
||
| `@Loop(Unroll, N)` / `@Loop(Vectorize)` / `@Loop(Fuse)` | Loop transformation hints |
|
||
| `@Data(Prefetch)` / `@Data(Restrict)` | Memory locality hints |
|
||
| `@Align(N)` / `@Pack` / `@ConstExpr` | Hardware layout and compile-time evaluation |
|
||
|
||
---
|
||
|
||
## Phase 3a: C++ Generator Implementation
|
||
|
||
Implement the C++ code generator with proper memory management strategy handling. Currently `Orchestrator::saveFile()` falls back to `PythonGenerator` for `.cpp` files with a `// TODO: Implement CppGenerator when ready` — this phase fills that gap.
|
||
|
||
### Step 39: Basic C++ generator skeleton
|
||
- Create `CppGenerator` class inheriting from `ProjectionGenerator` in `Generator.h`
|
||
- Implement basic concept mappings: Module → `namespace`, Function → function signature with return type
|
||
- Wire into `Orchestrator::saveFile()` to replace the PythonGenerator fallback
|
||
- Test: `generate(module)` → basic C++ skeleton with correct function signatures; assert output contains `namespace`, `{`, `}`
|
||
|
||
### Step 40: Statement generation for C++
|
||
- Implement Assignment → `target = value;`, Return → `return value;`, IfStatement → `if (...) {...} else {...}`
|
||
- Handle C++-specific syntax: semicolons, braces, no colons
|
||
- Test: AST with assignments/returns/if → valid C++ code; assert semicolons present, braces balanced
|
||
|
||
### Step 41: Expression generation for C++
|
||
- Implement BinaryOperation → `left op right`, VariableReference → `varName`, Literals → C++ equivalents
|
||
- Handle operator precedence with parentheses where needed
|
||
- Test: AST with nested expressions → valid C++ expressions; assert parenthesization is correct
|
||
|
||
### Step 42: Type generation for C++
|
||
- Implement PrimitiveType → `int`, `double`, `bool`, `std::string`, etc.
|
||
- Implement complex types: ListType → `std::vector<T>`, MapType → `std::map<K,V>`, SetType → `std::unordered_set<T>`, OptionalType → `std::optional<T>`
|
||
- Test: AST with typed parameters → correct C++ type declarations
|
||
|
||
### Step 43: Memory strategy code generation
|
||
- Refactor Sprint 2's `DerefStrategy` class into the canonical annotation types:
|
||
- `@Deallocate(Explicit)` → raw pointers with explicit `new`/`delete`; flag missing deallocation points as "Missing Intent" errors
|
||
- `@Lifetime(RAII)` → `std::unique_ptr<T>`, RAII destructors, move semantics; inject destructor calls at scope end
|
||
- `@Reclaim(Tracing)` → `std::shared_ptr<T>` for GC-like reference counting; no explicit deallocation needed
|
||
- `@Owner(Single)` → `std::unique_ptr<T>` with strict single-ownership enforcement; reject aliasing at compile time
|
||
- `@Owner(Shared_ARC)` → `std::shared_ptr<T>` with deterministic ref-counting semantics
|
||
- Create new annotation classes: `DeallocateAnnotation`, `LifetimeAnnotation`, `ReclaimAnnotation`, `OwnerAnnotation`, `AllocateAnnotation`
|
||
- Keep `DerefStrategy` as deprecated wrapper that maps to the new types during transition
|
||
- Test: AST with each annotation type → C++ with correct memory management; `@Deallocate(Explicit)` produces raw pointers, `@Lifetime(RAII)` produces `unique_ptr`, `@Reclaim(Tracing)` produces `shared_ptr`
|
||
|
||
### Step 44: C++-specific idioms
|
||
- FunctionCall → function calls with proper argument passing (by value, by reference, by const reference)
|
||
- Handle C++-specific features: `const` correctness, `&` references, `&&` move references
|
||
- LangSpecific annotation for C++ → emit `#include` directives, `using namespace` declarations
|
||
- Test: AST → idiomatic C++ code; assert `const`, references appear where annotated
|
||
|
||
> **CHECKPOINT:** C++ generator produces valid, compilable code with correct memory management for all canonical annotation types. Stop here until this passes.
|
||
|
||
---
|
||
|
||
## Phase 3b: Tree-sitter Integration (Replace Placeholders)
|
||
|
||
Sprint 2 Steps 30-34 created placeholder tree-sitter integration in `Parser.h` with stub implementations. This phase replaces those stubs with real tree-sitter C bindings and proper CST-to-AST mapping.
|
||
|
||
### Step 45: Tree-sitter Python integration
|
||
- Link with `tree-sitter` and `tree-sitter-python` C libraries via CMake FetchContent
|
||
- Replace `TreeSitterParser::parsePython()` stub with real implementation
|
||
- Map Python CST nodes to SemAnno AST: `function_definition` → Function, `assignment` → Assignment, etc.
|
||
- Auto-annotate with `@Reclaim(Tracing)` since Python uses tracing GC
|
||
- Test: `parsePython("def f(x): return x + 1")` → AST with Function(name="f"), Parameter(name="x"), Return with BinaryOperation; memory annotation is `@Reclaim(Tracing)`
|
||
|
||
### Step 46: Tree-sitter C++ integration
|
||
- Link with `tree-sitter-cpp` library
|
||
- Replace `TreeSitterParser::parseCpp()` stub with real implementation
|
||
- Map C++ CST nodes: `function_definition` → Function, `declaration` → Variable, `if_statement` → IfStatement
|
||
- Detect memory patterns: `unique_ptr` → `@Lifetime(RAII)`, `shared_ptr` → `@Owner(Shared_ARC)`, raw `new/delete` → `@Deallocate(Explicit)`
|
||
- Test: `parseCpp("int f(int x) { return x + 1; }")` → correct AST; `std::unique_ptr<int> p` → `@Lifetime(RAII)` annotation
|
||
|
||
### Step 47: Tree-sitter Elisp integration
|
||
- Link with `tree-sitter-elisp` library
|
||
- Replace `TreeSitterParser::parseElisp()` stub with real implementation
|
||
- Map Elisp CST nodes: `defun` → Function, `defvar` → Variable, `if` → IfStatement
|
||
- Auto-annotate with `@Reclaim(Tracing)` (Elisp uses GC)
|
||
- Test: `parseElisp("(defun f (x) (+ x 1))")` → correct AST matching Python/C++ equivalents
|
||
|
||
### Step 48: CST to AST mapping refinement
|
||
- Handle complex constructs: nested functions, classes/structs, lambdas, comprehensions
|
||
- Preserve source location info (line/column) on AST nodes for error reporting
|
||
- Map language-specific constructs to nearest SemAnno equivalent with `LangSpecific` annotations for lossless round-trip
|
||
- Test: Complex multi-function Python/C++/Elisp → correct SemAnno AST with source locations
|
||
|
||
### Step 49: Error recovery and diagnostics
|
||
- Handle malformed source gracefully (partial parse, not crash)
|
||
- Report parse errors with line/column information and error node markers in AST
|
||
- Test: Invalid syntax → ParseResult with errors list and partial AST; assert no crashes, assert error positions correct
|
||
|
||
> **CHECKPOINT:** All three languages parse correctly to SemAnno AST via real tree-sitter. `parsePython` → AST → `PythonGenerator` produces semantically equivalent output. Stop here until this passes.
|
||
|
||
---
|
||
|
||
## Phase 3c: Classical Editing Mode
|
||
|
||
Add traditional text editing alongside the existing structured editing in the ImGui shell.
|
||
|
||
### Step 50: Text editor component
|
||
- Add raw text editor pane (ImGui multiline text input or integrate ImGuiColorTextEdit) alongside structured editor
|
||
- Toggle between structured view and classical text view
|
||
- Test: Can type text in classical mode; text persists across view switches
|
||
|
||
### Step 51: Text → AST synchronization
|
||
- When text changes, re-parse via tree-sitter and update the in-memory AST
|
||
- When AST changes (via structured editing or agent API), regenerate text and update the text pane
|
||
- Debounce re-parsing to avoid thrashing on every keystroke
|
||
- Test: Edit text → AST updates within debounce window; edit AST via structured editor → text updates immediately
|
||
|
||
### Step 52: Syntax highlighting
|
||
- Use tree-sitter's incremental parsing for syntax highlighting in the text pane
|
||
- Highlight keywords, strings, comments, types, function names with distinct colors
|
||
- Test: Python/C++/Elisp code highlighted correctly; assert color spans match tree-sitter node types
|
||
|
||
### Step 53: Classical editing operations
|
||
- Copy/paste, undo/redo, find/replace in text mode
|
||
- Text undo/redo integrates with orchestrator's operation journal (text edit → AST mutation → journal entry)
|
||
- Test: Ctrl+Z in text mode undoes both text and AST; Ctrl+F finds and replaces text
|
||
|
||
### Step 54: Emacs-style keybindings
|
||
- Ctrl+S → save, Ctrl+X Ctrl+F → open file, Ctrl+G → cancel, M-x → command palette
|
||
- Keybinding layer configurable (Emacs mode vs. standard mode)
|
||
- Test: Emacs keybindings trigger correct orchestrator RPCs
|
||
|
||
> **CHECKPOINT:** Classical text editing mode works with bidirectional AST sync. Users can freely switch between structured and text editing. Stop here until this passes.
|
||
|
||
---
|
||
|
||
## Phase 3d: Emacs Integration (Complete Placeholders)
|
||
|
||
Sprint 2 Steps 26-29 built Emacs daemon spawning, `sendToEmacs()`, `loadFile()`, and `saveFile()` in `Orchestrator.h`. This phase completes the integration with a proper splash screen, robust command routing, and real buffer management (Sprint 2 implementations use basic `emacsclient -e` calls without error handling or multi-buffer tracking).
|
||
|
||
### Step 55: Emacs splash screen
|
||
- Create `whetstone-splash.el` that displays on Emacs daemon startup
|
||
- Show Whetstone version, available commands, recent files, and keybinding cheat sheet
|
||
- Test: `startEmacsDaemon()` → Emacs buffer contains splash content; assert buffer name is `*Whetstone*`
|
||
|
||
### Step 56: Robust command integration
|
||
- Replace raw `sendToEmacs()` string concatenation with proper Elisp command builder (escape special chars, handle errors)
|
||
- `M-x whetstone-find-file` → `loadFile()` RPC, `C-x C-s` → `saveFile()` RPC
|
||
- Error handling: if Emacs daemon dies, detect and restart automatically
|
||
- Test: Emacs commands trigger orchestrator RPCs; simulate daemon crash → auto-restart
|
||
|
||
### Step 57: Buffer management
|
||
- Track open buffers in orchestrator (map of path → buffer state)
|
||
- Switch between buffers, save/close individual buffers
|
||
- Sync buffer list between Emacs and ImGui file tree
|
||
- Test: Open 3 files → all appear in buffer list; close one → removed from list; switch → correct content displayed
|
||
|
||
### Step 58: Mode-specific behavior
|
||
- `whetstone-python-mode`, `whetstone-cpp-mode`, `whetstone-elisp-mode` with language-appropriate features
|
||
- Mode activates based on file extension; sets tree-sitter parser and generator accordingly
|
||
- Test: Open `.py` → Python mode active; open `.cpp` → C++ mode active; verify correct generator used
|
||
|
||
> **CHECKPOINT:** Emacs integration is robust with splash screen, error recovery, multi-buffer management, and language-aware modes. Stop here until this passes.
|
||
|
||
---
|
||
|
||
## Phase 3e: Agent API (Extend Existing)
|
||
|
||
Sprint 2 Steps 34-38 built basic Agent API with RPC-based `insertNode`, `deleteNode`, `setProperty`, and `insertSubtree` in the orchestrator. This phase adds a proper network transport (WebSocket), authentication, and richer query capabilities for external AI agents.
|
||
|
||
### Step 59: WebSocket agent endpoint
|
||
- Add WebSocket server (via WebSocket++ or Boost.Beast) alongside existing stdin/stdout JSON-RPC
|
||
- Agents connect via `ws://localhost:PORT/agent` and send JSON-RPC messages
|
||
- Session management: track connected agents, assign session IDs
|
||
- Test: Agent connects via WebSocket, sends `ping`, receives `pong`; assert session ID assigned
|
||
|
||
### Step 60: AST query API
|
||
- `getAST(nodeId)` → return AST subtree as JSON (already exists as `getAST` RPC — extend with depth limit and filtering)
|
||
- `findNodes(pattern)` → pattern-based node search (by concept type, property values, annotation presence)
|
||
- `getSubtree(nodeId, depth)` → return limited-depth subtree for large ASTs
|
||
- Test: Agent queries return correct AST data; `findNodes({concept: "Function"})` returns all functions
|
||
|
||
### Step 61: AST mutation API (extend)
|
||
- Extend existing `insertNode`/`deleteNode`/`setProperty` with validation and lock checking
|
||
- `updateNode(nodeId, properties)` → bulk property update with journal recording
|
||
- All mutations check memory annotation consistency and `OptimizationLock` warnings
|
||
- Test: Agent mutations update AST correctly; locked node mutation produces warning (not rejection)
|
||
|
||
### Step 62: Context API
|
||
- `getInScopeSymbols(nodeId)` → return variables/functions/parameters visible at that AST position
|
||
- `getCallHierarchy(functionId)` → return callers and callees across the module
|
||
- `getDependencyGraph(nodeId)` → return data flow dependencies
|
||
- Test: Agent gets correct scope at different AST positions; call hierarchy matches actual function calls
|
||
|
||
### Step 63: Batch operations (extend)
|
||
- Extend existing `insertSubtree()` with full transactional `applySequence(mutations[])` → atomic all-or-nothing
|
||
- On failure mid-sequence, roll back all preceding mutations in the batch
|
||
- Test: Batch of 5 mutations applies atomically; batch with error at step 3 → all 5 rolled back
|
||
|
||
> **CHECKPOINT:** Agents can connect via WebSocket, query AST with rich patterns, mutate with validation, get scope/call context, and apply atomic batches. Stop here until this passes.
|
||
|
||
---
|
||
|
||
## Phase 3f: Advanced Memory Management
|
||
|
||
Enhanced memory strategy annotations using the canonical system from `annotations/Memory strategy.md`. Implements validation, inference, cross-language projection, and optimization hints.
|
||
|
||
### Step 64: Memory annotation validation
|
||
- Validate all canonical memory annotations for consistency:
|
||
- `@Owner(Single)` on a variable that is aliased → error
|
||
- `@Deallocate(Explicit)` without a corresponding deallocation point → "Missing Intent" error (the Unfilled Node Constraint)
|
||
- `@Reclaim(Tracing)` on a real-time-critical path with `@Policy(Perf: Critical)` → warning
|
||
- Conflicting annotations on parent/child (e.g., parent `@Owner(Single)`, child `@Owner(Shared_ARC)`) → error
|
||
- Test: Invalid/conflicting annotations flagged with specific error messages; valid annotations pass
|
||
|
||
### Step 65: Memory strategy inference
|
||
- Infer appropriate annotation from usage patterns (per `Memory strategy.md` Section 2):
|
||
- Python source → `@Reclaim(Tracing)` (ref counting + cycle detection)
|
||
- C++ `unique_ptr` patterns → `@Lifetime(RAII)` or `@Owner(Single)`
|
||
- C++ `shared_ptr` patterns → `@Owner(Shared_ARC)`
|
||
- C raw `malloc`/`free` → `@Deallocate(Explicit)`
|
||
- Rust ownership patterns → `@Owner(Single)`
|
||
- Immutable data with no mutation → `@Allocate(Static)` candidate
|
||
- Surface inferred annotations in the IDE as suggestions (not auto-applied)
|
||
- Test: System suggests correct strategies based on usage analysis; Python function → `@Reclaim(Tracing)` suggested
|
||
|
||
### Step 66: Cross-language memory projection
|
||
- Implement the "Lossless Projection Logic" from `Memory strategy.md` Section 3:
|
||
- **High → Low** (Python → C): `@Reclaim(Tracing)` → inject `INC_REF`/`DEC_REF` shim with `free()` on zero
|
||
- **Low → High** (C → Java): `@Deallocate(Explicit)` → wrap in `try-with-resources` or `Cleaner` API
|
||
- **Strict → Permissive** (Rust → Python): `@Owner(Single)` → preserve in AST metadata but don't enforce in generated Python
|
||
- **C++ specifics**: `@Lifetime(RAII)` → inject destructor calls at scope end; `@Owner(Shared_ARC)` → `std::shared_ptr`
|
||
- Preserve all annotations through round-trips (annotation survives even if target language doesn't enforce it)
|
||
- Test: Python module with `@Reclaim(Tracing)` → C++ with ref-counting shim → back to Python with annotation intact
|
||
|
||
### Step 67: Optimization annotations
|
||
- Implement annotations from `annotations/6 optimization and intent.md`:
|
||
- `@Hot` / `@Cold` → C++ `[[likely]]`/`[[unlikely]]`, `__attribute__((hot))` / `__attribute__((cold))`
|
||
- `@Inline(Always|Never|Hint)` → C++ `[[gnu::always_inline]]` / `[[gnu::noinline]]`
|
||
- `@Pure` → C++ `[[gnu::pure]]` or `[[nodiscard]]`; enable aggressive constant folding
|
||
- `@ConstExpr` → C++ `constexpr`
|
||
- Integrate with code generation for all three target languages
|
||
- Test: `@Hot` function → C++ output contains `__attribute__((hot))`; `@Pure` function → `[[nodiscard]]`; `@ConstExpr` → `constexpr`
|
||
|
||
> **CHECKPOINT:** Memory annotations are validated, inferred, projected across languages per the canonical spec, and optimization hints influence code generation. Stop here until this passes.
|
||
|
||
---
|
||
|
||
## Phase 3g: Optimization Pipeline
|
||
|
||
Transformation engine that operates on the AST respecting memory annotations and optimization locks.
|
||
|
||
### Step 68: Transformation engine
|
||
- AST transformation framework: define transforms as AST → AST functions with pre/post conditions
|
||
- Built-in transforms: constant folding, dead code elimination, loop invariant hoisting
|
||
- Each transform checks `OptimizationLock` before modifying (warn, don't reject)
|
||
- Respect `@Loop` annotations: `@Loop(Vectorize)` → don't break iteration independence; `@Loop(Fuse)` → attempt adjacent loop fusion
|
||
- Test: Constant folding simplifies `3 + 4` → `7` in AST; locked node emits warning but still transforms
|
||
|
||
### Step 69: Strategy-aware optimization
|
||
- Transforms respect memory annotations:
|
||
- `@Owner(Single)` → can inline/move but not duplicate ownership (no aliasing)
|
||
- `@Deallocate(Explicit)` → cannot reorder around deallocation points
|
||
- `@Reclaim(Tracing)` → can freely restructure (GC handles cleanup)
|
||
- `@Allocate(Static)` → cannot dynamically allocate; optimize for fixed buffers
|
||
- Respect `@Data` annotations: `@Data(Restrict)` allows more aggressive alias analysis
|
||
- Test: Optimization of `@Owner(Single)` variable doesn't create aliasing; `@Reclaim(Tracing)` enables free restructuring
|
||
|
||
### Step 70: Cross-strategy validation
|
||
- After optimization, verify memory annotation invariants still hold:
|
||
- `@Deallocate(Explicit)`: no use-after-free, no leaked allocations
|
||
- `@Owner(Single)`: no double-move, no aliasing
|
||
- `@Owner(Shared_ARC)`: no circular references without weak refs
|
||
- `@Lifetime(RAII)`: destructor reachable on all code paths
|
||
- Emit diagnostics with AST node references
|
||
- Test: Intentionally break invariants → validator catches each violation type
|
||
|
||
### Step 71: Incremental optimization with rollback
|
||
- Apply optimizations incrementally, recording each transform in the operation journal
|
||
- Undo individual transforms or entire optimization passes
|
||
- Track provenance: each optimized node knows which transform created it
|
||
- Test: Apply 3 transforms → undo middle one → AST reflects only transforms 1 and 3
|
||
|
||
> **CHECKPOINT:** Optimization pipeline transforms AST safely, respects locks and memory annotations, validates invariants, and supports incremental rollback. Stop here until this passes.
|
||
|
||
---
|
||
|
||
## Phase 3h: Integration & Validation
|
||
|
||
Full system integration testing and documentation. Consolidated from original 7 steps to 4 focused steps.
|
||
|
||
### Step 72: End-to-end pipeline test
|
||
- Full workflow: Python file → tree-sitter parse → AST → optimize → C++ generation → compile with g++/clang
|
||
- Verify: generated C++ compiles without errors and produces correct output
|
||
- Round-trip: Python → AST → C++ → AST → Python preserves semantics and all memory annotations
|
||
- Test: `Calculator.py` → parse → optimize → `Calculator.cpp` → compile → run → correct output; round-trip preserves annotations
|
||
|
||
### Step 73: Error handling and edge cases
|
||
- Graceful handling of all error conditions across the full pipeline
|
||
- Edge cases: empty modules, deeply nested ASTs, circular references, very large files
|
||
- Error reporting with source location, AST node context, and actionable messages
|
||
- Test: Each error condition → meaningful error message (not crash); recovery to valid state
|
||
|
||
### Step 74: Performance profiling and benchmarks
|
||
- Profile bottlenecks: parsing time, generation time, optimization time, UI responsiveness
|
||
- Benchmark generated code vs. hand-written equivalents
|
||
- Set baseline metrics for future regression testing
|
||
- Test: Parse/generate cycle completes in < 100ms for typical module; UI stays responsive during background operations
|
||
|
||
### Step 75: API documentation
|
||
- Document all RPC methods (JSON-RPC + WebSocket agent API) with request/response schemas
|
||
- Document all canonical memory annotations with examples for each language target
|
||
- Document optimization transforms with before/after AST examples
|
||
- Test: Every public RPC method has a documented example that can be copy-pasted and run
|
||
|
||
> **CHECKPOINT:** Complete system validated end-to-end. All features work together. Documentation complete. Stop here — Sprint 3 is done.
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
| Phase | Global Steps | What You Have When Done |
|
||
|-------|-------------|------------------------|
|
||
| 3a: C++ Generator | 39–44 | C++ code generator with canonical memory annotation support |
|
||
| 3b: Tree-sitter (replace stubs) | 45–49 | Real parsing for Python, C++, and Elisp via tree-sitter C bindings |
|
||
| 3c: Classical Mode | 50–54 | Traditional text editing with bidirectional AST sync |
|
||
| 3d: Emacs Integration (complete) | 55–58 | Robust Emacs integration with splash screen, error recovery, multi-buffer |
|
||
| 3e: Agent API (extend) | 59–63 | WebSocket-based agent API with rich queries, validation, atomic batches |
|
||
| 3f: Memory Management | 64–67 | Canonical memory annotation validation, inference, cross-language projection, `@Hot`/`@Cold`/`@Inline`/`@Pure` |
|
||
| 3g: Optimization | 68–71 | Strategy-aware optimization pipeline with incremental rollback |
|
||
| 3h: Integration | 72–75 | End-to-end validation, profiling, and documentation |
|
||
|
||
## Dependencies
|
||
|
||
| Dependency | Version | Purpose |
|
||
|------------|---------|---------|
|
||
| tree-sitter | 0.22+ | Parsing Python, C++, and Elisp (C library) |
|
||
| tree-sitter-python | latest | Python grammar |
|
||
| tree-sitter-cpp | latest | C++ grammar |
|
||
| tree-sitter-elisp | latest | Elisp grammar |
|
||
| Dear ImGui | 1.90+ | GUI rendering |
|
||
| SDL2 | latest | Window/input backend |
|
||
| WebSocket++ | latest | Agent API network transport |
|
||
| spdlog | latest | Logging |
|
||
|
||
## Migration Notes
|
||
|
||
### Sprint 2 `DerefStrategy` → Sprint 3 Canonical Annotations
|
||
|
||
Sprint 2 used a single `DerefStrategy` class with a `strategy` string field. Sprint 3 replaces this with the canonical annotation classes from `annotations/Memory strategy.md`:
|
||
|
||
| Sprint 2 (`DerefStrategy.strategy`) | Sprint 3 Annotation Class | C++ Code Generation |
|
||
|---|---|---|
|
||
| `"imperative"` | `DeallocateAnnotation(Explicit)` | Raw pointers, explicit `new`/`delete` |
|
||
| `"streamed"` | `LifetimeAnnotation(RAII)` + `OwnerAnnotation(Single)` | `unique_ptr`, RAII destructors, move semantics |
|
||
| `"batched"` | `ReclaimAnnotation(Tracing)` + `OwnerAnnotation(Shared_ARC)` | `shared_ptr`, reference counting |
|
||
| `"content-addressed"` | `AllocateAnnotation(Static)` + `@ConstExpr` | `const` immutable objects, compile-time where possible |
|
||
|
||
### New annotation classes to implement in Step 43:
|
||
|
||
```
|
||
DeallocateAnnotation — strategy: "Explicit"
|
||
LifetimeAnnotation — strategy: "RAII"
|
||
ReclaimAnnotation — strategy: "Tracing" | "Cycle" | "Escape"
|
||
OwnerAnnotation — strategy: "Single" | "Shared_ARC"
|
||
AllocateAnnotation — strategy: "Static" | "Register" | "Allocator"
|
||
```
|
||
|
||
### Backward compatibility
|
||
|
||
- Keep `DerefStrategy` as a deprecated typedef/wrapper that maps old strategy strings to new annotation types
|
||
- Serialization: read both old `"DerefStrategy"` JSON nodes and new annotation names
|
||
- Generators: accept both old and new annotations, emit only new annotation names in output
|
||
- Sprint 2 test files are not modified (they use the old naming and are stubs anyway)
|
||
|
||
### Optimization annotation classes to implement in Step 67:
|
||
|
||
From `annotations/6 optimization and intent.md`:
|
||
|
||
```
|
||
HotColdAnnotation — hint: "Hot" | "Cold"
|
||
InlineAnnotation — mode: "Always" | "Never" | "Hint"
|
||
PureAnnotation — (no parameters)
|
||
TailCallAnnotation — (no parameters)
|
||
LoopAnnotation — transform: "Unroll(N)" | "Vectorize" | "Fuse"
|
||
DataAnnotation — hint: "Prefetch" | "Restrict"
|
||
AlignAnnotation — bytes: N
|
||
PackAnnotation — (no parameters)
|
||
ConstExprAnnotation — (no parameters)
|
||
```
|