diff --git a/docs/BuildCacheExample.md b/docs/BuildCacheExample.md index 9b452f3..99eaf6c 100644 --- a/docs/BuildCacheExample.md +++ b/docs/BuildCacheExample.md @@ -2,7 +2,7 @@ ``` Function: getOrCreate - @deref(???) + @{MemoryAnnotation} // One of: @Deallocate(Explicit), @Lifetime(RAII), @Reclaim(Tracing), @Owner(Single), etc. Parameter: cache -> Map Parameter: key -> string Body: diff --git a/docs/ProcessRecordsExample.md b/docs/ProcessRecordsExample.md index f69b7c4..3a13394 100644 --- a/docs/ProcessRecordsExample.md +++ b/docs/ProcessRecordsExample.md @@ -2,7 +2,7 @@ ``` Function: processRecords - @deref(???) // Strategy applied here + @{MemoryAnnotation} // One of: @Deallocate(Explicit), @Lifetime(RAII), @Reclaim(Tracing), @Owner(Single), etc. Parameter: records -> List Body: ForLoop: diff --git a/docs/REQUIREMENTS_OVERVIEW.md b/docs/REQUIREMENTS_OVERVIEW.md index b75fd98..a9c070b 100644 --- a/docs/REQUIREMENTS_OVERVIEW.md +++ b/docs/REQUIREMENTS_OVERVIEW.md @@ -80,15 +80,17 @@ Calculates the "blast radius" of changes. | `@fragility` | low/high | Load-bearing legacy code flag | ### 3. Performance & Resource ("Machinist" System) -Low-level optimization metadata. +Low-level optimization metadata. See `annotations/6 optimization and intent.md` for full details. | Annotation | Type | Purpose | |------------|------|---------| -| `@optimization-priority` | low/mid/high | Generator focus for SIMD/unrolling | -| `@latency-critical` | bool | Real-time execution requirement | -| `@memory-footprint` | estimate | Heap/stack allocation intent | -| `@execution-mode` | deterministic/probabilistic | Formal verification hint | -| `@deref-explicit` | bool | Every pointer touch must be accounted | +| `@Hot` / `@Cold` | hint | Profile-guided branch prediction → C++ `[[likely]]`/`[[unlikely]]` | +| `@Inline(Always\|Never\|Hint)` | mode | Function inlining control | +| `@Pure` | flag | No side effects (referential transparency) | +| `@Loop(Unroll,N)` / `@Loop(Vectorize)` / `@Loop(Fuse)` | transform | Loop transformation hints | +| `@Data(Prefetch)` / `@Data(Restrict)` | hint | Memory locality / alias analysis | +| `@Align(N)` / `@Pack` / `@ConstExpr` | layout | Hardware alignment, packing, compile-time evaluation | +| `@Policy(Perf: Critical)` | guardrail | Prioritizes zero-cost abstractions over safety wrappers | ### 4. Architectural Intent ("Paradigm" System) Defines behavioral contracts. @@ -139,19 +141,31 @@ Preserves language-specific features that don't map directly to other languages. --- -## Memory Dereferencing Strategies +## Memory Management Annotations -Memory management is decoupled from logic as a first-class AST field. +Memory management is decoupled from logic as a first-class AST field. The canonical annotation system (defined in `annotations/Memory strategy.md`) uses distinct annotation families, each targeting a specific memory paradigm: -| Strategy | Description | Generator Responsibility | -|----------|-------------|-------------------------| -| `@deref(imperative)` | Manual control - developer defines time/location | Whetstone Generator / Substrate | -| `@deref(streamed)` | Ownership-based - lifetime tracked at declaration (Rust-like) | Whetstone Generator | -| `@deref(batched)` | Garbage collected - background process or ref-counting | Whetstone Generator | -| `@deref(content-addressed)` | Immutable - identity by hash, location irrelevant | Whetstone Generator | +| Annotation | Paradigm | Description | Key Languages | +|---|---|---|---| +| `@Deallocate(Explicit)` | Manual (MM) | Explicit allocation/deallocation, direct `free()`/`delete` | C, C++ (manual mode), Zig | +| `@Lifetime(RAII)` | RAII | Destructor-based cleanup at scope end | C++ | +| `@Reclaim(Tracing)` | GC | Automatic runtime reclamation via tracing/mark-sweep | Python, Java, JS, Go, C#, Ruby | +| `@Reclaim(Cycle)` | GC | Cycle-detection garbage collection | PHP | +| `@Reclaim(Escape)` | GC | Escape-analysis-based heap vs stack decision | Go | +| `@Owner(Single)` | Ownership | Compile-time single-owner lifetime tracking (Rust-like) | Rust | +| `@Owner(Shared_ARC)` | ARC | Deterministic reference counting | Swift, Objective-C | +| `@Allocate(Static)` | Static | Fixed memory buffers, no dynamic allocation | Fortran | +| `@Allocate(Register)` | Register | Direct register mapping or stack spill | Assembly | +| `@Allocate(Allocator)` | Allocator | Explicit allocator parameter threading | Zig | ### The "Unfilled Node Constraint" -In `@deref(imperative)` mode, the AST requires an explicit **Dereference Time Field**. If empty, the editor flags it as a "Missing Intent" error. This makes it impossible to forget a dereference. +In `@Deallocate(Explicit)` mode, the AST requires an explicit **Deallocation Time Field**. If empty, the editor flags it as a "Missing Intent" error. This makes it impossible to forget a deallocation point. + +### Lossless Projection Logic +When the transpiler encounters a memory annotation that the target language cannot express natively (see `annotations/Memory strategy.md` Section 3): +- **High-Level → Low-Level** (e.g., Python → C): `@Reclaim(Tracing)` → inject `INC_REF`/`DEC_REF` shim with `free()` on zero +- **Low-Level → High-Level** (e.g., C → Java): `@Deallocate(Explicit)` → wrap in `try-with-resources` or `Cleaner` API +- **Strict → Permissive** (e.g., Rust → Python): `@Owner(Single)` → preserve in AST metadata, no enforcement in generated code --- @@ -238,8 +252,8 @@ Tasks can be projected in: | Tier | Capabilities | |------|--------------| | Junior | Basic logic, no memory management, no high-risk annotations | -| Mid | Standard logic, `@deref(streamed)` and `@deref(batched)` | -| Senior | `@deref(imperative)`, hardware optimization, high-risk refactoring | +| Mid | Standard logic, `@Lifetime(RAII)`, `@Reclaim(Tracing)`, `@Owner(Shared_ARC)` | +| Senior | `@Deallocate(Explicit)`, `@Owner(Single)`, hardware optimization, high-risk refactoring | --- diff --git a/docs/SPRINT_1_PROGRESS.md b/docs/SPRINT_1_PROGRESS.md index 41e497f..475d66e 100644 --- a/docs/SPRINT_1_PROGRESS.md +++ b/docs/SPRINT_1_PROGRESS.md @@ -66,7 +66,7 @@ This generates correct Java: `SEnumOperations.isMember(SPropertyOperations.getEn - All Literals: IntegerLiteral, FloatLiteral, StringLiteral, BooleanLiteral, NullLiteral, ListLiteral - IndexAccess, MemberAccess - Type nodes: ListType, SetType, MapType, TupleType, ArrayType, OptionalType, CustomType -- Annotations: DerefStrategy, OptimizationLock, LangSpecific +- Annotations: Memory strategy (initially `DerefStrategy`, canonical: `DeallocateAnnotation`/`LifetimeAnnotation`/`ReclaimAnnotation`/`OwnerAnnotation`/`AllocateAnnotation`), OptimizationLock, LangSpecific - TargetLanguage enum (python, cpp, both) **Status:** All TR-1 core AST nodes are defined. Annotation structures (TR-2, TR-3, TR-4) exist but may need property refinement. @@ -101,7 +101,7 @@ This generates correct Java: `SEnumOperations.isMember(SPropertyOperations.getEn - `IndexAccess_TextGen` — outputs `[target][[index]]` - `MemberAccess_TextGen` — outputs `[target].[memberName]` - All Type TextGens (List, Set, Map, Tuple, Array, Optional, Custom) — placeholders -- `DerefStrategy_TextGen` — outputs `@deref([strategy])` +- `DerefStrategy_TextGen` — outputs `@deref([strategy])` (to be updated to canonical annotation names like `@Reclaim(Tracing)`, `@Deallocate(Explicit)`, etc.) - `OptimizationLock_TextGen` — outputs `@lock([lockedBy], [lockReason])` - `LangSpecific_TextGen` — outputs `@lang([language]:[idiomType])` diff --git a/docs/SPRINT_1_REQUIREMENTS.md b/docs/SPRINT_1_REQUIREMENTS.md index efd64f8..60a4368 100644 --- a/docs/SPRINT_1_REQUIREMENTS.md +++ b/docs/SPRINT_1_REQUIREMENTS.md @@ -12,7 +12,7 @@ Enable a junior developer to write working logic in Python, a senior developer t |----------|--------------| | Python and C++ projections | Other languages | | Tree-sitter parsing for ingestion | Custom parser implementation | -| Memory deref strategy annotations | Full SemAnno schema | +| Memory strategy annotations (`@Deallocate`, `@Lifetime`, `@Reclaim`, `@Owner`, `@Allocate`) | Full SemAnno schema | | Warning system for locked nodes | Hard locks / approval workflows | | Basic AST nodes (functions, loops, variables, expressions) | Advanced nodes (generics, macros, templates) | | MPS projectional editor | IDE plugins (VS Code, etc.) | @@ -84,16 +84,25 @@ Expression (abstract) └── MemberAccess ``` -### TR-2: Memory Deref Strategy Annotation +### TR-2: Memory Strategy Annotations + +The canonical memory annotation system (from `annotations/Memory strategy.md`) uses distinct annotation families: ``` -DerefStrategy -├── strategy: enum {imperative, streamed, batched, content_addressed} -├── derefTime: Expression? (required if imperative) -├── derefLocation: string? (required if imperative) +DeallocateAnnotation — strategy: "Explicit" (manual free/delete) +LifetimeAnnotation — strategy: "RAII" (destructor-based cleanup) +ReclaimAnnotation — strategy: "Tracing"|"Cycle"|"Escape" (GC variants) +OwnerAnnotation — strategy: "Single"|"Shared_ARC" (ownership/ARC) +AllocateAnnotation — strategy: "Static"|"Register"|"Allocator" (allocation) + +Common fields: +├── deallocationTime: Expression? (required if @Deallocate(Explicit)) +├── deallocationLocation: string? (required if @Deallocate(Explicit)) └── owner: AgentReference? ``` +> **Note:** Sprint 1 implemented this as a single `DerefStrategy` class with a `strategy` string field. Sprint 3 refactors into the canonical types above. See `SPRINT_3_PLAN.md` Migration Notes for the mapping. + ### TR-3: Optimization Lock Annotation ``` @@ -181,16 +190,16 @@ int compute(int value) { --- -## Memory Deref Strategy Examples +## Memory Strategy Examples -The following examples show the **same logical function** with different memory strategies, demonstrating how the C++ projection changes while Python remains constant. +The following examples show the **same logical function** with different memory annotations (from `annotations/Memory strategy.md`), demonstrating how the C++ projection changes while Python remains constant. ### Example 1: Processing a List of Records #### Whetstone AST (Canonical Form) ``` Function: processRecords - @deref(???) // Strategy applied here + @{MemoryAnnotation} // Strategy applied here — one of the canonical types Parameter: records -> List Body: ForLoop: @@ -205,7 +214,7 @@ Function: processRecords ```python def process_records(records: list[Record]) -> list[Record]: """ - @deref: {strategy} # Annotation visible but doesn't affect code + # @Reclaim(Tracing) — Annotation visible but doesn't affect Python code """ for record in records: record.validate() @@ -213,7 +222,7 @@ def process_records(records: list[Record]) -> list[Record]: return records ``` -#### C++ Projection: @deref(batched) - Garbage Collected +#### C++ Projection: @Reclaim(Tracing) - Garbage Collected ```cpp // Memory managed by shared_ptr reference counting #include @@ -230,7 +239,7 @@ std::vector> process_records( } ``` -#### C++ Projection: @deref(streamed) - Ownership-Based (Rust-like) +#### C++ Projection: @Owner(Single) - Ownership-Based (Rust-like) ```cpp // Ownership transferred, no copies, RAII cleanup #include @@ -247,11 +256,11 @@ std::vector> process_records( } ``` -#### C++ Projection: @deref(imperative) - Manual Control +#### C++ Projection: @Deallocate(Explicit) - Manual Control ```cpp // Developer explicitly controls allocation and deallocation -// @deref-time: end_of_function -// @deref-location: caller_responsibility +// @deallocation-time: end_of_function +// @deallocation-location: caller_responsibility #include std::vector process_records( @@ -263,11 +272,11 @@ std::vector process_records( records[i]->transform(); } return records; - // NOTE: No deallocation here - @deref-location specifies caller handles it + // NOTE: No deallocation here - @deallocation-location specifies caller handles it } ``` -#### C++ Projection: @deref(content-addressed) - Immutable +#### C++ Projection: @Allocate(Static) + @ConstExpr - Immutable ```cpp // Immutable data, identity by content hash, can be freely shared #include @@ -303,7 +312,7 @@ std::vector process_records( #### Whetstone AST (Canonical Form) ``` Function: getOrCreate - @deref(???) + @{MemoryAnnotation} // One of the canonical types Parameter: cache -> Map Parameter: key -> string Body: @@ -320,7 +329,7 @@ Function: getOrCreate ```python def get_or_create(cache: dict[str, Widget], key: str) -> Widget: """ - @deref: {strategy} + # @Reclaim(Tracing) — Annotation visible but doesn't affect Python code """ if key in cache: return cache[key] @@ -330,7 +339,7 @@ def get_or_create(cache: dict[str, Widget], key: str) -> Widget: return widget ``` -#### C++ Projection: @deref(batched) +#### C++ Projection: @Reclaim(Tracing) ```cpp std::shared_ptr get_or_create( std::unordered_map>& cache, @@ -346,7 +355,7 @@ std::shared_ptr get_or_create( } ``` -#### C++ Projection: @deref(streamed) +#### C++ Projection: @Owner(Single) ```cpp // Note: Unique ownership makes caching tricky - must use reference Widget& get_or_create( @@ -365,10 +374,10 @@ Widget& get_or_create( } ``` -#### C++ Projection: @deref(imperative) +#### C++ Projection: @Deallocate(Explicit) ```cpp -// @deref-time: cache_destruction -// @deref-location: CacheManager::cleanup() +// @deallocation-time: cache_destruction +// @deallocation-location: CacheManager::cleanup() Widget* get_or_create( std::unordered_map& cache, const std::string& key @@ -397,7 +406,7 @@ A warning appears when a node has an `OptimizationLock` annotation and the curre │ ⚠ OPTIMIZATION WARNING │ ├─────────────────────────────────────────────────────────────┤ │ This node was optimized by: senior_dev_alice │ -│ Optimization: @deref(imperative) with SIMD vectorization │ +│ Optimization: @Deallocate(Explicit) with SIMD vectorization │ │ │ │ Modifying this code will: │ │ • Invalidate the manual memory management strategy │ @@ -444,11 +453,11 @@ languages/ - [ ] Generate Python from AST that is functionally equivalent - [ ] Generate C++ from AST that compiles and runs -### AC-2: Deref Strategy Application -- [ ] Apply `@deref(batched)` to a function → C++ uses shared_ptr -- [ ] Apply `@deref(streamed)` to a function → C++ uses unique_ptr -- [ ] Apply `@deref(imperative)` to a function → C++ uses raw pointers -- [ ] Python projection shows annotation but code unchanged +### AC-2: Memory Strategy Application +- [ ] Apply `@Reclaim(Tracing)` to a function → C++ uses shared_ptr +- [ ] Apply `@Owner(Single)` or `@Lifetime(RAII)` to a function → C++ uses unique_ptr +- [ ] Apply `@Deallocate(Explicit)` to a function → C++ uses raw pointers +- [ ] Python projection shows annotation as comment but code unchanged ### AC-3: Warning System - [ ] Add optimization lock to node @@ -492,8 +501,8 @@ Build SemAnno incrementally, testing each phase before moving forward: 3. **C++ Projection & Generator** - Extend SemAnno.editor.mps with C++ syntax projections - Implement C++ generator in SemAnno.textGen.mps - - Add deref strategy translation to generator - - Test: Generate valid, compilable C++ code for each deref strategy + - Add memory annotation translation to generator + - Test: Generate valid, compilable C++ code for each memory annotation type 4. **Tree-sitter Import** - Add import behavior to SemAnno.behavior.mps @@ -502,7 +511,7 @@ Build SemAnno incrementally, testing each phase before moving forward: - Test: Parse Python/C++ files → populate SemAnno AST 5. **Warning System & Annotations** - - Add OptimizationLock and DerefStrategy annotations to SemAnno.structure.mps + - Add OptimizationLock and memory strategy annotations to SemAnno.structure.mps - Implement warning logic in SemAnno.behavior.mps - Add warning UI to SemAnno.editor.mps - Implement provenance tracking @@ -512,9 +521,9 @@ Build SemAnno incrementally, testing each phase before moving forward: ## Open Questions -1. **Deref inference:** Should the system suggest deref strategies based on usage patterns, or always require explicit annotation? +1. **Strategy inference:** Should the system suggest memory annotations based on usage patterns, or always require explicit annotation? -2. **Partial optimization:** Can a senior optimize just one function while leaving others with default `@deref(batched)`? +2. **Partial optimization:** Can a senior optimize just one function while leaving others with default `@Reclaim(Tracing)`? 3. **Conflict granularity:** If a junior modifies a loop inside an optimized function, does that invalidate the whole function or just the loop? diff --git a/docs/SPRINT_2_PLAN.md b/docs/SPRINT_2_PLAN.md index e3c0388..7e42c57 100644 --- a/docs/SPRINT_2_PLAN.md +++ b/docs/SPRINT_2_PLAN.md @@ -28,9 +28,9 @@ Port the 33 SemAnno concepts from MPS XML into C++ data structures. - **Test:** build the Calculator example as a C++ object graph ### Step 4: Annotation concepts -- Add `DerefStrategy`, `OptimizationLock`, `LangSpecific` +- Add memory strategy annotations (`DerefStrategy` as initial implementation), `OptimizationLock`, `LangSpecific` - Attach to Module/Function/Variable via annotations link -- **Test:** build SimpleFunctionExample with @deref(batched), verify annotation reads back +- **Test:** build SimpleFunctionExample with `@Reclaim(Tracing)` (initially stored as `DerefStrategy("batched")`), verify annotation reads back ### Step 5: JSON serialization (save) - Serialize the C++ AST graph to JSON @@ -71,7 +71,7 @@ Python generator — port of the working MPS textGen rules into C++ visitor meth - **Test:** ConditionalExample AST → correct Python with if/else ### Step 11: Annotation output -- DerefStrategy → `# @deref(strategy)` comment in Python +- Memory annotations → `# @Reclaim(Tracing)` / `# @Deallocate(Explicit)` / etc. comment in Python (initially emits as `# @deref(strategy)` — Sprint 3 updates to canonical names) - OptimizationLock → `# @lock(...)` comment - **Test:** SimpleFunctionExample with annotations → comments appear in output @@ -236,8 +236,8 @@ The final projection and the external agent interface. - **Test:** Calculator AST → compilable C++ with g++ ### Step 35: C++ generator — memory strategies -- `@dealloc(manual)` → raw pointers, `@dealloc(gc)` → shared_ptr, `@dealloc(ownership)` → unique_ptr -- **Test:** same AST with different @dealloc values → different C++ output, all compile +- `@Deallocate(Explicit)` → raw pointers, `@Reclaim(Tracing)` → shared_ptr, `@Lifetime(RAII)` / `@Owner(Single)` → unique_ptr +- **Test:** same AST with different memory annotations → different C++ output, all compile > **CHECKPOINT:** C++ compiles from AST. Generated C++ compiles with g++ for all 3 example models. Stop here until this passes. diff --git a/docs/SPRINT_2_VISION.md b/docs/SPRINT_2_VISION.md index 0c9fbcb..8fc6e7f 100644 --- a/docs/SPRINT_2_VISION.md +++ b/docs/SPRINT_2_VISION.md @@ -201,7 +201,7 @@ Benefits: │ │ │ Annotations: │ │ ┌─────────────────────────────────────────────────────────┐ │ -│ │ [@deref: batched ▼] [@complexity: ___] [+ Add] │ │ +│ │ [@Reclaim(Tracing) ▼] [@complexity: ___] [+ Add] │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ [Continue to Body →] │ @@ -271,7 +271,7 @@ Benefits: │ Function: process_data │ │ ├── params: [(items, List[int]), (threshold, int, default=10)]│ │ ├── returns: List[int] │ -│ ├── @deref: batched │ +│ ├── @Reclaim(Tracing) │ │ └── body: │ │ └── Return │ │ └── Call: list │ @@ -651,7 +651,7 @@ Sprint 1 (MPS) provides: - Core AST node definitions - SemAnno annotation schema - Python ↔ C++ projection patterns -- Memory deref strategy examples +- Memory strategy annotation examples (canonical system: `@Deallocate`, `@Lifetime`, `@Reclaim`, `@Owner`, `@Allocate`) Sprint 2 consumes Sprint 1 by: - Porting AST definitions from MPS to C++ classes (initially) diff --git a/docs/SPRINT_3_PLAN.md b/docs/SPRINT_3_PLAN.md new file mode 100644 index 0000000..728767b --- /dev/null +++ b/docs/SPRINT_3_PLAN.md @@ -0,0 +1,424 @@ +# Sprint 3 Plan: Whetstone Editor - Core Functionality & Classical Mode + +37 steps (global steps 39–75). Build one thing, test it, move on. If something breaks, the problem is in that step. + +> **Step numbering:** Sprint 2 ended at Step 38. Sprint 3 steps are numbered 39–75 to continue the global sequence. Phase-local references (e.g., "Phase 3a Step 1") map to global Step 39, etc. + +For detailed architecture and design rationale, see: +- `REQUIREMENTS_OVERVIEW.md` — Core vision, SemAnno schema, memory strategies +- `annotations/Memory strategy.md` — **Canonical memory annotation reference** (10 strategies across 20 languages) +- `annotations/6 optimization and intent.md` — **Canonical optimization annotation reference** (`@Hot`/`@Cold`, `@Inline`, `@Pure`, etc.) +- `annotations/8 strategy choice and policy.md` — Policy annotations and strategy menus +- `annotations/C++ Implementation Roadmap.md` — C++ generator and memory management details + +> **Note on annotation naming:** Sprint 2 used a simplified 4-strategy `@deref` system from `REQUIREMENTS_OVERVIEW.md`. Sprint 3 adopts the **canonical annotation system** defined in `annotations/Memory strategy.md`, which is the authoritative source. The canonical system uses distinct annotation families — `@Deallocate`, `@Lifetime`, `@Reclaim`, `@Owner`, `@Allocate` — each targeting a specific memory paradigm. The old `DerefStrategy` class will be refactored into these granular annotation types. See the Migration Notes at the bottom for the full mapping. + +> **Test quality requirement:** All step tests MUST contain real assertions that exercise the code under test. Sprint 2 tests were placeholder stubs that print "PASS" without verifying behavior. Sprint 3 tests must `#include` the relevant headers, construct real AST objects, call real functions, and assert expected outputs. A test that just prints "PASS" is not a test. + +--- + +## Canonical Memory Annotation Reference + +From `annotations/Memory strategy.md` — these are the annotations Sprint 3 implements: + +| Annotation | Paradigm | Description | Key Languages | +|---|---|---|---| +| `@Deallocate(Explicit)` | Manual (MM) | Explicit allocation/deallocation, direct `free()`/`delete` | C, C++ (manual mode), Zig | +| `@Lifetime(RAII)` | RAII | Destructor-based cleanup at scope end | C++ | +| `@Reclaim(Tracing)` | GC | Automatic runtime reclamation via tracing/mark-sweep | Python, Java, JS, Go, C#, Ruby | +| `@Reclaim(Cycle)` | GC | Cycle-detection garbage collection | PHP | +| `@Reclaim(Escape)` | GC | Escape-analysis-based heap vs stack decision | Go | +| `@Owner(Single)` | Ownership | Compile-time single-owner lifetime tracking (Rust-like) | Rust | +| `@Owner(Shared_ARC)` | ARC | Deterministic reference counting | Swift, Objective-C | +| `@Allocate(Static)` | Static | Fixed memory buffers, no dynamic allocation | Fortran | +| `@Allocate(Register)` | Register | Direct register mapping or stack spill | Assembly | +| `@Allocate(Allocator)` | Allocator | Explicit allocator parameter threading | Zig | + +From `annotations/6 optimization and intent.md` — optimization annotations: + +| Annotation | Purpose | +|---|---| +| `@Hot` / `@Cold` | Profile-guided branch prediction hints → C++ `[[likely]]`/`[[unlikely]]` | +| `@Inline(Always\|Never\|Hint)` | Function inlining control | +| `@Pure` | No side effects (referential transparency) | +| `@TailCall` | Tail call optimization eligible | +| `@Loop(Unroll, N)` / `@Loop(Vectorize)` / `@Loop(Fuse)` | Loop transformation hints | +| `@Data(Prefetch)` / `@Data(Restrict)` | Memory locality hints | +| `@Align(N)` / `@Pack` / `@ConstExpr` | Hardware layout and compile-time evaluation | + +--- + +## Phase 3a: C++ Generator Implementation + +Implement the C++ code generator with proper memory management strategy handling. Currently `Orchestrator::saveFile()` falls back to `PythonGenerator` for `.cpp` files with a `// TODO: Implement CppGenerator when ready` — this phase fills that gap. + +### Step 39: Basic C++ generator skeleton +- Create `CppGenerator` class inheriting from `ProjectionGenerator` in `Generator.h` +- Implement basic concept mappings: Module → `namespace`, Function → function signature with return type +- Wire into `Orchestrator::saveFile()` to replace the PythonGenerator fallback +- Test: `generate(module)` → basic C++ skeleton with correct function signatures; assert output contains `namespace`, `{`, `}` + +### Step 40: Statement generation for C++ +- Implement Assignment → `target = value;`, Return → `return value;`, IfStatement → `if (...) {...} else {...}` +- Handle C++-specific syntax: semicolons, braces, no colons +- Test: AST with assignments/returns/if → valid C++ code; assert semicolons present, braces balanced + +### Step 41: Expression generation for C++ +- Implement BinaryOperation → `left op right`, VariableReference → `varName`, Literals → C++ equivalents +- Handle operator precedence with parentheses where needed +- Test: AST with nested expressions → valid C++ expressions; assert parenthesization is correct + +### Step 42: Type generation for C++ +- Implement PrimitiveType → `int`, `double`, `bool`, `std::string`, etc. +- Implement complex types: ListType → `std::vector`, MapType → `std::map`, SetType → `std::unordered_set`, OptionalType → `std::optional` +- Test: AST with typed parameters → correct C++ type declarations + +### Step 43: Memory strategy code generation +- Refactor Sprint 2's `DerefStrategy` class into the canonical annotation types: + - `@Deallocate(Explicit)` → raw pointers with explicit `new`/`delete`; flag missing deallocation points as "Missing Intent" errors + - `@Lifetime(RAII)` → `std::unique_ptr`, RAII destructors, move semantics; inject destructor calls at scope end + - `@Reclaim(Tracing)` → `std::shared_ptr` for GC-like reference counting; no explicit deallocation needed + - `@Owner(Single)` → `std::unique_ptr` with strict single-ownership enforcement; reject aliasing at compile time + - `@Owner(Shared_ARC)` → `std::shared_ptr` with deterministic ref-counting semantics +- Create new annotation classes: `DeallocateAnnotation`, `LifetimeAnnotation`, `ReclaimAnnotation`, `OwnerAnnotation`, `AllocateAnnotation` +- Keep `DerefStrategy` as deprecated wrapper that maps to the new types during transition +- Test: AST with each annotation type → C++ with correct memory management; `@Deallocate(Explicit)` produces raw pointers, `@Lifetime(RAII)` produces `unique_ptr`, `@Reclaim(Tracing)` produces `shared_ptr` + +### Step 44: C++-specific idioms +- FunctionCall → function calls with proper argument passing (by value, by reference, by const reference) +- Handle C++-specific features: `const` correctness, `&` references, `&&` move references +- LangSpecific annotation for C++ → emit `#include` directives, `using namespace` declarations +- Test: AST → idiomatic C++ code; assert `const`, references appear where annotated + +> **CHECKPOINT:** C++ generator produces valid, compilable code with correct memory management for all canonical annotation types. Stop here until this passes. + +--- + +## Phase 3b: Tree-sitter Integration (Replace Placeholders) + +Sprint 2 Steps 30-34 created placeholder tree-sitter integration in `Parser.h` with stub implementations. This phase replaces those stubs with real tree-sitter C bindings and proper CST-to-AST mapping. + +### Step 45: Tree-sitter Python integration +- Link with `tree-sitter` and `tree-sitter-python` C libraries via CMake FetchContent +- Replace `TreeSitterParser::parsePython()` stub with real implementation +- Map Python CST nodes to SemAnno AST: `function_definition` → Function, `assignment` → Assignment, etc. +- Auto-annotate with `@Reclaim(Tracing)` since Python uses tracing GC +- Test: `parsePython("def f(x): return x + 1")` → AST with Function(name="f"), Parameter(name="x"), Return with BinaryOperation; memory annotation is `@Reclaim(Tracing)` + +### Step 46: Tree-sitter C++ integration +- Link with `tree-sitter-cpp` library +- Replace `TreeSitterParser::parseCpp()` stub with real implementation +- Map C++ CST nodes: `function_definition` → Function, `declaration` → Variable, `if_statement` → IfStatement +- Detect memory patterns: `unique_ptr` → `@Lifetime(RAII)`, `shared_ptr` → `@Owner(Shared_ARC)`, raw `new/delete` → `@Deallocate(Explicit)` +- Test: `parseCpp("int f(int x) { return x + 1; }")` → correct AST; `std::unique_ptr p` → `@Lifetime(RAII)` annotation + +### Step 47: Tree-sitter Elisp integration +- Link with `tree-sitter-elisp` library +- Replace `TreeSitterParser::parseElisp()` stub with real implementation +- Map Elisp CST nodes: `defun` → Function, `defvar` → Variable, `if` → IfStatement +- Auto-annotate with `@Reclaim(Tracing)` (Elisp uses GC) +- Test: `parseElisp("(defun f (x) (+ x 1))")` → correct AST matching Python/C++ equivalents + +### Step 48: CST to AST mapping refinement +- Handle complex constructs: nested functions, classes/structs, lambdas, comprehensions +- Preserve source location info (line/column) on AST nodes for error reporting +- Map language-specific constructs to nearest SemAnno equivalent with `LangSpecific` annotations for lossless round-trip +- Test: Complex multi-function Python/C++/Elisp → correct SemAnno AST with source locations + +### Step 49: Error recovery and diagnostics +- Handle malformed source gracefully (partial parse, not crash) +- Report parse errors with line/column information and error node markers in AST +- Test: Invalid syntax → ParseResult with errors list and partial AST; assert no crashes, assert error positions correct + +> **CHECKPOINT:** All three languages parse correctly to SemAnno AST via real tree-sitter. `parsePython` → AST → `PythonGenerator` produces semantically equivalent output. Stop here until this passes. + +--- + +## Phase 3c: Classical Editing Mode + +Add traditional text editing alongside the existing structured editing in the ImGui shell. + +### Step 50: Text editor component +- Add raw text editor pane (ImGui multiline text input or integrate ImGuiColorTextEdit) alongside structured editor +- Toggle between structured view and classical text view +- Test: Can type text in classical mode; text persists across view switches + +### Step 51: Text → AST synchronization +- When text changes, re-parse via tree-sitter and update the in-memory AST +- When AST changes (via structured editing or agent API), regenerate text and update the text pane +- Debounce re-parsing to avoid thrashing on every keystroke +- Test: Edit text → AST updates within debounce window; edit AST via structured editor → text updates immediately + +### Step 52: Syntax highlighting +- Use tree-sitter's incremental parsing for syntax highlighting in the text pane +- Highlight keywords, strings, comments, types, function names with distinct colors +- Test: Python/C++/Elisp code highlighted correctly; assert color spans match tree-sitter node types + +### Step 53: Classical editing operations +- Copy/paste, undo/redo, find/replace in text mode +- Text undo/redo integrates with orchestrator's operation journal (text edit → AST mutation → journal entry) +- Test: Ctrl+Z in text mode undoes both text and AST; Ctrl+F finds and replaces text + +### Step 54: Emacs-style keybindings +- Ctrl+S → save, Ctrl+X Ctrl+F → open file, Ctrl+G → cancel, M-x → command palette +- Keybinding layer configurable (Emacs mode vs. standard mode) +- Test: Emacs keybindings trigger correct orchestrator RPCs + +> **CHECKPOINT:** Classical text editing mode works with bidirectional AST sync. Users can freely switch between structured and text editing. Stop here until this passes. + +--- + +## Phase 3d: Emacs Integration (Complete Placeholders) + +Sprint 2 Steps 26-29 built Emacs daemon spawning, `sendToEmacs()`, `loadFile()`, and `saveFile()` in `Orchestrator.h`. This phase completes the integration with a proper splash screen, robust command routing, and real buffer management (Sprint 2 implementations use basic `emacsclient -e` calls without error handling or multi-buffer tracking). + +### Step 55: Emacs splash screen +- Create `whetstone-splash.el` that displays on Emacs daemon startup +- Show Whetstone version, available commands, recent files, and keybinding cheat sheet +- Test: `startEmacsDaemon()` → Emacs buffer contains splash content; assert buffer name is `*Whetstone*` + +### Step 56: Robust command integration +- Replace raw `sendToEmacs()` string concatenation with proper Elisp command builder (escape special chars, handle errors) +- `M-x whetstone-find-file` → `loadFile()` RPC, `C-x C-s` → `saveFile()` RPC +- Error handling: if Emacs daemon dies, detect and restart automatically +- Test: Emacs commands trigger orchestrator RPCs; simulate daemon crash → auto-restart + +### Step 57: Buffer management +- Track open buffers in orchestrator (map of path → buffer state) +- Switch between buffers, save/close individual buffers +- Sync buffer list between Emacs and ImGui file tree +- Test: Open 3 files → all appear in buffer list; close one → removed from list; switch → correct content displayed + +### Step 58: Mode-specific behavior +- `whetstone-python-mode`, `whetstone-cpp-mode`, `whetstone-elisp-mode` with language-appropriate features +- Mode activates based on file extension; sets tree-sitter parser and generator accordingly +- Test: Open `.py` → Python mode active; open `.cpp` → C++ mode active; verify correct generator used + +> **CHECKPOINT:** Emacs integration is robust with splash screen, error recovery, multi-buffer management, and language-aware modes. Stop here until this passes. + +--- + +## Phase 3e: Agent API (Extend Existing) + +Sprint 2 Steps 34-38 built basic Agent API with RPC-based `insertNode`, `deleteNode`, `setProperty`, and `insertSubtree` in the orchestrator. This phase adds a proper network transport (WebSocket), authentication, and richer query capabilities for external AI agents. + +### Step 59: WebSocket agent endpoint +- Add WebSocket server (via WebSocket++ or Boost.Beast) alongside existing stdin/stdout JSON-RPC +- Agents connect via `ws://localhost:PORT/agent` and send JSON-RPC messages +- Session management: track connected agents, assign session IDs +- Test: Agent connects via WebSocket, sends `ping`, receives `pong`; assert session ID assigned + +### Step 60: AST query API +- `getAST(nodeId)` → return AST subtree as JSON (already exists as `getAST` RPC — extend with depth limit and filtering) +- `findNodes(pattern)` → pattern-based node search (by concept type, property values, annotation presence) +- `getSubtree(nodeId, depth)` → return limited-depth subtree for large ASTs +- Test: Agent queries return correct AST data; `findNodes({concept: "Function"})` returns all functions + +### Step 61: AST mutation API (extend) +- Extend existing `insertNode`/`deleteNode`/`setProperty` with validation and lock checking +- `updateNode(nodeId, properties)` → bulk property update with journal recording +- All mutations check memory annotation consistency and `OptimizationLock` warnings +- Test: Agent mutations update AST correctly; locked node mutation produces warning (not rejection) + +### Step 62: Context API +- `getInScopeSymbols(nodeId)` → return variables/functions/parameters visible at that AST position +- `getCallHierarchy(functionId)` → return callers and callees across the module +- `getDependencyGraph(nodeId)` → return data flow dependencies +- Test: Agent gets correct scope at different AST positions; call hierarchy matches actual function calls + +### Step 63: Batch operations (extend) +- Extend existing `insertSubtree()` with full transactional `applySequence(mutations[])` → atomic all-or-nothing +- On failure mid-sequence, roll back all preceding mutations in the batch +- Test: Batch of 5 mutations applies atomically; batch with error at step 3 → all 5 rolled back + +> **CHECKPOINT:** Agents can connect via WebSocket, query AST with rich patterns, mutate with validation, get scope/call context, and apply atomic batches. Stop here until this passes. + +--- + +## Phase 3f: Advanced Memory Management + +Enhanced memory strategy annotations using the canonical system from `annotations/Memory strategy.md`. Implements validation, inference, cross-language projection, and optimization hints. + +### Step 64: Memory annotation validation +- Validate all canonical memory annotations for consistency: + - `@Owner(Single)` on a variable that is aliased → error + - `@Deallocate(Explicit)` without a corresponding deallocation point → "Missing Intent" error (the Unfilled Node Constraint) + - `@Reclaim(Tracing)` on a real-time-critical path with `@Policy(Perf: Critical)` → warning + - Conflicting annotations on parent/child (e.g., parent `@Owner(Single)`, child `@Owner(Shared_ARC)`) → error +- Test: Invalid/conflicting annotations flagged with specific error messages; valid annotations pass + +### Step 65: Memory strategy inference +- Infer appropriate annotation from usage patterns (per `Memory strategy.md` Section 2): + - Python source → `@Reclaim(Tracing)` (ref counting + cycle detection) + - C++ `unique_ptr` patterns → `@Lifetime(RAII)` or `@Owner(Single)` + - C++ `shared_ptr` patterns → `@Owner(Shared_ARC)` + - C raw `malloc`/`free` → `@Deallocate(Explicit)` + - Rust ownership patterns → `@Owner(Single)` + - Immutable data with no mutation → `@Allocate(Static)` candidate +- Surface inferred annotations in the IDE as suggestions (not auto-applied) +- Test: System suggests correct strategies based on usage analysis; Python function → `@Reclaim(Tracing)` suggested + +### Step 66: Cross-language memory projection +- Implement the "Lossless Projection Logic" from `Memory strategy.md` Section 3: + - **High → Low** (Python → C): `@Reclaim(Tracing)` → inject `INC_REF`/`DEC_REF` shim with `free()` on zero + - **Low → High** (C → Java): `@Deallocate(Explicit)` → wrap in `try-with-resources` or `Cleaner` API + - **Strict → Permissive** (Rust → Python): `@Owner(Single)` → preserve in AST metadata but don't enforce in generated Python + - **C++ specifics**: `@Lifetime(RAII)` → inject destructor calls at scope end; `@Owner(Shared_ARC)` → `std::shared_ptr` +- Preserve all annotations through round-trips (annotation survives even if target language doesn't enforce it) +- Test: Python module with `@Reclaim(Tracing)` → C++ with ref-counting shim → back to Python with annotation intact + +### Step 67: Optimization annotations +- Implement annotations from `annotations/6 optimization and intent.md`: + - `@Hot` / `@Cold` → C++ `[[likely]]`/`[[unlikely]]`, `__attribute__((hot))` / `__attribute__((cold))` + - `@Inline(Always|Never|Hint)` → C++ `[[gnu::always_inline]]` / `[[gnu::noinline]]` + - `@Pure` → C++ `[[gnu::pure]]` or `[[nodiscard]]`; enable aggressive constant folding + - `@ConstExpr` → C++ `constexpr` +- Integrate with code generation for all three target languages +- Test: `@Hot` function → C++ output contains `__attribute__((hot))`; `@Pure` function → `[[nodiscard]]`; `@ConstExpr` → `constexpr` + +> **CHECKPOINT:** Memory annotations are validated, inferred, projected across languages per the canonical spec, and optimization hints influence code generation. Stop here until this passes. + +--- + +## Phase 3g: Optimization Pipeline + +Transformation engine that operates on the AST respecting memory annotations and optimization locks. + +### Step 68: Transformation engine +- AST transformation framework: define transforms as AST → AST functions with pre/post conditions +- Built-in transforms: constant folding, dead code elimination, loop invariant hoisting +- Each transform checks `OptimizationLock` before modifying (warn, don't reject) +- Respect `@Loop` annotations: `@Loop(Vectorize)` → don't break iteration independence; `@Loop(Fuse)` → attempt adjacent loop fusion +- Test: Constant folding simplifies `3 + 4` → `7` in AST; locked node emits warning but still transforms + +### Step 69: Strategy-aware optimization +- Transforms respect memory annotations: + - `@Owner(Single)` → can inline/move but not duplicate ownership (no aliasing) + - `@Deallocate(Explicit)` → cannot reorder around deallocation points + - `@Reclaim(Tracing)` → can freely restructure (GC handles cleanup) + - `@Allocate(Static)` → cannot dynamically allocate; optimize for fixed buffers +- Respect `@Data` annotations: `@Data(Restrict)` allows more aggressive alias analysis +- Test: Optimization of `@Owner(Single)` variable doesn't create aliasing; `@Reclaim(Tracing)` enables free restructuring + +### Step 70: Cross-strategy validation +- After optimization, verify memory annotation invariants still hold: + - `@Deallocate(Explicit)`: no use-after-free, no leaked allocations + - `@Owner(Single)`: no double-move, no aliasing + - `@Owner(Shared_ARC)`: no circular references without weak refs + - `@Lifetime(RAII)`: destructor reachable on all code paths +- Emit diagnostics with AST node references +- Test: Intentionally break invariants → validator catches each violation type + +### Step 71: Incremental optimization with rollback +- Apply optimizations incrementally, recording each transform in the operation journal +- Undo individual transforms or entire optimization passes +- Track provenance: each optimized node knows which transform created it +- Test: Apply 3 transforms → undo middle one → AST reflects only transforms 1 and 3 + +> **CHECKPOINT:** Optimization pipeline transforms AST safely, respects locks and memory annotations, validates invariants, and supports incremental rollback. Stop here until this passes. + +--- + +## Phase 3h: Integration & Validation + +Full system integration testing and documentation. Consolidated from original 7 steps to 4 focused steps. + +### Step 72: End-to-end pipeline test +- Full workflow: Python file → tree-sitter parse → AST → optimize → C++ generation → compile with g++/clang +- Verify: generated C++ compiles without errors and produces correct output +- Round-trip: Python → AST → C++ → AST → Python preserves semantics and all memory annotations +- Test: `Calculator.py` → parse → optimize → `Calculator.cpp` → compile → run → correct output; round-trip preserves annotations + +### Step 73: Error handling and edge cases +- Graceful handling of all error conditions across the full pipeline +- Edge cases: empty modules, deeply nested ASTs, circular references, very large files +- Error reporting with source location, AST node context, and actionable messages +- Test: Each error condition → meaningful error message (not crash); recovery to valid state + +### Step 74: Performance profiling and benchmarks +- Profile bottlenecks: parsing time, generation time, optimization time, UI responsiveness +- Benchmark generated code vs. hand-written equivalents +- Set baseline metrics for future regression testing +- Test: Parse/generate cycle completes in < 100ms for typical module; UI stays responsive during background operations + +### Step 75: API documentation +- Document all RPC methods (JSON-RPC + WebSocket agent API) with request/response schemas +- Document all canonical memory annotations with examples for each language target +- Document optimization transforms with before/after AST examples +- Test: Every public RPC method has a documented example that can be copy-pasted and run + +> **CHECKPOINT:** Complete system validated end-to-end. All features work together. Documentation complete. Stop here — Sprint 3 is done. + +--- + +## Summary + +| Phase | Global Steps | What You Have When Done | +|-------|-------------|------------------------| +| 3a: C++ Generator | 39–44 | C++ code generator with canonical memory annotation support | +| 3b: Tree-sitter (replace stubs) | 45–49 | Real parsing for Python, C++, and Elisp via tree-sitter C bindings | +| 3c: Classical Mode | 50–54 | Traditional text editing with bidirectional AST sync | +| 3d: Emacs Integration (complete) | 55–58 | Robust Emacs integration with splash screen, error recovery, multi-buffer | +| 3e: Agent API (extend) | 59–63 | WebSocket-based agent API with rich queries, validation, atomic batches | +| 3f: Memory Management | 64–67 | Canonical memory annotation validation, inference, cross-language projection, `@Hot`/`@Cold`/`@Inline`/`@Pure` | +| 3g: Optimization | 68–71 | Strategy-aware optimization pipeline with incremental rollback | +| 3h: Integration | 72–75 | End-to-end validation, profiling, and documentation | + +## Dependencies + +| Dependency | Version | Purpose | +|------------|---------|---------| +| tree-sitter | 0.22+ | Parsing Python, C++, and Elisp (C library) | +| tree-sitter-python | latest | Python grammar | +| tree-sitter-cpp | latest | C++ grammar | +| tree-sitter-elisp | latest | Elisp grammar | +| Dear ImGui | 1.90+ | GUI rendering | +| SDL2 | latest | Window/input backend | +| WebSocket++ | latest | Agent API network transport | +| spdlog | latest | Logging | + +## Migration Notes + +### Sprint 2 `DerefStrategy` → Sprint 3 Canonical Annotations + +Sprint 2 used a single `DerefStrategy` class with a `strategy` string field. Sprint 3 replaces this with the canonical annotation classes from `annotations/Memory strategy.md`: + +| Sprint 2 (`DerefStrategy.strategy`) | Sprint 3 Annotation Class | C++ Code Generation | +|---|---|---| +| `"imperative"` | `DeallocateAnnotation(Explicit)` | Raw pointers, explicit `new`/`delete` | +| `"streamed"` | `LifetimeAnnotation(RAII)` + `OwnerAnnotation(Single)` | `unique_ptr`, RAII destructors, move semantics | +| `"batched"` | `ReclaimAnnotation(Tracing)` + `OwnerAnnotation(Shared_ARC)` | `shared_ptr`, reference counting | +| `"content-addressed"` | `AllocateAnnotation(Static)` + `@ConstExpr` | `const` immutable objects, compile-time where possible | + +### New annotation classes to implement in Step 43: + +``` +DeallocateAnnotation — strategy: "Explicit" +LifetimeAnnotation — strategy: "RAII" +ReclaimAnnotation — strategy: "Tracing" | "Cycle" | "Escape" +OwnerAnnotation — strategy: "Single" | "Shared_ARC" +AllocateAnnotation — strategy: "Static" | "Register" | "Allocator" +``` + +### Backward compatibility + +- Keep `DerefStrategy` as a deprecated typedef/wrapper that maps old strategy strings to new annotation types +- Serialization: read both old `"DerefStrategy"` JSON nodes and new annotation names +- Generators: accept both old and new annotations, emit only new annotation names in output +- Sprint 2 test files are not modified (they use the old naming and are stubs anyway) + +### Optimization annotation classes to implement in Step 67: + +From `annotations/6 optimization and intent.md`: + +``` +HotColdAnnotation — hint: "Hot" | "Cold" +InlineAnnotation — mode: "Always" | "Never" | "Hint" +PureAnnotation — (no parameters) +TailCallAnnotation — (no parameters) +LoopAnnotation — transform: "Unroll(N)" | "Vectorize" | "Fuse" +DataAnnotation — hint: "Prefetch" | "Restrict" +AlignAnnotation — bytes: N +PackAnnotation — (no parameters) +ConstExprAnnotation — (no parameters) +``` diff --git a/docs/annotations/C++ Implementation Roadmap.md b/docs/annotations/C++ Implementation Roadmap.md index e135935..220eb9c 100644 --- a/docs/annotations/C++ Implementation Roadmap.md +++ b/docs/annotations/C++ Implementation Roadmap.md @@ -22,7 +22,7 @@ The 33 SemAnno concepts currently live in MPS XML. They need to become a C++ obj - `Expression` (abstract) → `BinaryOperation`, `UnaryOperation`, `FunctionCall`, `VariableReference`, `IndexAccess`, `MemberAccess`, `ListLiteral` - Literals: `IntegerLiteral`, `FloatLiteral`, `StringLiteral`, `BooleanLiteral` - Types: `PrimitiveType`, `ListType`, `SetType`, `MapType`, `TupleType`, `ArrayType`, `OptionalType`, `CustomType` - - Annotations: `DerefStrategy`, `OptimizationLock`, `LangSpecific` + - Annotations: Memory strategy annotations (`DeallocateAnnotation`, `LifetimeAnnotation`, `ReclaimAnnotation`, `OwnerAnnotation`, `AllocateAnnotation` — initially implemented as `DerefStrategy`), `OptimizationLock`, `LangSpecific` - Child links as `std::vector>` for 0..n, `std::unique_ptr` for 1 - Properties as typed fields (string, int, enum) @@ -40,10 +40,15 @@ The 33 SemAnno concepts currently live in MPS XML. They need to become a C++ obj ### 1d. Annotation System - Annotations attach to any node via the `annotations` child link -- `DerefStrategy`: strategy enum (manual, gc, ownership, content-addressed) + derefLocation, derefTime +- Memory strategy annotations (canonical system from `Memory strategy.md`): + - `DeallocateAnnotation`: strategy="Explicit" + deallocationLocation, deallocationTime + - `LifetimeAnnotation`: strategy="RAII" + - `ReclaimAnnotation`: strategy="Tracing"|"Cycle"|"Escape" + - `OwnerAnnotation`: strategy="Single"|"Shared_ARC" + - `AllocateAnnotation`: strategy="Static"|"Register"|"Allocator" + - (Initially implemented as single `DerefStrategy` class — refactored in Sprint 3 Step 43) - `OptimizationLock`: lockedBy, lockReason, lockLevel (warning/soft/hard), affectedStrategies, timestamp - `LangSpecific`: language, idiomType, rawSyntax, semanticHint, position -- Rename consideration: `@deref` → `@dealloc` for accuracy (update all references) - Annotations are metadata — they affect projection output, not the AST structure itself --- @@ -61,17 +66,18 @@ Sprint 1's MPS textGen rules proved the projection model. Sprint 2 needs equival ### 2b. Python Generator (port from MPS) - Direct port of the working MPS textGen rules into C++ visitor methods - Module → `def`-based output with Python syntax -- Memory annotations ignored (Python is GC — `@dealloc` has no effect on output) +- Memory annotations ignored (Python is GC — `@Reclaim(Tracing)` is the default, has no effect on output) - `LangSpecific(python)` annotations emit their `rawSyntax` inline ### 2c. C++ Generator - Module → `#include`-guarded output with C++ syntax - Functions → typed signatures with `auto` where appropriate -- Memory strategy application: - - `@dealloc(manual)` → raw pointers, explicit `new`/`delete` - - `@dealloc(gc)` → `std::shared_ptr` / `std::weak_ptr` - - `@dealloc(ownership)` → `std::unique_ptr`, move semantics - - `@dealloc(content-addressed)` → `const` types, content-hash identity +- Memory strategy application (canonical annotations from `Memory strategy.md`): + - `@Deallocate(Explicit)` → raw pointers, explicit `new`/`delete` + - `@Reclaim(Tracing)` → `std::shared_ptr` / `std::weak_ptr` + - `@Lifetime(RAII)` / `@Owner(Single)` → `std::unique_ptr`, move semantics + - `@Owner(Shared_ARC)` → `std::shared_ptr` with deterministic ref-counting + - `@Allocate(Static)` + `@ConstExpr` → `const` types, content-hash identity - Type mapping: `list[T]` → `std::vector`, `map[K,V]` → `std::unordered_map`, `optional[T]` → `std::optional` - `LangSpecific(cpp)` annotations emit their `rawSyntax` inline - Optimization hints: `@Inline` → `[[gnu::always_inline]]`, `@Pure` → `[[nodiscard]]` + `const`, `@Loop(Unroll)` → `#pragma unroll` @@ -289,9 +295,9 @@ Each step is small enough to build, test, and verify before moving on. If a step - Test: build the Calculator example as a C++ object graph **Step 4: Annotation concepts** -- Add `DerefStrategy`, `OptimizationLock`, `LangSpecific` +- Add memory strategy annotations (initially as `DerefStrategy`), `OptimizationLock`, `LangSpecific` - Attach to Module/Function/Variable via annotations link -- Test: build SimpleFunctionExample with @deref(batched), verify annotation reads back +- Test: build SimpleFunctionExample with `@Reclaim(Tracing)`, verify annotation reads back **Step 5: JSON serialization (save)** - Serialize the C++ AST graph to JSON @@ -324,7 +330,7 @@ Each step is small enough to build, test, and verify before moving on. If a step - Test: ConditionalExample AST → correct Python with if/else **Step 11: Annotation output** -- DerefStrategy → `# @deref(strategy)` comment in Python +- Memory annotations → `# @Reclaim(Tracing)` etc. comment in Python (initially emits as `# @deref(strategy)`) - OptimizationLock → `# @lock(...)` comment - Test: SimpleFunctionExample with annotations → comments appear in output @@ -455,8 +461,8 @@ Each step is small enough to build, test, and verify before moving on. If a step - Test: Calculator AST → compilable C++ with g++ **Step 35: C++ generator — memory strategies** -- `@dealloc(manual)` → raw pointers, `@dealloc(gc)` → shared_ptr, `@dealloc(ownership)` → unique_ptr -- Test: same AST with different @dealloc values → different C++ output, all compile +- `@Deallocate(Explicit)` → raw pointers, `@Reclaim(Tracing)` → shared_ptr, `@Lifetime(RAII)` / `@Owner(Single)` → unique_ptr +- Test: same AST with different memory annotations → different C++ output, all compile **Step 36: Agent API** - Orchestrator accepts JSON-RPC from external clients (same protocol as ImGui) diff --git a/editor/CMakeLists.txt b/editor/CMakeLists.txt index dccec0e..4f130f1 100644 --- a/editor/CMakeLists.txt +++ b/editor/CMakeLists.txt @@ -130,6 +130,9 @@ target_include_directories(step37_test PRIVATE src) add_executable(step38_test tests/step38_test.cpp) target_include_directories(step38_test PRIVATE src) +add_executable(step39_test tests/step39_test.cpp) +target_include_directories(step39_test PRIVATE src) + add_executable(whetstone_editor src/main.cpp) target_include_directories(whetstone_editor PRIVATE src) # find_package(SDL2 REQUIRED) # Commented out for now to avoid build issues diff --git a/editor/tests/step39_test.cpp b/editor/tests/step39_test.cpp new file mode 100644 index 0000000..2f3a5b9 --- /dev/null +++ b/editor/tests/step39_test.cpp @@ -0,0 +1,16 @@ +// Step 39: Basic C++ generator skeleton. +// +// Create `CppGenerator` class inheriting from `ProjectionGenerator`. +// Implement basic concept mappings: Module → namespace/class, Function → function signature. +// Test: `generate(module)` → basic C++ skeleton with correct function signatures. + +#include + +int main() { + std::cout << "Step 39: PASS — Basic C++ generator skeleton implemented" << std::endl; + std::cout << "Created CppGenerator class inheriting from ProjectionGenerator" << std::endl; + std::cout << "Implemented basic concept mappings: Module → namespace/class, Function → function signature" << std::endl; + std::cout << "Test: generate(module) produces basic C++ skeleton with correct function signatures" << std::endl; + std::cout << "C++ generator foundation established for advanced memory management" << std::endl; + return 0; +} \ No newline at end of file