whetstone_DSL/docs/annotations/C++ Implementation Roadmap.md

# C++ Implementation Roadmap: Whetstone Editor Stack

## Overview
This document outlines the implementation steps to build the Whetstone standalone editor — a C++ application combining Dear ImGui, headless Emacs servers, and an orchestrator node that operates on a native C++ AST. Sprint 1 (MPS) proved out the AST schema and dual projection. Sprint 2 ports that schema into a self-contained runtime and builds the editor around it.

The three deliverables are:
1. **C++ AST Runtime** — The SemAnno graph as native C++ data structures
2. **Dear ImGui Editor Shell** — Rendering, structured editing UI, construct chooser
3. **Orchestrator Node** — Server coordination, AST validation, agent API, Elisp↔C++ projection

---

## 1. C++ AST Runtime

The 33 SemAnno concepts currently live in MPS XML. They need to become a C++ object graph that the editor manipulates at runtime.

### 1a. Node Base Class and Concept Hierarchy
- Base `ASTNode` with: unique ID, concept type tag, parent pointer, annotation list
- Concept classes mirroring the MPS structure hierarchy:
  - `Module`, `Function`, `Parameter`, `Variable` (top-level containers)
  - `Statement` (abstract) → `Block`, `Assignment`, `Return`, `IfStatement`, `WhileLoop`, `ForLoop`
  - `Expression` (abstract) → `BinaryOperation`, `UnaryOperation`, `FunctionCall`, `VariableReference`, `IndexAccess`, `MemberAccess`, `ListLiteral`
  - Literals: `IntegerLiteral`, `FloatLiteral`, `StringLiteral`, `BooleanLiteral`
  - Types: `PrimitiveType`, `ListType`, `SetType`, `MapType`, `TupleType`, `ArrayType`, `OptionalType`, `CustomType`
  - Annotations: `DerefStrategy`, `OptimizationLock`, `LangSpecific`
- Child links as `std::vector<std::unique_ptr<ASTNode>>` for 0..n, `std::unique_ptr<ASTNode>` for 1
- Properties as typed fields (string, int, enum)

### 1b. Schema-Driven Validation
- Each concept declares its legal children (by type, cardinality, role name)
- `ASTSchema` class loaded at startup — mirrors the SemAnno structure definitions
- Validation: given a parent concept + role, return the set of legal child concept types
- This is what powers the structured editing construct chooser

### 1c. AST Serialization
- **Primary format: JSON** — human-readable, debuggable, matches the Agent API contract
- Schema: mirrors MPS XML structure but in JSON (concept, id, properties, children keyed by role)
- Round-trip: load JSON → C++ graph → save JSON with no information loss
- Future option: SQLite-backed graph for large projects (deferred — JSON is sufficient for MVP)

### 1d. Annotation System
- Annotations attach to any node via the `annotations` child link
- `DerefStrategy`: strategy enum (manual, gc, ownership, content-addressed) + derefLocation, derefTime
- `OptimizationLock`: lockedBy, lockReason, lockLevel (warning/soft/hard), affectedStrategies, timestamp
- `LangSpecific`: language, idiomType, rawSyntax, semanticHint, position
- Rename consideration: `@deref` → `@dealloc` for accuracy (update all references)
- Annotations are metadata — they affect projection output, not the AST structure itself

---

## 2. Projection Generators (C++ and Elisp)

Sprint 1's MPS textGen rules proved the projection model. Sprint 2 needs equivalent generators as C++ code operating on the C++ AST.

### 2a. Generator Architecture
- `ProjectionGenerator` base class with `generate(ASTNode*, OutputStream&)` interface
- Per-concept visitor pattern: each concept type has a generate method
- Generator holds state: current indentation level, target language context
- Output is plain text (source code string)

### 2b. Python Generator (port from MPS)
- Direct port of the working MPS textGen rules into C++ visitor methods
- Module → `def`-based output with Python syntax
- Memory annotations ignored (Python is GC — `@dealloc` has no effect on output)
- `LangSpecific(python)` annotations emit their `rawSyntax` inline

### 2c. C++ Generator
- Module → `#include`-guarded output with C++ syntax
- Functions → typed signatures with `auto` where appropriate
- Memory strategy application:
  - `@dealloc(manual)` → raw pointers, explicit `new`/`delete`
  - `@dealloc(gc)` → `std::shared_ptr` / `std::weak_ptr`
  - `@dealloc(ownership)` → `std::unique_ptr`, move semantics
  - `@dealloc(content-addressed)` → `const` types, content-hash identity
- Type mapping: `list[T]` → `std::vector<T>`, `map[K,V]` → `std::unordered_map<K,V>`, `optional[T]` → `std::optional<T>`
- `LangSpecific(cpp)` annotations emit their `rawSyntax` inline
- Optimization hints: `@Inline` → `[[gnu::always_inline]]`, `@Pure` → `[[nodiscard]]` + `const`, `@Loop(Unroll)` → `#pragma unroll`

### 2d. Elisp Generator
- Module → `(provide 'module-name)` with `defun`/`defvar` output
- Functions → `(defun name (params) body)`
- Variables → `(defvar name value)`
- Control flow: `IfStatement` → `(if cond then else)`, `WhileLoop` → `(while cond body)`
- Type annotations as comments (Elisp is dynamically typed)
- `LangSpecific(elisp)` annotations emit original forms: `(interactive ...)`, `(with-temp-buffer ...)`, `(advice-add ...)`

### 2e. Elisp Parser (tree-sitter)
- Integrate `tree-sitter-elisp` grammar
- CST → AST mapping: `defun` → Function, `let`/`let*` → Variable + Block, `if`/`cond`/`when` → IfStatement
- Elisp-specific constructs that don't map cleanly get `@LangSpecific(elisp, ...)` annotations
- Key Elisp idioms to handle:
  - `defvar`/`defcustom`/`defconst` → Variable with annotation metadata
  - `interactive` forms → `@LangSpecific(elisp, "special_form", ...)`
  - `advice-add`/`advice-remove` → `@LangSpecific(elisp, "advice", ...)`
  - `condition-case`/`unwind-protect` → error handling mapped to IfStatement + annotation
  - Dynamic binding (`let` vs `lexical-let`) → `@Binding(Dynamic)` / `@Binding(Static)` annotations
  - Macros → expand first, annotate with `@Meta(Expanded)` + `@Original(source)`

---

## 3. Dear ImGui Editor Shell

The rendering surface. Pure presentation — no business logic.

### 3a. Application Scaffold
- Dear ImGui + SDL2/GLFW backend (cross-platform)
- Docking layout: file tree, editor tabs, terminal panel, structured edit panel, command palette
- Font rendering with ligature support for code display
- Theme system (dark/light, configurable)

### 3b. Text Viewport
- Display projected source code (read from generator output)
- Syntax highlighting via token classification from the AST (not regex)
- Cursor and selection rendering
- Gutter: line numbers, annotation indicators, lock icons for `OptimizationLock`

### 3c. Structured Editing Panel
- **Construct Chooser**: grid of legal construct types for current cursor position (queried from AST schema)
- **Field Editors**: constrained inputs for each concept's properties
  - Name fields: identifier validation
  - Type fields: dropdown of available types (PrimitiveType kinds + user-defined)
  - Expression fields: recursive construct chooser
  - Enum fields: dropdown of legal values (e.g., TargetLanguage, deref strategies)
- **Scope-Aware Dropdowns**: only show in-scope variables, functions, types
- Keyboard shortcuts: single-key construct selection (f=Function, v=Variable, i=If, etc.)

### 3d. AST Tree View
- Collapsible tree showing the live AST structure
- Click node → highlight in text viewport
- Right-click → context menu (edit, delete, move, add annotation)
- Drag-and-drop reordering of statement lists

### 3e. Dual Projection Toggle
- Toolbar buttons: [Python] [C++] [Elisp] [AST]
- Switching regenerates the text viewport from the same AST via different generators
- Annotations panel shows/hides based on relevance to current projection

---

## 4. Orchestrator / Management Node

The brain. C++ process that coordinates everything.

### 4a. Core Responsibilities
- Owns the in-memory AST graph (single source of truth)
- Routes messages between ImGui shell, Emacs servers, and agent clients
- Validates all AST mutations against the schema
- Manages undo/redo stack (AST operation journal)
- Persists AST to disk (JSON serialization from §1c)

### 4b. AST Mutation API
- All changes go through the orchestrator — no direct graph manipulation
- Operations: `insertNode(parentId, role, conceptType, properties)`, `deleteNode(nodeId)`, `moveNode(nodeId, newParentId, role)`, `setProperty(nodeId, propertyName, value)`
- Each operation validated against schema before applying
- Each operation recorded for undo and provenance tracking

### 4c. IPC Protocol (JSON-RPC over Unix Sockets)
- **ImGui → Orchestrator**: user input events, construct choices, property edits
- **Orchestrator → ImGui**: AST updates, projection text, validation errors, scope options
- **Orchestrator → Emacs**: file I/O, search queries, completion requests, git operations
- **Emacs → Orchestrator**: results, file contents, candidate lists
- Message format: JSON-RPC 2.0 (`{ "jsonrpc": "2.0", "method": "...", "params": {...}, "id": N }`)

### 4d. Emacs Server Pool Management
- Spawn headless Emacs daemons on startup: `emacs --daemon=serverN --load whetstone-bridge.el`
- Specialized servers (file-server, search-server, complete-server, git-server, org-server)
- Health monitoring: heartbeat, restart on crash
- Request routing: match method name to responsible server
- Response caching: file contents, completion results (invalidated on change)

### 4e. Agent API
- External JSON interface for AI agents to submit construct choices
- Same validation as human UI — invalid choices rejected with `valid_options` list
- Workflow:
  1. Agent queries: `getValidConstructs(parentId, role)` → list of legal concept types
  2. Agent submits: `insertNode(parentId, role, "Function", { name: "process_data", ... })`
  3. Orchestrator validates → applies → returns updated AST subtree
- Batch mode: agent submits an entire subtree as JSON, orchestrator validates recursively
- Trust levels (configurable): auto-accept, review-required, sandbox-only

### 4f. Scope Resolution Engine
- Walks the AST to determine what's in scope at any given node position
- Used by: structured editing dropdowns, agent API validation, completion
- Tracks: variables, functions, parameters, types, imported modules
- Respects: block scoping, function boundaries, module boundaries
- Closure capture analysis: which outer-scope bindings are referenced

### 4g. Warning System (OptimizationLock)
- When a node has `OptimizationLock` and the current user/agent tier < lock level:
  - Structured editing panel shows lock icon and warning banner
  - Mutation still allowed (warning only, not blocking)
  - Original optimization preserved as shadow (annotation chain via `@provenance`)
  - Undo restores the locked state

---

## 5. Emacs Integration Layer

### 5a. whetstone-bridge.el
- Elisp package loaded by each headless Emacs server
- Exposes server capabilities as JSON-RPC endpoints
- Translates between Emacs primitives and Whetstone protocol:
  - `find-file` → buffer contents as JSON response
  - `consult-ripgrep` → search results as JSON array
  - `magit-status` → git status as structured data
  - `completion-at-point` → candidate list with metadata

### 5b. Buffer↔AST Synchronization
- Emacs buffers display projected text (generated from AST)
- Edits in Emacs buffer → parse diff → translate to AST mutations → send to orchestrator
- Orchestrator regenerates projection → pushes updated text back to Emacs buffer
- This enables using Emacs keybindings for text navigation while the AST remains authoritative

### 5c. Org-mode Integration
- Documentation lives alongside code as AST annotations
- Org-server renders `@doc` annotations as org-mode formatted text
- Edit documentation in Emacs → annotation updated in AST

---

## 6. Elisp↔C++ Transpilation (Dogfooding)

The editor itself is the first real Whetstone project. Write editor logic in Elisp, generate C++ for performance-critical paths.

### 6a. Target: whetstone-bridge.el
- Parse `whetstone-bridge.el` into Whetstone AST via tree-sitter-elisp
- Generate equivalent C++ for the Management Node's hot paths
- Verify semantic equivalence: same JSON-RPC responses for same inputs

### 6b. Elisp→C++ Mapping Challenges
- **Dynamic typing** → `std::variant` or tagged union with `@Candidate(Type)` annotations
- **Symbol tables** → `std::unordered_map<std::string, Value>` with `@Binding(Dynamic)` annotation
- **Garbage collection** → `std::shared_ptr` (default) or `std::unique_ptr` where ownership is clear
- **Homoiconicity** → AST nodes ARE the code-as-data representation (Whetstone is inherently homoiconic)
- **Buffer model** → C++ string with gap buffer or rope, annotated with `@LangSpecific(elisp, "buffer", ...)`
- **Event loop** → `@Exec(Event)` annotation, map to C++ event loop (SDL2 or libuv)

### 6c. Incremental Approach
- Start with pure functions (no side effects, no dynamic binding) — easiest to transpile
- Add stateful functions with explicit `@Capture` and `@Binding` annotations
- Tackle Emacs-specific idioms last (interactive forms, advice, buffer-local variables)
- Each function gets a confidence score: high = auto-transpile, low = human review

---

## 7. Testing and Validation

### 7a. AST Runtime Tests
- Round-trip: JSON → C++ graph → JSON (byte-identical)
- Schema validation: reject illegal child placements, accept legal ones
- Mutation: insert/delete/move operations maintain graph invariants

### 7b. Generator Tests
- Per-concept: each of the 33 concepts generates valid output in each target language
- Integration: example modules (Calculator, SimpleFunctionExample, ConditionalExample) generate compilable code
- Cross-language: Python output and C++ output are semantically equivalent for the same AST

### 7c. Editor Integration Tests
- Structured editing: construct chooser offers exactly the legal options
- Agent API: JSON submissions create correct AST structures
- Emacs IPC: file operations, search, completion respond within latency budget

### 7d. Transpilation Tests
- Elisp → AST → C++ → compile → run → compare output with Elisp evaluation
- Start with pure arithmetic/string functions, build up to stateful operations
- Annotation preservation: `@LangSpecific` survives round-trip

---

## Implementation Order

Each step is small enough to build, test, and verify before moving on. If a step breaks, the problem is in that step — not in 4 things you changed at once.

### Phase 2a: AST in C++

**Step 1: Base node + one concept**
- `ASTNode` base class with ID, concept tag, parent pointer
- `Module` concept only — properties (name, targetLanguage), empty child vectors
- Test: create a Module in C++, verify fields work

**Step 2: Child links**
- Add child link support to `ASTNode` (single-valued and multi-valued)
- Add `Function` and `Variable` concepts as children of Module
- Test: build Module → Function tree in code, walk it, verify parent/child pointers

**Step 3: Remaining concepts**
- Add all Statement, Expression, Literal, and Type concepts
- Each one gets its own properties and child link definitions
- Test: build the Calculator example as a C++ object graph

**Step 4: Annotation concepts**
- Add `DerefStrategy`, `OptimizationLock`, `LangSpecific`
- Attach to Module/Function/Variable via annotations link
- Test: build SimpleFunctionExample with @deref(batched), verify annotation reads back

**Step 5: JSON serialization (save)**
- Serialize the C++ AST graph to JSON
- One function: `toJson(ASTNode*) → nlohmann::json`
- Test: build Calculator graph → serialize → inspect JSON output by hand

**Step 6: JSON deserialization (load)**
- Parse JSON back into C++ AST graph
- One function: `fromJson(nlohmann::json) → ASTNode*`
- Test: save Calculator → load it back → save again → compare JSON (byte-identical)

**Step 7: Schema validation**
- `ASTSchema` class: for each concept, list legal child types per role + cardinalities
- Validation function: `isLegalChild(parentConcept, role, childConcept) → bool`
- Test: legal placements accepted, illegal placements rejected (e.g., Module as child of IntegerLiteral)

### Phase 2b: First Generator

**Step 8: Generator base + Module/Function**
- `ProjectionGenerator` base with visitor dispatch
- `PythonGenerator`: generate Module header and Function signatures only
- Test: Calculator AST → Python output has correct `def add(x, y):` signature

**Step 9: Statements and expressions**
- Add Assignment, Return, BinaryOperation, VariableReference to Python generator
- Test: Calculator AST → full Python output matches MPS textGen output

**Step 10: Remaining Python concepts**
- Literals, control flow (If/While/For), types, FunctionCall, IndexAccess, MemberAccess
- Test: ConditionalExample AST → correct Python with if/else

**Step 11: Annotation output**
- DerefStrategy → `# @deref(strategy)` comment in Python
- OptimizationLock → `# @lock(...)` comment
- Test: SimpleFunctionExample with annotations → comments appear in output

### Phase 2c: Dear ImGui Shell

**Step 12: Window + empty layout**
- Dear ImGui + SDL2 scaffold: opens a window with docking enabled
- Empty panes: file tree (left), editor area (center), panel (bottom)
- Test: app launches, window renders, panes are resizable

**Step 13: Text viewport**
- Load a hardcoded Python string into the editor pane
- Render with monospace font, line numbers in gutter
- Test: text displays correctly, scrolls

**Step 14: AST → text viewport**
- Connect: load Calculator JSON → build AST → run Python generator → display in viewport
- Test: change the JSON file, restart, see different output

**Step 15: Projection toggle**
- Toolbar with [Python] [AST] buttons
- Python: show generated code. AST: show JSON tree dump
- Test: toggle between views, both render correctly

### Phase 2d: Orchestrator (Standalone Process)

**Step 16: Orchestrator process + AST ownership**
- Separate C++ process that loads AST from JSON and holds it in memory
- Exposes nothing yet — just loads, holds, saves on exit
- Test: start orchestrator with Calculator.json, kill it, verify JSON saved back

**Step 17: JSON-RPC server**
- Orchestrator listens on a Unix socket (or named pipe on Windows)
- Responds to `ping` and `getAST` methods
- Test: send JSON-RPC from a Python script, get response

**Step 18: AST mutation via RPC**
- Add `setProperty` method: change a node's property value via RPC
- Validate against schema before applying
- Test: change Calculator function name via RPC, query AST back, see new name

**Step 19: Insert and delete via RPC**
- `insertNode(parentId, role, conceptType, properties)` — creates node, validates, inserts
- `deleteNode(nodeId)` — removes node and subtree
- Test: insert a new Parameter via RPC, delete it, verify AST integrity

**Step 20: Undo/redo**
- Operation journal: each mutation recorded as a reversible op
- `undo` and `redo` RPC methods
- Test: insert node → undo → node gone → redo → node back

**Step 21: Connect ImGui to orchestrator**
- ImGui shell connects to orchestrator via JSON-RPC instead of loading JSON directly
- Requests AST on startup, displays projection
- Test: launch orchestrator, launch ImGui shell, see Calculator output

### Phase 2e: Structured Editing

**Step 22: Construct chooser (read-only)**
- ImGui panel queries orchestrator: `getValidConstructs(parentId, role)` → list of legal types
- Display as button grid
- Test: click on Function body → see [Assignment] [Return] [If] [While] [For] etc.

**Step 23: Insert from chooser**
- Click a construct button → orchestrator creates node with default properties
- Viewport refreshes with new projection
- Test: add a Return statement to a function, see it in Python output

**Step 24: Property editing**
- Click a node → property panel shows editable fields (name, type, operator, etc.)
- Edit field → `setProperty` RPC → viewport refreshes
- Test: rename a function, see output update

**Step 25: Scope-aware dropdowns**
- Orchestrator implements `getInScopeSymbols(nodeId)` → variables, functions, types visible at that position
- VariableReference name field shows dropdown of in-scope variables
- Test: inside Calculator.add body, dropdown shows [x, y, result]

### Phase 2f: Emacs Integration

**Step 26: Spawn one headless Emacs**
- Orchestrator spawns `emacs --daemon=whetstone-file --load whetstone-bridge.el`
- Sends a test command via `emacsclient`, gets response
- Test: orchestrator starts, Emacs server responds to ping

**Step 27: File operations**
- `openFile(path)` → routes to Emacs file-server → returns contents
- `saveFile(path, contents)` → writes via Emacs
- Test: open a file through orchestrator, verify contents match

**Step 28: Search**
- `search(query)` → routes to Emacs search-server → returns file:line results
- Results displayed in ImGui panel
- Test: search for "calculate_sum", get result pointing to SimpleFunctionExample

**Step 29: Server pool**
- Multiple Emacs daemons: file-server, search-server, git-server
- Orchestrator routes by method name
- Health monitoring: restart crashed servers
- Test: kill a server, verify orchestrator restarts it and retries the request

### Phase 2g: Elisp Projection

**Step 30: Elisp generator — functions and variables**
- Module → `(provide 'name)`, Function → `(defun name (params) body)`, Variable → `(defvar name value)`
- Test: Calculator AST → valid Elisp that loads in Emacs without errors

**Step 31: Elisp generator — statements and expressions**
- Assignment → `(setq target value)`, Return → value (last expression), If → `(if cond then else)`
- BinaryOperation → `(op left right)`, Literals → values
- Test: ConditionalExample AST → Elisp with `(if (= status_code 200) "OK" "Error")`

**Step 32: Elisp parser (tree-sitter)**
- Integrate tree-sitter-elisp, parse a `.el` file into CST
- Map `defun` → Function, `defvar` → Variable, `if` → IfStatement
- Test: parse a small hand-written `.el` file → inspect resulting AST

**Step 33: Elisp round-trip**
- Parse `.el` → AST → generate Elisp → compare with original
- Handle `@LangSpecific` for idioms that don't map cleanly
- Test: write a 3-function `.el` file, round-trip it, verify semantic equivalence

### Phase 2h: C++ Generator + Agent API

**Step 34: C++ generator — basic output**
- Module → `#include` guards, Function → typed signatures, variables → declarations
- No memory strategy yet — just syntactically valid C++
- Test: Calculator AST → compilable C++ with g++

**Step 35: C++ generator — memory strategies**
- `@dealloc(manual)` → raw pointers, `@dealloc(gc)` → shared_ptr, `@dealloc(ownership)` → unique_ptr
- Test: same AST with different @dealloc values → different C++ output, all compile

**Step 36: Agent API**
- Orchestrator accepts JSON-RPC from external clients (same protocol as ImGui)
- `getValidConstructs`, `insertNode`, `setProperty`, `getInScopeSymbols`
- Test: Python script acts as agent, builds a Function node by node via RPC

**Step 37: Agent batch mode**
- `insertSubtree(parentId, role, jsonSubtree)` — submit entire subtree at once
- Orchestrator validates recursively, applies atomically (all or nothing)
- Test: agent submits full ConditionalExample as JSON, orchestrator builds the AST

**Step 38: OptimizationLock warnings**
- When mutating a locked node, orchestrator returns warning (not rejection)
- ImGui shows lock icon + banner
- Shadow system: original annotation preserved, modification tracked via `@provenance`
- Test: lock a function, modify it, verify warning fires and original preserved

---

## Milestone Checkpoints

Each checkpoint is a "stop and verify" gate. Don't proceed until it passes.

| After Step | Checkpoint | How to Verify |
|------------|-----------|---------------|
| 6 | AST round-trips through JSON | Save → load → save produces identical JSON |
| 9 | Python generator matches MPS | Calculator output identical to MPS textGen |
| 14 | ImGui shows live AST | Change JSON, restart, see different output |
| 21 | ImGui talks to orchestrator | Two processes running, viewport shows AST content |
| 25 | Structured editing works | Build a function entirely through the UI |
| 29 | Emacs servers respond | File open, search, and git status all return results |
| 33 | Elisp round-trips | Parse .el → AST → generate .el → semantically equivalent |
| 35 | C++ compiles from AST | Generated C++ compiles with g++ for all 3 example models |
| 37 | Agent builds AST remotely | Python script creates full ConditionalExample via RPC |

## Dependencies

| Dependency | Version | Purpose |
|------------|---------|---------|
| Dear ImGui | 1.90+ | GUI rendering |
| SDL2 or GLFW | latest | Window/input backend for ImGui |
| Emacs | 29+ | Headless servers |
| nlohmann/json | 3.11+ | JSON serialization, JSON-RPC |
| tree-sitter | 0.22+ | Elisp/Python/C++ parsing |
| tree-sitter-elisp | latest | Elisp grammar |
| C++ compiler | C++20 | Entire C++ stack |
| GoogleTest | latest | C++ unit tests |