whetstone_DSL/docs/SPRINT_2_PLAN.md

# Sprint 2 Plan: Whetstone Editor Stack

38 steps. Build one thing, test it, move on. If something breaks, the problem is in that step.

For detailed architecture and design rationale, see:
- `SPRINT_2_VISION.md` — Architecture diagrams, structured editing UI, agent API design
- `annotations/C++ Implementation Roadmap.md` — Full technical specs for each component (§1–§7)

---

## Phase 2a: AST in C++

Port the 33 SemAnno concepts from MPS XML into C++ data structures.

### Step 1: Base node + one concept
- `ASTNode` base class with ID, concept type tag, parent pointer
- `Module` concept only — properties (name, targetLanguage), empty child vectors
- **Test:** create a Module in C++, verify fields work

### Step 2: Child links
- Add child link support to `ASTNode` (single-valued and multi-valued)
- Add `Function` and `Variable` concepts as children of Module
- **Test:** build Module → Function tree in code, walk it, verify parent/child pointers

### Step 3: Remaining concepts
- Add all Statement, Expression, Literal, and Type concepts
- Each one gets its own properties and child link definitions
- **Test:** build the Calculator example as a C++ object graph

### Step 4: Annotation concepts
- Add memory strategy annotations (`DerefStrategy` as initial implementation), `OptimizationLock`, `LangSpecific`
- Attach to Module/Function/Variable via annotations link
- **Test:** build SimpleFunctionExample with `@Reclaim(Tracing)` (initially stored as `DerefStrategy("batched")`), verify annotation reads back

### Step 5: JSON serialization (save)
- Serialize the C++ AST graph to JSON
- One function: `toJson(ASTNode*) → nlohmann::json`
- **Test:** build Calculator graph → serialize → inspect JSON output by hand

### Step 6: JSON deserialization (load)
- Parse JSON back into C++ AST graph
- One function: `fromJson(nlohmann::json) → ASTNode*`
- **Test:** save Calculator → load it back → save again → compare JSON (byte-identical)

> **CHECKPOINT:** AST round-trips through JSON. Save → load → save produces identical output. Stop here until this passes.

### Step 7: Schema validation
- `ASTSchema` class: for each concept, list legal child types per role + cardinalities
- Validation function: `isLegalChild(parentConcept, role, childConcept) → bool`
- **Test:** legal placements accepted, illegal placements rejected (e.g., Module as child of IntegerLiteral)

---

## Phase 2b: First Generator

Python generator — port of the working MPS textGen rules into C++ visitor methods.

### Step 8: Generator base + Module/Function
- `ProjectionGenerator` base with visitor dispatch
- `PythonGenerator`: generate Module header and Function signatures only
- **Test:** Calculator AST → Python output has correct `def add(x, y):` signature

### Step 9: Statements and expressions
- Add Assignment, Return, BinaryOperation, VariableReference to Python generator
- **Test:** Calculator AST → full Python output matches MPS textGen output

> **CHECKPOINT:** Python generator matches MPS. Calculator output identical to what MPS produces. Stop here until this passes.

### Step 10: Remaining Python concepts
- Literals, control flow (If/While/For), types, FunctionCall, IndexAccess, MemberAccess
- **Test:** ConditionalExample AST → correct Python with if/else

### Step 11: Annotation output
- Memory annotations → `# @Reclaim(Tracing)` / `# @Deallocate(Explicit)` / etc. comment in Python (initially emits as `# @deref(strategy)` — Sprint 3 updates to canonical names)
- OptimizationLock → `# @lock(...)` comment
- **Test:** SimpleFunctionExample with annotations → comments appear in output

---

## Phase 2c: Dear ImGui Shell

Pure presentation layer — no business logic.

### Step 12: Window + empty layout
- Dear ImGui + SDL2 scaffold: opens a window with docking enabled
- Empty panes: file tree (left), editor area (center), panel (bottom)
- **Test:** app launches, window renders, panes are resizable

### Step 13: Text viewport
- Load a hardcoded Python string into the editor pane
- Render with monospace font, line numbers in gutter
- **Test:** text displays correctly, scrolls

### Step 14: AST → text viewport
- Connect: load Calculator JSON → build AST → run Python generator → display in viewport
- **Test:** change the JSON file, restart, see different output

> **CHECKPOINT:** ImGui shows live AST. Change JSON, restart, see different output. Stop here until this passes.

### Step 15: Projection toggle
- Toolbar with [Python] [AST] buttons
- Python: show generated code. AST: show JSON tree dump
- **Test:** toggle between views, both render correctly

---

## Phase 2d: Orchestrator (Standalone Process)

The brain — separate C++ process that owns the AST and coordinates everything.

### Step 16: Orchestrator process + AST ownership
- Separate C++ process that loads AST from JSON and holds it in memory
- Exposes nothing yet — just loads, holds, saves on exit
- **Test:** start orchestrator with Calculator.json, kill it, verify JSON saved back

### Step 17: JSON-RPC server
- Orchestrator listens on a Unix socket (or named pipe on Windows)
- Responds to `ping` and `getAST` methods
- **Test:** send JSON-RPC from a Python script, get response

### Step 18: AST mutation via RPC
- Add `setProperty` method: change a node's property value via RPC
- Validate against schema before applying
- **Test:** change Calculator function name via RPC, query AST back, see new name

### Step 19: Insert and delete via RPC
- `insertNode(parentId, role, conceptType, properties)` — creates node, validates, inserts
- `deleteNode(nodeId)` — removes node and subtree
- **Test:** insert a new Parameter via RPC, delete it, verify AST integrity

### Step 20: Undo/redo
- Operation journal: each mutation recorded as a reversible op
- `undo` and `redo` RPC methods
- **Test:** insert node → undo → node gone → redo → node back

### Step 21: Connect ImGui to orchestrator
- ImGui shell connects to orchestrator via JSON-RPC instead of loading JSON directly
- Requests AST on startup, displays projection
- **Test:** launch orchestrator, launch ImGui shell, see Calculator output

> **CHECKPOINT:** ImGui talks to orchestrator. Two processes running, viewport shows AST content. Stop here until this passes.

---

## Phase 2e: Structured Editing

Build code by choosing from valid options — no typing syntax.

### Step 22: Construct chooser (read-only)
- ImGui panel queries orchestrator: `getValidConstructs(parentId, role)` → list of legal types
- Display as button grid
- **Test:** click on Function body → see [Assignment] [Return] [If] [While] [For] etc.

### Step 23: Insert from chooser
- Click a construct button → orchestrator creates node with default properties
- Viewport refreshes with new projection
- **Test:** add a Return statement to a function, see it in Python output

### Step 24: Property editing
- Click a node → property panel shows editable fields (name, type, operator, etc.)
- Edit field → `setProperty` RPC → viewport refreshes
- **Test:** rename a function, see output update

### Step 25: Scope-aware dropdowns
- Orchestrator implements `getInScopeSymbols(nodeId)` → variables, functions, types visible at that position
- VariableReference name field shows dropdown of in-scope variables
- **Test:** inside Calculator.add body, dropdown shows [x, y, result]

> **CHECKPOINT:** Structured editing works. Build a function entirely through the UI. Stop here until this passes.

---

## Phase 2f: Emacs Integration

Headless Emacs servers for file ops, search, git, completion.

### Step 26: Spawn one headless Emacs
- Orchestrator spawns `emacs --daemon=whetstone-file --load whetstone-bridge.el`
- Sends a test command via `emacsclient`, gets response
- **Test:** orchestrator starts, Emacs server responds to ping

### Step 27: File operations
- `openFile(path)` → routes to Emacs file-server → returns contents
- `saveFile(path, contents)` → writes via Emacs
- **Test:** open a file through orchestrator, verify contents match

### Step 28: Search
- `search(query)` → routes to Emacs search-server → returns file:line results
- Results displayed in ImGui panel
- **Test:** search for "calculate_sum", get result pointing to SimpleFunctionExample

### Step 29: Server pool
- Multiple Emacs daemons: file-server, search-server, git-server
- Orchestrator routes by method name
- Health monitoring: restart crashed servers
- **Test:** kill a server, verify orchestrator restarts it and retries the request

> **CHECKPOINT:** Emacs servers respond. File open, search, and git status all return results. Stop here until this passes.

---

## Phase 2g: Elisp Projection

Generate Elisp from AST, and parse Elisp into AST.

### Step 30: Elisp generator — functions and variables
- Module → `(provide 'name)`, Function → `(defun name (params) body)`, Variable → `(defvar name value)`
- **Test:** Calculator AST → valid Elisp that loads in Emacs without errors

### Step 31: Elisp generator — statements and expressions
- Assignment → `(setq target value)`, Return → value (last expression), If → `(if cond then else)`
- BinaryOperation → `(op left right)`, Literals → values
- **Test:** ConditionalExample AST → Elisp with `(if (= status_code 200) "OK" "Error")`

### Step 32: Elisp parser (tree-sitter)
- Integrate tree-sitter-elisp, parse a `.el` file into CST
- Map `defun` → Function, `defvar` → Variable, `if` → IfStatement
- **Test:** parse a small hand-written `.el` file → inspect resulting AST

### Step 33: Elisp round-trip
- Parse `.el` → AST → generate Elisp → compare with original
- Handle `@LangSpecific` for idioms that don't map cleanly
- **Test:** write a 3-function `.el` file, round-trip it, verify semantic equivalence

> **CHECKPOINT:** Elisp round-trips. Parse .el → AST → generate .el → semantically equivalent. Stop here until this passes.

---

## Phase 2h: C++ Generator + Agent API

The final projection and the external agent interface.

### Step 34: C++ generator — basic output
- Module → `#include` guards, Function → typed signatures, variables → declarations
- No memory strategy yet — just syntactically valid C++
- **Test:** Calculator AST → compilable C++ with g++

### Step 35: C++ generator — memory strategies
- `@Deallocate(Explicit)` → raw pointers, `@Reclaim(Tracing)` → shared_ptr, `@Lifetime(RAII)` / `@Owner(Single)` → unique_ptr
- **Test:** same AST with different memory annotations → different C++ output, all compile

> **CHECKPOINT:** C++ compiles from AST. Generated C++ compiles with g++ for all 3 example models. Stop here until this passes.

### Step 36: Agent API
- Orchestrator accepts JSON-RPC from external clients (same protocol as ImGui)
- `getValidConstructs`, `insertNode`, `setProperty`, `getInScopeSymbols`
- **Test:** Python script acts as agent, builds a Function node by node via RPC

### Step 37: Agent batch mode
- `insertSubtree(parentId, role, jsonSubtree)` — submit entire subtree at once
- Orchestrator validates recursively, applies atomically (all or nothing)
- **Test:** agent submits full ConditionalExample as JSON, orchestrator builds the AST

> **CHECKPOINT:** Agent builds AST remotely. Python script creates full ConditionalExample via RPC. Stop here until this passes.

### Step 38: OptimizationLock warnings
- When mutating a locked node, orchestrator returns warning (not rejection)
- ImGui shows lock icon + banner
- Shadow system: original annotation preserved, modification tracked via `@provenance`
- **Test:** lock a function, modify it, verify warning fires and original preserved

---

## Summary

| Phase | Steps | What You Have When Done |
|-------|-------|------------------------|
| 2a: AST in C++ | 1–7 | 33 concepts as C++ objects, JSON round-trip, schema validation |
| 2b: First Generator | 8–11 | Python output matching MPS for all examples |
| 2c: Dear ImGui Shell | 12–15 | Window showing live AST projection with toggle |
| 2d: Orchestrator | 16–21 | Separate process, RPC mutations, undo/redo, ImGui connected |
| 2e: Structured Editing | 22–25 | Build code through UI, scope-aware dropdowns |
| 2f: Emacs Integration | 26–29 | File ops, search, git via headless Emacs servers |
| 2g: Elisp Projection | 30–33 | Generate and parse Elisp, round-trip verified |
| 2h: C++ + Agent API | 34–38 | C++ generator with memory strategies, agent RPC interface |

## Dependencies

| Dependency | Version | Purpose |
|------------|---------|---------|
| Dear ImGui | 1.90+ | GUI rendering |
| SDL2 or GLFW | latest | Window/input backend for ImGui |
| Emacs | 29+ | Headless servers |
| nlohmann/json | 3.11+ | JSON serialization, JSON-RPC |
| tree-sitter | 0.22+ | Elisp/Python/C++ parsing |
| tree-sitter-elisp | latest | Elisp grammar |
| C++ compiler | C++20 | Entire C++ stack |
| GoogleTest | latest | C++ unit tests |