# Sprint 2 Vision: Whetstone Editor & Structured Editing ## Vision Statement Build a next-generation code editor that combines Emacs's powerful libraries with a modern, symbol-rich UI, and introduce **structured editing** where humans and AI agents construct code by choosing from legal primitives rather than typing syntax. --- ## Goals 1. **Hybrid Editor Architecture** - Dear ImGui shell + headless Emacs servers 2. **Structured Editing UI** - Choice-based code construction, syntax errors impossible 3. **Unified Human/Agent Interface** - Same API for clicking and scripting 4. **Elisp ↔ C++ Projection** - Dogfood Whetstone by building the editor with it 5. **Proof of Concept** - Demonstrate AST-native development accelerates iteration --- ## Architecture: Dear ImGui + Headless Emacs ### Why Hybrid? | Approach | Pros | Cons | |----------|------|------| | Fork Emacs | Full compatibility | Massive codebase, GUI baggage | | Pure Dear ImGui | Clean slate, full control | Lose Emacs libraries | | **Hybrid** | Best of both | Complexity in coordination | The hybrid approach gives us: - **Emacs libraries** without Emacs's GUI limitations - **Dear ImGui flexibility** without reimplementing text processing - **Parallelism** via multiple Emacs server instances - **Upstream compatibility** - Emacs updates just work ### System Architecture ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Whetstone Editor │ ├─────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Dear ImGui Shell (C++) │ │ │ │ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ │ │ File │ │ Editor │ │ Terminal │ │ Structured │ │ │ │ │ │ Tree │ │ Tabs │ │ Panel │ │ Edit Panel │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ 📁 src │ │ [foo.py] │ │ $ _ │ │ [Function ▼] │ │ │ │ │ │ 📁 lib │ │ [bar.cpp]│ │ │ │ [Variable ▼] │ │ │ │ │ │ 📄 main │ │ │ │ │ │ [Expression ▼] │ │ │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │ │ │ │ │ │ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ │ │ Command Palette (Emacs-style, keyboard-driven) │ │ │ │ │ │ > find-file > magit-status > consult-ripgrep │ │ │ │ │ └──────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ IPC (JSON-RPC / Shared Memory) │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Management Node (C++) │ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │ │ │ │ Whetstone │ │ Server │ │ Agent │ │ │ │ │ │ AST Store │ │ Pool Mgr │ │ Queue │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ [AST Nodes] │ │ spawn() │ │ [Choice sequences from │ │ │ │ │ │ [Annotations│ │ route() │ │ human clicks or │ │ │ │ │ │ [Projections│ │ sync() │ │ agent submissions] │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ │ emacsclient protocol │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Headless Emacs Server Pool │ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ │ │ Server 1 │ │ Server 2 │ │ Server N │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ File I/O │ │ Search │ │ Git (Magit) │ │ │ │ │ │ TRAMP │ │ Completion │ │ Org-mode │ │ │ │ │ │ Dired │ │ Helm/Vertico│ │ LSP bridge │ │ │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ │ (Each runs: emacs --daemon=serverN --load whetstone-bridge.el) │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────┘ ``` ### Component Responsibilities | Component | Language | Responsibility | |-----------|----------|----------------| | **Dear ImGui Shell** | C++ | All rendering, keyboard/mouse input, widgets | | **Management Node** | C++ | AST storage, server coordination, agent queue | | **Emacs Servers** | Elisp | Text processing, file ops, search, completion | **Future Goal:** Once Whetstone is mature, rewrite the Management Node in Whetstone AST and generate C++ from it. This is the ultimate dogfooding - the editor's core logic defined in the system it's built to create. ### Communication Protocol ``` ┌────────────┐ JSON-RPC ┌─────────────┐ │ ImGui Shell│ ◄─────────────────────────► │ Mgmt Node │ └────────────┘ └─────────────┘ │ │ emacsclient -e '(elisp-expr)' ▼ ┌─────────────┐ │ Emacs Server│ └─────────────┘ ``` **Example flow - User searches for file:** 1. User presses `Ctrl+P` in ImGui shell 2. ImGui sends `{ "method": "find-file", "query": "" }` to Management Node 3. Management Node routes to Search server: `emacsclient -s search -e '(consult-find "")'` 4. Emacs returns candidates 5. Management Node sends candidates to ImGui 6. ImGui renders searchable dropdown 7. User selects file 8. Management Node loads file into AST, sends to ImGui for rendering --- ## Structured Editing: The Core Innovation ### The Problem with Text-Based Coding ``` Traditional flow: Human/Agent generates text → Parser validates → Errors → Fix → Repeat Problems: - Syntax errors waste time - Agents hallucinate invalid syntax - Autocompletion is probabilistic (might suggest wrong things) - Context is implicit (what's in scope? what types are valid?) ``` ### The Solution: Choice-Based Construction **You never type code. You choose from valid options at each step.** ``` Structured flow: Choose construct → Fill constrained fields → Choose next construct → Done Benefits: - Syntax errors IMPOSSIBLE - Every option shown is LEGAL - Context is EXPLICIT (only in-scope items offered) - Agents submit choice sequences (no parsing, no errors) ``` ### Structured Editing UI Flow #### Step 1: Choose Construct Type ``` ┌─────────────────────────────────────────────────────────────────┐ │ What do you want to create? │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Function │ │ Class │ │ Variable │ │ Import │ │ │ │ λ │ │ ◇ │ │ x= │ │ ↓ │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ If │ │ For │ │ While │ │ Return │ │ │ │ ? │ │ ↻ │ │ ↺ │ │ ← │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ [Keyboard: f=Function, c=Class, v=Variable, i=Import...] │ └─────────────────────────────────────────────────────────────────┘ ``` #### Step 2: Fill Constrained Fields (Function selected) ``` ┌─────────────────────────────────────────────────────────────────┐ │ Define Function │ │ │ │ Name: [process_data________] ← Text input, identifier rules│ │ │ │ Parameters: │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Name Type Default │ │ │ │ [items ] [List[int] ▼] [None ▼] │ │ │ │ [threshold] [int ▼] [10 ] │ │ │ │ │ │ │ │ [+ Add Parameter] │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ Return Type: [List[int] ▼] [None] [Auto-infer] │ │ │ │ Annotations: │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ [@deref: batched ▼] [@complexity: ___] [+ Add] │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ [Continue to Body →] │ └─────────────────────────────────────────────────────────────────┘ ``` #### Step 3: Build Function Body ``` ┌─────────────────────────────────────────────────────────────────┐ │ Function: process_data(items: List[int], threshold: int) │ │ │ │ Body: (Click to add statement) │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ 1. [+ Add Statement] │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ ───────────────────────────────────────────────────────────── │ │ Choose statement type: │ │ │ │ [Variable] [If] [For] [While] [Return] [Call] [Assign] │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` #### Step 4: Add Variable (Variable selected) ``` ┌─────────────────────────────────────────────────────────────────┐ │ Create Variable │ │ │ │ Name: [result_________] │ │ │ │ Initialize with: │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Literal │ │ Variable │ │ Call │ │ BinaryOp │ │ │ │ 42 │ │ x │ │ f() │ │ a+b │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` #### Step 5: Build Expression (Call selected) ``` ┌─────────────────────────────────────────────────────────────────┐ │ Function Call │ │ │ │ Function: [filter ▼] ← Dropdown: in-scope functions │ │ ┌─────────────────┐ │ │ │ filter │ ← Built-in │ │ │ map │ ← Built-in │ │ │ process_data │ ← Current function (recursion) │ │ │ helper_func │ ← Defined above │ │ └─────────────────┘ │ │ │ │ Arguments: │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Arg 1 (callable): [lambda ▼] → [x > threshold] │ │ │ │ Arg 2 (iterable): [items ▼] ← Only iterables shown │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ [Done - Add to Body] │ └─────────────────────────────────────────────────────────────────┘ ``` #### Result: Complete Function ``` ┌─────────────────────────────────────────────────────────────────┐ │ Function: process_data │ │ ├── params: [(items, List[int]), (threshold, int, default=10)]│ │ ├── returns: List[int] │ │ ├── @deref: batched │ │ └── body: │ │ └── Return │ │ └── Call: list │ │ └── Call: filter │ │ ├── Lambda: x > threshold │ │ └── Var: items │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Preview (Python): │ │ │ │ │ │ │ │ def process_data(items: List[int], threshold: int = 10): │ │ │ │ return list(filter(lambda x: x > threshold, items)) │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ [Edit] [View as C++] [Add Annotation] [Done] │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## Agent API: Scripted Choices The structured editing UI has a direct JSON equivalent. Agents don't generate code text - they submit choice sequences. ### Human Interaction (UI) ``` Click: Function → Type: "process_data" → Click: Add Param → ... ``` ### Agent Interaction (JSON) ```json { "construct": "Function", "name": "process_data", "params": [ { "name": "items", "type": "List[int]" }, { "name": "threshold", "type": "int", "default": "10" } ], "return_type": "List[int]", "annotations": [ { "type": "deref", "strategy": "batched" } ], "body": [ { "statement": "Return", "expression": { "type": "Call", "function": "list", "args": [ { "type": "Call", "function": "filter", "args": [ { "type": "Lambda", "params": ["x"], "body": { "type": "BinaryOp", "left": "x", "op": ">", "right": "threshold" } }, { "type": "VariableRef", "name": "items" } ] } ] } } ] } ``` ### Validation Guarantees The Management Node validates each choice: ``` Agent submits: { "function": "nonexistent_func", ... } System responds: { "error": "InvalidChoice", "field": "function", "valid_options": ["filter", "map", "process_data", "helper_func"], "reason": "Function 'nonexistent_func' not in scope" } ``` **No parsing errors. No syntax errors. Only semantic validation.** ### Agent Workflow: Context-Aware Queuing A local agent with project context can pre-compute likely choices: ``` ┌─────────────────────────────────────────────────────────────────┐ │ Agent Context │ │ ├── Current file: data_processor.py │ │ ├── Cursor: inside class DataProcessor │ │ ├── In scope: self, items, config, logger │ │ └── Recent pattern: user adding validation methods │ │ │ │ Agent predicts user wants: validation method │ │ Pre-queued choices: │ │ 1. Function (method) │ │ 2. Name: "validate_items" (inferred from context) │ │ 3. Param: items (already in scope) │ │ 4. Return: bool (validation pattern) │ │ 5. Body: If → condition → Return True/False │ │ │ │ User sees: "Create validate_items method? [Yes] [Modify] [No]"│ └─────────────────────────────────────────────────────────────────┘ ``` Instead of: ``` Agent: "Here's the code I generated..." Human: "Almost, but change X" Agent: "Here's the updated code..." Human: "Now there's a syntax error" Agent: "Sorry, here's the fix..." ``` It becomes: ``` Agent: "I suggest these choices: [validate_items, bool, If pattern]" Human: Clicks [Modify] → changes return type to Optional[str] Agent: Updates downstream choices automatically Human: Clicks [Accept] Done. No back-and-forth. No errors. ``` --- ## Elisp ↔ C++ Projection ### Why These Languages? | Language | Role in Editor | |----------|----------------| | **Elisp** | Emacs server scripting, configuration, extensions | | **C++** | Dear ImGui shell, performance-critical paths | Building Elisp ↔ C++ transpilation lets us **dogfood** Whetstone: - Write editor logic in Elisp (comfortable, existing packages) - Generate C++ for performance-critical rendering - Same AST, different projections ### Language Mapping | Elisp | Whetstone AST | C++ | |-------|---------------|-----| | `(defun name (args) body)` | Function node | `RetType name(Args) { body }` | | `(let ((x val)) body)` | Variable + Block | `{ auto x = val; body }` | | `(if cond then else)` | IfStatement | `if (cond) { then } else { else }` | | `(lambda (x) body)` | Lambda | `[](auto x) { return body; }` | | `(mapcar fn list)` | Call: map | `std::transform(...)` | | `'(1 2 3)` | ListLiteral | `std::vector{1, 2, 3}` | ### Elisp-Specific Idioms ``` @lang_specific(elisp, "special_form", "(interactive \"sPrompt: \")", hint="user_input") @lang_specific(elisp, "macro", "(with-temp-buffer ...)", hint="scoped_resource") @lang_specific(elisp, "advice", "(advice-add 'fn :around ...)", hint="aspect_oriented") ``` ### C++-Specific Idioms ``` @lang_specific(cpp, "template", "template", hint="generic") @lang_specific(cpp, "raii", "std::lock_guard", hint="scoped_resource") @lang_specific(cpp, "constexpr", "constexpr", hint="compile_time") ``` --- ## Dear ImGui Integration ### What is Dear ImGui? Immediate-mode GUI library. Every frame, you declare what to draw: ```cpp // C++ Dear ImGui void render() { if (ImGui::Button("Compile")) { runCompile(); } ImGui::SliderInt("Opt Level", &optLevel, 0, 3); ImGui::Text("Status: %s", status.c_str()); } ``` ### Elisp Wrapper (whetstone-imgui.el) ```elisp ;; Elisp Dear ImGui bindings (defun render () (imgui-button "Compile" #'run-compile) (imgui-slider-int "Opt Level" 'opt-level 0 3) (imgui-text "Status: %s" status)) ``` ### AST-Defined UI The Whetstone AST can define UI elements: ``` UIPanel: "Build Settings" ├── @projection: imgui ├── children: │ ├── UIButton │ │ ├── label: "Compile" │ │ ├── icon: "hammer" │ │ └── action: → BuildSystem.compile() │ ├── UISlider │ │ ├── label: "Optimization Level" │ │ ├── binding: → config.optLevel │ │ ├── min: 0 │ │ └── max: 3 │ └── UIText │ ├── template: "Status: {}" │ └── binding: → BuildSystem.status ``` **Same AST, multiple UI projections:** | Projection | Output | |------------|--------| | `@projection(imgui)` | Native Dear ImGui widgets | | `@projection(emacs)` | Text-based `[Compile] Opt: [===----] 2` | | `@projection(html)` | Web form with `