added requirements docs
This commit is contained in:
Bill
328
docs-20260203T163105Z-3-001/docs/REQUIREMENTS_OVERVIEW.md
Normal file
328
docs-20260203T163105Z-3-001/docs/REQUIREMENTS_OVERVIEW.md
Normal file
@@ -0,0 +1,328 @@
|
|||||||
|
# Whetstone DSL: Project Requirements Overview
|
||||||
|
|
||||||
|
## Vision Statement
|
||||||
|
|
||||||
|
**Source code should be a structured database of logic, not a text file of syntax.**
|
||||||
|
|
||||||
|
Whetstone is a Universal Logic Database that stores code as a semantic graph (AST), with traditional programming languages serving as **projections** of that canonical representation. This enables lossless transpilation, massively parallel development (human and AI agents), and complete separation of intent from implementation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Core Philosophy
|
||||||
|
|
||||||
|
### The Problem: Text Blindness
|
||||||
|
Current software engineering couples **Intent** (what we want to happen) with **Implementation** (how the machine does it) through language-specific syntax. This creates:
|
||||||
|
- Lock-in to specific languages and ecosystems
|
||||||
|
- Loss of semantic information during translation
|
||||||
|
- Inability to express cross-cutting concerns (ownership, risk, optimization hints)
|
||||||
|
- Barriers to parallel development by multiple agents
|
||||||
|
|
||||||
|
### The Solution: Projectional Editing
|
||||||
|
Move the "Source of Truth" from text files to a **Logical Graph Database**:
|
||||||
|
|
||||||
|
| Text-Based | Graph-Based (Whetstone) |
|
||||||
|
|------------|-------------------------|
|
||||||
|
| Linear | Structural |
|
||||||
|
| Ambiguous | Explicit |
|
||||||
|
| Syntax-heavy | Meaning-heavy |
|
||||||
|
| Language-specific | Universal |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture: The Three Layers
|
||||||
|
|
||||||
|
### Layer 1: The Logic Database (Definition)
|
||||||
|
A schema of pure logical concepts stored in a graph/tree database (JetBrains MPS).
|
||||||
|
|
||||||
|
**Core Principle:** This layer must be a **superset** of all target languages. It must express concepts like memory ownership (Rust-style) even when targeting languages that don't require them (Python).
|
||||||
|
|
||||||
|
**Contents:**
|
||||||
|
- AST node concepts (expressions, statements, functions, types)
|
||||||
|
- Semantic annotations (SemAnno schema)
|
||||||
|
- Hypergraph relationships (nodes participate in multiple relationships)
|
||||||
|
|
||||||
|
### Layer 2: Semantic Generators (Bridge)
|
||||||
|
Generators that translate the Logic Database into idiomatic text for specific runtimes.
|
||||||
|
|
||||||
|
**Example - SharedMemoryBlock concept:**
|
||||||
|
```
|
||||||
|
Generator A (Python): x = [1, 2, 3] # Note: Shared usage implied
|
||||||
|
Generator B (Rust): let x = Arc::new(Mutex::new(vec![1, 2, 3]));
|
||||||
|
Generator C (Verilog): reg [31:0] memory [0:2];
|
||||||
|
```
|
||||||
|
|
||||||
|
### Layer 3: Runtime (Metal)
|
||||||
|
Standard toolchains (GCC, LLVM, Python interpreter) compile/execute the generated code. Whetstone does not replace compilers; it feeds them optimized, correct input.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SemAnno: Universal Semantic Annotation Schema
|
||||||
|
|
||||||
|
Six annotation systems provide metadata for agents (human and AI):
|
||||||
|
|
||||||
|
### 1. Complexity & Intelligence ("Rank" System)
|
||||||
|
Defines who/what is qualified to modify a node.
|
||||||
|
|
||||||
|
| Annotation | Type | Purpose |
|
||||||
|
|------------|------|---------|
|
||||||
|
| `@complexity-score` | int | Derived from cyclomatic complexity, nesting, mutation |
|
||||||
|
| `@intelligence-requirement` | junior/mid/senior | Agent tier required for modification |
|
||||||
|
| `@review-lock` | bool | Requires second agent verification |
|
||||||
|
|
||||||
|
### 2. Dependency & Impact ("Radius" System)
|
||||||
|
Calculates the "blast radius" of changes.
|
||||||
|
|
||||||
|
| Annotation | Type | Purpose |
|
||||||
|
|------------|------|---------|
|
||||||
|
| `@dependency-count` | int | Incoming references to this node |
|
||||||
|
| `@dependency-level` | 0-100 | Normalized importance score |
|
||||||
|
| `@downstream-impact` | list | Affected domains [Network, Security, Billing] |
|
||||||
|
| `@fragility` | low/high | Load-bearing legacy code flag |
|
||||||
|
|
||||||
|
### 3. Performance & Resource ("Machinist" System)
|
||||||
|
Low-level optimization metadata.
|
||||||
|
|
||||||
|
| Annotation | Type | Purpose |
|
||||||
|
|------------|------|---------|
|
||||||
|
| `@optimization-priority` | low/mid/high | Generator focus for SIMD/unrolling |
|
||||||
|
| `@latency-critical` | bool | Real-time execution requirement |
|
||||||
|
| `@memory-footprint` | estimate | Heap/stack allocation intent |
|
||||||
|
| `@execution-mode` | deterministic/probabilistic | Formal verification hint |
|
||||||
|
| `@deref-explicit` | bool | Every pointer touch must be accounted |
|
||||||
|
|
||||||
|
### 4. Architectural Intent ("Paradigm" System)
|
||||||
|
Defines behavioral contracts.
|
||||||
|
|
||||||
|
| Annotation | Type | Purpose |
|
||||||
|
|------------|------|---------|
|
||||||
|
| `@intent` | Pure/Stateful/Parallel/Legacy | Primary behavioral contract |
|
||||||
|
| `@preferred-view` | language | Optimal projection for readability |
|
||||||
|
| `@lifecycle` | owner/borrower/transient | Ownership model hints |
|
||||||
|
|
||||||
|
### 5. Security & Risk ("Sentinel" System)
|
||||||
|
Granular risk tracking.
|
||||||
|
|
||||||
|
| Annotation | Type | Purpose |
|
||||||
|
|------------|------|---------|
|
||||||
|
| `@risk-type` | identifier | BufferOverflow, RaceCondition, SideChannel |
|
||||||
|
| `@trust-boundary` | marker | Secure environment transition point |
|
||||||
|
| `@provenance` | hash | Chain of agent modifications |
|
||||||
|
|
||||||
|
### 6. Projective Metadata ("Lens" System)
|
||||||
|
Controls rendering in projectional environments.
|
||||||
|
|
||||||
|
| Annotation | Type | Purpose |
|
||||||
|
|------------|------|---------|
|
||||||
|
| `@view-mask` | category | Toggle annotation visibility by type |
|
||||||
|
| `@cognitive-level` | 0-100 | Detail density filter |
|
||||||
|
| `@projection-alias` | string | Localized renaming without AST change |
|
||||||
|
| `@hidden-detail` | bool | Collapse boilerplate for reviews |
|
||||||
|
|
||||||
|
### 7. Language-Specific Idioms ("Polyglot" System)
|
||||||
|
Preserves language-specific features that don't map directly to other languages.
|
||||||
|
|
||||||
|
| Annotation | Type | Purpose |
|
||||||
|
|------------|------|---------|
|
||||||
|
| `@lang_specific` | structured | Captures idiom with language, type, raw syntax, and semantic hint |
|
||||||
|
|
||||||
|
**Architecture:**
|
||||||
|
```
|
||||||
|
@lang_specific(python, "decorator", "@lru_cache(maxsize=128)", hint="memoization")
|
||||||
|
@lang_specific(cpp, "template", "template<typename T>", hint="generic")
|
||||||
|
```
|
||||||
|
|
||||||
|
**Projection Behavior:**
|
||||||
|
- In native language: renders as original syntax
|
||||||
|
- In foreign language: renders as comment with semantic hint
|
||||||
|
|
||||||
|
**Long-term Vision:** These annotations enable future semantic equivalence detection. When the system understands that `@lru_cache` and a C++ `std::map`-based cache serve the same purpose ("memoization"), it can generate idiomatic implementations in either direction rather than just commenting.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Memory Dereferencing Strategies
|
||||||
|
|
||||||
|
Memory management is decoupled from logic as a first-class AST field.
|
||||||
|
|
||||||
|
| Strategy | Description | Generator Responsibility |
|
||||||
|
|----------|-------------|-------------------------|
|
||||||
|
| `@deref(imperative)` | Manual control - developer defines time/location | Whetstone Generator / Substrate |
|
||||||
|
| `@deref(streamed)` | Ownership-based - lifetime tracked at declaration (Rust-like) | Whetstone Generator |
|
||||||
|
| `@deref(batched)` | Garbage collected - background process or ref-counting | Whetstone Generator |
|
||||||
|
| `@deref(content-addressed)` | Immutable - identity by hash, location irrelevant | Whetstone Generator |
|
||||||
|
|
||||||
|
### The "Unfilled Node Constraint"
|
||||||
|
In `@deref(imperative)` mode, the AST requires an explicit **Dereference Time Field**. If empty, the editor flags it as a "Missing Intent" error. This makes it impossible to forget a dereference.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Strategy Toggles
|
||||||
|
|
||||||
|
Declarative optimization patterns applicable without manual implementation:
|
||||||
|
|
||||||
|
| Toggle | Effect |
|
||||||
|
|--------|--------|
|
||||||
|
| `@strategy(TCO)` | Convert recursive logic to iterative machine code |
|
||||||
|
| `@strategy(Memoize)` | Inject thread-safe cache for function results |
|
||||||
|
| `@strategy(Unroll)` | Duplicate loop bodies per target-hardware heuristic |
|
||||||
|
| `@strategy(SSO)` | Enable Small String/Buffer Optimization |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Transpilation Modes
|
||||||
|
|
||||||
|
### Direct Transpilation ("Wrapper")
|
||||||
|
Maps nodes 1-to-1. High compatibility, poor performance.
|
||||||
|
- Python dynamic list → `std::vector<std::any>` with runtime type-checking
|
||||||
|
|
||||||
|
### Idiomatic Transpilation ("Intent")
|
||||||
|
Analyzes usage patterns. High performance, hardware sympathy.
|
||||||
|
- Python list (append-only, sequential read) → `std::vector<T>` or Memory Arena
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Resolution Slider
|
||||||
|
|
||||||
|
Four levels from intent to execution:
|
||||||
|
|
||||||
|
| Level | Description | Example |
|
||||||
|
|-------|-------------|---------|
|
||||||
|
| 1 | Pseudocode/Intent | "Find a user in a list" |
|
||||||
|
| 2 | High-Level (JS/Python) | `Array.find()` |
|
||||||
|
| 3 | Intermediate (Rust/Java) | Type safety + ownership scopes |
|
||||||
|
| 4 | Execution Substrate (C++/C) | B-Trees, SIMD, Branch Hints |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Multi-Agent Collaboration
|
||||||
|
|
||||||
|
### The Round-Trip Problem
|
||||||
|
When a systems engineer optimizes code that a high-level developer later modifies.
|
||||||
|
|
||||||
|
### Solution: Persistent Metadata Separation
|
||||||
|
- **Logic Field:** Algorithmic intent (editable by junior)
|
||||||
|
- **Optimization Field:** Systems-level instructions (requires senior)
|
||||||
|
|
||||||
|
### Conflict Management
|
||||||
|
When high-level changes break low-level optimizations:
|
||||||
|
1. **Shadowed Metadata:** Old optimizations hidden but not deleted
|
||||||
|
2. **Alerting:** "Performance Guardrail" warning shown
|
||||||
|
3. **Diff Lens:** Visualizes where intent changes invalidated hardware instructions
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Hypergraph Requirement
|
||||||
|
|
||||||
|
Standard ASTs are strictly hierarchical (Parent → Child). Real-world logic is a **Hypergraph**:
|
||||||
|
|
||||||
|
- A Variable is **owned by** Function A
|
||||||
|
- But **used by** Function B
|
||||||
|
- And **monitored by** Agent C
|
||||||
|
|
||||||
|
**Implementation:** Smart References (Hyperedges) allow a single logical node to participate in multiple relationships simultaneously.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Agent-Native Development
|
||||||
|
|
||||||
|
### Task Decomposition Model
|
||||||
|
1. **High-context coordinator agent** builds semantic graph and queues tasks
|
||||||
|
2. **Small Language Models (8B)** handle tightly scoped tasks
|
||||||
|
3. **Human developers** work from the same task queue
|
||||||
|
|
||||||
|
### Projection Optimization
|
||||||
|
Tasks can be projected in:
|
||||||
|
- **Human-optimized view:** Natural language descriptions, readable code
|
||||||
|
- **AI-optimized view:** Structured data, explicit constraints, minimal ambiguity
|
||||||
|
|
||||||
|
### Agent Tiers
|
||||||
|
| Tier | Capabilities |
|
||||||
|
|------|--------------|
|
||||||
|
| Junior | Basic logic, no memory management, no high-risk annotations |
|
||||||
|
| Mid | Standard logic, `@deref(streamed)` and `@deref(batched)` |
|
||||||
|
| Senior | `@deref(imperative)`, hardware optimization, high-risk refactoring |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Target Applications
|
||||||
|
|
||||||
|
### Legacy Modernization
|
||||||
|
Parse old Fortran/COBOL → Logic Database → Generate modern C++/Rust/Julia
|
||||||
|
|
||||||
|
### Cross-Platform Logic
|
||||||
|
Write business logic once → Generate Kotlin (Android), Swift (iOS), Rust (backend)
|
||||||
|
|
||||||
|
### Agentic Coding
|
||||||
|
Agents manipulate Graph Nodes directly (no syntax errors from text generation)
|
||||||
|
|
||||||
|
### Parallel Development
|
||||||
|
Multiple developers/agents work on same codebase with explicit conflict boundaries
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Technical Principles
|
||||||
|
|
||||||
|
### Behavior Over Implementation
|
||||||
|
A node's primary definition is its **Behavioral Contract** (e.g., "Sort this collection") rather than implementation (e.g., "Quicksort with pointer increments").
|
||||||
|
|
||||||
|
### Substrate-Agnostic Core Nodes
|
||||||
|
Core AST nodes must not assume specific memory models or hardware constraints.
|
||||||
|
- **Bad:** LoopNode requires a "Pointer" field
|
||||||
|
- **Good:** LoopNode requires an "IteratorIntent" - realization is metadata
|
||||||
|
|
||||||
|
### The Semantic Boundary
|
||||||
|
Strict boundary between **What** (Logic) and **How** (Execution Strategy). Future capabilities are implemented by adding Strategy Engines to existing nodes, not re-architecting nodes.
|
||||||
|
|
||||||
|
### Strictest Common Denominator
|
||||||
|
The Logic Database enforces strict rules by default (explicit ownership), while Generators can "relax" rules for permissive languages or "poly-fill" rules for strict languages.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The C Primitives Substrate (Long-term Vision)
|
||||||
|
|
||||||
|
All high-level language features ultimately resolve to C-level primitives:
|
||||||
|
|
||||||
|
| High-Level Feature | C Primitive Resolution |
|
||||||
|
|-------------------|------------------------|
|
||||||
|
| Python `@lru_cache` | Hash table + function pointer + malloc/free |
|
||||||
|
| C++ `template<T>` | N compiled copies with concrete types |
|
||||||
|
| Python decorator | Function pointer manipulation |
|
||||||
|
| C++ RAII | Constructor/destructor at scope boundaries |
|
||||||
|
| Rust ownership | Compile-time tracking → equivalent to manual C |
|
||||||
|
| Garbage collection | Reference counting or mark-sweep algorithms |
|
||||||
|
|
||||||
|
**Architectural Implication:**
|
||||||
|
|
||||||
|
```
|
||||||
|
Level 4: Language Idioms @py_decl, @cpp_tmpl, @rust_lifetime
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Level 3: Universal AST Function, Loop, Variable, Expression
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Level 2: Semantic Intent "memoize", "parameterize by type", "scope-bound cleanup"
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Level 1: C Primitives pointers, structs, malloc/free, function calls
|
||||||
|
```
|
||||||
|
|
||||||
|
**End State:** Languages become pure *syntactic projections* over C primitives + semantic annotations. This enables true lossless transpilation because the canonical form captures both the intent (semantic annotations) and the implementation (C primitives).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Platform
|
||||||
|
|
||||||
|
**JetBrains MPS (Meta Programming System)**
|
||||||
|
- Projectional editor (edit AST directly, not text)
|
||||||
|
- Language workbench for defining DSLs
|
||||||
|
- Built-in support for generators
|
||||||
|
- Extensible type system and constraints
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
1. **Lossless Round-Trip:** Python → AST → C++ → AST → Python produces semantically equivalent code
|
||||||
|
2. **Parallel Safety:** Junior developer cannot accidentally break senior's optimizations
|
||||||
|
3. **Agent Compatibility:** Tasks can be completed by SLMs with sufficient annotation context
|
||||||
|
4. **Performance Parity:** Generated C++ matches hand-written C++ for common patterns
|
||||||
|
5. **Adoption Path:** Existing codebases can be incrementally imported via tree-sitter parsing
|
||||||
548
docs-20260203T163105Z-3-001/docs/SPRINT_1_REQUIREMENTS.md
Normal file
548
docs-20260203T163105Z-3-001/docs/SPRINT_1_REQUIREMENTS.md
Normal file
@@ -0,0 +1,548 @@
|
|||||||
|
# Sprint 1: Python ↔ C++ Dual Projection
|
||||||
|
|
||||||
|
## Sprint Goal
|
||||||
|
|
||||||
|
Enable a junior developer to write working logic in Python, a senior developer to optimize it in C++, and the junior developer to continue modifying safe areas while receiving warnings about optimization-locked regions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
| In Scope | Out of Scope |
|
||||||
|
|----------|--------------|
|
||||||
|
| Python and C++ projections | Other languages |
|
||||||
|
| Tree-sitter parsing for ingestion | Custom parser implementation |
|
||||||
|
| Memory deref strategy annotations | Full SemAnno schema |
|
||||||
|
| Warning system for locked nodes | Hard locks / approval workflows |
|
||||||
|
| Basic AST nodes (functions, loops, variables, expressions) | Advanced nodes (generics, macros, templates) |
|
||||||
|
| MPS projectional editor | IDE plugins (VS Code, etc.) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## User Stories
|
||||||
|
|
||||||
|
### US-1: Python Developer Writes Logic
|
||||||
|
**As a** junior Python developer
|
||||||
|
**I want to** write business logic in a Python-like projection
|
||||||
|
**So that** I can focus on correctness without worrying about memory management
|
||||||
|
|
||||||
|
### US-2: C++ Developer Optimizes
|
||||||
|
**As a** senior C++ developer
|
||||||
|
**I want to** switch to a C++ projection and add optimization annotations
|
||||||
|
**So that** the generated code is production-ready and performant
|
||||||
|
|
||||||
|
### US-3: Python Developer Sees Warnings
|
||||||
|
**As a** junior Python developer
|
||||||
|
**I want to** see warnings when I try to modify optimized nodes
|
||||||
|
**So that** I understand the impact of my changes without being blocked
|
||||||
|
|
||||||
|
### US-4: Import Existing Code
|
||||||
|
**As a** developer
|
||||||
|
**I want to** import existing Python or C++ files into the AST
|
||||||
|
**So that** I can work with legacy code in Whetstone
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Technical Requirements
|
||||||
|
|
||||||
|
### TR-1: Core AST Nodes
|
||||||
|
|
||||||
|
Define MPS concepts for a minimal but complete AST:
|
||||||
|
|
||||||
|
```
|
||||||
|
Module
|
||||||
|
├── Function
|
||||||
|
│ ├── name: string
|
||||||
|
│ ├── parameters: Parameter[]
|
||||||
|
│ ├── returnType: TypeReference
|
||||||
|
│ ├── body: Statement[]
|
||||||
|
│ └── annotations: SemAnno[]
|
||||||
|
├── Variable
|
||||||
|
│ ├── name: string
|
||||||
|
│ ├── type: TypeReference
|
||||||
|
│ ├── initializer: Expression?
|
||||||
|
│ └── annotations: SemAnno[]
|
||||||
|
└── annotations: SemAnno[]
|
||||||
|
|
||||||
|
Statement (abstract)
|
||||||
|
├── Assignment
|
||||||
|
├── IfStatement
|
||||||
|
├── WhileLoop
|
||||||
|
├── ForLoop
|
||||||
|
├── Return
|
||||||
|
├── ExpressionStatement
|
||||||
|
└── Block
|
||||||
|
|
||||||
|
Expression (abstract)
|
||||||
|
├── BinaryOperation
|
||||||
|
├── UnaryOperation
|
||||||
|
├── FunctionCall
|
||||||
|
├── VariableReference
|
||||||
|
├── Literal (int, float, string, bool)
|
||||||
|
├── ListLiteral
|
||||||
|
├── IndexAccess
|
||||||
|
└── MemberAccess
|
||||||
|
```
|
||||||
|
|
||||||
|
### TR-2: Memory Deref Strategy Annotation
|
||||||
|
|
||||||
|
```
|
||||||
|
DerefStrategy
|
||||||
|
├── strategy: enum {imperative, streamed, batched, content_addressed}
|
||||||
|
├── derefTime: Expression? (required if imperative)
|
||||||
|
├── derefLocation: string? (required if imperative)
|
||||||
|
└── owner: AgentReference?
|
||||||
|
```
|
||||||
|
|
||||||
|
### TR-3: Optimization Lock Annotation
|
||||||
|
|
||||||
|
```
|
||||||
|
OptimizationLock
|
||||||
|
├── lockedBy: AgentReference
|
||||||
|
├── lockReason: string
|
||||||
|
├── lockLevel: enum {warning, soft, hard} // Sprint 1: warning only
|
||||||
|
├── affectedStrategies: string[]
|
||||||
|
└── timestamp: datetime
|
||||||
|
```
|
||||||
|
|
||||||
|
### TR-4: Language-Specific Idiom Annotation
|
||||||
|
|
||||||
|
Placeholder annotations that preserve language-specific features during import without requiring full semantic understanding.
|
||||||
|
|
||||||
|
```
|
||||||
|
LangSpecific
|
||||||
|
├── language: enum {python, cpp, rust, ...}
|
||||||
|
├── idiomType: string // "decorator", "template", "attribute", "pragma", etc.
|
||||||
|
├── rawSyntax: string // The original syntax as written
|
||||||
|
├── semanticHint: string? // Optional: "memoization", "generic", "compile-hint"
|
||||||
|
└── position: enum {before, after, wrapping} // Where it attaches to the node
|
||||||
|
```
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
|
||||||
|
```
|
||||||
|
// Python decorator captured during import
|
||||||
|
@lang_specific(python, "decorator", "@lru_cache(maxsize=128)", hint="memoization", position=before)
|
||||||
|
|
||||||
|
// C++ template captured during import
|
||||||
|
@lang_specific(cpp, "template", "template<typename T>", hint="generic", position=before)
|
||||||
|
|
||||||
|
// C++ pragma
|
||||||
|
@lang_specific(cpp, "pragma", "#pragma omp parallel for", hint="parallelization", position=before)
|
||||||
|
|
||||||
|
// Python type hint that has no C++ equivalent
|
||||||
|
@lang_specific(python, "type_hint", "-> Generator[int, None, None]", hint="generator", position=after)
|
||||||
|
|
||||||
|
// C++ attribute
|
||||||
|
@lang_specific(cpp, "attribute", "[[nodiscard]]", hint="return_value_check", position=before)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Projection Behavior:**
|
||||||
|
|
||||||
|
| In Native Projection | In Foreign Projection |
|
||||||
|
|---------------------|----------------------|
|
||||||
|
| Rendered as original syntax | Rendered as comment with semantic hint |
|
||||||
|
|
||||||
|
**Python projection of a C++ template function:**
|
||||||
|
```python
|
||||||
|
# cpp_template: template<typename T> (generic)
|
||||||
|
# NOTE: Python version uses dynamic typing instead
|
||||||
|
def compute(value):
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
**C++ projection of a Python decorated function:**
|
||||||
|
```cpp
|
||||||
|
// python_decorator: @lru_cache(maxsize=128) (memoization)
|
||||||
|
// NOTE: Implement memoization manually or use std::map cache
|
||||||
|
int compute(int value) {
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### TR-5: Tree-sitter Integration
|
||||||
|
|
||||||
|
- Python grammar: `tree-sitter-python`
|
||||||
|
- C++ grammar: `tree-sitter-cpp`
|
||||||
|
- Map tree-sitter CST nodes to Whetstone AST concepts
|
||||||
|
- Preserve source locations for round-trip fidelity
|
||||||
|
|
||||||
|
### TR-6: Generators
|
||||||
|
|
||||||
|
**Python Generator:**
|
||||||
|
- Emit idiomatic Python from AST
|
||||||
|
- Ignore deref strategies (Python is GC'd)
|
||||||
|
- Emit annotations as comments or type hints where applicable
|
||||||
|
|
||||||
|
**C++ Generator:**
|
||||||
|
- Emit idiomatic C++ from AST
|
||||||
|
- Translate deref strategies to appropriate constructs
|
||||||
|
- Inject required includes and memory management code
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Memory Deref Strategy Examples
|
||||||
|
|
||||||
|
The following examples show the **same logical function** with different memory strategies, demonstrating how the C++ projection changes while Python remains constant.
|
||||||
|
|
||||||
|
### Example 1: Processing a List of Records
|
||||||
|
|
||||||
|
#### Whetstone AST (Canonical Form)
|
||||||
|
```
|
||||||
|
Function: processRecords
|
||||||
|
@deref(???) // Strategy applied here
|
||||||
|
Parameter: records -> List<Record>
|
||||||
|
Body:
|
||||||
|
ForLoop:
|
||||||
|
iterator: record in records
|
||||||
|
body:
|
||||||
|
Call: record.validate()
|
||||||
|
Call: record.transform()
|
||||||
|
Return: records
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Python Projection (Same for all strategies)
|
||||||
|
```python
|
||||||
|
def process_records(records: list[Record]) -> list[Record]:
|
||||||
|
"""
|
||||||
|
@deref: {strategy} # Annotation visible but doesn't affect code
|
||||||
|
"""
|
||||||
|
for record in records:
|
||||||
|
record.validate()
|
||||||
|
record.transform()
|
||||||
|
return records
|
||||||
|
```
|
||||||
|
|
||||||
|
#### C++ Projection: @deref(batched) - Garbage Collected
|
||||||
|
```cpp
|
||||||
|
// Memory managed by shared_ptr reference counting
|
||||||
|
#include <memory>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
std::vector<std::shared_ptr<Record>> process_records(
|
||||||
|
std::vector<std::shared_ptr<Record>> records
|
||||||
|
) {
|
||||||
|
for (auto& record : records) {
|
||||||
|
record->validate();
|
||||||
|
record->transform();
|
||||||
|
}
|
||||||
|
return records;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### C++ Projection: @deref(streamed) - Ownership-Based (Rust-like)
|
||||||
|
```cpp
|
||||||
|
// Ownership transferred, no copies, RAII cleanup
|
||||||
|
#include <vector>
|
||||||
|
#include <memory>
|
||||||
|
|
||||||
|
std::vector<std::unique_ptr<Record>> process_records(
|
||||||
|
std::vector<std::unique_ptr<Record>> records // Takes ownership
|
||||||
|
) {
|
||||||
|
for (auto& record : records) {
|
||||||
|
record->validate();
|
||||||
|
record->transform();
|
||||||
|
}
|
||||||
|
return records; // Transfers ownership to caller
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### C++ Projection: @deref(imperative) - Manual Control
|
||||||
|
```cpp
|
||||||
|
// Developer explicitly controls allocation and deallocation
|
||||||
|
// @deref-time: end_of_function
|
||||||
|
// @deref-location: caller_responsibility
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
std::vector<Record*> process_records(
|
||||||
|
std::vector<Record*> records, // Raw pointers - caller owns
|
||||||
|
size_t count
|
||||||
|
) {
|
||||||
|
for (size_t i = 0; i < count; ++i) {
|
||||||
|
records[i]->validate();
|
||||||
|
records[i]->transform();
|
||||||
|
}
|
||||||
|
return records;
|
||||||
|
// NOTE: No deallocation here - @deref-location specifies caller handles it
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### C++ Projection: @deref(content-addressed) - Immutable
|
||||||
|
```cpp
|
||||||
|
// Immutable data, identity by content hash, can be freely shared
|
||||||
|
#include <vector>
|
||||||
|
#include <functional>
|
||||||
|
|
||||||
|
struct ImmutableRecord {
|
||||||
|
const std::string data;
|
||||||
|
const size_t hash;
|
||||||
|
|
||||||
|
ImmutableRecord transform() const {
|
||||||
|
// Returns NEW record, original unchanged
|
||||||
|
return ImmutableRecord{transformed_data, new_hash};
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
std::vector<ImmutableRecord> process_records(
|
||||||
|
const std::vector<ImmutableRecord>& records // Immutable reference
|
||||||
|
) {
|
||||||
|
std::vector<ImmutableRecord> results;
|
||||||
|
results.reserve(records.size());
|
||||||
|
for (const auto& record : records) {
|
||||||
|
record.validate(); // Throws if invalid, doesn't mutate
|
||||||
|
results.push_back(record.transform()); // New record
|
||||||
|
}
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Example 2: Building a Cache
|
||||||
|
|
||||||
|
#### Whetstone AST (Canonical Form)
|
||||||
|
```
|
||||||
|
Function: getOrCreate
|
||||||
|
@deref(???)
|
||||||
|
Parameter: cache -> Map<string, Widget>
|
||||||
|
Parameter: key -> string
|
||||||
|
Body:
|
||||||
|
IfStatement:
|
||||||
|
condition: key in cache
|
||||||
|
then: Return cache[key]
|
||||||
|
else:
|
||||||
|
Assignment: widget = Widget.create(key)
|
||||||
|
Assignment: cache[key] = widget
|
||||||
|
Return: widget
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Python Projection
|
||||||
|
```python
|
||||||
|
def get_or_create(cache: dict[str, Widget], key: str) -> Widget:
|
||||||
|
"""
|
||||||
|
@deref: {strategy}
|
||||||
|
"""
|
||||||
|
if key in cache:
|
||||||
|
return cache[key]
|
||||||
|
else:
|
||||||
|
widget = Widget.create(key)
|
||||||
|
cache[key] = widget
|
||||||
|
return widget
|
||||||
|
```
|
||||||
|
|
||||||
|
#### C++ Projection: @deref(batched)
|
||||||
|
```cpp
|
||||||
|
std::shared_ptr<Widget> get_or_create(
|
||||||
|
std::unordered_map<std::string, std::shared_ptr<Widget>>& cache,
|
||||||
|
const std::string& key
|
||||||
|
) {
|
||||||
|
auto it = cache.find(key);
|
||||||
|
if (it != cache.end()) {
|
||||||
|
return it->second;
|
||||||
|
}
|
||||||
|
auto widget = std::make_shared<Widget>(Widget::create(key));
|
||||||
|
cache[key] = widget;
|
||||||
|
return widget;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### C++ Projection: @deref(streamed)
|
||||||
|
```cpp
|
||||||
|
// Note: Unique ownership makes caching tricky - must use reference
|
||||||
|
Widget& get_or_create(
|
||||||
|
std::unordered_map<std::string, std::unique_ptr<Widget>>& cache,
|
||||||
|
const std::string& key
|
||||||
|
) {
|
||||||
|
auto it = cache.find(key);
|
||||||
|
if (it != cache.end()) {
|
||||||
|
return *it->second;
|
||||||
|
}
|
||||||
|
auto [inserted_it, _] = cache.emplace(
|
||||||
|
key,
|
||||||
|
std::make_unique<Widget>(Widget::create(key))
|
||||||
|
);
|
||||||
|
return *inserted_it->second;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### C++ Projection: @deref(imperative)
|
||||||
|
```cpp
|
||||||
|
// @deref-time: cache_destruction
|
||||||
|
// @deref-location: CacheManager::cleanup()
|
||||||
|
Widget* get_or_create(
|
||||||
|
std::unordered_map<std::string, Widget*>& cache,
|
||||||
|
const std::string& key
|
||||||
|
) {
|
||||||
|
auto it = cache.find(key);
|
||||||
|
if (it != cache.end()) {
|
||||||
|
return it->second;
|
||||||
|
}
|
||||||
|
Widget* widget = new Widget(Widget::create(key)); // Manual allocation
|
||||||
|
cache[key] = widget;
|
||||||
|
return widget;
|
||||||
|
// Caller note: Deallocation handled by CacheManager::cleanup()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Warning System Specification
|
||||||
|
|
||||||
|
### Warning Triggers
|
||||||
|
A warning appears when a node has an `OptimizationLock` annotation and the current user's tier is below the lock level.
|
||||||
|
|
||||||
|
### Warning Display
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ ⚠ OPTIMIZATION WARNING │
|
||||||
|
├─────────────────────────────────────────────────────────────┤
|
||||||
|
│ This node was optimized by: senior_dev_alice │
|
||||||
|
│ Optimization: @deref(imperative) with SIMD vectorization │
|
||||||
|
│ │
|
||||||
|
│ Modifying this code will: │
|
||||||
|
│ • Invalidate the manual memory management strategy │
|
||||||
|
│ • Disable the 4x SIMD optimization │
|
||||||
|
│ │
|
||||||
|
│ [Proceed Anyway] [View C++ Projection] [Cancel] │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Warning Metadata
|
||||||
|
When user proceeds despite warning:
|
||||||
|
1. Original optimization is **shadowed** (preserved but inactive)
|
||||||
|
2. `@provenance` updated with modification chain
|
||||||
|
3. Notification queued for original optimizer
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File Structure for Sprint 1
|
||||||
|
|
||||||
|
```
|
||||||
|
languages/
|
||||||
|
├── SemAnno/
|
||||||
|
│ └── models/
|
||||||
|
│ └── SemAnno.structure.mps # Update with new concepts
|
||||||
|
├── WhetstoneCore/ # NEW: Core AST language
|
||||||
|
│ ├── models/
|
||||||
|
│ │ ├── WhetstoneCore.structure.mps
|
||||||
|
│ │ ├── WhetstoneCore.editor.mps
|
||||||
|
│ │ ├── WhetstoneCore.constraints.mps
|
||||||
|
│ │ └── WhetstoneCore.typesystem.mps
|
||||||
|
│ └── WhetstoneCore.mpl
|
||||||
|
├── PythonProjection/ # NEW: Python view
|
||||||
|
│ ├── models/
|
||||||
|
│ │ ├── PythonProjection.editor.mps
|
||||||
|
│ │ └── PythonProjection.textGen.mps
|
||||||
|
│ └── PythonProjection.mpl
|
||||||
|
├── CppProjection/ # NEW: C++ view
|
||||||
|
│ ├── models/
|
||||||
|
│ │ ├── CppProjection.editor.mps
|
||||||
|
│ │ └── CppProjection.textGen.mps
|
||||||
|
│ └── CppProjection.mpl
|
||||||
|
└── TreeSitterImport/ # NEW: Import from source
|
||||||
|
├── models/
|
||||||
|
│ ├── TreeSitterImport.behavior.mps
|
||||||
|
│ └── TreeSitterImport.plugin.mps
|
||||||
|
└── TreeSitterImport.mpl
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
### AC-1: Round-Trip Parsing
|
||||||
|
- [ ] Parse Python file with tree-sitter → Whetstone AST
|
||||||
|
- [ ] Parse C++ file with tree-sitter → Whetstone AST
|
||||||
|
- [ ] Generate Python from AST that is functionally equivalent
|
||||||
|
- [ ] Generate C++ from AST that compiles and runs
|
||||||
|
|
||||||
|
### AC-2: Deref Strategy Application
|
||||||
|
- [ ] Apply `@deref(batched)` to a function → C++ uses shared_ptr
|
||||||
|
- [ ] Apply `@deref(streamed)` to a function → C++ uses unique_ptr
|
||||||
|
- [ ] Apply `@deref(imperative)` to a function → C++ uses raw pointers
|
||||||
|
- [ ] Python projection shows annotation but code unchanged
|
||||||
|
|
||||||
|
### AC-3: Warning System
|
||||||
|
- [ ] Add optimization lock to node
|
||||||
|
- [ ] Junior user attempts edit → warning appears
|
||||||
|
- [ ] Warning shows what will be invalidated
|
||||||
|
- [ ] User can proceed (warning only, not blocking)
|
||||||
|
- [ ] Provenance updated after edit
|
||||||
|
|
||||||
|
### AC-4: Dual Projection Editing
|
||||||
|
- [ ] Open AST in Python projection → edit logic
|
||||||
|
- [ ] Switch to C++ projection → same logic, different syntax
|
||||||
|
- [ ] Edit in C++ projection → Python projection updates
|
||||||
|
- [ ] Annotations visible in both projections
|
||||||
|
|
||||||
|
### AC-5: Language-Specific Idiom Preservation
|
||||||
|
- [ ] Import Python file with decorators → `@lang_specific` annotations created
|
||||||
|
- [ ] Import C++ file with templates → `@lang_specific` annotations created
|
||||||
|
- [ ] View Python decorator in C++ projection → shows as comment with hint
|
||||||
|
- [ ] View C++ template in Python projection → shows as comment with hint
|
||||||
|
- [ ] Re-export to original language → idiom syntax restored exactly
|
||||||
|
- [ ] Semantic hints populated for known patterns (lru_cache → "memoization")
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
1. **Week 1-2: Core AST Nodes**
|
||||||
|
- Define WhetstoneCore language structure
|
||||||
|
- Implement basic editors for each node type
|
||||||
|
- Manual AST creation works in MPS
|
||||||
|
|
||||||
|
2. **Week 3-4: Python Projection**
|
||||||
|
- Python-syntax editor for AST nodes
|
||||||
|
- Python text generator
|
||||||
|
- Manual round-trip: type Python-like → see generated .py
|
||||||
|
|
||||||
|
3. **Week 5-6: C++ Projection**
|
||||||
|
- C++ syntax editor for AST nodes
|
||||||
|
- C++ text generator with deref strategy translation
|
||||||
|
- Deref strategy annotations affect generated code
|
||||||
|
|
||||||
|
4. **Week 7-8: Tree-sitter Import**
|
||||||
|
- Integrate tree-sitter-python
|
||||||
|
- Integrate tree-sitter-cpp
|
||||||
|
- Parse source files → populate AST
|
||||||
|
|
||||||
|
5. **Week 9-10: Warning System**
|
||||||
|
- OptimizationLock annotation
|
||||||
|
- Warning UI in editor
|
||||||
|
- Provenance tracking
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
1. **Deref inference:** Should the system suggest deref strategies based on usage patterns, or always require explicit annotation?
|
||||||
|
|
||||||
|
2. **Partial optimization:** Can a senior optimize just one function while leaving others with default `@deref(batched)`?
|
||||||
|
|
||||||
|
3. **Conflict granularity:** If a junior modifies a loop inside an optimized function, does that invalidate the whole function or just the loop?
|
||||||
|
|
||||||
|
4. **Tree-sitter fidelity:** How do we handle Python/C++ features that don't map cleanly to each other (e.g., Python decorators, C++ templates)?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
| Dependency | Purpose | Source |
|
||||||
|
|------------|---------|--------|
|
||||||
|
| JetBrains MPS 2023.2+ | Language workbench | jetbrains.com/mps |
|
||||||
|
| tree-sitter | Parsing | github.com/tree-sitter |
|
||||||
|
| tree-sitter-python | Python grammar | github.com/tree-sitter/tree-sitter-python |
|
||||||
|
| tree-sitter-cpp | C++ grammar | github.com/tree-sitter/tree-sitter-cpp |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Metrics
|
||||||
|
|
||||||
|
| Metric | Target |
|
||||||
|
|--------|--------|
|
||||||
|
| Parse success rate (Python) | >95% of valid Python files |
|
||||||
|
| Parse success rate (C++) | >90% of valid C++ files |
|
||||||
|
| Round-trip fidelity | Semantically equivalent output |
|
||||||
|
| Warning accuracy | 100% of locked nodes trigger warnings |
|
||||||
|
| Generator correctness | Generated C++ compiles without errors |
|
||||||
668
docs-20260203T163105Z-3-001/docs/SPRINT_2_VISION.md
Normal file
668
docs-20260203T163105Z-3-001/docs/SPRINT_2_VISION.md
Normal file
@@ -0,0 +1,668 @@
|
|||||||
|
# Sprint 2 Vision: Whetstone Editor & Structured Editing
|
||||||
|
|
||||||
|
## Vision Statement
|
||||||
|
|
||||||
|
Build a next-generation code editor that combines Emacs's powerful libraries with a modern, symbol-rich UI, and introduce **structured editing** where humans and AI agents construct code by choosing from legal primitives rather than typing syntax.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
1. **Hybrid Editor Architecture** - Dear ImGui shell + headless Emacs servers
|
||||||
|
2. **Structured Editing UI** - Choice-based code construction, syntax errors impossible
|
||||||
|
3. **Unified Human/Agent Interface** - Same API for clicking and scripting
|
||||||
|
4. **Elisp ↔ C++ Projection** - Dogfood Whetstone by building the editor with it
|
||||||
|
5. **Proof of Concept** - Demonstrate AST-native development accelerates iteration
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture: Dear ImGui + Headless Emacs
|
||||||
|
|
||||||
|
### Why Hybrid?
|
||||||
|
|
||||||
|
| Approach | Pros | Cons |
|
||||||
|
|----------|------|------|
|
||||||
|
| Fork Emacs | Full compatibility | Massive codebase, GUI baggage |
|
||||||
|
| Pure Dear ImGui | Clean slate, full control | Lose Emacs libraries |
|
||||||
|
| **Hybrid** | Best of both | Complexity in coordination |
|
||||||
|
|
||||||
|
The hybrid approach gives us:
|
||||||
|
- **Emacs libraries** without Emacs's GUI limitations
|
||||||
|
- **Dear ImGui flexibility** without reimplementing text processing
|
||||||
|
- **Parallelism** via multiple Emacs server instances
|
||||||
|
- **Upstream compatibility** - Emacs updates just work
|
||||||
|
|
||||||
|
### System Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Whetstone Editor │
|
||||||
|
├─────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Dear ImGui Shell (C++) │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │
|
||||||
|
│ │ │ File │ │ Editor │ │ Terminal │ │ Structured │ │ │
|
||||||
|
│ │ │ Tree │ │ Tabs │ │ Panel │ │ Edit Panel │ │ │
|
||||||
|
│ │ │ │ │ │ │ │ │ │ │ │
|
||||||
|
│ │ │ 📁 src │ │ [foo.py] │ │ $ _ │ │ [Function ▼] │ │ │
|
||||||
|
│ │ │ 📁 lib │ │ [bar.cpp]│ │ │ │ [Variable ▼] │ │ │
|
||||||
|
│ │ │ 📄 main │ │ │ │ │ │ [Expression ▼] │ │ │
|
||||||
|
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
|
||||||
|
│ │ │ Command Palette (Emacs-style, keyboard-driven) │ │ │
|
||||||
|
│ │ │ > find-file > magit-status > consult-ripgrep │ │ │
|
||||||
|
│ │ └──────────────────────────────────────────────────────────┘ │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ IPC (JSON-RPC / Shared Memory) │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Management Node (C++) │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │
|
||||||
|
│ │ │ Whetstone │ │ Server │ │ Agent │ │ │
|
||||||
|
│ │ │ AST Store │ │ Pool Mgr │ │ Queue │ │ │
|
||||||
|
│ │ │ │ │ │ │ │ │ │
|
||||||
|
│ │ │ [AST Nodes] │ │ spawn() │ │ [Choice sequences from │ │ │
|
||||||
|
│ │ │ [Annotations│ │ route() │ │ human clicks or │ │ │
|
||||||
|
│ │ │ [Projections│ │ sync() │ │ agent submissions] │ │ │
|
||||||
|
│ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ emacsclient protocol │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Headless Emacs Server Pool │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||||||
|
│ │ │ Server 1 │ │ Server 2 │ │ Server N │ │ │
|
||||||
|
│ │ │ │ │ │ │ │ │ │
|
||||||
|
│ │ │ File I/O │ │ Search │ │ Git (Magit) │ │ │
|
||||||
|
│ │ │ TRAMP │ │ Completion │ │ Org-mode │ │ │
|
||||||
|
│ │ │ Dired │ │ Helm/Vertico│ │ LSP bridge │ │ │
|
||||||
|
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ (Each runs: emacs --daemon=serverN --load whetstone-bridge.el) │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Component Responsibilities
|
||||||
|
|
||||||
|
| Component | Language | Responsibility |
|
||||||
|
|-----------|----------|----------------|
|
||||||
|
| **Dear ImGui Shell** | C++ | All rendering, keyboard/mouse input, widgets |
|
||||||
|
| **Management Node** | C++ | AST storage, server coordination, agent queue |
|
||||||
|
| **Emacs Servers** | Elisp | Text processing, file ops, search, completion |
|
||||||
|
|
||||||
|
**Future Goal:** Once Whetstone is mature, rewrite the Management Node in Whetstone AST and generate C++ from it. This is the ultimate dogfooding - the editor's core logic defined in the system it's built to create.
|
||||||
|
|
||||||
|
### Communication Protocol
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────┐ JSON-RPC ┌─────────────┐
|
||||||
|
│ ImGui Shell│ ◄─────────────────────────► │ Mgmt Node │
|
||||||
|
└────────────┘ └─────────────┘
|
||||||
|
│
|
||||||
|
│ emacsclient -e '(elisp-expr)'
|
||||||
|
▼
|
||||||
|
┌─────────────┐
|
||||||
|
│ Emacs Server│
|
||||||
|
└─────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example flow - User searches for file:**
|
||||||
|
|
||||||
|
1. User presses `Ctrl+P` in ImGui shell
|
||||||
|
2. ImGui sends `{ "method": "find-file", "query": "" }` to Management Node
|
||||||
|
3. Management Node routes to Search server: `emacsclient -s search -e '(consult-find "")'`
|
||||||
|
4. Emacs returns candidates
|
||||||
|
5. Management Node sends candidates to ImGui
|
||||||
|
6. ImGui renders searchable dropdown
|
||||||
|
7. User selects file
|
||||||
|
8. Management Node loads file into AST, sends to ImGui for rendering
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Structured Editing: The Core Innovation
|
||||||
|
|
||||||
|
### The Problem with Text-Based Coding
|
||||||
|
|
||||||
|
```
|
||||||
|
Traditional flow:
|
||||||
|
Human/Agent generates text → Parser validates → Errors → Fix → Repeat
|
||||||
|
|
||||||
|
Problems:
|
||||||
|
- Syntax errors waste time
|
||||||
|
- Agents hallucinate invalid syntax
|
||||||
|
- Autocompletion is probabilistic (might suggest wrong things)
|
||||||
|
- Context is implicit (what's in scope? what types are valid?)
|
||||||
|
```
|
||||||
|
|
||||||
|
### The Solution: Choice-Based Construction
|
||||||
|
|
||||||
|
**You never type code. You choose from valid options at each step.**
|
||||||
|
|
||||||
|
```
|
||||||
|
Structured flow:
|
||||||
|
Choose construct → Fill constrained fields → Choose next construct → Done
|
||||||
|
|
||||||
|
Benefits:
|
||||||
|
- Syntax errors IMPOSSIBLE
|
||||||
|
- Every option shown is LEGAL
|
||||||
|
- Context is EXPLICIT (only in-scope items offered)
|
||||||
|
- Agents submit choice sequences (no parsing, no errors)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Structured Editing UI Flow
|
||||||
|
|
||||||
|
#### Step 1: Choose Construct Type
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ What do you want to create? │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||||
|
│ │ Function │ │ Class │ │ Variable │ │ Import │ │
|
||||||
|
│ │ λ │ │ ◇ │ │ x= │ │ ↓ │ │
|
||||||
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||||
|
│ │ If │ │ For │ │ While │ │ Return │ │
|
||||||
|
│ │ ? │ │ ↻ │ │ ↺ │ │ ← │ │
|
||||||
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||||
|
│ │
|
||||||
|
│ [Keyboard: f=Function, c=Class, v=Variable, i=Import...] │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 2: Fill Constrained Fields (Function selected)
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Define Function │
|
||||||
|
│ │
|
||||||
|
│ Name: [process_data________] ← Text input, identifier rules│
|
||||||
|
│ │
|
||||||
|
│ Parameters: │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Name Type Default │ │
|
||||||
|
│ │ [items ] [List[int] ▼] [None ▼] │ │
|
||||||
|
│ │ [threshold] [int ▼] [10 ] │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ [+ Add Parameter] │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ Return Type: [List[int] ▼] [None] [Auto-infer] │
|
||||||
|
│ │
|
||||||
|
│ Annotations: │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ [@deref: batched ▼] [@complexity: ___] [+ Add] │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ [Continue to Body →] │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 3: Build Function Body
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Function: process_data(items: List[int], threshold: int) │
|
||||||
|
│ │
|
||||||
|
│ Body: (Click to add statement) │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ 1. [+ Add Statement] │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ───────────────────────────────────────────────────────────── │
|
||||||
|
│ Choose statement type: │
|
||||||
|
│ │
|
||||||
|
│ [Variable] [If] [For] [While] [Return] [Call] [Assign] │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 4: Add Variable (Variable selected)
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Create Variable │
|
||||||
|
│ │
|
||||||
|
│ Name: [result_________] │
|
||||||
|
│ │
|
||||||
|
│ Initialize with: │
|
||||||
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||||
|
│ │ Literal │ │ Variable │ │ Call │ │ BinaryOp │ │
|
||||||
|
│ │ 42 │ │ x │ │ f() │ │ a+b │ │
|
||||||
|
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 5: Build Expression (Call selected)
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Function Call │
|
||||||
|
│ │
|
||||||
|
│ Function: [filter ▼] ← Dropdown: in-scope functions │
|
||||||
|
│ ┌─────────────────┐ │
|
||||||
|
│ │ filter │ ← Built-in │
|
||||||
|
│ │ map │ ← Built-in │
|
||||||
|
│ │ process_data │ ← Current function (recursion) │
|
||||||
|
│ │ helper_func │ ← Defined above │
|
||||||
|
│ └─────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ Arguments: │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Arg 1 (callable): [lambda ▼] → [x > threshold] │ │
|
||||||
|
│ │ Arg 2 (iterable): [items ▼] ← Only iterables shown │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ [Done - Add to Body] │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Result: Complete Function
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Function: process_data │
|
||||||
|
│ ├── params: [(items, List[int]), (threshold, int, default=10)]│
|
||||||
|
│ ├── returns: List[int] │
|
||||||
|
│ ├── @deref: batched │
|
||||||
|
│ └── body: │
|
||||||
|
│ └── Return │
|
||||||
|
│ └── Call: list │
|
||||||
|
│ └── Call: filter │
|
||||||
|
│ ├── Lambda: x > threshold │
|
||||||
|
│ └── Var: items │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Preview (Python): │ │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ def process_data(items: List[int], threshold: int = 10): │ │
|
||||||
|
│ │ return list(filter(lambda x: x > threshold, items)) │ │
|
||||||
|
│ └─────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ [Edit] [View as C++] [Add Annotation] [Done] │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Agent API: Scripted Choices
|
||||||
|
|
||||||
|
The structured editing UI has a direct JSON equivalent. Agents don't generate code text - they submit choice sequences.
|
||||||
|
|
||||||
|
### Human Interaction (UI)
|
||||||
|
```
|
||||||
|
Click: Function → Type: "process_data" → Click: Add Param → ...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Agent Interaction (JSON)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"construct": "Function",
|
||||||
|
"name": "process_data",
|
||||||
|
"params": [
|
||||||
|
{ "name": "items", "type": "List[int]" },
|
||||||
|
{ "name": "threshold", "type": "int", "default": "10" }
|
||||||
|
],
|
||||||
|
"return_type": "List[int]",
|
||||||
|
"annotations": [
|
||||||
|
{ "type": "deref", "strategy": "batched" }
|
||||||
|
],
|
||||||
|
"body": [
|
||||||
|
{
|
||||||
|
"statement": "Return",
|
||||||
|
"expression": {
|
||||||
|
"type": "Call",
|
||||||
|
"function": "list",
|
||||||
|
"args": [
|
||||||
|
{
|
||||||
|
"type": "Call",
|
||||||
|
"function": "filter",
|
||||||
|
"args": [
|
||||||
|
{
|
||||||
|
"type": "Lambda",
|
||||||
|
"params": ["x"],
|
||||||
|
"body": { "type": "BinaryOp", "left": "x", "op": ">", "right": "threshold" }
|
||||||
|
},
|
||||||
|
{ "type": "VariableRef", "name": "items" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Validation Guarantees
|
||||||
|
|
||||||
|
The Management Node validates each choice:
|
||||||
|
|
||||||
|
```
|
||||||
|
Agent submits: { "function": "nonexistent_func", ... }
|
||||||
|
System responds: {
|
||||||
|
"error": "InvalidChoice",
|
||||||
|
"field": "function",
|
||||||
|
"valid_options": ["filter", "map", "process_data", "helper_func"],
|
||||||
|
"reason": "Function 'nonexistent_func' not in scope"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**No parsing errors. No syntax errors. Only semantic validation.**
|
||||||
|
|
||||||
|
### Agent Workflow: Context-Aware Queuing
|
||||||
|
|
||||||
|
A local agent with project context can pre-compute likely choices:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Agent Context │
|
||||||
|
│ ├── Current file: data_processor.py │
|
||||||
|
│ ├── Cursor: inside class DataProcessor │
|
||||||
|
│ ├── In scope: self, items, config, logger │
|
||||||
|
│ └── Recent pattern: user adding validation methods │
|
||||||
|
│ │
|
||||||
|
│ Agent predicts user wants: validation method │
|
||||||
|
│ Pre-queued choices: │
|
||||||
|
│ 1. Function (method) │
|
||||||
|
│ 2. Name: "validate_items" (inferred from context) │
|
||||||
|
│ 3. Param: items (already in scope) │
|
||||||
|
│ 4. Return: bool (validation pattern) │
|
||||||
|
│ 5. Body: If → condition → Return True/False │
|
||||||
|
│ │
|
||||||
|
│ User sees: "Create validate_items method? [Yes] [Modify] [No]"│
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
Instead of:
|
||||||
|
```
|
||||||
|
Agent: "Here's the code I generated..."
|
||||||
|
Human: "Almost, but change X"
|
||||||
|
Agent: "Here's the updated code..."
|
||||||
|
Human: "Now there's a syntax error"
|
||||||
|
Agent: "Sorry, here's the fix..."
|
||||||
|
```
|
||||||
|
|
||||||
|
It becomes:
|
||||||
|
```
|
||||||
|
Agent: "I suggest these choices: [validate_items, bool, If pattern]"
|
||||||
|
Human: Clicks [Modify] → changes return type to Optional[str]
|
||||||
|
Agent: Updates downstream choices automatically
|
||||||
|
Human: Clicks [Accept]
|
||||||
|
Done. No back-and-forth. No errors.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Elisp ↔ C++ Projection
|
||||||
|
|
||||||
|
### Why These Languages?
|
||||||
|
|
||||||
|
| Language | Role in Editor |
|
||||||
|
|----------|----------------|
|
||||||
|
| **Elisp** | Emacs server scripting, configuration, extensions |
|
||||||
|
| **C++** | Dear ImGui shell, performance-critical paths |
|
||||||
|
|
||||||
|
Building Elisp ↔ C++ transpilation lets us **dogfood** Whetstone:
|
||||||
|
- Write editor logic in Elisp (comfortable, existing packages)
|
||||||
|
- Generate C++ for performance-critical rendering
|
||||||
|
- Same AST, different projections
|
||||||
|
|
||||||
|
### Language Mapping
|
||||||
|
|
||||||
|
| Elisp | Whetstone AST | C++ |
|
||||||
|
|-------|---------------|-----|
|
||||||
|
| `(defun name (args) body)` | Function node | `RetType name(Args) { body }` |
|
||||||
|
| `(let ((x val)) body)` | Variable + Block | `{ auto x = val; body }` |
|
||||||
|
| `(if cond then else)` | IfStatement | `if (cond) { then } else { else }` |
|
||||||
|
| `(lambda (x) body)` | Lambda | `[](auto x) { return body; }` |
|
||||||
|
| `(mapcar fn list)` | Call: map | `std::transform(...)` |
|
||||||
|
| `'(1 2 3)` | ListLiteral | `std::vector{1, 2, 3}` |
|
||||||
|
|
||||||
|
### Elisp-Specific Idioms
|
||||||
|
|
||||||
|
```
|
||||||
|
@lang_specific(elisp, "special_form", "(interactive \"sPrompt: \")", hint="user_input")
|
||||||
|
@lang_specific(elisp, "macro", "(with-temp-buffer ...)", hint="scoped_resource")
|
||||||
|
@lang_specific(elisp, "advice", "(advice-add 'fn :around ...)", hint="aspect_oriented")
|
||||||
|
```
|
||||||
|
|
||||||
|
### C++-Specific Idioms
|
||||||
|
|
||||||
|
```
|
||||||
|
@lang_specific(cpp, "template", "template<typename T>", hint="generic")
|
||||||
|
@lang_specific(cpp, "raii", "std::lock_guard<std::mutex>", hint="scoped_resource")
|
||||||
|
@lang_specific(cpp, "constexpr", "constexpr", hint="compile_time")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dear ImGui Integration
|
||||||
|
|
||||||
|
### What is Dear ImGui?
|
||||||
|
|
||||||
|
Immediate-mode GUI library. Every frame, you declare what to draw:
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
// C++ Dear ImGui
|
||||||
|
void render() {
|
||||||
|
if (ImGui::Button("Compile")) {
|
||||||
|
runCompile();
|
||||||
|
}
|
||||||
|
ImGui::SliderInt("Opt Level", &optLevel, 0, 3);
|
||||||
|
ImGui::Text("Status: %s", status.c_str());
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Elisp Wrapper (whetstone-imgui.el)
|
||||||
|
|
||||||
|
```elisp
|
||||||
|
;; Elisp Dear ImGui bindings
|
||||||
|
(defun render ()
|
||||||
|
(imgui-button "Compile" #'run-compile)
|
||||||
|
(imgui-slider-int "Opt Level" 'opt-level 0 3)
|
||||||
|
(imgui-text "Status: %s" status))
|
||||||
|
```
|
||||||
|
|
||||||
|
### AST-Defined UI
|
||||||
|
|
||||||
|
The Whetstone AST can define UI elements:
|
||||||
|
|
||||||
|
```
|
||||||
|
UIPanel: "Build Settings"
|
||||||
|
├── @projection: imgui
|
||||||
|
├── children:
|
||||||
|
│ ├── UIButton
|
||||||
|
│ │ ├── label: "Compile"
|
||||||
|
│ │ ├── icon: "hammer"
|
||||||
|
│ │ └── action: → BuildSystem.compile()
|
||||||
|
│ ├── UISlider
|
||||||
|
│ │ ├── label: "Optimization Level"
|
||||||
|
│ │ ├── binding: → config.optLevel
|
||||||
|
│ │ ├── min: 0
|
||||||
|
│ │ └── max: 3
|
||||||
|
│ └── UIText
|
||||||
|
│ ├── template: "Status: {}"
|
||||||
|
│ └── binding: → BuildSystem.status
|
||||||
|
```
|
||||||
|
|
||||||
|
**Same AST, multiple UI projections:**
|
||||||
|
|
||||||
|
| Projection | Output |
|
||||||
|
|------------|--------|
|
||||||
|
| `@projection(imgui)` | Native Dear ImGui widgets |
|
||||||
|
| `@projection(emacs)` | Text-based `[Compile] Opt: [===----] 2` |
|
||||||
|
| `@projection(html)` | Web form with `<button>`, `<input type="range">` |
|
||||||
|
| `@projection(tui)` | ncurses terminal UI |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Implementation Phases
|
||||||
|
|
||||||
|
### Phase 2a: Editor Shell (4 weeks)
|
||||||
|
|
||||||
|
| Week | Deliverable |
|
||||||
|
|------|-------------|
|
||||||
|
| 1 | Dear ImGui scaffold with file tree, editor pane, command palette |
|
||||||
|
| 2 | Management Node: spawn/manage headless Emacs servers |
|
||||||
|
| 3 | IPC: route commands ImGui ↔ Management ↔ Emacs |
|
||||||
|
| 4 | Integration: Helm/Vertico search, file operations via TRAMP |
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- [ ] Open files via Emacs file dialog
|
||||||
|
- [ ] Search with Consult/Helm results rendered in ImGui
|
||||||
|
- [ ] Basic text editing (Emacs buffers displayed in ImGui)
|
||||||
|
|
||||||
|
### Phase 2b: Structured Editing (4 weeks)
|
||||||
|
|
||||||
|
| Week | Deliverable |
|
||||||
|
|------|-------------|
|
||||||
|
| 5 | Structured editing panel: construct chooser UI |
|
||||||
|
| 6 | Field editors: constrained inputs for names, types, expressions |
|
||||||
|
| 7 | Scope awareness: dropdown shows only valid in-scope options |
|
||||||
|
| 8 | Agent API: JSON choice submission, validation responses |
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- [ ] Create function via UI without typing syntax
|
||||||
|
- [ ] Agent submits JSON → valid AST created
|
||||||
|
- [ ] Invalid choices rejected with helpful errors
|
||||||
|
|
||||||
|
### Phase 2c: Elisp ↔ C++ (4 weeks)
|
||||||
|
|
||||||
|
| Week | Deliverable |
|
||||||
|
|------|-------------|
|
||||||
|
| 9 | Elisp tree-sitter parser integration |
|
||||||
|
| 10 | Elisp projection (AST → Elisp text) |
|
||||||
|
| 11 | C++ projection improvements for Elisp idiom mapping |
|
||||||
|
| 12 | Round-trip testing: Elisp → AST → C++ → AST → Elisp |
|
||||||
|
|
||||||
|
**Exit Criteria:**
|
||||||
|
- [ ] Parse whetstone-bridge.el into AST
|
||||||
|
- [ ] Generate equivalent C++ for performance-critical functions
|
||||||
|
- [ ] Elisp `@lang_specific` annotations preserved in C++ as comments
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Technical Decisions
|
||||||
|
|
||||||
|
### Management Node: Why C++?
|
||||||
|
|
||||||
|
| Option | Pros | Cons |
|
||||||
|
|--------|------|------|
|
||||||
|
| **C++** | Same as ImGui, no FFI boundary, mature | Memory safety requires discipline |
|
||||||
|
| Rust | Memory safe, good IPC libs | Another language in stack |
|
||||||
|
| Elisp | Single ecosystem | Performance, threading limitations |
|
||||||
|
|
||||||
|
C++ provides:
|
||||||
|
- **Zero FFI overhead** with Dear ImGui (same language)
|
||||||
|
- Mature threading primitives for managing multiple Emacs servers
|
||||||
|
- Excellent JSON libraries (nlohmann/json, rapidjson)
|
||||||
|
- Direct integration with the rendering layer
|
||||||
|
|
||||||
|
**Dogfooding path:** Once Whetstone matures, the Management Node becomes our first major dogfooding target. We'll define its logic in Whetstone AST and generate C++ from it. This proves the system works for real software, not just toy examples.
|
||||||
|
|
||||||
|
### IPC: JSON-RPC over Unix Sockets
|
||||||
|
|
||||||
|
```
|
||||||
|
ImGui Shell ←──JSON-RPC──→ Management Node ←──emacsclient──→ Emacs Servers
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why JSON-RPC:**
|
||||||
|
- Language agnostic
|
||||||
|
- Human readable (debuggable)
|
||||||
|
- Well-supported in C++ and Elisp
|
||||||
|
- LSP uses it (familiar pattern)
|
||||||
|
|
||||||
|
### Emacs Server Specialization
|
||||||
|
|
||||||
|
| Server | Packages Loaded | Purpose |
|
||||||
|
|--------|-----------------|---------|
|
||||||
|
| `file-server` | TRAMP, dired, recentf | File operations |
|
||||||
|
| `search-server` | Consult, ripgrep, fd | Fast search |
|
||||||
|
| `complete-server` | Company, Corfu, Cape | Completion |
|
||||||
|
| `git-server` | Magit, diff-hl | Version control |
|
||||||
|
| `org-server` | Org-mode, org-roam | Documentation |
|
||||||
|
|
||||||
|
**Benefit:** Each server loads only what it needs. Parallel operations don't block each other.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risks & Mitigations
|
||||||
|
|
||||||
|
| Risk | Impact | Mitigation |
|
||||||
|
|------|--------|------------|
|
||||||
|
| Emacs IPC latency | Sluggish UI | Async commands, local caching |
|
||||||
|
| ImGui learning curve | Slow UI development | Start simple, iterate |
|
||||||
|
| Elisp ↔ C++ semantic gap | Lossy transpilation | `@lang_specific` annotations |
|
||||||
|
| Scope complexity | Wrong options offered | Conservative scoping, user override |
|
||||||
|
| Agent choice explosion | Too many options | Context-aware filtering, ML ranking |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Success Metrics
|
||||||
|
|
||||||
|
| Metric | Target |
|
||||||
|
|--------|--------|
|
||||||
|
| File open latency | < 100ms (via Emacs server) |
|
||||||
|
| Search result latency | < 50ms first results |
|
||||||
|
| Structured edit: construct creation | < 10 clicks for simple function |
|
||||||
|
| Agent choice acceptance rate | > 80% first suggestion accepted |
|
||||||
|
| Syntax errors (structured mode) | 0 (by design) |
|
||||||
|
| Elisp → C++ round-trip fidelity | Semantically equivalent |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Questions for Sprint 2
|
||||||
|
|
||||||
|
1. **Emacs package compatibility** - Which GUI-dependent packages (if any) are critical? Can they be adapted?
|
||||||
|
|
||||||
|
2. **Structured editing escape hatch** - Should users be able to drop into raw text mode? When?
|
||||||
|
|
||||||
|
3. **Agent trust levels** - Can agents auto-accept choices, or always require human confirmation?
|
||||||
|
|
||||||
|
4. **Incremental adoption** - Can structured editing work alongside traditional text editing in same file?
|
||||||
|
|
||||||
|
5. **Performance budget** - What's acceptable latency for Emacs server round-trips?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
| Dependency | Version | Purpose |
|
||||||
|
|------------|---------|---------|
|
||||||
|
| Dear ImGui | 1.90+ | GUI rendering |
|
||||||
|
| Emacs | 29+ | Headless servers |
|
||||||
|
| nlohmann/json | 3.11+ | JSON-RPC in C++ |
|
||||||
|
| tree-sitter | 0.22+ | Elisp parsing |
|
||||||
|
| tree-sitter-elisp | latest | Elisp grammar |
|
||||||
|
| C++ compiler | C++20 | Management node, ImGui shell |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Relationship to Sprint 1
|
||||||
|
|
||||||
|
Sprint 1 (MPS) provides:
|
||||||
|
- Core AST node definitions
|
||||||
|
- SemAnno annotation schema
|
||||||
|
- Python ↔ C++ projection patterns
|
||||||
|
- Memory deref strategy examples
|
||||||
|
|
||||||
|
Sprint 2 consumes Sprint 1 by:
|
||||||
|
- Porting AST definitions from MPS to C++ classes (initially)
|
||||||
|
- Reusing annotation schema for structured editing constraints
|
||||||
|
- Extending projections to include Elisp
|
||||||
|
- Building the editor that will eventually replace MPS for Whetstone development
|
||||||
|
|
||||||
|
**The Bootstrap Path:**
|
||||||
|
```
|
||||||
|
Sprint 1: MPS defines AST → generates Python/C++
|
||||||
|
Sprint 2: C++ editor (hand-written) uses AST definitions
|
||||||
|
Sprint 3: Editor logic defined IN Whetstone AST → generates C++
|
||||||
|
Sprint 4: Self-hosting complete - Whetstone builds Whetstone
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user