Remove problematic editor definitions pending structure.mps ID fixes
The editor definitions referenced concepts with duplicate IDs in structure.mps. The structure file needs ID correction before editors can be properly created. To fix: 1. In MPS, open the SemAnno language 2. Right-click on problematic concepts (showing duplicate ID errors) 3. Select 'Correct ID' intention to auto-generate unique IDs 4. Once structure is clean, recreate editor definitions See notes below for details. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
Bill
328
docs/REQUIREMENTS_OVERVIEW.md
Normal file
328
docs/REQUIREMENTS_OVERVIEW.md
Normal file
@@ -0,0 +1,328 @@
|
||||
# Whetstone DSL: Project Requirements Overview
|
||||
|
||||
## Vision Statement
|
||||
|
||||
**Source code should be a structured database of logic, not a text file of syntax.**
|
||||
|
||||
Whetstone is a Universal Logic Database that stores code as a semantic graph (AST), with traditional programming languages serving as **projections** of that canonical representation. This enables lossless transpilation, massively parallel development (human and AI agents), and complete separation of intent from implementation.
|
||||
|
||||
---
|
||||
|
||||
## Core Philosophy
|
||||
|
||||
### The Problem: Text Blindness
|
||||
Current software engineering couples **Intent** (what we want to happen) with **Implementation** (how the machine does it) through language-specific syntax. This creates:
|
||||
- Lock-in to specific languages and ecosystems
|
||||
- Loss of semantic information during translation
|
||||
- Inability to express cross-cutting concerns (ownership, risk, optimization hints)
|
||||
- Barriers to parallel development by multiple agents
|
||||
|
||||
### The Solution: Projectional Editing
|
||||
Move the "Source of Truth" from text files to a **Logical Graph Database**:
|
||||
|
||||
| Text-Based | Graph-Based (Whetstone) |
|
||||
|------------|-------------------------|
|
||||
| Linear | Structural |
|
||||
| Ambiguous | Explicit |
|
||||
| Syntax-heavy | Meaning-heavy |
|
||||
| Language-specific | Universal |
|
||||
|
||||
---
|
||||
|
||||
## Architecture: The Three Layers
|
||||
|
||||
### Layer 1: The Logic Database (Definition)
|
||||
A schema of pure logical concepts stored in a graph/tree database (JetBrains MPS).
|
||||
|
||||
**Core Principle:** This layer must be a **superset** of all target languages. It must express concepts like memory ownership (Rust-style) even when targeting languages that don't require them (Python).
|
||||
|
||||
**Contents:**
|
||||
- AST node concepts (expressions, statements, functions, types)
|
||||
- Semantic annotations (SemAnno schema)
|
||||
- Hypergraph relationships (nodes participate in multiple relationships)
|
||||
|
||||
### Layer 2: Semantic Generators (Bridge)
|
||||
Generators that translate the Logic Database into idiomatic text for specific runtimes.
|
||||
|
||||
**Example - SharedMemoryBlock concept:**
|
||||
```
|
||||
Generator A (Python): x = [1, 2, 3] # Note: Shared usage implied
|
||||
Generator B (Rust): let x = Arc::new(Mutex::new(vec![1, 2, 3]));
|
||||
Generator C (Verilog): reg [31:0] memory [0:2];
|
||||
```
|
||||
|
||||
### Layer 3: Runtime (Metal)
|
||||
Standard toolchains (GCC, LLVM, Python interpreter) compile/execute the generated code. Whetstone does not replace compilers; it feeds them optimized, correct input.
|
||||
|
||||
---
|
||||
|
||||
## SemAnno: Universal Semantic Annotation Schema
|
||||
|
||||
Six annotation systems provide metadata for agents (human and AI):
|
||||
|
||||
### 1. Complexity & Intelligence ("Rank" System)
|
||||
Defines who/what is qualified to modify a node.
|
||||
|
||||
| Annotation | Type | Purpose |
|
||||
|------------|------|---------|
|
||||
| `@complexity-score` | int | Derived from cyclomatic complexity, nesting, mutation |
|
||||
| `@intelligence-requirement` | junior/mid/senior | Agent tier required for modification |
|
||||
| `@review-lock` | bool | Requires second agent verification |
|
||||
|
||||
### 2. Dependency & Impact ("Radius" System)
|
||||
Calculates the "blast radius" of changes.
|
||||
|
||||
| Annotation | Type | Purpose |
|
||||
|------------|------|---------|
|
||||
| `@dependency-count` | int | Incoming references to this node |
|
||||
| `@dependency-level` | 0-100 | Normalized importance score |
|
||||
| `@downstream-impact` | list | Affected domains [Network, Security, Billing] |
|
||||
| `@fragility` | low/high | Load-bearing legacy code flag |
|
||||
|
||||
### 3. Performance & Resource ("Machinist" System)
|
||||
Low-level optimization metadata.
|
||||
|
||||
| Annotation | Type | Purpose |
|
||||
|------------|------|---------|
|
||||
| `@optimization-priority` | low/mid/high | Generator focus for SIMD/unrolling |
|
||||
| `@latency-critical` | bool | Real-time execution requirement |
|
||||
| `@memory-footprint` | estimate | Heap/stack allocation intent |
|
||||
| `@execution-mode` | deterministic/probabilistic | Formal verification hint |
|
||||
| `@deref-explicit` | bool | Every pointer touch must be accounted |
|
||||
|
||||
### 4. Architectural Intent ("Paradigm" System)
|
||||
Defines behavioral contracts.
|
||||
|
||||
| Annotation | Type | Purpose |
|
||||
|------------|------|---------|
|
||||
| `@intent` | Pure/Stateful/Parallel/Legacy | Primary behavioral contract |
|
||||
| `@preferred-view` | language | Optimal projection for readability |
|
||||
| `@lifecycle` | owner/borrower/transient | Ownership model hints |
|
||||
|
||||
### 5. Security & Risk ("Sentinel" System)
|
||||
Granular risk tracking.
|
||||
|
||||
| Annotation | Type | Purpose |
|
||||
|------------|------|---------|
|
||||
| `@risk-type` | identifier | BufferOverflow, RaceCondition, SideChannel |
|
||||
| `@trust-boundary` | marker | Secure environment transition point |
|
||||
| `@provenance` | hash | Chain of agent modifications |
|
||||
|
||||
### 6. Projective Metadata ("Lens" System)
|
||||
Controls rendering in projectional environments.
|
||||
|
||||
| Annotation | Type | Purpose |
|
||||
|------------|------|---------|
|
||||
| `@view-mask` | category | Toggle annotation visibility by type |
|
||||
| `@cognitive-level` | 0-100 | Detail density filter |
|
||||
| `@projection-alias` | string | Localized renaming without AST change |
|
||||
| `@hidden-detail` | bool | Collapse boilerplate for reviews |
|
||||
|
||||
### 7. Language-Specific Idioms ("Polyglot" System)
|
||||
Preserves language-specific features that don't map directly to other languages.
|
||||
|
||||
| Annotation | Type | Purpose |
|
||||
|------------|------|---------|
|
||||
| `@lang_specific` | structured | Captures idiom with language, type, raw syntax, and semantic hint |
|
||||
|
||||
**Architecture:**
|
||||
```
|
||||
@lang_specific(python, "decorator", "@lru_cache(maxsize=128)", hint="memoization")
|
||||
@lang_specific(cpp, "template", "template<typename T>", hint="generic")
|
||||
```
|
||||
|
||||
**Projection Behavior:**
|
||||
- In native language: renders as original syntax
|
||||
- In foreign language: renders as comment with semantic hint
|
||||
|
||||
**Long-term Vision:** These annotations enable future semantic equivalence detection. When the system understands that `@lru_cache` and a C++ `std::map`-based cache serve the same purpose ("memoization"), it can generate idiomatic implementations in either direction rather than just commenting.
|
||||
|
||||
---
|
||||
|
||||
## Memory Dereferencing Strategies
|
||||
|
||||
Memory management is decoupled from logic as a first-class AST field.
|
||||
|
||||
| Strategy | Description | Generator Responsibility |
|
||||
|----------|-------------|-------------------------|
|
||||
| `@deref(imperative)` | Manual control - developer defines time/location | Whetstone Generator / Substrate |
|
||||
| `@deref(streamed)` | Ownership-based - lifetime tracked at declaration (Rust-like) | Whetstone Generator |
|
||||
| `@deref(batched)` | Garbage collected - background process or ref-counting | Whetstone Generator |
|
||||
| `@deref(content-addressed)` | Immutable - identity by hash, location irrelevant | Whetstone Generator |
|
||||
|
||||
### The "Unfilled Node Constraint"
|
||||
In `@deref(imperative)` mode, the AST requires an explicit **Dereference Time Field**. If empty, the editor flags it as a "Missing Intent" error. This makes it impossible to forget a dereference.
|
||||
|
||||
---
|
||||
|
||||
## Strategy Toggles
|
||||
|
||||
Declarative optimization patterns applicable without manual implementation:
|
||||
|
||||
| Toggle | Effect |
|
||||
|--------|--------|
|
||||
| `@strategy(TCO)` | Convert recursive logic to iterative machine code |
|
||||
| `@strategy(Memoize)` | Inject thread-safe cache for function results |
|
||||
| `@strategy(Unroll)` | Duplicate loop bodies per target-hardware heuristic |
|
||||
| `@strategy(SSO)` | Enable Small String/Buffer Optimization |
|
||||
|
||||
---
|
||||
|
||||
## Transpilation Modes
|
||||
|
||||
### Direct Transpilation ("Wrapper")
|
||||
Maps nodes 1-to-1. High compatibility, poor performance.
|
||||
- Python dynamic list → `std::vector<std::any>` with runtime type-checking
|
||||
|
||||
### Idiomatic Transpilation ("Intent")
|
||||
Analyzes usage patterns. High performance, hardware sympathy.
|
||||
- Python list (append-only, sequential read) → `std::vector<T>` or Memory Arena
|
||||
|
||||
---
|
||||
|
||||
## The Resolution Slider
|
||||
|
||||
Four levels from intent to execution:
|
||||
|
||||
| Level | Description | Example |
|
||||
|-------|-------------|---------|
|
||||
| 1 | Pseudocode/Intent | "Find a user in a list" |
|
||||
| 2 | High-Level (JS/Python) | `Array.find()` |
|
||||
| 3 | Intermediate (Rust/Java) | Type safety + ownership scopes |
|
||||
| 4 | Execution Substrate (C++/C) | B-Trees, SIMD, Branch Hints |
|
||||
|
||||
---
|
||||
|
||||
## Multi-Agent Collaboration
|
||||
|
||||
### The Round-Trip Problem
|
||||
When a systems engineer optimizes code that a high-level developer later modifies.
|
||||
|
||||
### Solution: Persistent Metadata Separation
|
||||
- **Logic Field:** Algorithmic intent (editable by junior)
|
||||
- **Optimization Field:** Systems-level instructions (requires senior)
|
||||
|
||||
### Conflict Management
|
||||
When high-level changes break low-level optimizations:
|
||||
1. **Shadowed Metadata:** Old optimizations hidden but not deleted
|
||||
2. **Alerting:** "Performance Guardrail" warning shown
|
||||
3. **Diff Lens:** Visualizes where intent changes invalidated hardware instructions
|
||||
|
||||
---
|
||||
|
||||
## Hypergraph Requirement
|
||||
|
||||
Standard ASTs are strictly hierarchical (Parent → Child). Real-world logic is a **Hypergraph**:
|
||||
|
||||
- A Variable is **owned by** Function A
|
||||
- But **used by** Function B
|
||||
- And **monitored by** Agent C
|
||||
|
||||
**Implementation:** Smart References (Hyperedges) allow a single logical node to participate in multiple relationships simultaneously.
|
||||
|
||||
---
|
||||
|
||||
## Agent-Native Development
|
||||
|
||||
### Task Decomposition Model
|
||||
1. **High-context coordinator agent** builds semantic graph and queues tasks
|
||||
2. **Small Language Models (8B)** handle tightly scoped tasks
|
||||
3. **Human developers** work from the same task queue
|
||||
|
||||
### Projection Optimization
|
||||
Tasks can be projected in:
|
||||
- **Human-optimized view:** Natural language descriptions, readable code
|
||||
- **AI-optimized view:** Structured data, explicit constraints, minimal ambiguity
|
||||
|
||||
### Agent Tiers
|
||||
| Tier | Capabilities |
|
||||
|------|--------------|
|
||||
| Junior | Basic logic, no memory management, no high-risk annotations |
|
||||
| Mid | Standard logic, `@deref(streamed)` and `@deref(batched)` |
|
||||
| Senior | `@deref(imperative)`, hardware optimization, high-risk refactoring |
|
||||
|
||||
---
|
||||
|
||||
## Target Applications
|
||||
|
||||
### Legacy Modernization
|
||||
Parse old Fortran/COBOL → Logic Database → Generate modern C++/Rust/Julia
|
||||
|
||||
### Cross-Platform Logic
|
||||
Write business logic once → Generate Kotlin (Android), Swift (iOS), Rust (backend)
|
||||
|
||||
### Agentic Coding
|
||||
Agents manipulate Graph Nodes directly (no syntax errors from text generation)
|
||||
|
||||
### Parallel Development
|
||||
Multiple developers/agents work on same codebase with explicit conflict boundaries
|
||||
|
||||
---
|
||||
|
||||
## Technical Principles
|
||||
|
||||
### Behavior Over Implementation
|
||||
A node's primary definition is its **Behavioral Contract** (e.g., "Sort this collection") rather than implementation (e.g., "Quicksort with pointer increments").
|
||||
|
||||
### Substrate-Agnostic Core Nodes
|
||||
Core AST nodes must not assume specific memory models or hardware constraints.
|
||||
- **Bad:** LoopNode requires a "Pointer" field
|
||||
- **Good:** LoopNode requires an "IteratorIntent" - realization is metadata
|
||||
|
||||
### The Semantic Boundary
|
||||
Strict boundary between **What** (Logic) and **How** (Execution Strategy). Future capabilities are implemented by adding Strategy Engines to existing nodes, not re-architecting nodes.
|
||||
|
||||
### Strictest Common Denominator
|
||||
The Logic Database enforces strict rules by default (explicit ownership), while Generators can "relax" rules for permissive languages or "poly-fill" rules for strict languages.
|
||||
|
||||
---
|
||||
|
||||
## The C Primitives Substrate (Long-term Vision)
|
||||
|
||||
All high-level language features ultimately resolve to C-level primitives:
|
||||
|
||||
| High-Level Feature | C Primitive Resolution |
|
||||
|-------------------|------------------------|
|
||||
| Python `@lru_cache` | Hash table + function pointer + malloc/free |
|
||||
| C++ `template<T>` | N compiled copies with concrete types |
|
||||
| Python decorator | Function pointer manipulation |
|
||||
| C++ RAII | Constructor/destructor at scope boundaries |
|
||||
| Rust ownership | Compile-time tracking → equivalent to manual C |
|
||||
| Garbage collection | Reference counting or mark-sweep algorithms |
|
||||
|
||||
**Architectural Implication:**
|
||||
|
||||
```
|
||||
Level 4: Language Idioms @py_decl, @cpp_tmpl, @rust_lifetime
|
||||
│
|
||||
▼
|
||||
Level 3: Universal AST Function, Loop, Variable, Expression
|
||||
│
|
||||
▼
|
||||
Level 2: Semantic Intent "memoize", "parameterize by type", "scope-bound cleanup"
|
||||
│
|
||||
▼
|
||||
Level 1: C Primitives pointers, structs, malloc/free, function calls
|
||||
```
|
||||
|
||||
**End State:** Languages become pure *syntactic projections* over C primitives + semantic annotations. This enables true lossless transpilation because the canonical form captures both the intent (semantic annotations) and the implementation (C primitives).
|
||||
|
||||
---
|
||||
|
||||
## Implementation Platform
|
||||
|
||||
**JetBrains MPS (Meta Programming System)**
|
||||
- Projectional editor (edit AST directly, not text)
|
||||
- Language workbench for defining DSLs
|
||||
- Built-in support for generators
|
||||
- Extensible type system and constraints
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **Lossless Round-Trip:** Python → AST → C++ → AST → Python produces semantically equivalent code
|
||||
2. **Parallel Safety:** Junior developer cannot accidentally break senior's optimizations
|
||||
3. **Agent Compatibility:** Tasks can be completed by SLMs with sufficient annotation context
|
||||
4. **Performance Parity:** Generated C++ matches hand-written C++ for common patterns
|
||||
5. **Adoption Path:** Existing codebases can be incrementally imported via tree-sitter parsing
|
||||
544
docs/SPRINT_1_REQUIREMENTS.md
Normal file
544
docs/SPRINT_1_REQUIREMENTS.md
Normal file
@@ -0,0 +1,544 @@
|
||||
# Sprint 1: Python ↔ C++ Dual Projection
|
||||
|
||||
## Sprint Goal
|
||||
|
||||
Enable a junior developer to write working logic in Python, a senior developer to optimize it in C++, and the junior developer to continue modifying safe areas while receiving warnings about optimization-locked regions.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
| In Scope | Out of Scope |
|
||||
|----------|--------------|
|
||||
| Python and C++ projections | Other languages |
|
||||
| Tree-sitter parsing for ingestion | Custom parser implementation |
|
||||
| Memory deref strategy annotations | Full SemAnno schema |
|
||||
| Warning system for locked nodes | Hard locks / approval workflows |
|
||||
| Basic AST nodes (functions, loops, variables, expressions) | Advanced nodes (generics, macros, templates) |
|
||||
| MPS projectional editor | IDE plugins (VS Code, etc.) |
|
||||
|
||||
---
|
||||
|
||||
## User Stories
|
||||
|
||||
### US-1: Python Developer Writes Logic
|
||||
**As a** junior Python developer
|
||||
**I want to** write business logic in a Python-like projection
|
||||
**So that** I can focus on correctness without worrying about memory management
|
||||
|
||||
### US-2: C++ Developer Optimizes
|
||||
**As a** senior C++ developer
|
||||
**I want to** switch to a C++ projection and add optimization annotations
|
||||
**So that** the generated code is production-ready and performant
|
||||
|
||||
### US-3: Python Developer Sees Warnings
|
||||
**As a** junior Python developer
|
||||
**I want to** see warnings when I try to modify optimized nodes
|
||||
**So that** I understand the impact of my changes without being blocked
|
||||
|
||||
### US-4: Import Existing Code
|
||||
**As a** developer
|
||||
**I want to** import existing Python or C++ files into the AST
|
||||
**So that** I can work with legacy code in Whetstone
|
||||
|
||||
---
|
||||
|
||||
## Technical Requirements
|
||||
|
||||
### TR-1: Core AST Nodes
|
||||
|
||||
Define MPS concepts for a minimal but complete AST:
|
||||
|
||||
```
|
||||
Module
|
||||
├── Function
|
||||
│ ├── name: string
|
||||
│ ├── parameters: Parameter[]
|
||||
│ ├── returnType: TypeReference
|
||||
│ ├── body: Statement[]
|
||||
│ └── annotations: SemAnno[]
|
||||
├── Variable
|
||||
│ ├── name: string
|
||||
│ ├── type: TypeReference
|
||||
│ ├── initializer: Expression?
|
||||
│ └── annotations: SemAnno[]
|
||||
└── annotations: SemAnno[]
|
||||
|
||||
Statement (abstract)
|
||||
├── Assignment
|
||||
├── IfStatement
|
||||
├── WhileLoop
|
||||
├── ForLoop
|
||||
├── Return
|
||||
├── ExpressionStatement
|
||||
└── Block
|
||||
|
||||
Expression (abstract)
|
||||
├── BinaryOperation
|
||||
├── UnaryOperation
|
||||
├── FunctionCall
|
||||
├── VariableReference
|
||||
├── Literal (int, float, string, bool)
|
||||
├── ListLiteral
|
||||
├── IndexAccess
|
||||
└── MemberAccess
|
||||
```
|
||||
|
||||
### TR-2: Memory Deref Strategy Annotation
|
||||
|
||||
```
|
||||
DerefStrategy
|
||||
├── strategy: enum {imperative, streamed, batched, content_addressed}
|
||||
├── derefTime: Expression? (required if imperative)
|
||||
├── derefLocation: string? (required if imperative)
|
||||
└── owner: AgentReference?
|
||||
```
|
||||
|
||||
### TR-3: Optimization Lock Annotation
|
||||
|
||||
```
|
||||
OptimizationLock
|
||||
├── lockedBy: AgentReference
|
||||
├── lockReason: string
|
||||
├── lockLevel: enum {warning, soft, hard} // Sprint 1: warning only
|
||||
├── affectedStrategies: string[]
|
||||
└── timestamp: datetime
|
||||
```
|
||||
|
||||
### TR-4: Language-Specific Idiom Annotation
|
||||
|
||||
Placeholder annotations that preserve language-specific features during import without requiring full semantic understanding.
|
||||
|
||||
```
|
||||
LangSpecific
|
||||
├── language: enum {python, cpp, rust, ...}
|
||||
├── idiomType: string // "decorator", "template", "attribute", "pragma", etc.
|
||||
├── rawSyntax: string // The original syntax as written
|
||||
├── semanticHint: string? // Optional: "memoization", "generic", "compile-hint"
|
||||
└── position: enum {before, after, wrapping} // Where it attaches to the node
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
|
||||
```
|
||||
// Python decorator captured during import
|
||||
@lang_specific(python, "decorator", "@lru_cache(maxsize=128)", hint="memoization", position=before)
|
||||
|
||||
// C++ template captured during import
|
||||
@lang_specific(cpp, "template", "template<typename T>", hint="generic", position=before)
|
||||
|
||||
// C++ pragma
|
||||
@lang_specific(cpp, "pragma", "#pragma omp parallel for", hint="parallelization", position=before)
|
||||
|
||||
// Python type hint that has no C++ equivalent
|
||||
@lang_specific(python, "type_hint", "-> Generator[int, None, None]", hint="generator", position=after)
|
||||
|
||||
// C++ attribute
|
||||
@lang_specific(cpp, "attribute", "[[nodiscard]]", hint="return_value_check", position=before)
|
||||
```
|
||||
|
||||
**Projection Behavior:**
|
||||
|
||||
| In Native Projection | In Foreign Projection |
|
||||
|---------------------|----------------------|
|
||||
| Rendered as original syntax | Rendered as comment with semantic hint |
|
||||
|
||||
**Python projection of a C++ template function:**
|
||||
```python
|
||||
# cpp_template: template<typename T> (generic)
|
||||
# NOTE: Python version uses dynamic typing instead
|
||||
def compute(value):
|
||||
...
|
||||
```
|
||||
|
||||
**C++ projection of a Python decorated function:**
|
||||
```cpp
|
||||
// python_decorator: @lru_cache(maxsize=128) (memoization)
|
||||
// NOTE: Implement memoization manually or use std::map cache
|
||||
int compute(int value) {
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
### TR-5: Tree-sitter Integration
|
||||
|
||||
- Python grammar: `tree-sitter-python`
|
||||
- C++ grammar: `tree-sitter-cpp`
|
||||
- Map tree-sitter CST nodes to Whetstone AST concepts
|
||||
- Preserve source locations for round-trip fidelity
|
||||
|
||||
### TR-6: Generators
|
||||
|
||||
**Python Generator:**
|
||||
- Emit idiomatic Python from AST
|
||||
- Ignore deref strategies (Python is GC'd)
|
||||
- Emit annotations as comments or type hints where applicable
|
||||
|
||||
**C++ Generator:**
|
||||
- Emit idiomatic C++ from AST
|
||||
- Translate deref strategies to appropriate constructs
|
||||
- Inject required includes and memory management code
|
||||
|
||||
---
|
||||
|
||||
## Memory Deref Strategy Examples
|
||||
|
||||
The following examples show the **same logical function** with different memory strategies, demonstrating how the C++ projection changes while Python remains constant.
|
||||
|
||||
### Example 1: Processing a List of Records
|
||||
|
||||
#### Whetstone AST (Canonical Form)
|
||||
```
|
||||
Function: processRecords
|
||||
@deref(???) // Strategy applied here
|
||||
Parameter: records -> List<Record>
|
||||
Body:
|
||||
ForLoop:
|
||||
iterator: record in records
|
||||
body:
|
||||
Call: record.validate()
|
||||
Call: record.transform()
|
||||
Return: records
|
||||
```
|
||||
|
||||
#### Python Projection (Same for all strategies)
|
||||
```python
|
||||
def process_records(records: list[Record]) -> list[Record]:
|
||||
"""
|
||||
@deref: {strategy} # Annotation visible but doesn't affect code
|
||||
"""
|
||||
for record in records:
|
||||
record.validate()
|
||||
record.transform()
|
||||
return records
|
||||
```
|
||||
|
||||
#### C++ Projection: @deref(batched) - Garbage Collected
|
||||
```cpp
|
||||
// Memory managed by shared_ptr reference counting
|
||||
#include <memory>
|
||||
#include <vector>
|
||||
|
||||
std::vector<std::shared_ptr<Record>> process_records(
|
||||
std::vector<std::shared_ptr<Record>> records
|
||||
) {
|
||||
for (auto& record : records) {
|
||||
record->validate();
|
||||
record->transform();
|
||||
}
|
||||
return records;
|
||||
}
|
||||
```
|
||||
|
||||
#### C++ Projection: @deref(streamed) - Ownership-Based (Rust-like)
|
||||
```cpp
|
||||
// Ownership transferred, no copies, RAII cleanup
|
||||
#include <vector>
|
||||
#include <memory>
|
||||
|
||||
std::vector<std::unique_ptr<Record>> process_records(
|
||||
std::vector<std::unique_ptr<Record>> records // Takes ownership
|
||||
) {
|
||||
for (auto& record : records) {
|
||||
record->validate();
|
||||
record->transform();
|
||||
}
|
||||
return records; // Transfers ownership to caller
|
||||
}
|
||||
```
|
||||
|
||||
#### C++ Projection: @deref(imperative) - Manual Control
|
||||
```cpp
|
||||
// Developer explicitly controls allocation and deallocation
|
||||
// @deref-time: end_of_function
|
||||
// @deref-location: caller_responsibility
|
||||
#include <vector>
|
||||
|
||||
std::vector<Record*> process_records(
|
||||
std::vector<Record*> records, // Raw pointers - caller owns
|
||||
size_t count
|
||||
) {
|
||||
for (size_t i = 0; i < count; ++i) {
|
||||
records[i]->validate();
|
||||
records[i]->transform();
|
||||
}
|
||||
return records;
|
||||
// NOTE: No deallocation here - @deref-location specifies caller handles it
|
||||
}
|
||||
```
|
||||
|
||||
#### C++ Projection: @deref(content-addressed) - Immutable
|
||||
```cpp
|
||||
// Immutable data, identity by content hash, can be freely shared
|
||||
#include <vector>
|
||||
#include <functional>
|
||||
|
||||
struct ImmutableRecord {
|
||||
const std::string data;
|
||||
const size_t hash;
|
||||
|
||||
ImmutableRecord transform() const {
|
||||
// Returns NEW record, original unchanged
|
||||
return ImmutableRecord{transformed_data, new_hash};
|
||||
}
|
||||
};
|
||||
|
||||
std::vector<ImmutableRecord> process_records(
|
||||
const std::vector<ImmutableRecord>& records // Immutable reference
|
||||
) {
|
||||
std::vector<ImmutableRecord> results;
|
||||
results.reserve(records.size());
|
||||
for (const auto& record : records) {
|
||||
record.validate(); // Throws if invalid, doesn't mutate
|
||||
results.push_back(record.transform()); // New record
|
||||
}
|
||||
return results;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Example 2: Building a Cache
|
||||
|
||||
#### Whetstone AST (Canonical Form)
|
||||
```
|
||||
Function: getOrCreate
|
||||
@deref(???)
|
||||
Parameter: cache -> Map<string, Widget>
|
||||
Parameter: key -> string
|
||||
Body:
|
||||
IfStatement:
|
||||
condition: key in cache
|
||||
then: Return cache[key]
|
||||
else:
|
||||
Assignment: widget = Widget.create(key)
|
||||
Assignment: cache[key] = widget
|
||||
Return: widget
|
||||
```
|
||||
|
||||
#### Python Projection
|
||||
```python
|
||||
def get_or_create(cache: dict[str, Widget], key: str) -> Widget:
|
||||
"""
|
||||
@deref: {strategy}
|
||||
"""
|
||||
if key in cache:
|
||||
return cache[key]
|
||||
else:
|
||||
widget = Widget.create(key)
|
||||
cache[key] = widget
|
||||
return widget
|
||||
```
|
||||
|
||||
#### C++ Projection: @deref(batched)
|
||||
```cpp
|
||||
std::shared_ptr<Widget> get_or_create(
|
||||
std::unordered_map<std::string, std::shared_ptr<Widget>>& cache,
|
||||
const std::string& key
|
||||
) {
|
||||
auto it = cache.find(key);
|
||||
if (it != cache.end()) {
|
||||
return it->second;
|
||||
}
|
||||
auto widget = std::make_shared<Widget>(Widget::create(key));
|
||||
cache[key] = widget;
|
||||
return widget;
|
||||
}
|
||||
```
|
||||
|
||||
#### C++ Projection: @deref(streamed)
|
||||
```cpp
|
||||
// Note: Unique ownership makes caching tricky - must use reference
|
||||
Widget& get_or_create(
|
||||
std::unordered_map<std::string, std::unique_ptr<Widget>>& cache,
|
||||
const std::string& key
|
||||
) {
|
||||
auto it = cache.find(key);
|
||||
if (it != cache.end()) {
|
||||
return *it->second;
|
||||
}
|
||||
auto [inserted_it, _] = cache.emplace(
|
||||
key,
|
||||
std::make_unique<Widget>(Widget::create(key))
|
||||
);
|
||||
return *inserted_it->second;
|
||||
}
|
||||
```
|
||||
|
||||
#### C++ Projection: @deref(imperative)
|
||||
```cpp
|
||||
// @deref-time: cache_destruction
|
||||
// @deref-location: CacheManager::cleanup()
|
||||
Widget* get_or_create(
|
||||
std::unordered_map<std::string, Widget*>& cache,
|
||||
const std::string& key
|
||||
) {
|
||||
auto it = cache.find(key);
|
||||
if (it != cache.end()) {
|
||||
return it->second;
|
||||
}
|
||||
Widget* widget = new Widget(Widget::create(key)); // Manual allocation
|
||||
cache[key] = widget;
|
||||
return widget;
|
||||
// Caller note: Deallocation handled by CacheManager::cleanup()
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Warning System Specification
|
||||
|
||||
### Warning Triggers
|
||||
A warning appears when a node has an `OptimizationLock` annotation and the current user's tier is below the lock level.
|
||||
|
||||
### Warning Display
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ⚠ OPTIMIZATION WARNING │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ This node was optimized by: senior_dev_alice │
|
||||
│ Optimization: @deref(imperative) with SIMD vectorization │
|
||||
│ │
|
||||
│ Modifying this code will: │
|
||||
│ • Invalidate the manual memory management strategy │
|
||||
│ • Disable the 4x SIMD optimization │
|
||||
│ │
|
||||
│ [Proceed Anyway] [View C++ Projection] [Cancel] │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Warning Metadata
|
||||
When user proceeds despite warning:
|
||||
1. Original optimization is **shadowed** (preserved but inactive)
|
||||
2. `@provenance` updated with modification chain
|
||||
3. Notification queued for original optimizer
|
||||
|
||||
---
|
||||
|
||||
## File Structure for Sprint 1
|
||||
|
||||
The SemAnno language is built incrementally across the sprint:
|
||||
|
||||
```
|
||||
languages/
|
||||
└── SemAnno/
|
||||
├── models/
|
||||
│ ├── SemAnno.structure.mps # Core AST + annotation concepts
|
||||
│ ├── SemAnno.editor.mps # Python & C++ projections
|
||||
│ ├── SemAnno.behavior.mps # Concept methods & helpers
|
||||
│ ├── SemAnno.typesystem.mps # Type rules & inference
|
||||
│ ├── SemAnno.textGen.mps # Python & C++ code generators
|
||||
│ └── SemAnno.constraints.mps # Validation constraints
|
||||
├── tests/
|
||||
│ └── SemAnno.tests.mps # Language feature tests
|
||||
└── SemAnno.mpl
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### AC-1: Round-Trip Parsing
|
||||
- [ ] Parse Python file with tree-sitter → Whetstone AST
|
||||
- [ ] Parse C++ file with tree-sitter → Whetstone AST
|
||||
- [ ] Generate Python from AST that is functionally equivalent
|
||||
- [ ] Generate C++ from AST that compiles and runs
|
||||
|
||||
### AC-2: Deref Strategy Application
|
||||
- [ ] Apply `@deref(batched)` to a function → C++ uses shared_ptr
|
||||
- [ ] Apply `@deref(streamed)` to a function → C++ uses unique_ptr
|
||||
- [ ] Apply `@deref(imperative)` to a function → C++ uses raw pointers
|
||||
- [ ] Python projection shows annotation but code unchanged
|
||||
|
||||
### AC-3: Warning System
|
||||
- [ ] Add optimization lock to node
|
||||
- [ ] Junior user attempts edit → warning appears
|
||||
- [ ] Warning shows what will be invalidated
|
||||
- [ ] User can proceed (warning only, not blocking)
|
||||
- [ ] Provenance updated after edit
|
||||
|
||||
### AC-4: Dual Projection Editing
|
||||
- [ ] Open AST in Python projection → edit logic
|
||||
- [ ] Switch to C++ projection → same logic, different syntax
|
||||
- [ ] Edit in C++ projection → Python projection updates
|
||||
- [ ] Annotations visible in both projections
|
||||
|
||||
### AC-5: Language-Specific Idiom Preservation
|
||||
- [ ] Import Python file with decorators → `@lang_specific` annotations created
|
||||
- [ ] Import C++ file with templates → `@lang_specific` annotations created
|
||||
- [ ] View Python decorator in C++ projection → shows as comment with hint
|
||||
- [ ] View C++ template in Python projection → shows as comment with hint
|
||||
- [ ] Re-export to original language → idiom syntax restored exactly
|
||||
- [ ] Semantic hints populated for known patterns (lru_cache → "memoization")
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
Build SemAnno incrementally, testing each phase before moving forward:
|
||||
|
||||
1. **Week 1-2: Core AST Structure**
|
||||
- Add core concepts to SemAnno.structure.mps (Module, Function, Variable, Statement, Expression, Type nodes)
|
||||
- Create basic editors in SemAnno.editor.mps for each node type
|
||||
- Manual AST creation works in MPS
|
||||
- Test: Create a simple function with statements and expressions
|
||||
|
||||
2. **Week 3-4: Python Projection & Generator**
|
||||
- Extend SemAnno.editor.mps with Python-syntax projections
|
||||
- Implement Python generator in SemAnno.textGen.mps
|
||||
- Manual round-trip: type Python-like code → see generated .py file
|
||||
- Test: Generate valid, runnable Python code
|
||||
|
||||
3. **Week 5-6: C++ Projection & Generator**
|
||||
- Extend SemAnno.editor.mps with C++ syntax projections
|
||||
- Implement C++ generator in SemAnno.textGen.mps
|
||||
- Add deref strategy translation to generator
|
||||
- Test: Generate valid, compilable C++ code for each deref strategy
|
||||
|
||||
4. **Week 7-8: Tree-sitter Import**
|
||||
- Add import behavior to SemAnno.behavior.mps
|
||||
- Integrate tree-sitter-python for Python parsing
|
||||
- Integrate tree-sitter-cpp for C++ parsing
|
||||
- Test: Parse Python/C++ files → populate SemAnno AST
|
||||
|
||||
5. **Week 9-10: Warning System & Annotations**
|
||||
- Add OptimizationLock and DerefStrategy annotations to SemAnno.structure.mps
|
||||
- Implement warning logic in SemAnno.behavior.mps
|
||||
- Add warning UI to SemAnno.editor.mps
|
||||
- Implement provenance tracking
|
||||
- Test: Locked nodes show warnings, allow edits with provenance updates
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Deref inference:** Should the system suggest deref strategies based on usage patterns, or always require explicit annotation?
|
||||
|
||||
2. **Partial optimization:** Can a senior optimize just one function while leaving others with default `@deref(batched)`?
|
||||
|
||||
3. **Conflict granularity:** If a junior modifies a loop inside an optimized function, does that invalidate the whole function or just the loop?
|
||||
|
||||
4. **Tree-sitter fidelity:** How do we handle Python/C++ features that don't map cleanly to each other (e.g., Python decorators, C++ templates)?
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Purpose | Source |
|
||||
|------------|---------|--------|
|
||||
| JetBrains MPS 2023.2+ | Language workbench | jetbrains.com/mps |
|
||||
| tree-sitter | Parsing | github.com/tree-sitter |
|
||||
| tree-sitter-python | Python grammar | github.com/tree-sitter/tree-sitter-python |
|
||||
| tree-sitter-cpp | C++ grammar | github.com/tree-sitter/tree-sitter-cpp |
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| Parse success rate (Python) | >95% of valid Python files |
|
||||
| Parse success rate (C++) | >90% of valid C++ files |
|
||||
| Round-trip fidelity | Semantically equivalent output |
|
||||
| Warning accuracy | 100% of locked nodes trigger warnings |
|
||||
| Generator correctness | Generated C++ compiles without errors |
|
||||
668
docs/SPRINT_2_VISION.md
Normal file
668
docs/SPRINT_2_VISION.md
Normal file
@@ -0,0 +1,668 @@
|
||||
# Sprint 2 Vision: Whetstone Editor & Structured Editing
|
||||
|
||||
## Vision Statement
|
||||
|
||||
Build a next-generation code editor that combines Emacs's powerful libraries with a modern, symbol-rich UI, and introduce **structured editing** where humans and AI agents construct code by choosing from legal primitives rather than typing syntax.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Hybrid Editor Architecture** - Dear ImGui shell + headless Emacs servers
|
||||
2. **Structured Editing UI** - Choice-based code construction, syntax errors impossible
|
||||
3. **Unified Human/Agent Interface** - Same API for clicking and scripting
|
||||
4. **Elisp ↔ C++ Projection** - Dogfood Whetstone by building the editor with it
|
||||
5. **Proof of Concept** - Demonstrate AST-native development accelerates iteration
|
||||
|
||||
---
|
||||
|
||||
## Architecture: Dear ImGui + Headless Emacs
|
||||
|
||||
### Why Hybrid?
|
||||
|
||||
| Approach | Pros | Cons |
|
||||
|----------|------|------|
|
||||
| Fork Emacs | Full compatibility | Massive codebase, GUI baggage |
|
||||
| Pure Dear ImGui | Clean slate, full control | Lose Emacs libraries |
|
||||
| **Hybrid** | Best of both | Complexity in coordination |
|
||||
|
||||
The hybrid approach gives us:
|
||||
- **Emacs libraries** without Emacs's GUI limitations
|
||||
- **Dear ImGui flexibility** without reimplementing text processing
|
||||
- **Parallelism** via multiple Emacs server instances
|
||||
- **Upstream compatibility** - Emacs updates just work
|
||||
|
||||
### System Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ Whetstone Editor │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Dear ImGui Shell (C++) │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │
|
||||
│ │ │ File │ │ Editor │ │ Terminal │ │ Structured │ │ │
|
||||
│ │ │ Tree │ │ Tabs │ │ Panel │ │ Edit Panel │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ 📁 src │ │ [foo.py] │ │ $ _ │ │ [Function ▼] │ │ │
|
||||
│ │ │ 📁 lib │ │ [bar.cpp]│ │ │ │ [Variable ▼] │ │ │
|
||||
│ │ │ 📄 main │ │ │ │ │ │ [Expression ▼] │ │ │
|
||||
│ │ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ Command Palette (Emacs-style, keyboard-driven) │ │ │
|
||||
│ │ │ > find-file > magit-status > consult-ripgrep │ │ │
|
||||
│ │ └──────────────────────────────────────────────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ IPC (JSON-RPC / Shared Memory) │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Management Node (C++) │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │
|
||||
│ │ │ Whetstone │ │ Server │ │ Agent │ │ │
|
||||
│ │ │ AST Store │ │ Pool Mgr │ │ Queue │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ [AST Nodes] │ │ spawn() │ │ [Choice sequences from │ │ │
|
||||
│ │ │ [Annotations│ │ route() │ │ human clicks or │ │ │
|
||||
│ │ │ [Projections│ │ sync() │ │ agent submissions] │ │ │
|
||||
│ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ │ emacsclient protocol │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Headless Emacs Server Pool │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||||
│ │ │ Server 1 │ │ Server 2 │ │ Server N │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ File I/O │ │ Search │ │ Git (Magit) │ │ │
|
||||
│ │ │ TRAMP │ │ Completion │ │ Org-mode │ │ │
|
||||
│ │ │ Dired │ │ Helm/Vertico│ │ LSP bridge │ │ │
|
||||
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ (Each runs: emacs --daemon=serverN --load whetstone-bridge.el) │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Component Responsibilities
|
||||
|
||||
| Component | Language | Responsibility |
|
||||
|-----------|----------|----------------|
|
||||
| **Dear ImGui Shell** | C++ | All rendering, keyboard/mouse input, widgets |
|
||||
| **Management Node** | C++ | AST storage, server coordination, agent queue |
|
||||
| **Emacs Servers** | Elisp | Text processing, file ops, search, completion |
|
||||
|
||||
**Future Goal:** Once Whetstone is mature, rewrite the Management Node in Whetstone AST and generate C++ from it. This is the ultimate dogfooding - the editor's core logic defined in the system it's built to create.
|
||||
|
||||
### Communication Protocol
|
||||
|
||||
```
|
||||
┌────────────┐ JSON-RPC ┌─────────────┐
|
||||
│ ImGui Shell│ ◄─────────────────────────► │ Mgmt Node │
|
||||
└────────────┘ └─────────────┘
|
||||
│
|
||||
│ emacsclient -e '(elisp-expr)'
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ Emacs Server│
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
**Example flow - User searches for file:**
|
||||
|
||||
1. User presses `Ctrl+P` in ImGui shell
|
||||
2. ImGui sends `{ "method": "find-file", "query": "" }` to Management Node
|
||||
3. Management Node routes to Search server: `emacsclient -s search -e '(consult-find "")'`
|
||||
4. Emacs returns candidates
|
||||
5. Management Node sends candidates to ImGui
|
||||
6. ImGui renders searchable dropdown
|
||||
7. User selects file
|
||||
8. Management Node loads file into AST, sends to ImGui for rendering
|
||||
|
||||
---
|
||||
|
||||
## Structured Editing: The Core Innovation
|
||||
|
||||
### The Problem with Text-Based Coding
|
||||
|
||||
```
|
||||
Traditional flow:
|
||||
Human/Agent generates text → Parser validates → Errors → Fix → Repeat
|
||||
|
||||
Problems:
|
||||
- Syntax errors waste time
|
||||
- Agents hallucinate invalid syntax
|
||||
- Autocompletion is probabilistic (might suggest wrong things)
|
||||
- Context is implicit (what's in scope? what types are valid?)
|
||||
```
|
||||
|
||||
### The Solution: Choice-Based Construction
|
||||
|
||||
**You never type code. You choose from valid options at each step.**
|
||||
|
||||
```
|
||||
Structured flow:
|
||||
Choose construct → Fill constrained fields → Choose next construct → Done
|
||||
|
||||
Benefits:
|
||||
- Syntax errors IMPOSSIBLE
|
||||
- Every option shown is LEGAL
|
||||
- Context is EXPLICIT (only in-scope items offered)
|
||||
- Agents submit choice sequences (no parsing, no errors)
|
||||
```
|
||||
|
||||
### Structured Editing UI Flow
|
||||
|
||||
#### Step 1: Choose Construct Type
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ What do you want to create? │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Function │ │ Class │ │ Variable │ │ Import │ │
|
||||
│ │ λ │ │ ◇ │ │ x= │ │ ↓ │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ If │ │ For │ │ While │ │ Return │ │
|
||||
│ │ ? │ │ ↻ │ │ ↺ │ │ ← │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
│ [Keyboard: f=Function, c=Class, v=Variable, i=Import...] │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Step 2: Fill Constrained Fields (Function selected)
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Define Function │
|
||||
│ │
|
||||
│ Name: [process_data________] ← Text input, identifier rules│
|
||||
│ │
|
||||
│ Parameters: │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Name Type Default │ │
|
||||
│ │ [items ] [List[int] ▼] [None ▼] │ │
|
||||
│ │ [threshold] [int ▼] [10 ] │ │
|
||||
│ │ │ │
|
||||
│ │ [+ Add Parameter] │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Return Type: [List[int] ▼] [None] [Auto-infer] │
|
||||
│ │
|
||||
│ Annotations: │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ [@deref: batched ▼] [@complexity: ___] [+ Add] │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ [Continue to Body →] │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Step 3: Build Function Body
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Function: process_data(items: List[int], threshold: int) │
|
||||
│ │
|
||||
│ Body: (Click to add statement) │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ 1. [+ Add Statement] │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ───────────────────────────────────────────────────────────── │
|
||||
│ Choose statement type: │
|
||||
│ │
|
||||
│ [Variable] [If] [For] [While] [Return] [Call] [Assign] │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Step 4: Add Variable (Variable selected)
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Create Variable │
|
||||
│ │
|
||||
│ Name: [result_________] │
|
||||
│ │
|
||||
│ Initialize with: │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Literal │ │ Variable │ │ Call │ │ BinaryOp │ │
|
||||
│ │ 42 │ │ x │ │ f() │ │ a+b │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Step 5: Build Expression (Call selected)
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Function Call │
|
||||
│ │
|
||||
│ Function: [filter ▼] ← Dropdown: in-scope functions │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ filter │ ← Built-in │
|
||||
│ │ map │ ← Built-in │
|
||||
│ │ process_data │ ← Current function (recursion) │
|
||||
│ │ helper_func │ ← Defined above │
|
||||
│ └─────────────────┘ │
|
||||
│ │
|
||||
│ Arguments: │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Arg 1 (callable): [lambda ▼] → [x > threshold] │ │
|
||||
│ │ Arg 2 (iterable): [items ▼] ← Only iterables shown │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ [Done - Add to Body] │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
#### Result: Complete Function
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Function: process_data │
|
||||
│ ├── params: [(items, List[int]), (threshold, int, default=10)]│
|
||||
│ ├── returns: List[int] │
|
||||
│ ├── @deref: batched │
|
||||
│ └── body: │
|
||||
│ └── Return │
|
||||
│ └── Call: list │
|
||||
│ └── Call: filter │
|
||||
│ ├── Lambda: x > threshold │
|
||||
│ └── Var: items │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Preview (Python): │ │
|
||||
│ │ │ │
|
||||
│ │ def process_data(items: List[int], threshold: int = 10): │ │
|
||||
│ │ return list(filter(lambda x: x > threshold, items)) │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ [Edit] [View as C++] [Add Annotation] [Done] │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Agent API: Scripted Choices
|
||||
|
||||
The structured editing UI has a direct JSON equivalent. Agents don't generate code text - they submit choice sequences.
|
||||
|
||||
### Human Interaction (UI)
|
||||
```
|
||||
Click: Function → Type: "process_data" → Click: Add Param → ...
|
||||
```
|
||||
|
||||
### Agent Interaction (JSON)
|
||||
```json
|
||||
{
|
||||
"construct": "Function",
|
||||
"name": "process_data",
|
||||
"params": [
|
||||
{ "name": "items", "type": "List[int]" },
|
||||
{ "name": "threshold", "type": "int", "default": "10" }
|
||||
],
|
||||
"return_type": "List[int]",
|
||||
"annotations": [
|
||||
{ "type": "deref", "strategy": "batched" }
|
||||
],
|
||||
"body": [
|
||||
{
|
||||
"statement": "Return",
|
||||
"expression": {
|
||||
"type": "Call",
|
||||
"function": "list",
|
||||
"args": [
|
||||
{
|
||||
"type": "Call",
|
||||
"function": "filter",
|
||||
"args": [
|
||||
{
|
||||
"type": "Lambda",
|
||||
"params": ["x"],
|
||||
"body": { "type": "BinaryOp", "left": "x", "op": ">", "right": "threshold" }
|
||||
},
|
||||
{ "type": "VariableRef", "name": "items" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Validation Guarantees
|
||||
|
||||
The Management Node validates each choice:
|
||||
|
||||
```
|
||||
Agent submits: { "function": "nonexistent_func", ... }
|
||||
System responds: {
|
||||
"error": "InvalidChoice",
|
||||
"field": "function",
|
||||
"valid_options": ["filter", "map", "process_data", "helper_func"],
|
||||
"reason": "Function 'nonexistent_func' not in scope"
|
||||
}
|
||||
```
|
||||
|
||||
**No parsing errors. No syntax errors. Only semantic validation.**
|
||||
|
||||
### Agent Workflow: Context-Aware Queuing
|
||||
|
||||
A local agent with project context can pre-compute likely choices:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Agent Context │
|
||||
│ ├── Current file: data_processor.py │
|
||||
│ ├── Cursor: inside class DataProcessor │
|
||||
│ ├── In scope: self, items, config, logger │
|
||||
│ └── Recent pattern: user adding validation methods │
|
||||
│ │
|
||||
│ Agent predicts user wants: validation method │
|
||||
│ Pre-queued choices: │
|
||||
│ 1. Function (method) │
|
||||
│ 2. Name: "validate_items" (inferred from context) │
|
||||
│ 3. Param: items (already in scope) │
|
||||
│ 4. Return: bool (validation pattern) │
|
||||
│ 5. Body: If → condition → Return True/False │
|
||||
│ │
|
||||
│ User sees: "Create validate_items method? [Yes] [Modify] [No]"│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Instead of:
|
||||
```
|
||||
Agent: "Here's the code I generated..."
|
||||
Human: "Almost, but change X"
|
||||
Agent: "Here's the updated code..."
|
||||
Human: "Now there's a syntax error"
|
||||
Agent: "Sorry, here's the fix..."
|
||||
```
|
||||
|
||||
It becomes:
|
||||
```
|
||||
Agent: "I suggest these choices: [validate_items, bool, If pattern]"
|
||||
Human: Clicks [Modify] → changes return type to Optional[str]
|
||||
Agent: Updates downstream choices automatically
|
||||
Human: Clicks [Accept]
|
||||
Done. No back-and-forth. No errors.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Elisp ↔ C++ Projection
|
||||
|
||||
### Why These Languages?
|
||||
|
||||
| Language | Role in Editor |
|
||||
|----------|----------------|
|
||||
| **Elisp** | Emacs server scripting, configuration, extensions |
|
||||
| **C++** | Dear ImGui shell, performance-critical paths |
|
||||
|
||||
Building Elisp ↔ C++ transpilation lets us **dogfood** Whetstone:
|
||||
- Write editor logic in Elisp (comfortable, existing packages)
|
||||
- Generate C++ for performance-critical rendering
|
||||
- Same AST, different projections
|
||||
|
||||
### Language Mapping
|
||||
|
||||
| Elisp | Whetstone AST | C++ |
|
||||
|-------|---------------|-----|
|
||||
| `(defun name (args) body)` | Function node | `RetType name(Args) { body }` |
|
||||
| `(let ((x val)) body)` | Variable + Block | `{ auto x = val; body }` |
|
||||
| `(if cond then else)` | IfStatement | `if (cond) { then } else { else }` |
|
||||
| `(lambda (x) body)` | Lambda | `[](auto x) { return body; }` |
|
||||
| `(mapcar fn list)` | Call: map | `std::transform(...)` |
|
||||
| `'(1 2 3)` | ListLiteral | `std::vector{1, 2, 3}` |
|
||||
|
||||
### Elisp-Specific Idioms
|
||||
|
||||
```
|
||||
@lang_specific(elisp, "special_form", "(interactive \"sPrompt: \")", hint="user_input")
|
||||
@lang_specific(elisp, "macro", "(with-temp-buffer ...)", hint="scoped_resource")
|
||||
@lang_specific(elisp, "advice", "(advice-add 'fn :around ...)", hint="aspect_oriented")
|
||||
```
|
||||
|
||||
### C++-Specific Idioms
|
||||
|
||||
```
|
||||
@lang_specific(cpp, "template", "template<typename T>", hint="generic")
|
||||
@lang_specific(cpp, "raii", "std::lock_guard<std::mutex>", hint="scoped_resource")
|
||||
@lang_specific(cpp, "constexpr", "constexpr", hint="compile_time")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dear ImGui Integration
|
||||
|
||||
### What is Dear ImGui?
|
||||
|
||||
Immediate-mode GUI library. Every frame, you declare what to draw:
|
||||
|
||||
```cpp
|
||||
// C++ Dear ImGui
|
||||
void render() {
|
||||
if (ImGui::Button("Compile")) {
|
||||
runCompile();
|
||||
}
|
||||
ImGui::SliderInt("Opt Level", &optLevel, 0, 3);
|
||||
ImGui::Text("Status: %s", status.c_str());
|
||||
}
|
||||
```
|
||||
|
||||
### Elisp Wrapper (whetstone-imgui.el)
|
||||
|
||||
```elisp
|
||||
;; Elisp Dear ImGui bindings
|
||||
(defun render ()
|
||||
(imgui-button "Compile" #'run-compile)
|
||||
(imgui-slider-int "Opt Level" 'opt-level 0 3)
|
||||
(imgui-text "Status: %s" status))
|
||||
```
|
||||
|
||||
### AST-Defined UI
|
||||
|
||||
The Whetstone AST can define UI elements:
|
||||
|
||||
```
|
||||
UIPanel: "Build Settings"
|
||||
├── @projection: imgui
|
||||
├── children:
|
||||
│ ├── UIButton
|
||||
│ │ ├── label: "Compile"
|
||||
│ │ ├── icon: "hammer"
|
||||
│ │ └── action: → BuildSystem.compile()
|
||||
│ ├── UISlider
|
||||
│ │ ├── label: "Optimization Level"
|
||||
│ │ ├── binding: → config.optLevel
|
||||
│ │ ├── min: 0
|
||||
│ │ └── max: 3
|
||||
│ └── UIText
|
||||
│ ├── template: "Status: {}"
|
||||
│ └── binding: → BuildSystem.status
|
||||
```
|
||||
|
||||
**Same AST, multiple UI projections:**
|
||||
|
||||
| Projection | Output |
|
||||
|------------|--------|
|
||||
| `@projection(imgui)` | Native Dear ImGui widgets |
|
||||
| `@projection(emacs)` | Text-based `[Compile] Opt: [===----] 2` |
|
||||
| `@projection(html)` | Web form with `<button>`, `<input type="range">` |
|
||||
| `@projection(tui)` | ncurses terminal UI |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 2a: Editor Shell (4 weeks)
|
||||
|
||||
| Week | Deliverable |
|
||||
|------|-------------|
|
||||
| 1 | Dear ImGui scaffold with file tree, editor pane, command palette |
|
||||
| 2 | Management Node: spawn/manage headless Emacs servers |
|
||||
| 3 | IPC: route commands ImGui ↔ Management ↔ Emacs |
|
||||
| 4 | Integration: Helm/Vertico search, file operations via TRAMP |
|
||||
|
||||
**Exit Criteria:**
|
||||
- [ ] Open files via Emacs file dialog
|
||||
- [ ] Search with Consult/Helm results rendered in ImGui
|
||||
- [ ] Basic text editing (Emacs buffers displayed in ImGui)
|
||||
|
||||
### Phase 2b: Structured Editing (4 weeks)
|
||||
|
||||
| Week | Deliverable |
|
||||
|------|-------------|
|
||||
| 5 | Structured editing panel: construct chooser UI |
|
||||
| 6 | Field editors: constrained inputs for names, types, expressions |
|
||||
| 7 | Scope awareness: dropdown shows only valid in-scope options |
|
||||
| 8 | Agent API: JSON choice submission, validation responses |
|
||||
|
||||
**Exit Criteria:**
|
||||
- [ ] Create function via UI without typing syntax
|
||||
- [ ] Agent submits JSON → valid AST created
|
||||
- [ ] Invalid choices rejected with helpful errors
|
||||
|
||||
### Phase 2c: Elisp ↔ C++ (4 weeks)
|
||||
|
||||
| Week | Deliverable |
|
||||
|------|-------------|
|
||||
| 9 | Elisp tree-sitter parser integration |
|
||||
| 10 | Elisp projection (AST → Elisp text) |
|
||||
| 11 | C++ projection improvements for Elisp idiom mapping |
|
||||
| 12 | Round-trip testing: Elisp → AST → C++ → AST → Elisp |
|
||||
|
||||
**Exit Criteria:**
|
||||
- [ ] Parse whetstone-bridge.el into AST
|
||||
- [ ] Generate equivalent C++ for performance-critical functions
|
||||
- [ ] Elisp `@lang_specific` annotations preserved in C++ as comments
|
||||
|
||||
---
|
||||
|
||||
## Technical Decisions
|
||||
|
||||
### Management Node: Why C++?
|
||||
|
||||
| Option | Pros | Cons |
|
||||
|--------|------|------|
|
||||
| **C++** | Same as ImGui, no FFI boundary, mature | Memory safety requires discipline |
|
||||
| Rust | Memory safe, good IPC libs | Another language in stack |
|
||||
| Elisp | Single ecosystem | Performance, threading limitations |
|
||||
|
||||
C++ provides:
|
||||
- **Zero FFI overhead** with Dear ImGui (same language)
|
||||
- Mature threading primitives for managing multiple Emacs servers
|
||||
- Excellent JSON libraries (nlohmann/json, rapidjson)
|
||||
- Direct integration with the rendering layer
|
||||
|
||||
**Dogfooding path:** Once Whetstone matures, the Management Node becomes our first major dogfooding target. We'll define its logic in Whetstone AST and generate C++ from it. This proves the system works for real software, not just toy examples.
|
||||
|
||||
### IPC: JSON-RPC over Unix Sockets
|
||||
|
||||
```
|
||||
ImGui Shell ←──JSON-RPC──→ Management Node ←──emacsclient──→ Emacs Servers
|
||||
```
|
||||
|
||||
**Why JSON-RPC:**
|
||||
- Language agnostic
|
||||
- Human readable (debuggable)
|
||||
- Well-supported in C++ and Elisp
|
||||
- LSP uses it (familiar pattern)
|
||||
|
||||
### Emacs Server Specialization
|
||||
|
||||
| Server | Packages Loaded | Purpose |
|
||||
|--------|-----------------|---------|
|
||||
| `file-server` | TRAMP, dired, recentf | File operations |
|
||||
| `search-server` | Consult, ripgrep, fd | Fast search |
|
||||
| `complete-server` | Company, Corfu, Cape | Completion |
|
||||
| `git-server` | Magit, diff-hl | Version control |
|
||||
| `org-server` | Org-mode, org-roam | Documentation |
|
||||
|
||||
**Benefit:** Each server loads only what it needs. Parallel operations don't block each other.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Emacs IPC latency | Sluggish UI | Async commands, local caching |
|
||||
| ImGui learning curve | Slow UI development | Start simple, iterate |
|
||||
| Elisp ↔ C++ semantic gap | Lossy transpilation | `@lang_specific` annotations |
|
||||
| Scope complexity | Wrong options offered | Conservative scoping, user override |
|
||||
| Agent choice explosion | Too many options | Context-aware filtering, ML ranking |
|
||||
|
||||
---
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| File open latency | < 100ms (via Emacs server) |
|
||||
| Search result latency | < 50ms first results |
|
||||
| Structured edit: construct creation | < 10 clicks for simple function |
|
||||
| Agent choice acceptance rate | > 80% first suggestion accepted |
|
||||
| Syntax errors (structured mode) | 0 (by design) |
|
||||
| Elisp → C++ round-trip fidelity | Semantically equivalent |
|
||||
|
||||
---
|
||||
|
||||
## Open Questions for Sprint 2
|
||||
|
||||
1. **Emacs package compatibility** - Which GUI-dependent packages (if any) are critical? Can they be adapted?
|
||||
|
||||
2. **Structured editing escape hatch** - Should users be able to drop into raw text mode? When?
|
||||
|
||||
3. **Agent trust levels** - Can agents auto-accept choices, or always require human confirmation?
|
||||
|
||||
4. **Incremental adoption** - Can structured editing work alongside traditional text editing in same file?
|
||||
|
||||
5. **Performance budget** - What's acceptable latency for Emacs server round-trips?
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
| Dependency | Version | Purpose |
|
||||
|------------|---------|---------|
|
||||
| Dear ImGui | 1.90+ | GUI rendering |
|
||||
| Emacs | 29+ | Headless servers |
|
||||
| nlohmann/json | 3.11+ | JSON-RPC in C++ |
|
||||
| tree-sitter | 0.22+ | Elisp parsing |
|
||||
| tree-sitter-elisp | latest | Elisp grammar |
|
||||
| C++ compiler | C++20 | Management node, ImGui shell |
|
||||
|
||||
---
|
||||
|
||||
## Relationship to Sprint 1
|
||||
|
||||
Sprint 1 (MPS) provides:
|
||||
- Core AST node definitions
|
||||
- SemAnno annotation schema
|
||||
- Python ↔ C++ projection patterns
|
||||
- Memory deref strategy examples
|
||||
|
||||
Sprint 2 consumes Sprint 1 by:
|
||||
- Porting AST definitions from MPS to C++ classes (initially)
|
||||
- Reusing annotation schema for structured editing constraints
|
||||
- Extending projections to include Elisp
|
||||
- Building the editor that will eventually replace MPS for Whetstone development
|
||||
|
||||
**The Bootstrap Path:**
|
||||
```
|
||||
Sprint 1: MPS defines AST → generates Python/C++
|
||||
Sprint 2: C++ editor (hand-written) uses AST definitions
|
||||
Sprint 3: Editor logic defined IN Whetstone AST → generates C++
|
||||
Sprint 4: Self-hosting complete - Whetstone builds Whetstone
|
||||
```
|
||||
Reference in New Issue
Block a user