diff --git a/docs-20260203T163105Z-3-001/docs/REQUIREMENTS_OVERVIEW.md b/docs-20260203T163105Z-3-001/docs/REQUIREMENTS_OVERVIEW.md new file mode 100644 index 0000000..b75fd98 --- /dev/null +++ b/docs-20260203T163105Z-3-001/docs/REQUIREMENTS_OVERVIEW.md @@ -0,0 +1,328 @@ +# Whetstone DSL: Project Requirements Overview + +## Vision Statement + +**Source code should be a structured database of logic, not a text file of syntax.** + +Whetstone is a Universal Logic Database that stores code as a semantic graph (AST), with traditional programming languages serving as **projections** of that canonical representation. This enables lossless transpilation, massively parallel development (human and AI agents), and complete separation of intent from implementation. + +--- + +## Core Philosophy + +### The Problem: Text Blindness +Current software engineering couples **Intent** (what we want to happen) with **Implementation** (how the machine does it) through language-specific syntax. This creates: +- Lock-in to specific languages and ecosystems +- Loss of semantic information during translation +- Inability to express cross-cutting concerns (ownership, risk, optimization hints) +- Barriers to parallel development by multiple agents + +### The Solution: Projectional Editing +Move the "Source of Truth" from text files to a **Logical Graph Database**: + +| Text-Based | Graph-Based (Whetstone) | +|------------|-------------------------| +| Linear | Structural | +| Ambiguous | Explicit | +| Syntax-heavy | Meaning-heavy | +| Language-specific | Universal | + +--- + +## Architecture: The Three Layers + +### Layer 1: The Logic Database (Definition) +A schema of pure logical concepts stored in a graph/tree database (JetBrains MPS). + +**Core Principle:** This layer must be a **superset** of all target languages. It must express concepts like memory ownership (Rust-style) even when targeting languages that don't require them (Python). + +**Contents:** +- AST node concepts (expressions, statements, functions, types) +- Semantic annotations (SemAnno schema) +- Hypergraph relationships (nodes participate in multiple relationships) + +### Layer 2: Semantic Generators (Bridge) +Generators that translate the Logic Database into idiomatic text for specific runtimes. + +**Example - SharedMemoryBlock concept:** +``` +Generator A (Python): x = [1, 2, 3] # Note: Shared usage implied +Generator B (Rust): let x = Arc::new(Mutex::new(vec![1, 2, 3])); +Generator C (Verilog): reg [31:0] memory [0:2]; +``` + +### Layer 3: Runtime (Metal) +Standard toolchains (GCC, LLVM, Python interpreter) compile/execute the generated code. Whetstone does not replace compilers; it feeds them optimized, correct input. + +--- + +## SemAnno: Universal Semantic Annotation Schema + +Six annotation systems provide metadata for agents (human and AI): + +### 1. Complexity & Intelligence ("Rank" System) +Defines who/what is qualified to modify a node. + +| Annotation | Type | Purpose | +|------------|------|---------| +| `@complexity-score` | int | Derived from cyclomatic complexity, nesting, mutation | +| `@intelligence-requirement` | junior/mid/senior | Agent tier required for modification | +| `@review-lock` | bool | Requires second agent verification | + +### 2. Dependency & Impact ("Radius" System) +Calculates the "blast radius" of changes. + +| Annotation | Type | Purpose | +|------------|------|---------| +| `@dependency-count` | int | Incoming references to this node | +| `@dependency-level` | 0-100 | Normalized importance score | +| `@downstream-impact` | list | Affected domains [Network, Security, Billing] | +| `@fragility` | low/high | Load-bearing legacy code flag | + +### 3. Performance & Resource ("Machinist" System) +Low-level optimization metadata. + +| Annotation | Type | Purpose | +|------------|------|---------| +| `@optimization-priority` | low/mid/high | Generator focus for SIMD/unrolling | +| `@latency-critical` | bool | Real-time execution requirement | +| `@memory-footprint` | estimate | Heap/stack allocation intent | +| `@execution-mode` | deterministic/probabilistic | Formal verification hint | +| `@deref-explicit` | bool | Every pointer touch must be accounted | + +### 4. Architectural Intent ("Paradigm" System) +Defines behavioral contracts. + +| Annotation | Type | Purpose | +|------------|------|---------| +| `@intent` | Pure/Stateful/Parallel/Legacy | Primary behavioral contract | +| `@preferred-view` | language | Optimal projection for readability | +| `@lifecycle` | owner/borrower/transient | Ownership model hints | + +### 5. Security & Risk ("Sentinel" System) +Granular risk tracking. + +| Annotation | Type | Purpose | +|------------|------|---------| +| `@risk-type` | identifier | BufferOverflow, RaceCondition, SideChannel | +| `@trust-boundary` | marker | Secure environment transition point | +| `@provenance` | hash | Chain of agent modifications | + +### 6. Projective Metadata ("Lens" System) +Controls rendering in projectional environments. + +| Annotation | Type | Purpose | +|------------|------|---------| +| `@view-mask` | category | Toggle annotation visibility by type | +| `@cognitive-level` | 0-100 | Detail density filter | +| `@projection-alias` | string | Localized renaming without AST change | +| `@hidden-detail` | bool | Collapse boilerplate for reviews | + +### 7. Language-Specific Idioms ("Polyglot" System) +Preserves language-specific features that don't map directly to other languages. + +| Annotation | Type | Purpose | +|------------|------|---------| +| `@lang_specific` | structured | Captures idiom with language, type, raw syntax, and semantic hint | + +**Architecture:** +``` +@lang_specific(python, "decorator", "@lru_cache(maxsize=128)", hint="memoization") +@lang_specific(cpp, "template", "template", hint="generic") +``` + +**Projection Behavior:** +- In native language: renders as original syntax +- In foreign language: renders as comment with semantic hint + +**Long-term Vision:** These annotations enable future semantic equivalence detection. When the system understands that `@lru_cache` and a C++ `std::map`-based cache serve the same purpose ("memoization"), it can generate idiomatic implementations in either direction rather than just commenting. + +--- + +## Memory Dereferencing Strategies + +Memory management is decoupled from logic as a first-class AST field. + +| Strategy | Description | Generator Responsibility | +|----------|-------------|-------------------------| +| `@deref(imperative)` | Manual control - developer defines time/location | Whetstone Generator / Substrate | +| `@deref(streamed)` | Ownership-based - lifetime tracked at declaration (Rust-like) | Whetstone Generator | +| `@deref(batched)` | Garbage collected - background process or ref-counting | Whetstone Generator | +| `@deref(content-addressed)` | Immutable - identity by hash, location irrelevant | Whetstone Generator | + +### The "Unfilled Node Constraint" +In `@deref(imperative)` mode, the AST requires an explicit **Dereference Time Field**. If empty, the editor flags it as a "Missing Intent" error. This makes it impossible to forget a dereference. + +--- + +## Strategy Toggles + +Declarative optimization patterns applicable without manual implementation: + +| Toggle | Effect | +|--------|--------| +| `@strategy(TCO)` | Convert recursive logic to iterative machine code | +| `@strategy(Memoize)` | Inject thread-safe cache for function results | +| `@strategy(Unroll)` | Duplicate loop bodies per target-hardware heuristic | +| `@strategy(SSO)` | Enable Small String/Buffer Optimization | + +--- + +## Transpilation Modes + +### Direct Transpilation ("Wrapper") +Maps nodes 1-to-1. High compatibility, poor performance. +- Python dynamic list → `std::vector` with runtime type-checking + +### Idiomatic Transpilation ("Intent") +Analyzes usage patterns. High performance, hardware sympathy. +- Python list (append-only, sequential read) → `std::vector` or Memory Arena + +--- + +## The Resolution Slider + +Four levels from intent to execution: + +| Level | Description | Example | +|-------|-------------|---------| +| 1 | Pseudocode/Intent | "Find a user in a list" | +| 2 | High-Level (JS/Python) | `Array.find()` | +| 3 | Intermediate (Rust/Java) | Type safety + ownership scopes | +| 4 | Execution Substrate (C++/C) | B-Trees, SIMD, Branch Hints | + +--- + +## Multi-Agent Collaboration + +### The Round-Trip Problem +When a systems engineer optimizes code that a high-level developer later modifies. + +### Solution: Persistent Metadata Separation +- **Logic Field:** Algorithmic intent (editable by junior) +- **Optimization Field:** Systems-level instructions (requires senior) + +### Conflict Management +When high-level changes break low-level optimizations: +1. **Shadowed Metadata:** Old optimizations hidden but not deleted +2. **Alerting:** "Performance Guardrail" warning shown +3. **Diff Lens:** Visualizes where intent changes invalidated hardware instructions + +--- + +## Hypergraph Requirement + +Standard ASTs are strictly hierarchical (Parent → Child). Real-world logic is a **Hypergraph**: + +- A Variable is **owned by** Function A +- But **used by** Function B +- And **monitored by** Agent C + +**Implementation:** Smart References (Hyperedges) allow a single logical node to participate in multiple relationships simultaneously. + +--- + +## Agent-Native Development + +### Task Decomposition Model +1. **High-context coordinator agent** builds semantic graph and queues tasks +2. **Small Language Models (8B)** handle tightly scoped tasks +3. **Human developers** work from the same task queue + +### Projection Optimization +Tasks can be projected in: +- **Human-optimized view:** Natural language descriptions, readable code +- **AI-optimized view:** Structured data, explicit constraints, minimal ambiguity + +### Agent Tiers +| Tier | Capabilities | +|------|--------------| +| Junior | Basic logic, no memory management, no high-risk annotations | +| Mid | Standard logic, `@deref(streamed)` and `@deref(batched)` | +| Senior | `@deref(imperative)`, hardware optimization, high-risk refactoring | + +--- + +## Target Applications + +### Legacy Modernization +Parse old Fortran/COBOL → Logic Database → Generate modern C++/Rust/Julia + +### Cross-Platform Logic +Write business logic once → Generate Kotlin (Android), Swift (iOS), Rust (backend) + +### Agentic Coding +Agents manipulate Graph Nodes directly (no syntax errors from text generation) + +### Parallel Development +Multiple developers/agents work on same codebase with explicit conflict boundaries + +--- + +## Technical Principles + +### Behavior Over Implementation +A node's primary definition is its **Behavioral Contract** (e.g., "Sort this collection") rather than implementation (e.g., "Quicksort with pointer increments"). + +### Substrate-Agnostic Core Nodes +Core AST nodes must not assume specific memory models or hardware constraints. +- **Bad:** LoopNode requires a "Pointer" field +- **Good:** LoopNode requires an "IteratorIntent" - realization is metadata + +### The Semantic Boundary +Strict boundary between **What** (Logic) and **How** (Execution Strategy). Future capabilities are implemented by adding Strategy Engines to existing nodes, not re-architecting nodes. + +### Strictest Common Denominator +The Logic Database enforces strict rules by default (explicit ownership), while Generators can "relax" rules for permissive languages or "poly-fill" rules for strict languages. + +--- + +## The C Primitives Substrate (Long-term Vision) + +All high-level language features ultimately resolve to C-level primitives: + +| High-Level Feature | C Primitive Resolution | +|-------------------|------------------------| +| Python `@lru_cache` | Hash table + function pointer + malloc/free | +| C++ `template` | N compiled copies with concrete types | +| Python decorator | Function pointer manipulation | +| C++ RAII | Constructor/destructor at scope boundaries | +| Rust ownership | Compile-time tracking → equivalent to manual C | +| Garbage collection | Reference counting or mark-sweep algorithms | + +**Architectural Implication:** + +``` +Level 4: Language Idioms @py_decl, @cpp_tmpl, @rust_lifetime + │ + ▼ +Level 3: Universal AST Function, Loop, Variable, Expression + │ + ▼ +Level 2: Semantic Intent "memoize", "parameterize by type", "scope-bound cleanup" + │ + ▼ +Level 1: C Primitives pointers, structs, malloc/free, function calls +``` + +**End State:** Languages become pure *syntactic projections* over C primitives + semantic annotations. This enables true lossless transpilation because the canonical form captures both the intent (semantic annotations) and the implementation (C primitives). + +--- + +## Implementation Platform + +**JetBrains MPS (Meta Programming System)** +- Projectional editor (edit AST directly, not text) +- Language workbench for defining DSLs +- Built-in support for generators +- Extensible type system and constraints + +--- + +## Success Criteria + +1. **Lossless Round-Trip:** Python → AST → C++ → AST → Python produces semantically equivalent code +2. **Parallel Safety:** Junior developer cannot accidentally break senior's optimizations +3. **Agent Compatibility:** Tasks can be completed by SLMs with sufficient annotation context +4. **Performance Parity:** Generated C++ matches hand-written C++ for common patterns +5. **Adoption Path:** Existing codebases can be incrementally imported via tree-sitter parsing diff --git a/docs-20260203T163105Z-3-001/docs/SPRINT_1_REQUIREMENTS.md b/docs-20260203T163105Z-3-001/docs/SPRINT_1_REQUIREMENTS.md new file mode 100644 index 0000000..fbbca3c --- /dev/null +++ b/docs-20260203T163105Z-3-001/docs/SPRINT_1_REQUIREMENTS.md @@ -0,0 +1,548 @@ +# Sprint 1: Python ↔ C++ Dual Projection + +## Sprint Goal + +Enable a junior developer to write working logic in Python, a senior developer to optimize it in C++, and the junior developer to continue modifying safe areas while receiving warnings about optimization-locked regions. + +--- + +## Scope + +| In Scope | Out of Scope | +|----------|--------------| +| Python and C++ projections | Other languages | +| Tree-sitter parsing for ingestion | Custom parser implementation | +| Memory deref strategy annotations | Full SemAnno schema | +| Warning system for locked nodes | Hard locks / approval workflows | +| Basic AST nodes (functions, loops, variables, expressions) | Advanced nodes (generics, macros, templates) | +| MPS projectional editor | IDE plugins (VS Code, etc.) | + +--- + +## User Stories + +### US-1: Python Developer Writes Logic +**As a** junior Python developer +**I want to** write business logic in a Python-like projection +**So that** I can focus on correctness without worrying about memory management + +### US-2: C++ Developer Optimizes +**As a** senior C++ developer +**I want to** switch to a C++ projection and add optimization annotations +**So that** the generated code is production-ready and performant + +### US-3: Python Developer Sees Warnings +**As a** junior Python developer +**I want to** see warnings when I try to modify optimized nodes +**So that** I understand the impact of my changes without being blocked + +### US-4: Import Existing Code +**As a** developer +**I want to** import existing Python or C++ files into the AST +**So that** I can work with legacy code in Whetstone + +--- + +## Technical Requirements + +### TR-1: Core AST Nodes + +Define MPS concepts for a minimal but complete AST: + +``` +Module +├── Function +│ ├── name: string +│ ├── parameters: Parameter[] +│ ├── returnType: TypeReference +│ ├── body: Statement[] +│ └── annotations: SemAnno[] +├── Variable +│ ├── name: string +│ ├── type: TypeReference +│ ├── initializer: Expression? +│ └── annotations: SemAnno[] +└── annotations: SemAnno[] + +Statement (abstract) +├── Assignment +├── IfStatement +├── WhileLoop +├── ForLoop +├── Return +├── ExpressionStatement +└── Block + +Expression (abstract) +├── BinaryOperation +├── UnaryOperation +├── FunctionCall +├── VariableReference +├── Literal (int, float, string, bool) +├── ListLiteral +├── IndexAccess +└── MemberAccess +``` + +### TR-2: Memory Deref Strategy Annotation + +``` +DerefStrategy +├── strategy: enum {imperative, streamed, batched, content_addressed} +├── derefTime: Expression? (required if imperative) +├── derefLocation: string? (required if imperative) +└── owner: AgentReference? +``` + +### TR-3: Optimization Lock Annotation + +``` +OptimizationLock +├── lockedBy: AgentReference +├── lockReason: string +├── lockLevel: enum {warning, soft, hard} // Sprint 1: warning only +├── affectedStrategies: string[] +└── timestamp: datetime +``` + +### TR-4: Language-Specific Idiom Annotation + +Placeholder annotations that preserve language-specific features during import without requiring full semantic understanding. + +``` +LangSpecific +├── language: enum {python, cpp, rust, ...} +├── idiomType: string // "decorator", "template", "attribute", "pragma", etc. +├── rawSyntax: string // The original syntax as written +├── semanticHint: string? // Optional: "memoization", "generic", "compile-hint" +└── position: enum {before, after, wrapping} // Where it attaches to the node +``` + +**Examples:** + +``` +// Python decorator captured during import +@lang_specific(python, "decorator", "@lru_cache(maxsize=128)", hint="memoization", position=before) + +// C++ template captured during import +@lang_specific(cpp, "template", "template", hint="generic", position=before) + +// C++ pragma +@lang_specific(cpp, "pragma", "#pragma omp parallel for", hint="parallelization", position=before) + +// Python type hint that has no C++ equivalent +@lang_specific(python, "type_hint", "-> Generator[int, None, None]", hint="generator", position=after) + +// C++ attribute +@lang_specific(cpp, "attribute", "[[nodiscard]]", hint="return_value_check", position=before) +``` + +**Projection Behavior:** + +| In Native Projection | In Foreign Projection | +|---------------------|----------------------| +| Rendered as original syntax | Rendered as comment with semantic hint | + +**Python projection of a C++ template function:** +```python +# cpp_template: template (generic) +# NOTE: Python version uses dynamic typing instead +def compute(value): + ... +``` + +**C++ projection of a Python decorated function:** +```cpp +// python_decorator: @lru_cache(maxsize=128) (memoization) +// NOTE: Implement memoization manually or use std::map cache +int compute(int value) { + ... +} +``` + +### TR-5: Tree-sitter Integration + +- Python grammar: `tree-sitter-python` +- C++ grammar: `tree-sitter-cpp` +- Map tree-sitter CST nodes to Whetstone AST concepts +- Preserve source locations for round-trip fidelity + +### TR-6: Generators + +**Python Generator:** +- Emit idiomatic Python from AST +- Ignore deref strategies (Python is GC'd) +- Emit annotations as comments or type hints where applicable + +**C++ Generator:** +- Emit idiomatic C++ from AST +- Translate deref strategies to appropriate constructs +- Inject required includes and memory management code + +--- + +## Memory Deref Strategy Examples + +The following examples show the **same logical function** with different memory strategies, demonstrating how the C++ projection changes while Python remains constant. + +### Example 1: Processing a List of Records + +#### Whetstone AST (Canonical Form) +``` +Function: processRecords + @deref(???) // Strategy applied here + Parameter: records -> List + Body: + ForLoop: + iterator: record in records + body: + Call: record.validate() + Call: record.transform() + Return: records +``` + +#### Python Projection (Same for all strategies) +```python +def process_records(records: list[Record]) -> list[Record]: + """ + @deref: {strategy} # Annotation visible but doesn't affect code + """ + for record in records: + record.validate() + record.transform() + return records +``` + +#### C++ Projection: @deref(batched) - Garbage Collected +```cpp +// Memory managed by shared_ptr reference counting +#include +#include + +std::vector> process_records( + std::vector> records +) { + for (auto& record : records) { + record->validate(); + record->transform(); + } + return records; +} +``` + +#### C++ Projection: @deref(streamed) - Ownership-Based (Rust-like) +```cpp +// Ownership transferred, no copies, RAII cleanup +#include +#include + +std::vector> process_records( + std::vector> records // Takes ownership +) { + for (auto& record : records) { + record->validate(); + record->transform(); + } + return records; // Transfers ownership to caller +} +``` + +#### C++ Projection: @deref(imperative) - Manual Control +```cpp +// Developer explicitly controls allocation and deallocation +// @deref-time: end_of_function +// @deref-location: caller_responsibility +#include + +std::vector process_records( + std::vector records, // Raw pointers - caller owns + size_t count +) { + for (size_t i = 0; i < count; ++i) { + records[i]->validate(); + records[i]->transform(); + } + return records; + // NOTE: No deallocation here - @deref-location specifies caller handles it +} +``` + +#### C++ Projection: @deref(content-addressed) - Immutable +```cpp +// Immutable data, identity by content hash, can be freely shared +#include +#include + +struct ImmutableRecord { + const std::string data; + const size_t hash; + + ImmutableRecord transform() const { + // Returns NEW record, original unchanged + return ImmutableRecord{transformed_data, new_hash}; + } +}; + +std::vector process_records( + const std::vector& records // Immutable reference +) { + std::vector results; + results.reserve(records.size()); + for (const auto& record : records) { + record.validate(); // Throws if invalid, doesn't mutate + results.push_back(record.transform()); // New record + } + return results; +} +``` + +--- + +### Example 2: Building a Cache + +#### Whetstone AST (Canonical Form) +``` +Function: getOrCreate + @deref(???) + Parameter: cache -> Map + Parameter: key -> string + Body: + IfStatement: + condition: key in cache + then: Return cache[key] + else: + Assignment: widget = Widget.create(key) + Assignment: cache[key] = widget + Return: widget +``` + +#### Python Projection +```python +def get_or_create(cache: dict[str, Widget], key: str) -> Widget: + """ + @deref: {strategy} + """ + if key in cache: + return cache[key] + else: + widget = Widget.create(key) + cache[key] = widget + return widget +``` + +#### C++ Projection: @deref(batched) +```cpp +std::shared_ptr get_or_create( + std::unordered_map>& cache, + const std::string& key +) { + auto it = cache.find(key); + if (it != cache.end()) { + return it->second; + } + auto widget = std::make_shared(Widget::create(key)); + cache[key] = widget; + return widget; +} +``` + +#### C++ Projection: @deref(streamed) +```cpp +// Note: Unique ownership makes caching tricky - must use reference +Widget& get_or_create( + std::unordered_map>& cache, + const std::string& key +) { + auto it = cache.find(key); + if (it != cache.end()) { + return *it->second; + } + auto [inserted_it, _] = cache.emplace( + key, + std::make_unique(Widget::create(key)) + ); + return *inserted_it->second; +} +``` + +#### C++ Projection: @deref(imperative) +```cpp +// @deref-time: cache_destruction +// @deref-location: CacheManager::cleanup() +Widget* get_or_create( + std::unordered_map& cache, + const std::string& key +) { + auto it = cache.find(key); + if (it != cache.end()) { + return it->second; + } + Widget* widget = new Widget(Widget::create(key)); // Manual allocation + cache[key] = widget; + return widget; + // Caller note: Deallocation handled by CacheManager::cleanup() +} +``` + +--- + +## Warning System Specification + +### Warning Triggers +A warning appears when a node has an `OptimizationLock` annotation and the current user's tier is below the lock level. + +### Warning Display +``` +┌─────────────────────────────────────────────────────────────┐ +│ ⚠ OPTIMIZATION WARNING │ +├─────────────────────────────────────────────────────────────┤ +│ This node was optimized by: senior_dev_alice │ +│ Optimization: @deref(imperative) with SIMD vectorization │ +│ │ +│ Modifying this code will: │ +│ • Invalidate the manual memory management strategy │ +│ • Disable the 4x SIMD optimization │ +│ │ +│ [Proceed Anyway] [View C++ Projection] [Cancel] │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Warning Metadata +When user proceeds despite warning: +1. Original optimization is **shadowed** (preserved but inactive) +2. `@provenance` updated with modification chain +3. Notification queued for original optimizer + +--- + +## File Structure for Sprint 1 + +``` +languages/ +├── SemAnno/ +│ └── models/ +│ └── SemAnno.structure.mps # Update with new concepts +├── WhetstoneCore/ # NEW: Core AST language +│ ├── models/ +│ │ ├── WhetstoneCore.structure.mps +│ │ ├── WhetstoneCore.editor.mps +│ │ ├── WhetstoneCore.constraints.mps +│ │ └── WhetstoneCore.typesystem.mps +│ └── WhetstoneCore.mpl +├── PythonProjection/ # NEW: Python view +│ ├── models/ +│ │ ├── PythonProjection.editor.mps +│ │ └── PythonProjection.textGen.mps +│ └── PythonProjection.mpl +├── CppProjection/ # NEW: C++ view +│ ├── models/ +│ │ ├── CppProjection.editor.mps +│ │ └── CppProjection.textGen.mps +│ └── CppProjection.mpl +└── TreeSitterImport/ # NEW: Import from source + ├── models/ + │ ├── TreeSitterImport.behavior.mps + │ └── TreeSitterImport.plugin.mps + └── TreeSitterImport.mpl +``` + +--- + +## Acceptance Criteria + +### AC-1: Round-Trip Parsing +- [ ] Parse Python file with tree-sitter → Whetstone AST +- [ ] Parse C++ file with tree-sitter → Whetstone AST +- [ ] Generate Python from AST that is functionally equivalent +- [ ] Generate C++ from AST that compiles and runs + +### AC-2: Deref Strategy Application +- [ ] Apply `@deref(batched)` to a function → C++ uses shared_ptr +- [ ] Apply `@deref(streamed)` to a function → C++ uses unique_ptr +- [ ] Apply `@deref(imperative)` to a function → C++ uses raw pointers +- [ ] Python projection shows annotation but code unchanged + +### AC-3: Warning System +- [ ] Add optimization lock to node +- [ ] Junior user attempts edit → warning appears +- [ ] Warning shows what will be invalidated +- [ ] User can proceed (warning only, not blocking) +- [ ] Provenance updated after edit + +### AC-4: Dual Projection Editing +- [ ] Open AST in Python projection → edit logic +- [ ] Switch to C++ projection → same logic, different syntax +- [ ] Edit in C++ projection → Python projection updates +- [ ] Annotations visible in both projections + +### AC-5: Language-Specific Idiom Preservation +- [ ] Import Python file with decorators → `@lang_specific` annotations created +- [ ] Import C++ file with templates → `@lang_specific` annotations created +- [ ] View Python decorator in C++ projection → shows as comment with hint +- [ ] View C++ template in Python projection → shows as comment with hint +- [ ] Re-export to original language → idiom syntax restored exactly +- [ ] Semantic hints populated for known patterns (lru_cache → "memoization") + +--- + +## Implementation Order + +1. **Week 1-2: Core AST Nodes** + - Define WhetstoneCore language structure + - Implement basic editors for each node type + - Manual AST creation works in MPS + +2. **Week 3-4: Python Projection** + - Python-syntax editor for AST nodes + - Python text generator + - Manual round-trip: type Python-like → see generated .py + +3. **Week 5-6: C++ Projection** + - C++ syntax editor for AST nodes + - C++ text generator with deref strategy translation + - Deref strategy annotations affect generated code + +4. **Week 7-8: Tree-sitter Import** + - Integrate tree-sitter-python + - Integrate tree-sitter-cpp + - Parse source files → populate AST + +5. **Week 9-10: Warning System** + - OptimizationLock annotation + - Warning UI in editor + - Provenance tracking + +--- + +## Open Questions + +1. **Deref inference:** Should the system suggest deref strategies based on usage patterns, or always require explicit annotation? + +2. **Partial optimization:** Can a senior optimize just one function while leaving others with default `@deref(batched)`? + +3. **Conflict granularity:** If a junior modifies a loop inside an optimized function, does that invalidate the whole function or just the loop? + +4. **Tree-sitter fidelity:** How do we handle Python/C++ features that don't map cleanly to each other (e.g., Python decorators, C++ templates)? + +--- + +## Dependencies + +| Dependency | Purpose | Source | +|------------|---------|--------| +| JetBrains MPS 2023.2+ | Language workbench | jetbrains.com/mps | +| tree-sitter | Parsing | github.com/tree-sitter | +| tree-sitter-python | Python grammar | github.com/tree-sitter/tree-sitter-python | +| tree-sitter-cpp | C++ grammar | github.com/tree-sitter/tree-sitter-cpp | + +--- + +## Success Metrics + +| Metric | Target | +|--------|--------| +| Parse success rate (Python) | >95% of valid Python files | +| Parse success rate (C++) | >90% of valid C++ files | +| Round-trip fidelity | Semantically equivalent output | +| Warning accuracy | 100% of locked nodes trigger warnings | +| Generator correctness | Generated C++ compiles without errors | diff --git a/docs-20260203T163105Z-3-001/docs/SPRINT_2_VISION.md b/docs-20260203T163105Z-3-001/docs/SPRINT_2_VISION.md new file mode 100644 index 0000000..0c9fbcb --- /dev/null +++ b/docs-20260203T163105Z-3-001/docs/SPRINT_2_VISION.md @@ -0,0 +1,668 @@ +# Sprint 2 Vision: Whetstone Editor & Structured Editing + +## Vision Statement + +Build a next-generation code editor that combines Emacs's powerful libraries with a modern, symbol-rich UI, and introduce **structured editing** where humans and AI agents construct code by choosing from legal primitives rather than typing syntax. + +--- + +## Goals + +1. **Hybrid Editor Architecture** - Dear ImGui shell + headless Emacs servers +2. **Structured Editing UI** - Choice-based code construction, syntax errors impossible +3. **Unified Human/Agent Interface** - Same API for clicking and scripting +4. **Elisp ↔ C++ Projection** - Dogfood Whetstone by building the editor with it +5. **Proof of Concept** - Demonstrate AST-native development accelerates iteration + +--- + +## Architecture: Dear ImGui + Headless Emacs + +### Why Hybrid? + +| Approach | Pros | Cons | +|----------|------|------| +| Fork Emacs | Full compatibility | Massive codebase, GUI baggage | +| Pure Dear ImGui | Clean slate, full control | Lose Emacs libraries | +| **Hybrid** | Best of both | Complexity in coordination | + +The hybrid approach gives us: +- **Emacs libraries** without Emacs's GUI limitations +- **Dear ImGui flexibility** without reimplementing text processing +- **Parallelism** via multiple Emacs server instances +- **Upstream compatibility** - Emacs updates just work + +### System Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ Whetstone Editor │ +├─────────────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Dear ImGui Shell (C++) │ │ +│ │ │ │ +│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │ │ +│ │ │ File │ │ Editor │ │ Terminal │ │ Structured │ │ │ +│ │ │ Tree │ │ Tabs │ │ Panel │ │ Edit Panel │ │ │ +│ │ │ │ │ │ │ │ │ │ │ │ +│ │ │ 📁 src │ │ [foo.py] │ │ $ _ │ │ [Function ▼] │ │ │ +│ │ │ 📁 lib │ │ [bar.cpp]│ │ │ │ [Variable ▼] │ │ │ +│ │ │ 📄 main │ │ │ │ │ │ [Expression ▼] │ │ │ +│ │ └──────────┘ └──────────┘ └──────────┘ └──────────────────┘ │ │ +│ │ │ │ +│ │ ┌──────────────────────────────────────────────────────────┐ │ │ +│ │ │ Command Palette (Emacs-style, keyboard-driven) │ │ │ +│ │ │ > find-file > magit-status > consult-ripgrep │ │ │ +│ │ └──────────────────────────────────────────────────────────┘ │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ IPC (JSON-RPC / Shared Memory) │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Management Node (C++) │ │ +│ │ │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │ +│ │ │ Whetstone │ │ Server │ │ Agent │ │ │ +│ │ │ AST Store │ │ Pool Mgr │ │ Queue │ │ │ +│ │ │ │ │ │ │ │ │ │ +│ │ │ [AST Nodes] │ │ spawn() │ │ [Choice sequences from │ │ │ +│ │ │ [Annotations│ │ route() │ │ human clicks or │ │ │ +│ │ │ [Projections│ │ sync() │ │ agent submissions] │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ │ +│ │ │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ emacsclient protocol │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ Headless Emacs Server Pool │ │ +│ │ │ │ +│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ +│ │ │ Server 1 │ │ Server 2 │ │ Server N │ │ │ +│ │ │ │ │ │ │ │ │ │ +│ │ │ File I/O │ │ Search │ │ Git (Magit) │ │ │ +│ │ │ TRAMP │ │ Completion │ │ Org-mode │ │ │ +│ │ │ Dired │ │ Helm/Vertico│ │ LSP bridge │ │ │ +│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ +│ │ │ │ +│ │ (Each runs: emacs --daemon=serverN --load whetstone-bridge.el) │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +### Component Responsibilities + +| Component | Language | Responsibility | +|-----------|----------|----------------| +| **Dear ImGui Shell** | C++ | All rendering, keyboard/mouse input, widgets | +| **Management Node** | C++ | AST storage, server coordination, agent queue | +| **Emacs Servers** | Elisp | Text processing, file ops, search, completion | + +**Future Goal:** Once Whetstone is mature, rewrite the Management Node in Whetstone AST and generate C++ from it. This is the ultimate dogfooding - the editor's core logic defined in the system it's built to create. + +### Communication Protocol + +``` +┌────────────┐ JSON-RPC ┌─────────────┐ +│ ImGui Shell│ ◄─────────────────────────► │ Mgmt Node │ +└────────────┘ └─────────────┘ + │ + │ emacsclient -e '(elisp-expr)' + ▼ + ┌─────────────┐ + │ Emacs Server│ + └─────────────┘ +``` + +**Example flow - User searches for file:** + +1. User presses `Ctrl+P` in ImGui shell +2. ImGui sends `{ "method": "find-file", "query": "" }` to Management Node +3. Management Node routes to Search server: `emacsclient -s search -e '(consult-find "")'` +4. Emacs returns candidates +5. Management Node sends candidates to ImGui +6. ImGui renders searchable dropdown +7. User selects file +8. Management Node loads file into AST, sends to ImGui for rendering + +--- + +## Structured Editing: The Core Innovation + +### The Problem with Text-Based Coding + +``` +Traditional flow: + Human/Agent generates text → Parser validates → Errors → Fix → Repeat + +Problems: + - Syntax errors waste time + - Agents hallucinate invalid syntax + - Autocompletion is probabilistic (might suggest wrong things) + - Context is implicit (what's in scope? what types are valid?) +``` + +### The Solution: Choice-Based Construction + +**You never type code. You choose from valid options at each step.** + +``` +Structured flow: + Choose construct → Fill constrained fields → Choose next construct → Done + +Benefits: + - Syntax errors IMPOSSIBLE + - Every option shown is LEGAL + - Context is EXPLICIT (only in-scope items offered) + - Agents submit choice sequences (no parsing, no errors) +``` + +### Structured Editing UI Flow + +#### Step 1: Choose Construct Type +``` +┌─────────────────────────────────────────────────────────────────┐ +│ What do you want to create? │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Function │ │ Class │ │ Variable │ │ Import │ │ +│ │ λ │ │ ◇ │ │ x= │ │ ↓ │ │ +│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ If │ │ For │ │ While │ │ Return │ │ +│ │ ? │ │ ↻ │ │ ↺ │ │ ← │ │ +│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ +│ │ +│ [Keyboard: f=Function, c=Class, v=Variable, i=Import...] │ +└─────────────────────────────────────────────────────────────────┘ +``` + +#### Step 2: Fill Constrained Fields (Function selected) +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Define Function │ +│ │ +│ Name: [process_data________] ← Text input, identifier rules│ +│ │ +│ Parameters: │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Name Type Default │ │ +│ │ [items ] [List[int] ▼] [None ▼] │ │ +│ │ [threshold] [int ▼] [10 ] │ │ +│ │ │ │ +│ │ [+ Add Parameter] │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ Return Type: [List[int] ▼] [None] [Auto-infer] │ +│ │ +│ Annotations: │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ [@deref: batched ▼] [@complexity: ___] [+ Add] │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ [Continue to Body →] │ +└─────────────────────────────────────────────────────────────────┘ +``` + +#### Step 3: Build Function Body +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Function: process_data(items: List[int], threshold: int) │ +│ │ +│ Body: (Click to add statement) │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ 1. [+ Add Statement] │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ ───────────────────────────────────────────────────────────── │ +│ Choose statement type: │ +│ │ +│ [Variable] [If] [For] [While] [Return] [Call] [Assign] │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +#### Step 4: Add Variable (Variable selected) +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Create Variable │ +│ │ +│ Name: [result_________] │ +│ │ +│ Initialize with: │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Literal │ │ Variable │ │ Call │ │ BinaryOp │ │ +│ │ 42 │ │ x │ │ f() │ │ a+b │ │ +│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ +│ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +#### Step 5: Build Expression (Call selected) +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Function Call │ +│ │ +│ Function: [filter ▼] ← Dropdown: in-scope functions │ +│ ┌─────────────────┐ │ +│ │ filter │ ← Built-in │ +│ │ map │ ← Built-in │ +│ │ process_data │ ← Current function (recursion) │ +│ │ helper_func │ ← Defined above │ +│ └─────────────────┘ │ +│ │ +│ Arguments: │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Arg 1 (callable): [lambda ▼] → [x > threshold] │ │ +│ │ Arg 2 (iterable): [items ▼] ← Only iterables shown │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ [Done - Add to Body] │ +└─────────────────────────────────────────────────────────────────┘ +``` + +#### Result: Complete Function +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Function: process_data │ +│ ├── params: [(items, List[int]), (threshold, int, default=10)]│ +│ ├── returns: List[int] │ +│ ├── @deref: batched │ +│ └── body: │ +│ └── Return │ +│ └── Call: list │ +│ └── Call: filter │ +│ ├── Lambda: x > threshold │ +│ └── Var: items │ +│ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Preview (Python): │ │ +│ │ │ │ +│ │ def process_data(items: List[int], threshold: int = 10): │ │ +│ │ return list(filter(lambda x: x > threshold, items)) │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ +│ [Edit] [View as C++] [Add Annotation] [Done] │ +└─────────────────────────────────────────────────────────────────┘ +``` + +--- + +## Agent API: Scripted Choices + +The structured editing UI has a direct JSON equivalent. Agents don't generate code text - they submit choice sequences. + +### Human Interaction (UI) +``` +Click: Function → Type: "process_data" → Click: Add Param → ... +``` + +### Agent Interaction (JSON) +```json +{ + "construct": "Function", + "name": "process_data", + "params": [ + { "name": "items", "type": "List[int]" }, + { "name": "threshold", "type": "int", "default": "10" } + ], + "return_type": "List[int]", + "annotations": [ + { "type": "deref", "strategy": "batched" } + ], + "body": [ + { + "statement": "Return", + "expression": { + "type": "Call", + "function": "list", + "args": [ + { + "type": "Call", + "function": "filter", + "args": [ + { + "type": "Lambda", + "params": ["x"], + "body": { "type": "BinaryOp", "left": "x", "op": ">", "right": "threshold" } + }, + { "type": "VariableRef", "name": "items" } + ] + } + ] + } + } + ] +} +``` + +### Validation Guarantees + +The Management Node validates each choice: + +``` +Agent submits: { "function": "nonexistent_func", ... } +System responds: { + "error": "InvalidChoice", + "field": "function", + "valid_options": ["filter", "map", "process_data", "helper_func"], + "reason": "Function 'nonexistent_func' not in scope" +} +``` + +**No parsing errors. No syntax errors. Only semantic validation.** + +### Agent Workflow: Context-Aware Queuing + +A local agent with project context can pre-compute likely choices: + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Agent Context │ +│ ├── Current file: data_processor.py │ +│ ├── Cursor: inside class DataProcessor │ +│ ├── In scope: self, items, config, logger │ +│ └── Recent pattern: user adding validation methods │ +│ │ +│ Agent predicts user wants: validation method │ +│ Pre-queued choices: │ +│ 1. Function (method) │ +│ 2. Name: "validate_items" (inferred from context) │ +│ 3. Param: items (already in scope) │ +│ 4. Return: bool (validation pattern) │ +│ 5. Body: If → condition → Return True/False │ +│ │ +│ User sees: "Create validate_items method? [Yes] [Modify] [No]"│ +└─────────────────────────────────────────────────────────────────┘ +``` + +Instead of: +``` +Agent: "Here's the code I generated..." +Human: "Almost, but change X" +Agent: "Here's the updated code..." +Human: "Now there's a syntax error" +Agent: "Sorry, here's the fix..." +``` + +It becomes: +``` +Agent: "I suggest these choices: [validate_items, bool, If pattern]" +Human: Clicks [Modify] → changes return type to Optional[str] +Agent: Updates downstream choices automatically +Human: Clicks [Accept] +Done. No back-and-forth. No errors. +``` + +--- + +## Elisp ↔ C++ Projection + +### Why These Languages? + +| Language | Role in Editor | +|----------|----------------| +| **Elisp** | Emacs server scripting, configuration, extensions | +| **C++** | Dear ImGui shell, performance-critical paths | + +Building Elisp ↔ C++ transpilation lets us **dogfood** Whetstone: +- Write editor logic in Elisp (comfortable, existing packages) +- Generate C++ for performance-critical rendering +- Same AST, different projections + +### Language Mapping + +| Elisp | Whetstone AST | C++ | +|-------|---------------|-----| +| `(defun name (args) body)` | Function node | `RetType name(Args) { body }` | +| `(let ((x val)) body)` | Variable + Block | `{ auto x = val; body }` | +| `(if cond then else)` | IfStatement | `if (cond) { then } else { else }` | +| `(lambda (x) body)` | Lambda | `[](auto x) { return body; }` | +| `(mapcar fn list)` | Call: map | `std::transform(...)` | +| `'(1 2 3)` | ListLiteral | `std::vector{1, 2, 3}` | + +### Elisp-Specific Idioms + +``` +@lang_specific(elisp, "special_form", "(interactive \"sPrompt: \")", hint="user_input") +@lang_specific(elisp, "macro", "(with-temp-buffer ...)", hint="scoped_resource") +@lang_specific(elisp, "advice", "(advice-add 'fn :around ...)", hint="aspect_oriented") +``` + +### C++-Specific Idioms + +``` +@lang_specific(cpp, "template", "template", hint="generic") +@lang_specific(cpp, "raii", "std::lock_guard", hint="scoped_resource") +@lang_specific(cpp, "constexpr", "constexpr", hint="compile_time") +``` + +--- + +## Dear ImGui Integration + +### What is Dear ImGui? + +Immediate-mode GUI library. Every frame, you declare what to draw: + +```cpp +// C++ Dear ImGui +void render() { + if (ImGui::Button("Compile")) { + runCompile(); + } + ImGui::SliderInt("Opt Level", &optLevel, 0, 3); + ImGui::Text("Status: %s", status.c_str()); +} +``` + +### Elisp Wrapper (whetstone-imgui.el) + +```elisp +;; Elisp Dear ImGui bindings +(defun render () + (imgui-button "Compile" #'run-compile) + (imgui-slider-int "Opt Level" 'opt-level 0 3) + (imgui-text "Status: %s" status)) +``` + +### AST-Defined UI + +The Whetstone AST can define UI elements: + +``` +UIPanel: "Build Settings" +├── @projection: imgui +├── children: +│ ├── UIButton +│ │ ├── label: "Compile" +│ │ ├── icon: "hammer" +│ │ └── action: → BuildSystem.compile() +│ ├── UISlider +│ │ ├── label: "Optimization Level" +│ │ ├── binding: → config.optLevel +│ │ ├── min: 0 +│ │ └── max: 3 +│ └── UIText +│ ├── template: "Status: {}" +│ └── binding: → BuildSystem.status +``` + +**Same AST, multiple UI projections:** + +| Projection | Output | +|------------|--------| +| `@projection(imgui)` | Native Dear ImGui widgets | +| `@projection(emacs)` | Text-based `[Compile] Opt: [===----] 2` | +| `@projection(html)` | Web form with `