whetstone_DSL/docs/REQUIREMENTS_OVERVIEW.md

# Whetstone DSL: Project Requirements Overview

## Vision Statement

**Source code should be a structured database of logic, not a text file of syntax.**

Whetstone is a Universal Logic Database that stores code as a semantic graph (AST), with traditional programming languages serving as **projections** of that canonical representation. This enables lossless transpilation, massively parallel development (human and AI agents), and complete separation of intent from implementation.

---

## Core Philosophy

### The Problem: Text Blindness
Current software engineering couples **Intent** (what we want to happen) with **Implementation** (how the machine does it) through language-specific syntax. This creates:
- Lock-in to specific languages and ecosystems
- Loss of semantic information during translation
- Inability to express cross-cutting concerns (ownership, risk, optimization hints)
- Barriers to parallel development by multiple agents

### The Solution: Projectional Editing
Move the "Source of Truth" from text files to a **Logical Graph Database**:

| Text-Based | Graph-Based (Whetstone) |
|------------|-------------------------|
| Linear | Structural |
| Ambiguous | Explicit |
| Syntax-heavy | Meaning-heavy |
| Language-specific | Universal |

---

## Architecture: The Three Layers

### Layer 1: The Logic Database (Definition)
A schema of pure logical concepts stored in a graph/tree database (JetBrains MPS).

**Core Principle:** This layer must be a **superset** of all target languages. It must express concepts like memory ownership (Rust-style) even when targeting languages that don't require them (Python).

**Contents:**
- AST node concepts (expressions, statements, functions, types)
- Semantic annotations (SemAnno schema)
- Hypergraph relationships (nodes participate in multiple relationships)

### Layer 2: Semantic Generators (Bridge)
Generators that translate the Logic Database into idiomatic text for specific runtimes.

**Example - SharedMemoryBlock concept:**
```
Generator A (Python):   x = [1, 2, 3]  # Note: Shared usage implied
Generator B (Rust):     let x = Arc::new(Mutex::new(vec![1, 2, 3]));
Generator C (Verilog):  reg [31:0] memory [0:2];
```

### Layer 3: Runtime (Metal)
Standard toolchains (GCC, LLVM, Python interpreter) compile/execute the generated code. Whetstone does not replace compilers; it feeds them optimized, correct input.

---

## SemAnno: Universal Semantic Annotation Schema

Six annotation systems provide metadata for agents (human and AI):

### 1. Complexity & Intelligence ("Rank" System)
Defines who/what is qualified to modify a node.

| Annotation | Type | Purpose |
|------------|------|---------|
| `@complexity-score` | int | Derived from cyclomatic complexity, nesting, mutation |
| `@intelligence-requirement` | junior/mid/senior | Agent tier required for modification |
| `@review-lock` | bool | Requires second agent verification |

### 2. Dependency & Impact ("Radius" System)
Calculates the "blast radius" of changes.

| Annotation | Type | Purpose |
|------------|------|---------|
| `@dependency-count` | int | Incoming references to this node |
| `@dependency-level` | 0-100 | Normalized importance score |
| `@downstream-impact` | list | Affected domains [Network, Security, Billing] |
| `@fragility` | low/high | Load-bearing legacy code flag |

### 3. Performance & Resource ("Machinist" System)
Low-level optimization metadata. See `annotations/6 optimization and intent.md` for full details.

| Annotation | Type | Purpose |
|------------|------|---------|
| `@Hot` / `@Cold` | hint | Profile-guided branch prediction → C++ `[[likely]]`/`[[unlikely]]` |
| `@Inline(Always\|Never\|Hint)` | mode | Function inlining control |
| `@Pure` | flag | No side effects (referential transparency) |
| `@Loop(Unroll,N)` / `@Loop(Vectorize)` / `@Loop(Fuse)` | transform | Loop transformation hints |
| `@Data(Prefetch)` / `@Data(Restrict)` | hint | Memory locality / alias analysis |
| `@Align(N)` / `@Pack` / `@ConstExpr` | layout | Hardware alignment, packing, compile-time evaluation |
| `@Policy(Perf: Critical)` | guardrail | Prioritizes zero-cost abstractions over safety wrappers |

### 4. Architectural Intent ("Paradigm" System)
Defines behavioral contracts.

| Annotation | Type | Purpose |
|------------|------|---------|
| `@intent` | Pure/Stateful/Parallel/Legacy | Primary behavioral contract |
| `@preferred-view` | language | Optimal projection for readability |
| `@lifecycle` | owner/borrower/transient | Ownership model hints |

### 5. Security & Risk ("Sentinel" System)
Granular risk tracking.

| Annotation | Type | Purpose |
|------------|------|---------|
| `@risk-type` | identifier | BufferOverflow, RaceCondition, SideChannel |
| `@trust-boundary` | marker | Secure environment transition point |
| `@provenance` | hash | Chain of agent modifications |

### 6. Projective Metadata ("Lens" System)
Controls rendering in projectional environments.

| Annotation | Type | Purpose |
|------------|------|---------|
| `@view-mask` | category | Toggle annotation visibility by type |
| `@cognitive-level` | 0-100 | Detail density filter |
| `@projection-alias` | string | Localized renaming without AST change |
| `@hidden-detail` | bool | Collapse boilerplate for reviews |

### 7. Language-Specific Idioms ("Polyglot" System)
Preserves language-specific features that don't map directly to other languages.

| Annotation | Type | Purpose |
|------------|------|---------|
| `@lang_specific` | structured | Captures idiom with language, type, raw syntax, and semantic hint |

**Architecture:**
```
@lang_specific(python, "decorator", "@lru_cache(maxsize=128)", hint="memoization")
@lang_specific(cpp, "template", "template<typename T>", hint="generic")
```

**Projection Behavior:**
- In native language: renders as original syntax
- In foreign language: renders as comment with semantic hint

**Long-term Vision:** These annotations enable future semantic equivalence detection. When the system understands that `@lru_cache` and a C++ `std::map`-based cache serve the same purpose ("memoization"), it can generate idiomatic implementations in either direction rather than just commenting.

---

## Memory Management Annotations

Memory management is decoupled from logic as a first-class AST field. The canonical annotation system (defined in `annotations/Memory strategy.md`) uses distinct annotation families, each targeting a specific memory paradigm:

| Annotation | Paradigm | Description | Key Languages |
|---|---|---|---|
| `@Deallocate(Explicit)` | Manual (MM) | Explicit allocation/deallocation, direct `free()`/`delete` | C, C++ (manual mode), Zig |
| `@Lifetime(RAII)` | RAII | Destructor-based cleanup at scope end | C++ |
| `@Reclaim(Tracing)` | GC | Automatic runtime reclamation via tracing/mark-sweep | Python, Java, JS, Go, C#, Ruby |
| `@Reclaim(Cycle)` | GC | Cycle-detection garbage collection | PHP |
| `@Reclaim(Escape)` | GC | Escape-analysis-based heap vs stack decision | Go |
| `@Owner(Single)` | Ownership | Compile-time single-owner lifetime tracking (Rust-like) | Rust |
| `@Owner(Shared_ARC)` | ARC | Deterministic reference counting | Swift, Objective-C |
| `@Allocate(Static)` | Static | Fixed memory buffers, no dynamic allocation | Fortran |
| `@Allocate(Register)` | Register | Direct register mapping or stack spill | Assembly |
| `@Allocate(Allocator)` | Allocator | Explicit allocator parameter threading | Zig |

### The "Unfilled Node Constraint"
In `@Deallocate(Explicit)` mode, the AST requires an explicit **Deallocation Time Field**. If empty, the editor flags it as a "Missing Intent" error. This makes it impossible to forget a deallocation point.

### Lossless Projection Logic
When the transpiler encounters a memory annotation that the target language cannot express natively (see `annotations/Memory strategy.md` Section 3):
- **High-Level → Low-Level** (e.g., Python → C): `@Reclaim(Tracing)` → inject `INC_REF`/`DEC_REF` shim with `free()` on zero
- **Low-Level → High-Level** (e.g., C → Java): `@Deallocate(Explicit)` → wrap in `try-with-resources` or `Cleaner` API
- **Strict → Permissive** (e.g., Rust → Python): `@Owner(Single)` → preserve in AST metadata, no enforcement in generated code

---

## Strategy Toggles

Declarative optimization patterns applicable without manual implementation:

| Toggle | Effect |
|--------|--------|
| `@strategy(TCO)` | Convert recursive logic to iterative machine code |
| `@strategy(Memoize)` | Inject thread-safe cache for function results |
| `@strategy(Unroll)` | Duplicate loop bodies per target-hardware heuristic |
| `@strategy(SSO)` | Enable Small String/Buffer Optimization |

---

## Transpilation Modes

### Direct Transpilation ("Wrapper")
Maps nodes 1-to-1. High compatibility, poor performance.
- Python dynamic list → `std::vector<std::any>` with runtime type-checking

### Idiomatic Transpilation ("Intent")
Analyzes usage patterns. High performance, hardware sympathy.
- Python list (append-only, sequential read) → `std::vector<T>` or Memory Arena

---

## The Resolution Slider

Four levels from intent to execution:

| Level | Description | Example |
|-------|-------------|---------|
| 1 | Pseudocode/Intent | "Find a user in a list" |
| 2 | High-Level (JS/Python) | `Array.find()` |
| 3 | Intermediate (Rust/Java) | Type safety + ownership scopes |
| 4 | Execution Substrate (C++/C) | B-Trees, SIMD, Branch Hints |

---

## Multi-Agent Collaboration

### The Round-Trip Problem
When a systems engineer optimizes code that a high-level developer later modifies.

### Solution: Persistent Metadata Separation
- **Logic Field:** Algorithmic intent (editable by junior)
- **Optimization Field:** Systems-level instructions (requires senior)

### Conflict Management
When high-level changes break low-level optimizations:
1. **Shadowed Metadata:** Old optimizations hidden but not deleted
2. **Alerting:** "Performance Guardrail" warning shown
3. **Diff Lens:** Visualizes where intent changes invalidated hardware instructions

---

## Hypergraph Requirement

Standard ASTs are strictly hierarchical (Parent → Child). Real-world logic is a **Hypergraph**:

- A Variable is **owned by** Function A
- But **used by** Function B
- And **monitored by** Agent C

**Implementation:** Smart References (Hyperedges) allow a single logical node to participate in multiple relationships simultaneously.

---

## Agent-Native Development

### Task Decomposition Model
1. **High-context coordinator agent** builds semantic graph and queues tasks
2. **Small Language Models (8B)** handle tightly scoped tasks
3. **Human developers** work from the same task queue

### Projection Optimization
Tasks can be projected in:
- **Human-optimized view:** Natural language descriptions, readable code
- **AI-optimized view:** Structured data, explicit constraints, minimal ambiguity

### Agent Tiers
| Tier | Capabilities |
|------|--------------|
| Junior | Basic logic, no memory management, no high-risk annotations |
| Mid | Standard logic, `@Lifetime(RAII)`, `@Reclaim(Tracing)`, `@Owner(Shared_ARC)` |
| Senior | `@Deallocate(Explicit)`, `@Owner(Single)`, hardware optimization, high-risk refactoring |

---

## Target Applications

### Legacy Modernization
Parse old Fortran/COBOL → Logic Database → Generate modern C++/Rust/Julia

### Cross-Platform Logic
Write business logic once → Generate Kotlin (Android), Swift (iOS), Rust (backend)

### Agentic Coding
Agents manipulate Graph Nodes directly (no syntax errors from text generation)

### Parallel Development
Multiple developers/agents work on same codebase with explicit conflict boundaries

---

## Technical Principles

### Behavior Over Implementation
A node's primary definition is its **Behavioral Contract** (e.g., "Sort this collection") rather than implementation (e.g., "Quicksort with pointer increments").

### Substrate-Agnostic Core Nodes
Core AST nodes must not assume specific memory models or hardware constraints.
- **Bad:** LoopNode requires a "Pointer" field
- **Good:** LoopNode requires an "IteratorIntent" - realization is metadata

### The Semantic Boundary
Strict boundary between **What** (Logic) and **How** (Execution Strategy). Future capabilities are implemented by adding Strategy Engines to existing nodes, not re-architecting nodes.

### Strictest Common Denominator
The Logic Database enforces strict rules by default (explicit ownership), while Generators can "relax" rules for permissive languages or "poly-fill" rules for strict languages.

---

## The C Primitives Substrate (Long-term Vision)

All high-level language features ultimately resolve to C-level primitives:

| High-Level Feature | C Primitive Resolution |
|-------------------|------------------------|
| Python `@lru_cache` | Hash table + function pointer + malloc/free |
| C++ `template<T>` | N compiled copies with concrete types |
| Python decorator | Function pointer manipulation |
| C++ RAII | Constructor/destructor at scope boundaries |
| Rust ownership | Compile-time tracking → equivalent to manual C |
| Garbage collection | Reference counting or mark-sweep algorithms |

**Architectural Implication:**

```
Level 4: Language Idioms     @py_decl, @cpp_tmpl, @rust_lifetime
            │
            ▼
Level 3: Universal AST       Function, Loop, Variable, Expression
            │
            ▼
Level 2: Semantic Intent     "memoize", "parameterize by type", "scope-bound cleanup"
            │
            ▼
Level 1: C Primitives        pointers, structs, malloc/free, function calls
```

**End State:** Languages become pure *syntactic projections* over C primitives + semantic annotations. This enables true lossless transpilation because the canonical form captures both the intent (semantic annotations) and the implementation (C primitives).

---

## Implementation Platform

**JetBrains MPS (Meta Programming System)**
- Projectional editor (edit AST directly, not text)
- Language workbench for defining DSLs
- Built-in support for generators
- Extensible type system and constraints

---

## Success Criteria

1. **Lossless Round-Trip:** Python → AST → C++ → AST → Python produces semantically equivalent code
2. **Parallel Safety:** Junior developer cannot accidentally break senior's optimizations
3. **Agent Compatibility:** Tasks can be completed by SLMs with sufficient annotation context
4. **Performance Parity:** Generated C++ matches hand-written C++ for common patterns
5. **Adoption Path:** Existing codebases can be incrementally imported via tree-sitter parsing