531 lines
25 KiB
Markdown
531 lines
25 KiB
Markdown
|
|
# Sprint 14 Plan: Language Batch 1 — C, WebAssembly, Common Lisp, Scheme
|
||
|
|
|
||
|
|
## Context
|
||
|
|
|
||
|
|
Universal transpilation requires universal language coverage. Sprint 14 adds four
|
||
|
|
languages chosen for strategic reasons:
|
||
|
|
|
||
|
|
- **C** — the lingua franca of legacy code. Billions of lines of C exist that need
|
||
|
|
modernization. C is also the bridge to assembly, OS kernels, and embedded systems.
|
||
|
|
Critical for the legacy ingestion pipeline (Sprint 20).
|
||
|
|
- **WebAssembly (WAT)** — the universal compilation target for the web. If Whetstone
|
||
|
|
can generate WASM text format, any Whetstone-managed project can target the browser.
|
||
|
|
Also relevant for plugin systems and sandboxed execution.
|
||
|
|
- **Common Lisp** — the most powerful macro system in any language. Representing
|
||
|
|
Lisp's homoiconic code-as-data in the AST tests whether our universal AST is truly
|
||
|
|
universal. Also covers the Lisp family for AI/research codebases.
|
||
|
|
- **Scheme** — minimal Lisp dialect, heavily used in teaching and language research.
|
||
|
|
Shares core concepts with Common Lisp but with different conventions (hygiene,
|
||
|
|
continuations, tail-call guarantees). Tests that our Lisp representation isn't
|
||
|
|
just "Common Lisp with a different name."
|
||
|
|
|
||
|
|
Each language follows the established pattern: parser (regex-based, no new tree-sitter
|
||
|
|
dependency), generator, pipeline integration, cross-language projection tests.
|
||
|
|
|
||
|
|
**After Sprint 14:** 14 language parsers and generators.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 14a: C Parser + Generator (Steps 361-366)
|
||
|
|
|
||
|
|
*C is structurally close to C++ but simpler — no classes, no templates, no namespaces
|
||
|
|
(pre-C23). The parser can share infrastructure with CppParser but must handle C-specific
|
||
|
|
constructs: raw pointers everywhere, malloc/free, function pointers, preprocessor-heavy
|
||
|
|
idioms, and typedef structs.*
|
||
|
|
|
||
|
|
### Step 361: C Parser — Functions, Structs, Enums (12 tests)
|
||
|
|
**Goal:** Parse C source into Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/CParser.h`** (~450 lines)
|
||
|
|
- `parseC(source) -> unique_ptr<Module>`
|
||
|
|
- `parseCWithDiagnostics(source) -> pair<Module, vector<ParseDiagnostic>>`
|
||
|
|
|
||
|
|
**Constructs parsed:**
|
||
|
|
- `int foo(int x, char* s) { ... }` → Function
|
||
|
|
- `struct Point { int x; int y; };` → ClassDeclaration (isStruct=true)
|
||
|
|
- `typedef struct { ... } Name;` → ClassDeclaration + TypeAlias
|
||
|
|
- `enum Color { RED, GREEN, BLUE };` → EnumDeclaration
|
||
|
|
- `enum { FLAG_A = 1, FLAG_B = 2 };` → anonymous EnumDeclaration
|
||
|
|
- Variable declarations with type: `int x = 5;`, `char* name;`
|
||
|
|
- `#include`, `#define`, `#pragma` → preprocessor nodes (from Sprint 12d)
|
||
|
|
- `static`, `extern`, `const` qualifiers on functions and variables
|
||
|
|
- Function pointer types: `void (*callback)(int)` → typed parameter
|
||
|
|
|
||
|
|
**Differences from C++ parser:**
|
||
|
|
- No `class` keyword (only `struct`)
|
||
|
|
- No `namespace`, no `template`, no `virtual`
|
||
|
|
- `struct` requires typedef for type name in older C
|
||
|
|
- Default struct access is public (same as C++)
|
||
|
|
|
||
|
|
**Modifies: `editor/src/ast/Parser.h`** — include CParser.h
|
||
|
|
**Modifies: `editor/src/Pipeline.h`** — add "c" routing
|
||
|
|
|
||
|
|
**Tests:** function parsing, struct parsing, typedef struct, enum parsing,
|
||
|
|
preprocessor directives, static/extern qualifiers, function pointers,
|
||
|
|
variable with pointer type, multiple functions in file, backward compat
|
||
|
|
(C++ tests unaffected), C-style comments, header guard pattern.
|
||
|
|
|
||
|
|
### Step 362: C Generator (12 tests)
|
||
|
|
**Goal:** Generate idiomatic C from Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/CGenerator.h`** (~400 lines)
|
||
|
|
- Extends `ProjectionGenerator` and `SemannoAnnotationImpl<CGenerator>`
|
||
|
|
- C syntax: no classes (structs only), function declarations, typedef
|
||
|
|
- Type mapping: int, float, double, char, void, size_t, pointer types
|
||
|
|
- Memory: explicit malloc/free patterns (guided by @Owner annotations)
|
||
|
|
- String handling: char* with strlen/strcpy patterns
|
||
|
|
- Array syntax: fixed-size arrays, pointer arithmetic
|
||
|
|
- `commentPrefix() -> "// "` (C99+) or `"/* "` for pre-C99
|
||
|
|
|
||
|
|
**Cross-language mapping:**
|
||
|
|
- Python class → C struct + associated functions
|
||
|
|
- Java/C++ class → C struct + function pointers (vtable pattern) for methods
|
||
|
|
- Python list → C array with size parameter
|
||
|
|
- Exceptions → error return codes + errno pattern
|
||
|
|
- Smart pointers → raw pointers with @Owner annotation to track ownership
|
||
|
|
|
||
|
|
**Modifies: `editor/src/ast/Generator.h`** — include CGenerator.h
|
||
|
|
**Modifies: `editor/src/Pipeline.h`** — add "c" generator routing
|
||
|
|
**Modifies: `editor/src/CrossLanguageProjector.h`** — C adaptations
|
||
|
|
|
||
|
|
**Tests:** function generation, struct output, typedef struct output,
|
||
|
|
enum output, preprocessor output, pointer types, malloc/free for @Owner(Manual),
|
||
|
|
cross-language: Python → C, cross-language: C++ class → C struct + functions,
|
||
|
|
Semanno annotation comments, commentPrefix, C-style string handling.
|
||
|
|
|
||
|
|
### Step 363: C Memory Annotation Mapping (12 tests)
|
||
|
|
**Goal:** C has explicit manual memory management. The annotation system should
|
||
|
|
map cleanly to C patterns and provide meaningful guidance.
|
||
|
|
|
||
|
|
**Modifies: `editor/src/MemoryStrategyInference.h`**
|
||
|
|
- C language defaults: Owner(Manual), Lifetime(Scope), Reclaim(Explicit)
|
||
|
|
- Inference rules for C:
|
||
|
|
- Function with malloc → @Owner(Manual) + @Reclaim(Explicit)
|
||
|
|
- Local variable → @Lifetime(Scope)
|
||
|
|
- Function parameter pointer → @Owner(Borrowed)
|
||
|
|
- Return pointer → @Owner(Transferred) or @Owner(Borrowed) based on convention
|
||
|
|
- Static variable → @Lifetime(Static)
|
||
|
|
|
||
|
|
**Modifies: `editor/src/AnnotationInference.h`**
|
||
|
|
- C-specific inference patterns:
|
||
|
|
- `goto` usage → @Complexity(high) + @Risk(medium)
|
||
|
|
- No bounds checking on arrays → @BoundsCheck(unchecked)
|
||
|
|
- Void pointer casts → @Risk(high) + @Ambiguity(medium)
|
||
|
|
- Header guard → @Synthetic(guard)
|
||
|
|
|
||
|
|
**Tests:** C defaults applied, malloc → Owner(Manual), local → Lifetime(Scope),
|
||
|
|
parameter pointer → Owner(Borrowed), goto detected as complexity signal,
|
||
|
|
void pointer cast flagged, static variable lifetime, header guard pattern,
|
||
|
|
inference confidence reasonable, cross-language: C Owner(Manual) →
|
||
|
|
Rust Owner(Box) projection, C Owner(Borrowed) → Rust &T.
|
||
|
|
|
||
|
|
### Step 364: C Integration + Self-Hosting Prep (8 tests)
|
||
|
|
**Goal:** Full C pipeline and cross-language validation.
|
||
|
|
|
||
|
|
**Tests:**
|
||
|
|
1. Parse C source → generate C → roundtrip produces equivalent output
|
||
|
|
2. Cross-language: C struct → Python class with __init__
|
||
|
|
3. Cross-language: Python class → C struct + constructor function
|
||
|
|
4. Cross-language: C with manual memory → Rust with ownership annotations preserved
|
||
|
|
5. Pipeline.run() with C source + target C++ → struct becomes class
|
||
|
|
6. Annotation inference on C code: malloc, arrays, function pointers all annotated
|
||
|
|
7. Parse simplified C header from Whetstone dependencies → verify AST
|
||
|
|
8. MCP tools recognize "c" language, tools/list unchanged (language is a parameter)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 14b: WebAssembly Text Format (Steps 365-369)
|
||
|
|
|
||
|
|
*WASM text format (WAT) is an S-expression syntax representing a stack-based VM.
|
||
|
|
This is structurally different from all previous languages — it's lower-level than C,
|
||
|
|
stack-based rather than expression-based, and has explicit type annotations on every
|
||
|
|
value. Representing WAT in the AST proves the AST can handle compilation targets,
|
||
|
|
not just source languages.*
|
||
|
|
|
||
|
|
### Step 365: WAT Parser (12 tests)
|
||
|
|
**Goal:** Parse WebAssembly text format into Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/WatParser.h`** (~400 lines)
|
||
|
|
- `parseWat(source) -> unique_ptr<Module>`
|
||
|
|
- `parseWatWithDiagnostics(source) -> pair<Module, vector<ParseDiagnostic>>`
|
||
|
|
|
||
|
|
**WAT constructs → AST mapping:**
|
||
|
|
- `(module ...)` → Module
|
||
|
|
- `(func $name (param $x i32) (result i32) ...)` → Function
|
||
|
|
- WAT types: i32, i64, f32, f64, funcref, externref
|
||
|
|
- Parameters have explicit types, always
|
||
|
|
- Result type is explicit
|
||
|
|
- `(local $name i32)` → Variable
|
||
|
|
- `(export "name" (func $name))` → Export annotation
|
||
|
|
- `(import "module" "name" (func ...))` → Import node
|
||
|
|
- `(global $name (mut i32) ...)` → Variable (isGlobal=true)
|
||
|
|
- `(memory 1)` → module-level memory declaration (custom annotation)
|
||
|
|
- `(table ...)` → module-level table declaration
|
||
|
|
- Instructions as statements: `i32.add`, `local.get`, `call`, `if`, `block`, `loop`
|
||
|
|
|
||
|
|
**Design decision:** WAT instructions map to existing AST nodes where possible
|
||
|
|
(BinaryOperation for i32.add, FunctionCall for call, IfStatement for if/else).
|
||
|
|
Stack semantics are represented via annotations (@Exec(stack)) rather than
|
||
|
|
trying to model the stack in the AST.
|
||
|
|
|
||
|
|
**Modifies: `editor/src/ast/Parser.h`** — include WatParser.h
|
||
|
|
**Modifies: `editor/src/Pipeline.h`** — add "wat" / "wasm" routing
|
||
|
|
|
||
|
|
**Tests:** module parsing, function with params and result, local variables,
|
||
|
|
export declarations, import declarations, global variables, i32.add → BinaryOperation,
|
||
|
|
call → FunctionCall, if/else → IfStatement, nested blocks, empty module,
|
||
|
|
WAT-specific types preserved in annotations.
|
||
|
|
|
||
|
|
### Step 366: WAT Generator (12 tests)
|
||
|
|
**Goal:** Generate valid WAT from Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/WatGenerator.h`** (~400 lines)
|
||
|
|
- Extends `ProjectionGenerator` and `SemannoAnnotationImpl<WatGenerator>`
|
||
|
|
- S-expression output format
|
||
|
|
- Type mapping to WASM types: int→i32, float→f32, double→f64, bool→i32
|
||
|
|
- Function output: `(func $name (param $x i32) (result i32) ...)`
|
||
|
|
- Expressions → stack instructions: `a + b` → `local.get $a local.get $b i32.add`
|
||
|
|
- Control flow: if/else → `(if (result i32) ... (then ...) (else ...))`
|
||
|
|
- Memory operations guided by annotations
|
||
|
|
- `commentPrefix() -> ";; "`
|
||
|
|
|
||
|
|
**Cross-language mapping:**
|
||
|
|
- Any language → WAT: flatten expressions to stack operations
|
||
|
|
- Classes → function tables + struct-in-memory layout
|
||
|
|
- Strings → memory offset + length (WAT has no native strings)
|
||
|
|
- Exceptions → trap or error code return
|
||
|
|
|
||
|
|
**Modifies: `editor/src/ast/Generator.h`** — include WatGenerator.h
|
||
|
|
**Modifies: `editor/src/Pipeline.h`** — add "wat" generator routing
|
||
|
|
|
||
|
|
**Tests:** function generation, S-expression formatting, type mapping,
|
||
|
|
arithmetic → stack instructions, if/else → WAT if, function call → call,
|
||
|
|
export declaration, import declaration, Semanno comments with ;; prefix,
|
||
|
|
cross-language: Python function → WAT function, round-trip parse → generate,
|
||
|
|
valid WAT structure (parentheses balanced).
|
||
|
|
|
||
|
|
### Step 367: WAT Annotation Mapping (12 tests)
|
||
|
|
**Goal:** Meaningful annotations for WASM's unique characteristics.
|
||
|
|
|
||
|
|
**Annotation mappings for WAT:**
|
||
|
|
- All functions → @Exec(stack) (stack-based execution model)
|
||
|
|
- Memory operations → @Owner(Manual) (WASM has linear memory, no GC by default)
|
||
|
|
- Exported functions → @Visibility(public)
|
||
|
|
- Imported functions → @Link(import, module="name")
|
||
|
|
- Table-based dispatch → @Shim(vtable)
|
||
|
|
- Memory size declarations → @Align + @Layout annotations
|
||
|
|
|
||
|
|
**Modifies: `editor/src/MemoryStrategyInference.h`** — WAT defaults
|
||
|
|
**Modifies: `editor/src/AnnotationInference.h`** — WAT-specific patterns
|
||
|
|
|
||
|
|
**Tests:** WAT function gets @Exec(stack), memory access gets @Owner(Manual),
|
||
|
|
export → @Visibility(public), import → @Link, defaults applied correctly,
|
||
|
|
inference confidence, cross-language: WAT annotations preserved when
|
||
|
|
projecting to C, annotation roundtrip through Semanno format.
|
||
|
|
|
||
|
|
### Step 368: WAT Cross-Language Projection (12 tests)
|
||
|
|
**Goal:** Validate that WAT works as both source and target in cross-language
|
||
|
|
projection scenarios.
|
||
|
|
|
||
|
|
**Tests:**
|
||
|
|
1. Python function → WAT: arithmetic expression flattened to stack ops
|
||
|
|
2. C function → WAT: direct mapping (both low-level, closest match)
|
||
|
|
3. WAT → Python: stack ops reconstructed to expression tree
|
||
|
|
4. WAT → C: near-direct mapping
|
||
|
|
5. WAT → Rust: memory annotations guide ownership
|
||
|
|
6. Java class → WAT: method table + memory layout
|
||
|
|
7. Round-trip: Python → WAT → Python produces equivalent logic
|
||
|
|
8. Memory annotations preserved across WAT ↔ C projection
|
||
|
|
9. WAT with imports → any language: imports become function declarations
|
||
|
|
10. Complex: multi-function WAT module → C output compiles conceptually
|
||
|
|
11. Semanno roundtrip on WAT source
|
||
|
|
12. Pipeline.run() with WAT source succeeds
|
||
|
|
|
||
|
|
### Step 369: Phase 14b Integration (8 tests)
|
||
|
|
**Goal:** WAT fully integrated into the transpilation pipeline.
|
||
|
|
|
||
|
|
**Tests:**
|
||
|
|
1. Parse WAT → generate WAT → output is valid WAT
|
||
|
|
2. Cross-language: C → WAT → C roundtrip preserves logic
|
||
|
|
3. 12 languages all project to WAT as target
|
||
|
|
4. WAT annotations meaningful and complete
|
||
|
|
5. Compact AST for WAT module has correct node names
|
||
|
|
6. Diagnostics work on WAT (parse errors detected)
|
||
|
|
7. Semanno sidecar save/load for WAT files
|
||
|
|
8. MCP tools handle "wat" language parameter
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 14c: Common Lisp Parser + Generator (Steps 370-373)
|
||
|
|
|
||
|
|
*Common Lisp is homoiconic: code is data, data is code. The S-expression syntax means
|
||
|
|
the AST representation is almost 1:1 with the source. This is both simpler (no complex
|
||
|
|
parsing) and harder (macros, multiple return values, condition system, CLOS). This phase
|
||
|
|
tests whether our universal AST handles a fundamentally different programming paradigm.*
|
||
|
|
|
||
|
|
### Step 370: Common Lisp Parser (12 tests)
|
||
|
|
**Goal:** Parse Common Lisp source into Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/CommonLispParser.h`** (~400 lines)
|
||
|
|
- `parseCommonLisp(source) -> unique_ptr<Module>`
|
||
|
|
- `parseCommonLispWithDiagnostics(source) -> pair<Module, vector<ParseDiagnostic>>`
|
||
|
|
|
||
|
|
**CL constructs → AST mapping:**
|
||
|
|
- `(defun name (params) body)` → Function
|
||
|
|
- `(defmethod name ((x class)) body)` → MethodDeclaration
|
||
|
|
- `(defclass name (supers) (slots))` → ClassDeclaration (CLOS)
|
||
|
|
- `(defvar *name* value)` / `(defparameter *name* value)` → Variable (isGlobal=true)
|
||
|
|
- `(let ((x 1) (y 2)) body)` → Variable declarations + scope
|
||
|
|
- `(lambda (params) body)` → LambdaExpression
|
||
|
|
- `(if test then else)` → IfStatement
|
||
|
|
- `(cond (test1 result1) ...)` → chain of IfStatements
|
||
|
|
- `(loop ...)` / `(do ...)` / `(dotimes ...)` → ForStatement variants
|
||
|
|
- `(funcall fn args)` / `(apply fn args)` → FunctionCall
|
||
|
|
- `(defmacro name (params) body)` → MacroDefinition (from Sprint 12d)
|
||
|
|
- `(quote expr)` / `'expr` → annotated expression (@Meta(quoted))
|
||
|
|
- `(values x y z)` → multi-value return (annotated with @Contract)
|
||
|
|
|
||
|
|
**Conventions handled:**
|
||
|
|
- `*earmuffs*` for dynamic/special variables
|
||
|
|
- `+constants+` for constants
|
||
|
|
- `predicate-p` naming convention
|
||
|
|
- `(declare (type fixnum x))` → type annotations
|
||
|
|
- Multiple package prefixes: `cl:defun`, `my-package:foo`
|
||
|
|
|
||
|
|
**Tests:** defun parsing, defclass parsing, lambda, let bindings, if/cond,
|
||
|
|
loop variants, defmacro, quote/quasiquote, earmuff detection, nested
|
||
|
|
S-expressions (deep nesting), mixed definitions in one file, CLOS method.
|
||
|
|
|
||
|
|
### Step 371: Common Lisp Generator (12 tests)
|
||
|
|
**Goal:** Generate idiomatic Common Lisp from Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/CommonLispGenerator.h`** (~400 lines)
|
||
|
|
- Extends `ProjectionGenerator` and `SemannoAnnotationImpl<CommonLispGenerator>`
|
||
|
|
- S-expression output: proper indentation, one form per line
|
||
|
|
- Type mapping: fixnum, single-float, double-float, string, t, nil, list, vector
|
||
|
|
- Memory: garbage collected (CL has GC), @Owner defaults to Shared_GC
|
||
|
|
- Convention adherence: earmuffs for specials, `-p` for predicates
|
||
|
|
- `commentPrefix() -> ";; "`
|
||
|
|
|
||
|
|
**Cross-language mapping:**
|
||
|
|
- Python class → CLOS defclass with defmethod
|
||
|
|
- C++ class → CLOS class + condition system for exceptions
|
||
|
|
- Java interface → CLOS generic function protocol
|
||
|
|
- Python list → CL list
|
||
|
|
- Static typing → (declare ...) type annotations
|
||
|
|
- Exceptions → CL condition/restart system
|
||
|
|
|
||
|
|
**Tests:** defun generation, defclass output, lambda output, let bindings,
|
||
|
|
S-expression formatting (correct indentation), type declarations from
|
||
|
|
annotations, cross-language: Python → Common Lisp, cross-language: Java →
|
||
|
|
Common Lisp, Semanno comments with ;; prefix, round-trip parse → generate,
|
||
|
|
CLOS method dispatch, condition system from exception annotations.
|
||
|
|
|
||
|
|
### Step 372: Lisp Annotation Mapping (12 tests)
|
||
|
|
**Goal:** Map Lisp's unique features to the annotation taxonomy.
|
||
|
|
|
||
|
|
**CL-specific annotations:**
|
||
|
|
- Macros → @Meta(quoted) + @Synthetic(macro)
|
||
|
|
- Dynamic variables → @Binding(dynamic) (vs @Binding(static) for lexical)
|
||
|
|
- Multiple return values → @Contract(returns=["x", "y"])
|
||
|
|
- Tail-recursive functions → @TailCall (CL doesn't guarantee TCO but hints are useful)
|
||
|
|
- CLOS methods → @Visibility + method combination annotations
|
||
|
|
- Special declarations → @Complexity(declared-type)
|
||
|
|
- `(optimize (speed 3) (safety 0))` → @Policy(perf=critical, style=literal)
|
||
|
|
|
||
|
|
**Modifies: `editor/src/MemoryStrategyInference.h`** — CL defaults (GC)
|
||
|
|
**Modifies: `editor/src/AnnotationInference.h`** — CL-specific patterns
|
||
|
|
|
||
|
|
**Tests:** macro → @Meta(quoted), dynamic var → @Binding(dynamic), multiple
|
||
|
|
values → @Contract, tail recursion detected, CLOS class → @Visibility,
|
||
|
|
optimize declaration → @Policy mapping, GC defaults, inference confidence,
|
||
|
|
cross-language: CL @Binding(dynamic) → Python (no direct equivalent, comment),
|
||
|
|
CL @Binding(dynamic) → C++ (thread_local hint).
|
||
|
|
|
||
|
|
### Step 373: Common Lisp Integration (8 tests)
|
||
|
|
**Goal:** Full CL pipeline and cross-language validation.
|
||
|
|
|
||
|
|
**Tests:**
|
||
|
|
1. Parse CL → generate CL → roundtrip equivalent
|
||
|
|
2. Cross-language: Python → Common Lisp (class → CLOS)
|
||
|
|
3. Cross-language: Common Lisp → Python (defclass → class)
|
||
|
|
4. Cross-language: Common Lisp → Java (CLOS → class hierarchy)
|
||
|
|
5. Annotation inference on CL code: macros, dynamic vars, tail calls
|
||
|
|
6. Semanno roundtrip on CL source
|
||
|
|
7. Pipeline.run() with CL source succeeds
|
||
|
|
8. Compact AST for Lisp module has meaningful node names
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Phase 14d: Scheme Parser + Generator (Steps 374-377)
|
||
|
|
|
||
|
|
*Scheme shares Lisp syntax but has different conventions: hygiene, first-class
|
||
|
|
continuations, guaranteed tail calls, and a minimalist standard library. The AST
|
||
|
|
must distinguish Scheme from Common Lisp despite surface similarity.*
|
||
|
|
|
||
|
|
### Step 374: Scheme Parser (12 tests)
|
||
|
|
**Goal:** Parse Scheme source into Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/SchemeParser.h`** (~350 lines)
|
||
|
|
- `parseScheme(source) -> unique_ptr<Module>`
|
||
|
|
- `parseSchemeWithDiagnostics(source) -> pair<Module, vector<ParseDiagnostic>>`
|
||
|
|
|
||
|
|
**Scheme constructs → AST mapping:**
|
||
|
|
- `(define (name params) body)` → Function
|
||
|
|
- `(define name value)` → Variable
|
||
|
|
- `(lambda (params) body)` → LambdaExpression
|
||
|
|
- `(let ((x 1)) body)` / `(let* ...)` / `(letrec ...)` → scoped Variables
|
||
|
|
- `(if test then else)` → IfStatement
|
||
|
|
- `(cond ...)` → chained IfStatements
|
||
|
|
- `(do ((var init step) ...) (test result) body)` → ForStatement
|
||
|
|
- `(call-with-current-continuation proc)` / `(call/cc proc)` → FunctionCall
|
||
|
|
with @Exec(continuation) annotation
|
||
|
|
- `(define-syntax name ...)` → MacroDefinition with @Meta(hygienic)
|
||
|
|
- `(syntax-rules ...)` → macro body
|
||
|
|
- `(values x y)` → multi-value return
|
||
|
|
- `(begin ...)` → statement sequence
|
||
|
|
|
||
|
|
**Differences from Common Lisp parser:**
|
||
|
|
- `define` instead of `defun`/`defvar`
|
||
|
|
- No CLOS (use records or closures for objects)
|
||
|
|
- Hygienic macros (define-syntax) instead of defmacro
|
||
|
|
- Guaranteed tail call optimization
|
||
|
|
- `#t`/`#f` instead of `t`/`nil` for booleans
|
||
|
|
- `'()` for empty list instead of `nil`
|
||
|
|
|
||
|
|
**Tests:** define function, define variable, lambda, let/let*/letrec,
|
||
|
|
if/cond, call/cc recognized, define-syntax, #t/#f booleans, nested
|
||
|
|
S-expressions, mixed definitions, tail-call pattern, Scheme-specific
|
||
|
|
comment `;` handling.
|
||
|
|
|
||
|
|
### Step 375: Scheme Generator (12 tests)
|
||
|
|
**Goal:** Generate idiomatic Scheme from Whetstone AST.
|
||
|
|
|
||
|
|
**New file: `editor/src/ast/SchemeGenerator.h`** (~350 lines)
|
||
|
|
- Extends `ProjectionGenerator` and `SemannoAnnotationImpl<SchemeGenerator>`
|
||
|
|
- S-expression output with Scheme conventions
|
||
|
|
- Type mapping: minimal (Scheme is dynamically typed) — number, string, boolean,
|
||
|
|
pair, list, vector, procedure
|
||
|
|
- Booleans: #t / #f
|
||
|
|
- Empty list: '()
|
||
|
|
- `commentPrefix() -> "; "`
|
||
|
|
|
||
|
|
**Cross-language mapping:**
|
||
|
|
- Python class → Scheme record type + procedures
|
||
|
|
- C struct → Scheme vector with accessor procedures
|
||
|
|
- Java interface → Scheme protocol (procedure signatures)
|
||
|
|
- Exceptions → Scheme `guard` / `with-exception-handler`
|
||
|
|
- Tail calls preserved where source had @TailCall
|
||
|
|
- Continuations: unique to Scheme, generated from @Exec(continuation) annotations
|
||
|
|
|
||
|
|
**Tests:** define generation, lambda output, let bindings, boolean #t/#f,
|
||
|
|
S-expression formatting, cross-language: Python → Scheme, cross-language:
|
||
|
|
Common Lisp → Scheme (closest mapping), cross-language: Scheme → Python,
|
||
|
|
Semanno comments with ; prefix, round-trip, tail-call preservation,
|
||
|
|
continuation annotations.
|
||
|
|
|
||
|
|
### Step 376: Scheme-Specific Annotations + CL Differentiation (12 tests)
|
||
|
|
**Goal:** Ensure Scheme and Common Lisp are properly differentiated in
|
||
|
|
annotations and cross-language projection.
|
||
|
|
|
||
|
|
**Scheme-specific annotations:**
|
||
|
|
- All functions checked for tail position → @TailCall where applicable
|
||
|
|
(Scheme guarantees TCO, so @TailCall is a correctness annotation, not just a hint)
|
||
|
|
- call/cc → @Exec(continuation)
|
||
|
|
- define-syntax → @Meta(hygienic) (vs CL's @Meta(quoted) for unhygienic macros)
|
||
|
|
- Proper tail recursion → @Loop(tail-recursive) (Scheme idiom for iteration)
|
||
|
|
|
||
|
|
**CL vs Scheme differentiation:**
|
||
|
|
- CL `defmacro` → @Meta(quoted), Scheme `define-syntax` → @Meta(hygienic)
|
||
|
|
- CL `*specials*` → @Binding(dynamic), Scheme uses `make-parameter` → @Binding(parameter)
|
||
|
|
- CL CLOS → ClassDeclaration, Scheme records → ClassDeclaration (isStruct=true)
|
||
|
|
- CL condition system → @Exception(condition), Scheme guard → @Exception(guard)
|
||
|
|
|
||
|
|
**Modifies: `editor/src/MemoryStrategyInference.h`** — Scheme defaults (GC)
|
||
|
|
**Modifies: `editor/src/AnnotationInference.h`** — Scheme-specific patterns
|
||
|
|
|
||
|
|
**Tests:** tail call detected in Scheme (guarantee vs hint), call/cc annotation,
|
||
|
|
hygienic macro annotation, CL→Scheme projection preserves semantics but changes
|
||
|
|
idiom, Scheme→CL projection, both languages GC defaults, differentiated
|
||
|
|
@Meta types, @Binding types differ, @Exception types differ, cross-language
|
||
|
|
annotation fidelity.
|
||
|
|
|
||
|
|
### Step 377: Phase 14d Integration + Sprint 14 Summary (8 tests)
|
||
|
|
**Goal:** All 4 new languages fully integrated.
|
||
|
|
|
||
|
|
**Tests:**
|
||
|
|
1. All 14 languages parse and generate without error
|
||
|
|
2. Cross-language matrix: C ↔ WAT ↔ Common Lisp ↔ Scheme all project correctly
|
||
|
|
3. Lisp → WAT: S-expression source → stack-based target
|
||
|
|
4. C → Common Lisp: manual memory → GC with annotation translation
|
||
|
|
5. Semanno format works for all 4 new languages with correct comment prefixes
|
||
|
|
6. Annotation inference produces language-appropriate defaults for each new language
|
||
|
|
7. Pipeline.run() succeeds for all 4 new languages
|
||
|
|
8. Sprint 14 totals: 14 parsers, 14 generators, all cross-language projections valid
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Step & Test Summary
|
||
|
|
|
||
|
|
| Phase | Steps | Tests | Theme |
|
||
|
|
|-------|-------|-------|-------|
|
||
|
|
| 14a | 361-364 | 44 | C parser, generator, memory mapping, integration |
|
||
|
|
| 14b | 365-369 | 56 | WAT parser, generator, annotations, cross-language, integration |
|
||
|
|
| 14c | 370-373 | 44 | Common Lisp parser, generator, annotations, integration |
|
||
|
|
| 14d | 374-377 | 44 | Scheme parser, generator, differentiation, integration |
|
||
|
|
| **Total** | **361-377** | **~188** | 17 steps |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Key Files
|
||
|
|
|
||
|
|
**New files:**
|
||
|
|
- `editor/src/ast/CParser.h` — C parser (regex-based)
|
||
|
|
- `editor/src/ast/CGenerator.h` — C code generator
|
||
|
|
- `editor/src/ast/WatParser.h` — WebAssembly text format parser
|
||
|
|
- `editor/src/ast/WatGenerator.h` — WAT code generator
|
||
|
|
- `editor/src/ast/CommonLispParser.h` — Common Lisp parser
|
||
|
|
- `editor/src/ast/CommonLispGenerator.h` — Common Lisp code generator
|
||
|
|
- `editor/src/ast/SchemeParser.h` — Scheme parser
|
||
|
|
- `editor/src/ast/SchemeGenerator.h` — Scheme code generator
|
||
|
|
- `editor/tests/step361_test.cpp` through `editor/tests/step377_test.cpp`
|
||
|
|
|
||
|
|
**Modified files:**
|
||
|
|
- `editor/src/ast/Parser.h` — include 4 new parsers
|
||
|
|
- `editor/src/ast/Generator.h` — include 4 new generators
|
||
|
|
- `editor/src/Pipeline.h` — routing for "c", "wat", "common-lisp", "scheme"
|
||
|
|
- `editor/src/CrossLanguageProjector.h` — adaptations for all 4 languages
|
||
|
|
- `editor/src/MemoryStrategyInference.h` — defaults for C (Manual), WAT (Manual), CL (GC), Scheme (GC)
|
||
|
|
- `editor/src/AnnotationInference.h` — language-specific inference patterns
|
||
|
|
- `editor/CMakeLists.txt` — test targets
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Language Design Notes
|
||
|
|
|
||
|
|
### Why regex-based parsers (no tree-sitter)?
|
||
|
|
Tree-sitter grammars exist for C and Scheme, but:
|
||
|
|
1. The architecture invariant says no new external dependencies beyond tree-sitter
|
||
|
|
2. The existing tree-sitter integration handles Python/C++/Java/Rust/Go/JS/TS
|
||
|
|
3. Adding tree-sitter grammars for 4 more languages bloats the build
|
||
|
|
4. Regex-based parsers for Kotlin/C# (Sprint 11d) proved the pattern works
|
||
|
|
5. For transpilation purposes, we need structural (not syntactic) fidelity —
|
||
|
|
regex parsers capture the structural skeleton, which is what matters for
|
||
|
|
the AST
|
||
|
|
|
||
|
|
### Lisp parser approach
|
||
|
|
Both Lisp parsers are S-expression readers at core:
|
||
|
|
1. Tokenize: atoms, strings, numbers, parentheses, quotes
|
||
|
|
2. Build S-expression tree
|
||
|
|
3. Walk tree, pattern-match on car of each form: `defun`→Function, `define`→Function, etc.
|
||
|
|
4. Unrecognized forms become generic FunctionCall nodes
|
||
|
|
|
||
|
|
This is simpler than regex parsing of C-like languages because S-expressions
|
||
|
|
have uniform syntax. The complexity is in semantic mapping, not parsing.
|
||
|
|
|
||
|
|
### WAT as compilation target
|
||
|
|
WAT is unique among our supported languages: it's primarily a target, not a source.
|
||
|
|
Most workflows will be "source language → WAT" rather than "WAT → source language."
|
||
|
|
The WAT→source direction is decompilation, which we support but with lower fidelity
|
||
|
|
(stack reconstruction to expression trees is lossy for complex code).
|