Add polyglot orchestrator feature requests, sprint plan, and test projects
Feature requests I-N cover: LanguageFitnessScorer, PolyglotFFIGlueGenerator, CrossLanguageSymbolIndex, LSP Orchestrator, DAP Orchestrator, Polyglot Test Harness. Sprint plan maps 75 steps across sprints 271-285 in 5 phases. Test projects: poly-sort (Rust+Python), poly-api (Go+TS), poly-parse (C++/Haskell), poly-pipeline (5 lang), poly-compiler (5 lang), poly-everything (all languages). Theoretical goal: lossless polyglot transpiling proven as a construction. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
267
docs/polyglot_orchestrator_sprint_plan.md
Normal file
267
docs/polyglot_orchestrator_sprint_plan.md
Normal file
@@ -0,0 +1,267 @@
|
||||
# Polyglot Orchestrator — Sprint Plan
|
||||
> Created: 2026-02-28
|
||||
> Follows: Sprint 270 (all 26 generator readiness gaps closed)
|
||||
> Goal: True lossless polyglot code generation with unified editing and debugging
|
||||
|
||||
---
|
||||
|
||||
## What We Are Building
|
||||
|
||||
A system where:
|
||||
1. A single spec drives code generation into multiple languages simultaneously
|
||||
2. Each language gets the component it is best suited for (fitness-scored)
|
||||
3. FFI boundaries between languages are generated, not written by hand
|
||||
4. A single editor session understands all languages via an LSP orchestrator
|
||||
5. A single debugger session can step across language boundaries via a DAP orchestrator
|
||||
6. A test harness proves behavioral parity across all language variants
|
||||
|
||||
The theoretical endpoint: a project with components in every supported Whetstone
|
||||
generation language, each component calling the next, with no hand-written glue,
|
||||
unified editing, and unified debugging. This proves lossless polyglot transpiling
|
||||
as a construction, not a claim.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Language Fitness Scorer (Sprint 271-272)
|
||||
**Steps 1883-1892**
|
||||
|
||||
The ability to analyze an AST subtree and recommend a ranked list of target languages
|
||||
based on computational shape. Prerequisite for everything else — you can't route
|
||||
generation to the right language without knowing what "right" means.
|
||||
|
||||
### Sprint 271: LanguageFitnessScorer core (Steps 1883-1887)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1883 | `ASTFeatureExtractor` | Extract scalar features from AST subtrees: mutation ratio, recursion shape, concurrency primitives, I/O pattern, type complexity |
|
||||
| 1884 | `LanguageIdiomProfile` | Static profiles for each supported language: ideal feature vector per language |
|
||||
| 1885 | `LanguageFitnessScorer` | Score AST features against profiles, return ranked list with rationale |
|
||||
| 1886 | `whetstone_score_language_fitness` | MCP tool wiring |
|
||||
| 1887 | Sprint 271 integration | Cross-component test: spec section → features → scores → ranked list |
|
||||
|
||||
### Sprint 272: Fitness-routed 2-language test projects (Steps 1888-1892)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1888 | `PolyglotProjectSpec` | Spec format extension: per-section language hints + fitness override |
|
||||
| 1889 | Test project: `poly-sort` | Rust (sort core) + Python (data gen/validation) — fitness-routed |
|
||||
| 1890 | Test project: `poly-api` | Go (HTTP server) + TypeScript (client) — fitness-routed |
|
||||
| 1891 | Test project: `poly-parse` | C++ (lexer/parser) + Haskell (AST transformation) — fitness-routed |
|
||||
| 1892 | Sprint 272 integration | All 3 test projects: generate → compile → run → output matches expected |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — FFI Glue + Symbol Index (Sprint 273-275)
|
||||
**Steps 1893-1907**
|
||||
|
||||
Generate the boundaries between languages from the shared AST. This is what makes
|
||||
the boundaries type-safe and eliminates hand-written glue code.
|
||||
|
||||
### Sprint 273: PolyglotFFIGlueGenerator (Steps 1893-1897)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1893 | `ABIBoundaryExtractor` | Identify AST nodes that cross language boundaries (exported functions, shared structs) |
|
||||
| 1894 | `CHeaderEmitter` | Emit C ABI header as neutral intermediary for any language pair |
|
||||
| 1895 | `RustPythonBindingEmitter` | Rust `extern "C"` + Python ctypes binding from same AST node |
|
||||
| 1896 | `GoCppBindingEmitter` | CGo + C++ extern from same AST node |
|
||||
| 1897 | Sprint 273 integration | Rust↔Python round-trip: Python calls Rust via generated binding |
|
||||
|
||||
### Sprint 274: More binding pairs + DWARF annotations (Steps 1898-1902)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1898 | `CppJSBindingEmitter` | C++ N-API binding from AST node |
|
||||
| 1899 | `RustGoBindingEmitter` | Rust FFI + CGo binding from AST node |
|
||||
| 1900 | `DWARFBoundaryAnnotator` | Annotate generated DWARF with cross-language AST node IDs (needed for DAP orchestrator) |
|
||||
| 1901 | `whetstone_generate_ffi_glue` | MCP tool wiring |
|
||||
| 1902 | Sprint 274 integration | Go↔C++ round-trip: Go calls C++ via generated binding |
|
||||
|
||||
### Sprint 275: Cross-Language Symbol Index (Steps 1903-1907)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1903 | `SCIPEmitter` | Emit SCIP-format symbol index from shared AST after multi-language generation |
|
||||
| 1904 | `CrossLanguageSymbolTable` | In-memory symbol table linking same-origin symbols across all generated languages |
|
||||
| 1905 | `SymbolIndexUpdater` | Incremental update on re-generation (don't rebuild from scratch on every change) |
|
||||
| 1906 | `whetstone_emit_symbol_index` | MCP tool wiring |
|
||||
| 1907 | Sprint 275 integration | poly-sort: generate Rust+Python → emit index → verify cross-language symbol links |
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — LSP Orchestration (Sprint 276-278)
|
||||
**Steps 1908-1922**
|
||||
|
||||
An LSP proxy that routes editor queries through the symbol index. The editor
|
||||
sees one language server; the orchestrator fans out to the right per-language
|
||||
server or resolves via the symbol index for cross-language queries.
|
||||
|
||||
### Sprint 276: LSP proxy core (Steps 1908-1912)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1908 | `LSPProxyServer` | LSP protocol handler that forwards to per-language servers |
|
||||
| 1909 | `LanguageServerRouter` | Route by file extension / detected language |
|
||||
| 1910 | `CrossLanguageBoundaryDetector` | Identify request targets that are cross-language boundary symbols |
|
||||
| 1911 | `CrossLanguageDefinitionResolver` | Resolve goto-definition across language seams via symbol index |
|
||||
| 1912 | Sprint 276 integration | poly-sort: goto-definition from Python call site lands on Rust source |
|
||||
|
||||
### Sprint 277: Hover + references + rename (Steps 1913-1917)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1913 | `CrossLanguageHoverProvider` | Hover at boundary shows both-side types + C ABI type |
|
||||
| 1914 | `CrossLanguageReferencesProvider` | Find all references across all language targets |
|
||||
| 1915 | `CrossLanguageRenameOrchestrator` | Rename propagates to all LSPs + updates symbol index |
|
||||
| 1916 | `LSPProxyConfig` | Config file: which languages, which LSP servers, which port |
|
||||
| 1917 | Sprint 277 integration | poly-api: rename shared type in Go+TS simultaneously |
|
||||
|
||||
### Sprint 278: LSP proxy hardening (Steps 1918-1922)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1918 | `LSPRequestCache` | Cache intra-language responses to reduce round-trip latency |
|
||||
| 1919 | `LSPServerHealthMonitor` | Detect crashed language servers, route to degraded mode |
|
||||
| 1920 | `LSPProxyDiagnosticAggregator` | Merge per-language diagnostics into unified list (no duplicates at boundaries) |
|
||||
| 1921 | LSP proxy test suite | End-to-end: proxy running, 3 language servers, cross-language queries correct |
|
||||
| 1922 | Sprint 278 integration | All 3 Phase 1 test projects: LSP proxy running, cross-language navigation verified |
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — DAP Orchestration (Sprint 279-281)
|
||||
**Steps 1923-1937**
|
||||
|
||||
A DAP proxy that presents a unified debugger session across all language runtimes.
|
||||
Stack traces are annotated, stepping across language boundaries works, breakpoints
|
||||
can be set on AST nodes (firing in any language target).
|
||||
|
||||
### Sprint 279: DAP proxy core + stack annotation (Steps 1923-1927)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1923 | `DAPProxyServer` | DAP protocol handler forwarding to per-language debug adapters |
|
||||
| 1924 | `DebugAdapterRouter` | Route by active frame language (detected from DWARF annotations) |
|
||||
| 1925 | `PolyglotStackTraceDecoder` | Annotate mixed-language stack with origin language + AST node ID per frame |
|
||||
| 1926 | `LanguageSeamFrameDetector` | Identify frames that sit at a language boundary (from DWARF annotations) |
|
||||
| 1927 | Sprint 279 integration | poly-sort: unified stack trace shows Rust + Python frames labeled correctly |
|
||||
|
||||
### Sprint 280: Cross-language step + breakpoints (Steps 1928-1932)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1928 | `CrossLanguageStepOverHandler` | Step-over at a boundary call executes foreign call atomically, pauses in caller |
|
||||
| 1929 | `CrossLanguageStepIntoHandler` | Step-into at a boundary call switches active debug adapter to foreign language |
|
||||
| 1930 | `ASTNodeBreakpointMapper` | Set breakpoint by AST node ID, fires in all language targets that implement that node |
|
||||
| 1931 | `BreakpointSyncOrchestrator` | Keep breakpoints in sync across all active debug adapters |
|
||||
| 1932 | Sprint 280 integration | poly-api: breakpoint on Go handler fires; step-into crosses into TS call |
|
||||
|
||||
### Sprint 281: DAP hardening + multi-adapter coordination (Steps 1933-1937)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1933 | `DAPSessionLifecycleManager` | Launch/attach/detach per-language adapters as the debug session progresses |
|
||||
| 1934 | `UnifiedVariableInspector` | Inspect variables across the boundary (translate types to editor representation) |
|
||||
| 1935 | `DAPHealthMonitor` | Handle adapter crashes, fall back to single-language mode gracefully |
|
||||
| 1936 | DAP proxy test suite | Step-into, step-over, breakpoint, variable inspect — all cross-language |
|
||||
| 1937 | Sprint 281 integration | poly-parse: C++↔Haskell debug session — step across AST transformation boundary |
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — The Proof (Sprint 282-285)
|
||||
**Steps 1938-1957**
|
||||
|
||||
The full polyglot test harness and the all-language test projects. This is the
|
||||
demonstration that the system is real — not a claim but a construction.
|
||||
|
||||
### Sprint 282: Polyglot test harness (Steps 1938-1942)
|
||||
|
||||
| Step | Component | What |
|
||||
|------|-----------|------|
|
||||
| 1938 | `PolyglotSpecRunner` | Generate N language variants from one spec, compile and run all |
|
||||
| 1939 | `BehavioralParityChecker` | Compare outputs across variants for same inputs, flag divergence |
|
||||
| 1940 | `ParityRegressionGuard` | Block new language additions that break existing parity |
|
||||
| 1941 | `whetstone_run_polyglot_suite` | MCP tool wiring |
|
||||
| 1942 | Sprint 282 integration | poly-sort: Python+Rust+Go+Haskell — all produce identical output for 1000 inputs |
|
||||
|
||||
### Sprint 283: Wacky project 1 — poly-pipeline (Steps 1943-1947)
|
||||
|
||||
A data pipeline where each stage is a different language:
|
||||
- Python (CSV ingestion + schema validation)
|
||||
- Rust (data normalization + outlier detection, perf-critical)
|
||||
- Go (fan-out routing to multiple sinks, concurrency)
|
||||
- Haskell (statistical transformation, pure function)
|
||||
- JavaScript/TypeScript (JSON serialization + HTTP push to consumer)
|
||||
|
||||
| Step | What |
|
||||
|------|------|
|
||||
| 1943 | Spec: poly-pipeline — all 5 components, interfaces defined in shared AST |
|
||||
| 1944 | Generate all 5 language targets + FFI glue at each boundary |
|
||||
| 1945 | Integration: end-to-end CSV in → HTTP push out, all 5 stages running |
|
||||
| 1946 | LSP + DAP: unified editing and debugging across all 5 languages |
|
||||
| 1947 | Parity check: replace each stage with a reference Python implementation, verify same output |
|
||||
|
||||
### Sprint 284: Wacky project 2 — poly-compiler (Steps 1948-1952)
|
||||
|
||||
A toy compiler where each phase is the language most suited to it:
|
||||
- C++ (lexer — character-level performance, no GC pressure)
|
||||
- Haskell (parser — recursive descent, algebraic data types)
|
||||
- Rust (type checker — borrow-safe, zero-cost ownership tracking)
|
||||
- Lisp/Scheme (optimizer — s-expression AST manipulation, homoiconicity)
|
||||
- Go (code emitter — goroutine-parallel emission, simple concurrency)
|
||||
|
||||
| Step | What |
|
||||
|------|------|
|
||||
| 1948 | Spec: poly-compiler — 5 phases, shared IR type defined in AST |
|
||||
| 1949 | Generate all 5 language targets + FFI glue |
|
||||
| 1950 | Integration: source string in → bytecode out, all 5 phases connected |
|
||||
| 1951 | LSP + DAP: cross-language navigation and debugging in the compiler itself |
|
||||
| 1952 | Parity check: each phase replaceable with Python reference implementation |
|
||||
|
||||
### Sprint 285: Wacky project 3 — poly-everything (Steps 1953-1957)
|
||||
|
||||
One project, every supported generation language. Each language owns one function.
|
||||
The chain: Language A calls B, B calls C, ... last language returns to A.
|
||||
Fitness scorer assigns each language to the function most suited to it.
|
||||
All FFI glue generated. All boundaries in the symbol index. Full LSP + DAP.
|
||||
|
||||
This is the theoretical proof: a single shared AST produced working code in every
|
||||
supported language, connected at runtime, navigable in a single editor session,
|
||||
debuggable in a single debugger session.
|
||||
|
||||
| Step | What |
|
||||
|------|------|
|
||||
| 1953 | Spec: poly-everything — one function per language, fitness-assigned |
|
||||
| 1954 | Generate all N language targets + all FFI glue in one pipeline run |
|
||||
| 1955 | Integration: full chain executes, output correct |
|
||||
| 1956 | LSP + DAP: navigate and debug across all N language boundaries |
|
||||
| 1957 | Parity report: all N language functions individually replaceable, parity confirmed |
|
||||
|
||||
---
|
||||
|
||||
## Success Definition
|
||||
|
||||
The sprint plan is complete when:
|
||||
|
||||
1. A spec with no target language hint produces a fitness-ranked recommendation
|
||||
2. A 2-language project (any pair) generates with correct FFI glue — no hand-written bindings
|
||||
3. Goto-definition from language A lands on the definition in language B
|
||||
4. A single debugger session steps across a language A → language B call
|
||||
5. poly-pipeline (5 languages) passes end-to-end with all stages running
|
||||
6. poly-compiler (5 languages) compiles and runs a "hello world" source string
|
||||
7. poly-everything (all languages) executes the full chain and parity is confirmed
|
||||
|
||||
At step 7, lossless polyglot transpiling is proven as a construction.
|
||||
|
||||
---
|
||||
|
||||
## Notes on Scope
|
||||
|
||||
- LSP and DAP orchestration layers are editor integrations, not MCP tools.
|
||||
They run as separate processes alongside the whetstone_mcp binary.
|
||||
- The SCIP symbol index is the shared data structure. It must be written to
|
||||
a known path on generation and read by both the LSP and DAP orchestrators.
|
||||
- Phase 5 test projects are not production software. They are proofs of concept
|
||||
designed to stress the boundaries of the orchestration system.
|
||||
- The practical value is not the test projects themselves but the integration
|
||||
layer capability they prove: any project that needs FFI between two environments
|
||||
can use Whetstone to generate the boundary from a shared spec.
|
||||
290
docs/polyglot_test_projects.md
Normal file
290
docs/polyglot_test_projects.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Polyglot Test Projects
|
||||
> Created: 2026-02-28
|
||||
> Purpose: Validate the polyglot orchestrator at each phase. Projects are ordered
|
||||
> by complexity — 2-language pairs first, then multi-language, then all-language.
|
||||
> None of these are production software. They are boundary stress tests.
|
||||
|
||||
---
|
||||
|
||||
## Tier 1: 2-Language Projects
|
||||
|
||||
These prove the basic pipeline: fitness scoring, FFI glue generation, symbol
|
||||
index, and cross-language LSP. They use language pairs with well-understood
|
||||
interop stories so failures are clearly attributable to the orchestrator.
|
||||
|
||||
---
|
||||
|
||||
### poly-sort
|
||||
|
||||
**Languages:** Rust + Python
|
||||
**Pattern:** Performance core in Rust, scripting/validation layer in Python
|
||||
|
||||
```
|
||||
Python (test runner)
|
||||
│ generates random arrays, validates output
|
||||
│ calls via ctypes (generated from shared AST)
|
||||
▼
|
||||
Rust (sort implementations)
|
||||
│ merge sort, quick sort, radix sort
|
||||
│ exposed as C ABI functions
|
||||
└─ returns sorted array to Python
|
||||
```
|
||||
|
||||
**Fitness rationale:**
|
||||
- Rust: flat array mutation, cache-coherent loops, zero GC pressure → systems fitness
|
||||
- Python: test orchestration, random generation, human-readable assertions → scripting fitness
|
||||
|
||||
**What it tests:**
|
||||
- `LanguageFitnessScorer` recommends Rust for the sort functions, Python for the harness
|
||||
- `RustPythonBindingEmitter` generates ctypes bindings for `sort_merge`, `sort_quick`, `sort_radix`
|
||||
- Symbol index links Python `sort_merge` call site to Rust `sort_merge` definition
|
||||
- Goto-definition from Python lands on Rust
|
||||
- Debugger can step from Python test into Rust sort function
|
||||
|
||||
**Acceptance test:**
|
||||
- 1000 random arrays of integers: all three sort variants produce output matching Python's `sorted()`
|
||||
- No hand-written FFI code in the project
|
||||
|
||||
---
|
||||
|
||||
### poly-api
|
||||
|
||||
**Languages:** Go + TypeScript
|
||||
**Pattern:** API server in Go, typed client in TypeScript
|
||||
|
||||
```
|
||||
TypeScript (browser/Node client)
|
||||
│ typed API client generated from shared AST
|
||||
│ calls HTTP endpoints
|
||||
▼
|
||||
Go (HTTP server)
|
||||
│ router, handlers, response serialization
|
||||
│ types defined in shared AST
|
||||
└─ serves JSON responses
|
||||
```
|
||||
|
||||
**Fitness rationale:**
|
||||
- Go: goroutine-based HTTP handling, simple concurrency, excellent standard library → concurrency fitness
|
||||
- TypeScript: strong types for API contracts, browser-compatible, React/Node ecosystem → typed scripting fitness
|
||||
|
||||
**What it tests:**
|
||||
- Shared type definitions (request/response structs) generated in both Go and TypeScript from same AST nodes
|
||||
- No OpenAPI spec written by hand — the shared AST is the source of truth
|
||||
- Symbol index links TypeScript `UserResponse` interface to Go `UserResponse` struct
|
||||
- Rename of a field in shared AST propagates to both Go and TypeScript simultaneously
|
||||
|
||||
**Acceptance test:**
|
||||
- TypeScript client calls Go server, receives typed response
|
||||
- Field rename via LSP orchestrator compiles cleanly in both languages
|
||||
- No hand-written type definitions or API client code
|
||||
|
||||
---
|
||||
|
||||
### poly-parse
|
||||
|
||||
**Languages:** C++ + Haskell
|
||||
**Pattern:** High-performance lexer/tokenizer in C++, algebraic parser in Haskell
|
||||
|
||||
```
|
||||
Haskell (parser)
|
||||
│ recursive descent, pattern matching on token ADT
|
||||
│ calls C++ tokenizer via FFI
|
||||
▼
|
||||
C++ (lexer)
|
||||
│ character-level scanning, zero allocation
|
||||
│ exposes token stream as C ABI iterator
|
||||
└─ returns tokens to Haskell
|
||||
```
|
||||
|
||||
**Fitness rationale:**
|
||||
- C++: character-level loops, cache-coherent buffers, zero allocation → systems fitness
|
||||
- Haskell: recursive ADT pattern matching, algebraic data types → functional fitness
|
||||
|
||||
**What it tests:**
|
||||
- C++ → Haskell FFI (unusual direction; most polyglot FFI is C calling higher-level)
|
||||
- `CHeaderEmitter` for the token stream iterator
|
||||
- `GoCppBindingEmitter` is not used here — this exercises a new binding pair (C++↔Haskell via C ABI)
|
||||
- Debugger: step from Haskell parser into C++ lexer and back
|
||||
|
||||
**Acceptance test:**
|
||||
- Parse a sample grammar (JSON or a simple expression language)
|
||||
- Output AST from Haskell matches reference Python parser for the same input
|
||||
- No hand-written FFI
|
||||
|
||||
---
|
||||
|
||||
## Tier 2: Multi-Language Projects
|
||||
|
||||
These stress the orchestrator beyond two-language pairs. Each adds languages
|
||||
that stress different aspects of the system (more binding pairs, more LSP
|
||||
servers, larger symbol index, more complex debug adapter coordination).
|
||||
|
||||
---
|
||||
|
||||
### poly-pipeline
|
||||
|
||||
**Languages:** Python + Rust + Go + Haskell + TypeScript (5 languages)
|
||||
**Pattern:** Data pipeline where each stage is the best language for that computation
|
||||
|
||||
```
|
||||
Python CSV ingestion, schema validation, error reporting
|
||||
↓ FFI (ctypes)
|
||||
Rust Data normalization, outlier detection, statistics
|
||||
↓ FFI (CGo)
|
||||
Go Fan-out routing to multiple sinks, concurrent dispatch
|
||||
↓ FFI (C ABI)
|
||||
Haskell Statistical transformation, pure functional reduction
|
||||
↓ FFI (N-API)
|
||||
TypeScript JSON serialization, HTTP push to downstream consumer
|
||||
```
|
||||
|
||||
**Fitness rationale:**
|
||||
- Python: schema parsing, human-readable error messages, ecosystem (pandas for validation) → scripting
|
||||
- Rust: numerical normalization on large arrays, no GC pauses at this stage → systems
|
||||
- Go: goroutines for fan-out, select for concurrent dispatch → concurrency
|
||||
- Haskell: pure reduction functions (fold, map), referential transparency at this stage → functional
|
||||
- TypeScript: JSON is native, HTTP client ecosystem mature, consumer-facing → typed scripting
|
||||
|
||||
**What it tests:**
|
||||
- 4 FFI boundaries generated from shared AST (Python↔Rust, Rust↔Go, Go↔Haskell, Haskell↔TS)
|
||||
- Symbol index with 5 language targets — navigation across 4 boundaries
|
||||
- LSP orchestrator running 5 language servers simultaneously
|
||||
- DAP orchestrator stepping through 5-language call chain
|
||||
- Parity: replace each stage with Python reference implementation, output must match
|
||||
|
||||
**Acceptance test:**
|
||||
- CSV file in → HTTP POST to mock consumer
|
||||
- All 5 stages running as one connected process (not microservices)
|
||||
- End-to-end wall time within 2x of a pure Python reference implementation
|
||||
|
||||
---
|
||||
|
||||
### poly-compiler
|
||||
|
||||
**Languages:** C++ + Haskell + Rust + Lisp (Scheme) + Go (5 languages)
|
||||
**Pattern:** A toy compiler where each phase is the language most fit for it
|
||||
|
||||
```
|
||||
C++ Lexer — character-level scanning, no allocations per token
|
||||
↓ C ABI
|
||||
Haskell Parser — recursive descent, algebraic token → AST types
|
||||
↓ C ABI
|
||||
Rust Type checker — ownership-tracking symbol table, zero-cost
|
||||
↓ C ABI
|
||||
Lisp/Scheme Optimizer — s-expression AST rewriting, pattern-matched rules
|
||||
↓ C ABI
|
||||
Go Code emitter — parallel emission of bytecode, concurrent writes
|
||||
```
|
||||
|
||||
**Fitness rationale:**
|
||||
- C++: character-level tight loops, arena allocation for token buffer → systems
|
||||
- Haskell: token ADTs, recursive descent naturally expressed as algebraic functions → functional
|
||||
- Rust: symbol table with lifetime-safe references, borrow checker prevents dangling refs → systems/safety
|
||||
- Lisp: the optimizer manipulates an AST that is itself S-expressions — homoiconicity → meta-programming
|
||||
- Go: parallel bytecode emission with goroutines, simple coordination → concurrency
|
||||
|
||||
**What it tests:**
|
||||
- The most philosophically interesting FFI direction: Lisp optimizing Rust's output then handing to Go
|
||||
- The Lisp binding pair (Scheme↔C ABI) is a new surface for the glue generator
|
||||
- poly-compiler is itself written in the languages it compiles — not intentionally self-hosting,
|
||||
but the structural similarity is notable
|
||||
|
||||
**Acceptance test:**
|
||||
- Compiles the following source string: `(let x 42) (+ x 1)` → bytecode `[PUSH 42, STORE x, LOAD x, PUSH 1, ADD]`
|
||||
- Each phase independently replaceable with Python reference implementation (parity check)
|
||||
|
||||
---
|
||||
|
||||
## Tier 3: The Proof
|
||||
|
||||
---
|
||||
|
||||
### poly-everything
|
||||
|
||||
**Languages:** All supported Whetstone generation languages
|
||||
**Pattern:** One function per language, fitness-assigned. Each calls the next.
|
||||
|
||||
The fitness scorer assigns each language to the function most suited to it.
|
||||
The chain is determined by the scorer, not specified manually.
|
||||
All FFI glue generated. All boundaries in the symbol index.
|
||||
Full LSP + DAP.
|
||||
|
||||
**Current supported generation languages in Whetstone:**
|
||||
Python, C++, Rust, Go, JavaScript/TypeScript, Java, Haskell, Elixir, Lisp/Scheme,
|
||||
Julia, PHP, Rust (embedded), WASM (via Rust), Kotlin, Swift, Zig
|
||||
|
||||
**Suggested chain** (fitness scorer should reproduce this independently):
|
||||
```
|
||||
Zig Memory-mapped file reader (ultra-low-level, no runtime)
|
||||
↓
|
||||
C++ Binary deserialization (struct parsing, no GC)
|
||||
↓
|
||||
Rust Validation and normalization (safety-critical, owned types)
|
||||
↓
|
||||
Haskell Schema transformation (pure functional, algebraic types)
|
||||
↓
|
||||
Lisp/Scheme Rule application (S-expression pattern matching)
|
||||
↓
|
||||
Julia Statistical analysis (array operations, broadcasting)
|
||||
↓
|
||||
Python Result formatting (readable output, ecosystem)
|
||||
↓
|
||||
Go HTTP dispatch (concurrency, fan-out)
|
||||
↓
|
||||
TypeScript JSON response (typed client contract)
|
||||
↓
|
||||
Java Persistent storage (JDBC, enterprise ecosystem)
|
||||
↓
|
||||
Kotlin Android notification (Kotlin coroutines, Android SDK)
|
||||
↓
|
||||
Swift iOS notification (Swift async/await, Apple SDK)
|
||||
↓
|
||||
Elixir Real-time broadcast (actor model, Phoenix channels)
|
||||
↓
|
||||
PHP CMS integration (WordPress hook, legacy ecosystem)
|
||||
↓
|
||||
WASM Browser computation (sandboxed, portable)
|
||||
└─ returns to Zig (closes the chain)
|
||||
```
|
||||
|
||||
**What it proves:**
|
||||
- The shared AST can represent computations across the full spectrum of
|
||||
language paradigms: systems, functional, OO, scripting, actor, array, meta
|
||||
- FFI glue can be generated for every adjacent pair in the chain
|
||||
- The symbol index spans all language targets
|
||||
- The LSP orchestrator can route queries across N language boundaries
|
||||
- The DAP orchestrator can present a coherent stack trace spanning all languages
|
||||
- Parity: every function replaceable by a Python reference implementation
|
||||
|
||||
**Acceptance test:**
|
||||
- Binary file in → push notification + browser computation + CMS update + database write
|
||||
- No hand-written FFI, no hand-written bindings, no hand-written type definitions
|
||||
- The only human artifact is the spec
|
||||
|
||||
**Parity acceptance test:**
|
||||
- Replace each language stage with a Python reference implementation
|
||||
- Output must be identical for all 100 test inputs
|
||||
- This is the formal proof of lossless transpiling
|
||||
|
||||
---
|
||||
|
||||
## What This Actually Means
|
||||
|
||||
If poly-everything passes, we have demonstrated:
|
||||
|
||||
1. **For integration layers**: Any environment boundary (Python↔C++, Go↔Java, Rust↔WASM)
|
||||
can be generated from a shared spec. This was previously only possible for major projects
|
||||
with dedicated platform integration teams (GraalVM, JPype, pybind11, etc.).
|
||||
|
||||
2. **For legacy modernization**: An existing system in language A can have a spec written
|
||||
from it, and a reimplementation in language B generated with correct type-safe boundaries.
|
||||
The language A and language B versions run in parallel and their outputs are compared
|
||||
automatically to verify the migration.
|
||||
|
||||
3. **For new projects**: Language selection stops being a permanent architectural decision.
|
||||
A component can be written in the fitness-optimal language and later regenerated in a
|
||||
different language if the fitness profile changes (e.g. scaling requirements shift
|
||||
from scripting-friendly to systems-performance).
|
||||
|
||||
The practical value is the integration layer capability. The theoretical value is
|
||||
the proof that language is a generation parameter, not an architectural commitment.
|
||||
Reference in New Issue
Block a user