bill/whetstone_DSL
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add polyglot orchestrator feature requests, sprint plan, and test projects

Browse Source 
Feature requests I-N cover: LanguageFitnessScorer, PolyglotFFIGlueGenerator,
CrossLanguageSymbolIndex, LSP Orchestrator, DAP Orchestrator, Polyglot Test Harness.
Sprint plan maps 75 steps across sprints 271-285 in 5 phases.
Test projects: poly-sort (Rust+Python), poly-api (Go+TS), poly-parse (C++/Haskell),
poly-pipeline (5 lang), poly-compiler (5 lang), poly-everything (all languages).
Theoretical goal: lossless polyglot transpiling proven as a construction.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
2026-02-28 23:01:40 -07:00
parent de5bdd358f
commit d1e883dd38
3 changed files with 922 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

365
FEATURE_REQUESTS.md
View File

@@ -2,6 +2,22 @@
> Backlog of feature ideas to triage into future sprints (e.g., Sprint 6/7). > Backlog of feature ideas to triage into future sprints (e.g., Sprint 6/7).
## Title Convention (Semantic Logging)
Feature request titles must include derivation source so provenance is explicit.
Required title prefix format:
- `[Derived:<source>] <feature title>`
Examples:
- `[Derived:DataPipeline-LoRA] Semantic Validation Gate APIs for Taskitem Promotion`
- `[Derived:UserFeedback-HiveMind] Resource Lock Constraints on TaskItems`
- `[Derived:RunLogs-NativeTools] MCP Tool-Call Completion Diagnostics`
Source tags should map to where the signal came from (run logs, user request, external project, postmortem, etc.), not just the proposed implementation area.
## Security Vulnerability Awareness (Dependencies) — IMPLEMENTED (Sprint 6, Steps 190–195) ## Security Vulnerability Awareness (Dependencies) — IMPLEMENTED (Sprint 6, Steps 190–195)
**Goal:** Warn when a dependency has known vulnerabilities and surface safer alternatives. **Goal:** Warn when a dependency has known vulnerabilities and surface safer alternatives.
@@ -141,6 +157,127 @@
- Provide recommended defaults for CLI tools vs GUI apps - Provide recommended defaults for CLI tools vs GUI apps
- Include guidance for minimizing startup latency and bundle size - Include guidance for minimizing startup latency and bundle size
## [Derived:PersonalVaultTool-MultiPlatform] Platform-Agnostic Projection via SemAnno Adapter Boundaries — PROPOSED
**Filed:** 2026-02-27
**Filed by:** personal_vault_tool session (2026-02-27)
**Priority:** High — required for any real-world cross-platform projection (Python → Android/Kotlin, Python → Swift/iOS, etc.)
**Source context:** `personal_vault_tool/vault_core.py` — clean business logic and platform I/O are entangled in the same class with no annotation boundary.
### Problem
Whetstone's projector today treats a source file as a unit. When projecting `VaultStore`
to Kotlin, it faithfully projects `sqlite3.connect()` — which doesn't exist on Android.
The projector has no way to distinguish:
- **Pure logic nodes** — `share_evaluate()` selector scoring, audit hash chain computation,
grant TTL enforcement — zero platform dependency, projectable to any target.
- **Platform adapter nodes** — `sqlite3.connect()`, `Fernet.encrypt()`, `socket.bind()` —
completely platform-specific, must be substituted per-target, not projected literally.
This is not a generator quality issue (GR-022/023/024). It is an architectural gap:
the AST has no annotation that marks where the platform boundary is.
### The Correct Architecture (already implied by SemAnno's design goals)
```
Source file with SemAnno annotations
│
├── @pure.logic nodes → project as-is to any language
│ share_evaluate()
│ audit_hash_chain()
│ grant_scope_check()
│
└── @platform.storage → generate abstract interface + per-target impl
│ _db() / read_records() / write_record() → Python: sqlite3
│ → Android: Room DAO
│ → iOS: CoreData
│
├── @platform.crypto → generate abstract interface + per-target impl
│ FernetEncryptor → Python: cryptography.Fernet
│ → Android: Android Keystore AES
│
└── @platform.network → omit on mobile (caller drives transport)
ThreadingHTTPServer
```
The platform adapters become **generated abstract interfaces** with per-target
implementations, not projected concrete calls.
### Feature Request A: SemAnno Platform Boundary Annotations
**New annotation tags:**
- `@pure.logic` — no I/O, no platform imports; freely projectable
- `@platform.storage` — interacts with a persistence layer; requires adapter
- `@platform.crypto` — interacts with a key/encryption service; requires adapter
- `@platform.network` — interacts with sockets/HTTP; usually omitted on mobile
- `@platform.os` — interacts with filesystem, env vars, signals
**Generator behavior:** annotated nodes are flagged before projection; the projector
emits an `AdapterInterface` node instead of projecting the implementation.
**Validation gate:** `whetstone_validate_taskitem` or a new `whetstone_check_platform_purity`
tool should verify that `@pure.logic` nodes have zero `@platform.*` imports in their
transitive dependency graph.
### Feature Request B: Platform Profile System
**New concept:** a named platform profile describes what substitutions exist for each
`@platform.*` category.
```json
{
"profile": "android-kotlin",
"substitutions": {
"platform.storage": "androidx.room.RoomDatabase",
"platform.crypto": "android.security.keystore.KeyGenParameterSpec",
"platform.network": null
}
}
```
Built-in profiles to start:
- `python-stdlib` (current, implicit)
- `android-kotlin` (Room + Android Keystore)
- `swift-ios` (CoreData + CryptoKit)
- `node-js` (better-sqlite3 + node:crypto)
**Generator behavior:** when projecting with a profile active, `@platform.*` nodes
are replaced by the profile's adapter interface stub rather than the source
implementation. The developer fills the stub; the logic layer is complete.
### Feature Request C: Multi-Target Projection Command
**Tool:** extend `whetstone_run_pipeline` or add `whetstone_project_multiplatform`
**Input:** annotated source file + list of target profiles
**Output:** one generated artifact per target, plus a shared pure-logic module
that all targets import unchanged.
**Acceptance criteria:**
- `personal_vault_tool/vault_core.py` (fully annotated) projects to:
- `out/python/vault_logic.py` + `out/python/adapters/sqlite_adapter.py`
- `out/kotlin/VaultLogic.kt` + `out/kotlin/adapters/RoomAdapter.kt` (stub)
- All `@pure.logic` methods are byte-identical in content between targets (modulo syntax).
- `@platform.*` methods emit only the interface signature + a `TODO` body in each target.
### Relationship to existing gaps
| Gap | Relationship |
|-----|-------------|
| GR-006 | Complex class generation — same root; multi-method class projection needed for VaultStore |
| GR-007 | Cross-language class node emission — must be closed before this works |
| GR-022/023/024 | Python generator string/import/comment bugs — must fix before Python→Python round-trip is clean enough to annotate |
### Suggested sprint scope
- Sprint N: SemAnno tags `@pure.logic`, `@platform.storage`, `@platform.crypto` (schema + parser + editor UI)
- Sprint N+1: Platform profile schema + substitution registry (2 built-in profiles: python-stdlib, android-kotlin)
- Sprint N+2: Projection gate — `whetstone_check_platform_purity` tool
- Sprint N+3: Multi-target pipeline output with stub generation
---
## Python/C++/JS-TS Support Enhancements <20> PROPOSED ## Python/C++/JS-TS Support Enhancements <20> PROPOSED
**Goal:** Expand first-class support for Python, C++, and JS/TS beyond current parse/generate. **Goal:** Expand first-class support for Python, C++, and JS/TS beyond current parse/generate.
@@ -218,6 +355,23 @@ HiveMind's scheduler knows the dispatch target node and enforces accordingly.
## Deterministic MCP Debugging Workflow for SLM Agents — PROPOSED ## Deterministic MCP Debugging Workflow for SLM Agents — PROPOSED
## [Derived:DataPipeline-LoRA] Semantic + Recursive Taskitem Validation APIs — PROPOSED
**Goal:** Expose first-class MCP/editor APIs to score task completion beyond test-pass and feed recursive quality signals back into taskitem generation.
**Why derived:** Current dataset curation needs semantic validity checks and recursive re-validation loops to avoid local minima in training data.
**Scope:**
- Semantic completion validator API (invariant checks, intent/diff consistency, equivalence hooks).
- Recursive taskitem quality scorer (self-containment, dependency correctness, blocker detectability).
- Promotion-grade result packet (`execution_pass`, `semantic_pass`, `recursive_pass`, confidence).
- MCP endpoints for batch scoring and audit export.
**Data-pipeline impact:**
- Reduces script-only heuristics.
- Enables consistent promotion gates across projects.
- Improves good/bad labeling quality for LoRA datasets.
**Filed by:** Whetstone execution session (2026-02-21) **Filed by:** Whetstone execution session (2026-02-21)
**Priority:** Critical for self-improving low-context agents **Priority:** Critical for self-improving low-context agents
**Goal:** Make generation + debugging robust enough that small/local models can complete tasks without expert intuition. **Goal:** Make generation + debugging robust enough that small/local models can complete tasks without expert intuition.
@@ -389,3 +543,214 @@ A fix may pass one test but silently break prior working behavior.
- deterministic regression guard set, - deterministic regression guard set,
- MCP tool schema checks, - MCP tool schema checks,
- with full repro packet trace and no manual intervention. - with full repro packet trace and no manual intervention.
---
## Polyglot Orchestrator — PROPOSED
> Origin: design session 2026-02-28. Goal: true lossless polyglot code generation —
> every component of a project written in the language best suited to it, with
> type-safe boundaries, unified editing (LSP), and unified debugging (DAP),
> all driven from a single shared AST.
>
> This is a multi-phase capability that extends Whetstone from a single-target
> code generator into a polyglot orchestration platform. The theoretical proof
> is a project where every supported generation language participates, each
> responsible for the component it is most fit for, with no manual FFI code.
### Feature Request I: Language Fitness Scorer
**Tool name:** `whetstone_score_language_fitness`
**Derived:** `[Derived:DesignSession-Polyglot] Language Fitness Scorer`
**Problem:**
Language selection today is implicit — the caller picks a target language. There is no
mechanism to analyze an AST subtree and recommend which language best fits its
computational shape. This means the caller must already know the answer, which defeats
the purpose of language-agnostic AST representation.
**Requirements:**
- Extract structural features from an AST node or spec section:
- mutation ratio (writes/reads per node)
- recursion shape (tail-recursive, tree-recursive, flat loop)
- concurrency primitives present (channels, shared memory, actors)
- memory lifetime pattern (arena, GC, borrow-shaped)
- type complexity (simple generics, dependent types, none)
- I/O pattern (blocking, async, event-driven)
- Score each supported generation language against a language idiom profile.
- Return ranked list with per-language score and feature explanation.
**Acceptance criteria:**
- Given an AST node representing a data-parallel reduction loop, scores Julia/Rust
above Python/Lisp.
- Given an AST node representing a supervision tree with restart policies, scores
Erlang/Elixir above C++/Go.
- Ranking rationale is human-readable (per-feature delta from ideal profile).
- Tool runs in < 50ms on AST subtrees up to 500 nodes.
---
### Feature Request J: Polyglot FFI Glue Generator
**Tool name:** `whetstone_generate_ffi_glue`
**Derived:** `[Derived:DesignSession-Polyglot] Polyglot FFI Glue Generator`
**Problem:**
When a project uses multiple languages, the boundary between them (FFI, C ABI,
ctypes, JNI, pybind11, etc.) is written by hand. This is the primary source of
polyglot integration failures — type mismatches, lifetime errors, and ABI drift
are all caught only at runtime. The shared AST already encodes the type information
needed to generate these boundaries correctly.
**Requirements:**
- Given an AST boundary node (a function or struct visible to two language targets):
- Emit the correct binding on both sides (e.g. Rust `extern "C"` + Python ctypes struct)
- Emit a C ABI header as the neutral intermediary when needed
- Annotate the boundary with source AST node ID for cross-language debug symbol linking
- Support at minimum: Rust↔Python, Go↔C++, C++↔JavaScript (N-API), Rust↔Go (CGo)
- Generated glue must compile and pass a round-trip type check
**Acceptance criteria:**
- A Rust function callable from Python via generated ctypes binding, where the
binding was produced entirely from the shared AST — no hand-written glue.
- Round-trip test: Python calls Rust, Rust returns value, Python receives correct type.
- Boundary annotations present in generated DWARF (for DAP orchestrator).
---
### Feature Request K: Cross-Language Symbol Index Emitter
**Tool name:** `whetstone_emit_symbol_index`
**Derived:** `[Derived:DesignSession-Polyglot] Cross-Language Symbol Index`
**Problem:**
LSPs are language-specific and share no state. Cross-language goto-definition,
rename, and find-references are broken in all polyglot projects. The shared AST
already contains the symbol graph — it just isn't being emitted in a queryable format.
**Requirements:**
- Emit a SCIP-format (Sourcegraph Code Intelligence Protocol) symbol index from
the shared AST after multi-language generation.
- Index must include:
- Definition sites (per-language file + line)
- Reference sites across all generated languages
- Type information at language boundaries
- Cross-language "same symbol" links (Rust `my_fn` ↔ Python `my_fn` ↔ C ABI `my_fn`)
- Index is updated incrementally on re-generation of any language target.
**Acceptance criteria:**
- A symbol defined in Rust and called from Python: goto-definition from Python
resolves to Rust source file and line.
- Rename in the shared AST propagates to all language targets and updates index.
- Index generation adds < 200ms to a full polyglot generation run.
---
### Feature Request L: LSP Orchestration Layer
**Tool name:** (editor integration, no standalone MCP tool)
**Derived:** `[Derived:DesignSession-Polyglot] LSP Orchestrator`
**Problem:**
Editors run one LSP per language. Cross-language queries (goto-definition, hover,
rename) are silently broken at language seams. There is no mechanism to route an
LSP request through the shared AST when the answer crosses a language boundary.
**Requirements:**
- A thin LSP proxy that sits between the editor and all per-language LSP servers.
- Intercepts requests at known cross-language boundary symbols (identified via symbol index).
- For intra-language requests: forwards directly to the appropriate language server.
- For cross-language requests: resolves via the SCIP symbol index and returns a
synthesized response to the editor.
- Supports: textDocument/definition, textDocument/references, textDocument/rename,
textDocument/hover (shows both-side type information at boundaries).
**Acceptance criteria:**
- Goto-definition from a Python call site lands on Rust function definition.
- Rename of a boundary symbol in the orchestrator propagates rename requests to
both the Rust LSP and the Python LSP simultaneously.
- Hover at a Rust↔Python boundary shows the C ABI type as well as both native types.
---
### Feature Request M: DAP Orchestration Layer
**Tool name:** (debugger integration, no standalone MCP tool)
**Derived:** `[Derived:DesignSession-Polyglot] DAP Orchestrator`
**Problem:**
Debuggers are language-specific. A polyglot stack trace (Rust → Go → Lisp) is
unreadable — native frames, goroutine scheduler frames, and Lisp continuations
are interleaved with no cross-language annotation. Stepping across a language
boundary is not possible.
**Requirements:**
- A DAP proxy sitting above per-language debug adapters (CodeLLDB for Rust,
Delve for Go, etc.).
- Uses DWARF boundary annotations (emitted by FFI Glue Generator) to identify
language seam frames in a mixed stack.
- Presents a unified, annotated stack trace: each frame labeled with its origin
language, AST node ID, and source location.
- Supports cross-language step-over: stepping past a boundary call executes
the foreign call and pauses at the next source line in the calling language.
- Breakpoints set by AST node ID fire in all generated language targets
simultaneously.
**Acceptance criteria:**
- Single debugger session, stack trace shows Rust frames and Python frames
correctly interleaved and labeled.
- Step-over at a Rust→Python call boundary pauses at the next Rust line
(Python call executes atomically).
- Breakpoint on AST boundary node fires when either the Rust or Python side
of the call is entered.
---
### Feature Request N: Polyglot Project Test Harness
**Tool name:** `whetstone_run_polyglot_suite`
**Derived:** `[Derived:DesignSession-Polyglot] Polyglot Test Harness`
**Problem:**
There is no systematic way to verify that a multi-language generation round-trip
is semantically correct — that the same intent expressed in N languages produces
equivalent observable behavior. Without this, "lossless polyglot transpiling"
is a claim, not a proof.
**Requirements:**
- Given a spec and a set of target languages, generate all language variants.
- Run each variant against the same black-box test suite (input/output pairs).
- Compare outputs across all variants; report any semantic divergence.
- Track parity over time: if adding a new language breaks an existing variant's
parity, block the generation.
**Acceptance criteria:**
- A sorting algorithm spec generates Python, Rust, Go, and Haskell variants.
- All four variants produce identical output for the same 1000 random inputs.
- Introducing a deliberate bug in the Rust generator causes the harness to flag
Rust as divergent without affecting the Python/Go/Haskell parity report.
---
### Rollout Plan (Polyglot Orchestrator)
1. **Phase 1 — Language Fitness** (Sprint 271-272): `LanguageFitnessScorer` + 2-language test projects
2. **Phase 2 — FFI + Symbol Index** (Sprint 273-275): `PolyglotFFIGlueGenerator` + `CrossLanguageSymbolIndexEmitter`
3. **Phase 3 — LSP Orchestration** (Sprint 276-278): LSP proxy + cross-language goto/rename/hover
4. **Phase 4 — DAP Orchestration** (Sprint 279-281): DAP proxy + unified stack traces + boundary breakpoints
5. **Phase 5 — Proof** (Sprint 282-285): Full polyglot test harness + all-language test projects
### Done definition for "lossless polyglot transpiling"
- A single spec generates N language variants (N = all supported Whetstone generators).
- All variants pass an identical black-box test suite.
- Cross-language calls work at runtime via generated FFI glue (no hand-written bindings).
- Goto-definition, rename, and hover work across all language seams in the editor.
- A unified debugger session can step through a call that crosses three or more language boundaries.
- The only hand-written artifact is the spec.

267
docs/polyglot_orchestrator_sprint_plan.md Normal file
View File

@@ -0,0 +1,267 @@
# Polyglot Orchestrator — Sprint Plan
> Created: 2026-02-28
> Follows: Sprint 270 (all 26 generator readiness gaps closed)
> Goal: True lossless polyglot code generation with unified editing and debugging
---
## What We Are Building
A system where:
1. A single spec drives code generation into multiple languages simultaneously
2. Each language gets the component it is best suited for (fitness-scored)
3. FFI boundaries between languages are generated, not written by hand
4. A single editor session understands all languages via an LSP orchestrator
5. A single debugger session can step across language boundaries via a DAP orchestrator
6. A test harness proves behavioral parity across all language variants
The theoretical endpoint: a project with components in every supported Whetstone
generation language, each component calling the next, with no hand-written glue,
unified editing, and unified debugging. This proves lossless polyglot transpiling
as a construction, not a claim.
---
## Phase 1 — Language Fitness Scorer (Sprint 271-272)
**Steps 1883-1892**
The ability to analyze an AST subtree and recommend a ranked list of target languages
based on computational shape. Prerequisite for everything else — you can't route
generation to the right language without knowing what "right" means.
### Sprint 271: LanguageFitnessScorer core (Steps 1883-1887)
| Step | Component | What |
|------|-----------|------|
| 1883 | `ASTFeatureExtractor` | Extract scalar features from AST subtrees: mutation ratio, recursion shape, concurrency primitives, I/O pattern, type complexity |
| 1884 | `LanguageIdiomProfile` | Static profiles for each supported language: ideal feature vector per language |
| 1885 | `LanguageFitnessScorer` | Score AST features against profiles, return ranked list with rationale |
| 1886 | `whetstone_score_language_fitness` | MCP tool wiring |
| 1887 | Sprint 271 integration | Cross-component test: spec section → features → scores → ranked list |
### Sprint 272: Fitness-routed 2-language test projects (Steps 1888-1892)
| Step | Component | What |
|------|-----------|------|
| 1888 | `PolyglotProjectSpec` | Spec format extension: per-section language hints + fitness override |
| 1889 | Test project: `poly-sort` | Rust (sort core) + Python (data gen/validation) — fitness-routed |
| 1890 | Test project: `poly-api` | Go (HTTP server) + TypeScript (client) — fitness-routed |
| 1891 | Test project: `poly-parse` | C++ (lexer/parser) + Haskell (AST transformation) — fitness-routed |
| 1892 | Sprint 272 integration | All 3 test projects: generate → compile → run → output matches expected |
---
## Phase 2 — FFI Glue + Symbol Index (Sprint 273-275)
**Steps 1893-1907**
Generate the boundaries between languages from the shared AST. This is what makes
the boundaries type-safe and eliminates hand-written glue code.
### Sprint 273: PolyglotFFIGlueGenerator (Steps 1893-1897)
| Step | Component | What |
|------|-----------|------|
| 1893 | `ABIBoundaryExtractor` | Identify AST nodes that cross language boundaries (exported functions, shared structs) |
| 1894 | `CHeaderEmitter` | Emit C ABI header as neutral intermediary for any language pair |
| 1895 | `RustPythonBindingEmitter` | Rust `extern "C"` + Python ctypes binding from same AST node |
| 1896 | `GoCppBindingEmitter` | CGo + C++ extern from same AST node |
| 1897 | Sprint 273 integration | Rust↔Python round-trip: Python calls Rust via generated binding |
### Sprint 274: More binding pairs + DWARF annotations (Steps 1898-1902)
| Step | Component | What |
|------|-----------|------|
| 1898 | `CppJSBindingEmitter` | C++ N-API binding from AST node |
| 1899 | `RustGoBindingEmitter` | Rust FFI + CGo binding from AST node |
| 1900 | `DWARFBoundaryAnnotator` | Annotate generated DWARF with cross-language AST node IDs (needed for DAP orchestrator) |
| 1901 | `whetstone_generate_ffi_glue` | MCP tool wiring |
| 1902 | Sprint 274 integration | Go↔C++ round-trip: Go calls C++ via generated binding |
### Sprint 275: Cross-Language Symbol Index (Steps 1903-1907)
| Step | Component | What |
|------|-----------|------|
| 1903 | `SCIPEmitter` | Emit SCIP-format symbol index from shared AST after multi-language generation |
| 1904 | `CrossLanguageSymbolTable` | In-memory symbol table linking same-origin symbols across all generated languages |
| 1905 | `SymbolIndexUpdater` | Incremental update on re-generation (don't rebuild from scratch on every change) |
| 1906 | `whetstone_emit_symbol_index` | MCP tool wiring |
| 1907 | Sprint 275 integration | poly-sort: generate Rust+Python → emit index → verify cross-language symbol links |
---
## Phase 3 — LSP Orchestration (Sprint 276-278)
**Steps 1908-1922**
An LSP proxy that routes editor queries through the symbol index. The editor
sees one language server; the orchestrator fans out to the right per-language
server or resolves via the symbol index for cross-language queries.
### Sprint 276: LSP proxy core (Steps 1908-1912)
| Step | Component | What |
|------|-----------|------|
| 1908 | `LSPProxyServer` | LSP protocol handler that forwards to per-language servers |
| 1909 | `LanguageServerRouter` | Route by file extension / detected language |
| 1910 | `CrossLanguageBoundaryDetector` | Identify request targets that are cross-language boundary symbols |
| 1911 | `CrossLanguageDefinitionResolver` | Resolve goto-definition across language seams via symbol index |
| 1912 | Sprint 276 integration | poly-sort: goto-definition from Python call site lands on Rust source |
### Sprint 277: Hover + references + rename (Steps 1913-1917)
| Step | Component | What |
|------|-----------|------|
| 1913 | `CrossLanguageHoverProvider` | Hover at boundary shows both-side types + C ABI type |
| 1914 | `CrossLanguageReferencesProvider` | Find all references across all language targets |
| 1915 | `CrossLanguageRenameOrchestrator` | Rename propagates to all LSPs + updates symbol index |
| 1916 | `LSPProxyConfig` | Config file: which languages, which LSP servers, which port |
| 1917 | Sprint 277 integration | poly-api: rename shared type in Go+TS simultaneously |
### Sprint 278: LSP proxy hardening (Steps 1918-1922)
| Step | Component | What |
|------|-----------|------|
| 1918 | `LSPRequestCache` | Cache intra-language responses to reduce round-trip latency |
| 1919 | `LSPServerHealthMonitor` | Detect crashed language servers, route to degraded mode |
| 1920 | `LSPProxyDiagnosticAggregator` | Merge per-language diagnostics into unified list (no duplicates at boundaries) |
| 1921 | LSP proxy test suite | End-to-end: proxy running, 3 language servers, cross-language queries correct |
| 1922 | Sprint 278 integration | All 3 Phase 1 test projects: LSP proxy running, cross-language navigation verified |
---
## Phase 4 — DAP Orchestration (Sprint 279-281)
**Steps 1923-1937**
A DAP proxy that presents a unified debugger session across all language runtimes.
Stack traces are annotated, stepping across language boundaries works, breakpoints
can be set on AST nodes (firing in any language target).
### Sprint 279: DAP proxy core + stack annotation (Steps 1923-1927)
| Step | Component | What |
|------|-----------|------|
| 1923 | `DAPProxyServer` | DAP protocol handler forwarding to per-language debug adapters |
| 1924 | `DebugAdapterRouter` | Route by active frame language (detected from DWARF annotations) |
| 1925 | `PolyglotStackTraceDecoder` | Annotate mixed-language stack with origin language + AST node ID per frame |
| 1926 | `LanguageSeamFrameDetector` | Identify frames that sit at a language boundary (from DWARF annotations) |
| 1927 | Sprint 279 integration | poly-sort: unified stack trace shows Rust + Python frames labeled correctly |
### Sprint 280: Cross-language step + breakpoints (Steps 1928-1932)
| Step | Component | What |
|------|-----------|------|
| 1928 | `CrossLanguageStepOverHandler` | Step-over at a boundary call executes foreign call atomically, pauses in caller |
| 1929 | `CrossLanguageStepIntoHandler` | Step-into at a boundary call switches active debug adapter to foreign language |
| 1930 | `ASTNodeBreakpointMapper` | Set breakpoint by AST node ID, fires in all language targets that implement that node |
| 1931 | `BreakpointSyncOrchestrator` | Keep breakpoints in sync across all active debug adapters |
| 1932 | Sprint 280 integration | poly-api: breakpoint on Go handler fires; step-into crosses into TS call |
### Sprint 281: DAP hardening + multi-adapter coordination (Steps 1933-1937)
| Step | Component | What |
|------|-----------|------|
| 1933 | `DAPSessionLifecycleManager` | Launch/attach/detach per-language adapters as the debug session progresses |
| 1934 | `UnifiedVariableInspector` | Inspect variables across the boundary (translate types to editor representation) |
| 1935 | `DAPHealthMonitor` | Handle adapter crashes, fall back to single-language mode gracefully |
| 1936 | DAP proxy test suite | Step-into, step-over, breakpoint, variable inspect — all cross-language |
| 1937 | Sprint 281 integration | poly-parse: C++↔Haskell debug session — step across AST transformation boundary |
---
## Phase 5 — The Proof (Sprint 282-285)
**Steps 1938-1957**
The full polyglot test harness and the all-language test projects. This is the
demonstration that the system is real — not a claim but a construction.
### Sprint 282: Polyglot test harness (Steps 1938-1942)
| Step | Component | What |
|------|-----------|------|
| 1938 | `PolyglotSpecRunner` | Generate N language variants from one spec, compile and run all |
| 1939 | `BehavioralParityChecker` | Compare outputs across variants for same inputs, flag divergence |
| 1940 | `ParityRegressionGuard` | Block new language additions that break existing parity |
| 1941 | `whetstone_run_polyglot_suite` | MCP tool wiring |
| 1942 | Sprint 282 integration | poly-sort: Python+Rust+Go+Haskell — all produce identical output for 1000 inputs |
### Sprint 283: Wacky project 1 — poly-pipeline (Steps 1943-1947)
A data pipeline where each stage is a different language:
- Python (CSV ingestion + schema validation)
- Rust (data normalization + outlier detection, perf-critical)
- Go (fan-out routing to multiple sinks, concurrency)
- Haskell (statistical transformation, pure function)
- JavaScript/TypeScript (JSON serialization + HTTP push to consumer)
| Step | What |
|------|------|
| 1943 | Spec: poly-pipeline — all 5 components, interfaces defined in shared AST |
| 1944 | Generate all 5 language targets + FFI glue at each boundary |
| 1945 | Integration: end-to-end CSV in → HTTP push out, all 5 stages running |
| 1946 | LSP + DAP: unified editing and debugging across all 5 languages |
| 1947 | Parity check: replace each stage with a reference Python implementation, verify same output |
### Sprint 284: Wacky project 2 — poly-compiler (Steps 1948-1952)
A toy compiler where each phase is the language most suited to it:
- C++ (lexer — character-level performance, no GC pressure)
- Haskell (parser — recursive descent, algebraic data types)
- Rust (type checker — borrow-safe, zero-cost ownership tracking)
- Lisp/Scheme (optimizer — s-expression AST manipulation, homoiconicity)
- Go (code emitter — goroutine-parallel emission, simple concurrency)
| Step | What |
|------|------|
| 1948 | Spec: poly-compiler — 5 phases, shared IR type defined in AST |
| 1949 | Generate all 5 language targets + FFI glue |
| 1950 | Integration: source string in → bytecode out, all 5 phases connected |
| 1951 | LSP + DAP: cross-language navigation and debugging in the compiler itself |
| 1952 | Parity check: each phase replaceable with Python reference implementation |
### Sprint 285: Wacky project 3 — poly-everything (Steps 1953-1957)
One project, every supported generation language. Each language owns one function.
The chain: Language A calls B, B calls C, ... last language returns to A.
Fitness scorer assigns each language to the function most suited to it.
All FFI glue generated. All boundaries in the symbol index. Full LSP + DAP.
This is the theoretical proof: a single shared AST produced working code in every
supported language, connected at runtime, navigable in a single editor session,
debuggable in a single debugger session.
| Step | What |
|------|------|
| 1953 | Spec: poly-everything — one function per language, fitness-assigned |
| 1954 | Generate all N language targets + all FFI glue in one pipeline run |
| 1955 | Integration: full chain executes, output correct |
| 1956 | LSP + DAP: navigate and debug across all N language boundaries |
| 1957 | Parity report: all N language functions individually replaceable, parity confirmed |
---
## Success Definition
The sprint plan is complete when:
1. A spec with no target language hint produces a fitness-ranked recommendation
2. A 2-language project (any pair) generates with correct FFI glue — no hand-written bindings
3. Goto-definition from language A lands on the definition in language B
4. A single debugger session steps across a language A → language B call
5. poly-pipeline (5 languages) passes end-to-end with all stages running
6. poly-compiler (5 languages) compiles and runs a "hello world" source string
7. poly-everything (all languages) executes the full chain and parity is confirmed
At step 7, lossless polyglot transpiling is proven as a construction.
---
## Notes on Scope
- LSP and DAP orchestration layers are editor integrations, not MCP tools.
They run as separate processes alongside the whetstone_mcp binary.
- The SCIP symbol index is the shared data structure. It must be written to
a known path on generation and read by both the LSP and DAP orchestrators.
- Phase 5 test projects are not production software. They are proofs of concept
designed to stress the boundaries of the orchestration system.
- The practical value is not the test projects themselves but the integration
layer capability they prove: any project that needs FFI between two environments
can use Whetstone to generate the boundary from a shared spec.

290
docs/polyglot_test_projects.md Normal file
View File

@@ -0,0 +1,290 @@
# Polyglot Test Projects
> Created: 2026-02-28
> Purpose: Validate the polyglot orchestrator at each phase. Projects are ordered
> by complexity — 2-language pairs first, then multi-language, then all-language.
> None of these are production software. They are boundary stress tests.
---
## Tier 1: 2-Language Projects
These prove the basic pipeline: fitness scoring, FFI glue generation, symbol
index, and cross-language LSP. They use language pairs with well-understood
interop stories so failures are clearly attributable to the orchestrator.
---
### poly-sort
**Languages:** Rust + Python
**Pattern:** Performance core in Rust, scripting/validation layer in Python
```
Python (test runner)
│ generates random arrays, validates output
│ calls via ctypes (generated from shared AST)
▼
Rust (sort implementations)
│ merge sort, quick sort, radix sort
│ exposed as C ABI functions
└─ returns sorted array to Python
```
**Fitness rationale:**
- Rust: flat array mutation, cache-coherent loops, zero GC pressure → systems fitness
- Python: test orchestration, random generation, human-readable assertions → scripting fitness
**What it tests:**
- `LanguageFitnessScorer` recommends Rust for the sort functions, Python for the harness
- `RustPythonBindingEmitter` generates ctypes bindings for `sort_merge`, `sort_quick`, `sort_radix`
- Symbol index links Python `sort_merge` call site to Rust `sort_merge` definition
- Goto-definition from Python lands on Rust
- Debugger can step from Python test into Rust sort function
**Acceptance test:**
- 1000 random arrays of integers: all three sort variants produce output matching Python's `sorted()`
- No hand-written FFI code in the project
---
### poly-api
**Languages:** Go + TypeScript
**Pattern:** API server in Go, typed client in TypeScript
```
TypeScript (browser/Node client)
│ typed API client generated from shared AST
│ calls HTTP endpoints
▼
Go (HTTP server)
│ router, handlers, response serialization
│ types defined in shared AST
└─ serves JSON responses
```
**Fitness rationale:**
- Go: goroutine-based HTTP handling, simple concurrency, excellent standard library → concurrency fitness
- TypeScript: strong types for API contracts, browser-compatible, React/Node ecosystem → typed scripting fitness
**What it tests:**
- Shared type definitions (request/response structs) generated in both Go and TypeScript from same AST nodes
- No OpenAPI spec written by hand — the shared AST is the source of truth
- Symbol index links TypeScript `UserResponse` interface to Go `UserResponse` struct
- Rename of a field in shared AST propagates to both Go and TypeScript simultaneously
**Acceptance test:**
- TypeScript client calls Go server, receives typed response
- Field rename via LSP orchestrator compiles cleanly in both languages
- No hand-written type definitions or API client code
---
### poly-parse
**Languages:** C++ + Haskell
**Pattern:** High-performance lexer/tokenizer in C++, algebraic parser in Haskell
```
Haskell (parser)
│ recursive descent, pattern matching on token ADT
│ calls C++ tokenizer via FFI
▼
C++ (lexer)
│ character-level scanning, zero allocation
│ exposes token stream as C ABI iterator
└─ returns tokens to Haskell
```
**Fitness rationale:**
- C++: character-level loops, cache-coherent buffers, zero allocation → systems fitness
- Haskell: recursive ADT pattern matching, algebraic data types → functional fitness
**What it tests:**
- C++ → Haskell FFI (unusual direction; most polyglot FFI is C calling higher-level)
- `CHeaderEmitter` for the token stream iterator
- `GoCppBindingEmitter` is not used here — this exercises a new binding pair (C++↔Haskell via C ABI)
- Debugger: step from Haskell parser into C++ lexer and back
**Acceptance test:**
- Parse a sample grammar (JSON or a simple expression language)
- Output AST from Haskell matches reference Python parser for the same input
- No hand-written FFI
---
## Tier 2: Multi-Language Projects
These stress the orchestrator beyond two-language pairs. Each adds languages
that stress different aspects of the system (more binding pairs, more LSP
servers, larger symbol index, more complex debug adapter coordination).
---
### poly-pipeline
**Languages:** Python + Rust + Go + Haskell + TypeScript (5 languages)
**Pattern:** Data pipeline where each stage is the best language for that computation
```
Python CSV ingestion, schema validation, error reporting
↓ FFI (ctypes)
Rust Data normalization, outlier detection, statistics
↓ FFI (CGo)
Go Fan-out routing to multiple sinks, concurrent dispatch
↓ FFI (C ABI)
Haskell Statistical transformation, pure functional reduction
↓ FFI (N-API)
TypeScript JSON serialization, HTTP push to downstream consumer
```
**Fitness rationale:**
- Python: schema parsing, human-readable error messages, ecosystem (pandas for validation) → scripting
- Rust: numerical normalization on large arrays, no GC pauses at this stage → systems
- Go: goroutines for fan-out, select for concurrent dispatch → concurrency
- Haskell: pure reduction functions (fold, map), referential transparency at this stage → functional
- TypeScript: JSON is native, HTTP client ecosystem mature, consumer-facing → typed scripting
**What it tests:**
- 4 FFI boundaries generated from shared AST (Python↔Rust, Rust↔Go, Go↔Haskell, Haskell↔TS)
- Symbol index with 5 language targets — navigation across 4 boundaries
- LSP orchestrator running 5 language servers simultaneously
- DAP orchestrator stepping through 5-language call chain
- Parity: replace each stage with Python reference implementation, output must match
**Acceptance test:**
- CSV file in → HTTP POST to mock consumer
- All 5 stages running as one connected process (not microservices)
- End-to-end wall time within 2x of a pure Python reference implementation
---
### poly-compiler
**Languages:** C++ + Haskell + Rust + Lisp (Scheme) + Go (5 languages)
**Pattern:** A toy compiler where each phase is the language most fit for it
```
C++ Lexer — character-level scanning, no allocations per token
↓ C ABI
Haskell Parser — recursive descent, algebraic token → AST types
↓ C ABI
Rust Type checker — ownership-tracking symbol table, zero-cost
↓ C ABI
Lisp/Scheme Optimizer — s-expression AST rewriting, pattern-matched rules
↓ C ABI
Go Code emitter — parallel emission of bytecode, concurrent writes
```
**Fitness rationale:**
- C++: character-level tight loops, arena allocation for token buffer → systems
- Haskell: token ADTs, recursive descent naturally expressed as algebraic functions → functional
- Rust: symbol table with lifetime-safe references, borrow checker prevents dangling refs → systems/safety
- Lisp: the optimizer manipulates an AST that is itself S-expressions — homoiconicity → meta-programming
- Go: parallel bytecode emission with goroutines, simple coordination → concurrency
**What it tests:**
- The most philosophically interesting FFI direction: Lisp optimizing Rust's output then handing to Go
- The Lisp binding pair (Scheme↔C ABI) is a new surface for the glue generator
- poly-compiler is itself written in the languages it compiles — not intentionally self-hosting,
but the structural similarity is notable
**Acceptance test:**
- Compiles the following source string: `(let x 42) (+ x 1)` → bytecode `[PUSH 42, STORE x, LOAD x, PUSH 1, ADD]`
- Each phase independently replaceable with Python reference implementation (parity check)
---
## Tier 3: The Proof
---
### poly-everything
**Languages:** All supported Whetstone generation languages
**Pattern:** One function per language, fitness-assigned. Each calls the next.
The fitness scorer assigns each language to the function most suited to it.
The chain is determined by the scorer, not specified manually.
All FFI glue generated. All boundaries in the symbol index.
Full LSP + DAP.
**Current supported generation languages in Whetstone:**
Python, C++, Rust, Go, JavaScript/TypeScript, Java, Haskell, Elixir, Lisp/Scheme,
Julia, PHP, Rust (embedded), WASM (via Rust), Kotlin, Swift, Zig
**Suggested chain** (fitness scorer should reproduce this independently):
```
Zig Memory-mapped file reader (ultra-low-level, no runtime)
↓
C++ Binary deserialization (struct parsing, no GC)
↓
Rust Validation and normalization (safety-critical, owned types)
↓
Haskell Schema transformation (pure functional, algebraic types)
↓
Lisp/Scheme Rule application (S-expression pattern matching)
↓
Julia Statistical analysis (array operations, broadcasting)
↓
Python Result formatting (readable output, ecosystem)
↓
Go HTTP dispatch (concurrency, fan-out)
↓
TypeScript JSON response (typed client contract)
↓
Java Persistent storage (JDBC, enterprise ecosystem)
↓
Kotlin Android notification (Kotlin coroutines, Android SDK)
↓
Swift iOS notification (Swift async/await, Apple SDK)
↓
Elixir Real-time broadcast (actor model, Phoenix channels)
↓
PHP CMS integration (WordPress hook, legacy ecosystem)
↓
WASM Browser computation (sandboxed, portable)
└─ returns to Zig (closes the chain)
```
**What it proves:**
- The shared AST can represent computations across the full spectrum of
language paradigms: systems, functional, OO, scripting, actor, array, meta
- FFI glue can be generated for every adjacent pair in the chain
- The symbol index spans all language targets
- The LSP orchestrator can route queries across N language boundaries
- The DAP orchestrator can present a coherent stack trace spanning all languages
- Parity: every function replaceable by a Python reference implementation
**Acceptance test:**
- Binary file in → push notification + browser computation + CMS update + database write
- No hand-written FFI, no hand-written bindings, no hand-written type definitions
- The only human artifact is the spec
**Parity acceptance test:**
- Replace each language stage with a Python reference implementation
- Output must be identical for all 100 test inputs
- This is the formal proof of lossless transpiling
---
## What This Actually Means
If poly-everything passes, we have demonstrated:
1. **For integration layers**: Any environment boundary (Python↔C++, Go↔Java, Rust↔WASM)
can be generated from a shared spec. This was previously only possible for major projects
with dedicated platform integration teams (GraalVM, JPype, pybind11, etc.).
2. **For legacy modernization**: An existing system in language A can have a spec written
from it, and a reimplementation in language B generated with correct type-safe boundaries.
The language A and language B versions run in parallel and their outputs are compared
automatically to verify the migration.
3. **For new projects**: Language selection stops being a permanent architectural decision.
A component can be written in the fitness-optimal language and later regenerated in a
different language if the fitness profile changes (e.g. scaling requirements shift
from scripting-friendly to systems-performance).
The practical value is the integration layer capability. The theoretical value is
the proof that language is a generation parameter, not an architectural commitment.