whetstone_DSL/docs/polyglot_orchestrator_sprint_plan.md

Polyglot Orchestrator — Sprint Plan

Created: 2026-02-28 Follows: Sprint 270 (all 26 generator readiness gaps closed) Goal: True lossless polyglot code generation with unified editing and debugging

---

What We Are Building

A system where:

1. A single spec drives code generation into multiple languages simultaneously
2. Each language gets the component it is best suited for (fitness-scored)
3. FFI boundaries between languages are generated, not written by hand
4. A single editor session understands all languages via an LSP orchestrator
5. A single debugger session can step across language boundaries via a DAP orchestrator
6. A test harness proves behavioral parity across all language variants

The theoretical endpoint: a project with components in every supported Whetstone generation language, each component calling the next, with no hand-written glue, unified editing, and unified debugging. This proves lossless polyglot transpiling as a construction, not a claim.

---

Phase 1 — Language Fitness Scorer (Sprint 271-272)

Steps 1883-1892

The ability to analyze an AST subtree and recommend a ranked list of target languages based on computational shape. Prerequisite for everything else — you can't route generation to the right language without knowing what "right" means.

Sprint 271: LanguageFitnessScorer core (Steps 1883-1887)

Step	Component	What
1883	ASTFeatureExtractor	Extract scalar features from AST subtrees: mutation ratio, recursion shape, concurrency primitives, I/O pattern, type complexity
1884	LanguageIdiomProfile	Static profiles for each supported language: ideal feature vector per language
1885	LanguageFitnessScorer	Score AST features against profiles, return ranked list with rationale
1886	whetstone_score_language_fitness	MCP tool wiring
1887	Sprint 271 integration	Cross-component test: spec section → features → scores → ranked list

Sprint 272: Fitness-routed 2-language test projects (Steps 1888-1892)

Step	Component	What
1888	PolyglotProjectSpec	Spec format extension: per-section language hints + fitness override
1889	Test project: poly-sort	Rust (sort core) + Python (data gen/validation) — fitness-routed
1890	Test project: poly-api	Go (HTTP server) + TypeScript (client) — fitness-routed
1891	Test project: poly-parse	C++ (lexer/parser) + Haskell (AST transformation) — fitness-routed
1892	Sprint 272 integration	All 3 test projects: generate → compile → run → output matches expected

---

Phase 2 — FFI Glue + Symbol Index (Sprint 273-275)

Steps 1893-1907

Generate the boundaries between languages from the shared AST. This is what makes the boundaries type-safe and eliminates hand-written glue code.

Sprint 273: PolyglotFFIGlueGenerator (Steps 1893-1897)

Step	Component	What
1893	ABIBoundaryExtractor	Identify AST nodes that cross language boundaries (exported functions, shared structs)
1894	CHeaderEmitter	Emit C ABI header as neutral intermediary for any language pair
1895	RustPythonBindingEmitter	Rust extern "C" + Python ctypes binding from same AST node
1896	GoCppBindingEmitter	CGo + C++ extern from same AST node
1897	Sprint 273 integration	Rust↔Python round-trip: Python calls Rust via generated binding

Sprint 274: More binding pairs + DWARF annotations (Steps 1898-1902)

Step	Component	What
1898	CppJSBindingEmitter	C++ N-API binding from AST node
1899	RustGoBindingEmitter	Rust FFI + CGo binding from AST node
1900	DWARFBoundaryAnnotator	Annotate generated DWARF with cross-language AST node IDs (needed for DAP orchestrator)
1901	whetstone_generate_ffi_glue	MCP tool wiring
1902	Sprint 274 integration	Go↔C++ round-trip: Go calls C++ via generated binding

Sprint 275: Cross-Language Symbol Index (Steps 1903-1907)

Step	Component	What
1903	SCIPEmitter	Emit SCIP-format symbol index from shared AST after multi-language generation
1904	CrossLanguageSymbolTable	In-memory symbol table linking same-origin symbols across all generated languages
1905	SymbolIndexUpdater	Incremental update on re-generation (don't rebuild from scratch on every change)
1906	whetstone_emit_symbol_index	MCP tool wiring
1907	Sprint 275 integration	poly-sort: generate Rust+Python → emit index → verify cross-language symbol links

---

Phase 3 — LSP Orchestration (Sprint 276-278)

Steps 1908-1922

An LSP proxy that routes editor queries through the symbol index. The editor sees one language server; the orchestrator fans out to the right per-language server or resolves via the symbol index for cross-language queries.

Sprint 276: LSP proxy core (Steps 1908-1912)

Step	Component	What
1908	LSPProxyServer	LSP protocol handler that forwards to per-language servers
1909	LanguageServerRouter	Route by file extension / detected language
1910	CrossLanguageBoundaryDetector	Identify request targets that are cross-language boundary symbols
1911	CrossLanguageDefinitionResolver	Resolve goto-definition across language seams via symbol index
1912	Sprint 276 integration	poly-sort: goto-definition from Python call site lands on Rust source

Sprint 277: Hover + references + rename (Steps 1913-1917)

Step	Component	What
1913	CrossLanguageHoverProvider	Hover at boundary shows both-side types + C ABI type
1914	CrossLanguageReferencesProvider	Find all references across all language targets
1915	CrossLanguageRenameOrchestrator	Rename propagates to all LSPs + updates symbol index
1916	LSPProxyConfig	Config file: which languages, which LSP servers, which port
1917	Sprint 277 integration	poly-api: rename shared type in Go+TS simultaneously

Sprint 278: LSP proxy hardening (Steps 1918-1922)

Step	Component	What
1918	LSPRequestCache	Cache intra-language responses to reduce round-trip latency
1919	LSPServerHealthMonitor	Detect crashed language servers, route to degraded mode
1920	LSPProxyDiagnosticAggregator	Merge per-language diagnostics into unified list (no duplicates at boundaries)
1921	LSP proxy test suite	End-to-end: proxy running, 3 language servers, cross-language queries correct
1922	Sprint 278 integration	All 3 Phase 1 test projects: LSP proxy running, cross-language navigation verified

---

Phase 4 — DAP Orchestration (Sprint 279-281)

Steps 1923-1937

A DAP proxy that presents a unified debugger session across all language runtimes. Stack traces are annotated, stepping across language boundaries works, breakpoints can be set on AST nodes (firing in any language target).

Sprint 279: DAP proxy core + stack annotation (Steps 1923-1927)

Step	Component	What
1923	DAPProxyServer	DAP protocol handler forwarding to per-language debug adapters
1924	DebugAdapterRouter	Route by active frame language (detected from DWARF annotations)
1925	PolyglotStackTraceDecoder	Annotate mixed-language stack with origin language + AST node ID per frame
1926	LanguageSeamFrameDetector	Identify frames that sit at a language boundary (from DWARF annotations)
1927	Sprint 279 integration	poly-sort: unified stack trace shows Rust + Python frames labeled correctly

Sprint 280: Cross-language step + breakpoints (Steps 1928-1932)

Step	Component	What
1928	CrossLanguageStepOverHandler	Step-over at a boundary call executes foreign call atomically, pauses in caller
1929	CrossLanguageStepIntoHandler	Step-into at a boundary call switches active debug adapter to foreign language
1930	ASTNodeBreakpointMapper	Set breakpoint by AST node ID, fires in all language targets that implement that node
1931	BreakpointSyncOrchestrator	Keep breakpoints in sync across all active debug adapters
1932	Sprint 280 integration	poly-api: breakpoint on Go handler fires; step-into crosses into TS call

Sprint 281: DAP hardening + multi-adapter coordination (Steps 1933-1937)

Step	Component	What
1933	DAPSessionLifecycleManager	Launch/attach/detach per-language adapters as the debug session progresses
1934	UnifiedVariableInspector	Inspect variables across the boundary (translate types to editor representation)
1935	DAPHealthMonitor	Handle adapter crashes, fall back to single-language mode gracefully
1936	DAP proxy test suite	Step-into, step-over, breakpoint, variable inspect — all cross-language
1937	Sprint 281 integration	poly-parse: C++↔Haskell debug session — step across AST transformation boundary

---

Phase 5 — The Proof (Sprint 282-285)

Steps 1938-1957

The full polyglot test harness and the all-language test projects. This is the demonstration that the system is real — not a claim but a construction.

Sprint 282: Polyglot test harness (Steps 1938-1942)

Step	Component	What
1938	PolyglotSpecRunner	Generate N language variants from one spec, compile and run all
1939	BehavioralParityChecker	Compare outputs across variants for same inputs, flag divergence
1940	ParityRegressionGuard	Block new language additions that break existing parity
1941	whetstone_run_polyglot_suite	MCP tool wiring
1942	Sprint 282 integration	poly-sort: Python+Rust+Go+Haskell — all produce identical output for 1000 inputs

Sprint 283: Wacky project 1 — poly-pipeline (Steps 1943-1947)

A data pipeline where each stage is a different language:

• Python (CSV ingestion + schema validation)
• Rust (data normalization + outlier detection, perf-critical)
• Go (fan-out routing to multiple sinks, concurrency)
• Haskell (statistical transformation, pure function)
• JavaScript/TypeScript (JSON serialization + HTTP push to consumer)

Step	What
1943	Spec: poly-pipeline — all 5 components, interfaces defined in shared AST
1944	Generate all 5 language targets + FFI glue at each boundary
1945	Integration: end-to-end CSV in → HTTP push out, all 5 stages running
1946	LSP + DAP: unified editing and debugging across all 5 languages
1947	Parity check: replace each stage with a reference Python implementation, verify same output

Sprint 284: Wacky project 2 — poly-compiler (Steps 1948-1952)

A toy compiler where each phase is the language most suited to it:

• C++ (lexer — character-level performance, no GC pressure)
• Haskell (parser — recursive descent, algebraic data types)
• Rust (type checker — borrow-safe, zero-cost ownership tracking)
• Lisp/Scheme (optimizer — s-expression AST manipulation, homoiconicity)
• Go (code emitter — goroutine-parallel emission, simple concurrency)

Step	What
1948	Spec: poly-compiler — 5 phases, shared IR type defined in AST
1949	Generate all 5 language targets + FFI glue
1950	Integration: source string in → bytecode out, all 5 phases connected
1951	LSP + DAP: cross-language navigation and debugging in the compiler itself
1952	Parity check: each phase replaceable with Python reference implementation

Sprint 285: Wacky project 3 — poly-everything (Steps 1953-1957)

One project, every supported generation language. Each language owns one function. The chain: Language A calls B, B calls C, ... last language returns to A. Fitness scorer assigns each language to the function most suited to it. All FFI glue generated. All boundaries in the symbol index. Full LSP + DAP.

This is the theoretical proof: a single shared AST produced working code in every supported language, connected at runtime, navigable in a single editor session, debuggable in a single debugger session.

Step	What
1953	Spec: poly-everything — one function per language, fitness-assigned
1954	Generate all N language targets + all FFI glue in one pipeline run
1955	Integration: full chain executes, output correct
1956	LSP + DAP: navigate and debug across all N language boundaries
1957	Parity report: all N language functions individually replaceable, parity confirmed

---

Success Definition

The sprint plan is complete when:

1. A spec with no target language hint produces a fitness-ranked recommendation
2. A 2-language project (any pair) generates with correct FFI glue — no hand-written bindings
3. Goto-definition from language A lands on the definition in language B
4. A single debugger session steps across a language A → language B call
5. poly-pipeline (5 languages) passes end-to-end with all stages running
6. poly-compiler (5 languages) compiles and runs a "hello world" source string
7. poly-everything (all languages) executes the full chain and parity is confirmed

At step 7, lossless polyglot transpiling is proven as a construction.

---

Notes on Scope

• LSP and DAP orchestration layers are editor integrations, not MCP tools. They run as separate processes alongside the whetstone_mcp binary.
• The SCIP symbol index is the shared data structure. It must be written to a known path on generation and read by both the LSP and DAP orchestrators.
• Phase 5 test projects are not production software. They are proofs of concept designed to stress the boundaries of the orchestration system.
• The practical value is not the test projects themselves but the integration layer capability they prove: any project that needs FFI between two environments can use Whetstone to generate the boundary from a shared spec.