From d1e883dd381093f4971002a5631a6af2eac4ce11 Mon Sep 17 00:00:00 2001 From: Bill Date: Sat, 28 Feb 2026 23:01:40 -0700 Subject: [PATCH] Add polyglot orchestrator feature requests, sprint plan, and test projects Feature requests I-N cover: LanguageFitnessScorer, PolyglotFFIGlueGenerator, CrossLanguageSymbolIndex, LSP Orchestrator, DAP Orchestrator, Polyglot Test Harness. Sprint plan maps 75 steps across sprints 271-285 in 5 phases. Test projects: poly-sort (Rust+Python), poly-api (Go+TS), poly-parse (C++/Haskell), poly-pipeline (5 lang), poly-compiler (5 lang), poly-everything (all languages). Theoretical goal: lossless polyglot transpiling proven as a construction. Co-Authored-By: Claude Sonnet 4.6 --- FEATURE_REQUESTS.md | 365 ++++++++++++++++++++++ docs/polyglot_orchestrator_sprint_plan.md | 267 ++++++++++++++++ docs/polyglot_test_projects.md | 290 +++++++++++++++++ 3 files changed, 922 insertions(+) create mode 100644 docs/polyglot_orchestrator_sprint_plan.md create mode 100644 docs/polyglot_test_projects.md diff --git a/FEATURE_REQUESTS.md b/FEATURE_REQUESTS.md index 7f66bb0..46e458f 100644 --- a/FEATURE_REQUESTS.md +++ b/FEATURE_REQUESTS.md @@ -2,6 +2,22 @@ > Backlog of feature ideas to triage into future sprints (e.g., Sprint 6/7). +## Title Convention (Semantic Logging) + +Feature request titles must include derivation source so provenance is explicit. + +Required title prefix format: + +- `[Derived:] ` + +Examples: + +- `[Derived:DataPipeline-LoRA] Semantic Validation Gate APIs for Taskitem Promotion` +- `[Derived:UserFeedback-HiveMind] Resource Lock Constraints on TaskItems` +- `[Derived:RunLogs-NativeTools] MCP Tool-Call Completion Diagnostics` + +Source tags should map to where the signal came from (run logs, user request, external project, postmortem, etc.), not just the proposed implementation area. + ## Security Vulnerability Awareness (Dependencies) — IMPLEMENTED (Sprint 6, Steps 190–195) **Goal:** Warn when a dependency has known vulnerabilities and surface safer alternatives. @@ -141,6 +157,127 @@ - Provide recommended defaults for CLI tools vs GUI apps - Include guidance for minimizing startup latency and bundle size +## [Derived:PersonalVaultTool-MultiPlatform] Platform-Agnostic Projection via SemAnno Adapter Boundaries — PROPOSED + +**Filed:** 2026-02-27 +**Filed by:** personal_vault_tool session (2026-02-27) +**Priority:** High — required for any real-world cross-platform projection (Python → Android/Kotlin, Python → Swift/iOS, etc.) +**Source context:** `personal_vault_tool/vault_core.py` — clean business logic and platform I/O are entangled in the same class with no annotation boundary. + +### Problem + +Whetstone's projector today treats a source file as a unit. When projecting `VaultStore` +to Kotlin, it faithfully projects `sqlite3.connect()` — which doesn't exist on Android. +The projector has no way to distinguish: + +- **Pure logic nodes** — `share_evaluate()` selector scoring, audit hash chain computation, + grant TTL enforcement — zero platform dependency, projectable to any target. +- **Platform adapter nodes** — `sqlite3.connect()`, `Fernet.encrypt()`, `socket.bind()` — + completely platform-specific, must be substituted per-target, not projected literally. + +This is not a generator quality issue (GR-022/023/024). It is an architectural gap: +the AST has no annotation that marks where the platform boundary is. + +### The Correct Architecture (already implied by SemAnno's design goals) + +``` +Source file with SemAnno annotations +│ +├── @pure.logic nodes → project as-is to any language +│ share_evaluate() +│ audit_hash_chain() +│ grant_scope_check() +│ +└── @platform.storage → generate abstract interface + per-target impl +│ _db() / read_records() / write_record() → Python: sqlite3 +│ → Android: Room DAO +│ → iOS: CoreData +│ +├── @platform.crypto → generate abstract interface + per-target impl +│ FernetEncryptor → Python: cryptography.Fernet +│ → Android: Android Keystore AES +│ +└── @platform.network → omit on mobile (caller drives transport) + ThreadingHTTPServer +``` + +The platform adapters become **generated abstract interfaces** with per-target +implementations, not projected concrete calls. + +### Feature Request A: SemAnno Platform Boundary Annotations + +**New annotation tags:** +- `@pure.logic` — no I/O, no platform imports; freely projectable +- `@platform.storage` — interacts with a persistence layer; requires adapter +- `@platform.crypto` — interacts with a key/encryption service; requires adapter +- `@platform.network` — interacts with sockets/HTTP; usually omitted on mobile +- `@platform.os` — interacts with filesystem, env vars, signals + +**Generator behavior:** annotated nodes are flagged before projection; the projector +emits an `AdapterInterface` node instead of projecting the implementation. + +**Validation gate:** `whetstone_validate_taskitem` or a new `whetstone_check_platform_purity` +tool should verify that `@pure.logic` nodes have zero `@platform.*` imports in their +transitive dependency graph. + +### Feature Request B: Platform Profile System + +**New concept:** a named platform profile describes what substitutions exist for each +`@platform.*` category. + +```json +{ + "profile": "android-kotlin", + "substitutions": { + "platform.storage": "androidx.room.RoomDatabase", + "platform.crypto": "android.security.keystore.KeyGenParameterSpec", + "platform.network": null + } +} +``` + +Built-in profiles to start: +- `python-stdlib` (current, implicit) +- `android-kotlin` (Room + Android Keystore) +- `swift-ios` (CoreData + CryptoKit) +- `node-js` (better-sqlite3 + node:crypto) + +**Generator behavior:** when projecting with a profile active, `@platform.*` nodes +are replaced by the profile's adapter interface stub rather than the source +implementation. The developer fills the stub; the logic layer is complete. + +### Feature Request C: Multi-Target Projection Command + +**Tool:** extend `whetstone_run_pipeline` or add `whetstone_project_multiplatform` + +**Input:** annotated source file + list of target profiles +**Output:** one generated artifact per target, plus a shared pure-logic module +that all targets import unchanged. + +**Acceptance criteria:** +- `personal_vault_tool/vault_core.py` (fully annotated) projects to: + - `out/python/vault_logic.py` + `out/python/adapters/sqlite_adapter.py` + - `out/kotlin/VaultLogic.kt` + `out/kotlin/adapters/RoomAdapter.kt` (stub) +- All `@pure.logic` methods are byte-identical in content between targets (modulo syntax). +- `@platform.*` methods emit only the interface signature + a `TODO` body in each target. + +### Relationship to existing gaps + +| Gap | Relationship | +|-----|-------------| +| GR-006 | Complex class generation — same root; multi-method class projection needed for VaultStore | +| GR-007 | Cross-language class node emission — must be closed before this works | +| GR-022/023/024 | Python generator string/import/comment bugs — must fix before Python→Python round-trip is clean enough to annotate | + +### Suggested sprint scope + +- Sprint N: SemAnno tags `@pure.logic`, `@platform.storage`, `@platform.crypto` (schema + parser + editor UI) +- Sprint N+1: Platform profile schema + substitution registry (2 built-in profiles: python-stdlib, android-kotlin) +- Sprint N+2: Projection gate — `whetstone_check_platform_purity` tool +- Sprint N+3: Multi-target pipeline output with stub generation + +--- + ## Python/C++/JS-TS Support Enhancements � PROPOSED **Goal:** Expand first-class support for Python, C++, and JS/TS beyond current parse/generate. @@ -218,6 +355,23 @@ HiveMind's scheduler knows the dispatch target node and enforces accordingly. ## Deterministic MCP Debugging Workflow for SLM Agents — PROPOSED +## [Derived:DataPipeline-LoRA] Semantic + Recursive Taskitem Validation APIs — PROPOSED + +**Goal:** Expose first-class MCP/editor APIs to score task completion beyond test-pass and feed recursive quality signals back into taskitem generation. + +**Why derived:** Current dataset curation needs semantic validity checks and recursive re-validation loops to avoid local minima in training data. + +**Scope:** +- Semantic completion validator API (invariant checks, intent/diff consistency, equivalence hooks). +- Recursive taskitem quality scorer (self-containment, dependency correctness, blocker detectability). +- Promotion-grade result packet (`execution_pass`, `semantic_pass`, `recursive_pass`, confidence). +- MCP endpoints for batch scoring and audit export. + +**Data-pipeline impact:** +- Reduces script-only heuristics. +- Enables consistent promotion gates across projects. +- Improves good/bad labeling quality for LoRA datasets. + **Filed by:** Whetstone execution session (2026-02-21) **Priority:** Critical for self-improving low-context agents **Goal:** Make generation + debugging robust enough that small/local models can complete tasks without expert intuition. @@ -389,3 +543,214 @@ A fix may pass one test but silently break prior working behavior. - deterministic regression guard set, - MCP tool schema checks, - with full repro packet trace and no manual intervention. + +--- + +## Polyglot Orchestrator — PROPOSED + +> Origin: design session 2026-02-28. Goal: true lossless polyglot code generation — +> every component of a project written in the language best suited to it, with +> type-safe boundaries, unified editing (LSP), and unified debugging (DAP), +> all driven from a single shared AST. +> +> This is a multi-phase capability that extends Whetstone from a single-target +> code generator into a polyglot orchestration platform. The theoretical proof +> is a project where every supported generation language participates, each +> responsible for the component it is most fit for, with no manual FFI code. + +### Feature Request I: Language Fitness Scorer + +**Tool name:** `whetstone_score_language_fitness` + +**Derived:** `[Derived:DesignSession-Polyglot] Language Fitness Scorer` + +**Problem:** +Language selection today is implicit — the caller picks a target language. There is no +mechanism to analyze an AST subtree and recommend which language best fits its +computational shape. This means the caller must already know the answer, which defeats +the purpose of language-agnostic AST representation. + +**Requirements:** +- Extract structural features from an AST node or spec section: + - mutation ratio (writes/reads per node) + - recursion shape (tail-recursive, tree-recursive, flat loop) + - concurrency primitives present (channels, shared memory, actors) + - memory lifetime pattern (arena, GC, borrow-shaped) + - type complexity (simple generics, dependent types, none) + - I/O pattern (blocking, async, event-driven) +- Score each supported generation language against a language idiom profile. +- Return ranked list with per-language score and feature explanation. + +**Acceptance criteria:** +- Given an AST node representing a data-parallel reduction loop, scores Julia/Rust + above Python/Lisp. +- Given an AST node representing a supervision tree with restart policies, scores + Erlang/Elixir above C++/Go. +- Ranking rationale is human-readable (per-feature delta from ideal profile). +- Tool runs in < 50ms on AST subtrees up to 500 nodes. + +--- + +### Feature Request J: Polyglot FFI Glue Generator + +**Tool name:** `whetstone_generate_ffi_glue` + +**Derived:** `[Derived:DesignSession-Polyglot] Polyglot FFI Glue Generator` + +**Problem:** +When a project uses multiple languages, the boundary between them (FFI, C ABI, +ctypes, JNI, pybind11, etc.) is written by hand. This is the primary source of +polyglot integration failures — type mismatches, lifetime errors, and ABI drift +are all caught only at runtime. The shared AST already encodes the type information +needed to generate these boundaries correctly. + +**Requirements:** +- Given an AST boundary node (a function or struct visible to two language targets): + - Emit the correct binding on both sides (e.g. Rust `extern "C"` + Python ctypes struct) + - Emit a C ABI header as the neutral intermediary when needed + - Annotate the boundary with source AST node ID for cross-language debug symbol linking +- Support at minimum: Rust↔Python, Go↔C++, C++↔JavaScript (N-API), Rust↔Go (CGo) +- Generated glue must compile and pass a round-trip type check + +**Acceptance criteria:** +- A Rust function callable from Python via generated ctypes binding, where the + binding was produced entirely from the shared AST — no hand-written glue. +- Round-trip test: Python calls Rust, Rust returns value, Python receives correct type. +- Boundary annotations present in generated DWARF (for DAP orchestrator). + +--- + +### Feature Request K: Cross-Language Symbol Index Emitter + +**Tool name:** `whetstone_emit_symbol_index` + +**Derived:** `[Derived:DesignSession-Polyglot] Cross-Language Symbol Index` + +**Problem:** +LSPs are language-specific and share no state. Cross-language goto-definition, +rename, and find-references are broken in all polyglot projects. The shared AST +already contains the symbol graph — it just isn't being emitted in a queryable format. + +**Requirements:** +- Emit a SCIP-format (Sourcegraph Code Intelligence Protocol) symbol index from + the shared AST after multi-language generation. +- Index must include: + - Definition sites (per-language file + line) + - Reference sites across all generated languages + - Type information at language boundaries + - Cross-language "same symbol" links (Rust `my_fn` ↔ Python `my_fn` ↔ C ABI `my_fn`) +- Index is updated incrementally on re-generation of any language target. + +**Acceptance criteria:** +- A symbol defined in Rust and called from Python: goto-definition from Python + resolves to Rust source file and line. +- Rename in the shared AST propagates to all language targets and updates index. +- Index generation adds < 200ms to a full polyglot generation run. + +--- + +### Feature Request L: LSP Orchestration Layer + +**Tool name:** (editor integration, no standalone MCP tool) + +**Derived:** `[Derived:DesignSession-Polyglot] LSP Orchestrator` + +**Problem:** +Editors run one LSP per language. Cross-language queries (goto-definition, hover, +rename) are silently broken at language seams. There is no mechanism to route an +LSP request through the shared AST when the answer crosses a language boundary. + +**Requirements:** +- A thin LSP proxy that sits between the editor and all per-language LSP servers. +- Intercepts requests at known cross-language boundary symbols (identified via symbol index). +- For intra-language requests: forwards directly to the appropriate language server. +- For cross-language requests: resolves via the SCIP symbol index and returns a + synthesized response to the editor. +- Supports: textDocument/definition, textDocument/references, textDocument/rename, + textDocument/hover (shows both-side type information at boundaries). + +**Acceptance criteria:** +- Goto-definition from a Python call site lands on Rust function definition. +- Rename of a boundary symbol in the orchestrator propagates rename requests to + both the Rust LSP and the Python LSP simultaneously. +- Hover at a Rust↔Python boundary shows the C ABI type as well as both native types. + +--- + +### Feature Request M: DAP Orchestration Layer + +**Tool name:** (debugger integration, no standalone MCP tool) + +**Derived:** `[Derived:DesignSession-Polyglot] DAP Orchestrator` + +**Problem:** +Debuggers are language-specific. A polyglot stack trace (Rust → Go → Lisp) is +unreadable — native frames, goroutine scheduler frames, and Lisp continuations +are interleaved with no cross-language annotation. Stepping across a language +boundary is not possible. + +**Requirements:** +- A DAP proxy sitting above per-language debug adapters (CodeLLDB for Rust, + Delve for Go, etc.). +- Uses DWARF boundary annotations (emitted by FFI Glue Generator) to identify + language seam frames in a mixed stack. +- Presents a unified, annotated stack trace: each frame labeled with its origin + language, AST node ID, and source location. +- Supports cross-language step-over: stepping past a boundary call executes + the foreign call and pauses at the next source line in the calling language. +- Breakpoints set by AST node ID fire in all generated language targets + simultaneously. + +**Acceptance criteria:** +- Single debugger session, stack trace shows Rust frames and Python frames + correctly interleaved and labeled. +- Step-over at a Rust→Python call boundary pauses at the next Rust line + (Python call executes atomically). +- Breakpoint on AST boundary node fires when either the Rust or Python side + of the call is entered. + +--- + +### Feature Request N: Polyglot Project Test Harness + +**Tool name:** `whetstone_run_polyglot_suite` + +**Derived:** `[Derived:DesignSession-Polyglot] Polyglot Test Harness` + +**Problem:** +There is no systematic way to verify that a multi-language generation round-trip +is semantically correct — that the same intent expressed in N languages produces +equivalent observable behavior. Without this, "lossless polyglot transpiling" +is a claim, not a proof. + +**Requirements:** +- Given a spec and a set of target languages, generate all language variants. +- Run each variant against the same black-box test suite (input/output pairs). +- Compare outputs across all variants; report any semantic divergence. +- Track parity over time: if adding a new language breaks an existing variant's + parity, block the generation. + +**Acceptance criteria:** +- A sorting algorithm spec generates Python, Rust, Go, and Haskell variants. +- All four variants produce identical output for the same 1000 random inputs. +- Introducing a deliberate bug in the Rust generator causes the harness to flag + Rust as divergent without affecting the Python/Go/Haskell parity report. + +--- + +### Rollout Plan (Polyglot Orchestrator) + +1. **Phase 1 — Language Fitness** (Sprint 271-272): `LanguageFitnessScorer` + 2-language test projects +2. **Phase 2 — FFI + Symbol Index** (Sprint 273-275): `PolyglotFFIGlueGenerator` + `CrossLanguageSymbolIndexEmitter` +3. **Phase 3 — LSP Orchestration** (Sprint 276-278): LSP proxy + cross-language goto/rename/hover +4. **Phase 4 — DAP Orchestration** (Sprint 279-281): DAP proxy + unified stack traces + boundary breakpoints +5. **Phase 5 — Proof** (Sprint 282-285): Full polyglot test harness + all-language test projects + +### Done definition for "lossless polyglot transpiling" + +- A single spec generates N language variants (N = all supported Whetstone generators). +- All variants pass an identical black-box test suite. +- Cross-language calls work at runtime via generated FFI glue (no hand-written bindings). +- Goto-definition, rename, and hover work across all language seams in the editor. +- A unified debugger session can step through a call that crosses three or more language boundaries. +- The only hand-written artifact is the spec. diff --git a/docs/polyglot_orchestrator_sprint_plan.md b/docs/polyglot_orchestrator_sprint_plan.md new file mode 100644 index 0000000..12ab64a --- /dev/null +++ b/docs/polyglot_orchestrator_sprint_plan.md @@ -0,0 +1,267 @@ +# Polyglot Orchestrator — Sprint Plan +> Created: 2026-02-28 +> Follows: Sprint 270 (all 26 generator readiness gaps closed) +> Goal: True lossless polyglot code generation with unified editing and debugging + +--- + +## What We Are Building + +A system where: +1. A single spec drives code generation into multiple languages simultaneously +2. Each language gets the component it is best suited for (fitness-scored) +3. FFI boundaries between languages are generated, not written by hand +4. A single editor session understands all languages via an LSP orchestrator +5. A single debugger session can step across language boundaries via a DAP orchestrator +6. A test harness proves behavioral parity across all language variants + +The theoretical endpoint: a project with components in every supported Whetstone +generation language, each component calling the next, with no hand-written glue, +unified editing, and unified debugging. This proves lossless polyglot transpiling +as a construction, not a claim. + +--- + +## Phase 1 — Language Fitness Scorer (Sprint 271-272) +**Steps 1883-1892** + +The ability to analyze an AST subtree and recommend a ranked list of target languages +based on computational shape. Prerequisite for everything else — you can't route +generation to the right language without knowing what "right" means. + +### Sprint 271: LanguageFitnessScorer core (Steps 1883-1887) + +| Step | Component | What | +|------|-----------|------| +| 1883 | `ASTFeatureExtractor` | Extract scalar features from AST subtrees: mutation ratio, recursion shape, concurrency primitives, I/O pattern, type complexity | +| 1884 | `LanguageIdiomProfile` | Static profiles for each supported language: ideal feature vector per language | +| 1885 | `LanguageFitnessScorer` | Score AST features against profiles, return ranked list with rationale | +| 1886 | `whetstone_score_language_fitness` | MCP tool wiring | +| 1887 | Sprint 271 integration | Cross-component test: spec section → features → scores → ranked list | + +### Sprint 272: Fitness-routed 2-language test projects (Steps 1888-1892) + +| Step | Component | What | +|------|-----------|------| +| 1888 | `PolyglotProjectSpec` | Spec format extension: per-section language hints + fitness override | +| 1889 | Test project: `poly-sort` | Rust (sort core) + Python (data gen/validation) — fitness-routed | +| 1890 | Test project: `poly-api` | Go (HTTP server) + TypeScript (client) — fitness-routed | +| 1891 | Test project: `poly-parse` | C++ (lexer/parser) + Haskell (AST transformation) — fitness-routed | +| 1892 | Sprint 272 integration | All 3 test projects: generate → compile → run → output matches expected | + +--- + +## Phase 2 — FFI Glue + Symbol Index (Sprint 273-275) +**Steps 1893-1907** + +Generate the boundaries between languages from the shared AST. This is what makes +the boundaries type-safe and eliminates hand-written glue code. + +### Sprint 273: PolyglotFFIGlueGenerator (Steps 1893-1897) + +| Step | Component | What | +|------|-----------|------| +| 1893 | `ABIBoundaryExtractor` | Identify AST nodes that cross language boundaries (exported functions, shared structs) | +| 1894 | `CHeaderEmitter` | Emit C ABI header as neutral intermediary for any language pair | +| 1895 | `RustPythonBindingEmitter` | Rust `extern "C"` + Python ctypes binding from same AST node | +| 1896 | `GoCppBindingEmitter` | CGo + C++ extern from same AST node | +| 1897 | Sprint 273 integration | Rust↔Python round-trip: Python calls Rust via generated binding | + +### Sprint 274: More binding pairs + DWARF annotations (Steps 1898-1902) + +| Step | Component | What | +|------|-----------|------| +| 1898 | `CppJSBindingEmitter` | C++ N-API binding from AST node | +| 1899 | `RustGoBindingEmitter` | Rust FFI + CGo binding from AST node | +| 1900 | `DWARFBoundaryAnnotator` | Annotate generated DWARF with cross-language AST node IDs (needed for DAP orchestrator) | +| 1901 | `whetstone_generate_ffi_glue` | MCP tool wiring | +| 1902 | Sprint 274 integration | Go↔C++ round-trip: Go calls C++ via generated binding | + +### Sprint 275: Cross-Language Symbol Index (Steps 1903-1907) + +| Step | Component | What | +|------|-----------|------| +| 1903 | `SCIPEmitter` | Emit SCIP-format symbol index from shared AST after multi-language generation | +| 1904 | `CrossLanguageSymbolTable` | In-memory symbol table linking same-origin symbols across all generated languages | +| 1905 | `SymbolIndexUpdater` | Incremental update on re-generation (don't rebuild from scratch on every change) | +| 1906 | `whetstone_emit_symbol_index` | MCP tool wiring | +| 1907 | Sprint 275 integration | poly-sort: generate Rust+Python → emit index → verify cross-language symbol links | + +--- + +## Phase 3 — LSP Orchestration (Sprint 276-278) +**Steps 1908-1922** + +An LSP proxy that routes editor queries through the symbol index. The editor +sees one language server; the orchestrator fans out to the right per-language +server or resolves via the symbol index for cross-language queries. + +### Sprint 276: LSP proxy core (Steps 1908-1912) + +| Step | Component | What | +|------|-----------|------| +| 1908 | `LSPProxyServer` | LSP protocol handler that forwards to per-language servers | +| 1909 | `LanguageServerRouter` | Route by file extension / detected language | +| 1910 | `CrossLanguageBoundaryDetector` | Identify request targets that are cross-language boundary symbols | +| 1911 | `CrossLanguageDefinitionResolver` | Resolve goto-definition across language seams via symbol index | +| 1912 | Sprint 276 integration | poly-sort: goto-definition from Python call site lands on Rust source | + +### Sprint 277: Hover + references + rename (Steps 1913-1917) + +| Step | Component | What | +|------|-----------|------| +| 1913 | `CrossLanguageHoverProvider` | Hover at boundary shows both-side types + C ABI type | +| 1914 | `CrossLanguageReferencesProvider` | Find all references across all language targets | +| 1915 | `CrossLanguageRenameOrchestrator` | Rename propagates to all LSPs + updates symbol index | +| 1916 | `LSPProxyConfig` | Config file: which languages, which LSP servers, which port | +| 1917 | Sprint 277 integration | poly-api: rename shared type in Go+TS simultaneously | + +### Sprint 278: LSP proxy hardening (Steps 1918-1922) + +| Step | Component | What | +|------|-----------|------| +| 1918 | `LSPRequestCache` | Cache intra-language responses to reduce round-trip latency | +| 1919 | `LSPServerHealthMonitor` | Detect crashed language servers, route to degraded mode | +| 1920 | `LSPProxyDiagnosticAggregator` | Merge per-language diagnostics into unified list (no duplicates at boundaries) | +| 1921 | LSP proxy test suite | End-to-end: proxy running, 3 language servers, cross-language queries correct | +| 1922 | Sprint 278 integration | All 3 Phase 1 test projects: LSP proxy running, cross-language navigation verified | + +--- + +## Phase 4 — DAP Orchestration (Sprint 279-281) +**Steps 1923-1937** + +A DAP proxy that presents a unified debugger session across all language runtimes. +Stack traces are annotated, stepping across language boundaries works, breakpoints +can be set on AST nodes (firing in any language target). + +### Sprint 279: DAP proxy core + stack annotation (Steps 1923-1927) + +| Step | Component | What | +|------|-----------|------| +| 1923 | `DAPProxyServer` | DAP protocol handler forwarding to per-language debug adapters | +| 1924 | `DebugAdapterRouter` | Route by active frame language (detected from DWARF annotations) | +| 1925 | `PolyglotStackTraceDecoder` | Annotate mixed-language stack with origin language + AST node ID per frame | +| 1926 | `LanguageSeamFrameDetector` | Identify frames that sit at a language boundary (from DWARF annotations) | +| 1927 | Sprint 279 integration | poly-sort: unified stack trace shows Rust + Python frames labeled correctly | + +### Sprint 280: Cross-language step + breakpoints (Steps 1928-1932) + +| Step | Component | What | +|------|-----------|------| +| 1928 | `CrossLanguageStepOverHandler` | Step-over at a boundary call executes foreign call atomically, pauses in caller | +| 1929 | `CrossLanguageStepIntoHandler` | Step-into at a boundary call switches active debug adapter to foreign language | +| 1930 | `ASTNodeBreakpointMapper` | Set breakpoint by AST node ID, fires in all language targets that implement that node | +| 1931 | `BreakpointSyncOrchestrator` | Keep breakpoints in sync across all active debug adapters | +| 1932 | Sprint 280 integration | poly-api: breakpoint on Go handler fires; step-into crosses into TS call | + +### Sprint 281: DAP hardening + multi-adapter coordination (Steps 1933-1937) + +| Step | Component | What | +|------|-----------|------| +| 1933 | `DAPSessionLifecycleManager` | Launch/attach/detach per-language adapters as the debug session progresses | +| 1934 | `UnifiedVariableInspector` | Inspect variables across the boundary (translate types to editor representation) | +| 1935 | `DAPHealthMonitor` | Handle adapter crashes, fall back to single-language mode gracefully | +| 1936 | DAP proxy test suite | Step-into, step-over, breakpoint, variable inspect — all cross-language | +| 1937 | Sprint 281 integration | poly-parse: C++↔Haskell debug session — step across AST transformation boundary | + +--- + +## Phase 5 — The Proof (Sprint 282-285) +**Steps 1938-1957** + +The full polyglot test harness and the all-language test projects. This is the +demonstration that the system is real — not a claim but a construction. + +### Sprint 282: Polyglot test harness (Steps 1938-1942) + +| Step | Component | What | +|------|-----------|------| +| 1938 | `PolyglotSpecRunner` | Generate N language variants from one spec, compile and run all | +| 1939 | `BehavioralParityChecker` | Compare outputs across variants for same inputs, flag divergence | +| 1940 | `ParityRegressionGuard` | Block new language additions that break existing parity | +| 1941 | `whetstone_run_polyglot_suite` | MCP tool wiring | +| 1942 | Sprint 282 integration | poly-sort: Python+Rust+Go+Haskell — all produce identical output for 1000 inputs | + +### Sprint 283: Wacky project 1 — poly-pipeline (Steps 1943-1947) + +A data pipeline where each stage is a different language: +- Python (CSV ingestion + schema validation) +- Rust (data normalization + outlier detection, perf-critical) +- Go (fan-out routing to multiple sinks, concurrency) +- Haskell (statistical transformation, pure function) +- JavaScript/TypeScript (JSON serialization + HTTP push to consumer) + +| Step | What | +|------|------| +| 1943 | Spec: poly-pipeline — all 5 components, interfaces defined in shared AST | +| 1944 | Generate all 5 language targets + FFI glue at each boundary | +| 1945 | Integration: end-to-end CSV in → HTTP push out, all 5 stages running | +| 1946 | LSP + DAP: unified editing and debugging across all 5 languages | +| 1947 | Parity check: replace each stage with a reference Python implementation, verify same output | + +### Sprint 284: Wacky project 2 — poly-compiler (Steps 1948-1952) + +A toy compiler where each phase is the language most suited to it: +- C++ (lexer — character-level performance, no GC pressure) +- Haskell (parser — recursive descent, algebraic data types) +- Rust (type checker — borrow-safe, zero-cost ownership tracking) +- Lisp/Scheme (optimizer — s-expression AST manipulation, homoiconicity) +- Go (code emitter — goroutine-parallel emission, simple concurrency) + +| Step | What | +|------|------| +| 1948 | Spec: poly-compiler — 5 phases, shared IR type defined in AST | +| 1949 | Generate all 5 language targets + FFI glue | +| 1950 | Integration: source string in → bytecode out, all 5 phases connected | +| 1951 | LSP + DAP: cross-language navigation and debugging in the compiler itself | +| 1952 | Parity check: each phase replaceable with Python reference implementation | + +### Sprint 285: Wacky project 3 — poly-everything (Steps 1953-1957) + +One project, every supported generation language. Each language owns one function. +The chain: Language A calls B, B calls C, ... last language returns to A. +Fitness scorer assigns each language to the function most suited to it. +All FFI glue generated. All boundaries in the symbol index. Full LSP + DAP. + +This is the theoretical proof: a single shared AST produced working code in every +supported language, connected at runtime, navigable in a single editor session, +debuggable in a single debugger session. + +| Step | What | +|------|------| +| 1953 | Spec: poly-everything — one function per language, fitness-assigned | +| 1954 | Generate all N language targets + all FFI glue in one pipeline run | +| 1955 | Integration: full chain executes, output correct | +| 1956 | LSP + DAP: navigate and debug across all N language boundaries | +| 1957 | Parity report: all N language functions individually replaceable, parity confirmed | + +--- + +## Success Definition + +The sprint plan is complete when: + +1. A spec with no target language hint produces a fitness-ranked recommendation +2. A 2-language project (any pair) generates with correct FFI glue — no hand-written bindings +3. Goto-definition from language A lands on the definition in language B +4. A single debugger session steps across a language A → language B call +5. poly-pipeline (5 languages) passes end-to-end with all stages running +6. poly-compiler (5 languages) compiles and runs a "hello world" source string +7. poly-everything (all languages) executes the full chain and parity is confirmed + +At step 7, lossless polyglot transpiling is proven as a construction. + +--- + +## Notes on Scope + +- LSP and DAP orchestration layers are editor integrations, not MCP tools. + They run as separate processes alongside the whetstone_mcp binary. +- The SCIP symbol index is the shared data structure. It must be written to + a known path on generation and read by both the LSP and DAP orchestrators. +- Phase 5 test projects are not production software. They are proofs of concept + designed to stress the boundaries of the orchestration system. +- The practical value is not the test projects themselves but the integration + layer capability they prove: any project that needs FFI between two environments + can use Whetstone to generate the boundary from a shared spec. diff --git a/docs/polyglot_test_projects.md b/docs/polyglot_test_projects.md new file mode 100644 index 0000000..a123c37 --- /dev/null +++ b/docs/polyglot_test_projects.md @@ -0,0 +1,290 @@ +# Polyglot Test Projects +> Created: 2026-02-28 +> Purpose: Validate the polyglot orchestrator at each phase. Projects are ordered +> by complexity — 2-language pairs first, then multi-language, then all-language. +> None of these are production software. They are boundary stress tests. + +--- + +## Tier 1: 2-Language Projects + +These prove the basic pipeline: fitness scoring, FFI glue generation, symbol +index, and cross-language LSP. They use language pairs with well-understood +interop stories so failures are clearly attributable to the orchestrator. + +--- + +### poly-sort + +**Languages:** Rust + Python +**Pattern:** Performance core in Rust, scripting/validation layer in Python + +``` +Python (test runner) + │ generates random arrays, validates output + │ calls via ctypes (generated from shared AST) + ▼ +Rust (sort implementations) + │ merge sort, quick sort, radix sort + │ exposed as C ABI functions + └─ returns sorted array to Python +``` + +**Fitness rationale:** +- Rust: flat array mutation, cache-coherent loops, zero GC pressure → systems fitness +- Python: test orchestration, random generation, human-readable assertions → scripting fitness + +**What it tests:** +- `LanguageFitnessScorer` recommends Rust for the sort functions, Python for the harness +- `RustPythonBindingEmitter` generates ctypes bindings for `sort_merge`, `sort_quick`, `sort_radix` +- Symbol index links Python `sort_merge` call site to Rust `sort_merge` definition +- Goto-definition from Python lands on Rust +- Debugger can step from Python test into Rust sort function + +**Acceptance test:** +- 1000 random arrays of integers: all three sort variants produce output matching Python's `sorted()` +- No hand-written FFI code in the project + +--- + +### poly-api + +**Languages:** Go + TypeScript +**Pattern:** API server in Go, typed client in TypeScript + +``` +TypeScript (browser/Node client) + │ typed API client generated from shared AST + │ calls HTTP endpoints + ▼ +Go (HTTP server) + │ router, handlers, response serialization + │ types defined in shared AST + └─ serves JSON responses +``` + +**Fitness rationale:** +- Go: goroutine-based HTTP handling, simple concurrency, excellent standard library → concurrency fitness +- TypeScript: strong types for API contracts, browser-compatible, React/Node ecosystem → typed scripting fitness + +**What it tests:** +- Shared type definitions (request/response structs) generated in both Go and TypeScript from same AST nodes +- No OpenAPI spec written by hand — the shared AST is the source of truth +- Symbol index links TypeScript `UserResponse` interface to Go `UserResponse` struct +- Rename of a field in shared AST propagates to both Go and TypeScript simultaneously + +**Acceptance test:** +- TypeScript client calls Go server, receives typed response +- Field rename via LSP orchestrator compiles cleanly in both languages +- No hand-written type definitions or API client code + +--- + +### poly-parse + +**Languages:** C++ + Haskell +**Pattern:** High-performance lexer/tokenizer in C++, algebraic parser in Haskell + +``` +Haskell (parser) + │ recursive descent, pattern matching on token ADT + │ calls C++ tokenizer via FFI + ▼ +C++ (lexer) + │ character-level scanning, zero allocation + │ exposes token stream as C ABI iterator + └─ returns tokens to Haskell +``` + +**Fitness rationale:** +- C++: character-level loops, cache-coherent buffers, zero allocation → systems fitness +- Haskell: recursive ADT pattern matching, algebraic data types → functional fitness + +**What it tests:** +- C++ → Haskell FFI (unusual direction; most polyglot FFI is C calling higher-level) +- `CHeaderEmitter` for the token stream iterator +- `GoCppBindingEmitter` is not used here — this exercises a new binding pair (C++↔Haskell via C ABI) +- Debugger: step from Haskell parser into C++ lexer and back + +**Acceptance test:** +- Parse a sample grammar (JSON or a simple expression language) +- Output AST from Haskell matches reference Python parser for the same input +- No hand-written FFI + +--- + +## Tier 2: Multi-Language Projects + +These stress the orchestrator beyond two-language pairs. Each adds languages +that stress different aspects of the system (more binding pairs, more LSP +servers, larger symbol index, more complex debug adapter coordination). + +--- + +### poly-pipeline + +**Languages:** Python + Rust + Go + Haskell + TypeScript (5 languages) +**Pattern:** Data pipeline where each stage is the best language for that computation + +``` +Python CSV ingestion, schema validation, error reporting + ↓ FFI (ctypes) +Rust Data normalization, outlier detection, statistics + ↓ FFI (CGo) +Go Fan-out routing to multiple sinks, concurrent dispatch + ↓ FFI (C ABI) +Haskell Statistical transformation, pure functional reduction + ↓ FFI (N-API) +TypeScript JSON serialization, HTTP push to downstream consumer +``` + +**Fitness rationale:** +- Python: schema parsing, human-readable error messages, ecosystem (pandas for validation) → scripting +- Rust: numerical normalization on large arrays, no GC pauses at this stage → systems +- Go: goroutines for fan-out, select for concurrent dispatch → concurrency +- Haskell: pure reduction functions (fold, map), referential transparency at this stage → functional +- TypeScript: JSON is native, HTTP client ecosystem mature, consumer-facing → typed scripting + +**What it tests:** +- 4 FFI boundaries generated from shared AST (Python↔Rust, Rust↔Go, Go↔Haskell, Haskell↔TS) +- Symbol index with 5 language targets — navigation across 4 boundaries +- LSP orchestrator running 5 language servers simultaneously +- DAP orchestrator stepping through 5-language call chain +- Parity: replace each stage with Python reference implementation, output must match + +**Acceptance test:** +- CSV file in → HTTP POST to mock consumer +- All 5 stages running as one connected process (not microservices) +- End-to-end wall time within 2x of a pure Python reference implementation + +--- + +### poly-compiler + +**Languages:** C++ + Haskell + Rust + Lisp (Scheme) + Go (5 languages) +**Pattern:** A toy compiler where each phase is the language most fit for it + +``` +C++ Lexer — character-level scanning, no allocations per token + ↓ C ABI +Haskell Parser — recursive descent, algebraic token → AST types + ↓ C ABI +Rust Type checker — ownership-tracking symbol table, zero-cost + ↓ C ABI +Lisp/Scheme Optimizer — s-expression AST rewriting, pattern-matched rules + ↓ C ABI +Go Code emitter — parallel emission of bytecode, concurrent writes +``` + +**Fitness rationale:** +- C++: character-level tight loops, arena allocation for token buffer → systems +- Haskell: token ADTs, recursive descent naturally expressed as algebraic functions → functional +- Rust: symbol table with lifetime-safe references, borrow checker prevents dangling refs → systems/safety +- Lisp: the optimizer manipulates an AST that is itself S-expressions — homoiconicity → meta-programming +- Go: parallel bytecode emission with goroutines, simple coordination → concurrency + +**What it tests:** +- The most philosophically interesting FFI direction: Lisp optimizing Rust's output then handing to Go +- The Lisp binding pair (Scheme↔C ABI) is a new surface for the glue generator +- poly-compiler is itself written in the languages it compiles — not intentionally self-hosting, + but the structural similarity is notable + +**Acceptance test:** +- Compiles the following source string: `(let x 42) (+ x 1)` → bytecode `[PUSH 42, STORE x, LOAD x, PUSH 1, ADD]` +- Each phase independently replaceable with Python reference implementation (parity check) + +--- + +## Tier 3: The Proof + +--- + +### poly-everything + +**Languages:** All supported Whetstone generation languages +**Pattern:** One function per language, fitness-assigned. Each calls the next. + +The fitness scorer assigns each language to the function most suited to it. +The chain is determined by the scorer, not specified manually. +All FFI glue generated. All boundaries in the symbol index. +Full LSP + DAP. + +**Current supported generation languages in Whetstone:** +Python, C++, Rust, Go, JavaScript/TypeScript, Java, Haskell, Elixir, Lisp/Scheme, +Julia, PHP, Rust (embedded), WASM (via Rust), Kotlin, Swift, Zig + +**Suggested chain** (fitness scorer should reproduce this independently): +``` +Zig Memory-mapped file reader (ultra-low-level, no runtime) + ↓ +C++ Binary deserialization (struct parsing, no GC) + ↓ +Rust Validation and normalization (safety-critical, owned types) + ↓ +Haskell Schema transformation (pure functional, algebraic types) + ↓ +Lisp/Scheme Rule application (S-expression pattern matching) + ↓ +Julia Statistical analysis (array operations, broadcasting) + ↓ +Python Result formatting (readable output, ecosystem) + ↓ +Go HTTP dispatch (concurrency, fan-out) + ↓ +TypeScript JSON response (typed client contract) + ↓ +Java Persistent storage (JDBC, enterprise ecosystem) + ↓ +Kotlin Android notification (Kotlin coroutines, Android SDK) + ↓ +Swift iOS notification (Swift async/await, Apple SDK) + ↓ +Elixir Real-time broadcast (actor model, Phoenix channels) + ↓ +PHP CMS integration (WordPress hook, legacy ecosystem) + ↓ +WASM Browser computation (sandboxed, portable) + └─ returns to Zig (closes the chain) +``` + +**What it proves:** +- The shared AST can represent computations across the full spectrum of + language paradigms: systems, functional, OO, scripting, actor, array, meta +- FFI glue can be generated for every adjacent pair in the chain +- The symbol index spans all language targets +- The LSP orchestrator can route queries across N language boundaries +- The DAP orchestrator can present a coherent stack trace spanning all languages +- Parity: every function replaceable by a Python reference implementation + +**Acceptance test:** +- Binary file in → push notification + browser computation + CMS update + database write +- No hand-written FFI, no hand-written bindings, no hand-written type definitions +- The only human artifact is the spec + +**Parity acceptance test:** +- Replace each language stage with a Python reference implementation +- Output must be identical for all 100 test inputs +- This is the formal proof of lossless transpiling + +--- + +## What This Actually Means + +If poly-everything passes, we have demonstrated: + +1. **For integration layers**: Any environment boundary (Python↔C++, Go↔Java, Rust↔WASM) + can be generated from a shared spec. This was previously only possible for major projects + with dedicated platform integration teams (GraalVM, JPype, pybind11, etc.). + +2. **For legacy modernization**: An existing system in language A can have a spec written + from it, and a reimplementation in language B generated with correct type-safe boundaries. + The language A and language B versions run in parallel and their outputs are compared + automatically to verify the migration. + +3. **For new projects**: Language selection stops being a permanent architectural decision. + A component can be written in the fitness-optimal language and later regenerated in a + different language if the fitness profile changes (e.g. scaling requirements shift + from scripting-friendly to systems-performance). + +The practical value is the integration layer capability. The theoretical value is +the proof that language is a generation parameter, not an architectural commitment.