whetstone_DSL/HANDOFF-2026-03-02-phase6-plan.md

Whetstone DSL — Session Handoff

Date: 2026-03-02 Session: Phase 6 Planning Last step completed: 1957 (Sprint 285 — poly-everything)

---

State on Entry

Sprints 284 and 285 were completed at the start of this session:

• Sprint 284 (1948–1952): poly-compiler — CompilerPhaseSpec, CompilerIRNode, CompilerPhaseRouter, CompilerPipelineRunner
• Sprint 285 (1953–1957): poly-everything — PolyglotFunctionMap, PolyglotExecutionContext, PolyglotResultAggregator, PolyglotOrchestrator
• Phase 5 (Sprints 282–285) fully complete

---

Key Decisions Made This Session

1. Gap Registry: Clean

All 26 GR-series gaps closed as of 2026-02-28. Compatibility ledger has one resolved entry. Nothing pending.

2. Phase 6: Library Dispatch

The system currently chooses languages but not libraries. This is the central gap: "write a matrix multiply in C++" and "use cublasSgemm" are not the same instruction.

Phase 6 closes this by making library + API selection fully deterministic:

• OperationTaxonomy — fixed registry of operation domains (serialization.json, compute.matrix, etc.)
• LibraryCapabilityLedger — per (library, operationDomain): score + known weaknesses + preferred APIs
• PerTaskLibrarySelector — deterministic max-score dispatch per task
• LibrarySymbolAdvisor — specific API function per (library, operationDomain)
• Integration into generate_taskitems — execution contracts gain selectedLibrary, operationDomain, preferredAPIs

No LLM or human in the selection loop at runtime. Annotation is one-time authoring.

Key insight: At deterministic compute speeds, "combinatorial explosion" across libraries × operations is irrelevant — the entire ledger evaluates in microseconds. We have essentially unlimited budget relative to what an LLM inference costs.

3. Per-Task Granularity (Not Per-Project)

Library selection is per-operation-domain, per-task — not per-project stack. A project using protobuf for IPC still selects nlohmann_json for a JSON serialization task (0.92 vs 0.60). The library that handles an operation best wins that task regardless of project stack.

4. Training Data Assessment

• Total: ~517K entries, 4.2 GB
• Core signal: mcp_calls_bad.jsonl (16K, failure modes), taskitem_pipeline_runs.jsonl (18.5K, pipeline structure), datasets/ benchmarks
• Will become wrong after Phase 6: The bulk of mcp_calls_good.jsonl and mcp_call_quality.jsonl — these train the SLM to make decisions that Phase 6 moves to deterministic compute. Training on them post-Phase-6 teaches wrong behavior.
• Action: Archive bad calls + pipeline structure + benchmark specs (~250 MB). Retire good call quality data (~350 MB) after Phase 6 is live. Discard logs/ execution traces (1.5 GB noise).

---

What to Do Next

Immediate: Start Sprint 286

Plan doc: docs/phase6_library_dispatch_sprint_plan.md

Sprint 286 (steps 1958–1962): OperationTaxonomy + LibraryCapabilityLedger

start_recording (session_id: sprint286-2026-03-02)
→ architect_intake (use phase6 plan Sprint 286 section as spec)
→ generate_taskitems
→ queue_ready
→ run_pipeline ×4
→ implement headers:
    OperationDomain.h
    LibraryCapabilityRecord.h
    LibraryCapabilityLedger.h
    OperationTaxonomy.h
    Sprint286IntegrationSummary.h
→ write tests: step1958_test.cpp – step1962_test.cpp
→ update CMakeLists (entries for step1958–step1962)
→ build + run tests
→ get_metrics → commit → update CLAUDE.md + MEMORY.md → write handoff

Initial library seeds for step1962 integration test:

• nlohmann_json: serialization.json=0.92
• protobuf: serialization.binary=0.95, serialization.json=0.60
• serde_json: serialization.json=0.98
• cublas: compute.matrix=0.99
• thrust: compute.parallel=0.97
• numpy: compute.matrix=0.95, compute.statistics=0.93
• tokio: network.async=0.97
• boost.asio: network.async=0.91

Sprint Sequence

Sprint	Steps	Theme
286	1958–1962	OperationTaxonomy + LibraryCapabilityLedger
287	1963–1967	PerTaskLibrarySelector
288	1968–1972	LibrarySymbolAdvisor
289	1973–1977	Integration into generate_taskitems
290	1978–1982	CUDA end-to-end proof

---

Files Written This Session

docs/phase6_library_dispatch_sprint_plan.md  — full Phase 6 plan (Sprints 286–290)
HANDOFF-2026-03-02-phase6-plan.md            — this file
CLAUDE.md                                    — updated to Phase 6
MEMORY.md                                    — updated to Sprint 285 complete, Phase 6 next

No code written this session — planning only.

---

Build Status

All tests passing through step1957. Branch: 001-core-ast-structure.

cmake --build editor/build-native --parallel  # builds all targets
./editor/build-native/step1957_test           # 5/5 pass