whetstone_DSL/docs/phase6_library_dispatch_sprint_plan.md

# Phase 6: Library Dispatch — Sprint Plan
**Steps 1958–1982 | Sprints 286–290**
**Written:** 2026-03-02

---

## Goal

Transform the task execution pipeline from **vanilla language programming** into
**library-aware, per-task deterministic dispatch**. After Phase 6, every taskitem
execution contract specifies not just a language but the exact library and API
function to use for that operation — chosen deterministically, with no LLM or
human in the loop for that decision.

---

## The Problem Being Solved

Current pipeline output (after Phase 5):
```
taskitem: "implement JSON serialization for UserRecord"
executionContract: { language: "C++", stepId: "serialize" }
```

Target pipeline output (after Phase 6):
```
taskitem: "implement JSON serialization for UserRecord"
executionContract: {
  language: "C++",
  library: "nlohmann_json",
  operationDomain: "serialization.json",
  preferredAPIs: ["nlohmann::json::dump()", "nlohmann::json::parse()"],
  capabilityScore: 0.92,
  avoidedLibraries: [{ name: "protobuf", reason: "no-pretty-print, slow-large-arrays" }],
  justification: "nlohmann scores 0.92 vs protobuf 0.60 for serialization.json"
}
```

The SLM receives a pre-decided contract. Its only job: write syntax around a
pre-specified API call. Smallest possible decision space → highest possible
accuracy → closest to determinism.

---

## Architecture

### New Components

```
OperationTaxonomy          — fixed registry of operation domains
                             anchored to what spec writers naturally write
                             e.g. serialization.json, compute.matrix, network.http.client,
                                  io.stream.audio, crypto.hash, db.query.sql, ...

LibraryCapabilityRecord    — per (library, operationDomain):
                             { score: float, preferredAPIs: [], knownWeaknesses: [],
                               notApplicable: bool, annotatedAt: date, source: str }

LibraryCapabilityLedger    — registry of LibraryCapabilityRecords
                             keyed by (libraryId, operationDomain)
                             one-time annotation, deterministic reads forever

PerTaskLibrarySelector     — given (normalizedRequirements, availableLibraries):
                             for each operation domain in the task:
                               rank available libraries by score
                               return LibrarySelection { library, domain, score,
                                                         preferredAPIs, avoidedLibraries }
                             fully deterministic: sort by score, pick max, log justification

LibrarySymbolAdvisor       — given (library, operationDomain):
                             return specific API symbols for that operation
                             seeded from: LSP symbol index, pre-built catalog, or both
                             deterministic lookup, no inference required

[Extended] generate_taskitems — execution contract gains:
                             selectedLibrary, operationDomain, capabilityScore,
                             preferredAPIs, avoidedLibraries, justification
```

### Selection Pipeline (fully deterministic after annotation)

```
spec text
  → architect_intake           normalize requirements, extract operation domains
  → language fitness scorer    pick language per operation (Phase 1, complete)
  → library capability ledger  pick library per operation domain
                               (deterministic: max score lookup)
  → library symbol advisor     pick specific API per operation
                               (deterministic: catalog lookup)
  → generate_taskitems         execution contract: language + library + APIs
  → SLM / code generator       fills in syntax only — no decisions required
```

---

## Operation Taxonomy Design Principles

1. **Anchored to spec language** — domains are what a spec writer writes, not what
   a library author writes. "serialize to JSON" not "invoke jackson.ObjectMapper".

2. **Domain granularity** — coarser than individual functions, finer than "IO":
   - `serialization.json` ≠ `serialization.binary` ≠ `serialization.xml`
   - `compute.matrix` ≠ `compute.ml.training` ≠ `compute.ml.inference`
   - `network.http.client` ≠ `network.http.server` ≠ `network.websocket`

3. **Hierarchical** — `serialization.json` is a child of `serialization`,
   enabling fallback scoring when a library annotates at the parent level.

4. **Finite and versioned** — the taxonomy is a versioned registry, not free text.
   New domains are added deliberately, not automatically.

---

## Library Capability Scoring Principles

- **Score range:** 0.00 (not applicable) to 1.00 (reference implementation)
- **Score meaning:**
  - 0.00 — not applicable for this domain
  - 0.60–0.75 — functional but with significant known weaknesses
  - 0.76–0.90 — good, minor weaknesses or API ergonomic issues
  - 0.91–1.00 — preferred reference for this domain
- **Annotation sources** (in priority order):
  1. Formal benchmark data (throughput, latency, memory, correctness coverage)
  2. Known CVE / correctness gaps from public vulnerability databases
  3. API surface analysis (does a direct function exist for the operation?)
  4. Community consensus from spec sheets / documentation
- **Annotation is one-time authoring** — the selector runs deterministically forever
- **Weaknesses are first-class** — a library can score 0.95 for binary serialization
  and explicitly 0.60 for JSON serialization; both entries exist in the ledger

---

## Sprint Breakdown

### Sprint 286 — OperationTaxonomy + LibraryCapabilityLedger (steps 1958–1962)

| Step | Component | Description |
|------|-----------|-------------|
| 1958 | OperationDomain | Enum/registry of operation domains with hierarchy; lookup, parent traversal |
| 1959 | LibraryCapabilityRecord | Struct + validator; score range enforcement, weakness list, API list |
| 1960 | LibraryCapabilityLedger | Registry keyed by (libraryId, operationDomain); get/set/has/score |
| 1961 | OperationTaxonomy | Full domain tree with 40+ initial entries; classify(text)→domain |
| 1962 | Integration | Seed ledger with 8 real libraries × their operation domains; verify lookups |

**Initial library set for seeding:**
- `nlohmann_json` — serialization.json: 0.92
- `protobuf` — serialization.binary: 0.95, serialization.json: 0.60
- `serde_json` (Rust) — serialization.json: 0.98
- `cublas` — compute.matrix: 0.99 (GPU), compute.matrix: 0.00 (CPU fallback)
- `thrust` — compute.parallel: 0.97
- `numpy` — compute.matrix: 0.95, compute.statistics: 0.93
- `tokio` (Rust) — network.async: 0.97, io.stream: 0.95
- `boost.asio` — network.async: 0.91, network.http.client: 0.78

---

### Sprint 287 — PerTaskLibrarySelector (steps 1963–1967)

| Step | Component | Description |
|------|-----------|-------------|
| 1963 | LibrarySelection | Struct: library, domain, score, preferredAPIs, avoidedLibraries, justification |
| 1964 | OperationClassifier | Classifies normalized requirement text → OperationDomain (deterministic) |
| 1965 | PerTaskLibrarySelector | Core selector: rank by score, return best + avoided list + justification |
| 1966 | SelectionJustificationLog | Structured log of every selection decision (auditable, deterministic replay) |
| 1967 | Integration | 8-task spec with mixed operations; verify correct library chosen per task |

---

### Sprint 288 — LibrarySymbolAdvisor (steps 1968–1972)

| Step | Component | Description |
|------|-----------|-------------|
| 1968 | LibrarySymbolRecord | Struct: functionName, signature, operationDomain, usagePattern, caveats |
| 1969 | LibrarySymbolCatalog | Registry of LibrarySymbolRecords; lookup by (library, operationDomain) |
| 1970 | LSPSymbolExtractor | Parses LSP completion/hover responses → LibrarySymbolRecords |
| 1971 | LibrarySymbolAdvisor | Given (library, operationDomain) → ranked symbol recommendations |
| 1972 | Integration | CUDA: advisee returns cublasSgemm for compute.matrix; nlohmann: json::parse |

---

### Sprint 289 — Integration into generate_taskitems (steps 1973–1977)

| Step | Component | Description |
|------|-----------|-------------|
| 1973 | EnrichedExecutionContract | Extends existing contract with library dispatch fields |
| 1974 | TaskitemLibraryAnnotator | Runs selector + advisor per taskitem; injects into contract |
| 1975 | AnnotatedTaskitemValidator | Validates library fields present when domain is known |
| 1976 | DispatchJustificationReport | Per-project report: every task → library choice → score → why |
| 1977 | Integration | Full pipeline: spec → intake → selector → advisor → enriched taskitems |

---

### Sprint 290 — CUDA End-to-End Proof (steps 1978–1982)

| Step | Component | Description |
|------|-----------|-------------|
| 1978 | CUDALibraryProfile | Full CUDA ledger entries: cublas, cufft, thrust, cudnn domains + scores |
| 1979 | CUDASymbolCatalog | 20+ CUDA API symbols with signatures and usage patterns |
| 1980 | ProjectStackDeclarator | Declares available libraries for a project (input to selector) |
| 1981 | ProofPipelineRunner | Runs full Phase 6 pipeline on a CUDA project spec |
| 1982 | Integration | Spec: "GPU matrix multiply + JSON result serialization"
                  Output: cublasSgemm for compute.matrix, nlohmann for serialization.json
                  Sprint290IntegrationSummary |

---

## What Changes for the SLM After Phase 6

**Before Phase 6** — SLM decides:
- What library to use (reasoning required)
- What API to call (knowledge required)
- Which edge cases the library handles (judgment required)

**After Phase 6** — SLM receives:
```
language: C++
library: cublas
operationDomain: compute.matrix
preferredAPIs: ["cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N, m, n, k, &alpha, A, lda, B, ldb, &beta, C, ldc)"]
knownWeaknesses: ["requires-device-memory-management", "no-automatic-handle-lifecycle"]
```

SLM job: write the function body that calls this API with these parameters.
No reasoning. No library knowledge. Syntax only.

---

## Impact on Training Data

After Phase 6 is live:
- **Retire:** `mcp_calls_good.jsonl`, `mcp_call_quality.jsonl` — these teach
  the SLM to make decisions that are now deterministic. Training on them teaches
  wrong behavior.
- **Archive:** `mcp_calls_bad.jsonl` — failure modes are architecture-independent.
- **Keep:** `taskitem_pipeline_runs.jsonl` — pipeline structure signal remains valid.
- **Keep:** `datasets/` benchmark specs — reusable eval sets regardless of architecture.

New training data target post-Phase 6: syntax-fill examples where the contract
is fully specified and the SLM is evaluated only on correctness of the generated
function body — not on the library/API decision.

---

## Success Criteria for Phase 6

1. Given a project spec with a declared library stack, every taskitem execution
   contract contains: `selectedLibrary`, `operationDomain`, `capabilityScore`,
   `preferredAPIs`, `avoidedLibraries`, `justification`
2. Selection is deterministic: same spec + same ledger = same output, always
3. No LLM or human consulted during selection
4. CUDA project proof: "GPU matrix multiply" → `cublasSgemm`, not a kernel loop
5. Mixed-stack proof: project uses protobuf for IPC but `serialization.json` task
   correctly selects `nlohmann_json` over protobuf (0.92 vs 0.60)