whetstone_DSL/sprint23_plan.md

# Sprint 23 Plan: Architect Mode + Tech Stack Selection

## Context

The endgame feature: describe a problem in natural language, and the system
proposes a technology stack, project structure, and annotated skeleton. This
is the architect's power tool — not replacing the architect, but giving them
a structured starting point they can review, modify, and approve.

Sprint 23 builds on everything: the annotation taxonomy (Sprints 10-11),
the skeleton AST (Sprint 11e), the workflow model (Sprint 12), the orchestrator
(Sprint 15), the language coverage (17+ languages), and the transpilation engine
(Sprint 21).

---

## Phase 23a: Problem Decomposition (Steps 471-476)

### Step 471: Problem Description Parser (12 tests)
- Accept natural language problem description
- Extract structured requirements:
  - Functional requirements (what it should do)
  - Non-functional requirements (performance, security, scalability)
  - Platform constraints (web, mobile, embedded, server, CLI)
  - Integration requirements (databases, APIs, existing systems)
- Output: StructuredRequirements JSON with confidence per extraction
- This is the one place where LLM assistance is expected in the architecture —
  the extraction is an MCP prompt template, not hardcoded NLP

### Step 472: Module Decomposition Engine (12 tests)
- Given StructuredRequirements, propose module structure:
  - Identify major subsystems (auth, data, API, UI, etc.)
  - Define module boundaries and interfaces
  - Estimate complexity per module
  - Identify cross-cutting concerns (logging, error handling, config)
- Output: ModuleGraph with nodes (modules) and edges (dependencies)
- Multiple decomposition strategies: microservice vs monolith vs layered

### Step 473: Technology Stack Selector (12 tests)
- Given requirements + module graph, recommend tech stack:
  - Language selection per module (guided by requirements):
    - Performance-critical → Rust or C++
    - Web frontend → TypeScript
    - Data processing → Python
    - System scripting → Go or Python
    - Database → SQL dialect based on requirements
    - Embedded → C or assembly
  - Framework suggestions (as annotations, not framework-specific code)
  - Database selection based on data requirements
- Configurable preferences: "I prefer Rust for backend", "must use PostgreSQL"
- Output: TechStackDecision with reasoning per choice

### Step 474: Skeleton Generation from Requirements (12 tests)
- Given module graph + tech stack:
  1. Create skeleton modules in chosen languages
  2. Create skeleton functions/classes with @Intent annotations from requirements
  3. Add routing annotations: @Complexity, @ContextWidth, @Automatability
  4. Add contract annotations: @Contract from functional requirements
  5. Add dependency annotations between modules
- Output: Multi-file skeleton project ready for workflow creation

### Step 475: Architect Review Interface (12 tests)
- Present the proposed stack + skeleton to the architect for review:
  - Tech stack summary with reasoning
  - Module dependency graph (visual in GUI, structured in MCP)
  - Per-module: language choice, complexity estimate, key annotations
  - Modification tools: change language, merge modules, add modules, adjust routing
- Architect approves → skeleton becomes the project spec
- Architect modifies → re-run affected selections

### Step 476: Phase 23a Integration (8 tests)
- Full flow: "Build a REST API for a bookstore with user auth, PostgreSQL,
  and a React frontend" → structured requirements → module decomposition →
  tech stack (TypeScript frontend, Python/Rust backend, PostgreSQL) →
  skeleton project → architect review → approved → workflow created
- 87+ MCP tools total

---

## Phase 23b: Project Templates + Scaffolding (Steps 477-481)

### Step 477: Architecture Templates (12 tests)
- Pre-built architecture patterns:
  - **REST API:** routes + handlers + models + middleware + database
  - **CLI Tool:** argument parsing + commands + output formatting
  - **Library:** public API + internal modules + tests + documentation stubs
  - **Microservice:** service + transport + storage + health check
  - **Full Stack:** frontend + backend + database + deployment config
- Templates are parameterized: entity names, field definitions, auth method
- Each template produces a fully-annotated skeleton project

### Step 478: Scaffold File Generation (12 tests)
- Generate actual project files on disk from approved skeleton:
  - Source files with skeleton code + Semanno annotations
  - Project configuration (package.json, Cargo.toml, pyproject.toml, CMakeLists.txt)
  - Directory structure following language conventions
  - .gitignore appropriate for selected stack
  - .whetstone/ directory with sidecar files and workflow state
- Uses existing fileCreate/fileWrite MCP infrastructure

### Step 479: Dependency + Build System Awareness (12 tests)
- Annotate dependencies between modules with build system info:
  - Python: imports and pip requirements
  - Rust: crate dependencies in Cargo.toml format
  - Node: npm packages in package.json format
  - C++: include paths and CMake targets
  - SQL: migration ordering
- Not full build system integration — annotation-level awareness for context

### Step 480: Multi-Language Project Orchestration (12 tests)
- Workflow creation for multi-language projects:
  - Each module may be a different language
  - Cross-language interfaces annotated with @Link and @Shim
  - Build order respects cross-language dependencies
  - FFI boundaries explicitly annotated for human review
- Orchestrator handles multi-language workflow as a single project

### Step 481: Phase 23b Integration + Sprint Summary (8 tests)
- REST API template → scaffold files on disk → workflow → orchestrate
- Multi-language project: Python service + Rust core + SQL schema
- Architect modifies tech stack mid-planning → skeleton regenerates
- All scaffolded files have proper annotations and structure
- Sprint 23 totals: architect mode operational, templates working

---

## Phase 23c: Self-Hosting — Architect/Worker Dispatch (Steps 482-487)

### Context

Phases 23a and 23b build architect mode for *user projects*. Phase 23c
turns it inward: can the system architect and dispatch *its own* sprint
steps to worker agents? This is the self-hosting proof for the
architect/worker thesis.

The fixed "12 tests per step" invariant was a useful scaffold during early
sprints, but it's wrong in principle. The number, type, and depth of tests
should emerge from the complexity of what's being built — not from a
template. A trivial utility struct needs 3 unit tests. A safety-critical
pipeline needs unit tests, integration tests, contract tests, fuzz
boundaries, and negative tests. Phase 23c replaces the fixed count with
a test plan generator that reasons about *what kind of testing* a step
needs.

### Testing taxonomy used by the test plan generator

| Type | Purpose | When to use |
|------|---------|-------------|
| **Unit** | Single function/method correctness | Every public API surface |
| **Negative** | Confirm rejection of bad input | Validators, parsers, security boundaries |
| **Integration** | Multiple components working together | Phase-end steps, pipeline steps |
| **Contract** | Pre/post condition verification | Functions with @Contract annotations |
| **Regression** | Prior steps still pass | Every step (run prior test binaries) |
| **Boundary** | Edge cases at type/size limits | Arithmetic, buffers, collections |
| **Property** | Invariant holds across random inputs | Sorting, serialization roundtrips |
| **Smoke** | Quick "does it build/link/run" | Build system changes, scaffolding |
| **Performance** | No regression in speed/memory | Hot paths, algorithms with @Complexity |

The test plan generator selects from these types based on the step
description, not a fixed formula. A step adding a data struct might get
4 unit tests and 1 boundary test. A step wiring an RPC pipeline might
get 3 unit tests, 4 integration tests, and 2 negative tests. The total
count varies.

---

### Step 482: Step Spec Expander (tests: unit, negative, boundary)
- Given a sprint plan step description (the markdown block), produce:
  1. **Test plan**: which test types apply, how many of each, named test
     function stubs with one-line descriptions of what each verifies
  2. **Header skeleton**: main struct/class outlines with field names,
     method signatures, and doc-comments derived from the step description
  3. **Build system entry**: CMake target (or Cargo.toml entry, etc.)
     appropriate for the project's build system
- Input: step markdown + project conventions + prior step headers
- Output: `StepSpec` containing test plan, header skeleton, build entry
- The expander does NOT write final implementations — it writes enough
  structure that a worker agent can fill in the bodies without guessing
  at API shapes or test naming conventions
- Complexity classification: the expander reads the step description and
  tags it as `trivial` (1-4 tests), `standard` (5-10 tests),
  `complex` (10-15 tests), or `pipeline` (integration suite, 6-10 tests)

### Step 483: Convention Extractor + Validator (tests: unit, negative, regression)
- Scan an existing codebase and extract a machine-readable conventions doc:
  - Test harness pattern (macros, framework, assertion style)
  - Naming conventions (struct names, file names, function names)
  - File organization (where headers go, where tests go, include paths)
  - Size limits (max lines per file, max functions per class)
  - Build system patterns (how targets are added, link libraries)
  - Code style (header-only vs .cpp, namespace usage, include guards)
- Output: `ProjectConventions` JSON that the step expander and worker
  validator consume
- **Validator mode**: given a completed step's files + conventions,
  report violations:
  - Header over size limit
  - Wrong test harness pattern
  - Missing include path
  - Struct/function naming doesn't match project style
  - Build target missing or malformed
- Violations are warnings, not hard errors — the architect decides which
  to enforce

### Step 484: Prior-Step Context Injector (tests: unit, integration)
- When generating or dispatching a step, automatically gather context
  from prior steps that the worker will need:
  - Read the previous step's header(s) to extract struct definitions,
    enum values, and public method signatures
  - Identify which types from prior steps the new step is expected to
    use, extend, or compose
  - Build a condensed "context brief" (not the full file — just the API
    surface: struct names, field types, method signatures)
- Output: `StepContext` containing:
  - prior step API summaries (types + methods, no implementations)
  - dependency graph (which prior headers this step needs to #include)
  - naming patterns observed (if prior steps use `FooReport`, this step
    should use `BarReport`, not `bar_report_t`)
- The context brief is small enough to fit in a worker agent's prompt
  without consuming most of its context window

### Step 485: Worker Dispatch + Result Validation (tests: unit, integration, negative)
- Orchestrate the full architect→worker flow:
  1. Architect provides step description (from sprint plan)
  2. Step expander produces `StepSpec` (test plan + skeleton + build entry)
  3. Context injector produces `StepContext` (prior step APIs)
  4. Convention extractor produces `ProjectConventions`
  5. Worker agent receives: StepSpec + StepContext + ProjectConventions
  6. Worker produces: completed header + completed test file + build entry
  7. Convention validator checks the worker's output
  8. Build system compiles and runs tests
  9. If tests pass and conventions are met → step is done
  10. If tests fail → worker gets error output and retries (bounded)
  11. If conventions violated → report to architect for decision
- This step models the dispatch loop, not the actual agent execution —
  it validates that the data structures and handoff protocol work
- Retry budget: configurable, default 3 attempts before escalating
  to architect

### Step 486: Test Plan Generator (tests: unit, boundary, negative)
- The intelligence behind "how many tests and what kind":
  - Reads step description and classifies complexity
  - Reads prior step test files to understand testing patterns
  - Proposes test types from the taxonomy (unit, negative, integration,
    contract, boundary, property, smoke, performance, regression)
  - For each proposed test: name, type, one-line description, expected
    routing (can a worker write this, or does the architect need to?)
- Heuristics:
  - Step mentions "detect" or "validate" → include negative tests
  - Step mentions "pipeline" or "workflow" → include integration tests
  - Step mentions "RPC" or "MCP" → include smoke tests (does it respond?)
  - Step mentions "score" or "arithmetic" → include boundary tests
  - Step is a phase-end integration → regression suite
  - Step adds a new struct → unit tests for construction/accessors
  - Step modifies existing behavior → regression tests mandatory
- Output: `TestPlan` with typed, named, described test stubs
- The test plan is reviewed by the architect before the worker starts —
  the architect can add/remove/modify tests before dispatch

### Step 487: Phase 23c Integration (tests: integration, regression)
- Full self-hosting loop on a real sprint step:
  1. Take a step description from an existing sprint plan
  2. Run convention extractor on the current codebase
  3. Run context injector for the step's position in the sprint
  4. Run step expander to produce spec
  5. Run test plan generator to produce typed test plan
  6. Validate that the spec + plan + context is sufficient for a
     worker to implement without asking questions
  7. Run convention validator on a known-good completed step to
     verify zero violations
  8. Run convention validator on a deliberately-broken step to
     verify it catches violations
- Verify the full data flow: sprint plan → step spec → worker brief →
  output validation
- This does NOT require actual LLM worker dispatch — it validates the
  data structures and protocol that make dispatch possible

---

## Revised Architecture Invariant

The old invariant was:
> "12 tests per step, 8 for integration" — consistent test density

The new invariant is:
> **Test plans are generated per-step from complexity analysis.**
> Test count and type emerge from what's being built, not from a template.
> The test plan generator proposes; the architect approves.
> Regression tests on prior steps are always included.

---

## Step & Test Summary

| Phase | Steps | Tests | Theme |
|-------|-------|-------|-------|
| 23a | 471-476 | ~68 | Problem decomposition, tech stack selection, skeleton generation |
| 23b | 477-481 | ~56 | Templates, scaffolding, build awareness, multi-language orchestration |
| 23c | 482-487 | varies | Self-hosting: step expansion, conventions, worker dispatch, test planning |
| **Total** | **471-487** | **varies** | 17 steps |

Phase 23c test counts are intentionally not fixed — they will be determined
by the test plan generator itself (Step 486) once it exists. Until then,
each step lists which test *types* apply rather than a count.