whetstone_DSL/sprint161_plan.md

# Sprint 161 Plan: Fix `whetstone_architect_intake` — Requirements Extraction

## Context

A/B test run on 2026-02-25 (see `docs/ab_test_ast_vs_language_first_2026-02-25.md`)
exposed a concrete failure in `whetstone_architect_intake`:

The tool was given a markdown spec with three sections — Goal, Requirements, and
Constraints. It returned only the two Constraints as `normalizedRequirements`.
The entire Requirements section (struct fields, class methods, behaviour rules)
was silently dropped.

**Why this matters:** `whetstone_generate_taskitems` receives normalizedRequirements
as its input. When intake drops functional requirements, taskitems are generated
from constraints alone — producing one generic placeholder task instead of a real
work breakdown. The downstream agent cannot complete the program without a smart
LLM filling the gap manually. That is the exact failure mode this sprint closes.

**Root cause:** The intake parser matches constraint-pattern sentences but does
not scan bullet-point lists under headings like "Requirements", "Features", or
"Behavior" for functional requirements. Field-list bullets (e.g.
"WorkItem has: job_id (string), priority (int)") are not parsed at all.

**No new MCP tools.** This is a quality fix to existing behaviour. The input/output
schema of `whetstone_architect_intake` is unchanged — only the content of
`normalizedRequirements` improves.

---

## Goals

1. Parse markdown bullet lists under recognised requirement headings into
   `NormalizedRequirement` objects with `kind: "requirement"` or `kind: "feature"`
2. Detect field-list syntax and emit one entry per field with `kind: "field"`
3. Wire the new parser into the existing intake handler without schema changes
4. Verify with the exact spec that failed in the A/B test: must produce ≥ 7
   normalized requirements and ≥ 5 taskitems (not 1 generic placeholder)

---

## Steps

### Step 1849: `RequirementsParser` — extract bullet lists from requirement sections (12 tests)

Create `editor/src/RequirementsParser.h`.

Scans markdown for headings whose lowercased, hyphenated anchor matches a
recognised set: `requirements`, `features`, `behavior`, `behaviour`,
`interface`, `api`. For each matching section, reads bullet items (`- text`
or `* text`) and produces one `ParsedRequirement` per item.

Detects field-list syntax: a bullet containing a colon followed by a
comma-separated list (e.g. `- WorkItem has: job_id (string), priority (int)`)
is split into one `ParsedRequirement` per field with `isField: true`,
`fieldName`, and `fieldType` populated.

```cpp
struct ParsedRequirement {
    std::string text;          // normalized lowercase
    std::string sourceText;    // original as written
    int         sourceLine;
    std::string sectionAnchor;
    std::string sectionTitle;
    bool        isField;
    std::string fieldName;     // when isField == true
    std::string fieldType;     // when isField == true
};

class RequirementsParser {
public:
    static std::vector<ParsedRequirement> parse(const std::string& markdown);
    static std::string anchorize(const std::string& heading);
    static bool isRequirementHeading(const std::string& anchor);
};
```

Tests (12): empty markdown returns empty, single bullet under Requirements heading
returns one entry, multiple bullets return multiple entries, bullet under
Constraints heading NOT returned (handled by existing parser), anchorize
lowercases and hyphenates, isRequirementHeading true for "requirements" and
"features", isRequirementHeading false for "constraints" and "goal", field-list
bullet produces multiple entries with isField==true, fieldName extracted correctly,
fieldType extracted correctly, sourceLine correct, sectionTitle preserved.

### Step 1850: `ArchitectIntakeProcessor` — merge all requirement kinds (10 tests)

Create `editor/src/ArchitectIntakeProcessor.h`.

Centralises the intake parsing logic currently inline in the MCP handler. Calls
both the existing constraint/conflict detection and `RequirementsParser`. Merges
results into one `normalizedRequirements` vector with all kinds present.

```cpp
struct IntakeResult {
    std::vector<nlohmann::json> normalizedRequirements; // kind: requirement/feature/field/constraint
    std::vector<nlohmann::json> conflicts;
    nlohmann::json              parsedSpec;
    int                         requirementCount;
    int                         fieldCount;
    int                         constraintCount;
    int                         conflictCount;
    bool                        hasConflicts;
};

class ArchitectIntakeProcessor {
public:
    static IntakeResult process(const std::string& markdown);
};
```

Tests (10): process returns IntakeResult, constraintCount matches constraint bullets,
requirementCount matches requirement bullets, fieldCount matches field-list extractions,
kind=="field" entries present for field-list bullets, kind=="constraint" entries still
present, conflictCount==0 when no conflicts, spec with only constraints produces no
requirement entries, all entries have requirementId assigned, ambiguous==false for
unambiguous bullets.

### Step 1851: Wire `ArchitectIntakeProcessor` into `whetstone_architect_intake` handler (8 tests)

Update the MCP handler for `whetstone_architect_intake` to call
`ArchitectIntakeProcessor::process()` instead of inline parsing.

No schema change — `inputSchema` unchanged, response field names unchanged.
Update description in `tools/claude/tools.json` to state that functional
requirements from bullet lists are now extracted alongside constraints.

Tests (8): tool still registered, inputSchema unchanged, response has
normalizedRequirements array, response has conflicts array, response has parsedSpec,
requirement entries present when Requirements section given, field entries present
for field-list bullets, constraint entries still present.

### Step 1852: End-to-end validation with the failing A/B test spec (10 tests)

Create `editor/src/Sprint161EndToEnd.h`.

Uses the exact spec from the failing 2026-02-25 A/B run as the primary input:

```
# Priority Work Queue
## Requirements
- WorkItem has: job_id (string), priority (int), payload (string)
- PriorityQueue supports: enqueue(item), dequeue() -> item, peek() -> item, size(), empty()
- Dequeue always returns the highest-priority item
- JSON serialization via nlohmann
## Constraints
- Header-only implementation
- No external dependencies beyond nlohmann and STL
```

Expected: normalizedRequirements contains ≥ 7 entries. Feeding the result to
`whetstone_generate_taskitems` must return ≥ 5 tasks. No task may have the
title "Intake Foundation Primary" (the generic fallback from the failed run).

Tests (10): intake produces > 2 normalizedRequirements, WorkItem field entries
present (kind=="field"), PriorityQueue entry present (kind=="requirement"),
constraint entries present (kind=="constraint"), no conflicts, taskitems call
returns success, taskCount >= 5, no task titled "Intake Foundation Primary",
at least one task has queueReady==true, planSummary.milestoneCount >= 1.

### Step 1853: Sprint 161 Integration Summary (8 tests)

Create `editor/src/Sprint161IntegrationSummary.h`.
Record: steps_completed=5 (1849-1853), failure_addressed="architect_intake
silently drops Requirements section bullet lists", ab_test_reference=
"docs/ab_test_ast_vs_language_first_2026-02-25.md", success=true.

Full sprint regression: steps 1849-1853 pass. Existing intake tests unaffected.

Tests (8): struct constructable, steps_completed==5, failure_addressed non-empty,
ab_test_reference non-empty, RequirementsParser works, ArchitectIntakeProcessor
works, end-to-end produces >=7 requirements, success==true.

---

## Architecture Gate

- Header-only C++, ≤ 600 lines per new file
- No new vcpkg dependencies (use `<string>`, `<vector>`, standard scanning only)
- External schema of `whetstone_architect_intake` must not change
- Existing constraint-detection tests from prior sprints must still pass
- The exact failing A/B test spec must be the primary integration test input