whetstone_DSL/sprint161_plan.md

Sprint 161 Plan: Fix whetstone_architect_intake — Requirements Extraction

Context

A/B test run on 2026-02-25 (see docs/ab_test_ast_vs_language_first_2026-02-25.md) exposed a concrete failure in whetstone_architect_intake:

The tool was given a markdown spec with three sections — Goal, Requirements, and Constraints. It returned only the two Constraints as normalizedRequirements. The entire Requirements section (struct fields, class methods, behaviour rules) was silently dropped.

Why this matters: whetstone_generate_taskitems receives normalizedRequirements as its input. When intake drops functional requirements, taskitems are generated from constraints alone — producing one generic placeholder task instead of a real work breakdown. The downstream agent cannot complete the program without a smart LLM filling the gap manually. That is the exact failure mode this sprint closes.

Root cause: The intake parser matches constraint-pattern sentences but does not scan bullet-point lists under headings like "Requirements", "Features", or "Behavior" for functional requirements. Field-list bullets (e.g. "WorkItem has: job_id (string), priority (int)") are not parsed at all.

No new MCP tools. This is a quality fix to existing behaviour. The input/output schema of whetstone_architect_intake is unchanged — only the content of normalizedRequirements improves.

---

Goals

1. Parse markdown bullet lists under recognised requirement headings into NormalizedRequirement objects with kind: "requirement" or kind: "feature"
2. Detect field-list syntax and emit one entry per field with kind: "field"
3. Wire the new parser into the existing intake handler without schema changes
4. Verify with the exact spec that failed in the A/B test: must produce ≥ 7 normalized requirements and ≥ 5 taskitems (not 1 generic placeholder)

---

Steps

Step 1849: RequirementsParser — extract bullet lists from requirement sections (12 tests)

Create editor/src/RequirementsParser.h.

Scans markdown for headings whose lowercased, hyphenated anchor matches a recognised set: requirements, features, behavior, behaviour, interface, api. For each matching section, reads bullet items (- text or * text) and produces one ParsedRequirement per item.

Detects field-list syntax: a bullet containing a colon followed by a comma-separated list (e.g. - WorkItem has: job_id (string), priority (int)) is split into one ParsedRequirement per field with isField: true, fieldName, and fieldType populated.

struct ParsedRequirement {
    std::string text;          // normalized lowercase
    std::string sourceText;    // original as written
    int         sourceLine;
    std::string sectionAnchor;
    std::string sectionTitle;
    bool        isField;
    std::string fieldName;     // when isField == true
    std::string fieldType;     // when isField == true
};

class RequirementsParser {
public:
    static std::vector<ParsedRequirement> parse(const std::string& markdown);
    static std::string anchorize(const std::string& heading);
    static bool isRequirementHeading(const std::string& anchor);
};

Tests (12): empty markdown returns empty, single bullet under Requirements heading returns one entry, multiple bullets return multiple entries, bullet under Constraints heading NOT returned (handled by existing parser), anchorize lowercases and hyphenates, isRequirementHeading true for "requirements" and "features", isRequirementHeading false for "constraints" and "goal", field-list bullet produces multiple entries with isField==true, fieldName extracted correctly, fieldType extracted correctly, sourceLine correct, sectionTitle preserved.

Step 1850: ArchitectIntakeProcessor — merge all requirement kinds (10 tests)

Create editor/src/ArchitectIntakeProcessor.h.

Centralises the intake parsing logic currently inline in the MCP handler. Calls both the existing constraint/conflict detection and RequirementsParser. Merges results into one normalizedRequirements vector with all kinds present.

struct IntakeResult {
    std::vector<nlohmann::json> normalizedRequirements; // kind: requirement/feature/field/constraint
    std::vector<nlohmann::json> conflicts;
    nlohmann::json              parsedSpec;
    int                         requirementCount;
    int                         fieldCount;
    int                         constraintCount;
    int                         conflictCount;
    bool                        hasConflicts;
};

class ArchitectIntakeProcessor {
public:
    static IntakeResult process(const std::string& markdown);
};

Tests (10): process returns IntakeResult, constraintCount matches constraint bullets, requirementCount matches requirement bullets, fieldCount matches field-list extractions, kind=="field" entries present for field-list bullets, kind=="constraint" entries still present, conflictCount==0 when no conflicts, spec with only constraints produces no requirement entries, all entries have requirementId assigned, ambiguous==false for unambiguous bullets.

Step 1851: Wire ArchitectIntakeProcessor into whetstone_architect_intake handler (8 tests)

Update the MCP handler for whetstone_architect_intake to call ArchitectIntakeProcessor::process() instead of inline parsing.

No schema change — inputSchema unchanged, response field names unchanged. Update description in tools/claude/tools.json to state that functional requirements from bullet lists are now extracted alongside constraints.

Tests (8): tool still registered, inputSchema unchanged, response has normalizedRequirements array, response has conflicts array, response has parsedSpec, requirement entries present when Requirements section given, field entries present for field-list bullets, constraint entries still present.

Step 1852: End-to-end validation with the failing A/B test spec (10 tests)

Create editor/src/Sprint161EndToEnd.h.

Uses the exact spec from the failing 2026-02-25 A/B run as the primary input:

# Priority Work Queue
## Requirements
- WorkItem has: job_id (string), priority (int), payload (string)
- PriorityQueue supports: enqueue(item), dequeue() -> item, peek() -> item, size(), empty()
- Dequeue always returns the highest-priority item
- JSON serialization via nlohmann
## Constraints
- Header-only implementation
- No external dependencies beyond nlohmann and STL

Expected: normalizedRequirements contains ≥ 7 entries. Feeding the result to whetstone_generate_taskitems must return ≥ 5 tasks. No task may have the title "Intake Foundation Primary" (the generic fallback from the failed run).

Tests (10): intake produces > 2 normalizedRequirements, WorkItem field entries present (kind=="field"), PriorityQueue entry present (kind=="requirement"), constraint entries present (kind=="constraint"), no conflicts, taskitems call returns success, taskCount >= 5, no task titled "Intake Foundation Primary", at least one task has queueReady==true, planSummary.milestoneCount >= 1.

Step 1853: Sprint 161 Integration Summary (8 tests)

Create editor/src/Sprint161IntegrationSummary.h. Record: steps_completed=5 (1849-1853), failure_addressed="architect_intake silently drops Requirements section bullet lists", ab_test_reference= "docs/ab_test_ast_vs_language_first_2026-02-25.md", success=true.

Full sprint regression: steps 1849-1853 pass. Existing intake tests unaffected.

Tests (8): struct constructable, steps_completed==5, failure_addressed non-empty, ab_test_reference non-empty, RequirementsParser works, ArchitectIntakeProcessor works, end-to-end produces >=7 requirements, success==true.

---

Architecture Gate

• Header-only C++, ≤ 600 lines per new file
• No new vcpkg dependencies (use <string>, <vector>, standard scanning only)
• External schema of whetstone_architect_intake must not change
• Existing constraint-detection tests from prior sprints must still pass
• The exact failing A/B test spec must be the primary integration test input