Add sprint 163-165 plans for max-strict MCP grammar completion

sprint163_plan.md

@@ -0,0 +1,104 @@
+# Sprint 163 Plan: Strict Grammar Foundation and Schema Normalization
+
+## Context
+
+Current MCP grammar coverage is complete (347 tools), but strictness is not maximal.
+The grammar generator still falls back to broad productions (`any-object`,
+`any-array`, `any-value`) for many nested argument shapes, which permits syntactic
+validity without full schema-level correctness.
+
+This sprint builds the strictness foundation:
+
+1. define a measurable strictness score
+2. normalize per-tool schemas into a canonical, recursion-safe form
+3. refactor grammar generation so strict handling is possible for every schema node
+
+No MCP tool names change. Tool contracts stay backward-compatible at the API level.
+
+---
+
+## Goals
+
+1. Add strictness metrics for all tool schemas and generated grammars
+2. Introduce canonical schema normalization for recursive codegen
+3. Eliminate fallback productions at top-level tool arguments
+4. Keep generated artifacts deterministic across runs
+
+---
+
+## Steps
+
+### Step 1859: Strictness audit utility and baseline report (10 tests)
+
+Add `tools/mcp/audit_grammar_strictness.py`.
+
+Input:
+- `tools/mcp/whetstone_tool_schemas.json`
+- `tools/mcp/grammars/per_tool_schemas.json`
+- `tools/mcp/grammars/*.gbnf`
+
+Output:
+- `tools/mcp/grammars/strictness_report.json`
+- per-tool metrics: fallback count, nested coverage, enum coverage, unsupported keywords
+
+Tests (10): deterministic output, counts align with 347 tools, missing-file handling,
+invalid schema handling, fallback detection, enum detection, nested object detection,
+array-item strictness detection, summary rollup correctness, JSON schema for report.
+
+### Step 1860: Canonical schema normalizer (12 tests)
+
+Add `tools/mcp/schema_normalizer.py`.
+
+Normalize each tool schema into a canonical AST-like structure:
+- explicit node kinds: object, array, string, integer, number, boolean, null
+- preserved constraints: required, additionalProperties, enum, const, pattern,
+  min/max, minItems/maxItems, oneOf/anyOf/allOf
+- `$ref` expansion within document scope
+
+Tests (12): nested object normalization, array-of-object normalization, enum/const
+preservation, required/optional split, additionalProperties preservation, oneOf,
+anyOf, allOf flattening semantics, `$ref` resolution, cycle guard, stable key order,
+unsupported keyword capture, malformed schema fallback with diagnostics.
+
+### Step 1861: Refactor grammar generator to use normalized schema IR (10 tests)
+
+Update `tools/mcp/generate_tool_grammars.py` to consume normalized schema IR from
+Step 1860 rather than ad-hoc field inspection.
+
+No strict keyword expansion yet; this step is structural refactor only.
+
+Tests (10): generated grammar parity for simple tools, deterministic ordering,
+dispatch includes all tools, per-tool files still generated for all tools,
+no regression in existing enum behavior, integer/number separation preserved,
+boolean/null handling preserved, optional field handling preserved, required field
+ordering stable, script exit code semantics unchanged.
+
+### Step 1862: Top-level strictness gate (8 tests)
+
+Disallow `any-object`/`any-array` at top-level argument properties unless explicitly
+annotated with `x-whetstone-allow-broad: true` in schema metadata.
+
+Tests (8): gate fails on broad top-level property, gate passes when annotated,
+gate passes when strict object spec exists, failure message includes tool+field,
+report includes waiver count, waivers are deterministic, unknown annotation ignored,
+CI-friendly nonzero exit on violation.
+
+### Step 1863: Sprint 163 Integration Summary (8 tests)
+
+Add `editor/src/Sprint163IntegrationSummary.h`.
+Record: steps_completed=5 (1859-1863), strictness_baseline_established=true,
+top_level_broad_fallback_blocked=true, success=true.
+
+Tests (8): constructable summary struct, steps_completed==5, baseline artifact exists,
+normalizer available, generator refactor active, strictness gate active, no tool-count
+regression (347), success==true.
+
+---
+
+## Architecture Gate
+
+- No MCP runtime behavior changes; this sprint is tooling + generation pipeline only
+- Generated grammar artifacts remain in `tools/mcp/grammars/`
+- Deterministic generation is mandatory (stable ordering)
+- All strictness checks must be machine-readable JSON
+