diff --git a/sprint163_plan.md b/sprint163_plan.md new file mode 100644 index 0000000..80a92dd --- /dev/null +++ b/sprint163_plan.md @@ -0,0 +1,104 @@ +# Sprint 163 Plan: Strict Grammar Foundation and Schema Normalization + +## Context + +Current MCP grammar coverage is complete (347 tools), but strictness is not maximal. +The grammar generator still falls back to broad productions (`any-object`, +`any-array`, `any-value`) for many nested argument shapes, which permits syntactic +validity without full schema-level correctness. + +This sprint builds the strictness foundation: + +1. define a measurable strictness score +2. normalize per-tool schemas into a canonical, recursion-safe form +3. refactor grammar generation so strict handling is possible for every schema node + +No MCP tool names change. Tool contracts stay backward-compatible at the API level. + +--- + +## Goals + +1. Add strictness metrics for all tool schemas and generated grammars +2. Introduce canonical schema normalization for recursive codegen +3. Eliminate fallback productions at top-level tool arguments +4. Keep generated artifacts deterministic across runs + +--- + +## Steps + +### Step 1859: Strictness audit utility and baseline report (10 tests) + +Add `tools/mcp/audit_grammar_strictness.py`. + +Input: +- `tools/mcp/whetstone_tool_schemas.json` +- `tools/mcp/grammars/per_tool_schemas.json` +- `tools/mcp/grammars/*.gbnf` + +Output: +- `tools/mcp/grammars/strictness_report.json` +- per-tool metrics: fallback count, nested coverage, enum coverage, unsupported keywords + +Tests (10): deterministic output, counts align with 347 tools, missing-file handling, +invalid schema handling, fallback detection, enum detection, nested object detection, +array-item strictness detection, summary rollup correctness, JSON schema for report. + +### Step 1860: Canonical schema normalizer (12 tests) + +Add `tools/mcp/schema_normalizer.py`. + +Normalize each tool schema into a canonical AST-like structure: +- explicit node kinds: object, array, string, integer, number, boolean, null +- preserved constraints: required, additionalProperties, enum, const, pattern, + min/max, minItems/maxItems, oneOf/anyOf/allOf +- `$ref` expansion within document scope + +Tests (12): nested object normalization, array-of-object normalization, enum/const +preservation, required/optional split, additionalProperties preservation, oneOf, +anyOf, allOf flattening semantics, `$ref` resolution, cycle guard, stable key order, +unsupported keyword capture, malformed schema fallback with diagnostics. + +### Step 1861: Refactor grammar generator to use normalized schema IR (10 tests) + +Update `tools/mcp/generate_tool_grammars.py` to consume normalized schema IR from +Step 1860 rather than ad-hoc field inspection. + +No strict keyword expansion yet; this step is structural refactor only. + +Tests (10): generated grammar parity for simple tools, deterministic ordering, +dispatch includes all tools, per-tool files still generated for all tools, +no regression in existing enum behavior, integer/number separation preserved, +boolean/null handling preserved, optional field handling preserved, required field +ordering stable, script exit code semantics unchanged. + +### Step 1862: Top-level strictness gate (8 tests) + +Disallow `any-object`/`any-array` at top-level argument properties unless explicitly +annotated with `x-whetstone-allow-broad: true` in schema metadata. + +Tests (8): gate fails on broad top-level property, gate passes when annotated, +gate passes when strict object spec exists, failure message includes tool+field, +report includes waiver count, waivers are deterministic, unknown annotation ignored, +CI-friendly nonzero exit on violation. + +### Step 1863: Sprint 163 Integration Summary (8 tests) + +Add `editor/src/Sprint163IntegrationSummary.h`. +Record: steps_completed=5 (1859-1863), strictness_baseline_established=true, +top_level_broad_fallback_blocked=true, success=true. + +Tests (8): constructable summary struct, steps_completed==5, baseline artifact exists, +normalizer available, generator refactor active, strictness gate active, no tool-count +regression (347), success==true. + +--- + +## Architecture Gate + +- No MCP runtime behavior changes; this sprint is tooling + generation pipeline only +- Generated grammar artifacts remain in `tools/mcp/grammars/` +- Deterministic generation is mandatory (stable ordering) +- All strictness checks must be machine-readable JSON + diff --git a/sprint164_plan.md b/sprint164_plan.md new file mode 100644 index 0000000..b1f9c6b --- /dev/null +++ b/sprint164_plan.md @@ -0,0 +1,100 @@ +# Sprint 164 Plan: Recursive Max-Strict GBNF and JSON Schema Dispatch + +## Context + +Sprint 163 establishes strictness metrics and normalized schemas. This sprint +implements maximal practical strictness in generated grammars by replacing broad +fallback productions with recursively generated rules across nested objects, +arrays, unions, and constraints. + +Target: generated artifacts should encode real argument shape constraints for each +tool, not only top-level scalar types. + +--- + +## Goals + +1. Generate recursive GBNF rules for nested object/array structures +2. Support union schemas (`oneOf`, `anyOf`, `allOf`) in both GBNF and JSON Schema +3. Encode common validation constraints directly in grammar where feasible +4. Reduce broad fallback usage to waived exceptions only + +--- + +## Steps + +### Step 1864: Recursive object/array rule generator (12 tests) + +Extend `tools/mcp/generate_tool_grammars.py` with recursive rule emission: +- object properties become named subrules +- arrays become item-specific list rules +- nested object arrays generate composed rules +- tuple arrays (`prefixItems`) supported where present + +Tests (12): nested object grammar, nested array grammar, array-of-object grammar, +object-in-array-in-object grammar, empty-object behavior, optional nested fields, +required nested fields, unique rule naming, no name collisions, deterministic output, +deep nesting limit guard, parseability of generated grammar. + +### Step 1865: Union support (`oneOf`/`anyOf`/`allOf`) (10 tests) + +Add union handling from normalized schema: +- `oneOf`: exclusive grammar alternatives +- `anyOf`: permissive alternatives with merged required handling +- `allOf`: merged intersection object for compatible branches + +Update `dispatch_schema.json` generation to preserve exact union semantics. + +Tests (10): oneOf generation, anyOf generation, allOf merge, conflicting allOf +detection, dispatch schema oneOf count preserved, per-tool union schema validity, +deterministic alternative ordering, error diagnostics for unsupported mixed unions, +no regression on non-union tools, parser acceptance for valid branch samples. + +### Step 1866: Constraint encoding and strict scalar handling (10 tests) + +Implement strict handling for: +- `const`, `enum` +- `pattern` (string) +- `minLength`/`maxLength` (string, with bounded grammar where feasible) +- `minimum`/`maximum`, `exclusiveMinimum`/`exclusiveMaximum` +- `minItems`/`maxItems` +- `additionalProperties: false` object closure + +Tests (10): enum/const enforced, bounded integer range handling, min/max items, +closed object rejects unknown keys, pattern passthrough metadata captured when not +grammatically representable, length constraints encoded or explicitly reported, +numeric constraint metadata propagation, strict scalar no-fallback on supported +cases, diagnostics for unsupported constraints, deterministic behavior. + +### Step 1867: Strictness regression suite (8 tests) + +Add `tools/mcp/test_strict_grammar_roundtrip.py` and golden fixtures for a focused +tool subset with complex arguments (e.g. `whetstone_set_environment`, +`whetstone_mutate`, `whetstone_generate_taskitems`, `whetstone_run_pipeline`, +`whetstone_validate_taskitem`). + +Tests (8): valid samples accepted, invalid shape rejected, invalid enum rejected, +invalid nested field rejected, unknown property rejected when closed, array item +type mismatch rejected, union mismatch rejected, golden output unchanged unless +intentional update. + +### Step 1868: Sprint 164 Integration Summary (8 tests) + +Add `editor/src/Sprint164IntegrationSummary.h`. +Record: steps_completed=5 (1864-1868), recursive_codegen_enabled=true, +union_support_enabled=true, strict_fallback_reduction_verified=true, success=true. + +Tests (8): constructable summary struct, steps_completed==5, recursive generation +active, union support active, strictness report improved vs Sprint 163 baseline, +fixtures pass, dispatch schema still covers all tools, success==true. + +--- + +## Architecture Gate + +- Keep generator output deterministic +- No removal of existing tools or contract fields +- Unsupported JSON Schema constructs must be explicitly reported, never silently + downgraded to broad fallback without a waiver reason +- Grammar generation runtime remains practical for 347 tools + diff --git a/sprint165_plan.md b/sprint165_plan.md new file mode 100644 index 0000000..d93d540 --- /dev/null +++ b/sprint165_plan.md @@ -0,0 +1,96 @@ +# Sprint 165 Plan: Enforcement, CI Gates, and Runtime Adoption of Max-Strict Grammars + +## Context + +After strict generation is implemented (Sprints 163-164), the remaining risk is +drift: schemas may evolve while grammar strictness regresses silently. This sprint +adds hard enforcement in CI and runtime preflight so strictness stays durable. + +--- + +## Goals + +1. Add CI gates that fail on strictness regressions +2. Version and fingerprint generated grammar artifacts +3. Validate runtime compatibility between MCP tool surface and strict artifacts +4. Provide an operational path for constrained decoding consumers + +--- + +## Steps + +### Step 1869: Strictness policy file and threshold gate (10 tests) + +Add `tools/mcp/grammars/strictness_policy.json` with hard thresholds: +- tool coverage must remain 100% +- waived broad fallbacks <= configured cap (target near-zero) +- unsupported keyword count non-increasing unless explicitly approved + +Add `tools/mcp/check_strictness_policy.py` to fail on policy violations. + +Tests (10): threshold pass/fail behavior, coverage regression detection, waiver cap, +unsupported keyword regression detection, policy parse errors, missing report errors, +stable diagnostics, machine-readable failure output, exit code correctness, +backward-compatible policy extension handling. + +### Step 1870: Artifact fingerprinting and drift detection (8 tests) + +Add generator outputs: +- `tools/mcp/grammars/manifest.json` (tool count, schema hash, grammar hash, timestamp) +- `tools/mcp/grammars/manifest.lock` (deterministic lock digest) + +Add `tools/mcp/verify_grammar_manifest.py` to compare generated artifacts against +current `whetstone_tool_schemas.json`. + +Tests (8): manifest creation, deterministic lock stability, mismatch detection on +schema change, mismatch detection on grammar edit, tool-count drift detection, +timestamp isolation from hash, lock verify pass path, lock verify fail path. + +### Step 1871: CI integration in build/test flow (8 tests) + +Wire strictness checks into project validation scripts: +- run grammar regeneration +- run strictness audit +- run policy check +- run manifest verify + +Fail CI on any violation. + +Tests (8): happy-path CI pass, strictness regression CI fail, manifest drift CI fail, +missing grammar file CI fail, unsupported keyword growth CI fail, waived fallback cap +CI fail, clear failure summaries, reproducible local invocation. + +### Step 1872: Runtime preflight strict compatibility check (10 tests) + +Update `tools/mcp/validate_slm_runtime.sh` to optionally enforce strict artifacts: +- verify local MCP tool surface matches manifest tool list +- verify dispatch schema and per-tool schemas loadable +- verify representative constrained tool call samples validate + +Add `--strict-grammars` mode (default on for CI, optional for local). + +Tests (10): strict mode success, strict mode fail on missing grammar, fail on tool +count mismatch, fail on schema mismatch hash, pass on aligned artifacts, readable +error output, non-strict mode backward compatibility, sample validation execution, +timeout-safe behavior, exit code correctness. + +### Step 1873: Sprint 165 Integration Summary (8 tests) + +Add `editor/src/Sprint165IntegrationSummary.h`. +Record: steps_completed=5 (1869-1873), ci_gate_active=true, +manifest_drift_detection_active=true, runtime_strict_preflight_active=true, +success=true. + +Tests (8): constructable summary struct, steps_completed==5, policy gate active, +manifest verification active, CI hook active, runtime strict mode active, +no coverage regression vs 347-tool baseline, success==true. + +--- + +## Architecture Gate + +- CI checks must be runnable locally and in automation without network dependency +- Generated artifacts remain committed and reproducible +- Strictness policy changes require explicit commit-level acknowledgment +- No runtime tool contract changes introduced in this sprint +