Add sprint 163-165 plans for max-strict MCP grammar completion
This commit is contained in:
Bill
104
sprint163_plan.md
Normal file
104
sprint163_plan.md
Normal file
@@ -0,0 +1,104 @@
|
|||||||
|
# Sprint 163 Plan: Strict Grammar Foundation and Schema Normalization
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Current MCP grammar coverage is complete (347 tools), but strictness is not maximal.
|
||||||
|
The grammar generator still falls back to broad productions (`any-object`,
|
||||||
|
`any-array`, `any-value`) for many nested argument shapes, which permits syntactic
|
||||||
|
validity without full schema-level correctness.
|
||||||
|
|
||||||
|
This sprint builds the strictness foundation:
|
||||||
|
|
||||||
|
1. define a measurable strictness score
|
||||||
|
2. normalize per-tool schemas into a canonical, recursion-safe form
|
||||||
|
3. refactor grammar generation so strict handling is possible for every schema node
|
||||||
|
|
||||||
|
No MCP tool names change. Tool contracts stay backward-compatible at the API level.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
1. Add strictness metrics for all tool schemas and generated grammars
|
||||||
|
2. Introduce canonical schema normalization for recursive codegen
|
||||||
|
3. Eliminate fallback productions at top-level tool arguments
|
||||||
|
4. Keep generated artifacts deterministic across runs
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
### Step 1859: Strictness audit utility and baseline report (10 tests)
|
||||||
|
|
||||||
|
Add `tools/mcp/audit_grammar_strictness.py`.
|
||||||
|
|
||||||
|
Input:
|
||||||
|
- `tools/mcp/whetstone_tool_schemas.json`
|
||||||
|
- `tools/mcp/grammars/per_tool_schemas.json`
|
||||||
|
- `tools/mcp/grammars/*.gbnf`
|
||||||
|
|
||||||
|
Output:
|
||||||
|
- `tools/mcp/grammars/strictness_report.json`
|
||||||
|
- per-tool metrics: fallback count, nested coverage, enum coverage, unsupported keywords
|
||||||
|
|
||||||
|
Tests (10): deterministic output, counts align with 347 tools, missing-file handling,
|
||||||
|
invalid schema handling, fallback detection, enum detection, nested object detection,
|
||||||
|
array-item strictness detection, summary rollup correctness, JSON schema for report.
|
||||||
|
|
||||||
|
### Step 1860: Canonical schema normalizer (12 tests)
|
||||||
|
|
||||||
|
Add `tools/mcp/schema_normalizer.py`.
|
||||||
|
|
||||||
|
Normalize each tool schema into a canonical AST-like structure:
|
||||||
|
- explicit node kinds: object, array, string, integer, number, boolean, null
|
||||||
|
- preserved constraints: required, additionalProperties, enum, const, pattern,
|
||||||
|
min/max, minItems/maxItems, oneOf/anyOf/allOf
|
||||||
|
- `$ref` expansion within document scope
|
||||||
|
|
||||||
|
Tests (12): nested object normalization, array-of-object normalization, enum/const
|
||||||
|
preservation, required/optional split, additionalProperties preservation, oneOf,
|
||||||
|
anyOf, allOf flattening semantics, `$ref` resolution, cycle guard, stable key order,
|
||||||
|
unsupported keyword capture, malformed schema fallback with diagnostics.
|
||||||
|
|
||||||
|
### Step 1861: Refactor grammar generator to use normalized schema IR (10 tests)
|
||||||
|
|
||||||
|
Update `tools/mcp/generate_tool_grammars.py` to consume normalized schema IR from
|
||||||
|
Step 1860 rather than ad-hoc field inspection.
|
||||||
|
|
||||||
|
No strict keyword expansion yet; this step is structural refactor only.
|
||||||
|
|
||||||
|
Tests (10): generated grammar parity for simple tools, deterministic ordering,
|
||||||
|
dispatch includes all tools, per-tool files still generated for all tools,
|
||||||
|
no regression in existing enum behavior, integer/number separation preserved,
|
||||||
|
boolean/null handling preserved, optional field handling preserved, required field
|
||||||
|
ordering stable, script exit code semantics unchanged.
|
||||||
|
|
||||||
|
### Step 1862: Top-level strictness gate (8 tests)
|
||||||
|
|
||||||
|
Disallow `any-object`/`any-array` at top-level argument properties unless explicitly
|
||||||
|
annotated with `x-whetstone-allow-broad: true` in schema metadata.
|
||||||
|
|
||||||
|
Tests (8): gate fails on broad top-level property, gate passes when annotated,
|
||||||
|
gate passes when strict object spec exists, failure message includes tool+field,
|
||||||
|
report includes waiver count, waivers are deterministic, unknown annotation ignored,
|
||||||
|
CI-friendly nonzero exit on violation.
|
||||||
|
|
||||||
|
### Step 1863: Sprint 163 Integration Summary (8 tests)
|
||||||
|
|
||||||
|
Add `editor/src/Sprint163IntegrationSummary.h`.
|
||||||
|
Record: steps_completed=5 (1859-1863), strictness_baseline_established=true,
|
||||||
|
top_level_broad_fallback_blocked=true, success=true.
|
||||||
|
|
||||||
|
Tests (8): constructable summary struct, steps_completed==5, baseline artifact exists,
|
||||||
|
normalizer available, generator refactor active, strictness gate active, no tool-count
|
||||||
|
regression (347), success==true.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Gate
|
||||||
|
|
||||||
|
- No MCP runtime behavior changes; this sprint is tooling + generation pipeline only
|
||||||
|
- Generated grammar artifacts remain in `tools/mcp/grammars/`
|
||||||
|
- Deterministic generation is mandatory (stable ordering)
|
||||||
|
- All strictness checks must be machine-readable JSON
|
||||||
|
|
||||||
100
sprint164_plan.md
Normal file
100
sprint164_plan.md
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
# Sprint 164 Plan: Recursive Max-Strict GBNF and JSON Schema Dispatch
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Sprint 163 establishes strictness metrics and normalized schemas. This sprint
|
||||||
|
implements maximal practical strictness in generated grammars by replacing broad
|
||||||
|
fallback productions with recursively generated rules across nested objects,
|
||||||
|
arrays, unions, and constraints.
|
||||||
|
|
||||||
|
Target: generated artifacts should encode real argument shape constraints for each
|
||||||
|
tool, not only top-level scalar types.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
1. Generate recursive GBNF rules for nested object/array structures
|
||||||
|
2. Support union schemas (`oneOf`, `anyOf`, `allOf`) in both GBNF and JSON Schema
|
||||||
|
3. Encode common validation constraints directly in grammar where feasible
|
||||||
|
4. Reduce broad fallback usage to waived exceptions only
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
### Step 1864: Recursive object/array rule generator (12 tests)
|
||||||
|
|
||||||
|
Extend `tools/mcp/generate_tool_grammars.py` with recursive rule emission:
|
||||||
|
- object properties become named subrules
|
||||||
|
- arrays become item-specific list rules
|
||||||
|
- nested object arrays generate composed rules
|
||||||
|
- tuple arrays (`prefixItems`) supported where present
|
||||||
|
|
||||||
|
Tests (12): nested object grammar, nested array grammar, array-of-object grammar,
|
||||||
|
object-in-array-in-object grammar, empty-object behavior, optional nested fields,
|
||||||
|
required nested fields, unique rule naming, no name collisions, deterministic output,
|
||||||
|
deep nesting limit guard, parseability of generated grammar.
|
||||||
|
|
||||||
|
### Step 1865: Union support (`oneOf`/`anyOf`/`allOf`) (10 tests)
|
||||||
|
|
||||||
|
Add union handling from normalized schema:
|
||||||
|
- `oneOf`: exclusive grammar alternatives
|
||||||
|
- `anyOf`: permissive alternatives with merged required handling
|
||||||
|
- `allOf`: merged intersection object for compatible branches
|
||||||
|
|
||||||
|
Update `dispatch_schema.json` generation to preserve exact union semantics.
|
||||||
|
|
||||||
|
Tests (10): oneOf generation, anyOf generation, allOf merge, conflicting allOf
|
||||||
|
detection, dispatch schema oneOf count preserved, per-tool union schema validity,
|
||||||
|
deterministic alternative ordering, error diagnostics for unsupported mixed unions,
|
||||||
|
no regression on non-union tools, parser acceptance for valid branch samples.
|
||||||
|
|
||||||
|
### Step 1866: Constraint encoding and strict scalar handling (10 tests)
|
||||||
|
|
||||||
|
Implement strict handling for:
|
||||||
|
- `const`, `enum`
|
||||||
|
- `pattern` (string)
|
||||||
|
- `minLength`/`maxLength` (string, with bounded grammar where feasible)
|
||||||
|
- `minimum`/`maximum`, `exclusiveMinimum`/`exclusiveMaximum`
|
||||||
|
- `minItems`/`maxItems`
|
||||||
|
- `additionalProperties: false` object closure
|
||||||
|
|
||||||
|
Tests (10): enum/const enforced, bounded integer range handling, min/max items,
|
||||||
|
closed object rejects unknown keys, pattern passthrough metadata captured when not
|
||||||
|
grammatically representable, length constraints encoded or explicitly reported,
|
||||||
|
numeric constraint metadata propagation, strict scalar no-fallback on supported
|
||||||
|
cases, diagnostics for unsupported constraints, deterministic behavior.
|
||||||
|
|
||||||
|
### Step 1867: Strictness regression suite (8 tests)
|
||||||
|
|
||||||
|
Add `tools/mcp/test_strict_grammar_roundtrip.py` and golden fixtures for a focused
|
||||||
|
tool subset with complex arguments (e.g. `whetstone_set_environment`,
|
||||||
|
`whetstone_mutate`, `whetstone_generate_taskitems`, `whetstone_run_pipeline`,
|
||||||
|
`whetstone_validate_taskitem`).
|
||||||
|
|
||||||
|
Tests (8): valid samples accepted, invalid shape rejected, invalid enum rejected,
|
||||||
|
invalid nested field rejected, unknown property rejected when closed, array item
|
||||||
|
type mismatch rejected, union mismatch rejected, golden output unchanged unless
|
||||||
|
intentional update.
|
||||||
|
|
||||||
|
### Step 1868: Sprint 164 Integration Summary (8 tests)
|
||||||
|
|
||||||
|
Add `editor/src/Sprint164IntegrationSummary.h`.
|
||||||
|
Record: steps_completed=5 (1864-1868), recursive_codegen_enabled=true,
|
||||||
|
union_support_enabled=true, strict_fallback_reduction_verified=true, success=true.
|
||||||
|
|
||||||
|
Tests (8): constructable summary struct, steps_completed==5, recursive generation
|
||||||
|
active, union support active, strictness report improved vs Sprint 163 baseline,
|
||||||
|
fixtures pass, dispatch schema still covers all tools, success==true.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Gate
|
||||||
|
|
||||||
|
- Keep generator output deterministic
|
||||||
|
- No removal of existing tools or contract fields
|
||||||
|
- Unsupported JSON Schema constructs must be explicitly reported, never silently
|
||||||
|
downgraded to broad fallback without a waiver reason
|
||||||
|
- Grammar generation runtime remains practical for 347 tools
|
||||||
|
|
||||||
96
sprint165_plan.md
Normal file
96
sprint165_plan.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
# Sprint 165 Plan: Enforcement, CI Gates, and Runtime Adoption of Max-Strict Grammars
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
After strict generation is implemented (Sprints 163-164), the remaining risk is
|
||||||
|
drift: schemas may evolve while grammar strictness regresses silently. This sprint
|
||||||
|
adds hard enforcement in CI and runtime preflight so strictness stays durable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
1. Add CI gates that fail on strictness regressions
|
||||||
|
2. Version and fingerprint generated grammar artifacts
|
||||||
|
3. Validate runtime compatibility between MCP tool surface and strict artifacts
|
||||||
|
4. Provide an operational path for constrained decoding consumers
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
### Step 1869: Strictness policy file and threshold gate (10 tests)
|
||||||
|
|
||||||
|
Add `tools/mcp/grammars/strictness_policy.json` with hard thresholds:
|
||||||
|
- tool coverage must remain 100%
|
||||||
|
- waived broad fallbacks <= configured cap (target near-zero)
|
||||||
|
- unsupported keyword count non-increasing unless explicitly approved
|
||||||
|
|
||||||
|
Add `tools/mcp/check_strictness_policy.py` to fail on policy violations.
|
||||||
|
|
||||||
|
Tests (10): threshold pass/fail behavior, coverage regression detection, waiver cap,
|
||||||
|
unsupported keyword regression detection, policy parse errors, missing report errors,
|
||||||
|
stable diagnostics, machine-readable failure output, exit code correctness,
|
||||||
|
backward-compatible policy extension handling.
|
||||||
|
|
||||||
|
### Step 1870: Artifact fingerprinting and drift detection (8 tests)
|
||||||
|
|
||||||
|
Add generator outputs:
|
||||||
|
- `tools/mcp/grammars/manifest.json` (tool count, schema hash, grammar hash, timestamp)
|
||||||
|
- `tools/mcp/grammars/manifest.lock` (deterministic lock digest)
|
||||||
|
|
||||||
|
Add `tools/mcp/verify_grammar_manifest.py` to compare generated artifacts against
|
||||||
|
current `whetstone_tool_schemas.json`.
|
||||||
|
|
||||||
|
Tests (8): manifest creation, deterministic lock stability, mismatch detection on
|
||||||
|
schema change, mismatch detection on grammar edit, tool-count drift detection,
|
||||||
|
timestamp isolation from hash, lock verify pass path, lock verify fail path.
|
||||||
|
|
||||||
|
### Step 1871: CI integration in build/test flow (8 tests)
|
||||||
|
|
||||||
|
Wire strictness checks into project validation scripts:
|
||||||
|
- run grammar regeneration
|
||||||
|
- run strictness audit
|
||||||
|
- run policy check
|
||||||
|
- run manifest verify
|
||||||
|
|
||||||
|
Fail CI on any violation.
|
||||||
|
|
||||||
|
Tests (8): happy-path CI pass, strictness regression CI fail, manifest drift CI fail,
|
||||||
|
missing grammar file CI fail, unsupported keyword growth CI fail, waived fallback cap
|
||||||
|
CI fail, clear failure summaries, reproducible local invocation.
|
||||||
|
|
||||||
|
### Step 1872: Runtime preflight strict compatibility check (10 tests)
|
||||||
|
|
||||||
|
Update `tools/mcp/validate_slm_runtime.sh` to optionally enforce strict artifacts:
|
||||||
|
- verify local MCP tool surface matches manifest tool list
|
||||||
|
- verify dispatch schema and per-tool schemas loadable
|
||||||
|
- verify representative constrained tool call samples validate
|
||||||
|
|
||||||
|
Add `--strict-grammars` mode (default on for CI, optional for local).
|
||||||
|
|
||||||
|
Tests (10): strict mode success, strict mode fail on missing grammar, fail on tool
|
||||||
|
count mismatch, fail on schema mismatch hash, pass on aligned artifacts, readable
|
||||||
|
error output, non-strict mode backward compatibility, sample validation execution,
|
||||||
|
timeout-safe behavior, exit code correctness.
|
||||||
|
|
||||||
|
### Step 1873: Sprint 165 Integration Summary (8 tests)
|
||||||
|
|
||||||
|
Add `editor/src/Sprint165IntegrationSummary.h`.
|
||||||
|
Record: steps_completed=5 (1869-1873), ci_gate_active=true,
|
||||||
|
manifest_drift_detection_active=true, runtime_strict_preflight_active=true,
|
||||||
|
success=true.
|
||||||
|
|
||||||
|
Tests (8): constructable summary struct, steps_completed==5, policy gate active,
|
||||||
|
manifest verification active, CI hook active, runtime strict mode active,
|
||||||
|
no coverage regression vs 347-tool baseline, success==true.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Gate
|
||||||
|
|
||||||
|
- CI checks must be runnable locally and in automation without network dependency
|
||||||
|
- Generated artifacts remain committed and reproducible
|
||||||
|
- Strictness policy changes require explicit commit-level acknowledgment
|
||||||
|
- No runtime tool contract changes introduced in this sprint
|
||||||
|
|
||||||
Reference in New Issue
Block a user