diff --git a/docs/constructive_editing_runtime_plan.md b/docs/constructive_editing_runtime_plan.md index af5f085..7fb8d3b 100644 --- a/docs/constructive_editing_runtime_plan.md +++ b/docs/constructive_editing_runtime_plan.md @@ -150,8 +150,13 @@ Parallel planning tranche (active): - `logs/taskitem_runs/TEST_ONLY_spec_planning_baseline_20260226/deterministic_spec_readiness.json` Planning runtime controls (active): -- `sprint232_plan.md` to `sprint235_plan.md` +- `sprint232_plan.md` to `sprint237_plan.md` - semantic bridge + intake augmentation + requirement injection + expansion gating are wired into: - `tools/mcp/run_sprint_taskitem_pipeline.sh` - current policy default is native-first semantic fallback: - `WSTONE_SEMANTIC_TASK_EXPANSION_MODE=fallback_only` +- fallback diagnostics + remediation metadata are emitted by: + - `tools/mcp/analyze_semantic_fallback_gaps.py` +- fallback budget gating is available via: + - `tools/mcp/check_semantic_fallback_budget.py` + - `tools/mcp/run_semantic_fallback_budget_gate.sh` diff --git a/docs/generator_readiness_gap_registry_2026-02-26.md b/docs/generator_readiness_gap_registry_2026-02-26.md index 2ac1f7a..edc963b 100644 --- a/docs/generator_readiness_gap_registry_2026-02-26.md +++ b/docs/generator_readiness_gap_registry_2026-02-26.md @@ -46,6 +46,7 @@ This is the canonical dated registry for "not production-ready" generator gaps. | GR-017 | Full-stack contract propagation gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/fullstack_contract_closure.json` | `partial` | Full-stack contract packet + strict validation added in benchmark orchestration and summary aggregation. Contract invalid rate is `0%` on valid catalog, but cross-layer semantic consistency is not yet equivalence-verified. | Semantic propagation verifier sprint | | GR-018 | Performance/security/rollout constrained refactor enforcement gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/results.jsonl` | `partial` | Hard checks now enforce migration rollback + data-loss policy, security deny-by-default, SLO p95 presence, and rollout staged+abort policy. Enforcement is contract-level; generator capability under these constraints is still weak in C++ AB path. | Constraint-aware generation follow-up | | GR-019 | Parity-blocked readiness load (gating without capability closure) | `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r7/summary.json`, `logs/taskitem_runs/challenging_subset_prod_20260226_r7/summary.json`, `docs/sprint225_227_execution_tracker_2026-02-26.md` | `partial` | Sprint 225-227 reduced blocked parity load from `6` to `0` on both tracked hard catalogs while keeping unresolved divergence at `0`. This closes immediate safety debt for current corpora, but robustness is still contingent on pattern-driven repair classes. | Generalize repairs beyond queue-shaped transpile outputs | +| GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Current measured slice remains `fallback_rate=1.0`, so gating now detects but does not yet fix root capability gaps. | Native decomposition upgrade + semantic rationale enforcement | ## What Was Covered Today (Sprints 175-184) diff --git a/docs/progress_log_2026-02-26.md b/docs/progress_log_2026-02-26.md index d0b968f..f2acf12 100644 --- a/docs/progress_log_2026-02-26.md +++ b/docs/progress_log_2026-02-26.md @@ -123,3 +123,27 @@ - `logs/taskitem_runs/TEST_ONLY_sprint235_probe_20260226_144846/00_summary.json` - `logs/taskitem_runs/TEST_ONLY_sprint235_probe_20260226_144854/00_summary.json` - `logs/taskitem_runs/TEST_ONLY_sprint235_richspec_20260226_144902/00_summary.json` + +## Sprint 236 Added (Same Day) + +- Added semantic fallback gap audit tool: + - `tools/mcp/analyze_semantic_fallback_gaps.py` +- Emits per-run metadata for remediation routing: + - `recommended_tools` + - `recommended_taskitem_constraints` +- Generated dated audit bundle: + - `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json` + - `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_tooling_recommendations.json` +- Current measured signal on sprint 235 sample runs: + - `fallback_rate=1.0` + - dominant root cause: `native_decomposition_and_semantic_signal_deficit` + +## Sprint 237 Added (Same Day) + +- Added semantic fallback budget gate: + - `tools/mcp/check_semantic_fallback_budget.py` +- Added single-command audit+gate wrapper: + - `tools/mcp/run_semantic_fallback_budget_gate.sh` +- Dated pass/fail artifacts captured: + - fail (strict budget): `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json` + - pass (relaxed budget): `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_pass_20260226/semantic_fallback_budget_gate.json` diff --git a/docs/sprint236_execution_tracker_2026-02-26.md b/docs/sprint236_execution_tracker_2026-02-26.md new file mode 100644 index 0000000..50d1c8b --- /dev/null +++ b/docs/sprint236_execution_tracker_2026-02-26.md @@ -0,0 +1,34 @@ +# Sprint 236 Execution Tracker - 2026-02-26 + +## Scope +- `sprint236_plan.md` + +## Implemented + +- Added semantic fallback audit tool: + - `tools/mcp/analyze_semantic_fallback_gaps.py` +- Analyzer outputs: + - `semantic_fallback_records.jsonl` + - `semantic_fallback_summary.json` + - `semantic_fallback_tooling_recommendations.json` +- Added deterministic root-cause classes: + - `native_decomposition_too_shallow` + - `semantic_rationale_missing` + - `native_decomposition_and_semantic_signal_deficit` +- Added per-run remediation metadata: + - `recommended_tools` + - `recommended_taskitem_constraints` + +## Baseline Audit Artifact + +- `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json` +- `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_tooling_recommendations.json` + +Observed in this slice: +- `record_count=5` +- `fallback_rate=1.0` +- dominant root cause: `native_decomposition_and_semantic_signal_deficit` + +## Explicit Completion Signal + +- Sprint 236: `DONE` (implemented + audit generated + dated metadata captured) diff --git a/docs/sprint237_execution_tracker_2026-02-26.md b/docs/sprint237_execution_tracker_2026-02-26.md new file mode 100644 index 0000000..40dc7cc --- /dev/null +++ b/docs/sprint237_execution_tracker_2026-02-26.md @@ -0,0 +1,26 @@ +# Sprint 237 Execution Tracker - 2026-02-26 + +## Scope +- `sprint237_plan.md` + +## Implemented + +- Added gate checker: + - `tools/mcp/check_semantic_fallback_budget.py` +- Added wrapper for audit+gate: + - `tools/mcp/run_semantic_fallback_budget_gate.sh` +- Added policy controls: + - `WSTONE_SEMANTIC_MAX_FALLBACK_RATE` (default `0.35`) + - `WSTONE_SEMANTIC_FALLBACK_MIN_RECORDS` (default `5`) + - `WSTONE_SEMANTIC_FALLBACK_INCLUDE_GLOB` (run selection) + +## Baseline Gate Artifacts + +- expected fail (strict budget): + - `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json` +- expected pass (relaxed budget): + - `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_pass_20260226/semantic_fallback_budget_gate.json` + +## Explicit Completion Signal + +- Sprint 237: `DONE` (implemented + pass/fail gate smoke verified) diff --git a/editor/src/Sprint236IntegrationSummary.h b/editor/src/Sprint236IntegrationSummary.h new file mode 100644 index 0000000..be0c801 --- /dev/null +++ b/editor/src/Sprint236IntegrationSummary.h @@ -0,0 +1,6 @@ +#pragma once + +// Sprint 236 integration summary: +// - Added semantic fallback gap analyzer over taskitem pipeline summaries. +// - Emits deterministic per-run root-cause classes and remediation metadata. +// - Produces dated audit artifacts for future sprint prioritization. diff --git a/editor/src/Sprint237IntegrationSummary.h b/editor/src/Sprint237IntegrationSummary.h new file mode 100644 index 0000000..1cd21d2 --- /dev/null +++ b/editor/src/Sprint237IntegrationSummary.h @@ -0,0 +1,6 @@ +#pragma once + +// Sprint 237 integration summary: +// - Added semantic fallback budget gate checker and wrapper runner. +// - Gate supports explicit fallback-rate budget and minimum sample enforcement. +// - Produces deterministic pass/fail artifact for CI/batch policy wiring. diff --git a/sprint236_plan.md b/sprint236_plan.md new file mode 100644 index 0000000..fec96bc --- /dev/null +++ b/sprint236_plan.md @@ -0,0 +1,11 @@ +# Sprint 236 Plan: Semantic Fallback Gap Audit + Tool Metadata + +## Goal +Produce deterministic telemetry showing where semantic fallback is compensating for weak native decomposition, and attach tool/constraint metadata to close those gaps. + +## Steps +- Step 2223: Add semantic fallback gap analyzer for pipeline summaries. +- Step 2224: Classify fallback root causes using native task and semantic-signal counts. +- Step 2225: Emit per-run remediation metadata: recommended tools + taskitem constraints. +- Step 2226: Generate dated TEST_ONLY audit artifacts from sprint 235 runs. +- Step 2227: Add `Sprint236IntegrationSummary.h` and execution tracker. diff --git a/sprint237_plan.md b/sprint237_plan.md new file mode 100644 index 0000000..d9e39de --- /dev/null +++ b/sprint237_plan.md @@ -0,0 +1,11 @@ +# Sprint 237 Plan: Semantic Fallback Budget Gate + +## Goal +Turn semantic fallback audit signal into an enforceable gate for CI and batch execution. + +## Steps +- Step 2228: Add fallback-budget gate checker. +- Step 2229: Add wrapper that runs audit + gate in one command. +- Step 2230: Support configurable fallback budget and minimum sample size. +- Step 2231: Emit deterministic gate artifact with pass/fail reason. +- Step 2232: Add `Sprint237IntegrationSummary.h` and execution tracker. diff --git a/tools/mcp/analyze_semantic_fallback_gaps.py b/tools/mcp/analyze_semantic_fallback_gaps.py new file mode 100755 index 0000000..86009c3 --- /dev/null +++ b/tools/mcp/analyze_semantic_fallback_gaps.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python3 +import argparse +import json +from dataclasses import dataclass, asdict +from pathlib import Path +from typing import Dict, List + + +@dataclass +class FallbackRecord: + run_id: str + input_file: str + timestamp: str + mode: str + enabled: bool + fallback_applied: bool + skipped_reason: str + native_task_count: int + native_semantic_signal_count: int + expanded_task_count: int + gap_class: str + likely_root_cause: str + recommended_tools: List[str] + recommended_taskitem_constraints: List[str] + + +def load_json(path: Path) -> Dict: + with path.open("r", encoding="utf-8") as f: + return json.load(f) + + +def classify_root_cause(native_task_count: int, native_semantic_signal_count: int) -> str: + if native_task_count <= 2 and native_semantic_signal_count == 0: + return "native_decomposition_and_semantic_signal_deficit" + if native_task_count <= 2: + return "native_decomposition_too_shallow" + if native_semantic_signal_count == 0: + return "semantic_rationale_missing" + return "native_decomposition_sufficient" + + +def classify_gap(expansion: Dict) -> str: + enabled = bool(expansion.get("enabled", False)) + fallback_applied = bool(expansion.get("fallback_applied", False)) + mode = str(expansion.get("mode", "")) + if not enabled: + return "semantic_expansion_disabled" + if fallback_applied: + return "semantic_fallback_triggered" + if mode == "fallback_only": + return "native_first_pass" + return "semantic_expansion_not_applied" + + +def recommendations(gap_class: str, root_cause: str) -> Dict[str, List[str]]: + if gap_class == "semantic_fallback_triggered": + tools = [ + "whetstone_architect_intake", + "whetstone_generate_taskitems", + "whetstone_validate_taskitem", + "whetstone_queue_ready", + ] + constraints = [ + "min_native_task_count>=5", + "require_semantic_reason_tokens=true", + "require_execution_specificity_score>=85", + "require_deterministic_rollback_replay_contract=true", + ] + if root_cause == "semantic_rationale_missing": + tools.append("whetstone_validate_taskitem") + constraints.append("require_reason_contains_semantic_risk_contract_capability=true") + return {"tools": tools, "constraints": constraints} + + if gap_class == "semantic_expansion_disabled": + return { + "tools": ["whetstone_generate_taskitems", "whetstone_queue_ready"], + "constraints": ["set_WSTONE_SEMANTIC_TASK_EXPANSION=1_for_complex_specs"], + } + + return { + "tools": ["whetstone_validate_taskitem"], + "constraints": ["monitor_semantic_fallback_rate<=0.35"], + } + + +def load_records(runs_root: Path, include_glob: str) -> List[FallbackRecord]: + records: List[FallbackRecord] = [] + for summary_path in sorted(runs_root.glob(f"{include_glob}/00_summary.json")): + try: + summary = load_json(summary_path) + except Exception: + continue + expansion = summary.get("semantic_task_expansion") or {} + if not isinstance(expansion, dict): + continue + + native_task_count = int(expansion.get("native_task_count", 0) or 0) + native_semantic_signal_count = int(expansion.get("native_semantic_signal_count", 0) or 0) + gap_class = classify_gap(expansion) + root_cause = classify_root_cause(native_task_count, native_semantic_signal_count) + rec = recommendations(gap_class, root_cause) + + records.append( + FallbackRecord( + run_id=summary_path.parent.name, + input_file=str(summary.get("input_file", "")), + timestamp=str(summary.get("timestamp", "")), + mode=str(expansion.get("mode", "")), + enabled=bool(expansion.get("enabled", False)), + fallback_applied=bool(expansion.get("fallback_applied", False)), + skipped_reason=str(expansion.get("skipped_reason", "")), + native_task_count=native_task_count, + native_semantic_signal_count=native_semantic_signal_count, + expanded_task_count=int(expansion.get("expanded_task_count", 0) or 0), + gap_class=gap_class, + likely_root_cause=root_cause, + recommended_tools=rec["tools"], + recommended_taskitem_constraints=rec["constraints"], + ) + ) + + return records + + +def write_json(path: Path, obj: Dict) -> None: + with path.open("w", encoding="utf-8") as f: + json.dump(obj, f, indent=2, sort_keys=True) + f.write("\n") + + +def write_jsonl(path: Path, rows: List[Dict]) -> None: + with path.open("w", encoding="utf-8") as f: + for row in rows: + f.write(json.dumps(row, sort_keys=True)) + f.write("\n") + + +def summarize(records: List[FallbackRecord]) -> Dict: + total = len(records) + fallback_count = sum(1 for r in records if r.fallback_applied) + enabled_count = sum(1 for r in records if r.enabled) + by_gap: Dict[str, int] = {} + by_root: Dict[str, int] = {} + tool_frequency: Dict[str, int] = {} + + for r in records: + by_gap[r.gap_class] = by_gap.get(r.gap_class, 0) + 1 + by_root[r.likely_root_cause] = by_root.get(r.likely_root_cause, 0) + 1 + for t in r.recommended_tools: + tool_frequency[t] = tool_frequency.get(t, 0) + 1 + + fallback_rate = (fallback_count / enabled_count) if enabled_count else 0.0 + return { + "record_count": total, + "enabled_count": enabled_count, + "fallback_applied_count": fallback_count, + "fallback_rate": round(fallback_rate, 4), + "gap_class_counts": by_gap, + "root_cause_counts": by_root, + "recommended_tool_frequency": tool_frequency, + "status": "ok" if total > 0 else "no_data", + } + + +def main() -> int: + parser = argparse.ArgumentParser(description="Analyze semantic fallback usage and emit remediation metadata.") + parser.add_argument("--runs-root", default="logs/taskitem_runs", help="Run root containing per-run 00_summary.json files") + parser.add_argument("--include-glob", default="*", help="Folder glob under runs-root to include") + parser.add_argument("--out-dir", required=True, help="Output directory for audit artifacts") + args = parser.parse_args() + + runs_root = Path(args.runs_root) + out_dir = Path(args.out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + + records = load_records(runs_root, args.include_glob) + rows = [asdict(r) for r in records] + write_jsonl(out_dir / "semantic_fallback_records.jsonl", rows) + + summary = summarize(records) + write_json(out_dir / "semantic_fallback_summary.json", summary) + + recommendations_payload = { + "top_recommendations": sorted( + summary.get("recommended_tool_frequency", {}).items(), + key=lambda kv: (-int(kv[1]), str(kv[0])), + ), + "records": rows, + } + write_json(out_dir / "semantic_fallback_tooling_recommendations.json", recommendations_payload) + + print( + json.dumps( + { + "status": summary.get("status", "no_data"), + "record_count": summary.get("record_count", 0), + "fallback_rate": summary.get("fallback_rate", 0.0), + "out_dir": str(out_dir), + }, + sort_keys=True, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/mcp/check_semantic_fallback_budget.py b/tools/mcp/check_semantic_fallback_budget.py new file mode 100755 index 0000000..072c23a --- /dev/null +++ b/tools/mcp/check_semantic_fallback_budget.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +import argparse +import json +from pathlib import Path +from typing import Dict + + +def load_json(path: Path) -> Dict: + with path.open("r", encoding="utf-8") as f: + return json.load(f) + + +def write_json(path: Path, obj: Dict) -> None: + with path.open("w", encoding="utf-8") as f: + json.dump(obj, f, indent=2, sort_keys=True) + f.write("\n") + + +def main() -> int: + parser = argparse.ArgumentParser(description="Check semantic fallback usage against a configured budget.") + parser.add_argument("--summary", required=True, help="Path to semantic_fallback_summary.json") + parser.add_argument("--out", required=True, help="Output gate evaluation JSON") + parser.add_argument("--max-fallback-rate", type=float, default=0.35, help="Max allowed fallback rate in [0,1]") + parser.add_argument("--min-record-count", type=int, default=5, help="Minimum records required before strict gating") + args = parser.parse_args() + + summary = load_json(Path(args.summary)) + fallback_rate = float(summary.get("fallback_rate", 0.0) or 0.0) + record_count = int(summary.get("record_count", 0) or 0) + + enforceable = record_count >= args.min_record_count + passed = True + reason = "insufficient_data" + if enforceable: + passed = fallback_rate <= args.max_fallback_rate + reason = "within_budget" if passed else "fallback_rate_exceeds_budget" + + result = { + "passed": bool(passed), + "reason": reason, + "enforceable": bool(enforceable), + "record_count": record_count, + "min_record_count": int(args.min_record_count), + "fallback_rate": fallback_rate, + "max_fallback_rate": float(args.max_fallback_rate), + } + write_json(Path(args.out), result) + + print(json.dumps(result, sort_keys=True)) + return 0 if passed else 9 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/mcp/run_semantic_fallback_budget_gate.sh b/tools/mcp/run_semantic_fallback_budget_gate.sh new file mode 100755 index 0000000..8de8618 --- /dev/null +++ b/tools/mcp/run_semantic_fallback_budget_gate.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +RUNS_ROOT="${WSTONE_RUNS_ROOT:-$ROOT_DIR/logs/taskitem_runs}" +INCLUDE_GLOB="${WSTONE_SEMANTIC_FALLBACK_INCLUDE_GLOB:-*}" +OUT_DIR="${1:-$RUNS_ROOT/TEST_ONLY_semantic_fallback_budget_gate_$(date +%Y%m%d_%H%M%S)}" +MAX_RATE="${WSTONE_SEMANTIC_MAX_FALLBACK_RATE:-0.35}" +MIN_RECORDS="${WSTONE_SEMANTIC_FALLBACK_MIN_RECORDS:-5}" + +mkdir -p "$OUT_DIR" + +python3 "$ROOT_DIR/tools/mcp/analyze_semantic_fallback_gaps.py" \ + --runs-root "$RUNS_ROOT" \ + --include-glob "$INCLUDE_GLOB" \ + --out-dir "$OUT_DIR" >/dev/null + +if python3 "$ROOT_DIR/tools/mcp/check_semantic_fallback_budget.py" \ + --summary "$OUT_DIR/semantic_fallback_summary.json" \ + --out "$OUT_DIR/semantic_fallback_budget_gate.json" \ + --max-fallback-rate "$MAX_RATE" \ + --min-record-count "$MIN_RECORDS" >/dev/null; then + jq -n \ + --arg out_dir "$OUT_DIR" \ + --argjson gate "$(cat "$OUT_DIR/semantic_fallback_budget_gate.json")" \ + '{status:"pass", out_dir:$out_dir, gate:$gate}' +else + jq -n \ + --arg out_dir "$OUT_DIR" \ + --argjson gate "$(cat "$OUT_DIR/semantic_fallback_budget_gate.json")" \ + '{status:"fail", out_dir:$out_dir, gate:$gate}' + exit 9 +fi