diff --git a/docs/constructive_editing_runtime_plan.md b/docs/constructive_editing_runtime_plan.md index 5a91ac4..176b5ff 100644 --- a/docs/constructive_editing_runtime_plan.md +++ b/docs/constructive_editing_runtime_plan.md @@ -150,7 +150,7 @@ Parallel planning tranche (active): - `logs/taskitem_runs/TEST_ONLY_spec_planning_baseline_20260226/deterministic_spec_readiness.json` Planning runtime controls (active): -- `sprint232_plan.md` to `sprint252_plan.md` +- `sprint232_plan.md` to `sprint253_plan.md` - semantic bridge + intake augmentation + requirement injection + expansion gating are wired into: - `tools/mcp/run_sprint_taskitem_pipeline.sh` - current policy default is native-first semantic fallback: @@ -207,3 +207,6 @@ Planning runtime controls (active): - closure ladder can route around historically ineffective raw-only mode: - `tools/mcp/analyze_raw_candidate_uplift_history.py` - `WSTONE_CLOSURE_LADDER_SKIP_RAW_WHEN_NO_UPLIFT` +- closure ladder outcomes can be aggregated in batch: + - `tools/mcp/run_native_profile_closure_ladder_batch.sh` + - `tools/mcp/analyze_closure_ladder_outcomes.py` diff --git a/docs/generator_readiness_gap_registry_2026-02-26.md b/docs/generator_readiness_gap_registry_2026-02-26.md index adc26c2..be30552 100644 --- a/docs/generator_readiness_gap_registry_2026-02-26.md +++ b/docs/generator_readiness_gap_registry_2026-02-26.md @@ -47,7 +47,7 @@ This is the canonical dated registry for "not production-ready" generator gaps. | GR-018 | Performance/security/rollout constrained refactor enforcement gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/results.jsonl` | `partial` | Hard checks now enforce migration rollback + data-loss policy, security deny-by-default, SLO p95 presence, and rollout staged+abort policy. Enforcement is contract-level; generator capability under these constraints is still weak in C++ AB path. | Constraint-aware generation follow-up | | GR-019 | Parity-blocked readiness load (gating without capability closure) | `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r7/summary.json`, `logs/taskitem_runs/challenging_subset_prod_20260226_r7/summary.json`, `docs/sprint225_227_execution_tracker_2026-02-26.md` | `partial` | Sprint 225-227 reduced blocked parity load from `6` to `0` on both tracked hard catalogs while keeping unresolved divergence at `0`. This closes immediate safety debt for current corpora, but robustness is still contingent on pattern-driven repair classes. | Generalize repairs beyond queue-shaped transpile outputs | | GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json`, `logs/taskitem_runs/TEST_ONLY_sprint239_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint240_retry_20260226_145631/00_summary.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md`, `docs/sprint238_execution_tracker_2026-02-26.md`, `docs/sprint239_execution_tracker_2026-02-26.md`, `docs/sprint240_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Sprint 238 added native decomposition hard gate in pipeline (`min task count`, `min semantic signal count`) so weak native generation can be blocked before fallback masking. Sprint 239 added native reason enrichment and improved semantic signal density (`0 -> 6`). Sprint 240 added native decomposition retry with explicit minimum-task policy; on current sample retry attempted but did not improve depth (`2 -> 2`). | Native decomposition task-depth upgrade (increase native task granularity beyond 2) | -| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`, `logs/taskitem_runs/TEST_ONLY_sprint250_closure_ladder_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint251_rawvariants_20260226_154355/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json` | `partial` | Sprint 241 introduced impact profiles/gates; 242-243 remediation loops; 244 first-pass autofill; 245 intrinsic constraints only (no uplift); 246 multishot closes hard sample; 247 single-shot shaping closes with multishot+autofill disabled; 248 raw candidate search found no uplift; 249 added no-uplift guardrail; 250 closure ladder selected minimal passing mode; 251 richer raw variants still no raw uplift (`selected_variant=0`); 252 added history-aware ladder routing to skip raw-only when historically ineffective. Residual gap remains in raw generator single-shot quality without overlays. | Improve raw single-shot generator output so profile closure does not rely on shaping/multishot/autofill overlays | +| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`, `logs/taskitem_runs/TEST_ONLY_sprint250_closure_ladder_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint251_rawvariants_20260226_154355/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint253_closure_ladder_batch_20260226/closure_ladder_batch_summary.json` | `partial` | Sprint 241 introduced impact profiles/gates; 242-243 remediation loops; 244 first-pass autofill; 245 intrinsic constraints only (no uplift); 246 multishot closure; 247 single-shot shaping closure; 248 raw candidate search no uplift; 249 no-uplift guardrail; 250 closure ladder; 251 richer raw variants no uplift; 252 history-aware ladder routing; 253 batch ladder analytics (`single_shot_shape=2`, `multishot=1`, `ok=3`). Residual gap remains in raw generator single-shot quality without overlays. | Improve raw single-shot generator output so profile closure does not rely on shaping/multishot/autofill overlays | ## What Was Covered Today (Sprints 175-184) diff --git a/docs/progress_log_2026-02-26.md b/docs/progress_log_2026-02-26.md index 09297e2..84a97dc 100644 --- a/docs/progress_log_2026-02-26.md +++ b/docs/progress_log_2026-02-26.md @@ -364,3 +364,15 @@ - `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json` - Verified forced-skip behavior: - attempted modes excluded `raw_only` and started at `single_shot_shape`. + +## Sprint 253 Added (Same Day) + +- Added closure-ladder batch analytics: + - `tools/mcp/analyze_closure_ladder_outcomes.py` + - `tools/mcp/run_native_profile_closure_ladder_batch.sh` +- Dated batch artifact: + - `logs/taskitem_runs/TEST_ONLY_sprint253_closure_ladder_batch_20260226/closure_ladder_batch_summary.json` +- Current batch result (3 specs): + - `selected_mode_counts.single_shot_shape=2` + - `selected_mode_counts.multishot=1` + - `status_counts.ok=3` diff --git a/docs/sprint253_execution_tracker_2026-02-26.md b/docs/sprint253_execution_tracker_2026-02-26.md new file mode 100644 index 0000000..8e0e776 --- /dev/null +++ b/docs/sprint253_execution_tracker_2026-02-26.md @@ -0,0 +1,30 @@ +# Sprint 253 Execution Tracker - 2026-02-26 + +## Scope +- `sprint253_plan.md` + +## Implemented + +New analyzer: +- `tools/mcp/analyze_closure_ladder_outcomes.py` + - Aggregates closure ladder summaries into mode/status distributions. + +New batch runner: +- `tools/mcp/run_native_profile_closure_ladder_batch.sh` + - Executes closure ladder across a provided spec list. + - Emits batch aggregate summary: + - `closure_ladder_batch_summary.json` + +## Validation Artifact + +- `logs/taskitem_runs/TEST_ONLY_sprint253_closure_ladder_batch_20260226/closure_ladder_batch_summary.json` + +Observed batch result (3 specs): +- `status_counts.ok=3` +- `selected_mode_counts`: + - `single_shot_shape=2` + - `multishot=1` + +## Explicit Completion Signal + +- Sprint 253: `DONE` (implemented + batch distribution verified) diff --git a/editor/src/Sprint253IntegrationSummary.h b/editor/src/Sprint253IntegrationSummary.h new file mode 100644 index 0000000..5c8e4b8 --- /dev/null +++ b/editor/src/Sprint253IntegrationSummary.h @@ -0,0 +1,6 @@ +#pragma once + +// Sprint 253 integration summary: +// - Added batch closure ladder execution and outcome aggregation tooling. +// - Produces selected-mode distribution across multiple specs. +// - Enables trend tracking of minimal closure strategy over time. diff --git a/sprint253_plan.md b/sprint253_plan.md new file mode 100644 index 0000000..599f813 --- /dev/null +++ b/sprint253_plan.md @@ -0,0 +1,11 @@ +# Sprint 253 Plan: Batch Closure Ladder Analytics + +## Goal +Measure closure-mode dependence across multiple specs and track which strategy is minimally required. + +## Steps +- Step 2310: Add closure ladder outcome analyzer. +- Step 2311: Add batch runner for closure ladder execution over spec lists. +- Step 2312: Emit mode-distribution summary across runs. +- Step 2313: Validate batch run on mixed hard/plan specs. +- Step 2314: Add `Sprint253IntegrationSummary.h` and execution tracker. diff --git a/tools/mcp/analyze_closure_ladder_outcomes.py b/tools/mcp/analyze_closure_ladder_outcomes.py new file mode 100755 index 0000000..63e6f54 --- /dev/null +++ b/tools/mcp/analyze_closure_ladder_outcomes.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +import argparse +import json +from pathlib import Path +from typing import Dict + + +def load_json(path: Path): + with path.open("r", encoding="utf-8") as f: + return json.load(f) + + +def main() -> int: + p = argparse.ArgumentParser(description="Aggregate closure ladder outcomes.") + p.add_argument("--runs-root", default="logs/taskitem_runs") + p.add_argument("--include-glob", default="TEST_ONLY_*closure_ladder*") + p.add_argument("--out", required=True) + args = p.parse_args() + + root = Path(args.runs_root) + summaries = sorted(root.glob(f"{args.include_glob}/closure_ladder_summary.json")) + + mode_counts: Dict[str, int] = {} + status_counts: Dict[str, int] = {} + rows = [] + for s in summaries: + try: + j = load_json(s) + except Exception: + continue + mode = str(j.get("selected_mode", "")) + status = str(j.get("status", "unknown")) + mode_counts[mode] = mode_counts.get(mode, 0) + 1 + status_counts[status] = status_counts.get(status, 0) + 1 + rows.append({ + "run_id": s.parent.name, + "status": status, + "selected_mode": mode, + "selected_run": str(j.get("selected_run", "")), + }) + + out = { + "status": "ok", + "record_count": len(rows), + "status_counts": status_counts, + "selected_mode_counts": mode_counts, + "records": rows, + } + + with Path(args.out).open("w", encoding="utf-8") as f: + json.dump(out, f, indent=2, sort_keys=True) + f.write("\n") + + print(json.dumps({"status": "ok", "record_count": len(rows), "out": args.out}, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/mcp/run_native_profile_closure_ladder_batch.sh b/tools/mcp/run_native_profile_closure_ladder_batch.sh new file mode 100755 index 0000000..3c88864 --- /dev/null +++ b/tools/mcp/run_native_profile_closure_ladder_batch.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +SPEC_LIST_FILE="${1:-}" +OUT_DIR="${2:-$ROOT_DIR/logs/taskitem_runs/TEST_ONLY_native_profile_closure_ladder_batch_$(date +%Y%m%d_%H%M%S)}" +if [[ -z "$SPEC_LIST_FILE" ]]; then + echo "usage: $0 [out_dir]" >&2 + exit 2 +fi +if [[ ! -f "$SPEC_LIST_FILE" ]]; then + echo "error: spec list not found: $SPEC_LIST_FILE" >&2 + exit 2 +fi +mkdir -p "$OUT_DIR" + +while IFS= read -r spec; do + [[ -z "$spec" ]] && continue + [[ "$spec" =~ ^# ]] && continue + if [[ ! -f "$spec" ]]; then + echo "skip missing spec: $spec" >&2 + continue + fi + safe_name="$(basename "$spec" | sed 's/[^A-Za-z0-9._-]/_/g')" + run_out="$OUT_DIR/$safe_name" + mkdir -p "$run_out" + "$ROOT_DIR/tools/mcp/run_native_profile_closure_ladder.sh" "$spec" "$run_out" >/dev/null || true +done < "$SPEC_LIST_FILE" + +python3 "$ROOT_DIR/tools/mcp/analyze_closure_ladder_outcomes.py" \ + --runs-root "$OUT_DIR" \ + --include-glob "*" \ + --out "$OUT_DIR/closure_ladder_batch_summary.json" >/dev/null + +jq -n --arg out_dir "$OUT_DIR" --argjson summary "$(cat "$OUT_DIR/closure_ladder_batch_summary.json")" '{status:"ok", out_dir:$out_dir, summary:$summary}'