Add richer raw variants and history-aware ladder routing (sprints 251-252)
This commit is contained in:
@@ -150,7 +150,7 @@ Parallel planning tranche (active):
|
|||||||
- `logs/taskitem_runs/TEST_ONLY_spec_planning_baseline_20260226/deterministic_spec_readiness.json`
|
- `logs/taskitem_runs/TEST_ONLY_spec_planning_baseline_20260226/deterministic_spec_readiness.json`
|
||||||
|
|
||||||
Planning runtime controls (active):
|
Planning runtime controls (active):
|
||||||
- `sprint232_plan.md` to `sprint250_plan.md`
|
- `sprint232_plan.md` to `sprint252_plan.md`
|
||||||
- semantic bridge + intake augmentation + requirement injection + expansion gating are wired into:
|
- semantic bridge + intake augmentation + requirement injection + expansion gating are wired into:
|
||||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||||
- current policy default is native-first semantic fallback:
|
- current policy default is native-first semantic fallback:
|
||||||
@@ -204,3 +204,6 @@ Planning runtime controls (active):
|
|||||||
- `tools/mcp/score_native_tasks_profile_coverage.py`
|
- `tools/mcp/score_native_tasks_profile_coverage.py`
|
||||||
- closure ladder can select the minimal passing strategy automatically:
|
- closure ladder can select the minimal passing strategy automatically:
|
||||||
- `tools/mcp/run_native_profile_closure_ladder.sh`
|
- `tools/mcp/run_native_profile_closure_ladder.sh`
|
||||||
|
- closure ladder can route around historically ineffective raw-only mode:
|
||||||
|
- `tools/mcp/analyze_raw_candidate_uplift_history.py`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_SKIP_RAW_WHEN_NO_UPLIFT`
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ This is the canonical dated registry for "not production-ready" generator gaps.
|
|||||||
| GR-018 | Performance/security/rollout constrained refactor enforcement gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/results.jsonl` | `partial` | Hard checks now enforce migration rollback + data-loss policy, security deny-by-default, SLO p95 presence, and rollout staged+abort policy. Enforcement is contract-level; generator capability under these constraints is still weak in C++ AB path. | Constraint-aware generation follow-up |
|
| GR-018 | Performance/security/rollout constrained refactor enforcement gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/results.jsonl` | `partial` | Hard checks now enforce migration rollback + data-loss policy, security deny-by-default, SLO p95 presence, and rollout staged+abort policy. Enforcement is contract-level; generator capability under these constraints is still weak in C++ AB path. | Constraint-aware generation follow-up |
|
||||||
| GR-019 | Parity-blocked readiness load (gating without capability closure) | `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r7/summary.json`, `logs/taskitem_runs/challenging_subset_prod_20260226_r7/summary.json`, `docs/sprint225_227_execution_tracker_2026-02-26.md` | `partial` | Sprint 225-227 reduced blocked parity load from `6` to `0` on both tracked hard catalogs while keeping unresolved divergence at `0`. This closes immediate safety debt for current corpora, but robustness is still contingent on pattern-driven repair classes. | Generalize repairs beyond queue-shaped transpile outputs |
|
| GR-019 | Parity-blocked readiness load (gating without capability closure) | `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r7/summary.json`, `logs/taskitem_runs/challenging_subset_prod_20260226_r7/summary.json`, `docs/sprint225_227_execution_tracker_2026-02-26.md` | `partial` | Sprint 225-227 reduced blocked parity load from `6` to `0` on both tracked hard catalogs while keeping unresolved divergence at `0`. This closes immediate safety debt for current corpora, but robustness is still contingent on pattern-driven repair classes. | Generalize repairs beyond queue-shaped transpile outputs |
|
||||||
| GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json`, `logs/taskitem_runs/TEST_ONLY_sprint239_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint240_retry_20260226_145631/00_summary.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md`, `docs/sprint238_execution_tracker_2026-02-26.md`, `docs/sprint239_execution_tracker_2026-02-26.md`, `docs/sprint240_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Sprint 238 added native decomposition hard gate in pipeline (`min task count`, `min semantic signal count`) so weak native generation can be blocked before fallback masking. Sprint 239 added native reason enrichment and improved semantic signal density (`0 -> 6`). Sprint 240 added native decomposition retry with explicit minimum-task policy; on current sample retry attempted but did not improve depth (`2 -> 2`). | Native decomposition task-depth upgrade (increase native task granularity beyond 2) |
|
| GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json`, `logs/taskitem_runs/TEST_ONLY_sprint239_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint240_retry_20260226_145631/00_summary.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md`, `docs/sprint238_execution_tracker_2026-02-26.md`, `docs/sprint239_execution_tracker_2026-02-26.md`, `docs/sprint240_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Sprint 238 added native decomposition hard gate in pipeline (`min task count`, `min semantic signal count`) so weak native generation can be blocked before fallback masking. Sprint 239 added native reason enrichment and improved semantic signal density (`0 -> 6`). Sprint 240 added native decomposition retry with explicit minimum-task policy; on current sample retry attempted but did not improve depth (`2 -> 2`). | Native decomposition task-depth upgrade (increase native task granularity beyond 2) |
|
||||||
| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`, `logs/taskitem_runs/TEST_ONLY_sprint250_closure_ladder_20260226/closure_ladder_summary.json` | `partial` | Sprint 241 introduced impact profiles/gates; 242-243 remediation loops; 244 first-pass autofill; 245 intrinsic constraints only (no uplift); 246 multishot closes hard sample; 247 single-shot shaping closes with multishot+autofill disabled; 248 raw candidate search found no uplift; 249 added no-uplift guardrail; 250 added automatic closure ladder and selected `single_shot_shape` as minimal passing mode on hard sample. Residual gap remains in raw generator single-shot quality without overlays. | Improve raw single-shot generator output so profile closure does not rely on shaping/multishot/autofill overlays |
|
| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`, `logs/taskitem_runs/TEST_ONLY_sprint250_closure_ladder_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint251_rawvariants_20260226_154355/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json` | `partial` | Sprint 241 introduced impact profiles/gates; 242-243 remediation loops; 244 first-pass autofill; 245 intrinsic constraints only (no uplift); 246 multishot closes hard sample; 247 single-shot shaping closes with multishot+autofill disabled; 248 raw candidate search found no uplift; 249 added no-uplift guardrail; 250 closure ladder selected minimal passing mode; 251 richer raw variants still no raw uplift (`selected_variant=0`); 252 added history-aware ladder routing to skip raw-only when historically ineffective. Residual gap remains in raw generator single-shot quality without overlays. | Improve raw single-shot generator output so profile closure does not rely on shaping/multishot/autofill overlays |
|
||||||
|
|
||||||
## What Was Covered Today (Sprints 175-184)
|
## What Was Covered Today (Sprints 175-184)
|
||||||
|
|
||||||
|
|||||||
@@ -334,3 +334,33 @@
|
|||||||
- Current measured result on hard fullstack sample:
|
- Current measured result on hard fullstack sample:
|
||||||
- selected mode: `single_shot_shape`
|
- selected mode: `single_shot_shape`
|
||||||
- selected run passes impact coverage with `failing_profile_count=0`
|
- selected run passes impact coverage with `failing_profile_count=0`
|
||||||
|
|
||||||
|
## Sprint 251 Added (Same Day)
|
||||||
|
|
||||||
|
- Added richer raw-variant synthesis for raw-only candidate search:
|
||||||
|
- `tools/mcp/synthesize_raw_candidate_requirement_variants.py`
|
||||||
|
- raw search now consumes profile-bundle variants
|
||||||
|
- Dated artifact:
|
||||||
|
- `logs/taskitem_runs/TEST_ONLY_sprint251_rawvariants_20260226_154355/00_summary.json`
|
||||||
|
- Current measured result:
|
||||||
|
- `available_variants=8`, `attempted_variants=8`
|
||||||
|
- `selected_variant=0`, `failing_profile_count 7 -> 7`
|
||||||
|
- no raw-only uplift on this hard sample.
|
||||||
|
|
||||||
|
## Sprint 252 Added (Same Day)
|
||||||
|
|
||||||
|
- Added raw uplift history analyzer:
|
||||||
|
- `tools/mcp/analyze_raw_candidate_uplift_history.py`
|
||||||
|
- Added closure-ladder policy routing based on historical raw uplift:
|
||||||
|
- `tools/mcp/run_native_profile_closure_ladder.sh`
|
||||||
|
- controls:
|
||||||
|
- `WSTONE_CLOSURE_LADDER_SKIP_RAW_WHEN_NO_UPLIFT`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_HISTORY_GLOB`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_HISTORY_MAX_RUNS`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_MIN_UPLIFT_RATE`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_MIN_RECORDS`
|
||||||
|
- Dated artifacts:
|
||||||
|
- `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_20260226/closure_ladder_summary.json`
|
||||||
|
- `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json`
|
||||||
|
- Verified forced-skip behavior:
|
||||||
|
- attempted modes excluded `raw_only` and started at `single_shot_shape`.
|
||||||
|
|||||||
29
docs/sprint251_execution_tracker_2026-02-26.md
Normal file
29
docs/sprint251_execution_tracker_2026-02-26.md
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
# Sprint 251 Execution Tracker - 2026-02-26
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
- `sprint251_plan.md`
|
||||||
|
|
||||||
|
## Implemented
|
||||||
|
|
||||||
|
New tool:
|
||||||
|
- `tools/mcp/synthesize_raw_candidate_requirement_variants.py`
|
||||||
|
- Builds profile-aware raw requirement bundles for candidate search.
|
||||||
|
|
||||||
|
Pipeline integration in `tools/mcp/run_sprint_taskitem_pipeline.sh`:
|
||||||
|
- Raw candidate search now consumes synthesized variant bundles.
|
||||||
|
- Added telemetry field:
|
||||||
|
- `native_raw_candidate_search.available_variants`
|
||||||
|
|
||||||
|
## Validation Artifact
|
||||||
|
|
||||||
|
- `logs/taskitem_runs/TEST_ONLY_sprint251_rawvariants_20260226_154355/00_summary.json`
|
||||||
|
|
||||||
|
Observed result on hard fullstack sample:
|
||||||
|
- `available_variants=8`
|
||||||
|
- `attempted_variants=8`
|
||||||
|
- selected variant remains baseline (`selected_variant=0`)
|
||||||
|
- `failing_profile_count` unchanged at `7`
|
||||||
|
|
||||||
|
## Explicit Completion Signal
|
||||||
|
|
||||||
|
- Sprint 251: `DONE` (implemented + measured; no raw-only uplift)
|
||||||
36
docs/sprint252_execution_tracker_2026-02-26.md
Normal file
36
docs/sprint252_execution_tracker_2026-02-26.md
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
# Sprint 252 Execution Tracker - 2026-02-26
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
- `sprint252_plan.md`
|
||||||
|
|
||||||
|
## Implemented
|
||||||
|
|
||||||
|
New analyzer:
|
||||||
|
- `tools/mcp/analyze_raw_candidate_uplift_history.py`
|
||||||
|
- Computes recent raw-search uplift rate from run summaries.
|
||||||
|
|
||||||
|
Closure ladder policy routing in `tools/mcp/run_native_profile_closure_ladder.sh`:
|
||||||
|
- Added controls:
|
||||||
|
- `WSTONE_CLOSURE_LADDER_SKIP_RAW_WHEN_NO_UPLIFT` (default `1`)
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_HISTORY_GLOB`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_HISTORY_MAX_RUNS`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_MIN_UPLIFT_RATE`
|
||||||
|
- `WSTONE_CLOSURE_LADDER_RAW_MIN_RECORDS` (default `3`)
|
||||||
|
- Emits policy artifact when enabled:
|
||||||
|
- `raw_uplift_history.json`
|
||||||
|
- Dynamically removes `raw_only` from ladder modes when uplift evidence is below threshold.
|
||||||
|
|
||||||
|
## Validation Artifacts
|
||||||
|
|
||||||
|
- policy-enabled run:
|
||||||
|
- `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_20260226/closure_ladder_summary.json`
|
||||||
|
- forced-skip run (demonstrates raw mode removed):
|
||||||
|
- `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json`
|
||||||
|
|
||||||
|
Observed forced-skip signal:
|
||||||
|
- `attempted_modes=[single_shot_shape,multishot,autofill]`
|
||||||
|
- selected mode remained `single_shot_shape`.
|
||||||
|
|
||||||
|
## Explicit Completion Signal
|
||||||
|
|
||||||
|
- Sprint 252: `DONE` (implemented + policy skip behavior verified)
|
||||||
6
editor/src/Sprint251IntegrationSummary.h
Normal file
6
editor/src/Sprint251IntegrationSummary.h
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
// Sprint 251 integration summary:
|
||||||
|
// - Added richer raw variant synthesis for candidate search.
|
||||||
|
// - Raw candidate search now consumes profile-bundle variants rather than single requirement slices.
|
||||||
|
// - Hard sample still shows no raw-only uplift; selected candidate remains baseline.
|
||||||
6
editor/src/Sprint252IntegrationSummary.h
Normal file
6
editor/src/Sprint252IntegrationSummary.h
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
// Sprint 252 integration summary:
|
||||||
|
// - Added raw uplift history analyzer for closure ladder policy routing.
|
||||||
|
// - Closure ladder can skip raw-only mode when historical uplift is weak.
|
||||||
|
// - Policy behavior is configurable via threshold and minimum-record controls.
|
||||||
11
sprint251_plan.md
Normal file
11
sprint251_plan.md
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
# Sprint 251 Plan: Rich Raw Variant Synthesis for Candidate Search
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
Increase raw-only search strength by synthesizing richer requirement bundles per active profile and testing if raw generator output improves without overlays.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
- Step 2300: Add raw candidate requirement variant synthesizer.
|
||||||
|
- Step 2301: Integrate variant bundles into raw candidate search loop.
|
||||||
|
- Step 2302: Surface available-variant telemetry in summary.
|
||||||
|
- Step 2303: Re-test hard fullstack sample for raw-only uplift.
|
||||||
|
- Step 2304: Add `Sprint251IntegrationSummary.h` and execution tracker.
|
||||||
11
sprint252_plan.md
Normal file
11
sprint252_plan.md
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
# Sprint 252 Plan: Closure Ladder Policy Routing from Raw Uplift History
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
Reduce wasted raw-only attempts by making closure ladder mode selection aware of recent raw candidate uplift performance.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
- Step 2305: Add raw uplift history analyzer.
|
||||||
|
- Step 2306: Add ladder policy controls for skip conditions.
|
||||||
|
- Step 2307: Skip `raw_only` when historical uplift rate is below threshold.
|
||||||
|
- Step 2308: Emit policy evidence artifact in ladder output directory.
|
||||||
|
- Step 2309: Add `Sprint252IntegrationSummary.h` and execution tracker.
|
||||||
64
tools/mcp/analyze_raw_candidate_uplift_history.py
Executable file
64
tools/mcp/analyze_raw_candidate_uplift_history.py
Executable file
@@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, List
|
||||||
|
|
||||||
|
|
||||||
|
def load_json(path: Path):
|
||||||
|
with path.open("r", encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
p = argparse.ArgumentParser(description="Analyze recent raw candidate search uplift history.")
|
||||||
|
p.add_argument("--runs-root", default="logs/taskitem_runs")
|
||||||
|
p.add_argument("--include-glob", default="*")
|
||||||
|
p.add_argument("--max-runs", type=int, default=20)
|
||||||
|
p.add_argument("--out", required=True)
|
||||||
|
args = p.parse_args()
|
||||||
|
|
||||||
|
root = Path(args.runs_root)
|
||||||
|
summaries: List[Path] = sorted(root.glob(f"{args.include_glob}/00_summary.json"), key=lambda x: x.stat().st_mtime, reverse=True)
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
for path in summaries[: max(0, args.max_runs)]:
|
||||||
|
try:
|
||||||
|
s = load_json(path)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
raw = s.get("native_raw_candidate_search") or {}
|
||||||
|
if not isinstance(raw, dict) or not raw.get("enabled", False):
|
||||||
|
continue
|
||||||
|
baseline = int(raw.get("baseline_failing_profile_count", 999) or 999)
|
||||||
|
best = int(raw.get("best_failing_profile_count", 999) or 999)
|
||||||
|
rows.append({
|
||||||
|
"run_id": path.parent.name,
|
||||||
|
"baseline_failing_profile_count": baseline,
|
||||||
|
"best_failing_profile_count": best,
|
||||||
|
"uplift": int(best < baseline),
|
||||||
|
})
|
||||||
|
|
||||||
|
total = len(rows)
|
||||||
|
uplift_count = sum(int(r.get("uplift", 0)) for r in rows)
|
||||||
|
no_uplift_count = total - uplift_count
|
||||||
|
|
||||||
|
out = {
|
||||||
|
"status": "ok",
|
||||||
|
"record_count": total,
|
||||||
|
"uplift_count": uplift_count,
|
||||||
|
"no_uplift_count": no_uplift_count,
|
||||||
|
"uplift_rate": round((uplift_count / total), 4) if total else 0.0,
|
||||||
|
"records": rows,
|
||||||
|
}
|
||||||
|
|
||||||
|
with Path(args.out).open("w", encoding="utf-8") as f:
|
||||||
|
json.dump(out, f, indent=2, sort_keys=True)
|
||||||
|
f.write("\n")
|
||||||
|
|
||||||
|
print(json.dumps({"status": "ok", "record_count": total, "uplift_rate": out["uplift_rate"], "out": args.out}, sort_keys=True))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -14,6 +14,31 @@ if [[ ! -f "$SPEC_PATH" ]]; then
|
|||||||
fi
|
fi
|
||||||
mkdir -p "$OUT_DIR"
|
mkdir -p "$OUT_DIR"
|
||||||
|
|
||||||
|
SKIP_RAW_WHEN_NO_UPLIFT="${WSTONE_CLOSURE_LADDER_SKIP_RAW_WHEN_NO_UPLIFT:-1}"
|
||||||
|
RAW_HISTORY_GLOB="${WSTONE_CLOSURE_LADDER_RAW_HISTORY_GLOB:-TEST_ONLY_sprint24*_raw*}"
|
||||||
|
RAW_HISTORY_MAX_RUNS="${WSTONE_CLOSURE_LADDER_RAW_HISTORY_MAX_RUNS:-20}"
|
||||||
|
RAW_MIN_UPLIFT_RATE="${WSTONE_CLOSURE_LADDER_RAW_MIN_UPLIFT_RATE:-0.05}"
|
||||||
|
RAW_MIN_RECORDS="${WSTONE_CLOSURE_LADDER_RAW_MIN_RECORDS:-3}"
|
||||||
|
|
||||||
|
effective_modes=(raw_only single_shot_shape multishot autofill)
|
||||||
|
if [[ "$SKIP_RAW_WHEN_NO_UPLIFT" == "1" ]]; then
|
||||||
|
python3 "$ROOT_DIR/tools/mcp/analyze_raw_candidate_uplift_history.py" \
|
||||||
|
--runs-root "$ROOT_DIR/logs/taskitem_runs" \
|
||||||
|
--include-glob "$RAW_HISTORY_GLOB" \
|
||||||
|
--max-runs "$RAW_HISTORY_MAX_RUNS" \
|
||||||
|
--out "$OUT_DIR/raw_uplift_history.json" >/dev/null || true
|
||||||
|
if [[ -f "$OUT_DIR/raw_uplift_history.json" ]]; then
|
||||||
|
raw_record_count="$(jq '.record_count // 0' "$OUT_DIR/raw_uplift_history.json")"
|
||||||
|
raw_uplift_rate="$(jq '.uplift_rate // 0' "$OUT_DIR/raw_uplift_history.json")"
|
||||||
|
if [[ "$raw_record_count" -ge "$RAW_MIN_RECORDS" ]]; then
|
||||||
|
awk_cmp="$(awk -v a="$raw_uplift_rate" -v b="$RAW_MIN_UPLIFT_RATE" 'BEGIN{if (a < b) print 1; else print 0;}')"
|
||||||
|
if [[ "$awk_cmp" == "1" ]]; then
|
||||||
|
effective_modes=(single_shot_shape multishot autofill)
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
run_mode() {
|
run_mode() {
|
||||||
local mode="$1"
|
local mode="$1"
|
||||||
local logfile="$OUT_DIR/${mode}.log"
|
local logfile="$OUT_DIR/${mode}.log"
|
||||||
@@ -65,12 +90,11 @@ run_mode() {
|
|||||||
return "$rc"
|
return "$rc"
|
||||||
}
|
}
|
||||||
|
|
||||||
modes=(raw_only single_shot_shape multishot autofill)
|
|
||||||
selected=""
|
selected=""
|
||||||
selected_run=""
|
selected_run=""
|
||||||
selected_rc=1
|
selected_rc=1
|
||||||
|
|
||||||
for mode in "${modes[@]}"; do
|
for mode in "${effective_modes[@]}"; do
|
||||||
if run_mode "$mode"; then
|
if run_mode "$mode"; then
|
||||||
selected="$mode"
|
selected="$mode"
|
||||||
selected_run="$(cat "$OUT_DIR/${mode}.run_dir")"
|
selected_run="$(cat "$OUT_DIR/${mode}.run_dir")"
|
||||||
@@ -85,7 +109,7 @@ jq -n \
|
|||||||
--arg selected_mode "$selected" \
|
--arg selected_mode "$selected" \
|
||||||
--arg selected_run "$selected_run" \
|
--arg selected_run "$selected_run" \
|
||||||
--argjson selected_rc "$selected_rc" \
|
--argjson selected_rc "$selected_rc" \
|
||||||
--argjson attempted_modes "$(printf '%s\n' "${modes[@]}" | jq -R . | jq -s .)" \
|
--argjson attempted_modes "$(printf '%s\n' "${effective_modes[@]}" | jq -R . | jq -s .)" \
|
||||||
'{
|
'{
|
||||||
status: (if $selected_rc == 0 then "ok" else "fail" end),
|
status: (if $selected_rc == 0 then "ok" else "fail" end),
|
||||||
spec:$spec,
|
spec:$spec,
|
||||||
|
|||||||
@@ -390,13 +390,12 @@ else
|
|||||||
'{attempted:$attempted, applied:$applied, initial_task_count:$initial_task_count, target_min_task_count:$target_min_task_count}')"
|
'{attempted:$attempted, applied:$applied, initial_task_count:$initial_task_count, target_min_task_count:$target_min_task_count}')"
|
||||||
fi
|
fi
|
||||||
if [[ "$NATIVE_RAW_CANDIDATE_SEARCH" == "1" ]]; then
|
if [[ "$NATIVE_RAW_CANDIDATE_SEARCH" == "1" ]]; then
|
||||||
if [[ "$(printf '%s' "$INTRINSIC_REQS_JSON" | jq 'length')" -eq 0 ]]; then
|
python3 "$ROOT_DIR/tools/mcp/synthesize_raw_candidate_requirement_variants.py" \
|
||||||
python3 "$ROOT_DIR/tools/mcp/synthesize_native_intrinsic_boost_requirements.py" \
|
--spec "$INPUT_FILE" \
|
||||||
--spec "$INPUT_FILE" \
|
--profiles "$NATIVE_IMPACT_COVERAGE_PROFILES" \
|
||||||
--profiles "$NATIVE_IMPACT_COVERAGE_PROFILES" \
|
--max-variants "$NATIVE_RAW_CANDIDATE_MAX_VARIANTS" \
|
||||||
--out "$OUT_DIR/02ae_raw_candidate_requirements.json" >/dev/null
|
--out "$OUT_DIR/02ae_raw_candidate_variants.json" >/dev/null
|
||||||
INTRINSIC_REQS_JSON="$(cat "$OUT_DIR/02ae_raw_candidate_requirements.json")"
|
VARIANTS_JSON="$(cat "$OUT_DIR/02ae_raw_candidate_variants.json")"
|
||||||
fi
|
|
||||||
|
|
||||||
printf '%s\n' "$TASKS" > "$OUT_DIR/02ae_candidate_0_tasks.json"
|
printf '%s\n' "$TASKS" > "$OUT_DIR/02ae_candidate_0_tasks.json"
|
||||||
python3 "$ROOT_DIR/tools/mcp/score_native_tasks_profile_coverage.py" \
|
python3 "$ROOT_DIR/tools/mcp/score_native_tasks_profile_coverage.py" \
|
||||||
@@ -412,13 +411,13 @@ if [[ "$NATIVE_RAW_CANDIDATE_SEARCH" == "1" ]]; then
|
|||||||
attempted=1
|
attempted=1
|
||||||
successful=1
|
successful=1
|
||||||
|
|
||||||
req_total="$(printf '%s' "$INTRINSIC_REQS_JSON" | jq 'length')"
|
variant_total="$(printf '%s' "$VARIANTS_JSON" | jq '.variants | length')"
|
||||||
max_variants="$NATIVE_RAW_CANDIDATE_MAX_VARIANTS"
|
max_variants="$NATIVE_RAW_CANDIDATE_MAX_VARIANTS"
|
||||||
idx=0
|
idx=0
|
||||||
variant=1
|
variant=1
|
||||||
while [[ "$variant" -lt "$max_variants" && "$idx" -lt "$req_total" ]]; do
|
while [[ "$variant" -lt "$max_variants" && "$idx" -lt "$variant_total" ]]; do
|
||||||
req_i="$(printf '%s' "$INTRINSIC_REQS_JSON" | jq ".[$idx]")"
|
req_bundle="$(printf '%s' "$VARIANTS_JSON" | jq ".variants[$idx]")"
|
||||||
cand_reqs="$(jq -nc --argjson base "$NORMALIZED_REQS" --argjson req "$req_i" '$base + [$req]')"
|
cand_reqs="$(jq -nc --argjson base "$NORMALIZED_REQS" --argjson bundle "$req_bundle" '$base + $bundle')"
|
||||||
cand_args="$(jq -nc --argjson nr "$cand_reqs" --argjson cf "$CONFLICTS" --arg strict "$STRICT_EXECUTION_CONTRACT" \
|
cand_args="$(jq -nc --argjson nr "$cand_reqs" --argjson cf "$CONFLICTS" --arg strict "$STRICT_EXECUTION_CONTRACT" \
|
||||||
'{normalizedRequirements:$nr,conflicts:$cf,strictExecutionContract:($strict == "1")}')"
|
'{normalizedRequirements:$nr,conflicts:$cf,strictExecutionContract:($strict == "1")}')"
|
||||||
cand_raw="$(call_tool "whetstone_generate_taskitems" "$cand_args")"
|
cand_raw="$(call_tool "whetstone_generate_taskitems" "$cand_args")"
|
||||||
@@ -457,7 +456,8 @@ if [[ "$NATIVE_RAW_CANDIDATE_SEARCH" == "1" ]]; then
|
|||||||
--argjson baseline_task_count "$baseline_task_count" \
|
--argjson baseline_task_count "$baseline_task_count" \
|
||||||
--argjson best_failing_profile_count "$best_fail" \
|
--argjson best_failing_profile_count "$best_fail" \
|
||||||
--argjson best_task_count "$best_task_count" \
|
--argjson best_task_count "$best_task_count" \
|
||||||
'{enabled:$enabled, attempted_variants:$attempted, successful_variants:$successful, selected_variant:$best_variant, baseline_failing_profile_count:$baseline_failing_profile_count, baseline_task_count:$baseline_task_count, best_failing_profile_count:$best_failing_profile_count, best_task_count:$best_task_count}')"
|
--argjson available_variants "$variant_total" \
|
||||||
|
'{enabled:$enabled, attempted_variants:$attempted, successful_variants:$successful, available_variants:$available_variants, selected_variant:$best_variant, baseline_failing_profile_count:$baseline_failing_profile_count, baseline_task_count:$baseline_task_count, best_failing_profile_count:$best_failing_profile_count, best_task_count:$best_task_count}')"
|
||||||
printf '%s\n' "$NATIVE_RAW_CANDIDATE_SEARCH_JSON" > "$OUT_DIR/02ae_raw_candidate_search.json"
|
printf '%s\n' "$NATIVE_RAW_CANDIDATE_SEARCH_JSON" > "$OUT_DIR/02ae_raw_candidate_search.json"
|
||||||
if [[ "$NATIVE_RAW_CANDIDATE_REQUIRE_UPLIFT" == "1" && "$best_fail" -ge "$baseline_fail" ]]; then
|
if [[ "$NATIVE_RAW_CANDIDATE_REQUIRE_UPLIFT" == "1" && "$best_fail" -ge "$baseline_fail" ]]; then
|
||||||
echo "error: raw candidate search did not improve failing profile count" >&2
|
echo "error: raw candidate search did not improve failing profile count" >&2
|
||||||
|
|||||||
131
tools/mcp/synthesize_raw_candidate_requirement_variants.py
Executable file
131
tools/mcp/synthesize_raw_candidate_requirement_variants.py
Executable file
@@ -0,0 +1,131 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, List
|
||||||
|
|
||||||
|
|
||||||
|
def load_json(path: Path):
|
||||||
|
with path.open("r", encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
|
||||||
|
|
||||||
|
def write_json(path: Path, obj) -> None:
|
||||||
|
with path.open("w", encoding="utf-8") as f:
|
||||||
|
json.dump(obj, f, indent=2, sort_keys=True)
|
||||||
|
f.write("\n")
|
||||||
|
|
||||||
|
|
||||||
|
def activate_profiles(profiles: List[Dict], spec_text: str) -> List[Dict]:
|
||||||
|
s = spec_text.lower()
|
||||||
|
out = []
|
||||||
|
for p in profiles:
|
||||||
|
keys = [str(k).lower() for k in (p.get("trigger_keywords") or [])]
|
||||||
|
if keys and any(k in s for k in keys):
|
||||||
|
out.append(p)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def req_from_profile(p: Dict, idx: int) -> Dict:
|
||||||
|
pid = str(p.get("id", f"profile_{idx}"))
|
||||||
|
min_tasks = int(p.get("min_native_task_count", 0) or 0)
|
||||||
|
reasons = [str(x) for x in (p.get("required_reason_keywords") or [])]
|
||||||
|
ops = [str(x) for x in (p.get("required_prerequisite_ops") or [])]
|
||||||
|
contract = [str(x) for x in (p.get("required_execution_contract") or [])]
|
||||||
|
text = (
|
||||||
|
f"raw candidate policy {pid}: produce >= {min_tasks} concrete taskitems; "
|
||||||
|
f"reasons include {', '.join(reasons) if reasons else 'none'}; "
|
||||||
|
f"ops include {', '.join(ops) if ops else 'none'}; "
|
||||||
|
f"execution contract include {', '.join(contract) if contract else 'none'}; "
|
||||||
|
"avoid generic or umbrella tasks."
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"requirementId": f"raw-variant-{pid}",
|
||||||
|
"kind": "constraint",
|
||||||
|
"normalizedText": text,
|
||||||
|
"anchor": "raw_candidate_variant",
|
||||||
|
"sourceLine": 0,
|
||||||
|
"ambiguous": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def global_decomposition_req(active_count: int, min_target: int) -> Dict:
|
||||||
|
return {
|
||||||
|
"requirementId": "raw-variant-global-decomposition",
|
||||||
|
"kind": "constraint",
|
||||||
|
"normalizedText": (
|
||||||
|
f"raw decomposition hard policy: output at least {min_target} concrete taskitems "
|
||||||
|
f"for {active_count} active impact profiles; each task must have deterministic executionContract and explicit prerequisiteOps."
|
||||||
|
),
|
||||||
|
"anchor": "raw_candidate_variant",
|
||||||
|
"sourceLine": 0,
|
||||||
|
"ambiguous": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def build_variants(active: List[Dict], max_variants: int) -> List[List[Dict]]:
|
||||||
|
if max_variants <= 0:
|
||||||
|
return []
|
||||||
|
|
||||||
|
profile_reqs = [req_from_profile(p, i) for i, p in enumerate(active)]
|
||||||
|
min_target = max(5, len(active) + 2)
|
||||||
|
g = global_decomposition_req(len(active), min_target)
|
||||||
|
|
||||||
|
variants: List[List[Dict]] = []
|
||||||
|
|
||||||
|
# Variant 1: global only
|
||||||
|
variants.append([g])
|
||||||
|
|
||||||
|
# Variant 2: global + all profiles
|
||||||
|
variants.append([g] + profile_reqs)
|
||||||
|
|
||||||
|
# Variant 3+: one profile at a time + global
|
||||||
|
for r in profile_reqs:
|
||||||
|
variants.append([g, r])
|
||||||
|
|
||||||
|
# Variant: paired profile bundles
|
||||||
|
for i in range(0, len(profile_reqs), 2):
|
||||||
|
bundle = [g] + profile_reqs[i:i+2]
|
||||||
|
variants.append(bundle)
|
||||||
|
|
||||||
|
# Deduplicate by requirementId tuple ordering
|
||||||
|
seen = set()
|
||||||
|
out: List[List[Dict]] = []
|
||||||
|
for v in variants:
|
||||||
|
key = tuple(x.get("requirementId", "") for x in v)
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
out.append(v)
|
||||||
|
if len(out) >= max_variants:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
p = argparse.ArgumentParser(description="Synthesize raw candidate requirement variants from active profiles.")
|
||||||
|
p.add_argument("--spec", required=True)
|
||||||
|
p.add_argument("--profiles", required=True)
|
||||||
|
p.add_argument("--max-variants", type=int, default=8)
|
||||||
|
p.add_argument("--out", required=True)
|
||||||
|
args = p.parse_args()
|
||||||
|
|
||||||
|
spec_text = Path(args.spec).read_text(encoding="utf-8", errors="ignore")
|
||||||
|
profiles_doc = load_json(Path(args.profiles))
|
||||||
|
profiles = list(profiles_doc.get("profiles") or [])
|
||||||
|
active = activate_profiles(profiles, spec_text)
|
||||||
|
|
||||||
|
variants = build_variants(active, args.max_variants)
|
||||||
|
payload = {
|
||||||
|
"status": "ok",
|
||||||
|
"active_profile_count": len(active),
|
||||||
|
"variant_count": len(variants),
|
||||||
|
"variants": variants,
|
||||||
|
}
|
||||||
|
write_json(Path(args.out), payload)
|
||||||
|
print(json.dumps({"status": "ok", "active_profile_count": len(active), "variant_count": len(variants), "out": args.out}, sort_keys=True))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
Reference in New Issue
Block a user