bill/whetstone_DSL
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add raw candidate search scoring and no-uplift guardrail (sprints 248-249)

Browse Source
This commit is contained in:
Bill
2026-02-26 15:41:20 -07:00
parent dfb10ea40f
commit 8e0018ef80
11 changed files with 328 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/constructive_editing_runtime_plan.md
View File

@@ -150,7 +150,7 @@ Parallel planning tranche (active):
- `logs/taskitem_runs/TEST_ONLY_spec_planning_baseline_20260226/deterministic_spec_readiness.json`
Planning runtime controls (active):
- `sprint232_plan.md` to `sprint247_plan.md`
- `sprint232_plan.md` to `sprint249_plan.md`
- semantic bridge + intake augmentation + requirement injection + expansion gating are wired into:
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
- current policy default is native-first semantic fallback:
@@ -197,3 +197,8 @@ Planning runtime controls (active):
- `WSTONE_NATIVE_SINGLESHOT_PROFILE_SHAPE`
- `tools/mcp/synthesize_single_shot_profile_shape_tasks.py`
- `native_single_shot_profile_shape`
- raw-only candidate search (no overlays) can benchmark intrinsic generator variants:
- `WSTONE_NATIVE_RAW_CANDIDATE_SEARCH`
- `WSTONE_NATIVE_RAW_CANDIDATE_MAX_VARIANTS`
- `WSTONE_NATIVE_RAW_CANDIDATE_REQUIRE_UPLIFT`
- `tools/mcp/score_native_tasks_profile_coverage.py`

2
docs/generator_readiness_gap_registry_2026-02-26.md
View File

@@ -47,7 +47,7 @@ This is the canonical dated registry for "not production-ready" generator gaps.
| GR-018 | Performance/security/rollout constrained refactor enforcement gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/results.jsonl` | `partial` | Hard checks now enforce migration rollback + data-loss policy, security deny-by-default, SLO p95 presence, and rollout staged+abort policy. Enforcement is contract-level; generator capability under these constraints is still weak in C++ AB path. | Constraint-aware generation follow-up |
| GR-019 | Parity-blocked readiness load (gating without capability closure) | `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r7/summary.json`, `logs/taskitem_runs/challenging_subset_prod_20260226_r7/summary.json`, `docs/sprint225_227_execution_tracker_2026-02-26.md` | `partial` | Sprint 225-227 reduced blocked parity load from `6` to `0` on both tracked hard catalogs while keeping unresolved divergence at `0`. This closes immediate safety debt for current corpora, but robustness is still contingent on pattern-driven repair classes. | Generalize repairs beyond queue-shaped transpile outputs |
| GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json`, `logs/taskitem_runs/TEST_ONLY_sprint239_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint240_retry_20260226_145631/00_summary.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md`, `docs/sprint238_execution_tracker_2026-02-26.md`, `docs/sprint239_execution_tracker_2026-02-26.md`, `docs/sprint240_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Sprint 238 added native decomposition hard gate in pipeline (`min task count`, `min semantic signal count`) so weak native generation can be blocked before fallback masking. Sprint 239 added native reason enrichment and improved semantic signal density (`0 -> 6`). Sprint 240 added native decomposition retry with explicit minimum-task policy; on current sample retry attempted but did not improve depth (`2 -> 2`). | Native decomposition task-depth upgrade (increase native task granularity beyond 2) |
| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json` | `partial` | Sprint 241 added one-by-one impact profiles and native impact coverage gate. Sprint 242-243 added remediation synthesis and task-bundle closure loop (`7 -> 0`). Sprint 244 added first-pass profile autofill. Sprint 245 intrinsic-constraint-only path showed no uplift (`7 -> 7`). Sprint 246 intrinsic multishot path closed hard sample with autofill disabled (`7 -> 0`). Sprint 247 added single-shot profile shaping and closes hard sample with both multishot and autofill disabled (`7 -> 0`) while enforced gate still fails when shaping is disabled (`7`). Remaining risk is that closure depends on deterministic shaping overlays, not purely raw generator single-shot behavior. | Improve raw single-shot generator output so profile closure does not rely on post-generation shaping overlays |
| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json` | `partial` | Sprint 241 introduced impact profiles/gates; 242-243 remediation loops; 244 first-pass autofill; 245 intrinsic constraints only (no uplift); 246 multishot closes hard sample; 247 single-shot shaping closes with multishot+autofill disabled; 248 raw candidate search over 8 variants found no uplift (`selected_variant=0`, `7 -> 7`); 249 added no-uplift guardrail to hard-fail ineffective raw search attempts. Residual gap remains in raw generator single-shot quality without overlays. | Improve raw single-shot generator output so profile closure does not rely on shaping/multishot/autofill overlays |
## What Was Covered Today (Sprints 175-184)

28
docs/progress_log_2026-02-26.md
View File

@@ -295,3 +295,31 @@
- Enforced gate verification:
- pass ON: `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`
- fail OFF: `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153242/06_native_impact_coverage.json`
## Sprint 248 Added (Same Day)
- Added raw generator candidate search path (no shaping/autofill/multishot overlays):
- `tools/mcp/score_native_tasks_profile_coverage.py`
- `WSTONE_NATIVE_RAW_CANDIDATE_SEARCH`
- `WSTONE_NATIVE_RAW_CANDIDATE_MAX_VARIANTS`
- summary packet: `native_raw_candidate_search`
- A/B artifacts:
- OFF: `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153902/00_summary.json`
- ON: `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`
- Current measured result:
- `attempted_variants=8`, `selected_variant=0`
- `failing_profile_count 7 -> 7`
- no raw-only uplift on this hard sample.
## Sprint 249 Added (Same Day)
- Added raw candidate no-uplift guardrail:
- `WSTONE_NATIVE_RAW_CANDIDATE_REQUIRE_UPLIFT`
- artifact: `02ae_raw_candidate_search.json`
- hard fail code: `17` when uplift is required but absent
- Dated artifacts:
- guard OFF: `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154026/00_summary.json`
- guard ON (expected fail): `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`
- Current measured result:
- baseline `7`, best `7` failing profiles
- run blocked correctly under require-uplift mode.

38
docs/sprint248_execution_tracker_2026-02-26.md Normal file
View File

@@ -0,0 +1,38 @@
# Sprint 248 Execution Tracker - 2026-02-26
## Scope
- `sprint248_plan.md`
## Implemented
New tool:
- `tools/mcp/score_native_tasks_profile_coverage.py`
- Scores any raw task set against active impact profiles.
Pipeline integration in `tools/mcp/run_sprint_taskitem_pipeline.sh`:
- Added controls:
- `WSTONE_NATIVE_RAW_CANDIDATE_SEARCH` (default `0`)
- `WSTONE_NATIVE_RAW_CANDIDATE_MAX_VARIANTS` (default `8`)
- Added raw candidate generation/scoring artifacts:
- `02ae_candidate_<n>.json`
- `02ae_candidate_<n>_tasks.json`
- `02ae_candidate_<n>_score.json`
- Added summary packet:
- `native_raw_candidate_search`
## A/B Validation (no overlays)
- candidate search OFF:
- `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153902/00_summary.json`
- candidate search ON:
- `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`
Observed result:
- attempted variants: `8`
- selected variant: `0` (base)
- `failing_profile_count: 7 -> 7`
- no raw candidate uplift on this hard sample.
## Explicit Completion Signal
- Sprint 248: `DONE` (implemented + measured; no uplift on current sample)

30
docs/sprint249_execution_tracker_2026-02-26.md Normal file
View File

@@ -0,0 +1,30 @@
# Sprint 249 Execution Tracker - 2026-02-26
## Scope
- `sprint249_plan.md`
## Implemented
Pipeline updates in `tools/mcp/run_sprint_taskitem_pipeline.sh`:
- Added `WSTONE_NATIVE_RAW_CANDIDATE_REQUIRE_UPLIFT` (default `0`).
- `native_raw_candidate_search` now reports:
- `baseline_failing_profile_count`
- `best_failing_profile_count`
- baseline/best task counts
- Added artifact:
- `02ae_raw_candidate_search.json`
- Added hard-fail behavior (`exit 17`) when uplift is required but not achieved.
## Validation Artifacts
- guard disabled (records no-uplift signal, continues):
- `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154026/00_summary.json`
- guard enabled (expected hard fail):
- `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`
Observed signal:
- baseline and best both `7` failing profiles (`selected_variant=0`), so no uplift.
## Explicit Completion Signal
- Sprint 249: `DONE` (implemented + expected no-uplift guard behavior verified)