Add adaptive raw top-gap retry loop (sprint 259)

This commit is contained in:
Bill
2026-02-26 16:33:19 -07:00
parent df0de8156b
commit ba1e0dc3e8
7 changed files with 141 additions and 1 deletions

View File

@@ -228,3 +228,7 @@ Planning runtime controls (active):
- `WSTONE_NATIVE_RAW_TOP_GAP_REQUIREMENTS`
- `WSTONE_NATIVE_RAW_TOP_GAP_MAX_SIGNALS`
- `native_raw_top_gap_requirements`
- raw candidate search can run adaptive retry with expanded top-gap requirements:
- `WSTONE_NATIVE_RAW_TOP_GAP_ADAPTIVE_RETRY`
- `WSTONE_NATIVE_RAW_TOP_GAP_RETRY_SIGNALS`
- `native_raw_adaptive_retry`

View File

@@ -47,7 +47,7 @@ This is the canonical dated registry for "not production-ready" generator gaps.
| GR-018 | Performance/security/rollout constrained refactor enforcement gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/results.jsonl` | `partial` | Hard checks now enforce migration rollback + data-loss policy, security deny-by-default, SLO p95 presence, and rollout staged+abort policy. Enforcement is contract-level; generator capability under these constraints is still weak in C++ AB path. | Constraint-aware generation follow-up |
| GR-019 | Parity-blocked readiness load (gating without capability closure) | `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r7/summary.json`, `logs/taskitem_runs/challenging_subset_prod_20260226_r7/summary.json`, `docs/sprint225_227_execution_tracker_2026-02-26.md` | `partial` | Sprint 225-227 reduced blocked parity load from `6` to `0` on both tracked hard catalogs while keeping unresolved divergence at `0`. This closes immediate safety debt for current corpora, but robustness is still contingent on pattern-driven repair classes. | Generalize repairs beyond queue-shaped transpile outputs |
| GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json`, `logs/taskitem_runs/TEST_ONLY_sprint239_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint240_retry_20260226_145631/00_summary.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md`, `docs/sprint238_execution_tracker_2026-02-26.md`, `docs/sprint239_execution_tracker_2026-02-26.md`, `docs/sprint240_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Sprint 238 added native decomposition hard gate in pipeline (`min task count`, `min semantic signal count`) so weak native generation can be blocked before fallback masking. Sprint 239 added native reason enrichment and improved semantic signal density (`0 -> 6`). Sprint 240 added native decomposition retry with explicit minimum-task policy; on current sample retry attempted but did not improve depth (`2 -> 2`). | Native decomposition task-depth upgrade (increase native task granularity beyond 2) |
| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`, `logs/taskitem_runs/TEST_ONLY_sprint250_closure_ladder_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint251_rawvariants_20260226_154355/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint253_closure_ladder_batch_20260226/closure_ladder_batch_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint254_closure_ladder_batch_20260226_r2/raw_gap_backlog.json`, `logs/taskitem_runs/TEST_ONLY_sprint255_rawharden_20260226_161924/00_summary.json`, `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162423/00_summary.json`, `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162633/02ae_raw_candidate_search.json`, `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162924/01e_raw_top_gap_requirements_report.json` | `partial` | Sprint 241 introduced impact profiles/gates; 242-243 remediation loops; 244 first-pass autofill; 245 intrinsic constraints only (no uplift); 246 multishot closure; 247 single-shot shaping closure; 248 raw candidate search no uplift; 249 no-uplift guardrail; 250 closure ladder; 251 richer raw variants no uplift; 252 history-aware ladder routing; 253 batch ladder analytics; 254 raw gap backlog synthesis from `raw_only` attempts identifies top missing raw signals; 255 adds first-pass raw top-gap hardening that closes hard-sample profile coverage (`failing_profile_count 7 -> 0`) by injecting missing ops/contracts/reasons and minimum task depth; 256 adds top-gap weighted candidate selection telemetry and tie-break policy; 257 adds hard fail policy when top-gap uplift is required but absent (`exit 18`); 258 injects backlog-driven top-gap normalized requirements before raw generation, but measured intrinsic output still shows no uplift on this sample (`best_top_gap_score=0`, `failing_profile_count=1`). | Add adaptive raw retry loop that iteratively re-synthesizes and expands top-gap requirements when uplift remains zero |
| GR-021 | Impact-specific native decomposition coverage not enforced uniformly | `docs/native_decomposition_impact_list_2026-02-26.md`, `tools/mcp/profiles/native_decomposition_impact_profiles.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_fullstack_impact_20260226_150416/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint241_native_impact_aggregate_20260226/native_impact_coverage_aggregate.json`, `logs/taskitem_runs/TEST_ONLY_sprint242_impact_remediation_loop_20260226/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint243_impact_remediation_tasks_loop_20260226_r2/remediation_loop_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_20260226_151651/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint244_autofill_enforce_20260226_151709/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint245_intrinsic_20260226_151835/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_20260226_153033/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint246_multishot_enforce_20260226_153045/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_20260226_153230/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint247_singleshot_enforce_20260226_153241/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint248_rawsearch_20260226_153903/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint249_rawguard_20260226_154028/02ae_raw_candidate_search.json`, `logs/taskitem_runs/TEST_ONLY_sprint250_closure_ladder_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint251_rawvariants_20260226_154355/00_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint252_closure_ladder_policy_skip2_20260226/closure_ladder_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint253_closure_ladder_batch_20260226/closure_ladder_batch_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint254_closure_ladder_batch_20260226_r2/raw_gap_backlog.json`, `logs/taskitem_runs/TEST_ONLY_sprint255_rawharden_20260226_161924/00_summary.json`, `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162423/00_summary.json`, `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162633/02ae_raw_candidate_search.json`, `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162924/01e_raw_top_gap_requirements_report.json`, `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163148/02af_raw_adaptive_retry_score.json` | `partial` | Sprint 241 introduced impact profiles/gates; 242-243 remediation loops; 244 first-pass autofill; 245 intrinsic constraints only (no uplift); 246 multishot closure; 247 single-shot shaping closure; 248 raw candidate search no uplift; 249 no-uplift guardrail; 250 closure ladder; 251 richer raw variants no uplift; 252 history-aware ladder routing; 253 batch ladder analytics; 254 raw gap backlog synthesis from `raw_only` attempts identifies top missing raw signals; 255 adds first-pass raw top-gap hardening that closes hard-sample profile coverage (`failing_profile_count 7 -> 0`) by injecting missing ops/contracts/reasons and minimum task depth; 256 adds top-gap weighted candidate selection telemetry and tie-break policy; 257 adds hard fail policy when top-gap uplift is required but absent (`exit 18`); 258 injects backlog-driven top-gap normalized requirements before raw generation; 259 adds adaptive retry with expanded top-gap requirements and deterministic retry scoring. Measured intrinsic output still shows no uplift on this sample (`retry_top_gap_score=0`, `retry_fail=1`, `applied=false`). | Add model/prompt-level intrinsic generation intervention keyed by missing-signal classes (beyond requirement text injection alone) |
## What Was Covered Today (Sprints 175-184)

View File

@@ -458,3 +458,19 @@
- Current measured result on hard sample:
- injection selected `5` top signals from `20`
- intrinsic raw outcome unchanged (`selected_variant=0`, `best_top_gap_score=0`, `failing_profile_count=1`).
## Sprint 259 Added (Same Day)
- Added adaptive raw top-gap retry loop:
- `WSTONE_NATIVE_RAW_TOP_GAP_ADAPTIVE_RETRY`
- `WSTONE_NATIVE_RAW_TOP_GAP_RETRY_SIGNALS`
- summary packet:
- `native_raw_adaptive_retry`
- Retry executes when weighted top-gap score has no uplift, then applies only if better.
- Dated A/B artifacts:
- OFF: `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163143/00_summary.json`
- ON: `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163148/00_summary.json`
- retry detail: `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163148/02af_raw_adaptive_retry_score.json`
- Current measured result on hard sample:
- retry attempted and succeeded operationally
- no intrinsic uplift (`retry_top_gap_score=0`, `retry_fail=1`, `applied=false`).

View File

@@ -0,0 +1,43 @@
# Sprint 259 Execution Tracker - 2026-02-26
## Scope
- `sprint259_plan.md`
## Implemented
Pipeline adaptive retry in `tools/mcp/run_sprint_taskitem_pipeline.sh`:
- Added:
- `WSTONE_NATIVE_RAW_TOP_GAP_ADAPTIVE_RETRY`
- `WSTONE_NATIVE_RAW_TOP_GAP_RETRY_SIGNALS`
- Retry trigger condition:
- weighted top-gap mode enabled
- `best_top_gap_score <= baseline_top_gap_score`
- Retry behavior:
- synthesize expanded top-gap requirements
- rerun raw task generation
- score retry candidate
- apply retry output only if it improves fail count, top-gap score, or task count tie-break
- Added summary packet:
- `native_raw_adaptive_retry`
## Validation Artifacts
Adaptive retry OFF:
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163143/00_summary.json`
Adaptive retry ON:
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163148/00_summary.json`
- retry artifacts:
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163148/02af_raw_adaptive_retry.json`
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163148/02af_raw_adaptive_retry_score.json`
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_163148/02af_raw_adaptive_retry_requirements_report.json`
Observed result on hard sample:
- retry attempted successfully
- `retry_top_gap_score=0`
- `retry_fail=1`
- `applied=false` (no uplift to apply)
## Explicit Completion Signal
- Sprint 259: `DONE` (implemented + validated; no uplift on this sample)

View File

@@ -0,0 +1,6 @@
#pragma once
// Sprint 259 integration summary:
// - Added adaptive raw retry loop driven by top-gap weighted no-uplift condition.
// - Retry synthesizes expanded top-gap requirements and re-scores candidate output.
// - Summary now includes native_raw_adaptive_retry telemetry with attempted/applied status.

11
sprint259_plan.md Normal file
View File

@@ -0,0 +1,11 @@
# Sprint 259 Plan: Adaptive Raw Top-Gap Retry Loop
## Goal
Automatically run a second intrinsic raw-generation attempt with expanded top-gap requirements when weighted raw search shows no uplift.
## Steps
- Step 2342: Add adaptive retry controls and retry signal budget.
- Step 2343: Trigger retry when weighted top-gap score does not improve baseline.
- Step 2344: Score retry result and apply if it improves fail count/top-gap score/task depth.
- Step 2345: Emit retry telemetry and artifacts in summary.
- Step 2346: Add `Sprint259IntegrationSummary.h` and execution tracker.

View File

@@ -58,6 +58,8 @@ NATIVE_RAW_TOP_GAP_BACKLOG_FILE="${WSTONE_NATIVE_RAW_TOP_GAP_BACKLOG_FILE:-}"
NATIVE_RAW_TOP_GAP_REQUIRE_UPLIFT="${WSTONE_NATIVE_RAW_TOP_GAP_REQUIRE_UPLIFT:-0}"
NATIVE_RAW_TOP_GAP_REQUIREMENTS="${WSTONE_NATIVE_RAW_TOP_GAP_REQUIREMENTS:-0}"
NATIVE_RAW_TOP_GAP_MAX_SIGNALS="${WSTONE_NATIVE_RAW_TOP_GAP_MAX_SIGNALS:-5}"
NATIVE_RAW_TOP_GAP_ADAPTIVE_RETRY="${WSTONE_NATIVE_RAW_TOP_GAP_ADAPTIVE_RETRY:-0}"
NATIVE_RAW_TOP_GAP_RETRY_SIGNALS="${WSTONE_NATIVE_RAW_TOP_GAP_RETRY_SIGNALS:-10}"
EXTRA_NORMALIZED_REQUIREMENTS_FILE="${WSTONE_EXTRA_NORMALIZED_REQUIREMENTS_FILE:-}"
EXTRA_TASKS_FILE="${WSTONE_EXTRA_TASKS_FILE:-}"
CAPABILITY_SIGNALS_JSON="${WSTONE_CAPABILITY_SIGNALS_JSON:-}"
@@ -122,6 +124,7 @@ NATIVE_MULTISHOT_JSON='{}'
NATIVE_RAW_CANDIDATE_SEARCH_JSON='{}'
NATIVE_RAW_HARDENING_JSON='{}'
NATIVE_RAW_TOP_GAP_REQUIREMENTS_JSON='{}'
NATIVE_RAW_ADAPTIVE_RETRY_JSON='{}'
INTRINSIC_REQS_JSON='[]'
EXTRA_NORMALIZED_REQUIREMENTS_JSON='[]'
EXTRA_TASKS_JSON='[]'
@@ -495,6 +498,61 @@ if [[ "$NATIVE_RAW_CANDIDATE_SEARCH" == "1" ]]; then
variant=$((variant + 1))
done
if [[ "$NATIVE_RAW_TOP_GAP_ADAPTIVE_RETRY" == "1" && "$use_top_gap_weighted_select" == true && "$best_top_gap_score" -le "$baseline_top_gap_score" ]]; then
python3 "$ROOT_DIR/tools/mcp/synthesize_raw_top_gap_requirements.py" \
--top-gaps "$NATIVE_RAW_TOP_GAP_BACKLOG_FILE" \
--max-signals "$NATIVE_RAW_TOP_GAP_RETRY_SIGNALS" \
--out "$OUT_DIR/02af_raw_adaptive_retry_requirements.json" \
--out-report "$OUT_DIR/02af_raw_adaptive_retry_requirements_report.json" >/dev/null
retry_top_gap_reqs="$(cat "$OUT_DIR/02af_raw_adaptive_retry_requirements.json")"
retry_cand_reqs="$(jq -nc --argjson base "$NORMALIZED_REQS" --argjson extra "$retry_top_gap_reqs" '$base + $extra')"
retry_cand_args="$(jq -nc --argjson nr "$retry_cand_reqs" --argjson cf "$CONFLICTS" --arg strict "$STRICT_EXECUTION_CONTRACT" \
'{normalizedRequirements:$nr,conflicts:$cf,strictExecutionContract:($strict == "1")}')"
retry_raw="$(call_tool "whetstone_generate_taskitems" "$retry_cand_args")"
printf '%s\n' "$retry_raw" > "$OUT_DIR/02af_raw_adaptive_retry_raw.ndjson.json"
retry_json="$(extract_tool_text_json "$retry_raw")"
printf '%s\n' "$retry_json" > "$OUT_DIR/02af_raw_adaptive_retry.json"
retry_applied=false
retry_success=false
retry_fail=999
retry_task_count=0
retry_top_gap_score=0
if [[ "$(printf '%s' "$retry_json" | jq -r '.success // false')" == "true" ]]; then
retry_success=true
retry_tasks="$(printf '%s' "$retry_json" | jq '.tasks // []')"
printf '%s\n' "$retry_tasks" > "$OUT_DIR/02af_raw_adaptive_retry_tasks.json"
python3 "$ROOT_DIR/tools/mcp/score_native_tasks_profile_coverage.py" \
--spec "$INPUT_FILE" \
--profiles "$NATIVE_IMPACT_COVERAGE_PROFILES" \
--tasks "$OUT_DIR/02af_raw_adaptive_retry_tasks.json" \
"${top_gap_score_args[@]}" \
--out "$OUT_DIR/02af_raw_adaptive_retry_score.json" >/dev/null
retry_fail="$(jq '.failing_profile_count // 999' "$OUT_DIR/02af_raw_adaptive_retry_score.json")"
retry_task_count="$(jq '.task_count // 0' "$OUT_DIR/02af_raw_adaptive_retry_score.json")"
retry_top_gap_score="$(jq '.top_gap_score // 0' "$OUT_DIR/02af_raw_adaptive_retry_score.json")"
if [[ "$retry_fail" -lt "$best_fail" || ( "$retry_fail" -eq "$best_fail" && "$retry_top_gap_score" -gt "$best_top_gap_score" ) || ( "$retry_fail" -eq "$best_fail" && "$retry_top_gap_score" -eq "$best_top_gap_score" && "$retry_task_count" -gt "$best_task_count" ) ]]; then
best_variant="adaptive_retry"
best_fail="$retry_fail"
best_task_count="$retry_task_count"
best_top_gap_score="$retry_top_gap_score"
TASKS="$retry_tasks"
retry_applied=true
fi
fi
NATIVE_RAW_ADAPTIVE_RETRY_JSON="$(jq -nc \
--argjson enabled true \
--argjson attempted true \
--argjson applied "$retry_applied" \
--argjson success "$retry_success" \
--argjson retry_fail "$retry_fail" \
--argjson retry_task_count "$retry_task_count" \
--argjson retry_top_gap_score "$retry_top_gap_score" \
--argjson retry_signal_count "$NATIVE_RAW_TOP_GAP_RETRY_SIGNALS" \
'{enabled:$enabled, attempted:$attempted, applied:$applied, success:$success, retry_signal_count:$retry_signal_count, retry_fail:$retry_fail, retry_task_count:$retry_task_count, retry_top_gap_score:$retry_top_gap_score}')"
else
NATIVE_RAW_ADAPTIVE_RETRY_JSON='{"enabled":false}'
fi
NATIVE_RAW_CANDIDATE_SEARCH_JSON="$(jq -nc \
--argjson enabled true \
--argjson attempted "$attempted" \
@@ -877,6 +935,7 @@ SUMMARY_JSON="$(jq -nc \
--argjson native_intrinsic_boost "$NATIVE_INTRINSIC_BOOST_JSON" \
--argjson native_raw_top_gap_requirements "$NATIVE_RAW_TOP_GAP_REQUIREMENTS_JSON" \
--argjson native_raw_candidate_search "$NATIVE_RAW_CANDIDATE_SEARCH_JSON" \
--argjson native_raw_adaptive_retry "$NATIVE_RAW_ADAPTIVE_RETRY_JSON" \
--argjson native_raw_hardening "$NATIVE_RAW_HARDENING_JSON" \
--argjson native_single_shot_profile_shape "$NATIVE_SINGLESHOT_PROFILE_SHAPE_JSON" \
--argjson native_multishot "$NATIVE_MULTISHOT_JSON" \
@@ -907,6 +966,7 @@ SUMMARY_JSON="$(jq -nc \
native_intrinsic_boost: $native_intrinsic_boost,
native_raw_top_gap_requirements: $native_raw_top_gap_requirements,
native_raw_candidate_search: $native_raw_candidate_search,
native_raw_adaptive_retry: $native_raw_adaptive_retry,
native_raw_hardening: $native_raw_hardening,
native_single_shot_profile_shape: $native_single_shot_profile_shape,
native_multishot: $native_multishot,