From 5eaaa7cc56f5bb810b6309ca830393adebd88053 Mon Sep 17 00:00:00 2001 From: Bill Date: Thu, 26 Feb 2026 14:54:15 -0700 Subject: [PATCH] Add native decomposition quality gate to pipeline (sprint 238) --- docs/constructive_editing_runtime_plan.md | 4 +- ...rator_readiness_gap_registry_2026-02-26.md | 2 +- docs/progress_log_2026-02-26.md | 13 ++++++ .../sprint238_execution_tracker_2026-02-26.md | 28 +++++++++++++ editor/src/Sprint238IntegrationSummary.h | 6 +++ sprint238_plan.md | 11 +++++ tools/mcp/run_sprint_taskitem_pipeline.sh | 40 +++++++++++++++++++ 7 files changed, 102 insertions(+), 2 deletions(-) create mode 100644 docs/sprint238_execution_tracker_2026-02-26.md create mode 100644 editor/src/Sprint238IntegrationSummary.h create mode 100644 sprint238_plan.md diff --git a/docs/constructive_editing_runtime_plan.md b/docs/constructive_editing_runtime_plan.md index 7fb8d3b..37a2116 100644 --- a/docs/constructive_editing_runtime_plan.md +++ b/docs/constructive_editing_runtime_plan.md @@ -150,7 +150,7 @@ Parallel planning tranche (active): - `logs/taskitem_runs/TEST_ONLY_spec_planning_baseline_20260226/deterministic_spec_readiness.json` Planning runtime controls (active): -- `sprint232_plan.md` to `sprint237_plan.md` +- `sprint232_plan.md` to `sprint238_plan.md` - semantic bridge + intake augmentation + requirement injection + expansion gating are wired into: - `tools/mcp/run_sprint_taskitem_pipeline.sh` - current policy default is native-first semantic fallback: @@ -160,3 +160,5 @@ Planning runtime controls (active): - fallback budget gating is available via: - `tools/mcp/check_semantic_fallback_budget.py` - `tools/mcp/run_semantic_fallback_budget_gate.sh` +- native decomposition quality gate is available in pipeline summary/hard-fail path: + - `native_decomposition_gate` diff --git a/docs/generator_readiness_gap_registry_2026-02-26.md b/docs/generator_readiness_gap_registry_2026-02-26.md index edc963b..86c2e1e 100644 --- a/docs/generator_readiness_gap_registry_2026-02-26.md +++ b/docs/generator_readiness_gap_registry_2026-02-26.md @@ -46,7 +46,7 @@ This is the canonical dated registry for "not production-ready" generator gaps. | GR-017 | Full-stack contract propagation gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/fullstack_contract_closure.json` | `partial` | Full-stack contract packet + strict validation added in benchmark orchestration and summary aggregation. Contract invalid rate is `0%` on valid catalog, but cross-layer semantic consistency is not yet equivalence-verified. | Semantic propagation verifier sprint | | GR-018 | Performance/security/rollout constrained refactor enforcement gap | `docs/gap_hunt_fullstack_multifile_2026-02-26.md`, `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r5/results.jsonl` | `partial` | Hard checks now enforce migration rollback + data-loss policy, security deny-by-default, SLO p95 presence, and rollout staged+abort policy. Enforcement is contract-level; generator capability under these constraints is still weak in C++ AB path. | Constraint-aware generation follow-up | | GR-019 | Parity-blocked readiness load (gating without capability closure) | `logs/taskitem_runs/challenging_fullstack_multifile_20260226_r7/summary.json`, `logs/taskitem_runs/challenging_subset_prod_20260226_r7/summary.json`, `docs/sprint225_227_execution_tracker_2026-02-26.md` | `partial` | Sprint 225-227 reduced blocked parity load from `6` to `0` on both tracked hard catalogs while keeping unresolved divergence at `0`. This closes immediate safety debt for current corpora, but robustness is still contingent on pattern-driven repair classes. | Generalize repairs beyond queue-shaped transpile outputs | -| GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Current measured slice remains `fallback_rate=1.0`, so gating now detects but does not yet fix root capability gaps. | Native decomposition upgrade + semantic rationale enforcement | +| GR-020 | Semantic fallback overuse masks weak native decomposition | `logs/taskitem_runs/TEST_ONLY_sprint236_semantic_fallback_audit_20260226/semantic_fallback_summary.json`, `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json`, `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json`, `docs/sprint236_execution_tracker_2026-02-26.md`, `docs/sprint237_execution_tracker_2026-02-26.md`, `docs/sprint238_execution_tracker_2026-02-26.md` | `partial` | Sprint 236 added deterministic fallback-gap auditing with tool + constraint metadata. Sprint 237 added fallback-budget hard gate (`max-fallback-rate`) and produced expected fail/pass artifacts. Sprint 238 added native decomposition hard gate in pipeline (`min task count`, `min semantic signal count`) so weak native generation can be blocked before fallback masking. Current measured slice still shows weak native decomposition (`2 tasks`, `0 semantic signals`) and needs generator-quality uplift to close. | Native decomposition upgrade + semantic rationale enforcement | ## What Was Covered Today (Sprints 175-184) diff --git a/docs/progress_log_2026-02-26.md b/docs/progress_log_2026-02-26.md index f2acf12..4d1a859 100644 --- a/docs/progress_log_2026-02-26.md +++ b/docs/progress_log_2026-02-26.md @@ -147,3 +147,16 @@ - Dated pass/fail artifacts captured: - fail (strict budget): `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_fail_20260226/semantic_fallback_budget_gate.json` - pass (relaxed budget): `logs/taskitem_runs/TEST_ONLY_sprint237_semantic_fallback_gate_pass_20260226/semantic_fallback_budget_gate.json` + +## Sprint 238 Added (Same Day) + +- Added native decomposition quality gate directly in pipeline: + - `WSTONE_NATIVE_DECOMP_HARD_GATE` + - `WSTONE_NATIVE_TASK_MIN_COUNT` + - `WSTONE_NATIVE_SEMANTIC_SIGNAL_MIN` +- Added run artifact and summary packet: + - `02b_native_decomposition_gate.json` + - `native_decomposition_gate` +- Dated gate verification artifacts: + - fail packet: `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json` + - pass packet: `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_pass_20260226.json` diff --git a/docs/sprint238_execution_tracker_2026-02-26.md b/docs/sprint238_execution_tracker_2026-02-26.md new file mode 100644 index 0000000..894afb7 --- /dev/null +++ b/docs/sprint238_execution_tracker_2026-02-26.md @@ -0,0 +1,28 @@ +# Sprint 238 Execution Tracker - 2026-02-26 + +## Scope +- `sprint238_plan.md` + +## Implemented + +Pipeline updates in `tools/mcp/run_sprint_taskitem_pipeline.sh`: +- Added native decomposition gate controls: + - `WSTONE_NATIVE_DECOMP_HARD_GATE` (default `0`) + - `WSTONE_NATIVE_TASK_MIN_COUNT` (default `0`) + - `WSTONE_NATIVE_SEMANTIC_SIGNAL_MIN` (default `0`) +- Added gate artifact output: + - `02b_native_decomposition_gate.json` +- Added summary packet: + - `native_decomposition_gate` +- Added hard-fail behavior when enabled and thresholds are not met. + +## Baseline Gate Artifacts + +- expected fail gate packet: + - `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_fail_20260226.json` +- expected pass gate packet: + - `logs/taskitem_runs/TEST_ONLY_sprint238_native_gate_pass_20260226.json` + +## Explicit Completion Signal + +- Sprint 238: `DONE` (implemented + hard-fail/pass smoke verified) diff --git a/editor/src/Sprint238IntegrationSummary.h b/editor/src/Sprint238IntegrationSummary.h new file mode 100644 index 0000000..733c52d --- /dev/null +++ b/editor/src/Sprint238IntegrationSummary.h @@ -0,0 +1,6 @@ +#pragma once + +// Sprint 238 integration summary: +// - Added native decomposition quality gate with optional hard-fail mode. +// - Gate checks min native task count and semantic signal count before fallback. +// - Pipeline now emits a native decomposition gate artifact and summary packet. diff --git a/sprint238_plan.md b/sprint238_plan.md new file mode 100644 index 0000000..ca7488d --- /dev/null +++ b/sprint238_plan.md @@ -0,0 +1,11 @@ +# Sprint 238 Plan: Native Decomposition Quality Gate + +## Goal +Detect and optionally block weak native decomposition before semantic fallback hides generation deficiencies. + +## Steps +- Step 2233: Add native decomposition gate packet to pipeline summary. +- Step 2234: Support configurable minimum native task count and semantic signal count. +- Step 2235: Add optional hard-fail mode for gate violations. +- Step 2236: Persist gate artifact for every pipeline run. +- Step 2237: Add `Sprint238IntegrationSummary.h` and execution tracker. diff --git a/tools/mcp/run_sprint_taskitem_pipeline.sh b/tools/mcp/run_sprint_taskitem_pipeline.sh index aeed54f..8371b68 100755 --- a/tools/mcp/run_sprint_taskitem_pipeline.sh +++ b/tools/mcp/run_sprint_taskitem_pipeline.sh @@ -33,6 +33,9 @@ SEMANTIC_COVERAGE_GATE="${WSTONE_SEMANTIC_COVERAGE_GATE:-0}" SEMANTIC_MIN_COVERAGE="${WSTONE_SEMANTIC_MIN_COVERAGE:-0.9}" SEMANTIC_REQUIRE_CAPS_FOR_COMPLEX="${WSTONE_SEMANTIC_REQUIRE_CAPS_FOR_COMPLEX:-1}" SEMANTIC_MIN_COMPLEXITY_SCORE="${WSTONE_SEMANTIC_MIN_COMPLEXITY_SCORE:-2}" +NATIVE_DECOMP_HARD_GATE="${WSTONE_NATIVE_DECOMP_HARD_GATE:-0}" +NATIVE_TASK_MIN_COUNT="${WSTONE_NATIVE_TASK_MIN_COUNT:-0}" +NATIVE_SEMANTIC_SIGNAL_MIN="${WSTONE_NATIVE_SEMANTIC_SIGNAL_MIN:-0}" CAPABILITY_SIGNALS_JSON="${WSTONE_CAPABILITY_SIGNALS_JSON:-}" if [[ -z "$CAPABILITY_SIGNALS_JSON" ]]; then CAPABILITY_SIGNALS_JSON='{}' @@ -84,6 +87,7 @@ SEMANTIC_AUGMENT_JSON='{}' SEMANTIC_REQUIREMENT_INJECTION_JSON='{}' SEMANTIC_TASK_EXPANSION_JSON='{}' SEMANTIC_GATE_JSON='{}' +NATIVE_DECOMP_GATE_JSON='{}' if [[ "$SEMANTIC_PLANNING_BRIDGE" == "1" ]]; then python3 "$ROOT_DIR/tools/mcp/markdown_to_semantic_annotations.py" \ --spec "$INPUT_FILE" \ @@ -288,6 +292,40 @@ fi TASKS="$(printf '%s' "$GEN_JSON" | jq '.tasks')" native_task_count="$(printf '%s' "$TASKS" | jq 'length')" native_semantic_signal_count="$(printf '%s' "$TASKS" | jq '[.[] | ((.reasons // [])[]? | tostring) | select(test("semantic|risk|contract|capability"; "i"))] | length')" +native_task_count_ok="true" +native_semantic_signal_ok="true" +if [[ "$native_task_count" -lt "$NATIVE_TASK_MIN_COUNT" ]]; then + native_task_count_ok="false" +fi +if [[ "$native_semantic_signal_count" -lt "$NATIVE_SEMANTIC_SIGNAL_MIN" ]]; then + native_semantic_signal_ok="false" +fi +native_gate_passed="true" +if [[ "$native_task_count_ok" != "true" || "$native_semantic_signal_ok" != "true" ]]; then + native_gate_passed="false" +fi +NATIVE_DECOMP_GATE_JSON="$(jq -nc \ + --argjson enabled "$([[ "$NATIVE_DECOMP_HARD_GATE" == "1" ]] && echo true || echo false)" \ + --argjson passed "$([[ "$native_gate_passed" == "true" ]] && echo true || echo false)" \ + --argjson native_task_count "$native_task_count" \ + --argjson native_semantic_signal_count "$native_semantic_signal_count" \ + --argjson min_task_count "$NATIVE_TASK_MIN_COUNT" \ + --argjson min_semantic_signal_count "$NATIVE_SEMANTIC_SIGNAL_MIN" \ + '{ + enabled:$enabled, + passed:$passed, + native_task_count:$native_task_count, + native_semantic_signal_count:$native_semantic_signal_count, + min_task_count:$min_task_count, + min_semantic_signal_count:$min_semantic_signal_count + }')" +printf '%s\n' "$NATIVE_DECOMP_GATE_JSON" > "$OUT_DIR/02b_native_decomposition_gate.json" +if [[ "$NATIVE_DECOMP_HARD_GATE" == "1" && "$native_gate_passed" != "true" ]]; then + echo "error: native decomposition gate failed" >&2 + echo "error: see $OUT_DIR/02b_native_decomposition_gate.json" >&2 + exit 10 +fi + semantic_fallback_needed="false" if [[ "$SEMANTIC_TASK_EXPANSION_MODE" == "always" ]]; then semantic_fallback_needed="true" @@ -383,6 +421,7 @@ SUMMARY_JSON="$(jq -nc \ --argjson semantic_intake_augment "$SEMANTIC_AUGMENT_JSON" \ --argjson semantic_requirement_injection "$SEMANTIC_REQUIREMENT_INJECTION_JSON" \ --argjson semantic_task_expansion "$SEMANTIC_TASK_EXPANSION_JSON" \ + --argjson native_decomposition_gate "$NATIVE_DECOMP_GATE_JSON" \ --argjson intake "$INTAKE_JSON" \ --argjson generated "$GEN_JSON" \ --argjson queue "$QUEUE_JSON" \ @@ -400,6 +439,7 @@ SUMMARY_JSON="$(jq -nc \ semantic_intake_augment: $semantic_intake_augment, semantic_requirement_injection: $semantic_requirement_injection, semantic_task_expansion: $semantic_task_expansion, + native_decomposition_gate: $native_decomposition_gate, intake: { success: ($intake.success // false), normalized_requirement_count: (($intake.normalizedRequirements // [])|length),