87 lines
3.4 KiB
Markdown
87 lines
3.4 KiB
Markdown
|
# Deterministic Gap Hunt Report - 2026-02-26
|
||
|
|
|
||
|
|
## 1) Strict Backfill Audit of Sprints 186-205
|
||
|
|
|
||
|
|
Reference:
|
||
|
|
- `docs/sprint186_205_intent_audit_2026-02-26.md`
|
||
|
|
- `docs/sprint186_205_closeout_report_2026-02-26.md`
|
||
|
|
|
||
|
|
Signal split:
|
||
|
|
- Pipeline closeout (`EXECUTION-READY`): `20/20`
|
||
|
|
- Strict intent closure:
|
||
|
|
- `CLOSED`: `1/20`
|
||
|
|
- `PARTIAL`: `10/20`
|
||
|
|
- `GAP`: `9/20`
|
||
|
|
|
||
|
|
Interpretation:
|
||
|
|
- Operational gates are green.
|
||
|
|
- Many original sprint intents remain partially implemented or scaffolded.
|
||
|
|
|
||
|
|
## 2) New Semantically Challenging Project Catalogs
|
||
|
|
|
||
|
|
Created:
|
||
|
|
- `datasets/project_benchmarks/challenging_deterministic_projects_2026-02-26.jsonl` (12 hard projects)
|
||
|
|
- `datasets/project_benchmarks/challenging_subset_prod_2026-02-26.jsonl` (focused production-loop subset)
|
||
|
|
- `datasets/project_benchmarks/challenging_projection_projects_2026-02-26.jsonl` (projection/environment constraints)
|
||
|
|
- `docs/challenging_deterministic_projects_2026-02-26.md`
|
||
|
|
|
||
|
|
## 3) Benchmark Results and New Gap Signals
|
||
|
|
|
||
|
|
### A) Hard matrix (12 projects x 4 languages)
|
||
|
|
Run:
|
||
|
|
- `logs/taskitem_runs/challenging_matrix_20260226`
|
||
|
|
|
||
|
|
Summary highlights:
|
||
|
|
- `total_runs=48`
|
||
|
|
- A/B Path A ready: `77.08%`
|
||
|
|
- A/B Path B ready: `25.0%`
|
||
|
|
- Path B top failures: `compile_failed:non_zero_exit`, `tests_failed:non_zero_exit` (36 each)
|
||
|
|
- Avg token ratio `PathB/PathA`: `13.67x`
|
||
|
|
|
||
|
|
### B) Focused production subset
|
||
|
|
Run:
|
||
|
|
- `logs/taskitem_runs/challenging_subset_prod_20260226`
|
||
|
|
|
||
|
|
Summary highlights:
|
||
|
|
- `total_runs=8`
|
||
|
|
- A/B path B still fails on cpp cases.
|
||
|
|
- Production loop reports `ready=100%` for all 8 cases, including cases where A/B path B failed compile/tests.
|
||
|
|
- Several production summaries show sparse/null gate evidence fields despite green outcome.
|
||
|
|
|
||
|
|
### C) Projection constraints subset
|
||
|
|
Run:
|
||
|
|
- `logs/taskitem_runs/challenging_projection_matrix_20260226`
|
||
|
|
|
||
|
|
Summary highlights:
|
||
|
|
- Projection-heavy specs are accepted as free-text, but constraint fields are not enforced as first-class contract inputs in benchmark orchestration.
|
||
|
|
- Confirms projection/environment constraints are under-modeled in current automated benchmark loop.
|
||
|
|
|
||
|
|
## 4) New Gap IDs (Post-186-205)
|
||
|
|
|
||
|
|
- `GR-011` False-green production gating
|
||
|
|
- Symptom: production loop marks ready where A/B compilation/tests fail on equivalent hard specs.
|
||
|
|
- Evidence: `challenging_subset_prod_20260226` vs A/B failures.
|
||
|
|
|
||
|
|
- `GR-012` Projection constraint contract missing in benchmark/execution path
|
||
|
|
- Symptom: target/environment budgets and forbidden APIs are not treated as first-class executable constraints.
|
||
|
|
- Evidence: projection catalog accepted, but orchestration path consumes mostly `spec` text.
|
||
|
|
|
||
|
|
- `GR-013` Language asymmetry under hard semantics
|
||
|
|
- Symptom: Python passes much more frequently than cpp/go/rust on equivalent hard specs.
|
||
|
|
- Evidence: `challenging_matrix_20260226` language breakdown.
|
||
|
|
|
||
|
|
- `GR-014` Token inefficiency and semantic collapse in language-first path
|
||
|
|
- Symptom: Path B token use is ~14x higher with poor readiness under hard tasks.
|
||
|
|
- Evidence: hard matrix summaries.
|
||
|
|
|
||
|
|
- `GR-015` Intent-vs-closeout drift risk
|
||
|
|
- Symptom: sprint closeout can be green while original intent remains partial.
|
||
|
|
- Evidence: strict intent audit vs closeout report divergence.
|
||
|
|
|
||
|
|
## 5) Recommended Next Sprint Theme
|
||
|
|
|
||
|
|
- Hard-gate production readiness on explicit compile/test evidence.
|
||
|
|
- Add projection-constraint schema to benchmark and pipeline tool inputs.
|
||
|
|
- Add cross-language parity gates for hard semantic specs.
|
||
|
|
- Add anti-false-green checks linking A/B and production evidence packets.
|