whetstone_DSL/docs/deterministic_gap_hunt_2026-02-26.md

Deterministic Gap Hunt Report - 2026-02-26

1) Strict Backfill Audit of Sprints 186-205

Reference:

• docs/sprint186_205_intent_audit_2026-02-26.md
• docs/sprint186_205_closeout_report_2026-02-26.md

Signal split:

• Pipeline closeout (EXECUTION-READY): 20/20
• Strict intent closure:
  • CLOSED: 1/20
  • PARTIAL: 10/20
  • GAP: 9/20

Interpretation:

• Operational gates are green.
• Many original sprint intents remain partially implemented or scaffolded.

2) New Semantically Challenging Project Catalogs

Created:

• datasets/project_benchmarks/challenging_deterministic_projects_2026-02-26.jsonl (12 hard projects)
• datasets/project_benchmarks/challenging_subset_prod_2026-02-26.jsonl (focused production-loop subset)
• datasets/project_benchmarks/challenging_projection_projects_2026-02-26.jsonl (projection/environment constraints)
• docs/challenging_deterministic_projects_2026-02-26.md

3) Benchmark Results and New Gap Signals

A) Hard matrix (12 projects x 4 languages)

Run:

• logs/taskitem_runs/challenging_matrix_20260226

Summary highlights:

• total_runs=48
• A/B Path A ready: 77.08%
• A/B Path B ready: 25.0%
• Path B top failures: compile_failed:non_zero_exit, tests_failed:non_zero_exit (36 each)
• Avg token ratio PathB/PathA: 13.67x

B) Focused production subset

Run:

• logs/taskitem_runs/challenging_subset_prod_20260226

Summary highlights:

• total_runs=8
• A/B path B still fails on cpp cases.
• Production loop reports ready=100% for all 8 cases, including cases where A/B path B failed compile/tests.
• Several production summaries show sparse/null gate evidence fields despite green outcome.

C) Projection constraints subset

Run:

• logs/taskitem_runs/challenging_projection_matrix_20260226

Summary highlights:

• Projection-heavy specs are accepted as free-text, but constraint fields are not enforced as first-class contract inputs in benchmark orchestration.
• Confirms projection/environment constraints are under-modeled in current automated benchmark loop.

4) New Gap IDs (Post-186-205)

• GR-011 False-green production gating

  • Symptom: production loop marks ready where A/B compilation/tests fail on equivalent hard specs.
  • Evidence: challenging_subset_prod_20260226 vs A/B failures.

• GR-012 Projection constraint contract missing in benchmark/execution path

  • Symptom: target/environment budgets and forbidden APIs are not treated as first-class executable constraints.
  • Evidence: projection catalog accepted, but orchestration path consumes mostly spec text.

• GR-013 Language asymmetry under hard semantics

  • Symptom: Python passes much more frequently than cpp/go/rust on equivalent hard specs.
  • Evidence: challenging_matrix_20260226 language breakdown.

• GR-014 Token inefficiency and semantic collapse in language-first path

  • Symptom: Path B token use is ~14x higher with poor readiness under hard tasks.
  • Evidence: hard matrix summaries.

• GR-015 Intent-vs-closeout drift risk

  • Symptom: sprint closeout can be green while original intent remains partial.
  • Evidence: strict intent audit vs closeout report divergence.

5) Recommended Next Sprint Theme

• Hard-gate production readiness on explicit compile/test evidence.
• Add projection-constraint schema to benchmark and pipeline tool inputs.
• Add cross-language parity gates for hard semantic specs.
• Add anti-false-green checks linking A/B and production evidence packets.