Add batch hardening report for drive and product specs
This commit is contained in:
@@ -18,6 +18,7 @@
|
|||||||
- `tools/mcp/spec_planning_readiness.py`
|
- `tools/mcp/spec_planning_readiness.py`
|
||||||
- `tools/mcp/spec_planning_hardener.py`
|
- `tools/mcp/spec_planning_hardener.py`
|
||||||
- `tools/mcp/run_spec_hardening_gate.sh`
|
- `tools/mcp/run_spec_hardening_gate.sh`
|
||||||
|
- `docs/spec_hardening_batch_report_2026-02-26.md`
|
||||||
|
|
||||||
## Dated Handoff
|
## Dated Handoff
|
||||||
|
|
||||||
|
|||||||
@@ -67,3 +67,8 @@
|
|||||||
- Demonstrated hard-gate delta on sprint sample:
|
- Demonstrated hard-gate delta on sprint sample:
|
||||||
- original spec blocked (`rc=7`, score `17`)
|
- original spec blocked (`rc=7`, score `17`)
|
||||||
- hardened spec passed (`rc=0`, score `100`)
|
- hardened spec passed (`rc=0`, score `100`)
|
||||||
|
- Executed batch hardening on real corpora:
|
||||||
|
- drive specs (7)
|
||||||
|
- example product meta specs (10)
|
||||||
|
- Batch report:
|
||||||
|
- `docs/spec_hardening_batch_report_2026-02-26.md`
|
||||||
|
|||||||
55
docs/spec_hardening_batch_report_2026-02-26.md
Normal file
55
docs/spec_hardening_batch_report_2026-02-26.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Spec Hardening Batch Report - 2026-02-26
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
Batches executed with:
|
||||||
|
- `tools/mcp/run_spec_hardening_gate.sh`
|
||||||
|
|
||||||
|
Corpora:
|
||||||
|
1. Drive project docs (`/home/bill/Documents/drive_projects/...`) - 7 specs
|
||||||
|
2. Example product meta specs (`example_projects/*/PROJECT_META.md`) - 10 specs
|
||||||
|
|
||||||
|
## Batch Artifacts
|
||||||
|
|
||||||
|
Drive batch:
|
||||||
|
- `logs/taskitem_runs/drive_spec_hardening_batch_20260226/results.jsonl`
|
||||||
|
- `logs/taskitem_runs/drive_spec_hardening_batch_20260226/summary.json`
|
||||||
|
- `logs/taskitem_runs/drive_spec_hardening_batch_20260226/quality_summary.json`
|
||||||
|
|
||||||
|
Example project meta batch:
|
||||||
|
- `logs/taskitem_runs/example_projectmeta_hardening_batch_20260226/results.jsonl`
|
||||||
|
- `logs/taskitem_runs/example_projectmeta_hardening_batch_20260226/summary.json`
|
||||||
|
|
||||||
|
## High-Density Outcomes
|
||||||
|
|
||||||
|
Readiness/hard-gate behavior:
|
||||||
|
- Original specs: all blocked by hard gate (`original_rc=7`)
|
||||||
|
- Hardened specs: all passed (`hardened_rc=0`)
|
||||||
|
- Readiness scores improved to `100` for all hardened outputs.
|
||||||
|
|
||||||
|
Drive batch deltas:
|
||||||
|
- before score range: `15..22`
|
||||||
|
- after score range: `100..100`
|
||||||
|
- delta range: `78..85`
|
||||||
|
|
||||||
|
Example product-meta deltas:
|
||||||
|
- before score range: `0..8`
|
||||||
|
- after score range: `100..100`
|
||||||
|
- delta range: `92..100`
|
||||||
|
|
||||||
|
## Important Limitation Discovered
|
||||||
|
|
||||||
|
Downstream taskitem metrics are currently low-discrimination across both batches:
|
||||||
|
- `task_count` consistently `2`
|
||||||
|
- `validation.average_score` consistently `87.5`
|
||||||
|
- `validation.average_execution_specificity_score` consistently `75.0`
|
||||||
|
- `validation.failing_count` consistently `0`
|
||||||
|
|
||||||
|
Interpretation:
|
||||||
|
- The new planning gate/hardening loop is working as intended for pre-execution quality control.
|
||||||
|
- But current sprint pipeline translation quality signal is too flat for heterogeneous product specs.
|
||||||
|
- Next required work: increase spec-to-taskitem semantic sensitivity (more variable decomposition and specificity based on domain constraints).
|
||||||
|
|
||||||
|
## Recommendation
|
||||||
|
|
||||||
|
Use real specs first (drive + product metas), then generate synthetic specs only to cover missing domains after translation-sensitivity improvements are in place.
|
||||||
Reference in New Issue
Block a user