Add batch hardening report for drive and product specs
This commit is contained in:
@@ -67,3 +67,8 @@
|
||||
- Demonstrated hard-gate delta on sprint sample:
|
||||
- original spec blocked (`rc=7`, score `17`)
|
||||
- hardened spec passed (`rc=0`, score `100`)
|
||||
- Executed batch hardening on real corpora:
|
||||
- drive specs (7)
|
||||
- example product meta specs (10)
|
||||
- Batch report:
|
||||
- `docs/spec_hardening_batch_report_2026-02-26.md`
|
||||
|
||||
55
docs/spec_hardening_batch_report_2026-02-26.md
Normal file
55
docs/spec_hardening_batch_report_2026-02-26.md
Normal file
@@ -0,0 +1,55 @@
|
||||
# Spec Hardening Batch Report - 2026-02-26
|
||||
|
||||
## Scope
|
||||
|
||||
Batches executed with:
|
||||
- `tools/mcp/run_spec_hardening_gate.sh`
|
||||
|
||||
Corpora:
|
||||
1. Drive project docs (`/home/bill/Documents/drive_projects/...`) - 7 specs
|
||||
2. Example product meta specs (`example_projects/*/PROJECT_META.md`) - 10 specs
|
||||
|
||||
## Batch Artifacts
|
||||
|
||||
Drive batch:
|
||||
- `logs/taskitem_runs/drive_spec_hardening_batch_20260226/results.jsonl`
|
||||
- `logs/taskitem_runs/drive_spec_hardening_batch_20260226/summary.json`
|
||||
- `logs/taskitem_runs/drive_spec_hardening_batch_20260226/quality_summary.json`
|
||||
|
||||
Example project meta batch:
|
||||
- `logs/taskitem_runs/example_projectmeta_hardening_batch_20260226/results.jsonl`
|
||||
- `logs/taskitem_runs/example_projectmeta_hardening_batch_20260226/summary.json`
|
||||
|
||||
## High-Density Outcomes
|
||||
|
||||
Readiness/hard-gate behavior:
|
||||
- Original specs: all blocked by hard gate (`original_rc=7`)
|
||||
- Hardened specs: all passed (`hardened_rc=0`)
|
||||
- Readiness scores improved to `100` for all hardened outputs.
|
||||
|
||||
Drive batch deltas:
|
||||
- before score range: `15..22`
|
||||
- after score range: `100..100`
|
||||
- delta range: `78..85`
|
||||
|
||||
Example product-meta deltas:
|
||||
- before score range: `0..8`
|
||||
- after score range: `100..100`
|
||||
- delta range: `92..100`
|
||||
|
||||
## Important Limitation Discovered
|
||||
|
||||
Downstream taskitem metrics are currently low-discrimination across both batches:
|
||||
- `task_count` consistently `2`
|
||||
- `validation.average_score` consistently `87.5`
|
||||
- `validation.average_execution_specificity_score` consistently `75.0`
|
||||
- `validation.failing_count` consistently `0`
|
||||
|
||||
Interpretation:
|
||||
- The new planning gate/hardening loop is working as intended for pre-execution quality control.
|
||||
- But current sprint pipeline translation quality signal is too flat for heterogeneous product specs.
|
||||
- Next required work: increase spec-to-taskitem semantic sensitivity (more variable decomposition and specificity based on domain constraints).
|
||||
|
||||
## Recommendation
|
||||
|
||||
Use real specs first (drive + product metas), then generate synthetic specs only to cover missing domains after translation-sensitivity improvements are in place.
|
||||
Reference in New Issue
Block a user