Add RSA extraction bootstrap and working notes
This commit is contained in:
117
docs/case_studies/whetstone_dsl_extraction_yield_snapshot.md
Normal file
117
docs/case_studies/whetstone_dsl_extraction_yield_snapshot.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# WhetstoneDSL Extraction Yield Snapshot
|
||||
|
||||
## Slice
|
||||
|
||||
Parser:
|
||||
|
||||
- `tools/extract_gate_rows.py`
|
||||
|
||||
Contracts:
|
||||
|
||||
- `semantic/rsa_gate_extraction_contracts_v0.json`
|
||||
|
||||
Input slice:
|
||||
|
||||
- first 50 run directories under
|
||||
`CLionProjects/whetstone_DSL/logs/taskitem_runs`
|
||||
|
||||
Output:
|
||||
|
||||
- `semantic/extracted_gate_rows_slice.ndjson`
|
||||
|
||||
## Summary
|
||||
|
||||
Observed counts from the 50-run slice:
|
||||
|
||||
- runs processed: `50`
|
||||
- tasks processed: `106`
|
||||
- rows emitted: `424`
|
||||
- rows with rejection flags: `330`
|
||||
|
||||
Per-gate emitted rows:
|
||||
|
||||
- `prereq_op_selector`: `106`
|
||||
- `target_file_selection`: `106`
|
||||
- `acceptance_command_selection`: `106`
|
||||
- `required_tool_selection`: `106`
|
||||
|
||||
Per-gate rejected rows:
|
||||
|
||||
- `prereq_op_selector`: `12`
|
||||
- `target_file_selection`: `106`
|
||||
- `acceptance_command_selection`: `106`
|
||||
- `required_tool_selection`: `106`
|
||||
|
||||
## Dominant Rejection Flags
|
||||
|
||||
- `sibling_uniform_target_files`: `106`
|
||||
- `sibling_uniform_acceptance_commands`: `106`
|
||||
- `sibling_uniform_required_tools`: `106`
|
||||
- `invalid_prerequisite_values:whetstone_generate_taskitems,whetstone_queue_ready,whetstone_validate_taskitem`: `12`
|
||||
|
||||
## Interpretation
|
||||
|
||||
### `prereq_op_selector`
|
||||
|
||||
This is the strongest current extraction surface.
|
||||
|
||||
It survives most of the slice cleanly, and the rejection cases are informative
|
||||
rather than noisy. The rejected rows come from schema-drift runs where
|
||||
`prerequisiteOps` was contaminated with tool ids such as:
|
||||
|
||||
- `whetstone_generate_taskitems`
|
||||
- `whetstone_queue_ready`
|
||||
- `whetstone_validate_taskitem`
|
||||
|
||||
That means the parser is catching real label pollution rather than inventing
|
||||
false negatives.
|
||||
|
||||
### `target_file_selection`
|
||||
|
||||
Present explicitly, but not yet trustworthy as broad supervision in this slice.
|
||||
|
||||
The dominant issue is sibling-uniform output: all tasks in a run often carry the
|
||||
same target-file list, which makes naive extraction look more like template
|
||||
reuse than local bounded choice.
|
||||
|
||||
### `acceptance_command_selection`
|
||||
|
||||
Present explicitly, but also highly template-like in this slice.
|
||||
|
||||
The same sibling-uniform pattern appears across runs, which suggests these rows
|
||||
should remain review-gated until we either:
|
||||
|
||||
- find more diverse artifact families
|
||||
- or factor the decision more tightly
|
||||
|
||||
### `required_tool_selection`
|
||||
|
||||
Same current status as acceptance commands.
|
||||
|
||||
The field exists and is easy to parse, but the observed slice suggests strong
|
||||
template carryover rather than high-confidence local decision variance.
|
||||
|
||||
## Immediate Conclusion
|
||||
|
||||
The current first supervised RSA gate should be:
|
||||
|
||||
- `prereq_op_selector`
|
||||
|
||||
The following remain promising, but should stay review-gated or filtered more
|
||||
aggressively before broader extraction:
|
||||
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `required_tool_selection`
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Tighten parser-side filtering for schema-drift prereq rows and optionally
|
||||
split accepted vs rejected outputs.
|
||||
|
||||
2. Search for artifact families with higher local variance for:
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `required_tool_selection`
|
||||
|
||||
3. Promote `prereq_op_selector` to the first extraction-ready gate dataset.
|
||||
Reference in New Issue
Block a user