Add RSA extraction bootstrap and working notes

This commit is contained in:
Bill Holcombe
2026-04-12 15:48:43 -06:00
parent 9b112b9675
commit 0253a92f28
11 changed files with 2572 additions and 1 deletions

View File

@@ -0,0 +1,117 @@
# WhetstoneDSL Extraction Yield Snapshot
## Slice
Parser:
- `tools/extract_gate_rows.py`
Contracts:
- `semantic/rsa_gate_extraction_contracts_v0.json`
Input slice:
- first 50 run directories under
`CLionProjects/whetstone_DSL/logs/taskitem_runs`
Output:
- `semantic/extracted_gate_rows_slice.ndjson`
## Summary
Observed counts from the 50-run slice:
- runs processed: `50`
- tasks processed: `106`
- rows emitted: `424`
- rows with rejection flags: `330`
Per-gate emitted rows:
- `prereq_op_selector`: `106`
- `target_file_selection`: `106`
- `acceptance_command_selection`: `106`
- `required_tool_selection`: `106`
Per-gate rejected rows:
- `prereq_op_selector`: `12`
- `target_file_selection`: `106`
- `acceptance_command_selection`: `106`
- `required_tool_selection`: `106`
## Dominant Rejection Flags
- `sibling_uniform_target_files`: `106`
- `sibling_uniform_acceptance_commands`: `106`
- `sibling_uniform_required_tools`: `106`
- `invalid_prerequisite_values:whetstone_generate_taskitems,whetstone_queue_ready,whetstone_validate_taskitem`: `12`
## Interpretation
### `prereq_op_selector`
This is the strongest current extraction surface.
It survives most of the slice cleanly, and the rejection cases are informative
rather than noisy. The rejected rows come from schema-drift runs where
`prerequisiteOps` was contaminated with tool ids such as:
- `whetstone_generate_taskitems`
- `whetstone_queue_ready`
- `whetstone_validate_taskitem`
That means the parser is catching real label pollution rather than inventing
false negatives.
### `target_file_selection`
Present explicitly, but not yet trustworthy as broad supervision in this slice.
The dominant issue is sibling-uniform output: all tasks in a run often carry the
same target-file list, which makes naive extraction look more like template
reuse than local bounded choice.
### `acceptance_command_selection`
Present explicitly, but also highly template-like in this slice.
The same sibling-uniform pattern appears across runs, which suggests these rows
should remain review-gated until we either:
- find more diverse artifact families
- or factor the decision more tightly
### `required_tool_selection`
Same current status as acceptance commands.
The field exists and is easy to parse, but the observed slice suggests strong
template carryover rather than high-confidence local decision variance.
## Immediate Conclusion
The current first supervised RSA gate should be:
- `prereq_op_selector`
The following remain promising, but should stay review-gated or filtered more
aggressively before broader extraction:
- `target_file_selection`
- `acceptance_command_selection`
- `required_tool_selection`
## Next Steps
1. Tighten parser-side filtering for schema-drift prereq rows and optionally
split accepted vs rejected outputs.
2. Search for artifact families with higher local variance for:
- `target_file_selection`
- `acceptance_command_selection`
- `required_tool_selection`
3. Promote `prereq_op_selector` to the first extraction-ready gate dataset.