whetstone_RSA/docs/case_studies/whetstone_dsl_extraction_yield_snapshot.md

WhetstoneDSL Extraction Yield Snapshot

Slice

Parser:

• tools/extract_gate_rows.py

Contracts:

• semantic/rsa_gate_extraction_contracts_v0.json

Input slice:

• first 50 run directories under CLionProjects/whetstone_DSL/logs/taskitem_runs

Output:

• semantic/extracted_gate_rows_slice.ndjson

Summary

Observed counts from the 50-run slice:

• runs processed: 50
• tasks processed: 106
• rows emitted: 424
• rows with rejection flags: 330

Per-gate emitted rows:

• prereq_op_selector: 106
• target_file_selection: 106
• acceptance_command_selection: 106
• required_tool_selection: 106

Per-gate rejected rows:

• prereq_op_selector: 12
• target_file_selection: 106
• acceptance_command_selection: 106
• required_tool_selection: 106

Dominant Rejection Flags

• sibling_uniform_target_files: 106
• sibling_uniform_acceptance_commands: 106
• sibling_uniform_required_tools: 106
• invalid_prerequisite_values:whetstone_generate_taskitems,whetstone_queue_ready,whetstone_validate_taskitem: 12

Interpretation

prereq_op_selector

This is the strongest current extraction surface.

It survives most of the slice cleanly, and the rejection cases are informative rather than noisy. The rejected rows come from schema-drift runs where prerequisiteOps was contaminated with tool ids such as:

• whetstone_generate_taskitems
• whetstone_queue_ready
• whetstone_validate_taskitem

That means the parser is catching real label pollution rather than inventing false negatives.

target_file_selection

Present explicitly, but not yet trustworthy as broad supervision in this slice.

The dominant issue is sibling-uniform output: all tasks in a run often carry the same target-file list, which makes naive extraction look more like template reuse than local bounded choice.

acceptance_command_selection

Present explicitly, but also highly template-like in this slice.

The same sibling-uniform pattern appears across runs, which suggests these rows should remain review-gated until we either:

• find more diverse artifact families
• or factor the decision more tightly

required_tool_selection

Same current status as acceptance commands.

The field exists and is easy to parse, but the observed slice suggests strong template carryover rather than high-confidence local decision variance.

Immediate Conclusion

The current first supervised RSA gate should be:

• prereq_op_selector

The following remain promising, but should stay review-gated or filtered more aggressively before broader extraction:

• target_file_selection
• acceptance_command_selection
• required_tool_selection

Next Steps

1. Tighten parser-side filtering for schema-drift prereq rows and optionally split accepted vs rejected outputs.

2. Search for artifact families with higher local variance for:

   • target_file_selection
   • acceptance_command_selection
   • required_tool_selection

3. Promote prereq_op_selector to the first extraction-ready gate dataset.