whetstone_RSA/docs/case_studies/whetstone_dsl_run_artifact_audit.md

WhetstoneDSL Run Artifact Audit

Purpose

This note audits surviving whetstone_DSL run artifacts for RSA gate extraction. The goal is not to treat every surviving corpus as direct training data. The goal is to identify which artifacts preserve explicit decision outputs, which preserve only surrounding context, and which are better treated as evaluation traces.

Audit Targets

Primary paths inspected:

• CLionProjects/whetstone_DSL/logs/taskitem_runs
• CLionProjects/whetstone_DSL/datasets/example_run_specs
• CLionProjects/whetstone_DSL/datasets/project_benchmarks

Representative files inspected:

• logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/01_intake.json
• logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems.json
• logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems_raw.ndjson.json
• logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/04_validate_taskitem.json
• logs/taskitem_runs/project_benchmark_matrix_20260225_192610/cp_001_python/ab_stdout.log
• logs/taskitem_runs/ast_once_vs_direct_20260225_193037/cp_001/base_payload.json
• datasets/example_run_specs/index.jsonl
• datasets/example_run_specs/runspec_py_cli_todo_wrong_tool_selection_01.md
• datasets/project_benchmarks/common_projects_100.jsonl

Source Classes

1. Structured taskitem runs

Best current extraction source:

• logs/taskitem_runs/*/01_intake.json
• logs/taskitem_runs/*/02_generate_taskitems.json
• logs/taskitem_runs/*/04_validate_taskitem.json

What these preserve:

• pre-decision intake context
• generated taskitems
• explicit task confidence and escalation signals
• explicit prerequisite operations
• explicit execution-contract fields
• validation and specificity feedback

Observed explicit fields in 02_generate_taskitems.json:

• confidence
• escalate
• prerequisiteOps
• queueReady
• reasons
• resourceLocks
• executionContract.acceptanceCommands
• executionContract.requiredTools
• executionContract.targetFiles
• executionContract.stepIds
• executionContract.riskLevel
• executionContract.executionSpecificityScore

Observed intake fields in 01_intake.json:

• normalized requirements
• parsed spec structure
• conflict and ambiguity signals

Observed validation fields in 04_validate_taskitem.json:

• gap and specificity diagnostics
• recommended tools
• execution-quality feedback

Assessment:

• highest value source for verification_type
• highest value source for prereq_op_selector
• highest value source for execution_contract_shape
• highest value source for child skeleton decisions such as:
  • target_file_selection
  • acceptance_command_selection
  • required_tool_selection

Constraint:

• these files preserve outputs, but not necessarily independence of each field
• some fields may be jointly emitted in one response rather than chosen as separate model calls
• extraction should therefore attach provenance and a decomposition method

2. Raw NDJSON taskitem responses

Representative path:

• logs/taskitem_runs/*/02_generate_taskitems_raw.ndjson.json

What these preserve:

• near-raw tool response wrapper
• text form of the generated taskitem JSON
• tool/runtime metadata

Assessment:

• useful provenance source
• useful for checking whether 02_generate_taskitems.json is a direct capture or a normalized downstream rewrite
• not a separate supervision source by itself

Constraint:

• this still reflects bundled output from a single taskitem-generation call
• it does not by itself prove per-field independence

3. Project benchmark matrix runs

Representative path:

• logs/taskitem_runs/project_benchmark_matrix_20260225_192610/*/ab_stdout.log

Observed shape:

• spec
• language
• strict mode
• path A / path B readiness
• failure reasons
• token accounting

Assessment:

• good evaluation corpus
• good source for comparing system modes and efficiency
• not a direct source for task-shaping gates such as prereq_op_selector or target_file_selection

Constraint:

• these logs summarize end-to-end path outcomes
• they do not preserve the intermediate bounded decisions we want to supervise

4. AST-once vs direct comparison runs

Representative path:

• logs/taskitem_runs/ast_once_vs_direct_20260225_193037/*/base_payload.json

Observed shape:

• AST node payload
• generated code
• compile/test/placeholder gates
• quality flags
• used symbols

Assessment:

• useful evaluation corpus for AST-first vs direct generation comparisons
• useful for downstream codegen quality measurement
• not a primary direct source for taskitem decision supervision

Constraint:

• preserves codegen result and gate outcomes
• does not preserve the earlier planning and task-shaping decisions

5. Example run specs

Representative paths:

• datasets/example_run_specs/index.jsonl
• datasets/example_run_specs/*.md

Observed shape:

• scenario family and variant metadata
• objective
• target behavior
• anti-pattern tags
• acceptance criteria
• data capture tags

Assessment:

• strong context/spec source
• useful for reconstructing scenario intent and mistake class
• not a direct source of labels for bounded decisions

6. Project benchmark inventories

Representative path:

• datasets/project_benchmarks/*.jsonl

Observed shape:

• project id
• category
• language
• natural-language spec

Assessment:

• good context reservoir
• good replay seed corpus
• not a direct supervision source

Extraction Suitability By Decision

Direct or near-direct candidates

• prereq_op_selector

  • candidate source: 02_generate_taskitems.json
  • source field: prerequisiteOps
  • note: may still need decomposition into slot-based labels rather than raw list copy

• execution_contract_shape

  • candidate source: 02_generate_taskitems.json
  • source field: executionContract
  • note: should be split into child decisions before training

• target_file_selection

  • candidate source: executionContract.targetFiles
  • note: must preserve local requirement context and touched-surface context

• acceptance_command_selection

  • candidate source: executionContract.acceptanceCommands
  • note: likely partly heuristic in some runs; requires provenance tagging

• required_tool_selection

  • candidate source: executionContract.requiredTools
  • note: likely mixed model-plus-template behavior

Reconstructible but not yet trusted as direct labels

• verification_type

  • may be reconstructible from execution-contract structure, validation shape, and downstream required tools
  • not yet observed as a single explicit standalone field in audited artifacts

• worker_type

  • current code surface exists in routing/orchestration layers:
    • editor/src/RoutingRules.h
    • editor/src/WorkflowOrchestrator.h
    • editor/src/TaskitemPoolPersistence.h
    • editor/src/headless_rpc/DispatchPart6.h
  • not yet observed as a clean explicit field in the audited taskitem-run artifacts used for task generation
  • may exist elsewhere in routing or orchestration artifacts, or may need reconstruction from execution contract and downstream path

Poor direct-fit sources for current Sprint 003 gates

• cpp_raising_profile
  • current code surface exists in C++-raising layers:
    • editor/src/mcp/RegisterCppRaisingTools.h
    • editor/src/cpp_ir/CppErrorModelMapping.h
    • editor/src/cpp_ir/CppBuildArtifactGenerator.h
    • editor/src/cpp_ir/CppAsyncMappingStrategy.h
    • editor/src/cpp_ir/CppTemplateRaisingPolicy.h
  • not surfaced in the audited generic taskitem corpora reviewed so far
  • likely needs a different source family tied to C++ raising flows rather than generic taskitem generation

Search Outcome Notes

Additional source search confirms:

• workerType is a real first-class concept in the current DSL routing stack
• safe-first|perf-first|interop-first is a real first-class concept in the current C++ raising stack
• the audited taskitem-generation artifacts do not yet show those values as clean serialized labels in the same way that prerequisiteOps and executionContract.* are preserved

Practical implication:

• prereq_op_selector and execution-contract child decisions can move toward extraction-spec work now
• worker_type and cpp_raising_profile need a second artifact search pass over routing and raising-specific run families before we treat them as extractable labels

Provenance Classes

Every extracted training row should carry one provenance tag:

• direct_explicit

  • field exists directly in the artifact in bounded form

• direct_bundled

  • field exists directly, but as one member of a larger jointly emitted object

• reconstructed

  • label inferred from multiple artifact fields or execution traces

• untrusted

  • label candidate exists, but source independence or semantic stability is too weak to use without further review

Current default classification:

• prerequisiteOps: direct_bundled
• executionContract.targetFiles: direct_bundled
• executionContract.acceptanceCommands: direct_bundled
• executionContract.requiredTools: direct_bundled
• verification_type: reconstructed
• worker_type: reconstructed
• cpp_raising_profile: untrusted

Immediate Recommendations

1. Use logs/taskitem_runs/*/{01_intake.json,02_generate_taskitems.json,04_validate_taskitem.json} as the first extraction corpus.

2. Treat datasets/example_run_specs and datasets/project_benchmarks as context reservoirs, not label sources.

3. Keep project benchmark matrix and AST/direct corpora for evaluation and counterfactual replay, not first-pass supervision.

4. Build extraction specs per gate rather than per full taskitem JSON.

5. Require each extraction spec to declare:

   • source files
   • context slice
   • label shape
   • provenance class
   • rejection conditions

Next Audit Steps

1. Search surviving routing/orchestration artifacts for explicit worker_type labels.

2. Search C++ raising artifacts for explicit cpp_raising_profile choices.

3. Draft per-gate extraction specs for:

   • prereq_op_selector
   • target_file_selection
   • acceptance_command_selection
   • required_tool_selection

4. Add rejection rules for low-trust bundled outputs before building any training set.