whetstone_RSA/docs/case_studies/whetstone_dsl_run_artifact_audit.md

# WhetstoneDSL Run Artifact Audit

## Purpose

This note audits surviving `whetstone_DSL` run artifacts for RSA gate extraction.
The goal is not to treat every surviving corpus as direct training data. The goal
is to identify which artifacts preserve explicit decision outputs, which preserve
only surrounding context, and which are better treated as evaluation traces.

## Audit Targets

Primary paths inspected:

- `CLionProjects/whetstone_DSL/logs/taskitem_runs`
- `CLionProjects/whetstone_DSL/datasets/example_run_specs`
- `CLionProjects/whetstone_DSL/datasets/project_benchmarks`

Representative files inspected:

- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/01_intake.json`
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems.json`
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems_raw.ndjson.json`
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/04_validate_taskitem.json`
- `logs/taskitem_runs/project_benchmark_matrix_20260225_192610/cp_001_python/ab_stdout.log`
- `logs/taskitem_runs/ast_once_vs_direct_20260225_193037/cp_001/base_payload.json`
- `datasets/example_run_specs/index.jsonl`
- `datasets/example_run_specs/runspec_py_cli_todo_wrong_tool_selection_01.md`
- `datasets/project_benchmarks/common_projects_100.jsonl`

## Source Classes

### 1. Structured taskitem runs

Best current extraction source:

- `logs/taskitem_runs/*/01_intake.json`
- `logs/taskitem_runs/*/02_generate_taskitems.json`
- `logs/taskitem_runs/*/04_validate_taskitem.json`

What these preserve:

- pre-decision intake context
- generated taskitems
- explicit task confidence and escalation signals
- explicit prerequisite operations
- explicit execution-contract fields
- validation and specificity feedback

Observed explicit fields in `02_generate_taskitems.json`:

- `confidence`
- `escalate`
- `prerequisiteOps`
- `queueReady`
- `reasons`
- `resourceLocks`
- `executionContract.acceptanceCommands`
- `executionContract.requiredTools`
- `executionContract.targetFiles`
- `executionContract.stepIds`
- `executionContract.riskLevel`
- `executionContract.executionSpecificityScore`

Observed intake fields in `01_intake.json`:

- normalized requirements
- parsed spec structure
- conflict and ambiguity signals

Observed validation fields in `04_validate_taskitem.json`:

- gap and specificity diagnostics
- recommended tools
- execution-quality feedback

Assessment:

- highest value source for `verification_type`
- highest value source for `prereq_op_selector`
- highest value source for `execution_contract_shape`
- highest value source for child skeleton decisions such as:
  - `target_file_selection`
  - `acceptance_command_selection`
  - `required_tool_selection`

Constraint:

- these files preserve outputs, but not necessarily independence of each field
- some fields may be jointly emitted in one response rather than chosen as
  separate model calls
- extraction should therefore attach provenance and a decomposition method

### 2. Raw NDJSON taskitem responses

Representative path:

- `logs/taskitem_runs/*/02_generate_taskitems_raw.ndjson.json`

What these preserve:

- near-raw tool response wrapper
- text form of the generated taskitem JSON
- tool/runtime metadata

Assessment:

- useful provenance source
- useful for checking whether `02_generate_taskitems.json` is a direct capture
  or a normalized downstream rewrite
- not a separate supervision source by itself

Constraint:

- this still reflects bundled output from a single taskitem-generation call
- it does not by itself prove per-field independence

### 3. Project benchmark matrix runs

Representative path:

- `logs/taskitem_runs/project_benchmark_matrix_20260225_192610/*/ab_stdout.log`

Observed shape:

- spec
- language
- strict mode
- path A / path B readiness
- failure reasons
- token accounting

Assessment:

- good evaluation corpus
- good source for comparing system modes and efficiency
- not a direct source for task-shaping gates such as `prereq_op_selector` or
  `target_file_selection`

Constraint:

- these logs summarize end-to-end path outcomes
- they do not preserve the intermediate bounded decisions we want to supervise

### 4. AST-once vs direct comparison runs

Representative path:

- `logs/taskitem_runs/ast_once_vs_direct_20260225_193037/*/base_payload.json`

Observed shape:

- AST node payload
- generated code
- compile/test/placeholder gates
- quality flags
- used symbols

Assessment:

- useful evaluation corpus for AST-first vs direct generation comparisons
- useful for downstream codegen quality measurement
- not a primary direct source for taskitem decision supervision

Constraint:

- preserves codegen result and gate outcomes
- does not preserve the earlier planning and task-shaping decisions

### 5. Example run specs

Representative paths:

- `datasets/example_run_specs/index.jsonl`
- `datasets/example_run_specs/*.md`

Observed shape:

- scenario family and variant metadata
- objective
- target behavior
- anti-pattern tags
- acceptance criteria
- data capture tags

Assessment:

- strong context/spec source
- useful for reconstructing scenario intent and mistake class
- not a direct source of labels for bounded decisions

### 6. Project benchmark inventories

Representative path:

- `datasets/project_benchmarks/*.jsonl`

Observed shape:

- project id
- category
- language
- natural-language spec

Assessment:

- good context reservoir
- good replay seed corpus
- not a direct supervision source

## Extraction Suitability By Decision

### Direct or near-direct candidates

- `prereq_op_selector`
  - candidate source: `02_generate_taskitems.json`
  - source field: `prerequisiteOps`
  - note: may still need decomposition into slot-based labels rather than raw
    list copy

- `execution_contract_shape`
  - candidate source: `02_generate_taskitems.json`
  - source field: `executionContract`
  - note: should be split into child decisions before training

- `target_file_selection`
  - candidate source: `executionContract.targetFiles`
  - note: must preserve local requirement context and touched-surface context

- `acceptance_command_selection`
  - candidate source: `executionContract.acceptanceCommands`
  - note: likely partly heuristic in some runs; requires provenance tagging

- `required_tool_selection`
  - candidate source: `executionContract.requiredTools`
  - note: likely mixed model-plus-template behavior

### Reconstructible but not yet trusted as direct labels

- `verification_type`
  - may be reconstructible from execution-contract structure, validation shape,
    and downstream required tools
  - not yet observed as a single explicit standalone field in audited artifacts

- `worker_type`
  - current code surface exists in routing/orchestration layers:
    - `editor/src/RoutingRules.h`
    - `editor/src/WorkflowOrchestrator.h`
    - `editor/src/TaskitemPoolPersistence.h`
    - `editor/src/headless_rpc/DispatchPart6.h`
  - not yet observed as a clean explicit field in the audited taskitem-run
    artifacts used for task generation
  - may exist elsewhere in routing or orchestration artifacts, or may need
    reconstruction from execution contract and downstream path

### Poor direct-fit sources for current Sprint 003 gates

- `cpp_raising_profile`
  - current code surface exists in C++-raising layers:
    - `editor/src/mcp/RegisterCppRaisingTools.h`
    - `editor/src/cpp_ir/CppErrorModelMapping.h`
    - `editor/src/cpp_ir/CppBuildArtifactGenerator.h`
    - `editor/src/cpp_ir/CppAsyncMappingStrategy.h`
    - `editor/src/cpp_ir/CppTemplateRaisingPolicy.h`
  - not surfaced in the audited generic taskitem corpora reviewed so far
  - likely needs a different source family tied to C++ raising flows rather than
    generic taskitem generation

## Search Outcome Notes

Additional source search confirms:

- `workerType` is a real first-class concept in the current DSL routing stack
- `safe-first|perf-first|interop-first` is a real first-class concept in the
  current C++ raising stack
- the audited taskitem-generation artifacts do not yet show those values as
  clean serialized labels in the same way that `prerequisiteOps` and
  `executionContract.*` are preserved

Practical implication:

- `prereq_op_selector` and execution-contract child decisions can move toward
  extraction-spec work now
- `worker_type` and `cpp_raising_profile` need a second artifact search pass
  over routing and raising-specific run families before we treat them as
  extractable labels

## Provenance Classes

Every extracted training row should carry one provenance tag:

- `direct_explicit`
  - field exists directly in the artifact in bounded form

- `direct_bundled`
  - field exists directly, but as one member of a larger jointly emitted object

- `reconstructed`
  - label inferred from multiple artifact fields or execution traces

- `untrusted`
  - label candidate exists, but source independence or semantic stability is too
    weak to use without further review

Current default classification:

- `prerequisiteOps`: `direct_bundled`
- `executionContract.targetFiles`: `direct_bundled`
- `executionContract.acceptanceCommands`: `direct_bundled`
- `executionContract.requiredTools`: `direct_bundled`
- `verification_type`: `reconstructed`
- `worker_type`: `reconstructed`
- `cpp_raising_profile`: `untrusted`

## Immediate Recommendations

1. Use `logs/taskitem_runs/*/{01_intake.json,02_generate_taskitems.json,04_validate_taskitem.json}`
   as the first extraction corpus.

2. Treat `datasets/example_run_specs` and `datasets/project_benchmarks` as
   context reservoirs, not label sources.

3. Keep project benchmark matrix and AST/direct corpora for evaluation and
   counterfactual replay, not first-pass supervision.

4. Build extraction specs per gate rather than per full taskitem JSON.

5. Require each extraction spec to declare:
   - source files
   - context slice
   - label shape
   - provenance class
   - rejection conditions

## Next Audit Steps

1. Search surviving routing/orchestration artifacts for explicit `worker_type`
   labels.

2. Search C++ raising artifacts for explicit `cpp_raising_profile` choices.

3. Draft per-gate extraction specs for:
   - `prereq_op_selector`
   - `target_file_selection`
   - `acceptance_command_selection`
   - `required_tool_selection`

4. Add rejection rules for low-trust bundled outputs before building any
   training set.