whetstone_RSA/docs/case_studies/whetstone_dsl_training_data_strategy.md

# WhetstoneDSL Training Data Strategy

## Purpose

This document records the data-handling rule for turning existing
`whetstone_DSL` outputs into RSA training data.

The immediate concern is that much of the existing material may be:

- whole JSON responses
- whole taskitem packets
- multi-decision LLM outputs

rather than clean one-decision-per-example supervision.

That means the data cannot be used naively.

## Core Rule

Do not train per-gate specialists directly on full taskitem or full-response
outputs without reconstructing:

- the exact local context for the decision
- the specific decision target being learned
- whether the label was independently chosen or entangled with other choices

## Why This Matters

The old pipeline was optimized for:

- one LLM or SLM making several bounded choices together
- minimizing token use by sharing context across those choices

The new RSA direction is different:

- one specialist per bounded decision
- narrower context per decision
- explicit gate contracts

So existing outputs may reflect:

- coupled decisions
- prompt-order effects
- hidden shared rationale
- labels that were only valid because other fields in the same response were
  chosen jointly

If we ignore that, the training data will be semantically wrong even if it looks
large.

## Safe Extraction Rule

For each candidate gate, build examples as:

- `decision_context`
- `decision_label`
- `decision_provenance`

and not as:

- full response blob -> label

### `decision_context`

Must include only the information that would have been available at the moment
that gate should fire.

Examples:

- `worker_type`
  - task title
  - task intent
  - maybe requirement summary
  - not downstream fields derived after routing

- `verification_type`
  - target files
  - task title
  - task intent
  - not later acceptance-command decisions unless those are part of the gate by
    contract

- `prereq_op_selector`
  - task description
  - uncertainty and dependency signals if the intended gate contract includes them
  - not later execution outcomes

### `decision_label`

Must match the new gate contract exactly.

If the historic data uses a different shape than the new gate, do not force it.

Instead:

- transform it carefully
- or discard it
- or redefine the gate

### `decision_provenance`

Each extracted example should track:

- source artifact
- source run id if available
- whether label was explicit or inferred
- whether the example came from:
  - human-authored heuristic baseline
  - LLM output
  - SLM output
  - deterministic reconstruction

This is necessary because not all labels have equal trust.

## Gate-Specific Warnings

### `worker_type`

This is likely recoverable from older taskitems, but only if the worker role was
explicitly present and chosen before downstream execution shaping.

Risk:

- label polluted by later execution-contract logic

### `verification_type`

This is relatively safe because it can often be reconstructed from target files
and acceptance intent.

Risk:

- historic labels may include joint decisions with acceptance-command shaping

### `prereq_op_selector`

This is the riskiest early gate for naive extraction.

Reason:

- older systems may have emitted a flat list of ops as one bundle
- the new semantic contract treats this as structured or factorized

Rule:

- do not flatten structured op bundles into arbitrary multiclass labels

### `cpp_raising_profile`

This is likely only useful if old runs actually recorded explicit profile choice
with enough IR context.

If not, this gate may need to start with synthetic or policy-authored data rather
than mined response logs.

## Easy/Challenging Project Corpora

The `100 easy` and `25 challenging` project sets are valuable, but they are not
automatically specialist training sets.

They are best treated as:

- context reservoirs
- replay/evaluation corpora
- example generators for extracting many per-gate decisions

The right pattern is:

1. replay or parse the project artifact
2. isolate one decision surface at a time
3. extract only the local context relevant to that gate
4. record the corresponding bounded label

This should produce a gate-native dataset rather than a project-native dataset.

## Taskitem as Self-Contained Work Unit — Design Constraint

**Added 2026-04-12.**

A taskitem is a concurrent work unit. By design it must include all information a
specialist needs to make its gate decision. The specialist does not reach back to
the original spec or the intake.

**Consequence for RSA gate design:**

The taskitem schema must carry every field that is discriminative for the gate.
`generate_taskitems` populates those fields from the intake contract. The RSA gate
reads those fields from the taskitem — no other context is available at decision time.

**Consequence for training data:**

The current taskitem schema does not carry discriminative fields for most RSA gates.
The run corpus has weak text (17 unique templates) because the schema was designed for
a different purpose — it records outputs, not inputs.

RSA gates are designed to operate on whatever input actually exists — including
mixed structured/unstructured content and incomplete directions. Waiting for ideal
input is not the RSA model.

The current corpus is the training corpus. Sprint plan step descriptions are the best
available text signal now. What the gate learns from imperfect inputs, and when it
abstains on insufficient inputs, is the gate's actual contract.

Richer taskitem fields (if added) give the gate more signal and may shift it toward a
deterministic backend — but that is operator choice, not a prerequisite for deployment.

## Compatibility Constraint

We should not assume the new specialists must fit perfectly into the exact old
taskitem bundle shape.

There are two valid outcomes:

### Option A: Fit specialists into current taskitems

Use when:

- the old taskitem structure already matches clean gate boundaries
- the decision context can be isolated without distortion

### Option B: Rebuild taskitems around explicit gates

Use when:

- the current taskitems bundle too many coupled decisions
- the old layout hides which context belongs to which choice
- structured gates are being forced into flat labels

This is not failure.
It may be the correct result of moving from one-model-many-decisions to
many-model-one-decision architecture.

## Recommended Next Step

Before mining historic runs, define extraction specs for the first four gates:

- `verification_type`
- `worker_type`
- `prereq_op_selector`
- `cpp_raising_profile`

Each extraction spec should state:

- allowed input fields
- forbidden leaked fields
- label shape
- trust level of recovered labels

Only after that should we parse old JSON outputs or taskitem runs.