whetstone_RSA/sprints/SPRINT-004-pipeline-decision-audit.md

# Sprint 004: Pipeline Decision Audit

## Status

Authored — not yet executed. Execute in the next session.

## Intent

Manually walk the full whetstone_DSL pipeline from raw project description to
generated code artifacts. At every stage, identify every decision being made,
record what inputs were available at the time of that decision, and evaluate the
optimal method for that decision.

This sprint produces a decision map, not code and not training data.

## Why This Sprint Exists

The current specialist fleet and RSA gate inventory were built top-down: we
identified gate candidates from the output schema and worked backward. This sprint
inverts that: start from the input and work forward, letting the pipeline tell us
what decisions actually exist.

The prior approach missed decisions that were collapsed into single LLM calls. A
single call to `generate_taskitems` currently makes at minimum:

- task decomposition (how many tasks, what granularity)
- task title assignment
- milestone grouping
- worker type selection
- verification type selection
- prerequisite op selection
- execution contract shaping
  - target file selection
  - acceptance command selection
  - required tool selection
- confidence scoring
- ambiguity annotation
- reason generation

These are not one decision. They are 10+ decisions bundled for token efficiency.
The audit's job is to unbundle them and evaluate each one independently.

## Scope

### In scope

- Walk every pipeline stage from spec text to final code artifact
- For each stage: list every decision made in that stage
- For each decision: record inputs available, current implementation, proposed method
- Identify decisions that are genuinely coupled vs accidentally bundled
- Note where current inputs are insufficient and what the gate should do (abstain,
  escalate, or decide from partial information)

### Not in scope

- Running any pipeline code
- Writing training data generators
- Training any specialist
- Modifying any whetstone_DSL source
- Designing gate schemas or contracts in detail

## Pipeline Stages To Audit

### Stage 0: Project description ingestion

Raw input: a project description in some format (markdown, structured JSON, free text).

Questions to answer:
- What formats are currently accepted?
- What decisions are implicit in accepting/rejecting an input format?
- Is format normalization a decision or a transformation?
- What is the minimum viable input for the pipeline to proceed?
- What does the pipeline do when input is partial or ambiguous?

### Stage 1: `architect_intake`

Current implementation: single LLM call over the raw spec.

Output fields to audit (each may represent a separate decision):
- `normalizedRequirements` — which lines become requirements, which are discarded
- `parsedSpec.goals` — goal extraction and normalization
- `parsedSpec.constraints` — constraint extraction
- `parsedSpec.acceptanceCriteria` — acceptance criterion extraction
- `parsedSpec.dependencies` — dependency extraction
- `conflictSignals` — conflict detection
- `ambiguitySignals` — ambiguity detection

For each output field:
- What inputs does this decision use?
- Is this deterministic given the input, or genuinely uncertain?
- What method is appropriate (rule, RSA, SLM, LLM)?
- What does failure or partial output mean for downstream stages?

### Stage 2: `generate_taskitems`

Current implementation: single LLM call over normalized requirements.

Decisions to unbundle and audit individually:

**Decomposition decisions**
- How many tasks to create
- What granularity to use
- How to group related work into milestones
- Whether a requirement maps to one task or multiple

**Per-task assignment decisions**
- `title` — what to call the task
- `milestoneId` — which milestone this task belongs to
- `workerType` — who does this work (implementer, reviewer, architect, qa)
- `verificationType` — how to verify completion (unit, integration, schema, smoke, docs)
- `prerequisiteOps` — what gates must clear before this task runs
- `confidence` — how confident is the system in this task
- `ambiguityCount` — how many ambiguous signals were present
- `reasons` — why was this task generated

**Execution contract decisions** (currently nested inside each task)
- `targetFiles` — which files this task touches
- `acceptanceCommands` — how to verify the task is done
- `requiredTools` — which tools are needed
- `crossProjectTargets` — does this task touch other projects
- `resourceLocks` — what resources does this task need exclusive access to

For each decision: same four questions as Stage 1.

### Stage 3: `queue_ready`

Current implementation: deterministic validation.

Questions to answer:
- Is this purely deterministic, or are there judgment calls embedded in it?
- What validation rules exist and are they all rule-complete?
- Are there cases where queue_ready makes a decision that should be RSA?

### Stage 4: Per-task execution (`run_pipeline`)

Current implementation: one LLM call per task step, given the execution contract.

This stage likely contains its own collapsed decision bundle. Questions to answer:
- What does the executor decide before writing any code?
- What decisions are made during code generation vs before it starts?
- What decisions are made per file vs per task?
- How does the executor handle execution contract fields that are missing or ambiguous?

### Stage 5: Verification

Current implementation: runs acceptance commands, reports pass/fail.

Questions to answer:
- Is the verification decision purely mechanical (pass/fail), or is there judgment?
- What happens when acceptance commands are absent or malformed?
- Is there a triage decision when verification fails?

## For Each Decision: What To Record

Each discovered decision should be recorded as a decision record with these fields:

```
decision_id:       short unique identifier (e.g., "intake.conflict_detection")
stage:             which pipeline stage this fires in
inputs_available:  what information exists at decision time
current_impl:      how it is currently implemented (LLM / rule / heuristic / none)
output_type:       boolean / enum / structured / free-text / numeric
output_values:     the bounded set of outputs, if applicable
coupling:          which other decisions does this one depend on or influence
proposed_method:   deterministic / RSA / SLM / LLM / human
rationale:         why that method fits this decision
rsa_notes:         if RSA: what the gate input features are, what abstain looks like
open_questions:    anything unresolved
```

## Code Surfaces To Examine

These are the primary source files for understanding what decisions currently exist:

| Surface | Location |
|---------|----------|
| `architect_intake` tool | `editor/src/mcp/RegisterArchitectIntakeTools.h` |
| `generate_taskitems` tool | `editor/src/mcp/RegisterTaskitemGeneratorTools.h` |
| Taskitem generator logic | `editor/src/TaskitemGeneratorV2.h` |
| Confidence and ambiguity | `editor/src/TaskitemConfidenceAmbiguity.h` |
| Queue validation | `editor/src/mcp/RegisterQueueTools.h` |
| Pipeline execution | `editor/src/mcp/RegisterPipelineTools.h` |
| Routing | `editor/src/RoutingEngine.h` |
| Execution contract | `editor/src/ExecutionContractBuilder.h` (if exists) |

For each file: read the decision logic, not just the schema. The schema tells you
what fields exist. The logic tells you what actually determines their values.

## Output of This Sprint

One document: `docs/case_studies/whetstone_dsl_pipeline_decision_map.md`

Structure:
- One section per pipeline stage
- Within each section: one subsection per decision record
- Summary table at the top: all decisions, their current impl, and proposed method

This document becomes the authoritative input for:
- RSA gate inventory (replaces the current incomplete inventory)
- Training data collection strategy (once we know what inputs each gate sees)
- Sprint 005: first gate implementation from the decision map

## Acceptance Criteria

- Every stage from project description to generated artifact is covered
- Every output field of every pipeline tool is traced to at least one decision record
- No decision record has `proposed_method: LLM` without a written rationale for why
  a smaller or more bounded approach is insufficient
- The summary table exists and is navigable
- At least one decision currently implemented as a single LLM call is shown to contain
  three or more separable decisions

## Execution Notes for Next Session

Start with Stage 1 (`architect_intake`) — it is the entry point and its outputs
constrain everything downstream. Read the source, not just the schema.

Do not assume a field is a single decision because it has a single key in the output
JSON. JSON keys are output slots. Decisions are the reasoning steps that fill them.

When a decision is genuinely coupled to another (output of A determines the valid
space of B), record the coupling explicitly rather than merging the decisions. Coupled
decisions that are *always* coupled may warrant a structured output gate. Decisions
that are *sometimes* coupled may warrant independent gates with a dependency edge.