Sprint 003 findings + Sprint 004 pipeline decision audit plan
- split_prereq_op_dataset.py: extract accepted/rejected/schema-drift rows per gate - gen_prereq_op_rsa_data.py: convert accepted rows to Fabricate TSV format - 1000-run extracted dataset: 1297 accepted prereq_op rows, 24 schema-drift - Key finding: needs_validate_intake always True; effective gate is binary (needs_resolve_dependencies 77%, needs_architect_review 20%) - Key finding: run corpus text is template-driven (17 unique inputs) — not suitable for text-based training; RSA exists before ideal input is available - ROADMAP: gate lifecycle framing — goal is choice of backend, not forced determinism; RSA operates on imperfect/mixed/incomplete input by design - training_data_strategy.md: taskitem-as-self-contained-work-unit constraint - SPRINT-003: updated with 1000-run results, text audit findings, design constraints - SPRINT-004: pipeline decision audit — manual walkthrough of every decision from project description to generated artifact; execute next session Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill Holcombe
@@ -2,7 +2,7 @@
|
||||
|
||||
## Status
|
||||
|
||||
Active
|
||||
Active — pending intake contract design (see Architectural Constraint below)
|
||||
|
||||
## Intent
|
||||
|
||||
@@ -19,6 +19,47 @@ The output of this sprint should be concrete enough that later work can proceed
|
||||
without re-deriving what the first real gates are or how they map into semantic
|
||||
entities.
|
||||
|
||||
## Architectural Constraint — Intake Contract
|
||||
|
||||
**Discovered this session.** This constraint changes how RSA training data is sourced.
|
||||
|
||||
The current whetstone_DSL run corpus predates a planned intake contract. The intended
|
||||
design is that external tools (HiveMind, gleaner, constcad, etc.) submit project
|
||||
descriptions in a **structured contract format** that explicitly carries the fields
|
||||
needed to make RSA gate decisions bounded and classifiable.
|
||||
|
||||
Without the contract, the intake produces ~18 unique template outputs regardless of
|
||||
project. With the contract, every project submission becomes a varied, labeled training
|
||||
example with known decision-relevant fields.
|
||||
|
||||
**Correct pipeline (with contract):**
|
||||
|
||||
```
|
||||
External tool
|
||||
↓ submits structured project description (contract fields)
|
||||
Intake contract schema ← explicit per-field semantics
|
||||
↓
|
||||
architect_intake reads contract, produces normalized form
|
||||
↓
|
||||
RSA gates fire on contract fields ← bounded, varied, discriminative
|
||||
↓
|
||||
generate_taskitems ← deterministic given gate outputs
|
||||
```
|
||||
|
||||
**Implications for RSA design:**
|
||||
|
||||
- Gate input features are **contract fields**, not free-form text
|
||||
- Training data comes from **contract submissions**, not ad hoc specs
|
||||
- The current corpus is valid for label schema validation only
|
||||
- Sprint plan step descriptions remain a valid interim text source for baseline training
|
||||
- The contract schema must be defined before RSA training data can be collected at scale
|
||||
|
||||
**Next design milestone (not this sprint):**
|
||||
|
||||
Define the intake contract schema: what fields must an external tool provide, and which
|
||||
fields are the discriminative inputs for each immediate RSA gate (`verification_type`,
|
||||
`worker_type`, `prereq_op_selector`, `cpp_raising_profile`).
|
||||
|
||||
## Why This Sprint Exists
|
||||
|
||||
`SPRINT-002-rsa-semantic-bootstrap.md` established the architectural direction,
|
||||
@@ -317,9 +358,143 @@ Turn `prereq_op_selector` into the first accepted dataset slice, with:
|
||||
|
||||
Status:
|
||||
|
||||
- partially complete
|
||||
- `prereq_op_selector` is now represented as structured/factorized in
|
||||
`semantic/rsa_semantic_bootstrap_v0.json`
|
||||
- **complete** for the 50-run slice
|
||||
- split produced by `tools/split_prereq_op_dataset.py`
|
||||
- outputs at `semantic/prereq_op_dataset/`
|
||||
- `accepted.ndjson` — 94 rows
|
||||
- `rejected.ndjson` — 12 rows (all schema-drift)
|
||||
- `schema_drift.ndjson` — 12 rows (same set; all rejections are schema-drift)
|
||||
|
||||
### Key Findings From Split
|
||||
|
||||
**Label distribution (accepted rows):**
|
||||
|
||||
| Combo | Count |
|
||||
|-------|-------|
|
||||
| validate\_intake only | 13 |
|
||||
| validate\_intake + resolve\_dependencies | 47 |
|
||||
| validate\_intake + architect\_review | 34 |
|
||||
|
||||
**`needs_validate_intake` is always True** across all 94 accepted rows.
|
||||
|
||||
This means:
|
||||
- `validate_intake` is not a decision surface — it is a constant output in this corpus.
|
||||
- The effective gate is a 3-way classification:
|
||||
- `validate_only` (13 rows, 13.8%)
|
||||
- `validate+resolve` (47 rows, 50%)
|
||||
- `validate+architect` (34 rows, 36.2%)
|
||||
- Or equivalently, two independent binary decisions:
|
||||
- `needs_resolve_dependencies` (47/94 = 50%)
|
||||
- `needs_architect_review` (34/94 = 36%)
|
||||
|
||||
**Schema-drift rows carry valid label signal.** The 12 rejected rows have correct
|
||||
`needs_*` booleans. The only issue is `source_values` contains tool IDs mixed
|
||||
with valid op names. Salvage path: re-extract only the valid op names and strip
|
||||
the tool IDs, then accept the row.
|
||||
|
||||
### Decision Required Before Broadening
|
||||
|
||||
Before running over more runs, decide:
|
||||
|
||||
1. **Hard-filter or salvage schema-drift rows?**
|
||||
- Hard-filter: discard 12 rows, keep extraction simple.
|
||||
- Salvage: strip invalid values from `source_values`, promote to accepted.
|
||||
- Recommendation: hard-filter for the first training run. Salvage later if
|
||||
the accepted corpus stays small after broadening.
|
||||
|
||||
2. **Flat 3-way vs factorized binary formulation?**
|
||||
- Flat 3-way: treat as single multiclass with 3 outputs (ignoring validate_intake).
|
||||
- Factorized binary: two independent classifiers for `architect_review` and
|
||||
`resolve_dependencies`. `validate_intake` is dropped as a constant.
|
||||
- Recommendation: factorized binary. The two decisions appear semantically
|
||||
orthogonal in the data.
|
||||
|
||||
### 1000-Run Extraction Results
|
||||
|
||||
Output at `semantic/prereq_op_dataset_1000/` (1000 runs, 1321 tasks):
|
||||
|
||||
- accepted: 1297 rows
|
||||
- rejected: 24 rows (all schema-drift)
|
||||
- schema drift variants:
|
||||
- tool-ID drift (`whetstone_generate_taskitems`, etc.): 12 rows
|
||||
- `manual-approval` drift: 12 rows (undocumented op name in `prerequisiteOps`)
|
||||
|
||||
**Label distribution (1000-run accepted):**
|
||||
|
||||
| Combo | Count | Pct |
|
||||
|-------|-------|-----|
|
||||
| validate\_intake + resolve\_dependencies | 996 | 76.8% |
|
||||
| validate\_intake + architect\_review | 256 | 19.7% |
|
||||
| validate\_intake only | 44 | 3.4% |
|
||||
| all three | 1 | 0.1% |
|
||||
|
||||
`needs_validate_intake` remains always True at 1000-run scale.
|
||||
|
||||
Effective gate at 1000 runs:
|
||||
- `needs_resolve_dependencies`: 997/1297 = 76.9% positive
|
||||
- `needs_architect_review`: 257/1297 = 19.8% positive
|
||||
|
||||
Class imbalance for `needs_architect_review` (~80/20) is manageable for training.
|
||||
The `validate_intake` constant can be dropped from the label space.
|
||||
|
||||
Both schema-drift variants are **salvageable**: valid bits (`needs_*`) are correct;
|
||||
only `source_values` contains noise. Recommendation: hard-filter for first training
|
||||
run, evaluate whether 24 extra rows are worth a salvage pass after.
|
||||
|
||||
### Text Feature Audit — Critical Finding
|
||||
|
||||
Attempted text composition from extracted row context fields: `title`, `reasons`,
|
||||
`normalized_requirements`. Result:
|
||||
|
||||
- unique texts across 1297 rows: **17** (title+reasons), **18** (acceptance criteria)
|
||||
- top text appears **697 times** ("Execution Readiness Primary | high_confidence_clear_path")
|
||||
- structured features (confidence, dependency count) are also nearly constant
|
||||
|
||||
**Root cause:** The whetstone_DSL corpus uses a small set of canonical intake specs
|
||||
(~18 unique templates) run many times. All tasks generated from the same intake share
|
||||
the same context text. The step description text — the actual discriminative feature —
|
||||
is not stored in run artifacts.
|
||||
|
||||
**Conclusion:**
|
||||
|
||||
The run corpus captures **labels** correctly but not the **text features** needed to
|
||||
train a text-based prereq_op specialist.
|
||||
|
||||
The correct training data source for text input remains sprint plan step descriptions
|
||||
(as used by `gen_prereq_op_data.py` in the specialist fleet). The run corpus contributes:
|
||||
- validated label schema
|
||||
- label frequency distribution (validate-intake constant, resolve 77%, architect 20%)
|
||||
- schema-drift pattern documentation
|
||||
- corpus for future label-correction of heuristic sprint-plan data
|
||||
|
||||
**Data strategy going forward:**
|
||||
|
||||
| Source | Role |
|
||||
|--------|------|
|
||||
| Sprint plan step descriptions | Primary text input for training |
|
||||
| Run corpus (`accepted.ndjson`) | Label validation and frequency baseline |
|
||||
| Run corpus schema-drift rows | Schema-drift taxonomy for robustness testing |
|
||||
|
||||
The sprint plan approach is already implemented in:
|
||||
`CLionProjects/whetstone_DSL/specialists/scripts/gen_prereq_op_data.py`
|
||||
|
||||
That script uses keyword heuristics to assign labels. The RSA extraction work validates
|
||||
that the heuristic label schema is correct; improving label accuracy requires linking
|
||||
sprint plan steps back to run artifacts (not yet feasible).
|
||||
|
||||
### Next Active Task
|
||||
|
||||
Proceed to training using existing sprint plan data with the factorized binary formulation:
|
||||
|
||||
1. Run `gen_prereq_op_data.py` to regenerate sprint plan TSVs.
|
||||
2. Split into two binary TSVs (resolve / architect) using `prereq_op_binary_split.py`.
|
||||
3. Train both specialists with `train_specialist_pt.py`.
|
||||
4. Evaluate combined accuracy against the 4-way eval set.
|
||||
|
||||
The RSA-extracted `accepted.ndjson` dataset remains available as a label-validation
|
||||
reference and can be used for post-training audit (do sprint-plan-trained specialists
|
||||
generalize to real run labels?).
|
||||
|
||||
- `execution_contract_shape` skeleton is now defined in
|
||||
`docs/case_studies/whetstone_dsl_execution_contract_shape_skeleton.md`
|
||||
- `target_file_selection` and `workstream_grouping` still need explicit skeleton field sets
|
||||
|
||||
224
sprints/SPRINT-004-pipeline-decision-audit.md
Normal file
224
sprints/SPRINT-004-pipeline-decision-audit.md
Normal file
@@ -0,0 +1,224 @@
|
||||
# Sprint 004: Pipeline Decision Audit
|
||||
|
||||
## Status
|
||||
|
||||
Authored — not yet executed. Execute in the next session.
|
||||
|
||||
## Intent
|
||||
|
||||
Manually walk the full whetstone_DSL pipeline from raw project description to
|
||||
generated code artifacts. At every stage, identify every decision being made,
|
||||
record what inputs were available at the time of that decision, and evaluate the
|
||||
optimal method for that decision.
|
||||
|
||||
This sprint produces a decision map, not code and not training data.
|
||||
|
||||
## Why This Sprint Exists
|
||||
|
||||
The current specialist fleet and RSA gate inventory were built top-down: we
|
||||
identified gate candidates from the output schema and worked backward. This sprint
|
||||
inverts that: start from the input and work forward, letting the pipeline tell us
|
||||
what decisions actually exist.
|
||||
|
||||
The prior approach missed decisions that were collapsed into single LLM calls. A
|
||||
single call to `generate_taskitems` currently makes at minimum:
|
||||
|
||||
- task decomposition (how many tasks, what granularity)
|
||||
- task title assignment
|
||||
- milestone grouping
|
||||
- worker type selection
|
||||
- verification type selection
|
||||
- prerequisite op selection
|
||||
- execution contract shaping
|
||||
- target file selection
|
||||
- acceptance command selection
|
||||
- required tool selection
|
||||
- confidence scoring
|
||||
- ambiguity annotation
|
||||
- reason generation
|
||||
|
||||
These are not one decision. They are 10+ decisions bundled for token efficiency.
|
||||
The audit's job is to unbundle them and evaluate each one independently.
|
||||
|
||||
## Scope
|
||||
|
||||
### In scope
|
||||
|
||||
- Walk every pipeline stage from spec text to final code artifact
|
||||
- For each stage: list every decision made in that stage
|
||||
- For each decision: record inputs available, current implementation, proposed method
|
||||
- Identify decisions that are genuinely coupled vs accidentally bundled
|
||||
- Note where current inputs are insufficient and what the gate should do (abstain,
|
||||
escalate, or decide from partial information)
|
||||
|
||||
### Not in scope
|
||||
|
||||
- Running any pipeline code
|
||||
- Writing training data generators
|
||||
- Training any specialist
|
||||
- Modifying any whetstone_DSL source
|
||||
- Designing gate schemas or contracts in detail
|
||||
|
||||
## Pipeline Stages To Audit
|
||||
|
||||
### Stage 0: Project description ingestion
|
||||
|
||||
Raw input: a project description in some format (markdown, structured JSON, free text).
|
||||
|
||||
Questions to answer:
|
||||
- What formats are currently accepted?
|
||||
- What decisions are implicit in accepting/rejecting an input format?
|
||||
- Is format normalization a decision or a transformation?
|
||||
- What is the minimum viable input for the pipeline to proceed?
|
||||
- What does the pipeline do when input is partial or ambiguous?
|
||||
|
||||
### Stage 1: `architect_intake`
|
||||
|
||||
Current implementation: single LLM call over the raw spec.
|
||||
|
||||
Output fields to audit (each may represent a separate decision):
|
||||
- `normalizedRequirements` — which lines become requirements, which are discarded
|
||||
- `parsedSpec.goals` — goal extraction and normalization
|
||||
- `parsedSpec.constraints` — constraint extraction
|
||||
- `parsedSpec.acceptanceCriteria` — acceptance criterion extraction
|
||||
- `parsedSpec.dependencies` — dependency extraction
|
||||
- `conflictSignals` — conflict detection
|
||||
- `ambiguitySignals` — ambiguity detection
|
||||
|
||||
For each output field:
|
||||
- What inputs does this decision use?
|
||||
- Is this deterministic given the input, or genuinely uncertain?
|
||||
- What method is appropriate (rule, RSA, SLM, LLM)?
|
||||
- What does failure or partial output mean for downstream stages?
|
||||
|
||||
### Stage 2: `generate_taskitems`
|
||||
|
||||
Current implementation: single LLM call over normalized requirements.
|
||||
|
||||
Decisions to unbundle and audit individually:
|
||||
|
||||
**Decomposition decisions**
|
||||
- How many tasks to create
|
||||
- What granularity to use
|
||||
- How to group related work into milestones
|
||||
- Whether a requirement maps to one task or multiple
|
||||
|
||||
**Per-task assignment decisions**
|
||||
- `title` — what to call the task
|
||||
- `milestoneId` — which milestone this task belongs to
|
||||
- `workerType` — who does this work (implementer, reviewer, architect, qa)
|
||||
- `verificationType` — how to verify completion (unit, integration, schema, smoke, docs)
|
||||
- `prerequisiteOps` — what gates must clear before this task runs
|
||||
- `confidence` — how confident is the system in this task
|
||||
- `ambiguityCount` — how many ambiguous signals were present
|
||||
- `reasons` — why was this task generated
|
||||
|
||||
**Execution contract decisions** (currently nested inside each task)
|
||||
- `targetFiles` — which files this task touches
|
||||
- `acceptanceCommands` — how to verify the task is done
|
||||
- `requiredTools` — which tools are needed
|
||||
- `crossProjectTargets` — does this task touch other projects
|
||||
- `resourceLocks` — what resources does this task need exclusive access to
|
||||
|
||||
For each decision: same four questions as Stage 1.
|
||||
|
||||
### Stage 3: `queue_ready`
|
||||
|
||||
Current implementation: deterministic validation.
|
||||
|
||||
Questions to answer:
|
||||
- Is this purely deterministic, or are there judgment calls embedded in it?
|
||||
- What validation rules exist and are they all rule-complete?
|
||||
- Are there cases where queue_ready makes a decision that should be RSA?
|
||||
|
||||
### Stage 4: Per-task execution (`run_pipeline`)
|
||||
|
||||
Current implementation: one LLM call per task step, given the execution contract.
|
||||
|
||||
This stage likely contains its own collapsed decision bundle. Questions to answer:
|
||||
- What does the executor decide before writing any code?
|
||||
- What decisions are made during code generation vs before it starts?
|
||||
- What decisions are made per file vs per task?
|
||||
- How does the executor handle execution contract fields that are missing or ambiguous?
|
||||
|
||||
### Stage 5: Verification
|
||||
|
||||
Current implementation: runs acceptance commands, reports pass/fail.
|
||||
|
||||
Questions to answer:
|
||||
- Is the verification decision purely mechanical (pass/fail), or is there judgment?
|
||||
- What happens when acceptance commands are absent or malformed?
|
||||
- Is there a triage decision when verification fails?
|
||||
|
||||
## For Each Decision: What To Record
|
||||
|
||||
Each discovered decision should be recorded as a decision record with these fields:
|
||||
|
||||
```
|
||||
decision_id: short unique identifier (e.g., "intake.conflict_detection")
|
||||
stage: which pipeline stage this fires in
|
||||
inputs_available: what information exists at decision time
|
||||
current_impl: how it is currently implemented (LLM / rule / heuristic / none)
|
||||
output_type: boolean / enum / structured / free-text / numeric
|
||||
output_values: the bounded set of outputs, if applicable
|
||||
coupling: which other decisions does this one depend on or influence
|
||||
proposed_method: deterministic / RSA / SLM / LLM / human
|
||||
rationale: why that method fits this decision
|
||||
rsa_notes: if RSA: what the gate input features are, what abstain looks like
|
||||
open_questions: anything unresolved
|
||||
```
|
||||
|
||||
## Code Surfaces To Examine
|
||||
|
||||
These are the primary source files for understanding what decisions currently exist:
|
||||
|
||||
| Surface | Location |
|
||||
|---------|----------|
|
||||
| `architect_intake` tool | `editor/src/mcp/RegisterArchitectIntakeTools.h` |
|
||||
| `generate_taskitems` tool | `editor/src/mcp/RegisterTaskitemGeneratorTools.h` |
|
||||
| Taskitem generator logic | `editor/src/TaskitemGeneratorV2.h` |
|
||||
| Confidence and ambiguity | `editor/src/TaskitemConfidenceAmbiguity.h` |
|
||||
| Queue validation | `editor/src/mcp/RegisterQueueTools.h` |
|
||||
| Pipeline execution | `editor/src/mcp/RegisterPipelineTools.h` |
|
||||
| Routing | `editor/src/RoutingEngine.h` |
|
||||
| Execution contract | `editor/src/ExecutionContractBuilder.h` (if exists) |
|
||||
|
||||
For each file: read the decision logic, not just the schema. The schema tells you
|
||||
what fields exist. The logic tells you what actually determines their values.
|
||||
|
||||
## Output of This Sprint
|
||||
|
||||
One document: `docs/case_studies/whetstone_dsl_pipeline_decision_map.md`
|
||||
|
||||
Structure:
|
||||
- One section per pipeline stage
|
||||
- Within each section: one subsection per decision record
|
||||
- Summary table at the top: all decisions, their current impl, and proposed method
|
||||
|
||||
This document becomes the authoritative input for:
|
||||
- RSA gate inventory (replaces the current incomplete inventory)
|
||||
- Training data collection strategy (once we know what inputs each gate sees)
|
||||
- Sprint 005: first gate implementation from the decision map
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Every stage from project description to generated artifact is covered
|
||||
- Every output field of every pipeline tool is traced to at least one decision record
|
||||
- No decision record has `proposed_method: LLM` without a written rationale for why
|
||||
a smaller or more bounded approach is insufficient
|
||||
- The summary table exists and is navigable
|
||||
- At least one decision currently implemented as a single LLM call is shown to contain
|
||||
three or more separable decisions
|
||||
|
||||
## Execution Notes for Next Session
|
||||
|
||||
Start with Stage 1 (`architect_intake`) — it is the entry point and its outputs
|
||||
constrain everything downstream. Read the source, not just the schema.
|
||||
|
||||
Do not assume a field is a single decision because it has a single key in the output
|
||||
JSON. JSON keys are output slots. Decisions are the reasoning steps that fill them.
|
||||
|
||||
When a decision is genuinely coupled to another (output of A determines the valid
|
||||
space of B), record the coupling explicitly rather than merging the decisions. Coupled
|
||||
decisions that are *always* coupled may warrant a structured output gate. Decisions
|
||||
that are *sometimes* coupled may warrant independent gates with a dependency edge.
|
||||
Reference in New Issue
Block a user