Sprint 003 findings + Sprint 004 pipeline decision audit plan

- split_prereq_op_dataset.py: extract accepted/rejected/schema-drift rows per gate
- gen_prereq_op_rsa_data.py: convert accepted rows to Fabricate TSV format
- 1000-run extracted dataset: 1297 accepted prereq_op rows, 24 schema-drift
- Key finding: needs_validate_intake always True; effective gate is binary
  (needs_resolve_dependencies 77%, needs_architect_review 20%)
- Key finding: run corpus text is template-driven (17 unique inputs) — not
  suitable for text-based training; RSA exists before ideal input is available
- ROADMAP: gate lifecycle framing — goal is choice of backend, not forced
  determinism; RSA operates on imperfect/mixed/incomplete input by design
- training_data_strategy.md: taskitem-as-self-contained-work-unit constraint
- SPRINT-003: updated with 1000-run results, text audit findings, design constraints
- SPRINT-004: pipeline decision audit — manual walkthrough of every decision
  from project description to generated artifact; execute next session

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill Holcombe
2026-04-12 21:43:17 -06:00
parent 0253a92f28
commit c4569de9ca
18 changed files with 10209 additions and 4 deletions

View File

@@ -173,6 +173,37 @@ The right pattern is:
This should produce a gate-native dataset rather than a project-native dataset.
## Taskitem as Self-Contained Work Unit — Design Constraint
**Added 2026-04-12.**
A taskitem is a concurrent work unit. By design it must include all information a
specialist needs to make its gate decision. The specialist does not reach back to
the original spec or the intake.
**Consequence for RSA gate design:**
The taskitem schema must carry every field that is discriminative for the gate.
`generate_taskitems` populates those fields from the intake contract. The RSA gate
reads those fields from the taskitem — no other context is available at decision time.
**Consequence for training data:**
The current taskitem schema does not carry discriminative fields for most RSA gates.
The run corpus has weak text (17 unique templates) because the schema was designed for
a different purpose — it records outputs, not inputs.
RSA gates are designed to operate on whatever input actually exists — including
mixed structured/unstructured content and incomplete directions. Waiting for ideal
input is not the RSA model.
The current corpus is the training corpus. Sprint plan step descriptions are the best
available text signal now. What the gate learns from imperfect inputs, and when it
abstains on insufficient inputs, is the gate's actual contract.
Richer taskitem fields (if added) give the gate more signal and may shift it toward a
deterministic backend — but that is operator choice, not a prerequisite for deployment.
## Compatibility Constraint
We should not assume the new specialists must fit perfectly into the exact old