whetstone_RSA/HANDOFF-2026-04-12-session2.md

Handoff: RSA Session 2 — 2026-04-12

What Was Done

Extraction and dataset split

• Built tools/split_prereq_op_dataset.py

  • splits extracted gate rows into accepted / rejected / schema-drift buckets
  • for prereq_op_selector only

• Ran extract_gate_rows.py over 1000 runs (from 50-run slice)

  • 1297 accepted rows, 24 rejected (all schema-drift)
  • outputs at semantic/prereq_op_dataset_1000/

• Built tools/gen_prereq_op_rsa_data.py

  • converts accepted rows to Fabricate TSV format
  • outputs at semantic/prereq_op_tsv/

Key findings from dataset analysis

needs_validate_intake is always True (1297/1297 accepted rows). It is a constant, not a decision surface. Drop it from the label space.

Effective gate is two binary classifiers:

• needs_resolve_dependencies: 997/1297 = 76.9% positive
• needs_architect_review: 257/1297 = 19.8% positive

Text features are useless from run corpus. Only 17 unique texts across 1297 rows — the corpus uses ~18 canonical intake spec templates. Every task generated from the same intake shares the same context text. The discriminative signal (step description) is not stored in run artifacts.

Schema-drift variants found:

• tool-ID drift: whetstone_generate_taskitems etc. in prerequisiteOps (12 rows)
• manual-approval drift: undocumented op name in prerequisiteOps (12 rows) Both are salvageable (valid bits are correct), recommended to hard-filter for now.

Training baseline

Sprint plan data (204 examples, 36 eval) split into two binary TSVs and trained:

Specialist	best_acc	steps
prereq_review (needs architect_review)	75.0%	4000
prereq_approval (needs manual_approval)	69.4%	4000

These are weak baselines — small dataset, heuristic labels, wrong gate framing (approval ≠ resolve_dependencies). Not production-ready.

Architectural Insights Captured (Most Important Part of This Session)

1. Intake contract

The original design intention is that external tools (HiveMind, gleaner, etc.) submit project descriptions in a structured contract format that explicitly carries decision-relevant fields. The current corpus predates this contract.

With the contract, every project submission is a varied, labeled training example with known discriminative fields. Without it, the intake produces ~18 template outputs regardless of project.

Reference: project.json manifests are the closest existing contract artifact. The llm_touchpoints array with determinism_score is the RSA entropy signal.

2. Taskitem as self-contained work unit

A taskitem is a concurrent work unit. By design it must include all information the specialist needs. The specialist does not reach back to the original spec.

This means the taskitem schema must carry discriminative RSA fields. generate_taskitems populates them; the RSA gate reads them from the taskitem.

The current schema lacks these fields — that is why the run corpus has weak text.

Adding RSA-relevant fields to the taskitem schema is the prerequisite for collecting real training data at scale.

3. Contract can collapse RSA gates to deterministic

If the contract carries architectural_surface, cross_component_deps, security_sensitive etc. explicitly, some gates that currently require ML become deterministic rules. The RSA entropy threshold shifts as the contract gets richer.

What Is Recorded Where

Finding	Location
Sprint status + findings	sprints/SPRINT-003-...md (updated)
Taskitem-as-work-unit constraint	docs/case_studies/whetstone_dsl_training_data_strategy.md
Intake contract constraint	sprints/SPRINT-003-...md (Architectural Constraint section)
Split tool	tools/split_prereq_op_dataset.py
TSV generator (RSA-sourced)	tools/gen_prereq_op_rsa_data.py
1000-run extracted dataset	semantic/prereq_op_dataset_1000/
Trained baseline checkpoints	/mnt/storage/fabricate_runs/whetstone_prereq_review_binary/
	/mnt/storage/fabricate_runs/whetstone_prereq_approval_binary/

What Is NOT Done (Next Session)

Execute SPRINT-004: Pipeline Decision Audit.

Full sprint document: sprints/SPRINT-004-pipeline-decision-audit.md

The goal is a manual walkthrough of the full whetstone_DSL pipeline from raw project description to generated code. At every stage, record every decision being made, what inputs were available, and what the optimal method is (deterministic / RSA / SLM / LLM / human).

Current LLM calls bundle many decisions together for token efficiency. The audit unbundles them. generate_taskitems alone contains 10+ separable decisions.

Output: docs/case_studies/whetstone_dsl_pipeline_decision_map.md

This document replaces the current gate inventory as the authoritative RSA input. Training data collection strategy follows from it, not before it.

Start here next session

Read editor/src/mcp/RegisterArchitectIntakeTools.h first — it is the pipeline entry point and its outputs constrain everything downstream.

Do Not Do Next Session

• Do not run any pipeline code
• Do not write training data generators
• Do not train any specialist
• Do not modify whetstone_DSL source
• Do not collect more run artifacts until the decision map is complete