whetstone_DSL/specialists/HANDOFF-2026-04-19.md

Specialist Fleet Handoff — 2026-04-19

Session Summary

Full pilot sweep + automatability decomposition + worker_type data improvement. All results committed.

---

Fleet Status

Gate	Best Acc	Tier	Verdict	Action
automatability	100% (5-way)	2-stage	SOLVED	Deploy decomposed pipeline
worker_type	92.2%	small_plus	scale_more	More implementer data
verification_type	84.3%	medium	no_gain_stop_scaling	More/sharper data
prereq_op	72.2%	small_plus	decompose_or_relabel	Blocked — schema gap

Prior session baselines: worker_type 88.5%, verification_type 84.9%, prereq_op 69.4%, automatability 69.6%. Note: those baselines used different (smaller) datasets and are not directly comparable — the sweep is now on a clean unified pipeline.

---

What Was Done

1. Full pilot sweep (sweep_all_gates.sh --pilot-only)

All 4 gates, small_plus + medium tiers, 3000 steps each. Results: specialists/eval/results/*_sweep.json

Key finding: medium tier did not improve over small_plus for any gate. Capacity is not the bottleneck anywhere in the fleet.

2. Automatability — 2-stage decomposition (SOLVED)

Root cause: Directional Collapse (Pattern #1 in whetstone_RSA/docs/diagnostic_patterns.md). deterministic was mapping 100% of errors to template — the two classes are indistinguishable in taskitem title text. Adding data or scaling both fail.

Fix: 2-stage pipeline:

• Stage 1 binary: low_entropy {deterministic, template} vs needs_routing {specialist, slm, llm, human}
• Stage 2 4-way: specialist / slm / llm / human

Results: Stage 1 = 100%, Stage 2 = 100%, combined 5-way = 100%.

Script: specialists/scripts/split_automatability_decompose.py Split TSVs: specialists/data/generated/automatability_binary_*.tsv, automatability_routed_*.tsv Result: specialists/eval/results/automatability_decompose.json

Note on low_entropy bucket: deterministic vs template is NOT distinguishable from taskitem titles alone — that boundary requires features not in the current input schema (how a task is implemented, not what it builds). If the downstream system ever needs to distinguish them, the fix is Upstream Feature Starvation (Pattern #6): enrich the taskitem schema, not the model.

3. Worker_type — combined dataset

Previous train: 300 examples (sprint plan titles only). Added: worker_type_projects_train.tsv (361 examples, title | intent format). Combined: 661 train / 115 eval.

Results: 80.8% → 92.2% overall. implementer 42.9% → 75.0%. reviewer and architect are now perfect (100%).

Combined TSVs: specialists/data/generated/worker_type_combined_*.tsv Result: specialists/eval/results/worker_type_combined_sweep.json

Remaining implementer confusion is Label Boundary Drift (Pattern #10): the heuristic keyword classifier labels "implementer" by keyword absence (default), producing noisy boundaries on ambiguous titles. The short-text format limits how much a model can learn without richer features.

4. Diagnostic patterns document

whetstone_RSA/docs/diagnostic_patterns.md — committed to whetstone_RSA. 10 named patterns covering the most common gate failure modes, each with: observable signal, mechanism, detection test, specific fix, failure conditions, real example from this project.

Written at a level usable by developers and smaller models without deep ML background. Motivated by the question: how small a model / how junior a developer does it take to apply the insight that solved automatability?

---

What Is NOT Done — Next Session

Priority 1: worker_type implementer improvement

Option A (recommended): Generate more data with explicit construction-verb examples. The current data labels "implementer" by keyword absence — it's the default. Add synthetic examples with clear implementer signals: "Implement X", "Build X", "Write X".

Option B: Human relabeling of the 52 eval examples. If reviewers consistently disagree on >30% of errors, apply Label Boundary Drift fix (rewrite label criteria).

Target: get implementer above 85% before declaring worker_type deployable.

Priority 2: verification_type — more data

Currently 83.7% (small_plus), 84.3% (medium). Capacity plateau confirmed. Confusion concentrates in unit↔integration↔schema — semantically adjacent. These have the projects equivalent: verification_type_projects_train.tsv (unused). Try the same combined-dataset approach that worked for worker_type.

Priority 3: prereq_op — blocked, do not pursue until schema change

Root cause: Upstream Feature Starvation (Pattern #6). The taskitem schema does not carry discriminative fields at inference time. No amount of data or model work fixes this. See whetstone_RSA/docs/diagnostic_patterns.md Pattern #6.

Next action is in whetstone_DSL Sprint 004 (pipeline decision audit), not here.

Not on the list

• automatability: SOLVED, nothing to do
• scaling any gate: capacity plateau confirmed fleet-wide, not the lever

---

Run Commands

# Full pilot sweep from scratch
cd /home/bill/Documents/CLionProjects/whetstone_DSL
bash specialists/scripts/sweep_all_gates.sh --pilot-only 2>&1 | tee specialists/eval/results/sweep_run.log

# Re-run automatability decompose (skip training, models already exist)
.venv/bin/python3 specialists/scripts/split_automatability_decompose.py --skip-train

# Run worker_type combined sweep only
.venv/bin/python3 specialists/scripts/capacity_sweep.py \
  --name worker_type_combined \
  --train specialists/data/generated/worker_type_combined_train.tsv \
  --eval  specialists/data/generated/worker_type_combined_eval.tsv \
  --labels "implementer,reviewer,architect,qa" \
  --runs-dir /mnt/storage/fabricate_runs \
  --results-out specialists/eval/results/worker_type_combined_sweep.json \
  --pilot-only

Checkpoints

All live at /mnt/storage/fabricate_runs/:

• whetstone_automatability_stage1_binary/checkpoint.pt — stage 1 (binary)
• whetstone_automatability_stage2_4way/checkpoint.pt — stage 2 (4-way)
• whetstone_worker_type_combined_small_plus/checkpoint.pt — worker_type 92.2%
• whetstone_verification_type_small_plus/checkpoint.pt — verification_type 83.7%