whetstone_DSL/specialists/HANDOFF-2026-04-20.md

# Specialist Fleet Handoff — 2026-04-20

## Session Summary

worker_type implementer improvement: fixed heuristic label noise, added synthetic
implementer data, retrained. Implementer accuracy at target. Heuristic fix is a
significant data quality improvement that affects all future sprint plan data.

---

## Fleet Status

| Gate | Best Acc | Tier | Checkpoint | Next Action |
|------|----------|------|------------|-------------|
| automatability | **100%** | 2-stage | `/mnt/storage/fabricate_runs/whetstone_automatability_stage*` | SOLVED — deploy |
| worker_type | **89.2%** (v2) | small_plus | `/mnt/storage/fabricate_runs/whetstone_worker_type_v2_small_plus/checkpoint.pt` | See Priority 1 below |
| verification_type | 84.3% | small_plus | `/mnt/storage/fabricate_runs/whetstone_verification_type_small_plus/checkpoint.pt` | See Priority 2 below |
| prereq_op | 72.2% | small_plus | blocked | Blocked — schema gap |

Prior session baseline (v1 combined): worker_type 92.2% overall / 75.0% implementer.
This session (v2): 89.2% overall / 85.7% implementer.

---

## What Was Done

### 1. Heuristic fix — gen_worker_type_data.py

**Root cause found:** `classify()` used substring matching (`tok in t`), causing
multi-word token false positives:
- `whetstone_preview_*` tools labeled reviewer because "review" ⊆ "preview"
- `whetstone_*_triage_*` tools labeled reviewer because "triage" ⊆ "failure_triage"
- `whetstone_*_test_*` tools labeled qa because "test" ⊆ "test_output"
- Tasks with "scanner", "auditor", "diagnostics" labeled reviewer/qa incorrectly

**Fix:** switched to `re.compile(r'\b' + re.escape(tok) + r'\b')` word-boundary
matching for all three token sets (QA, ARCHITECT, REVIEWER).

**Impact:** 67 label changes in the sprint plan corpus. All were false positives
moving from reviewer/qa/architect → implementer. The fix is correct for all
examined cases.

**File changed:** `specialists/scripts/gen_worker_type_data.py`

### 2. Synthetic implementer data — gen_implementer_synthetic.py

**Problem:** implementer was labeled by keyword *absence* (default). Model never
saw strong positive implementer signal, causing 75% accuracy on the class.

**Fix:** added `specialists/scripts/gen_implementer_synthetic.py` — generates 158
synthetic implementer examples in two formats:
- Short title format (sprint-plan style): "Implement X", "Build X", "Add X", etc.
- Title | intent format (projects style): "Build X | Implement Y with Z features"

Includes "ambiguous-noun" examples: tasks where the construction verb is explicit
but the object contains words like "audit", "review", "triage" (e.g. "Build audit
log service"). These conflict with the heuristic labeling on the same patterns —
see Priority 1 below.

**Output files:**
- `specialists/data/generated/worker_type_implementer_synthetic.train.tsv` (135 rows)
- `specialists/data/generated/worker_type_implementer_synthetic.eval.tsv` (23 rows)
- `specialists/data/generated/worker_type_v2_train.tsv` (combined + synthetic, 690 rows)
- `specialists/data/generated/worker_type_v2_eval.tsv` (combined + synthetic, 120 rows)

### 3. v2 sweep results (pilot-only, 3000 steps)

| Tier | Overall | implementer | reviewer | architect | qa |
|------|---------|-------------|----------|-----------|-----|
| small_plus | 89.2% | **85.7%** | 92.3% | 91.3% | 90.9% |
| medium | 85.8% | — | — | — | — |

Confusion (small_plus):
- implementer → reviewer: 4  (7 errors total on 49 eval)
- implementer → architect: 3
- architect → implementer: 2
- qa → implementer: 2
- reviewer → implementer: 1

Capacity plateau confirmed: medium worse than small_plus. Data, not capacity, is the
bottleneck.

---

## Priority 1: Resolve the synthetic data conflict (worker_type)

The synthetic implementer data includes "Build audit log service",
"Implement review scoring engine" etc. labeled as implementer. But the heuristic
also labels similar sprint plan steps as reviewer (because "audit"/"review" match
REVIEWER_TOKENS with word boundaries). This creates conflicting labels in the
combined dataset.

**Two options:**

**Option A (recommended): Remove ambiguous-noun examples from synthetic set.**
Keep only synthetic examples where the construction verb is paired with unambiguous
nouns (auth, billing, webhook, scheduler, etc.). No conflict with heuristic labels.
Simpler and more conservative.

**Option B: Extend heuristic with construction-verb override.**
Add a CONSTRUCTION_VERBS set. If the text starts with "Build", "Implement", "Add",
"Write", etc., classify as implementer regardless of other tokens.
More aggressive, requires re-checking all 67 changed labels.

To apply Option A: edit `gen_implementer_synthetic.py`, remove the
"Ambiguous-noun implementer" section from `SHORT_TITLES` and `TITLE_INTENT_TEMPLATES`,
then re-run to regenerate v2 files and retrain.

Current 85.7% implementer is already at target (handoff said >85%). But the
conflicting labels are a data quality issue worth resolving before declaring done.

### Priority 2: verification_type — combined dataset

Currently 84.3% (capacity plateau confirmed). Try the same approach that worked for
worker_type: combine sprint plan TSV with `verification_type_projects_train.tsv`.

```bash
cat specialists/data/generated/verification_type_train.tsv \
    specialists/data/generated/verification_type_projects_train.tsv \
    > specialists/data/generated/verification_type_combined_train.tsv

cat specialists/data/generated/verification_type_eval.tsv \
    specialists/data/generated/verification_type_projects_eval.tsv \
    > specialists/data/generated/verification_type_combined_eval.tsv

.venv/bin/python3 specialists/scripts/capacity_sweep.py \
  --name verification_type_combined \
  --train specialists/data/generated/verification_type_combined_train.tsv \
  --eval  specialists/data/generated/verification_type_combined_eval.tsv \
  --labels "unit,integration,schema,smoke,docs" \
  --runs-dir /mnt/storage/fabricate_runs \
  --results-out specialists/eval/results/verification_type_combined_sweep.json \
  --pilot-only
```

### Priority 3: prereq_op — still blocked

No progress possible until whetstone_DSL Sprint 004 (pipeline decision audit) adds
discriminative fields to the taskitem schema. Do not pursue.

---

## Eval Set Quality Note

The synthetic eval (23 of 120 rows) is drawn from the same template pool as the
synthetic train. This weakens eval as a generalization test. Once the conflicting
labels are resolved and the dataset stabilizes, hold out a completely fresh eval set
that was never touched during data generation iteration.

---

## Run Commands

```bash
cd /home/bill/Documents/CLionProjects/whetstone_DSL

# Re-run v2 sweep after resolving synthetic conflict (Option A)
.venv/bin/python3 specialists/scripts/capacity_sweep.py \
  --name worker_type_v2 \
  --train specialists/data/generated/worker_type_v2_train.tsv \
  --eval  specialists/data/generated/worker_type_v2_eval.tsv \
  --labels "implementer,reviewer,architect,qa" \
  --runs-dir /mnt/storage/fabricate_runs \
  --results-out specialists/eval/results/worker_type_v2_sweep.json \
  --pilot-only

# Error inspection (loads v2 small_plus checkpoint)
.venv/bin/python3 /tmp/show_errors.py
```

## Checkpoints

- `whetstone_worker_type_v2_small_plus/checkpoint.pt` — 89.2% overall, 85.7% implementer
- `whetstone_automatability_stage1_binary/checkpoint.pt` — stage 1 (binary) 100%
- `whetstone_automatability_stage2_4way/checkpoint.pt` — stage 2 (4-way) 100%
- `whetstone_verification_type_small_plus/checkpoint.pt` — 84.3% (needs more data)