7.4 KiB
Specialist Fleet Handoff — 2026-04-20
Session Summary
worker_type implementer improvement: fixed heuristic label noise, added synthetic implementer data, retrained. Implementer accuracy at target. Heuristic fix is a significant data quality improvement that affects all future sprint plan data.
Fleet Status
| Gate | Best Acc | Tier | Checkpoint | Next Action |
|---|---|---|---|---|
| automatability | 100% | 2-stage | /mnt/storage/fabricate_runs/whetstone_automatability_stage* |
SOLVED — deploy |
| worker_type | 89.2% (v2) | small_plus | /mnt/storage/fabricate_runs/whetstone_worker_type_v2_small_plus/checkpoint.pt |
See Priority 1 below |
| verification_type | 84.3% | small_plus | /mnt/storage/fabricate_runs/whetstone_verification_type_small_plus/checkpoint.pt |
See Priority 2 below |
| prereq_op | 72.2% | small_plus | blocked | Blocked — schema gap |
Prior session baseline (v1 combined): worker_type 92.2% overall / 75.0% implementer. This session (v2): 89.2% overall / 85.7% implementer.
What Was Done
1. Heuristic fix — gen_worker_type_data.py
Root cause found: classify() used substring matching (tok in t), causing
multi-word token false positives:
whetstone_preview_*tools labeled reviewer because "review" ⊆ "preview"whetstone_*_triage_*tools labeled reviewer because "triage" ⊆ "failure_triage"whetstone_*_test_*tools labeled qa because "test" ⊆ "test_output"- Tasks with "scanner", "auditor", "diagnostics" labeled reviewer/qa incorrectly
Fix: switched to re.compile(r'\b' + re.escape(tok) + r'\b') word-boundary
matching for all three token sets (QA, ARCHITECT, REVIEWER).
Impact: 67 label changes in the sprint plan corpus. All were false positives moving from reviewer/qa/architect → implementer. The fix is correct for all examined cases.
File changed: specialists/scripts/gen_worker_type_data.py
2. Synthetic implementer data — gen_implementer_synthetic.py
Problem: implementer was labeled by keyword absence (default). Model never saw strong positive implementer signal, causing 75% accuracy on the class.
Fix: added specialists/scripts/gen_implementer_synthetic.py — generates 158
synthetic implementer examples in two formats:
- Short title format (sprint-plan style): "Implement X", "Build X", "Add X", etc.
- Title | intent format (projects style): "Build X | Implement Y with Z features"
Includes "ambiguous-noun" examples: tasks where the construction verb is explicit but the object contains words like "audit", "review", "triage" (e.g. "Build audit log service"). These conflict with the heuristic labeling on the same patterns — see Priority 1 below.
Output files:
specialists/data/generated/worker_type_implementer_synthetic.train.tsv(135 rows)specialists/data/generated/worker_type_implementer_synthetic.eval.tsv(23 rows)specialists/data/generated/worker_type_v2_train.tsv(combined + synthetic, 690 rows)specialists/data/generated/worker_type_v2_eval.tsv(combined + synthetic, 120 rows)
3. v2 sweep results (pilot-only, 3000 steps)
| Tier | Overall | implementer | reviewer | architect | qa |
|---|---|---|---|---|---|
| small_plus | 89.2% | 85.7% | 92.3% | 91.3% | 90.9% |
| medium | 85.8% | — | — | — | — |
Confusion (small_plus):
- implementer → reviewer: 4 (7 errors total on 49 eval)
- implementer → architect: 3
- architect → implementer: 2
- qa → implementer: 2
- reviewer → implementer: 1
Capacity plateau confirmed: medium worse than small_plus. Data, not capacity, is the bottleneck.
Priority 1: Resolve the synthetic data conflict (worker_type)
The synthetic implementer data includes "Build audit log service", "Implement review scoring engine" etc. labeled as implementer. But the heuristic also labels similar sprint plan steps as reviewer (because "audit"/"review" match REVIEWER_TOKENS with word boundaries). This creates conflicting labels in the combined dataset.
Two options:
Option A (recommended): Remove ambiguous-noun examples from synthetic set. Keep only synthetic examples where the construction verb is paired with unambiguous nouns (auth, billing, webhook, scheduler, etc.). No conflict with heuristic labels. Simpler and more conservative.
Option B: Extend heuristic with construction-verb override. Add a CONSTRUCTION_VERBS set. If the text starts with "Build", "Implement", "Add", "Write", etc., classify as implementer regardless of other tokens. More aggressive, requires re-checking all 67 changed labels.
To apply Option A: edit gen_implementer_synthetic.py, remove the
"Ambiguous-noun implementer" section from SHORT_TITLES and TITLE_INTENT_TEMPLATES,
then re-run to regenerate v2 files and retrain.
Current 85.7% implementer is already at target (handoff said >85%). But the conflicting labels are a data quality issue worth resolving before declaring done.
Priority 2: verification_type — combined dataset
Currently 84.3% (capacity plateau confirmed). Try the same approach that worked for
worker_type: combine sprint plan TSV with verification_type_projects_train.tsv.
cat specialists/data/generated/verification_type_train.tsv \
specialists/data/generated/verification_type_projects_train.tsv \
> specialists/data/generated/verification_type_combined_train.tsv
cat specialists/data/generated/verification_type_eval.tsv \
specialists/data/generated/verification_type_projects_eval.tsv \
> specialists/data/generated/verification_type_combined_eval.tsv
.venv/bin/python3 specialists/scripts/capacity_sweep.py \
--name verification_type_combined \
--train specialists/data/generated/verification_type_combined_train.tsv \
--eval specialists/data/generated/verification_type_combined_eval.tsv \
--labels "unit,integration,schema,smoke,docs" \
--runs-dir /mnt/storage/fabricate_runs \
--results-out specialists/eval/results/verification_type_combined_sweep.json \
--pilot-only
Priority 3: prereq_op — still blocked
No progress possible until whetstone_DSL Sprint 004 (pipeline decision audit) adds discriminative fields to the taskitem schema. Do not pursue.
Eval Set Quality Note
The synthetic eval (23 of 120 rows) is drawn from the same template pool as the synthetic train. This weakens eval as a generalization test. Once the conflicting labels are resolved and the dataset stabilizes, hold out a completely fresh eval set that was never touched during data generation iteration.
Run Commands
cd /home/bill/Documents/CLionProjects/whetstone_DSL
# Re-run v2 sweep after resolving synthetic conflict (Option A)
.venv/bin/python3 specialists/scripts/capacity_sweep.py \
--name worker_type_v2 \
--train specialists/data/generated/worker_type_v2_train.tsv \
--eval specialists/data/generated/worker_type_v2_eval.tsv \
--labels "implementer,reviewer,architect,qa" \
--runs-dir /mnt/storage/fabricate_runs \
--results-out specialists/eval/results/worker_type_v2_sweep.json \
--pilot-only
# Error inspection (loads v2 small_plus checkpoint)
.venv/bin/python3 /tmp/show_errors.py
Checkpoints
whetstone_worker_type_v2_small_plus/checkpoint.pt— 89.2% overall, 85.7% implementerwhetstone_automatability_stage1_binary/checkpoint.pt— stage 1 (binary) 100%whetstone_automatability_stage2_4way/checkpoint.pt— stage 2 (4-way) 100%whetstone_verification_type_small_plus/checkpoint.pt— 84.3% (needs more data)