Specialist fleet handoff 2026-04-19 + remaining eval artifacts

Handoff covers: automatability SOLVED (2-stage, 100%), worker_type 92.2% (combined dataset), verification_type 84.3% (more data needed), prereq_op blocked on taskitem schema gap. Next: worker_type implementer data gen, verification_type combined dataset, prereq_op deferred to Sprint 004. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 13:46:08 -06:00
parent 15125dce25
commit 486940cbe4
9 changed files with 817 additions and 0 deletions
--- a/specialists/HANDOFF-2026-04-19.md
+++ b/specialists/HANDOFF-2026-04-19.md
@@ -0,0 +1,149 @@
+# Specialist Fleet Handoff — 2026-04-19
+
+## Session Summary
+
+Full pilot sweep + automatability decomposition + worker_type data improvement.
+All results committed.
+
+---
+
+## Fleet Status
+
+| Gate | Best Acc | Tier | Verdict | Action |
+|------|----------|------|---------|--------|
+| automatability | **100%** (5-way) | 2-stage | **SOLVED** | Deploy decomposed pipeline |
+| worker_type | **92.2%** | small_plus | scale_more | More implementer data |
+| verification_type | 84.3% | medium | no_gain_stop_scaling | More/sharper data |
+| prereq_op | 72.2% | small_plus | decompose_or_relabel | Blocked — schema gap |
+
+Prior session baselines: worker_type 88.5%, verification_type 84.9%, prereq_op 69.4%,
+automatability 69.6%. Note: those baselines used different (smaller) datasets and
+are not directly comparable — the sweep is now on a clean unified pipeline.
+
+---
+
+## What Was Done
+
+### 1. Full pilot sweep (sweep_all_gates.sh --pilot-only)
+
+All 4 gates, small_plus + medium tiers, 3000 steps each.
+Results: `specialists/eval/results/*_sweep.json`
+
+Key finding: medium tier did not improve over small_plus for any gate.
+Capacity is not the bottleneck anywhere in the fleet.
+
+### 2. Automatability — 2-stage decomposition (SOLVED)
+
+Root cause: Directional Collapse (Pattern #1 in `whetstone_RSA/docs/diagnostic_patterns.md`).
+`deterministic` was mapping 100% of errors to `template` — the two classes are
+indistinguishable in taskitem title text. Adding data or scaling both fail.
+
+Fix: 2-stage pipeline:
+- Stage 1 binary: `low_entropy {deterministic, template}` vs `needs_routing {specialist, slm, llm, human}`
+- Stage 2 4-way: `specialist / slm / llm / human`
+
+Results: Stage 1 = 100%, Stage 2 = 100%, combined 5-way = 100%.
+
+Script: `specialists/scripts/split_automatability_decompose.py`
+Split TSVs: `specialists/data/generated/automatability_binary_*.tsv`, `automatability_routed_*.tsv`
+Result: `specialists/eval/results/automatability_decompose.json`
+
+Note on `low_entropy` bucket: `deterministic` vs `template` is NOT distinguishable
+from taskitem titles alone — that boundary requires features not in the current
+input schema (how a task is implemented, not what it builds). If the downstream
+system ever needs to distinguish them, the fix is Upstream Feature Starvation
+(Pattern #6): enrich the taskitem schema, not the model.
+
+### 3. Worker_type — combined dataset
+
+Previous train: 300 examples (sprint plan titles only).
+Added: `worker_type_projects_train.tsv` (361 examples, `title | intent` format).
+Combined: 661 train / 115 eval.
+
+Results: 80.8% → 92.2% overall. `implementer` 42.9% → 75.0%.
+`reviewer` and `architect` are now perfect (100%).
+
+Combined TSVs: `specialists/data/generated/worker_type_combined_*.tsv`
+Result: `specialists/eval/results/worker_type_combined_sweep.json`
+
+Remaining `implementer` confusion is Label Boundary Drift (Pattern #10):
+the heuristic keyword classifier labels "implementer" by keyword absence (default),
+producing noisy boundaries on ambiguous titles. The short-text format limits how
+much a model can learn without richer features.
+
+### 4. Diagnostic patterns document
+
+`whetstone_RSA/docs/diagnostic_patterns.md` — committed to whetstone_RSA.
+10 named patterns covering the most common gate failure modes, each with:
+observable signal, mechanism, detection test, specific fix, failure conditions,
+real example from this project.
+
+Written at a level usable by developers and smaller models without deep ML
+background. Motivated by the question: how small a model / how junior a developer
+does it take to apply the insight that solved automatability?
+
+---
+
+## What Is NOT Done — Next Session
+
+### Priority 1: worker_type implementer improvement
+
+Option A (recommended): Generate more data with explicit construction-verb examples.
+The current data labels "implementer" by keyword absence — it's the default. Add
+synthetic examples with clear implementer signals: "Implement X", "Build X", "Write X".
+
+Option B: Human relabeling of the 52 eval examples. If reviewers consistently
+disagree on >30% of errors, apply Label Boundary Drift fix (rewrite label criteria).
+
+Target: get `implementer` above 85% before declaring worker_type deployable.
+
+### Priority 2: verification_type — more data
+
+Currently 83.7% (small_plus), 84.3% (medium). Capacity plateau confirmed.
+Confusion concentrates in `unit`↔`integration`↔`schema` — semantically adjacent.
+These have the projects equivalent: `verification_type_projects_train.tsv` (unused).
+Try the same combined-dataset approach that worked for worker_type.
+
+### Priority 3: prereq_op — blocked, do not pursue until schema change
+
+Root cause: Upstream Feature Starvation (Pattern #6). The taskitem schema does not
+carry discriminative fields at inference time. No amount of data or model work
+fixes this. See `whetstone_RSA/docs/diagnostic_patterns.md` Pattern #6.
+
+Next action is in whetstone_DSL Sprint 004 (pipeline decision audit), not here.
+
+### Not on the list
+
+- automatability: SOLVED, nothing to do
+- scaling any gate: capacity plateau confirmed fleet-wide, not the lever
+
+---
+
+## Run Commands
+
+```bash
+# Full pilot sweep from scratch
+cd /home/bill/Documents/CLionProjects/whetstone_DSL
+bash specialists/scripts/sweep_all_gates.sh --pilot-only 2>&1 | tee specialists/eval/results/sweep_run.log
+
+# Re-run automatability decompose (skip training, models already exist)
+.venv/bin/python3 specialists/scripts/split_automatability_decompose.py --skip-train
+
+# Run worker_type combined sweep only
+.venv/bin/python3 specialists/scripts/capacity_sweep.py \
+  --name worker_type_combined \
+  --train specialists/data/generated/worker_type_combined_train.tsv \
+  --eval  specialists/data/generated/worker_type_combined_eval.tsv \
+  --labels "implementer,reviewer,architect,qa" \
+  --runs-dir /mnt/storage/fabricate_runs \
+  --results-out specialists/eval/results/worker_type_combined_sweep.json \
+  --pilot-only
+```
+
+## Checkpoints
+
+All live at `/mnt/storage/fabricate_runs/`:
+- `whetstone_automatability_stage1_binary/checkpoint.pt` — stage 1 (binary)
+- `whetstone_automatability_stage2_4way/checkpoint.pt` — stage 2 (4-way)
+- `whetstone_worker_type_combined_small_plus/checkpoint.pt` — worker_type 92.2%
+- `whetstone_verification_type_small_plus/checkpoint.pt` — verification_type 83.7%
--- a/specialists/eval/results/automatability_medium.json
+++ b/specialists/eval/results/automatability_medium.json
@@ -0,0 +1,82 @@
+{
+  "overall_accuracy": 78.3,
+  "n_eval": 46,
+  "per_class": {
+    "deterministic": {
+      "acc": 0.0,
+      "n": 10
+    },
+    "template": {
+      "acc": 100.0,
+      "n": 7
+    },
+    "specialist": {
+      "acc": 100.0,
+      "n": 9
+    },
+    "slm": {
+      "acc": 100.0,
+      "n": 7
+    },
+    "llm": {
+      "acc": 100.0,
+      "n": 4
+    },
+    "human": {
+      "acc": 100.0,
+      "n": 9
+    }
+  },
+  "confusion_summary": [
+    {
+      "gt": "deterministic",
+      "pred": "template",
+      "count": 10,
+      "gt_total": 10
+    }
+  ],
+  "threshold_analysis": [
+    {
+      "threshold": 0.5,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 78.3
+    },
+    {
+      "threshold": 0.6,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 78.3
+    },
+    {
+      "threshold": 0.7,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 78.3
+    },
+    {
+      "threshold": 0.8,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 78.3
+    },
+    {
+      "threshold": 0.9,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 78.3
+    }
+  ],
+  "guardrail_threshold": null,
+  "guardrail_abstain_rate": null,
+  "eval_latency_ms_per_example": 5.345,
+  "model_config": {
+    "hidden_dim": 384,
+    "layers": 3,
+    "heads": 6,
+    "n_labels": 6,
+    "seq_len": 96,
+    "vocab_size": 133
+  },
+  "verdict": "decompose_or_relabel"
+}
--- a/specialists/eval/results/automatability_small_plus.json
+++ b/specialists/eval/results/automatability_small_plus.json
@@ -0,0 +1,88 @@
+{
+  "overall_accuracy": 69.6,
+  "n_eval": 46,
+  "per_class": {
+    "deterministic": {
+      "acc": 0.0,
+      "n": 10
+    },
+    "template": {
+      "acc": 42.857142857142854,
+      "n": 7
+    },
+    "specialist": {
+      "acc": 100.0,
+      "n": 9
+    },
+    "slm": {
+      "acc": 100.0,
+      "n": 7
+    },
+    "llm": {
+      "acc": 100.0,
+      "n": 4
+    },
+    "human": {
+      "acc": 100.0,
+      "n": 9
+    }
+  },
+  "confusion_summary": [
+    {
+      "gt": "deterministic",
+      "pred": "llm",
+      "count": 10,
+      "gt_total": 10
+    },
+    {
+      "gt": "template",
+      "pred": "llm",
+      "count": 4,
+      "gt_total": 7
+    }
+  ],
+  "threshold_analysis": [
+    {
+      "threshold": 0.5,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 69.6
+    },
+    {
+      "threshold": 0.6,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 69.6
+    },
+    {
+      "threshold": 0.7,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 69.6
+    },
+    {
+      "threshold": 0.8,
+      "accepted": 46,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 69.6
+    },
+    {
+      "threshold": 0.9,
+      "accepted": 32,
+      "abstain_rate": 0.304,
+      "accepted_accuracy": 100.0
+    }
+  ],
+  "guardrail_threshold": 0.9,
+  "guardrail_abstain_rate": 0.304,
+  "eval_latency_ms_per_example": 5.137,
+  "model_config": {
+    "hidden_dim": 256,
+    "layers": 2,
+    "heads": 4,
+    "n_labels": 6,
+    "seq_len": 96,
+    "vocab_size": 133
+  },
+  "verdict": "decompose_or_relabel"
+}
--- a/specialists/eval/results/verification_type_medium.json
+++ b/specialists/eval/results/verification_type_medium.json
@@ -0,0 +1,126 @@
+{
+  "overall_accuracy": 83.7,
+  "n_eval": 172,
+  "per_class": {
+    "unit": {
+      "acc": 75.75757575757575,
+      "n": 33
+    },
+    "integration": {
+      "acc": 82.35294117647058,
+      "n": 34
+    },
+    "schema": {
+      "acc": 76.47058823529412,
+      "n": 34
+    },
+    "smoke": {
+      "acc": 87.5,
+      "n": 32
+    },
+    "docs": {
+      "acc": 94.87179487179486,
+      "n": 39
+    }
+  },
+  "confusion_summary": [
+    {
+      "gt": "unit",
+      "pred": "schema",
+      "count": 5,
+      "gt_total": 33
+    },
+    {
+      "gt": "integration",
+      "pred": "unit",
+      "count": 5,
+      "gt_total": 34
+    },
+    {
+      "gt": "schema",
+      "pred": "integration",
+      "count": 5,
+      "gt_total": 34
+    },
+    {
+      "gt": "unit",
+      "pred": "integration",
+      "count": 3,
+      "gt_total": 33
+    },
+    {
+      "gt": "schema",
+      "pred": "unit",
+      "count": 3,
+      "gt_total": 34
+    },
+    {
+      "gt": "smoke",
+      "pred": "integration",
+      "count": 2,
+      "gt_total": 32
+    },
+    {
+      "gt": "smoke",
+      "pred": "docs",
+      "count": 2,
+      "gt_total": 32
+    },
+    {
+      "gt": "docs",
+      "pred": "integration",
+      "count": 2,
+      "gt_total": 39
+    },
+    {
+      "gt": "integration",
+      "pred": "smoke",
+      "count": 1,
+      "gt_total": 34
+    }
+  ],
+  "threshold_analysis": [
+    {
+      "threshold": 0.5,
+      "accepted": 172,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 83.7
+    },
+    {
+      "threshold": 0.6,
+      "accepted": 171,
+      "abstain_rate": 0.006,
+      "accepted_accuracy": 83.6
+    },
+    {
+      "threshold": 0.7,
+      "accepted": 171,
+      "abstain_rate": 0.006,
+      "accepted_accuracy": 83.6
+    },
+    {
+      "threshold": 0.8,
+      "accepted": 170,
+      "abstain_rate": 0.012,
+      "accepted_accuracy": 84.1
+    },
+    {
+      "threshold": 0.9,
+      "accepted": 167,
+      "abstain_rate": 0.029,
+      "accepted_accuracy": 85.0
+    }
+  ],
+  "guardrail_threshold": null,
+  "guardrail_abstain_rate": null,
+  "eval_latency_ms_per_example": 1.243,
+  "model_config": {
+    "hidden_dim": 384,
+    "layers": 3,
+    "heads": 6,
+    "n_labels": 5,
+    "seq_len": 96,
+    "vocab_size": 1279
+  },
+  "verdict": "scale_more"
+}
--- a/specialists/eval/results/verification_type_small_plus.json
+++ b/specialists/eval/results/verification_type_small_plus.json
@@ -0,0 +1,132 @@
+{
+  "overall_accuracy": 83.7,
+  "n_eval": 172,
+  "per_class": {
+    "unit": {
+      "acc": 75.75757575757575,
+      "n": 33
+    },
+    "integration": {
+      "acc": 76.47058823529412,
+      "n": 34
+    },
+    "schema": {
+      "acc": 76.47058823529412,
+      "n": 34
+    },
+    "smoke": {
+      "acc": 93.75,
+      "n": 32
+    },
+    "docs": {
+      "acc": 94.87179487179486,
+      "n": 39
+    }
+  },
+  "confusion_summary": [
+    {
+      "gt": "unit",
+      "pred": "schema",
+      "count": 5,
+      "gt_total": 33
+    },
+    {
+      "gt": "integration",
+      "pred": "unit",
+      "count": 5,
+      "gt_total": 34
+    },
+    {
+      "gt": "schema",
+      "pred": "unit",
+      "count": 4,
+      "gt_total": 34
+    },
+    {
+      "gt": "schema",
+      "pred": "integration",
+      "count": 4,
+      "gt_total": 34
+    },
+    {
+      "gt": "unit",
+      "pred": "integration",
+      "count": 3,
+      "gt_total": 33
+    },
+    {
+      "gt": "integration",
+      "pred": "schema",
+      "count": 2,
+      "gt_total": 34
+    },
+    {
+      "gt": "docs",
+      "pred": "integration",
+      "count": 2,
+      "gt_total": 39
+    },
+    {
+      "gt": "integration",
+      "pred": "smoke",
+      "count": 1,
+      "gt_total": 34
+    },
+    {
+      "gt": "smoke",
+      "pred": "integration",
+      "count": 1,
+      "gt_total": 32
+    },
+    {
+      "gt": "smoke",
+      "pred": "docs",
+      "count": 1,
+      "gt_total": 32
+    }
+  ],
+  "threshold_analysis": [
+    {
+      "threshold": 0.5,
+      "accepted": 172,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 83.7
+    },
+    {
+      "threshold": 0.6,
+      "accepted": 171,
+      "abstain_rate": 0.006,
+      "accepted_accuracy": 83.6
+    },
+    {
+      "threshold": 0.7,
+      "accepted": 170,
+      "abstain_rate": 0.012,
+      "accepted_accuracy": 83.5
+    },
+    {
+      "threshold": 0.8,
+      "accepted": 164,
+      "abstain_rate": 0.047,
+      "accepted_accuracy": 84.8
+    },
+    {
+      "threshold": 0.9,
+      "accepted": 160,
+      "abstain_rate": 0.07,
+      "accepted_accuracy": 85.0
+    }
+  ],
+  "guardrail_threshold": null,
+  "guardrail_abstain_rate": null,
+  "eval_latency_ms_per_example": 4.976,
+  "model_config": {
+    "hidden_dim": 256,
+    "layers": 2,
+    "heads": 4,
+    "n_labels": 5,
+    "seq_len": 96,
+    "vocab_size": 1279
+  },
+  "verdict": "scale_more"
+}
--- a/specialists/eval/results/worker_type_combined_small_plus.json
+++ b/specialists/eval/results/worker_type_combined_small_plus.json
@@ -0,0 +1,98 @@
+{
+  "overall_accuracy": 91.3,
+  "n_eval": 115,
+  "per_class": {
+    "implementer": {
+      "acc": 75.0,
+      "n": 32
+    },
+    "reviewer": {
+      "acc": 100.0,
+      "n": 29
+    },
+    "architect": {
+      "acc": 100.0,
+      "n": 29
+    },
+    "qa": {
+      "acc": 92.0,
+      "n": 25
+    }
+  },
+  "confusion_summary": [
+    {
+      "gt": "implementer",
+      "pred": "reviewer",
+      "count": 6,
+      "gt_total": 32
+    },
+    {
+      "gt": "implementer",
+      "pred": "architect",
+      "count": 1,
+      "gt_total": 32
+    },
+    {
+      "gt": "implementer",
+      "pred": "qa",
+      "count": 1,
+      "gt_total": 32
+    },
+    {
+      "gt": "qa",
+      "pred": "implementer",
+      "count": 1,
+      "gt_total": 25
+    },
+    {
+      "gt": "qa",
+      "pred": "architect",
+      "count": 1,
+      "gt_total": 25
+    }
+  ],
+  "threshold_analysis": [
+    {
+      "threshold": 0.5,
+      "accepted": 115,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 91.3
+    },
+    {
+      "threshold": 0.6,
+      "accepted": 113,
+      "abstain_rate": 0.017,
+      "accepted_accuracy": 92.0
+    },
+    {
+      "threshold": 0.7,
+      "accepted": 111,
+      "abstain_rate": 0.035,
+      "accepted_accuracy": 91.9
+    },
+    {
+      "threshold": 0.8,
+      "accepted": 109,
+      "abstain_rate": 0.052,
+      "accepted_accuracy": 91.7
+    },
+    {
+      "threshold": 0.9,
+      "accepted": 109,
+      "abstain_rate": 0.052,
+      "accepted_accuracy": 91.7
+    }
+  ],
+  "guardrail_threshold": null,
+  "guardrail_abstain_rate": null,
+  "eval_latency_ms_per_example": 1.993,
+  "model_config": {
+    "hidden_dim": 256,
+    "layers": 2,
+    "heads": 4,
+    "n_labels": 4,
+    "seq_len": 96,
+    "vocab_size": 1184
+  },
+  "verdict": "scale_more"
+}
--- a/specialists/eval/results/worker_type_small_plus.json
+++ b/specialists/eval/results/worker_type_small_plus.json
@@ -0,0 +1,86 @@
+{
+  "overall_accuracy": 80.8,
+  "n_eval": 52,
+  "per_class": {
+    "implementer": {
+      "acc": 42.857142857142854,
+      "n": 14
+    },
+    "reviewer": {
+      "acc": 100.0,
+      "n": 8
+    },
+    "architect": {
+      "acc": 100.0,
+      "n": 16
+    },
+    "qa": {
+      "acc": 85.71428571428571,
+      "n": 14
+    }
+  },
+  "confusion_summary": [
+    {
+      "gt": "implementer",
+      "pred": "reviewer",
+      "count": 5,
+      "gt_total": 14
+    },
+    {
+      "gt": "implementer",
+      "pred": "architect",
+      "count": 3,
+      "gt_total": 14
+    },
+    {
+      "gt": "qa",
+      "pred": "architect",
+      "count": 2,
+      "gt_total": 14
+    }
+  ],
+  "threshold_analysis": [
+    {
+      "threshold": 0.5,
+      "accepted": 52,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 80.8
+    },
+    {
+      "threshold": 0.6,
+      "accepted": 52,
+      "abstain_rate": 0.0,
+      "accepted_accuracy": 80.8
+    },
+    {
+      "threshold": 0.7,
+      "accepted": 51,
+      "abstain_rate": 0.019,
+      "accepted_accuracy": 82.4
+    },
+    {
+      "threshold": 0.8,
+      "accepted": 51,
+      "abstain_rate": 0.019,
+      "accepted_accuracy": 82.4
+    },
+    {
+      "threshold": 0.9,
+      "accepted": 48,
+      "abstain_rate": 0.077,
+      "accepted_accuracy": 85.4
+    }
+  ],
+  "guardrail_threshold": null,
+  "guardrail_abstain_rate": null,
+  "eval_latency_ms_per_example": 4.296,
+  "model_config": {
+    "hidden_dim": 256,
+    "layers": 2,
+    "heads": 4,
+    "n_labels": 4,
+    "seq_len": 96,
+    "vocab_size": 550
+  },
+  "verdict": "scale_more"
+}
--- a/specialists/runs/prereq_approval_history.json
+++ b/specialists/runs/prereq_approval_history.json
@@ -0,0 +1,28 @@
+{
+  "history": [
+    {
+      "step": 1000,
+      "overall": 88.89,
+      "loss": 1.3032,
+      "elapsed_s": 49.1
+    },
+    {
+      "step": 2000,
+      "overall": 88.89,
+      "loss": 1.1894,
+      "elapsed_s": 44.9
+    },
+    {
+      "step": 3000,
+      "overall": 86.11,
+      "loss": 1.0969,
+      "elapsed_s": 44.3
+    },
+    {
+      "step": 4000,
+      "overall": 86.11,
+      "loss": 1.0206,
+      "elapsed_s": 44.3
+    }
+  ]
+}
--- a/specialists/runs/prereq_review_history.json
+++ b/specialists/runs/prereq_review_history.json
@@ -0,0 +1,28 @@
+{
+  "history": [
+    {
+      "step": 1000,
+      "overall": 88.89,
+      "loss": 1.4074,
+      "elapsed_s": 55.7
+    },
+    {
+      "step": 2000,
+      "overall": 86.11,
+      "loss": 1.4714,
+      "elapsed_s": 45.2
+    },
+    {
+      "step": 3000,
+      "overall": 86.11,
+      "loss": 1.5002,
+      "elapsed_s": 44.9
+    },
+    {
+      "step": 4000,
+      "overall": 86.11,
+      "loss": 1.5354,
+      "elapsed_s": 44.8
+    }
+  ]
+}