Files

Bill 15125dce25 Specialist fleet: pilot sweep results + automatability decomposition

Pilot sweep (all 4 gates, small_plus + medium):
  verification_type  84.3%  (medium, no_gain_stop_scaling)
  worker_type        80.8%  (small_plus, scale_more)
  automatability     78.3%  (medium, decompose_or_relabel)
  prereq_op          72.2%  (small_plus, decompose_or_relabel)

Automatability: 2-stage decomposition resolves Directional Collapse.
  Stage 1 binary (low_entropy vs needs_routing): 100%
  Stage 2 4-way (specialist/slm/llm/human): 100%
  Combined 5-way pipeline: 100%
  Script: specialists/scripts/split_automatability_decompose.py

Worker_type: combined dataset (sprint plans + projects) improves
  implementer class from 42.9% → 75.0%, overall 80.8% → 92.2%.
  Combined train/eval TSVs added.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-19 13:44:38 -06:00

3.9 KiB

Raw Blame History

Specialist Fleet Handoff — 2026-04-18

Status at Session End

Sweep was killed mid-run to conserve solar battery. State:

Gate	Tier	Status	Accuracy
verification_type	small_plus	COMPLETE	81.4% — verdict: `scale_more`
verification_type	medium	KILLED	Partial checkpoint, do not trust
worker_type	—	NOT STARTED	—
prereq_op	—	NOT STARTED	—
automatability	—	NOT STARTED	—

First Task Next Session

From the whetstone_DSL root, run the full pilot sweep fresh:

cd /home/bill/Documents/CLionProjects/whetstone_DSL
bash specialists/scripts/sweep_all_gates.sh --pilot-only 2>&1 | tee specialists/eval/results/sweep_run.log

The --reset flag inside capacity_sweep.py ensures the broken verification_type medium checkpoint from this session doesn't interfere. All 4 gates run from scratch.

Expected runtime: ~25–35 minutes total (small_plus + medium × 4 gates, 3000 steps each).

Results land in:

specialists/eval/results/<gate>_sweep.json — per-gate sweep table
specialists/eval/results/summary.json — combined best-tier per gate

Fixes Applied This Session (All Saved to Disk)

Venv path — sweep_all_gates.sh now uses whetstone DSL venv: PYTHON="/home/bill/Documents/CLionProjects/whetstone_DSL/.venv/bin/python3"
Venv path — capacity_sweep.py + prereq_op_binary_split.py now use sys.executable instead of the deleted WhetstoneAI_Fabricate/.venv path.
Eval JSON parse — eval_specialist_pt.py now prints single-line JSON as last line: print(json.dumps(result)) # single line — capacity_sweep.py reads last line as JSON
Large tier removed — sweep_all_gates.sh defaults to TIERS="small_plus,medium". Large (512/4/8, 13M params) takes ~247s/block — impractical for a pilot.
Nested tensor slowpath — train_specialist_pt.py has enable_nested_tensor=False on TransformerEncoder. This eliminated ~40% overhead AND improved accuracy.
Inductor compile — torch.compile(mode="reduce-overhead") enabled with TF32. python3.12-dev is installed. Actual speedup: ~62s → ~45s/block (~25%). Note: torch.randint dynamic sampling prevents full CUDA graph replay; partial re-recording still happens. Pre-generated static batch pool would fix this but wasn't implemented (one-time training cost doesn't justify complexity).

Baseline Accuracy (from prior session, before this session's fixes)

Gate	Prior Best	Labels
worker_type	88.5%	implementer, reviewer, architect, qa
verification_type	84.9%	unit, integration, schema, smoke, docs
prereq_op	69.4%	standard, needs_review, needs_approval, full_gates
automatability	69.6%	deterministic, template, specialist, slm, llm, human

Target: 98-99% (deploy_direct verdict) or 95%+ at ≤0.30 abstain rate (deploy_with_guardrails).

After the Sweep

Check specialists/eval/results/summary.json. For any gate with verdict scale_more:

Try full 15k steps: edit capacity_sweep.py MAX_STEPS or run the gate individually
Or generate more training data — this is the higher-leverage move for low-accuracy gates

Gates most likely to need more data: prereq_op (4-way, hard to separate) and automatability (6-way, some classes are similar). Consider a data gen script that uses an LLM to generate examples per label given a description.

Model Architecture (for reference)

Small_plus tier: hidden=256, layers=2, heads=4 (~1.2M params) Medium tier: hidden=384, layers=3, heads=6 (~3.0M params)

Encoder-only transformer, mean-pool over non-padding positions, linear classification head. Tokenizer: whitespace split, lowercase. Seq len: 96 tokens. Checkpoint format: checkpoint.pt — contains model, optimizer, vocab, config, labels, step.

Checkpoints live in: /mnt/storage/fabricate_runs/whetstone_<gate>_<tier>/checkpoint.pt

3.9 KiB Raw Blame History Unescape Escape