whetstone_DSL/specialists/HANDOFF-2026-03-31.md

# Specialist Fleet Handoff — 2026-03-31

## Where We Are

Switched training infrastructure from Fabricate (C++ binary + SQLite) to **pure PyTorch**.
Fabricate was causing uninterruptible D-state IO blocking because `/mnt/storage` is a spinning
HDD and Fabricate writes continuous SQLite WAL data (~100MB/min) even with morphisms disabled.

## What Was Built This Session

### New Files
- `specialists/scripts/train_specialist_pt.py` — PyTorch training script (replaces `run_grokking_until.py`)
- `specialists/eval/eval_specialist_pt.py` — PyTorch eval script (replaces `eval_specialist.py`)

### Updated Files
- `specialists/scripts/capacity_sweep.py` — now calls `train_specialist_pt.py` + `eval_specialist_pt.py`
- `specialists/scripts/prereq_op_binary_split.py` — same, `evaluate_combined` rewritten in PyTorch
- Both scripts now use `checkpoint.pt` instead of `checkpoint.bin` + `fabricate.db`
- TIERS updated: `heads` field added (4/6/8 for small+/medium/large) — PyTorch supports multi-head unlike Fabricate

### PyTorch training script details
- No SQLite, no WAL. Saves `checkpoint.pt` + `history.json` only.
- History JSON format compatible with `capacity_sweep.py`: `{"history": [{"step": N, "overall": float, "loss": float}]}`
- Same CLI interface as `run_grokking_until.py` for drop-in use by sweep scripts
- Grokking detection + early stop implemented (same logic: loss < threshold OR acc jump ≥ threshold)
- Checkpoint includes: model weights, optimizer state, vocab, config, labels, step count

## Performance Issue — PARTIALLY FIXED

**Root cause:** DataLoader restarts ~833×/1000 steps when dataset has 300 examples + batch_size=256.
Each restart has Python overhead → 38s per 1000-step block (expected ~4s).

**Fix applied (untimed):** Load all training data onto GPU once as a tensor, sample via
`torch.randint` each step instead of DataLoader. This eliminates all Python/CPU overhead per step.

```python
all_x = torch.stack([row[0] for row in train_ds]).to(device)
all_y = torch.tensor([row[1] for row in train_ds], ..., device=device)
# per step:
idx = torch.randint(0, n_train, (batch_size,), device=device)
x, y = all_x[idx], all_y[idx]
```

**This fix is in the code but was NOT timed before session ended.**

## First Task Next Session

1. Time the fixed training script:
```bash
cd /home/bill/Documents/CLionProjects/whetstone_DSL
time /home/bill/Documents/WhetstoneAI_Fabricate/.venv/bin/python3 \
  specialists/scripts/train_specialist_pt.py \
  --dataset specialists/data/generated/worker_type_train.tsv \
  --eval-dataset specialists/data/generated/worker_type_eval.tsv \
  --labels "implementer,reviewer,architect,qa" \
  --out-dir /tmp/pt_timing_test \
  --hidden-dim 256 --layers 2 --heads 4 \
  --lr 0.001 --weight-decay 0.01 \
  --max-steps 3000 --batch-size 256 \
  --stop-after-grokking-blocks 99 \
  --reset
```

Expected: ~4-8 seconds per 1000-step block (was 38s before fix), so ~12-25s for 3000 steps.
If correct, pilot sweep (all 4 gates × 3 tiers) should complete in ~5-10 minutes.

2. If timing is good, run:
```bash
bash specialists/scripts/sweep_all_gates.sh --pilot-only
```

3. Analyze pilot results in `specialists/eval/results/summary.json`, decide which gates
   need full training vs already good enough.

## Current Baseline Accuracies (small Fabricate models, unchanged)
- `worker_type`:        88.5%  (implementer residual-category issue, 64% implementer acc)
- `verification_type`:  84.9%  (schema→integration confusion, capacity limited)
- `prereq_op`:          69.4%  (standard residual-category, over-fires approval gate)
- `automatability`:     69.6%  (deterministic 0% — pure residual-category collapse)

Target: 98-99% for all gates. Larger PyTorch models (no multi-head constraint) should help.

## Key Architecture Notes
- Fabricate constraint: `gpu_transformer` engine only supports `heads=1`
- PyTorch: any heads where `hidden_dim % heads == 0` — we now use 4/6/8
- Run dirs are still on `/mnt/storage/fabricate_runs/` — this is fine, `.pt` files are
  tiny (few MB) and written infrequently (once per 1000 steps), not continuously
- The old `whetstone_prereq_op_large` run dir on the HDD may still have a D-state
  Fabricate process associated. Check `pgrep fabricate` at session start; kill if present.

## Venv
All scripts use: `/home/bill/Documents/WhetstoneAI_Fabricate/.venv/bin/python3`
PyTorch 2.10+cu128 is installed there. CUDA is available (RTX 3060).