87 lines
3.9 KiB
Markdown
87 lines
3.9 KiB
Markdown
|
# Specialist Fleet Handoff — 2026-04-18
|
|||
|
|
|
|||
|
|
## Status at Session End
|
|||
|
|
|
|||
|
|
Sweep was killed mid-run to conserve solar battery. State:
|
|||
|
|
|
|||
|
|
| Gate | Tier | Status | Accuracy |
|
|||
|
|
|------|------|--------|----------|
|
|||
|
|
| verification_type | small_plus | COMPLETE | 81.4% — verdict: `scale_more` |
|
|||
|
|
| verification_type | medium | KILLED | Partial checkpoint, do not trust |
|
|||
|
|
| worker_type | — | NOT STARTED | — |
|
|||
|
|
| prereq_op | — | NOT STARTED | — |
|
|||
|
|
| automatability | — | NOT STARTED | — |
|
|||
|
|
|
|||
|
|
## First Task Next Session
|
|||
|
|
|
|||
|
|
From the whetstone_DSL root, run the full pilot sweep fresh:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
cd /home/bill/Documents/CLionProjects/whetstone_DSL
|
|||
|
|
bash specialists/scripts/sweep_all_gates.sh --pilot-only 2>&1 | tee specialists/eval/results/sweep_run.log
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The `--reset` flag inside `capacity_sweep.py` ensures the broken `verification_type` medium
|
|||
|
|
checkpoint from this session doesn't interfere. All 4 gates run from scratch.
|
|||
|
|
|
|||
|
|
Expected runtime: ~25–35 minutes total (small_plus + medium × 4 gates, 3000 steps each).
|
|||
|
|
|
|||
|
|
Results land in:
|
|||
|
|
- `specialists/eval/results/<gate>_sweep.json` — per-gate sweep table
|
|||
|
|
- `specialists/eval/results/summary.json` — combined best-tier per gate
|
|||
|
|
|
|||
|
|
## Fixes Applied This Session (All Saved to Disk)
|
|||
|
|
|
|||
|
|
1. **Venv path** — `sweep_all_gates.sh` now uses whetstone DSL venv:
|
|||
|
|
`PYTHON="/home/bill/Documents/CLionProjects/whetstone_DSL/.venv/bin/python3"`
|
|||
|
|
|
|||
|
|
2. **Venv path** — `capacity_sweep.py` + `prereq_op_binary_split.py` now use `sys.executable`
|
|||
|
|
instead of the deleted `WhetstoneAI_Fabricate/.venv` path.
|
|||
|
|
|
|||
|
|
3. **Eval JSON parse** — `eval_specialist_pt.py` now prints single-line JSON as last line:
|
|||
|
|
`print(json.dumps(result)) # single line — capacity_sweep.py reads last line as JSON`
|
|||
|
|
|
|||
|
|
4. **Large tier removed** — `sweep_all_gates.sh` defaults to `TIERS="small_plus,medium"`.
|
|||
|
|
Large (512/4/8, 13M params) takes ~247s/block — impractical for a pilot.
|
|||
|
|
|
|||
|
|
5. **Nested tensor slowpath** — `train_specialist_pt.py` has `enable_nested_tensor=False`
|
|||
|
|
on TransformerEncoder. This eliminated ~40% overhead AND improved accuracy.
|
|||
|
|
|
|||
|
|
6. **Inductor compile** — `torch.compile(mode="reduce-overhead")` enabled with TF32.
|
|||
|
|
`python3.12-dev` is installed. Actual speedup: ~62s → ~45s/block (~25%).
|
|||
|
|
Note: `torch.randint` dynamic sampling prevents full CUDA graph replay; partial
|
|||
|
|
re-recording still happens. Pre-generated static batch pool would fix this but
|
|||
|
|
wasn't implemented (one-time training cost doesn't justify complexity).
|
|||
|
|
|
|||
|
|
## Baseline Accuracy (from prior session, before this session's fixes)
|
|||
|
|
|
|||
|
|
| Gate | Prior Best | Labels |
|
|||
|
|
|------|-----------|--------|
|
|||
|
|
| worker_type | 88.5% | implementer, reviewer, architect, qa |
|
|||
|
|
| verification_type | 84.9% | unit, integration, schema, smoke, docs |
|
|||
|
|
| prereq_op | 69.4% | standard, needs_review, needs_approval, full_gates |
|
|||
|
|
| automatability | 69.6% | deterministic, template, specialist, slm, llm, human |
|
|||
|
|
|
|||
|
|
Target: 98-99% (deploy_direct verdict) or 95%+ at ≤0.30 abstain rate (deploy_with_guardrails).
|
|||
|
|
|
|||
|
|
## After the Sweep
|
|||
|
|
|
|||
|
|
Check `specialists/eval/results/summary.json`. For any gate with verdict `scale_more`:
|
|||
|
|
- Try full 15k steps: edit capacity_sweep.py `MAX_STEPS` or run the gate individually
|
|||
|
|
- Or generate more training data — this is the higher-leverage move for low-accuracy gates
|
|||
|
|
|
|||
|
|
Gates most likely to need more data: `prereq_op` (4-way, hard to separate) and `automatability`
|
|||
|
|
(6-way, some classes are similar). Consider a data gen script that uses an LLM to generate
|
|||
|
|
examples per label given a description.
|
|||
|
|
|
|||
|
|
## Model Architecture (for reference)
|
|||
|
|
|
|||
|
|
Small_plus tier: hidden=256, layers=2, heads=4 (~1.2M params)
|
|||
|
|
Medium tier: hidden=384, layers=3, heads=6 (~3.0M params)
|
|||
|
|
|
|||
|
|
Encoder-only transformer, mean-pool over non-padding positions, linear classification head.
|
|||
|
|
Tokenizer: whitespace split, lowercase. Seq len: 96 tokens.
|
|||
|
|
Checkpoint format: `checkpoint.pt` — contains model, optimizer, vocab, config, labels, step.
|
|||
|
|
|
|||
|
|
Checkpoints live in: `/mnt/storage/fabricate_runs/whetstone_<gate>_<tier>/checkpoint.pt`
|