specialists: switch training to PyTorch, add capacity sweep infrastructure
- train_specialist_pt.py: pure PyTorch trainer replacing Fabricate binary. No SQLite/WAL — saves checkpoint.pt + history.json only. Fabricate was causing D-state IO blocking on the HDD due to continuous WAL writes. - eval_specialist_pt.py: PyTorch eval with confusion matrix, threshold analysis, guardrail assessment, and deployment verdict. - capacity_sweep.py: updated to call PyTorch scripts; TIERS now include heads field (4/6/8) since PyTorch supports multi-head unlike Fabricate. - prereq_op_binary_split.py: evaluate_combined rewritten in PyTorch. - Performance fix in train_specialist_pt.py: load training data onto GPU once and sample via torch.randint instead of DataLoader (eliminates Python/CPU overhead on tiny datasets). Fix applied, not yet timed. Next session: time the fix, then run sweep_all_gates.sh --pilot-only. See HANDOFF-2026-03-31.md for full context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
56
specialists/scripts/train_prereq_op.sh
Executable file
56
specialists/scripts/train_prereq_op.sh
Executable file
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
# Train the prereq_op_selector 4-way specialist.
|
||||
#
|
||||
# Classes:
|
||||
# 0 = standard [validate-intake, resolve-dependencies]
|
||||
# 1 = needs_review [validate-intake, architect-review]
|
||||
# 2 = needs_approval [validate-intake, manual-approval]
|
||||
# 3 = full_gates [validate-intake, architect-review, manual-approval]
|
||||
#
|
||||
# Usage: bash specialists/scripts/train_prereq_op.sh
|
||||
# (run from whetstone_DSL root)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
FABRICATE_ROOT="/home/bill/Documents/WhetstoneAI_Fabricate"
|
||||
|
||||
TRAIN_DATA="$ROOT/specialists/data/generated/prereq_op_train.tsv"
|
||||
EVAL_DATA="$ROOT/specialists/data/generated/prereq_op_eval.tsv"
|
||||
OUT_DIR="/mnt/storage/fabricate_runs/whetstone_prereq_op"
|
||||
HISTORY="$ROOT/specialists/runs/prereq_op_history.json"
|
||||
RUN_LOG="$ROOT/specialists/runs/prereq_op_training.log"
|
||||
|
||||
mkdir -p "$ROOT/specialists/runs" "$OUT_DIR"
|
||||
|
||||
if [[ ! -f "$TRAIN_DATA" ]]; then
|
||||
echo "Missing training data. Run:" >&2
|
||||
echo " python3 specialists/scripts/gen_prereq_op_data.py" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Training prereq_op_selector specialist..."
|
||||
echo " train: $TRAIN_DATA"
|
||||
echo " eval: $EVAL_DATA"
|
||||
echo " out: $OUT_DIR"
|
||||
echo " log: $RUN_LOG"
|
||||
echo ""
|
||||
|
||||
cd "$FABRICATE_ROOT"
|
||||
|
||||
.venv/bin/python3 run_grokking_until.py \
|
||||
--dataset "$TRAIN_DATA" \
|
||||
--eval-dataset "$EVAL_DATA" \
|
||||
--out-dir "$OUT_DIR" \
|
||||
--labels "standard,needs_review,needs_approval,full_gates" \
|
||||
--lr 0.001 \
|
||||
--weight-decay 0.01 \
|
||||
--max-steps 5000 \
|
||||
--grok-loss-threshold 0.10 \
|
||||
--grok-acc-jump 15.0 \
|
||||
--stop-after-grokking-blocks 3 \
|
||||
--history-out "$HISTORY" \
|
||||
2>&1 | tee "$RUN_LOG"
|
||||
|
||||
echo ""
|
||||
echo "Done. Checkpoint at: $OUT_DIR/checkpoint.bin"
|
||||
Reference in New Issue
Block a user