3fc9deac5b
- train_specialist_pt.py: pure PyTorch trainer replacing Fabricate binary. No SQLite/WAL — saves checkpoint.pt + history.json only. Fabricate was causing D-state IO blocking on the HDD due to continuous WAL writes. - eval_specialist_pt.py: PyTorch eval with confusion matrix, threshold analysis, guardrail assessment, and deployment verdict. - capacity_sweep.py: updated to call PyTorch scripts; TIERS now include heads field (4/6/8) since PyTorch supports multi-head unlike Fabricate. - prereq_op_binary_split.py: evaluate_combined rewritten in PyTorch. - Performance fix in train_specialist_pt.py: load training data onto GPU once and sample via torch.randint instead of DataLoader (eliminates Python/CPU overhead on tiny datasets). Fix applied, not yet timed. Next session: time the fix, then run sweep_all_gates.sh --pilot-only. See HANDOFF-2026-03-31.md for full context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
65 lines
2.2 KiB
Bash
Executable File
65 lines
2.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Train the automatability_strategy 6-way specialist.
|
|
#
|
|
# Classes (matching AutomatabilityAnnotation.strategy in Annotation.h):
|
|
# 0 = deterministic — pure formula/rule, zero parameters
|
|
# 1 = template — fixed-pattern code generation
|
|
# 2 = specialist — bounded probabilistic (~800KB model)
|
|
# 3 = slm — complex transformation, local model (~1-7GB)
|
|
# 4 = llm — full generative reasoning (14B+)
|
|
# 5 = human — requires manual judgment
|
|
#
|
|
# Input format: "name=ClassName"
|
|
#
|
|
# NOTE: 6-way classification with sparse tiers (human=5, specialist=8 raw
|
|
# examples). Grokking may take longer — max-steps set to 10000.
|
|
# If it doesn't grok by 10k, lower LR to 0.0005 and retry.
|
|
#
|
|
# Usage: bash specialists/scripts/train_automatability.sh
|
|
# (run from whetstone_DSL root)
|
|
|
|
set -euo pipefail
|
|
|
|
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
|
FABRICATE_ROOT="/home/bill/Documents/WhetstoneAI_Fabricate"
|
|
|
|
TRAIN_DATA="$ROOT/specialists/data/generated/automatability_train.tsv"
|
|
EVAL_DATA="$ROOT/specialists/data/generated/automatability_eval.tsv"
|
|
OUT_DIR="/mnt/storage/fabricate_runs/whetstone_automatability"
|
|
HISTORY="$ROOT/specialists/runs/automatability_history.json"
|
|
RUN_LOG="$ROOT/specialists/runs/automatability_training.log"
|
|
|
|
mkdir -p "$ROOT/specialists/runs" "$OUT_DIR"
|
|
|
|
if [[ ! -f "$TRAIN_DATA" ]]; then
|
|
echo "Missing training data. Run:" >&2
|
|
echo " python3 specialists/scripts/gen_automatability_data.py" >&2
|
|
exit 1
|
|
fi
|
|
|
|
echo "Training automatability_strategy specialist..."
|
|
echo " train: $TRAIN_DATA"
|
|
echo " eval: $EVAL_DATA"
|
|
echo " out: $OUT_DIR"
|
|
echo " log: $RUN_LOG"
|
|
echo ""
|
|
|
|
cd "$FABRICATE_ROOT"
|
|
|
|
.venv/bin/python3 run_grokking_until.py \
|
|
--dataset "$TRAIN_DATA" \
|
|
--eval-dataset "$EVAL_DATA" \
|
|
--out-dir "$OUT_DIR" \
|
|
--labels "deterministic,template,specialist,slm,llm,human" \
|
|
--lr 0.001 \
|
|
--weight-decay 0.01 \
|
|
--max-steps 10000 \
|
|
--grok-loss-threshold 0.10 \
|
|
--grok-acc-jump 15.0 \
|
|
--stop-after-grokking-blocks 3 \
|
|
--history-out "$HISTORY" \
|
|
2>&1 | tee "$RUN_LOG"
|
|
|
|
echo ""
|
|
echo "Done. Checkpoint at: $OUT_DIR/checkpoint.bin"
|