# Whetstone Specialist Fleet **What this is:** A system of ~800KB transformer specialists that replace bounded-output LLM decisions inside the Whetstone MCP pipeline. Each specialist handles exactly one classification or scoring decision, runs in under a millisecond, and can be held resident in GPU VRAM alongside thousands of peers. This is not a LoRA. It is not a fine-tuned 14B model. It is a different architecture for a different problem. --- ## Origin: WhetstoneAI Fabricate Experiments This project grew out of the WhetstoneAI_Fabricate experiments at `/home/bill/Documents/WhetstoneAI_Fabricate/`. Those experiments were training tiny transformer models (~213K parameters) to solve structured reasoning problems — math ordering, logic gates, state machines, modular arithmetic, DAG traversal. Each model was a binary classifier (true/false) over reasoning chains. The key finding was the **Routed-Specialist Architecture (RSA)**: - A small router model (5-way classifier) reads a raw question and picks which domain expert to consult - The chosen specialist (binary classifier) answers the question - End-to-end composed accuracy: 96.3% in-distribution, 76.7% OOD after hardening - Total model size: router (~800KB) + 5 specialists (~800KB each) = ~5MB - Inference: microseconds per query on GPU The critical realization: **the Whetstone taskitem pipeline is already doing this same structure, but with a 14B LLM handling all the decisions.** Every tool call involves the LLM choosing from bounded option sets — verification type, prerequisite ops, confidence tier, worker role, quality score tier. These are exactly the classifier-shaped problems that the Fabricate architecture solves at 800KB. The WhetstoneAI_Fabricate binary and training harness are reused here unchanged. No new training infrastructure was written. --- ## The AST Is Already Rich — Specialists Predict Its Fields Whetstone is not a C++ code generator. C++, Python, Rust, and all other output languages are *projections* of an AST-as-source-of-truth. The AST uses language-neutral `conceptType` names (`Function`, `ClassDeclaration`, `IfStatement`) that map across 19+ languages through a grammar layer. This means specialists operate at the AST level and cover all languages simultaneously — there are no per-language variants for decisions that live in the AST. More importantly: the Whetstone AST already carries annotation metadata that most ASTs never have. The annotation schema in `editor/src/ast/Annotation.h` (60+ types) includes: ``` RiskAnnotation.dependentCount int how many callers/consumers downstream CallHierarchyInfo.callerIds list which functions call this one BlastRadiusReport.riskLevel enum none/low/medium/high (blast radius) IntentAnnotation.category enum validation/transformation/io/... ComplexityAnnotation.cognitiveComplexity int 1-10 scale AutomatabilityAnnotation.strategy enum deterministic|template|specialist|slm|llm|human AutomatabilityAnnotation.confidence float 0.0–1.0 ``` NOTE: Current `Annotation.h` is missing `specialist` in this enum. Add it before training the `automatability_strategy` specialist. The 6 tiers in order: formula → template → specialist (~800KB bounded classifier) → slm → llm → human. **`AutomatabilityAnnotation.strategy` is the Tacit Codex entropy classification built directly into the AST schema.** It encodes whether each node's decisions should be handled by a formula, template instantiation, a tiny specialist, a small LLM, a large LLM, or a human. Training the `automatability_strategy` specialist means populating this field automatically across the entire codebase — every node gets its entropy tier assigned without an LLM call. `RiskAnnotation.dependentCount` is the "how many things downstream depend on this?" metric the system already tracks. Combined with blast radius analysis (`BlastRadiusAnalyzer.h`) and call hierarchy (`ContextAPI.h`), the AST already measures change risk. These are input features for the `change_risk_tier` specialist. ## The Core Premise ### What an MCP tool call currently does When you call `whetstone_generate_taskitems`, the LLM receives context and makes roughly these decisions: ``` How many milestones? → integer 1-6 Verification type? → unit | integration | schema | smoke | docs Prereq ops needed? → combo of 4 possible ops Worker type? → implementer | reviewer | architect | qa Confidence level? → integer 0-100 (computed from formula) Should escalate? → bool ...and then the generative parts: Task title text? → open text Acceptance criteria text? → open text ``` The first group are **bounded decisions** — the output is always one choice from a small fixed set. These are classifier problems. The second group are **generative decisions** — the output is open text. These still require the LLM. ### What the specialist fleet does instead Every bounded decision becomes a specialist call. All specialists fire in parallel before or instead of the LLM call. The LLM, if called at all, only handles the open-text portions — with the bounded decisions already pre-filled. ``` [specialist: verification_type] ──► unit/integration/schema/smoke/docs ~0.5ms [specialist: prereq_op_selector] ──► [validate-intake, resolve-deps] ~0.5ms [specialist: confidence_tier] ──► high/medium/low ~0.5ms [specialist: worker_type] ──► implementer ~0.5ms ↓ all parallel, all in <2ms total [LLM: generate task title] ──► "ResourceLocks schema extension" ~3s [LLM: write acceptance criteria] ──► "..." ~2s ``` ### Why this scales An RTX 3060 has 12GB VRAM. One quantized 14B LLM takes ~8GB. One specialist takes ~800KB. The same card that holds one LLM can hold **~14,000 specialists resident simultaneously**. 50 bounded decision types × 20 domain variants each = 1,000 specialists = 800MB VRAM. All 1,000 can run in a single batched forward pass. --- ## The Philosophical Anchor (Tacit Codex) This project is an instance of the broader [Tacit Codex](../tacit_codex/) work. The central claim: **"LLM required" is not a permanent category. It is a label for decisions we haven't formalized yet.** Every decision in software generation is either: 1. **Deterministic** — given the same inputs, always the same output (e.g., filename = `FeatureName.h`, MCP tool name = `whetstone_` + snake_case verb + noun) 2. **Statistical** — the output is one of N known options, predictable from features (e.g., verification type from step description) → specialist 3. **"Human mind magic"** — the residual category for decisions that feel non-mechanical The third category is not permanent. It marks where the decision space hasn't been decomposed fine enough yet. "Write a function body" feels like magic until you decompose it into: structural pattern (specialist) × error handling style (specialist) × data structure choice (specialist) × loop form (specialist) × variable naming pattern (specialist), all composed through deterministic template instantiation. **The LLM is the scaffolding.** It operates at high entropy while the specialists are being built. Every LLM decision is an implicit training example for the specialist that will eventually replace it. The Tacit Codex entropy ledger tracks which decisions are still unformalized — the specialist fleet is what the collapsed version looks like. --- ## Current State (as of 2026-03-28) Two specialists trained and validated: | Specialist | Classes | Accuracy | Training data | Checkpoint | |-----------|---------|----------|---------------|------------| | `verification_type` | unit/integration/schema/smoke/docs (5-way) | **83.7%** | 978 examples from 260 sprint plans | `/mnt/storage/fabricate_runs/whetstone_verification_type/` | | `confidence_tier` | high/medium/low (3-way) | **100%** | 1020 synthetic examples from formula | `/mnt/storage/fabricate_runs/whetstone_confidence_tier/` | The `verification_type` specialist was trained entirely from the sprint plan files that already exist in this repo — no external data, no hand-labeling. The sprint step descriptions (`### Step NNN: `) are labeled by heuristic rules that classify each description into its verification type. The `confidence_tier` specialist was trained on synthetic data generated from the actual formula in `TaskitemConfidenceAmbiguity.h`. It learned the formula exactly. --- ## Quick Start ```bash # From whetstone_DSL root # Generate training data (sprint plans → TSV) python3 specialists/scripts/gen_verification_type_data.py --sprint-dir . # Train bash specialists/scripts/train_verification_type.sh # Infer on a single example python3 specialists/scripts/infer.py \ --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \ --labels "unit,integration,schema,smoke,docs" \ --text "whetstone_detect_conflicts MCP tool" # → integration (conf=1.00) # Interactive REPL (omit --text) python3 specialists/scripts/infer.py \ --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \ --labels "unit,integration,schema,smoke,docs" # Evaluate accuracy on a held-out TSV python3 specialists/scripts/infer.py \ --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \ --labels "unit,integration,schema,smoke,docs" \ --eval-tsv specialists/data/generated/verification_type_eval.tsv ``` --- ## Technical Notes for Next Agent **Training data format** (TSV, required by the Fabricate binary): ``` \t\t ``` - `integer_label`: index into `--labels` list (0-based) - `hops`: always `0` for Whetstone specialists (unused, required by format) - `text`: the input text to classify **The `--labels` flag** maps integer indices to token strings the model emits. `--labels "unit,integration,schema,smoke,docs"` means label 0 = "unit", 1 = "integration", etc. **The label column must be an integer, not a string.** String labels cause a crash. **Fabricate binary**: `/home/bill/Documents/WhetstoneAI_Fabricate/build/whetstone_fabricate` **Training harness**: `/home/bill/Documents/WhetstoneAI_Fabricate/run_grokking_until.py` **Eval tool**: `/home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_multiclass.py` **Inference helpers**: `/home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_with_vocab.py` **Model architecture** (fixed for all specialists): `--hidden-dim 128 --layers 1 --heads 1 --seq-len 96 --batch-size 32` This produces ~213K parameters, ~800KB checkpoint. **Grokking** is the normal training outcome. The model memorizes first (high training accuracy, low eval accuracy), then at some step (usually 1000-3000) the eval accuracy jumps sharply. The harness detects this and stops automatically. Do not be alarmed by "Fresh start" at the top of training logs — this means no prior checkpoint exists in the output dir, not that the weights are random (they're initialized from scratch as intended). **Known issue from RSA work**: If you ever resume training a specialist from an existing checkpoint while changing the `--labels` order, the token rows in the checkpoint will be misaligned. Use `/home/bill/Documents/WhetstoneAI_Fabricate/tools/swap_label_rows.py` to fix this before resuming. See `WhetstoneAI_Fabricate/HANDOFF-2026-03-28-RSA.md` for the full story of this bug. --- ## Files in This Directory ``` specialists/ README.md ← this file ROADMAP.md ← phased implementation plan scripts/ gen_verification_type_data.py extract sprint plans → labeled TSV gen_confidence_tier_data.py generate synthetic formula data → TSV train_verification_type.sh train verification_type specialist train_confidence_tier.sh train confidence_tier specialist infer.py single/batch/eval/REPL inference data/ generated/ TSVs produced by gen_* scripts (committed) mined/ data extracted from JSONL pipeline runs (future) docs/ ARCHITECTURE.md full decision-point inventory + design rationale runs/ *_history.json grokking history per specialist (committed) *_training.log full training logs (gitignore these) ```