whetstone_DSL/specialists

Whetstone Specialist Fleet

What this is: A system of ~800KB transformer specialists that replace bounded-output LLM decisions inside the Whetstone MCP pipeline. Each specialist handles exactly one classification or scoring decision, runs in under a millisecond, and can be held resident in GPU VRAM alongside thousands of peers.

This is not a LoRA. It is not a fine-tuned 14B model. It is a different architecture for a different problem.

---

Origin: WhetstoneAI Fabricate Experiments

This project grew out of the WhetstoneAI_Fabricate experiments at /home/bill/Documents/WhetstoneAI_Fabricate/.

Those experiments were training tiny transformer models (~213K parameters) to solve structured reasoning problems — math ordering, logic gates, state machines, modular arithmetic, DAG traversal. Each model was a binary classifier (true/false) over reasoning chains. The key finding was the Routed-Specialist Architecture (RSA):

• A small router model (5-way classifier) reads a raw question and picks which domain expert to consult
• The chosen specialist (binary classifier) answers the question
• End-to-end composed accuracy: 96.3% in-distribution, 76.7% OOD after hardening
• Total model size: router (~800KB) + 5 specialists (~800KB each) = ~5MB
• Inference: microseconds per query on GPU

The critical realization: the Whetstone taskitem pipeline is already doing this same structure, but with a 14B LLM handling all the decisions. Every tool call involves the LLM choosing from bounded option sets — verification type, prerequisite ops, confidence tier, worker role, quality score tier. These are exactly the classifier-shaped problems that the Fabricate architecture solves at 800KB.

The WhetstoneAI_Fabricate binary and training harness are reused here unchanged. No new training infrastructure was written.

---

The AST Is Already Rich — Specialists Predict Its Fields

Whetstone is not a C++ code generator. C++, Python, Rust, and all other output languages are projections of an AST-as-source-of-truth. The AST uses language-neutral conceptType names (Function, ClassDeclaration, IfStatement) that map across 19+ languages through a grammar layer. This means specialists operate at the AST level and cover all languages simultaneously — there are no per-language variants for decisions that live in the AST.

More importantly: the Whetstone AST already carries annotation metadata that most ASTs never have. The annotation schema in editor/src/ast/Annotation.h (60+ types) includes:

RiskAnnotation.dependentCount         int    how many callers/consumers downstream
CallHierarchyInfo.callerIds           list   which functions call this one
BlastRadiusReport.riskLevel           enum   none/low/medium/high (blast radius)
IntentAnnotation.category             enum   validation/transformation/io/...
ComplexityAnnotation.cognitiveComplexity  int  1-10 scale
AutomatabilityAnnotation.strategy     enum   deterministic|template|specialist|slm|llm|human
AutomatabilityAnnotation.confidence   float  0.0–1.0

NOTE: Current Annotation.h is missing specialist in this enum. Add it before training the automatability_strategy specialist. The 6 tiers in order: formula → template → specialist (~800KB bounded classifier) → slm → llm → human.

AutomatabilityAnnotation.strategy is the Tacit Codex entropy classification built directly into the AST schema. It encodes whether each node's decisions should be handled by a formula, template instantiation, a tiny specialist, a small LLM, a large LLM, or a human. Training the automatability_strategy specialist means populating this field automatically across the entire codebase — every node gets its entropy tier assigned without an LLM call.

RiskAnnotation.dependentCount is the "how many things downstream depend on this?" metric the system already tracks. Combined with blast radius analysis (BlastRadiusAnalyzer.h) and call hierarchy (ContextAPI.h), the AST already measures change risk. These are input features for the change_risk_tier specialist.

The Core Premise

What an MCP tool call currently does

When you call whetstone_generate_taskitems, the LLM receives context and makes roughly these decisions:

How many milestones?         → integer 1-6
Verification type?           → unit | integration | schema | smoke | docs
Prereq ops needed?           → combo of 4 possible ops
Worker type?                 → implementer | reviewer | architect | qa
Confidence level?            → integer 0-100 (computed from formula)
Should escalate?             → bool
...and then the generative parts:
Task title text?             → open text
Acceptance criteria text?    → open text

The first group are bounded decisions — the output is always one choice from a small fixed set. These are classifier problems. The second group are generative decisions — the output is open text. These still require the LLM.

What the specialist fleet does instead

Every bounded decision becomes a specialist call. All specialists fire in parallel before or instead of the LLM call. The LLM, if called at all, only handles the open-text portions — with the bounded decisions already pre-filled.

[specialist: verification_type]  ──► unit/integration/schema/smoke/docs   ~0.5ms
[specialist: prereq_op_selector] ──► [validate-intake, resolve-deps]       ~0.5ms
[specialist: confidence_tier]    ──► high/medium/low                       ~0.5ms
[specialist: worker_type]        ──► implementer                           ~0.5ms
                     ↓ all parallel, all in <2ms total
[LLM: generate task title]       ──► "ResourceLocks schema extension"      ~3s
[LLM: write acceptance criteria] ──► "..."                                 ~2s

Why this scales

An RTX 3060 has 12GB VRAM. One quantized 14B LLM takes ~8GB. One specialist takes ~800KB. The same card that holds one LLM can hold ~14,000 specialists resident simultaneously.

50 bounded decision types × 20 domain variants each = 1,000 specialists = 800MB VRAM. All 1,000 can run in a single batched forward pass.

---

The Philosophical Anchor (Tacit Codex)

This project is an instance of the broader Tacit Codex work.

The central claim: "LLM required" is not a permanent category. It is a label for decisions we haven't formalized yet.

Every decision in software generation is either:

1. Deterministic — given the same inputs, always the same output (e.g., filename = FeatureName.h, MCP tool name = whetstone_ + snake_case verb + noun)
2. Statistical — the output is one of N known options, predictable from features (e.g., verification type from step description) → specialist
3. "Human mind magic" — the residual category for decisions that feel non-mechanical

The third category is not permanent. It marks where the decision space hasn't been decomposed fine enough yet. "Write a function body" feels like magic until you decompose it into: structural pattern (specialist) × error handling style (specialist) × data structure choice (specialist) × loop form (specialist) × variable naming pattern (specialist), all composed through deterministic template instantiation.

The LLM is the scaffolding. It operates at high entropy while the specialists are being built. Every LLM decision is an implicit training example for the specialist that will eventually replace it. The Tacit Codex entropy ledger tracks which decisions are still unformalized — the specialist fleet is what the collapsed version looks like.

---

Current State (as of 2026-03-28)

Two specialists trained and validated:

Specialist	Classes	Accuracy	Training data	Checkpoint
verification_type	unit/integration/schema/smoke/docs (5-way)	83.7%	978 examples from 260 sprint plans	/mnt/storage/fabricate_runs/whetstone_verification_type/
confidence_tier	high/medium/low (3-way)	100%	1020 synthetic examples from formula	/mnt/storage/fabricate_runs/whetstone_confidence_tier/

The verification_type specialist was trained entirely from the sprint plan files that already exist in this repo — no external data, no hand-labeling. The sprint step descriptions (### Step NNN: <description>) are labeled by heuristic rules that classify each description into its verification type.

The confidence_tier specialist was trained on synthetic data generated from the actual formula in TaskitemConfidenceAmbiguity.h. It learned the formula exactly.

---

Quick Start

# From whetstone_DSL root

# Generate training data (sprint plans → TSV)
python3 specialists/scripts/gen_verification_type_data.py --sprint-dir .

# Train
bash specialists/scripts/train_verification_type.sh

# Infer on a single example
python3 specialists/scripts/infer.py \
  --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \
  --labels "unit,integration,schema,smoke,docs" \
  --text "whetstone_detect_conflicts MCP tool"
# → integration (conf=1.00)

# Interactive REPL (omit --text)
python3 specialists/scripts/infer.py \
  --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \
  --labels "unit,integration,schema,smoke,docs"

# Evaluate accuracy on a held-out TSV
python3 specialists/scripts/infer.py \
  --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \
  --labels "unit,integration,schema,smoke,docs" \
  --eval-tsv specialists/data/generated/verification_type_eval.tsv

---

Technical Notes for Next Agent

Training data format (TSV, required by the Fabricate binary):

<integer_label>\t<hops>\t<text>

• integer_label: index into --labels list (0-based)
• hops: always 0 for Whetstone specialists (unused, required by format)
• text: the input text to classify

The --labels flag maps integer indices to token strings the model emits. --labels "unit,integration,schema,smoke,docs" means label 0 = "unit", 1 = "integration", etc. The label column must be an integer, not a string. String labels cause a crash.

Fabricate binary: /home/bill/Documents/WhetstoneAI_Fabricate/build/whetstone_fabricate Training harness: /home/bill/Documents/WhetstoneAI_Fabricate/run_grokking_until.py Eval tool: /home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_multiclass.py Inference helpers: /home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_with_vocab.py

Model architecture (fixed for all specialists): --hidden-dim 128 --layers 1 --heads 1 --seq-len 96 --batch-size 32 This produces ~213K parameters, ~800KB checkpoint.

Grokking is the normal training outcome. The model memorizes first (high training accuracy, low eval accuracy), then at some step (usually 1000-3000) the eval accuracy jumps sharply. The harness detects this and stops automatically. Do not be alarmed by "Fresh start" at the top of training logs — this means no prior checkpoint exists in the output dir, not that the weights are random (they're initialized from scratch as intended).

Known issue from RSA work: If you ever resume training a specialist from an existing checkpoint while changing the --labels order, the token rows in the checkpoint will be misaligned. Use /home/bill/Documents/WhetstoneAI_Fabricate/tools/swap_label_rows.py to fix this before resuming. See WhetstoneAI_Fabricate/HANDOFF-2026-03-28-RSA.md for the full story of this bug.

---

Files in This Directory

specialists/
  README.md                    ← this file
  ROADMAP.md                   ← phased implementation plan
  scripts/
    gen_verification_type_data.py   extract sprint plans → labeled TSV
    gen_confidence_tier_data.py     generate synthetic formula data → TSV
    train_verification_type.sh      train verification_type specialist
    train_confidence_tier.sh        train confidence_tier specialist
    infer.py                        single/batch/eval/REPL inference
  data/
    generated/                 TSVs produced by gen_* scripts (committed)
    mined/                     data extracted from JSONL pipeline runs (future)
  docs/
    ARCHITECTURE.md            full decision-point inventory + design rationale
  runs/
    *_history.json             grokking history per specialist (committed)
    *_training.log             full training logs (gitignore these)