whetstone_DSL/specialists/README.md

# Whetstone Specialist Fleet

**What this is:** A system of ~800KB transformer specialists that replace bounded-output
LLM decisions inside the Whetstone MCP pipeline. Each specialist handles exactly one
classification or scoring decision, runs in under a millisecond, and can be held
resident in GPU VRAM alongside thousands of peers.

This is not a LoRA. It is not a fine-tuned 14B model. It is a different architecture
for a different problem.

---

## Origin: WhetstoneAI Fabricate Experiments

This project grew out of the WhetstoneAI_Fabricate experiments at
`/home/bill/Documents/WhetstoneAI_Fabricate/`.

Those experiments were training tiny transformer models (~213K parameters) to solve
structured reasoning problems — math ordering, logic gates, state machines, modular
arithmetic, DAG traversal. Each model was a binary classifier (true/false) over
reasoning chains. The key finding was the **Routed-Specialist Architecture (RSA)**:

- A small router model (5-way classifier) reads a raw question and picks which
  domain expert to consult
- The chosen specialist (binary classifier) answers the question
- End-to-end composed accuracy: 96.3% in-distribution, 76.7% OOD after hardening
- Total model size: router (~800KB) + 5 specialists (~800KB each) = ~5MB
- Inference: microseconds per query on GPU

The critical realization: **the Whetstone taskitem pipeline is already doing this
same structure, but with a 14B LLM handling all the decisions.** Every tool call
involves the LLM choosing from bounded option sets — verification type, prerequisite
ops, confidence tier, worker role, quality score tier. These are exactly the
classifier-shaped problems that the Fabricate architecture solves at 800KB.

The WhetstoneAI_Fabricate binary and training harness are reused here unchanged.
No new training infrastructure was written.

---

## The AST Is Already Rich — Specialists Predict Its Fields

Whetstone is not a C++ code generator. C++, Python, Rust, and all other output
languages are *projections* of an AST-as-source-of-truth. The AST uses
language-neutral `conceptType` names (`Function`, `ClassDeclaration`, `IfStatement`)
that map across 19+ languages through a grammar layer. This means specialists operate
at the AST level and cover all languages simultaneously — there are no per-language
variants for decisions that live in the AST.

More importantly: the Whetstone AST already carries annotation metadata that most
ASTs never have. The annotation schema in `editor/src/ast/Annotation.h` (60+ types)
includes:

```
RiskAnnotation.dependentCount         int    how many callers/consumers downstream
CallHierarchyInfo.callerIds           list   which functions call this one
BlastRadiusReport.riskLevel           enum   none/low/medium/high (blast radius)
IntentAnnotation.category             enum   validation/transformation/io/...
ComplexityAnnotation.cognitiveComplexity  int  1-10 scale
AutomatabilityAnnotation.strategy     enum   deterministic|template|specialist|slm|llm|human
AutomatabilityAnnotation.confidence   float  0.0–1.0
```

NOTE: Current `Annotation.h` is missing `specialist` in this enum. Add it before
training the `automatability_strategy` specialist. The 6 tiers in order:
formula → template → specialist (~800KB bounded classifier) → slm → llm → human.

**`AutomatabilityAnnotation.strategy` is the Tacit Codex entropy classification
built directly into the AST schema.** It encodes whether each node's decisions
should be handled by a formula, template instantiation, a tiny specialist,
a small LLM, a large LLM, or a human. Training the `automatability_strategy`
specialist means populating this field automatically across the entire codebase —
every node gets its entropy tier assigned without an LLM call.

`RiskAnnotation.dependentCount` is the "how many things downstream depend on this?"
metric the system already tracks. Combined with blast radius analysis
(`BlastRadiusAnalyzer.h`) and call hierarchy (`ContextAPI.h`), the AST already
measures change risk. These are input features for the `change_risk_tier` specialist.

## The Core Premise

### What an MCP tool call currently does

When you call `whetstone_generate_taskitems`, the LLM receives context and makes
roughly these decisions:

```
How many milestones?         → integer 1-6
Verification type?           → unit | integration | schema | smoke | docs
Prereq ops needed?           → combo of 4 possible ops
Worker type?                 → implementer | reviewer | architect | qa
Confidence level?            → integer 0-100 (computed from formula)
Should escalate?             → bool
...and then the generative parts:
Task title text?             → open text
Acceptance criteria text?    → open text
```

The first group are **bounded decisions** — the output is always one choice from
a small fixed set. These are classifier problems. The second group are **generative
decisions** — the output is open text. These still require the LLM.

### What the specialist fleet does instead

Every bounded decision becomes a specialist call. All specialists fire in parallel
before or instead of the LLM call. The LLM, if called at all, only handles the
open-text portions — with the bounded decisions already pre-filled.

```
[specialist: verification_type]  ──► unit/integration/schema/smoke/docs   ~0.5ms
[specialist: prereq_op_selector] ──► [validate-intake, resolve-deps]       ~0.5ms
[specialist: confidence_tier]    ──► high/medium/low                       ~0.5ms
[specialist: worker_type]        ──► implementer                           ~0.5ms
                     ↓ all parallel, all in <2ms total
[LLM: generate task title]       ──► "ResourceLocks schema extension"      ~3s
[LLM: write acceptance criteria] ──► "..."                                 ~2s
```

### Why this scales

An RTX 3060 has 12GB VRAM. One quantized 14B LLM takes ~8GB. One specialist takes
~800KB. The same card that holds one LLM can hold **~14,000 specialists resident
simultaneously**.

50 bounded decision types × 20 domain variants each = 1,000 specialists = 800MB VRAM.
All 1,000 can run in a single batched forward pass.

---

## The Philosophical Anchor (Tacit Codex)

This project is an instance of the broader
[Tacit Codex](../tacit_codex/) work.

The central claim: **"LLM required" is not a permanent category. It is a label for
decisions we haven't formalized yet.**

Every decision in software generation is either:
1. **Deterministic** — given the same inputs, always the same output (e.g., filename
   = `FeatureName.h`, MCP tool name = `whetstone_` + snake_case verb + noun)
2. **Statistical** — the output is one of N known options, predictable from features
   (e.g., verification type from step description) → specialist
3. **"Human mind magic"** — the residual category for decisions that feel non-mechanical

The third category is not permanent. It marks where the decision space hasn't been
decomposed fine enough yet. "Write a function body" feels like magic until you
decompose it into: structural pattern (specialist) × error handling style (specialist)
× data structure choice (specialist) × loop form (specialist) × variable naming
pattern (specialist), all composed through deterministic template instantiation.

**The LLM is the scaffolding.** It operates at high entropy while the specialists
are being built. Every LLM decision is an implicit training example for the specialist
that will eventually replace it. The Tacit Codex entropy ledger tracks which decisions
are still unformalized — the specialist fleet is what the collapsed version looks like.

---

## Current State (as of 2026-03-28)

Two specialists trained and validated:

| Specialist | Classes | Accuracy | Training data | Checkpoint |
|-----------|---------|----------|---------------|------------|
| `verification_type` | unit/integration/schema/smoke/docs (5-way) | **83.7%** | 978 examples from 260 sprint plans | `/mnt/storage/fabricate_runs/whetstone_verification_type/` |
| `confidence_tier` | high/medium/low (3-way) | **100%** | 1020 synthetic examples from formula | `/mnt/storage/fabricate_runs/whetstone_confidence_tier/` |

The `verification_type` specialist was trained entirely from the sprint plan files
that already exist in this repo — no external data, no hand-labeling. The sprint
step descriptions (`### Step NNN: <description>`) are labeled by heuristic rules
that classify each description into its verification type.

The `confidence_tier` specialist was trained on synthetic data generated from the
actual formula in `TaskitemConfidenceAmbiguity.h`. It learned the formula exactly.

---

## Quick Start

```bash
# From whetstone_DSL root

# Generate training data (sprint plans → TSV)
python3 specialists/scripts/gen_verification_type_data.py --sprint-dir .

# Train
bash specialists/scripts/train_verification_type.sh

# Infer on a single example
python3 specialists/scripts/infer.py \
  --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \
  --labels "unit,integration,schema,smoke,docs" \
  --text "whetstone_detect_conflicts MCP tool"
# → integration (conf=1.00)

# Interactive REPL (omit --text)
python3 specialists/scripts/infer.py \
  --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \
  --labels "unit,integration,schema,smoke,docs"

# Evaluate accuracy on a held-out TSV
python3 specialists/scripts/infer.py \
  --run-dir /mnt/storage/fabricate_runs/whetstone_verification_type \
  --labels "unit,integration,schema,smoke,docs" \
  --eval-tsv specialists/data/generated/verification_type_eval.tsv
```

---

## Technical Notes for Next Agent

**Training data format** (TSV, required by the Fabricate binary):
```
<integer_label>\t<hops>\t<text>
```
- `integer_label`: index into `--labels` list (0-based)
- `hops`: always `0` for Whetstone specialists (unused, required by format)
- `text`: the input text to classify

**The `--labels` flag** maps integer indices to token strings the model emits.
`--labels "unit,integration,schema,smoke,docs"` means label 0 = "unit", 1 = "integration", etc.
**The label column must be an integer, not a string.** String labels cause a crash.

**Fabricate binary**: `/home/bill/Documents/WhetstoneAI_Fabricate/build/whetstone_fabricate`
**Training harness**: `/home/bill/Documents/WhetstoneAI_Fabricate/run_grokking_until.py`
**Eval tool**: `/home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_multiclass.py`
**Inference helpers**: `/home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_with_vocab.py`

**Model architecture** (fixed for all specialists):
`--hidden-dim 128 --layers 1 --heads 1 --seq-len 96 --batch-size 32`
This produces ~213K parameters, ~800KB checkpoint.

**Grokking** is the normal training outcome. The model memorizes first (high training
accuracy, low eval accuracy), then at some step (usually 1000-3000) the eval accuracy
jumps sharply. The harness detects this and stops automatically. Do not be alarmed by
"Fresh start" at the top of training logs — this means no prior checkpoint exists in
the output dir, not that the weights are random (they're initialized from scratch as
intended).

**Known issue from RSA work**: If you ever resume training a specialist from an
existing checkpoint while changing the `--labels` order, the token rows in the
checkpoint will be misaligned. Use
`/home/bill/Documents/WhetstoneAI_Fabricate/tools/swap_label_rows.py` to fix this
before resuming. See `WhetstoneAI_Fabricate/HANDOFF-2026-03-28-RSA.md` for the
full story of this bug.

---

## Files in This Directory

```
specialists/
  README.md                    ← this file
  ROADMAP.md                   ← phased implementation plan
  scripts/
    gen_verification_type_data.py   extract sprint plans → labeled TSV
    gen_confidence_tier_data.py     generate synthetic formula data → TSV
    train_verification_type.sh      train verification_type specialist
    train_confidence_tier.sh        train confidence_tier specialist
    infer.py                        single/batch/eval/REPL inference
  data/
    generated/                 TSVs produced by gen_* scripts (committed)
    mined/                     data extracted from JSONL pipeline runs (future)
  docs/
    ARCHITECTURE.md            full decision-point inventory + design rationale
  runs/
    *_history.json             grokking history per specialist (committed)
    *_training.log             full training logs (gitignore these)
```