whetstone_DSL/specialists/docs/ARCHITECTURE.md

Whetstone Specialist Fleet — Architecture

Date: 2026-03-28 Status: Phase 1 — first specialist training

---

The Core Idea

The current Whetstone pipeline makes every bounded decision with one large LLM call. Each call is ~seconds, sequential, and expensive. Many of these decisions are bounded-output classifications — the exact problem that WhetstoneAI Fabricate's ~213K-parameter (~800KB) specialists solve in microseconds.

Hypothesis: Replace bounded-output LLM decisions in the Whetstone taskitem pipeline with a fleet of micro-specialists, each ~800KB, running in parallel.

Why It Works

The WhetstoneAI RSA experiment demonstrated:

• 96.3% in-distribution accuracy on domain classification
• 76.7% OOD accuracy after replay hardening (up from 44.8%)
• Inference time: microseconds per query on GPU, 1–5ms on CPU (numpy)
• Model size: ~213K params, ~800KB per checkpoint
• RTX 3060 (12GB VRAM): can hold ~14,000 specialists resident simultaneously

The Whetstone taskitem pipeline already has:

• Bounded enum decision points (verification type, prerequisite ops, worker type...)
• Recorded task completions (18,519 pipeline runs)
• Concurrency/dependency architecture (lock-based pool with parallel workers)

This means the specialists can map 1:1 onto the pipeline's decision points and run in parallel with the existing task dependency graph.

---

Decision Points Amenable to Specialists

Tier 1: Text-Input Classifiers (primary target)

These decisions receive a task description and output a bounded choice. They currently run inside the LLM and are the highest-value targets.

Specialist	Input	Output	Classes
verification_type	task step description	verification type	unit / integration / schema / smoke / docs
prereq_op_selector	task description	prerequisite op set	validate-intake / resolve-dependencies / architect-review / manual-approval
worker_type	task description	worker role	implementer / reviewer / architect / qa
gap_class	taskitem issues list	gap classification	under_constrained / env_gap / planning_gap / cross_project / generator_cap

Tier 2: Structural Specialists (formula approximators)

These learn the deterministic scoring formulas — useful for ultra-fast pre-screening without spinning up the full validator.

Specialist	Input	Output	Notes
confidence_tier	(conflict_count, ambiguity_count, deps, prereqs)	high / medium / low	Learns the deduction formula
escalation_predictor	(confidence, ambiguity_count)	escalate: bool	Binary: confidence < 60 OR ambiguity > 0
quality_score_tier	task features	pass / warn / fail	Learns SelfContainmentScorer cutoffs

Tier 3: Future Specialists (need data collection)

Specialist	Input	Output	Blocker
library_selector	task intent + available libs	selected library	No labeled data yet
milestone_count	requirement count + conflicts	1 / 2 / 3 / 4+ milestones	Need more diverse pipeline runs
task_position_to_dependency	task index + total count	has_dep: bool	Imbalanced data (97% no-dep)

---

Architecture: Fleet Model

┌─────────────────────────────────────────────────────────────────┐
│ Whetstone MCP Request                                           │
│ (generate_taskitems, validate_taskitem, ...)                    │
└────────────────────────┬────────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│ Specialist Dispatcher (Python: infer.py)                        │
│                                                                 │
│  Task text ──► verification_type_specialist ──► unit            │
│  Task text ──► prereq_op_specialist         ──► [validate, ...]  │
│  Task text ──► worker_type_specialist       ──► implementer     │
│  Features  ──► confidence_tier_specialist   ──► high            │
│                                                                 │
│  All specialists run in parallel via thread pool                │
└──────────────┬──────────────────────────────────────────────────┘
               │ aggregated results
               ▼
┌─────────────────────────────────────────────────────────────────┐
│ Whetstone C++ Runtime                                           │
│ (enriches taskitem fields with specialist predictions)          │
└─────────────────────────────────────────────────────────────────┘

Specialist Model Spec (all identical architecture)

Architecture: Transformer (D=128, L=1, heads=1, seq_len=96)
Parameters:   ~213K
Size:         ~832KB per checkpoint
Inference:    <1ms CPU (numpy), ~50µs GPU
Training:     1K–5K steps, grokking detection, weight decay

Reuses the WhetstoneAI Fabricate binary (whetstone_fabricate) and training harness (run_grokking_until.py) without modification.

---

Training Pipeline

sprint_plans/*.md
        │
        ▼
scripts/gen_verification_type_data.py   ← TSV extractor (one per specialist)
        │
        ▼
data/generated/verification_type_train.tsv   (label<TAB>0<TAB>text)
data/generated/verification_type_eval.tsv
        │
        ▼
scripts/train_verification_type.sh           ← wraps run_grokking_until.py
        │
        ▼
/mnt/storage/fabricate_runs/whetstone_verification_type/
  checkpoint.bin   (weights + optional Adam state)
  fabricate.db     (vocabulary)
        │
        ▼
specialists/scripts/infer.py                 ← Python inference wrapper
        │
        ▼
Whetstone MCP tool integration (future)

---

Data Sources

Current (18,519 pipeline run records)

training_data/lora/taskitem_pipeline_runs.jsonl

Limitation: Almost entirely homogeneous (2 task types repeated, no text diversity). Useful for structural/formula specialists (Tier 2). NOT useful for text-based classifiers.

Primary for Text Specialists

sprint*_plan.md files (260 files, ~2,600 labeled step descriptions)

Each step description has an implicit verification type based on its content:

• ### Step NNN: whetstone_foo MCP tool → integration
• ### Step NNN: Schema for X → schema
• ### Step NNN: Sprint N integration summary + regression → smoke
• ### Step NNN: X documentation packet → docs
• Everything else → unit

Future Data Collection

Add richer recording to run_sprint_taskitem_pipeline.sh:

• Capture per-step verification type decisions
• Record library selection choices
• Record milestone count decisions

---

Current Status

Specialist	Data Source	Training Status	Accuracy
verification_type	sprint plans (978 train, 172 eval)	TRAINING	TBD
All others	—	planned	—

---

Integration Plan (Future)

1. Phase 1 (now): Train verification_type specialist, validate accuracy
2. Phase 2: Wire into whetstone_validate_taskitem as a pre-prediction step
3. Phase 3: Train prereq_op and confidence_tier specialists
4. Phase 4: Build Python dispatcher that runs all specialists in parallel
5. Phase 5: Expose as new MCP tool whetstone_specialist_predict
6. Phase 6: Replace LLM calls for bounded decisions with specialist fleet

---

Connection to WhetstoneAI Fabricate

This project reuses the Fabricate infrastructure directly:

• Binary: /home/bill/Documents/WhetstoneAI_Fabricate/build/whetstone_fabricate
• Harness: /home/bill/Documents/WhetstoneAI_Fabricate/run_grokking_until.py
• Eval tools: /home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_multiclass.py
• Inference: /home/bill/Documents/WhetstoneAI_Fabricate/tools/eval_with_vocab.py

No code changes to Fabricate are needed. It is consumed as a training and inference substrate.