whetstone_RSA/docs/case_studies/whetstone_dsl.md

# Case Study: WhetstoneDSL

## Why This Is The First Case Study

WhetstoneDSL is already using the Routed-Specialist Architecture for bounded editor
and workflow decisions. That makes it the right proving ground for `whetstone_RSA`.

Relevant existing work:

- `/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/HANDOFF-2026-03-28.md`
- `/home/bill/Documents/WhetstoneAI_Fabricate/HANDOFF-2026-03-29.md`

## What Is Already Proven

The current specialist fleet established that tiny transformers can replace some
LLM calls in the MCP pipeline when the outputs are bounded.

Documented examples:

- `verification_type`
  - 5-way classifier
  - `83.7%` in the initial handoff, later `84.9%` at step 4000 in the current history
- `confidence_tier`
  - 3-way classifier
  - 100% held-out accuracy
- `worker_type`
  - 4-way classifier
  - `88.5%` at step 4000 in the current history
- `prereq_op`
  - 4-way classifier
  - `69.4%` and flat across steps 1000-4000 in the current history
- `automatability`
  - 6-way classifier
  - `69.6%` and flat across steps 1000-4000 in the current history

Shared properties:

- bounded vocabulary
- deterministic downstream meaning
- AST-first architecture
- millisecond-scale inference
- ~800KB checkpoint size

## Important Constraint

Whetstone is AST-first, not language-first.

That means RSA gates should generally live at the AST or orchestration layer, not
be duplicated per target language. The model should choose bounded structural
decisions; deterministic projection handles language output.

## Current Opportunity

The current Whetstone specialist work suggests there are at least three gate bands:

- low-entropy gates that are formula-like and should stay deterministic
- medium-entropy gates that fit the ~800KB specialist tier
- higher-entropy bounded gates that may need 10x to 16x larger models, better
  schemas, or multi-stage decomposition

This project should turn those bands into a measurable sizing policy.

## Current Snapshot

The active case study is already informative enough to shape the policy:

- `confidence_tier` is effectively deterministic in disguise and should be judged
  as a formula baseline, not as evidence for model capability
- `verification_type` and `worker_type` look viable at the tiny tier, but should
  be evaluated through abstain and retry policy rather than raw accuracy alone
- `prereq_op` and `automatability` are currently below an obvious auto-accept bar
  at the tiny tier

Just as important, the current larger-model runs are not yet producing accuracy
comparisons. The logs show startup on `hidden=512, layers=4`, but the runs abort
before any substantive training result is recorded. That means the current frontier
is:

- tiny-tier accuracy is known
- large-tier cost is partially visible
- large-tier quality is not yet known

So the correct immediate move is not "chase bigger until better." It is to adopt a
deployment policy that measures accepted-risk performance while the larger-tier
infrastructure is being stabilized.

## Interpretation

This case study strongly suggests that raw held-out accuracy is not enough to drive
the roadmap.

For WhetstoneDSL, the more relevant questions are:

- which gates are reversible?
- which gates are validated by deterministic checks later?
- which mistakes silently poison later stages?
- which gates can abstain cheaply and retry with more context?

That is why `whetstone_RSA` should treat deployment quality as a system metric,
not a model-only metric.

## Candidate Gate Inventory

Immediate WhetstoneDSL gate families:

- `verification_type`
- `confidence_tier`
- `prereq_op_selector`
- `worker_type`
- `gap_class`
- `escalation_predictor`
- `automatability_strategy`

These are good initial benchmarks because they already have:

- clear output enums
- architectural value
- known downstream consequences
- mixed difficulty levels

## What To Measure

For each gate:

- output cardinality
- slot count
- ambiguity of the input surface
- context width required
- label boundary sharpness
- train/eval accuracy
- calibration quality
- failure concentration by class
- smallest model size that clears target quality

## Working Hypothesis

The main predictor will not just be the number of output labels.

A gate can be 3-way and still difficult if it depends on:

- long context
- latent intent reconstruction
- weakly separable labels
- hidden state
- domain-specific lexical variation

So the sizing function should be based on effective entropy, not enum size alone.

## Why This Matters

If Whetstone can tag nodes, route tasks, select templates, and choose review paths
through bounded specialists, then LLM use becomes residual instead of default.

That would make the AST-first environment cheaper, faster, more local, and easier
to retrain per project.