156 lines
4.7 KiB
Markdown
156 lines
4.7 KiB
Markdown
# Case Study: WhetstoneDSL
|
|
|
|
## Why This Is The First Case Study
|
|
|
|
WhetstoneDSL is already using the Routed-Specialist Architecture for bounded editor
|
|
and workflow decisions. That makes it the right proving ground for `whetstone_RSA`.
|
|
|
|
Relevant existing work:
|
|
|
|
- `/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/HANDOFF-2026-03-28.md`
|
|
- `/home/bill/Documents/WhetstoneAI_Fabricate/HANDOFF-2026-03-29.md`
|
|
|
|
## What Is Already Proven
|
|
|
|
The current specialist fleet established that tiny transformers can replace some
|
|
LLM calls in the MCP pipeline when the outputs are bounded.
|
|
|
|
Documented examples:
|
|
|
|
- `verification_type`
|
|
- 5-way classifier
|
|
- `83.7%` in the initial handoff, later `84.9%` at step 4000 in the current history
|
|
- `confidence_tier`
|
|
- 3-way classifier
|
|
- 100% held-out accuracy
|
|
- `worker_type`
|
|
- 4-way classifier
|
|
- `88.5%` at step 4000 in the current history
|
|
- `prereq_op`
|
|
- 4-way classifier
|
|
- `69.4%` and flat across steps 1000-4000 in the current history
|
|
- `automatability`
|
|
- 6-way classifier
|
|
- `69.6%` and flat across steps 1000-4000 in the current history
|
|
|
|
Shared properties:
|
|
|
|
- bounded vocabulary
|
|
- deterministic downstream meaning
|
|
- AST-first architecture
|
|
- millisecond-scale inference
|
|
- ~800KB checkpoint size
|
|
|
|
## Important Constraint
|
|
|
|
Whetstone is AST-first, not language-first.
|
|
|
|
That means RSA gates should generally live at the AST or orchestration layer, not
|
|
be duplicated per target language. The model should choose bounded structural
|
|
decisions; deterministic projection handles language output.
|
|
|
|
## Current Opportunity
|
|
|
|
The current Whetstone specialist work suggests there are at least three gate bands:
|
|
|
|
- low-entropy gates that are formula-like and should stay deterministic
|
|
- medium-entropy gates that fit the ~800KB specialist tier
|
|
- higher-entropy bounded gates that may need 10x to 16x larger models, better
|
|
schemas, or multi-stage decomposition
|
|
|
|
This project should turn those bands into a measurable sizing policy.
|
|
|
|
## Current Snapshot
|
|
|
|
The active case study is already informative enough to shape the policy:
|
|
|
|
- `confidence_tier` is effectively deterministic in disguise and should be judged
|
|
as a formula baseline, not as evidence for model capability
|
|
- `verification_type` and `worker_type` look viable at the tiny tier, but should
|
|
be evaluated through abstain and retry policy rather than raw accuracy alone
|
|
- `prereq_op` and `automatability` are currently below an obvious auto-accept bar
|
|
at the tiny tier
|
|
|
|
Just as important, the current larger-model runs are not yet producing accuracy
|
|
comparisons. The logs show startup on `hidden=512, layers=4`, but the runs abort
|
|
before any substantive training result is recorded. That means the current frontier
|
|
is:
|
|
|
|
- tiny-tier accuracy is known
|
|
- large-tier cost is partially visible
|
|
- large-tier quality is not yet known
|
|
|
|
So the correct immediate move is not "chase bigger until better." It is to adopt a
|
|
deployment policy that measures accepted-risk performance while the larger-tier
|
|
infrastructure is being stabilized.
|
|
|
|
## Interpretation
|
|
|
|
This case study strongly suggests that raw held-out accuracy is not enough to drive
|
|
the roadmap.
|
|
|
|
For WhetstoneDSL, the more relevant questions are:
|
|
|
|
- which gates are reversible?
|
|
- which gates are validated by deterministic checks later?
|
|
- which mistakes silently poison later stages?
|
|
- which gates can abstain cheaply and retry with more context?
|
|
|
|
That is why `whetstone_RSA` should treat deployment quality as a system metric,
|
|
not a model-only metric.
|
|
|
|
## Candidate Gate Inventory
|
|
|
|
Immediate WhetstoneDSL gate families:
|
|
|
|
- `verification_type`
|
|
- `confidence_tier`
|
|
- `prereq_op_selector`
|
|
- `worker_type`
|
|
- `gap_class`
|
|
- `escalation_predictor`
|
|
- `automatability_strategy`
|
|
|
|
These are good initial benchmarks because they already have:
|
|
|
|
- clear output enums
|
|
- architectural value
|
|
- known downstream consequences
|
|
- mixed difficulty levels
|
|
|
|
## What To Measure
|
|
|
|
For each gate:
|
|
|
|
- output cardinality
|
|
- slot count
|
|
- ambiguity of the input surface
|
|
- context width required
|
|
- label boundary sharpness
|
|
- train/eval accuracy
|
|
- calibration quality
|
|
- failure concentration by class
|
|
- smallest model size that clears target quality
|
|
|
|
## Working Hypothesis
|
|
|
|
The main predictor will not just be the number of output labels.
|
|
|
|
A gate can be 3-way and still difficult if it depends on:
|
|
|
|
- long context
|
|
- latent intent reconstruction
|
|
- weakly separable labels
|
|
- hidden state
|
|
- domain-specific lexical variation
|
|
|
|
So the sizing function should be based on effective entropy, not enum size alone.
|
|
|
|
## Why This Matters
|
|
|
|
If Whetstone can tag nodes, route tasks, select templates, and choose review paths
|
|
through bounded specialists, then LLM use becomes residual instead of default.
|
|
|
|
That would make the AST-first environment cheaper, faster, more local, and easier
|
|
to retrain per project.
|