Initial whetstone_RSA architecture and C++ scaffold
This commit is contained in:
Bill Holcombe
155
docs/case_studies/whetstone_dsl.md
Normal file
155
docs/case_studies/whetstone_dsl.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Case Study: WhetstoneDSL
|
||||
|
||||
## Why This Is The First Case Study
|
||||
|
||||
WhetstoneDSL is already using the Routed-Specialist Architecture for bounded editor
|
||||
and workflow decisions. That makes it the right proving ground for `whetstone_RSA`.
|
||||
|
||||
Relevant existing work:
|
||||
|
||||
- `/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/HANDOFF-2026-03-28.md`
|
||||
- `/home/bill/Documents/WhetstoneAI_Fabricate/HANDOFF-2026-03-29.md`
|
||||
|
||||
## What Is Already Proven
|
||||
|
||||
The current specialist fleet established that tiny transformers can replace some
|
||||
LLM calls in the MCP pipeline when the outputs are bounded.
|
||||
|
||||
Documented examples:
|
||||
|
||||
- `verification_type`
|
||||
- 5-way classifier
|
||||
- `83.7%` in the initial handoff, later `84.9%` at step 4000 in the current history
|
||||
- `confidence_tier`
|
||||
- 3-way classifier
|
||||
- 100% held-out accuracy
|
||||
- `worker_type`
|
||||
- 4-way classifier
|
||||
- `88.5%` at step 4000 in the current history
|
||||
- `prereq_op`
|
||||
- 4-way classifier
|
||||
- `69.4%` and flat across steps 1000-4000 in the current history
|
||||
- `automatability`
|
||||
- 6-way classifier
|
||||
- `69.6%` and flat across steps 1000-4000 in the current history
|
||||
|
||||
Shared properties:
|
||||
|
||||
- bounded vocabulary
|
||||
- deterministic downstream meaning
|
||||
- AST-first architecture
|
||||
- millisecond-scale inference
|
||||
- ~800KB checkpoint size
|
||||
|
||||
## Important Constraint
|
||||
|
||||
Whetstone is AST-first, not language-first.
|
||||
|
||||
That means RSA gates should generally live at the AST or orchestration layer, not
|
||||
be duplicated per target language. The model should choose bounded structural
|
||||
decisions; deterministic projection handles language output.
|
||||
|
||||
## Current Opportunity
|
||||
|
||||
The current Whetstone specialist work suggests there are at least three gate bands:
|
||||
|
||||
- low-entropy gates that are formula-like and should stay deterministic
|
||||
- medium-entropy gates that fit the ~800KB specialist tier
|
||||
- higher-entropy bounded gates that may need 10x to 16x larger models, better
|
||||
schemas, or multi-stage decomposition
|
||||
|
||||
This project should turn those bands into a measurable sizing policy.
|
||||
|
||||
## Current Snapshot
|
||||
|
||||
The active case study is already informative enough to shape the policy:
|
||||
|
||||
- `confidence_tier` is effectively deterministic in disguise and should be judged
|
||||
as a formula baseline, not as evidence for model capability
|
||||
- `verification_type` and `worker_type` look viable at the tiny tier, but should
|
||||
be evaluated through abstain and retry policy rather than raw accuracy alone
|
||||
- `prereq_op` and `automatability` are currently below an obvious auto-accept bar
|
||||
at the tiny tier
|
||||
|
||||
Just as important, the current larger-model runs are not yet producing accuracy
|
||||
comparisons. The logs show startup on `hidden=512, layers=4`, but the runs abort
|
||||
before any substantive training result is recorded. That means the current frontier
|
||||
is:
|
||||
|
||||
- tiny-tier accuracy is known
|
||||
- large-tier cost is partially visible
|
||||
- large-tier quality is not yet known
|
||||
|
||||
So the correct immediate move is not "chase bigger until better." It is to adopt a
|
||||
deployment policy that measures accepted-risk performance while the larger-tier
|
||||
infrastructure is being stabilized.
|
||||
|
||||
## Interpretation
|
||||
|
||||
This case study strongly suggests that raw held-out accuracy is not enough to drive
|
||||
the roadmap.
|
||||
|
||||
For WhetstoneDSL, the more relevant questions are:
|
||||
|
||||
- which gates are reversible?
|
||||
- which gates are validated by deterministic checks later?
|
||||
- which mistakes silently poison later stages?
|
||||
- which gates can abstain cheaply and retry with more context?
|
||||
|
||||
That is why `whetstone_RSA` should treat deployment quality as a system metric,
|
||||
not a model-only metric.
|
||||
|
||||
## Candidate Gate Inventory
|
||||
|
||||
Immediate WhetstoneDSL gate families:
|
||||
|
||||
- `verification_type`
|
||||
- `confidence_tier`
|
||||
- `prereq_op_selector`
|
||||
- `worker_type`
|
||||
- `gap_class`
|
||||
- `escalation_predictor`
|
||||
- `automatability_strategy`
|
||||
|
||||
These are good initial benchmarks because they already have:
|
||||
|
||||
- clear output enums
|
||||
- architectural value
|
||||
- known downstream consequences
|
||||
- mixed difficulty levels
|
||||
|
||||
## What To Measure
|
||||
|
||||
For each gate:
|
||||
|
||||
- output cardinality
|
||||
- slot count
|
||||
- ambiguity of the input surface
|
||||
- context width required
|
||||
- label boundary sharpness
|
||||
- train/eval accuracy
|
||||
- calibration quality
|
||||
- failure concentration by class
|
||||
- smallest model size that clears target quality
|
||||
|
||||
## Working Hypothesis
|
||||
|
||||
The main predictor will not just be the number of output labels.
|
||||
|
||||
A gate can be 3-way and still difficult if it depends on:
|
||||
|
||||
- long context
|
||||
- latent intent reconstruction
|
||||
- weakly separable labels
|
||||
- hidden state
|
||||
- domain-specific lexical variation
|
||||
|
||||
So the sizing function should be based on effective entropy, not enum size alone.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
If Whetstone can tag nodes, route tasks, select templates, and choose review paths
|
||||
through bounded specialists, then LLM use becomes residual instead of default.
|
||||
|
||||
That would make the AST-first environment cheaper, faster, more local, and easier
|
||||
to retrain per project.
|
||||
Reference in New Issue
Block a user