Initial whetstone_RSA architecture and C++ scaffold
This commit is contained in:
Bill Holcombe
265
docs/adoption_model.md
Normal file
265
docs/adoption_model.md
Normal file
@@ -0,0 +1,265 @@
|
||||
# Adoption Model
|
||||
|
||||
## Purpose
|
||||
|
||||
This document explains where `whetstone_RSA` sits in the broader tool ecosystem and
|
||||
how other systems should decide when to use it.
|
||||
|
||||
The key architectural claim is:
|
||||
|
||||
`whetstone_RSA` belongs directly above deterministic tooling.
|
||||
|
||||
It should be used when a problem is no longer fully deterministic, but the output
|
||||
contract is still bounded enough that open-ended generation is wasteful.
|
||||
|
||||
## Placement In The Stack
|
||||
|
||||
The intended execution stack is:
|
||||
|
||||
```text
|
||||
deterministic
|
||||
↓
|
||||
RSA
|
||||
↓
|
||||
SLM
|
||||
↓
|
||||
LLM
|
||||
↓
|
||||
human
|
||||
```
|
||||
|
||||
This is not just a cost ladder. It is a structural ladder.
|
||||
|
||||
- `deterministic`
|
||||
- same valid input -> same output
|
||||
- formulas, rules, template expansion, exact policies
|
||||
|
||||
- `RSA`
|
||||
- output contract is bounded
|
||||
- inputs may be structured, unstructured, or hybrid
|
||||
- confidence, abstain, and escalation matter
|
||||
|
||||
- `SLM`
|
||||
- output is open-ended but still locally constrained
|
||||
|
||||
- `LLM`
|
||||
- output is open-ended and requires broader synthesis or reasoning
|
||||
|
||||
- `human`
|
||||
- problem is not yet formalized enough to automate safely
|
||||
|
||||
## Why RSA Sits Right Above Deterministic
|
||||
|
||||
RSA is the first non-deterministic layer that still preserves strong structure:
|
||||
|
||||
- bounded outputs
|
||||
- schema validation
|
||||
- abstain support
|
||||
- deterministic downstream execution
|
||||
- easy retraining
|
||||
|
||||
That makes it the natural default immediately after deterministic methods stop being
|
||||
enough.
|
||||
|
||||
In many systems, the wrong jump is being made:
|
||||
|
||||
```text
|
||||
deterministic -> LLM
|
||||
```
|
||||
|
||||
The RSA layer exists to fill the large missing middle.
|
||||
|
||||
## Two Ways A Project Uses RSA
|
||||
|
||||
There are two distinct adoption modes.
|
||||
|
||||
### 1. Runtime Consumer
|
||||
|
||||
A project calls `whetstone_RSA` during execution to make a bounded decision.
|
||||
|
||||
Examples:
|
||||
|
||||
- choose a workflow type
|
||||
- classify a validation tier
|
||||
- select a bounded CAD operation family
|
||||
- route a request to a deterministic tool
|
||||
|
||||
### 2. Architectural Producer
|
||||
|
||||
A project decides that the software it is generating should itself include an RSA
|
||||
gate.
|
||||
|
||||
This is especially relevant for `whetstone_DSL`.
|
||||
|
||||
`whetstone_DSL` should not only call RSA for its own internal bounded decisions.
|
||||
It should also be able to say:
|
||||
|
||||
- this decision surface in the generated software is bounded
|
||||
- deterministic rules are insufficient
|
||||
- LLM use would be wasteful
|
||||
- therefore the generated system should contain an RSA gate here
|
||||
|
||||
That means RSA is both:
|
||||
|
||||
- a runtime library
|
||||
- a target architectural pattern
|
||||
|
||||
## Suitability Assessment
|
||||
|
||||
Projects should be able to assess whether a decision surface should become an RSA
|
||||
gate.
|
||||
|
||||
The suitability question is:
|
||||
|
||||
Should this decision live in deterministic code, an RSA gate, an SLM, an LLM, or a
|
||||
human workflow?
|
||||
|
||||
## Conditions That Favor RSA
|
||||
|
||||
Recommend RSA when most of these are true:
|
||||
|
||||
- output vocabulary is finite or can be made finite
|
||||
- downstream execution is deterministic or template-driven
|
||||
- input space may be structured or unstructured, but is not rule-complete
|
||||
- many inputs map into a small number of actions
|
||||
- confidence matters
|
||||
- abstain or retry is acceptable
|
||||
- domain retraining is practical
|
||||
|
||||
## Conditions That Disfavor RSA
|
||||
|
||||
Do not recommend RSA when:
|
||||
|
||||
- the decision is already deterministic
|
||||
- the output is genuinely open-ended
|
||||
- the label space is unstable and not yet separable from policy
|
||||
- the task is synthesis, not selection or routing
|
||||
- no meaningful abstain or escalation path exists
|
||||
|
||||
## Suitability Output
|
||||
|
||||
The library should eventually support a structured suitability assessment.
|
||||
|
||||
Example conceptual result:
|
||||
|
||||
```text
|
||||
GateSuitability
|
||||
recommended
|
||||
target_layer
|
||||
rationale
|
||||
bounded_output_confidence
|
||||
deterministic_baseline_available
|
||||
label_stability_risk
|
||||
suggested_gate_shape
|
||||
```
|
||||
|
||||
Where `target_layer` is one of:
|
||||
|
||||
- `deterministic`
|
||||
- `rsa`
|
||||
- `slm`
|
||||
- `llm`
|
||||
- `human`
|
||||
|
||||
And `suggested_gate_shape` might be:
|
||||
|
||||
- `binary`
|
||||
- `multiclass`
|
||||
- `factorized`
|
||||
- `hierarchical`
|
||||
|
||||
## Contractual Boundary
|
||||
|
||||
One of the strongest properties of RSA is that it should remain contractual.
|
||||
|
||||
The consuming system should not care exactly which learned mechanism fills the gate,
|
||||
as long as the contract is preserved:
|
||||
|
||||
- bounded schema
|
||||
- confidence
|
||||
- abstain behavior
|
||||
- policy recommendation
|
||||
- traceability
|
||||
|
||||
That means the architecture should remain open to future replacements:
|
||||
|
||||
- transformer specialist
|
||||
- RL policy
|
||||
- classical ML classifier
|
||||
- retrieval-backed scorer
|
||||
- hybrid deterministic + learned controller
|
||||
|
||||
The important thing is not the current model family.
|
||||
The important thing is the gate contract.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
If the contract is stable, the learning mechanism can improve without forcing
|
||||
downstream systems to change their architecture.
|
||||
|
||||
That gives RSA two important advantages:
|
||||
|
||||
- it is light enough to retrain quickly when a domain changes
|
||||
- it is abstract enough to swap in a different decision engine later
|
||||
|
||||
This is a major reason to keep `whetstone_RSA` as a standalone library rather than
|
||||
embedding it into one project's internal logic.
|
||||
|
||||
## Ecosystem Role
|
||||
|
||||
In the current tool ecosystem:
|
||||
|
||||
- `whetstone_RSA`
|
||||
- provides the bounded-decision contract, runtime, diagnosis, and suitability model
|
||||
|
||||
- `whetstone_DSL`
|
||||
- consumes RSA for its own internal bounded decisions
|
||||
- may also generate software that itself contains RSA gates
|
||||
|
||||
- `WHIMP`, `constcad`, and similar systems
|
||||
- consume RSA where design or modeling decisions are bounded but not deterministic
|
||||
|
||||
- `hivemind`
|
||||
- can identify high-ROI entropy surfaces where RSA gates are worth creating
|
||||
|
||||
This avoids circular abstraction problems because:
|
||||
|
||||
- RSA owns the generic bounded-decision mechanism
|
||||
- each tool owns its domain-specific gates
|
||||
- no domain project needs to be the conceptual home of RSA
|
||||
|
||||
## Design Rule
|
||||
|
||||
When a system encounters a decision that is not rule-complete but is still bounded
|
||||
enough to avoid open-ended generation, the default answer should be:
|
||||
|
||||
Try RSA before escalating to SLM or LLM.
|
||||
|
||||
## Ontology Note
|
||||
|
||||
The preferred contract vocabulary for this library is defined in
|
||||
`docs/decision_contract_ontology.md`.
|
||||
|
||||
RSA should be understood through orthogonal fields such as:
|
||||
|
||||
- `input_structure`
|
||||
- `input_modality`
|
||||
- `output_topology`
|
||||
- `output_constraint`
|
||||
- `policy_stability`
|
||||
|
||||
not through vague single-label categories.
|
||||
|
||||
Backend family choice is a separate decision layer. See `docs/backend_selection.md`.
|
||||
|
||||
## Implication For WhetstoneDSL
|
||||
|
||||
`whetstone_DSL` should eventually be able to do both:
|
||||
|
||||
- use RSA during generation
|
||||
- recognize when generated software should include an RSA gate
|
||||
|
||||
That second capability is important.
|
||||
|
||||
It means Whetstone is not only a consumer of the library. It becomes a producer of
|
||||
RSA-native architectures in the software it builds.
|
||||
186
docs/architecture.md
Normal file
186
docs/architecture.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# Architecture
|
||||
|
||||
## Purpose
|
||||
|
||||
`whetstone_RSA` should be a callable library for bounded fuzzy decisions, not a
|
||||
general chat model.
|
||||
|
||||
The reusable unit is not "a small transformer." The reusable unit is:
|
||||
|
||||
- a finite decision space
|
||||
- a compact policy over that space
|
||||
- a deterministic executor behind it
|
||||
|
||||
## Core Runtime
|
||||
|
||||
The library should expose one high-level operation:
|
||||
|
||||
```text
|
||||
decide(input, schema, context) -> decision_result
|
||||
```
|
||||
|
||||
Where:
|
||||
|
||||
- `input` is natural language or a compact structured description
|
||||
- `schema` defines the legal decision space
|
||||
- `context` carries app-specific state
|
||||
|
||||
The runtime returns:
|
||||
|
||||
- `decision_id`
|
||||
- `slots`
|
||||
- `confidence`
|
||||
- `abstain`
|
||||
- `escalation_target`
|
||||
- `trace`
|
||||
|
||||
## Main Components
|
||||
|
||||
### 1. Decision Schema
|
||||
|
||||
Defines the bounded output surface:
|
||||
|
||||
- enum labels
|
||||
- slot definitions
|
||||
- required vs optional fields
|
||||
- validity constraints
|
||||
- allowed escalation targets
|
||||
|
||||
This is the contract that keeps the model narrow.
|
||||
|
||||
### 2. Input Encoder
|
||||
|
||||
Normalizes app inputs into a stable inference shape.
|
||||
|
||||
Examples:
|
||||
|
||||
- plain user text
|
||||
- AST node metadata
|
||||
- taskitem summaries
|
||||
- form descriptions
|
||||
- UI state plus text
|
||||
|
||||
### 3. Policy Layer
|
||||
|
||||
A tiny learned model that scores legal outcomes.
|
||||
|
||||
This may start as a transformer specialist, but the interface should not hard-code
|
||||
one model family. The policy layer should be swappable.
|
||||
|
||||
### 4. Constraint Layer
|
||||
|
||||
Deterministic checks that reject illegal or inconsistent outputs.
|
||||
|
||||
Examples:
|
||||
|
||||
- invalid enum value
|
||||
- missing required slot
|
||||
- impossible field combination
|
||||
- out-of-policy tool choice
|
||||
|
||||
### 5. Executor
|
||||
|
||||
Takes a valid decision and calls the deterministic downstream system:
|
||||
|
||||
- tool invocation
|
||||
- template expansion
|
||||
- AST mutation selection
|
||||
- workflow dispatch
|
||||
- form submission preparation
|
||||
|
||||
### 6. Confidence and Escalation
|
||||
|
||||
Every decision gate needs a native abstain path.
|
||||
|
||||
Outputs should support:
|
||||
|
||||
- `proceed`
|
||||
- `retry_with_context`
|
||||
- `escalate_to_larger_model`
|
||||
- `escalate_to_human`
|
||||
|
||||
### 7. Gate Diagnosis
|
||||
|
||||
The runtime should eventually support gate diagnosis as a first-class library
|
||||
capability.
|
||||
|
||||
This means the library should not only run gates, but also help determine why a
|
||||
gate is underperforming and what restructuring options are plausible.
|
||||
|
||||
See `docs/gate_failure_taxonomy.md`.
|
||||
|
||||
## Library Boundaries
|
||||
|
||||
The core library should stay domain-agnostic.
|
||||
|
||||
Recommended split:
|
||||
|
||||
- `rsa-core`: schemas, inference interface, scoring, confidence, tracing
|
||||
- `rsa-runtime`: model loading, quantization support, inference backends
|
||||
- `rsa-eval`: benchmarks, calibration, error analysis
|
||||
- `rsa-domains-whetstone`: WhetstoneDSL gate schemas and adapters
|
||||
|
||||
## Data Model
|
||||
|
||||
Each gate should be described by metadata that is independent of any one model:
|
||||
|
||||
- `gate_id`
|
||||
- `task_family`
|
||||
- `output_cardinality`
|
||||
- `slot_schema`
|
||||
- `input_modalities`
|
||||
- `requires_world_knowledge`
|
||||
- `requires_stateful_context`
|
||||
- `deterministic_fallback_available`
|
||||
- `label_stability`
|
||||
- `candidate_factorizations`
|
||||
- `quality_target`
|
||||
- `latency_budget_ms`
|
||||
|
||||
This metadata matters because model size should be chosen from gate properties,
|
||||
not from intuition alone.
|
||||
|
||||
## Initial API Sketch
|
||||
|
||||
```text
|
||||
GateDefinition
|
||||
id
|
||||
labels
|
||||
slots
|
||||
constraints
|
||||
escalation_policy
|
||||
|
||||
DecisionRequest
|
||||
gate_id
|
||||
input_text
|
||||
context
|
||||
|
||||
DecisionResult
|
||||
decision_id
|
||||
slots
|
||||
confidence
|
||||
abstain
|
||||
escalation_target
|
||||
trace
|
||||
|
||||
GateDiagnosis
|
||||
primary_failure_class
|
||||
secondary_failure_classes
|
||||
recommended_probes
|
||||
recommended_interventions
|
||||
confidence
|
||||
```
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- open-ended answer generation
|
||||
- internet-grounded knowledge retrieval
|
||||
- replacing deterministic business logic
|
||||
- hiding uncertainty instead of exposing it
|
||||
|
||||
## Guiding Principle
|
||||
|
||||
Intelligence should be concentrated at the routing boundary.
|
||||
|
||||
Once the route is chosen and validated, the rest of the system should become
|
||||
ordinary software.
|
||||
226
docs/backend_selection.md
Normal file
226
docs/backend_selection.md
Normal file
@@ -0,0 +1,226 @@
|
||||
# Backend Selection
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines backend selection as a first-class concern for
|
||||
`whetstone_RSA`.
|
||||
|
||||
The key distinction is:
|
||||
|
||||
- RSA decides whether a problem should be handled as a bounded decision contract
|
||||
- backend selection decides which implementation family should fill that contract
|
||||
|
||||
These are different questions and should not be collapsed together.
|
||||
|
||||
## Core Rule
|
||||
|
||||
Do not define RSA by one backend family.
|
||||
|
||||
RSA is model-agnostic.
|
||||
The backend should be selected based on gate characteristics, deployment needs, and
|
||||
measured results.
|
||||
|
||||
## Decision Layers
|
||||
|
||||
There are three separate decisions:
|
||||
|
||||
### 1. Layer Suitability
|
||||
|
||||
Question:
|
||||
|
||||
Should this problem be:
|
||||
|
||||
- `deterministic`
|
||||
- `rsa`
|
||||
- `slm`
|
||||
- `llm`
|
||||
- `human`
|
||||
|
||||
This is handled by suitability assessment.
|
||||
|
||||
### 2. Gate Shape Selection
|
||||
|
||||
Question:
|
||||
|
||||
If the problem is an RSA candidate, what shape should the gate take?
|
||||
|
||||
- `binary`
|
||||
- `multiclass`
|
||||
- `factorized`
|
||||
- `hierarchical`
|
||||
|
||||
This is handled by gate design and diagnosis.
|
||||
|
||||
### 3. Backend Selection
|
||||
|
||||
Question:
|
||||
|
||||
If the problem is an RSA gate, what implementation family should fill it?
|
||||
|
||||
- `transformer`
|
||||
- `gradient_boosting`
|
||||
- `random_forest`
|
||||
- `bayesian`
|
||||
- `linear_model`
|
||||
- `rl_policy`
|
||||
- `hybrid`
|
||||
|
||||
This document is about the third decision.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Different gates have different data characteristics.
|
||||
|
||||
Examples:
|
||||
|
||||
- `unstructured + textual + bounded`
|
||||
- transformer may be the natural default
|
||||
|
||||
- `structured + numeric + bounded`
|
||||
- gradient boosting, random forest, or linear models may be stronger and cheaper
|
||||
|
||||
- `small data + strong priors`
|
||||
- Bayesian or simpler classifiers may be better
|
||||
|
||||
- `reward-shaped routing with delayed payoff`
|
||||
- RL or bandit-style policies may become useful
|
||||
|
||||
So backend choice should not be hard-coded.
|
||||
|
||||
## What RSA Should Own
|
||||
|
||||
`whetstone_RSA` should own:
|
||||
|
||||
- the gate contract
|
||||
- suitability assessment
|
||||
- diagnosis
|
||||
- policy behavior
|
||||
- the runtime interface for backends
|
||||
|
||||
It should not assume:
|
||||
|
||||
- transformer by default in all cases
|
||||
- one training stack
|
||||
- one benchmark methodology
|
||||
|
||||
## What Backend Selection Should Do
|
||||
|
||||
Backend selection should answer:
|
||||
|
||||
- which backend families are plausible for this gate?
|
||||
- which should be tried first?
|
||||
- which benchmark signals matter?
|
||||
- whether existing model-selection tooling should be used
|
||||
|
||||
## Selection Variables
|
||||
|
||||
Backend recommendation should consider at least these variables:
|
||||
|
||||
### Contract variables
|
||||
|
||||
- `input_structure`
|
||||
- `input_modality`
|
||||
- `output_topology`
|
||||
- `output_constraint`
|
||||
- `policy_stability`
|
||||
|
||||
### Data variables
|
||||
|
||||
- dataset size
|
||||
- class balance
|
||||
- feature sparsity
|
||||
- sequence length
|
||||
- label noise
|
||||
- factorization availability
|
||||
|
||||
### Deployment variables
|
||||
|
||||
- latency budget
|
||||
- memory budget
|
||||
- retraining frequency
|
||||
- calibration needs
|
||||
- explainability needs
|
||||
- online adaptation needs
|
||||
|
||||
### Runtime variables
|
||||
|
||||
- deterministic fallback availability
|
||||
- confidence thresholding support
|
||||
- retry/escalation policy
|
||||
- traceability requirements
|
||||
|
||||
## Suggested Types
|
||||
|
||||
Future implementation should likely include types along these lines:
|
||||
|
||||
```text
|
||||
BackendFamily
|
||||
transformer
|
||||
gradient_boosting
|
||||
random_forest
|
||||
bayesian
|
||||
linear_model
|
||||
rl_policy
|
||||
hybrid
|
||||
```
|
||||
|
||||
```text
|
||||
BackendSuitability
|
||||
gate_id
|
||||
candidate_backends
|
||||
recommended_backend
|
||||
rationale
|
||||
confidence
|
||||
```
|
||||
|
||||
```text
|
||||
BackendBenchmarkResult
|
||||
gate_id
|
||||
backend_family
|
||||
metrics
|
||||
training_cost
|
||||
runtime_cost
|
||||
calibration
|
||||
notes
|
||||
```
|
||||
|
||||
## Use Existing Tools Where Practical
|
||||
|
||||
Backend selection is not a place to reinvent the entire applied ML ecosystem.
|
||||
|
||||
In many cases, existing tools or benchmark harnesses can already compare:
|
||||
|
||||
- random forest
|
||||
- boosted trees
|
||||
- linear models
|
||||
- Naive Bayes
|
||||
- simple neural models
|
||||
|
||||
So `whetstone_RSA` should be willing to consume benchmark results from external
|
||||
model-selection or AutoML-style workflows instead of owning every search strategy
|
||||
itself.
|
||||
|
||||
The important thing is that the result plugs back into the RSA contract layer.
|
||||
|
||||
## Recommended Architectural Boundary
|
||||
|
||||
Keep backend selection adjacent to the RSA core, not inside the core identity.
|
||||
|
||||
That means:
|
||||
|
||||
- `GateSuitability` answers whether RSA is the right container
|
||||
- `GateDiagnosis` helps debug and restructure the gate
|
||||
- `BackendSuitability` recommends what implementation family should fill it
|
||||
|
||||
This separation keeps the architecture clean.
|
||||
|
||||
## Immediate Follow-Up
|
||||
|
||||
This should eventually be reflected in code by:
|
||||
|
||||
- adding a backend family enum
|
||||
- adding backend suitability types
|
||||
- extending the runtime contract so gates do not assume one backend
|
||||
- optionally adding hooks for external benchmark/model-selection tooling
|
||||
|
||||
Until then, this document is the explicit placeholder so the work does not get lost.
|
||||
155
docs/case_studies/whetstone_dsl.md
Normal file
155
docs/case_studies/whetstone_dsl.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Case Study: WhetstoneDSL
|
||||
|
||||
## Why This Is The First Case Study
|
||||
|
||||
WhetstoneDSL is already using the Routed-Specialist Architecture for bounded editor
|
||||
and workflow decisions. That makes it the right proving ground for `whetstone_RSA`.
|
||||
|
||||
Relevant existing work:
|
||||
|
||||
- `/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/HANDOFF-2026-03-28.md`
|
||||
- `/home/bill/Documents/WhetstoneAI_Fabricate/HANDOFF-2026-03-29.md`
|
||||
|
||||
## What Is Already Proven
|
||||
|
||||
The current specialist fleet established that tiny transformers can replace some
|
||||
LLM calls in the MCP pipeline when the outputs are bounded.
|
||||
|
||||
Documented examples:
|
||||
|
||||
- `verification_type`
|
||||
- 5-way classifier
|
||||
- `83.7%` in the initial handoff, later `84.9%` at step 4000 in the current history
|
||||
- `confidence_tier`
|
||||
- 3-way classifier
|
||||
- 100% held-out accuracy
|
||||
- `worker_type`
|
||||
- 4-way classifier
|
||||
- `88.5%` at step 4000 in the current history
|
||||
- `prereq_op`
|
||||
- 4-way classifier
|
||||
- `69.4%` and flat across steps 1000-4000 in the current history
|
||||
- `automatability`
|
||||
- 6-way classifier
|
||||
- `69.6%` and flat across steps 1000-4000 in the current history
|
||||
|
||||
Shared properties:
|
||||
|
||||
- bounded vocabulary
|
||||
- deterministic downstream meaning
|
||||
- AST-first architecture
|
||||
- millisecond-scale inference
|
||||
- ~800KB checkpoint size
|
||||
|
||||
## Important Constraint
|
||||
|
||||
Whetstone is AST-first, not language-first.
|
||||
|
||||
That means RSA gates should generally live at the AST or orchestration layer, not
|
||||
be duplicated per target language. The model should choose bounded structural
|
||||
decisions; deterministic projection handles language output.
|
||||
|
||||
## Current Opportunity
|
||||
|
||||
The current Whetstone specialist work suggests there are at least three gate bands:
|
||||
|
||||
- low-entropy gates that are formula-like and should stay deterministic
|
||||
- medium-entropy gates that fit the ~800KB specialist tier
|
||||
- higher-entropy bounded gates that may need 10x to 16x larger models, better
|
||||
schemas, or multi-stage decomposition
|
||||
|
||||
This project should turn those bands into a measurable sizing policy.
|
||||
|
||||
## Current Snapshot
|
||||
|
||||
The active case study is already informative enough to shape the policy:
|
||||
|
||||
- `confidence_tier` is effectively deterministic in disguise and should be judged
|
||||
as a formula baseline, not as evidence for model capability
|
||||
- `verification_type` and `worker_type` look viable at the tiny tier, but should
|
||||
be evaluated through abstain and retry policy rather than raw accuracy alone
|
||||
- `prereq_op` and `automatability` are currently below an obvious auto-accept bar
|
||||
at the tiny tier
|
||||
|
||||
Just as important, the current larger-model runs are not yet producing accuracy
|
||||
comparisons. The logs show startup on `hidden=512, layers=4`, but the runs abort
|
||||
before any substantive training result is recorded. That means the current frontier
|
||||
is:
|
||||
|
||||
- tiny-tier accuracy is known
|
||||
- large-tier cost is partially visible
|
||||
- large-tier quality is not yet known
|
||||
|
||||
So the correct immediate move is not "chase bigger until better." It is to adopt a
|
||||
deployment policy that measures accepted-risk performance while the larger-tier
|
||||
infrastructure is being stabilized.
|
||||
|
||||
## Interpretation
|
||||
|
||||
This case study strongly suggests that raw held-out accuracy is not enough to drive
|
||||
the roadmap.
|
||||
|
||||
For WhetstoneDSL, the more relevant questions are:
|
||||
|
||||
- which gates are reversible?
|
||||
- which gates are validated by deterministic checks later?
|
||||
- which mistakes silently poison later stages?
|
||||
- which gates can abstain cheaply and retry with more context?
|
||||
|
||||
That is why `whetstone_RSA` should treat deployment quality as a system metric,
|
||||
not a model-only metric.
|
||||
|
||||
## Candidate Gate Inventory
|
||||
|
||||
Immediate WhetstoneDSL gate families:
|
||||
|
||||
- `verification_type`
|
||||
- `confidence_tier`
|
||||
- `prereq_op_selector`
|
||||
- `worker_type`
|
||||
- `gap_class`
|
||||
- `escalation_predictor`
|
||||
- `automatability_strategy`
|
||||
|
||||
These are good initial benchmarks because they already have:
|
||||
|
||||
- clear output enums
|
||||
- architectural value
|
||||
- known downstream consequences
|
||||
- mixed difficulty levels
|
||||
|
||||
## What To Measure
|
||||
|
||||
For each gate:
|
||||
|
||||
- output cardinality
|
||||
- slot count
|
||||
- ambiguity of the input surface
|
||||
- context width required
|
||||
- label boundary sharpness
|
||||
- train/eval accuracy
|
||||
- calibration quality
|
||||
- failure concentration by class
|
||||
- smallest model size that clears target quality
|
||||
|
||||
## Working Hypothesis
|
||||
|
||||
The main predictor will not just be the number of output labels.
|
||||
|
||||
A gate can be 3-way and still difficult if it depends on:
|
||||
|
||||
- long context
|
||||
- latent intent reconstruction
|
||||
- weakly separable labels
|
||||
- hidden state
|
||||
- domain-specific lexical variation
|
||||
|
||||
So the sizing function should be based on effective entropy, not enum size alone.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
If Whetstone can tag nodes, route tasks, select templates, and choose review paths
|
||||
through bounded specialists, then LLM use becomes residual instead of default.
|
||||
|
||||
That would make the AST-first environment cheaper, faster, more local, and easier
|
||||
to retrain per project.
|
||||
290
docs/decision_contract_ontology.md
Normal file
290
docs/decision_contract_ontology.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Decision Contract Ontology
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the contract vocabulary for `whetstone_RSA`.
|
||||
|
||||
The goal is to describe decision problems using orthogonal fields rather than a
|
||||
single overloaded label.
|
||||
|
||||
This is the right approach because `whetstone_RSA` is an applied data science
|
||||
library. The ontology should describe:
|
||||
|
||||
- input shape
|
||||
- output shape
|
||||
- constraint type
|
||||
- policy stability
|
||||
|
||||
instead of relying on vague words like `messy`, `semantic`, or `tacit`.
|
||||
|
||||
## Core Rule
|
||||
|
||||
Do not classify decision problems with one field if multiple independent dimensions
|
||||
matter.
|
||||
|
||||
In particular:
|
||||
|
||||
- `structured` is not the same as `deterministic`
|
||||
- `bounded` is not the same as `discrete`
|
||||
- `continuous` is not the same as `open`
|
||||
- `unstructured` is not the same as `non-deterministic`
|
||||
|
||||
## Recommended Axes
|
||||
|
||||
### 1. `input_structure`
|
||||
|
||||
Describes whether the input arrives in normalized fields or not.
|
||||
|
||||
Allowed values:
|
||||
|
||||
- `structured`
|
||||
- `unstructured`
|
||||
- `hybrid`
|
||||
|
||||
Guidance:
|
||||
|
||||
- `structured`
|
||||
- typed fields, metrics, schemas, vectors, enums, AST packets
|
||||
- `unstructured`
|
||||
- free text, raw documents, rich descriptions, unconstrained traces
|
||||
- `hybrid`
|
||||
- mixed structured and unstructured inputs
|
||||
|
||||
### 2. `input_modality`
|
||||
|
||||
Describes the representational form of the input.
|
||||
|
||||
Allowed values:
|
||||
|
||||
- `textual`
|
||||
- `numeric`
|
||||
- `symbolic`
|
||||
- `multimodal`
|
||||
|
||||
Guidance:
|
||||
|
||||
- `textual`
|
||||
- natural language or tokenized text-like input
|
||||
- `numeric`
|
||||
- vectors, metrics, time series, scalar features
|
||||
- `symbolic`
|
||||
- AST nodes, typed graphs, enums, rule packets
|
||||
- `multimodal`
|
||||
- mixtures such as text plus metrics, geometry plus labels, etc.
|
||||
|
||||
### 3. `output_topology`
|
||||
|
||||
Describes the mathematical form of the output space.
|
||||
|
||||
Allowed values:
|
||||
|
||||
- `discrete`
|
||||
- `continuous`
|
||||
- `hybrid`
|
||||
|
||||
Guidance:
|
||||
|
||||
- `discrete`
|
||||
- classes, enums, branch choices, ranked finite actions
|
||||
- `continuous`
|
||||
- scalar scores, vectors, bounded numeric values
|
||||
- `hybrid`
|
||||
- mixed outputs such as class + score
|
||||
|
||||
### 4. `output_constraint`
|
||||
|
||||
Describes how tightly the output space is constrained.
|
||||
|
||||
Allowed values:
|
||||
|
||||
- `deterministic`
|
||||
- `bounded`
|
||||
- `open`
|
||||
|
||||
Guidance:
|
||||
|
||||
- `deterministic`
|
||||
- rule-complete; same valid input should produce the same output
|
||||
- `bounded`
|
||||
- output must remain inside a defined contract or finite action space
|
||||
- `open`
|
||||
- output is not contract-limited to a finite or strongly bounded space
|
||||
|
||||
### 5. `policy_stability`
|
||||
|
||||
Describes whether the target label or routing policy is stable over time.
|
||||
|
||||
Allowed values:
|
||||
|
||||
- `stable`
|
||||
- `mixed`
|
||||
- `drifting`
|
||||
|
||||
Guidance:
|
||||
|
||||
- `stable`
|
||||
- policy meaning is not expected to move much with tooling or context
|
||||
- `mixed`
|
||||
- some output semantics are stable, some depend on deployment choices
|
||||
- `drifting`
|
||||
- label meaning changes materially as tools, policy, or capability change
|
||||
|
||||
## Why Multiple Fields Matter
|
||||
|
||||
These distinctions are operationally important.
|
||||
|
||||
Examples:
|
||||
|
||||
- `structured + deterministic`
|
||||
- formula engine
|
||||
- validation rule
|
||||
- exact template expansion
|
||||
|
||||
- `structured + bounded + discrete`
|
||||
- policy classifier over metrics
|
||||
- thresholded routing decision
|
||||
- bounded model selection
|
||||
|
||||
- `unstructured + bounded + discrete`
|
||||
- text classification
|
||||
- intent router
|
||||
- tool selector
|
||||
|
||||
- `hybrid + bounded + discrete`
|
||||
- text + metrics -> bounded decision
|
||||
- telemetry + state + text -> policy route
|
||||
|
||||
- `structured + continuous`
|
||||
- score model
|
||||
- optimization cost estimate
|
||||
- risk surface regression
|
||||
|
||||
- `unstructured + open`
|
||||
- open-ended generation
|
||||
- code drafting
|
||||
- narrative synthesis
|
||||
|
||||
## RSA In This Ontology
|
||||
|
||||
RSA should be defined by contract properties, not by one input adjective.
|
||||
|
||||
RSA is most appropriate when:
|
||||
|
||||
- `output_constraint = bounded`
|
||||
- output is not fully `deterministic`
|
||||
- downstream execution is deterministic or strongly structured
|
||||
- confidence, abstain, and escalation matter
|
||||
|
||||
Typical RSA cases:
|
||||
|
||||
- `structured -> bounded`
|
||||
- `unstructured -> bounded`
|
||||
- `hybrid -> bounded`
|
||||
|
||||
Most RSA outputs will also be:
|
||||
|
||||
- `output_topology = discrete`
|
||||
|
||||
But that should not be assumed as a universal rule if bounded continuous or hybrid
|
||||
forms later become useful.
|
||||
|
||||
## What RSA Is Not
|
||||
|
||||
RSA is not:
|
||||
|
||||
- a synonym for transformer
|
||||
- a synonym for RL
|
||||
- a synonym for all learned decision-making
|
||||
- a synonym for all non-deterministic automation
|
||||
|
||||
RSA is specifically a bounded decision contract with policy behavior around:
|
||||
|
||||
- confidence
|
||||
- abstain
|
||||
- retry
|
||||
- escalation
|
||||
- deterministic downstream execution
|
||||
|
||||
## Example Contract Records
|
||||
|
||||
### Example A: Formula Confidence Tier
|
||||
|
||||
```text
|
||||
input_structure = structured
|
||||
input_modality = numeric
|
||||
output_topology = discrete
|
||||
output_constraint = deterministic
|
||||
policy_stability = stable
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- likely not RSA
|
||||
- should probably remain deterministic
|
||||
|
||||
### Example B: Verification Type From Step Text
|
||||
|
||||
```text
|
||||
input_structure = unstructured
|
||||
input_modality = textual
|
||||
output_topology = discrete
|
||||
output_constraint = bounded
|
||||
policy_stability = stable
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- good RSA candidate
|
||||
|
||||
### Example C: Automatability Routing Tier
|
||||
|
||||
```text
|
||||
input_structure = hybrid
|
||||
input_modality = symbolic
|
||||
output_topology = discrete
|
||||
output_constraint = bounded
|
||||
policy_stability = drifting
|
||||
```
|
||||
|
||||
Interpretation:
|
||||
|
||||
- possible RSA candidate
|
||||
- but needs stability analysis before treating it as an ordinary gate
|
||||
|
||||
## Design Consequence
|
||||
|
||||
Suitability assessment should use these fields directly.
|
||||
|
||||
The library should be able to answer:
|
||||
|
||||
- should this stay deterministic?
|
||||
- should this become an RSA gate?
|
||||
- is the output too open for RSA?
|
||||
- is the policy too unstable for safe gate deployment?
|
||||
|
||||
That assessment becomes much clearer when the contract is described with multiple
|
||||
orthogonal fields.
|
||||
|
||||
## Preferred Vocabulary
|
||||
|
||||
Prefer:
|
||||
|
||||
- `structured`
|
||||
- `unstructured`
|
||||
- `hybrid`
|
||||
- `discrete`
|
||||
- `continuous`
|
||||
- `deterministic`
|
||||
- `bounded`
|
||||
- `open`
|
||||
- `stable`
|
||||
- `drifting`
|
||||
|
||||
Avoid as core ontology terms:
|
||||
|
||||
- `messy`
|
||||
- `semantic`
|
||||
- `tacit`
|
||||
|
||||
Those words may still be useful in prose, but they are too vague for the type
|
||||
system.
|
||||
344
docs/diagnosis_api.md
Normal file
344
docs/diagnosis_api.md
Normal file
@@ -0,0 +1,344 @@
|
||||
# Diagnosis API
|
||||
|
||||
## Purpose
|
||||
|
||||
This document sketches the library-facing diagnosis API for `whetstone_RSA`.
|
||||
|
||||
The goal is to make gate diagnosis executable as a runtime capability:
|
||||
|
||||
- ingest gate metadata and evaluation evidence
|
||||
- run standard probes
|
||||
- classify likely failure modes
|
||||
- recommend interventions
|
||||
|
||||
This is still a design artifact, not a final implementation commitment.
|
||||
|
||||
## Design Principles
|
||||
|
||||
- diagnosis should be model-agnostic
|
||||
- probes should be pluggable
|
||||
- recommendations should be structured
|
||||
- the API should work with partial evidence
|
||||
- case studies should inform the design, not hard-code domain assumptions
|
||||
|
||||
## Core Types
|
||||
|
||||
### `GateDefinition`
|
||||
|
||||
Represents the bounded decision contract.
|
||||
|
||||
Suggested fields:
|
||||
|
||||
```text
|
||||
GateDefinition
|
||||
gate_id
|
||||
task_family
|
||||
labels
|
||||
slots
|
||||
constraints
|
||||
risk_tier
|
||||
label_stability
|
||||
deterministic_baseline_available
|
||||
supports_abstain
|
||||
candidate_factorizations
|
||||
metadata
|
||||
```
|
||||
|
||||
### `GateEvidence`
|
||||
|
||||
Represents the observed behavior of a gate at a given model tier or policy setup.
|
||||
|
||||
Suggested fields:
|
||||
|
||||
```text
|
||||
GateEvidence
|
||||
gate_id
|
||||
model_tier
|
||||
raw_accuracy
|
||||
accept_rate
|
||||
accuracy_on_accepted
|
||||
abstain_rate
|
||||
retry_recovery_rate
|
||||
escalation_rate
|
||||
silent_error_rate
|
||||
calibration
|
||||
confusion_summary
|
||||
latency_ms
|
||||
compute_cost
|
||||
notes
|
||||
```
|
||||
|
||||
### `ProbeRequest`
|
||||
|
||||
Specifies a diagnosis run.
|
||||
|
||||
```text
|
||||
ProbeRequest
|
||||
gate_definition
|
||||
evidence
|
||||
enabled_probes
|
||||
policy_context
|
||||
```
|
||||
|
||||
### `ProbeResult`
|
||||
|
||||
Represents the output of one diagnostic probe.
|
||||
|
||||
```text
|
||||
ProbeResult
|
||||
probe_id
|
||||
status
|
||||
signals
|
||||
recommendation_hints
|
||||
confidence
|
||||
```
|
||||
|
||||
### `GateDiagnosis`
|
||||
|
||||
Represents the synthesized diagnosis result.
|
||||
|
||||
```text
|
||||
GateDiagnosis
|
||||
gate_id
|
||||
primary_failure_class
|
||||
secondary_failure_classes
|
||||
supporting_signals
|
||||
recommended_probes
|
||||
recommended_interventions
|
||||
recommended_policy
|
||||
confidence
|
||||
```
|
||||
|
||||
## Failure Class Enum
|
||||
|
||||
Suggested starting enum:
|
||||
|
||||
```text
|
||||
FailureClass
|
||||
capacity_limited_gate
|
||||
factorizable_gate
|
||||
non_stationary_gate
|
||||
missing_context_gate
|
||||
deterministic_disguised_as_ml_gate
|
||||
label_space_mismatch
|
||||
guardrail_limited_gate
|
||||
```
|
||||
|
||||
## Probe Interface
|
||||
|
||||
Each probe should have the same basic contract.
|
||||
|
||||
```text
|
||||
Probe
|
||||
id()
|
||||
supports(gate_definition, evidence) -> bool
|
||||
run(request) -> ProbeResult
|
||||
```
|
||||
|
||||
This lets the library add probes gradually without changing the diagnosis surface.
|
||||
|
||||
## Initial Probe Set
|
||||
|
||||
### `confidence_threshold_probe`
|
||||
|
||||
Purpose:
|
||||
|
||||
Estimate whether the gate is already useful under abstain and confidence thresholding.
|
||||
|
||||
Inputs:
|
||||
|
||||
- confidence-bearing predictions
|
||||
- risk tier
|
||||
- accepted-policy thresholds
|
||||
|
||||
Outputs:
|
||||
|
||||
- projected accept rate
|
||||
- projected accepted precision
|
||||
- silent error estimate
|
||||
- guardrail suitability hint
|
||||
|
||||
### `factorization_probe`
|
||||
|
||||
Purpose:
|
||||
|
||||
Estimate whether a weak multiclass gate has a plausible lower-entropy factorization.
|
||||
|
||||
Inputs:
|
||||
|
||||
- label schema
|
||||
- confusion summary
|
||||
- candidate factorization metadata if available
|
||||
|
||||
Outputs:
|
||||
|
||||
- factorization plausibility score
|
||||
- candidate sub-gate structure
|
||||
- comparison hint versus flat multiclass form
|
||||
|
||||
### `deterministic_rule_probe`
|
||||
|
||||
Purpose:
|
||||
|
||||
Estimate whether the gate should actually live in ordinary software.
|
||||
|
||||
Inputs:
|
||||
|
||||
- deterministic baseline metadata
|
||||
- structured features if available
|
||||
- current gate evidence
|
||||
|
||||
Outputs:
|
||||
|
||||
- deterministic plausibility score
|
||||
- recommendation to replace, hybridize, or keep learned routing
|
||||
|
||||
### `schema_stability_probe`
|
||||
|
||||
Purpose:
|
||||
|
||||
Estimate whether the labels are ontology-stable or deployment-policy-dependent.
|
||||
|
||||
Inputs:
|
||||
|
||||
- label definitions
|
||||
- versioned notes or temporal evaluation evidence
|
||||
- capability metadata
|
||||
|
||||
Outputs:
|
||||
|
||||
- stability rating
|
||||
- ontology-vs-policy warning
|
||||
- recommendation to split labels or add a policy layer
|
||||
|
||||
### `context_width_probe`
|
||||
|
||||
Purpose:
|
||||
|
||||
Estimate whether weak performance is caused by missing context rather than missing
|
||||
capacity.
|
||||
|
||||
Inputs:
|
||||
|
||||
- input modality metadata
|
||||
- evidence across different context budgets if available
|
||||
|
||||
Outputs:
|
||||
|
||||
- context sensitivity score
|
||||
- minimum context recommendation
|
||||
|
||||
### `confusion_structure_probe`
|
||||
|
||||
Purpose:
|
||||
|
||||
Classify whether the error pattern is adjacent, hierarchical, compositional, or
|
||||
diffuse.
|
||||
|
||||
Inputs:
|
||||
|
||||
- confusion summary
|
||||
- label metadata
|
||||
|
||||
Outputs:
|
||||
|
||||
- confusion topology
|
||||
- merge, split, or hierarchy recommendation
|
||||
|
||||
## Diagnosis Flow
|
||||
|
||||
The library should support a default synthesis flow:
|
||||
|
||||
1. Validate gate metadata.
|
||||
2. Run applicable probes.
|
||||
3. Aggregate probe signals.
|
||||
4. Rank likely failure classes.
|
||||
5. Emit structured intervention recommendations.
|
||||
|
||||
The result should not be a paragraph. It should be machine-usable.
|
||||
|
||||
## Intervention Enum
|
||||
|
||||
Suggested intervention vocabulary:
|
||||
|
||||
```text
|
||||
Intervention
|
||||
keep_current_gate
|
||||
deploy_with_guardrails
|
||||
raise_confidence_threshold
|
||||
add_retry_with_context
|
||||
add_deterministic_prepass
|
||||
replace_with_deterministic_logic
|
||||
factor_into_subgates
|
||||
switch_to_hierarchical_gate
|
||||
revise_label_schema
|
||||
enrich_input_context
|
||||
evaluate_larger_model_tier
|
||||
```
|
||||
|
||||
## Policy Recommendation Shape
|
||||
|
||||
Suggested policy structure:
|
||||
|
||||
```text
|
||||
PolicyRecommendation
|
||||
deploy_mode
|
||||
threshold_hint
|
||||
abstain_required
|
||||
retry_strategy
|
||||
escalation_targets
|
||||
```
|
||||
|
||||
Where `deploy_mode` might be:
|
||||
|
||||
- `auto_accept`
|
||||
- `guarded_accept`
|
||||
- `abstain_first`
|
||||
- `research_only`
|
||||
|
||||
## Logging Requirements
|
||||
|
||||
If diagnosis is going to work over time, live gate runs need structured logging.
|
||||
|
||||
Minimum useful fields:
|
||||
|
||||
```text
|
||||
GateRunLog
|
||||
gate_id
|
||||
model_tier
|
||||
input_signature
|
||||
confidence
|
||||
accepted
|
||||
abstained
|
||||
retry_used
|
||||
escalation_target
|
||||
final_outcome
|
||||
deterministic_check_outcome
|
||||
override_outcome
|
||||
timestamp
|
||||
```
|
||||
|
||||
## Case-Study Mapping
|
||||
|
||||
The case study suggests how diagnosis should behave:
|
||||
|
||||
- a gate like `confidence_tier` should trigger strong
|
||||
`deterministic_disguised_as_ml_gate` signals
|
||||
- a gate like `verification_type` or `worker_type` may resolve as
|
||||
`guardrail_limited_gate`
|
||||
- a gate like `prereq_op` should make `factorization_probe` a prominent candidate
|
||||
- a gate like `automatability` should make `schema_stability_probe` prominent
|
||||
|
||||
These mappings are examples, not hard-coded rules.
|
||||
|
||||
## Near-Term Value
|
||||
|
||||
Even before implementation, this API sketch clarifies what the library must preserve:
|
||||
|
||||
- gate metadata
|
||||
- evidence summaries
|
||||
- structured probe results
|
||||
- structured interventions
|
||||
|
||||
That keeps `whetstone_RSA` pointed toward a diagnosable routing system rather than
|
||||
just a pile of tiny models.
|
||||
114
docs/entropy_model_sizing.md
Normal file
114
docs/entropy_model_sizing.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# Entropy To Model Sizing
|
||||
|
||||
## Goal
|
||||
|
||||
Estimate the smallest model tier that can fill a given bounded decision gate with
|
||||
acceptable accuracy and calibration.
|
||||
|
||||
This project needs a practical rule of the form:
|
||||
|
||||
```text
|
||||
gate entropy score -> recommended model tier
|
||||
```
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Right now model choice is too manual:
|
||||
|
||||
- an ~800KB specialist works for some gates
|
||||
- some gates appear to exceed that capacity
|
||||
- the next attempts are 10x and 16x larger
|
||||
|
||||
That is directionally right, but it should become measurable.
|
||||
|
||||
## Working Definition
|
||||
|
||||
`Gate entropy` is not just Shannon entropy over labels.
|
||||
|
||||
For this project, it should mean the difficulty of mapping available input context
|
||||
onto the correct bounded output under realistic variation.
|
||||
|
||||
## Proposed Factors
|
||||
|
||||
Each gate gets a score assembled from observable features:
|
||||
|
||||
- `output_cardinality`
|
||||
- `class_balance`
|
||||
- `input_length_distribution`
|
||||
- `context_width_required`
|
||||
- `slot_count`
|
||||
- `slot_interdependence`
|
||||
- `label_boundary_fuzziness`
|
||||
- `lexical_variation`
|
||||
- `hidden_state_dependence`
|
||||
- `world_knowledge_dependence`
|
||||
- `error_cost`
|
||||
- `abstain_availability`
|
||||
|
||||
Some of these increase model demand directly. Others change the acceptable
|
||||
confidence threshold and escalation policy.
|
||||
|
||||
## First Experimental Plan
|
||||
|
||||
For each Whetstone gate:
|
||||
|
||||
1. Define a fixed training and evaluation set.
|
||||
2. Train multiple model tiers on the same gate.
|
||||
3. Measure accuracy, calibration, confusion concentration, and latency.
|
||||
4. Record the smallest tier that clears the target.
|
||||
|
||||
Suggested initial tiers:
|
||||
|
||||
- deterministic baseline when possible
|
||||
- tiny specialist baseline: current ~213K-param / ~800KB tier
|
||||
- medium specialist: ~10x current size
|
||||
- large specialist: ~16x current size
|
||||
|
||||
## Deliverable
|
||||
|
||||
Build a baseline table like:
|
||||
|
||||
```text
|
||||
gate_id | entropy_score | smallest_passing_tier | accuracy | ece | latency_ms
|
||||
```
|
||||
|
||||
Then fit a first-pass policy:
|
||||
|
||||
```text
|
||||
if entropy <= A -> deterministic
|
||||
if A < entropy <= B -> tiny specialist
|
||||
if B < entropy <= C -> medium specialist
|
||||
if entropy > C -> large specialist or escalate
|
||||
```
|
||||
|
||||
## Important Caveat
|
||||
|
||||
A larger model should not be the only answer.
|
||||
|
||||
Before moving up a tier, the evaluation should check:
|
||||
|
||||
- is the schema wrong?
|
||||
- is the label set under-enumerated?
|
||||
- is important context missing?
|
||||
- are we forcing a fuzzy gate where a deterministic pre-pass should exist?
|
||||
|
||||
Model size should increase only after the gate definition itself is defensible.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which entropy factors are predictive enough to automate tier selection?
|
||||
- Is calibration a better gating signal than raw accuracy for deployment?
|
||||
- Can some gates be factorized into two smaller gates instead of one larger model?
|
||||
- Does AST-native input encoding reduce effective entropy compared with raw text?
|
||||
- Can Hivemind use failure and abstention data to identify where new deterministic
|
||||
tools should be built?
|
||||
|
||||
## Connection To Hivemind
|
||||
|
||||
This is where the projects meet:
|
||||
|
||||
- Hivemind identifies high-entropy surfaces with poor deterministic support
|
||||
- `whetstone_RSA` tries to collapse some of those surfaces into bounded gates
|
||||
- residual high-entropy zones become candidates for larger models or new tools
|
||||
|
||||
That makes entropy estimation useful for both deployment and roadmap planning.
|
||||
227
docs/evaluation_policy.md
Normal file
227
docs/evaluation_policy.md
Normal file
@@ -0,0 +1,227 @@
|
||||
# Evaluation Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines how bounded decision gates should be evaluated inside
|
||||
`whetstone_RSA`.
|
||||
|
||||
The central rule is:
|
||||
|
||||
Do not optimize for raw accuracy in isolation.
|
||||
|
||||
Optimize for final correctness at acceptable latency and cost, while keeping
|
||||
silent wrong decisions below the gate's risk budget.
|
||||
|
||||
## Why Raw Accuracy Is Insufficient
|
||||
|
||||
A gate with `70%` raw top-1 accuracy may still be useful if:
|
||||
|
||||
- it handles easy cases confidently
|
||||
- it abstains on risky cases
|
||||
- retries recover a meaningful share of the abstentions
|
||||
- downstream deterministic checks catch many residual mistakes
|
||||
|
||||
A gate with `97%` raw accuracy may still be a poor deployment choice if:
|
||||
|
||||
- it requires 10x to 16x model size
|
||||
- latency becomes unacceptable
|
||||
- it blocks parallel deployment
|
||||
- the extra accuracy only reduces low-cost mistakes
|
||||
|
||||
For bounded systems, the real question is:
|
||||
|
||||
What is the expected cost of using this gate in the pipeline?
|
||||
|
||||
## Required Metrics
|
||||
|
||||
Every gate evaluation should report these metrics:
|
||||
|
||||
- `raw_accuracy`
|
||||
- `accept_rate`
|
||||
- `accuracy_on_accepted`
|
||||
- `abstain_rate`
|
||||
- `retry_recovery_rate`
|
||||
- `escalation_rate`
|
||||
- `silent_error_rate`
|
||||
- `avg_latency_ms`
|
||||
- `avg_compute_cost`
|
||||
|
||||
Optional but recommended:
|
||||
|
||||
- calibration error
|
||||
- confusion concentration
|
||||
- class-weighted error
|
||||
- deterministic-check catch rate
|
||||
- rollback or repair rate
|
||||
|
||||
## Definitions
|
||||
|
||||
### Raw Accuracy
|
||||
|
||||
Top-1 accuracy on the full evaluation set with no abstention policy.
|
||||
|
||||
Useful for research. Not enough for deployment.
|
||||
|
||||
### Accept Rate
|
||||
|
||||
Fraction of inputs the gate handles automatically without abstaining.
|
||||
|
||||
### Accuracy On Accepted
|
||||
|
||||
Accuracy measured only on predictions that pass the acceptance threshold.
|
||||
|
||||
This is often more important than raw accuracy.
|
||||
|
||||
### Retry Recovery Rate
|
||||
|
||||
Fraction of abstained or low-confidence cases that become correct after:
|
||||
|
||||
- adding richer context
|
||||
- running a deterministic pre-pass
|
||||
- using a larger specialist
|
||||
|
||||
### Silent Error Rate
|
||||
|
||||
Wrong decisions that are accepted automatically and are not caught by later
|
||||
deterministic checks before they can do damage.
|
||||
|
||||
This is usually the key deployment risk metric.
|
||||
|
||||
## Gate Risk Tiers
|
||||
|
||||
Every gate should be assigned a risk tier before model sizing decisions are made.
|
||||
|
||||
### Tier 1: Low Risk
|
||||
|
||||
Characteristics:
|
||||
|
||||
- mistakes are easy to reverse
|
||||
- downstream deterministic checks exist
|
||||
- retries are cheap
|
||||
- wrong branch selection does not poison later stages
|
||||
|
||||
Policy:
|
||||
|
||||
- lower raw accuracy can be acceptable
|
||||
- moderate abstain rate is fine
|
||||
- optimize for cheap throughput
|
||||
|
||||
### Tier 2: Medium Risk
|
||||
|
||||
Characteristics:
|
||||
|
||||
- mistakes cause wasted work or moderate workflow churn
|
||||
- some downstream correction exists, but not always
|
||||
- retries have non-trivial cost
|
||||
|
||||
Policy:
|
||||
|
||||
- accepted predictions need stronger precision
|
||||
- confidence thresholds should be conservative
|
||||
- escalation should be available
|
||||
|
||||
### Tier 3: High Risk
|
||||
|
||||
Characteristics:
|
||||
|
||||
- mistakes silently corrupt planning or structure
|
||||
- correction is costly or delayed
|
||||
- wrong decisions can contaminate later stages
|
||||
|
||||
Policy:
|
||||
|
||||
- silent error rate must be very low
|
||||
- abstain aggressively
|
||||
- escalate early
|
||||
- raw accuracy matters less than safe acceptance behavior
|
||||
|
||||
## Decision Rule
|
||||
|
||||
For a given gate, accept a smaller model if it wins on the system frontier:
|
||||
|
||||
- lower average cost
|
||||
- lower or equal silent error rate
|
||||
- acceptable final correctness after retry and escalation
|
||||
- acceptable latency
|
||||
|
||||
Do not move to a larger model just because it raises raw accuracy.
|
||||
|
||||
Move to a larger model only if:
|
||||
|
||||
- silent errors remain too high after thresholding
|
||||
- retries do not recover enough cases
|
||||
- the gate is structurally valid and still under-capacity
|
||||
|
||||
## Pre-Scale Checks
|
||||
|
||||
Before scaling a gate from tiny to medium or large, check:
|
||||
|
||||
- is the output schema well-defined?
|
||||
- are the labels under-enumerated?
|
||||
- are there ambiguous labels that should be split?
|
||||
- is important context missing from the input encoding?
|
||||
- can the gate be decomposed into two simpler gates?
|
||||
- should part of the logic be deterministic instead?
|
||||
|
||||
Use `docs/gate_failure_taxonomy.md` to classify which of these checks should be
|
||||
prioritized for a given weak gate.
|
||||
|
||||
If any of those fail, bigger models may just hide a design problem.
|
||||
|
||||
## Recommended Evaluation Loop
|
||||
|
||||
1. Train and measure raw accuracy.
|
||||
2. Calibrate confidence thresholds.
|
||||
3. Measure `accuracy_on_accepted` and `silent_error_rate`.
|
||||
4. Add deterministic checks and retry path.
|
||||
5. Measure final correctness and average cost.
|
||||
6. Only then compare larger model tiers.
|
||||
|
||||
## Working Heuristic
|
||||
|
||||
Use this default posture until domain-specific policy replaces it:
|
||||
|
||||
- low-risk gates: a base model in the `70-85%` range may still be deployable with
|
||||
abstain and retry
|
||||
- medium-risk gates: target stronger accepted precision, not just higher raw accuracy
|
||||
- high-risk gates: prioritize low silent-error rate over broad auto-accept coverage
|
||||
|
||||
These are not hard thresholds. They are a reminder that the deployment objective is
|
||||
system utility, not benchmark vanity.
|
||||
|
||||
## WhetstoneDSL Case Study
|
||||
|
||||
Current tiny-tier WhetstoneDSL results already show why this policy is needed:
|
||||
|
||||
- `confidence_tier` reaches `100%`, but is essentially a formula task
|
||||
- `verification_type` reaches roughly `85%`
|
||||
- `worker_type` reaches roughly `88.5%`
|
||||
- `prereq_op` is flat at roughly `69.4%`
|
||||
- `automatability` is flat at roughly `69.6%`
|
||||
|
||||
The correct interpretation is not:
|
||||
|
||||
- "`69%` is bad, so scale immediately"
|
||||
|
||||
The correct interpretation is:
|
||||
|
||||
- determine whether those `69%` gates can be made safe with acceptance thresholds,
|
||||
decomposition, retries, and deterministic guards
|
||||
- then decide whether larger specialists are economically justified
|
||||
|
||||
## What To Log
|
||||
|
||||
Each live decision should eventually log:
|
||||
|
||||
- gate id
|
||||
- model tier
|
||||
- confidence
|
||||
- accepted or abstained
|
||||
- retry used or not
|
||||
- escalation target
|
||||
- final chosen action
|
||||
- deterministic validation outcome
|
||||
- final human or system override outcome
|
||||
|
||||
That turns deployment into a continuous training and policy dataset instead of a
|
||||
one-time benchmark.
|
||||
391
docs/gate_failure_taxonomy.md
Normal file
391
docs/gate_failure_taxonomy.md
Normal file
@@ -0,0 +1,391 @@
|
||||
# Gate Failure Taxonomy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines a first-pass taxonomy for why bounded decision gates fail and
|
||||
how `whetstone_RSA` should reason about those failures.
|
||||
|
||||
The goal is to make gate diagnosis a library capability rather than an ad hoc human
|
||||
judgment call.
|
||||
|
||||
This taxonomy is motivated by the `whetstone_DSL` case study, but it is written as
|
||||
an RSA-library abstraction.
|
||||
|
||||
## Core Claim
|
||||
|
||||
When a bounded gate performs poorly, the default answer should not be "use a larger
|
||||
model."
|
||||
|
||||
The library should first determine what kind of failure is happening.
|
||||
|
||||
Different failure types require different responses:
|
||||
|
||||
- some need more capacity
|
||||
- some need better confidence policy
|
||||
- some should be decomposed
|
||||
- some should become deterministic
|
||||
- some have unstable labels and should not be treated as ordinary classifiers
|
||||
|
||||
## Failure Classes
|
||||
|
||||
### 1. `capacity_limited_gate`
|
||||
|
||||
Description:
|
||||
|
||||
The gate is well-defined and reasonably stationary, but the current model tier does
|
||||
not have enough capacity to separate the decision surface reliably.
|
||||
|
||||
Characteristic symptoms:
|
||||
|
||||
- performance improves with training but plateaus below target
|
||||
- label boundaries appear coherent under manual inspection
|
||||
- confusion is broad rather than concentrated in one structural pattern
|
||||
- richer models or larger context are plausible fixes
|
||||
|
||||
Likely root causes:
|
||||
|
||||
- too little model capacity
|
||||
- too little representational depth
|
||||
- long-range dependencies exceed current context handling
|
||||
- input compression is too lossy for the chosen tier
|
||||
|
||||
Recommended probes:
|
||||
|
||||
- larger-model comparison
|
||||
- context-width probe
|
||||
- calibration probe
|
||||
- feature ablation probe
|
||||
|
||||
Recommended interventions:
|
||||
|
||||
- move to a larger specialist tier
|
||||
- add better structured features
|
||||
- use a richer encoder while preserving the same gate schema
|
||||
|
||||
### 2. `factorizable_gate`
|
||||
|
||||
Description:
|
||||
|
||||
The gate is represented as one multiclass decision, but the actual semantics are a
|
||||
composition of smaller independent or semi-independent decisions.
|
||||
|
||||
Characteristic symptoms:
|
||||
|
||||
- flat multiclass performance despite apparently simple semantics
|
||||
- classes can be expressed as Cartesian products or hierarchical branches
|
||||
- confusion concentrates between composite labels
|
||||
- downstream action is itself assembled from multiple sub-decisions
|
||||
|
||||
Likely root causes:
|
||||
|
||||
- a product decision has been collapsed into one label
|
||||
- one model is being asked to infer multiple policy axes at once
|
||||
- the current evaluation obscures easier sub-structure
|
||||
|
||||
Recommended probes:
|
||||
|
||||
- factorization probe
|
||||
- hierarchical-gate probe
|
||||
- per-axis confidence probe
|
||||
|
||||
Recommended interventions:
|
||||
|
||||
- split into multiple smaller gates
|
||||
- add a deterministic combiner over sub-gates
|
||||
- replace flat multiclass routing with hierarchical routing
|
||||
|
||||
### 3. `non_stationary_gate`
|
||||
|
||||
Description:
|
||||
|
||||
The label semantics change over time because they depend on evolving tooling,
|
||||
policies, or environment state rather than a stable ontology.
|
||||
|
||||
Characteristic symptoms:
|
||||
|
||||
- labels become stale as the surrounding system improves
|
||||
- class definitions drift without changing the input domain
|
||||
- retraining is needed because the target policy moved, not because the model forgot
|
||||
- disagreements cluster at capability boundaries rather than lexical boundaries
|
||||
|
||||
Likely root causes:
|
||||
|
||||
- output labels encode current deployment policy
|
||||
- the gate depends on tool availability or maturity
|
||||
- class meaning is roadmap-dependent
|
||||
|
||||
Recommended probes:
|
||||
|
||||
- schema-stability probe
|
||||
- temporal re-label probe
|
||||
- policy-variant probe
|
||||
|
||||
Recommended interventions:
|
||||
|
||||
- separate stable ontology fields from changing policy fields
|
||||
- turn part of the decision into a policy layer above the model
|
||||
- use hierarchical routing where the first stage predicts a stable property and a
|
||||
later stage applies current deployment policy
|
||||
|
||||
### 4. `missing_context_gate`
|
||||
|
||||
Description:
|
||||
|
||||
The gate might be learnable, but the current input encoding omits the context needed
|
||||
to make the decision safely.
|
||||
|
||||
Characteristic symptoms:
|
||||
|
||||
- short-text performance is poor but humans request surrounding context
|
||||
- errors cluster on inputs that depend on hidden project or workflow state
|
||||
- adding metadata is expected to help more than scaling parameters
|
||||
- the same surface text would require different answers in different contexts
|
||||
|
||||
Likely root causes:
|
||||
|
||||
- important state is absent from the input
|
||||
- text summaries collapse policy-critical structure
|
||||
- the gate implicitly depends on project metadata, not just user language
|
||||
|
||||
Recommended probes:
|
||||
|
||||
- context-width probe
|
||||
- metadata augmentation probe
|
||||
- stateful-context probe
|
||||
|
||||
Recommended interventions:
|
||||
|
||||
- enrich inputs with structured metadata
|
||||
- separate text understanding from stateful decision logic
|
||||
- require abstention when needed context is unavailable
|
||||
|
||||
### 5. `deterministic_disguised_as_ml_gate`
|
||||
|
||||
Description:
|
||||
|
||||
The gate looks statistical at first glance, but the output is largely determined by
|
||||
known rules or a stable formula.
|
||||
|
||||
Characteristic symptoms:
|
||||
|
||||
- very high performance with synthetic or structural encodings
|
||||
- human explanation reduces to a crisp rule
|
||||
- model errors mostly reflect encoding mismatch, not true ambiguity
|
||||
- downstream users already trust a formula more than the classifier
|
||||
|
||||
Likely root causes:
|
||||
|
||||
- a deterministic rule was not implemented yet
|
||||
- the learned model is being used as a proxy for explicit policy
|
||||
- training data was easier to generate than writing the rule
|
||||
|
||||
Recommended probes:
|
||||
|
||||
- deterministic-rule probe
|
||||
- symbolic baseline probe
|
||||
- feature-rule extraction probe
|
||||
|
||||
Recommended interventions:
|
||||
|
||||
- replace with deterministic logic
|
||||
- keep the model only as a convenience wrapper over explicit rules
|
||||
- use the learned gate only when the rule inputs are partially missing
|
||||
|
||||
### 6. `label_space_mismatch`
|
||||
|
||||
Description:
|
||||
|
||||
The gate's labels are underspecified, overlapping, misordered, or otherwise not
|
||||
aligned with the actual decision boundary the system needs.
|
||||
|
||||
Characteristic symptoms:
|
||||
|
||||
- confusion clusters around neighboring or semantically overlapping labels
|
||||
- multiple labels seem equally valid for the same input
|
||||
- label guidelines are hard to explain concisely
|
||||
- accuracy does not reflect practical usefulness because some "mistakes" are close
|
||||
|
||||
Likely root causes:
|
||||
|
||||
- labels mix ontology and policy
|
||||
- labels are too coarse or too fine
|
||||
- adjacent classes are not operationally distinct
|
||||
|
||||
Recommended probes:
|
||||
|
||||
- confusion-structure probe
|
||||
- class-merge probe
|
||||
- hierarchical-label probe
|
||||
|
||||
Recommended interventions:
|
||||
|
||||
- merge or split labels
|
||||
- add hierarchy to the label space
|
||||
- redefine evaluation so high-cost confusions are distinguished from near-miss ones
|
||||
|
||||
### 7. `guardrail_limited_gate`
|
||||
|
||||
Description:
|
||||
|
||||
The gate is good enough to be useful, but not good enough to be trusted without a
|
||||
deployment policy around it.
|
||||
|
||||
Characteristic symptoms:
|
||||
|
||||
- raw accuracy is moderate but confidence is informative
|
||||
- abstention can isolate hard cases
|
||||
- downstream checks can catch many residual errors
|
||||
- accepted predictions are much stronger than overall predictions
|
||||
|
||||
Likely root causes:
|
||||
|
||||
- the model is solving an economically useful subset of the problem
|
||||
- deployment policy has not yet been calibrated
|
||||
|
||||
Recommended probes:
|
||||
|
||||
- confidence-threshold probe
|
||||
- abstain-coverage probe
|
||||
- retry-recovery probe
|
||||
- deterministic-check catch-rate probe
|
||||
|
||||
Recommended interventions:
|
||||
|
||||
- deploy with guardrails
|
||||
- set confidence thresholds by risk tier
|
||||
- add retry and escalation paths
|
||||
|
||||
## Diagnostic Probes
|
||||
|
||||
The taxonomy is only useful if the library has standard probes.
|
||||
|
||||
### `confidence_threshold_probe`
|
||||
|
||||
Question:
|
||||
|
||||
Does the gate become safe and useful when low-confidence cases abstain?
|
||||
|
||||
Outputs:
|
||||
|
||||
- accept rate
|
||||
- accuracy on accepted
|
||||
- silent error estimate
|
||||
|
||||
### `factorization_probe`
|
||||
|
||||
Question:
|
||||
|
||||
Can a weak multiclass gate be re-expressed as multiple smaller gates with a better
|
||||
combined policy?
|
||||
|
||||
Outputs:
|
||||
|
||||
- candidate factorization structure
|
||||
- estimated per-axis difficulty
|
||||
- combined accuracy or accepted-precision comparison
|
||||
|
||||
### `deterministic_rule_probe`
|
||||
|
||||
Question:
|
||||
|
||||
Is the gate mostly encoding a crisp rule that should live in ordinary software?
|
||||
|
||||
Outputs:
|
||||
|
||||
- symbolic baseline score
|
||||
- overlap between model errors and rule violations
|
||||
|
||||
### `schema_stability_probe`
|
||||
|
||||
Question:
|
||||
|
||||
Are the labels stable over time, or do they move with deployment policy and tool
|
||||
availability?
|
||||
|
||||
Outputs:
|
||||
|
||||
- stability rating
|
||||
- ontology-vs-policy split recommendation
|
||||
|
||||
### `context_width_probe`
|
||||
|
||||
Question:
|
||||
|
||||
Is performance limited by missing project or workflow context?
|
||||
|
||||
Outputs:
|
||||
|
||||
- delta from richer context
|
||||
- recommended minimum context set
|
||||
|
||||
### `confusion_structure_probe`
|
||||
|
||||
Question:
|
||||
|
||||
Are the observed errors random, adjacent, hierarchical, or compositional?
|
||||
|
||||
Outputs:
|
||||
|
||||
- confusion topology
|
||||
- merge, split, or hierarchy recommendation
|
||||
|
||||
## Recommended Response Order
|
||||
|
||||
The library should approach gate triage in this order:
|
||||
|
||||
1. Check whether the gate is deterministic in disguise.
|
||||
2. Check whether the label space is stable and coherent.
|
||||
3. Check whether the gate is factorizable.
|
||||
4. Check whether critical context is missing.
|
||||
5. Check whether a guarded deployment policy is already sufficient.
|
||||
6. Only then treat scale as the primary lever.
|
||||
|
||||
## Library Implications
|
||||
|
||||
To support this taxonomy, `whetstone_RSA` should eventually represent each gate with
|
||||
enough metadata for diagnosis.
|
||||
|
||||
Suggested gate metadata:
|
||||
|
||||
- `gate_id`
|
||||
- `label_schema`
|
||||
- `label_stability`
|
||||
- `task_family`
|
||||
- `input_modalities`
|
||||
- `supports_abstain`
|
||||
- `deterministic_baseline_available`
|
||||
- `candidate_factorizations`
|
||||
- `risk_tier`
|
||||
|
||||
Suggested diagnostic result shape:
|
||||
|
||||
```text
|
||||
GateDiagnosis
|
||||
primary_failure_class
|
||||
secondary_failure_classes
|
||||
supporting_signals
|
||||
recommended_probes
|
||||
recommended_interventions
|
||||
confidence
|
||||
```
|
||||
|
||||
## Case-Study Motivators
|
||||
|
||||
The case study motivates this taxonomy, but does not constrain it.
|
||||
|
||||
- A gate like `prereq_op` suggests `factorizable_gate`
|
||||
- A gate like `automatability` suggests `non_stationary_gate`
|
||||
- A gate like `confidence_tier` suggests `deterministic_disguised_as_ml_gate`
|
||||
- A gate like `verification_type` or `worker_type` may become
|
||||
`guardrail_limited_gate`
|
||||
|
||||
These examples are evidence of recurring patterns, not bespoke exceptions.
|
||||
|
||||
## Summary
|
||||
|
||||
The main architectural insight is simple:
|
||||
|
||||
Bounded-gate failure is not one problem.
|
||||
|
||||
If the RSA library can classify failure types before scaling models blindly, it will
|
||||
be cheaper, safer, and easier to generalize across domains.
|
||||
171
docs/gate_rubric.md
Normal file
171
docs/gate_rubric.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# Gate Rubric
|
||||
|
||||
## Purpose
|
||||
|
||||
This document gives a provisional deployment rubric for the current WhetstoneDSL
|
||||
specialist gates.
|
||||
|
||||
The goal is not to rank models by benchmark score. The goal is to decide, per gate:
|
||||
|
||||
- deploy now
|
||||
- deploy with guardrails
|
||||
- wait for larger-tier comparison
|
||||
- redesign the gate before scaling
|
||||
|
||||
## Rubric Buckets
|
||||
|
||||
### Deploy Now
|
||||
|
||||
Use when:
|
||||
|
||||
- the gate is effectively deterministic or near-deterministic
|
||||
- wrong decisions are easy to catch or reverse
|
||||
- the current model already clears the practical quality bar
|
||||
|
||||
### Deploy With Guardrails
|
||||
|
||||
Use when:
|
||||
|
||||
- the gate is useful at the current size
|
||||
- raw accuracy is not enough for blind auto-accept
|
||||
- confidence thresholding, retry, or deterministic checks can contain risk
|
||||
|
||||
### Wait For Large Run
|
||||
|
||||
Use when:
|
||||
|
||||
- the current tiny-tier result is too weak for safe deployment
|
||||
- the gate definition appears defensible
|
||||
- larger-tier quality data is actively being generated
|
||||
|
||||
### Redesign Before Scaling
|
||||
|
||||
Use when:
|
||||
|
||||
- the label space appears structurally muddy
|
||||
- the gate may really be multiple smaller gates
|
||||
- important input context is missing
|
||||
- bigger models would mostly compensate for a bad formulation
|
||||
|
||||
## Current WhetstoneDSL Gates
|
||||
|
||||
### `confidence_tier`
|
||||
|
||||
- Tiny-tier result: `100%`
|
||||
- Provisional bucket: `Deploy Now`
|
||||
- Rationale:
|
||||
- this is very close to a formula task
|
||||
- it should be treated as a deterministic baseline whenever possible
|
||||
- specialist use is acceptable, but only as a convenience layer over a known rule
|
||||
|
||||
### `verification_type`
|
||||
|
||||
- Tiny-tier result: about `84.9%`
|
||||
- Provisional bucket: `Deploy With Guardrails`
|
||||
- Rationale:
|
||||
- bounded 5-way task with real signal at the tiny tier
|
||||
- likely useful if accepted predictions have high precision
|
||||
- reversible in many workflows because verification type can be validated or
|
||||
corrected downstream
|
||||
- Recommended next step:
|
||||
- evaluate confidence-thresholded precision
|
||||
- inspect confusion concentration, especially adjacent classes
|
||||
|
||||
### `worker_type`
|
||||
|
||||
- Tiny-tier result: about `88.5%`
|
||||
- Provisional bucket: `Deploy With Guardrails`
|
||||
- Rationale:
|
||||
- strong enough to be operationally useful if low-confidence cases abstain
|
||||
- wrong worker selection is often recoverable
|
||||
- likely a good candidate for first-pass routing plus retry
|
||||
- Recommended next step:
|
||||
- measure accepted precision and abstain coverage
|
||||
- add a cheap retry path with richer context
|
||||
|
||||
### `prereq_op`
|
||||
|
||||
- Tiny-tier result: about `69.4%`, flat across steps 1000-4000
|
||||
- Provisional bucket: `Redesign Before Scaling`
|
||||
- Rationale:
|
||||
- the current 4-way label is really a composition of two latent yes/no decisions:
|
||||
`needs architect review?` and `needs manual approval?`
|
||||
- the current generator maps those two axes into one of:
|
||||
`standard`, `needs_review`, `needs_approval`, `full_gates`
|
||||
- that means the model is being asked to infer a product label from short text,
|
||||
even though the natural decision structure is factorized
|
||||
- labels are heuristic and keyword-driven, which likely adds noise
|
||||
- scaling the model may help, but it may also just memorize fuzzy boundaries
|
||||
- Why it is hard:
|
||||
- short descriptions often under-specify whether a task is architectural,
|
||||
production-sensitive, both, or neither
|
||||
- many cues are policy-level rather than lexical
|
||||
- some examples mix "phase/integration/framework" signals with release/security
|
||||
signals, making the combined class boundary weak
|
||||
- the downstream action is itself compositional, which suggests the predictor
|
||||
should also be compositional
|
||||
- Recommended decomposition:
|
||||
- gate A: `requires_architect_review` → `yes/no`
|
||||
- gate B: `requires_manual_approval` → `yes/no`
|
||||
- deterministic combiner:
|
||||
- `00` -> `standard`
|
||||
- `10` -> `needs_review`
|
||||
- `01` -> `needs_approval`
|
||||
- `11` -> `full_gates`
|
||||
- Why decomposition is attractive:
|
||||
- each model solves a cleaner binary problem
|
||||
- outputs align directly with the policy semantics
|
||||
- confidence can be calibrated independently per axis
|
||||
- one axis may be deterministic or near-deterministic in some contexts
|
||||
- What to test before scaling:
|
||||
- binary performance for each axis
|
||||
- whether richer structured inputs outperform plain step text
|
||||
- whether some approval decisions should be deterministic policy checks instead
|
||||
|
||||
### `automatability`
|
||||
|
||||
- Tiny-tier result: about `69.6%`, flat across steps 1000-4000
|
||||
- Provisional bucket: `Wait For Large Run` and `Redesign Before Scaling`
|
||||
- Rationale:
|
||||
- this gate is conceptually important, but the label boundaries are likely moving
|
||||
targets because they depend on what tools and templates the system already has
|
||||
- adjacent classes are semantically close
|
||||
- the cost of some confusions is very asymmetric
|
||||
- Why it is hard:
|
||||
- the classes are not purely descriptive; they are partly roadmap-dependent
|
||||
- a node can move from `llm` to `specialist` or `template` as tooling improves
|
||||
- the difference between `template`, `specialist`, `slm`, and `llm` is partly an
|
||||
operational routing policy, not just an intrinsic property of the node
|
||||
- Recommended next step:
|
||||
- wait for current large runs to finish
|
||||
- inspect confusion matrix, not just headline accuracy
|
||||
- consider a hierarchical formulation
|
||||
- Possible decomposition:
|
||||
- gate A: `is deterministic/template sufficient?`
|
||||
- gate B: `if learned, is output bounded or open-vocabulary?`
|
||||
- gate C: `if open-vocabulary, does small-model routing suffice or is full LLM needed?`
|
||||
|
||||
## Immediate Operational Policy
|
||||
|
||||
Until the larger runs complete, use this posture:
|
||||
|
||||
- `confidence_tier`: deterministic or auto-deploy
|
||||
- `verification_type`: candidate for guarded deployment
|
||||
- `worker_type`: candidate for guarded deployment
|
||||
- `prereq_op`: do not blindly scale; test decomposed formulation first
|
||||
- `automatability`: keep training and collect richer evidence before committing
|
||||
|
||||
## Main Lesson
|
||||
|
||||
When a bounded gate performs poorly, the first question should not be:
|
||||
|
||||
"How much bigger should the model be?"
|
||||
|
||||
The first questions should be:
|
||||
|
||||
- is the gate actually one gate?
|
||||
- is the label space stable?
|
||||
- is the input missing policy-critical structure?
|
||||
- should part of the logic be deterministic?
|
||||
|
||||
That is the point of this rubric.
|
||||
290
docs/probe_execution_flow.md
Normal file
290
docs/probe_execution_flow.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Probe Execution Flow
|
||||
|
||||
## Purpose
|
||||
|
||||
This document describes how `whetstone_RSA` should execute gate diagnosis in
|
||||
practice.
|
||||
|
||||
The taxonomy and diagnosis API define what the library needs to represent.
|
||||
This document defines how the pieces fit together during a diagnosis run.
|
||||
|
||||
## High-Level Flow
|
||||
|
||||
```text
|
||||
GateDefinition
|
||||
+
|
||||
GateEvidence
|
||||
+
|
||||
PolicyContext
|
||||
↓
|
||||
Probe selection
|
||||
↓
|
||||
Applicable probes run
|
||||
↓
|
||||
Probe results normalized
|
||||
↓
|
||||
Failure-class synthesis
|
||||
↓
|
||||
Intervention ranking
|
||||
↓
|
||||
Policy recommendation
|
||||
↓
|
||||
GateDiagnosis
|
||||
```
|
||||
|
||||
## Inputs
|
||||
|
||||
### 1. `GateDefinition`
|
||||
|
||||
Static metadata about the gate:
|
||||
|
||||
- labels
|
||||
- slots
|
||||
- constraints
|
||||
- risk tier
|
||||
- supports abstain
|
||||
- deterministic baseline availability
|
||||
- label stability
|
||||
- candidate factorizations
|
||||
|
||||
### 2. `GateEvidence`
|
||||
|
||||
Observed evidence for one gate and one model/policy configuration:
|
||||
|
||||
- raw accuracy
|
||||
- accepted accuracy
|
||||
- abstain rate
|
||||
- retry recovery
|
||||
- silent error rate
|
||||
- calibration
|
||||
- confusion summary
|
||||
- latency
|
||||
- cost
|
||||
|
||||
### 3. `PolicyContext`
|
||||
|
||||
Deployment context that affects recommendations:
|
||||
|
||||
- risk budget
|
||||
- acceptable latency
|
||||
- acceptable compute cost
|
||||
- available escalation targets
|
||||
- whether deterministic checks exist downstream
|
||||
|
||||
## Step 1: Validate Evidence
|
||||
|
||||
Before diagnosis starts, the library should verify the evidence is usable.
|
||||
|
||||
Checks:
|
||||
|
||||
- gate id matches between definition and evidence
|
||||
- labels in confusion summary match the schema
|
||||
- metrics are internally consistent
|
||||
- risk tier and policy context are present
|
||||
|
||||
If evidence is incomplete, diagnosis should still proceed when possible, but the
|
||||
missing fields should reduce confidence in the result.
|
||||
|
||||
## Step 2: Select Applicable Probes
|
||||
|
||||
Not every probe applies to every gate.
|
||||
|
||||
Examples:
|
||||
|
||||
- `deterministic_rule_probe` applies only when structured features or a deterministic
|
||||
baseline are available
|
||||
- `factorization_probe` applies when labels suggest composition or a candidate
|
||||
factorization is registered
|
||||
- `schema_stability_probe` applies when labels may drift with policy or capability
|
||||
- `confidence_threshold_probe` applies only if confidence-bearing outputs exist
|
||||
|
||||
The library should ask each probe:
|
||||
|
||||
```text
|
||||
supports(gate_definition, evidence) -> bool
|
||||
```
|
||||
|
||||
and only run probes that opt in.
|
||||
|
||||
## Step 3: Run Probes
|
||||
|
||||
Each applicable probe returns:
|
||||
|
||||
- probe-specific signals
|
||||
- recommendation hints
|
||||
- a confidence score
|
||||
|
||||
Probe outputs should be normalized into a standard envelope before synthesis.
|
||||
|
||||
Example normalized structure:
|
||||
|
||||
```text
|
||||
ProbeResult
|
||||
probe_id
|
||||
status
|
||||
signals
|
||||
recommendation_hints
|
||||
confidence
|
||||
```
|
||||
|
||||
## Step 4: Aggregate Signals
|
||||
|
||||
The library should not map one probe directly to one final diagnosis.
|
||||
|
||||
Instead it should synthesize across probe outputs.
|
||||
|
||||
Examples:
|
||||
|
||||
- strong `factorization_probe` + compositional confusion pattern
|
||||
-> `factorizable_gate`
|
||||
- strong `schema_stability_probe` + drifting label semantics
|
||||
-> `non_stationary_gate`
|
||||
- strong `confidence_threshold_probe` + weak raw accuracy but high accepted precision
|
||||
-> `guardrail_limited_gate`
|
||||
- strong `deterministic_rule_probe`
|
||||
-> `deterministic_disguised_as_ml_gate`
|
||||
|
||||
This step should produce:
|
||||
|
||||
- ranked failure classes
|
||||
- supporting signals
|
||||
- unresolved ambiguities
|
||||
|
||||
## Step 5: Rank Interventions
|
||||
|
||||
Once failure classes are ranked, the library should rank interventions.
|
||||
|
||||
Interventions should come from a controlled vocabulary:
|
||||
|
||||
- `keep_current_gate`
|
||||
- `deploy_with_guardrails`
|
||||
- `raise_confidence_threshold`
|
||||
- `add_retry_with_context`
|
||||
- `add_deterministic_prepass`
|
||||
- `replace_with_deterministic_logic`
|
||||
- `factor_into_subgates`
|
||||
- `switch_to_hierarchical_gate`
|
||||
- `revise_label_schema`
|
||||
- `enrich_input_context`
|
||||
- `evaluate_larger_model_tier`
|
||||
|
||||
The order matters.
|
||||
|
||||
The library should prefer lower-cost, structurally-corrective interventions before
|
||||
defaulting to bigger models.
|
||||
|
||||
## Step 6: Emit Policy Recommendation
|
||||
|
||||
The final diagnosis should include a recommended deployment posture.
|
||||
|
||||
Possible deployment modes:
|
||||
|
||||
- `auto_accept`
|
||||
- `guarded_accept`
|
||||
- `abstain_first`
|
||||
- `research_only`
|
||||
|
||||
This recommendation should consider:
|
||||
|
||||
- risk tier
|
||||
- silent error estimate
|
||||
- accepted precision
|
||||
- latency budget
|
||||
- escalation availability
|
||||
|
||||
## Example Flow Patterns
|
||||
|
||||
### Pattern A: Good Gate With Policy Gaps
|
||||
|
||||
Observed:
|
||||
|
||||
- moderate raw accuracy
|
||||
- strong accepted precision under thresholding
|
||||
- downstream deterministic checks exist
|
||||
|
||||
Likely result:
|
||||
|
||||
- primary failure class: `guardrail_limited_gate`
|
||||
- intervention: `deploy_with_guardrails`
|
||||
|
||||
### Pattern B: Flat Multiclass Gate
|
||||
|
||||
Observed:
|
||||
|
||||
- low raw accuracy
|
||||
- confusion concentrates between composite labels
|
||||
- factorization metadata exists
|
||||
|
||||
Likely result:
|
||||
|
||||
- primary failure class: `factorizable_gate`
|
||||
- intervention: `factor_into_subgates`
|
||||
|
||||
### Pattern C: Formula-Like Gate
|
||||
|
||||
Observed:
|
||||
|
||||
- high score under structural encoding
|
||||
- symbolic baseline exists
|
||||
- class meaning is crisp
|
||||
|
||||
Likely result:
|
||||
|
||||
- primary failure class: `deterministic_disguised_as_ml_gate`
|
||||
- intervention: `replace_with_deterministic_logic`
|
||||
|
||||
### Pattern D: Drifting Capability Boundary
|
||||
|
||||
Observed:
|
||||
|
||||
- disagreements cluster between adjacent routing tiers
|
||||
- label meanings depend on current tooling maturity
|
||||
|
||||
Likely result:
|
||||
|
||||
- primary failure class: `non_stationary_gate`
|
||||
- intervention: `revise_label_schema` or separate ontology from policy
|
||||
|
||||
## Confidence Model
|
||||
|
||||
The diagnosis should carry its own confidence.
|
||||
|
||||
Diagnosis confidence should depend on:
|
||||
|
||||
- probe coverage
|
||||
- evidence completeness
|
||||
- signal agreement across probes
|
||||
- stability of the ranked recommendation
|
||||
|
||||
Low-confidence diagnosis is still useful if it says:
|
||||
|
||||
- evidence insufficient
|
||||
- collect richer context
|
||||
- run a missing probe
|
||||
|
||||
## Failure Handling
|
||||
|
||||
Diagnosis should fail soft.
|
||||
|
||||
If probes cannot reach a strong conclusion, the output should still be structured:
|
||||
|
||||
```text
|
||||
GateDiagnosis
|
||||
primary_failure_class = unknown
|
||||
recommended_probes = [...]
|
||||
recommended_interventions = [collect_more_evidence]
|
||||
confidence = low
|
||||
```
|
||||
|
||||
That is better than forcing a false diagnosis.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
This execution flow is what keeps `whetstone_RSA` from becoming just a model zoo.
|
||||
|
||||
The library’s value is not only that it can run tiny models. It is that it can
|
||||
help decide:
|
||||
|
||||
- whether a gate is worth scaling
|
||||
- whether a gate should be restructured
|
||||
- whether the problem should leave ML entirely
|
||||
159
docs/probe_registry.md
Normal file
159
docs/probe_registry.md
Normal file
@@ -0,0 +1,159 @@
|
||||
# Probe Registry
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the probe registry concept for `whetstone_RSA`.
|
||||
|
||||
The registry is how the diagnosis runtime discovers and executes diagnostic probes
|
||||
without hard-coding them into one monolithic diagnosis function.
|
||||
|
||||
## Why A Registry Exists
|
||||
|
||||
The library should be able to add probes incrementally.
|
||||
|
||||
Examples:
|
||||
|
||||
- `confidence_threshold_probe`
|
||||
- `factorization_probe`
|
||||
- `deterministic_rule_probe`
|
||||
- `schema_stability_probe`
|
||||
- `context_width_probe`
|
||||
- `confusion_structure_probe`
|
||||
|
||||
Some applications may also add domain-specific probes later.
|
||||
|
||||
The registry keeps that extensible while preserving a stable diagnosis surface.
|
||||
|
||||
## Probe Contract
|
||||
|
||||
Every probe should satisfy the same conceptual interface:
|
||||
|
||||
```text
|
||||
Probe
|
||||
id() -> string
|
||||
supports(gate_definition, evidence) -> bool
|
||||
run(probe_request) -> probe_result
|
||||
```
|
||||
|
||||
## Probe Metadata
|
||||
|
||||
Each registered probe should expose metadata:
|
||||
|
||||
```text
|
||||
ProbeMetadata
|
||||
probe_id
|
||||
description
|
||||
required_inputs
|
||||
optional_inputs
|
||||
supported_failure_classes
|
||||
produces_signals
|
||||
```
|
||||
|
||||
This lets the runtime explain why a probe was or was not run.
|
||||
|
||||
## Registry Responsibilities
|
||||
|
||||
The probe registry should support:
|
||||
|
||||
- registration
|
||||
- lookup by id
|
||||
- listing available probes
|
||||
- filtering applicable probes
|
||||
- validating probe dependencies if any are added later
|
||||
|
||||
## Default Probe Set
|
||||
|
||||
Initial probes expected in the registry:
|
||||
|
||||
- `confidence_threshold_probe`
|
||||
- `factorization_probe`
|
||||
- `deterministic_rule_probe`
|
||||
- `schema_stability_probe`
|
||||
- `context_width_probe`
|
||||
- `confusion_structure_probe`
|
||||
|
||||
## Probe Request Envelope
|
||||
|
||||
The runtime should present probes with a uniform request:
|
||||
|
||||
```text
|
||||
ProbeRequest
|
||||
gate_definition
|
||||
evidence
|
||||
policy_context
|
||||
trace_context
|
||||
```
|
||||
|
||||
This keeps probes focused on diagnosis logic rather than runtime plumbing.
|
||||
|
||||
## Probe Result Envelope
|
||||
|
||||
Each probe should return:
|
||||
|
||||
```text
|
||||
ProbeResult
|
||||
probe_id
|
||||
status
|
||||
signals
|
||||
recommendation_hints
|
||||
confidence
|
||||
```
|
||||
|
||||
Suggested `status` values:
|
||||
|
||||
- `success`
|
||||
- `not_applicable`
|
||||
- `insufficient_evidence`
|
||||
- `error`
|
||||
|
||||
## Applicability Rules
|
||||
|
||||
Probes should opt in explicitly.
|
||||
|
||||
Examples:
|
||||
|
||||
- `confidence_threshold_probe` should decline if confidence-bearing outputs are absent
|
||||
- `factorization_probe` should decline if labels have no registered structure and no
|
||||
confusion evidence is available
|
||||
- `deterministic_rule_probe` should decline if no structured features or baseline rule exists
|
||||
|
||||
This keeps diagnosis honest and prevents invented certainty.
|
||||
|
||||
## Synthesis Boundary
|
||||
|
||||
The registry executes probes.
|
||||
It does not decide the final diagnosis.
|
||||
|
||||
Final diagnosis should happen in a separate synthesis step that:
|
||||
|
||||
- aggregates probe signals
|
||||
- ranks failure classes
|
||||
- ranks interventions
|
||||
|
||||
That separation matters because:
|
||||
|
||||
- probes are local
|
||||
- diagnosis is global
|
||||
|
||||
## Future-Proofing
|
||||
|
||||
The registry design should tolerate:
|
||||
|
||||
- additional probes
|
||||
- domain-specific probes
|
||||
- partial evidence
|
||||
- multiple runtime implementations
|
||||
|
||||
The registry should not assume one model family, one training stack, or one
|
||||
application domain.
|
||||
|
||||
## Minimal Viable Implementation
|
||||
|
||||
The first probe registry can be very small:
|
||||
|
||||
- static in-memory registration
|
||||
- simple `supports` checks
|
||||
- direct execution
|
||||
- JSON-serializable results
|
||||
|
||||
That is enough to validate the architecture before adding plugin systems or dynamic loading.
|
||||
293
docs/runtime_contract.md
Normal file
293
docs/runtime_contract.md
Normal file
@@ -0,0 +1,293 @@
|
||||
# Runtime Contract
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the minimal runtime contract for `whetstone_RSA`.
|
||||
|
||||
The project already has:
|
||||
|
||||
- gate schemas
|
||||
- evaluation policy
|
||||
- failure taxonomy
|
||||
- diagnosis API sketch
|
||||
- probe execution flow
|
||||
|
||||
The missing piece is the execution boundary:
|
||||
|
||||
- what the runtime receives
|
||||
- what it returns
|
||||
- what extension points exist
|
||||
|
||||
This document defines that contract without binding the project to a specific
|
||||
implementation language.
|
||||
|
||||
## Runtime Roles
|
||||
|
||||
The runtime has three primary roles:
|
||||
|
||||
### 1. Decision Runtime
|
||||
|
||||
Runs a bounded gate against an input request and returns a structured decision.
|
||||
|
||||
### 2. Diagnosis Runtime
|
||||
|
||||
Runs probes against gate evidence and returns a structured diagnosis.
|
||||
|
||||
### 3. Policy Runtime
|
||||
|
||||
Combines gate output and diagnosis output into a deployment posture:
|
||||
|
||||
- accept
|
||||
- abstain
|
||||
- retry
|
||||
- escalate
|
||||
|
||||
Backend selection is a related but distinct concern and should remain adjacent to
|
||||
the RSA runtime rather than defining the RSA core. See `docs/backend_selection.md`.
|
||||
|
||||
## Core Runtime Calls
|
||||
|
||||
### `decide`
|
||||
|
||||
Purpose:
|
||||
|
||||
Run a bounded decision gate.
|
||||
|
||||
Conceptual signature:
|
||||
|
||||
```text
|
||||
decide(decision_request) -> decision_result
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
DecisionRequest
|
||||
gate_id
|
||||
input
|
||||
context
|
||||
policy_context
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
DecisionResult
|
||||
gate_id
|
||||
decision_id
|
||||
slots
|
||||
confidence
|
||||
abstain
|
||||
escalation_target
|
||||
trace
|
||||
```
|
||||
|
||||
### `diagnose`
|
||||
|
||||
Purpose:
|
||||
|
||||
Run gate diagnosis based on metadata and evidence.
|
||||
|
||||
Conceptual signature:
|
||||
|
||||
```text
|
||||
diagnose(diagnosis_request) -> gate_diagnosis
|
||||
```
|
||||
|
||||
Input:
|
||||
|
||||
```text
|
||||
DiagnosisRequest
|
||||
gate_definition
|
||||
evidence
|
||||
policy_context
|
||||
enabled_probes
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
GateDiagnosis
|
||||
gate_id
|
||||
primary_failure_class
|
||||
secondary_failure_classes
|
||||
supporting_signals
|
||||
recommended_probes
|
||||
recommended_interventions
|
||||
recommended_policy
|
||||
confidence
|
||||
```
|
||||
|
||||
### `recommend_policy`
|
||||
|
||||
Purpose:
|
||||
|
||||
Convert a gate decision and a diagnosis into an action posture.
|
||||
|
||||
Conceptual signature:
|
||||
|
||||
```text
|
||||
recommend_policy(decision_result, gate_diagnosis, policy_context) -> policy_action
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```text
|
||||
PolicyAction
|
||||
mode
|
||||
threshold_applied
|
||||
retry_strategy
|
||||
escalation_target
|
||||
rationale
|
||||
```
|
||||
|
||||
## Runtime Responsibilities
|
||||
|
||||
The runtime should own:
|
||||
|
||||
- schema validation
|
||||
- gate lookup
|
||||
- model-tier lookup
|
||||
- probe lookup
|
||||
- evidence normalization
|
||||
- policy recommendation
|
||||
|
||||
The runtime should not own:
|
||||
|
||||
- domain-specific business logic after execution
|
||||
- arbitrary open-ended generation
|
||||
- hidden manual policy that bypasses the declared schema
|
||||
|
||||
## Extension Points
|
||||
|
||||
The runtime should expose three registries:
|
||||
|
||||
### Gate Registry
|
||||
|
||||
Maps `gate_id` to:
|
||||
|
||||
- gate definition
|
||||
- supported model tiers
|
||||
- deterministic baseline if any
|
||||
- candidate factorizations
|
||||
|
||||
### Probe Registry
|
||||
|
||||
Maps `probe_id` to:
|
||||
|
||||
- applicability rules
|
||||
- execution logic
|
||||
- output normalizer
|
||||
|
||||
### Policy Registry
|
||||
|
||||
Maps deployment mode or risk policy to:
|
||||
|
||||
- threshold rules
|
||||
- retry rules
|
||||
- escalation rules
|
||||
|
||||
## Minimal Validation Rules
|
||||
|
||||
### For `decide`
|
||||
|
||||
- gate must exist
|
||||
- decision schema must validate
|
||||
- model tier must be available if the gate is learned
|
||||
- inputs must satisfy required structure
|
||||
|
||||
### For `diagnose`
|
||||
|
||||
- gate definition must validate
|
||||
- evidence must validate
|
||||
- enabled probes must exist
|
||||
- probe applicability must be checked before execution
|
||||
|
||||
### For `recommend_policy`
|
||||
|
||||
- decision result and diagnosis must refer to the same gate
|
||||
- risk tier must be available
|
||||
- policy mode must be valid for the gate
|
||||
|
||||
## Trace Requirements
|
||||
|
||||
The runtime should preserve structured trace output.
|
||||
|
||||
### Decision Trace
|
||||
|
||||
Should include:
|
||||
|
||||
- selected gate
|
||||
- selected model tier
|
||||
- confidence
|
||||
- thresholding outcome
|
||||
- abstain or escalation outcome
|
||||
|
||||
### Diagnosis Trace
|
||||
|
||||
Should include:
|
||||
|
||||
- probes run
|
||||
- probe outputs
|
||||
- synthesized failure class ranking
|
||||
- intervention ranking
|
||||
|
||||
Trace should be inspectable by humans and consumable by tools.
|
||||
|
||||
## Recommended Runtime States
|
||||
|
||||
For implementation planning, treat runtime work as these internal stages:
|
||||
|
||||
```text
|
||||
request_received
|
||||
schema_validated
|
||||
gate_loaded
|
||||
decision_executed
|
||||
evidence_loaded
|
||||
probes_executed
|
||||
diagnosis_synthesized
|
||||
policy_computed
|
||||
response_emitted
|
||||
```
|
||||
|
||||
This is useful for debugging and logging even before the runtime is complex.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
The runtime should fail in a structured way.
|
||||
|
||||
Example error kinds:
|
||||
|
||||
- `unknown_gate`
|
||||
- `invalid_schema`
|
||||
- `missing_model_tier`
|
||||
- `invalid_evidence`
|
||||
- `unknown_probe`
|
||||
- `inapplicable_probe`
|
||||
- `diagnosis_insufficient_evidence`
|
||||
|
||||
Errors should be explicit objects, not plain strings.
|
||||
|
||||
## Implementation Guidance
|
||||
|
||||
The first implementation should stay minimal:
|
||||
|
||||
- one gate registry abstraction
|
||||
- one probe registry abstraction
|
||||
- one diagnosis synthesis path
|
||||
- one JSON-serializable result format
|
||||
|
||||
Do not overbuild orchestration first.
|
||||
|
||||
The value of the runtime is that it makes bounded gate execution and diagnosis
|
||||
uniform across applications.
|
||||
|
||||
## Relationship To Case Studies
|
||||
|
||||
Case studies should populate the runtime through data:
|
||||
|
||||
- gate definitions
|
||||
- evidence bundles
|
||||
- diagnosis outputs
|
||||
|
||||
They should not force case-specific branches into the runtime contract.
|
||||
223
docs/whetstone_ast_first_integration.md
Normal file
223
docs/whetstone_ast_first_integration.md
Normal file
@@ -0,0 +1,223 @@
|
||||
# Whetstone-AST-First Integration
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines how `whetstone_RSA` should relate to `whetstone_DSL`.
|
||||
|
||||
The key rule is:
|
||||
|
||||
`whetstone_RSA` should be Whetstone-AST-first, not merely AST-first.
|
||||
|
||||
That means the semantic source of truth for RSA contracts should live in the
|
||||
Whetstone AST and annotation system, not in ad hoc C++ structs or generic parser
|
||||
ASTs.
|
||||
|
||||
## Why Generic AST Is Not Enough
|
||||
|
||||
A generic AST is mostly syntax structure.
|
||||
|
||||
It can tell you things like:
|
||||
|
||||
- function
|
||||
- class
|
||||
- parameter
|
||||
- if-statement
|
||||
- call expression
|
||||
|
||||
That is useful, but it is not enough for RSA.
|
||||
|
||||
RSA needs first-class semantic contracts for things like:
|
||||
|
||||
- input structure
|
||||
- input modality
|
||||
- output topology
|
||||
- output constraint
|
||||
- policy stability
|
||||
- suitability assessment
|
||||
- gate shape
|
||||
- confidence behavior
|
||||
- abstain policy
|
||||
- escalation policy
|
||||
- diagnosis metadata
|
||||
|
||||
Those are not reliable consequences of syntax alone.
|
||||
They are authored semantic data.
|
||||
|
||||
## Why Whetstone AST Is The Right Source
|
||||
|
||||
The Whetstone AST already exists to carry language-neutral semantic intent beyond
|
||||
what ordinary source code or parser ASTs can express cleanly.
|
||||
|
||||
That makes it the right authoring substrate for RSA because:
|
||||
|
||||
- annotations are first-class
|
||||
- concepts are language-neutral
|
||||
- contracts can be preserved across projections
|
||||
- comments do not need to carry architectural meaning
|
||||
- generated C++ can remain a projection target instead of becoming the semantic home
|
||||
|
||||
In short:
|
||||
|
||||
- generic AST gives structure
|
||||
- Whetstone AST gives structure plus semantic contract capacity
|
||||
|
||||
RSA needs the second one.
|
||||
|
||||
## Architectural Rule
|
||||
|
||||
The semantic source of truth for RSA should live in `whetstone_DSL`.
|
||||
|
||||
The C++ runtime in `whetstone_RSA` should be treated as a projection target for:
|
||||
|
||||
- runtime data types
|
||||
- execution logic
|
||||
- probe registry interfaces
|
||||
- policy and diagnosis runtime behavior
|
||||
|
||||
This keeps the architecture aligned with the rest of the Whetstone program:
|
||||
|
||||
- AST is source of truth
|
||||
- target languages are projections
|
||||
|
||||
## Proposed Split Of Responsibilities
|
||||
|
||||
### `whetstone_DSL`
|
||||
|
||||
Owns:
|
||||
|
||||
- RSA semantic ontology
|
||||
- gate definitions as semantic contracts
|
||||
- suitability metadata
|
||||
- diagnosis metadata
|
||||
- probe declarations
|
||||
- projection rules into target runtimes
|
||||
|
||||
### `whetstone_RSA`
|
||||
|
||||
Owns:
|
||||
|
||||
- C++ runtime implementation
|
||||
- probe execution engine
|
||||
- diagnosis engine
|
||||
- policy runtime
|
||||
- compiled/runtime-facing representations of projected contracts
|
||||
|
||||
## What Should Exist In The Whetstone AST
|
||||
|
||||
RSA needs explicit semantic objects, not just ordinary code nodes.
|
||||
|
||||
Examples of first-class RSA concepts that should eventually exist in Whetstone form:
|
||||
|
||||
- `GateDefinition`
|
||||
- `GateEvidence`
|
||||
- `GateDiagnosis`
|
||||
- `GateSuitability`
|
||||
- `PolicyRecommendation`
|
||||
- `ProbeDefinition`
|
||||
- `DecisionContract`
|
||||
|
||||
These should be represented as semantic entities with structured fields, not as
|
||||
freeform comments embedded in generated C++.
|
||||
|
||||
## Why This Is Better Than Authoring In C++
|
||||
|
||||
If RSA contracts are authored directly in C++, the semantics become fragile:
|
||||
|
||||
- comments carry architectural meaning
|
||||
- relationships are implicit
|
||||
- code review sees implementation before contract
|
||||
- projection to other targets becomes harder
|
||||
|
||||
If RSA contracts are authored in the Whetstone AST:
|
||||
|
||||
- the semantics are explicit
|
||||
- annotations stay attached to the real concept
|
||||
- multiple targets can be generated later
|
||||
- runtime code becomes easier to keep honest
|
||||
|
||||
This is especially important because RSA is intended to remain model-agnostic.
|
||||
|
||||
The contract is primary.
|
||||
The current transformer-based runtime is secondary.
|
||||
|
||||
## Model-Agnostic Benefit
|
||||
|
||||
Because RSA is contractual rather than model-defined, the AST layer should describe:
|
||||
|
||||
- what the gate promises
|
||||
- what the gate consumes
|
||||
- how confidence and abstention behave
|
||||
- how diagnosis works
|
||||
|
||||
without hard-coding one backend.
|
||||
|
||||
That preserves future replacement paths such as:
|
||||
|
||||
- transformer specialist
|
||||
- classical ML classifier
|
||||
- RL policy
|
||||
- retrieval-backed scorer
|
||||
- deterministic-plus-learned hybrid
|
||||
|
||||
The Whetstone AST should capture the contract.
|
||||
The C++ runtime should implement one or more backends against it.
|
||||
|
||||
## Near-Term Practical Rule
|
||||
|
||||
Right now, some C++ scaffolding already exists in `whetstone_RSA`.
|
||||
|
||||
That is fine as a bootstrap.
|
||||
|
||||
But it should be treated as:
|
||||
|
||||
- provisional runtime code
|
||||
- useful for exercising the contract shape
|
||||
- not the long-term semantic source of truth
|
||||
|
||||
As soon as the Whetstone-side semantic representation exists, the C++ layer should
|
||||
be aligned to it rather than continuing as an independently evolving hand-authored
|
||||
model.
|
||||
|
||||
## How To Use WhetstoneDSL Here
|
||||
|
||||
The right use of `whetstone_DSL` is not "parse some C++ and hope the meaning falls
|
||||
out."
|
||||
|
||||
The right use is:
|
||||
|
||||
1. define RSA concepts explicitly in Whetstone semantic form
|
||||
2. attach first-class metadata and annotations there
|
||||
3. project those contracts into C++ runtime types and adapters
|
||||
|
||||
If existing Whetstone MCP tools can help bootstrap some of the current C++ structs
|
||||
into AST-native representations, that is useful.
|
||||
|
||||
But the goal is not mechanical conversion for its own sake.
|
||||
The goal is semantic alignment.
|
||||
|
||||
If direct manual authoring in Whetstone form lands closer to the intended contract,
|
||||
that is better than a weak automatic conversion.
|
||||
|
||||
## Practical Guidance For The Current Stage
|
||||
|
||||
At this stage of the project:
|
||||
|
||||
- keep the current C++ runtime scaffold because it is useful for validating runtime shape
|
||||
- do not expand C++ into the permanent semantic home of the system
|
||||
- prioritize defining the RSA domain model in Whetstone semantic terms
|
||||
- use MCP-assisted conversion only when it lands close to the intended semantic contract
|
||||
|
||||
This is still early enough that correctness of semantic shape matters more than
|
||||
automation of the conversion path.
|
||||
|
||||
## Implication
|
||||
|
||||
`whetstone_RSA` is not just "a C++ library with some docs."
|
||||
|
||||
It should become:
|
||||
|
||||
- a Whetstone-authored semantic subsystem
|
||||
- with a C++ runtime projection
|
||||
|
||||
That is the architecture most consistent with both Whetstone and the long-term
|
||||
goals of RSA.
|
||||
Reference in New Issue
Block a user