Initial whetstone_RSA architecture and C++ scaffold
This commit is contained in:
Bill Holcombe
226
docs/backend_selection.md
Normal file
226
docs/backend_selection.md
Normal file
@@ -0,0 +1,226 @@
|
||||
# Backend Selection
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines backend selection as a first-class concern for
|
||||
`whetstone_RSA`.
|
||||
|
||||
The key distinction is:
|
||||
|
||||
- RSA decides whether a problem should be handled as a bounded decision contract
|
||||
- backend selection decides which implementation family should fill that contract
|
||||
|
||||
These are different questions and should not be collapsed together.
|
||||
|
||||
## Core Rule
|
||||
|
||||
Do not define RSA by one backend family.
|
||||
|
||||
RSA is model-agnostic.
|
||||
The backend should be selected based on gate characteristics, deployment needs, and
|
||||
measured results.
|
||||
|
||||
## Decision Layers
|
||||
|
||||
There are three separate decisions:
|
||||
|
||||
### 1. Layer Suitability
|
||||
|
||||
Question:
|
||||
|
||||
Should this problem be:
|
||||
|
||||
- `deterministic`
|
||||
- `rsa`
|
||||
- `slm`
|
||||
- `llm`
|
||||
- `human`
|
||||
|
||||
This is handled by suitability assessment.
|
||||
|
||||
### 2. Gate Shape Selection
|
||||
|
||||
Question:
|
||||
|
||||
If the problem is an RSA candidate, what shape should the gate take?
|
||||
|
||||
- `binary`
|
||||
- `multiclass`
|
||||
- `factorized`
|
||||
- `hierarchical`
|
||||
|
||||
This is handled by gate design and diagnosis.
|
||||
|
||||
### 3. Backend Selection
|
||||
|
||||
Question:
|
||||
|
||||
If the problem is an RSA gate, what implementation family should fill it?
|
||||
|
||||
- `transformer`
|
||||
- `gradient_boosting`
|
||||
- `random_forest`
|
||||
- `bayesian`
|
||||
- `linear_model`
|
||||
- `rl_policy`
|
||||
- `hybrid`
|
||||
|
||||
This document is about the third decision.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Different gates have different data characteristics.
|
||||
|
||||
Examples:
|
||||
|
||||
- `unstructured + textual + bounded`
|
||||
- transformer may be the natural default
|
||||
|
||||
- `structured + numeric + bounded`
|
||||
- gradient boosting, random forest, or linear models may be stronger and cheaper
|
||||
|
||||
- `small data + strong priors`
|
||||
- Bayesian or simpler classifiers may be better
|
||||
|
||||
- `reward-shaped routing with delayed payoff`
|
||||
- RL or bandit-style policies may become useful
|
||||
|
||||
So backend choice should not be hard-coded.
|
||||
|
||||
## What RSA Should Own
|
||||
|
||||
`whetstone_RSA` should own:
|
||||
|
||||
- the gate contract
|
||||
- suitability assessment
|
||||
- diagnosis
|
||||
- policy behavior
|
||||
- the runtime interface for backends
|
||||
|
||||
It should not assume:
|
||||
|
||||
- transformer by default in all cases
|
||||
- one training stack
|
||||
- one benchmark methodology
|
||||
|
||||
## What Backend Selection Should Do
|
||||
|
||||
Backend selection should answer:
|
||||
|
||||
- which backend families are plausible for this gate?
|
||||
- which should be tried first?
|
||||
- which benchmark signals matter?
|
||||
- whether existing model-selection tooling should be used
|
||||
|
||||
## Selection Variables
|
||||
|
||||
Backend recommendation should consider at least these variables:
|
||||
|
||||
### Contract variables
|
||||
|
||||
- `input_structure`
|
||||
- `input_modality`
|
||||
- `output_topology`
|
||||
- `output_constraint`
|
||||
- `policy_stability`
|
||||
|
||||
### Data variables
|
||||
|
||||
- dataset size
|
||||
- class balance
|
||||
- feature sparsity
|
||||
- sequence length
|
||||
- label noise
|
||||
- factorization availability
|
||||
|
||||
### Deployment variables
|
||||
|
||||
- latency budget
|
||||
- memory budget
|
||||
- retraining frequency
|
||||
- calibration needs
|
||||
- explainability needs
|
||||
- online adaptation needs
|
||||
|
||||
### Runtime variables
|
||||
|
||||
- deterministic fallback availability
|
||||
- confidence thresholding support
|
||||
- retry/escalation policy
|
||||
- traceability requirements
|
||||
|
||||
## Suggested Types
|
||||
|
||||
Future implementation should likely include types along these lines:
|
||||
|
||||
```text
|
||||
BackendFamily
|
||||
transformer
|
||||
gradient_boosting
|
||||
random_forest
|
||||
bayesian
|
||||
linear_model
|
||||
rl_policy
|
||||
hybrid
|
||||
```
|
||||
|
||||
```text
|
||||
BackendSuitability
|
||||
gate_id
|
||||
candidate_backends
|
||||
recommended_backend
|
||||
rationale
|
||||
confidence
|
||||
```
|
||||
|
||||
```text
|
||||
BackendBenchmarkResult
|
||||
gate_id
|
||||
backend_family
|
||||
metrics
|
||||
training_cost
|
||||
runtime_cost
|
||||
calibration
|
||||
notes
|
||||
```
|
||||
|
||||
## Use Existing Tools Where Practical
|
||||
|
||||
Backend selection is not a place to reinvent the entire applied ML ecosystem.
|
||||
|
||||
In many cases, existing tools or benchmark harnesses can already compare:
|
||||
|
||||
- random forest
|
||||
- boosted trees
|
||||
- linear models
|
||||
- Naive Bayes
|
||||
- simple neural models
|
||||
|
||||
So `whetstone_RSA` should be willing to consume benchmark results from external
|
||||
model-selection or AutoML-style workflows instead of owning every search strategy
|
||||
itself.
|
||||
|
||||
The important thing is that the result plugs back into the RSA contract layer.
|
||||
|
||||
## Recommended Architectural Boundary
|
||||
|
||||
Keep backend selection adjacent to the RSA core, not inside the core identity.
|
||||
|
||||
That means:
|
||||
|
||||
- `GateSuitability` answers whether RSA is the right container
|
||||
- `GateDiagnosis` helps debug and restructure the gate
|
||||
- `BackendSuitability` recommends what implementation family should fill it
|
||||
|
||||
This separation keeps the architecture clean.
|
||||
|
||||
## Immediate Follow-Up
|
||||
|
||||
This should eventually be reflected in code by:
|
||||
|
||||
- adding a backend family enum
|
||||
- adding backend suitability types
|
||||
- extending the runtime contract so gates do not assume one backend
|
||||
- optionally adding hooks for external benchmark/model-selection tooling
|
||||
|
||||
Until then, this document is the explicit placeholder so the work does not get lost.
|
||||
Reference in New Issue
Block a user