227 lines
4.6 KiB
Markdown
227 lines
4.6 KiB
Markdown
# Backend Selection
|
|
|
|
## Purpose
|
|
|
|
This document defines backend selection as a first-class concern for
|
|
`whetstone_RSA`.
|
|
|
|
The key distinction is:
|
|
|
|
- RSA decides whether a problem should be handled as a bounded decision contract
|
|
- backend selection decides which implementation family should fill that contract
|
|
|
|
These are different questions and should not be collapsed together.
|
|
|
|
## Core Rule
|
|
|
|
Do not define RSA by one backend family.
|
|
|
|
RSA is model-agnostic.
|
|
The backend should be selected based on gate characteristics, deployment needs, and
|
|
measured results.
|
|
|
|
## Decision Layers
|
|
|
|
There are three separate decisions:
|
|
|
|
### 1. Layer Suitability
|
|
|
|
Question:
|
|
|
|
Should this problem be:
|
|
|
|
- `deterministic`
|
|
- `rsa`
|
|
- `slm`
|
|
- `llm`
|
|
- `human`
|
|
|
|
This is handled by suitability assessment.
|
|
|
|
### 2. Gate Shape Selection
|
|
|
|
Question:
|
|
|
|
If the problem is an RSA candidate, what shape should the gate take?
|
|
|
|
- `binary`
|
|
- `multiclass`
|
|
- `factorized`
|
|
- `hierarchical`
|
|
|
|
This is handled by gate design and diagnosis.
|
|
|
|
### 3. Backend Selection
|
|
|
|
Question:
|
|
|
|
If the problem is an RSA gate, what implementation family should fill it?
|
|
|
|
- `transformer`
|
|
- `gradient_boosting`
|
|
- `random_forest`
|
|
- `bayesian`
|
|
- `linear_model`
|
|
- `rl_policy`
|
|
- `hybrid`
|
|
|
|
This document is about the third decision.
|
|
|
|
## Why This Matters
|
|
|
|
Different gates have different data characteristics.
|
|
|
|
Examples:
|
|
|
|
- `unstructured + textual + bounded`
|
|
- transformer may be the natural default
|
|
|
|
- `structured + numeric + bounded`
|
|
- gradient boosting, random forest, or linear models may be stronger and cheaper
|
|
|
|
- `small data + strong priors`
|
|
- Bayesian or simpler classifiers may be better
|
|
|
|
- `reward-shaped routing with delayed payoff`
|
|
- RL or bandit-style policies may become useful
|
|
|
|
So backend choice should not be hard-coded.
|
|
|
|
## What RSA Should Own
|
|
|
|
`whetstone_RSA` should own:
|
|
|
|
- the gate contract
|
|
- suitability assessment
|
|
- diagnosis
|
|
- policy behavior
|
|
- the runtime interface for backends
|
|
|
|
It should not assume:
|
|
|
|
- transformer by default in all cases
|
|
- one training stack
|
|
- one benchmark methodology
|
|
|
|
## What Backend Selection Should Do
|
|
|
|
Backend selection should answer:
|
|
|
|
- which backend families are plausible for this gate?
|
|
- which should be tried first?
|
|
- which benchmark signals matter?
|
|
- whether existing model-selection tooling should be used
|
|
|
|
## Selection Variables
|
|
|
|
Backend recommendation should consider at least these variables:
|
|
|
|
### Contract variables
|
|
|
|
- `input_structure`
|
|
- `input_modality`
|
|
- `output_topology`
|
|
- `output_constraint`
|
|
- `policy_stability`
|
|
|
|
### Data variables
|
|
|
|
- dataset size
|
|
- class balance
|
|
- feature sparsity
|
|
- sequence length
|
|
- label noise
|
|
- factorization availability
|
|
|
|
### Deployment variables
|
|
|
|
- latency budget
|
|
- memory budget
|
|
- retraining frequency
|
|
- calibration needs
|
|
- explainability needs
|
|
- online adaptation needs
|
|
|
|
### Runtime variables
|
|
|
|
- deterministic fallback availability
|
|
- confidence thresholding support
|
|
- retry/escalation policy
|
|
- traceability requirements
|
|
|
|
## Suggested Types
|
|
|
|
Future implementation should likely include types along these lines:
|
|
|
|
```text
|
|
BackendFamily
|
|
transformer
|
|
gradient_boosting
|
|
random_forest
|
|
bayesian
|
|
linear_model
|
|
rl_policy
|
|
hybrid
|
|
```
|
|
|
|
```text
|
|
BackendSuitability
|
|
gate_id
|
|
candidate_backends
|
|
recommended_backend
|
|
rationale
|
|
confidence
|
|
```
|
|
|
|
```text
|
|
BackendBenchmarkResult
|
|
gate_id
|
|
backend_family
|
|
metrics
|
|
training_cost
|
|
runtime_cost
|
|
calibration
|
|
notes
|
|
```
|
|
|
|
## Use Existing Tools Where Practical
|
|
|
|
Backend selection is not a place to reinvent the entire applied ML ecosystem.
|
|
|
|
In many cases, existing tools or benchmark harnesses can already compare:
|
|
|
|
- random forest
|
|
- boosted trees
|
|
- linear models
|
|
- Naive Bayes
|
|
- simple neural models
|
|
|
|
So `whetstone_RSA` should be willing to consume benchmark results from external
|
|
model-selection or AutoML-style workflows instead of owning every search strategy
|
|
itself.
|
|
|
|
The important thing is that the result plugs back into the RSA contract layer.
|
|
|
|
## Recommended Architectural Boundary
|
|
|
|
Keep backend selection adjacent to the RSA core, not inside the core identity.
|
|
|
|
That means:
|
|
|
|
- `GateSuitability` answers whether RSA is the right container
|
|
- `GateDiagnosis` helps debug and restructure the gate
|
|
- `BackendSuitability` recommends what implementation family should fill it
|
|
|
|
This separation keeps the architecture clean.
|
|
|
|
## Immediate Follow-Up
|
|
|
|
This should eventually be reflected in code by:
|
|
|
|
- adding a backend family enum
|
|
- adding backend suitability types
|
|
- extending the runtime contract so gates do not assume one backend
|
|
- optionally adding hooks for external benchmark/model-selection tooling
|
|
|
|
Until then, this document is the explicit placeholder so the work does not get lost.
|