115 lines
3.2 KiB
Markdown
115 lines
3.2 KiB
Markdown
# Entropy To Model Sizing
|
|
|
|
## Goal
|
|
|
|
Estimate the smallest model tier that can fill a given bounded decision gate with
|
|
acceptable accuracy and calibration.
|
|
|
|
This project needs a practical rule of the form:
|
|
|
|
```text
|
|
gate entropy score -> recommended model tier
|
|
```
|
|
|
|
## Why This Matters
|
|
|
|
Right now model choice is too manual:
|
|
|
|
- an ~800KB specialist works for some gates
|
|
- some gates appear to exceed that capacity
|
|
- the next attempts are 10x and 16x larger
|
|
|
|
That is directionally right, but it should become measurable.
|
|
|
|
## Working Definition
|
|
|
|
`Gate entropy` is not just Shannon entropy over labels.
|
|
|
|
For this project, it should mean the difficulty of mapping available input context
|
|
onto the correct bounded output under realistic variation.
|
|
|
|
## Proposed Factors
|
|
|
|
Each gate gets a score assembled from observable features:
|
|
|
|
- `output_cardinality`
|
|
- `class_balance`
|
|
- `input_length_distribution`
|
|
- `context_width_required`
|
|
- `slot_count`
|
|
- `slot_interdependence`
|
|
- `label_boundary_fuzziness`
|
|
- `lexical_variation`
|
|
- `hidden_state_dependence`
|
|
- `world_knowledge_dependence`
|
|
- `error_cost`
|
|
- `abstain_availability`
|
|
|
|
Some of these increase model demand directly. Others change the acceptable
|
|
confidence threshold and escalation policy.
|
|
|
|
## First Experimental Plan
|
|
|
|
For each Whetstone gate:
|
|
|
|
1. Define a fixed training and evaluation set.
|
|
2. Train multiple model tiers on the same gate.
|
|
3. Measure accuracy, calibration, confusion concentration, and latency.
|
|
4. Record the smallest tier that clears the target.
|
|
|
|
Suggested initial tiers:
|
|
|
|
- deterministic baseline when possible
|
|
- tiny specialist baseline: current ~213K-param / ~800KB tier
|
|
- medium specialist: ~10x current size
|
|
- large specialist: ~16x current size
|
|
|
|
## Deliverable
|
|
|
|
Build a baseline table like:
|
|
|
|
```text
|
|
gate_id | entropy_score | smallest_passing_tier | accuracy | ece | latency_ms
|
|
```
|
|
|
|
Then fit a first-pass policy:
|
|
|
|
```text
|
|
if entropy <= A -> deterministic
|
|
if A < entropy <= B -> tiny specialist
|
|
if B < entropy <= C -> medium specialist
|
|
if entropy > C -> large specialist or escalate
|
|
```
|
|
|
|
## Important Caveat
|
|
|
|
A larger model should not be the only answer.
|
|
|
|
Before moving up a tier, the evaluation should check:
|
|
|
|
- is the schema wrong?
|
|
- is the label set under-enumerated?
|
|
- is important context missing?
|
|
- are we forcing a fuzzy gate where a deterministic pre-pass should exist?
|
|
|
|
Model size should increase only after the gate definition itself is defensible.
|
|
|
|
## Open Questions
|
|
|
|
- Which entropy factors are predictive enough to automate tier selection?
|
|
- Is calibration a better gating signal than raw accuracy for deployment?
|
|
- Can some gates be factorized into two smaller gates instead of one larger model?
|
|
- Does AST-native input encoding reduce effective entropy compared with raw text?
|
|
- Can Hivemind use failure and abstention data to identify where new deterministic
|
|
tools should be built?
|
|
|
|
## Connection To Hivemind
|
|
|
|
This is where the projects meet:
|
|
|
|
- Hivemind identifies high-entropy surfaces with poor deterministic support
|
|
- `whetstone_RSA` tries to collapse some of those surfaces into bounded gates
|
|
- residual high-entropy zones become candidates for larger models or new tools
|
|
|
|
That makes entropy estimation useful for both deployment and roadmap planning.
|