# Entropy To Model Sizing

## Goal

Estimate the smallest model tier that can fill a given bounded decision gate with
acceptable accuracy and calibration.

This project needs a practical rule of the form:

```text
gate entropy score -> recommended model tier
```

## Why This Matters

Right now model choice is too manual:

- an ~800KB specialist works for some gates
- some gates appear to exceed that capacity
- the next attempts are 10x and 16x larger

That is directionally right, but it should become measurable.

## Working Definition

`Gate entropy` is not just Shannon entropy over labels.

For this project, it should mean the difficulty of mapping available input context
onto the correct bounded output under realistic variation.

## Proposed Factors

Each gate gets a score assembled from observable features:

- `output_cardinality`
- `class_balance`
- `input_length_distribution`
- `context_width_required`
- `slot_count`
- `slot_interdependence`
- `label_boundary_fuzziness`
- `lexical_variation`
- `hidden_state_dependence`
- `world_knowledge_dependence`
- `error_cost`
- `abstain_availability`

Some of these increase model demand directly. Others change the acceptable
confidence threshold and escalation policy.

## First Experimental Plan

For each Whetstone gate:

1. Define a fixed training and evaluation set.
2. Train multiple model tiers on the same gate.
3. Measure accuracy, calibration, confusion concentration, and latency.
4. Record the smallest tier that clears the target.

Suggested initial tiers:

- deterministic baseline when possible
- tiny specialist baseline: current ~213K-param / ~800KB tier
- medium specialist: ~10x current size
- large specialist: ~16x current size

## Deliverable

Build a baseline table like:

```text
gate_id | entropy_score | smallest_passing_tier | accuracy | ece | latency_ms
```

Then fit a first-pass policy:

```text
if entropy <= A -> deterministic
if A < entropy <= B -> tiny specialist
if B < entropy <= C -> medium specialist
if entropy > C -> large specialist or escalate
```

## Important Caveat

A larger model should not be the only answer.

Before moving up a tier, the evaluation should check:

- is the schema wrong?
- is the label set under-enumerated?
- is important context missing?
- are we forcing a fuzzy gate where a deterministic pre-pass should exist?

Model size should increase only after the gate definition itself is defensible.

## Open Questions

- Which entropy factors are predictive enough to automate tier selection?
- Is calibration a better gating signal than raw accuracy for deployment?
- Can some gates be factorized into two smaller gates instead of one larger model?
- Does AST-native input encoding reduce effective entropy compared with raw text?
- Can Hivemind use failure and abstention data to identify where new deterministic
  tools should be built?

## Connection To Hivemind

This is where the projects meet:

- Hivemind identifies high-entropy surfaces with poor deterministic support
- `whetstone_RSA` tries to collapse some of those surfaces into bounded gates
- residual high-entropy zones become candidates for larger models or new tools

That makes entropy estimation useful for both deployment and roadmap planning.