whetstone_RSA/docs/entropy_model_sizing.md

Entropy To Model Sizing

Goal

Estimate the smallest model tier that can fill a given bounded decision gate with acceptable accuracy and calibration.

This project needs a practical rule of the form:

gate entropy score -> recommended model tier

Why This Matters

Right now model choice is too manual:

• an ~800KB specialist works for some gates
• some gates appear to exceed that capacity
• the next attempts are 10x and 16x larger

That is directionally right, but it should become measurable.

Working Definition

Gate entropy is not just Shannon entropy over labels.

For this project, it should mean the difficulty of mapping available input context onto the correct bounded output under realistic variation.

Proposed Factors

Each gate gets a score assembled from observable features:

• output_cardinality
• class_balance
• input_length_distribution
• context_width_required
• slot_count
• slot_interdependence
• label_boundary_fuzziness
• lexical_variation
• hidden_state_dependence
• world_knowledge_dependence
• error_cost
• abstain_availability

Some of these increase model demand directly. Others change the acceptable confidence threshold and escalation policy.

First Experimental Plan

For each Whetstone gate:

1. Define a fixed training and evaluation set.
2. Train multiple model tiers on the same gate.
3. Measure accuracy, calibration, confusion concentration, and latency.
4. Record the smallest tier that clears the target.

Suggested initial tiers:

• deterministic baseline when possible
• tiny specialist baseline: current ~213K-param / ~800KB tier
• medium specialist: ~10x current size
• large specialist: ~16x current size

Deliverable

Build a baseline table like:

gate_id | entropy_score | smallest_passing_tier | accuracy | ece | latency_ms

Then fit a first-pass policy:

if entropy <= A -> deterministic
if A < entropy <= B -> tiny specialist
if B < entropy <= C -> medium specialist
if entropy > C -> large specialist or escalate

Important Caveat

A larger model should not be the only answer.

Before moving up a tier, the evaluation should check:

• is the schema wrong?
• is the label set under-enumerated?
• is important context missing?
• are we forcing a fuzzy gate where a deterministic pre-pass should exist?

Model size should increase only after the gate definition itself is defensible.

Open Questions

• Which entropy factors are predictive enough to automate tier selection?
• Is calibration a better gating signal than raw accuracy for deployment?
• Can some gates be factorized into two smaller gates instead of one larger model?
• Does AST-native input encoding reduce effective entropy compared with raw text?
• Can Hivemind use failure and abstention data to identify where new deterministic tools should be built?

Connection To Hivemind

This is where the projects meet:

• Hivemind identifies high-entropy surfaces with poor deterministic support
• whetstone_RSA tries to collapse some of those surfaces into bounded gates
• residual high-entropy zones become candidates for larger models or new tools

That makes entropy estimation useful for both deployment and roadmap planning.