# Evaluation Policy ## Purpose This document defines how bounded decision gates should be evaluated inside `whetstone_RSA`. The central rule is: Do not optimize for raw accuracy in isolation. Optimize for final correctness at acceptable latency and cost, while keeping silent wrong decisions below the gate's risk budget. ## Why Raw Accuracy Is Insufficient A gate with `70%` raw top-1 accuracy may still be useful if: - it handles easy cases confidently - it abstains on risky cases - retries recover a meaningful share of the abstentions - downstream deterministic checks catch many residual mistakes A gate with `97%` raw accuracy may still be a poor deployment choice if: - it requires 10x to 16x model size - latency becomes unacceptable - it blocks parallel deployment - the extra accuracy only reduces low-cost mistakes For bounded systems, the real question is: What is the expected cost of using this gate in the pipeline? ## Required Metrics Every gate evaluation should report these metrics: - `raw_accuracy` - `accept_rate` - `accuracy_on_accepted` - `abstain_rate` - `retry_recovery_rate` - `escalation_rate` - `silent_error_rate` - `avg_latency_ms` - `avg_compute_cost` Optional but recommended: - calibration error - confusion concentration - class-weighted error - deterministic-check catch rate - rollback or repair rate ## Definitions ### Raw Accuracy Top-1 accuracy on the full evaluation set with no abstention policy. Useful for research. Not enough for deployment. ### Accept Rate Fraction of inputs the gate handles automatically without abstaining. ### Accuracy On Accepted Accuracy measured only on predictions that pass the acceptance threshold. This is often more important than raw accuracy. ### Retry Recovery Rate Fraction of abstained or low-confidence cases that become correct after: - adding richer context - running a deterministic pre-pass - using a larger specialist ### Silent Error Rate Wrong decisions that are accepted automatically and are not caught by later deterministic checks before they can do damage. This is usually the key deployment risk metric. ## Gate Risk Tiers Every gate should be assigned a risk tier before model sizing decisions are made. ### Tier 1: Low Risk Characteristics: - mistakes are easy to reverse - downstream deterministic checks exist - retries are cheap - wrong branch selection does not poison later stages Policy: - lower raw accuracy can be acceptable - moderate abstain rate is fine - optimize for cheap throughput ### Tier 2: Medium Risk Characteristics: - mistakes cause wasted work or moderate workflow churn - some downstream correction exists, but not always - retries have non-trivial cost Policy: - accepted predictions need stronger precision - confidence thresholds should be conservative - escalation should be available ### Tier 3: High Risk Characteristics: - mistakes silently corrupt planning or structure - correction is costly or delayed - wrong decisions can contaminate later stages Policy: - silent error rate must be very low - abstain aggressively - escalate early - raw accuracy matters less than safe acceptance behavior ## Decision Rule For a given gate, accept a smaller model if it wins on the system frontier: - lower average cost - lower or equal silent error rate - acceptable final correctness after retry and escalation - acceptable latency Do not move to a larger model just because it raises raw accuracy. Move to a larger model only if: - silent errors remain too high after thresholding - retries do not recover enough cases - the gate is structurally valid and still under-capacity ## Pre-Scale Checks Before scaling a gate from tiny to medium or large, check: - is the output schema well-defined? - are the labels under-enumerated? - are there ambiguous labels that should be split? - is important context missing from the input encoding? - can the gate be decomposed into two simpler gates? - should part of the logic be deterministic instead? Use `docs/gate_failure_taxonomy.md` to classify which of these checks should be prioritized for a given weak gate. If any of those fail, bigger models may just hide a design problem. ## Recommended Evaluation Loop 1. Train and measure raw accuracy. 2. Calibrate confidence thresholds. 3. Measure `accuracy_on_accepted` and `silent_error_rate`. 4. Add deterministic checks and retry path. 5. Measure final correctness and average cost. 6. Only then compare larger model tiers. ## Working Heuristic Use this default posture until domain-specific policy replaces it: - low-risk gates: a base model in the `70-85%` range may still be deployable with abstain and retry - medium-risk gates: target stronger accepted precision, not just higher raw accuracy - high-risk gates: prioritize low silent-error rate over broad auto-accept coverage These are not hard thresholds. They are a reminder that the deployment objective is system utility, not benchmark vanity. ## WhetstoneDSL Case Study Current tiny-tier WhetstoneDSL results already show why this policy is needed: - `confidence_tier` reaches `100%`, but is essentially a formula task - `verification_type` reaches roughly `85%` - `worker_type` reaches roughly `88.5%` - `prereq_op` is flat at roughly `69.4%` - `automatability` is flat at roughly `69.6%` The correct interpretation is not: - "`69%` is bad, so scale immediately" The correct interpretation is: - determine whether those `69%` gates can be made safe with acceptance thresholds, decomposition, retries, and deterministic guards - then decide whether larger specialists are economically justified ## What To Log Each live decision should eventually log: - gate id - model tier - confidence - accepted or abstained - retry used or not - escalation target - final chosen action - deterministic validation outcome - final human or system override outcome That turns deployment into a continuous training and policy dataset instead of a one-time benchmark.