# Gate Rubric ## Purpose This document gives a provisional deployment rubric for the current WhetstoneDSL specialist gates. The goal is not to rank models by benchmark score. The goal is to decide, per gate: - deploy now - deploy with guardrails - wait for larger-tier comparison - redesign the gate before scaling ## Rubric Buckets ### Deploy Now Use when: - the gate is effectively deterministic or near-deterministic - wrong decisions are easy to catch or reverse - the current model already clears the practical quality bar ### Deploy With Guardrails Use when: - the gate is useful at the current size - raw accuracy is not enough for blind auto-accept - confidence thresholding, retry, or deterministic checks can contain risk ### Wait For Large Run Use when: - the current tiny-tier result is too weak for safe deployment - the gate definition appears defensible - larger-tier quality data is actively being generated ### Redesign Before Scaling Use when: - the label space appears structurally muddy - the gate may really be multiple smaller gates - important input context is missing - bigger models would mostly compensate for a bad formulation ## Current WhetstoneDSL Gates ### `confidence_tier` - Tiny-tier result: `100%` - Provisional bucket: `Deploy Now` - Rationale: - this is very close to a formula task - it should be treated as a deterministic baseline whenever possible - specialist use is acceptable, but only as a convenience layer over a known rule ### `verification_type` - Tiny-tier result: about `84.9%` - Provisional bucket: `Deploy With Guardrails` - Rationale: - bounded 5-way task with real signal at the tiny tier - likely useful if accepted predictions have high precision - reversible in many workflows because verification type can be validated or corrected downstream - Recommended next step: - evaluate confidence-thresholded precision - inspect confusion concentration, especially adjacent classes ### `worker_type` - Tiny-tier result: about `88.5%` - Provisional bucket: `Deploy With Guardrails` - Rationale: - strong enough to be operationally useful if low-confidence cases abstain - wrong worker selection is often recoverable - likely a good candidate for first-pass routing plus retry - Recommended next step: - measure accepted precision and abstain coverage - add a cheap retry path with richer context ### `prereq_op` - Tiny-tier result: about `69.4%`, flat across steps 1000-4000 - Provisional bucket: `Redesign Before Scaling` - Rationale: - the current 4-way label is really a composition of two latent yes/no decisions: `needs architect review?` and `needs manual approval?` - the current generator maps those two axes into one of: `standard`, `needs_review`, `needs_approval`, `full_gates` - that means the model is being asked to infer a product label from short text, even though the natural decision structure is factorized - labels are heuristic and keyword-driven, which likely adds noise - scaling the model may help, but it may also just memorize fuzzy boundaries - Why it is hard: - short descriptions often under-specify whether a task is architectural, production-sensitive, both, or neither - many cues are policy-level rather than lexical - some examples mix "phase/integration/framework" signals with release/security signals, making the combined class boundary weak - the downstream action is itself compositional, which suggests the predictor should also be compositional - Recommended decomposition: - gate A: `requires_architect_review` → `yes/no` - gate B: `requires_manual_approval` → `yes/no` - deterministic combiner: - `00` -> `standard` - `10` -> `needs_review` - `01` -> `needs_approval` - `11` -> `full_gates` - Why decomposition is attractive: - each model solves a cleaner binary problem - outputs align directly with the policy semantics - confidence can be calibrated independently per axis - one axis may be deterministic or near-deterministic in some contexts - What to test before scaling: - binary performance for each axis - whether richer structured inputs outperform plain step text - whether some approval decisions should be deterministic policy checks instead ### `automatability` - Tiny-tier result: about `69.6%`, flat across steps 1000-4000 - Provisional bucket: `Wait For Large Run` and `Redesign Before Scaling` - Rationale: - this gate is conceptually important, but the label boundaries are likely moving targets because they depend on what tools and templates the system already has - adjacent classes are semantically close - the cost of some confusions is very asymmetric - Why it is hard: - the classes are not purely descriptive; they are partly roadmap-dependent - a node can move from `llm` to `specialist` or `template` as tooling improves - the difference between `template`, `specialist`, `slm`, and `llm` is partly an operational routing policy, not just an intrinsic property of the node - Recommended next step: - wait for current large runs to finish - inspect confusion matrix, not just headline accuracy - consider a hierarchical formulation - Possible decomposition: - gate A: `is deterministic/template sufficient?` - gate B: `if learned, is output bounded or open-vocabulary?` - gate C: `if open-vocabulary, does small-model routing suffice or is full LLM needed?` ## Immediate Operational Policy Until the larger runs complete, use this posture: - `confidence_tier`: deterministic or auto-deploy - `verification_type`: candidate for guarded deployment - `worker_type`: candidate for guarded deployment - `prereq_op`: do not blindly scale; test decomposed formulation first - `automatability`: keep training and collect richer evidence before committing ## Main Lesson When a bounded gate performs poorly, the first question should not be: "How much bigger should the model be?" The first questions should be: - is the gate actually one gate? - is the label space stable? - is the input missing policy-critical structure? - should part of the logic be deterministic? That is the point of this rubric.