# Gate Failure Taxonomy ## Purpose This document defines a first-pass taxonomy for why bounded decision gates fail and how `whetstone_RSA` should reason about those failures. The goal is to make gate diagnosis a library capability rather than an ad hoc human judgment call. This taxonomy is motivated by the `whetstone_DSL` case study, but it is written as an RSA-library abstraction. ## Core Claim When a bounded gate performs poorly, the default answer should not be "use a larger model." The library should first determine what kind of failure is happening. Different failure types require different responses: - some need more capacity - some need better confidence policy - some should be decomposed - some should become deterministic - some have unstable labels and should not be treated as ordinary classifiers ## Failure Classes ### 1. `capacity_limited_gate` Description: The gate is well-defined and reasonably stationary, but the current model tier does not have enough capacity to separate the decision surface reliably. Characteristic symptoms: - performance improves with training but plateaus below target - label boundaries appear coherent under manual inspection - confusion is broad rather than concentrated in one structural pattern - richer models or larger context are plausible fixes Likely root causes: - too little model capacity - too little representational depth - long-range dependencies exceed current context handling - input compression is too lossy for the chosen tier Recommended probes: - larger-model comparison - context-width probe - calibration probe - feature ablation probe Recommended interventions: - move to a larger specialist tier - add better structured features - use a richer encoder while preserving the same gate schema ### 2. `factorizable_gate` Description: The gate is represented as one multiclass decision, but the actual semantics are a composition of smaller independent or semi-independent decisions. Characteristic symptoms: - flat multiclass performance despite apparently simple semantics - classes can be expressed as Cartesian products or hierarchical branches - confusion concentrates between composite labels - downstream action is itself assembled from multiple sub-decisions Likely root causes: - a product decision has been collapsed into one label - one model is being asked to infer multiple policy axes at once - the current evaluation obscures easier sub-structure Recommended probes: - factorization probe - hierarchical-gate probe - per-axis confidence probe Recommended interventions: - split into multiple smaller gates - add a deterministic combiner over sub-gates - replace flat multiclass routing with hierarchical routing ### 3. `non_stationary_gate` Description: The label semantics change over time because they depend on evolving tooling, policies, or environment state rather than a stable ontology. Characteristic symptoms: - labels become stale as the surrounding system improves - class definitions drift without changing the input domain - retraining is needed because the target policy moved, not because the model forgot - disagreements cluster at capability boundaries rather than lexical boundaries Likely root causes: - output labels encode current deployment policy - the gate depends on tool availability or maturity - class meaning is roadmap-dependent Recommended probes: - schema-stability probe - temporal re-label probe - policy-variant probe Recommended interventions: - separate stable ontology fields from changing policy fields - turn part of the decision into a policy layer above the model - use hierarchical routing where the first stage predicts a stable property and a later stage applies current deployment policy ### 4. `missing_context_gate` Description: The gate might be learnable, but the current input encoding omits the context needed to make the decision safely. Characteristic symptoms: - short-text performance is poor but humans request surrounding context - errors cluster on inputs that depend on hidden project or workflow state - adding metadata is expected to help more than scaling parameters - the same surface text would require different answers in different contexts Likely root causes: - important state is absent from the input - text summaries collapse policy-critical structure - the gate implicitly depends on project metadata, not just user language Recommended probes: - context-width probe - metadata augmentation probe - stateful-context probe Recommended interventions: - enrich inputs with structured metadata - separate text understanding from stateful decision logic - require abstention when needed context is unavailable ### 5. `deterministic_disguised_as_ml_gate` Description: The gate looks statistical at first glance, but the output is largely determined by known rules or a stable formula. Characteristic symptoms: - very high performance with synthetic or structural encodings - human explanation reduces to a crisp rule - model errors mostly reflect encoding mismatch, not true ambiguity - downstream users already trust a formula more than the classifier Likely root causes: - a deterministic rule was not implemented yet - the learned model is being used as a proxy for explicit policy - training data was easier to generate than writing the rule Recommended probes: - deterministic-rule probe - symbolic baseline probe - feature-rule extraction probe Recommended interventions: - replace with deterministic logic - keep the model only as a convenience wrapper over explicit rules - use the learned gate only when the rule inputs are partially missing ### 6. `label_space_mismatch` Description: The gate's labels are underspecified, overlapping, misordered, or otherwise not aligned with the actual decision boundary the system needs. Characteristic symptoms: - confusion clusters around neighboring or semantically overlapping labels - multiple labels seem equally valid for the same input - label guidelines are hard to explain concisely - accuracy does not reflect practical usefulness because some "mistakes" are close Likely root causes: - labels mix ontology and policy - labels are too coarse or too fine - adjacent classes are not operationally distinct Recommended probes: - confusion-structure probe - class-merge probe - hierarchical-label probe Recommended interventions: - merge or split labels - add hierarchy to the label space - redefine evaluation so high-cost confusions are distinguished from near-miss ones ### 7. `guardrail_limited_gate` Description: The gate is good enough to be useful, but not good enough to be trusted without a deployment policy around it. Characteristic symptoms: - raw accuracy is moderate but confidence is informative - abstention can isolate hard cases - downstream checks can catch many residual errors - accepted predictions are much stronger than overall predictions Likely root causes: - the model is solving an economically useful subset of the problem - deployment policy has not yet been calibrated Recommended probes: - confidence-threshold probe - abstain-coverage probe - retry-recovery probe - deterministic-check catch-rate probe Recommended interventions: - deploy with guardrails - set confidence thresholds by risk tier - add retry and escalation paths ## Diagnostic Probes The taxonomy is only useful if the library has standard probes. ### `confidence_threshold_probe` Question: Does the gate become safe and useful when low-confidence cases abstain? Outputs: - accept rate - accuracy on accepted - silent error estimate ### `factorization_probe` Question: Can a weak multiclass gate be re-expressed as multiple smaller gates with a better combined policy? Outputs: - candidate factorization structure - estimated per-axis difficulty - combined accuracy or accepted-precision comparison ### `deterministic_rule_probe` Question: Is the gate mostly encoding a crisp rule that should live in ordinary software? Outputs: - symbolic baseline score - overlap between model errors and rule violations ### `schema_stability_probe` Question: Are the labels stable over time, or do they move with deployment policy and tool availability? Outputs: - stability rating - ontology-vs-policy split recommendation ### `context_width_probe` Question: Is performance limited by missing project or workflow context? Outputs: - delta from richer context - recommended minimum context set ### `confusion_structure_probe` Question: Are the observed errors random, adjacent, hierarchical, or compositional? Outputs: - confusion topology - merge, split, or hierarchy recommendation ## Recommended Response Order The library should approach gate triage in this order: 1. Check whether the gate is deterministic in disguise. 2. Check whether the label space is stable and coherent. 3. Check whether the gate is factorizable. 4. Check whether critical context is missing. 5. Check whether a guarded deployment policy is already sufficient. 6. Only then treat scale as the primary lever. ## Library Implications To support this taxonomy, `whetstone_RSA` should eventually represent each gate with enough metadata for diagnosis. Suggested gate metadata: - `gate_id` - `label_schema` - `label_stability` - `task_family` - `input_modalities` - `supports_abstain` - `deterministic_baseline_available` - `candidate_factorizations` - `risk_tier` Suggested diagnostic result shape: ```text GateDiagnosis primary_failure_class secondary_failure_classes supporting_signals recommended_probes recommended_interventions confidence ``` ## Case-Study Motivators The case study motivates this taxonomy, but does not constrain it. - A gate like `prereq_op` suggests `factorizable_gate` - A gate like `automatability` suggests `non_stationary_gate` - A gate like `confidence_tier` suggests `deterministic_disguised_as_ml_gate` - A gate like `verification_type` or `worker_type` may become `guardrail_limited_gate` These examples are evidence of recurring patterns, not bespoke exceptions. ## Summary The main architectural insight is simple: Bounded-gate failure is not one problem. If the RSA library can classify failure types before scaling models blindly, it will be cheaper, safer, and easier to generalize across domains.