whetstone_RSA/sprints/SPRINT-001-gate-failure-taxonomy.md

Sprint 001: Gate Failure Taxonomy

Status

In Progress

Intent

Define an abstract, library-level framework for diagnosing why bounded decision gates fail and what restructuring options whetstone_RSA should support.

This sprint is explicitly about the RSA library.

It is not about changing whetstone_DSL. whetstone_DSL is used only as a case study to surface recurring gate failure patterns that the library should handle generically.

Problem

Some bounded gates train well at tiny model size. Others plateau at mediocre accuracy, and it is unclear whether the right response is:

• a larger model
• a better confidence policy
• a decomposed gate structure
• a deterministic pre-pass
• richer input context
• or a relabeling of the decision space

Right now that diagnosis depends too much on a knowledgeable human noticing the pattern by inspection.

The RSA library should instead provide abstract mechanisms for:

• identifying failure signatures
• classifying likely root causes
• probing alternative gate structures
• recommending safer deployment policies

Scope

This sprint produces a first-pass taxonomy and design contract for gate diagnosis.

Primary output:

• docs/gate_failure_taxonomy.md

Supporting updates as needed:

• docs/architecture.md
• docs/evaluation_policy.md
• docs/gate_rubric.md

Case Study Constraint

Use whetstone_DSL/specialists as evidence only.

Do not propose direct code changes to whetstone_DSL as the sprint output. Instead, convert the observed patterns into reusable RSA abstractions.

Working Hypothesis

Poor bounded-gate performance can usually be grouped into a small number of failure types that should be recognized by the library:

• capacity_limited_gate
• factorizable_gate
• non_stationary_gate
• missing_context_gate
• deterministic_disguised_as_ml_gate
• label_space_mismatch

The library should be able to recommend different responses for each failure type without requiring a human to rediscover the pattern each time.

Deliverables

1. Failure Taxonomy

Define each failure class with:

• description
• characteristic symptoms
• likely root causes
• recommended probes
• recommended interventions

2. Probe Model

Define library-level diagnostic probes such as:

• confidence-threshold probe
• class-confusion probe
• factorization probe
• deterministic-rule probe
• schema-stability probe
• context-width probe

These are conceptual probes first, not necessarily implemented code in this sprint.

3. Restructuring Policy

Define abstract restructuring options available to RSA:

• keep as single multiclass gate
• factor into multiple smaller gates
• convert part of the problem into deterministic rules
• change to hierarchical routing
• abstain and escalate
• relabel or merge classes

4. Library Contract Implications

Describe what the runtime should eventually support so this taxonomy is usable:

• gate metadata sufficient for diagnosis
• pluggable probes
• structured recommendations
• logging fields that make diagnosis possible

Why This Matters

Without a failure taxonomy, model-size escalation becomes the default answer to any weak gate.

That is the wrong default.

The library should first ask:

• is this actually one gate?
• is the label space stable?
• is the model missing critical context?
• is part of this already deterministic?
• is the policy target changing as the system evolves?

Only after those questions are answered should model scale be the main lever.

WhetstoneDSL-Derived Evidence To Capture

prereq_op

Observed pattern:

• tiny-tier performance around 69%
• combined 4-way label built from two latent binary decisions
• heuristic keyword labeling over short text

Library lesson:

• multiclass gates may hide compositional structure
• factorization probes should be first-class

automatability

Observed pattern:

• tiny-tier performance around 69%
• labels depend partly on current system capabilities and tooling maturity

Library lesson:

• some gates are non-stationary
• schema-stability checks must be first-class

confidence_tier

Observed pattern:

• perfect or near-perfect behavior because the task is effectively formulaic

Library lesson:

• the library should detect when a gate belongs in deterministic tooling rather than learned routing

Acceptance Criteria

• A new taxonomy document exists in whetstone_RSA/docs/
• The taxonomy is framed in RSA-library terms, not WhetstoneDSL implementation terms
• The document includes at least five distinct failure classes
• Each class includes symptoms, probes, and recommended interventions
• The case study evidence is cited only as motivating evidence
• The sprint file is sufficient for a later session to resume work cleanly

Not In Scope

• modifying whetstone_DSL
• retraining any current specialist
• implementing all probes in code
• deciding final thresholds for every future application

Completed In This Sprint

• Wrote docs/gate_failure_taxonomy.md
• Updated docs/architecture.md to include gate diagnosis as a first-class concern
• Updated docs/evaluation_policy.md to route weak-gate triage through the taxonomy
• Wrote docs/gate_rubric.md to translate the case study into provisional policy

Next Step

Define the diagnosis API and probe contracts so the taxonomy can become executable library behavior rather than remaining documentation only.

Target artifact:

• docs/diagnosis_api.md