# Diagnosis API ## Purpose This document sketches the library-facing diagnosis API for `whetstone_RSA`. The goal is to make gate diagnosis executable as a runtime capability: - ingest gate metadata and evaluation evidence - run standard probes - classify likely failure modes - recommend interventions This is still a design artifact, not a final implementation commitment. ## Design Principles - diagnosis should be model-agnostic - probes should be pluggable - recommendations should be structured - the API should work with partial evidence - case studies should inform the design, not hard-code domain assumptions ## Core Types ### `GateDefinition` Represents the bounded decision contract. Suggested fields: ```text GateDefinition gate_id task_family labels slots constraints risk_tier label_stability deterministic_baseline_available supports_abstain candidate_factorizations metadata ``` ### `GateEvidence` Represents the observed behavior of a gate at a given model tier or policy setup. Suggested fields: ```text GateEvidence gate_id model_tier raw_accuracy accept_rate accuracy_on_accepted abstain_rate retry_recovery_rate escalation_rate silent_error_rate calibration confusion_summary latency_ms compute_cost notes ``` ### `ProbeRequest` Specifies a diagnosis run. ```text ProbeRequest gate_definition evidence enabled_probes policy_context ``` ### `ProbeResult` Represents the output of one diagnostic probe. ```text ProbeResult probe_id status signals recommendation_hints confidence ``` ### `GateDiagnosis` Represents the synthesized diagnosis result. ```text GateDiagnosis gate_id primary_failure_class secondary_failure_classes supporting_signals recommended_probes recommended_interventions recommended_policy confidence ``` ## Failure Class Enum Suggested starting enum: ```text FailureClass capacity_limited_gate factorizable_gate non_stationary_gate missing_context_gate deterministic_disguised_as_ml_gate label_space_mismatch guardrail_limited_gate ``` ## Probe Interface Each probe should have the same basic contract. ```text Probe id() supports(gate_definition, evidence) -> bool run(request) -> ProbeResult ``` This lets the library add probes gradually without changing the diagnosis surface. ## Initial Probe Set ### `confidence_threshold_probe` Purpose: Estimate whether the gate is already useful under abstain and confidence thresholding. Inputs: - confidence-bearing predictions - risk tier - accepted-policy thresholds Outputs: - projected accept rate - projected accepted precision - silent error estimate - guardrail suitability hint ### `factorization_probe` Purpose: Estimate whether a weak multiclass gate has a plausible lower-entropy factorization. Inputs: - label schema - confusion summary - candidate factorization metadata if available Outputs: - factorization plausibility score - candidate sub-gate structure - comparison hint versus flat multiclass form ### `deterministic_rule_probe` Purpose: Estimate whether the gate should actually live in ordinary software. Inputs: - deterministic baseline metadata - structured features if available - current gate evidence Outputs: - deterministic plausibility score - recommendation to replace, hybridize, or keep learned routing ### `schema_stability_probe` Purpose: Estimate whether the labels are ontology-stable or deployment-policy-dependent. Inputs: - label definitions - versioned notes or temporal evaluation evidence - capability metadata Outputs: - stability rating - ontology-vs-policy warning - recommendation to split labels or add a policy layer ### `context_width_probe` Purpose: Estimate whether weak performance is caused by missing context rather than missing capacity. Inputs: - input modality metadata - evidence across different context budgets if available Outputs: - context sensitivity score - minimum context recommendation ### `confusion_structure_probe` Purpose: Classify whether the error pattern is adjacent, hierarchical, compositional, or diffuse. Inputs: - confusion summary - label metadata Outputs: - confusion topology - merge, split, or hierarchy recommendation ## Diagnosis Flow The library should support a default synthesis flow: 1. Validate gate metadata. 2. Run applicable probes. 3. Aggregate probe signals. 4. Rank likely failure classes. 5. Emit structured intervention recommendations. The result should not be a paragraph. It should be machine-usable. ## Intervention Enum Suggested intervention vocabulary: ```text Intervention keep_current_gate deploy_with_guardrails raise_confidence_threshold add_retry_with_context add_deterministic_prepass replace_with_deterministic_logic factor_into_subgates switch_to_hierarchical_gate revise_label_schema enrich_input_context evaluate_larger_model_tier ``` ## Policy Recommendation Shape Suggested policy structure: ```text PolicyRecommendation deploy_mode threshold_hint abstain_required retry_strategy escalation_targets ``` Where `deploy_mode` might be: - `auto_accept` - `guarded_accept` - `abstain_first` - `research_only` ## Logging Requirements If diagnosis is going to work over time, live gate runs need structured logging. Minimum useful fields: ```text GateRunLog gate_id model_tier input_signature confidence accepted abstained retry_used escalation_target final_outcome deterministic_check_outcome override_outcome timestamp ``` ## Case-Study Mapping The case study suggests how diagnosis should behave: - a gate like `confidence_tier` should trigger strong `deterministic_disguised_as_ml_gate` signals - a gate like `verification_type` or `worker_type` may resolve as `guardrail_limited_gate` - a gate like `prereq_op` should make `factorization_probe` a prominent candidate - a gate like `automatability` should make `schema_stability_probe` prominent These mappings are examples, not hard-coded rules. ## Near-Term Value Even before implementation, this API sketch clarifies what the library must preserve: - gate metadata - evidence summaries - structured probe results - structured interventions That keeps `whetstone_RSA` pointed toward a diagnosable routing system rather than just a pile of tiny models.