whetstone_RSA/docs/diagnosis_api.md

Diagnosis API

Purpose

This document sketches the library-facing diagnosis API for whetstone_RSA.

The goal is to make gate diagnosis executable as a runtime capability:

• ingest gate metadata and evaluation evidence
• run standard probes
• classify likely failure modes
• recommend interventions

This is still a design artifact, not a final implementation commitment.

Design Principles

• diagnosis should be model-agnostic
• probes should be pluggable
• recommendations should be structured
• the API should work with partial evidence
• case studies should inform the design, not hard-code domain assumptions

Core Types

GateDefinition

Represents the bounded decision contract.

Suggested fields:

GateDefinition
  gate_id
  task_family
  labels
  slots
  constraints
  risk_tier
  label_stability
  deterministic_baseline_available
  supports_abstain
  candidate_factorizations
  metadata

GateEvidence

Represents the observed behavior of a gate at a given model tier or policy setup.

Suggested fields:

GateEvidence
  gate_id
  model_tier
  raw_accuracy
  accept_rate
  accuracy_on_accepted
  abstain_rate
  retry_recovery_rate
  escalation_rate
  silent_error_rate
  calibration
  confusion_summary
  latency_ms
  compute_cost
  notes

ProbeRequest

Specifies a diagnosis run.

ProbeRequest
  gate_definition
  evidence
  enabled_probes
  policy_context

ProbeResult

Represents the output of one diagnostic probe.

ProbeResult
  probe_id
  status
  signals
  recommendation_hints
  confidence

GateDiagnosis

Represents the synthesized diagnosis result.

GateDiagnosis
  gate_id
  primary_failure_class
  secondary_failure_classes
  supporting_signals
  recommended_probes
  recommended_interventions
  recommended_policy
  confidence

Failure Class Enum

Suggested starting enum:

FailureClass
  capacity_limited_gate
  factorizable_gate
  non_stationary_gate
  missing_context_gate
  deterministic_disguised_as_ml_gate
  label_space_mismatch
  guardrail_limited_gate

Probe Interface

Each probe should have the same basic contract.

Probe
  id()
  supports(gate_definition, evidence) -> bool
  run(request) -> ProbeResult

This lets the library add probes gradually without changing the diagnosis surface.

Initial Probe Set

confidence_threshold_probe

Purpose:

Estimate whether the gate is already useful under abstain and confidence thresholding.

Inputs:

• confidence-bearing predictions
• risk tier
• accepted-policy thresholds

Outputs:

• projected accept rate
• projected accepted precision
• silent error estimate
• guardrail suitability hint

factorization_probe

Purpose:

Estimate whether a weak multiclass gate has a plausible lower-entropy factorization.

Inputs:

• label schema
• confusion summary
• candidate factorization metadata if available

Outputs:

• factorization plausibility score
• candidate sub-gate structure
• comparison hint versus flat multiclass form

deterministic_rule_probe

Purpose:

Estimate whether the gate should actually live in ordinary software.

Inputs:

• deterministic baseline metadata
• structured features if available
• current gate evidence

Outputs:

• deterministic plausibility score
• recommendation to replace, hybridize, or keep learned routing

schema_stability_probe

Purpose:

Estimate whether the labels are ontology-stable or deployment-policy-dependent.

Inputs:

• label definitions
• versioned notes or temporal evaluation evidence
• capability metadata

Outputs:

• stability rating
• ontology-vs-policy warning
• recommendation to split labels or add a policy layer

context_width_probe

Purpose:

Estimate whether weak performance is caused by missing context rather than missing capacity.

Inputs:

• input modality metadata
• evidence across different context budgets if available

Outputs:

• context sensitivity score
• minimum context recommendation

confusion_structure_probe

Purpose:

Classify whether the error pattern is adjacent, hierarchical, compositional, or diffuse.

Inputs:

• confusion summary
• label metadata

Outputs:

• confusion topology
• merge, split, or hierarchy recommendation

Diagnosis Flow

The library should support a default synthesis flow:

1. Validate gate metadata.
2. Run applicable probes.
3. Aggregate probe signals.
4. Rank likely failure classes.
5. Emit structured intervention recommendations.

The result should not be a paragraph. It should be machine-usable.

Intervention Enum

Suggested intervention vocabulary:

Intervention
  keep_current_gate
  deploy_with_guardrails
  raise_confidence_threshold
  add_retry_with_context
  add_deterministic_prepass
  replace_with_deterministic_logic
  factor_into_subgates
  switch_to_hierarchical_gate
  revise_label_schema
  enrich_input_context
  evaluate_larger_model_tier

Policy Recommendation Shape

Suggested policy structure:

PolicyRecommendation
  deploy_mode
  threshold_hint
  abstain_required
  retry_strategy
  escalation_targets

Where deploy_mode might be:

• auto_accept
• guarded_accept
• abstain_first
• research_only

Logging Requirements

If diagnosis is going to work over time, live gate runs need structured logging.

Minimum useful fields:

GateRunLog
  gate_id
  model_tier
  input_signature
  confidence
  accepted
  abstained
  retry_used
  escalation_target
  final_outcome
  deterministic_check_outcome
  override_outcome
  timestamp

Case-Study Mapping

The case study suggests how diagnosis should behave:

• a gate like confidence_tier should trigger strong deterministic_disguised_as_ml_gate signals
• a gate like verification_type or worker_type may resolve as guardrail_limited_gate
• a gate like prereq_op should make factorization_probe a prominent candidate
• a gate like automatability should make schema_stability_probe prominent

These mappings are examples, not hard-coded rules.

Near-Term Value

Even before implementation, this API sketch clarifies what the library must preserve:

• gate metadata
• evidence summaries
• structured probe results
• structured interventions

That keeps whetstone_RSA pointed toward a diagnosable routing system rather than just a pile of tiny models.