whetstone_RSA/docs/diagnosis_api.md

# Diagnosis API

## Purpose

This document sketches the library-facing diagnosis API for `whetstone_RSA`.

The goal is to make gate diagnosis executable as a runtime capability:

- ingest gate metadata and evaluation evidence
- run standard probes
- classify likely failure modes
- recommend interventions

This is still a design artifact, not a final implementation commitment.

## Design Principles

- diagnosis should be model-agnostic
- probes should be pluggable
- recommendations should be structured
- the API should work with partial evidence
- case studies should inform the design, not hard-code domain assumptions

## Core Types

### `GateDefinition`

Represents the bounded decision contract.

Suggested fields:

```text
GateDefinition
  gate_id
  task_family
  labels
  slots
  constraints
  risk_tier
  label_stability
  deterministic_baseline_available
  supports_abstain
  candidate_factorizations
  metadata
```

### `GateEvidence`

Represents the observed behavior of a gate at a given model tier or policy setup.

Suggested fields:

```text
GateEvidence
  gate_id
  model_tier
  raw_accuracy
  accept_rate
  accuracy_on_accepted
  abstain_rate
  retry_recovery_rate
  escalation_rate
  silent_error_rate
  calibration
  confusion_summary
  latency_ms
  compute_cost
  notes
```

### `ProbeRequest`

Specifies a diagnosis run.

```text
ProbeRequest
  gate_definition
  evidence
  enabled_probes
  policy_context
```

### `ProbeResult`

Represents the output of one diagnostic probe.

```text
ProbeResult
  probe_id
  status
  signals
  recommendation_hints
  confidence
```

### `GateDiagnosis`

Represents the synthesized diagnosis result.

```text
GateDiagnosis
  gate_id
  primary_failure_class
  secondary_failure_classes
  supporting_signals
  recommended_probes
  recommended_interventions
  recommended_policy
  confidence
```

## Failure Class Enum

Suggested starting enum:

```text
FailureClass
  capacity_limited_gate
  factorizable_gate
  non_stationary_gate
  missing_context_gate
  deterministic_disguised_as_ml_gate
  label_space_mismatch
  guardrail_limited_gate
```

## Probe Interface

Each probe should have the same basic contract.

```text
Probe
  id()
  supports(gate_definition, evidence) -> bool
  run(request) -> ProbeResult
```

This lets the library add probes gradually without changing the diagnosis surface.

## Initial Probe Set

### `confidence_threshold_probe`

Purpose:

Estimate whether the gate is already useful under abstain and confidence thresholding.

Inputs:

- confidence-bearing predictions
- risk tier
- accepted-policy thresholds

Outputs:

- projected accept rate
- projected accepted precision
- silent error estimate
- guardrail suitability hint

### `factorization_probe`

Purpose:

Estimate whether a weak multiclass gate has a plausible lower-entropy factorization.

Inputs:

- label schema
- confusion summary
- candidate factorization metadata if available

Outputs:

- factorization plausibility score
- candidate sub-gate structure
- comparison hint versus flat multiclass form

### `deterministic_rule_probe`

Purpose:

Estimate whether the gate should actually live in ordinary software.

Inputs:

- deterministic baseline metadata
- structured features if available
- current gate evidence

Outputs:

- deterministic plausibility score
- recommendation to replace, hybridize, or keep learned routing

### `schema_stability_probe`

Purpose:

Estimate whether the labels are ontology-stable or deployment-policy-dependent.

Inputs:

- label definitions
- versioned notes or temporal evaluation evidence
- capability metadata

Outputs:

- stability rating
- ontology-vs-policy warning
- recommendation to split labels or add a policy layer

### `context_width_probe`

Purpose:

Estimate whether weak performance is caused by missing context rather than missing
capacity.

Inputs:

- input modality metadata
- evidence across different context budgets if available

Outputs:

- context sensitivity score
- minimum context recommendation

### `confusion_structure_probe`

Purpose:

Classify whether the error pattern is adjacent, hierarchical, compositional, or
diffuse.

Inputs:

- confusion summary
- label metadata

Outputs:

- confusion topology
- merge, split, or hierarchy recommendation

## Diagnosis Flow

The library should support a default synthesis flow:

1. Validate gate metadata.
2. Run applicable probes.
3. Aggregate probe signals.
4. Rank likely failure classes.
5. Emit structured intervention recommendations.

The result should not be a paragraph. It should be machine-usable.

## Intervention Enum

Suggested intervention vocabulary:

```text
Intervention
  keep_current_gate
  deploy_with_guardrails
  raise_confidence_threshold
  add_retry_with_context
  add_deterministic_prepass
  replace_with_deterministic_logic
  factor_into_subgates
  switch_to_hierarchical_gate
  revise_label_schema
  enrich_input_context
  evaluate_larger_model_tier
```

## Policy Recommendation Shape

Suggested policy structure:

```text
PolicyRecommendation
  deploy_mode
  threshold_hint
  abstain_required
  retry_strategy
  escalation_targets
```

Where `deploy_mode` might be:

- `auto_accept`
- `guarded_accept`
- `abstain_first`
- `research_only`

## Logging Requirements

If diagnosis is going to work over time, live gate runs need structured logging.

Minimum useful fields:

```text
GateRunLog
  gate_id
  model_tier
  input_signature
  confidence
  accepted
  abstained
  retry_used
  escalation_target
  final_outcome
  deterministic_check_outcome
  override_outcome
  timestamp
```

## Case-Study Mapping

The case study suggests how diagnosis should behave:

- a gate like `confidence_tier` should trigger strong
  `deterministic_disguised_as_ml_gate` signals
- a gate like `verification_type` or `worker_type` may resolve as
  `guardrail_limited_gate`
- a gate like `prereq_op` should make `factorization_probe` a prominent candidate
- a gate like `automatability` should make `schema_stability_probe` prominent

These mappings are examples, not hard-coded rules.

## Near-Term Value

Even before implementation, this API sketch clarifies what the library must preserve:

- gate metadata
- evidence summaries
- structured probe results
- structured interventions

That keeps `whetstone_RSA` pointed toward a diagnosable routing system rather than
just a pile of tiny models.