Initial whetstone_RSA architecture and C++ scaffold

docs/diagnosis_api.md

@@ -0,0 +1,344 @@
+# Diagnosis API
+
+## Purpose
+
+This document sketches the library-facing diagnosis API for `whetstone_RSA`.
+
+The goal is to make gate diagnosis executable as a runtime capability:
+
+- ingest gate metadata and evaluation evidence
+- run standard probes
+- classify likely failure modes
+- recommend interventions
+
+This is still a design artifact, not a final implementation commitment.
+
+## Design Principles
+
+- diagnosis should be model-agnostic
+- probes should be pluggable
+- recommendations should be structured
+- the API should work with partial evidence
+- case studies should inform the design, not hard-code domain assumptions
+
+## Core Types
+
+### `GateDefinition`
+
+Represents the bounded decision contract.
+
+Suggested fields:
+
+```text
+GateDefinition
+  gate_id
+  task_family
+  labels
+  slots
+  constraints
+  risk_tier
+  label_stability
+  deterministic_baseline_available
+  supports_abstain
+  candidate_factorizations
+  metadata
+```
+
+### `GateEvidence`
+
+Represents the observed behavior of a gate at a given model tier or policy setup.
+
+Suggested fields:
+
+```text
+GateEvidence
+  gate_id
+  model_tier
+  raw_accuracy
+  accept_rate
+  accuracy_on_accepted
+  abstain_rate
+  retry_recovery_rate
+  escalation_rate
+  silent_error_rate
+  calibration
+  confusion_summary
+  latency_ms
+  compute_cost
+  notes
+```
+
+### `ProbeRequest`
+
+Specifies a diagnosis run.
+
+```text
+ProbeRequest
+  gate_definition
+  evidence
+  enabled_probes
+  policy_context
+```
+
+### `ProbeResult`
+
+Represents the output of one diagnostic probe.
+
+```text
+ProbeResult
+  probe_id
+  status
+  signals
+  recommendation_hints
+  confidence
+```
+
+### `GateDiagnosis`
+
+Represents the synthesized diagnosis result.
+
+```text
+GateDiagnosis
+  gate_id
+  primary_failure_class
+  secondary_failure_classes
+  supporting_signals
+  recommended_probes
+  recommended_interventions
+  recommended_policy
+  confidence
+```
+
+## Failure Class Enum
+
+Suggested starting enum:
+
+```text
+FailureClass
+  capacity_limited_gate
+  factorizable_gate
+  non_stationary_gate
+  missing_context_gate
+  deterministic_disguised_as_ml_gate
+  label_space_mismatch
+  guardrail_limited_gate
+```
+
+## Probe Interface
+
+Each probe should have the same basic contract.
+
+```text
+Probe
+  id()
+  supports(gate_definition, evidence) -> bool
+  run(request) -> ProbeResult
+```
+
+This lets the library add probes gradually without changing the diagnosis surface.
+
+## Initial Probe Set
+
+### `confidence_threshold_probe`
+
+Purpose:
+
+Estimate whether the gate is already useful under abstain and confidence thresholding.
+
+Inputs:
+
+- confidence-bearing predictions
+- risk tier
+- accepted-policy thresholds
+
+Outputs:
+
+- projected accept rate
+- projected accepted precision
+- silent error estimate
+- guardrail suitability hint
+
+### `factorization_probe`
+
+Purpose:
+
+Estimate whether a weak multiclass gate has a plausible lower-entropy factorization.
+
+Inputs:
+
+- label schema
+- confusion summary
+- candidate factorization metadata if available
+
+Outputs:
+
+- factorization plausibility score
+- candidate sub-gate structure
+- comparison hint versus flat multiclass form
+
+### `deterministic_rule_probe`
+
+Purpose:
+
+Estimate whether the gate should actually live in ordinary software.
+
+Inputs:
+
+- deterministic baseline metadata
+- structured features if available
+- current gate evidence
+
+Outputs:
+
+- deterministic plausibility score
+- recommendation to replace, hybridize, or keep learned routing
+
+### `schema_stability_probe`
+
+Purpose:
+
+Estimate whether the labels are ontology-stable or deployment-policy-dependent.
+
+Inputs:
+
+- label definitions
+- versioned notes or temporal evaluation evidence
+- capability metadata
+
+Outputs:
+
+- stability rating
+- ontology-vs-policy warning
+- recommendation to split labels or add a policy layer
+
+### `context_width_probe`
+
+Purpose:
+
+Estimate whether weak performance is caused by missing context rather than missing
+capacity.
+
+Inputs:
+
+- input modality metadata
+- evidence across different context budgets if available
+
+Outputs:
+
+- context sensitivity score
+- minimum context recommendation
+
+### `confusion_structure_probe`
+
+Purpose:
+
+Classify whether the error pattern is adjacent, hierarchical, compositional, or
+diffuse.
+
+Inputs:
+
+- confusion summary
+- label metadata
+
+Outputs:
+
+- confusion topology
+- merge, split, or hierarchy recommendation
+
+## Diagnosis Flow
+
+The library should support a default synthesis flow:
+
+1. Validate gate metadata.
+2. Run applicable probes.
+3. Aggregate probe signals.
+4. Rank likely failure classes.
+5. Emit structured intervention recommendations.
+
+The result should not be a paragraph. It should be machine-usable.
+
+## Intervention Enum
+
+Suggested intervention vocabulary:
+
+```text
+Intervention
+  keep_current_gate
+  deploy_with_guardrails
+  raise_confidence_threshold
+  add_retry_with_context
+  add_deterministic_prepass
+  replace_with_deterministic_logic
+  factor_into_subgates
+  switch_to_hierarchical_gate
+  revise_label_schema
+  enrich_input_context
+  evaluate_larger_model_tier
+```
+
+## Policy Recommendation Shape
+
+Suggested policy structure:
+
+```text
+PolicyRecommendation
+  deploy_mode
+  threshold_hint
+  abstain_required
+  retry_strategy
+  escalation_targets
+```
+
+Where `deploy_mode` might be:
+
+- `auto_accept`
+- `guarded_accept`
+- `abstain_first`
+- `research_only`
+
+## Logging Requirements
+
+If diagnosis is going to work over time, live gate runs need structured logging.
+
+Minimum useful fields:
+
+```text
+GateRunLog
+  gate_id
+  model_tier
+  input_signature
+  confidence
+  accepted
+  abstained
+  retry_used
+  escalation_target
+  final_outcome
+  deterministic_check_outcome
+  override_outcome
+  timestamp
+```
+
+## Case-Study Mapping
+
+The case study suggests how diagnosis should behave:
+
+- a gate like `confidence_tier` should trigger strong
+  `deterministic_disguised_as_ml_gate` signals
+- a gate like `verification_type` or `worker_type` may resolve as
+  `guardrail_limited_gate`
+- a gate like `prereq_op` should make `factorization_probe` a prominent candidate
+- a gate like `automatability` should make `schema_stability_probe` prominent
+
+These mappings are examples, not hard-coded rules.
+
+## Near-Term Value
+
+Even before implementation, this API sketch clarifies what the library must preserve:
+
+- gate metadata
+- evidence summaries
+- structured probe results
+- structured interventions
+
+That keeps `whetstone_RSA` pointed toward a diagnosable routing system rather than
+just a pile of tiny models.