whetstone_RSA/docs/probe_registry.md

Probe Registry

Purpose

This document defines the probe registry concept for whetstone_RSA.

The registry is how the diagnosis runtime discovers and executes diagnostic probes without hard-coding them into one monolithic diagnosis function.

Why A Registry Exists

The library should be able to add probes incrementally.

Examples:

• confidence_threshold_probe
• factorization_probe
• deterministic_rule_probe
• schema_stability_probe
• context_width_probe
• confusion_structure_probe

Some applications may also add domain-specific probes later.

The registry keeps that extensible while preserving a stable diagnosis surface.

Probe Contract

Every probe should satisfy the same conceptual interface:

Probe
  id() -> string
  supports(gate_definition, evidence) -> bool
  run(probe_request) -> probe_result

Probe Metadata

Each registered probe should expose metadata:

ProbeMetadata
  probe_id
  description
  required_inputs
  optional_inputs
  supported_failure_classes
  produces_signals

This lets the runtime explain why a probe was or was not run.

Registry Responsibilities

The probe registry should support:

• registration
• lookup by id
• listing available probes
• filtering applicable probes
• validating probe dependencies if any are added later

Default Probe Set

Initial probes expected in the registry:

• confidence_threshold_probe
• factorization_probe
• deterministic_rule_probe
• schema_stability_probe
• context_width_probe
• confusion_structure_probe

Probe Request Envelope

The runtime should present probes with a uniform request:

ProbeRequest
  gate_definition
  evidence
  policy_context
  trace_context

This keeps probes focused on diagnosis logic rather than runtime plumbing.

Probe Result Envelope

Each probe should return:

ProbeResult
  probe_id
  status
  signals
  recommendation_hints
  confidence

Suggested status values:

• success
• not_applicable
• insufficient_evidence
• error

Applicability Rules

Probes should opt in explicitly.

Examples:

• confidence_threshold_probe should decline if confidence-bearing outputs are absent
• factorization_probe should decline if labels have no registered structure and no confusion evidence is available
• deterministic_rule_probe should decline if no structured features or baseline rule exists

This keeps diagnosis honest and prevents invented certainty.

Synthesis Boundary

The registry executes probes. It does not decide the final diagnosis.

Final diagnosis should happen in a separate synthesis step that:

• aggregates probe signals
• ranks failure classes
• ranks interventions

That separation matters because:

• probes are local
• diagnosis is global

Future-Proofing

The registry design should tolerate:

• additional probes
• domain-specific probes
• partial evidence
• multiple runtime implementations

The registry should not assume one model family, one training stack, or one application domain.

Minimal Viable Implementation

The first probe registry can be very small:

• static in-memory registration
• simple supports checks
• direct execution
• JSON-serializable results

That is enough to validate the architecture before adding plugin systems or dynamic loading.