160 lines
3.2 KiB
Markdown
160 lines
3.2 KiB
Markdown
# Probe Registry
|
|
|
|
## Purpose
|
|
|
|
This document defines the probe registry concept for `whetstone_RSA`.
|
|
|
|
The registry is how the diagnosis runtime discovers and executes diagnostic probes
|
|
without hard-coding them into one monolithic diagnosis function.
|
|
|
|
## Why A Registry Exists
|
|
|
|
The library should be able to add probes incrementally.
|
|
|
|
Examples:
|
|
|
|
- `confidence_threshold_probe`
|
|
- `factorization_probe`
|
|
- `deterministic_rule_probe`
|
|
- `schema_stability_probe`
|
|
- `context_width_probe`
|
|
- `confusion_structure_probe`
|
|
|
|
Some applications may also add domain-specific probes later.
|
|
|
|
The registry keeps that extensible while preserving a stable diagnosis surface.
|
|
|
|
## Probe Contract
|
|
|
|
Every probe should satisfy the same conceptual interface:
|
|
|
|
```text
|
|
Probe
|
|
id() -> string
|
|
supports(gate_definition, evidence) -> bool
|
|
run(probe_request) -> probe_result
|
|
```
|
|
|
|
## Probe Metadata
|
|
|
|
Each registered probe should expose metadata:
|
|
|
|
```text
|
|
ProbeMetadata
|
|
probe_id
|
|
description
|
|
required_inputs
|
|
optional_inputs
|
|
supported_failure_classes
|
|
produces_signals
|
|
```
|
|
|
|
This lets the runtime explain why a probe was or was not run.
|
|
|
|
## Registry Responsibilities
|
|
|
|
The probe registry should support:
|
|
|
|
- registration
|
|
- lookup by id
|
|
- listing available probes
|
|
- filtering applicable probes
|
|
- validating probe dependencies if any are added later
|
|
|
|
## Default Probe Set
|
|
|
|
Initial probes expected in the registry:
|
|
|
|
- `confidence_threshold_probe`
|
|
- `factorization_probe`
|
|
- `deterministic_rule_probe`
|
|
- `schema_stability_probe`
|
|
- `context_width_probe`
|
|
- `confusion_structure_probe`
|
|
|
|
## Probe Request Envelope
|
|
|
|
The runtime should present probes with a uniform request:
|
|
|
|
```text
|
|
ProbeRequest
|
|
gate_definition
|
|
evidence
|
|
policy_context
|
|
trace_context
|
|
```
|
|
|
|
This keeps probes focused on diagnosis logic rather than runtime plumbing.
|
|
|
|
## Probe Result Envelope
|
|
|
|
Each probe should return:
|
|
|
|
```text
|
|
ProbeResult
|
|
probe_id
|
|
status
|
|
signals
|
|
recommendation_hints
|
|
confidence
|
|
```
|
|
|
|
Suggested `status` values:
|
|
|
|
- `success`
|
|
- `not_applicable`
|
|
- `insufficient_evidence`
|
|
- `error`
|
|
|
|
## Applicability Rules
|
|
|
|
Probes should opt in explicitly.
|
|
|
|
Examples:
|
|
|
|
- `confidence_threshold_probe` should decline if confidence-bearing outputs are absent
|
|
- `factorization_probe` should decline if labels have no registered structure and no
|
|
confusion evidence is available
|
|
- `deterministic_rule_probe` should decline if no structured features or baseline rule exists
|
|
|
|
This keeps diagnosis honest and prevents invented certainty.
|
|
|
|
## Synthesis Boundary
|
|
|
|
The registry executes probes.
|
|
It does not decide the final diagnosis.
|
|
|
|
Final diagnosis should happen in a separate synthesis step that:
|
|
|
|
- aggregates probe signals
|
|
- ranks failure classes
|
|
- ranks interventions
|
|
|
|
That separation matters because:
|
|
|
|
- probes are local
|
|
- diagnosis is global
|
|
|
|
## Future-Proofing
|
|
|
|
The registry design should tolerate:
|
|
|
|
- additional probes
|
|
- domain-specific probes
|
|
- partial evidence
|
|
- multiple runtime implementations
|
|
|
|
The registry should not assume one model family, one training stack, or one
|
|
application domain.
|
|
|
|
## Minimal Viable Implementation
|
|
|
|
The first probe registry can be very small:
|
|
|
|
- static in-memory registration
|
|
- simple `supports` checks
|
|
- direct execution
|
|
- JSON-serializable results
|
|
|
|
That is enough to validate the architecture before adding plugin systems or dynamic loading.
|