whetstone_RSA/docs/whetstone_ast_first_integration.md

# Whetstone-AST-First Integration

## Purpose

This document defines how `whetstone_RSA` should relate to `whetstone_DSL`.

The key rule is:

`whetstone_RSA` should be Whetstone-AST-first, not merely AST-first.

That means the semantic source of truth for RSA contracts should live in the
Whetstone AST and annotation system, not in ad hoc C++ structs or generic parser
ASTs.

## Why Generic AST Is Not Enough

A generic AST is mostly syntax structure.

It can tell you things like:

- function
- class
- parameter
- if-statement
- call expression

That is useful, but it is not enough for RSA.

RSA needs first-class semantic contracts for things like:

- input structure
- input modality
- output topology
- output constraint
- policy stability
- suitability assessment
- gate shape
- confidence behavior
- abstain policy
- escalation policy
- diagnosis metadata

Those are not reliable consequences of syntax alone.
They are authored semantic data.

## Why Whetstone AST Is The Right Source

The Whetstone AST already exists to carry language-neutral semantic intent beyond
what ordinary source code or parser ASTs can express cleanly.

That makes it the right authoring substrate for RSA because:

- annotations are first-class
- concepts are language-neutral
- contracts can be preserved across projections
- comments do not need to carry architectural meaning
- generated C++ can remain a projection target instead of becoming the semantic home

In short:

- generic AST gives structure
- Whetstone AST gives structure plus semantic contract capacity

RSA needs the second one.

## Architectural Rule

The semantic source of truth for RSA should live in `whetstone_DSL`.

The C++ runtime in `whetstone_RSA` should be treated as a projection target for:

- runtime data types
- execution logic
- probe registry interfaces
- policy and diagnosis runtime behavior

This keeps the architecture aligned with the rest of the Whetstone program:

- AST is source of truth
- target languages are projections

## Proposed Split Of Responsibilities

### `whetstone_DSL`

Owns:

- RSA semantic ontology
- gate definitions as semantic contracts
- suitability metadata
- diagnosis metadata
- probe declarations
- projection rules into target runtimes

### `whetstone_RSA`

Owns:

- C++ runtime implementation
- probe execution engine
- diagnosis engine
- policy runtime
- compiled/runtime-facing representations of projected contracts

## What Should Exist In The Whetstone AST

RSA needs explicit semantic objects, not just ordinary code nodes.

Examples of first-class RSA concepts that should eventually exist in Whetstone form:

- `GateDefinition`
- `GateEvidence`
- `GateDiagnosis`
- `GateSuitability`
- `PolicyRecommendation`
- `ProbeDefinition`
- `DecisionContract`

These should be represented as semantic entities with structured fields, not as
freeform comments embedded in generated C++.

## Why This Is Better Than Authoring In C++

If RSA contracts are authored directly in C++, the semantics become fragile:

- comments carry architectural meaning
- relationships are implicit
- code review sees implementation before contract
- projection to other targets becomes harder

If RSA contracts are authored in the Whetstone AST:

- the semantics are explicit
- annotations stay attached to the real concept
- multiple targets can be generated later
- runtime code becomes easier to keep honest

This is especially important because RSA is intended to remain model-agnostic.

The contract is primary.
The current transformer-based runtime is secondary.

## Model-Agnostic Benefit

Because RSA is contractual rather than model-defined, the AST layer should describe:

- what the gate promises
- what the gate consumes
- how confidence and abstention behave
- how diagnosis works

without hard-coding one backend.

That preserves future replacement paths such as:

- transformer specialist
- classical ML classifier
- RL policy
- retrieval-backed scorer
- deterministic-plus-learned hybrid

The Whetstone AST should capture the contract.
The C++ runtime should implement one or more backends against it.

## Near-Term Practical Rule

Right now, some C++ scaffolding already exists in `whetstone_RSA`.

That is fine as a bootstrap.

But it should be treated as:

- provisional runtime code
- useful for exercising the contract shape
- not the long-term semantic source of truth

As soon as the Whetstone-side semantic representation exists, the C++ layer should
be aligned to it rather than continuing as an independently evolving hand-authored
model.

## How To Use WhetstoneDSL Here

The right use of `whetstone_DSL` is not "parse some C++ and hope the meaning falls
out."

The right use is:

1. define RSA concepts explicitly in Whetstone semantic form
2. attach first-class metadata and annotations there
3. project those contracts into C++ runtime types and adapters

If existing Whetstone MCP tools can help bootstrap some of the current C++ structs
into AST-native representations, that is useful.

But the goal is not mechanical conversion for its own sake.
The goal is semantic alignment.

If direct manual authoring in Whetstone form lands closer to the intended contract,
that is better than a weak automatic conversion.

## Practical Guidance For The Current Stage

At this stage of the project:

- keep the current C++ runtime scaffold because it is useful for validating runtime shape
- do not expand C++ into the permanent semantic home of the system
- prioritize defining the RSA domain model in Whetstone semantic terms
- use MCP-assisted conversion only when it lands close to the intended semantic contract

This is still early enough that correctness of semantic shape matters more than
automation of the conversion path.

## Implication

`whetstone_RSA` is not just "a C++ library with some docs."

It should become:

- a Whetstone-authored semantic subsystem
- with a C++ runtime projection

That is the architecture most consistent with both Whetstone and the long-term
goals of RSA.

## Bootstrap Mapping From The Current C++ Scaffold

The current C++ code is useful bootstrap material, but it should now be treated as
input to the semantic model rather than the long-term definition of that model.

This section inventories the current scaffold and classifies each piece as either:

- semantic contract that should move into Whetstone-authored form
- runtime behavior that should remain in projected C++
- bootstrap/demo code that should not define long-term architecture

### Current C++ Semantic Surface

From `include/whetstone_rsa/types.h`, the current scaffold already expresses the
core semantic vocabulary that should be re-authored in Whetstone form:

- ontology enums
  - `InputStructure`
  - `InputModality`
  - `OutputTopology`
  - `OutputConstraint`
  - `LabelStability`
  - `TargetLayer`
- diagnosis enums
  - `FailureClass`
  - `Intervention`
  - `DeployMode`
  - `RiskTier`
- contract entities
  - `GateDefinition`
  - `GateEvidence`
  - `PolicyContext`
  - `ProbeRequest`
  - `ProbeResult`
  - `GateDiagnosis`
  - `GateSuitability`
  - `PolicyRecommendation`

These should not remain primarily hand-authored C++ concepts.

They should become semantic entities in the Whetstone source of truth.

### Recommended Whetstone Semantic Entities

The current scaffold suggests the following first semantic model.

#### 1. `DecisionContract`

This should become the semantic home for the ontology-level fields that currently
sit inside `GateDefinition`.

Proposed fields:

- `gate_id`
- `task_family`
- `input_structure`
- `input_modality`
- `output_topology`
- `output_constraint`
- `risk_tier`
- `policy_stability`
- `deterministic_baseline_available`
- `supports_abstain`
- `metadata`

This is the contract boundary, independent of backend family.

#### 2. `DecisionSurface`

The current `GateDefinition.labels` plus `candidate_factorizations` really describe
the shape of the output space, not just the gate identity.

Proposed fields:

- `output_labels`
- `slot_schema`
- `candidate_factorizations`
- `candidate_gate_shapes`
- `invalid_combinations`
- `deterministic_prepass_options`

This gives the AST layer a place to describe flat multiclass, hierarchical, or
factorized designs without forcing that structure to remain implicit in C++.

#### 3. `DecisionRequestModel`

The current C++ scaffold does not yet have a first-class request packet type
separate from evaluation/demo inputs.

That should be corrected in the AST-first model.

Proposed fields:

- `packet_id`
- `text_fields`
- `symbolic_fields`
- `numeric_fields`
- `context_fields`
- `state_fields`
- `provenance`

This entity should support both enum-style gates and bounded structured outputs.

#### 4. `DecisionResultModel`

The current scaffold implicitly assumes label selection and policy behavior, but the
AST-first model should make result structure explicit.

Proposed fields:

- `decision_id`
- `selected_label`
- `filled_slots`
- `confidence`
- `abstain`
- `retry_strategy`
- `escalation_target`
- `trace_packet`

This should later project into both C++ runtime structs and packaging artifacts.

#### 5. `DiagnosisModel`

The current `GateDiagnosis` is already close to a semantic object and should be
preserved almost directly, but moved into Whetstone-authored form.

Proposed fields:

- `primary_failure_class`
- `secondary_failure_classes`
- `supporting_signals`
- `recommended_probes`
- `recommended_interventions`
- `recommended_policy`
- `confidence`

#### 6. `ProbeDefinition`

The current `Probe` interface and `ProbeRequest` / `ProbeResult` pair indicate that
probes are part of the semantic model, not just implementation detail.

The AST layer should explicitly represent:

- `probe_id`
- `applies_to_gate_shapes`
- `required_evidence_fields`
- `produced_signals`
- `failure_hints`
- `intervention_hints`

Concrete runtime probes such as `confidence_threshold_probe` should then be C++
implementations of these declared probe contracts.

#### 7. `SuitabilityAssessment`

The current `GateSuitability` type should become a semantic output model for the
pre-RSA decision that determines whether the problem belongs in the RSA layer at all.

Proposed fields:

- `recommended`
- `target_layer`
- `rationale`
- `bounded_output_confidence`
- `deterministic_baseline_available`
- `label_stability_risk`
- `suggested_gate_shape`

#### 8. `BackendSuitability`

This does not yet exist in code, but the roadmap now requires it.

Proposed fields:

- `gate_id`
- `candidate_backends`
- `recommended_backend`
- `rationale`
- `confidence`
- `benchmark_refs`

This must be semantic-first, not added ad hoc in C++ later.

#### 9. `GatePackage`

The runtime needs a durable packaged form for serving decisions after offline
selection and calibration.

Proposed fields:

- `gate_id`
- `contract_ref`
- `gate_shape`
- `backend_family`
- `artifact_ref`
- `thresholds`
- `abstain_policy`
- `escalation_policy`
- `trace_schema`
- `version`

### What Should Remain Runtime-Only In C++

The following concepts should remain projected runtime behavior rather than become
the semantic source of truth:

- `Probe` virtual interface
- `ProbeRegistry`
- `DiagnosisEngine`
- `SuitabilityAssessor`
- backend adapter implementations
- calibration code
- packaging loaders
- benchmark runners

These are execution mechanisms, not the semantic model itself.

### What In The Current Scaffold Is Only Bootstrap

The following parts are useful but should not steer the long-term semantic design:

- `examples/diagnose_demo.cpp`
- the current hand-authored stringification helpers
- any current field arrangement that exists only because it was convenient for the
  first C++ scaffold

### Immediate Migration Map

The practical next move should be:

1. Represent the current C++ contract entities above as Whetstone semantic objects.
2. Mark which current fields are semantic contract versus runtime-only detail.
3. Treat `types.h` as a temporary projected contract snapshot, not the authoring
   home of the model.
4. Add projection rules so future C++ runtime types are derived from the semantic
   source rather than independently evolved.

### Most Important Boundary To Preserve

The current scaffold already mixes two different layers that must stay separate:

- semantic contract
- runtime execution strategy

If that separation is preserved now, `whetstone_RSA` can become genuinely AST-first.
If it is not preserved, the project will keep drifting into a C++-first runtime
with AST plans attached afterward.

## First RSA Semantic Model

This section defines the first concrete semantic model that should become the
source of truth for `whetstone_RSA`.

It is intentionally small and close to the current scaffold so migration can
start immediately without pretending the architecture is already finished.

The design rule is:

- semantic entities describe contract and meaning
- projected C++ types describe runtime execution

### Model Goals

The first semantic model must support all of the following from the beginning:

- bounded enum decisions
- bounded structured slot-filling decisions
- suitability assessment before a gate is adopted
- diagnosis after a gate underperforms
- gate-shape comparison
- backend-selection metadata
- runtime packaging metadata

### Semantic Entity Set

The first semantic model should consist of these entities:

- `DecisionContract`
- `DecisionSurface`
- `DecisionRequestModel`
- `DecisionResultModel`
- `SuitabilityAssessment`
- `DiagnosisModel`
- `ProbeDefinition`
- `BackendSuitability`
- `GatePackage`

### 1. `DecisionContract`

This is the semantic identity of an RSA gate.

It answers:

- what decision surface exists
- what kind of inputs it consumes
- what kind of bounded outputs it promises
- what policy constraints govern deployment

Proposed fields:

```text
DecisionContract
  gate_id
  task_family
  input_structure
  input_modality
  output_topology
  output_constraint
  risk_tier
  policy_stability
  deterministic_baseline_available
  supports_abstain
  metadata
```

Notes:

- `policy_stability` should replace C++ naming that implies only labels drift.
- this entity should not include backend-family decisions
- this entity should not include benchmark evidence

### 2. `DecisionSurface`

This is the semantic description of the output space.

It is separate from `DecisionContract` because two gates can share similar
contract properties but differ in shape:

- flat enum choice
- factorized decision
- hierarchical route
- bounded slot fill

Proposed fields:

```text
DecisionSurface
  gate_id
  primary_mode            // enum | structured | hybrid
  output_labels
  slot_schema
  candidate_gate_shapes   // binary | multiclass | factorized | hierarchical
  candidate_factorizations
  invalid_combinations
  deterministic_prepass_options
  downstream_executor_kind
```

Notes:

- `slot_schema` is required from version 0 even if many early gates are enum-only
- `invalid_combinations` should make structured outputs safe by contract
- `downstream_executor_kind` should record what deterministic system consumes the result

### 3. `DecisionRequestModel`

This is the semantic input packet model for runtime use.

It should support hybrid packets directly instead of forcing a project to flatten
everything into one string.

Proposed fields:

```text
DecisionRequestModel
  packet_id
  gate_id
  text_fields
  symbolic_fields
  numeric_fields
  context_fields
  state_fields
  provenance
```

Expected usage:

- `text_fields`
  - natural language summaries
  - extracted comments
  - short requirement packets
- `symbolic_fields`
  - AST node kinds
  - annotations
  - enums
  - dependency graphs
- `numeric_fields`
  - counts
  - scores
  - ratios
  - latency or complexity features

### 4. `DecisionResultModel`

This is the canonical semantic output model for runtime decisions.

Proposed fields:

```text
DecisionResultModel
  decision_id
  gate_id
  selected_label
  filled_slots
  confidence
  abstain
  retry_strategy
  escalation_target
  trace_packet
```

Notes:

- enum gates mainly use `selected_label`
- structured gates mainly use `filled_slots`
- hybrid outputs may use both

### 5. `SuitabilityAssessment`

This is the semantic result of asking whether a problem belongs in RSA at all.

Proposed fields:

```text
SuitabilityAssessment
  gate_id
  recommended
  target_layer            // deterministic | rsa | slm | llm | human
  rationale
  bounded_output_confidence
  deterministic_baseline_available
  policy_stability_risk
  suggested_gate_shape
```

This entity is intentionally separate from diagnosis and backend selection.

### 6. `DiagnosisModel`

This is the semantic result of evaluating a weak or risky gate.

Proposed fields:

```text
DiagnosisModel
  gate_id
  primary_failure_class
  secondary_failure_classes
  supporting_signals
  recommended_probes
  recommended_interventions
  recommended_policy
  confidence
```

This should remain close to the current C++ `GateDiagnosis`.

### 7. `ProbeDefinition`

This is the semantic declaration of a diagnostic probe.

The semantic layer should define what a probe means before runtime C++ implements
how it executes.

Proposed fields:

```text
ProbeDefinition
  probe_id
  applies_to_gate_shapes
  required_contract_fields
  required_evidence_fields
  produced_signals
  failure_hints
  intervention_hints
```

Concrete examples:

- `confidence_threshold_probe`
- `class_confusion_probe`
- `factorization_probe`
- `deterministic_rule_probe`
- `context_width_probe`

### 8. `BackendSuitability`

This is the semantic result of backend comparison after a gate has already been
judged to belong in RSA.

Proposed fields:

```text
BackendSuitability
  gate_id
  candidate_backends
  recommended_backend
  rationale
  confidence
  benchmark_refs
```

Expected backend families:

- `linear_model`
- `gradient_boosting`
- `random_forest`
- `bayesian`
- `transformer`
- `hybrid`
- `rl_policy`

`rl_policy` should exist in the model now even if it remains an unimplemented
runtime placeholder for the first version.

### 9. `GatePackage`

This is the semantic deployment artifact description.

It exists so runtime loading is a projection of semantic packaging decisions,
not a one-off C++ concern.

Proposed fields:

```text
GatePackage
  gate_id
  contract_ref
  surface_ref
  gate_shape
  backend_family
  artifact_ref
  thresholds
  abstain_policy
  escalation_policy
  trace_schema
  version
```

### Semantic Relationships

The first model should follow these relationships:

```text
DecisionContract
  owns high-level decision identity

DecisionSurface
  refines output shape for a contract

DecisionRequestModel
  is evaluated against DecisionContract + DecisionSurface

DecisionResultModel
  must satisfy DecisionSurface constraints

SuitabilityAssessment
  determines whether the decision belongs in RSA at all

DiagnosisModel
  explains weak performance or risky deployment

ProbeDefinition
  declares reusable diagnosis mechanisms

BackendSuitability
  recommends which implementation family should fill an RSA contract

GatePackage
  records the chosen runtime projection of a validated contract
```

### Projection Map To The Current C++ Scaffold

The current scaffold should be treated as the first runtime projection target.

#### Project directly from semantics into current C++ types

```text
DecisionContract        -> GateDefinition core fields
DecisionSurface         -> GateDefinition.labels + candidate_factorizations + future slot fields
DecisionResultModel     -> future runtime result structs
SuitabilityAssessment   -> GateSuitability
DiagnosisModel          -> GateDiagnosis
ProbeDefinition         -> Probe metadata + ProbeRequest/ProbeResult contracts
```

#### Runtime-only C++ types that should remain projected

```text
Probe
ProbeRegistry
DiagnosisEngine
SuitabilityAssessor
ConfidenceThresholdProbe
```

These are implementations, not semantic source objects.

### Immediate Type Migration Rule

Until semantic authoring is fully in place, the current C++ fields should be
interpreted according to this split:

- semantic-now
  - ontology enums
  - gate contract fields
  - diagnosis enums
  - suitability result fields
- runtime-now
  - probe interfaces
  - engine wiring
  - confidence-threshold implementation
  - demo executable

When a new field is proposed, the default question should be:

- does this describe semantic meaning?
  - if yes, add it to the semantic model first
- does this describe runtime execution detail?
  - if yes, keep it in projected C++

### Immediate Next Conversion Step

The next concrete implementation step after this document should be:

1. represent `DecisionContract`, `DecisionSurface`, and `ProbeDefinition` as the
   first Whetstone-authored RSA semantic forms
2. map each field in `types.h` into one of those forms
3. mark missing fields needed for structured slot outputs and backend selection
4. treat `types.h` as a temporary projection snapshot until semantic-first
   generation replaces hand editing

## Bootstrap Artifact

The first machine-readable semantic bootstrap artifact now lives at:

- `semantic/rsa_semantic_bootstrap_v0.json`

It is intentionally limited to the first three semantic entities needed to begin
realignment:

- `DecisionContract`
- `DecisionSurface`
- `ProbeDefinition`

That file should be treated as the current semantic bootstrap source ahead of any
further handwritten expansion of C++ contract types.