whetstone_RSA/docs/whetstone_ast_first_integration.md

# Whetstone-AST-First Integration

## Purpose

This document defines how `whetstone_RSA` should relate to `whetstone_DSL`.

The key rule is:

`whetstone_RSA` should be Whetstone-AST-first, not merely AST-first.

That means the semantic source of truth for RSA contracts should live in the
Whetstone AST and annotation system, not in ad hoc C++ structs or generic parser
ASTs.

## Why Generic AST Is Not Enough

A generic AST is mostly syntax structure.

It can tell you things like:

- function
- class
- parameter
- if-statement
- call expression

That is useful, but it is not enough for RSA.

RSA needs first-class semantic contracts for things like:

- input structure
- input modality
- output topology
- output constraint
- policy stability
- suitability assessment
- gate shape
- confidence behavior
- abstain policy
- escalation policy
- diagnosis metadata

Those are not reliable consequences of syntax alone.
They are authored semantic data.

## Why Whetstone AST Is The Right Source

The Whetstone AST already exists to carry language-neutral semantic intent beyond
what ordinary source code or parser ASTs can express cleanly.

That makes it the right authoring substrate for RSA because:

- annotations are first-class
- concepts are language-neutral
- contracts can be preserved across projections
- comments do not need to carry architectural meaning
- generated C++ can remain a projection target instead of becoming the semantic home

In short:

- generic AST gives structure
- Whetstone AST gives structure plus semantic contract capacity

RSA needs the second one.

## Architectural Rule

The semantic source of truth for RSA should live in `whetstone_DSL`.

The C++ runtime in `whetstone_RSA` should be treated as a projection target for:

- runtime data types
- execution logic
- probe registry interfaces
- policy and diagnosis runtime behavior

This keeps the architecture aligned with the rest of the Whetstone program:

- AST is source of truth
- target languages are projections

## Proposed Split Of Responsibilities

### `whetstone_DSL`

Owns:

- RSA semantic ontology
- gate definitions as semantic contracts
- suitability metadata
- diagnosis metadata
- probe declarations
- projection rules into target runtimes

### `whetstone_RSA`

Owns:

- C++ runtime implementation
- probe execution engine
- diagnosis engine
- policy runtime
- compiled/runtime-facing representations of projected contracts

## What Should Exist In The Whetstone AST

RSA needs explicit semantic objects, not just ordinary code nodes.

Examples of first-class RSA concepts that should eventually exist in Whetstone form:

- `GateDefinition`
- `GateEvidence`
- `GateDiagnosis`
- `GateSuitability`
- `PolicyRecommendation`
- `ProbeDefinition`
- `DecisionContract`

These should be represented as semantic entities with structured fields, not as
freeform comments embedded in generated C++.

## Why This Is Better Than Authoring In C++

If RSA contracts are authored directly in C++, the semantics become fragile:

- comments carry architectural meaning
- relationships are implicit
- code review sees implementation before contract
- projection to other targets becomes harder

If RSA contracts are authored in the Whetstone AST:

- the semantics are explicit
- annotations stay attached to the real concept
- multiple targets can be generated later
- runtime code becomes easier to keep honest

This is especially important because RSA is intended to remain model-agnostic.

The contract is primary.
The current transformer-based runtime is secondary.

## Model-Agnostic Benefit

Because RSA is contractual rather than model-defined, the AST layer should describe:

- what the gate promises
- what the gate consumes
- how confidence and abstention behave
- how diagnosis works

without hard-coding one backend.

That preserves future replacement paths such as:

- transformer specialist
- classical ML classifier
- RL policy
- retrieval-backed scorer
- deterministic-plus-learned hybrid

The Whetstone AST should capture the contract.
The C++ runtime should implement one or more backends against it.

## Near-Term Practical Rule

Right now, some C++ scaffolding already exists in `whetstone_RSA`.

That is fine as a bootstrap.

But it should be treated as:

- provisional runtime code
- useful for exercising the contract shape
- not the long-term semantic source of truth

As soon as the Whetstone-side semantic representation exists, the C++ layer should
be aligned to it rather than continuing as an independently evolving hand-authored
model.

## How To Use WhetstoneDSL Here

The right use of `whetstone_DSL` is not "parse some C++ and hope the meaning falls
out."

The right use is:

1. define RSA concepts explicitly in Whetstone semantic form
2. attach first-class metadata and annotations there
3. project those contracts into C++ runtime types and adapters

If existing Whetstone MCP tools can help bootstrap some of the current C++ structs
into AST-native representations, that is useful.

But the goal is not mechanical conversion for its own sake.
The goal is semantic alignment.

If direct manual authoring in Whetstone form lands closer to the intended contract,
that is better than a weak automatic conversion.

## Practical Guidance For The Current Stage

At this stage of the project:

- keep the current C++ runtime scaffold because it is useful for validating runtime shape
- do not expand C++ into the permanent semantic home of the system
- prioritize defining the RSA domain model in Whetstone semantic terms
- use MCP-assisted conversion only when it lands close to the intended semantic contract

This is still early enough that correctness of semantic shape matters more than
automation of the conversion path.

## Implication

`whetstone_RSA` is not just "a C++ library with some docs."

It should become:

- a Whetstone-authored semantic subsystem
- with a C++ runtime projection

That is the architecture most consistent with both Whetstone and the long-term
goals of RSA.