whetstone_RSA/docs/whetstone_ast_first_integration.md

Whetstone-AST-First Integration

Purpose

This document defines how whetstone_RSA should relate to whetstone_DSL.

The key rule is:

whetstone_RSA should be Whetstone-AST-first, not merely AST-first.

That means the semantic source of truth for RSA contracts should live in the Whetstone AST and annotation system, not in ad hoc C++ structs or generic parser ASTs.

Why Generic AST Is Not Enough

A generic AST is mostly syntax structure.

It can tell you things like:

• function
• class
• parameter
• if-statement
• call expression

That is useful, but it is not enough for RSA.

RSA needs first-class semantic contracts for things like:

• input structure
• input modality
• output topology
• output constraint
• policy stability
• suitability assessment
• gate shape
• confidence behavior
• abstain policy
• escalation policy
• diagnosis metadata

Those are not reliable consequences of syntax alone. They are authored semantic data.

Why Whetstone AST Is The Right Source

The Whetstone AST already exists to carry language-neutral semantic intent beyond what ordinary source code or parser ASTs can express cleanly.

That makes it the right authoring substrate for RSA because:

• annotations are first-class
• concepts are language-neutral
• contracts can be preserved across projections
• comments do not need to carry architectural meaning
• generated C++ can remain a projection target instead of becoming the semantic home

In short:

• generic AST gives structure
• Whetstone AST gives structure plus semantic contract capacity

RSA needs the second one.

Architectural Rule

The semantic source of truth for RSA should live in whetstone_DSL.

The C++ runtime in whetstone_RSA should be treated as a projection target for:

• runtime data types
• execution logic
• probe registry interfaces
• policy and diagnosis runtime behavior

This keeps the architecture aligned with the rest of the Whetstone program:

• AST is source of truth
• target languages are projections

Proposed Split Of Responsibilities

whetstone_DSL

Owns:

• RSA semantic ontology
• gate definitions as semantic contracts
• suitability metadata
• diagnosis metadata
• probe declarations
• projection rules into target runtimes

whetstone_RSA

Owns:

• C++ runtime implementation
• probe execution engine
• diagnosis engine
• policy runtime
• compiled/runtime-facing representations of projected contracts

What Should Exist In The Whetstone AST

RSA needs explicit semantic objects, not just ordinary code nodes.

Examples of first-class RSA concepts that should eventually exist in Whetstone form:

• GateDefinition
• GateEvidence
• GateDiagnosis
• GateSuitability
• PolicyRecommendation
• ProbeDefinition
• DecisionContract

These should be represented as semantic entities with structured fields, not as freeform comments embedded in generated C++.

Why This Is Better Than Authoring In C++

If RSA contracts are authored directly in C++, the semantics become fragile:

• comments carry architectural meaning
• relationships are implicit
• code review sees implementation before contract
• projection to other targets becomes harder

If RSA contracts are authored in the Whetstone AST:

• the semantics are explicit
• annotations stay attached to the real concept
• multiple targets can be generated later
• runtime code becomes easier to keep honest

This is especially important because RSA is intended to remain model-agnostic.

The contract is primary. The current transformer-based runtime is secondary.

Model-Agnostic Benefit

Because RSA is contractual rather than model-defined, the AST layer should describe:

• what the gate promises
• what the gate consumes
• how confidence and abstention behave
• how diagnosis works

without hard-coding one backend.

That preserves future replacement paths such as:

• transformer specialist
• classical ML classifier
• RL policy
• retrieval-backed scorer
• deterministic-plus-learned hybrid

The Whetstone AST should capture the contract. The C++ runtime should implement one or more backends against it.

Near-Term Practical Rule

Right now, some C++ scaffolding already exists in whetstone_RSA.

That is fine as a bootstrap.

But it should be treated as:

• provisional runtime code
• useful for exercising the contract shape
• not the long-term semantic source of truth

As soon as the Whetstone-side semantic representation exists, the C++ layer should be aligned to it rather than continuing as an independently evolving hand-authored model.

How To Use WhetstoneDSL Here

The right use of whetstone_DSL is not "parse some C++ and hope the meaning falls out."

The right use is:

1. define RSA concepts explicitly in Whetstone semantic form
2. attach first-class metadata and annotations there
3. project those contracts into C++ runtime types and adapters

If existing Whetstone MCP tools can help bootstrap some of the current C++ structs into AST-native representations, that is useful.

But the goal is not mechanical conversion for its own sake. The goal is semantic alignment.

If direct manual authoring in Whetstone form lands closer to the intended contract, that is better than a weak automatic conversion.

Practical Guidance For The Current Stage

At this stage of the project:

• keep the current C++ runtime scaffold because it is useful for validating runtime shape
• do not expand C++ into the permanent semantic home of the system
• prioritize defining the RSA domain model in Whetstone semantic terms
• use MCP-assisted conversion only when it lands close to the intended semantic contract

This is still early enough that correctness of semantic shape matters more than automation of the conversion path.

Implication

whetstone_RSA is not just "a C++ library with some docs."

It should become:

• a Whetstone-authored semantic subsystem
• with a C++ runtime projection

That is the architecture most consistent with both Whetstone and the long-term goals of RSA.