whetstone_RSA/docs/whetstone_ast_first_integration.md

Whetstone-AST-First Integration

Purpose

This document defines how whetstone_RSA should relate to whetstone_DSL.

The key rule is:

whetstone_RSA should be Whetstone-AST-first, not merely AST-first.

That means the semantic source of truth for RSA contracts should live in the Whetstone AST and annotation system, not in ad hoc C++ structs or generic parser ASTs.

Why Generic AST Is Not Enough

A generic AST is mostly syntax structure.

It can tell you things like:

• function
• class
• parameter
• if-statement
• call expression

That is useful, but it is not enough for RSA.

RSA needs first-class semantic contracts for things like:

• input structure
• input modality
• output topology
• output constraint
• policy stability
• suitability assessment
• gate shape
• confidence behavior
• abstain policy
• escalation policy
• diagnosis metadata

Those are not reliable consequences of syntax alone. They are authored semantic data.

Why Whetstone AST Is The Right Source

The Whetstone AST already exists to carry language-neutral semantic intent beyond what ordinary source code or parser ASTs can express cleanly.

That makes it the right authoring substrate for RSA because:

• annotations are first-class
• concepts are language-neutral
• contracts can be preserved across projections
• comments do not need to carry architectural meaning
• generated C++ can remain a projection target instead of becoming the semantic home

In short:

• generic AST gives structure
• Whetstone AST gives structure plus semantic contract capacity

RSA needs the second one.

Architectural Rule

The semantic source of truth for RSA should live in whetstone_DSL.

The C++ runtime in whetstone_RSA should be treated as a projection target for:

• runtime data types
• execution logic
• probe registry interfaces
• policy and diagnosis runtime behavior

This keeps the architecture aligned with the rest of the Whetstone program:

• AST is source of truth
• target languages are projections

Proposed Split Of Responsibilities

whetstone_DSL

Owns:

• RSA semantic ontology
• gate definitions as semantic contracts
• suitability metadata
• diagnosis metadata
• probe declarations
• projection rules into target runtimes

whetstone_RSA

Owns:

• C++ runtime implementation
• probe execution engine
• diagnosis engine
• policy runtime
• compiled/runtime-facing representations of projected contracts

What Should Exist In The Whetstone AST

RSA needs explicit semantic objects, not just ordinary code nodes.

Examples of first-class RSA concepts that should eventually exist in Whetstone form:

• GateDefinition
• GateEvidence
• GateDiagnosis
• GateSuitability
• PolicyRecommendation
• ProbeDefinition
• DecisionContract

These should be represented as semantic entities with structured fields, not as freeform comments embedded in generated C++.

Why This Is Better Than Authoring In C++

If RSA contracts are authored directly in C++, the semantics become fragile:

• comments carry architectural meaning
• relationships are implicit
• code review sees implementation before contract
• projection to other targets becomes harder

If RSA contracts are authored in the Whetstone AST:

• the semantics are explicit
• annotations stay attached to the real concept
• multiple targets can be generated later
• runtime code becomes easier to keep honest

This is especially important because RSA is intended to remain model-agnostic.

The contract is primary. The current transformer-based runtime is secondary.

Model-Agnostic Benefit

Because RSA is contractual rather than model-defined, the AST layer should describe:

• what the gate promises
• what the gate consumes
• how confidence and abstention behave
• how diagnosis works

without hard-coding one backend.

That preserves future replacement paths such as:

• transformer specialist
• classical ML classifier
• RL policy
• retrieval-backed scorer
• deterministic-plus-learned hybrid

The Whetstone AST should capture the contract. The C++ runtime should implement one or more backends against it.

Near-Term Practical Rule

Right now, some C++ scaffolding already exists in whetstone_RSA.

That is fine as a bootstrap.

But it should be treated as:

• provisional runtime code
• useful for exercising the contract shape
• not the long-term semantic source of truth

As soon as the Whetstone-side semantic representation exists, the C++ layer should be aligned to it rather than continuing as an independently evolving hand-authored model.

How To Use WhetstoneDSL Here

The right use of whetstone_DSL is not "parse some C++ and hope the meaning falls out."

The right use is:

1. define RSA concepts explicitly in Whetstone semantic form
2. attach first-class metadata and annotations there
3. project those contracts into C++ runtime types and adapters

If existing Whetstone MCP tools can help bootstrap some of the current C++ structs into AST-native representations, that is useful.

But the goal is not mechanical conversion for its own sake. The goal is semantic alignment.

If direct manual authoring in Whetstone form lands closer to the intended contract, that is better than a weak automatic conversion.

Practical Guidance For The Current Stage

At this stage of the project:

• keep the current C++ runtime scaffold because it is useful for validating runtime shape
• do not expand C++ into the permanent semantic home of the system
• prioritize defining the RSA domain model in Whetstone semantic terms
• use MCP-assisted conversion only when it lands close to the intended semantic contract

This is still early enough that correctness of semantic shape matters more than automation of the conversion path.

Implication

whetstone_RSA is not just "a C++ library with some docs."

It should become:

• a Whetstone-authored semantic subsystem
• with a C++ runtime projection

That is the architecture most consistent with both Whetstone and the long-term goals of RSA.

Bootstrap Mapping From The Current C++ Scaffold

The current C++ code is useful bootstrap material, but it should now be treated as input to the semantic model rather than the long-term definition of that model.

This section inventories the current scaffold and classifies each piece as either:

• semantic contract that should move into Whetstone-authored form
• runtime behavior that should remain in projected C++
• bootstrap/demo code that should not define long-term architecture

Current C++ Semantic Surface

From include/whetstone_rsa/types.h, the current scaffold already expresses the core semantic vocabulary that should be re-authored in Whetstone form:

• ontology enums
  • InputStructure
  • InputModality
  • OutputTopology
  • OutputConstraint
  • LabelStability
  • TargetLayer
• diagnosis enums
  • FailureClass
  • Intervention
  • DeployMode
  • RiskTier
• contract entities
  • GateDefinition
  • GateEvidence
  • PolicyContext
  • ProbeRequest
  • ProbeResult
  • GateDiagnosis
  • GateSuitability
  • PolicyRecommendation

These should not remain primarily hand-authored C++ concepts.

They should become semantic entities in the Whetstone source of truth.

Recommended Whetstone Semantic Entities

The current scaffold suggests the following first semantic model.

1. DecisionContract

This should become the semantic home for the ontology-level fields that currently sit inside GateDefinition.

Proposed fields:

• gate_id
• task_family
• input_structure
• input_modality
• output_topology
• output_constraint
• risk_tier
• policy_stability
• deterministic_baseline_available
• supports_abstain
• metadata

This is the contract boundary, independent of backend family.

2. DecisionSurface

The current GateDefinition.labels plus candidate_factorizations really describe the shape of the output space, not just the gate identity.

Proposed fields:

• output_labels
• slot_schema
• candidate_factorizations
• candidate_gate_shapes
• invalid_combinations
• deterministic_prepass_options

This gives the AST layer a place to describe flat multiclass, hierarchical, or factorized designs without forcing that structure to remain implicit in C++.

3. DecisionRequestModel

The current C++ scaffold does not yet have a first-class request packet type separate from evaluation/demo inputs.

That should be corrected in the AST-first model.

Proposed fields:

• packet_id
• text_fields
• symbolic_fields
• numeric_fields
• context_fields
• state_fields
• provenance

This entity should support both enum-style gates and bounded structured outputs.

4. DecisionResultModel

The current scaffold implicitly assumes label selection and policy behavior, but the AST-first model should make result structure explicit.

Proposed fields:

• decision_id
• selected_label
• filled_slots
• confidence
• abstain
• retry_strategy
• escalation_target
• trace_packet

This should later project into both C++ runtime structs and packaging artifacts.

5. DiagnosisModel

The current GateDiagnosis is already close to a semantic object and should be preserved almost directly, but moved into Whetstone-authored form.

Proposed fields:

• primary_failure_class
• secondary_failure_classes
• supporting_signals
• recommended_probes
• recommended_interventions
• recommended_policy
• confidence

6. ProbeDefinition

The current Probe interface and ProbeRequest / ProbeResult pair indicate that probes are part of the semantic model, not just implementation detail.

The AST layer should explicitly represent:

• probe_id
• applies_to_gate_shapes
• required_evidence_fields
• produced_signals
• failure_hints
• intervention_hints

Concrete runtime probes such as confidence_threshold_probe should then be C++ implementations of these declared probe contracts.

7. SuitabilityAssessment

The current GateSuitability type should become a semantic output model for the pre-RSA decision that determines whether the problem belongs in the RSA layer at all.

Proposed fields:

• recommended
• target_layer
• rationale
• bounded_output_confidence
• deterministic_baseline_available
• label_stability_risk
• suggested_gate_shape

8. BackendSuitability

This does not yet exist in code, but the roadmap now requires it.

Proposed fields:

• gate_id
• candidate_backends
• recommended_backend
• rationale
• confidence
• benchmark_refs

This must be semantic-first, not added ad hoc in C++ later.

9. GatePackage

The runtime needs a durable packaged form for serving decisions after offline selection and calibration.

Proposed fields:

• gate_id
• contract_ref
• gate_shape
• backend_family
• artifact_ref
• thresholds
• abstain_policy
• escalation_policy
• trace_schema
• version

What Should Remain Runtime-Only In C++

The following concepts should remain projected runtime behavior rather than become the semantic source of truth:

• Probe virtual interface
• ProbeRegistry
• DiagnosisEngine
• SuitabilityAssessor
• backend adapter implementations
• calibration code
• packaging loaders
• benchmark runners

These are execution mechanisms, not the semantic model itself.

What In The Current Scaffold Is Only Bootstrap

The following parts are useful but should not steer the long-term semantic design:

• examples/diagnose_demo.cpp
• the current hand-authored stringification helpers
• any current field arrangement that exists only because it was convenient for the first C++ scaffold

Immediate Migration Map

The practical next move should be:

1. Represent the current C++ contract entities above as Whetstone semantic objects.
2. Mark which current fields are semantic contract versus runtime-only detail.
3. Treat types.h as a temporary projected contract snapshot, not the authoring home of the model.
4. Add projection rules so future C++ runtime types are derived from the semantic source rather than independently evolved.

Most Important Boundary To Preserve

The current scaffold already mixes two different layers that must stay separate:

• semantic contract
• runtime execution strategy

If that separation is preserved now, whetstone_RSA can become genuinely AST-first. If it is not preserved, the project will keep drifting into a C++-first runtime with AST plans attached afterward.

First RSA Semantic Model

This section defines the first concrete semantic model that should become the source of truth for whetstone_RSA.

It is intentionally small and close to the current scaffold so migration can start immediately without pretending the architecture is already finished.

The design rule is:

• semantic entities describe contract and meaning
• projected C++ types describe runtime execution

Model Goals

The first semantic model must support all of the following from the beginning:

• bounded enum decisions
• bounded structured slot-filling decisions
• suitability assessment before a gate is adopted
• diagnosis after a gate underperforms
• gate-shape comparison
• backend-selection metadata
• runtime packaging metadata

Semantic Entity Set

The first semantic model should consist of these entities:

• DecisionContract
• DecisionSurface
• DecisionRequestModel
• DecisionResultModel
• SuitabilityAssessment
• DiagnosisModel
• ProbeDefinition
• BackendSuitability
• GatePackage

1. DecisionContract

This is the semantic identity of an RSA gate.

It answers:

• what decision surface exists
• what kind of inputs it consumes
• what kind of bounded outputs it promises
• what policy constraints govern deployment

Proposed fields:

DecisionContract
  gate_id
  task_family
  input_structure
  input_modality
  output_topology
  output_constraint
  risk_tier
  policy_stability
  deterministic_baseline_available
  supports_abstain
  metadata

Notes:

• policy_stability should replace C++ naming that implies only labels drift.
• this entity should not include backend-family decisions
• this entity should not include benchmark evidence

2. DecisionSurface

This is the semantic description of the output space.

It is separate from DecisionContract because two gates can share similar contract properties but differ in shape:

• flat enum choice
• factorized decision
• hierarchical route
• bounded slot fill

Proposed fields:

DecisionSurface
  gate_id
  primary_mode            // enum | structured | hybrid
  output_labels
  slot_schema
  candidate_gate_shapes   // binary | multiclass | factorized | hierarchical
  candidate_factorizations
  invalid_combinations
  deterministic_prepass_options
  downstream_executor_kind

Notes:

• slot_schema is required from version 0 even if many early gates are enum-only
• invalid_combinations should make structured outputs safe by contract
• downstream_executor_kind should record what deterministic system consumes the result

3. DecisionRequestModel

This is the semantic input packet model for runtime use.

It should support hybrid packets directly instead of forcing a project to flatten everything into one string.

Proposed fields:

DecisionRequestModel
  packet_id
  gate_id
  text_fields
  symbolic_fields
  numeric_fields
  context_fields
  state_fields
  provenance

Expected usage:

• text_fields
  • natural language summaries
  • extracted comments
  • short requirement packets
• symbolic_fields
  • AST node kinds
  • annotations
  • enums
  • dependency graphs
• numeric_fields
  • counts
  • scores
  • ratios
  • latency or complexity features

4. DecisionResultModel

This is the canonical semantic output model for runtime decisions.

Proposed fields:

DecisionResultModel
  decision_id
  gate_id
  selected_label
  filled_slots
  confidence
  abstain
  retry_strategy
  escalation_target
  trace_packet

Notes:

• enum gates mainly use selected_label
• structured gates mainly use filled_slots
• hybrid outputs may use both

5. SuitabilityAssessment

This is the semantic result of asking whether a problem belongs in RSA at all.

Proposed fields:

SuitabilityAssessment
  gate_id
  recommended
  target_layer            // deterministic | rsa | slm | llm | human
  rationale
  bounded_output_confidence
  deterministic_baseline_available
  policy_stability_risk
  suggested_gate_shape

This entity is intentionally separate from diagnosis and backend selection.

6. DiagnosisModel

This is the semantic result of evaluating a weak or risky gate.

Proposed fields:

DiagnosisModel
  gate_id
  primary_failure_class
  secondary_failure_classes
  supporting_signals
  recommended_probes
  recommended_interventions
  recommended_policy
  confidence

This should remain close to the current C++ GateDiagnosis.

7. ProbeDefinition

This is the semantic declaration of a diagnostic probe.

The semantic layer should define what a probe means before runtime C++ implements how it executes.

Proposed fields:

ProbeDefinition
  probe_id
  applies_to_gate_shapes
  required_contract_fields
  required_evidence_fields
  produced_signals
  failure_hints
  intervention_hints

Concrete examples:

• confidence_threshold_probe
• class_confusion_probe
• factorization_probe
• deterministic_rule_probe
• context_width_probe

8. BackendSuitability

This is the semantic result of backend comparison after a gate has already been judged to belong in RSA.

Proposed fields:

BackendSuitability
  gate_id
  candidate_backends
  recommended_backend
  rationale
  confidence
  benchmark_refs

Expected backend families:

• linear_model
• gradient_boosting
• random_forest
• bayesian
• transformer
• hybrid
• rl_policy

rl_policy should exist in the model now even if it remains an unimplemented runtime placeholder for the first version.

9. GatePackage

This is the semantic deployment artifact description.

It exists so runtime loading is a projection of semantic packaging decisions, not a one-off C++ concern.

Proposed fields:

GatePackage
  gate_id
  contract_ref
  surface_ref
  gate_shape
  backend_family
  artifact_ref
  thresholds
  abstain_policy
  escalation_policy
  trace_schema
  version

Semantic Relationships

The first model should follow these relationships:

DecisionContract
  owns high-level decision identity

DecisionSurface
  refines output shape for a contract

DecisionRequestModel
  is evaluated against DecisionContract + DecisionSurface

DecisionResultModel
  must satisfy DecisionSurface constraints

SuitabilityAssessment
  determines whether the decision belongs in RSA at all

DiagnosisModel
  explains weak performance or risky deployment

ProbeDefinition
  declares reusable diagnosis mechanisms

BackendSuitability
  recommends which implementation family should fill an RSA contract

GatePackage
  records the chosen runtime projection of a validated contract

Projection Map To The Current C++ Scaffold

The current scaffold should be treated as the first runtime projection target.

Project directly from semantics into current C++ types

DecisionContract        -> GateDefinition core fields
DecisionSurface         -> GateDefinition.labels + candidate_factorizations + future slot fields
DecisionResultModel     -> future runtime result structs
SuitabilityAssessment   -> GateSuitability
DiagnosisModel          -> GateDiagnosis
ProbeDefinition         -> Probe metadata + ProbeRequest/ProbeResult contracts

Runtime-only C++ types that should remain projected

Probe
ProbeRegistry
DiagnosisEngine
SuitabilityAssessor
ConfidenceThresholdProbe

These are implementations, not semantic source objects.

Immediate Type Migration Rule

Until semantic authoring is fully in place, the current C++ fields should be interpreted according to this split:

• semantic-now
  • ontology enums
  • gate contract fields
  • diagnosis enums
  • suitability result fields
• runtime-now
  • probe interfaces
  • engine wiring
  • confidence-threshold implementation
  • demo executable

When a new field is proposed, the default question should be:

• does this describe semantic meaning?
  • if yes, add it to the semantic model first
• does this describe runtime execution detail?
  • if yes, keep it in projected C++

Immediate Next Conversion Step

The next concrete implementation step after this document should be:

1. represent DecisionContract, DecisionSurface, and ProbeDefinition as the first Whetstone-authored RSA semantic forms
2. map each field in types.h into one of those forms
3. mark missing fields needed for structured slot outputs and backend selection
4. treat types.h as a temporary projection snapshot until semantic-first generation replaces hand editing

Bootstrap Artifact

The first machine-readable semantic bootstrap artifact now lives at:

• semantic/rsa_semantic_bootstrap_v0.json

It is intentionally limited to the first three semantic entities needed to begin realignment:

• DecisionContract
• DecisionSurface
• ProbeDefinition

That file should be treated as the current semantic bootstrap source ahead of any further handwritten expansion of C++ contract types.