# Whetstone-AST-First Integration ## Purpose This document defines how `whetstone_RSA` should relate to `whetstone_DSL`. The key rule is: `whetstone_RSA` should be Whetstone-AST-first, not merely AST-first. That means the semantic source of truth for RSA contracts should live in the Whetstone AST and annotation system, not in ad hoc C++ structs or generic parser ASTs. ## Why Generic AST Is Not Enough A generic AST is mostly syntax structure. It can tell you things like: - function - class - parameter - if-statement - call expression That is useful, but it is not enough for RSA. RSA needs first-class semantic contracts for things like: - input structure - input modality - output topology - output constraint - policy stability - suitability assessment - gate shape - confidence behavior - abstain policy - escalation policy - diagnosis metadata Those are not reliable consequences of syntax alone. They are authored semantic data. ## Why Whetstone AST Is The Right Source The Whetstone AST already exists to carry language-neutral semantic intent beyond what ordinary source code or parser ASTs can express cleanly. That makes it the right authoring substrate for RSA because: - annotations are first-class - concepts are language-neutral - contracts can be preserved across projections - comments do not need to carry architectural meaning - generated C++ can remain a projection target instead of becoming the semantic home In short: - generic AST gives structure - Whetstone AST gives structure plus semantic contract capacity RSA needs the second one. ## Architectural Rule The semantic source of truth for RSA should live in `whetstone_DSL`. The C++ runtime in `whetstone_RSA` should be treated as a projection target for: - runtime data types - execution logic - probe registry interfaces - policy and diagnosis runtime behavior This keeps the architecture aligned with the rest of the Whetstone program: - AST is source of truth - target languages are projections ## Proposed Split Of Responsibilities ### `whetstone_DSL` Owns: - RSA semantic ontology - gate definitions as semantic contracts - suitability metadata - diagnosis metadata - probe declarations - projection rules into target runtimes ### `whetstone_RSA` Owns: - C++ runtime implementation - probe execution engine - diagnosis engine - policy runtime - compiled/runtime-facing representations of projected contracts ## What Should Exist In The Whetstone AST RSA needs explicit semantic objects, not just ordinary code nodes. Examples of first-class RSA concepts that should eventually exist in Whetstone form: - `GateDefinition` - `GateEvidence` - `GateDiagnosis` - `GateSuitability` - `PolicyRecommendation` - `ProbeDefinition` - `DecisionContract` These should be represented as semantic entities with structured fields, not as freeform comments embedded in generated C++. ## Why This Is Better Than Authoring In C++ If RSA contracts are authored directly in C++, the semantics become fragile: - comments carry architectural meaning - relationships are implicit - code review sees implementation before contract - projection to other targets becomes harder If RSA contracts are authored in the Whetstone AST: - the semantics are explicit - annotations stay attached to the real concept - multiple targets can be generated later - runtime code becomes easier to keep honest This is especially important because RSA is intended to remain model-agnostic. The contract is primary. The current transformer-based runtime is secondary. ## Model-Agnostic Benefit Because RSA is contractual rather than model-defined, the AST layer should describe: - what the gate promises - what the gate consumes - how confidence and abstention behave - how diagnosis works without hard-coding one backend. That preserves future replacement paths such as: - transformer specialist - classical ML classifier - RL policy - retrieval-backed scorer - deterministic-plus-learned hybrid The Whetstone AST should capture the contract. The C++ runtime should implement one or more backends against it. ## Near-Term Practical Rule Right now, some C++ scaffolding already exists in `whetstone_RSA`. That is fine as a bootstrap. But it should be treated as: - provisional runtime code - useful for exercising the contract shape - not the long-term semantic source of truth As soon as the Whetstone-side semantic representation exists, the C++ layer should be aligned to it rather than continuing as an independently evolving hand-authored model. ## How To Use WhetstoneDSL Here The right use of `whetstone_DSL` is not "parse some C++ and hope the meaning falls out." The right use is: 1. define RSA concepts explicitly in Whetstone semantic form 2. attach first-class metadata and annotations there 3. project those contracts into C++ runtime types and adapters If existing Whetstone MCP tools can help bootstrap some of the current C++ structs into AST-native representations, that is useful. But the goal is not mechanical conversion for its own sake. The goal is semantic alignment. If direct manual authoring in Whetstone form lands closer to the intended contract, that is better than a weak automatic conversion. ## Practical Guidance For The Current Stage At this stage of the project: - keep the current C++ runtime scaffold because it is useful for validating runtime shape - do not expand C++ into the permanent semantic home of the system - prioritize defining the RSA domain model in Whetstone semantic terms - use MCP-assisted conversion only when it lands close to the intended semantic contract This is still early enough that correctness of semantic shape matters more than automation of the conversion path. ## Implication `whetstone_RSA` is not just "a C++ library with some docs." It should become: - a Whetstone-authored semantic subsystem - with a C++ runtime projection That is the architecture most consistent with both Whetstone and the long-term goals of RSA. ## Bootstrap Mapping From The Current C++ Scaffold The current C++ code is useful bootstrap material, but it should now be treated as input to the semantic model rather than the long-term definition of that model. This section inventories the current scaffold and classifies each piece as either: - semantic contract that should move into Whetstone-authored form - runtime behavior that should remain in projected C++ - bootstrap/demo code that should not define long-term architecture ### Current C++ Semantic Surface From `include/whetstone_rsa/types.h`, the current scaffold already expresses the core semantic vocabulary that should be re-authored in Whetstone form: - ontology enums - `InputStructure` - `InputModality` - `OutputTopology` - `OutputConstraint` - `LabelStability` - `TargetLayer` - diagnosis enums - `FailureClass` - `Intervention` - `DeployMode` - `RiskTier` - contract entities - `GateDefinition` - `GateEvidence` - `PolicyContext` - `ProbeRequest` - `ProbeResult` - `GateDiagnosis` - `GateSuitability` - `PolicyRecommendation` These should not remain primarily hand-authored C++ concepts. They should become semantic entities in the Whetstone source of truth. ### Recommended Whetstone Semantic Entities The current scaffold suggests the following first semantic model. #### 1. `DecisionContract` This should become the semantic home for the ontology-level fields that currently sit inside `GateDefinition`. Proposed fields: - `gate_id` - `task_family` - `input_structure` - `input_modality` - `output_topology` - `output_constraint` - `risk_tier` - `policy_stability` - `deterministic_baseline_available` - `supports_abstain` - `metadata` This is the contract boundary, independent of backend family. #### 2. `DecisionSurface` The current `GateDefinition.labels` plus `candidate_factorizations` really describe the shape of the output space, not just the gate identity. Proposed fields: - `output_labels` - `slot_schema` - `candidate_factorizations` - `candidate_gate_shapes` - `invalid_combinations` - `deterministic_prepass_options` This gives the AST layer a place to describe flat multiclass, hierarchical, or factorized designs without forcing that structure to remain implicit in C++. #### 3. `DecisionRequestModel` The current C++ scaffold does not yet have a first-class request packet type separate from evaluation/demo inputs. That should be corrected in the AST-first model. Proposed fields: - `packet_id` - `text_fields` - `symbolic_fields` - `numeric_fields` - `context_fields` - `state_fields` - `provenance` This entity should support both enum-style gates and bounded structured outputs. #### 4. `DecisionResultModel` The current scaffold implicitly assumes label selection and policy behavior, but the AST-first model should make result structure explicit. Proposed fields: - `decision_id` - `selected_label` - `filled_slots` - `confidence` - `abstain` - `retry_strategy` - `escalation_target` - `trace_packet` This should later project into both C++ runtime structs and packaging artifacts. #### 5. `DiagnosisModel` The current `GateDiagnosis` is already close to a semantic object and should be preserved almost directly, but moved into Whetstone-authored form. Proposed fields: - `primary_failure_class` - `secondary_failure_classes` - `supporting_signals` - `recommended_probes` - `recommended_interventions` - `recommended_policy` - `confidence` #### 6. `ProbeDefinition` The current `Probe` interface and `ProbeRequest` / `ProbeResult` pair indicate that probes are part of the semantic model, not just implementation detail. The AST layer should explicitly represent: - `probe_id` - `applies_to_gate_shapes` - `required_evidence_fields` - `produced_signals` - `failure_hints` - `intervention_hints` Concrete runtime probes such as `confidence_threshold_probe` should then be C++ implementations of these declared probe contracts. #### 7. `SuitabilityAssessment` The current `GateSuitability` type should become a semantic output model for the pre-RSA decision that determines whether the problem belongs in the RSA layer at all. Proposed fields: - `recommended` - `target_layer` - `rationale` - `bounded_output_confidence` - `deterministic_baseline_available` - `label_stability_risk` - `suggested_gate_shape` #### 8. `BackendSuitability` This does not yet exist in code, but the roadmap now requires it. Proposed fields: - `gate_id` - `candidate_backends` - `recommended_backend` - `rationale` - `confidence` - `benchmark_refs` This must be semantic-first, not added ad hoc in C++ later. #### 9. `GatePackage` The runtime needs a durable packaged form for serving decisions after offline selection and calibration. Proposed fields: - `gate_id` - `contract_ref` - `gate_shape` - `backend_family` - `artifact_ref` - `thresholds` - `abstain_policy` - `escalation_policy` - `trace_schema` - `version` ### What Should Remain Runtime-Only In C++ The following concepts should remain projected runtime behavior rather than become the semantic source of truth: - `Probe` virtual interface - `ProbeRegistry` - `DiagnosisEngine` - `SuitabilityAssessor` - backend adapter implementations - calibration code - packaging loaders - benchmark runners These are execution mechanisms, not the semantic model itself. ### What In The Current Scaffold Is Only Bootstrap The following parts are useful but should not steer the long-term semantic design: - `examples/diagnose_demo.cpp` - the current hand-authored stringification helpers - any current field arrangement that exists only because it was convenient for the first C++ scaffold ### Immediate Migration Map The practical next move should be: 1. Represent the current C++ contract entities above as Whetstone semantic objects. 2. Mark which current fields are semantic contract versus runtime-only detail. 3. Treat `types.h` as a temporary projected contract snapshot, not the authoring home of the model. 4. Add projection rules so future C++ runtime types are derived from the semantic source rather than independently evolved. ### Most Important Boundary To Preserve The current scaffold already mixes two different layers that must stay separate: - semantic contract - runtime execution strategy If that separation is preserved now, `whetstone_RSA` can become genuinely AST-first. If it is not preserved, the project will keep drifting into a C++-first runtime with AST plans attached afterward. ## First RSA Semantic Model This section defines the first concrete semantic model that should become the source of truth for `whetstone_RSA`. It is intentionally small and close to the current scaffold so migration can start immediately without pretending the architecture is already finished. The design rule is: - semantic entities describe contract and meaning - projected C++ types describe runtime execution ### Model Goals The first semantic model must support all of the following from the beginning: - bounded enum decisions - bounded structured slot-filling decisions - suitability assessment before a gate is adopted - diagnosis after a gate underperforms - gate-shape comparison - backend-selection metadata - runtime packaging metadata ### Semantic Entity Set The first semantic model should consist of these entities: - `DecisionContract` - `DecisionSurface` - `DecisionRequestModel` - `DecisionResultModel` - `SuitabilityAssessment` - `DiagnosisModel` - `ProbeDefinition` - `BackendSuitability` - `GatePackage` ### 1. `DecisionContract` This is the semantic identity of an RSA gate. It answers: - what decision surface exists - what kind of inputs it consumes - what kind of bounded outputs it promises - what policy constraints govern deployment Proposed fields: ```text DecisionContract gate_id task_family input_structure input_modality output_topology output_constraint risk_tier policy_stability deterministic_baseline_available supports_abstain metadata ``` Notes: - `policy_stability` should replace C++ naming that implies only labels drift. - this entity should not include backend-family decisions - this entity should not include benchmark evidence ### 2. `DecisionSurface` This is the semantic description of the output space. It is separate from `DecisionContract` because two gates can share similar contract properties but differ in shape: - flat enum choice - factorized decision - hierarchical route - bounded slot fill Proposed fields: ```text DecisionSurface gate_id primary_mode // enum | structured | hybrid output_labels slot_schema candidate_gate_shapes // binary | multiclass | factorized | hierarchical candidate_factorizations invalid_combinations deterministic_prepass_options downstream_executor_kind ``` Notes: - `slot_schema` is required from version 0 even if many early gates are enum-only - `invalid_combinations` should make structured outputs safe by contract - `downstream_executor_kind` should record what deterministic system consumes the result ### 3. `DecisionRequestModel` This is the semantic input packet model for runtime use. It should support hybrid packets directly instead of forcing a project to flatten everything into one string. Proposed fields: ```text DecisionRequestModel packet_id gate_id text_fields symbolic_fields numeric_fields context_fields state_fields provenance ``` Expected usage: - `text_fields` - natural language summaries - extracted comments - short requirement packets - `symbolic_fields` - AST node kinds - annotations - enums - dependency graphs - `numeric_fields` - counts - scores - ratios - latency or complexity features ### 4. `DecisionResultModel` This is the canonical semantic output model for runtime decisions. Proposed fields: ```text DecisionResultModel decision_id gate_id selected_label filled_slots confidence abstain retry_strategy escalation_target trace_packet ``` Notes: - enum gates mainly use `selected_label` - structured gates mainly use `filled_slots` - hybrid outputs may use both ### 5. `SuitabilityAssessment` This is the semantic result of asking whether a problem belongs in RSA at all. Proposed fields: ```text SuitabilityAssessment gate_id recommended target_layer // deterministic | rsa | slm | llm | human rationale bounded_output_confidence deterministic_baseline_available policy_stability_risk suggested_gate_shape ``` This entity is intentionally separate from diagnosis and backend selection. ### 6. `DiagnosisModel` This is the semantic result of evaluating a weak or risky gate. Proposed fields: ```text DiagnosisModel gate_id primary_failure_class secondary_failure_classes supporting_signals recommended_probes recommended_interventions recommended_policy confidence ``` This should remain close to the current C++ `GateDiagnosis`. ### 7. `ProbeDefinition` This is the semantic declaration of a diagnostic probe. The semantic layer should define what a probe means before runtime C++ implements how it executes. Proposed fields: ```text ProbeDefinition probe_id applies_to_gate_shapes required_contract_fields required_evidence_fields produced_signals failure_hints intervention_hints ``` Concrete examples: - `confidence_threshold_probe` - `class_confusion_probe` - `factorization_probe` - `deterministic_rule_probe` - `context_width_probe` ### 8. `BackendSuitability` This is the semantic result of backend comparison after a gate has already been judged to belong in RSA. Proposed fields: ```text BackendSuitability gate_id candidate_backends recommended_backend rationale confidence benchmark_refs ``` Expected backend families: - `linear_model` - `gradient_boosting` - `random_forest` - `bayesian` - `transformer` - `hybrid` - `rl_policy` `rl_policy` should exist in the model now even if it remains an unimplemented runtime placeholder for the first version. ### 9. `GatePackage` This is the semantic deployment artifact description. It exists so runtime loading is a projection of semantic packaging decisions, not a one-off C++ concern. Proposed fields: ```text GatePackage gate_id contract_ref surface_ref gate_shape backend_family artifact_ref thresholds abstain_policy escalation_policy trace_schema version ``` ### Semantic Relationships The first model should follow these relationships: ```text DecisionContract owns high-level decision identity DecisionSurface refines output shape for a contract DecisionRequestModel is evaluated against DecisionContract + DecisionSurface DecisionResultModel must satisfy DecisionSurface constraints SuitabilityAssessment determines whether the decision belongs in RSA at all DiagnosisModel explains weak performance or risky deployment ProbeDefinition declares reusable diagnosis mechanisms BackendSuitability recommends which implementation family should fill an RSA contract GatePackage records the chosen runtime projection of a validated contract ``` ### Projection Map To The Current C++ Scaffold The current scaffold should be treated as the first runtime projection target. #### Project directly from semantics into current C++ types ```text DecisionContract -> GateDefinition core fields DecisionSurface -> GateDefinition.labels + candidate_factorizations + future slot fields DecisionResultModel -> future runtime result structs SuitabilityAssessment -> GateSuitability DiagnosisModel -> GateDiagnosis ProbeDefinition -> Probe metadata + ProbeRequest/ProbeResult contracts ``` #### Runtime-only C++ types that should remain projected ```text Probe ProbeRegistry DiagnosisEngine SuitabilityAssessor ConfidenceThresholdProbe ``` These are implementations, not semantic source objects. ### Immediate Type Migration Rule Until semantic authoring is fully in place, the current C++ fields should be interpreted according to this split: - semantic-now - ontology enums - gate contract fields - diagnosis enums - suitability result fields - runtime-now - probe interfaces - engine wiring - confidence-threshold implementation - demo executable When a new field is proposed, the default question should be: - does this describe semantic meaning? - if yes, add it to the semantic model first - does this describe runtime execution detail? - if yes, keep it in projected C++ ### Immediate Next Conversion Step The next concrete implementation step after this document should be: 1. represent `DecisionContract`, `DecisionSurface`, and `ProbeDefinition` as the first Whetstone-authored RSA semantic forms 2. map each field in `types.h` into one of those forms 3. mark missing fields needed for structured slot outputs and backend selection 4. treat `types.h` as a temporary projection snapshot until semantic-first generation replaces hand editing ## Bootstrap Artifact The first machine-readable semantic bootstrap artifact now lives at: - `semantic/rsa_semantic_bootstrap_v0.json` It is intentionally limited to the first three semantic entities needed to begin realignment: - `DecisionContract` - `DecisionSurface` - `ProbeDefinition` That file should be treated as the current semantic bootstrap source ahead of any further handwritten expansion of C++ contract types.