WIP: save progress 2026-04-12
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill Holcombe
299
ROADMAP.md
299
ROADMAP.md
@@ -6,12 +6,115 @@
|
||||
- Define the bounded gate schema contract.
|
||||
- Document the WhetstoneDSL case study and current specialist fleet.
|
||||
|
||||
### Phase 0A: AST-First Realignment
|
||||
|
||||
Before `whetstone_RSA` grows much further as a hand-authored C++ library, realign
|
||||
the project so its semantic source of truth is Whetstone-authored and C++ becomes
|
||||
a projection target.
|
||||
|
||||
This phase exists to prevent the current bootstrap scaffold from hardening into the
|
||||
wrong long-term architecture.
|
||||
|
||||
#### Intent
|
||||
|
||||
- make RSA semantically AST-first
|
||||
- keep C++ as runtime projection and execution layer
|
||||
- preserve compatibility with the current scaffold while changing the source of truth
|
||||
|
||||
#### Architectural rule
|
||||
|
||||
The long-term source of truth for RSA should live in Whetstone semantic forms, not
|
||||
in hand-authored C++ structs.
|
||||
|
||||
The C++ code in this repository should be treated as:
|
||||
|
||||
- runtime implementation
|
||||
- projected contract types
|
||||
- backend execution surface
|
||||
- diagnosis and policy engine
|
||||
|
||||
not the canonical definition of RSA semantics.
|
||||
|
||||
#### 0A.1 Semantic entities to define first
|
||||
|
||||
Define Whetstone-authored semantic objects for at least:
|
||||
|
||||
- `GateDefinition`
|
||||
- `DecisionRequest`
|
||||
- `DecisionResult`
|
||||
- `StructuredDecisionResult`
|
||||
- `GateSuitability`
|
||||
- `GateDiagnosis`
|
||||
- `ProbeDefinition`
|
||||
- `BackendSuitability`
|
||||
- `GatePackage`
|
||||
|
||||
These should carry the contract metadata already identified in the docs:
|
||||
|
||||
- input structure and modality
|
||||
- output topology and constraint
|
||||
- policy stability
|
||||
- gate shape
|
||||
- confidence, abstain, and escalation policy
|
||||
- trace fields
|
||||
- backend family and packaging metadata
|
||||
|
||||
#### 0A.2 Projection boundary
|
||||
|
||||
Define a deliberate projection boundary from Whetstone semantic forms into:
|
||||
|
||||
- C++ runtime contract structs
|
||||
- schema artifacts
|
||||
- benchmark configuration artifacts
|
||||
- packaged runtime manifests
|
||||
|
||||
The important architectural constraint is:
|
||||
|
||||
- semantics are authored once
|
||||
- runtime targets are projected from that source
|
||||
|
||||
#### 0A.3 Migration rule for the current scaffold
|
||||
|
||||
Do not continue expanding hand-authored C++ contract definitions unless they are
|
||||
clearly temporary runtime placeholders.
|
||||
|
||||
When a contract type changes, the preferred direction should become:
|
||||
|
||||
- update semantic source
|
||||
- regenerate or realign projected C++ representation
|
||||
|
||||
rather than:
|
||||
|
||||
- evolve C++ first
|
||||
- treat docs and later AST work as catch-up
|
||||
|
||||
#### 0A.4 Immediate AST-first deliverables
|
||||
|
||||
- define the first RSA semantic model in Whetstone terms
|
||||
- map current C++ runtime types to those semantic entities
|
||||
- identify what current fields are semantic contract versus runtime-only detail
|
||||
- define the projection rules into C++
|
||||
- update roadmap phases so future implementation assumes semantic-first authoring
|
||||
|
||||
#### 0A.5 Scope constraint
|
||||
|
||||
This does not require immediate deep modification of `whetstone_DSL` runtime code.
|
||||
|
||||
It does require that future RSA design work stop assuming C++ is the authoring
|
||||
home of the system.
|
||||
|
||||
The immediate goal is architectural realignment, not premature full conversion.
|
||||
|
||||
## Phase 1: Gate Inventory
|
||||
|
||||
- Enumerate candidate gates from WhetstoneDSL.
|
||||
- Record output vocabulary, input shape, and downstream deterministic executor.
|
||||
- Split gates into deterministic, specialist, and escalation-first candidates.
|
||||
|
||||
Primary case-study inventory artifact:
|
||||
|
||||
- `docs/case_studies/whetstone_dsl_gate_inventory.md`
|
||||
|
||||
## Phase 2: Entropy Baseline
|
||||
|
||||
- Define a first-pass gate entropy score.
|
||||
@@ -31,6 +134,194 @@
|
||||
- Implement abstain and escalation policies.
|
||||
- Support pluggable inference backends.
|
||||
|
||||
### Phase 4A: Post-Suitability RSA Runtime Plan
|
||||
|
||||
This phase begins only after a decision surface has already been judged to belong
|
||||
in the `rsa` layer rather than `deterministic`, `slm`, `llm`, or `human`.
|
||||
|
||||
The focus here is not broad suitability triage. The focus is:
|
||||
|
||||
- how an RSA gate is represented
|
||||
- how its shape is chosen
|
||||
- how its backend is selected
|
||||
- how it is packaged for efficient runtime use
|
||||
|
||||
#### Design constraints
|
||||
|
||||
- Keep the library focused on `whetstone_RSA`, not direct `whetstone_DSL` changes.
|
||||
- Assume RSA semantic contracts are authored AST-first and projected into runtime
|
||||
targets rather than defined primarily in handwritten C++.
|
||||
- Support both bounded single-choice classification and bounded structured outputs
|
||||
with slots from the beginning.
|
||||
- Include `rl_policy` and bandit-style policies as contract placeholders, but do
|
||||
not make them a primary implementation target until classical ML and transformer
|
||||
backends are working and benchmarked.
|
||||
- Treat hybrid inputs as first-class. A gate may consume text, symbolic state, and
|
||||
numeric features together.
|
||||
- Use case studies only to surface reusable abstractions. Do not let one case study
|
||||
become the semantic definition of the library.
|
||||
|
||||
#### 4A.1 Contract and packet model
|
||||
|
||||
- Define a canonical `GateDefinition` that supports:
|
||||
- bounded enum decisions
|
||||
- bounded structured slot-filling decisions
|
||||
- constraints
|
||||
- confidence policy
|
||||
- abstain policy
|
||||
- escalation policy
|
||||
- trace metadata
|
||||
- Define a canonical `DecisionRequest` packet that supports:
|
||||
- textual fields
|
||||
- symbolic fields
|
||||
- numeric features
|
||||
- state and provenance metadata
|
||||
- Define `DecisionResult` and `StructuredDecisionResult` so programs can consume
|
||||
either plain label outputs or bounded slot outputs through argument-oriented
|
||||
runtime calls rather than project-specific ad hoc interfaces.
|
||||
|
||||
#### 4A.2 Gate-shape selection
|
||||
|
||||
The library should not assume that all bounded problems should be represented as
|
||||
one flat multiclass classifier.
|
||||
|
||||
For each RSA candidate gate, support shape analysis across:
|
||||
|
||||
- `binary`
|
||||
- `multiclass`
|
||||
- `factorized`
|
||||
- `hierarchical`
|
||||
|
||||
Initial implementation goals:
|
||||
|
||||
- define metadata that describes candidate gate shapes
|
||||
- define diagnostics that detect when a flat label space hides compositional
|
||||
structure
|
||||
- support shape-specific constraints so invalid state combinations are prevented
|
||||
by contract rather than patched later
|
||||
|
||||
#### 4A.3 Backend-selection layer
|
||||
|
||||
Backend selection remains separate from RSA identity.
|
||||
|
||||
The library should support a backend contract for at least:
|
||||
|
||||
- `linear_model`
|
||||
- `gradient_boosting`
|
||||
- `random_forest`
|
||||
- `bayesian`
|
||||
- `transformer`
|
||||
- `hybrid`
|
||||
- `rl_policy` placeholder
|
||||
|
||||
Initial implementation goals:
|
||||
|
||||
- define a common backend interface for train, infer, calibrate, and trace
|
||||
- allow external benchmark or AutoML-style workflows to produce backend
|
||||
recommendations without forcing one internal training stack
|
||||
- keep backend artifacts swappable as long as the gate contract is preserved
|
||||
|
||||
#### 4A.4 Benchmark harness
|
||||
|
||||
Build a reusable harness that compares both gate shape and backend family for the
|
||||
same bounded decision contract.
|
||||
|
||||
For each gate, the harness should compare:
|
||||
|
||||
- flat multiclass against factorized or hierarchical alternatives
|
||||
- classical ML backends against tiny transformer backends
|
||||
- deterministic baselines where any part of the gate can be rule-complete
|
||||
|
||||
Required outputs:
|
||||
|
||||
- accepted-decision accuracy
|
||||
- calibration quality
|
||||
- abstain quality
|
||||
- invalid-output rate
|
||||
- downstream deterministic validation success
|
||||
- latency
|
||||
- memory footprint
|
||||
- training and retraining cost
|
||||
|
||||
#### 4A.5 Runtime packaging
|
||||
|
||||
AutoML or scripted comparison should be offline or occasional, not on the hot path.
|
||||
|
||||
The runtime should load a selected gate package that includes:
|
||||
|
||||
- gate contract
|
||||
- chosen gate shape
|
||||
- chosen backend family
|
||||
- trained artifact reference
|
||||
- confidence thresholds
|
||||
- abstain and escalation policy
|
||||
- trace schema and version metadata
|
||||
|
||||
The steady-state runtime model is:
|
||||
|
||||
- benchmark and select offline
|
||||
- package once
|
||||
- load the chosen gate into memory
|
||||
- serve repeated decisions efficiently
|
||||
- re-run selection only when schema, data, policy, or hardware conditions change
|
||||
|
||||
#### 4A.6 Invocation surface
|
||||
|
||||
The library should expose explicit runtime calls rather than force consumers to
|
||||
rebuild invocation conventions for each gate.
|
||||
|
||||
Initial target surface:
|
||||
|
||||
- `decide_class(request, gate_definition)`
|
||||
- `decide_structured(request, gate_definition)`
|
||||
- `diagnose_gate(gate_definition, evidence, benchmark_results)`
|
||||
- `recommend_backend(gate_definition, benchmark_results)`
|
||||
- `recommend_gate_shape(gate_definition, benchmark_results)`
|
||||
|
||||
The exact function names may change, but the core idea should remain: consumers
|
||||
pass structured arguments into a reusable RSA runtime surface.
|
||||
|
||||
#### 4A.7 Diagnosis loop
|
||||
|
||||
When a gate underperforms, the first response should not be automatic model-size
|
||||
escalation.
|
||||
|
||||
The runtime and benchmark surfaces should instead help answer:
|
||||
|
||||
- should this gate stay flat or be factorized?
|
||||
- is part of the task actually deterministic?
|
||||
- is the label space unstable?
|
||||
- is the backend family mismatched to the input packet?
|
||||
- should the gate abstain earlier?
|
||||
|
||||
#### 4A.8 First case-study role
|
||||
|
||||
Use the memory-strategy decision only as the first end-to-end exercise for the
|
||||
abstract mechanism.
|
||||
|
||||
The case study should help validate:
|
||||
|
||||
- mixed symbolic/text/numeric packet design
|
||||
- shape comparison between flat and decomposed gates
|
||||
- backend comparison between classical ML and tiny transformers
|
||||
- packaging of both enum-style and slot-filling outputs
|
||||
|
||||
It should not be allowed to overfit the library to one memory-management ontology.
|
||||
|
||||
#### 4A.9 Immediate implementation order
|
||||
|
||||
- define the AST-first semantic source for RSA contracts
|
||||
- define projection rules from semantic entities into C++ runtime types
|
||||
- extend schema types so both enum and structured-slot gates are first-class
|
||||
- define canonical request/result packet types
|
||||
- define gate-shape metadata and recommendation types
|
||||
- define backend-family contracts, including placeholder RL/bandit support
|
||||
- build the benchmark report format
|
||||
- implement classical ML adapters first
|
||||
- implement transformer adapter second
|
||||
- add packaging and runtime loading
|
||||
- add diagnosis and recommendation APIs on top of benchmark evidence
|
||||
|
||||
## Phase 5: Whetstone Integration
|
||||
|
||||
- Expose WhetstoneDSL gates through the library.
|
||||
@@ -46,6 +337,12 @@
|
||||
## Exit Criteria For Version 0
|
||||
|
||||
- At least one reusable gate schema format is stable.
|
||||
- Both enum-style and bounded structured-slot gates are supported by the contract.
|
||||
- At least three WhetstoneDSL gates are benchmarked across size tiers.
|
||||
- At least one gate has been compared across multiple gate shapes, not just
|
||||
multiple model sizes.
|
||||
- At least one gate has been compared across classical ML and transformer
|
||||
backends under the same contract.
|
||||
- A baseline entropy-to-tier recommendation table exists.
|
||||
- One real application can call the library instead of hard-coding model selection.
|
||||
- One real application can call the library instead of hard-coding model
|
||||
selection or backend choice.
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
# WhetstoneDSL Execution Contract Shape Skeleton
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the first semantic skeleton for the structured decision
|
||||
surface currently spread across `whetstone_DSL` execution-contract heuristics.
|
||||
|
||||
The intent is not to finalize training data or runtime wiring yet.
|
||||
|
||||
The intent is to make the contract explicit enough that:
|
||||
|
||||
- we know what the decision is
|
||||
- we know which parts are sub-decisions
|
||||
- we know which parts can stay deterministic
|
||||
- we can later extract training data without flattening the structure incorrectly
|
||||
|
||||
## Why This Needs A Skeleton First
|
||||
|
||||
The current `whetstone_DSL` execution contract is assembled from heuristics in
|
||||
`RegisterArchitectIntakeTools.h`.
|
||||
|
||||
That logic currently bundles several distinct choices together:
|
||||
|
||||
- target files
|
||||
- required tools
|
||||
- acceptance commands
|
||||
- verification type
|
||||
- step sequence
|
||||
- cross-project targets
|
||||
- resource locks
|
||||
|
||||
If we treat that whole bundle as one opaque label, the gate becomes unusable for
|
||||
RSA.
|
||||
|
||||
If we flatten it into arbitrary string labels, we lose the structure and make the
|
||||
training problem worse.
|
||||
|
||||
So the correct intermediate step is a semantic skeleton.
|
||||
|
||||
## Entity Name
|
||||
|
||||
Recommended semantic entity:
|
||||
|
||||
- `ExecutionContractShapeDecision`
|
||||
|
||||
This is a skeleton decision, not yet a finalized RSA gate package.
|
||||
|
||||
## Contract Boundary
|
||||
|
||||
`ExecutionContractShapeDecision` answers:
|
||||
|
||||
- what execution-shape packet should be attached to a taskitem before downstream
|
||||
deterministic execution or validation
|
||||
|
||||
It does **not** answer:
|
||||
|
||||
- whether the task is globally good or bad
|
||||
- how the task should be routed to human/LLM/SLM layers
|
||||
- what final generated code should look like
|
||||
|
||||
## Field Set
|
||||
|
||||
### Core identity fields
|
||||
|
||||
- `decision_id`
|
||||
- `task_id`
|
||||
- `source_case_study`
|
||||
- `contract_version`
|
||||
|
||||
These fields are bookkeeping only.
|
||||
|
||||
### Input context fields
|
||||
|
||||
These are the fields allowed to influence the decision.
|
||||
|
||||
- `task_title`
|
||||
- `task_intent`
|
||||
- `requirement_ids`
|
||||
- `requirement_kinds`
|
||||
- `ambiguity_signals`
|
||||
- `conflict_signals`
|
||||
- `workspace_root`
|
||||
- `project_root`
|
||||
- `cross_project_intent`
|
||||
- `known_target_hints`
|
||||
|
||||
Rule:
|
||||
|
||||
- do not include downstream execution outcomes here
|
||||
- do not include later compile/test results here
|
||||
|
||||
## Structured output fields
|
||||
|
||||
### 1. `target_files`
|
||||
|
||||
Meaning:
|
||||
|
||||
- which files this task most directly acts on
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Notes:
|
||||
|
||||
- bounded structured selection
|
||||
- likely future child decision:
|
||||
`TargetFileSelectionDecision`
|
||||
|
||||
### 2. `required_tools`
|
||||
|
||||
Meaning:
|
||||
|
||||
- which named tools or MCP operations are required by the contract
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Examples:
|
||||
|
||||
- `whetstone_generate_taskitems`
|
||||
- `whetstone_validate_taskitem`
|
||||
|
||||
Notes:
|
||||
|
||||
- often partly deterministic from explicit references
|
||||
- could remain deterministic longer than other fields
|
||||
|
||||
### 3. `acceptance_commands`
|
||||
|
||||
Meaning:
|
||||
|
||||
- which concrete validation commands should be run for this task
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Examples:
|
||||
|
||||
- `cargo test`
|
||||
- `pytest`
|
||||
- `ctest`
|
||||
|
||||
Notes:
|
||||
|
||||
- bounded structured choice
|
||||
- highly sensitive to target-file inference
|
||||
|
||||
### 4. `verification_type`
|
||||
|
||||
Meaning:
|
||||
|
||||
- the dominant verification mode implied by the task
|
||||
|
||||
Type:
|
||||
|
||||
- enum
|
||||
|
||||
Allowed values:
|
||||
|
||||
- `unit`
|
||||
- `schema`
|
||||
- `docs`
|
||||
- `smoke`
|
||||
|
||||
Notes:
|
||||
|
||||
- this is the clearest immediate child gate
|
||||
- should remain explicit instead of being buried inside the larger contract
|
||||
|
||||
### 5. `step_sequence`
|
||||
|
||||
Meaning:
|
||||
|
||||
- the bounded sequence of execution phases expected for the task
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Examples:
|
||||
|
||||
- `inspect_scope`
|
||||
- `implement_changes`
|
||||
- `apply_changes`
|
||||
- `run_unit_verification`
|
||||
- `run_smoke_verification`
|
||||
- `review_doc_alignment`
|
||||
|
||||
Notes:
|
||||
|
||||
- structured output
|
||||
- good later candidate for a bounded sequence or hierarchical gate
|
||||
|
||||
### 6. `cross_project_targets`
|
||||
|
||||
Meaning:
|
||||
|
||||
- external project artifacts the task is explicitly allowed or expected to touch
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Notes:
|
||||
|
||||
- should be empty by default
|
||||
- this field must not be inferred casually from vague mentions
|
||||
|
||||
### 7. `resource_locks`
|
||||
|
||||
Meaning:
|
||||
|
||||
- coarse-grained lock names needed to avoid conflicting execution
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Notes:
|
||||
|
||||
- likely deterministic in early versions
|
||||
- still part of the execution-contract shape because it affects runtime behavior
|
||||
|
||||
## Child Decision Decomposition
|
||||
|
||||
`ExecutionContractShapeDecision` should not remain monolithic forever.
|
||||
|
||||
The expected decomposition is:
|
||||
|
||||
- `VerificationTypeGate`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `AcceptanceCommandSelectionDecision`
|
||||
- `StepSequenceDecision`
|
||||
- `CrossProjectTargetDecision`
|
||||
- `ResourceLockPolicy`
|
||||
|
||||
This parent skeleton exists so those child decisions have a coherent semantic
|
||||
container.
|
||||
|
||||
## Deterministic Baseline Split
|
||||
|
||||
Not every field in the execution contract should become a specialist.
|
||||
|
||||
### Likely deterministic for longer
|
||||
|
||||
- explicit tool references
|
||||
- explicit target file mentions
|
||||
- resource locks when directly tied to file or subsystem identity
|
||||
|
||||
### Stronger specialist candidates
|
||||
|
||||
- `verification_type`
|
||||
- target-file selection when multiple plausible files exist
|
||||
- acceptance-command selection when language/build context is mixed
|
||||
- step-sequence selection
|
||||
|
||||
## Invalid States
|
||||
|
||||
The skeleton should define invalid or suspicious combinations.
|
||||
|
||||
Examples:
|
||||
|
||||
- `verification_type = docs` with only code target files and no doc targets
|
||||
- `acceptance_commands = ["cargo test"]` with only Python target files
|
||||
- non-empty `cross_project_targets` without cross-project intent
|
||||
- empty `target_files` plus non-empty language-specific acceptance commands
|
||||
|
||||
These invalid states are important because they give the later gate contract a
|
||||
real constraint surface instead of a bag of fields.
|
||||
|
||||
## Training Extraction Guidance
|
||||
|
||||
If historic JSON/taskitem runs are mined later, extraction should happen per
|
||||
child decision, not by treating the full execution contract as one class label.
|
||||
|
||||
Safe pattern:
|
||||
|
||||
1. reconstruct task-local context
|
||||
2. extract `verification_type`
|
||||
3. extract `target_files`
|
||||
4. extract `acceptance_commands`
|
||||
5. extract `step_sequence`
|
||||
6. preserve provenance for each extracted field
|
||||
|
||||
Unsafe pattern:
|
||||
|
||||
- one whole JSON contract becomes one giant class label
|
||||
|
||||
## Minimal Machine-Readable Shape
|
||||
|
||||
The eventual machine-readable skeleton should support at least:
|
||||
|
||||
```text
|
||||
ExecutionContractShapeDecision
|
||||
input_context:
|
||||
task_title
|
||||
task_intent
|
||||
ambiguity_signals
|
||||
conflict_signals
|
||||
workspace_root
|
||||
project_root
|
||||
cross_project_intent
|
||||
known_target_hints
|
||||
output:
|
||||
target_files[]
|
||||
required_tools[]
|
||||
acceptance_commands[]
|
||||
verification_type
|
||||
step_sequence[]
|
||||
cross_project_targets[]
|
||||
resource_locks[]
|
||||
constraints:
|
||||
invalid_combinations[]
|
||||
```
|
||||
|
||||
## Immediate Next Step
|
||||
|
||||
Define equivalent skeletons for:
|
||||
|
||||
- `TargetFileSelectionDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
|
||||
Then update the semantic bootstrap artifact so `ExecutionContractShapeDecision`
|
||||
exists as an explicit skeleton entity rather than only as prose.
|
||||
456
docs/case_studies/whetstone_dsl_gate_extraction_specs_v0.md
Normal file
456
docs/case_studies/whetstone_dsl_gate_extraction_specs_v0.md
Normal file
@@ -0,0 +1,456 @@
|
||||
# WhetstoneDSL Gate Extraction Specs v0
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines first-pass extraction specs for turning surviving
|
||||
`whetstone_DSL` artifacts into RSA-aligned training rows.
|
||||
|
||||
These specs are gate-native, not taskitem-native.
|
||||
Each spec describes:
|
||||
|
||||
- source artifacts
|
||||
- allowed context
|
||||
- forbidden leaked context
|
||||
- label shape
|
||||
- provenance expectation
|
||||
- rejection conditions
|
||||
|
||||
The immediate goal is to support the first high-confidence extraction surfaces
|
||||
identified in the run-artifact audit.
|
||||
|
||||
## Source Priority
|
||||
|
||||
Primary extraction corpus:
|
||||
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs/*/01_intake.json`
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs/*/02_generate_taskitems.json`
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs/*/04_validate_taskitem.json`
|
||||
|
||||
Supporting context reservoirs:
|
||||
|
||||
- `CLionProjects/whetstone_DSL/datasets/example_run_specs/*`
|
||||
- `CLionProjects/whetstone_DSL/datasets/project_benchmarks/*`
|
||||
|
||||
These supporting corpora may supply spec context, but not labels.
|
||||
|
||||
## Global Extraction Rules
|
||||
|
||||
### Row shape
|
||||
|
||||
Every extracted row should contain:
|
||||
|
||||
- `gate_id`
|
||||
- `decision_context`
|
||||
- `decision_label`
|
||||
- `decision_provenance`
|
||||
- `source_artifacts`
|
||||
- `rejection_flags`
|
||||
|
||||
### Provenance values
|
||||
|
||||
Use one of:
|
||||
|
||||
- `direct_explicit`
|
||||
- `direct_bundled`
|
||||
- `reconstructed`
|
||||
- `untrusted`
|
||||
|
||||
### Global rejection rules
|
||||
|
||||
Reject a candidate row when:
|
||||
|
||||
- the candidate label depends on downstream execution results
|
||||
- the candidate label is only inferable from post-hoc failure or success
|
||||
- the candidate label shape does not match the RSA gate contract
|
||||
- required pre-decision context is missing
|
||||
- the artifact mixes unrelated tasks without isolatable per-task context
|
||||
|
||||
## Spec: `prereq_op_selector`
|
||||
|
||||
### Status
|
||||
|
||||
First extraction target.
|
||||
This is the strongest current gate candidate in surviving taskitem runs.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- optional: `04_validate_taskitem.json`
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- `01_intake.json`
|
||||
- normalized requirements
|
||||
- parsed spec sections
|
||||
- ambiguity signals
|
||||
- conflict signals
|
||||
- `02_generate_taskitems.json`
|
||||
- `title`
|
||||
- `milestoneId`
|
||||
- `dependencyTaskIds`
|
||||
- `confidence`
|
||||
- `ambiguityCount`
|
||||
- `reasons`
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].prerequisiteOps`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- dependency ids
|
||||
- local ambiguity and conflict signals
|
||||
- intake-level normalized requirements
|
||||
- milestone grouping if already known at task generation time
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- acceptance commands
|
||||
- target files
|
||||
- required tools
|
||||
- downstream execution results
|
||||
- validation pass/fail outcome
|
||||
- generated code
|
||||
|
||||
### Label shape
|
||||
|
||||
Do not train as a flat multiclass label.
|
||||
|
||||
Use a factorized structured label:
|
||||
|
||||
- `needs_validate_intake`: boolean
|
||||
- `needs_architect_review`: boolean
|
||||
- `needs_resolve_dependencies`: boolean
|
||||
|
||||
Allowed source values:
|
||||
|
||||
- `validate-intake`
|
||||
- `architect-review`
|
||||
- `resolve-dependencies`
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
Reason:
|
||||
|
||||
- the list exists explicitly, but was emitted as part of a larger taskitem
|
||||
bundle rather than an isolated gate invocation
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- `prerequisiteOps` contains values outside the accepted slot vocabulary
|
||||
- the task cannot be isolated from sibling tasks in the artifact
|
||||
- confidence or dependency signals are missing and the context becomes too thin
|
||||
- the list appears to be post-processed rather than source-emitted
|
||||
|
||||
## Spec: `target_file_selection`
|
||||
|
||||
### Status
|
||||
|
||||
High-confidence child decision under `execution_contract_shape`.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- optional: run-spec markdown from `datasets/example_run_specs`
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- normalized requirements
|
||||
- parsed spec
|
||||
- task title
|
||||
- milestone id
|
||||
- dependency ids
|
||||
- local reasons
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].executionContract.targetFiles`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- relevant normalized requirements
|
||||
- milestone grouping
|
||||
- dependency ids
|
||||
- optionally scenario family metadata from run-spec corpus
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- acceptance commands
|
||||
- required tools
|
||||
- execution specificity score
|
||||
- validation diagnostics
|
||||
- downstream worker assignment
|
||||
- final implementation output
|
||||
|
||||
### Label shape
|
||||
|
||||
Structured bounded list of repository paths.
|
||||
|
||||
Recommended first-pass representation:
|
||||
|
||||
- `target_files`: ordered list of paths
|
||||
|
||||
Potential later factorization:
|
||||
|
||||
- `primary_surface`
|
||||
- `secondary_surfaces`
|
||||
- `script_surface_present`
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- target files include unrelated repository surfaces not justified by task text
|
||||
- path list is empty
|
||||
- path list appears copied across all sibling tasks without local task variation
|
||||
- path list depends on downstream implementation or validation feedback
|
||||
|
||||
## Spec: `acceptance_command_selection`
|
||||
|
||||
### Status
|
||||
|
||||
Promising child decision under `execution_contract_shape`, but more leakage risk
|
||||
than target-file selection.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- `04_validate_taskitem.json` for audit only, not label derivation
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- normalized requirements
|
||||
- task title
|
||||
- target files
|
||||
- milestone id
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].executionContract.acceptanceCommands`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- target files
|
||||
- normalized requirements that imply verification surface
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- validation outcome
|
||||
- execution success or failure
|
||||
- downstream generated code
|
||||
- specificity score used as a target proxy
|
||||
|
||||
### Label shape
|
||||
|
||||
Structured bounded list of command identifiers.
|
||||
|
||||
Recommended first-pass representation:
|
||||
|
||||
- `acceptance_commands`: ordered list of commands
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- commands are obviously copied template defaults unrelated to task surface
|
||||
- commands require information only known after code edits
|
||||
- commands are inconsistent with target files or implied verification surface
|
||||
|
||||
## Spec: `required_tool_selection`
|
||||
|
||||
### Status
|
||||
|
||||
Useful, but higher risk of mixing model choices with framework templates.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- `04_validate_taskitem.json` for audit only
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- normalized requirements
|
||||
- task title
|
||||
- target files
|
||||
- prerequisite ops
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].executionContract.requiredTools`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- prerequisite ops
|
||||
- target files
|
||||
- normalized requirements
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- validation-recommended tools as label source
|
||||
- downstream execution traces
|
||||
- final worker selection
|
||||
|
||||
### Label shape
|
||||
|
||||
Structured bounded list of tool ids.
|
||||
|
||||
Recommended first-pass representation:
|
||||
|
||||
- `required_tools`: ordered list of tool names
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- the tool list is clearly framework boilerplate with no task-local variation
|
||||
- the list includes post-hoc validator recommendations instead of generation-time
|
||||
selections
|
||||
- tool names cannot be mapped to a stable tool registry
|
||||
|
||||
## Spec: `verification_type`
|
||||
|
||||
### Status
|
||||
|
||||
Not yet a direct label source.
|
||||
Treat as reconstruction target only after child execution-contract fields are
|
||||
stable.
|
||||
|
||||
### Candidate reconstruction inputs
|
||||
|
||||
- target files
|
||||
- acceptance commands
|
||||
- task title
|
||||
- normalized requirements
|
||||
|
||||
### Current label shape
|
||||
|
||||
Bounded enum:
|
||||
|
||||
- `unit`
|
||||
- `schema`
|
||||
- `docs`
|
||||
- `smoke`
|
||||
|
||||
### Current provenance
|
||||
|
||||
- `reconstructed`
|
||||
|
||||
### Blocking issue
|
||||
|
||||
No audited artifact yet shows `verification_type` as a clean standalone field.
|
||||
|
||||
## Spec: `worker_type`
|
||||
|
||||
### Status
|
||||
|
||||
Second-pass extraction target.
|
||||
|
||||
### Current evidence
|
||||
|
||||
- explicit source-code surface exists in:
|
||||
- `editor/src/RoutingRules.h`
|
||||
- `editor/src/WorkflowOrchestrator.h`
|
||||
- `editor/src/TaskitemPoolPersistence.h`
|
||||
- `editor/src/headless_rpc/DispatchPart6.h`
|
||||
|
||||
### Current provenance
|
||||
|
||||
- `reconstructed`
|
||||
|
||||
### Blocking issue
|
||||
|
||||
No clean serialized label found yet in the audited taskitem-generation artifacts.
|
||||
|
||||
### Next search target
|
||||
|
||||
Routing and orchestration run families, not generic taskitem-generation runs.
|
||||
|
||||
## Spec: `cpp_raising_profile`
|
||||
|
||||
### Status
|
||||
|
||||
Second-pass extraction target.
|
||||
|
||||
### Current evidence
|
||||
|
||||
- explicit source-code surface exists in:
|
||||
- `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
- `editor/src/cpp_ir/CppErrorModelMapping.h`
|
||||
- `editor/src/cpp_ir/CppBuildArtifactGenerator.h`
|
||||
- `editor/src/cpp_ir/CppAsyncMappingStrategy.h`
|
||||
- `editor/src/cpp_ir/CppTemplateRaisingPolicy.h`
|
||||
|
||||
### Current label shape
|
||||
|
||||
Bounded enum:
|
||||
|
||||
- `safe-first`
|
||||
- `perf-first`
|
||||
- `interop-first`
|
||||
|
||||
### Current provenance
|
||||
|
||||
- `untrusted`
|
||||
|
||||
### Blocking issue
|
||||
|
||||
No clean serialized profile label found yet in the audited generic taskitem
|
||||
corpora.
|
||||
|
||||
### Next search target
|
||||
|
||||
C++-raising-specific run artifacts and MCP execution traces.
|
||||
|
||||
## Immediate Implementation Order
|
||||
|
||||
1. `prereq_op_selector`
|
||||
2. `target_file_selection`
|
||||
3. `acceptance_command_selection`
|
||||
4. `required_tool_selection`
|
||||
5. `verification_type` reconstruction rule
|
||||
6. `worker_type` artifact search
|
||||
7. `cpp_raising_profile` artifact search
|
||||
374
docs/case_studies/whetstone_dsl_gate_inventory.md
Normal file
374
docs/case_studies/whetstone_dsl_gate_inventory.md
Normal file
@@ -0,0 +1,374 @@
|
||||
# WhetstoneDSL Gate Inventory
|
||||
|
||||
## Purpose
|
||||
|
||||
This document grounds `whetstone_RSA` Phase 1 in the actual decision surfaces
|
||||
currently present in `whetstone_DSL`.
|
||||
|
||||
It is intentionally narrower than a general architecture document.
|
||||
|
||||
The goal is to answer:
|
||||
|
||||
- what bounded decisions already exist in the `whetstone_DSL` taskitem and
|
||||
AST-codegen pipeline
|
||||
- which of those are already deterministic
|
||||
- which are immediate RSA case-study candidates
|
||||
- which should become semantic skeletons first before any specialist training
|
||||
|
||||
This inventory is based on the current code surfaces in:
|
||||
|
||||
- `editor/src/TaskitemGeneratorV2.h`
|
||||
- `editor/src/TaskitemConfidenceAmbiguity.h`
|
||||
- `editor/src/RoutingEngine.h`
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- `editor/src/ast/ProjectionGenerator.h`
|
||||
- `editor/src/CrossLanguageProjector.h`
|
||||
- `editor/src/ConstrainedProjectionGate.h`
|
||||
- `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
|
||||
## Architectural Observation
|
||||
|
||||
The strongest current RSA surface in `whetstone_DSL` is not the final
|
||||
language-emission step.
|
||||
|
||||
Once the AST and target language are fixed, most current code generation is
|
||||
already deterministic:
|
||||
|
||||
- target generator dispatch is deterministic
|
||||
- AST concept dispatch is deterministic
|
||||
- environment gating is deterministic
|
||||
- C++ raising profile validation is deterministic
|
||||
|
||||
That means the immediate RSA case-study value is higher in:
|
||||
|
||||
- taskitem generation
|
||||
- execution-contract shaping
|
||||
- routing
|
||||
- pre-codegen bounded profile decisions
|
||||
|
||||
than in raw syntax emission.
|
||||
|
||||
## Current Decision Surfaces
|
||||
|
||||
### A. Taskitem decomposition and planning
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- milestone grouping
|
||||
- workstream grouping
|
||||
- workstream title shape
|
||||
- workstream intent aggregation
|
||||
- uncertainty scoring
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- largely heuristic and deterministic in `ScopeMilestoneDecomposer`
|
||||
- not yet expressed as explicit RSA gate contracts
|
||||
|
||||
Assessment:
|
||||
|
||||
- milestone grouping is currently deterministic and should remain so until a real
|
||||
ambiguity signal shows otherwise
|
||||
- uncertainty scoring is currently formula-like and should remain deterministic
|
||||
- title shaping is mixed: parts are deterministic formatting, but some grouping
|
||||
and summary choices are latent semantic decisions
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- weak immediate training target
|
||||
- good semantic-skeleton target
|
||||
|
||||
Suggested skeleton entities:
|
||||
|
||||
- `WorkstreamGroupingDecision`
|
||||
- `MilestoneGroupingDecision`
|
||||
- `WorkstreamSummaryPolicy`
|
||||
|
||||
### B. Taskitem worker-type assignment
|
||||
|
||||
Observed decision:
|
||||
|
||||
- `implementer | reviewer | architect | qa`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- keyword heuristic in `TaskitemGeneratorV2::inferWorkerType`
|
||||
- file comment explicitly says this is a placeholder to be replaced by a
|
||||
`worker_type` specialist
|
||||
|
||||
Assessment:
|
||||
|
||||
- clear bounded enum
|
||||
- direct downstream deterministic meaning
|
||||
- already known from specialist experiments
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- immediate case-study gate
|
||||
|
||||
Recommended gate class:
|
||||
|
||||
- bounded multiclass gate
|
||||
|
||||
Likely failure risks:
|
||||
|
||||
- label-space mismatch
|
||||
- missing context gate
|
||||
|
||||
### C. Taskitem prerequisite-op assignment
|
||||
|
||||
Observed decision:
|
||||
|
||||
- `validate-intake`
|
||||
- `architect-review`
|
||||
- `resolve-dependencies`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- deterministic default plus uncertainty-based add-on in
|
||||
`TaskitemGeneratorV2::inferPrerequisiteOps`
|
||||
- comments explicitly note this is a placeholder for a future
|
||||
`prereq_op_selector` specialist
|
||||
|
||||
Assessment:
|
||||
|
||||
- this is not purely one flat label
|
||||
- it is closer to a bounded structured output or factorized multi-label gate
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- immediate case-study gate
|
||||
- especially important for factorization probes
|
||||
|
||||
Recommended gate shape:
|
||||
|
||||
- factorized or multi-label, not naive flat multiclass
|
||||
|
||||
Likely failure risks:
|
||||
|
||||
- factorizable gate
|
||||
- label-space mismatch
|
||||
|
||||
### D. Confidence / ambiguity / escalation
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- confidence score `0..100`
|
||||
- ambiguity count
|
||||
- escalate `true|false`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- formulaic in `TaskitemConfidenceAmbiguity`
|
||||
|
||||
Assessment:
|
||||
|
||||
- this is the strongest current example of a deterministic-disguised-as-ML gate
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- important as negative evidence
|
||||
- should stay deterministic unless a later richer policy surface emerges
|
||||
|
||||
Recommended disposition:
|
||||
|
||||
- deterministic layer, not RSA
|
||||
|
||||
### E. Routing decision
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- worker type:
|
||||
`deterministic | template | slm | llm | human`
|
||||
- context width:
|
||||
`local | file | project | cross-project`
|
||||
- review required
|
||||
- agent role:
|
||||
`linter | refactor | generator`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- deterministic rule cascade in `RoutingEngine`
|
||||
- explicit override path plus heuristic fallbacks
|
||||
|
||||
Assessment:
|
||||
|
||||
- routing is already modeled as a bounded decision surface
|
||||
- the current engine is a rule baseline
|
||||
- this is a strong later RSA case study, but not the first one
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- important second-wave case study
|
||||
- especially useful for comparing deterministic baseline versus RSA gate quality
|
||||
|
||||
Recommended disposition:
|
||||
|
||||
- keep current deterministic routing as baseline
|
||||
- use it later as benchmark data for RSA substitution or hybrid routing
|
||||
|
||||
### F. Execution-contract shaping
|
||||
|
||||
Observed decisions in `RegisterArchitectIntakeTools`:
|
||||
|
||||
- target file inference
|
||||
- required tool inference
|
||||
- acceptance-command inference
|
||||
- cross-project target inference
|
||||
- verification type inference:
|
||||
`unit | schema | docs | smoke`
|
||||
- step-id inference:
|
||||
`inspect_scope`, `implement_changes`, `apply_changes`,
|
||||
`run_unit_verification`, `run_smoke_verification`, etc.
|
||||
- resource-lock inference
|
||||
|
||||
Assessment by subtype:
|
||||
|
||||
- target-file inference:
|
||||
mixed, partly lexical and partly semantic
|
||||
- required-tool inference:
|
||||
bounded and meaningful
|
||||
- verification type:
|
||||
strong existing specialist case
|
||||
- step-id inference:
|
||||
bounded structured sequence; good future structured-output RSA case
|
||||
- resource locks:
|
||||
likely deterministic unless concurrency semantics become richer
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- `verification_type` is immediate
|
||||
- execution-contract shaping is a strong structured-output frontier
|
||||
|
||||
Recommended split:
|
||||
|
||||
- immediate RSA gates:
|
||||
- `verification_type`
|
||||
- semantic skeleton first:
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `AcceptanceCommandSelectionDecision`
|
||||
- `StepSequenceDecision`
|
||||
|
||||
### G. AST projection / code generation
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- target language selection
|
||||
- AST node dispatch to language-specific visitor
|
||||
- annotation-preserving cross-language projection
|
||||
- environment constraint gating
|
||||
- C++ raising profile:
|
||||
`safe-first | perf-first | interop-first`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- almost entirely deterministic once target language and profile are provided
|
||||
|
||||
Assessment:
|
||||
|
||||
- generator dispatch itself is not the best current RSA surface
|
||||
- the richer opportunity is in bounded pre-codegen policy selection, not in
|
||||
token-level syntax generation
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- immediate gate candidate:
|
||||
`cpp_raising_profile`
|
||||
- skeleton-first targets:
|
||||
- `ProjectionProfileDecision`
|
||||
- `ErrorModelPolicyDecision`
|
||||
- `OwnershipMappingProfileDecision`
|
||||
- `AsyncMappingPolicyDecision`
|
||||
|
||||
## Inventory Table
|
||||
|
||||
| Decision surface | Current form | Best current layer |
|
||||
| --- | --- | --- |
|
||||
| `worker_type` | bounded enum | RSA immediate |
|
||||
| `prereq_op_selector` | bounded structured / factorized | RSA immediate |
|
||||
| `verification_type` | bounded enum | RSA immediate |
|
||||
| `confidence_tier` / `escalate` | formula-like | deterministic |
|
||||
| milestone grouping | heuristic grouping | semantic skeleton first |
|
||||
| workstream summary/title | mixed heuristic | semantic skeleton first |
|
||||
| routing layer selection | bounded enum with strong rules | deterministic baseline, later RSA |
|
||||
| target-file inference | mixed bounded selection | semantic skeleton first |
|
||||
| acceptance-command inference | bounded selection | semantic skeleton first |
|
||||
| step-sequence inference | bounded structured sequence | semantic skeleton first |
|
||||
| resource locks | bounded selection | deterministic |
|
||||
| generator dispatch by language | deterministic | deterministic |
|
||||
| C++ raising profile | bounded enum | RSA immediate or near-immediate |
|
||||
|
||||
## Recommended Implementation Order
|
||||
|
||||
### 1. Immediate RSA gate cases
|
||||
|
||||
Use the already-observed bounded gates first:
|
||||
|
||||
- `verification_type`
|
||||
- `worker_type`
|
||||
- `prereq_op_selector`
|
||||
- `cpp_raising_profile`
|
||||
|
||||
These are the best first fit because they already have:
|
||||
|
||||
- small output vocabularies
|
||||
- deterministic downstream meaning
|
||||
- clear failure consequences
|
||||
- real code paths in `whetstone_DSL`
|
||||
|
||||
### 2. Skeleton before specialist
|
||||
|
||||
Before training more specialists, define explicit semantic skeletons for:
|
||||
|
||||
- workstream grouping
|
||||
- execution contract shaping
|
||||
- target-file selection
|
||||
- acceptance-command selection
|
||||
- step-sequence selection
|
||||
|
||||
These are not yet clean enough as training targets because the semantic contract
|
||||
is still implicit in heuristics.
|
||||
|
||||
### 3. Second-wave RSA cases
|
||||
|
||||
After the skeleton pass:
|
||||
|
||||
- routing layer selection
|
||||
- routing context-width selection
|
||||
- richer codegen profile selection
|
||||
- backend-policy decisions in IR raising
|
||||
|
||||
## Implication For Sprint 002
|
||||
|
||||
Sprint 002 should not jump straight from the current C++ RSA scaffold into model
|
||||
training.
|
||||
|
||||
It should first ensure the semantic model can express at least these categories:
|
||||
|
||||
- bounded enum gate
|
||||
- factorized / structured-output gate
|
||||
- deterministic baseline attached to a candidate gate
|
||||
- downstream deterministic executor contract
|
||||
- failure diagnosis for factorization and schema drift
|
||||
|
||||
That semantic coverage is required to represent the real `whetstone_DSL`
|
||||
surfaces above.
|
||||
|
||||
## Concrete Next Step
|
||||
|
||||
Define Whetstone-authored semantic entities for the first four case-study gates:
|
||||
|
||||
- `VerificationTypeGate`
|
||||
- `WorkerTypeGate`
|
||||
- `PrereqOpGate`
|
||||
- `CppRaisingProfileGate`
|
||||
|
||||
and define skeleton semantic entities for:
|
||||
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
|
||||
That would let `whetstone_RSA` move from generic architecture language into a
|
||||
usable case-study-grounded semantic bootstrap.
|
||||
348
docs/case_studies/whetstone_dsl_run_artifact_audit.md
Normal file
348
docs/case_studies/whetstone_dsl_run_artifact_audit.md
Normal file
@@ -0,0 +1,348 @@
|
||||
# WhetstoneDSL Run Artifact Audit
|
||||
|
||||
## Purpose
|
||||
|
||||
This note audits surviving `whetstone_DSL` run artifacts for RSA gate extraction.
|
||||
The goal is not to treat every surviving corpus as direct training data. The goal
|
||||
is to identify which artifacts preserve explicit decision outputs, which preserve
|
||||
only surrounding context, and which are better treated as evaluation traces.
|
||||
|
||||
## Audit Targets
|
||||
|
||||
Primary paths inspected:
|
||||
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs`
|
||||
- `CLionProjects/whetstone_DSL/datasets/example_run_specs`
|
||||
- `CLionProjects/whetstone_DSL/datasets/project_benchmarks`
|
||||
|
||||
Representative files inspected:
|
||||
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/01_intake.json`
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems.json`
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems_raw.ndjson.json`
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/04_validate_taskitem.json`
|
||||
- `logs/taskitem_runs/project_benchmark_matrix_20260225_192610/cp_001_python/ab_stdout.log`
|
||||
- `logs/taskitem_runs/ast_once_vs_direct_20260225_193037/cp_001/base_payload.json`
|
||||
- `datasets/example_run_specs/index.jsonl`
|
||||
- `datasets/example_run_specs/runspec_py_cli_todo_wrong_tool_selection_01.md`
|
||||
- `datasets/project_benchmarks/common_projects_100.jsonl`
|
||||
|
||||
## Source Classes
|
||||
|
||||
### 1. Structured taskitem runs
|
||||
|
||||
Best current extraction source:
|
||||
|
||||
- `logs/taskitem_runs/*/01_intake.json`
|
||||
- `logs/taskitem_runs/*/02_generate_taskitems.json`
|
||||
- `logs/taskitem_runs/*/04_validate_taskitem.json`
|
||||
|
||||
What these preserve:
|
||||
|
||||
- pre-decision intake context
|
||||
- generated taskitems
|
||||
- explicit task confidence and escalation signals
|
||||
- explicit prerequisite operations
|
||||
- explicit execution-contract fields
|
||||
- validation and specificity feedback
|
||||
|
||||
Observed explicit fields in `02_generate_taskitems.json`:
|
||||
|
||||
- `confidence`
|
||||
- `escalate`
|
||||
- `prerequisiteOps`
|
||||
- `queueReady`
|
||||
- `reasons`
|
||||
- `resourceLocks`
|
||||
- `executionContract.acceptanceCommands`
|
||||
- `executionContract.requiredTools`
|
||||
- `executionContract.targetFiles`
|
||||
- `executionContract.stepIds`
|
||||
- `executionContract.riskLevel`
|
||||
- `executionContract.executionSpecificityScore`
|
||||
|
||||
Observed intake fields in `01_intake.json`:
|
||||
|
||||
- normalized requirements
|
||||
- parsed spec structure
|
||||
- conflict and ambiguity signals
|
||||
|
||||
Observed validation fields in `04_validate_taskitem.json`:
|
||||
|
||||
- gap and specificity diagnostics
|
||||
- recommended tools
|
||||
- execution-quality feedback
|
||||
|
||||
Assessment:
|
||||
|
||||
- highest value source for `verification_type`
|
||||
- highest value source for `prereq_op_selector`
|
||||
- highest value source for `execution_contract_shape`
|
||||
- highest value source for child skeleton decisions such as:
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `required_tool_selection`
|
||||
|
||||
Constraint:
|
||||
|
||||
- these files preserve outputs, but not necessarily independence of each field
|
||||
- some fields may be jointly emitted in one response rather than chosen as
|
||||
separate model calls
|
||||
- extraction should therefore attach provenance and a decomposition method
|
||||
|
||||
### 2. Raw NDJSON taskitem responses
|
||||
|
||||
Representative path:
|
||||
|
||||
- `logs/taskitem_runs/*/02_generate_taskitems_raw.ndjson.json`
|
||||
|
||||
What these preserve:
|
||||
|
||||
- near-raw tool response wrapper
|
||||
- text form of the generated taskitem JSON
|
||||
- tool/runtime metadata
|
||||
|
||||
Assessment:
|
||||
|
||||
- useful provenance source
|
||||
- useful for checking whether `02_generate_taskitems.json` is a direct capture
|
||||
or a normalized downstream rewrite
|
||||
- not a separate supervision source by itself
|
||||
|
||||
Constraint:
|
||||
|
||||
- this still reflects bundled output from a single taskitem-generation call
|
||||
- it does not by itself prove per-field independence
|
||||
|
||||
### 3. Project benchmark matrix runs
|
||||
|
||||
Representative path:
|
||||
|
||||
- `logs/taskitem_runs/project_benchmark_matrix_20260225_192610/*/ab_stdout.log`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- spec
|
||||
- language
|
||||
- strict mode
|
||||
- path A / path B readiness
|
||||
- failure reasons
|
||||
- token accounting
|
||||
|
||||
Assessment:
|
||||
|
||||
- good evaluation corpus
|
||||
- good source for comparing system modes and efficiency
|
||||
- not a direct source for task-shaping gates such as `prereq_op_selector` or
|
||||
`target_file_selection`
|
||||
|
||||
Constraint:
|
||||
|
||||
- these logs summarize end-to-end path outcomes
|
||||
- they do not preserve the intermediate bounded decisions we want to supervise
|
||||
|
||||
### 4. AST-once vs direct comparison runs
|
||||
|
||||
Representative path:
|
||||
|
||||
- `logs/taskitem_runs/ast_once_vs_direct_20260225_193037/*/base_payload.json`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- AST node payload
|
||||
- generated code
|
||||
- compile/test/placeholder gates
|
||||
- quality flags
|
||||
- used symbols
|
||||
|
||||
Assessment:
|
||||
|
||||
- useful evaluation corpus for AST-first vs direct generation comparisons
|
||||
- useful for downstream codegen quality measurement
|
||||
- not a primary direct source for taskitem decision supervision
|
||||
|
||||
Constraint:
|
||||
|
||||
- preserves codegen result and gate outcomes
|
||||
- does not preserve the earlier planning and task-shaping decisions
|
||||
|
||||
### 5. Example run specs
|
||||
|
||||
Representative paths:
|
||||
|
||||
- `datasets/example_run_specs/index.jsonl`
|
||||
- `datasets/example_run_specs/*.md`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- scenario family and variant metadata
|
||||
- objective
|
||||
- target behavior
|
||||
- anti-pattern tags
|
||||
- acceptance criteria
|
||||
- data capture tags
|
||||
|
||||
Assessment:
|
||||
|
||||
- strong context/spec source
|
||||
- useful for reconstructing scenario intent and mistake class
|
||||
- not a direct source of labels for bounded decisions
|
||||
|
||||
### 6. Project benchmark inventories
|
||||
|
||||
Representative path:
|
||||
|
||||
- `datasets/project_benchmarks/*.jsonl`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- project id
|
||||
- category
|
||||
- language
|
||||
- natural-language spec
|
||||
|
||||
Assessment:
|
||||
|
||||
- good context reservoir
|
||||
- good replay seed corpus
|
||||
- not a direct supervision source
|
||||
|
||||
## Extraction Suitability By Decision
|
||||
|
||||
### Direct or near-direct candidates
|
||||
|
||||
- `prereq_op_selector`
|
||||
- candidate source: `02_generate_taskitems.json`
|
||||
- source field: `prerequisiteOps`
|
||||
- note: may still need decomposition into slot-based labels rather than raw
|
||||
list copy
|
||||
|
||||
- `execution_contract_shape`
|
||||
- candidate source: `02_generate_taskitems.json`
|
||||
- source field: `executionContract`
|
||||
- note: should be split into child decisions before training
|
||||
|
||||
- `target_file_selection`
|
||||
- candidate source: `executionContract.targetFiles`
|
||||
- note: must preserve local requirement context and touched-surface context
|
||||
|
||||
- `acceptance_command_selection`
|
||||
- candidate source: `executionContract.acceptanceCommands`
|
||||
- note: likely partly heuristic in some runs; requires provenance tagging
|
||||
|
||||
- `required_tool_selection`
|
||||
- candidate source: `executionContract.requiredTools`
|
||||
- note: likely mixed model-plus-template behavior
|
||||
|
||||
### Reconstructible but not yet trusted as direct labels
|
||||
|
||||
- `verification_type`
|
||||
- may be reconstructible from execution-contract structure, validation shape,
|
||||
and downstream required tools
|
||||
- not yet observed as a single explicit standalone field in audited artifacts
|
||||
|
||||
- `worker_type`
|
||||
- current code surface exists in routing/orchestration layers:
|
||||
- `editor/src/RoutingRules.h`
|
||||
- `editor/src/WorkflowOrchestrator.h`
|
||||
- `editor/src/TaskitemPoolPersistence.h`
|
||||
- `editor/src/headless_rpc/DispatchPart6.h`
|
||||
- not yet observed as a clean explicit field in the audited taskitem-run
|
||||
artifacts used for task generation
|
||||
- may exist elsewhere in routing or orchestration artifacts, or may need
|
||||
reconstruction from execution contract and downstream path
|
||||
|
||||
### Poor direct-fit sources for current Sprint 003 gates
|
||||
|
||||
- `cpp_raising_profile`
|
||||
- current code surface exists in C++-raising layers:
|
||||
- `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
- `editor/src/cpp_ir/CppErrorModelMapping.h`
|
||||
- `editor/src/cpp_ir/CppBuildArtifactGenerator.h`
|
||||
- `editor/src/cpp_ir/CppAsyncMappingStrategy.h`
|
||||
- `editor/src/cpp_ir/CppTemplateRaisingPolicy.h`
|
||||
- not surfaced in the audited generic taskitem corpora reviewed so far
|
||||
- likely needs a different source family tied to C++ raising flows rather than
|
||||
generic taskitem generation
|
||||
|
||||
## Search Outcome Notes
|
||||
|
||||
Additional source search confirms:
|
||||
|
||||
- `workerType` is a real first-class concept in the current DSL routing stack
|
||||
- `safe-first|perf-first|interop-first` is a real first-class concept in the
|
||||
current C++ raising stack
|
||||
- the audited taskitem-generation artifacts do not yet show those values as
|
||||
clean serialized labels in the same way that `prerequisiteOps` and
|
||||
`executionContract.*` are preserved
|
||||
|
||||
Practical implication:
|
||||
|
||||
- `prereq_op_selector` and execution-contract child decisions can move toward
|
||||
extraction-spec work now
|
||||
- `worker_type` and `cpp_raising_profile` need a second artifact search pass
|
||||
over routing and raising-specific run families before we treat them as
|
||||
extractable labels
|
||||
|
||||
## Provenance Classes
|
||||
|
||||
Every extracted training row should carry one provenance tag:
|
||||
|
||||
- `direct_explicit`
|
||||
- field exists directly in the artifact in bounded form
|
||||
|
||||
- `direct_bundled`
|
||||
- field exists directly, but as one member of a larger jointly emitted object
|
||||
|
||||
- `reconstructed`
|
||||
- label inferred from multiple artifact fields or execution traces
|
||||
|
||||
- `untrusted`
|
||||
- label candidate exists, but source independence or semantic stability is too
|
||||
weak to use without further review
|
||||
|
||||
Current default classification:
|
||||
|
||||
- `prerequisiteOps`: `direct_bundled`
|
||||
- `executionContract.targetFiles`: `direct_bundled`
|
||||
- `executionContract.acceptanceCommands`: `direct_bundled`
|
||||
- `executionContract.requiredTools`: `direct_bundled`
|
||||
- `verification_type`: `reconstructed`
|
||||
- `worker_type`: `reconstructed`
|
||||
- `cpp_raising_profile`: `untrusted`
|
||||
|
||||
## Immediate Recommendations
|
||||
|
||||
1. Use `logs/taskitem_runs/*/{01_intake.json,02_generate_taskitems.json,04_validate_taskitem.json}`
|
||||
as the first extraction corpus.
|
||||
|
||||
2. Treat `datasets/example_run_specs` and `datasets/project_benchmarks` as
|
||||
context reservoirs, not label sources.
|
||||
|
||||
3. Keep project benchmark matrix and AST/direct corpora for evaluation and
|
||||
counterfactual replay, not first-pass supervision.
|
||||
|
||||
4. Build extraction specs per gate rather than per full taskitem JSON.
|
||||
|
||||
5. Require each extraction spec to declare:
|
||||
- source files
|
||||
- context slice
|
||||
- label shape
|
||||
- provenance class
|
||||
- rejection conditions
|
||||
|
||||
## Next Audit Steps
|
||||
|
||||
1. Search surviving routing/orchestration artifacts for explicit `worker_type`
|
||||
labels.
|
||||
|
||||
2. Search C++ raising artifacts for explicit `cpp_raising_profile` choices.
|
||||
|
||||
3. Draft per-gate extraction specs for:
|
||||
- `prereq_op_selector`
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `required_tool_selection`
|
||||
|
||||
4. Add rejection rules for low-trust bundled outputs before building any
|
||||
training set.
|
||||
367
docs/case_studies/whetstone_dsl_semantic_mapping.md
Normal file
367
docs/case_studies/whetstone_dsl_semantic_mapping.md
Normal file
@@ -0,0 +1,367 @@
|
||||
# WhetstoneDSL Semantic Mapping
|
||||
|
||||
## Purpose
|
||||
|
||||
This document maps the current `whetstone_DSL` implementation surfaces into the
|
||||
semantic entities being established for `whetstone_RSA`.
|
||||
|
||||
It exists to make Sprint 003 executable.
|
||||
|
||||
The goal is not to restate architecture in the abstract. The goal is to answer:
|
||||
|
||||
- which `whetstone_DSL` files currently embody the first RSA-relevant decisions
|
||||
- which RSA semantic entities those files imply
|
||||
- which parts are already deterministic baselines
|
||||
- which parts should become explicit gate contracts
|
||||
- which parts should remain skeletons before training
|
||||
|
||||
## Mapping Rule
|
||||
|
||||
Each `whetstone_DSL` surface should be classified into one of four roles:
|
||||
|
||||
- semantic source candidate
|
||||
- deterministic baseline
|
||||
- runtime/executor consumer
|
||||
- training-data source candidate
|
||||
|
||||
This keeps the case study from collapsing implementation, semantics, and data
|
||||
collection into one bucket.
|
||||
|
||||
## Surface Map
|
||||
|
||||
### 1. `editor/src/TaskitemGeneratorV2.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- generate taskitems from decomposed workstreams
|
||||
- infer `workerType`
|
||||
- infer `prerequisiteOps`
|
||||
- mark queue readiness
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- `worker_type` is a first-class `DecisionContract` plus `DecisionSurface`
|
||||
- `prereq_op_selector` is a first-class `DecisionContract` plus structured
|
||||
`DecisionSurface`
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- current keyword worker heuristic is the baseline policy
|
||||
- current prerequisite-op defaulting behavior is the baseline policy
|
||||
|
||||
Training-data value:
|
||||
|
||||
- high, once paired with richer context and accepted downstream contracts
|
||||
- especially useful for:
|
||||
- task description -> `worker_type`
|
||||
- task description -> prerequisite-op set
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `WorkerTypeGate`
|
||||
- `PrereqOpGate`
|
||||
|
||||
### 2. `editor/src/TaskitemConfidenceAmbiguity.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- compute confidence score
|
||||
- count ambiguity
|
||||
- derive escalation flag
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- this is primarily a policy/baseline artifact, not a first training gate
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- extremely high
|
||||
- should remain attached to future gate records as deterministic baseline
|
||||
evidence
|
||||
|
||||
Training-data value:
|
||||
|
||||
- low as a primary specialist target
|
||||
- high as a supervision source for:
|
||||
- deterministic-baseline detection
|
||||
- guardrail-policy comparison
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `DeterministicPolicyBaseline`
|
||||
- not an initial RSA gate
|
||||
|
||||
### 3. `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- intake normalization and taskitem generation orchestration
|
||||
- execution-contract shaping
|
||||
- verification-type inference
|
||||
- target-file inference
|
||||
- acceptance-command inference
|
||||
- step-sequence inference
|
||||
- cross-project-target inference
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- `verification_type` is a first-class immediate gate
|
||||
- execution contract shaping is a structured semantic frontier
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- very high for current heuristics
|
||||
- should be preserved as baseline before any specialist substitution
|
||||
|
||||
Training-data value:
|
||||
|
||||
- very high if taskitem runs with context and final MCP choices still exist
|
||||
- likely strongest surface for:
|
||||
- `verification_type`
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `execution_contract_shape`
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `VerificationTypeGate`
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `AcceptanceCommandSelectionDecision`
|
||||
- `StepSequenceDecision`
|
||||
|
||||
### 4. `editor/src/ScopeMilestoneDecomposer.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- milestone grouping
|
||||
- workstream grouping
|
||||
- grouped summary/title generation
|
||||
- uncertainty scoring
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- grouping decisions are real semantic decisions
|
||||
- but they are not yet clean enough as direct training targets
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- high
|
||||
|
||||
Training-data value:
|
||||
|
||||
- medium
|
||||
- useful after the grouping contract is made explicit
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `MilestoneGroupingDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
- `WorkstreamSummaryPolicy`
|
||||
|
||||
### 5. `editor/src/RoutingEngine.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- choose worker layer
|
||||
- choose context width
|
||||
- set review flag
|
||||
- derive agent role
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- future second-wave RSA surface
|
||||
- useful as a meta-layer above particular gates
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- extremely high
|
||||
- keep this baseline intact for later RSA-versus-rule comparison
|
||||
|
||||
Training-data value:
|
||||
|
||||
- medium to high once real historical routing outcomes are assembled
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `LayerSuitabilityDecision`
|
||||
- `RoutingContextDecision`
|
||||
|
||||
Note:
|
||||
|
||||
- these sit one layer above RSA identity itself, so they should not be confused
|
||||
with an RSA gate contract
|
||||
|
||||
### 6. `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- choose and validate C++ raising profile
|
||||
- run deterministic C++ raising and packaging pipeline
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- `cpp_raising_profile` is a clean bounded gate candidate
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- current manual/explicit profile selection is the baseline
|
||||
|
||||
Training-data value:
|
||||
|
||||
- medium
|
||||
- strongest when paired with IR traits, risk annotations, and downstream review
|
||||
outcomes
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `CppRaisingProfileGate`
|
||||
|
||||
### 7. `editor/src/ConstrainedProjectionGate.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- block generation when environment constraints are violated
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- this is executor-side deterministic gating
|
||||
- it is not currently an RSA gate candidate
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- extremely high
|
||||
|
||||
Training-data value:
|
||||
|
||||
- useful as supervision for invalid-output prevention
|
||||
- not useful as a primary specialist target
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `DeterministicConstraintGate`
|
||||
|
||||
### 8. `editor/src/ast/ProjectionGenerator.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- dispatch AST node kinds to language-specific visitors
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- almost none at the current stage
|
||||
- this is deterministic target-language projection
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- total
|
||||
|
||||
Training-data value:
|
||||
|
||||
- not a primary RSA training surface
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- no immediate RSA semantic entity
|
||||
- leave in deterministic projection layer
|
||||
|
||||
### 9. `editor/src/CrossLanguageProjector.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- clone AST into target language
|
||||
- preserve/adapt annotations
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- good future source for bounded policy choices around adaptation profiles
|
||||
- current implementation is deterministic
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- high
|
||||
|
||||
Training-data value:
|
||||
|
||||
- low now
|
||||
- potentially higher later if multiple adaptation profiles exist
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- future `ProjectionProfileDecision`
|
||||
- not an immediate Sprint 003 gate
|
||||
|
||||
## Semantic Entity Alignment
|
||||
|
||||
### Immediate gate entities
|
||||
|
||||
| WhetstoneDSL surface | RSA entity | Status |
|
||||
| --- | --- | --- |
|
||||
| `TaskitemGeneratorV2::inferWorkerType` | `WorkerTypeGate` | bootstrap now |
|
||||
| `TaskitemGeneratorV2::inferPrerequisiteOps` | `PrereqOpGate` | bootstrap now |
|
||||
| `RegisterArchitectIntakeTools::inferVerificationType` | `VerificationTypeGate` | bootstrap now |
|
||||
| `RegisterCppRaisingTools` profile enum | `CppRaisingProfileGate` | bootstrap now |
|
||||
|
||||
### Skeleton-first entities
|
||||
|
||||
| WhetstoneDSL surface | RSA entity | Status |
|
||||
| --- | --- | --- |
|
||||
| execution-contract heuristics | `ExecutionContractShapeDecision` | skeleton now |
|
||||
| target-file inference | `TargetFileSelectionDecision` | skeleton now |
|
||||
| grouping heuristics | `WorkstreamGroupingDecision` | skeleton now |
|
||||
| grouped summary logic | `WorkstreamSummaryPolicy` | skeleton now |
|
||||
| step-sequence inference | `StepSequenceDecision` | skeleton now |
|
||||
|
||||
### Deterministic baselines to preserve
|
||||
|
||||
| WhetstoneDSL surface | RSA role |
|
||||
| --- | --- |
|
||||
| `TaskitemConfidenceAmbiguity` | deterministic baseline / guardrail policy |
|
||||
| `RoutingEngine` | deterministic meta-routing baseline |
|
||||
| `ConstrainedProjectionGate` | deterministic constraint gate |
|
||||
| `ProjectionGenerator` dispatch | deterministic projection executor |
|
||||
| `CrossLanguageProjector` current adaptation | deterministic projection baseline |
|
||||
|
||||
## What This Means For Integration
|
||||
|
||||
The current integration path should be:
|
||||
|
||||
1. capture gate semantics first
|
||||
2. preserve deterministic baselines as explicit comparison points
|
||||
3. identify data sources for each gate
|
||||
4. defer runtime substitution until the gate contracts are stable
|
||||
|
||||
The wrong integration path would be:
|
||||
|
||||
1. wire RSA runtime into `whetstone_DSL`
|
||||
2. guess what the gates mean later
|
||||
|
||||
This case study is strong enough now to avoid that mistake.
|
||||
|
||||
## Training Data Notes
|
||||
|
||||
Potential high-value training data sources still likely exist in or around
|
||||
`whetstone_DSL`, depending on what survived deletion:
|
||||
|
||||
- taskitem pipeline run logs with context and final MCP choices
|
||||
- specialist training data under `whetstone_DSL/specialists`
|
||||
- the `100 easy` and `25 challenging` project sets already used for examples and
|
||||
evaluation
|
||||
- sprint plan corpus already used for `verification_type`
|
||||
|
||||
Immediate implication:
|
||||
|
||||
- do not block semantic bootstrap on recovering training data
|
||||
- but treat recovered taskitem/MCP-choice runs as likely first-class supervision
|
||||
once the gate contracts are fixed
|
||||
|
||||
## Next Mapping Step
|
||||
|
||||
The next document after this one should define skeleton field sets for:
|
||||
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
|
||||
Those are the structured surfaces most likely to become later RSA gates once the
|
||||
semantic contract is explicit enough to benchmark cleanly.
|
||||
218
docs/case_studies/whetstone_dsl_training_data_strategy.md
Normal file
218
docs/case_studies/whetstone_dsl_training_data_strategy.md
Normal file
@@ -0,0 +1,218 @@
|
||||
# WhetstoneDSL Training Data Strategy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document records the data-handling rule for turning existing
|
||||
`whetstone_DSL` outputs into RSA training data.
|
||||
|
||||
The immediate concern is that much of the existing material may be:
|
||||
|
||||
- whole JSON responses
|
||||
- whole taskitem packets
|
||||
- multi-decision LLM outputs
|
||||
|
||||
rather than clean one-decision-per-example supervision.
|
||||
|
||||
That means the data cannot be used naively.
|
||||
|
||||
## Core Rule
|
||||
|
||||
Do not train per-gate specialists directly on full taskitem or full-response
|
||||
outputs without reconstructing:
|
||||
|
||||
- the exact local context for the decision
|
||||
- the specific decision target being learned
|
||||
- whether the label was independently chosen or entangled with other choices
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The old pipeline was optimized for:
|
||||
|
||||
- one LLM or SLM making several bounded choices together
|
||||
- minimizing token use by sharing context across those choices
|
||||
|
||||
The new RSA direction is different:
|
||||
|
||||
- one specialist per bounded decision
|
||||
- narrower context per decision
|
||||
- explicit gate contracts
|
||||
|
||||
So existing outputs may reflect:
|
||||
|
||||
- coupled decisions
|
||||
- prompt-order effects
|
||||
- hidden shared rationale
|
||||
- labels that were only valid because other fields in the same response were
|
||||
chosen jointly
|
||||
|
||||
If we ignore that, the training data will be semantically wrong even if it looks
|
||||
large.
|
||||
|
||||
## Safe Extraction Rule
|
||||
|
||||
For each candidate gate, build examples as:
|
||||
|
||||
- `decision_context`
|
||||
- `decision_label`
|
||||
- `decision_provenance`
|
||||
|
||||
and not as:
|
||||
|
||||
- full response blob -> label
|
||||
|
||||
### `decision_context`
|
||||
|
||||
Must include only the information that would have been available at the moment
|
||||
that gate should fire.
|
||||
|
||||
Examples:
|
||||
|
||||
- `worker_type`
|
||||
- task title
|
||||
- task intent
|
||||
- maybe requirement summary
|
||||
- not downstream fields derived after routing
|
||||
|
||||
- `verification_type`
|
||||
- target files
|
||||
- task title
|
||||
- task intent
|
||||
- not later acceptance-command decisions unless those are part of the gate by
|
||||
contract
|
||||
|
||||
- `prereq_op_selector`
|
||||
- task description
|
||||
- uncertainty and dependency signals if the intended gate contract includes them
|
||||
- not later execution outcomes
|
||||
|
||||
### `decision_label`
|
||||
|
||||
Must match the new gate contract exactly.
|
||||
|
||||
If the historic data uses a different shape than the new gate, do not force it.
|
||||
|
||||
Instead:
|
||||
|
||||
- transform it carefully
|
||||
- or discard it
|
||||
- or redefine the gate
|
||||
|
||||
### `decision_provenance`
|
||||
|
||||
Each extracted example should track:
|
||||
|
||||
- source artifact
|
||||
- source run id if available
|
||||
- whether label was explicit or inferred
|
||||
- whether the example came from:
|
||||
- human-authored heuristic baseline
|
||||
- LLM output
|
||||
- SLM output
|
||||
- deterministic reconstruction
|
||||
|
||||
This is necessary because not all labels have equal trust.
|
||||
|
||||
## Gate-Specific Warnings
|
||||
|
||||
### `worker_type`
|
||||
|
||||
This is likely recoverable from older taskitems, but only if the worker role was
|
||||
explicitly present and chosen before downstream execution shaping.
|
||||
|
||||
Risk:
|
||||
|
||||
- label polluted by later execution-contract logic
|
||||
|
||||
### `verification_type`
|
||||
|
||||
This is relatively safe because it can often be reconstructed from target files
|
||||
and acceptance intent.
|
||||
|
||||
Risk:
|
||||
|
||||
- historic labels may include joint decisions with acceptance-command shaping
|
||||
|
||||
### `prereq_op_selector`
|
||||
|
||||
This is the riskiest early gate for naive extraction.
|
||||
|
||||
Reason:
|
||||
|
||||
- older systems may have emitted a flat list of ops as one bundle
|
||||
- the new semantic contract treats this as structured or factorized
|
||||
|
||||
Rule:
|
||||
|
||||
- do not flatten structured op bundles into arbitrary multiclass labels
|
||||
|
||||
### `cpp_raising_profile`
|
||||
|
||||
This is likely only useful if old runs actually recorded explicit profile choice
|
||||
with enough IR context.
|
||||
|
||||
If not, this gate may need to start with synthetic or policy-authored data rather
|
||||
than mined response logs.
|
||||
|
||||
## Easy/Challenging Project Corpora
|
||||
|
||||
The `100 easy` and `25 challenging` project sets are valuable, but they are not
|
||||
automatically specialist training sets.
|
||||
|
||||
They are best treated as:
|
||||
|
||||
- context reservoirs
|
||||
- replay/evaluation corpora
|
||||
- example generators for extracting many per-gate decisions
|
||||
|
||||
The right pattern is:
|
||||
|
||||
1. replay or parse the project artifact
|
||||
2. isolate one decision surface at a time
|
||||
3. extract only the local context relevant to that gate
|
||||
4. record the corresponding bounded label
|
||||
|
||||
This should produce a gate-native dataset rather than a project-native dataset.
|
||||
|
||||
## Compatibility Constraint
|
||||
|
||||
We should not assume the new specialists must fit perfectly into the exact old
|
||||
taskitem bundle shape.
|
||||
|
||||
There are two valid outcomes:
|
||||
|
||||
### Option A: Fit specialists into current taskitems
|
||||
|
||||
Use when:
|
||||
|
||||
- the old taskitem structure already matches clean gate boundaries
|
||||
- the decision context can be isolated without distortion
|
||||
|
||||
### Option B: Rebuild taskitems around explicit gates
|
||||
|
||||
Use when:
|
||||
|
||||
- the current taskitems bundle too many coupled decisions
|
||||
- the old layout hides which context belongs to which choice
|
||||
- structured gates are being forced into flat labels
|
||||
|
||||
This is not failure.
|
||||
It may be the correct result of moving from one-model-many-decisions to
|
||||
many-model-one-decision architecture.
|
||||
|
||||
## Recommended Next Step
|
||||
|
||||
Before mining historic runs, define extraction specs for the first four gates:
|
||||
|
||||
- `verification_type`
|
||||
- `worker_type`
|
||||
- `prereq_op_selector`
|
||||
- `cpp_raising_profile`
|
||||
|
||||
Each extraction spec should state:
|
||||
|
||||
- allowed input fields
|
||||
- forbidden leaked fields
|
||||
- label shape
|
||||
- trust level of recovered labels
|
||||
|
||||
Only after that should we parse old JSON outputs or taskitem runs.
|
||||
@@ -221,3 +221,652 @@ It should become:
|
||||
|
||||
That is the architecture most consistent with both Whetstone and the long-term
|
||||
goals of RSA.
|
||||
|
||||
## Bootstrap Mapping From The Current C++ Scaffold
|
||||
|
||||
The current C++ code is useful bootstrap material, but it should now be treated as
|
||||
input to the semantic model rather than the long-term definition of that model.
|
||||
|
||||
This section inventories the current scaffold and classifies each piece as either:
|
||||
|
||||
- semantic contract that should move into Whetstone-authored form
|
||||
- runtime behavior that should remain in projected C++
|
||||
- bootstrap/demo code that should not define long-term architecture
|
||||
|
||||
### Current C++ Semantic Surface
|
||||
|
||||
From `include/whetstone_rsa/types.h`, the current scaffold already expresses the
|
||||
core semantic vocabulary that should be re-authored in Whetstone form:
|
||||
|
||||
- ontology enums
|
||||
- `InputStructure`
|
||||
- `InputModality`
|
||||
- `OutputTopology`
|
||||
- `OutputConstraint`
|
||||
- `LabelStability`
|
||||
- `TargetLayer`
|
||||
- diagnosis enums
|
||||
- `FailureClass`
|
||||
- `Intervention`
|
||||
- `DeployMode`
|
||||
- `RiskTier`
|
||||
- contract entities
|
||||
- `GateDefinition`
|
||||
- `GateEvidence`
|
||||
- `PolicyContext`
|
||||
- `ProbeRequest`
|
||||
- `ProbeResult`
|
||||
- `GateDiagnosis`
|
||||
- `GateSuitability`
|
||||
- `PolicyRecommendation`
|
||||
|
||||
These should not remain primarily hand-authored C++ concepts.
|
||||
|
||||
They should become semantic entities in the Whetstone source of truth.
|
||||
|
||||
### Recommended Whetstone Semantic Entities
|
||||
|
||||
The current scaffold suggests the following first semantic model.
|
||||
|
||||
#### 1. `DecisionContract`
|
||||
|
||||
This should become the semantic home for the ontology-level fields that currently
|
||||
sit inside `GateDefinition`.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `gate_id`
|
||||
- `task_family`
|
||||
- `input_structure`
|
||||
- `input_modality`
|
||||
- `output_topology`
|
||||
- `output_constraint`
|
||||
- `risk_tier`
|
||||
- `policy_stability`
|
||||
- `deterministic_baseline_available`
|
||||
- `supports_abstain`
|
||||
- `metadata`
|
||||
|
||||
This is the contract boundary, independent of backend family.
|
||||
|
||||
#### 2. `DecisionSurface`
|
||||
|
||||
The current `GateDefinition.labels` plus `candidate_factorizations` really describe
|
||||
the shape of the output space, not just the gate identity.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `output_labels`
|
||||
- `slot_schema`
|
||||
- `candidate_factorizations`
|
||||
- `candidate_gate_shapes`
|
||||
- `invalid_combinations`
|
||||
- `deterministic_prepass_options`
|
||||
|
||||
This gives the AST layer a place to describe flat multiclass, hierarchical, or
|
||||
factorized designs without forcing that structure to remain implicit in C++.
|
||||
|
||||
#### 3. `DecisionRequestModel`
|
||||
|
||||
The current C++ scaffold does not yet have a first-class request packet type
|
||||
separate from evaluation/demo inputs.
|
||||
|
||||
That should be corrected in the AST-first model.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `packet_id`
|
||||
- `text_fields`
|
||||
- `symbolic_fields`
|
||||
- `numeric_fields`
|
||||
- `context_fields`
|
||||
- `state_fields`
|
||||
- `provenance`
|
||||
|
||||
This entity should support both enum-style gates and bounded structured outputs.
|
||||
|
||||
#### 4. `DecisionResultModel`
|
||||
|
||||
The current scaffold implicitly assumes label selection and policy behavior, but the
|
||||
AST-first model should make result structure explicit.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `decision_id`
|
||||
- `selected_label`
|
||||
- `filled_slots`
|
||||
- `confidence`
|
||||
- `abstain`
|
||||
- `retry_strategy`
|
||||
- `escalation_target`
|
||||
- `trace_packet`
|
||||
|
||||
This should later project into both C++ runtime structs and packaging artifacts.
|
||||
|
||||
#### 5. `DiagnosisModel`
|
||||
|
||||
The current `GateDiagnosis` is already close to a semantic object and should be
|
||||
preserved almost directly, but moved into Whetstone-authored form.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `primary_failure_class`
|
||||
- `secondary_failure_classes`
|
||||
- `supporting_signals`
|
||||
- `recommended_probes`
|
||||
- `recommended_interventions`
|
||||
- `recommended_policy`
|
||||
- `confidence`
|
||||
|
||||
#### 6. `ProbeDefinition`
|
||||
|
||||
The current `Probe` interface and `ProbeRequest` / `ProbeResult` pair indicate that
|
||||
probes are part of the semantic model, not just implementation detail.
|
||||
|
||||
The AST layer should explicitly represent:
|
||||
|
||||
- `probe_id`
|
||||
- `applies_to_gate_shapes`
|
||||
- `required_evidence_fields`
|
||||
- `produced_signals`
|
||||
- `failure_hints`
|
||||
- `intervention_hints`
|
||||
|
||||
Concrete runtime probes such as `confidence_threshold_probe` should then be C++
|
||||
implementations of these declared probe contracts.
|
||||
|
||||
#### 7. `SuitabilityAssessment`
|
||||
|
||||
The current `GateSuitability` type should become a semantic output model for the
|
||||
pre-RSA decision that determines whether the problem belongs in the RSA layer at all.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `recommended`
|
||||
- `target_layer`
|
||||
- `rationale`
|
||||
- `bounded_output_confidence`
|
||||
- `deterministic_baseline_available`
|
||||
- `label_stability_risk`
|
||||
- `suggested_gate_shape`
|
||||
|
||||
#### 8. `BackendSuitability`
|
||||
|
||||
This does not yet exist in code, but the roadmap now requires it.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `gate_id`
|
||||
- `candidate_backends`
|
||||
- `recommended_backend`
|
||||
- `rationale`
|
||||
- `confidence`
|
||||
- `benchmark_refs`
|
||||
|
||||
This must be semantic-first, not added ad hoc in C++ later.
|
||||
|
||||
#### 9. `GatePackage`
|
||||
|
||||
The runtime needs a durable packaged form for serving decisions after offline
|
||||
selection and calibration.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `gate_id`
|
||||
- `contract_ref`
|
||||
- `gate_shape`
|
||||
- `backend_family`
|
||||
- `artifact_ref`
|
||||
- `thresholds`
|
||||
- `abstain_policy`
|
||||
- `escalation_policy`
|
||||
- `trace_schema`
|
||||
- `version`
|
||||
|
||||
### What Should Remain Runtime-Only In C++
|
||||
|
||||
The following concepts should remain projected runtime behavior rather than become
|
||||
the semantic source of truth:
|
||||
|
||||
- `Probe` virtual interface
|
||||
- `ProbeRegistry`
|
||||
- `DiagnosisEngine`
|
||||
- `SuitabilityAssessor`
|
||||
- backend adapter implementations
|
||||
- calibration code
|
||||
- packaging loaders
|
||||
- benchmark runners
|
||||
|
||||
These are execution mechanisms, not the semantic model itself.
|
||||
|
||||
### What In The Current Scaffold Is Only Bootstrap
|
||||
|
||||
The following parts are useful but should not steer the long-term semantic design:
|
||||
|
||||
- `examples/diagnose_demo.cpp`
|
||||
- the current hand-authored stringification helpers
|
||||
- any current field arrangement that exists only because it was convenient for the
|
||||
first C++ scaffold
|
||||
|
||||
### Immediate Migration Map
|
||||
|
||||
The practical next move should be:
|
||||
|
||||
1. Represent the current C++ contract entities above as Whetstone semantic objects.
|
||||
2. Mark which current fields are semantic contract versus runtime-only detail.
|
||||
3. Treat `types.h` as a temporary projected contract snapshot, not the authoring
|
||||
home of the model.
|
||||
4. Add projection rules so future C++ runtime types are derived from the semantic
|
||||
source rather than independently evolved.
|
||||
|
||||
### Most Important Boundary To Preserve
|
||||
|
||||
The current scaffold already mixes two different layers that must stay separate:
|
||||
|
||||
- semantic contract
|
||||
- runtime execution strategy
|
||||
|
||||
If that separation is preserved now, `whetstone_RSA` can become genuinely AST-first.
|
||||
If it is not preserved, the project will keep drifting into a C++-first runtime
|
||||
with AST plans attached afterward.
|
||||
|
||||
## First RSA Semantic Model
|
||||
|
||||
This section defines the first concrete semantic model that should become the
|
||||
source of truth for `whetstone_RSA`.
|
||||
|
||||
It is intentionally small and close to the current scaffold so migration can
|
||||
start immediately without pretending the architecture is already finished.
|
||||
|
||||
The design rule is:
|
||||
|
||||
- semantic entities describe contract and meaning
|
||||
- projected C++ types describe runtime execution
|
||||
|
||||
### Model Goals
|
||||
|
||||
The first semantic model must support all of the following from the beginning:
|
||||
|
||||
- bounded enum decisions
|
||||
- bounded structured slot-filling decisions
|
||||
- suitability assessment before a gate is adopted
|
||||
- diagnosis after a gate underperforms
|
||||
- gate-shape comparison
|
||||
- backend-selection metadata
|
||||
- runtime packaging metadata
|
||||
|
||||
### Semantic Entity Set
|
||||
|
||||
The first semantic model should consist of these entities:
|
||||
|
||||
- `DecisionContract`
|
||||
- `DecisionSurface`
|
||||
- `DecisionRequestModel`
|
||||
- `DecisionResultModel`
|
||||
- `SuitabilityAssessment`
|
||||
- `DiagnosisModel`
|
||||
- `ProbeDefinition`
|
||||
- `BackendSuitability`
|
||||
- `GatePackage`
|
||||
|
||||
### 1. `DecisionContract`
|
||||
|
||||
This is the semantic identity of an RSA gate.
|
||||
|
||||
It answers:
|
||||
|
||||
- what decision surface exists
|
||||
- what kind of inputs it consumes
|
||||
- what kind of bounded outputs it promises
|
||||
- what policy constraints govern deployment
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionContract
|
||||
gate_id
|
||||
task_family
|
||||
input_structure
|
||||
input_modality
|
||||
output_topology
|
||||
output_constraint
|
||||
risk_tier
|
||||
policy_stability
|
||||
deterministic_baseline_available
|
||||
supports_abstain
|
||||
metadata
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `policy_stability` should replace C++ naming that implies only labels drift.
|
||||
- this entity should not include backend-family decisions
|
||||
- this entity should not include benchmark evidence
|
||||
|
||||
### 2. `DecisionSurface`
|
||||
|
||||
This is the semantic description of the output space.
|
||||
|
||||
It is separate from `DecisionContract` because two gates can share similar
|
||||
contract properties but differ in shape:
|
||||
|
||||
- flat enum choice
|
||||
- factorized decision
|
||||
- hierarchical route
|
||||
- bounded slot fill
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionSurface
|
||||
gate_id
|
||||
primary_mode // enum | structured | hybrid
|
||||
output_labels
|
||||
slot_schema
|
||||
candidate_gate_shapes // binary | multiclass | factorized | hierarchical
|
||||
candidate_factorizations
|
||||
invalid_combinations
|
||||
deterministic_prepass_options
|
||||
downstream_executor_kind
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `slot_schema` is required from version 0 even if many early gates are enum-only
|
||||
- `invalid_combinations` should make structured outputs safe by contract
|
||||
- `downstream_executor_kind` should record what deterministic system consumes the result
|
||||
|
||||
### 3. `DecisionRequestModel`
|
||||
|
||||
This is the semantic input packet model for runtime use.
|
||||
|
||||
It should support hybrid packets directly instead of forcing a project to flatten
|
||||
everything into one string.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionRequestModel
|
||||
packet_id
|
||||
gate_id
|
||||
text_fields
|
||||
symbolic_fields
|
||||
numeric_fields
|
||||
context_fields
|
||||
state_fields
|
||||
provenance
|
||||
```
|
||||
|
||||
Expected usage:
|
||||
|
||||
- `text_fields`
|
||||
- natural language summaries
|
||||
- extracted comments
|
||||
- short requirement packets
|
||||
- `symbolic_fields`
|
||||
- AST node kinds
|
||||
- annotations
|
||||
- enums
|
||||
- dependency graphs
|
||||
- `numeric_fields`
|
||||
- counts
|
||||
- scores
|
||||
- ratios
|
||||
- latency or complexity features
|
||||
|
||||
### 4. `DecisionResultModel`
|
||||
|
||||
This is the canonical semantic output model for runtime decisions.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionResultModel
|
||||
decision_id
|
||||
gate_id
|
||||
selected_label
|
||||
filled_slots
|
||||
confidence
|
||||
abstain
|
||||
retry_strategy
|
||||
escalation_target
|
||||
trace_packet
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- enum gates mainly use `selected_label`
|
||||
- structured gates mainly use `filled_slots`
|
||||
- hybrid outputs may use both
|
||||
|
||||
### 5. `SuitabilityAssessment`
|
||||
|
||||
This is the semantic result of asking whether a problem belongs in RSA at all.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
SuitabilityAssessment
|
||||
gate_id
|
||||
recommended
|
||||
target_layer // deterministic | rsa | slm | llm | human
|
||||
rationale
|
||||
bounded_output_confidence
|
||||
deterministic_baseline_available
|
||||
policy_stability_risk
|
||||
suggested_gate_shape
|
||||
```
|
||||
|
||||
This entity is intentionally separate from diagnosis and backend selection.
|
||||
|
||||
### 6. `DiagnosisModel`
|
||||
|
||||
This is the semantic result of evaluating a weak or risky gate.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DiagnosisModel
|
||||
gate_id
|
||||
primary_failure_class
|
||||
secondary_failure_classes
|
||||
supporting_signals
|
||||
recommended_probes
|
||||
recommended_interventions
|
||||
recommended_policy
|
||||
confidence
|
||||
```
|
||||
|
||||
This should remain close to the current C++ `GateDiagnosis`.
|
||||
|
||||
### 7. `ProbeDefinition`
|
||||
|
||||
This is the semantic declaration of a diagnostic probe.
|
||||
|
||||
The semantic layer should define what a probe means before runtime C++ implements
|
||||
how it executes.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
ProbeDefinition
|
||||
probe_id
|
||||
applies_to_gate_shapes
|
||||
required_contract_fields
|
||||
required_evidence_fields
|
||||
produced_signals
|
||||
failure_hints
|
||||
intervention_hints
|
||||
```
|
||||
|
||||
Concrete examples:
|
||||
|
||||
- `confidence_threshold_probe`
|
||||
- `class_confusion_probe`
|
||||
- `factorization_probe`
|
||||
- `deterministic_rule_probe`
|
||||
- `context_width_probe`
|
||||
|
||||
### 8. `BackendSuitability`
|
||||
|
||||
This is the semantic result of backend comparison after a gate has already been
|
||||
judged to belong in RSA.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
BackendSuitability
|
||||
gate_id
|
||||
candidate_backends
|
||||
recommended_backend
|
||||
rationale
|
||||
confidence
|
||||
benchmark_refs
|
||||
```
|
||||
|
||||
Expected backend families:
|
||||
|
||||
- `linear_model`
|
||||
- `gradient_boosting`
|
||||
- `random_forest`
|
||||
- `bayesian`
|
||||
- `transformer`
|
||||
- `hybrid`
|
||||
- `rl_policy`
|
||||
|
||||
`rl_policy` should exist in the model now even if it remains an unimplemented
|
||||
runtime placeholder for the first version.
|
||||
|
||||
### 9. `GatePackage`
|
||||
|
||||
This is the semantic deployment artifact description.
|
||||
|
||||
It exists so runtime loading is a projection of semantic packaging decisions,
|
||||
not a one-off C++ concern.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
GatePackage
|
||||
gate_id
|
||||
contract_ref
|
||||
surface_ref
|
||||
gate_shape
|
||||
backend_family
|
||||
artifact_ref
|
||||
thresholds
|
||||
abstain_policy
|
||||
escalation_policy
|
||||
trace_schema
|
||||
version
|
||||
```
|
||||
|
||||
### Semantic Relationships
|
||||
|
||||
The first model should follow these relationships:
|
||||
|
||||
```text
|
||||
DecisionContract
|
||||
owns high-level decision identity
|
||||
|
||||
DecisionSurface
|
||||
refines output shape for a contract
|
||||
|
||||
DecisionRequestModel
|
||||
is evaluated against DecisionContract + DecisionSurface
|
||||
|
||||
DecisionResultModel
|
||||
must satisfy DecisionSurface constraints
|
||||
|
||||
SuitabilityAssessment
|
||||
determines whether the decision belongs in RSA at all
|
||||
|
||||
DiagnosisModel
|
||||
explains weak performance or risky deployment
|
||||
|
||||
ProbeDefinition
|
||||
declares reusable diagnosis mechanisms
|
||||
|
||||
BackendSuitability
|
||||
recommends which implementation family should fill an RSA contract
|
||||
|
||||
GatePackage
|
||||
records the chosen runtime projection of a validated contract
|
||||
```
|
||||
|
||||
### Projection Map To The Current C++ Scaffold
|
||||
|
||||
The current scaffold should be treated as the first runtime projection target.
|
||||
|
||||
#### Project directly from semantics into current C++ types
|
||||
|
||||
```text
|
||||
DecisionContract -> GateDefinition core fields
|
||||
DecisionSurface -> GateDefinition.labels + candidate_factorizations + future slot fields
|
||||
DecisionResultModel -> future runtime result structs
|
||||
SuitabilityAssessment -> GateSuitability
|
||||
DiagnosisModel -> GateDiagnosis
|
||||
ProbeDefinition -> Probe metadata + ProbeRequest/ProbeResult contracts
|
||||
```
|
||||
|
||||
#### Runtime-only C++ types that should remain projected
|
||||
|
||||
```text
|
||||
Probe
|
||||
ProbeRegistry
|
||||
DiagnosisEngine
|
||||
SuitabilityAssessor
|
||||
ConfidenceThresholdProbe
|
||||
```
|
||||
|
||||
These are implementations, not semantic source objects.
|
||||
|
||||
### Immediate Type Migration Rule
|
||||
|
||||
Until semantic authoring is fully in place, the current C++ fields should be
|
||||
interpreted according to this split:
|
||||
|
||||
- semantic-now
|
||||
- ontology enums
|
||||
- gate contract fields
|
||||
- diagnosis enums
|
||||
- suitability result fields
|
||||
- runtime-now
|
||||
- probe interfaces
|
||||
- engine wiring
|
||||
- confidence-threshold implementation
|
||||
- demo executable
|
||||
|
||||
When a new field is proposed, the default question should be:
|
||||
|
||||
- does this describe semantic meaning?
|
||||
- if yes, add it to the semantic model first
|
||||
- does this describe runtime execution detail?
|
||||
- if yes, keep it in projected C++
|
||||
|
||||
### Immediate Next Conversion Step
|
||||
|
||||
The next concrete implementation step after this document should be:
|
||||
|
||||
1. represent `DecisionContract`, `DecisionSurface`, and `ProbeDefinition` as the
|
||||
first Whetstone-authored RSA semantic forms
|
||||
2. map each field in `types.h` into one of those forms
|
||||
3. mark missing fields needed for structured slot outputs and backend selection
|
||||
4. treat `types.h` as a temporary projection snapshot until semantic-first
|
||||
generation replaces hand editing
|
||||
|
||||
## Bootstrap Artifact
|
||||
|
||||
The first machine-readable semantic bootstrap artifact now lives at:
|
||||
|
||||
- `semantic/rsa_semantic_bootstrap_v0.json`
|
||||
|
||||
It is intentionally limited to the first three semantic entities needed to begin
|
||||
realignment:
|
||||
|
||||
- `DecisionContract`
|
||||
- `DecisionSurface`
|
||||
- `ProbeDefinition`
|
||||
|
||||
That file should be treated as the current semantic bootstrap source ahead of any
|
||||
further handwritten expansion of C++ contract types.
|
||||
|
||||
490
semantic/rsa_semantic_bootstrap_v0.json
Normal file
490
semantic/rsa_semantic_bootstrap_v0.json
Normal file
@@ -0,0 +1,490 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "whetstone-rsa/semantic-bootstrap-v0",
|
||||
"title": "RSA Semantic Bootstrap V0",
|
||||
"version": "0.1.0-bootstrap",
|
||||
"status": "bootstrap_source_of_truth",
|
||||
"notes": [
|
||||
"This file is the first machine-readable semantic bootstrap artifact for whetstone_RSA.",
|
||||
"It is intentionally smaller than the full target semantic model.",
|
||||
"It defines the first three semantic entities called out in Sprint 002: DecisionContract, DecisionSurface, and ProbeDefinition.",
|
||||
"Until AST-native authoring is fully in place, this artifact should be treated as the semantic source of truth ahead of handwritten C++ contract growth."
|
||||
],
|
||||
"enums": {
|
||||
"risk_tier": [
|
||||
"low",
|
||||
"medium",
|
||||
"high"
|
||||
],
|
||||
"input_structure": [
|
||||
"structured",
|
||||
"unstructured",
|
||||
"hybrid"
|
||||
],
|
||||
"input_modality": [
|
||||
"textual",
|
||||
"numeric",
|
||||
"symbolic",
|
||||
"multimodal"
|
||||
],
|
||||
"output_topology": [
|
||||
"discrete",
|
||||
"continuous",
|
||||
"hybrid"
|
||||
],
|
||||
"output_constraint": [
|
||||
"deterministic",
|
||||
"bounded",
|
||||
"open"
|
||||
],
|
||||
"policy_stability": [
|
||||
"stable",
|
||||
"mixed",
|
||||
"non_stationary"
|
||||
],
|
||||
"gate_shape": [
|
||||
"binary",
|
||||
"multiclass",
|
||||
"factorized",
|
||||
"hierarchical"
|
||||
],
|
||||
"backend_family": [
|
||||
"linear_model",
|
||||
"gradient_boosting",
|
||||
"random_forest",
|
||||
"bayesian",
|
||||
"transformer",
|
||||
"hybrid",
|
||||
"rl_policy"
|
||||
],
|
||||
"failure_class": [
|
||||
"unknown",
|
||||
"capacity_limited_gate",
|
||||
"factorizable_gate",
|
||||
"non_stationary_gate",
|
||||
"missing_context_gate",
|
||||
"deterministic_disguised_as_ml_gate",
|
||||
"label_space_mismatch",
|
||||
"guardrail_limited_gate"
|
||||
],
|
||||
"intervention": [
|
||||
"collect_more_evidence",
|
||||
"keep_current_gate",
|
||||
"deploy_with_guardrails",
|
||||
"raise_confidence_threshold",
|
||||
"add_retry_with_context",
|
||||
"add_deterministic_prepass",
|
||||
"replace_with_deterministic_logic",
|
||||
"factor_into_subgates",
|
||||
"switch_to_hierarchical_gate",
|
||||
"revise_label_schema",
|
||||
"enrich_input_context",
|
||||
"evaluate_larger_model_tier"
|
||||
]
|
||||
},
|
||||
"entities": {
|
||||
"DecisionContract": {
|
||||
"purpose": "Semantic identity of an RSA decision gate, independent of runtime backend choice.",
|
||||
"fields": [
|
||||
{
|
||||
"name": "gate_id",
|
||||
"type": "string",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "task_family",
|
||||
"type": "string",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "input_structure",
|
||||
"type": "enum:input_structure",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "input_modality",
|
||||
"type": "enum:input_modality",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "output_topology",
|
||||
"type": "enum:output_topology",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "output_constraint",
|
||||
"type": "enum:output_constraint",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "risk_tier",
|
||||
"type": "enum:risk_tier",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "policy_stability",
|
||||
"type": "enum:policy_stability",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "deterministic_baseline_available",
|
||||
"type": "boolean",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "supports_abstain",
|
||||
"type": "boolean",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "metadata",
|
||||
"type": "map<string,string>",
|
||||
"required": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"DecisionSurface": {
|
||||
"purpose": "Semantic description of output shape, factorization, and bounded slot structure for a gate.",
|
||||
"fields": [
|
||||
{
|
||||
"name": "gate_id",
|
||||
"type": "string",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "primary_mode",
|
||||
"type": "enum",
|
||||
"allowed_values": [
|
||||
"enum",
|
||||
"structured",
|
||||
"hybrid"
|
||||
],
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "output_labels",
|
||||
"type": "array<string>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "slot_schema",
|
||||
"type": "array<object>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "candidate_gate_shapes",
|
||||
"type": "array<enum:gate_shape>",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "candidate_factorizations",
|
||||
"type": "array<array<string>>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "invalid_combinations",
|
||||
"type": "array<object>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "deterministic_prepass_options",
|
||||
"type": "array<string>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "downstream_executor_kind",
|
||||
"type": "string",
|
||||
"required": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"ProbeDefinition": {
|
||||
"purpose": "Semantic declaration of a diagnosis probe before any runtime C++ implementation.",
|
||||
"fields": [
|
||||
{
|
||||
"name": "probe_id",
|
||||
"type": "string",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "applies_to_gate_shapes",
|
||||
"type": "array<enum:gate_shape>",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"name": "required_contract_fields",
|
||||
"type": "array<string>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "required_evidence_fields",
|
||||
"type": "array<string>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "produced_signals",
|
||||
"type": "array<string>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "failure_hints",
|
||||
"type": "array<enum:failure_class>",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "intervention_hints",
|
||||
"type": "array<enum:intervention>",
|
||||
"required": false
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"projection_targets": {
|
||||
"cpp_runtime": {
|
||||
"notes": [
|
||||
"The current C++ scaffold is a projection target, not the authoring home of the model.",
|
||||
"Future changes should update this semantic artifact first, then realign projected C++ types."
|
||||
],
|
||||
"field_map": {
|
||||
"DecisionContract": {
|
||||
"target_type": "whetstone::rsa::GateDefinition",
|
||||
"target_fields": [
|
||||
"gate_id",
|
||||
"task_family",
|
||||
"input_structure",
|
||||
"input_modality",
|
||||
"output_topology",
|
||||
"output_constraint",
|
||||
"risk_tier",
|
||||
"label_stability",
|
||||
"deterministic_baseline_available",
|
||||
"supports_abstain",
|
||||
"metadata"
|
||||
]
|
||||
},
|
||||
"DecisionSurface": {
|
||||
"target_type": "whetstone::rsa::GateDefinition",
|
||||
"target_fields": [
|
||||
"labels",
|
||||
"candidate_factorizations"
|
||||
],
|
||||
"notes": [
|
||||
"slot_schema and invalid_combinations are semantic-first fields not yet fully represented in the current C++ scaffold.",
|
||||
"candidate_gate_shapes and downstream_executor_kind also need future projected support."
|
||||
]
|
||||
},
|
||||
"ProbeDefinition": {
|
||||
"target_type": "whetstone::rsa::Probe plus ProbeRequest/ProbeResult contract",
|
||||
"target_fields": [
|
||||
"probe_id",
|
||||
"failure_hints",
|
||||
"intervention_hints"
|
||||
],
|
||||
"notes": [
|
||||
"The current runtime does not yet expose semantic-first probe declarations separate from implementations."
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"bootstrap_instances": {
|
||||
"DecisionContract": [
|
||||
{
|
||||
"gate_id": "verification_type",
|
||||
"task_family": "taskitem_verification_routing",
|
||||
"input_structure": "unstructured",
|
||||
"input_modality": "textual",
|
||||
"output_topology": "discrete",
|
||||
"output_constraint": "bounded",
|
||||
"risk_tier": "medium",
|
||||
"policy_stability": "stable",
|
||||
"deterministic_baseline_available": true,
|
||||
"supports_abstain": true,
|
||||
"metadata": {
|
||||
"case_study": "whetstone_dsl",
|
||||
"source_surface": "editor/src/mcp/RegisterArchitectIntakeTools.h",
|
||||
"inventory_disposition": "rsa_immediate",
|
||||
"downstream_executor_kind": "verification_contract"
|
||||
}
|
||||
},
|
||||
{
|
||||
"gate_id": "worker_type",
|
||||
"task_family": "taskitem_worker_routing",
|
||||
"input_structure": "unstructured",
|
||||
"input_modality": "textual",
|
||||
"output_topology": "discrete",
|
||||
"output_constraint": "bounded",
|
||||
"risk_tier": "medium",
|
||||
"policy_stability": "mixed",
|
||||
"deterministic_baseline_available": true,
|
||||
"supports_abstain": true,
|
||||
"metadata": {
|
||||
"case_study": "whetstone_dsl",
|
||||
"source_surface": "editor/src/TaskitemGeneratorV2.h",
|
||||
"inventory_disposition": "rsa_immediate",
|
||||
"downstream_executor_kind": "workflow_routing"
|
||||
}
|
||||
},
|
||||
{
|
||||
"gate_id": "prereq_op_selector",
|
||||
"task_family": "taskitem_execution_readiness",
|
||||
"input_structure": "hybrid",
|
||||
"input_modality": "symbolic",
|
||||
"output_topology": "hybrid",
|
||||
"output_constraint": "bounded",
|
||||
"risk_tier": "medium",
|
||||
"policy_stability": "stable",
|
||||
"deterministic_baseline_available": true,
|
||||
"supports_abstain": true,
|
||||
"metadata": {
|
||||
"case_study": "whetstone_dsl",
|
||||
"source_surface": "editor/src/TaskitemGeneratorV2.h",
|
||||
"inventory_disposition": "rsa_immediate",
|
||||
"downstream_executor_kind": "execution_contract",
|
||||
"structured_output_reason": "multi_op_selection"
|
||||
}
|
||||
},
|
||||
{
|
||||
"gate_id": "cpp_raising_profile",
|
||||
"task_family": "cpp_projection_profile_selection",
|
||||
"input_structure": "structured",
|
||||
"input_modality": "symbolic",
|
||||
"output_topology": "discrete",
|
||||
"output_constraint": "bounded",
|
||||
"risk_tier": "high",
|
||||
"policy_stability": "stable",
|
||||
"deterministic_baseline_available": true,
|
||||
"supports_abstain": true,
|
||||
"metadata": {
|
||||
"case_study": "whetstone_dsl",
|
||||
"source_surface": "editor/src/mcp/RegisterCppRaisingTools.h",
|
||||
"inventory_disposition": "rsa_immediate",
|
||||
"downstream_executor_kind": "cpp_ir_raising"
|
||||
}
|
||||
}
|
||||
],
|
||||
"DecisionSurface": [
|
||||
{
|
||||
"gate_id": "verification_type",
|
||||
"primary_mode": "enum",
|
||||
"output_labels": [
|
||||
"unit",
|
||||
"schema",
|
||||
"smoke",
|
||||
"docs"
|
||||
],
|
||||
"candidate_gate_shapes": [
|
||||
"multiclass"
|
||||
],
|
||||
"candidate_factorizations": [],
|
||||
"deterministic_prepass_options": [
|
||||
"target_file_suffix_rules",
|
||||
"title_keyword_prepass"
|
||||
],
|
||||
"downstream_executor_kind": "verification_policy"
|
||||
},
|
||||
{
|
||||
"gate_id": "worker_type",
|
||||
"primary_mode": "enum",
|
||||
"output_labels": [
|
||||
"implementer",
|
||||
"reviewer",
|
||||
"architect",
|
||||
"qa"
|
||||
],
|
||||
"candidate_gate_shapes": [
|
||||
"multiclass"
|
||||
],
|
||||
"candidate_factorizations": [],
|
||||
"deterministic_prepass_options": [
|
||||
"title_keyword_rules"
|
||||
],
|
||||
"downstream_executor_kind": "workflow_routing"
|
||||
},
|
||||
{
|
||||
"gate_id": "prereq_op_selector",
|
||||
"primary_mode": "structured",
|
||||
"slot_schema": [
|
||||
{
|
||||
"name": "validate_intake",
|
||||
"type": "boolean"
|
||||
},
|
||||
{
|
||||
"name": "architect_review",
|
||||
"type": "boolean"
|
||||
},
|
||||
{
|
||||
"name": "resolve_dependencies",
|
||||
"type": "boolean"
|
||||
}
|
||||
],
|
||||
"candidate_gate_shapes": [
|
||||
"factorized",
|
||||
"hierarchical"
|
||||
],
|
||||
"candidate_factorizations": [
|
||||
[
|
||||
"validate_intake",
|
||||
"architect_review",
|
||||
"resolve_dependencies"
|
||||
]
|
||||
],
|
||||
"invalid_combinations": [],
|
||||
"deterministic_prepass_options": [
|
||||
"always_require_validate_intake"
|
||||
],
|
||||
"downstream_executor_kind": "execution_contract"
|
||||
},
|
||||
{
|
||||
"gate_id": "cpp_raising_profile",
|
||||
"primary_mode": "enum",
|
||||
"output_labels": [
|
||||
"safe-first",
|
||||
"perf-first",
|
||||
"interop-first"
|
||||
],
|
||||
"candidate_gate_shapes": [
|
||||
"multiclass"
|
||||
],
|
||||
"candidate_factorizations": [],
|
||||
"deterministic_prepass_options": [
|
||||
"environment_policy_override",
|
||||
"ffi_presence_prepass"
|
||||
],
|
||||
"downstream_executor_kind": "cpp_ir_raising"
|
||||
}
|
||||
],
|
||||
"ProbeDefinition": [
|
||||
{
|
||||
"probe_id": "confidence_threshold_probe",
|
||||
"applies_to_gate_shapes": [
|
||||
"binary",
|
||||
"multiclass",
|
||||
"factorized",
|
||||
"hierarchical"
|
||||
],
|
||||
"required_contract_fields": [
|
||||
"supports_abstain"
|
||||
],
|
||||
"required_evidence_fields": [
|
||||
"accept_rate",
|
||||
"accuracy_on_accepted"
|
||||
],
|
||||
"produced_signals": [
|
||||
"accept_rate",
|
||||
"accuracy_on_accepted",
|
||||
"silent_error_rate"
|
||||
],
|
||||
"failure_hints": [
|
||||
"guardrail_limited_gate",
|
||||
"capacity_limited_gate"
|
||||
],
|
||||
"intervention_hints": [
|
||||
"deploy_with_guardrails",
|
||||
"raise_confidence_threshold",
|
||||
"evaluate_larger_model_tier"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
202
sprints/SPRINT-002-rsa-semantic-bootstrap.md
Normal file
202
sprints/SPRINT-002-rsa-semantic-bootstrap.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Sprint 002: RSA Semantic Bootstrap
|
||||
|
||||
## Status
|
||||
|
||||
Planned
|
||||
|
||||
## Intent
|
||||
|
||||
Define the first Whetstone-authored semantic model for `whetstone_RSA` and make it
|
||||
the new source-of-truth target for future RSA contract evolution.
|
||||
|
||||
This sprint exists to stop the project from continuing as a C++-first library by
|
||||
inertia.
|
||||
|
||||
## Problem
|
||||
|
||||
`whetstone_RSA` currently has a useful C++ scaffold, but the semantic center of
|
||||
gravity is still wrong.
|
||||
|
||||
The project has already concluded that RSA should be Whetstone-AST-first and that
|
||||
the C++ layer should be a runtime projection. That conclusion is documented, but it
|
||||
is not yet operationalized.
|
||||
|
||||
Right now:
|
||||
|
||||
- the semantic contract is still primarily embodied in `include/whetstone_rsa/types.h`
|
||||
- diagnosis/runtime interfaces are defined in C++ first
|
||||
- backend-selection and structured-slot support are planned but not yet represented
|
||||
in a semantic-first source model
|
||||
|
||||
If that continues, the bootstrap scaffold will become the architecture by default.
|
||||
|
||||
## Scope
|
||||
|
||||
This sprint is about semantic bootstrap only.
|
||||
|
||||
Primary outputs:
|
||||
|
||||
- a first concrete RSA semantic model
|
||||
- a mapping from the existing C++ scaffold into that semantic model
|
||||
- a projection boundary definition for future C++ runtime types
|
||||
|
||||
Supporting documents to update if needed:
|
||||
|
||||
- `docs/whetstone_ast_first_integration.md`
|
||||
- `ROADMAP.md`
|
||||
|
||||
This sprint does **not** require immediate deep changes to `whetstone_DSL`
|
||||
runtime code.
|
||||
|
||||
## Architectural Constraint
|
||||
|
||||
The semantic source of truth for RSA should live in Whetstone-authored forms.
|
||||
|
||||
The C++ code in `whetstone_RSA` should remain:
|
||||
|
||||
- runtime implementation
|
||||
- projected contract types
|
||||
- diagnosis/runtime execution machinery
|
||||
- backend adapter surface
|
||||
|
||||
It should not remain the primary authoring home of RSA semantics.
|
||||
|
||||
## Sprint Goals
|
||||
|
||||
### 1. Define the first semantic entity set
|
||||
|
||||
Establish the minimum initial semantic model for RSA:
|
||||
|
||||
- `DecisionContract`
|
||||
- `DecisionSurface`
|
||||
- `DecisionRequestModel`
|
||||
- `DecisionResultModel`
|
||||
- `SuitabilityAssessment`
|
||||
- `DiagnosisModel`
|
||||
- `ProbeDefinition`
|
||||
- `BackendSuitability`
|
||||
- `GatePackage`
|
||||
|
||||
### 2. Separate semantic contract from runtime behavior
|
||||
|
||||
Classify the current C++ scaffold into:
|
||||
|
||||
- semantic contract entities
|
||||
- runtime-only execution types
|
||||
- bootstrap/demo code
|
||||
|
||||
### 3. Define the first projection boundary
|
||||
|
||||
State clearly what will eventually be projected from semantic source into:
|
||||
|
||||
- C++ runtime contract structs
|
||||
- runtime manifests
|
||||
- schema artifacts
|
||||
- benchmark configuration artifacts
|
||||
|
||||
### 4. Prepare version-0 support for both enum and structured-slot gates
|
||||
|
||||
The semantic model must support:
|
||||
|
||||
- bounded enum outputs
|
||||
- bounded structured outputs with slots
|
||||
|
||||
from the beginning, even if early runtime examples are enum-dominant.
|
||||
|
||||
### 5. Leave RL/bandit as semantic placeholders only
|
||||
|
||||
Include `rl_policy` in backend-family semantics now so the contract stays open,
|
||||
but keep first implementation effort focused on:
|
||||
|
||||
- classical ML families
|
||||
- tiny transformer families
|
||||
|
||||
## Concrete Deliverables
|
||||
|
||||
### Deliverable A: Semantic model definition
|
||||
|
||||
Produce a concrete first semantic model with entity names, field groups, and
|
||||
relationship rules.
|
||||
|
||||
### Deliverable B: C++ scaffold mapping
|
||||
|
||||
Map the current public C++ surface into the semantic model, especially:
|
||||
|
||||
- `types.h`
|
||||
- `probe.h`
|
||||
- `probe_registry.h`
|
||||
- `diagnosis_engine.h`
|
||||
- `suitability_assessor.h`
|
||||
|
||||
### Deliverable C: Missing-field inventory
|
||||
|
||||
Identify what the current scaffold is missing for:
|
||||
|
||||
- structured slot outputs
|
||||
- backend suitability
|
||||
- deployment packaging
|
||||
- argument-oriented runtime invocation
|
||||
|
||||
### Deliverable D: Projection strategy
|
||||
|
||||
Define how future C++ types should be treated:
|
||||
|
||||
- temporary projection snapshots
|
||||
- not semantic source
|
||||
- updated from semantic-first changes rather than evolved independently
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A first semantic model is explicitly documented in RSA docs
|
||||
- The model supports both enum and structured-slot gates
|
||||
- The current C++ scaffold is mapped into semantic vs runtime-only layers
|
||||
- Backend suitability and gate packaging exist in the semantic model even if not
|
||||
fully implemented in code
|
||||
- The roadmap and AST-first integration docs point to the same next step
|
||||
- A later session could begin implementing semantic-first RSA without re-deriving
|
||||
the architecture
|
||||
|
||||
## Not In Scope
|
||||
|
||||
- full implementation of semantic authoring inside `whetstone_DSL`
|
||||
- replacing the current C++ runtime scaffold
|
||||
- immediate automatic projection generation
|
||||
- training any model
|
||||
- wiring RSA into `whetstone_DSL` taskitem generation
|
||||
|
||||
## Case Study Rule
|
||||
|
||||
Use existing RSA C++ types and WhetstoneDSL decision surfaces only as evidence and
|
||||
bootstrap input.
|
||||
|
||||
Do not let any one case study define RSA semantics by itself.
|
||||
|
||||
## Recommended Execution Order
|
||||
|
||||
1. Inventory the existing C++ public surface.
|
||||
2. Define the minimum semantic entity set.
|
||||
3. Map each current C++ field into a semantic or runtime-only bucket.
|
||||
4. Identify missing fields required by version-0 goals.
|
||||
5. Define the projection boundary into C++.
|
||||
6. Record the next implementation step for semantic authoring.
|
||||
|
||||
## Exit Condition
|
||||
|
||||
This sprint is complete when the project no longer needs to answer:
|
||||
|
||||
"What is the first semantic model for RSA, and how does the current C++ scaffold
|
||||
map into it?"
|
||||
|
||||
That question should be settled well enough that implementation can start from the
|
||||
semantic model instead of revisiting architecture.
|
||||
|
||||
## Next Step
|
||||
|
||||
Represent the first three RSA semantic entities as Whetstone-authored forms:
|
||||
|
||||
- `DecisionContract`
|
||||
- `DecisionSurface`
|
||||
- `ProbeDefinition`
|
||||
|
||||
Then map `include/whetstone_rsa/types.h` onto those entities as a temporary C++
|
||||
projection target.
|
||||
351
sprints/SPRINT-003-whetstone-dsl-gate-semantic-bootstrap.md
Normal file
351
sprints/SPRINT-003-whetstone-dsl-gate-semantic-bootstrap.md
Normal file
@@ -0,0 +1,351 @@
|
||||
# Sprint 003: WhetstoneDSL Gate Semantic Bootstrap
|
||||
|
||||
## Status
|
||||
|
||||
Active
|
||||
|
||||
## Intent
|
||||
|
||||
Turn the current RSA architecture and gate inventory into an executable semantic
|
||||
bootstrap focused on the first concrete `whetstone_DSL` case-study gates.
|
||||
|
||||
This sprint exists to close the gap between:
|
||||
|
||||
- generic RSA architecture
|
||||
- generic semantic bootstrap language
|
||||
- actual `whetstone_DSL` decision surfaces
|
||||
|
||||
The output of this sprint should be concrete enough that later work can proceed
|
||||
without re-deriving what the first real gates are or how they map into semantic
|
||||
entities.
|
||||
|
||||
## Why This Sprint Exists
|
||||
|
||||
`SPRINT-002-rsa-semantic-bootstrap.md` established the architectural direction,
|
||||
but it is still too abstract to drive direct implementation.
|
||||
|
||||
We now have a concrete inventory in:
|
||||
|
||||
- `docs/case_studies/whetstone_dsl_gate_inventory.md`
|
||||
|
||||
The next step is to operationalize that inventory into:
|
||||
|
||||
- first concrete case-study gate records
|
||||
- explicit semantic mapping rules
|
||||
- clear enum-vs-structured-output boundaries
|
||||
- acceptance criteria for what counts as a completed bootstrap
|
||||
|
||||
## Scope
|
||||
|
||||
This sprint is still semantic-first.
|
||||
|
||||
Primary outputs:
|
||||
|
||||
- a detailed sprint plan for the first case-study gate set
|
||||
- semantic records for the first concrete `whetstone_DSL` gates
|
||||
- mapping notes from `whetstone_DSL` decision surfaces into RSA entities
|
||||
- explicit skeleton targets for later structured-output gates
|
||||
|
||||
This sprint does **not** require:
|
||||
|
||||
- model training
|
||||
- wiring RSA runtime into `whetstone_DSL`
|
||||
- replacing current `whetstone_DSL` heuristics
|
||||
- projecting full Whetstone-authored AST entities yet
|
||||
|
||||
## Primary Gate Set
|
||||
|
||||
These are the first concrete RSA case-study gates for bootstrap:
|
||||
|
||||
### Immediate RSA gates
|
||||
|
||||
- `verification_type`
|
||||
- `worker_type`
|
||||
- `prereq_op_selector`
|
||||
- `cpp_raising_profile`
|
||||
|
||||
### Skeleton-first semantic decisions
|
||||
|
||||
- `execution_contract_shape`
|
||||
- `workstream_grouping`
|
||||
- `target_file_selection`
|
||||
|
||||
## Gate Definitions To Lock Down
|
||||
|
||||
### 1. `verification_type`
|
||||
|
||||
Source surface:
|
||||
|
||||
- `RegisterArchitectIntakeTools.h`
|
||||
|
||||
Decision form:
|
||||
|
||||
- bounded enum
|
||||
|
||||
Expected outputs:
|
||||
|
||||
- `unit`
|
||||
- `schema`
|
||||
- `docs`
|
||||
- `smoke`
|
||||
|
||||
Contract interpretation:
|
||||
|
||||
- unstructured textual input
|
||||
- bounded discrete output
|
||||
- stable policy
|
||||
- deterministic downstream verification behavior
|
||||
|
||||
### 2. `worker_type`
|
||||
|
||||
Source surface:
|
||||
|
||||
- `TaskitemGeneratorV2.h`
|
||||
|
||||
Decision form:
|
||||
|
||||
- bounded enum
|
||||
|
||||
Expected outputs:
|
||||
|
||||
- `implementer`
|
||||
- `reviewer`
|
||||
- `architect`
|
||||
- `qa`
|
||||
|
||||
Contract interpretation:
|
||||
|
||||
- unstructured or hybrid textual input
|
||||
- bounded discrete output
|
||||
- mixed context sensitivity
|
||||
- deterministic downstream routing behavior
|
||||
|
||||
### 3. `prereq_op_selector`
|
||||
|
||||
Source surface:
|
||||
|
||||
- `TaskitemGeneratorV2.h`
|
||||
|
||||
Decision form:
|
||||
|
||||
- bounded structured output
|
||||
- factorized / multi-label candidate
|
||||
|
||||
Expected atomic outputs:
|
||||
|
||||
- `validate-intake`
|
||||
- `architect-review`
|
||||
- `resolve-dependencies`
|
||||
|
||||
Contract interpretation:
|
||||
|
||||
- hybrid semantic gate
|
||||
- not a good flat multiclass default
|
||||
- primary probe target for factorization analysis
|
||||
|
||||
### 4. `cpp_raising_profile`
|
||||
|
||||
Source surface:
|
||||
|
||||
- `RegisterCppRaisingTools.h`
|
||||
|
||||
Decision form:
|
||||
|
||||
- bounded enum
|
||||
|
||||
Expected outputs:
|
||||
|
||||
- `safe-first`
|
||||
- `perf-first`
|
||||
- `interop-first`
|
||||
|
||||
Contract interpretation:
|
||||
|
||||
- structured symbolic input
|
||||
- bounded discrete output
|
||||
- deterministic downstream execution and artifact generation
|
||||
|
||||
## Skeleton Decisions To Represent But Not Train Yet
|
||||
|
||||
### `execution_contract_shape`
|
||||
|
||||
Why skeleton first:
|
||||
|
||||
- output is structured
|
||||
- current semantics are implicit across many heuristics
|
||||
- not yet stable enough as a direct training target
|
||||
|
||||
Required semantic fields:
|
||||
|
||||
- target files
|
||||
- required tools
|
||||
- acceptance commands
|
||||
- step sequence
|
||||
- cross-project targets
|
||||
- resource locks
|
||||
|
||||
### `workstream_grouping`
|
||||
|
||||
Why skeleton first:
|
||||
|
||||
- current logic mixes lexical grouping with semantic bundling
|
||||
- good future structured-output gate
|
||||
- not yet clean enough for direct benchmark comparisons
|
||||
|
||||
### `target_file_selection`
|
||||
|
||||
Why skeleton first:
|
||||
|
||||
- current logic is bounded but still heavily heuristic
|
||||
- needs clearer contract and evaluation rules before training
|
||||
|
||||
## Deliverables
|
||||
|
||||
### Deliverable A: Case-study gate records
|
||||
|
||||
Update the semantic bootstrap so it explicitly records:
|
||||
|
||||
- gate identity
|
||||
- ontology fields
|
||||
- output labels or slot schema
|
||||
- candidate gate shape
|
||||
- deterministic baseline availability
|
||||
- downstream executor kind
|
||||
|
||||
for the four immediate gates.
|
||||
|
||||
Status:
|
||||
|
||||
- complete in `semantic/rsa_semantic_bootstrap_v0.json`
|
||||
|
||||
### Deliverable B: Mapping rules
|
||||
|
||||
Write down the mapping from current `whetstone_DSL` code surfaces into RSA
|
||||
semantic entities:
|
||||
|
||||
- `TaskitemGeneratorV2`
|
||||
- `TaskitemConfidenceAmbiguity`
|
||||
- `RegisterArchitectIntakeTools`
|
||||
- `RegisterCppRaisingTools`
|
||||
- `RoutingEngine`
|
||||
|
||||
Status:
|
||||
|
||||
- complete in `docs/case_studies/whetstone_dsl_semantic_mapping.md`
|
||||
|
||||
### Deliverable C: Structured-output bootstrap boundary
|
||||
|
||||
Define how `prereq_op_selector` and `execution_contract_shape` are represented
|
||||
without collapsing them into flat labels.
|
||||
|
||||
Status:
|
||||
|
||||
- partially complete
|
||||
- `execution_contract_shape` skeleton captured in
|
||||
`docs/case_studies/whetstone_dsl_execution_contract_shape_skeleton.md`
|
||||
- extraction rules for `prereq_op_selector` and execution-contract child
|
||||
decisions captured in
|
||||
`docs/case_studies/whetstone_dsl_gate_extraction_specs_v0.md`
|
||||
|
||||
### Deliverable D: Artifact-grounded extraction path
|
||||
|
||||
Define which surviving `whetstone_DSL` corpora can support RSA-aligned
|
||||
extraction, and which should remain context or evaluation corpora.
|
||||
|
||||
Status:
|
||||
|
||||
- partially complete
|
||||
- audit captured in `docs/case_studies/whetstone_dsl_run_artifact_audit.md`
|
||||
- first extraction rules captured in
|
||||
`docs/case_studies/whetstone_dsl_gate_extraction_specs_v0.md`
|
||||
|
||||
## Active Next Task
|
||||
|
||||
Turn the first extraction specs into machine-readable extraction contracts for:
|
||||
|
||||
- `prereq_op_selector`
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `required_tool_selection`
|
||||
|
||||
Status:
|
||||
|
||||
- partially complete
|
||||
- `prereq_op_selector` is now represented as structured/factorized in
|
||||
`semantic/rsa_semantic_bootstrap_v0.json`
|
||||
- `execution_contract_shape` skeleton is now defined in
|
||||
`docs/case_studies/whetstone_dsl_execution_contract_shape_skeleton.md`
|
||||
- `target_file_selection` and `workstream_grouping` still need explicit skeleton field sets
|
||||
|
||||
### Deliverable D: Probe-readiness notes
|
||||
|
||||
Record which probes should apply first to each gate:
|
||||
|
||||
- `confidence_threshold_probe`
|
||||
- `factorization_probe`
|
||||
- `schema_stability_probe`
|
||||
- `deterministic_rule_probe`
|
||||
- `context_width_probe`
|
||||
|
||||
Status:
|
||||
|
||||
- partially complete
|
||||
- immediate-gate probe candidates are recorded in the gate inventory and
|
||||
bootstrap notes
|
||||
- still needs a single consolidated probe-readiness artifact
|
||||
|
||||
## Recommended Execution Order
|
||||
|
||||
1. Extend the semantic bootstrap artifact with the first four case-study gates.
|
||||
2. Record mapping notes from each current `whetstone_DSL` implementation surface.
|
||||
3. Represent `prereq_op_selector` as structured/factorized rather than flat.
|
||||
4. Represent skeleton-only decisions for execution-contract shaping and grouping.
|
||||
5. Record which current gates should remain deterministic.
|
||||
6. Leave runtime wiring and training for later sprints.
|
||||
|
||||
Current progress:
|
||||
|
||||
- steps 1, 2, and 3 are complete
|
||||
- `execution_contract_shape` is complete enough for first-pass skeleton treatment
|
||||
- `target_file_selection` is now the active task
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- the sprint names a fixed first gate set rather than speaking about gates in the abstract
|
||||
- the semantic bootstrap artifact contains explicit records for:
|
||||
- `verification_type`
|
||||
- `worker_type`
|
||||
- `prereq_op_selector`
|
||||
- `cpp_raising_profile`
|
||||
- each gate is classified as enum, structured, or factorized
|
||||
- at least three skeleton-first decision entities are identified and justified
|
||||
- deterministic-baseline gates are explicitly separated from RSA gates
|
||||
- the sprint is actionable without reopening architecture debate
|
||||
|
||||
## Not In Scope
|
||||
|
||||
- training a new specialist
|
||||
- modifying `whetstone_DSL` runtime code
|
||||
- adding new RSA C++ runtime classes unless needed for semantic alignment
|
||||
- making Whetstone AST the full live authoring source this sprint
|
||||
|
||||
## Exit Condition
|
||||
|
||||
This sprint is complete when a later session can answer all of the following
|
||||
without ambiguity:
|
||||
|
||||
- what are the first real `whetstone_DSL` gates in RSA terms
|
||||
- which are enum versus structured-output
|
||||
- which should remain deterministic
|
||||
- which structured decisions need skeleton treatment before training
|
||||
- what the first semantic bootstrap records look like
|
||||
|
||||
## Immediate First Task
|
||||
|
||||
Extend `semantic/rsa_semantic_bootstrap_v0.json` with a case-study gate section
|
||||
covering:
|
||||
|
||||
- `verification_type`
|
||||
- `worker_type`
|
||||
- `prereq_op_selector`
|
||||
- `cpp_raising_profile`
|
||||
Reference in New Issue
Block a user