WIP: save progress 2026-04-12
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill Holcombe
@@ -0,0 +1,327 @@
|
||||
# WhetstoneDSL Execution Contract Shape Skeleton
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the first semantic skeleton for the structured decision
|
||||
surface currently spread across `whetstone_DSL` execution-contract heuristics.
|
||||
|
||||
The intent is not to finalize training data or runtime wiring yet.
|
||||
|
||||
The intent is to make the contract explicit enough that:
|
||||
|
||||
- we know what the decision is
|
||||
- we know which parts are sub-decisions
|
||||
- we know which parts can stay deterministic
|
||||
- we can later extract training data without flattening the structure incorrectly
|
||||
|
||||
## Why This Needs A Skeleton First
|
||||
|
||||
The current `whetstone_DSL` execution contract is assembled from heuristics in
|
||||
`RegisterArchitectIntakeTools.h`.
|
||||
|
||||
That logic currently bundles several distinct choices together:
|
||||
|
||||
- target files
|
||||
- required tools
|
||||
- acceptance commands
|
||||
- verification type
|
||||
- step sequence
|
||||
- cross-project targets
|
||||
- resource locks
|
||||
|
||||
If we treat that whole bundle as one opaque label, the gate becomes unusable for
|
||||
RSA.
|
||||
|
||||
If we flatten it into arbitrary string labels, we lose the structure and make the
|
||||
training problem worse.
|
||||
|
||||
So the correct intermediate step is a semantic skeleton.
|
||||
|
||||
## Entity Name
|
||||
|
||||
Recommended semantic entity:
|
||||
|
||||
- `ExecutionContractShapeDecision`
|
||||
|
||||
This is a skeleton decision, not yet a finalized RSA gate package.
|
||||
|
||||
## Contract Boundary
|
||||
|
||||
`ExecutionContractShapeDecision` answers:
|
||||
|
||||
- what execution-shape packet should be attached to a taskitem before downstream
|
||||
deterministic execution or validation
|
||||
|
||||
It does **not** answer:
|
||||
|
||||
- whether the task is globally good or bad
|
||||
- how the task should be routed to human/LLM/SLM layers
|
||||
- what final generated code should look like
|
||||
|
||||
## Field Set
|
||||
|
||||
### Core identity fields
|
||||
|
||||
- `decision_id`
|
||||
- `task_id`
|
||||
- `source_case_study`
|
||||
- `contract_version`
|
||||
|
||||
These fields are bookkeeping only.
|
||||
|
||||
### Input context fields
|
||||
|
||||
These are the fields allowed to influence the decision.
|
||||
|
||||
- `task_title`
|
||||
- `task_intent`
|
||||
- `requirement_ids`
|
||||
- `requirement_kinds`
|
||||
- `ambiguity_signals`
|
||||
- `conflict_signals`
|
||||
- `workspace_root`
|
||||
- `project_root`
|
||||
- `cross_project_intent`
|
||||
- `known_target_hints`
|
||||
|
||||
Rule:
|
||||
|
||||
- do not include downstream execution outcomes here
|
||||
- do not include later compile/test results here
|
||||
|
||||
## Structured output fields
|
||||
|
||||
### 1. `target_files`
|
||||
|
||||
Meaning:
|
||||
|
||||
- which files this task most directly acts on
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Notes:
|
||||
|
||||
- bounded structured selection
|
||||
- likely future child decision:
|
||||
`TargetFileSelectionDecision`
|
||||
|
||||
### 2. `required_tools`
|
||||
|
||||
Meaning:
|
||||
|
||||
- which named tools or MCP operations are required by the contract
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Examples:
|
||||
|
||||
- `whetstone_generate_taskitems`
|
||||
- `whetstone_validate_taskitem`
|
||||
|
||||
Notes:
|
||||
|
||||
- often partly deterministic from explicit references
|
||||
- could remain deterministic longer than other fields
|
||||
|
||||
### 3. `acceptance_commands`
|
||||
|
||||
Meaning:
|
||||
|
||||
- which concrete validation commands should be run for this task
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Examples:
|
||||
|
||||
- `cargo test`
|
||||
- `pytest`
|
||||
- `ctest`
|
||||
|
||||
Notes:
|
||||
|
||||
- bounded structured choice
|
||||
- highly sensitive to target-file inference
|
||||
|
||||
### 4. `verification_type`
|
||||
|
||||
Meaning:
|
||||
|
||||
- the dominant verification mode implied by the task
|
||||
|
||||
Type:
|
||||
|
||||
- enum
|
||||
|
||||
Allowed values:
|
||||
|
||||
- `unit`
|
||||
- `schema`
|
||||
- `docs`
|
||||
- `smoke`
|
||||
|
||||
Notes:
|
||||
|
||||
- this is the clearest immediate child gate
|
||||
- should remain explicit instead of being buried inside the larger contract
|
||||
|
||||
### 5. `step_sequence`
|
||||
|
||||
Meaning:
|
||||
|
||||
- the bounded sequence of execution phases expected for the task
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Examples:
|
||||
|
||||
- `inspect_scope`
|
||||
- `implement_changes`
|
||||
- `apply_changes`
|
||||
- `run_unit_verification`
|
||||
- `run_smoke_verification`
|
||||
- `review_doc_alignment`
|
||||
|
||||
Notes:
|
||||
|
||||
- structured output
|
||||
- good later candidate for a bounded sequence or hierarchical gate
|
||||
|
||||
### 6. `cross_project_targets`
|
||||
|
||||
Meaning:
|
||||
|
||||
- external project artifacts the task is explicitly allowed or expected to touch
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Notes:
|
||||
|
||||
- should be empty by default
|
||||
- this field must not be inferred casually from vague mentions
|
||||
|
||||
### 7. `resource_locks`
|
||||
|
||||
Meaning:
|
||||
|
||||
- coarse-grained lock names needed to avoid conflicting execution
|
||||
|
||||
Type:
|
||||
|
||||
- `array<string>`
|
||||
|
||||
Notes:
|
||||
|
||||
- likely deterministic in early versions
|
||||
- still part of the execution-contract shape because it affects runtime behavior
|
||||
|
||||
## Child Decision Decomposition
|
||||
|
||||
`ExecutionContractShapeDecision` should not remain monolithic forever.
|
||||
|
||||
The expected decomposition is:
|
||||
|
||||
- `VerificationTypeGate`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `AcceptanceCommandSelectionDecision`
|
||||
- `StepSequenceDecision`
|
||||
- `CrossProjectTargetDecision`
|
||||
- `ResourceLockPolicy`
|
||||
|
||||
This parent skeleton exists so those child decisions have a coherent semantic
|
||||
container.
|
||||
|
||||
## Deterministic Baseline Split
|
||||
|
||||
Not every field in the execution contract should become a specialist.
|
||||
|
||||
### Likely deterministic for longer
|
||||
|
||||
- explicit tool references
|
||||
- explicit target file mentions
|
||||
- resource locks when directly tied to file or subsystem identity
|
||||
|
||||
### Stronger specialist candidates
|
||||
|
||||
- `verification_type`
|
||||
- target-file selection when multiple plausible files exist
|
||||
- acceptance-command selection when language/build context is mixed
|
||||
- step-sequence selection
|
||||
|
||||
## Invalid States
|
||||
|
||||
The skeleton should define invalid or suspicious combinations.
|
||||
|
||||
Examples:
|
||||
|
||||
- `verification_type = docs` with only code target files and no doc targets
|
||||
- `acceptance_commands = ["cargo test"]` with only Python target files
|
||||
- non-empty `cross_project_targets` without cross-project intent
|
||||
- empty `target_files` plus non-empty language-specific acceptance commands
|
||||
|
||||
These invalid states are important because they give the later gate contract a
|
||||
real constraint surface instead of a bag of fields.
|
||||
|
||||
## Training Extraction Guidance
|
||||
|
||||
If historic JSON/taskitem runs are mined later, extraction should happen per
|
||||
child decision, not by treating the full execution contract as one class label.
|
||||
|
||||
Safe pattern:
|
||||
|
||||
1. reconstruct task-local context
|
||||
2. extract `verification_type`
|
||||
3. extract `target_files`
|
||||
4. extract `acceptance_commands`
|
||||
5. extract `step_sequence`
|
||||
6. preserve provenance for each extracted field
|
||||
|
||||
Unsafe pattern:
|
||||
|
||||
- one whole JSON contract becomes one giant class label
|
||||
|
||||
## Minimal Machine-Readable Shape
|
||||
|
||||
The eventual machine-readable skeleton should support at least:
|
||||
|
||||
```text
|
||||
ExecutionContractShapeDecision
|
||||
input_context:
|
||||
task_title
|
||||
task_intent
|
||||
ambiguity_signals
|
||||
conflict_signals
|
||||
workspace_root
|
||||
project_root
|
||||
cross_project_intent
|
||||
known_target_hints
|
||||
output:
|
||||
target_files[]
|
||||
required_tools[]
|
||||
acceptance_commands[]
|
||||
verification_type
|
||||
step_sequence[]
|
||||
cross_project_targets[]
|
||||
resource_locks[]
|
||||
constraints:
|
||||
invalid_combinations[]
|
||||
```
|
||||
|
||||
## Immediate Next Step
|
||||
|
||||
Define equivalent skeletons for:
|
||||
|
||||
- `TargetFileSelectionDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
|
||||
Then update the semantic bootstrap artifact so `ExecutionContractShapeDecision`
|
||||
exists as an explicit skeleton entity rather than only as prose.
|
||||
456
docs/case_studies/whetstone_dsl_gate_extraction_specs_v0.md
Normal file
456
docs/case_studies/whetstone_dsl_gate_extraction_specs_v0.md
Normal file
@@ -0,0 +1,456 @@
|
||||
# WhetstoneDSL Gate Extraction Specs v0
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines first-pass extraction specs for turning surviving
|
||||
`whetstone_DSL` artifacts into RSA-aligned training rows.
|
||||
|
||||
These specs are gate-native, not taskitem-native.
|
||||
Each spec describes:
|
||||
|
||||
- source artifacts
|
||||
- allowed context
|
||||
- forbidden leaked context
|
||||
- label shape
|
||||
- provenance expectation
|
||||
- rejection conditions
|
||||
|
||||
The immediate goal is to support the first high-confidence extraction surfaces
|
||||
identified in the run-artifact audit.
|
||||
|
||||
## Source Priority
|
||||
|
||||
Primary extraction corpus:
|
||||
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs/*/01_intake.json`
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs/*/02_generate_taskitems.json`
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs/*/04_validate_taskitem.json`
|
||||
|
||||
Supporting context reservoirs:
|
||||
|
||||
- `CLionProjects/whetstone_DSL/datasets/example_run_specs/*`
|
||||
- `CLionProjects/whetstone_DSL/datasets/project_benchmarks/*`
|
||||
|
||||
These supporting corpora may supply spec context, but not labels.
|
||||
|
||||
## Global Extraction Rules
|
||||
|
||||
### Row shape
|
||||
|
||||
Every extracted row should contain:
|
||||
|
||||
- `gate_id`
|
||||
- `decision_context`
|
||||
- `decision_label`
|
||||
- `decision_provenance`
|
||||
- `source_artifacts`
|
||||
- `rejection_flags`
|
||||
|
||||
### Provenance values
|
||||
|
||||
Use one of:
|
||||
|
||||
- `direct_explicit`
|
||||
- `direct_bundled`
|
||||
- `reconstructed`
|
||||
- `untrusted`
|
||||
|
||||
### Global rejection rules
|
||||
|
||||
Reject a candidate row when:
|
||||
|
||||
- the candidate label depends on downstream execution results
|
||||
- the candidate label is only inferable from post-hoc failure or success
|
||||
- the candidate label shape does not match the RSA gate contract
|
||||
- required pre-decision context is missing
|
||||
- the artifact mixes unrelated tasks without isolatable per-task context
|
||||
|
||||
## Spec: `prereq_op_selector`
|
||||
|
||||
### Status
|
||||
|
||||
First extraction target.
|
||||
This is the strongest current gate candidate in surviving taskitem runs.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- optional: `04_validate_taskitem.json`
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- `01_intake.json`
|
||||
- normalized requirements
|
||||
- parsed spec sections
|
||||
- ambiguity signals
|
||||
- conflict signals
|
||||
- `02_generate_taskitems.json`
|
||||
- `title`
|
||||
- `milestoneId`
|
||||
- `dependencyTaskIds`
|
||||
- `confidence`
|
||||
- `ambiguityCount`
|
||||
- `reasons`
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].prerequisiteOps`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- dependency ids
|
||||
- local ambiguity and conflict signals
|
||||
- intake-level normalized requirements
|
||||
- milestone grouping if already known at task generation time
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- acceptance commands
|
||||
- target files
|
||||
- required tools
|
||||
- downstream execution results
|
||||
- validation pass/fail outcome
|
||||
- generated code
|
||||
|
||||
### Label shape
|
||||
|
||||
Do not train as a flat multiclass label.
|
||||
|
||||
Use a factorized structured label:
|
||||
|
||||
- `needs_validate_intake`: boolean
|
||||
- `needs_architect_review`: boolean
|
||||
- `needs_resolve_dependencies`: boolean
|
||||
|
||||
Allowed source values:
|
||||
|
||||
- `validate-intake`
|
||||
- `architect-review`
|
||||
- `resolve-dependencies`
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
Reason:
|
||||
|
||||
- the list exists explicitly, but was emitted as part of a larger taskitem
|
||||
bundle rather than an isolated gate invocation
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- `prerequisiteOps` contains values outside the accepted slot vocabulary
|
||||
- the task cannot be isolated from sibling tasks in the artifact
|
||||
- confidence or dependency signals are missing and the context becomes too thin
|
||||
- the list appears to be post-processed rather than source-emitted
|
||||
|
||||
## Spec: `target_file_selection`
|
||||
|
||||
### Status
|
||||
|
||||
High-confidence child decision under `execution_contract_shape`.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- optional: run-spec markdown from `datasets/example_run_specs`
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- normalized requirements
|
||||
- parsed spec
|
||||
- task title
|
||||
- milestone id
|
||||
- dependency ids
|
||||
- local reasons
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].executionContract.targetFiles`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- relevant normalized requirements
|
||||
- milestone grouping
|
||||
- dependency ids
|
||||
- optionally scenario family metadata from run-spec corpus
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- acceptance commands
|
||||
- required tools
|
||||
- execution specificity score
|
||||
- validation diagnostics
|
||||
- downstream worker assignment
|
||||
- final implementation output
|
||||
|
||||
### Label shape
|
||||
|
||||
Structured bounded list of repository paths.
|
||||
|
||||
Recommended first-pass representation:
|
||||
|
||||
- `target_files`: ordered list of paths
|
||||
|
||||
Potential later factorization:
|
||||
|
||||
- `primary_surface`
|
||||
- `secondary_surfaces`
|
||||
- `script_surface_present`
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- target files include unrelated repository surfaces not justified by task text
|
||||
- path list is empty
|
||||
- path list appears copied across all sibling tasks without local task variation
|
||||
- path list depends on downstream implementation or validation feedback
|
||||
|
||||
## Spec: `acceptance_command_selection`
|
||||
|
||||
### Status
|
||||
|
||||
Promising child decision under `execution_contract_shape`, but more leakage risk
|
||||
than target-file selection.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- `04_validate_taskitem.json` for audit only, not label derivation
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- normalized requirements
|
||||
- task title
|
||||
- target files
|
||||
- milestone id
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].executionContract.acceptanceCommands`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- target files
|
||||
- normalized requirements that imply verification surface
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- validation outcome
|
||||
- execution success or failure
|
||||
- downstream generated code
|
||||
- specificity score used as a target proxy
|
||||
|
||||
### Label shape
|
||||
|
||||
Structured bounded list of command identifiers.
|
||||
|
||||
Recommended first-pass representation:
|
||||
|
||||
- `acceptance_commands`: ordered list of commands
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- commands are obviously copied template defaults unrelated to task surface
|
||||
- commands require information only known after code edits
|
||||
- commands are inconsistent with target files or implied verification surface
|
||||
|
||||
## Spec: `required_tool_selection`
|
||||
|
||||
### Status
|
||||
|
||||
Useful, but higher risk of mixing model choices with framework templates.
|
||||
|
||||
### Source artifacts
|
||||
|
||||
- `01_intake.json`
|
||||
- `02_generate_taskitems.json`
|
||||
- `04_validate_taskitem.json` for audit only
|
||||
|
||||
### Source fields
|
||||
|
||||
Context candidates:
|
||||
|
||||
- normalized requirements
|
||||
- task title
|
||||
- target files
|
||||
- prerequisite ops
|
||||
|
||||
Label source:
|
||||
|
||||
- `02_generate_taskitems.json.tasks[*].executionContract.requiredTools`
|
||||
|
||||
### Allowed context
|
||||
|
||||
- task title
|
||||
- local task intent
|
||||
- prerequisite ops
|
||||
- target files
|
||||
- normalized requirements
|
||||
|
||||
### Forbidden leaked context
|
||||
|
||||
- validation-recommended tools as label source
|
||||
- downstream execution traces
|
||||
- final worker selection
|
||||
|
||||
### Label shape
|
||||
|
||||
Structured bounded list of tool ids.
|
||||
|
||||
Recommended first-pass representation:
|
||||
|
||||
- `required_tools`: ordered list of tool names
|
||||
|
||||
### Provenance
|
||||
|
||||
Default provenance:
|
||||
|
||||
- `direct_bundled`
|
||||
|
||||
### Rejection rules
|
||||
|
||||
Reject row if:
|
||||
|
||||
- the tool list is clearly framework boilerplate with no task-local variation
|
||||
- the list includes post-hoc validator recommendations instead of generation-time
|
||||
selections
|
||||
- tool names cannot be mapped to a stable tool registry
|
||||
|
||||
## Spec: `verification_type`
|
||||
|
||||
### Status
|
||||
|
||||
Not yet a direct label source.
|
||||
Treat as reconstruction target only after child execution-contract fields are
|
||||
stable.
|
||||
|
||||
### Candidate reconstruction inputs
|
||||
|
||||
- target files
|
||||
- acceptance commands
|
||||
- task title
|
||||
- normalized requirements
|
||||
|
||||
### Current label shape
|
||||
|
||||
Bounded enum:
|
||||
|
||||
- `unit`
|
||||
- `schema`
|
||||
- `docs`
|
||||
- `smoke`
|
||||
|
||||
### Current provenance
|
||||
|
||||
- `reconstructed`
|
||||
|
||||
### Blocking issue
|
||||
|
||||
No audited artifact yet shows `verification_type` as a clean standalone field.
|
||||
|
||||
## Spec: `worker_type`
|
||||
|
||||
### Status
|
||||
|
||||
Second-pass extraction target.
|
||||
|
||||
### Current evidence
|
||||
|
||||
- explicit source-code surface exists in:
|
||||
- `editor/src/RoutingRules.h`
|
||||
- `editor/src/WorkflowOrchestrator.h`
|
||||
- `editor/src/TaskitemPoolPersistence.h`
|
||||
- `editor/src/headless_rpc/DispatchPart6.h`
|
||||
|
||||
### Current provenance
|
||||
|
||||
- `reconstructed`
|
||||
|
||||
### Blocking issue
|
||||
|
||||
No clean serialized label found yet in the audited taskitem-generation artifacts.
|
||||
|
||||
### Next search target
|
||||
|
||||
Routing and orchestration run families, not generic taskitem-generation runs.
|
||||
|
||||
## Spec: `cpp_raising_profile`
|
||||
|
||||
### Status
|
||||
|
||||
Second-pass extraction target.
|
||||
|
||||
### Current evidence
|
||||
|
||||
- explicit source-code surface exists in:
|
||||
- `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
- `editor/src/cpp_ir/CppErrorModelMapping.h`
|
||||
- `editor/src/cpp_ir/CppBuildArtifactGenerator.h`
|
||||
- `editor/src/cpp_ir/CppAsyncMappingStrategy.h`
|
||||
- `editor/src/cpp_ir/CppTemplateRaisingPolicy.h`
|
||||
|
||||
### Current label shape
|
||||
|
||||
Bounded enum:
|
||||
|
||||
- `safe-first`
|
||||
- `perf-first`
|
||||
- `interop-first`
|
||||
|
||||
### Current provenance
|
||||
|
||||
- `untrusted`
|
||||
|
||||
### Blocking issue
|
||||
|
||||
No clean serialized profile label found yet in the audited generic taskitem
|
||||
corpora.
|
||||
|
||||
### Next search target
|
||||
|
||||
C++-raising-specific run artifacts and MCP execution traces.
|
||||
|
||||
## Immediate Implementation Order
|
||||
|
||||
1. `prereq_op_selector`
|
||||
2. `target_file_selection`
|
||||
3. `acceptance_command_selection`
|
||||
4. `required_tool_selection`
|
||||
5. `verification_type` reconstruction rule
|
||||
6. `worker_type` artifact search
|
||||
7. `cpp_raising_profile` artifact search
|
||||
374
docs/case_studies/whetstone_dsl_gate_inventory.md
Normal file
374
docs/case_studies/whetstone_dsl_gate_inventory.md
Normal file
@@ -0,0 +1,374 @@
|
||||
# WhetstoneDSL Gate Inventory
|
||||
|
||||
## Purpose
|
||||
|
||||
This document grounds `whetstone_RSA` Phase 1 in the actual decision surfaces
|
||||
currently present in `whetstone_DSL`.
|
||||
|
||||
It is intentionally narrower than a general architecture document.
|
||||
|
||||
The goal is to answer:
|
||||
|
||||
- what bounded decisions already exist in the `whetstone_DSL` taskitem and
|
||||
AST-codegen pipeline
|
||||
- which of those are already deterministic
|
||||
- which are immediate RSA case-study candidates
|
||||
- which should become semantic skeletons first before any specialist training
|
||||
|
||||
This inventory is based on the current code surfaces in:
|
||||
|
||||
- `editor/src/TaskitemGeneratorV2.h`
|
||||
- `editor/src/TaskitemConfidenceAmbiguity.h`
|
||||
- `editor/src/RoutingEngine.h`
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- `editor/src/ast/ProjectionGenerator.h`
|
||||
- `editor/src/CrossLanguageProjector.h`
|
||||
- `editor/src/ConstrainedProjectionGate.h`
|
||||
- `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
|
||||
## Architectural Observation
|
||||
|
||||
The strongest current RSA surface in `whetstone_DSL` is not the final
|
||||
language-emission step.
|
||||
|
||||
Once the AST and target language are fixed, most current code generation is
|
||||
already deterministic:
|
||||
|
||||
- target generator dispatch is deterministic
|
||||
- AST concept dispatch is deterministic
|
||||
- environment gating is deterministic
|
||||
- C++ raising profile validation is deterministic
|
||||
|
||||
That means the immediate RSA case-study value is higher in:
|
||||
|
||||
- taskitem generation
|
||||
- execution-contract shaping
|
||||
- routing
|
||||
- pre-codegen bounded profile decisions
|
||||
|
||||
than in raw syntax emission.
|
||||
|
||||
## Current Decision Surfaces
|
||||
|
||||
### A. Taskitem decomposition and planning
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- milestone grouping
|
||||
- workstream grouping
|
||||
- workstream title shape
|
||||
- workstream intent aggregation
|
||||
- uncertainty scoring
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- largely heuristic and deterministic in `ScopeMilestoneDecomposer`
|
||||
- not yet expressed as explicit RSA gate contracts
|
||||
|
||||
Assessment:
|
||||
|
||||
- milestone grouping is currently deterministic and should remain so until a real
|
||||
ambiguity signal shows otherwise
|
||||
- uncertainty scoring is currently formula-like and should remain deterministic
|
||||
- title shaping is mixed: parts are deterministic formatting, but some grouping
|
||||
and summary choices are latent semantic decisions
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- weak immediate training target
|
||||
- good semantic-skeleton target
|
||||
|
||||
Suggested skeleton entities:
|
||||
|
||||
- `WorkstreamGroupingDecision`
|
||||
- `MilestoneGroupingDecision`
|
||||
- `WorkstreamSummaryPolicy`
|
||||
|
||||
### B. Taskitem worker-type assignment
|
||||
|
||||
Observed decision:
|
||||
|
||||
- `implementer | reviewer | architect | qa`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- keyword heuristic in `TaskitemGeneratorV2::inferWorkerType`
|
||||
- file comment explicitly says this is a placeholder to be replaced by a
|
||||
`worker_type` specialist
|
||||
|
||||
Assessment:
|
||||
|
||||
- clear bounded enum
|
||||
- direct downstream deterministic meaning
|
||||
- already known from specialist experiments
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- immediate case-study gate
|
||||
|
||||
Recommended gate class:
|
||||
|
||||
- bounded multiclass gate
|
||||
|
||||
Likely failure risks:
|
||||
|
||||
- label-space mismatch
|
||||
- missing context gate
|
||||
|
||||
### C. Taskitem prerequisite-op assignment
|
||||
|
||||
Observed decision:
|
||||
|
||||
- `validate-intake`
|
||||
- `architect-review`
|
||||
- `resolve-dependencies`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- deterministic default plus uncertainty-based add-on in
|
||||
`TaskitemGeneratorV2::inferPrerequisiteOps`
|
||||
- comments explicitly note this is a placeholder for a future
|
||||
`prereq_op_selector` specialist
|
||||
|
||||
Assessment:
|
||||
|
||||
- this is not purely one flat label
|
||||
- it is closer to a bounded structured output or factorized multi-label gate
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- immediate case-study gate
|
||||
- especially important for factorization probes
|
||||
|
||||
Recommended gate shape:
|
||||
|
||||
- factorized or multi-label, not naive flat multiclass
|
||||
|
||||
Likely failure risks:
|
||||
|
||||
- factorizable gate
|
||||
- label-space mismatch
|
||||
|
||||
### D. Confidence / ambiguity / escalation
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- confidence score `0..100`
|
||||
- ambiguity count
|
||||
- escalate `true|false`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- formulaic in `TaskitemConfidenceAmbiguity`
|
||||
|
||||
Assessment:
|
||||
|
||||
- this is the strongest current example of a deterministic-disguised-as-ML gate
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- important as negative evidence
|
||||
- should stay deterministic unless a later richer policy surface emerges
|
||||
|
||||
Recommended disposition:
|
||||
|
||||
- deterministic layer, not RSA
|
||||
|
||||
### E. Routing decision
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- worker type:
|
||||
`deterministic | template | slm | llm | human`
|
||||
- context width:
|
||||
`local | file | project | cross-project`
|
||||
- review required
|
||||
- agent role:
|
||||
`linter | refactor | generator`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- deterministic rule cascade in `RoutingEngine`
|
||||
- explicit override path plus heuristic fallbacks
|
||||
|
||||
Assessment:
|
||||
|
||||
- routing is already modeled as a bounded decision surface
|
||||
- the current engine is a rule baseline
|
||||
- this is a strong later RSA case study, but not the first one
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- important second-wave case study
|
||||
- especially useful for comparing deterministic baseline versus RSA gate quality
|
||||
|
||||
Recommended disposition:
|
||||
|
||||
- keep current deterministic routing as baseline
|
||||
- use it later as benchmark data for RSA substitution or hybrid routing
|
||||
|
||||
### F. Execution-contract shaping
|
||||
|
||||
Observed decisions in `RegisterArchitectIntakeTools`:
|
||||
|
||||
- target file inference
|
||||
- required tool inference
|
||||
- acceptance-command inference
|
||||
- cross-project target inference
|
||||
- verification type inference:
|
||||
`unit | schema | docs | smoke`
|
||||
- step-id inference:
|
||||
`inspect_scope`, `implement_changes`, `apply_changes`,
|
||||
`run_unit_verification`, `run_smoke_verification`, etc.
|
||||
- resource-lock inference
|
||||
|
||||
Assessment by subtype:
|
||||
|
||||
- target-file inference:
|
||||
mixed, partly lexical and partly semantic
|
||||
- required-tool inference:
|
||||
bounded and meaningful
|
||||
- verification type:
|
||||
strong existing specialist case
|
||||
- step-id inference:
|
||||
bounded structured sequence; good future structured-output RSA case
|
||||
- resource locks:
|
||||
likely deterministic unless concurrency semantics become richer
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- `verification_type` is immediate
|
||||
- execution-contract shaping is a strong structured-output frontier
|
||||
|
||||
Recommended split:
|
||||
|
||||
- immediate RSA gates:
|
||||
- `verification_type`
|
||||
- semantic skeleton first:
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `AcceptanceCommandSelectionDecision`
|
||||
- `StepSequenceDecision`
|
||||
|
||||
### G. AST projection / code generation
|
||||
|
||||
Observed decisions:
|
||||
|
||||
- target language selection
|
||||
- AST node dispatch to language-specific visitor
|
||||
- annotation-preserving cross-language projection
|
||||
- environment constraint gating
|
||||
- C++ raising profile:
|
||||
`safe-first | perf-first | interop-first`
|
||||
|
||||
Current implementation state:
|
||||
|
||||
- almost entirely deterministic once target language and profile are provided
|
||||
|
||||
Assessment:
|
||||
|
||||
- generator dispatch itself is not the best current RSA surface
|
||||
- the richer opportunity is in bounded pre-codegen policy selection, not in
|
||||
token-level syntax generation
|
||||
|
||||
RSA relevance:
|
||||
|
||||
- immediate gate candidate:
|
||||
`cpp_raising_profile`
|
||||
- skeleton-first targets:
|
||||
- `ProjectionProfileDecision`
|
||||
- `ErrorModelPolicyDecision`
|
||||
- `OwnershipMappingProfileDecision`
|
||||
- `AsyncMappingPolicyDecision`
|
||||
|
||||
## Inventory Table
|
||||
|
||||
| Decision surface | Current form | Best current layer |
|
||||
| --- | --- | --- |
|
||||
| `worker_type` | bounded enum | RSA immediate |
|
||||
| `prereq_op_selector` | bounded structured / factorized | RSA immediate |
|
||||
| `verification_type` | bounded enum | RSA immediate |
|
||||
| `confidence_tier` / `escalate` | formula-like | deterministic |
|
||||
| milestone grouping | heuristic grouping | semantic skeleton first |
|
||||
| workstream summary/title | mixed heuristic | semantic skeleton first |
|
||||
| routing layer selection | bounded enum with strong rules | deterministic baseline, later RSA |
|
||||
| target-file inference | mixed bounded selection | semantic skeleton first |
|
||||
| acceptance-command inference | bounded selection | semantic skeleton first |
|
||||
| step-sequence inference | bounded structured sequence | semantic skeleton first |
|
||||
| resource locks | bounded selection | deterministic |
|
||||
| generator dispatch by language | deterministic | deterministic |
|
||||
| C++ raising profile | bounded enum | RSA immediate or near-immediate |
|
||||
|
||||
## Recommended Implementation Order
|
||||
|
||||
### 1. Immediate RSA gate cases
|
||||
|
||||
Use the already-observed bounded gates first:
|
||||
|
||||
- `verification_type`
|
||||
- `worker_type`
|
||||
- `prereq_op_selector`
|
||||
- `cpp_raising_profile`
|
||||
|
||||
These are the best first fit because they already have:
|
||||
|
||||
- small output vocabularies
|
||||
- deterministic downstream meaning
|
||||
- clear failure consequences
|
||||
- real code paths in `whetstone_DSL`
|
||||
|
||||
### 2. Skeleton before specialist
|
||||
|
||||
Before training more specialists, define explicit semantic skeletons for:
|
||||
|
||||
- workstream grouping
|
||||
- execution contract shaping
|
||||
- target-file selection
|
||||
- acceptance-command selection
|
||||
- step-sequence selection
|
||||
|
||||
These are not yet clean enough as training targets because the semantic contract
|
||||
is still implicit in heuristics.
|
||||
|
||||
### 3. Second-wave RSA cases
|
||||
|
||||
After the skeleton pass:
|
||||
|
||||
- routing layer selection
|
||||
- routing context-width selection
|
||||
- richer codegen profile selection
|
||||
- backend-policy decisions in IR raising
|
||||
|
||||
## Implication For Sprint 002
|
||||
|
||||
Sprint 002 should not jump straight from the current C++ RSA scaffold into model
|
||||
training.
|
||||
|
||||
It should first ensure the semantic model can express at least these categories:
|
||||
|
||||
- bounded enum gate
|
||||
- factorized / structured-output gate
|
||||
- deterministic baseline attached to a candidate gate
|
||||
- downstream deterministic executor contract
|
||||
- failure diagnosis for factorization and schema drift
|
||||
|
||||
That semantic coverage is required to represent the real `whetstone_DSL`
|
||||
surfaces above.
|
||||
|
||||
## Concrete Next Step
|
||||
|
||||
Define Whetstone-authored semantic entities for the first four case-study gates:
|
||||
|
||||
- `VerificationTypeGate`
|
||||
- `WorkerTypeGate`
|
||||
- `PrereqOpGate`
|
||||
- `CppRaisingProfileGate`
|
||||
|
||||
and define skeleton semantic entities for:
|
||||
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
|
||||
That would let `whetstone_RSA` move from generic architecture language into a
|
||||
usable case-study-grounded semantic bootstrap.
|
||||
348
docs/case_studies/whetstone_dsl_run_artifact_audit.md
Normal file
348
docs/case_studies/whetstone_dsl_run_artifact_audit.md
Normal file
@@ -0,0 +1,348 @@
|
||||
# WhetstoneDSL Run Artifact Audit
|
||||
|
||||
## Purpose
|
||||
|
||||
This note audits surviving `whetstone_DSL` run artifacts for RSA gate extraction.
|
||||
The goal is not to treat every surviving corpus as direct training data. The goal
|
||||
is to identify which artifacts preserve explicit decision outputs, which preserve
|
||||
only surrounding context, and which are better treated as evaluation traces.
|
||||
|
||||
## Audit Targets
|
||||
|
||||
Primary paths inspected:
|
||||
|
||||
- `CLionProjects/whetstone_DSL/logs/taskitem_runs`
|
||||
- `CLionProjects/whetstone_DSL/datasets/example_run_specs`
|
||||
- `CLionProjects/whetstone_DSL/datasets/project_benchmarks`
|
||||
|
||||
Representative files inspected:
|
||||
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/01_intake.json`
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems.json`
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/02_generate_taskitems_raw.ndjson.json`
|
||||
- `logs/taskitem_runs/01a_fallback_intake_spec_20260226_162113/04_validate_taskitem.json`
|
||||
- `logs/taskitem_runs/project_benchmark_matrix_20260225_192610/cp_001_python/ab_stdout.log`
|
||||
- `logs/taskitem_runs/ast_once_vs_direct_20260225_193037/cp_001/base_payload.json`
|
||||
- `datasets/example_run_specs/index.jsonl`
|
||||
- `datasets/example_run_specs/runspec_py_cli_todo_wrong_tool_selection_01.md`
|
||||
- `datasets/project_benchmarks/common_projects_100.jsonl`
|
||||
|
||||
## Source Classes
|
||||
|
||||
### 1. Structured taskitem runs
|
||||
|
||||
Best current extraction source:
|
||||
|
||||
- `logs/taskitem_runs/*/01_intake.json`
|
||||
- `logs/taskitem_runs/*/02_generate_taskitems.json`
|
||||
- `logs/taskitem_runs/*/04_validate_taskitem.json`
|
||||
|
||||
What these preserve:
|
||||
|
||||
- pre-decision intake context
|
||||
- generated taskitems
|
||||
- explicit task confidence and escalation signals
|
||||
- explicit prerequisite operations
|
||||
- explicit execution-contract fields
|
||||
- validation and specificity feedback
|
||||
|
||||
Observed explicit fields in `02_generate_taskitems.json`:
|
||||
|
||||
- `confidence`
|
||||
- `escalate`
|
||||
- `prerequisiteOps`
|
||||
- `queueReady`
|
||||
- `reasons`
|
||||
- `resourceLocks`
|
||||
- `executionContract.acceptanceCommands`
|
||||
- `executionContract.requiredTools`
|
||||
- `executionContract.targetFiles`
|
||||
- `executionContract.stepIds`
|
||||
- `executionContract.riskLevel`
|
||||
- `executionContract.executionSpecificityScore`
|
||||
|
||||
Observed intake fields in `01_intake.json`:
|
||||
|
||||
- normalized requirements
|
||||
- parsed spec structure
|
||||
- conflict and ambiguity signals
|
||||
|
||||
Observed validation fields in `04_validate_taskitem.json`:
|
||||
|
||||
- gap and specificity diagnostics
|
||||
- recommended tools
|
||||
- execution-quality feedback
|
||||
|
||||
Assessment:
|
||||
|
||||
- highest value source for `verification_type`
|
||||
- highest value source for `prereq_op_selector`
|
||||
- highest value source for `execution_contract_shape`
|
||||
- highest value source for child skeleton decisions such as:
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `required_tool_selection`
|
||||
|
||||
Constraint:
|
||||
|
||||
- these files preserve outputs, but not necessarily independence of each field
|
||||
- some fields may be jointly emitted in one response rather than chosen as
|
||||
separate model calls
|
||||
- extraction should therefore attach provenance and a decomposition method
|
||||
|
||||
### 2. Raw NDJSON taskitem responses
|
||||
|
||||
Representative path:
|
||||
|
||||
- `logs/taskitem_runs/*/02_generate_taskitems_raw.ndjson.json`
|
||||
|
||||
What these preserve:
|
||||
|
||||
- near-raw tool response wrapper
|
||||
- text form of the generated taskitem JSON
|
||||
- tool/runtime metadata
|
||||
|
||||
Assessment:
|
||||
|
||||
- useful provenance source
|
||||
- useful for checking whether `02_generate_taskitems.json` is a direct capture
|
||||
or a normalized downstream rewrite
|
||||
- not a separate supervision source by itself
|
||||
|
||||
Constraint:
|
||||
|
||||
- this still reflects bundled output from a single taskitem-generation call
|
||||
- it does not by itself prove per-field independence
|
||||
|
||||
### 3. Project benchmark matrix runs
|
||||
|
||||
Representative path:
|
||||
|
||||
- `logs/taskitem_runs/project_benchmark_matrix_20260225_192610/*/ab_stdout.log`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- spec
|
||||
- language
|
||||
- strict mode
|
||||
- path A / path B readiness
|
||||
- failure reasons
|
||||
- token accounting
|
||||
|
||||
Assessment:
|
||||
|
||||
- good evaluation corpus
|
||||
- good source for comparing system modes and efficiency
|
||||
- not a direct source for task-shaping gates such as `prereq_op_selector` or
|
||||
`target_file_selection`
|
||||
|
||||
Constraint:
|
||||
|
||||
- these logs summarize end-to-end path outcomes
|
||||
- they do not preserve the intermediate bounded decisions we want to supervise
|
||||
|
||||
### 4. AST-once vs direct comparison runs
|
||||
|
||||
Representative path:
|
||||
|
||||
- `logs/taskitem_runs/ast_once_vs_direct_20260225_193037/*/base_payload.json`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- AST node payload
|
||||
- generated code
|
||||
- compile/test/placeholder gates
|
||||
- quality flags
|
||||
- used symbols
|
||||
|
||||
Assessment:
|
||||
|
||||
- useful evaluation corpus for AST-first vs direct generation comparisons
|
||||
- useful for downstream codegen quality measurement
|
||||
- not a primary direct source for taskitem decision supervision
|
||||
|
||||
Constraint:
|
||||
|
||||
- preserves codegen result and gate outcomes
|
||||
- does not preserve the earlier planning and task-shaping decisions
|
||||
|
||||
### 5. Example run specs
|
||||
|
||||
Representative paths:
|
||||
|
||||
- `datasets/example_run_specs/index.jsonl`
|
||||
- `datasets/example_run_specs/*.md`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- scenario family and variant metadata
|
||||
- objective
|
||||
- target behavior
|
||||
- anti-pattern tags
|
||||
- acceptance criteria
|
||||
- data capture tags
|
||||
|
||||
Assessment:
|
||||
|
||||
- strong context/spec source
|
||||
- useful for reconstructing scenario intent and mistake class
|
||||
- not a direct source of labels for bounded decisions
|
||||
|
||||
### 6. Project benchmark inventories
|
||||
|
||||
Representative path:
|
||||
|
||||
- `datasets/project_benchmarks/*.jsonl`
|
||||
|
||||
Observed shape:
|
||||
|
||||
- project id
|
||||
- category
|
||||
- language
|
||||
- natural-language spec
|
||||
|
||||
Assessment:
|
||||
|
||||
- good context reservoir
|
||||
- good replay seed corpus
|
||||
- not a direct supervision source
|
||||
|
||||
## Extraction Suitability By Decision
|
||||
|
||||
### Direct or near-direct candidates
|
||||
|
||||
- `prereq_op_selector`
|
||||
- candidate source: `02_generate_taskitems.json`
|
||||
- source field: `prerequisiteOps`
|
||||
- note: may still need decomposition into slot-based labels rather than raw
|
||||
list copy
|
||||
|
||||
- `execution_contract_shape`
|
||||
- candidate source: `02_generate_taskitems.json`
|
||||
- source field: `executionContract`
|
||||
- note: should be split into child decisions before training
|
||||
|
||||
- `target_file_selection`
|
||||
- candidate source: `executionContract.targetFiles`
|
||||
- note: must preserve local requirement context and touched-surface context
|
||||
|
||||
- `acceptance_command_selection`
|
||||
- candidate source: `executionContract.acceptanceCommands`
|
||||
- note: likely partly heuristic in some runs; requires provenance tagging
|
||||
|
||||
- `required_tool_selection`
|
||||
- candidate source: `executionContract.requiredTools`
|
||||
- note: likely mixed model-plus-template behavior
|
||||
|
||||
### Reconstructible but not yet trusted as direct labels
|
||||
|
||||
- `verification_type`
|
||||
- may be reconstructible from execution-contract structure, validation shape,
|
||||
and downstream required tools
|
||||
- not yet observed as a single explicit standalone field in audited artifacts
|
||||
|
||||
- `worker_type`
|
||||
- current code surface exists in routing/orchestration layers:
|
||||
- `editor/src/RoutingRules.h`
|
||||
- `editor/src/WorkflowOrchestrator.h`
|
||||
- `editor/src/TaskitemPoolPersistence.h`
|
||||
- `editor/src/headless_rpc/DispatchPart6.h`
|
||||
- not yet observed as a clean explicit field in the audited taskitem-run
|
||||
artifacts used for task generation
|
||||
- may exist elsewhere in routing or orchestration artifacts, or may need
|
||||
reconstruction from execution contract and downstream path
|
||||
|
||||
### Poor direct-fit sources for current Sprint 003 gates
|
||||
|
||||
- `cpp_raising_profile`
|
||||
- current code surface exists in C++-raising layers:
|
||||
- `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
- `editor/src/cpp_ir/CppErrorModelMapping.h`
|
||||
- `editor/src/cpp_ir/CppBuildArtifactGenerator.h`
|
||||
- `editor/src/cpp_ir/CppAsyncMappingStrategy.h`
|
||||
- `editor/src/cpp_ir/CppTemplateRaisingPolicy.h`
|
||||
- not surfaced in the audited generic taskitem corpora reviewed so far
|
||||
- likely needs a different source family tied to C++ raising flows rather than
|
||||
generic taskitem generation
|
||||
|
||||
## Search Outcome Notes
|
||||
|
||||
Additional source search confirms:
|
||||
|
||||
- `workerType` is a real first-class concept in the current DSL routing stack
|
||||
- `safe-first|perf-first|interop-first` is a real first-class concept in the
|
||||
current C++ raising stack
|
||||
- the audited taskitem-generation artifacts do not yet show those values as
|
||||
clean serialized labels in the same way that `prerequisiteOps` and
|
||||
`executionContract.*` are preserved
|
||||
|
||||
Practical implication:
|
||||
|
||||
- `prereq_op_selector` and execution-contract child decisions can move toward
|
||||
extraction-spec work now
|
||||
- `worker_type` and `cpp_raising_profile` need a second artifact search pass
|
||||
over routing and raising-specific run families before we treat them as
|
||||
extractable labels
|
||||
|
||||
## Provenance Classes
|
||||
|
||||
Every extracted training row should carry one provenance tag:
|
||||
|
||||
- `direct_explicit`
|
||||
- field exists directly in the artifact in bounded form
|
||||
|
||||
- `direct_bundled`
|
||||
- field exists directly, but as one member of a larger jointly emitted object
|
||||
|
||||
- `reconstructed`
|
||||
- label inferred from multiple artifact fields or execution traces
|
||||
|
||||
- `untrusted`
|
||||
- label candidate exists, but source independence or semantic stability is too
|
||||
weak to use without further review
|
||||
|
||||
Current default classification:
|
||||
|
||||
- `prerequisiteOps`: `direct_bundled`
|
||||
- `executionContract.targetFiles`: `direct_bundled`
|
||||
- `executionContract.acceptanceCommands`: `direct_bundled`
|
||||
- `executionContract.requiredTools`: `direct_bundled`
|
||||
- `verification_type`: `reconstructed`
|
||||
- `worker_type`: `reconstructed`
|
||||
- `cpp_raising_profile`: `untrusted`
|
||||
|
||||
## Immediate Recommendations
|
||||
|
||||
1. Use `logs/taskitem_runs/*/{01_intake.json,02_generate_taskitems.json,04_validate_taskitem.json}`
|
||||
as the first extraction corpus.
|
||||
|
||||
2. Treat `datasets/example_run_specs` and `datasets/project_benchmarks` as
|
||||
context reservoirs, not label sources.
|
||||
|
||||
3. Keep project benchmark matrix and AST/direct corpora for evaluation and
|
||||
counterfactual replay, not first-pass supervision.
|
||||
|
||||
4. Build extraction specs per gate rather than per full taskitem JSON.
|
||||
|
||||
5. Require each extraction spec to declare:
|
||||
- source files
|
||||
- context slice
|
||||
- label shape
|
||||
- provenance class
|
||||
- rejection conditions
|
||||
|
||||
## Next Audit Steps
|
||||
|
||||
1. Search surviving routing/orchestration artifacts for explicit `worker_type`
|
||||
labels.
|
||||
|
||||
2. Search C++ raising artifacts for explicit `cpp_raising_profile` choices.
|
||||
|
||||
3. Draft per-gate extraction specs for:
|
||||
- `prereq_op_selector`
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `required_tool_selection`
|
||||
|
||||
4. Add rejection rules for low-trust bundled outputs before building any
|
||||
training set.
|
||||
367
docs/case_studies/whetstone_dsl_semantic_mapping.md
Normal file
367
docs/case_studies/whetstone_dsl_semantic_mapping.md
Normal file
@@ -0,0 +1,367 @@
|
||||
# WhetstoneDSL Semantic Mapping
|
||||
|
||||
## Purpose
|
||||
|
||||
This document maps the current `whetstone_DSL` implementation surfaces into the
|
||||
semantic entities being established for `whetstone_RSA`.
|
||||
|
||||
It exists to make Sprint 003 executable.
|
||||
|
||||
The goal is not to restate architecture in the abstract. The goal is to answer:
|
||||
|
||||
- which `whetstone_DSL` files currently embody the first RSA-relevant decisions
|
||||
- which RSA semantic entities those files imply
|
||||
- which parts are already deterministic baselines
|
||||
- which parts should become explicit gate contracts
|
||||
- which parts should remain skeletons before training
|
||||
|
||||
## Mapping Rule
|
||||
|
||||
Each `whetstone_DSL` surface should be classified into one of four roles:
|
||||
|
||||
- semantic source candidate
|
||||
- deterministic baseline
|
||||
- runtime/executor consumer
|
||||
- training-data source candidate
|
||||
|
||||
This keeps the case study from collapsing implementation, semantics, and data
|
||||
collection into one bucket.
|
||||
|
||||
## Surface Map
|
||||
|
||||
### 1. `editor/src/TaskitemGeneratorV2.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- generate taskitems from decomposed workstreams
|
||||
- infer `workerType`
|
||||
- infer `prerequisiteOps`
|
||||
- mark queue readiness
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- `worker_type` is a first-class `DecisionContract` plus `DecisionSurface`
|
||||
- `prereq_op_selector` is a first-class `DecisionContract` plus structured
|
||||
`DecisionSurface`
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- current keyword worker heuristic is the baseline policy
|
||||
- current prerequisite-op defaulting behavior is the baseline policy
|
||||
|
||||
Training-data value:
|
||||
|
||||
- high, once paired with richer context and accepted downstream contracts
|
||||
- especially useful for:
|
||||
- task description -> `worker_type`
|
||||
- task description -> prerequisite-op set
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `WorkerTypeGate`
|
||||
- `PrereqOpGate`
|
||||
|
||||
### 2. `editor/src/TaskitemConfidenceAmbiguity.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- compute confidence score
|
||||
- count ambiguity
|
||||
- derive escalation flag
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- this is primarily a policy/baseline artifact, not a first training gate
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- extremely high
|
||||
- should remain attached to future gate records as deterministic baseline
|
||||
evidence
|
||||
|
||||
Training-data value:
|
||||
|
||||
- low as a primary specialist target
|
||||
- high as a supervision source for:
|
||||
- deterministic-baseline detection
|
||||
- guardrail-policy comparison
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `DeterministicPolicyBaseline`
|
||||
- not an initial RSA gate
|
||||
|
||||
### 3. `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- intake normalization and taskitem generation orchestration
|
||||
- execution-contract shaping
|
||||
- verification-type inference
|
||||
- target-file inference
|
||||
- acceptance-command inference
|
||||
- step-sequence inference
|
||||
- cross-project-target inference
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- `verification_type` is a first-class immediate gate
|
||||
- execution contract shaping is a structured semantic frontier
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- very high for current heuristics
|
||||
- should be preserved as baseline before any specialist substitution
|
||||
|
||||
Training-data value:
|
||||
|
||||
- very high if taskitem runs with context and final MCP choices still exist
|
||||
- likely strongest surface for:
|
||||
- `verification_type`
|
||||
- `target_file_selection`
|
||||
- `acceptance_command_selection`
|
||||
- `execution_contract_shape`
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `VerificationTypeGate`
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `AcceptanceCommandSelectionDecision`
|
||||
- `StepSequenceDecision`
|
||||
|
||||
### 4. `editor/src/ScopeMilestoneDecomposer.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- milestone grouping
|
||||
- workstream grouping
|
||||
- grouped summary/title generation
|
||||
- uncertainty scoring
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- grouping decisions are real semantic decisions
|
||||
- but they are not yet clean enough as direct training targets
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- high
|
||||
|
||||
Training-data value:
|
||||
|
||||
- medium
|
||||
- useful after the grouping contract is made explicit
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `MilestoneGroupingDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
- `WorkstreamSummaryPolicy`
|
||||
|
||||
### 5. `editor/src/RoutingEngine.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- choose worker layer
|
||||
- choose context width
|
||||
- set review flag
|
||||
- derive agent role
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- future second-wave RSA surface
|
||||
- useful as a meta-layer above particular gates
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- extremely high
|
||||
- keep this baseline intact for later RSA-versus-rule comparison
|
||||
|
||||
Training-data value:
|
||||
|
||||
- medium to high once real historical routing outcomes are assembled
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `LayerSuitabilityDecision`
|
||||
- `RoutingContextDecision`
|
||||
|
||||
Note:
|
||||
|
||||
- these sit one layer above RSA identity itself, so they should not be confused
|
||||
with an RSA gate contract
|
||||
|
||||
### 6. `editor/src/mcp/RegisterCppRaisingTools.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- choose and validate C++ raising profile
|
||||
- run deterministic C++ raising and packaging pipeline
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- `cpp_raising_profile` is a clean bounded gate candidate
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- current manual/explicit profile selection is the baseline
|
||||
|
||||
Training-data value:
|
||||
|
||||
- medium
|
||||
- strongest when paired with IR traits, risk annotations, and downstream review
|
||||
outcomes
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `CppRaisingProfileGate`
|
||||
|
||||
### 7. `editor/src/ConstrainedProjectionGate.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- block generation when environment constraints are violated
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- this is executor-side deterministic gating
|
||||
- it is not currently an RSA gate candidate
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- extremely high
|
||||
|
||||
Training-data value:
|
||||
|
||||
- useful as supervision for invalid-output prevention
|
||||
- not useful as a primary specialist target
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- `DeterministicConstraintGate`
|
||||
|
||||
### 8. `editor/src/ast/ProjectionGenerator.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- dispatch AST node kinds to language-specific visitors
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- almost none at the current stage
|
||||
- this is deterministic target-language projection
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- total
|
||||
|
||||
Training-data value:
|
||||
|
||||
- not a primary RSA training surface
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- no immediate RSA semantic entity
|
||||
- leave in deterministic projection layer
|
||||
|
||||
### 9. `editor/src/CrossLanguageProjector.h`
|
||||
|
||||
Current responsibility:
|
||||
|
||||
- clone AST into target language
|
||||
- preserve/adapt annotations
|
||||
|
||||
RSA semantic implications:
|
||||
|
||||
- good future source for bounded policy choices around adaptation profiles
|
||||
- current implementation is deterministic
|
||||
|
||||
Deterministic baseline value:
|
||||
|
||||
- high
|
||||
|
||||
Training-data value:
|
||||
|
||||
- low now
|
||||
- potentially higher later if multiple adaptation profiles exist
|
||||
|
||||
Recommended semantic extraction:
|
||||
|
||||
- future `ProjectionProfileDecision`
|
||||
- not an immediate Sprint 003 gate
|
||||
|
||||
## Semantic Entity Alignment
|
||||
|
||||
### Immediate gate entities
|
||||
|
||||
| WhetstoneDSL surface | RSA entity | Status |
|
||||
| --- | --- | --- |
|
||||
| `TaskitemGeneratorV2::inferWorkerType` | `WorkerTypeGate` | bootstrap now |
|
||||
| `TaskitemGeneratorV2::inferPrerequisiteOps` | `PrereqOpGate` | bootstrap now |
|
||||
| `RegisterArchitectIntakeTools::inferVerificationType` | `VerificationTypeGate` | bootstrap now |
|
||||
| `RegisterCppRaisingTools` profile enum | `CppRaisingProfileGate` | bootstrap now |
|
||||
|
||||
### Skeleton-first entities
|
||||
|
||||
| WhetstoneDSL surface | RSA entity | Status |
|
||||
| --- | --- | --- |
|
||||
| execution-contract heuristics | `ExecutionContractShapeDecision` | skeleton now |
|
||||
| target-file inference | `TargetFileSelectionDecision` | skeleton now |
|
||||
| grouping heuristics | `WorkstreamGroupingDecision` | skeleton now |
|
||||
| grouped summary logic | `WorkstreamSummaryPolicy` | skeleton now |
|
||||
| step-sequence inference | `StepSequenceDecision` | skeleton now |
|
||||
|
||||
### Deterministic baselines to preserve
|
||||
|
||||
| WhetstoneDSL surface | RSA role |
|
||||
| --- | --- |
|
||||
| `TaskitemConfidenceAmbiguity` | deterministic baseline / guardrail policy |
|
||||
| `RoutingEngine` | deterministic meta-routing baseline |
|
||||
| `ConstrainedProjectionGate` | deterministic constraint gate |
|
||||
| `ProjectionGenerator` dispatch | deterministic projection executor |
|
||||
| `CrossLanguageProjector` current adaptation | deterministic projection baseline |
|
||||
|
||||
## What This Means For Integration
|
||||
|
||||
The current integration path should be:
|
||||
|
||||
1. capture gate semantics first
|
||||
2. preserve deterministic baselines as explicit comparison points
|
||||
3. identify data sources for each gate
|
||||
4. defer runtime substitution until the gate contracts are stable
|
||||
|
||||
The wrong integration path would be:
|
||||
|
||||
1. wire RSA runtime into `whetstone_DSL`
|
||||
2. guess what the gates mean later
|
||||
|
||||
This case study is strong enough now to avoid that mistake.
|
||||
|
||||
## Training Data Notes
|
||||
|
||||
Potential high-value training data sources still likely exist in or around
|
||||
`whetstone_DSL`, depending on what survived deletion:
|
||||
|
||||
- taskitem pipeline run logs with context and final MCP choices
|
||||
- specialist training data under `whetstone_DSL/specialists`
|
||||
- the `100 easy` and `25 challenging` project sets already used for examples and
|
||||
evaluation
|
||||
- sprint plan corpus already used for `verification_type`
|
||||
|
||||
Immediate implication:
|
||||
|
||||
- do not block semantic bootstrap on recovering training data
|
||||
- but treat recovered taskitem/MCP-choice runs as likely first-class supervision
|
||||
once the gate contracts are fixed
|
||||
|
||||
## Next Mapping Step
|
||||
|
||||
The next document after this one should define skeleton field sets for:
|
||||
|
||||
- `ExecutionContractShapeDecision`
|
||||
- `TargetFileSelectionDecision`
|
||||
- `WorkstreamGroupingDecision`
|
||||
|
||||
Those are the structured surfaces most likely to become later RSA gates once the
|
||||
semantic contract is explicit enough to benchmark cleanly.
|
||||
218
docs/case_studies/whetstone_dsl_training_data_strategy.md
Normal file
218
docs/case_studies/whetstone_dsl_training_data_strategy.md
Normal file
@@ -0,0 +1,218 @@
|
||||
# WhetstoneDSL Training Data Strategy
|
||||
|
||||
## Purpose
|
||||
|
||||
This document records the data-handling rule for turning existing
|
||||
`whetstone_DSL` outputs into RSA training data.
|
||||
|
||||
The immediate concern is that much of the existing material may be:
|
||||
|
||||
- whole JSON responses
|
||||
- whole taskitem packets
|
||||
- multi-decision LLM outputs
|
||||
|
||||
rather than clean one-decision-per-example supervision.
|
||||
|
||||
That means the data cannot be used naively.
|
||||
|
||||
## Core Rule
|
||||
|
||||
Do not train per-gate specialists directly on full taskitem or full-response
|
||||
outputs without reconstructing:
|
||||
|
||||
- the exact local context for the decision
|
||||
- the specific decision target being learned
|
||||
- whether the label was independently chosen or entangled with other choices
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The old pipeline was optimized for:
|
||||
|
||||
- one LLM or SLM making several bounded choices together
|
||||
- minimizing token use by sharing context across those choices
|
||||
|
||||
The new RSA direction is different:
|
||||
|
||||
- one specialist per bounded decision
|
||||
- narrower context per decision
|
||||
- explicit gate contracts
|
||||
|
||||
So existing outputs may reflect:
|
||||
|
||||
- coupled decisions
|
||||
- prompt-order effects
|
||||
- hidden shared rationale
|
||||
- labels that were only valid because other fields in the same response were
|
||||
chosen jointly
|
||||
|
||||
If we ignore that, the training data will be semantically wrong even if it looks
|
||||
large.
|
||||
|
||||
## Safe Extraction Rule
|
||||
|
||||
For each candidate gate, build examples as:
|
||||
|
||||
- `decision_context`
|
||||
- `decision_label`
|
||||
- `decision_provenance`
|
||||
|
||||
and not as:
|
||||
|
||||
- full response blob -> label
|
||||
|
||||
### `decision_context`
|
||||
|
||||
Must include only the information that would have been available at the moment
|
||||
that gate should fire.
|
||||
|
||||
Examples:
|
||||
|
||||
- `worker_type`
|
||||
- task title
|
||||
- task intent
|
||||
- maybe requirement summary
|
||||
- not downstream fields derived after routing
|
||||
|
||||
- `verification_type`
|
||||
- target files
|
||||
- task title
|
||||
- task intent
|
||||
- not later acceptance-command decisions unless those are part of the gate by
|
||||
contract
|
||||
|
||||
- `prereq_op_selector`
|
||||
- task description
|
||||
- uncertainty and dependency signals if the intended gate contract includes them
|
||||
- not later execution outcomes
|
||||
|
||||
### `decision_label`
|
||||
|
||||
Must match the new gate contract exactly.
|
||||
|
||||
If the historic data uses a different shape than the new gate, do not force it.
|
||||
|
||||
Instead:
|
||||
|
||||
- transform it carefully
|
||||
- or discard it
|
||||
- or redefine the gate
|
||||
|
||||
### `decision_provenance`
|
||||
|
||||
Each extracted example should track:
|
||||
|
||||
- source artifact
|
||||
- source run id if available
|
||||
- whether label was explicit or inferred
|
||||
- whether the example came from:
|
||||
- human-authored heuristic baseline
|
||||
- LLM output
|
||||
- SLM output
|
||||
- deterministic reconstruction
|
||||
|
||||
This is necessary because not all labels have equal trust.
|
||||
|
||||
## Gate-Specific Warnings
|
||||
|
||||
### `worker_type`
|
||||
|
||||
This is likely recoverable from older taskitems, but only if the worker role was
|
||||
explicitly present and chosen before downstream execution shaping.
|
||||
|
||||
Risk:
|
||||
|
||||
- label polluted by later execution-contract logic
|
||||
|
||||
### `verification_type`
|
||||
|
||||
This is relatively safe because it can often be reconstructed from target files
|
||||
and acceptance intent.
|
||||
|
||||
Risk:
|
||||
|
||||
- historic labels may include joint decisions with acceptance-command shaping
|
||||
|
||||
### `prereq_op_selector`
|
||||
|
||||
This is the riskiest early gate for naive extraction.
|
||||
|
||||
Reason:
|
||||
|
||||
- older systems may have emitted a flat list of ops as one bundle
|
||||
- the new semantic contract treats this as structured or factorized
|
||||
|
||||
Rule:
|
||||
|
||||
- do not flatten structured op bundles into arbitrary multiclass labels
|
||||
|
||||
### `cpp_raising_profile`
|
||||
|
||||
This is likely only useful if old runs actually recorded explicit profile choice
|
||||
with enough IR context.
|
||||
|
||||
If not, this gate may need to start with synthetic or policy-authored data rather
|
||||
than mined response logs.
|
||||
|
||||
## Easy/Challenging Project Corpora
|
||||
|
||||
The `100 easy` and `25 challenging` project sets are valuable, but they are not
|
||||
automatically specialist training sets.
|
||||
|
||||
They are best treated as:
|
||||
|
||||
- context reservoirs
|
||||
- replay/evaluation corpora
|
||||
- example generators for extracting many per-gate decisions
|
||||
|
||||
The right pattern is:
|
||||
|
||||
1. replay or parse the project artifact
|
||||
2. isolate one decision surface at a time
|
||||
3. extract only the local context relevant to that gate
|
||||
4. record the corresponding bounded label
|
||||
|
||||
This should produce a gate-native dataset rather than a project-native dataset.
|
||||
|
||||
## Compatibility Constraint
|
||||
|
||||
We should not assume the new specialists must fit perfectly into the exact old
|
||||
taskitem bundle shape.
|
||||
|
||||
There are two valid outcomes:
|
||||
|
||||
### Option A: Fit specialists into current taskitems
|
||||
|
||||
Use when:
|
||||
|
||||
- the old taskitem structure already matches clean gate boundaries
|
||||
- the decision context can be isolated without distortion
|
||||
|
||||
### Option B: Rebuild taskitems around explicit gates
|
||||
|
||||
Use when:
|
||||
|
||||
- the current taskitems bundle too many coupled decisions
|
||||
- the old layout hides which context belongs to which choice
|
||||
- structured gates are being forced into flat labels
|
||||
|
||||
This is not failure.
|
||||
It may be the correct result of moving from one-model-many-decisions to
|
||||
many-model-one-decision architecture.
|
||||
|
||||
## Recommended Next Step
|
||||
|
||||
Before mining historic runs, define extraction specs for the first four gates:
|
||||
|
||||
- `verification_type`
|
||||
- `worker_type`
|
||||
- `prereq_op_selector`
|
||||
- `cpp_raising_profile`
|
||||
|
||||
Each extraction spec should state:
|
||||
|
||||
- allowed input fields
|
||||
- forbidden leaked fields
|
||||
- label shape
|
||||
- trust level of recovered labels
|
||||
|
||||
Only after that should we parse old JSON outputs or taskitem runs.
|
||||
@@ -221,3 +221,652 @@ It should become:
|
||||
|
||||
That is the architecture most consistent with both Whetstone and the long-term
|
||||
goals of RSA.
|
||||
|
||||
## Bootstrap Mapping From The Current C++ Scaffold
|
||||
|
||||
The current C++ code is useful bootstrap material, but it should now be treated as
|
||||
input to the semantic model rather than the long-term definition of that model.
|
||||
|
||||
This section inventories the current scaffold and classifies each piece as either:
|
||||
|
||||
- semantic contract that should move into Whetstone-authored form
|
||||
- runtime behavior that should remain in projected C++
|
||||
- bootstrap/demo code that should not define long-term architecture
|
||||
|
||||
### Current C++ Semantic Surface
|
||||
|
||||
From `include/whetstone_rsa/types.h`, the current scaffold already expresses the
|
||||
core semantic vocabulary that should be re-authored in Whetstone form:
|
||||
|
||||
- ontology enums
|
||||
- `InputStructure`
|
||||
- `InputModality`
|
||||
- `OutputTopology`
|
||||
- `OutputConstraint`
|
||||
- `LabelStability`
|
||||
- `TargetLayer`
|
||||
- diagnosis enums
|
||||
- `FailureClass`
|
||||
- `Intervention`
|
||||
- `DeployMode`
|
||||
- `RiskTier`
|
||||
- contract entities
|
||||
- `GateDefinition`
|
||||
- `GateEvidence`
|
||||
- `PolicyContext`
|
||||
- `ProbeRequest`
|
||||
- `ProbeResult`
|
||||
- `GateDiagnosis`
|
||||
- `GateSuitability`
|
||||
- `PolicyRecommendation`
|
||||
|
||||
These should not remain primarily hand-authored C++ concepts.
|
||||
|
||||
They should become semantic entities in the Whetstone source of truth.
|
||||
|
||||
### Recommended Whetstone Semantic Entities
|
||||
|
||||
The current scaffold suggests the following first semantic model.
|
||||
|
||||
#### 1. `DecisionContract`
|
||||
|
||||
This should become the semantic home for the ontology-level fields that currently
|
||||
sit inside `GateDefinition`.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `gate_id`
|
||||
- `task_family`
|
||||
- `input_structure`
|
||||
- `input_modality`
|
||||
- `output_topology`
|
||||
- `output_constraint`
|
||||
- `risk_tier`
|
||||
- `policy_stability`
|
||||
- `deterministic_baseline_available`
|
||||
- `supports_abstain`
|
||||
- `metadata`
|
||||
|
||||
This is the contract boundary, independent of backend family.
|
||||
|
||||
#### 2. `DecisionSurface`
|
||||
|
||||
The current `GateDefinition.labels` plus `candidate_factorizations` really describe
|
||||
the shape of the output space, not just the gate identity.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `output_labels`
|
||||
- `slot_schema`
|
||||
- `candidate_factorizations`
|
||||
- `candidate_gate_shapes`
|
||||
- `invalid_combinations`
|
||||
- `deterministic_prepass_options`
|
||||
|
||||
This gives the AST layer a place to describe flat multiclass, hierarchical, or
|
||||
factorized designs without forcing that structure to remain implicit in C++.
|
||||
|
||||
#### 3. `DecisionRequestModel`
|
||||
|
||||
The current C++ scaffold does not yet have a first-class request packet type
|
||||
separate from evaluation/demo inputs.
|
||||
|
||||
That should be corrected in the AST-first model.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `packet_id`
|
||||
- `text_fields`
|
||||
- `symbolic_fields`
|
||||
- `numeric_fields`
|
||||
- `context_fields`
|
||||
- `state_fields`
|
||||
- `provenance`
|
||||
|
||||
This entity should support both enum-style gates and bounded structured outputs.
|
||||
|
||||
#### 4. `DecisionResultModel`
|
||||
|
||||
The current scaffold implicitly assumes label selection and policy behavior, but the
|
||||
AST-first model should make result structure explicit.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `decision_id`
|
||||
- `selected_label`
|
||||
- `filled_slots`
|
||||
- `confidence`
|
||||
- `abstain`
|
||||
- `retry_strategy`
|
||||
- `escalation_target`
|
||||
- `trace_packet`
|
||||
|
||||
This should later project into both C++ runtime structs and packaging artifacts.
|
||||
|
||||
#### 5. `DiagnosisModel`
|
||||
|
||||
The current `GateDiagnosis` is already close to a semantic object and should be
|
||||
preserved almost directly, but moved into Whetstone-authored form.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `primary_failure_class`
|
||||
- `secondary_failure_classes`
|
||||
- `supporting_signals`
|
||||
- `recommended_probes`
|
||||
- `recommended_interventions`
|
||||
- `recommended_policy`
|
||||
- `confidence`
|
||||
|
||||
#### 6. `ProbeDefinition`
|
||||
|
||||
The current `Probe` interface and `ProbeRequest` / `ProbeResult` pair indicate that
|
||||
probes are part of the semantic model, not just implementation detail.
|
||||
|
||||
The AST layer should explicitly represent:
|
||||
|
||||
- `probe_id`
|
||||
- `applies_to_gate_shapes`
|
||||
- `required_evidence_fields`
|
||||
- `produced_signals`
|
||||
- `failure_hints`
|
||||
- `intervention_hints`
|
||||
|
||||
Concrete runtime probes such as `confidence_threshold_probe` should then be C++
|
||||
implementations of these declared probe contracts.
|
||||
|
||||
#### 7. `SuitabilityAssessment`
|
||||
|
||||
The current `GateSuitability` type should become a semantic output model for the
|
||||
pre-RSA decision that determines whether the problem belongs in the RSA layer at all.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `recommended`
|
||||
- `target_layer`
|
||||
- `rationale`
|
||||
- `bounded_output_confidence`
|
||||
- `deterministic_baseline_available`
|
||||
- `label_stability_risk`
|
||||
- `suggested_gate_shape`
|
||||
|
||||
#### 8. `BackendSuitability`
|
||||
|
||||
This does not yet exist in code, but the roadmap now requires it.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `gate_id`
|
||||
- `candidate_backends`
|
||||
- `recommended_backend`
|
||||
- `rationale`
|
||||
- `confidence`
|
||||
- `benchmark_refs`
|
||||
|
||||
This must be semantic-first, not added ad hoc in C++ later.
|
||||
|
||||
#### 9. `GatePackage`
|
||||
|
||||
The runtime needs a durable packaged form for serving decisions after offline
|
||||
selection and calibration.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
- `gate_id`
|
||||
- `contract_ref`
|
||||
- `gate_shape`
|
||||
- `backend_family`
|
||||
- `artifact_ref`
|
||||
- `thresholds`
|
||||
- `abstain_policy`
|
||||
- `escalation_policy`
|
||||
- `trace_schema`
|
||||
- `version`
|
||||
|
||||
### What Should Remain Runtime-Only In C++
|
||||
|
||||
The following concepts should remain projected runtime behavior rather than become
|
||||
the semantic source of truth:
|
||||
|
||||
- `Probe` virtual interface
|
||||
- `ProbeRegistry`
|
||||
- `DiagnosisEngine`
|
||||
- `SuitabilityAssessor`
|
||||
- backend adapter implementations
|
||||
- calibration code
|
||||
- packaging loaders
|
||||
- benchmark runners
|
||||
|
||||
These are execution mechanisms, not the semantic model itself.
|
||||
|
||||
### What In The Current Scaffold Is Only Bootstrap
|
||||
|
||||
The following parts are useful but should not steer the long-term semantic design:
|
||||
|
||||
- `examples/diagnose_demo.cpp`
|
||||
- the current hand-authored stringification helpers
|
||||
- any current field arrangement that exists only because it was convenient for the
|
||||
first C++ scaffold
|
||||
|
||||
### Immediate Migration Map
|
||||
|
||||
The practical next move should be:
|
||||
|
||||
1. Represent the current C++ contract entities above as Whetstone semantic objects.
|
||||
2. Mark which current fields are semantic contract versus runtime-only detail.
|
||||
3. Treat `types.h` as a temporary projected contract snapshot, not the authoring
|
||||
home of the model.
|
||||
4. Add projection rules so future C++ runtime types are derived from the semantic
|
||||
source rather than independently evolved.
|
||||
|
||||
### Most Important Boundary To Preserve
|
||||
|
||||
The current scaffold already mixes two different layers that must stay separate:
|
||||
|
||||
- semantic contract
|
||||
- runtime execution strategy
|
||||
|
||||
If that separation is preserved now, `whetstone_RSA` can become genuinely AST-first.
|
||||
If it is not preserved, the project will keep drifting into a C++-first runtime
|
||||
with AST plans attached afterward.
|
||||
|
||||
## First RSA Semantic Model
|
||||
|
||||
This section defines the first concrete semantic model that should become the
|
||||
source of truth for `whetstone_RSA`.
|
||||
|
||||
It is intentionally small and close to the current scaffold so migration can
|
||||
start immediately without pretending the architecture is already finished.
|
||||
|
||||
The design rule is:
|
||||
|
||||
- semantic entities describe contract and meaning
|
||||
- projected C++ types describe runtime execution
|
||||
|
||||
### Model Goals
|
||||
|
||||
The first semantic model must support all of the following from the beginning:
|
||||
|
||||
- bounded enum decisions
|
||||
- bounded structured slot-filling decisions
|
||||
- suitability assessment before a gate is adopted
|
||||
- diagnosis after a gate underperforms
|
||||
- gate-shape comparison
|
||||
- backend-selection metadata
|
||||
- runtime packaging metadata
|
||||
|
||||
### Semantic Entity Set
|
||||
|
||||
The first semantic model should consist of these entities:
|
||||
|
||||
- `DecisionContract`
|
||||
- `DecisionSurface`
|
||||
- `DecisionRequestModel`
|
||||
- `DecisionResultModel`
|
||||
- `SuitabilityAssessment`
|
||||
- `DiagnosisModel`
|
||||
- `ProbeDefinition`
|
||||
- `BackendSuitability`
|
||||
- `GatePackage`
|
||||
|
||||
### 1. `DecisionContract`
|
||||
|
||||
This is the semantic identity of an RSA gate.
|
||||
|
||||
It answers:
|
||||
|
||||
- what decision surface exists
|
||||
- what kind of inputs it consumes
|
||||
- what kind of bounded outputs it promises
|
||||
- what policy constraints govern deployment
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionContract
|
||||
gate_id
|
||||
task_family
|
||||
input_structure
|
||||
input_modality
|
||||
output_topology
|
||||
output_constraint
|
||||
risk_tier
|
||||
policy_stability
|
||||
deterministic_baseline_available
|
||||
supports_abstain
|
||||
metadata
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `policy_stability` should replace C++ naming that implies only labels drift.
|
||||
- this entity should not include backend-family decisions
|
||||
- this entity should not include benchmark evidence
|
||||
|
||||
### 2. `DecisionSurface`
|
||||
|
||||
This is the semantic description of the output space.
|
||||
|
||||
It is separate from `DecisionContract` because two gates can share similar
|
||||
contract properties but differ in shape:
|
||||
|
||||
- flat enum choice
|
||||
- factorized decision
|
||||
- hierarchical route
|
||||
- bounded slot fill
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionSurface
|
||||
gate_id
|
||||
primary_mode // enum | structured | hybrid
|
||||
output_labels
|
||||
slot_schema
|
||||
candidate_gate_shapes // binary | multiclass | factorized | hierarchical
|
||||
candidate_factorizations
|
||||
invalid_combinations
|
||||
deterministic_prepass_options
|
||||
downstream_executor_kind
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `slot_schema` is required from version 0 even if many early gates are enum-only
|
||||
- `invalid_combinations` should make structured outputs safe by contract
|
||||
- `downstream_executor_kind` should record what deterministic system consumes the result
|
||||
|
||||
### 3. `DecisionRequestModel`
|
||||
|
||||
This is the semantic input packet model for runtime use.
|
||||
|
||||
It should support hybrid packets directly instead of forcing a project to flatten
|
||||
everything into one string.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionRequestModel
|
||||
packet_id
|
||||
gate_id
|
||||
text_fields
|
||||
symbolic_fields
|
||||
numeric_fields
|
||||
context_fields
|
||||
state_fields
|
||||
provenance
|
||||
```
|
||||
|
||||
Expected usage:
|
||||
|
||||
- `text_fields`
|
||||
- natural language summaries
|
||||
- extracted comments
|
||||
- short requirement packets
|
||||
- `symbolic_fields`
|
||||
- AST node kinds
|
||||
- annotations
|
||||
- enums
|
||||
- dependency graphs
|
||||
- `numeric_fields`
|
||||
- counts
|
||||
- scores
|
||||
- ratios
|
||||
- latency or complexity features
|
||||
|
||||
### 4. `DecisionResultModel`
|
||||
|
||||
This is the canonical semantic output model for runtime decisions.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DecisionResultModel
|
||||
decision_id
|
||||
gate_id
|
||||
selected_label
|
||||
filled_slots
|
||||
confidence
|
||||
abstain
|
||||
retry_strategy
|
||||
escalation_target
|
||||
trace_packet
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- enum gates mainly use `selected_label`
|
||||
- structured gates mainly use `filled_slots`
|
||||
- hybrid outputs may use both
|
||||
|
||||
### 5. `SuitabilityAssessment`
|
||||
|
||||
This is the semantic result of asking whether a problem belongs in RSA at all.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
SuitabilityAssessment
|
||||
gate_id
|
||||
recommended
|
||||
target_layer // deterministic | rsa | slm | llm | human
|
||||
rationale
|
||||
bounded_output_confidence
|
||||
deterministic_baseline_available
|
||||
policy_stability_risk
|
||||
suggested_gate_shape
|
||||
```
|
||||
|
||||
This entity is intentionally separate from diagnosis and backend selection.
|
||||
|
||||
### 6. `DiagnosisModel`
|
||||
|
||||
This is the semantic result of evaluating a weak or risky gate.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
DiagnosisModel
|
||||
gate_id
|
||||
primary_failure_class
|
||||
secondary_failure_classes
|
||||
supporting_signals
|
||||
recommended_probes
|
||||
recommended_interventions
|
||||
recommended_policy
|
||||
confidence
|
||||
```
|
||||
|
||||
This should remain close to the current C++ `GateDiagnosis`.
|
||||
|
||||
### 7. `ProbeDefinition`
|
||||
|
||||
This is the semantic declaration of a diagnostic probe.
|
||||
|
||||
The semantic layer should define what a probe means before runtime C++ implements
|
||||
how it executes.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
ProbeDefinition
|
||||
probe_id
|
||||
applies_to_gate_shapes
|
||||
required_contract_fields
|
||||
required_evidence_fields
|
||||
produced_signals
|
||||
failure_hints
|
||||
intervention_hints
|
||||
```
|
||||
|
||||
Concrete examples:
|
||||
|
||||
- `confidence_threshold_probe`
|
||||
- `class_confusion_probe`
|
||||
- `factorization_probe`
|
||||
- `deterministic_rule_probe`
|
||||
- `context_width_probe`
|
||||
|
||||
### 8. `BackendSuitability`
|
||||
|
||||
This is the semantic result of backend comparison after a gate has already been
|
||||
judged to belong in RSA.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
BackendSuitability
|
||||
gate_id
|
||||
candidate_backends
|
||||
recommended_backend
|
||||
rationale
|
||||
confidence
|
||||
benchmark_refs
|
||||
```
|
||||
|
||||
Expected backend families:
|
||||
|
||||
- `linear_model`
|
||||
- `gradient_boosting`
|
||||
- `random_forest`
|
||||
- `bayesian`
|
||||
- `transformer`
|
||||
- `hybrid`
|
||||
- `rl_policy`
|
||||
|
||||
`rl_policy` should exist in the model now even if it remains an unimplemented
|
||||
runtime placeholder for the first version.
|
||||
|
||||
### 9. `GatePackage`
|
||||
|
||||
This is the semantic deployment artifact description.
|
||||
|
||||
It exists so runtime loading is a projection of semantic packaging decisions,
|
||||
not a one-off C++ concern.
|
||||
|
||||
Proposed fields:
|
||||
|
||||
```text
|
||||
GatePackage
|
||||
gate_id
|
||||
contract_ref
|
||||
surface_ref
|
||||
gate_shape
|
||||
backend_family
|
||||
artifact_ref
|
||||
thresholds
|
||||
abstain_policy
|
||||
escalation_policy
|
||||
trace_schema
|
||||
version
|
||||
```
|
||||
|
||||
### Semantic Relationships
|
||||
|
||||
The first model should follow these relationships:
|
||||
|
||||
```text
|
||||
DecisionContract
|
||||
owns high-level decision identity
|
||||
|
||||
DecisionSurface
|
||||
refines output shape for a contract
|
||||
|
||||
DecisionRequestModel
|
||||
is evaluated against DecisionContract + DecisionSurface
|
||||
|
||||
DecisionResultModel
|
||||
must satisfy DecisionSurface constraints
|
||||
|
||||
SuitabilityAssessment
|
||||
determines whether the decision belongs in RSA at all
|
||||
|
||||
DiagnosisModel
|
||||
explains weak performance or risky deployment
|
||||
|
||||
ProbeDefinition
|
||||
declares reusable diagnosis mechanisms
|
||||
|
||||
BackendSuitability
|
||||
recommends which implementation family should fill an RSA contract
|
||||
|
||||
GatePackage
|
||||
records the chosen runtime projection of a validated contract
|
||||
```
|
||||
|
||||
### Projection Map To The Current C++ Scaffold
|
||||
|
||||
The current scaffold should be treated as the first runtime projection target.
|
||||
|
||||
#### Project directly from semantics into current C++ types
|
||||
|
||||
```text
|
||||
DecisionContract -> GateDefinition core fields
|
||||
DecisionSurface -> GateDefinition.labels + candidate_factorizations + future slot fields
|
||||
DecisionResultModel -> future runtime result structs
|
||||
SuitabilityAssessment -> GateSuitability
|
||||
DiagnosisModel -> GateDiagnosis
|
||||
ProbeDefinition -> Probe metadata + ProbeRequest/ProbeResult contracts
|
||||
```
|
||||
|
||||
#### Runtime-only C++ types that should remain projected
|
||||
|
||||
```text
|
||||
Probe
|
||||
ProbeRegistry
|
||||
DiagnosisEngine
|
||||
SuitabilityAssessor
|
||||
ConfidenceThresholdProbe
|
||||
```
|
||||
|
||||
These are implementations, not semantic source objects.
|
||||
|
||||
### Immediate Type Migration Rule
|
||||
|
||||
Until semantic authoring is fully in place, the current C++ fields should be
|
||||
interpreted according to this split:
|
||||
|
||||
- semantic-now
|
||||
- ontology enums
|
||||
- gate contract fields
|
||||
- diagnosis enums
|
||||
- suitability result fields
|
||||
- runtime-now
|
||||
- probe interfaces
|
||||
- engine wiring
|
||||
- confidence-threshold implementation
|
||||
- demo executable
|
||||
|
||||
When a new field is proposed, the default question should be:
|
||||
|
||||
- does this describe semantic meaning?
|
||||
- if yes, add it to the semantic model first
|
||||
- does this describe runtime execution detail?
|
||||
- if yes, keep it in projected C++
|
||||
|
||||
### Immediate Next Conversion Step
|
||||
|
||||
The next concrete implementation step after this document should be:
|
||||
|
||||
1. represent `DecisionContract`, `DecisionSurface`, and `ProbeDefinition` as the
|
||||
first Whetstone-authored RSA semantic forms
|
||||
2. map each field in `types.h` into one of those forms
|
||||
3. mark missing fields needed for structured slot outputs and backend selection
|
||||
4. treat `types.h` as a temporary projection snapshot until semantic-first
|
||||
generation replaces hand editing
|
||||
|
||||
## Bootstrap Artifact
|
||||
|
||||
The first machine-readable semantic bootstrap artifact now lives at:
|
||||
|
||||
- `semantic/rsa_semantic_bootstrap_v0.json`
|
||||
|
||||
It is intentionally limited to the first three semantic entities needed to begin
|
||||
realignment:
|
||||
|
||||
- `DecisionContract`
|
||||
- `DecisionSurface`
|
||||
- `ProbeDefinition`
|
||||
|
||||
That file should be treated as the current semantic bootstrap source ahead of any
|
||||
further handwritten expansion of C++ contract types.
|
||||
|
||||
Reference in New Issue
Block a user