whetstone_DSL/sprint10_plan.md

Sprint 10 Plan: Semantic Annotation Taxonomy & Environment Layer

Context

Sprint 9 delivered a complete agent-first MCP tooling layer (34 tools, 244/244 tests, steps 245-265). The agent API surface is mature: headless mode, multi-file projects, diagnostics, search/rename, undo/redo, save, compact AST, batch queries.

Sprint 10 shifts focus to what agents actually do with that API: annotating code with rich semantic metadata from the full annotation taxonomy (docs/annotations/ Subjects 1-8) and modeling runtime environments. This is the core Whetstone value proposition — the semantic gap bridge between languages.

What's already started:

• Steps 266-270 TDD tests written (Phase 10a-10b)
• 5 semantic annotation AST classes in Annotation.h (Intent, Complexity, Risk, Contract, SemanticTag)
• SidecarPersistence.h with save/load logic
• Semantic annotation RPC methods in HeadlessAgentRPCHandler.h
• MCP tool registrations and prompt templates in MCPServer.h
• CompactAST.h semantic summary extraction

What's deferred to Sprint 11: LSP client, language-specific code actions, GUI overhaul, TUI mode, plugin system.

---

Phase 10a: Semantic Annotation Core + Sidecar Persistence (Steps 266-268)

TDD tests already written. Partial implementation exists.

Step 266: Semantic Annotation AST Node Types (12 tests)

Goal: 5 new AI-oriented annotation types with JSON roundtrip and compact AST support.

Types: IntentAnnotation, ComplexityAnnotation, RiskAnnotation, ContractAnnotation, SemanticTagAnnotation

Implementation already present in:

• editor/src/ast/Annotation.h (lines 145-184) — class definitions
• editor/src/ast/Serialization.h — toJson/fromJson dispatch
• editor/src/CompactAST.h — extractSemanticSummary()

Work remaining: Wire up any missing serialization, ensure all 12 tests pass.

Step 267: Sidecar AST Persistence (12 tests)

Goal: Save/load annotated ASTs as .whetstone/<filename>.ast.json sidecar files.

RPC methods: saveAnnotatedAST, loadAnnotatedAST, listAnnotatedFiles

Implementation already present in:

• editor/src/SidecarPersistence.h — sidecarPath(), saveSidecarAST(), loadSidecarAST()

Work remaining: Wire RPC handlers in HeadlessAgentRPCHandler.h, register MCP tools (whetstone_save_annotated_ast, whetstone_load_annotated_ast, whetstone_list_annotated_files), permissions in AgentPermissionPolicy.h.

Step 268: Phase 10a Integration Tests (8 tests)

Goal: End-to-end: annotate functions → compact AST shows semantic → save sidecar → reopen → load → verify.

Tests cover: multi-file sidecar, all 5 types in compact view, idempotent save/load, annotations don't pollute source output.

---

Phase 10b: Annotation Agent Interface (Steps 269-271)

Steps 269-270 TDD tests already written.

Step 269: Annotation RPC Methods (12 tests)

Goal: High-level set/get/remove/discover RPCs so agents don't need low-level applyMutation calls.

RPC methods:

• setSemanticAnnotation(nodeId, type, fields) — set or update
• getSemanticAnnotations(nodeId?) — query per-node or project-wide
• removeSemanticAnnotation(nodeId, type) — remove specific type
• getUnannotatedNodes(type?, includeHints?) — find gaps with static analysis hints

MCP tools: whetstone_set_semantic_annotation, whetstone_get_semantic_annotations, whetstone_remove_semantic_annotation, whetstone_get_unannotated_nodes

Implementation partially present in HeadlessAgentRPCHandler.h and MCPServer.h.

Step 270: Annotation Prompt Templates (12 tests)

Goal: MCP prompt templates that guide frontier models through annotation workflow.

Prompts: annotate_intent, annotate_complexity, annotate_risk, annotate_contracts, annotate_full

Each prompt mentions relevant tools (setSemanticAnnotation, getUnannotatedNodes, save sidecar).

getUnannotatedNodes hints: bodyStatements count, sideEffectHint heuristic.

Step 271: Phase 10b Integration Tests (8 tests)

Goal: Full agent annotation session: discover unannotated → set annotations → query → remove → re-query → save sidecar.

Tests cover: multi-function annotation workflow, prompt template content validation, permission enforcement across the pipeline, hint accuracy.

---

Phase 10c: Type System, Execution & Scope Annotations (Steps 272-277)

Implements annotation Subjects 2, 3, and 4 from docs/annotations/.

Step 272: Type System Annotations — Layout & Constraints (12 tests)

Goal: AST nodes for type metadata that drives cross-language projection.

New annotation classes (from Type System and Variance.md):

• BitWidthAnnotation — fields: width (int), e.g. 32 for Java int
• EndianAnnotation — fields: order ("big"|"little")
• LayoutAnnotation — fields: mode ("packed"|"aligned"), alignment (int)
• NullabilityAnnotation — fields: nullable (bool), strategy ("strict"|"nullable")
• VarianceAnnotation — fields: variance ("covariant"|"contravariant"|"invariant")

Deliverables: Class definitions, JSON roundtrip serialization, compact AST integration, 12 tests.

Step 273: Type System Annotations — Identity & Mutability (12 tests)

Goal: Type identity and immutability depth annotations.

New annotation classes:

• IdentityAnnotation — fields: mode ("nominal"|"structural")
• MutAnnotation — fields: depth ("shallow"|"deep"|"interior")
• TypeStateAnnotation — fields: state ("erased"|"reified")

Plus: C++ generator integration — MutAnnotation drives const qualification, IdentityAnnotation adds branding comments. 8 annotation types total for Subject 2.

Step 274: Concurrency Annotations — Primitives & Memory Model (12 tests)

Goal: Execution model annotations for thread safety and hardware semantics.

New annotation classes (from Exec model and Concurrency.md):

• AtomicAnnotation — fields: consistency ("seq_cst"|"relaxed"|"acquire"|"release")
• SyncAnnotation — fields: primitive ("monitor"|"spin"|"semaphore")
• ThreadModelAnnotation — fields: model ("green"|"os"|"fiber")
• MemoryBarrierAnnotation — fields: (marker, no extra fields)

C++ generator: AtomicAnnotation → std::atomic with memory order, SyncAnnotation → std::mutex/std::lock_guard comments.

Step 275: Async, Parallelism & Error Handling Annotations (12 tests)

Goal: Control flow and error model annotations.

New annotation classes:

• ExecAnnotation — fields: mode ("async"|"event"), runtimeHint (string)
• BlockingAnnotation — fields: kind ("io"|"compute")
• ParallelAnnotation — fields: kind ("data"|"task")
• TrapAnnotation — fields: signal (string)
• ExceptionAnnotation — fields: style ("checked"|"unchecked")
• PanicAnnotation — fields: behavior ("abort"|"unwind")

C++ generator: ExecAnnotation(async) → // async: coroutine candidate, ParallelAnnotation(data) → // SIMD-safe / #pragma omp.

Step 276: Scope & Namespace Annotations (12 tests)

Goal: Symbol binding, closure capture, and visibility annotations.

New annotation classes (from Scope & Namespace resolution.md):

• BindingAnnotation — fields: time ("static"|"dynamic")
• LookupAnnotation — fields: mode ("lexical"|"hoisted")
• CaptureAnnotation — fields: strategy ("value"|"ref"|"move")
• VisibilityAnnotation — fields: level ("private"|"internal"|"friend"|"public")
• NamespaceAnnotation — fields: style ("qualified"|"flat")
• ScopeAnnotation — fields: kind ("local"|"global_leaked"|"singleton")

C++ generator: CaptureAnnotation → lambda capture list comment, VisibilityAnnotation → access specifier hint.

Step 277: Phase 10c Integration Tests (8 tests)

Goal: Cross-subject annotation workflows — annotate a module with type, concurrency, and scope metadata, verify compact AST, sidecar roundtrip, and C++ generator output reflects annotations.

Tests cover:

1. Type annotations drive C++ const/alignment output
2. Concurrency annotations produce thread-safety comments
3. Scope annotations influence lambda capture hints
4. All Subject 2-4 annotations survive sidecar save/load
5. Compact AST includes type/exec/scope summaries
6. Mixed annotations on single function (type + concurrency + scope)
7. getSemanticAnnotations returns Subject 2-4 types
8. setSemanticAnnotation works for new types via RPC

---

Phase 10d: Shims, Optimization, Meta-Programming & Policy Annotations (Steps 278-283)

Implements annotation Subjects 5, 6 (gaps), 7, and 8 from docs/annotations/.

Step 278: Shim & Escape Hatch Annotations (12 tests)

Goal: Preserve platform-specific and inexpressible constructs.

New annotation classes (from Shims and Escape Hatches.md):

• IntrinsicAnnotation — fields: instruction (string), arch (string)
• RawAnnotation — fields: language (string), code (string)
• CallingConvAnnotation — fields: convention ("stdcall"|"cdecl"|"fastcall")
• LinkAnnotation — fields: symbolName (string), library (string)
• ShimAnnotation — fields: strategy ("vtable"|"trampoline"|"union_tag"|"cast")
• PointerArithmeticAnnotation — fields: (marker)
• OpaqueAnnotation — fields: reason (string)

Step 279: Platform & Provenance Annotations (12 tests)

Goal: Conditional compilation and transformation history tracking.

New annotation classes:

• TargetAnnotation — fields: platform (string), arch (string)
• FeatureAnnotation — fields: flag (string), enabled (bool)
• OriginalAnnotation — fields: sourceCode (string), sourceLanguage (string)
• MappingAnnotation — fields: history (vector), transformations applied

C++ generator: TargetAnnotation → #ifdef guards, OriginalAnnotation → preserved-source comment block.

Step 280: Optimization Annotation Completion (12 tests)

Goal: Fill gaps in Subject 6 — loop hints, data locality, hardware mapping, safety overrides.

New annotation classes (from 6 optimization and intent.md):

• TailCallAnnotation — fields: (marker, for TCO eligibility)
• LoopAnnotation — fields: hint ("unroll"|"vectorize"|"fuse"), factor (int, optional)
• DataAnnotation — fields: hint ("prefetch"|"restrict")
• AlignAnnotation — fields: bytes (int)
• PackAnnotation — fields: (marker)
• BoundsCheckAnnotation — fields: enabled (bool)
• OverflowAnnotation — fields: behavior ("wrap"|"saturation"|"panic")

C++ generator: LoopAnnotation(vectorize) → #pragma omp simd, AlignAnnotation → alignas(N), PackAnnotation → #pragma pack(push, 1), OverflowAnnotation → wrapping arithmetic comment.

Step 281: Meta-Programming Annotations (12 tests)

Goal: Code-as-data, macro expansion tracking, reflection policies.

New annotation classes (from Meta-Programming & Homoiconicity.md):

• MetaAnnotation — fields: state ("quoted"|"unquoted"), phase ("compile"|"runtime")
• SymbolAnnotation — fields: mode ("gensym"|"interned")
• EvaluateAnnotation — fields: phase ("compile_time"|"runtime")
• TemplateAnnotation — fields: specialization ("trait"|"monomorphize"|"erasure")
• SyntheticAnnotation — fields: generator (string), isStructuralRisk (bool)

Step 282: Strategy & Policy Annotations (12 tests)

Goal: Decision support — multiple valid transformations, guardrails, conflict markers.

New annotation classes (from 8 strategy choice and policy.md):

• PolicyAnnotation — fields: strictness ("high"|"low"), perf ("critical"|"normal"), style ("idiomatic"|"literal"), binaryStable (bool)
• AmbiguityAnnotation — fields: intent (string), options (vector)
• CandidateAnnotation — fields: inferredTypes (vector)
• TradeoffAnnotation — fields: reason (string), safetyCost (string), perfCost (string)
• ChoiceAnnotation — fields: choiceId (string), options (vector)
• DecisionAnnotation — fields: choiceId (string), selection (string), author (string), reason (string)

Step 283: Phase 10d Integration Tests (8 tests)

Goal: Full Subject 5-8 annotation workflow with sidecar persistence and generator output.

Tests cover:

1. Shim annotations on FFI boundary function
2. OriginalAnnotation preserves legacy source through projection
3. Optimization annotations drive C++ pragma output
4. Meta-programming annotations mark macro-expanded code
5. Policy + Choice + Decision annotation workflow (strategy menu)
6. All Subject 5-8 annotations survive sidecar roundtrip
7. setSemanticAnnotation RPC works for all new types
8. Annotation taxonomy completeness check: all 8 subjects represented

---

Phase 10e: Environment Layer (Steps 284-289)

Implements the Environment Layer feature request from docs/annotations/Environment Layer.md.

Step 284: EnvironmentSpec Schema & AST Node (12 tests)

Goal: First-class EnvironmentSpec attached to modules.

New class: EnvironmentSpec (extends ASTNode) Fields:

• envId (string) — "posix_process", "browser", "jvm", "python_cpython", "emacs_runtime"
• envVersion (string, optional)
• capabilities (vector) — required capabilities
• constraints (vector) — forbidden behaviors
• scheduler (string) — "event_loop"|"threads"|"fibers"|"coroutines"|"single_thread"
• memory (string) — "manual"|"raii"|"refcount"|"tracing_gc"|"region"
• bindingTimes (json object) — per-feature: "compile"|"link"|"load"|"runtime"
• exceptions (string) — "unwind"|"checked"|"typed"|"untyped"
• ffi (string) — "c_abi"|"dynamic_linking"|"none"

Deliverables: Class definition, JSON roundtrip, attachment to Module via setEnvironment/getEnvironment RPC, compact AST env summary.

Step 285: Capability Vocabulary & Validation (12 tests)

Goal: Fixed capability vocabulary with validation and per-node capability requirements.

Capability vocabulary (initial set): io.fs, io.net, io.stdinout, clock.time, timers, event_loop, threads, process, signals, modules, dynamic_loading, reflection, gc, jit, ui.dom, editor.emacs, host.imgui

New class: CapabilityRequirement (extends Annotation)

• fields: capability (string), required (bool)
• Attaches to any node that requires a specific env capability

Validation: validateCapabilities(module) checks all CapabilityRequirements against module's EnvironmentSpec.capabilities, returns diagnostics for unmet requirements.

Step 286: Environment-Aware Pipeline Hooks (12 tests)

Goal: Transforms receive EnvironmentSpec and adjust behavior.

Changes to Pipeline:

• runPipeline gains optional env parameter (or reads from module's EnvironmentSpec)
• projectLanguage checks env.scheduler and env.memory to choose lowering strategy
• Diagnostics flag env-incompatible annotations (e.g., @Parallel(Data) in single_thread env)

New diagnostic codes: E05xx for environment-related issues.

Step 287: Environment-Aware Lowering Decisions (12 tests)

Goal: Concrete code generation changes driven by environment model.

Lowering rules:

• env.scheduler=event_loop + ExecAnnotation(async) → callback-based pattern in C++
• env.scheduler=threads + ExecAnnotation(async) → std::async/std::future in C++
• env.memory=tracing_gc → suppress explicit deallocation in C++ output
• env.memory=manual → inject explicit delete calls
• env.exceptions=unwind → try/catch in C++; env.exceptions=typed → std::expected

Tests: Same AST with different EnvironmentSpecs generates different C++ output.

Step 288: Host Boundary AST Nodes (12 tests)

Goal: Explicit nodes for host-environment interactions.

New AST node types:

• HostCall (extends Expression) — fields: capability (string), name (string), maps to explicit env boundary
• ScheduleTask (extends Statement) — fields: queue ("microtask"|"macrotask"|"thread"), body (Statement children)
• ModuleLoad (extends Statement) — fields: moduleName (string), mode ("static"|"dynamic")

Serialization, compact AST, and generator support for all 3 types.

C++ generator: HostCall → capability-namespaced function call, ScheduleTask → thread/task dispatch, ModuleLoad → #include (static) or dlopen (dynamic).

Step 289: Phase 10e Integration Tests (8 tests)

Goal: End-to-end environment-aware workflow.

Tests cover:

1. Set EnvironmentSpec on module, verify in compact AST
2. CapabilityRequirement diagnostic for missing capability
3. Same module + different env → different C++ output (threads vs event_loop)
4. HostCall generates correct capability-namespaced code
5. ScheduleTask + env.scheduler drives dispatch pattern
6. Environment survives sidecar save/load
7. Pipeline with env parameter applies correct lowering
8. Full workflow: set env → annotate → generate → verify output

---

Step & Test Summary

Phase	Steps	Tests	Theme
10a	266-268	32	Semantic annotations + sidecar
10b	269-271	32	Agent annotation interface
10c	272-277	80	Type, execution, scope annotations (Subjects 2-4)
10d	278-283	80	Shims, optimization, meta, policy (Subjects 5-8)
10e	284-289	80	Environment layer
Total	266-289	304

MCP tool count projection: ~42 tools (34 current + 4 sidecar + 4 semantic RPC + environment tools)

---

Key Files to Modify

File	Changes
editor/src/ast/Annotation.h	~50 new annotation classes across Phases 10c-10e
editor/src/ast/Serialization.h	toJson/fromJson dispatch for all new types
editor/src/CompactAST.h	Semantic summary extraction for new annotation subjects
editor/src/HeadlessAgentRPCHandler.h	New RPC methods (sidecar, env, extended setSemanticAnnotation)
editor/src/AgentPermissionPolicy.h	Permissions for new methods
editor/src/MCPServer.h	Tool/prompt registrations
editor/src/SidecarPersistence.h	Already exists, may need extensions
editor/src/ast/ProjectionGenerator.h	Visit methods for generator-relevant annotations
editor/src/ast/CppGeneratorTypes.h	Annotation-driven C++ output
editor/CMakeLists.txt	Test targets for steps 266-289

New files:

• editor/src/EnvironmentSpec.h — EnvironmentSpec class, capability vocabulary, validation
• editor/src/ast/HostBoundary.h — HostCall, ScheduleTask, ModuleLoad AST nodes
• editor/tests/step271_test.cpp through editor/tests/step289_test.cpp

---

Verification

After each phase:

1. Build: cd editor/build-native && cmake .. && make -j$(nproc)
2. Run step tests: ./step26X_test for each step in the phase
3. Verify test counts match expected (12 per step, 8 for integration)
4. Check MCP tool count via tools/list in integration test

Full sprint verification:

• All 304 tests pass across steps 266-289
• Annotation taxonomy covers all 8 subjects from docs/annotations/
• Environment Layer meets acceptance criteria from Environment Layer.md
• Sidecar persistence works for all annotation types
• C++ generator output reflects annotation metadata

---

Sprint 11 Preview (deferred from original Sprint 10)

1. LSP client for headless mode (clangd, pyright, rust-analyzer)
2. Language-specific code actions (Go imports, Rust borrow fixes)
3. GUI overhaul: pin floating windows, sane defaults, usable color scheme
4. Terminal UI mode (TUI) as alternative to ImGui GUI
5. Plugin system: load language support and tools dynamically