diff --git a/ARCHITECT.md b/ARCHITECT.md new file mode 100644 index 0000000..2609478 --- /dev/null +++ b/ARCHITECT.md @@ -0,0 +1,107 @@ +# Whetstone DSL — Architect Notes + +## Core Thesis + +AI+Human teams will always outperform AI alone. Millions of years of cognitive evolution +means humans belong on the team — not as button-pushers on an assembly line, but as +architects, reviewers, and decision-makers where ambiguity, judgment, and reflection matter. + +## The Three Pillars + +### 1. Semantic Annotations as Workflow Routing +Annotations are not just metadata. They are **routing signals** that determine: +- Who handles a task (human architect, human engineer, LLM, SLM, deterministic template) +- What context is needed (local, file-wide, project-wide, cross-project) +- What kind of thinking is required (mechanical, analytical, creative, architectural) +- When to defer (ambiguity too high, risk too high, human review required) + +The annotation taxonomy makes the cost/complexity of each task **knowable before spending +tokens or compute**. This is the core optimization: don't send a PhD to push buttons on +the assembly line. + +### 2. Universal Transpilation +Whetstone's AST is language-agnostic. The vision: +- Ingest legacy code in any supported language +- Annotate it with semantic meaning (intent, complexity, risk, contracts) +- Transpile to modern, safer, more efficient implementations in any target language +- Eventually: describe a problem, the system chooses the tech stack + +This requires broad language coverage. Languages to support (in rough priority): +- **Done:** Python, C++, JavaScript, TypeScript, Java, Rust, Go, Elisp +- **Sprint 11:** Kotlin, C# +- **Near-term:** C, WebAssembly, Common Lisp, Scheme +- **Mid-term:** F#, VB.NET, SQL stacks (PostgreSQL, T-SQL, MySQL) +- **Later:** x86 assembly, ARM assembly, additional as needed + +### 3. Professional GUI for Human-in-the-Loop +The GUI is not an afterthought. If the thesis is that humans must be part of the team, +then the interface must make what's happening **intuitive to understand**. It should look +like a finished product from a professional company. + +Current issues: +- Floating windows need to be docked +- Key combination symbols not implemented +- Color scheme needs work +- Navigation needs to be clean and discoverable + +## The Skeleton AST — How Architects Model Projects + +Before code exists, the architect (human or agent) creates a **skeleton AST**: modules, +functions, and classes with names, signatures, and annotations — but no implementations. +The annotations on skeleton nodes describe: +- **What** it should do: @Intent("parse HTTP headers and extract auth tokens") +- **How hard** it is: @Complexity(high), @Ambiguity(medium) +- **Who should do it**: @Automatability(llm), @Review(required, human) +- **What context is needed**: @ContextWidth(project) +- **What order**: @Priority(high, blockedBy=["auth_module"]) +- **What state it's in**: @ImplementationStatus(skeleton) + +The skeleton + annotations = project specification = task queue. This is the bridge +between "describe a problem" and "execute the solution." The architect doesn't write +code — they write the structure and intent. The orchestration engine reads the +annotations and routes each skeleton to the right worker. + +This also means every project modeling decision is captured as structured data (which +annotations the architect chose, how they decomposed the problem, what they marked +for human review). Post-Sprint 25, this becomes training data for teaching AI how +experienced engineers think about projects before implementation begins. + +## Architecture Invariants + +- **Header-only C++** — all implementation in .h files, no .cpp (except mcp_main.cpp and test files) +- **600-line file limit** — split when approaching, use Extended/Impl pattern +- **12 tests per step, 8 for integration** — consistent test density +- **MCP-first** — every capability exposed as an MCP tool +- **Headless-first** — all functionality works without GUI +- **No external dependencies beyond tree-sitter** — regex parsers for languages without tree-sitter grammars + +## Key Metrics + +- Current: 38+ MCP tools, 67+ annotation types, 8 languages, ~296 steps, ~3000+ tests +- Sprint 11 target: 40+ tools, 10 languages, validation complete +- Sprint 15 target: Workflow orchestration operational, 14+ languages +- Sprint 20 target: Claude Code plugin usable end-to-end, professional GUI +- Sprint 25 target: Self-hosting, legacy code ingestion, tech stack selection + +## What "Done" Looks Like + +A developer says: "I have a legacy C++ codebase. I want to modernize it, add safety +guarantees, and port the networking layer to Rust." + +Whetstone: +1. Ingests the C++ code, builds annotated AST +2. Architect (human or agent) reviews annotations, marks routing decisions +3. Task queue populates: deterministic renames go to templates, straightforward ports + go to SLM, complex architectural decisions surface for human review +4. Engineer approves the queue, work executes in parallel +5. Cross-language transpilation produces the Rust networking layer +6. Validation catches conflicts, diagnostics guide fixes +7. Human reviews the 5% that needed judgment, not the 95% that didn't + +## Sprint Philosophy + +- Quality over speed. Build what's right, not what's fast. +- Languages are additive and quick — keep adding them throughout. +- The orchestration layer is the hard architectural work — get it right. +- The GUI is load-bearing for the thesis — it must be professional. +- Self-hosting is the ultimate proof: Whetstone annotates and transpiles itself. diff --git a/feature-requests.md b/feature-requests.md new file mode 100644 index 0000000..e987a62 --- /dev/null +++ b/feature-requests.md @@ -0,0 +1,239 @@ +# Feature Requests: C++ AST & Parser Gaps + +These are specific C++ language constructs that the current parser/AST cannot represent, identified by examining what would be needed to parse Whetstone's own codebase through its own pipeline. Sprint 11c covers some of these (class, template, lambda basics). The rest are candidates for Sprint 12-13. + +--- + +## Covered by Sprint 11c (in progress) + +These are already planned but noted for completeness: + +- **class/struct declarations** → ClassDeclaration node (fields, methods, inheritance) +- **templates** → GenericType + TypeParameter nodes +- **virtual methods** → MethodDeclaration with isVirtual/isOverride +- **lambdas** → LambdaExpression with capture list + +--- + +## Sprint 12 Candidates: Core C++ Constructs + +### 1. Multiple & Virtual Inheritance + +Whetstone uses diamond inheritance extensively: +```cpp +class KotlinGenerator : public ProjectionGenerator, + public virtual AnnotationVisitorExtended, + public SemannoAnnotationImpl { ... }; +``` + +**Needed:** +- ClassDeclaration.superClasses (vector, not single superClass) +- Inheritance qualifier: `public`, `protected`, `private` +- Virtual inheritance flag per base class +- Diamond resolution representation + +### 2. CRTP (Curiously Recurring Template Pattern) + +```cpp +template +class SemannoAnnotationImpl : public virtual AnnotationVisitorExtended { ... }; +``` + +The AST needs to represent: +- Template class declarations (not just template functions) +- Self-referential type parameters (`class Foo : Base`) +- CRTP as a recognizable pattern for annotation/inference + +### 3. Preprocessor Directives + +Whetstone headers use: +```cpp +#pragma once +#include "ast/ASTNode.h" +#define TEST(name) { ... } +#define CHECK(cond, msg) if (!(cond)) { ... } +``` + +**Needed:** +- Include directive nodes (path, system vs local) +- Pragma nodes +- Macro definition nodes (name, parameters, body) +- Macro expansion tracking (`@Synthetic` annotation could mark expanded nodes) + +### 4. Enum Class (Scoped Enums) + +```cpp +enum class DiagnosticSeverity { Error = 1, Warning = 2, Info = 3, Hint = 4 }; +``` + +**Needed:** +- EnumDeclaration node (name, isScoped, underlyingType) +- EnumMember node (name, value) + +### 5. Namespace Declarations + +```cpp +namespace whetstone { ... } +using namespace std; +``` + +**Needed:** +- NamespaceDeclaration node (name, children) +- UsingDirective node (namespace or specific symbol) + +### 6. Type Aliases & Using Declarations + +```cpp +using json = nlohmann::json; +using Diagnostic = AnnotationValidator::Diagnostic; +typedef std::vector StringVec; +``` + +**Needed:** +- TypeAlias node (name, targetType) +- Nested type access representation (Foo::Bar::Baz) + +### 7. Static Members & Methods + +```cpp +class Foo { + static bool isPowerOf2(int v); + static const int MAX_WIDTH = 128; +}; +``` + +**Needed:** +- MethodDeclaration.isStatic (exists in Sprint 11c plan) +- Static data member representation +- Static method call vs instance method call distinction + +### 8. Const & Constexpr + +```cpp +const std::string& getMessage() const; +constexpr int MAX = 256; +static constexpr bool isPowerOf2(int v) { ... } +``` + +**Needed:** +- Method const qualifier (method doesn't modify `this`) +- constexpr function/variable distinction +- const reference parameter types + +### 9. Smart Pointers & RAII Patterns + +```cpp +std::unique_ptr ast; +std::make_unique(); +auto* raw = new Function(); +``` + +**Needed:** +- Template type instantiation in expressions (not just declarations) +- Smart pointer construction/move as recognizable patterns +- `new`/`delete` expression nodes + +--- + +## Sprint 13 Candidates: Advanced C++ + +### 10. `auto` Type Deduction + +```cpp +auto entry = SemannoParser::parse(emitted); +auto* a = static_cast(anno); +const auto& ct = anno->conceptType; +``` + +**Needed:** +- AutoType node (deferred type resolution) +- `auto*`, `auto&`, `const auto&` variants + +### 11. static_cast / dynamic_cast / reinterpret_cast + +```cpp +auto* a = static_cast(anno); +``` + +**Needed:** +- CastExpression node (kind: static/dynamic/reinterpret/const, targetType, operand) +- This is critical for Whetstone's own downcast-heavy validation pattern + +### 12. Range-Based For Loops + +```cpp +for (const auto& role : node->childRoles()) { ... } +for (auto* child : node->allChildren()) { ... } +``` + +**Needed:** +- RangeForStatement node (variable, iterable, body) +- Distinguish from C-style for loops + +### 13. Structured Bindings (C++17) + +```cpp +auto [key, value] = pair; +``` + +**Needed:** +- StructuredBinding node (names[], initializer) + +### 14. std::initializer_list + +```cpp +bool isOneOf(const std::string& val, std::initializer_list options); +``` + +**Needed:** +- Initializer list expression node +- Brace-enclosed initializer `{"a", "b", "c"}` + +### 15. Operator Overloading + +```cpp +bool operator<(const DiagnosticKey& other) const; +friend std::ostream& operator<<(std::ostream& os, const Foo& f); +``` + +**Needed:** +- OperatorDeclaration node (operator symbol, parameters) +- Friend declaration support + +### 16. Exception Handling + +```cpp +try { ... } catch (const std::exception& e) { ... } +throw std::runtime_error("error"); +noexcept specifier +``` + +**Needed:** +- TryCatchStatement node +- ThrowExpression node +- Noexcept specifier on functions + +### 17. STL Container Usage Patterns + +```cpp +std::vector items; +std::map present; +std::set seen; +``` + +**Needed:** +- Generic type instantiation in variable declarations +- Container method calls as recognizable patterns (push_back, insert, find, count, etc.) +- Iterator patterns + +--- + +## Self-Hosting Milestone + +When items 1-11 are implemented, Whetstone should be able to parse its own header files through its own pipeline. This would be the ultimate integration test: annotate Whetstone's source with its own annotation taxonomy, validate it with its own validator, and generate projections of itself. + +Key files to target first (simplest to most complex): +1. `AnnotationConflictExtended.h` — simple structs and free functions +2. `AnnotationValidatorExtended.h` — single class, static_cast pattern +3. `TransformEngineExtended.h` — inheritance, virtual methods +4. `SemannoAnnotationImpl.h` — CRTP, virtual inheritance (hardest) diff --git a/progress.md b/progress.md index a73502b..ba12114 100644 --- a/progress.md +++ b/progress.md @@ -880,3 +880,33 @@ Full Phase 11a integration validation now passes end-to-end. - Semanno sidecar save/load integration with expected counts. - Language comment prefix conventions. - Marker annotation emission without property parentheses. + +--- + +# Roadmap Planning — Sprints 12-25+ + +## Status: Planning Complete (Sprints 12-19 detailed, 20-25 in roadmap.md) + +Sprint 11 revised: Phase 11e changed from "Training Data Pipeline" to "Workflow +Annotation Foundation" — routing annotation types (Subject 9), skeleton AST, and +inference-to-routing bridge replace training data export (deferred to post-25). + +### Planning documents created: +- `ARCHITECT.md` — Core thesis, skeleton AST concept, architecture invariants, key metrics +- `roadmap.md` — Sprint 12-25+ high-level roadmap with post-25 training data notes +- `sprint11_plan.md` — Revised: 11e is now workflow annotation foundation +- `sprint12_plan.md` — Workflow Model + C++ Depth (22 steps, ~248 tests, 56+ tools) +- `sprint13_plan.md` — GUI Overhaul Phase 1 (19 steps, ~212 tests) +- `sprint14_plan.md` — Languages: C, WebAssembly, Common Lisp, Scheme (17 steps, ~188 tests) +- `sprint15_plan.md` — Orchestration Engine (16 steps, ~180 tests, 68+ tools) +- `sprint16_plan.md` — C++ Depth + Self-Hosting Phase 1 (11 steps, ~124 tests) +- `sprint17_plan.md` — Languages: F#, VB.NET, SQL dialects (12 steps, ~136 tests) +- `sprint18_plan.md` — Claude Code Plugin + MCP Workflow Tools (11 steps, ~124 tests, 75+ tools) +- `sprint19_plan.md` — GUI Phase 2: Workflow Visualization (10 steps, ~112 tests) + +### Key architectural decisions: +1. **Skeleton AST** = project specification before code exists (annotations on empty functions/classes) +2. **Annotations are routing signals** — @Automatability, @ContextWidth, @Ambiguity determine worker dispatch +3. **Orchestrator prepares context but never calls LLMs** — model-agnostic, cost-controlled +4. **Plugin is MCP protocol** — not a proprietary binary, any MCP client can drive workflows +5. **Training data deferred to post-25** — real workflow decisions are better training signal than synthetic pairs diff --git a/roadmap.md b/roadmap.md new file mode 100644 index 0000000..a6090ff --- /dev/null +++ b/roadmap.md @@ -0,0 +1,111 @@ +# Whetstone DSL — Sprint Roadmap (Sprints 11-25) + +## Sprint 11: Dial It In (in progress) +Finish the annotation foundation: validation for all 67 types, parser deepening for +class/async/lambda, Kotlin + C# languages, and generalized annotation inference. +Replaces training data export with workflow-relevant annotation types. + +## Sprint 12: Workflow Model + C++ Depth +Introduce WorkItem, TaskAnnotation, and routing annotation types (@ContextWidth, +@ReviewRequired, @DeferToHuman, @Automatable). Add C++ constructs from feature-requests +items 1-5 (multiple inheritance, CRTP, preprocessor, enum class, namespaces). + +## Sprint 13: GUI Overhaul Phase 1 +Dock all floating windows, implement key combination symbols, professional color scheme, +clean navigation. The human-in-the-loop thesis requires a GUI that makes the system +intuitive to understand. Should look like a finished product. + +## Sprint 14: Language Batch 1 — C, WebAssembly, Common Lisp, Scheme +C is critical for legacy ingestion. WASM for web deployment targets. Lisp/Scheme for +functional paradigm coverage and macro system representation in the AST. + +## Sprint 15: Orchestration Engine +Routing logic: annotation → dispatch decision (deterministic/SLM/LLM/human). Task queue +with priority and dependency management. Context size estimation from annotations. +Review gates and approval workflows. + +## Sprint 16: C++ Depth + Self-Hosting Phase 1 +Feature-requests items 6-11 (type aliases, static members, const/constexpr, smart +pointers, auto type deduction, casts). Begin self-hosting: parse simplest Whetstone +headers through our own pipeline. + +## Sprint 17: Language Batch 2 — .NET Family + SQL +F#, VB.NET for server-side .NET coverage. SQL dialect support (PostgreSQL, T-SQL, MySQL) +for full-stack transpilation scenarios. + +## Sprint 18: Claude Code Plugin + MCP Workflow Tools +Workflow-aware MCP tools that understand task queues and routing. Claude Code plugin that +an engineer can install and immediately use for annotation-driven project workflows. +End-to-end: describe project → annotate → route → execute. + +## Sprint 19: GUI Phase 2 — Workflow Visualization +Task queue visualization, annotation overlay on code, workflow status dashboard, human +review interface. The architect should see the project as a structured flow, not a +wall of text. + +## Sprint 20: Legacy Code Ingestion +Parse legacy code (old C, old C++, old Java), infer annotations, suggest modernization +paths. Safe transpilation to modern equivalents with human review at decision points. + +## Sprint 21: Cross-Language Transpilation Engine +Beyond syntactic projection: algorithm-level translation guided by semantic annotations. +Safety and efficiency annotations steer the translation. Verify behavioral equivalence +where possible. + +## Sprint 22: Language Batch 3 — Assembly + Remaining Gaps +x86 and ARM assembly representation. C++ feature-requests items 12-17 (range-for, +structured bindings, operator overloading, exception handling, STL patterns). +Fill remaining language gaps identified during real-world use. + +## Sprint 23: Architect Mode + Tech Stack Selection +Describe a problem in natural language → system suggests tech stack, project structure, +and annotation skeleton. Architecture-level templates. Project scaffolding from +annotated descriptions. + +## Sprint 24: Security + Static Analysis +Security annotations and validation. Vulnerability pattern detection (OWASP top 10). +Dependency audit annotations. Security review as a routing category in the +orchestration engine. + +## Sprint 25: Integration, Self-Hosting, Polish +Full self-hosting: Whetstone parses, annotates, and transpiles its own source. +End-to-end workflow testing across all supported languages. Performance optimization. +Release preparation. + +--- + +## Post-25: Training Data Harvest + Project Modeling Intelligence + +### Training Data from Real Workflows +By Sprint 25, the system has accumulated real-world signal across every layer: +- **Architect modeling decisions**: skeleton ASTs, annotation choices, how humans decompose + problems into annotated task queues. This is the rarest and most valuable data — how + experienced engineers *think about* projects before code exists. +- **Routing outcomes**: which annotation patterns led to correct dispatch (deterministic vs + SLM vs LLM vs human), and which routing decisions had to be corrected by humans. +- **Transpilation pairs**: verified source→target translations across 20+ languages, validated + by the system and reviewed by humans. Not synthetic — real production transpilations. +- **Human review decisions**: what architects approved, rejected, modified. The diff between + what the system proposed and what the human accepted is pure training signal. +- **Orchestration traces**: full workflow recordings from project description through + completion, including where the system deferred to humans and why. + +### Project Modeling as a Learnable Skill +The architect phase (skeleton AST + annotations = project specification) generates a +unique dataset: how humans model projects *before implementation begins*. This includes: +- How a problem description maps to module decomposition +- Which annotations an experienced architect applies to which skeleton nodes +- How dependency structure is decided +- Where the architect marks "this needs human review" vs "this is automatable" +- How the architect estimates complexity, risk, and context requirements + +This is the data needed to eventually train a model that can propose project structures +from problem descriptions (Sprint 23's "Tech Stack Selection"), but grounded in real +human decisions rather than synthetic examples. + +### Sprint 26+: Fine-Tuning Pipeline +- Export accumulated workflow decisions, routing outcomes, and transpilation pairs +- Build fine-tuning datasets for routing model improvement +- Build fine-tuning datasets for project modeling (skeleton generation from descriptions) +- Feedback loop: improved models → better routing → better outcomes → better training data +- The system improves its own judgment from historical human decisions diff --git a/sprint11_plan.md b/sprint11_plan.md new file mode 100644 index 0000000..44b7d06 --- /dev/null +++ b/sprint11_plan.md @@ -0,0 +1,616 @@ +# Sprint 11 Plan: Dial It In + +## Context + +Sprint 10 completed the semantic annotation taxonomy (67+ types across 8 subjects) and environment layer, bringing the test count to 200/200 (steps 284-289). The agent API surface is mature: 38 MCP tools, headless mode, 10-phase annotation taxonomy, environment-aware code generation. + +Sprint 11 closes all remaining gaps: 55 annotation types are silently dropped during code generation (only 12 have codegen visitors), only 5 types have validation rules, parsers don't handle classes/generics/async, and the training data infrastructure exists but only covers memory annotations. Sprint 11 closes all these gaps, adds Kotlin and C# as new languages, introduces the Semanno inline comment standard, and builds the machinery for exporting training data. + +**What's already built:** +- 67+ annotation AST classes in Annotation.h (Subjects 1-8) +- 8 language parsers (Python, C++, Elisp, JavaScript, TypeScript, Java, Rust, Go) +- 8 language generators with ProjectionGenerator visitor dispatch +- 38 MCP tools, sidecar persistence, environment layer +- AnnotationValidator with 5 memory-only validation rules +- AnnotationConflict with ancestor-only same-type detection +- TransformEngine with constant folding + DCE +- MemoryStrategyInference for memory annotation inference +- TraceGenerator with 6 scenario types + +**Deliverables:** +- **0 annotation types silently dropped** during codegen (currently 55 are dropped) +- **10 languages** (currently 8): + Kotlin, C# +- **9 new AST node types**: ClassDeclaration, InterfaceDeclaration, MethodDeclaration, GenericType, TypeParameter, AsyncFunction, AwaitExpression, LambdaExpression, DecoratorAnnotation +- **All 10 parsers deepened** with class/async/lambda support +- **Semanno inline comment format**: `// @semanno:type(key=value)` — parseable, roundtrippable, language-aware +- **Validation rules for all annotation types** (E0600-E1299) +- **Cross-type conflict detection** (e.g., @Pure + @Blocking, @Atomic + @Sync) +- **Generalized annotation inference** (all 8 subjects, not just memory) +- **Training data export** to HuggingFace/OpenAI/HuggingFace formats +- **40+ MCP tools** (currently 38) + +--- + +## Phase 11a: Semanno Format + Annotation Codegen (Steps 290-296) + +*All 55 previously-dropped annotation types now generate Semanno inline comments in all generators.* + +### Step 290: SemannoFormat.h — Comment Format Standard (12 tests) +**Goal:** Define and implement the `@semanno:type(key=value)` inline comment format as a parseable, roundtrippable standard. + +**New file: `editor/src/SemannoFormat.h`** (~600 lines) +- `SemannoEmitter::emit(const ASTNode* anno) -> string` — emits `@semanno:type(key="value")` for all 67+ annotation types +- `SemannoParser::parse(string line) -> SemannoEntry{type, properties}` — parses any comment format (`//`, `#`, `;;`, etc.) +- `SemannoParser::isSemannoComment(line) -> bool` — quick detection +- Internal helpers: value escaping, vector serialization (`;`-separated) + +**Tests:** emit/parse roundtrip for each annotation subject, property escaping, multi-property annotations. + +### Step 291: ProjectionGenerator — Subject 2-4 Visitors (12 tests) +**Goal:** Add dispatch branches for type system, concurrency, and scope annotations. + +**New file: `editor/src/ast/AnnotationVisitors.h`** (~150 lines) +- `AnnotationVisitorExtended` — pure virtual interface with 56 visitor methods for Subjects 2-8 + +**Modifies: `editor/src/ast/ProjectionGenerator.h`** +- Include `AnnotationVisitors.h`, inherit from `AnnotationVisitorExtended` +- Add 24 dispatch branches in `dispatchGenerate()` for Subject 2-4 types +- Default implementations return `""` for backward compatibility + +**Tests:** dispatch recognizes all Subject 2-4 annotation types, returns non-empty for visitors with implementations. + +### Step 292: ProjectionGenerator — Subject 5-8 + Semantic Visitors (12 tests) +**Goal:** Complete the dispatch table — no annotation type returns "Unknown". + +**Modifies: `editor/src/ast/ProjectionGenerator.h`** +- Add 32 more dispatch branches for Subjects 5-8, semantic core, and CapabilityRequirement +- New AST node dispatch: ClassDeclaration, InterfaceDeclaration, MethodDeclaration, GenericType, TypeParameter, AsyncFunction, AwaitExpression, LambdaExpression, DecoratorAnnotation + +**Tests:** exhaustive dispatch test — AST with one of each 56 new types, none returns "Unknown". + +### Step 293: Generator Implementations — Python/C++/Rust/Go (12 tests) +**Goal:** All 4 generators emit Semanno comments for every annotation type. + +**New file: `editor/src/SemannoAnnotationImpl.h`** (~200 lines) +- CRTP mixin `SemannoAnnotationImpl` — provides default implementations for all 56 visitor methods +- Each method calls `commentPrefix() + SemannoEmitter::emit(anno)` +- Derived generators only need to provide `commentPrefix()` (e.g., `"# "` for Python, `"// "` for C++) + +**Modifies:** `PythonGenerator.h`, `CppGenerator.h`, `RustGenerator.h`, `GoGenerator.h` +- Each adds `public SemannoAnnotationImpl` mixin +- Each adds `std::string commentPrefix() const` method + +**Tests:** each generator emits Semanno comments for mixed old+new annotations. + +### Step 294: Generator Implementations — Java/JS/Elisp (12 tests) +**Goal:** Remaining 3 generators get the same SemannoAnnotationImpl mixin. + +**Modifies:** `JavaGenerator.h`, `JavaScriptGenerator.h`, `ElispGenerator.h` +- Same mixin pattern: inherit `SemannoAnnotationImpl`, add `commentPrefix()` +- Elisp uses `";; "` prefix + +**Tests:** each generator emits Semanno comments, parse-back roundtrip works. + +### Step 295: Semanno Sidecar Integration (12 tests) +**Goal:** Save/load `.semanno` sidecar files alongside `.ast.json` sidecars. + +**New file: `editor/src/SemannoSidecar.h`** (~150 lines) +- Format: `L: @semanno:type(key=value)` per line +- `saveSemannoSidecar(workspaceRoot, filePath, module)` — walks AST, emits annotations +- `loadSemannoSidecar(workspaceRoot, filePath)` — parses sidecar back to entries + +**Modifies:** `SidecarPersistence.h` — import SemannoSidecar +**Modifies:** `HeadlessAgentRPCHandler.h` — `exportSemanno` RPC method + +**Tests:** save/load roundtrip, alongside `.ast.json` sidecar, RPC method. + +### Step 296: Phase 11a Integration Tests (8 tests) +**Goal:** Full pipeline verification — no annotation type is silently dropped. + +**Tests cover:** +1. Parse → annotate with Subject 2-8 → generate → verify Semanno comments present +2. Cross-language: annotations preserved through projection +3. All 10 generators produce non-empty Semanno for every annotation type (exhaustive) +4. Semanno roundtrip: generate code → parse comments back → match original annotations +5. Mixed legacy + new annotations in single module +6. Semanno sidecar alongside AST sidecar +7. Generator comment prefix matches target language convention +8. Empty annotations (markers like @TailCall) produce correct Semanno + +--- + +## Phase 11b: Validation & Conflict Completion (Steps 297-301) + +*Extends validation from 5 to 67 types, adds cross-type conflict detection.* + +### Step 297: AnnotationValidator — Subject 2-4 Rules (12 tests) +**Goal:** Validation rules for type system, concurrency, and scope annotations. + +**New file: `editor/src/AnnotationValidatorExtended.h`** (~300 lines) +- `AnnotationValidatorExtended::validateExtended(root) -> vector` +- New diagnostic codes: E0600-E0699 (type system), E0700-E0799 (concurrency), E0800-E0899 (scope) + +**Rules:** +- E0600: BitWidth must be power of 2 (1, 2, 4, 8, 16, 32, 64, 128) +- E0601: Endian order must be "big" or "little" +- E0602: Layout mode must be "packed" or "aligned" +- E0603: Nullability strategy must be "strict" or "nullable" +- E0700: Atomic consistency must be valid ordering (seq_cst, relaxed, acquire, release) +- E0701: Sync primitive must be "monitor", "spin", or "semaphore" +- E0704: ExecAnnotation(async) needs ThreadModel on module or ancestor +- E0800: Binding time must be "static" or "dynamic" +- E0804: Visibility level must be standard keyword + +**Modifies:** `AnnotationValidator.h` — include extended, call new validators. + +**Tests:** each rule with valid/invalid inputs, structured diagnostic code verification. + +### Step 298: AnnotationValidator — Subject 5-8 Rules (12 tests) +**Goal:** Complete validation coverage for remaining 32 annotation types. + +**Modifies: `AnnotationValidatorExtended.h`** — add rules for remaining types. +- E0900: CallingConv convention must be "stdcall", "cdecl", or "fastcall" +- E0901: Shim strategy must be valid (vtable, trampoline, union_tag, cast) +- E1000: LoopAnnotation hint must be "unroll", "vectorize", or "fuse" +- E1001: AlignAnnotation bytes must be power of 2 +- E1100: Meta state must be "quoted" or "unquoted" +- E1200: Policy strictness must be "high" or "low" + +**Tests:** each Subject 5-8 rule, no false positives on valid annotations. + +### Step 299: Cross-Type Conflict Detection (12 tests) +**Goal:** Detect semantic conflicts between different annotation families on the same node. + +**New file: `editor/src/AnnotationConflictExtended.h`** (~200 lines) +- `collectCrossTypeConflicts(node) -> vector` + +**Conflict pairs:** +1. `@Pure` + `@Blocking` — purity violated by blocking behavior +2. `@Atomic` + `@Sync` — conflicting synchronization models +3. `@Capture(move)` + `@Owner(Shared_ARC)` — move semantics incompatible with shared ref counting +4. `@ConstExpr` + `@Exec(async)` — compile-time eval can't be async +5. `@Inline(Always)` + `@TailCall` — inlining defeats tail call optimization +6. `@Pack` + `@Layout(aligned)` — packing and alignment contradict + +**Modifies:** `AnnotationConflict.h` — include extended. + +**Tests:** each conflict pair detected, no false positives, recursive detection. + +### Step 300: TransformEngine Enhancements (12 tests) +**Goal:** New transforms that interact with the annotation taxonomy. + +**New file: `editor/src/TransformEngineExtended.h`** (~250 lines) +- `floatConstantFolding()` — folds BinaryOperation on FloatLiterals +- `deadVariableElimination()` — removes unreferenced Variable declarations +- `annotationAwareOptimize()` — respects @OptimizationLock (skip), @BoundsCheck (preserve) + +**Tests:** float folding, dead var removal, OptimizationLock blocks optimization, BoundsCheck preserved. + +### Step 301: Phase 11b Integration Tests (8 tests) +**Goal:** Full pipeline with validation and conflict detection. + +**Tests cover:** +1. Full pipeline with validation: conflicting annotations → correct E-codes +2. No regressions on original E01xx-E05xx diagnostics +3. Structured diagnostics include all new E06xx-E12xx codes +4. Cross-type conflicts reported with clear messages +5. Annotation-aware optimization respects locks +6. Combined validation + conflict + transform pipeline +7. All 67 annotation types have at least one validation rule +8. Quick-fix suggestions for common validation errors + +--- + +## Phase 11c: Parser Deepening (Steps 302-308) + +*9 new AST node types, all 8 existing parsers deepened with class/async/lambda support.* + +### Step 302: New AST Nodes — Class/Interface/Generic (12 tests) +**Goal:** Structured types for OOP and generic programming. + +**New file: `editor/src/ast/ClassDeclaration.h`** (~100 lines) +- `ClassDeclaration` — name, superClass, isAbstract; children: interfaces, fields, methods, annotations +- `InterfaceDeclaration` — name; children: methods, annotations +- `MethodDeclaration` (extends Function) — className, isStatic, visibility, isOverride, isVirtual + +**New file: `editor/src/ast/GenericType.h`** (~60 lines) +- `GenericType` — baseName; children: typeParameters +- `TypeParameter` — name, constraint + +**Tests:** construction, property access, child roles, serialization roundtrip. + +### Step 303: New AST Nodes — Async/Lambda/Decorator (12 tests) +**Goal:** Modern language constructs as first-class AST nodes. + +**New file: `editor/src/ast/AsyncNodes.h`** (~100 lines) +- `AsyncFunction` (extends Function) — isAsync flag +- `AwaitExpression` — children: expression +- `LambdaExpression` — captureList (vector); children: parameters, body +- `DecoratorAnnotation` — name; children: arguments + +**Tests:** construction, serialization, compact AST names. Can run in parallel with Step 302. + +### Step 304: Serialization + Dispatch for New Nodes (12 tests) +**Goal:** Full JSON roundtrip and generator dispatch for all 9 new node types. + +**Modifies: `editor/src/ast/Serialization.h`** +- `propertiesToJson()` — 9 new branches for new node types +- `createNode()` — 9 new factory entries +- `setPropertiesFromJson()` — 9 new deserialization branches + +**Modifies: `editor/src/ast/ProjectionGenerator.h`** +- Add visitor methods + `dispatchGenerate()` branches for all 9 new node types + +**Tests:** JSON roundtrip for each type, dispatch returns non-empty, nested structures. + +### Step 305: Python + Java Parser Deepening (12 tests) +**Goal:** Parse classes, async, lambdas from Python and Java source. + +**Modifies: `editor/src/ast/PythonParser.h`** +- `class_definition` → ClassDeclaration +- `async def` → AsyncFunction +- `await` → AwaitExpression +- `lambda` → LambdaExpression +- `@decorator` → DecoratorAnnotation + +**Modifies: `editor/src/ast/JavaParser.h`** +- `class_declaration` → ClassDeclaration +- `interface_declaration` → InterfaceDeclaration +- `type_parameters` → GenericType + +**Tests:** backward compatibility (existing function-level tests pass), new constructs parsed. + +### Step 306: TypeScript/JavaScript + Rust Parser Deepening (12 tests) +**Goal:** Parse classes, async/await, arrow functions, traits. + +**Modifies: `editor/src/ast/JavaScriptParser.h`** +- class declarations, async/await, arrow functions → LambdaExpression + +**Modifies: `editor/src/ast/RustParser.h`** +- struct → ClassDeclaration, impl → MethodDeclaration +- trait → InterfaceDeclaration, async fn → AsyncFunction, closures → LambdaExpression + +**Tests:** each new construct parsed correctly, mixed with existing constructs. + +### Step 307: Go + C++ + Elisp Parser Deepening (12 tests) +**Goal:** Complete parser deepening for remaining 3 languages. + +**Modifies: `editor/src/ast/GoParser.h`** +- struct/interface types, method receivers, goroutines + +**Modifies: `editor/src/ast/CppParser.h`** +- class/struct → ClassDeclaration, templates → GenericType, virtual methods → MethodDeclaration, lambdas → LambdaExpression + +**Modifies: `editor/src/ast/ElispParser.h`** +- lambda forms → LambdaExpression + +**Tests:** each language's new constructs parsed, no regressions. + +### Step 308: Generator Updates + Phase 11c Integration (8 tests) +**Goal:** All 8 generators produce language-appropriate output for new AST node types. + +**Modifies:** All 8 generators — add visitor implementations for ClassDeclaration, InterfaceDeclaration, MethodDeclaration, GenericType, TypeParameter, AsyncFunction, AwaitExpression, LambdaExpression, DecoratorAnnotation. + +**Tests:** +1. Parse class → generate → output has language-appropriate class syntax +2. Async roundtrip: parse async def → generate → contains async def +3. Generic type roundtrip +4. Lambda/closure roundtrip +5. Decorator roundtrip (Python → Python) +6. Class with methods and fields +7. Interface with abstract methods +8. Mixed: class + async + lambda in single module + +--- + +## Phase 11d: New Languages — Kotlin + C# (Steps 309-313) + +*Brings total from 8 to 10 language parsers and generators.* + +### Step 309: Kotlin Parser (12 tests) +**Goal:** Parse Kotlin source into Whetstone AST. + +**New file: `editor/src/ast/KotlinParser.h`** (~400 lines) +- Regex-based parser (no tree-sitter dependency) +- Parses: `fun`, `suspend fun` (→ AsyncFunction), `class`, `data class` (→ ClassDeclaration), `val`/`var` (→ Variable) +- `parseKotlin()` and `parseKotlinWithDiagnostics()` entry points + +**Modifies:** `Parser.h` — include KotlinParser.h +**Modifies:** `Pipeline.h` — add `"kotlin"` routing in parse() + +**Tests:** function parsing, class parsing, suspend function → AsyncFunction, data class, type inference. + +### Step 310: Kotlin Generator (12 tests) +**Goal:** Generate idiomatic Kotlin from Whetstone AST. + +**New file: `editor/src/ast/KotlinGenerator.h`** (~500 lines) +- Extends `ProjectionGenerator` and `SemannoAnnotationImpl` +- Kotlin syntax: `fun`, `val`, `listOf()`, `for..in`, `when`, nullable `?` +- Type mapping: Int, Double, String, Boolean, Unit, Array<>, Pair<> + +**Modifies:** `Generator.h` — include KotlinGenerator.h +**Modifies:** `Pipeline.h` — add `"kotlin"` routing in generate() + +**Tests:** function generation, class output, cross-language: Python → Kotlin, Kotlin → Java projection. + +### Step 311: C# Parser (12 tests) +**Goal:** Parse C# source into Whetstone AST. + +**New file: `editor/src/ast/CSharpParser.h`** (~450 lines) +- Regex-based parser (no tree-sitter dependency) +- Parses: methods (with visibility/return-type detection), `async` methods (→ AsyncFunction), `class`, `interface` +- Handles `using` directives, `namespace` wrappers + +**Modifies:** `Parser.h` — include CSharpParser.h +**Modifies:** `Pipeline.h` — add `"csharp"` routing in parse() + +**Tests:** method parsing, class parsing, interface parsing, async methods, generics. + +### Step 312: C# Generator (12 tests) +**Goal:** Generate idiomatic C# from Whetstone AST. + +**New file: `editor/src/ast/CSharpGenerator.h`** (~500 lines) +- Extends `ProjectionGenerator` and `SemannoAnnotationImpl` +- C# syntax: `public void`, Allman braces, `foreach..in`, semicolons +- Type mapping: int, float, double, string, bool, void, List<>, HashSet<>, Dictionary<> + +**Modifies:** `Generator.h` — include CSharpGenerator.h +**Modifies:** `Pipeline.h` — add `"csharp"` routing in generate() + +**Tests:** method generation, class output, cross-language: Java → C#, C# → TypeScript projection. + +### Step 313: New Languages Integration + MCP (8 tests) +**Goal:** Full integration of Kotlin + C# across the pipeline. + +**Modifies:** +- `MCPServer.h` — MCP describes 10 languages in tool descriptions +- `CrossLanguageProjector.h` — Kotlin + C# in adaptReclaimStrategy, generic annotation cloning +- `MemoryStrategyInference.h` — Kotlin defaults (GC/Tracing) +- `HeadlessAgentRPCHandler.h` — Kotlin + C# accepted in parse/generate routes + +**Tests:** +1. MCP tools/list mentions Kotlin and C# +2. Cross-language projection: Python → Kotlin with annotation preservation +3. Cross-language projection: Java → C# with annotation preservation +4. Pipeline.run() with Kotlin source +5. Pipeline.run() with C# source +6. MemoryStrategyInference returns correct defaults for Kotlin +7. All 10 generators registered and produce output +8. Final language count verification: 10 parsers, 10 generators + +--- + +## Phase 11e: Workflow Annotation Foundation (Steps 314-319) + +*Generalized inference engine + routing annotation types + skeleton AST support. +Lays the groundwork for Sprint 12's workflow model.* + +### Step 314: Generalized Annotation Inference Engine (12 tests) +**Goal:** Infer annotations across all 8 subjects, not just memory. + +**New file: `editor/src/AnnotationInference.h`** (~400 lines) +- `AnnotationInference::inferAll(root) -> vector` +- `InferredAnnotation` struct: annotationType, targetNodeId, confidence, reasoning +- Delegates memory annotations to existing `MemoryStrategyInference` (backward compat) + +**Pattern-based rules:** +- loops → @Loop +- async functions → @Exec(async) +- class methods → @Visibility +- recursive functions → @TailCall +- pure functions (no side effects) → @Pure +- try/catch patterns → @Exception +- goroutines/parallel patterns → @Parallel +- nested complexity → @Complexity + +**Tests:** each pattern detected correctly, confidence scores reasonable, memory delegation works. + +### Step 315: Routing Annotation Types (12 tests) +**Goal:** New annotation types that describe *how a task should be handled*, not just +what the code does. These are the vocabulary Sprint 12's workflow model will route on. + +**Modifies: `editor/src/ast/Annotation.h`** — 6 new annotation classes (Subject 9: Workflow) +- `ContextWidthAnnotation` — width: "local" | "file" | "project" | "cross-project" + (how much context is needed to implement or review this node) +- `ReviewAnnotation` — required: bool, reviewer: "human" | "agent" | "either", + reason: string (whether this task needs human review and why) +- `AmbiguityAnnotation` — level: "none" | "low" | "medium" | "high", + description: string (how ambiguous the requirements are) +- `AutomatabilityAnnotation` — strategy: "deterministic" | "template" | "slm" | "llm" | "human", + confidence: float (what kind of worker should handle this) +- `PriorityAnnotation` — level: "critical" | "high" | "medium" | "low", + blockedBy: vector (task ordering hints) +- `ImplementationStatusAnnotation` — status: "skeleton" | "partial" | "complete" | "needs-review", + assignee: string (tracks where a node is in the workflow) + +**Modifies: `editor/src/ast/Serialization.h`** — propertiesToJson/createNode/setPropertiesFromJson +for all 6 new types +**Modifies: `editor/src/CompactAST.h`** — workflow annotations in extractSemanticSummary +**Modifies: `editor/src/SidecarPersistence.h`** — workflow annotations in isSemanticAnnotation + +**Tests:** construction, property access, JSON roundtrip, compact AST summary, sidecar +recognition, validation of enum values. + +### Step 316: Skeleton AST Support (12 tests) +**Goal:** Modules and functions with annotations but no implementation — the architect's +project specification. A skeleton AST is the project model before code exists. + +**New file: `editor/src/SkeletonAST.h`** (~300 lines) +- `createSkeletonModule(name, language) -> Module*` — empty module with metadata +- `addSkeletonFunction(module, name, params, returnType, annotations) -> Function*` + — function with signature and annotations but empty body (body = single + PlaceholderStatement or no children) +- `addSkeletonClass(module, name, methods[], fields[]) -> ClassDeclaration*` + — class structure with method signatures but no implementations +- `isSkeletonNode(node) -> bool` — checks if a node has no implementation +- `getSkeletonSummary(module) -> SkeletonSummary` — counts: total nodes, + implemented, skeleton, by-annotation-type +- `skeletonToTaskList(module) -> vector` — converts skeleton nodes + into a flat task list with routing annotations extracted + +**SkeletonTask struct:** +- nodeId, nodeName, nodeType +- contextWidth (from @ContextWidth or inferred) +- automatability (from @Automatability or inferred) +- priority (from @Priority or default) +- reviewRequired (from @Review or inferred from @Ambiguity) +- status (from @ImplementationStatus) +- dependencies (from @Priority.blockedBy) + +**Tests:** create skeleton module, add skeleton functions with annotations, skeleton +detection, summary counts, task list generation, annotation extraction, empty module +handling, dependency ordering. + +### Step 317: Architect Tooling — RPC + MCP (12 tests) +**Goal:** Expose skeleton creation and project modeling via RPC and MCP tools. + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** +- `createSkeleton` — create a new skeleton module with language and description +- `addSkeletonNode` — add a function/class skeleton with annotations to a module +- `getProjectModel` — return the skeleton summary + task list for a buffer +- `inferAnnotations` — run AnnotationInference on a buffer, return suggestions + +**Modifies: `editor/src/AgentPermissionPolicy.h`** +- `getProjectModel` / `inferAnnotations` as read-only +- `createSkeleton` / `addSkeletonNode` as mutation (Refactor/Generator) + +**Modifies: `editor/src/MCPServer.h`** — registerWorkflowTools() +- `whetstone_create_skeleton` — create skeleton module +- `whetstone_add_skeleton_node` — add annotated skeleton function/class +- `whetstone_get_project_model` — get skeleton summary + task list +- `whetstone_infer_annotations` — auto-infer annotations on existing code + +**Tool count:** 42+ (was 38): +4 workflow tools + +**Tests:** RPC dispatch, MCP tool registration, skeleton creation through MCP, +project model response structure, infer annotations on Python code, permission +enforcement. + +### Step 318: Inference-to-Routing Bridge (12 tests) +**Goal:** Connect annotation inference to routing decisions. When inference runs on +code (or a skeleton is created manually), the routing annotations should be derivable. + +**Modifies: `editor/src/AnnotationInference.h`** +- `inferRoutingAnnotations(node) -> vector` — infers workflow + annotations from code patterns: + - High cyclomatic complexity → @Ambiguity(high) + @Review(required, human) + - Simple getter/setter → @Automatability(deterministic) + - Cross-file dependencies detected → @ContextWidth(project) + - Async + error handling → @Automatability(llm) + @ContextWidth(file) + - Single local transform → @Automatability(template) + @ContextWidth(local) +- `estimateContextTokens(node, width) -> int` — rough token count needed for + the given context width (local: node only, file: buffer, project: all buffers) +- `suggestWorkerType(annotations) -> string` — given a node's annotations, + suggest "deterministic" | "slm" | "llm" | "human" + +**Tests:** complexity → ambiguity mapping, getter → deterministic, cross-file → +project width, token estimation reasonable, worker suggestion matches annotations, +manual annotations override inferred, combined inference + routing pipeline. + +### Step 319: Phase 11e Integration + Sprint Summary (8 tests) +**Goal:** Full workflow annotation pipeline and sprint verification. + +**Tests cover:** +1. Parse Python → infer annotations across all 8 subjects → verify suggestions +2. Create skeleton module → add annotated functions → get project model → verify task list +3. Infer routing annotations: simple function → deterministic, complex function → llm +4. Skeleton task list respects dependency ordering from @Priority.blockedBy +5. MCP tools/list returns 42+ tools with 4 workflow tools present +6. Sprint totals: 10 parsers, 10 generators, 42+ MCP tools, 0 annotation types dropped +7. Inference + routing + skeleton combined: create skeleton → infer → route → verify +8. Semanno roundtrip includes workflow annotations (Subject 9) + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 11a | 290-296 | 80 | Semanno format + annotation codegen for all 56 missing types | +| 11b | 297-301 | 56 | Validation, conflict detection, transforms for all annotations | +| 11c | 302-308 | 80 | New AST nodes (class/generic/async) + 8 parser deepening | +| 11d | 309-313 | 56 | Kotlin + C# parsers and generators | +| 11e | 314-319 | 68 | Workflow annotation foundation (inference, routing types, skeleton AST) | +| **Total** | **290-319** | **~340** | 30 steps | + +**MCP tool count projection:** 42+ tools (38 current + 4 workflow tools) + +--- + +## Key Files to Modify + +| File | Changes | +|------|---------| +| `editor/src/ast/ProjectionGenerator.h` | 56 new visitor dispatch branches + 9 new AST node visitors | +| `editor/src/ast/Annotation.h` | Source of truth for all annotation classes; +6 workflow annotations (Subject 9) | +| `editor/src/ast/Serialization.h` | propertiesToJson/createNode/setPropertiesFromJson for 9 new AST node types + 6 workflow annotations | +| `editor/src/AnnotationValidator.h` | Include extended validator, expand from 5 to 67 validated types | +| `editor/src/AnnotationConflict.h` | Include extended, add cross-type conflict detection | +| `editor/src/TransformEngine.h` | Extended with float folding, dead variable elimination, annotation-aware optimization | +| `editor/src/ast/Parser.h` | Include KotlinParser.h + CSharpParser.h | +| `editor/src/Pipeline.h` | Add Kotlin + C# language routing in parse() and generate() | +| `editor/src/MCPServer.h` | Register 4 workflow tools | +| `editor/src/HeadlessAgentRPCHandler.h` | createSkeleton, addSkeletonNode, getProjectModel, inferAnnotations RPCs | +| `editor/src/CompactAST.h` | Workflow annotations in extractSemanticSummary | +| `editor/src/SidecarPersistence.h` | Workflow annotations in isSemanticAnnotation | +| `editor/src/CrossLanguageProjector.h` | Kotlin + C# support, generic annotation cloning | +| `editor/src/MemoryStrategyInference.h` | Kotlin language defaults | +| `editor/src/ast/PythonGenerator.h` | SemannoAnnotationImpl mixin + commentPrefix() | +| `editor/src/ast/CppGenerator.h` | SemannoAnnotationImpl mixin + commentPrefix() | +| `editor/src/ast/RustGenerator.h` | SemannoAnnotationImpl mixin + commentPrefix() | +| `editor/src/ast/GoGenerator.h` | SemannoAnnotationImpl mixin + commentPrefix() | +| `editor/src/ast/JavaGenerator.h` | SemannoAnnotationImpl mixin + commentPrefix() | +| `editor/src/ast/JavaScriptGenerator.h` | SemannoAnnotationImpl mixin + commentPrefix() | +| `editor/src/ast/ElispGenerator.h` | SemannoAnnotationImpl mixin + commentPrefix() | +| `editor/src/ast/Generator.h` | Include KotlinGenerator.h + CSharpGenerator.h | +| `editor/CMakeLists.txt` | Test targets for steps 290-319 | + +**New files:** +- `editor/src/SemannoFormat.h` — SemannoEmitter + SemannoParser (11a, done) +- `editor/src/SemannoAnnotationImpl.h` — CRTP mixin for generators (11a, done) +- `editor/src/SemannoSidecar.h` — .semanno sidecar save/load (11a, done) +- `editor/src/ast/AnnotationVisitors.h` — AnnotationVisitorExtended interface (11a, done) +- `editor/src/AnnotationValidatorExtended.h` — E0600-E1299 validation rules +- `editor/src/AnnotationConflictExtended.h` — cross-type conflict detection +- `editor/src/TransformEngineExtended.h` — float folding, dead var, annotation-aware +- `editor/src/ast/ClassDeclaration.h` — ClassDeclaration, InterfaceDeclaration, MethodDeclaration +- `editor/src/ast/GenericType.h` — GenericType, TypeParameter +- `editor/src/ast/AsyncNodes.h` — AsyncFunction, AwaitExpression, LambdaExpression, DecoratorAnnotation +- `editor/src/ast/KotlinParser.h` — regex-based Kotlin parser +- `editor/src/ast/KotlinGenerator.h` — Kotlin code generator +- `editor/src/ast/CSharpParser.h` — regex-based C# parser +- `editor/src/ast/CSharpGenerator.h` — C# code generator +- `editor/src/AnnotationInference.h` — generalized multi-subject annotation inference + routing bridge +- `editor/src/SkeletonAST.h` — skeleton module/function creation, project model, task list generation +- `editor/tests/step290_test.cpp` through `editor/tests/step319_test.cpp` + +--- + +## Verification + +After each phase: +1. Build all new test targets: `cmake --build . --target stepNNN_test` +2. Run each test: `./stepNNN_test` — expect "Results: 12/12" (or 8/8 for integration) +3. Phase integration test validates no regressions on earlier steps + +Full sprint verification: +- All ~340 tests pass across steps 290-319 +- Run full test suite (steps 245-319) to verify no regressions +- MCP `tools/list` returns 42+ tools +- Skeleton creation + project model extraction works end-to-end +- Annotation inference covers all 8 subjects + routing annotations +- Semanno roundtrip: generate code with annotations → parse comments back → match +- All 10 languages parse and generate correctly +- 0 annotation types silently dropped during codegen + +--- + +## Sprint 12 Preview + +See `roadmap.md` for the full Sprint 12-25+ roadmap. + +Sprint 12 builds directly on 11e's foundation: +- WorkItem model (extends SkeletonTask with execution state, assignment, results) +- Task queue with dependency resolution and priority scheduling +- Routing engine that reads annotations and dispatches to workers +- C++ feature-requests items 1-5 for self-hosting progress diff --git a/sprint12_plan.md b/sprint12_plan.md new file mode 100644 index 0000000..14754fd --- /dev/null +++ b/sprint12_plan.md @@ -0,0 +1,889 @@ +# Sprint 12 Plan: Workflow Model + C++ Depth + +## Context + +Sprint 11 completed the annotation foundation: Semanno format for all 67+ types, +validation rules across all 8 subjects, parser deepening with 9 new AST node types, +Kotlin + C# as languages 9 and 10, and the workflow annotation foundation (Subject 9 +routing annotations, skeleton AST, annotation inference, inference-to-routing bridge). + +Sprint 12 builds the **execution layer** on top of that foundation. The skeleton AST +from Sprint 11e describes *what* needs to be done; Sprint 12 adds *how* it gets done. +WorkItems track execution state, the routing engine reads annotations and dispatches +to the right worker type, and review gates ensure humans stay in the loop where they +should be. + +In parallel, Sprint 12 deepens C++ AST coverage with 5 construct families from +`feature-requests.md`, progressing toward the self-hosting milestone where Whetstone +can parse its own headers. + +**What Sprint 11 delivered that Sprint 12 builds on:** +- `SkeletonAST.h` — skeleton modules, task list generation, SkeletonTask struct +- `AnnotationInference.h` — multi-subject inference + routing bridge + suggestWorkerType() +- Subject 9 annotations: @ContextWidth, @Review, @Ambiguity, @Automatability, @Priority, @ImplementationStatus +- `ResponseBudget.h` — budget truncation + continuation tokens (Sprint 9c) +- `AgentPermissionPolicy.h` — 3-role system (Linter, Refactor, Generator) +- `Pipeline.h` — 8-stage parse→infer→validate→optimize→generate flow +- 42+ MCP tools, 10 language parsers/generators, 73+ annotation types + +**Sprint 12 Deliverables:** +- **WorkItem execution model** with lifecycle state tracking +- **Task queue** with dependency resolution and priority scheduling +- **Routing engine** that maps annotations → worker dispatch decisions +- **Worker abstractions** (deterministic, agent, human) with context estimation +- **Review gates** with auto-approve rules and human review queue +- **7 new C++ AST node types**: enhanced ClassDeclaration (multiple inheritance), + TemplateClass, IncludeDirective, PragmaDirective, MacroDefinition, EnumDeclaration, + NamespaceDeclaration +- **C++ parser deepened** with inheritance, preprocessor, enum, namespace support +- **50+ MCP tools** (42 current + 8 workflow/routing tools) + +--- + +## Phase 12a: WorkItem Model + Task Queue (Steps 320-325) + +*The execution model that turns skeleton tasks into trackable, assignable work.* + +### Step 320: WorkItem.h — Core Execution Model (12 tests) +**Goal:** Define the WorkItem lifecycle — the unit of work that flows through the +routing engine, gets assigned to a worker, produces results, and optionally goes +through human review. + +**New file: `editor/src/WorkItem.h`** (~350 lines) + +**WorkItem struct** (extends SkeletonTask concept): +``` +WorkItem: + // Identity (from SkeletonTask) + id: string // unique work item ID (wi-001, wi-002, ...) + nodeId: string // AST node this work item targets + nodeName: string // human-readable name + nodeType: string // "function" | "class" | "method" | ... + bufferId: string // which buffer this belongs to + + // Routing (from annotations, set by routing engine) + contextWidth: string // "local" | "file" | "project" | "cross-project" + workerType: string // "deterministic" | "template" | "slm" | "llm" | "human" + reviewRequired: bool // whether output needs review before acceptance + priority: string // "critical" | "high" | "medium" | "low" + dependencies: vector // work item IDs that must complete first + + // Lifecycle + status: string // "pending" | "ready" | "assigned" | "in-progress" | + // "review" | "complete" | "rejected" + assignee: string // worker ID or "human:" + createdAt: string // ISO timestamp + assignedAt: string + completedAt: string + + // Result (populated after execution) + result: WorkItemResult +``` + +**WorkItemResult struct:** +``` +WorkItemResult: + generatedCode: string // the code produced by the worker + astJson: json // the AST subtree produced + diagnostics: vector // any diagnostics from validation + confidence: float // worker's confidence in the result (0.0-1.0) + tokensBudget: int // context tokens consumed + tokensGenerated: int // output tokens produced + reasoning: string // worker's explanation of decisions made +``` + +**Status transition rules:** +- `pending` → `ready` (when all dependencies complete) +- `ready` → `assigned` (when routing engine dispatches) +- `assigned` → `in-progress` (when worker begins) +- `in-progress` → `review` (if reviewRequired) or `complete` (if auto-approved) +- `review` → `complete` (human approves) or `rejected` (human rejects) +- `rejected` → `ready` (re-enters queue for re-routing with feedback) + +**Helper functions:** +- `createWorkItem(skeletonTask, bufferId) -> WorkItem` — from SkeletonTask +- `isTerminal(status) -> bool` — complete or rejected-and-abandoned +- `canTransition(from, to) -> bool` — validates state machine +- `transitionWorkItem(item, newStatus) -> bool` — applies transition with timestamp +- `workItemToJson(item) -> json` / `workItemFromJson(json) -> WorkItem` + +**Tests:** construction from SkeletonTask, status transitions (valid and invalid), +JSON roundtrip, timestamp population, result attachment, dependency tracking, +terminal state detection, ID generation. + +### Step 321: TaskQueue.h — Priority Queue with Dependencies (12 tests) +**Goal:** Ordered queue that respects both priority levels and dependency chains. +Items whose dependencies are unsatisfied stay blocked; ready items are ordered +by priority then creation time. + +**New file: `editor/src/TaskQueue.h`** (~300 lines) + +**TaskQueue class:** +- `enqueue(WorkItem) -> void` — adds item, sets status to pending or ready +- `getReady() -> vector` — returns items whose dependencies are all + complete, ordered by priority (critical first) then creation time +- `peek() -> optional` — next ready item without removing +- `dequeue() -> optional` — removes and returns next ready item, + transitions to assigned +- `complete(itemId) -> bool` — marks complete, triggers dependency re-evaluation + (blocked items may become ready) +- `reject(itemId, feedback) -> bool` — marks rejected, re-enqueues as ready + with feedback attached +- `getBlocked() -> vector` — items waiting on dependencies +- `getByStatus(status) -> vector` — filter by lifecycle status +- `size() -> int` / `readyCount() -> int` / `blockedCount() -> int` +- `getItem(itemId) -> optional` — lookup by ID +- `updateItem(itemId, WorkItem) -> bool` — replace item (for status updates) + +**Dependency resolution:** +- When `complete(itemId)` is called, scan all blocked items +- For each blocked item, check if all dependencies are now complete +- If so, transition from pending → ready +- O(n) scan is fine for project-scale queues (hundreds, not millions) + +**Priority ordering:** +- critical=0, high=1, medium=2, low=3 (lower number = higher priority) +- Ties broken by createdAt (earlier first) + +**Tests:** enqueue/dequeue ordering, priority ordering (critical before low), +dependency blocking (B blocked by A, A completes → B becomes ready), reject and +re-enqueue, getReady returns only unblocked items, empty queue behavior, +getByStatus filtering, size tracking, completion cascading (A→B→C chain), +mixed priorities with dependencies. + +### Step 322: WorkflowState.h — Project-Level Workflow Tracking (12 tests) +**Goal:** Top-level state that manages the full workflow lifecycle for a project: +from skeleton creation through routing, execution, review, and completion. + +**New file: `editor/src/WorkflowState.h`** (~300 lines) + +**WorkflowPhase enum:** +- `Modeling` — architect is creating/annotating skeletons +- `Routing` — routing engine is assigning worker types +- `Executing` — workers are producing results +- `Reviewing` — human review in progress +- `Complete` — all work items complete + +**WorkflowState class:** +- `queue: TaskQueue` — the project's task queue +- `phase: WorkflowPhase` — current workflow phase +- `projectName: string` +- `createdAt: string` + +**Methods:** +- `populateFromSkeleton(module) -> int` — walks skeleton AST, creates WorkItems + from SkeletonTask list, enqueues all, returns count +- `routeAll(RoutingEngine&) -> int` — routes all ready items (Sprint 12b), + returns count routed +- `getStats() -> WorkflowStats` — counts by status, by worker type, by priority +- `getPhase() -> WorkflowPhase` — auto-computed from item statuses +- `getHistory(itemId) -> vector` — audit trail for an item +- `toJson() / fromJson()` — full state serialization + +**WorkflowStats struct:** +``` +WorkflowStats: + total: int + pending: int, ready: int, assigned: int, inProgress: int + review: int, complete: int, rejected: int + byWorkerType: map // deterministic: 5, llm: 3, human: 2 + byPriority: map // critical: 1, high: 4, medium: 5 + completionPercent: float +``` + +**StatusChange struct** (audit trail): +``` +StatusChange: + itemId: string + fromStatus: string + toStatus: string + timestamp: string + actor: string // "routing-engine" | "worker:det-001" | "human:bill" + note: string // rejection reason, routing rationale, etc. +``` + +**Tests:** populateFromSkeleton creates correct item count, phase auto-detection +(all pending → Modeling, some assigned → Executing, some in review → Reviewing, +all complete → Complete), stats accuracy, history tracking, JSON roundtrip, +empty workflow behavior, phase transitions. + +### Step 323: Workflow Sidecar Persistence (12 tests) +**Goal:** Save and load workflow state to `.whetstone/.workflow.json` +so workflows survive across sessions. + +**New file: `editor/src/WorkflowPersistence.h`** (~200 lines) +- `workflowSidecarPath(workspaceRoot, projectName) -> string` + — returns `.whetstone/.workflow.json` +- `saveWorkflow(workspaceRoot, state) -> SaveResult` + — serializes WorkflowState to JSON, writes to sidecar path +- `loadWorkflow(workspaceRoot, projectName) -> optional` + — reads sidecar, deserializes, returns nullopt if not found +- `deleteWorkflow(workspaceRoot, projectName) -> bool` + — removes sidecar file + +**SaveResult struct:** +``` +SaveResult: + success: bool + path: string + bytesWritten: int + itemCount: int +``` + +**Tests:** save/load roundtrip, all work item fields preserved, status history +preserved, missing file returns nullopt, save creates parent directories, +delete removes file, multiple workflows in same workspace, workflow with +results attached, empty workflow save/load. + +### Step 324: Workflow RPC + MCP Tools (12 tests) +**Goal:** Expose workflow management through RPC methods and MCP tools so agents +can create, inspect, and advance workflows. + +**Modifies: `editor/src/HeadlessEditorState.h`** +- Add `WorkflowState` member (optional, null until created) +- Include WorkflowState.h, WorkflowPersistence.h + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** — 8 new RPC methods: +- `createWorkflow` — create WorkflowState, populate from active buffer's skeleton + - params: projectName + - returns: { itemCount, phase, stats } +- `getWorkflowState` — return current workflow stats + phase + - returns: { phase, stats, readyCount, blockedCount } +- `getReadyTasks` — return work items ready for assignment + - returns: { items: [...], count } +- `getWorkItem` — return a single work item's full details + - params: itemId + - returns: { workItem with result if present } +- `assignTask` — move a ready task to assigned status + - params: itemId, assignee + - returns: { success, item } +- `completeTask` — mark a task complete with result + - params: itemId, result (generatedCode, confidence, reasoning) + - returns: { success, newlyReady: [...] } +- `rejectTask` — reject a task back to the queue + - params: itemId, reason + - returns: { success } +- `saveWorkflow` — persist workflow to sidecar + - returns: { success, path, bytesWritten } + +**Modifies: `editor/src/AgentPermissionPolicy.h`** +- `getWorkflowState` / `getReadyTasks` / `getWorkItem` — read-only (all roles) +- `createWorkflow` / `assignTask` / `completeTask` / `rejectTask` / `saveWorkflow` + — mutation (Refactor/Generator only) + +**Modifies: `editor/src/MCPServer.h`** — registerWorkflowExecutionTools() +- `whetstone_create_workflow` — create and populate workflow +- `whetstone_get_workflow_state` — inspect current workflow +- `whetstone_get_ready_tasks` — get assignable items +- `whetstone_get_work_item` — inspect single item +- `whetstone_assign_task` — assign a ready task +- `whetstone_complete_task` — mark task complete with result +- `whetstone_reject_task` — reject task with feedback +- `whetstone_save_workflow` — persist to sidecar + +**Tool count:** 50+ (42 from Sprint 11 + 8 workflow execution tools) + +**Tests:** createWorkflow from skeleton module, getReadyTasks ordering, assignTask +transitions, completeTask triggers dependency cascade, rejectTask re-enqueues, +getWorkItem returns full details, Linter role can read but not mutate, MCP tool +registration (8 new tools, 50+ total), saveWorkflow round-trip. + +### Step 325: Phase 12a Integration Tests (8 tests) +**Goal:** End-to-end workflow lifecycle from skeleton to completion. + +**Tests:** +1. Create skeleton module → createWorkflow → verify item count matches skeleton +2. Priority ordering: critical task appears before low in getReadyTasks +3. Dependency chain: A blocks B blocks C → complete A → B ready → complete B → C ready +4. Full lifecycle: create → assign → in-progress → complete with result → verify +5. Rejection flow: assign → reject with feedback → re-appears in ready queue +6. WorkflowState persistence: save → reload → verify all items and history preserved +7. Stats accuracy: mixed statuses → getWorkflowState returns correct counts +8. Combined: skeleton with 5 functions, mixed priorities and dependencies, + complete in correct order, verify final phase = Complete + +--- + +## Phase 12b: Routing Engine + Workers (Steps 326-331) + +*The brain that reads annotations and decides who handles what.* + +### Step 326: RoutingEngine.h — Annotation-to-Dispatch Logic (12 tests) +**Goal:** Given a WorkItem with routing annotations, produce a RoutingDecision +that says what kind of worker should handle it and how much context to provide. + +**New file: `editor/src/RoutingEngine.h`** (~350 lines) + +**RoutingDecision struct:** +``` +RoutingDecision: + workerType: string // "deterministic" | "template" | "slm" | "llm" | "human" + contextWidth: string // "local" | "file" | "project" | "cross-project" + contextBudgetTokens: int // estimated tokens for context window + reviewRequired: bool // whether result needs human review + confidence: float // engine's confidence in this routing (0.0-1.0) + reasoning: string // human-readable explanation of the decision + agentRole: string // "linter" | "refactor" | "generator" +``` + +**RoutingEngine class:** +- `route(WorkItem) -> RoutingDecision` — main entry point +- `routeBatch(vector) -> vector` — batch routing + +**Routing rules (priority order):** +1. **Explicit annotation override** — if @Automatability is set on the node, + use it directly (human architect has spoken) +2. **Ambiguity gate** — @Ambiguity(high) → human, regardless of other signals +3. **Review gate** — @Review(required, human) → sets reviewRequired=true +4. **Complexity routing:** + - @Complexity(cognitive<=2) + @ContextWidth(local) → deterministic + - @Complexity(cognitive<=4) + @ContextWidth(local|file) → slm + - @Complexity(cognitive<=7) → llm + - @Complexity(cognitive>7) → llm + reviewRequired +5. **Context width escalation:** + - cross-project → llm (needs large context window) + - project → llm or slm depending on complexity + - file → slm unless complex + - local → deterministic or template unless complex +6. **Pattern defaults** (if no annotations): + - Skeleton with no body → template if simple signature, slm otherwise + - Function with existing body (modification) → llm + - Getter/setter/accessor patterns → deterministic + +**Context budget estimation:** +- local: ~500 tokens (node + immediate siblings) +- file: ~2000 tokens (buffer content, budget-truncated) +- project: ~8000 tokens (all buffers, compact AST summaries) +- cross-project: ~16000 tokens (project + external references) +- Uses `estimateContextTokens()` from Sprint 11e AnnotationInference + +**Tests:** explicit @Automatability overrides inferred routing, high ambiguity → +human, low complexity + local → deterministic, high complexity → llm, cross-project +→ llm, review annotation sets reviewRequired, batch routing, no-annotation defaults, +getter pattern → deterministic, confidence reflects annotation coverage, context +budget reasonable for each width. + +### Step 327: WorkerRegistry.h — Worker Abstractions (12 tests) +**Goal:** Define the worker interface and provide concrete implementations for +deterministic and pass-through workers. LLM/SLM workers are stubs that prepare +context — actual model invocation happens externally via MCP. + +**New file: `editor/src/WorkerRegistry.h`** (~400 lines) + +**WorkerInterface (abstract):** +``` +class WorkerInterface: + type() -> string // "deterministic" | "template" | "slm" | "llm" | "human" + canHandle(WorkItem) -> bool // whether this worker accepts the item + execute(WorkItem, context) -> WorkItemResult // produce result +``` + +**WorkerContext struct:** +``` +WorkerContext: + nodeAst: json // the target node's AST (always present) + bufferContent: string // current buffer text (file+ context) + projectSummary: json // compact AST summaries of all buffers (project+ context) + annotations: vector // all annotations on the target node + diagnostics: vector // relevant diagnostics + skeletonIntent: string // @Intent annotation value if present + feedbackFromRejection: string // if this is a re-routed rejected item +``` + +**DeterministicWorker** — uses Pipeline for code generation: +- Handles: simple skeleton functions with clear signatures and @Intent +- Execute: parses intent into AST manipulations, runs Pipeline.generate() +- Confidence: 0.9+ for pattern-matched templates, 0.5 for heuristic fills + +**TemplateWorker** — uses language-specific templates: +- Handles: getters, setters, constructors, simple CRUD patterns +- Execute: pattern-matches skeleton signature → fills template +- Confidence: 0.95 for exact pattern match + +**AgentWorker** — prepares context for external LLM/SLM invocation: +- Handles: anything too complex for deterministic/template +- Execute: assembles WorkerContext at appropriate context width, + returns a "prepared" result with context bundle (the actual model + invocation happens externally through MCP) +- Subclasses: `SLMAgentWorker` (narrow context), `LLMAgentWorker` (wide context) +- Confidence: 0.0 (deferred to external model) + +**HumanWorker** — marks item for human review: +- Handles: items routed to human +- Execute: returns result with status="needs-human", includes context bundle + and reasoning for why this was routed to human +- Confidence: 0.0 (deferred to human) + +**WorkerRegistry class:** +- `registerWorker(WorkerInterface*) -> void` +- `getWorker(workerType) -> WorkerInterface*` +- `getDefaultRegistry() -> WorkerRegistry` — pre-populated with all 5 worker types + +**Tests:** deterministic worker produces code for simple skeleton, template worker +fills getter pattern, agent worker prepares context bundle, human worker marks for +review, registry lookup, canHandle filtering, worker type strings, context assembly +at local/file/project widths, rejection feedback included in context, pipeline +integration for deterministic worker. + +### Step 328: Context Assembly + Budget (12 tests) +**Goal:** Build the context window for each work item based on its @ContextWidth, +respecting token budgets and using the ResponseBudget system for truncation. + +**New file: `editor/src/ContextAssembler.h`** (~250 lines) + +**ContextAssembler class:** +- `assembleContext(workItem, bufferStates, contextWidth, budget) -> WorkerContext` + — main entry point; builds context at the specified width within budget + +**Assembly strategies by width:** +- **local:** target node AST + immediate sibling nodes + parent annotations +- **file:** target node + full buffer content (editBuf), truncated if over budget +- **project:** target node + compact AST summaries of all open buffers + + import graph for the target file + cross-file symbols +- **cross-project:** project context + external dependency annotations + (from sidecar files in workspace) + +**Budget enforcement:** +- Estimate each context component's token cost (json.size() / 4) +- If total exceeds budget, prioritize: target node > annotations > diagnostics > + siblings > buffer content > project summaries +- Use ResponseBudget::applyBudget() for array truncation +- Return actual tokens used in WorkerContext + +**Integration with existing systems:** +- Uses `CompactAST::toJsonCompact()` for project-wide summaries +- Uses `ProjectState::ImportGraph` for cross-file references +- Uses `ResponseBudget::applyBudget()` for truncation +- Uses `AnnotationInference::estimateContextTokens()` for estimates + +**Tests:** local context contains only target node, file context includes buffer +content, project context includes other buffer summaries, budget truncation +activates when over limit, priority ordering preserved under truncation, import +graph included in project context, cross-file symbols included, empty project +returns minimal context, token estimate within 20% of actual, budget=0 means +unlimited. + +### Step 329: Routing RPC + MCP Tools (12 tests) +**Goal:** Expose routing and worker dispatch through RPC and MCP tools. + +**Modifies: `editor/src/HeadlessEditorState.h`** +- Add `RoutingEngine` and `WorkerRegistry` members +- Initialize default routing rules and worker registry + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** — 4 new RPC methods: +- `routeTask` — route a single work item, return RoutingDecision + - params: itemId + - returns: { decision: { workerType, contextWidth, budget, reasoning } } +- `routeAllReady` — route all ready items in the workflow + - returns: { routed: int, decisions: [...] } +- `executeTask` — assign worker and execute (deterministic/template only; + agent/human items are prepared but not executed) + - params: itemId + - returns: { result, workerType, autoApproved } +- `getRoutingExplanation` — explain why a specific item was routed a certain way + - params: itemId + - returns: { reasoning, annotationsUsed, rulesApplied } + +**Modifies: `editor/src/AgentPermissionPolicy.h`** +- `getRoutingExplanation` — read-only +- `routeTask` / `routeAllReady` / `executeTask` — mutation (Refactor/Generator) + +**Modifies: `editor/src/MCPServer.h`** — registerRoutingTools() +- `whetstone_route_task` — route single item +- `whetstone_route_all_ready` — batch route +- `whetstone_execute_task` — execute deterministic/template tasks +- `whetstone_get_routing_explanation` — explain routing decision + +**Tool count:** 54+ (50 + 4 routing tools) + +**Tests:** routeTask returns valid decision, routeAllReady batch processes queue, +executeTask runs deterministic worker and produces code, agent items marked as +prepared not executed, human items marked for review, routing explanation includes +annotation references, Linter role can read explanation but not route, MCP tool +registration (4 new tools). + +### Step 330: Review Gates (12 tests) +**Goal:** Configurable rules for auto-approving low-risk results and surfacing +high-risk results for human review. + +**New file: `editor/src/ReviewGate.h`** (~200 lines) + +**ReviewPolicy struct:** +``` +ReviewPolicy: + autoApproveRules: vector + defaultAction: string // "require-review" | "auto-approve" +``` + +**AutoApproveRule struct:** +``` +AutoApproveRule: + workerType: string // match worker type ("deterministic", "*") + maxComplexity: int // max cognitive complexity to auto-approve + minConfidence: float // minimum worker confidence to auto-approve + riskLevel: string // max risk level ("none", "low") +``` + +**ReviewGate class:** +- `shouldAutoApprove(workItem, result, policy) -> ReviewDecision` +- `ReviewDecision`: approved (bool), reasoning (string), rule matched or why not +- Default policy: auto-approve deterministic + confidence>=0.9 + risk<=low; + everything else requires review + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** +- `setReviewPolicy` — configure auto-approve rules +- `getReviewPolicy` — inspect current review policy + +**Modifies: `editor/src/MCPServer.h`** +- `whetstone_set_review_policy` + `whetstone_get_review_policy` + +**Tool count:** 56+ (54 + 2 review tools) + +**Tests:** deterministic + high confidence → auto-approve, llm result → require +review, low confidence → require review even if deterministic, high risk overrides +auto-approve, custom policy respected, default policy behavior, explicit +@Review(required) overrides auto-approve, rejection with feedback, policy +serialization roundtrip, empty policy uses defaults, MCP tool registration. + +### Step 331: Phase 12b Integration Tests (8 tests) +**Goal:** Full routing pipeline from skeleton through execution and review. + +**Tests:** +1. Create skeleton → populate workflow → route all → verify worker type assignments +2. Simple getter skeleton → route → deterministic → execute → auto-approve → complete +3. Complex function skeleton → route → llm → prepare context → verify context width +4. High ambiguity skeleton → route → human → verify marked for review +5. Dependency chain with mixed routing: A(deterministic)→B(llm)→C(human) in order +6. Rejection flow: execute → reject → re-route with feedback in context +7. Review policy: set strict policy → nothing auto-approved → set permissive → deterministic auto-approved +8. Full lifecycle: 5-function skeleton with mixed complexity → workflow → route → execute deterministic ones → verify stats + +--- + +## Phase 12c: C++ AST Depth — Inheritance + CRTP (Steps 332-336) + +*Multiple inheritance, virtual inheritance, and CRTP — Whetstone's own patterns.* + +### Step 332: Multiple Inheritance in ClassDeclaration (12 tests) +**Goal:** Upgrade ClassDeclaration from single `superClass` string to a vector +of base classes with access specifiers and virtual flags. + +**Modifies: `editor/src/ast/ClassDeclaration.h`** +- New struct `BaseClass { name, accessSpecifier, isVirtual }` + - accessSpecifier: "public" | "protected" | "private" (default: "private" for C++) + - isVirtual: bool (default: false) +- `ClassDeclaration` gains: `vector baseClasses` +- Backward compat: existing `superClass` string still works (converted to + single BaseClass with public access), deprecated but supported +- Helper: `addBase(name, access, isVirtual)`, `getBases() -> vector` +- Helper: `hasDiamondInheritance(classMap) -> bool` (detects shared bases) + +**Modifies: `editor/src/ast/Serialization.h`** +- `propertiesToJson` — serialize baseClasses array +- `setPropertiesFromJson` — deserialize baseClasses, handle legacy superClass +- `createNode` — no change (ClassDeclaration factory exists from Sprint 11c) + +**Tests:** single base (backward compat), multiple bases, access specifiers, +virtual inheritance flag, BaseClass JSON roundtrip, mixed virtual and non-virtual, +diamond detection (A→B, A→C, D→B+C), empty bases list, legacy superClass +migration, addBase helper. + +### Step 333: Template Class Declarations (12 tests) +**Goal:** Represent `template class Foo : public Bar { ... }` — +template parameters on classes (not just functions), including CRTP patterns. + +**Modifies: `editor/src/ast/GenericType.h`** +- `GenericType` gains: `isClassTemplate: bool` (vs function template) +- When attached to a ClassDeclaration, represents `template<...> class` +- TypeParameter gains: `isVariadic: bool` (for `typename... Args`) + +**New recognition helpers in `editor/src/ast/GenericType.h`:** +- `isCRTP(classDecl) -> bool` — detects when a class inherits from a base + parameterized by itself: `class Foo : public Base` +- `getTemplateParameters(classDecl) -> vector` — extracts + template params from attached GenericType children + +**Modifies: `editor/src/ast/Serialization.h`** +- GenericType: serialize `isClassTemplate` +- TypeParameter: serialize `isVariadic` + +**Modifies: `editor/src/CompactAST.h`** +- `getNodeName` for GenericType includes template indicator + +**Tests:** class template construction, CRTP detection (class Foo : Base), +non-CRTP base parameterized by other type, variadic type parameter, JSON roundtrip +for isClassTemplate, template class with multiple type params, GenericType +attached to ClassDeclaration, compact AST name, isCRTP with diamond inheritance, +template + multiple inheritance combined. + +### Step 334: C++ Parser — Inheritance + Templates (12 tests) +**Goal:** Deepen CppParser to handle multiple inheritance, virtual inheritance, +access specifiers, template class declarations, and CRTP patterns. + +**Modifies: `editor/src/ast/CppParser.h`** +- Parse `class Foo : public Bar, private virtual Baz { ... }` + → ClassDeclaration with baseClasses vector +- Parse `template class Foo { ... }` + → ClassDeclaration with GenericType child (isClassTemplate=true) +- Parse `template` + → TypeParameters including variadic +- Parse `class Foo : public Base` → CRTP detection +- Parse `struct` same as `class` but default public access + +**Backward compatibility:** existing class parsing (single base) continues +to work, now populates baseClasses[0] instead of superClass. + +**Tests:** single inheritance (backward compat), multiple inheritance parsed, +access specifiers detected, virtual inheritance flag, template class parsed, +CRTP pattern detected, struct vs class default access, nested class (inner class), +template with variadic, combined template + multiple inheritance, existing +function-level tests still pass, real Whetstone-style class signature. + +### Step 335: C++ Generator — Inheritance + Templates (12 tests) +**Goal:** CppGenerator outputs correct C++ syntax for multiple inheritance, +virtual inheritance, access specifiers, and template classes. + +**Modifies: `editor/src/ast/CppGenerator.h`** +- Generate `class Foo : public Bar, private virtual Baz { ... }` +- Generate `template class Foo { ... }` +- Generate `template` for variadic +- Access sections: `public:`, `protected:`, `private:` grouping for methods + +**Cross-language considerations:** +- Other generators handle multiple inheritance differently: + - Java/Kotlin/C# → single extends + multiple implements (interfaces) + - Python → multiple inheritance directly + - Rust → no inheritance, use traits + - Go → embedding +- Add `adaptInheritance(classDecl, targetLanguage) -> ClassDeclaration` helper + in CrossLanguageProjector that converts multiple inheritance to + target-appropriate pattern + +**Modifies: `editor/src/CrossLanguageProjector.h`** +- `adaptInheritance()` — converts C++ multiple inheritance to target patterns + +**Tests:** C++ multiple inheritance output, access specifier output, virtual +keyword present, template class output, CRTP output, cross-language: C++ → Java +(multiple bases → extends + implements), C++ → Python (direct), C++ → Rust +(traits), cross-language: Java → C++ (implements → base classes), roundtrip: +parse C++ → generate C++, combined template + inheritance output. + +### Step 336: Phase 12c Integration Tests (8 tests) +**Goal:** Full C++ inheritance + template roundtrip and cross-language projection. + +**Tests:** +1. Parse Whetstone-style class: `class KotlinGenerator : public ProjectionGenerator, public virtual AnnotationVisitorExtended, public SemannoAnnotationImpl` → verify AST structure +2. CRTP detection: `SemannoAnnotationImpl` recognized as CRTP +3. Round-trip: parse complex C++ class → generate → parse again → AST equivalent +4. Cross-language: C++ class with multiple inheritance → Java output (extends + implements) +5. Cross-language: C++ class with multiple inheritance → Python output (multiple bases) +6. Template class with 3 type params → parse → generate → verify +7. Diamond inheritance detection: parse class hierarchy → hasDiamondInheritance returns true +8. Mixed: class with templates, multiple inheritance, virtual methods, annotations → full pipeline + +--- + +## Phase 12d: C++ AST Depth — Preprocessor, Enums, Namespaces (Steps 337-341) + +*The remaining C++ constructs needed for self-hosting progress.* + +### Step 337: Preprocessor AST Nodes (12 tests) +**Goal:** Represent `#include`, `#pragma`, and `#define` as first-class AST nodes. + +**New file: `editor/src/ast/PreprocessorNodes.h`** (~150 lines) + +**IncludeDirective** (Statement-level): +- `path: string` — the include path ("ast/ASTNode.h" or ) +- `isSystem: bool` — angle brackets (true) vs quotes (false) + +**PragmaDirective** (Statement-level): +- `directive: string` — the pragma content ("once", "pack(push, 1)", etc.) + +**MacroDefinition** (Statement-level): +- `name: string` — macro name +- `parameters: vector` — macro params (empty for object-like macros) +- `body: string` — macro body text (unparsed — macros are fundamentally textual) +- `isFunctionLike: bool` — has parameter list + +**Design note:** Preprocessor directives are statement-level nodes added to Module's +children. They are *not* deeply parsed into sub-AST (macro bodies stay as text). +This matches the reality that preprocessor operates on text, not AST. For annotation +purposes, `@Synthetic(macro-expanded)` can mark nodes that result from macro expansion. + +**Modifies: `editor/src/ast/Serialization.h`** +- propertiesToJson/createNode/setPropertiesFromJson for all 3 types + +**Modifies: `editor/src/CompactAST.h`** +- getNodeName for IncludeDirective, PragmaDirective, MacroDefinition + +**Tests:** IncludeDirective construction + roundtrip, system vs local flag, +PragmaDirective construction + roundtrip, MacroDefinition with params, +function-like vs object-like distinction, compact AST names, module with +mixed preprocessor + function nodes, empty macro body, Serialization createNode +dispatch. + +### Step 338: EnumDeclaration + NamespaceDeclaration (12 tests) +**Goal:** Scoped enums and namespaces as first-class AST nodes. + +**New file: `editor/src/ast/EnumNamespaceNodes.h`** (~150 lines) + +**EnumDeclaration** (Statement-level, has children): +- `name: string` +- `isScoped: bool` — `enum class` (true) vs plain `enum` (false) +- `underlyingType: string` — optional (e.g., "int", "uint8_t") +- Children role "members": vector of EnumMember nodes + +**EnumMember** (leaf node): +- `name: string` +- `value: string` — optional explicit value ("1", "0xFF") + +**NamespaceDeclaration** (Statement-level, has children): +- `name: string` — namespace name (empty for anonymous) +- Children role "body": vector of statements/declarations inside the namespace + +**TypeAlias** (Statement-level): +- `aliasName: string` — the new name +- `targetType: string` — what it aliases +- `isUsing: bool` — `using` (true) vs `typedef` (false) + +**Modifies: `editor/src/ast/Serialization.h`** +- propertiesToJson/createNode/setPropertiesFromJson for all 4 types + +**Modifies: `editor/src/CompactAST.h`** +- getNodeName for EnumDeclaration, EnumMember, NamespaceDeclaration, TypeAlias + +**Tests:** EnumDeclaration construction, scoped vs unscoped, EnumMember with explicit +value, enum with 4 members as children, NamespaceDeclaration with body children, +TypeAlias using vs typedef, JSON roundtrip for all 4 types, compact AST names, +nested namespace (namespace inside namespace), enum inside namespace. + +### Step 339: C++ Parser — Preprocessor, Enum, Namespace (12 tests) +**Goal:** Parse these constructs from C++ source code. + +**Modifies: `editor/src/ast/CppParser.h`** +- Parse `#include "path"` and `#include ` → IncludeDirective +- Parse `#pragma once` and `#pragma ...` → PragmaDirective +- Parse `#define NAME(args) body` → MacroDefinition +- Parse `enum class Name : Type { A = 1, B = 2 }` → EnumDeclaration + EnumMembers +- Parse `enum Name { ... }` → EnumDeclaration (isScoped=false) +- Parse `namespace Name { ... }` → NamespaceDeclaration with body +- Parse `using Name = Type;` → TypeAlias (isUsing=true) +- Parse `typedef Type Name;` → TypeAlias (isUsing=false) + +**Tests:** #include system/local parsed, #pragma once, #define object-like, +#define function-like, enum class with values, plain enum, namespace with +contents, using alias, typedef, mixed file with preprocessor + namespace + +enum + functions, backward compat (existing C++ tests pass), real Whetstone +header snippet. + +### Step 340: C++ Generator — Preprocessor, Enum, Namespace (12 tests) +**Goal:** CppGenerator produces correct output for all new node types. + +**Modifies: `editor/src/ast/CppGenerator.h`** +- Generate `#include "path"` / `#include ` with correct brackets +- Generate `#pragma once` +- Generate `#define NAME(args) body` +- Generate `enum class Name : Type { A = 1, B };` +- Generate `namespace Name { ... }` +- Generate `using Name = Type;` + +**Cross-language considerations:** +- Python: no preprocessor, no enums (use class), no namespaces (use modules) +- Java: import instead of include, enum keyword, package instead of namespace +- Rust: use instead of include, enum, mod instead of namespace +- Add basic adaptation in generators for enum/namespace equivalents + +**Tests:** #include output, #pragma output, #define output, enum class output, +namespace output, using alias output, round-trip parse → generate → verify, +cross-language: C++ enum → Java enum, C++ namespace → Java package (comment), +C++ enum → Python class, C++ enum → Rust enum, mixed file generation. + +### Step 341: Phase 12d Integration + Sprint 12 Summary (8 tests) +**Goal:** Full C++ depth validation and sprint verification. + +**Tests:** +1. Parse realistic C++ header: includes + pragma + namespace + enum + class + functions → full AST +2. Round-trip: complex C++ header → AST → generate → parse again → equivalent +3. Enum inside namespace: parse → verify scoping in AST +4. Cross-language: C++ header → Java output with imports + package + enums adapted +5. Combined workflow + C++: create skeleton with C++ class patterns → workflow → route +6. All new AST nodes serialize/deserialize correctly (batch verification) +7. Sprint 12 totals: 56+ MCP tools, all new C++ constructs, workflow engine operational +8. C++ self-hosting progress: parse simplified Whetstone header fragment → verify structure + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 12a | 320-325 | 68 | WorkItem model, task queue, workflow state, persistence, RPC/MCP | +| 12b | 326-331 | 68 | Routing engine, workers, context assembly, review gates, RPC/MCP | +| 12c | 332-336 | 56 | C++ multiple inheritance, CRTP, template classes, cross-language | +| 12d | 337-341 | 56 | C++ preprocessor, enum, namespace, type alias, generators | +| **Total** | **320-341** | **~248** | 22 steps | + +**MCP tool count projection:** 56+ tools (42 from Sprint 11 + 8 workflow + 4 routing + 2 review) + +--- + +## Key Files to Modify + +| File | Changes | +|------|---------| +| `editor/src/HeadlessEditorState.h` | Add WorkflowState, RoutingEngine, WorkerRegistry members | +| `editor/src/HeadlessAgentRPCHandler.h` | 12 new RPC methods (8 workflow + 4 routing) | +| `editor/src/AgentPermissionPolicy.h` | Permission mapping for 12 new methods | +| `editor/src/MCPServer.h` | registerWorkflowExecutionTools() + registerRoutingTools() + registerReviewTools() | +| `editor/src/ast/ClassDeclaration.h` | BaseClass struct, baseClasses vector, diamond detection | +| `editor/src/ast/GenericType.h` | isClassTemplate, isVariadic, isCRTP detection | +| `editor/src/ast/CppParser.h` | Inheritance, templates, preprocessor, enum, namespace parsing | +| `editor/src/ast/CppGenerator.h` | Multiple inheritance, templates, preprocessor, enum, namespace output | +| `editor/src/ast/Serialization.h` | 7+ new node types in all 3 dispatch points | +| `editor/src/CompactAST.h` | getNodeName for all new node types | +| `editor/src/CrossLanguageProjector.h` | adaptInheritance() for cross-language inheritance mapping | +| `editor/CMakeLists.txt` | Test targets for steps 320-341 | + +**New files:** +- `editor/src/WorkItem.h` — WorkItem struct, WorkItemResult, lifecycle state machine +- `editor/src/TaskQueue.h` — Priority queue with dependency resolution +- `editor/src/WorkflowState.h` — Project-level workflow tracking, phase detection, stats +- `editor/src/WorkflowPersistence.h` — Workflow sidecar save/load +- `editor/src/RoutingEngine.h` — Annotation-to-dispatch routing logic +- `editor/src/WorkerRegistry.h` — Worker abstractions (deterministic, template, agent, human) +- `editor/src/ContextAssembler.h` — Context window assembly with budget enforcement +- `editor/src/ReviewGate.h` — Auto-approve rules and review policy +- `editor/src/ast/PreprocessorNodes.h` — IncludeDirective, PragmaDirective, MacroDefinition +- `editor/src/ast/EnumNamespaceNodes.h` — EnumDeclaration, EnumMember, NamespaceDeclaration, TypeAlias +- `editor/tests/step320_test.cpp` through `editor/tests/step341_test.cpp` + +--- + +## Verification + +After each phase: +1. Build all new test targets: `cmake --build . --target stepNNN_test` +2. Run each test: `./stepNNN_test` — expect "Results: 12/12" (or 8/8 for integration) +3. Phase integration test validates no regressions on earlier steps + +Full sprint verification: +- All ~248 tests pass across steps 320-341 +- Run full test suite (steps 245-341) to verify no regressions +- MCP `tools/list` returns 56+ tools +- Create skeleton → workflow → route → execute deterministic tasks → verify results +- Routing engine produces sensible decisions for varied annotation patterns +- C++ parser handles: multiple inheritance, CRTP, preprocessor, enum class, namespace +- Parse simplified Whetstone header → verify AST structure (self-hosting progress) +- Cross-language: C++ → Java/Python/Rust inheritance adaptation works +- All 10 languages parse and generate correctly (no regressions) + +--- + +## Dependencies on Sprint 11 + +Sprint 12 assumes all Sprint 11 phases are complete: +- **11a** (done): Semanno format, annotation codegen +- **11b**: Validation rules (E0600-E1299) — used by routing engine to assess risk +- **11c**: 9 new AST nodes including ClassDeclaration — Sprint 12c extends it +- **11d**: Kotlin + C# — 10 languages total +- **11e**: SkeletonAST, routing annotations, inference engine — Sprint 12a/12b build directly on these diff --git a/sprint13_plan.md b/sprint13_plan.md new file mode 100644 index 0000000..56f785e --- /dev/null +++ b/sprint13_plan.md @@ -0,0 +1,526 @@ +# Sprint 13 Plan: GUI Overhaul Phase 1 + +## Context + +The human-in-the-loop thesis is load-bearing. If humans are supposed to be architects +and reviewers — not button-pushers — then the interface must make the system intuitive +to understand. The current GUI has floating windows, no key combination symbols, and +doesn't look like a professional product. This sprint fixes that. + +Sprint 12 delivered the workflow model and routing engine. The GUI needs to surface +that workflow state — but the heavy workflow visualization comes in Sprint 19. This +sprint focuses on the structural foundation: docking, styling, navigation, and key +symbols so the editor feels like a real tool. + +**Current GUI state:** +- ImGui-based with SDL backend +- Text editor with syntax highlighting +- AST tree view panel +- Diagnostics panel +- Annotation panel +- Floating windows (not docked) +- No keyboard shortcut display or customization +- Default ImGui color scheme + +**Sprint 13 Deliverables:** +- All windows docked in a professional layout +- Keyboard shortcut system with displayable key combination symbols +- Professional dark color scheme with consistent visual language +- Clean navigation: panel switching, breadcrumbs, command palette +- Status bar with project/workflow/language info +- Resizable, saveable layout that persists across sessions + +--- + +## Phase 13a: Docking Layout (Steps 342-346) + +*Convert every floating window to a docked panel in a professional layout.* + +### Step 342: ImGui Docking Infrastructure (12 tests) +**Goal:** Set up the ImGui docking system and define the default layout. + +**Modifies: `editor/src/EditorState.h`** (or new `editor/src/DockingLayout.h`) +- Enable ImGui docking via `io.ConfigFlags |= ImGuiConfigFlags_DockingEnable` +- Define DockSpace as the root of the editor window +- Default layout: + ``` + ┌──────────────┬──────────────────────────────┬──────────────┐ + │ File Tree │ Code Editor │ AST View │ + │ │ │ │ + │ │ │ │ + │ ├──────────────────────────────┤ │ + │ │ Diagnostics / Output │ │ + └──────────────┴──────────────────────────────┴──────────────┘ + ``` +- Left dock: file tree / project navigator (20% width) +- Center: code editor (60% width, split top/bottom) +- Right dock: AST view / annotations (20% width) +- Bottom: diagnostics, output, workflow status (25% height) + +**Tests:** docking enabled, DockSpace renders, default layout has 4 regions, +panels render in correct regions, no floating windows, window positions persist +after frame, layout ID stable across frames, DockSpace fills viewport. + +### Step 343: Panel Registration System (12 tests) +**Goal:** Every panel registers itself with a panel manager that controls +docking position, visibility, and ordering. + +**New file: `editor/src/PanelManager.h`** (~300 lines) +- `PanelInfo` struct: id, title, defaultDock (left/center/right/bottom), + isVisible, order, icon +- `PanelManager` class: + - `registerPanel(id, title, dock, renderFn)` — register a panel + - `setVisible(id, bool)` — show/hide + - `isVisible(id) -> bool` + - `renderAll()` — render all visible panels in docked positions + - `getPanelList() -> vector` — for UI menus + - `togglePanel(id)` — flip visibility + - `focusPanel(id)` — bring panel to front in its dock + +**Registered panels:** +- `file-tree` — project file browser (left) +- `code-editor` — main text editor (center-top) +- `ast-view` — AST tree visualization (right) +- `annotations` — annotation list/editor (right, tabbed with AST) +- `diagnostics` — error/warning list (bottom) +- `output` — generated code output (bottom, tabbed with diagnostics) +- `workflow` — workflow status summary (bottom, tabbed) +- `properties` — selected node properties (right, tabbed) + +**Tests:** register 8 panels, all render without error, visibility toggle, +focus panel brings to front, default dock positions, panel ordering, +hidden panel doesn't render, getPanelList returns all registered. + +### Step 344: File Tree Panel (12 tests) +**Goal:** Replace any floating file browser with a docked file tree that +shows the workspace structure with icons and expand/collapse. + +**New file: `editor/src/panels/FileTreePanel.h`** (~250 lines) +- Tree view of workspace files using FileTree.h data +- Expand/collapse directories +- File type icons (or text indicators): source files, headers, configs +- Click to open file in editor (calls openFile RPC internally) +- Highlight active file +- Respect .gitignore (already in FileTree.h) +- Right-click context menu: new file, rename, delete (future) + +**Tests:** renders workspace tree, directories expandable, click opens file, +active file highlighted, .gitignore respected, empty workspace shows message, +nested directories render correctly, file count matches workspace index. + +### Step 345: Tabbed Bottom Panel (12 tests) +**Goal:** Diagnostics, output, and workflow share the bottom dock as tabs. + +**Modifies: existing diagnostic/output rendering** +- Tab bar: Diagnostics | Output | Workflow | Console +- Diagnostics tab: structured diagnostic list with severity icons, + click-to-navigate (jumps to error line in editor) +- Output tab: generated code with syntax highlighting +- Workflow tab: summary stats (placeholder for Sprint 19 deep visualization) +- Console tab: MCP tool log / agent interaction log +- Tab badge: unread count (e.g., "Diagnostics (3)" when 3 new errors) + +**Tests:** 4 tabs render, tab switching works, diagnostics click navigates +to line, output shows generated code, workflow shows stats placeholder, +badge count updates on new diagnostic, default tab is diagnostics, tabs +persist across frames. + +### Step 346: Phase 13a Integration — Layout Persistence (8 tests) +**Goal:** Save and restore dock layout across sessions. + +**Modifies: `editor/src/DockingLayout.h`** (or PanelManager) +- `saveLayout(path)` — serialize ImGui dock state + panel visibility to JSON +- `loadLayout(path)` — restore from JSON +- Auto-save on exit, auto-load on startup +- Default layout if no saved state exists +- Path: `.whetstone/layout.json` in workspace + +**Tests:** +1. Default layout applied on first run +2. Resize panel → save → reload → same size +3. Hide panel → save → reload → still hidden +4. Move panel to different dock → save → reload → same dock +5. Tab order preserved +6. Multiple workspaces have independent layouts +7. Corrupt layout file → falls back to default +8. Layout file location correct (.whetstone/layout.json) + +--- + +## Phase 13b: Color Scheme + Visual Polish (Steps 347-350) + +*Professional dark theme with consistent visual hierarchy.* + +### Step 347: Theme System (12 tests) +**Goal:** Centralized theme definition that every panel reads from. + +**New file: `editor/src/Theme.h`** (~300 lines) +- `WhetstoneTheme` struct with named semantic colors: + - `bg`, `bgAlt`, `bgPanel`, `bgPopup` + - `text`, `textDim`, `textAccent`, `textError`, `textWarning`, `textSuccess` + - `border`, `borderFocused` + - `accent`, `accentHover`, `accentActive` — primary brand color + - `selectionBg`, `selectionText` + - `scrollbar`, `scrollbarHover` + - `diagnostic_error`, `diagnostic_warning`, `diagnostic_info`, `diagnostic_hint` + - `syntax_keyword`, `syntax_string`, `syntax_number`, `syntax_comment`, + `syntax_function`, `syntax_type`, `syntax_operator`, `syntax_annotation` +- `applyTheme(theme)` — sets all ImGui style colors from theme +- `getDefaultDarkTheme() -> WhetstoneTheme` — professional dark theme +- `getDefaultLightTheme() -> WhetstoneTheme` — clean light alternative +- Font configuration: primary (monospace), UI (sans-serif), sizes + +**Color palette (dark theme):** +- Background: deep blue-grey (#1a1d23, #22252b, #2a2d35) +- Text: warm white (#e0e0e0), dim (#808890), accent (#7aa2f7) +- Borders: subtle (#3a3d45), focused (#7aa2f7) +- Errors: soft red (#f7768e), warnings: amber (#e0af68), + success: green (#9ece6a), info: blue (#7dcfff) +- Syntax: keywords blue (#7aa2f7), strings green (#9ece6a), + numbers orange (#ff9e64), comments grey (#565f89) + +**Tests:** dark theme applies without crash, all semantic colors defined, +no color is pure black or pure white (contrast), light theme alternative, +applyTheme sets ImGui colors, font sizes reasonable, theme roundtrip to JSON, +syntax colors distinct from each other, diagnostic colors match severity, +accent color used for focused elements. + +### Step 348: Syntax Highlighting Refresh (12 tests) +**Goal:** Update the text editor's syntax highlighting to use theme colors +and cover all 10 supported languages. + +**Modifies: syntax highlighting in text editor component** +- Use theme's syntax colors instead of hardcoded values +- Language-aware highlighting (keywords differ per language) +- Semanno comment highlighting: `@semanno:type(key=value)` gets distinct + annotation color +- Line numbers in `textDim` color +- Current line highlight with subtle `bgAlt` background +- Selection uses `selectionBg` / `selectionText` +- Matching bracket highlight + +**Tests:** Python keywords highlighted, C++ keywords highlighted, Kotlin +keywords highlighted, Semanno comments get annotation color, strings in +string color, numbers in number color, comments in comment color, current +line highlighted, line numbers dimmed, selection visible, theme change +updates highlighting, bracket matching visible. + +### Step 349: Icon System + Visual Indicators (12 tests) +**Goal:** Consistent iconography across all panels using Unicode symbols +(no external icon font dependency — keep it header-only). + +**New file: `editor/src/Icons.h`** (~100 lines) +- Unicode symbol constants for common items: + - File types: source, header, config, folder, image + - Diagnostics: error circle, warning triangle, info, hint + - Workflow: pending clock, in-progress spinner, complete check, + rejected cross, review eye + - AST: function, class, variable, annotation, module + - Navigation: arrow right, arrow down, search, settings +- `renderIcon(iconType, color)` — render icon with theme color +- Fallback: if Unicode doesn't render, use text abbreviations (ERR, WARN, etc.) + +**Tests:** all icon constants defined, renderIcon doesn't crash, error icon +is distinct from warning icon, file type icons cover 10 languages, workflow +status icons cover all states, fallback text works, icon + text alignment +correct, icon colors respect theme. + +### Step 350: Phase 13b Integration — Visual Consistency (8 tests) +**Goal:** Every panel uses the theme and icons consistently. + +**Tests:** +1. All panels use theme background colors (no default ImGui grey) +2. Diagnostic panel uses severity icons + colors from theme +3. AST panel uses node type icons +4. File tree uses folder/file icons +5. Status text uses textDim for secondary info +6. Focused panel border uses accent color +7. Theme switch (dark→light) updates all panels +8. No visual artifacts: no overlapping text, no clipped content, no invisible text + +--- + +## Phase 13c: Keyboard System + Key Symbols (Steps 351-355) + +*The key combination symbol system the user has been asking for.* + +### Step 351: Keybinding Registry (12 tests) +**Goal:** Centralized keybinding system where every action has a named binding +that can be displayed, customized, and persisted. + +**New file: `editor/src/KeybindingRegistry.h`** (~350 lines) + +**KeyCombo struct:** +- `modifiers: uint8_t` — bitmask: Ctrl, Alt, Shift, Super/Meta +- `key: int` — ImGui key code +- `toString() -> string` — "Ctrl+Shift+P", "Alt+F4", etc. +- `toSymbols() -> string` — "⌃⇧P", "⌥F4" (platform-aware symbols) + - macOS: ⌘ ⌥ ⌃ ⇧ + - Linux/Windows: Ctrl Alt Shift Super +- `matches(ImGuiIO&) -> bool` — check if currently pressed + +**KeyAction enum** (or string-based for extensibility): +- `OpenFile`, `SaveBuffer`, `SaveAll`, `CloseBuffer` +- `TogglePanel_FileTree`, `TogglePanel_AST`, `TogglePanel_Diagnostics` +- `FocusEditor`, `FocusCommandPalette` +- `NextDiagnostic`, `PrevDiagnostic` +- `RunPipeline`, `GenerateCode` +- `Undo`, `Redo` +- `FindInFile`, `FindInProject` +- `ToggleAnnotations`, `InferAnnotations` +- `CreateWorkflow`, `RouteAll` + +**KeybindingRegistry class:** +- `bind(action, combo)` — set binding for action +- `getBinding(action) -> KeyCombo` — get current binding +- `getSymbols(action) -> string` — get display symbols for an action +- `processInput(ImGuiIO&) -> optional` — check all bindings +- `getAll() -> map` — for settings UI +- `loadDefaults()` — standard keybindings +- `saveToJson(path)` / `loadFromJson(path)` — persistence + +**Default bindings:** +- Ctrl+S: SaveBuffer, Ctrl+Shift+S: SaveAll +- Ctrl+P: CommandPalette +- Ctrl+B: TogglePanel_FileTree +- Ctrl+Shift+D: TogglePanel_Diagnostics +- F5: RunPipeline +- Ctrl+Z: Undo, Ctrl+Shift+Z: Redo +- Ctrl+F: FindInFile, Ctrl+Shift+F: FindInProject +- F8: NextDiagnostic, Shift+F8: PrevDiagnostic + +**Tests:** bind and retrieve, default bindings loaded, key combo toString, +key combo toSymbols (platform-aware), processInput detects active combo, +save/load roundtrip, rebind action, conflict detection (same combo for +two actions), getSymbols returns displayable string, modifier bitmask +correct, matches detects key press. + +### Step 352: Key Symbol Rendering (12 tests) +**Goal:** Display key combination symbols inline in menus, tooltips, and +the command palette — like every professional editor does. + +**New file: `editor/src/KeySymbolRenderer.h`** (~150 lines) +- `renderKeySymbol(combo, theme)` — renders a styled key badge inline + - Background: slightly lighter than panel bg + - Border: subtle rounded rect + - Text: key symbol in small monospace font + - Example: `[⌃S]` rendered as a neat badge +- `renderActionWithKey(label, action, registry, theme)` — renders + "Save Buffer ⌃S" with right-aligned key symbols (menu item style) +- `renderKeyChord(combos, theme)` — for multi-key sequences + ("⌃K ⌃S" style chords) + +**Tests:** single key renders, modifier+key renders, platform symbols correct, +menu item layout (label left, key right), chord rendering, key badge has +background, theme colors used, empty binding shows nothing, special keys +(Enter, Escape, Tab, F-keys) render correctly, long label doesn't overlap +key badge. + +### Step 353: Command Palette (12 tests) +**Goal:** Ctrl+P opens a fuzzy-search command palette (VS Code style) that +lists all available actions with their key symbols. + +**New file: `editor/src/CommandPalette.h`** (~300 lines) +- Modal popup centered in editor +- Text input at top with fuzzy search +- List of matching actions below, each showing: + - Icon (from Icons.h) + - Action name ("Save Buffer", "Toggle AST Panel", "Run Pipeline") + - Key binding symbol (right-aligned) +- Enter executes selected action +- Escape closes +- Arrow keys navigate list +- Search matches action name, panel name, and command aliases +- Recently used actions appear first + +**Action sources:** +- All KeyAction entries from KeybindingRegistry +- Panel toggles from PanelManager +- RPC methods (RunPipeline, GenerateCode, InferAnnotations, etc.) +- File operations (Open, Save, Close) + +**Tests:** palette opens on Ctrl+P, fuzzy search filters actions, Enter +executes action, Escape closes, key symbols displayed, arrow navigation, +recent actions first, empty search shows all, partial match works, +palette closes after execution, multiple action sources merged, no +duplicate entries. + +### Step 354: Menu Bar with Key Symbols (12 tests) +**Goal:** Top menu bar (File, Edit, View, Tools, Workflow) with key +symbols on every menu item. + +**Modifies: main editor rendering** +- **File:** New | Open | Save (⌃S) | Save All (⌃⇧S) | Close +- **Edit:** Undo (⌃Z) | Redo (⌃⇧Z) | Find (⌃F) | Find in Project (⌃⇧F) +- **View:** Toggle File Tree (⌃B) | Toggle AST | Toggle Diagnostics (⌃⇧D) | + Toggle Annotations | Reset Layout +- **Tools:** Run Pipeline (F5) | Generate Code | Infer Annotations | + Validate | Export Semanno +- **Workflow:** Create Workflow | Route All | View Ready Tasks | Save Workflow +- **Help:** About | Keyboard Shortcuts + +Every menu item uses `renderActionWithKey()` for consistent key symbol display. + +**Tests:** all 5 menus render, File menu has Save with ⌃S, Edit menu has +Undo with ⌃Z, View menu toggles panels, Tools menu runs pipeline, menu +items execute correct actions, key symbols visible, menu closes after +selection, separator lines between groups, workflow menu items present, +submenu rendering. + +### Step 355: Phase 13c Integration — Keyboard Shortcuts Help Panel (8 tests) +**Goal:** A dedicated keyboard shortcuts panel and verification that the +entire keybinding system works end-to-end. + +**New panel: Keyboard Shortcuts** (accessible from Help menu or Ctrl+?) +- Grouped by category: File, Edit, View, Tools, Workflow, Navigation +- Each row: action name | key symbol | description +- Search/filter bar +- "Customize" button (opens keybinding editor — basic version) + +**Tests:** +1. All default bindings displayed in shortcuts panel +2. Ctrl+S triggers save action +3. Ctrl+P opens command palette +4. Command palette shows key symbols for all actions +5. Menu items show correct key symbols +6. Rebind action → shortcut panel updates +7. Key symbols render correctly on current platform +8. Full keyboard-driven workflow: Ctrl+P → type "pipeline" → Enter → pipeline runs + +--- + +## Phase 13d: Navigation + Status Bar (Steps 356-360) + +*Clean navigation: breadcrumbs, status bar, and go-to-definition.* + +### Step 356: Breadcrumb Navigation (12 tests) +**Goal:** Breadcrumb bar above the code editor showing the current location +in the AST hierarchy: `Module > Class > Method > Function`. + +**New file: `editor/src/panels/BreadcrumbBar.h`** (~150 lines) +- Displays path from module root to cursor position's AST node +- Each breadcrumb is clickable → navigates to that node +- Dropdown on each segment → shows siblings (other methods in class, etc.) +- Updates as cursor moves +- Uses compact AST names from CompactAST::getNodeName() + +**Tests:** breadcrumb shows module name, nested path updates with cursor, +click navigates to node, dropdown shows siblings, empty file shows module only, +deep nesting (4+ levels) renders without overflow, breadcrumb uses theme +colors, annotation nodes appear in path. + +### Step 357: Status Bar (12 tests) +**Goal:** Bottom status bar showing project state at a glance. + +**New file: `editor/src/panels/StatusBar.h`** (~200 lines) +- Left section: language indicator + file path +- Center section: cursor position (Ln 42, Col 8) | encoding | line ending +- Right section: workflow status (if active), MCP connection indicator, + diagnostic summary (3 errors, 2 warnings) + +**Status bar segments:** +- Language badge: colored badge with language name ("Python", "C++", "Kotlin") +- Workflow badge: phase indicator if workflow active + ("Executing 3/10" with progress ring) +- Diagnostic summary: clickable → jumps to diagnostics panel +- MCP indicator: green dot if MCP connected, grey if headless-only + +**Tests:** language badge shows current buffer language, cursor position +updates, diagnostic count accurate, workflow badge shows when active, +click diagnostic count focuses diagnostics panel, status bar uses theme +colors, language changes when switching buffers, encoding display, +all sections fit without overflow. + +### Step 358: Go-to-Definition + Go-to-Symbol (12 tests) +**Goal:** Navigate by AST structure, not just text search. + +**Modifies: `editor/src/panels/` or code editor component** +- **Go-to-Definition (F12):** Click on a function call → jump to its definition + (uses cross-file symbol resolution from Sprint 9d) +- **Go-to-Symbol (Ctrl+Shift+O):** Open a symbol picker showing all + functions/classes/variables in current file with icons +- **Go-to-Symbol in Project (Ctrl+T):** Same but across all open buffers +- Uses existing `searchProject` and `getInScopeSymbols` RPC infrastructure + +**Tests:** F12 on function call jumps to definition, Ctrl+Shift+O lists file +symbols, Ctrl+T lists project symbols, symbol list shows icons by type, +filter works on symbol list, cross-file jump opens target buffer, no match +shows message, symbols sorted by line number. + +### Step 359: Find and Replace Improvements (12 tests) +**Goal:** In-file and project-wide find/replace with result highlighting. + +**Modifies: text editor component** +- Ctrl+F: find bar appears at top of editor (not floating window) +- Result highlighting in editor (all matches highlighted, current match + distinct color) +- Match count display ("3 of 17") +- Enter: next match, Shift+Enter: previous match +- Ctrl+H: replace mode (adds replace input field) +- Ctrl+Shift+F: project-wide find (results in bottom panel) +- Search uses `searchProject` for AST-aware results when possible + +**Tests:** Ctrl+F opens find bar, typing filters matches, match highlighting +visible, match count accurate, Enter cycles through matches, Ctrl+H shows +replace field, replace works, project-wide search shows results in bottom +panel, Escape closes find bar, empty search clears highlights. + +### Step 360: Phase 13d Integration + Sprint 13 Summary (8 tests) +**Goal:** Full navigation workflow and sprint verification. + +**Tests:** +1. Full keyboard workflow: open file → navigate to function → go-to-definition → breadcrumb click back +2. Command palette → "find" → project-wide search → navigate to result +3. Status bar reflects buffer switches (language, cursor, diagnostics) +4. All panels docked, no floating windows +5. Theme consistent across all panels and navigation elements +6. Key symbols visible in menus, palette, and shortcuts panel +7. Layout save → restart → same layout restored +8. Sprint 13 totals: all panels docked, theme applied, keybindings functional, navigation operational + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 13a | 342-346 | 56 | Docking layout, panel system, file tree, tabbed bottom | +| 13b | 347-350 | 44 | Theme system, syntax highlighting, icons, visual consistency | +| 13c | 351-355 | 56 | Keybinding registry, key symbols, command palette, menus | +| 13d | 356-360 | 56 | Breadcrumbs, status bar, go-to-definition, find/replace | +| **Total** | **342-360** | **~212** | 19 steps | + +--- + +## Key Files + +**New files:** +- `editor/src/DockingLayout.h` — ImGui docking setup, default layout, persistence +- `editor/src/PanelManager.h` — panel registration, visibility, rendering +- `editor/src/Theme.h` — semantic color themes, dark/light, font configuration +- `editor/src/Icons.h` — Unicode icon constants, renderIcon +- `editor/src/KeybindingRegistry.h` — keybindings, key combos, persistence +- `editor/src/KeySymbolRenderer.h` — key badge rendering, menu item key display +- `editor/src/CommandPalette.h` — fuzzy-search command palette +- `editor/src/panels/FileTreePanel.h` — docked file tree browser +- `editor/src/panels/BreadcrumbBar.h` — AST breadcrumb navigation +- `editor/src/panels/StatusBar.h` — status bar with language/workflow/diagnostic info + +**Modified files:** +- `editor/src/EditorState.h` — docking enable, layout integration +- `editor/src/main.cpp` — theme application, panel manager initialization +- All existing panel renderers — migrate to PanelManager registration +- `editor/CMakeLists.txt` — test targets for steps 342-360 + +--- + +## Design Principles + +1. **No floating windows.** Every panel docks. Period. +2. **Key symbols everywhere.** If an action has a keybinding, its symbol is visible + wherever that action appears — menus, palette, tooltips, shortcuts panel. +3. **Theme is semantic.** Colors are named by purpose (textError, syntax_keyword) + not by value. Changing the theme changes everything. +4. **Keyboard-first.** Every action reachable by keyboard. Mouse is optional. +5. **Professional defaults.** First-run experience should look polished, not like + a developer's test harness. diff --git a/sprint14_plan.md b/sprint14_plan.md new file mode 100644 index 0000000..ad7ed09 --- /dev/null +++ b/sprint14_plan.md @@ -0,0 +1,530 @@ +# Sprint 14 Plan: Language Batch 1 — C, WebAssembly, Common Lisp, Scheme + +## Context + +Universal transpilation requires universal language coverage. Sprint 14 adds four +languages chosen for strategic reasons: + +- **C** — the lingua franca of legacy code. Billions of lines of C exist that need + modernization. C is also the bridge to assembly, OS kernels, and embedded systems. + Critical for the legacy ingestion pipeline (Sprint 20). +- **WebAssembly (WAT)** — the universal compilation target for the web. If Whetstone + can generate WASM text format, any Whetstone-managed project can target the browser. + Also relevant for plugin systems and sandboxed execution. +- **Common Lisp** — the most powerful macro system in any language. Representing + Lisp's homoiconic code-as-data in the AST tests whether our universal AST is truly + universal. Also covers the Lisp family for AI/research codebases. +- **Scheme** — minimal Lisp dialect, heavily used in teaching and language research. + Shares core concepts with Common Lisp but with different conventions (hygiene, + continuations, tail-call guarantees). Tests that our Lisp representation isn't + just "Common Lisp with a different name." + +Each language follows the established pattern: parser (regex-based, no new tree-sitter +dependency), generator, pipeline integration, cross-language projection tests. + +**After Sprint 14:** 14 language parsers and generators. + +--- + +## Phase 14a: C Parser + Generator (Steps 361-366) + +*C is structurally close to C++ but simpler — no classes, no templates, no namespaces +(pre-C23). The parser can share infrastructure with CppParser but must handle C-specific +constructs: raw pointers everywhere, malloc/free, function pointers, preprocessor-heavy +idioms, and typedef structs.* + +### Step 361: C Parser — Functions, Structs, Enums (12 tests) +**Goal:** Parse C source into Whetstone AST. + +**New file: `editor/src/ast/CParser.h`** (~450 lines) +- `parseC(source) -> unique_ptr` +- `parseCWithDiagnostics(source) -> pair>` + +**Constructs parsed:** +- `int foo(int x, char* s) { ... }` → Function +- `struct Point { int x; int y; };` → ClassDeclaration (isStruct=true) +- `typedef struct { ... } Name;` → ClassDeclaration + TypeAlias +- `enum Color { RED, GREEN, BLUE };` → EnumDeclaration +- `enum { FLAG_A = 1, FLAG_B = 2 };` → anonymous EnumDeclaration +- Variable declarations with type: `int x = 5;`, `char* name;` +- `#include`, `#define`, `#pragma` → preprocessor nodes (from Sprint 12d) +- `static`, `extern`, `const` qualifiers on functions and variables +- Function pointer types: `void (*callback)(int)` → typed parameter + +**Differences from C++ parser:** +- No `class` keyword (only `struct`) +- No `namespace`, no `template`, no `virtual` +- `struct` requires typedef for type name in older C +- Default struct access is public (same as C++) + +**Modifies: `editor/src/ast/Parser.h`** — include CParser.h +**Modifies: `editor/src/Pipeline.h`** — add "c" routing + +**Tests:** function parsing, struct parsing, typedef struct, enum parsing, +preprocessor directives, static/extern qualifiers, function pointers, +variable with pointer type, multiple functions in file, backward compat +(C++ tests unaffected), C-style comments, header guard pattern. + +### Step 362: C Generator (12 tests) +**Goal:** Generate idiomatic C from Whetstone AST. + +**New file: `editor/src/ast/CGenerator.h`** (~400 lines) +- Extends `ProjectionGenerator` and `SemannoAnnotationImpl` +- C syntax: no classes (structs only), function declarations, typedef +- Type mapping: int, float, double, char, void, size_t, pointer types +- Memory: explicit malloc/free patterns (guided by @Owner annotations) +- String handling: char* with strlen/strcpy patterns +- Array syntax: fixed-size arrays, pointer arithmetic +- `commentPrefix() -> "// "` (C99+) or `"/* "` for pre-C99 + +**Cross-language mapping:** +- Python class → C struct + associated functions +- Java/C++ class → C struct + function pointers (vtable pattern) for methods +- Python list → C array with size parameter +- Exceptions → error return codes + errno pattern +- Smart pointers → raw pointers with @Owner annotation to track ownership + +**Modifies: `editor/src/ast/Generator.h`** — include CGenerator.h +**Modifies: `editor/src/Pipeline.h`** — add "c" generator routing +**Modifies: `editor/src/CrossLanguageProjector.h`** — C adaptations + +**Tests:** function generation, struct output, typedef struct output, +enum output, preprocessor output, pointer types, malloc/free for @Owner(Manual), +cross-language: Python → C, cross-language: C++ class → C struct + functions, +Semanno annotation comments, commentPrefix, C-style string handling. + +### Step 363: C Memory Annotation Mapping (12 tests) +**Goal:** C has explicit manual memory management. The annotation system should +map cleanly to C patterns and provide meaningful guidance. + +**Modifies: `editor/src/MemoryStrategyInference.h`** +- C language defaults: Owner(Manual), Lifetime(Scope), Reclaim(Explicit) +- Inference rules for C: + - Function with malloc → @Owner(Manual) + @Reclaim(Explicit) + - Local variable → @Lifetime(Scope) + - Function parameter pointer → @Owner(Borrowed) + - Return pointer → @Owner(Transferred) or @Owner(Borrowed) based on convention + - Static variable → @Lifetime(Static) + +**Modifies: `editor/src/AnnotationInference.h`** +- C-specific inference patterns: + - `goto` usage → @Complexity(high) + @Risk(medium) + - No bounds checking on arrays → @BoundsCheck(unchecked) + - Void pointer casts → @Risk(high) + @Ambiguity(medium) + - Header guard → @Synthetic(guard) + +**Tests:** C defaults applied, malloc → Owner(Manual), local → Lifetime(Scope), +parameter pointer → Owner(Borrowed), goto detected as complexity signal, +void pointer cast flagged, static variable lifetime, header guard pattern, +inference confidence reasonable, cross-language: C Owner(Manual) → +Rust Owner(Box) projection, C Owner(Borrowed) → Rust &T. + +### Step 364: C Integration + Self-Hosting Prep (8 tests) +**Goal:** Full C pipeline and cross-language validation. + +**Tests:** +1. Parse C source → generate C → roundtrip produces equivalent output +2. Cross-language: C struct → Python class with __init__ +3. Cross-language: Python class → C struct + constructor function +4. Cross-language: C with manual memory → Rust with ownership annotations preserved +5. Pipeline.run() with C source + target C++ → struct becomes class +6. Annotation inference on C code: malloc, arrays, function pointers all annotated +7. Parse simplified C header from Whetstone dependencies → verify AST +8. MCP tools recognize "c" language, tools/list unchanged (language is a parameter) + +--- + +## Phase 14b: WebAssembly Text Format (Steps 365-369) + +*WASM text format (WAT) is an S-expression syntax representing a stack-based VM. +This is structurally different from all previous languages — it's lower-level than C, +stack-based rather than expression-based, and has explicit type annotations on every +value. Representing WAT in the AST proves the AST can handle compilation targets, +not just source languages.* + +### Step 365: WAT Parser (12 tests) +**Goal:** Parse WebAssembly text format into Whetstone AST. + +**New file: `editor/src/ast/WatParser.h`** (~400 lines) +- `parseWat(source) -> unique_ptr` +- `parseWatWithDiagnostics(source) -> pair>` + +**WAT constructs → AST mapping:** +- `(module ...)` → Module +- `(func $name (param $x i32) (result i32) ...)` → Function + - WAT types: i32, i64, f32, f64, funcref, externref + - Parameters have explicit types, always + - Result type is explicit +- `(local $name i32)` → Variable +- `(export "name" (func $name))` → Export annotation +- `(import "module" "name" (func ...))` → Import node +- `(global $name (mut i32) ...)` → Variable (isGlobal=true) +- `(memory 1)` → module-level memory declaration (custom annotation) +- `(table ...)` → module-level table declaration +- Instructions as statements: `i32.add`, `local.get`, `call`, `if`, `block`, `loop` + +**Design decision:** WAT instructions map to existing AST nodes where possible +(BinaryOperation for i32.add, FunctionCall for call, IfStatement for if/else). +Stack semantics are represented via annotations (@Exec(stack)) rather than +trying to model the stack in the AST. + +**Modifies: `editor/src/ast/Parser.h`** — include WatParser.h +**Modifies: `editor/src/Pipeline.h`** — add "wat" / "wasm" routing + +**Tests:** module parsing, function with params and result, local variables, +export declarations, import declarations, global variables, i32.add → BinaryOperation, +call → FunctionCall, if/else → IfStatement, nested blocks, empty module, +WAT-specific types preserved in annotations. + +### Step 366: WAT Generator (12 tests) +**Goal:** Generate valid WAT from Whetstone AST. + +**New file: `editor/src/ast/WatGenerator.h`** (~400 lines) +- Extends `ProjectionGenerator` and `SemannoAnnotationImpl` +- S-expression output format +- Type mapping to WASM types: int→i32, float→f32, double→f64, bool→i32 +- Function output: `(func $name (param $x i32) (result i32) ...)` +- Expressions → stack instructions: `a + b` → `local.get $a local.get $b i32.add` +- Control flow: if/else → `(if (result i32) ... (then ...) (else ...))` +- Memory operations guided by annotations +- `commentPrefix() -> ";; "` + +**Cross-language mapping:** +- Any language → WAT: flatten expressions to stack operations +- Classes → function tables + struct-in-memory layout +- Strings → memory offset + length (WAT has no native strings) +- Exceptions → trap or error code return + +**Modifies: `editor/src/ast/Generator.h`** — include WatGenerator.h +**Modifies: `editor/src/Pipeline.h`** — add "wat" generator routing + +**Tests:** function generation, S-expression formatting, type mapping, +arithmetic → stack instructions, if/else → WAT if, function call → call, +export declaration, import declaration, Semanno comments with ;; prefix, +cross-language: Python function → WAT function, round-trip parse → generate, +valid WAT structure (parentheses balanced). + +### Step 367: WAT Annotation Mapping (12 tests) +**Goal:** Meaningful annotations for WASM's unique characteristics. + +**Annotation mappings for WAT:** +- All functions → @Exec(stack) (stack-based execution model) +- Memory operations → @Owner(Manual) (WASM has linear memory, no GC by default) +- Exported functions → @Visibility(public) +- Imported functions → @Link(import, module="name") +- Table-based dispatch → @Shim(vtable) +- Memory size declarations → @Align + @Layout annotations + +**Modifies: `editor/src/MemoryStrategyInference.h`** — WAT defaults +**Modifies: `editor/src/AnnotationInference.h`** — WAT-specific patterns + +**Tests:** WAT function gets @Exec(stack), memory access gets @Owner(Manual), +export → @Visibility(public), import → @Link, defaults applied correctly, +inference confidence, cross-language: WAT annotations preserved when +projecting to C, annotation roundtrip through Semanno format. + +### Step 368: WAT Cross-Language Projection (12 tests) +**Goal:** Validate that WAT works as both source and target in cross-language +projection scenarios. + +**Tests:** +1. Python function → WAT: arithmetic expression flattened to stack ops +2. C function → WAT: direct mapping (both low-level, closest match) +3. WAT → Python: stack ops reconstructed to expression tree +4. WAT → C: near-direct mapping +5. WAT → Rust: memory annotations guide ownership +6. Java class → WAT: method table + memory layout +7. Round-trip: Python → WAT → Python produces equivalent logic +8. Memory annotations preserved across WAT ↔ C projection +9. WAT with imports → any language: imports become function declarations +10. Complex: multi-function WAT module → C output compiles conceptually +11. Semanno roundtrip on WAT source +12. Pipeline.run() with WAT source succeeds + +### Step 369: Phase 14b Integration (8 tests) +**Goal:** WAT fully integrated into the transpilation pipeline. + +**Tests:** +1. Parse WAT → generate WAT → output is valid WAT +2. Cross-language: C → WAT → C roundtrip preserves logic +3. 12 languages all project to WAT as target +4. WAT annotations meaningful and complete +5. Compact AST for WAT module has correct node names +6. Diagnostics work on WAT (parse errors detected) +7. Semanno sidecar save/load for WAT files +8. MCP tools handle "wat" language parameter + +--- + +## Phase 14c: Common Lisp Parser + Generator (Steps 370-373) + +*Common Lisp is homoiconic: code is data, data is code. The S-expression syntax means +the AST representation is almost 1:1 with the source. This is both simpler (no complex +parsing) and harder (macros, multiple return values, condition system, CLOS). This phase +tests whether our universal AST handles a fundamentally different programming paradigm.* + +### Step 370: Common Lisp Parser (12 tests) +**Goal:** Parse Common Lisp source into Whetstone AST. + +**New file: `editor/src/ast/CommonLispParser.h`** (~400 lines) +- `parseCommonLisp(source) -> unique_ptr` +- `parseCommonLispWithDiagnostics(source) -> pair>` + +**CL constructs → AST mapping:** +- `(defun name (params) body)` → Function +- `(defmethod name ((x class)) body)` → MethodDeclaration +- `(defclass name (supers) (slots))` → ClassDeclaration (CLOS) +- `(defvar *name* value)` / `(defparameter *name* value)` → Variable (isGlobal=true) +- `(let ((x 1) (y 2)) body)` → Variable declarations + scope +- `(lambda (params) body)` → LambdaExpression +- `(if test then else)` → IfStatement +- `(cond (test1 result1) ...)` → chain of IfStatements +- `(loop ...)` / `(do ...)` / `(dotimes ...)` → ForStatement variants +- `(funcall fn args)` / `(apply fn args)` → FunctionCall +- `(defmacro name (params) body)` → MacroDefinition (from Sprint 12d) +- `(quote expr)` / `'expr` → annotated expression (@Meta(quoted)) +- `(values x y z)` → multi-value return (annotated with @Contract) + +**Conventions handled:** +- `*earmuffs*` for dynamic/special variables +- `+constants+` for constants +- `predicate-p` naming convention +- `(declare (type fixnum x))` → type annotations +- Multiple package prefixes: `cl:defun`, `my-package:foo` + +**Tests:** defun parsing, defclass parsing, lambda, let bindings, if/cond, +loop variants, defmacro, quote/quasiquote, earmuff detection, nested +S-expressions (deep nesting), mixed definitions in one file, CLOS method. + +### Step 371: Common Lisp Generator (12 tests) +**Goal:** Generate idiomatic Common Lisp from Whetstone AST. + +**New file: `editor/src/ast/CommonLispGenerator.h`** (~400 lines) +- Extends `ProjectionGenerator` and `SemannoAnnotationImpl` +- S-expression output: proper indentation, one form per line +- Type mapping: fixnum, single-float, double-float, string, t, nil, list, vector +- Memory: garbage collected (CL has GC), @Owner defaults to Shared_GC +- Convention adherence: earmuffs for specials, `-p` for predicates +- `commentPrefix() -> ";; "` + +**Cross-language mapping:** +- Python class → CLOS defclass with defmethod +- C++ class → CLOS class + condition system for exceptions +- Java interface → CLOS generic function protocol +- Python list → CL list +- Static typing → (declare ...) type annotations +- Exceptions → CL condition/restart system + +**Tests:** defun generation, defclass output, lambda output, let bindings, +S-expression formatting (correct indentation), type declarations from +annotations, cross-language: Python → Common Lisp, cross-language: Java → +Common Lisp, Semanno comments with ;; prefix, round-trip parse → generate, +CLOS method dispatch, condition system from exception annotations. + +### Step 372: Lisp Annotation Mapping (12 tests) +**Goal:** Map Lisp's unique features to the annotation taxonomy. + +**CL-specific annotations:** +- Macros → @Meta(quoted) + @Synthetic(macro) +- Dynamic variables → @Binding(dynamic) (vs @Binding(static) for lexical) +- Multiple return values → @Contract(returns=["x", "y"]) +- Tail-recursive functions → @TailCall (CL doesn't guarantee TCO but hints are useful) +- CLOS methods → @Visibility + method combination annotations +- Special declarations → @Complexity(declared-type) +- `(optimize (speed 3) (safety 0))` → @Policy(perf=critical, style=literal) + +**Modifies: `editor/src/MemoryStrategyInference.h`** — CL defaults (GC) +**Modifies: `editor/src/AnnotationInference.h`** — CL-specific patterns + +**Tests:** macro → @Meta(quoted), dynamic var → @Binding(dynamic), multiple +values → @Contract, tail recursion detected, CLOS class → @Visibility, +optimize declaration → @Policy mapping, GC defaults, inference confidence, +cross-language: CL @Binding(dynamic) → Python (no direct equivalent, comment), +CL @Binding(dynamic) → C++ (thread_local hint). + +### Step 373: Common Lisp Integration (8 tests) +**Goal:** Full CL pipeline and cross-language validation. + +**Tests:** +1. Parse CL → generate CL → roundtrip equivalent +2. Cross-language: Python → Common Lisp (class → CLOS) +3. Cross-language: Common Lisp → Python (defclass → class) +4. Cross-language: Common Lisp → Java (CLOS → class hierarchy) +5. Annotation inference on CL code: macros, dynamic vars, tail calls +6. Semanno roundtrip on CL source +7. Pipeline.run() with CL source succeeds +8. Compact AST for Lisp module has meaningful node names + +--- + +## Phase 14d: Scheme Parser + Generator (Steps 374-377) + +*Scheme shares Lisp syntax but has different conventions: hygiene, first-class +continuations, guaranteed tail calls, and a minimalist standard library. The AST +must distinguish Scheme from Common Lisp despite surface similarity.* + +### Step 374: Scheme Parser (12 tests) +**Goal:** Parse Scheme source into Whetstone AST. + +**New file: `editor/src/ast/SchemeParser.h`** (~350 lines) +- `parseScheme(source) -> unique_ptr` +- `parseSchemeWithDiagnostics(source) -> pair>` + +**Scheme constructs → AST mapping:** +- `(define (name params) body)` → Function +- `(define name value)` → Variable +- `(lambda (params) body)` → LambdaExpression +- `(let ((x 1)) body)` / `(let* ...)` / `(letrec ...)` → scoped Variables +- `(if test then else)` → IfStatement +- `(cond ...)` → chained IfStatements +- `(do ((var init step) ...) (test result) body)` → ForStatement +- `(call-with-current-continuation proc)` / `(call/cc proc)` → FunctionCall + with @Exec(continuation) annotation +- `(define-syntax name ...)` → MacroDefinition with @Meta(hygienic) +- `(syntax-rules ...)` → macro body +- `(values x y)` → multi-value return +- `(begin ...)` → statement sequence + +**Differences from Common Lisp parser:** +- `define` instead of `defun`/`defvar` +- No CLOS (use records or closures for objects) +- Hygienic macros (define-syntax) instead of defmacro +- Guaranteed tail call optimization +- `#t`/`#f` instead of `t`/`nil` for booleans +- `'()` for empty list instead of `nil` + +**Tests:** define function, define variable, lambda, let/let*/letrec, +if/cond, call/cc recognized, define-syntax, #t/#f booleans, nested +S-expressions, mixed definitions, tail-call pattern, Scheme-specific +comment `;` handling. + +### Step 375: Scheme Generator (12 tests) +**Goal:** Generate idiomatic Scheme from Whetstone AST. + +**New file: `editor/src/ast/SchemeGenerator.h`** (~350 lines) +- Extends `ProjectionGenerator` and `SemannoAnnotationImpl` +- S-expression output with Scheme conventions +- Type mapping: minimal (Scheme is dynamically typed) — number, string, boolean, + pair, list, vector, procedure +- Booleans: #t / #f +- Empty list: '() +- `commentPrefix() -> "; "` + +**Cross-language mapping:** +- Python class → Scheme record type + procedures +- C struct → Scheme vector with accessor procedures +- Java interface → Scheme protocol (procedure signatures) +- Exceptions → Scheme `guard` / `with-exception-handler` +- Tail calls preserved where source had @TailCall +- Continuations: unique to Scheme, generated from @Exec(continuation) annotations + +**Tests:** define generation, lambda output, let bindings, boolean #t/#f, +S-expression formatting, cross-language: Python → Scheme, cross-language: +Common Lisp → Scheme (closest mapping), cross-language: Scheme → Python, +Semanno comments with ; prefix, round-trip, tail-call preservation, +continuation annotations. + +### Step 376: Scheme-Specific Annotations + CL Differentiation (12 tests) +**Goal:** Ensure Scheme and Common Lisp are properly differentiated in +annotations and cross-language projection. + +**Scheme-specific annotations:** +- All functions checked for tail position → @TailCall where applicable + (Scheme guarantees TCO, so @TailCall is a correctness annotation, not just a hint) +- call/cc → @Exec(continuation) +- define-syntax → @Meta(hygienic) (vs CL's @Meta(quoted) for unhygienic macros) +- Proper tail recursion → @Loop(tail-recursive) (Scheme idiom for iteration) + +**CL vs Scheme differentiation:** +- CL `defmacro` → @Meta(quoted), Scheme `define-syntax` → @Meta(hygienic) +- CL `*specials*` → @Binding(dynamic), Scheme uses `make-parameter` → @Binding(parameter) +- CL CLOS → ClassDeclaration, Scheme records → ClassDeclaration (isStruct=true) +- CL condition system → @Exception(condition), Scheme guard → @Exception(guard) + +**Modifies: `editor/src/MemoryStrategyInference.h`** — Scheme defaults (GC) +**Modifies: `editor/src/AnnotationInference.h`** — Scheme-specific patterns + +**Tests:** tail call detected in Scheme (guarantee vs hint), call/cc annotation, +hygienic macro annotation, CL→Scheme projection preserves semantics but changes +idiom, Scheme→CL projection, both languages GC defaults, differentiated +@Meta types, @Binding types differ, @Exception types differ, cross-language +annotation fidelity. + +### Step 377: Phase 14d Integration + Sprint 14 Summary (8 tests) +**Goal:** All 4 new languages fully integrated. + +**Tests:** +1. All 14 languages parse and generate without error +2. Cross-language matrix: C ↔ WAT ↔ Common Lisp ↔ Scheme all project correctly +3. Lisp → WAT: S-expression source → stack-based target +4. C → Common Lisp: manual memory → GC with annotation translation +5. Semanno format works for all 4 new languages with correct comment prefixes +6. Annotation inference produces language-appropriate defaults for each new language +7. Pipeline.run() succeeds for all 4 new languages +8. Sprint 14 totals: 14 parsers, 14 generators, all cross-language projections valid + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 14a | 361-364 | 44 | C parser, generator, memory mapping, integration | +| 14b | 365-369 | 56 | WAT parser, generator, annotations, cross-language, integration | +| 14c | 370-373 | 44 | Common Lisp parser, generator, annotations, integration | +| 14d | 374-377 | 44 | Scheme parser, generator, differentiation, integration | +| **Total** | **361-377** | **~188** | 17 steps | + +--- + +## Key Files + +**New files:** +- `editor/src/ast/CParser.h` — C parser (regex-based) +- `editor/src/ast/CGenerator.h` — C code generator +- `editor/src/ast/WatParser.h` — WebAssembly text format parser +- `editor/src/ast/WatGenerator.h` — WAT code generator +- `editor/src/ast/CommonLispParser.h` — Common Lisp parser +- `editor/src/ast/CommonLispGenerator.h` — Common Lisp code generator +- `editor/src/ast/SchemeParser.h` — Scheme parser +- `editor/src/ast/SchemeGenerator.h` — Scheme code generator +- `editor/tests/step361_test.cpp` through `editor/tests/step377_test.cpp` + +**Modified files:** +- `editor/src/ast/Parser.h` — include 4 new parsers +- `editor/src/ast/Generator.h` — include 4 new generators +- `editor/src/Pipeline.h` — routing for "c", "wat", "common-lisp", "scheme" +- `editor/src/CrossLanguageProjector.h` — adaptations for all 4 languages +- `editor/src/MemoryStrategyInference.h` — defaults for C (Manual), WAT (Manual), CL (GC), Scheme (GC) +- `editor/src/AnnotationInference.h` — language-specific inference patterns +- `editor/CMakeLists.txt` — test targets + +--- + +## Language Design Notes + +### Why regex-based parsers (no tree-sitter)? +Tree-sitter grammars exist for C and Scheme, but: +1. The architecture invariant says no new external dependencies beyond tree-sitter +2. The existing tree-sitter integration handles Python/C++/Java/Rust/Go/JS/TS +3. Adding tree-sitter grammars for 4 more languages bloats the build +4. Regex-based parsers for Kotlin/C# (Sprint 11d) proved the pattern works +5. For transpilation purposes, we need structural (not syntactic) fidelity — + regex parsers capture the structural skeleton, which is what matters for + the AST + +### Lisp parser approach +Both Lisp parsers are S-expression readers at core: +1. Tokenize: atoms, strings, numbers, parentheses, quotes +2. Build S-expression tree +3. Walk tree, pattern-match on car of each form: `defun`→Function, `define`→Function, etc. +4. Unrecognized forms become generic FunctionCall nodes + +This is simpler than regex parsing of C-like languages because S-expressions +have uniform syntax. The complexity is in semantic mapping, not parsing. + +### WAT as compilation target +WAT is unique among our supported languages: it's primarily a target, not a source. +Most workflows will be "source language → WAT" rather than "WAT → source language." +The WAT→source direction is decompilation, which we support but with lower fidelity +(stack reconstruction to expression trees is lossy for complex code). diff --git a/sprint15_plan.md b/sprint15_plan.md new file mode 100644 index 0000000..3f4ea4b --- /dev/null +++ b/sprint15_plan.md @@ -0,0 +1,700 @@ +# Sprint 15 Plan: Orchestration Engine + +## Context + +Sprint 12 built the workflow model: WorkItems, TaskQueue, RoutingEngine, Workers, +ReviewGates. Sprint 15 makes it operational. The difference: Sprint 12 defined the +*parts*; Sprint 15 makes them *run together* as a continuous pipeline. + +This is where the thesis becomes real. An architect creates a skeleton, annotates it, +and the orchestration engine takes over: routing tasks to the right worker, assembling +context, executing deterministic tasks, preparing bundles for LLM/SLM/human workers, +enforcing review gates, handling rejections, tracking progress, and surfacing the +right decisions to the right people at the right time. + +**What Sprint 12 delivered:** +- WorkItem lifecycle (pending → ready → assigned → in-progress → review → complete) +- TaskQueue with priority + dependency resolution +- RoutingEngine with annotation-based dispatch rules +- Workers: Deterministic, Template, Agent (SLM/LLM), Human +- ContextAssembler with budget enforcement +- ReviewGate with auto-approve policies +- WorkflowPersistence sidecar + +**What Sprint 15 adds:** +- **Orchestrator** — the main loop that advances the workflow automatically +- **Execution pipeline** — workers actually produce results that feed back in +- **Parallel dispatch** — independent tasks execute concurrently +- **Feedback loops** — rejection re-routes with accumulated context +- **MCP orchestration protocol** — Claude Code (or any MCP client) can drive + the workflow step-by-step or let it auto-advance +- **Progress streaming** — real-time workflow status updates via MCP +- **Dependency graph visualization data** — for Sprint 19's GUI + +--- + +## Phase 15a: Orchestrator Core (Steps 378-383) + +*The main loop that drives workflows forward.* + +### Step 378: Orchestrator.h — The Main Loop (12 tests) +**Goal:** Single class that owns the workflow lifecycle: advance ready tasks +through routing → context assembly → execution → review → completion. + +**New file: `editor/src/Orchestrator.h`** (~400 lines) + +**Orchestrator class:** +- `Orchestrator(workflowState, routingEngine, workerRegistry, contextAssembler, reviewGate)` +- `step() -> OrchestratorEvent` — advance one task one stage +- `advance() -> vector` — advance all advanceable tasks one stage each +- `runToCompletion() -> WorkflowStats` — loop advance() until nothing more can be done + without external input (all remaining tasks are human-routed or awaiting review) +- `runUntil(predicate) -> WorkflowStats` — advance until predicate says stop +- `getBlockers() -> vector` — what's preventing further progress + (human tasks, review queue, unresolved dependencies) + +**OrchestratorEvent struct:** +``` +OrchestratorEvent: + type: string // "routed" | "context-assembled" | "executed" | "auto-approved" + // | "sent-to-review" | "completed" | "rejected" | "blocked" + itemId: string + detail: json // event-specific data (routing decision, result, etc.) + timestamp: string +``` + +**BlockerInfo struct:** +``` +BlockerInfo: + type: string // "needs-human" | "needs-review" | "needs-external-model" + itemIds: vector // which items are blocked + description: string // human-readable explanation +``` + +**Orchestrator step() logic:** +1. Get ready tasks from queue (dependencies satisfied, not assigned) +2. For each ready task: + a. Route it (RoutingEngine → RoutingDecision) + b. Assemble context (ContextAssembler at decided width) + c. If deterministic/template → execute immediately, get result + d. If agent (SLM/LLM) → prepare context bundle, mark as "needs-external-model" + e. If human → mark as "needs-human" +3. For deterministic results: + a. Run through ReviewGate + b. If auto-approved → complete, trigger dependency cascade + c. If needs review → mark as "in-review" +4. Return all events from this step + +**Tests:** single deterministic task completes in one step(), dependency cascade +on completion, human task blocks at "needs-human", agent task blocks at +"needs-external-model", runToCompletion processes all deterministic tasks, events +emitted for each stage, getBlockers reports human/review items, empty workflow +returns immediately, mixed workflow (some deterministic, some human) processes +what it can. + +### Step 379: Parallel Dispatch + Batching (12 tests) +**Goal:** Independent tasks (no dependency between them) should be processed +in parallel within a single advance() call. + +**Modifies: `editor/src/Orchestrator.h`** +- `advance()` processes all ready tasks, not just the first +- Independent tasks in the same priority level are batched +- Deterministic tasks executed sequentially (single-threaded) but + logically parallel (all advance in one step) +- Context assembly for multiple tasks shares project-level context + (assemble once, reuse for all project-width items) + +**BatchResult struct:** +``` +BatchResult: + events: vector + itemsAdvanced: int + itemsBlocked: int + contextTokensSaved: int // tokens saved by sharing project context +``` + +- `advanceBatch() -> BatchResult` + +**Optimization: shared context** +- If 3 tasks all need project-width context, assemble project summary once +- Each task gets: shared project summary + task-specific local context +- Track tokens saved by this optimization + +**Tests:** 3 independent tasks all advance in one call, shared context assembled +once (not 3 times), batch result counts correct, dependent tasks wait (only +independent ones advance), priority ordering within batch, context tokens saved +reported, empty batch, single-item batch, mixed independent + dependent. + +### Step 380: Feedback Loop — Rejection Re-Routing (12 tests) +**Goal:** When a human rejects a work item, the rejection feedback becomes +part of the context for the next attempt. The routing engine may choose a +different worker type based on the feedback. + +**Modifies: `editor/src/Orchestrator.h`** and `editor/src/RoutingEngine.h` + +**Rejection flow:** +1. Human reviews result, provides rejection feedback (text + optional annotation changes) +2. Orchestrator marks item as rejected with feedback +3. Item re-enters ready queue with accumulated context: + - Original skeleton intent + - Previous attempt's result + - Rejection feedback + - Any annotation updates from the reviewer +4. Routing engine considers rejection history: + - If deterministic worker was rejected → escalate to SLM or LLM + - If SLM was rejected → escalate to LLM + - If LLM was rejected → escalate to human (with full context of attempts) + - If human was rejected → remains human with updated annotations + +**RejectionHistory in WorkItem:** +``` +RejectionHistory: + attempts: vector + +Attempt: + workerType: string + result: WorkItemResult + feedback: string + rejectedAt: string + rejectedBy: string +``` + +**Tests:** rejected item re-enters queue, feedback included in next context, +routing escalation (deterministic → slm → llm → human), rejection history +preserved, multiple rejections accumulate, annotation updates from reviewer +applied, re-routed item has wider context than original, escalation doesn't +skip levels, human rejection stays human, rejection count tracked. + +### Step 381: Progress Tracking + ETA (12 tests) +**Goal:** Track workflow progress with completion percentage, throughput +metrics, and rough time-to-completion estimates. + +**New file: `editor/src/WorkflowProgress.h`** (~200 lines) + +**ProgressSnapshot struct:** +``` +ProgressSnapshot: + totalItems: int + completedItems: int + completionPercent: float + itemsPerMinute: float // throughput (completed items / elapsed time) + estimatedRemainingMinutes: float // remaining / throughput + byWorkerType: map + currentPhase: string + blockers: vector + startedAt: string + lastActivityAt: string +``` + +**WorkerStats struct:** +``` +WorkerStats: + completed: int + avgConfidence: float + avgTokensUsed: int + rejectionRate: float +``` + +**WorkflowProgress class:** +- `recordEvent(OrchestratorEvent)` — update metrics on each event +- `getSnapshot() -> ProgressSnapshot` — current state +- `getTimeline() -> vector` — event log for visualization +- `getWorkerStats() -> map` + +**Tests:** completion percent accurate, throughput calculated from events, +ETA reasonable (within 2x of actual for uniform tasks), worker stats +per type, rejection rate tracked, timeline records all events, empty +workflow returns 0% with no ETA, blockers reported in snapshot, snapshot +JSON serialization, progress monotonically increases (except rejections). + +### Step 382: Orchestrator RPC + MCP (12 tests) +**Goal:** MCP tools for driving the orchestrator. An agent should be able +to step through a workflow, auto-advance deterministic tasks, or query +what's blocked and needs attention. + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** — 6 new RPC methods: +- `orchestrateStep` — advance one task one stage, return event +- `orchestrateAdvance` — advance all ready tasks, return batch result +- `orchestrateRunDeterministic` — run all deterministic/template tasks to + completion, stop at agent/human items +- `getBlockers` — what needs external input +- `getProgress` — current progress snapshot +- `submitExternalResult` — an agent submitting LLM/SLM output for a task + that was prepared by the orchestrator + +**Modifies: `editor/src/MCPServer.h`** — registerOrchestratorTools() +- `whetstone_orchestrate_step` +- `whetstone_orchestrate_advance` +- `whetstone_orchestrate_run_deterministic` +- `whetstone_get_blockers` +- `whetstone_get_progress` +- `whetstone_submit_result` + +**Tool count:** 62+ (56 from Sprint 12 + 6 orchestrator tools) + +**Key design: submitExternalResult** +This is how the Claude Code plugin (Sprint 18) will close the loop: +1. Orchestrator prepares context bundle for an LLM task +2. Claude Code receives the bundle via MCP +3. Claude Code sends the context to Claude, gets a response +4. Claude Code calls `submitExternalResult` with the generated code +5. Orchestrator runs it through review gate and continues + +**Tests:** step advances one task, advance processes batch, runDeterministic +completes all template tasks and stops, getBlockers reports human tasks, +getProgress returns valid snapshot, submitExternalResult accepts LLM output +and advances the task, Linter role can read progress but not orchestrate, +MCP tool registration, combined: run deterministic → get blockers → submit +result → progress updates. + +### Step 383: Phase 15a Integration Tests (8 tests) +**Goal:** End-to-end orchestration from skeleton to partial completion. + +**Tests:** +1. Create 5-function skeleton (2 simple, 2 medium, 1 complex) → workflow → + orchestrate → 2 deterministic complete, 2 agent-prepared, 1 human-blocked +2. Dependency chain respected: orchestrator doesn't advance B until A completes +3. Rejection escalation: deterministic result rejected → re-routed to LLM +4. Shared context optimization: 3 project-width tasks share context assembly +5. Progress tracking: completion % and throughput match actual events +6. submitExternalResult for agent task → review gate → auto-approve → complete +7. getBlockers shows exactly the human/review tasks +8. Workflow persistence: save mid-orchestration → reload → continue from same state + +--- + +## Phase 15b: MCP Orchestration Protocol (Steps 384-388) + +*Define the protocol that an MCP client (like Claude Code) uses to drive +a full project workflow. This is the contract Sprint 18 implements against.* + +### Step 384: Workflow Session Protocol (12 tests) +**Goal:** Define the stateful MCP session protocol for project workflows. + +**New file: `editor/src/WorkflowProtocol.h`** (~300 lines) + +**Protocol phases (from the MCP client's perspective):** +1. **Init:** `whetstone_create_skeleton` → skeleton module created +2. **Model:** Series of `whetstone_add_skeleton_node` calls → project modeled +3. **Annotate:** `whetstone_infer_annotations` + manual annotation adjustments +4. **Plan:** `whetstone_create_workflow` → work items created and queued +5. **Route:** `whetstone_orchestrate_advance` → routing decisions made +6. **Execute:** `whetstone_orchestrate_run_deterministic` → auto tasks complete +7. **Assist:** Loop: `whetstone_get_blockers` → prepare context → + call external model → `whetstone_submit_result` +8. **Review:** Human reviews items in review queue +9. **Complete:** All items complete, `whetstone_save_workflow` + +**WorkflowSession struct:** +``` +WorkflowSession: + sessionId: string + projectName: string + currentPhase: string // which protocol phase + startedAt: string + commands: vector // history of MCP commands issued +``` + +**Protocol helpers:** +- `getNextAction(session) -> ProtocolAction` — suggest what the MCP client + should do next based on current workflow state +- `validateTransition(from, to) -> bool` — is this phase transition valid +- `getSessionSummary(session) -> json` — current state for client display + +**Tests:** protocol phases in order, getNextAction suggests correctly at each +phase, invalid transition rejected, session tracks command history, summary +accurate at each phase, phase auto-detection from workflow state, session +persistence, protocol works with empty project. + +### Step 385: Context Bundle Format (12 tests) +**Goal:** Standardize the context bundle that agent workers prepare for +external model invocation. This is what the MCP client receives and +forwards to an LLM. + +**New file: `editor/src/ContextBundle.h`** (~200 lines) + +**ContextBundle struct:** +``` +ContextBundle: + taskDescription: string // human-readable task description + skeletonCode: string // the skeleton function/class to implement + intent: string // @Intent annotation value + constraints: vector // from @Contract, @Risk, @Policy annotations + contextCode: string // surrounding code (scope-appropriate) + projectSummary: string // compact summary of project structure + existingTests: string // any test expectations + previousAttempts: vector // from rejection history + tokenBudget: int // how many tokens the response should target + outputFormat: string // "code-only" | "code-with-explanation" +``` + +**AttemptSummary struct:** +``` +AttemptSummary: + workerType: string + generatedCode: string + feedback: string +``` + +**ContextBundle builders:** +- `buildBundle(workItem, workerContext, routingDecision) -> ContextBundle` +- `bundleToPrompt(bundle) -> string` — render as a prompt string that any + LLM can understand (not Claude-specific) +- `bundleToJson(bundle) -> json` — structured format for MCP transport +- `estimateBundleTokens(bundle) -> int` + +**Tests:** bundle contains all fields, bundleToPrompt readable and structured, +rejection history included when present, token estimate reasonable, budget +respected, constraints from annotations included, intent from @Intent, +empty fields handled gracefully, bundleToJson roundtrip, prompt format +model-agnostic (no model-specific tokens). + +### Step 386: Result Acceptance Protocol (12 tests) +**Goal:** Standardize how external results are validated, accepted, and +integrated back into the workflow. + +**Modifies: `editor/src/Orchestrator.h`** + +**ResultSubmission struct:** +``` +ResultSubmission: + itemId: string + generatedCode: string + confidence: float // model's self-assessed confidence (optional) + reasoning: string // model's explanation (optional) + suggestedAnnotations: vector // model may suggest annotations +``` + +**Acceptance pipeline:** +1. Parse generated code through Pipeline.parse() → verify it's valid syntax +2. Run AnnotationValidator on the result → check for violations +3. Run diagnostic pipeline → check for errors +4. Apply ReviewGate policy → auto-approve or send to review +5. If accepted: update WorkItem result, complete the task, cascade dependencies +6. If validation fails: reject with structured feedback, re-route + +**ResultAcceptance struct:** +``` +ResultAcceptance: + accepted: bool + validationPassed: bool + diagnosticCount: int + autoApproved: bool + reviewRequired: bool + validationErrors: vector +``` + +**Tests:** valid code accepted, syntax error rejected with feedback, annotation +violation flagged, diagnostic errors prevent auto-approve, confidence below +threshold sends to review, suggested annotations attached to node, acceptance +pipeline runs all checks in order, partial acceptance (code valid but review +needed), multiple submissions for same item (latest wins), cascade on acceptance. + +### Step 387: Orchestrator Event Stream (12 tests) +**Goal:** Real-time event stream that MCP clients can poll for workflow +progress. Foundation for Sprint 19's GUI visualization. + +**New file: `editor/src/EventStream.h`** (~200 lines) + +**EventStream class:** +- `emit(OrchestratorEvent)` — add event to stream +- `poll(sinceVersion) -> vector` — get events since version +- `subscribe(callback)` — register listener (for GUI integration later) +- `getVersion() -> int` — current stream position +- `getRecent(count) -> vector` — last N events + +**Event types emitted:** +- `workflow.created` — new workflow initialized +- `task.routed` — routing decision made (includes decision details) +- `task.context-assembled` — context prepared (includes token count) +- `task.executed` — worker produced result +- `task.auto-approved` — review gate approved +- `task.sent-to-review` — awaiting human review +- `task.completed` — fully done +- `task.rejected` — human rejected +- `task.escalated` — re-routed after rejection +- `workflow.progress` — periodic progress snapshot +- `workflow.blocked` — nothing more can advance +- `workflow.complete` — all items done + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** +- `getEventStream` — poll events since version +- `getRecentEvents` — get last N events + +**Modifies: `editor/src/MCPServer.h`** +- `whetstone_get_event_stream` + `whetstone_get_recent_events` + +**Tool count:** 64+ (62 + 2 event stream tools) + +**Tests:** events emitted during orchestration, poll returns only new events, +version tracking correct, subscribe callback fires, event types cover all +workflow transitions, getRecent returns correct count, empty stream returns +nothing, high-frequency polling doesn't duplicate, event JSON serialization, +MCP tool registration. + +### Step 388: Phase 15b Integration — Full Protocol Test (8 tests) +**Goal:** Simulate a complete MCP client session through the protocol. + +**Tests:** +1. Full protocol walkthrough: init → model → annotate → plan → route → execute → + submit external → review → complete +2. Event stream captures every transition in order +3. Context bundle for LLM task contains skeleton + intent + project summary +4. Result acceptance: valid code auto-approved for deterministic, review-required for LLM +5. Rejection → re-route → submit better result → accepted +6. Progress snapshot at each phase matches actual state +7. getNextAction correctly guides client through protocol +8. Session persistence: save mid-protocol → reload → resume + +--- + +## Phase 15c: Advanced Routing + Optimization (Steps 389-393) + +*Make the routing engine smarter and the orchestration more efficient.* + +### Step 389: Routing Rules Engine (12 tests) +**Goal:** Make routing rules configurable and composable rather than hardcoded. + +**New file: `editor/src/RoutingRules.h`** (~300 lines) + +**RoutingRule struct:** +``` +RoutingRule: + name: string + priority: int // lower = higher priority + conditions: vector // all must match (AND) + action: RoutingAction +``` + +**RuleCondition types:** +- `annotation(type, property, value)` — node has annotation with property match +- `complexity(op, threshold)` — cognitive complexity comparison +- `contextWidth(width)` — context width annotation match +- `nodeType(type)` — function vs class vs method +- `rejectionCount(op, threshold)` — number of previous rejections +- `language(lang)` — source language match +- `pattern(name)` — recognized pattern (getter, setter, constructor, etc.) + +**RoutingAction:** +- `workerType: string` +- `reviewRequired: bool` +- `contextWidthOverride: string` (optional) +- `budgetMultiplier: float` (optional, for giving complex tasks more tokens) + +**RulesEngine class:** +- `addRule(rule)` — add to rule set +- `evaluate(workItem) -> RoutingDecision` — first matching rule wins +- `getDefaultRules() -> vector` — sensible defaults +- `loadRules(json) / saveRules() -> json` — persistence + +**Tests:** first matching rule wins, conditions AND together, annotation +condition matches, complexity threshold, pattern matching, default rules +sensible, custom rules override defaults, no match falls through to default, +rule priority ordering, rules persist to JSON, budgetMultiplier applied, +rejection count escalation. + +### Step 390: Cost Estimation + Optimization (12 tests) +**Goal:** Before executing the workflow, estimate the total cost in tokens +and suggest optimizations. + +**New file: `editor/src/CostEstimator.h`** (~200 lines) + +**CostEstimate struct:** +``` +CostEstimate: + totalContextTokens: int // total context tokens across all tasks + totalOutputTokens: int // estimated output tokens + byWorkerType: map // tokens per worker type + deterministicTasks: int // tasks with zero token cost + estimatedCost: float // rough $ cost at current model pricing + optimizationSuggestions: vector +``` + +**CostEstimator class:** +- `estimate(workflowState, routingEngine) -> CostEstimate` — run routing + on all items, sum up context budgets +- `suggest(estimate) -> vector` — recommendations: + - "5 getter functions could use template worker (saving ~10k tokens)" + - "3 file-width tasks could be narrowed to local (saving ~4.5k tokens)" + - "Consider batching these 4 related tasks for shared context" + +**OptimizationSuggestion struct:** +``` +OptimizationSuggestion: + description: string + tokensSaved: int + itemIds: vector + action: string // "narrow-context" | "use-template" | "batch" | "skip-review" +``` + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** — `estimateCost` RPC +**Modifies: `editor/src/MCPServer.h`** — `whetstone_estimate_cost` + +**Tool count:** 65+ (64 + 1 cost tool) + +**Tests:** estimate returns reasonable token counts, deterministic tasks have +zero token cost, optimization suggestions generated, getter detection suggests +template, context narrowing suggested where possible, batch suggestion for +related tasks, empty workflow returns zero cost, large workflow (50 items) +completes in reasonable time, cost estimate JSON serialization, MCP tool +registration. + +### Step 391: Workflow Templates (12 tests) +**Goal:** Pre-built workflow templates for common project patterns. + +**New file: `editor/src/WorkflowTemplates.h`** (~250 lines) + +**Templates:** +- **CRUD API:** generates skeleton for create/read/update/delete operations + for a given entity. Deterministic routing for getters/setters, LLM for + business logic, human review for security-sensitive operations. +- **Module Refactor:** given existing code, creates work items for extracting + functions, renaming, restructuring. Mostly LLM with human review. +- **Cross-Language Port:** given source in language A, creates skeleton in + language B with annotation-guided work items for each function/class. + Template workers for simple functions, LLM for complex logic. +- **Test Suite Generation:** given production code, creates skeleton test + functions with @Intent annotations describing what to test. SLM for + simple unit tests, LLM for integration tests. +- **Legacy Modernization:** given old code, creates work items for updating + idioms, replacing deprecated patterns, adding safety annotations. + +**WorkflowTemplate class:** +- `getTemplates() -> vector` +- `applyTemplate(templateName, params) -> WorkflowState` — creates populated + workflow from template +- Templates are composable (e.g., cross-language port + test generation) + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** — `listTemplates`, `applyTemplate` +**Modifies: `editor/src/MCPServer.h`** — `whetstone_list_templates`, `whetstone_apply_template` + +**Tool count:** 67+ (65 + 2 template tools) + +**Tests:** 5 templates available, CRUD template creates correct skeleton, +cross-language template populates work items with annotation-guided routing, +template params validated, test generation template creates @Intent annotations, +applyTemplate produces valid WorkflowState, composing templates works, +MCP tool registration, template list includes descriptions, empty params +uses defaults. + +### Step 392: Dependency Graph Data (12 tests) +**Goal:** Export workflow dependency graph as structured data that Sprint 19's +GUI can visualize. + +**New file: `editor/src/DependencyGraph.h`** (~200 lines) + +**GraphData struct:** +``` +GraphData: + nodes: vector + edges: vector +``` + +**GraphNode:** +``` + id: string + label: string + type: string // "function" | "class" | "method" + status: string // WorkItem status + workerType: string // routing decision + priority: string +``` + +**GraphEdge:** +``` + from: string + to: string + type: string // "depends-on" | "blocks" +``` + +- `buildDependencyGraph(workflowState) -> GraphData` +- `graphToJson(graph) -> json` — for MCP transport +- `getCriticalPath(graph) -> vector` — longest dependency chain + (determines minimum completion time) + +**Modifies: `editor/src/HeadlessAgentRPCHandler.h`** — `getDependencyGraph` (workflow context) +**Modifies: `editor/src/MCPServer.h`** — `whetstone_get_dependency_graph` + +**Tool count:** 68+ (67 + 1 graph tool) + +**Tests:** graph has correct node count, edges match dependencies, critical +path is longest chain, node status matches workflow state, worker type +populated from routing, JSON serialization, empty workflow produces empty +graph, single-node graph, complex graph (diamond dependencies), graph +updates after task completion, MCP tool registration. + +### Step 393: Phase 15c Integration + Sprint 15 Summary (8 tests) +**Goal:** Full orchestration engine validation. + +**Tests:** +1. Custom routing rules: override default to use template for all simple functions → + verify template worker used +2. Cost estimation: 10-function workflow → estimate → suggestions include + "use template for getters" +3. CRUD template: create → route → execute deterministic → submit LLM results → + complete workflow +4. Dependency graph: 5-function chain → critical path = 5 → verify +5. Event stream: run full workflow → event stream captures all transitions +6. Protocol: full MCP client simulation through all 9 phases +7. Optimization: apply suggestion to narrow context → re-estimate → lower cost +8. Sprint 15 totals: 68+ MCP tools, orchestrator operational, protocol defined + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 15a | 378-383 | 68 | Orchestrator loop, parallel dispatch, feedback, progress, RPC | +| 15b | 384-388 | 56 | MCP protocol, context bundles, result acceptance, event stream | +| 15c | 389-393 | 56 | Rules engine, cost estimation, templates, dependency graph | +| **Total** | **378-393** | **~180** | 16 steps | + +**MCP tool count projection:** 68+ tools + +--- + +## Key Files + +**New files:** +- `editor/src/Orchestrator.h` — main orchestration loop, step/advance/runToCompletion +- `editor/src/WorkflowProgress.h` — progress tracking, ETA, worker stats +- `editor/src/WorkflowProtocol.h` — MCP session protocol, phase management +- `editor/src/ContextBundle.h` — standardized context for external model invocation +- `editor/src/EventStream.h` — real-time event stream for workflow transitions +- `editor/src/RoutingRules.h` — configurable routing rules engine +- `editor/src/CostEstimator.h` — token cost estimation and optimization suggestions +- `editor/src/WorkflowTemplates.h` — pre-built workflow templates (CRUD, refactor, port, test, modernize) +- `editor/src/DependencyGraph.h` — dependency graph export for visualization + +**Modified files:** +- `editor/src/HeadlessEditorState.h` — Orchestrator member +- `editor/src/HeadlessAgentRPCHandler.h` — ~10 new RPC methods +- `editor/src/MCPServer.h` — register 12+ new MCP tools +- `editor/src/RoutingEngine.h` — delegates to RoutingRules +- `editor/src/WorkItem.h` — RejectionHistory, AttemptSummary +- `editor/CMakeLists.txt` — test targets + +--- + +## Architectural Notes + +### Why the orchestrator doesn't call LLMs directly +The orchestrator prepares context bundles but never invokes a model. This is deliberate: +1. **Model-agnostic:** Any LLM/SLM can be used. The system doesn't prefer Claude, GPT, or local models. +2. **Cost control:** The human/agent decides when to spend tokens, not the orchestrator. +3. **Auditability:** Every token spent is traceable to an explicit submitExternalResult call. +4. **Offline capability:** The orchestrator can run all deterministic tasks without any API access. + +### The Sprint 18 connection +Sprint 15 defines the *server side* of the orchestration protocol. Sprint 18 builds +the *client side* — the Claude Code plugin that: +1. Connects to the MCP server +2. Follows the workflow protocol +3. Receives context bundles +4. Sends them to Claude (or another model) +5. Submits results back +6. Presents human review items to the engineer + +Sprint 15 is complete when an MCP client *could* drive a full workflow. Sprint 18 +builds the client that *does*. diff --git a/sprint16_plan.md b/sprint16_plan.md new file mode 100644 index 0000000..e31bef8 --- /dev/null +++ b/sprint16_plan.md @@ -0,0 +1,90 @@ +# Sprint 16 Plan: C++ Depth + Self-Hosting Phase 1 + +## Context + +Sprints 12d gave us preprocessor, enum, namespace, and multiple inheritance. +Sprint 16 adds the remaining constructs needed to parse Whetstone's own simpler +headers: type aliases, static members, const/constexpr, smart pointers, auto +type deduction, and cast expressions (feature-requests items 6-11). + +The self-hosting milestone: parse `AnnotationConflictExtended.h` (simple structs +and free functions) through our own pipeline, get a valid AST, annotate it, and +generate equivalent code. + +--- + +## Phase 16a: C++ Type System Depth (Steps 394-399) + +### Step 394: TypeAlias + Nested Type Access (12 tests) +- Parse `using json = nlohmann::json;` and `typedef vector StringVec;` +- Represent nested type access: `Foo::Bar::Baz` as qualified name +- Already have TypeAlias node from Sprint 12d — deepen parser recognition + +### Step 395: Static Members + Const/Constexpr (12 tests) +- MethodDeclaration gains `isConst` (method doesn't modify `this`) +- Variable gains `isConstexpr`, `isStatic` flags +- Parse `static constexpr int MAX = 256;` and `const string& getName() const;` +- Generator outputs const and constexpr qualifiers + +### Step 396: Smart Pointer Patterns (12 tests) +- Recognize `unique_ptr`, `shared_ptr`, `make_unique()` as patterns +- Map to ownership annotations: unique_ptr → @Owner(Unique), shared_ptr → @Owner(Shared_ARC) +- Parse `new`/`delete` expressions as FunctionCall nodes with @Reclaim annotations +- Cross-language: unique_ptr → Rust Box, shared_ptr → Rust Arc + +### Step 397: Auto Type Deduction (12 tests) +- New node or annotation: `auto` as deferred type +- Parse `auto x = compute();`, `auto* p = &x;`, `const auto& ref = vec;` +- Type inference hint via @TypeState annotation when deducible from context +- Generator outputs `auto` for C++, inferred types for other languages + +### Step 398: Cast Expressions (12 tests) +- CastExpression node: kind (static/dynamic/reinterpret/const), targetType, operand +- Parse `static_cast(bar)` and `dynamic_cast(base)` +- Critical for Whetstone's own downcast-heavy validation pattern +- Annotation: @Risk(medium) on reinterpret_cast, @Risk(low) on static_cast + +### Step 399: Phase 16a Integration (8 tests) +- Parse a Whetstone-style header fragment with all new constructs +- Roundtrip: parse → generate → parse → equivalent AST +- Cross-language projection for smart pointers and casts + +--- + +## Phase 16b: Self-Hosting Phase 1 (Steps 400-404) + +### Step 400: Self-Hosting Test Harness (12 tests) +- Infrastructure to feed actual Whetstone .h files through the pipeline +- Compare parsed AST against expected structure +- Track coverage: which constructs parsed vs skipped +- Graceful degradation: unparseable sections become opaque blocks, not errors + +### Step 401: Parse AnnotationConflictExtended.h (12 tests) +- The simplest Whetstone header: structs + free functions +- Verify: all struct definitions found, all function signatures captured +- Expected: some template-heavy code still opaque, but basic structure correct + +### Step 402: Parse AnnotationValidatorExtended.h (12 tests) +- Single class with static methods and static_cast patterns +- Verify: class structure, method signatures, static_cast recognized +- Annotation inference: @Complexity scores for validation methods + +### Step 403: Annotate + Generate from Self-Hosted AST (12 tests) +- Take parsed AST from step 401-402, run inference +- Generate C++ output, compare structural similarity +- Semanno annotations on Whetstone's own code + +### Step 404: Self-Hosting Progress Report + Phase Integration (8 tests) +- Coverage metrics: % of constructs in target files successfully parsed +- Gap analysis: what still can't parse (for Sprint 22) +- Sprint 16 totals verification + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 16a | 394-399 | 68 | Type aliases, const/constexpr, smart pointers, auto, casts | +| 16b | 400-404 | 56 | Self-hosting harness, parse own headers, annotate, generate | +| **Total** | **394-404** | **~124** | 11 steps | diff --git a/sprint17_plan.md b/sprint17_plan.md new file mode 100644 index 0000000..58e51ed --- /dev/null +++ b/sprint17_plan.md @@ -0,0 +1,113 @@ +# Sprint 17 Plan: Language Batch 2 — .NET Family + SQL + +## Context + +Full-stack transpilation needs server-side languages and database layers. +Sprint 17 adds F#, VB.NET, and SQL dialects. This covers the Microsoft +ecosystem and the data layer that every real project touches. + +**After Sprint 17:** 17 language parsers and generators. + +--- + +## Phase 17a: F# Parser + Generator (Steps 405-409) + +F# is a functional-first .NET language. ML-family syntax with type inference, +pattern matching, discriminated unions, and computation expressions. Tests +whether the AST handles ML-style functional programming (complements the +Lisp-style coverage from Sprint 14). + +### Step 405: F# Parser (12 tests) +- `let name params = body` → Function +- `let mutable x = 5` → Variable (isMutable) +- `type Name = | Case1 | Case2 of int` → EnumDeclaration (discriminated union) +- `type Name = { field: Type }` → ClassDeclaration (record) +- `match x with | pattern -> expr` → pattern match → IfStatement chain +- `async { ... }` → AsyncFunction with computation expression +- `module Name` → NamespaceDeclaration +- `|> pipeline |> operators` → chained FunctionCalls +- Significant whitespace handling (indentation-sensitive) + +### Step 406: F# Generator (12 tests) +- Extends ProjectionGenerator + SemannoAnnotationImpl +- F# conventions: `let`, pipe operators, pattern matching +- Type mapping: int, float, string, bool, unit, list, seq, option, result +- `commentPrefix() -> "// "` + +### Step 407: VB.NET Parser (12 tests) +- `Sub Name(params) ... End Sub` → Function (void return) +- `Function Name(params) As Type ... End Function` → Function +- `Class Name ... End Class` → ClassDeclaration +- `Interface Name ... End Interface` → InterfaceDeclaration +- `Module Name ... End Module` → NamespaceDeclaration +- `Dim x As Integer = 5` → Variable +- `If...Then...ElseIf...Else...End If` → IfStatement +- `For Each x In collection ... Next` → ForStatement + +### Step 408: VB.NET Generator (12 tests) +- VB syntax: `Sub`/`Function`, `Dim`, `End` blocks +- Type mapping: Integer, Double, String, Boolean, Object +- Case-insensitive output conventions +- `commentPrefix() -> "' "` + +### Step 409: .NET Integration (8 tests) +- Cross-language: C# ↔ F# ↔ VB.NET (all .NET, high fidelity) +- Cross-language: Python → F# (OOP → functional) +- Pipeline.run() for both new languages +- All 3 .NET languages share type system annotations + +--- + +## Phase 17b: SQL Dialect Support (Steps 410-416) + +SQL is fundamentally different from all other languages: it's declarative, +set-based, and schema-oriented. The AST needs table/column/query representations. +Supporting SQL means Whetstone can reason about the full stack: frontend → +backend → database. + +### Step 410: SQL AST Nodes (12 tests) +- New nodes: TableDeclaration, ColumnDefinition, SelectQuery, InsertStatement, + UpdateStatement, DeleteStatement, JoinClause, WhereClause, IndexDefinition +- These are new node types specific to SQL — not mappable to existing Function/Class + +### Step 411: PostgreSQL Parser (12 tests) +- CREATE TABLE, SELECT, INSERT, UPDATE, DELETE, JOIN, WHERE +- PostgreSQL-specific: SERIAL, JSONB, array types, DO blocks, PL/pgSQL functions +- Schema.Table notation + +### Step 412: PostgreSQL Generator (12 tests) +- Generate valid PostgreSQL DDL and DML +- Type mapping to PostgreSQL types +- `commentPrefix() -> "-- "` + +### Step 413: T-SQL Parser + Generator (12 tests) +- Microsoft SQL Server dialect: IDENTITY, TOP, NVARCHAR, stored procedures +- T-SQL-specific: BEGIN/END blocks, DECLARE, SET, PRINT +- Differences from PostgreSQL highlighted in annotations + +### Step 414: MySQL Parser + Generator (12 tests) +- MySQL dialect: AUTO_INCREMENT, ENGINE=InnoDB, backtick quoting +- MySQL-specific: LIMIT syntax, GROUP_CONCAT + +### Step 415: SQL Annotation Mapping (12 tests) +- @Risk(high) on DELETE without WHERE, DROP TABLE +- @Complexity from JOIN depth and subquery nesting +- @BoundsCheck from LIMIT/TOP presence +- @Contract(pre="index exists") for queries depending on indexes +- Cross-stack: Python ORM class ↔ SQL table definition + +### Step 416: Phase 17b Integration + Sprint Summary (8 tests) +- 17 languages total (14 + F# + VB.NET + SQL counts as 1 with 3 dialects) +- Cross-stack: Python class → C# class → SQL table → migrations +- Annotation inference on SQL (risk detection, complexity) +- Sprint 17 totals verification + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 17a | 405-409 | 56 | F# + VB.NET parsers, generators, .NET integration | +| 17b | 410-416 | 80 | SQL AST nodes, PostgreSQL, T-SQL, MySQL, annotations | +| **Total** | **405-416** | **~136** | 12 steps | diff --git a/sprint18_plan.md b/sprint18_plan.md new file mode 100644 index 0000000..07fd12d --- /dev/null +++ b/sprint18_plan.md @@ -0,0 +1,138 @@ +# Sprint 18 Plan: Claude Code Plugin + MCP Workflow Tools + +## Context + +This is the payoff sprint. Sprints 12 and 15 built the workflow model and +orchestration engine. Sprint 18 builds the **client side** — the thing an +engineer actually installs and uses. + +The deliverable: a Claude Code MCP tool configuration that an engineer adds +to their project, and suddenly Claude Code can orchestrate annotation-driven +workflows. The engineer says "modernize this module" and the system: +1. Creates a skeleton with inferred annotations +2. Routes tasks (deterministic for simple stuff, Claude for complex stuff) +3. Executes what it can automatically +4. Surfaces the hard decisions to the human with full context +5. Tracks progress and handles review + +**This sprint doesn't build a custom plugin binary.** It builds the MCP server +tools and the prompt templates that make the existing `whetstone_mcp` binary +work as an effective Claude Code tool server. The "plugin" is a configuration +file pointing Claude Code at the whetstone_mcp server. + +--- + +## Phase 18a: Claude Code Integration (Steps 417-422) + +### Step 417: MCP Server Configuration + Discovery (12 tests) +- Generate `.mcp.json` configuration file for Claude Code +- Server manifest: all 68+ tools organized by category with descriptions +- Tool categories: AST, Diagnostics, Workflow, Routing, Project, File ops +- Auto-detection: `whetstone_mcp` binary path resolution +- Workspace detection: auto-find project root and language + +### Step 418: Workflow Prompt Templates (12 tests) +- MCP prompt templates that guide Claude through the workflow protocol: + - `architect_project` — "Analyze this codebase and create an annotated skeleton" + - `modernize_module` — "Modernize this legacy code with safety annotations" + - `port_to_language` — "Port this module from {source} to {target}" + - `add_feature` — "Add {feature} to this project using the workflow system" + - `review_pending` — "Review and approve/reject pending work items" +- Each prompt includes the workflow protocol steps the agent should follow +- Prompts reference specific MCP tools by name + +### Step 419: Agent Workflow Loop (12 tests) +- The core interaction loop for an agent using the MCP tools: + 1. `whetstone_infer_annotations` on existing code + 2. `whetstone_create_skeleton` for new functionality + 3. `whetstone_create_workflow` to plan execution + 4. `whetstone_orchestrate_run_deterministic` for auto tasks + 5. `whetstone_get_blockers` to find what needs the agent + 6. For each agent task: read context bundle, generate code, `whetstone_submit_result` + 7. `whetstone_get_progress` to track completion +- Test that this loop works end-to-end through MCP + +### Step 420: Human Review Interface via MCP (12 tests) +- Tools for presenting review decisions to the engineer: + - `whetstone_get_review_queue` — items awaiting human review + - `whetstone_get_review_context` — full context for a review item + (original skeleton, generated code, annotations, confidence, reasoning) + - `whetstone_approve_item` — approve with optional feedback + - `whetstone_reject_item` — reject with required feedback +- Review context formatted for human readability (not just raw JSON) + +### Step 421: Workspace Onboarding Flow (12 tests) +- First-time setup experience for a new workspace: + 1. Detect project languages from file extensions + 2. Index workspace files + 3. Run annotation inference on key files + 4. Suggest initial annotations and workflow + 5. Generate `.whetstone/` directory with sidecar files +- `whetstone_onboard_workspace` tool that does this in one call + +### Step 422: Phase 18a Integration (8 tests) +- Full end-to-end: configure MCP → onboard workspace → create workflow → + orchestrate → agent completes tasks → human reviews → complete +- All MCP tools documented with clear descriptions +- 75+ MCP tools total + +--- + +## Phase 18b: Multi-Model Dispatch (Steps 423-427) + +The orchestration engine routes tasks to "slm", "llm", or "human" — but +which specific model? Phase 18b adds model selection awareness. + +### Step 423: Model Profile Registry (12 tests) +- ModelProfile struct: name, contextWindow, costPerToken, capabilities, speed +- Pre-configured profiles: claude-opus, claude-sonnet, claude-haiku, local-slm +- Routing engine maps: "llm" → claude-sonnet (default), "slm" → claude-haiku +- Configurable overrides in `.whetstone/config.json` + +### Step 424: Context Window Optimization (12 tests) +- Given a model's context window, optimize the context bundle: + - Small model (8k context) → aggressive truncation, local scope only + - Medium model (32k) → file scope with budget + - Large model (200k) → project scope, include related files +- Context assembly adapts automatically based on model profile + +### Step 425: Batch Submission for Agent Tasks (12 tests) +- When multiple agent tasks are ready, batch them for efficient submission +- Group by context overlap (tasks in same file share context) +- Batch format: shared context once + per-task instructions +- Reduces total token cost by ~30-50% for related tasks + +### Step 426: Cost Tracking + Reporting (12 tests) +- Track actual tokens used per task (from submitExternalResult metadata) +- Compare estimated vs actual costs +- Running cost total for the workflow +- Cost report: breakdown by worker type, language, task complexity + +### Step 427: Phase 18b Integration + Sprint Summary (8 tests) +- Model profiles select appropriate context window +- Batch submission reduces total tokens vs individual +- Cost tracking matches expected for a 10-task workflow +- Sprint 18 totals: 75+ tools, workflow operational end-to-end + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 18a | 417-422 | 68 | Claude Code config, prompts, agent loop, review, onboarding | +| 18b | 423-427 | 56 | Model profiles, context optimization, batching, cost tracking | +| **Total** | **417-427** | **~124** | 11 steps | + +--- + +## Key Architectural Decision + +The "plugin" is not a custom binary or VS Code extension. It is: +1. The existing `whetstone_mcp` binary (headless MCP server) +2. A `.mcp.json` configuration file that tells Claude Code where to find it +3. MCP prompt templates that guide the agent through the workflow protocol +4. MCP tools organized into a coherent workflow + +This means any MCP-compatible client (not just Claude Code) can drive Whetstone +workflows. The protocol is the product, not a proprietary integration. diff --git a/sprint19_plan.md b/sprint19_plan.md new file mode 100644 index 0000000..f7a868e --- /dev/null +++ b/sprint19_plan.md @@ -0,0 +1,102 @@ +# Sprint 19 Plan: GUI Phase 2 — Workflow Visualization + +## Context + +Sprint 13 gave us docked panels, themes, key symbols, and navigation. Sprint 19 +adds the workflow-specific visualization. The architect should see the project as +a **structured flow** — task dependencies, routing decisions, execution progress, +review queues — not just code and files. + +This is the GUI that makes the AI+Human thesis tangible. When an architect looks +at the screen, they should immediately understand: what's done, what's in progress, +what's blocked, what needs their attention, and why. + +--- + +## Phase 19a: Task Board Panel (Steps 428-432) + +### Step 428: Kanban-Style Task Board (12 tests) +- New docked panel: Task Board +- Columns: Pending | Ready | In Progress | Review | Complete +- Cards show: task name, worker type icon, priority badge, annotation tags +- Drag (or click) to manually reassign between columns +- Filter by: worker type, priority, language, file + +### Step 429: Task Detail View (12 tests) +- Click a task card → side panel with full details: + - Skeleton code (what was specified) + - Generated code (what the worker produced) + - Diff view: skeleton vs result + - Routing decision explanation + - Rejection history with feedback + - Annotations on the target node +- Approve/reject buttons for review items + +### Step 430: Dependency Graph Visualization (12 tests) +- Visual DAG rendering of task dependencies +- Uses DependencyGraph.h data from Sprint 15 +- Nodes colored by status (green=complete, blue=in-progress, grey=pending, red=blocked) +- Critical path highlighted +- Hover shows task details +- Click navigates to task in board and code editor + +### Step 431: Progress Dashboard (12 tests) +- Workflow progress ring (% complete) +- Throughput chart (items completed over time) +- Worker breakdown pie chart (deterministic vs SLM vs LLM vs human) +- Cost tracker (tokens used, estimated remaining) +- Blocker summary with action buttons + +### Step 432: Phase 19a Integration (8 tests) +- Task board syncs with live workflow state +- Dependency graph updates as tasks complete +- Progress dashboard reflects real orchestration events +- Full visual workflow: create → route → execute → review → complete, all visible + +--- + +## Phase 19b: Code Annotation Overlay (Steps 433-437) + +### Step 433: Inline Annotation Badges (12 tests) +- In the code editor, show annotation badges in the gutter (left margin) +- Each annotation type has an icon and color +- Click badge → expand to show full annotation details +- Multiple annotations on same line → stacked badges + +### Step 434: Annotation Heatmap (12 tests) +- Color-code the code editor background by annotation density/complexity +- High complexity regions: warm colors (orange/red tint) +- Well-annotated safe regions: cool colors (green/blue tint) +- Toggle on/off from View menu +- Helps architect quickly spot the hard parts of the codebase + +### Step 435: Workflow Status Overlay (12 tests) +- In multi-file view, each function/class has a status indicator: + - Skeleton (outline icon), In Progress (spinner), Complete (check), Review (eye) +- Color matches task board status +- Click status → opens task detail for that work item + +### Step 436: Review Comparison View (12 tests) +- Side-by-side or inline diff for review items: + - Left: skeleton / original code + - Right: generated code + - Inline annotations showing what changed and why +- Approve/reject buttons integrated into the diff view +- Keyboard shortcuts: Ctrl+Shift+A (approve), Ctrl+Shift+R (reject) + +### Step 437: Phase 19b Integration + Sprint Summary (8 tests) +- Annotation badges visible on all annotated code +- Heatmap correctly highlights complex regions +- Workflow overlay reflects live orchestration state +- Review comparison accessible from task board and code editor +- Sprint 19 totals: all visualization panels operational + +--- + +## Step & Test Summary + +| Phase | Steps | Tests | Theme | +|-------|-------|-------|-------| +| 19a | 428-432 | 56 | Task board, detail view, dependency graph, progress dashboard | +| 19b | 433-437 | 56 | Annotation badges, heatmap, workflow overlay, review diff | +| **Total** | **428-437** | **~112** | 10 steps |