From c9e96a5ccae34a6952cc7c6645fe18d54dde7b14 Mon Sep 17 00:00:00 2001 From: Bill Date: Wed, 18 Feb 2026 17:10:50 -0700 Subject: [PATCH] Sprint plans 42-45: review 42, write 43-45 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sprint 42 (reviewed): fixed HiveMind-specific example in smoke test step, added review notes clarifying asToolResponse usage and test file pattern. Sprint 43: Cross-File Context Assembly — WorkspaceFileIndex, ContextSliceAssembler, TokenBudgetEnforcer, whetstone_assemble_context MCP tool (Steps 674-678). Sprint 44: Taskitem Self-Containment Validator — PrerequisiteOpResolver, SelfContainmentScorer, TaskitemQualityAuditor, whetstone_validate_taskitem MCP tool (Steps 679-683). Directly addresses whether taskitem token reduction is real. Sprint 45: Agent Session Metrics + A/B Harness — AgentSessionRecorder, TaskCompletionMetrics, ABTestComparison, whetstone_start_recording + whetstone_get_metrics MCP tools (Steps 684-688). Produces the measured comparison between Whetstone-assisted and baseline agent runs. All sprints are generic editor tooling — no coupling to external systems. Co-Authored-By: Claude Sonnet 4.6 --- sprint42_plan.md | 186 ++++++++++++++++++++++++++++++++++ sprint43_plan.md | 217 ++++++++++++++++++++++++++++++++++++++++ sprint44_plan.md | 225 ++++++++++++++++++++++++++++++++++++++++++ sprint45_plan.md | 252 +++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 880 insertions(+) create mode 100644 sprint42_plan.md create mode 100644 sprint43_plan.md create mode 100644 sprint44_plan.md create mode 100644 sprint45_plan.md diff --git a/sprint42_plan.md b/sprint42_plan.md new file mode 100644 index 0000000..53d1d32 --- /dev/null +++ b/sprint42_plan.md @@ -0,0 +1,186 @@ +# Sprint 42 Plan: MCP Modeling Tools — Wire Unregistered Generators + +## Context + +Sprint 41 wired `whetstone_schema_to_cpp` and `whetstone_generate_dispatch_table` +(Steps 664-668, tool count 82→84). During that audit, two more existing classes +were found unregistered: + +- `whetstone_generate_project` (`ProjectSkeletonGenerator`, Step 639) +- `whetstone_generate_inference_job` (`InferenceJobGenerator`, Step 660) + +Both C++ classes are complete and tested. Both have `toolName()` methods. This +sprint is pure plumbing — wire them into the live MCP server following the Sprint 41 +pattern. No new logic is written. + +--- + +## Existing C++ Implementations (Do Not Rewrite) + +| Class | File | Step | Interface | +|-------|------|------|-----------| +| `ProjectSkeletonGenerator` | `editor/src/ProjectSkeletonGenerator.h` | 639 | `::generate(name, description, dependencies)` → `GeneratedProjectSkeleton{success, cmakeLists, mainCpp, moduleHeaders, dependencies, errors}`. Use `::asToolResponse(out)` to convert to JSON for MCP return. | +| `InferenceJobGenerator` | `editor/src/InferenceJobGenerator.h` | 660 | `::generate(goal, entropyScore, files, bounty)` → `json` directly (no `asToolResponse` needed — already returns JSON). | + +Both `toolName()` methods already return the correct MCP tool name strings. + +--- + +## Review Notes (from Sprint 41 agent review) + +- Original plan did not specify the `step672_test.cpp` file for the smoke test step. + This sprint follows the pattern established in Sprint 41 Step 667: a test file is + written for the rebuild+smoke step using the MCP Content-Length framing protocol. +- `InferenceJobGenerator::generate()` returns `json` directly; no `asToolResponse` + wrapper is needed (unlike `ProjectSkeletonGenerator`). +- Call order in `registerWhetstoneTools()`: add `registerModelingTools()` after + `registerCodegenTools()` — do not rearrange existing calls. +- Architecture invariant: header-only, 600-line limit, no new vcpkg dependencies. + +--- + +## Wiring Pattern + +Identical to Sprint 41 (`RegisterCodegenTools.h`): + +1. Create `editor/src/mcp/RegisterModelingTools.h` +2. Push tool definitions to `tools_`, assign lambdas to `toolHandlers_` +3. `#include "mcp/RegisterModelingTools.h"` in `MCPServer.h` (after RegisterCodegenTools.h) +4. `registerModelingTools();` in `registerWhetstoneTools()` in `RegisterOnboardingAndAllTools.h` +5. Add two entries to `tools/claude/tools.json` (tool count 84→86) +6. Rebuild `whetstone_mcp` and smoke test both tools + +--- + +## Steps + +### Step 669: `RegisterModelingTools.h` — wire `whetstone_generate_project` (12 tests) + +Create `editor/src/mcp/RegisterModelingTools.h`. Register `whetstone_generate_project`. + +Input schema (all args → `ProjectSkeletonGenerator::generate(name, description, dependencies)`): +```json +{ + "name": { "type": "string", "description": "Project name (required)." }, + "description": { "type": "string", "description": "Short project description." }, + "dependencies": { "type": "array", "description": "CMake find_package dependencies.", + "items": { "type": "string" } } +} +``` +Required: `["name", "description"]` + +Handler: call `ProjectSkeletonGenerator::generate(...)`, return +`ProjectSkeletonGenerator::asToolResponse(out)` as the MCP response JSON. + +Tests (12): name required (empty name → error), description required, valid inputs +succeed, cmakeLists non-empty on success, mainCpp non-empty on success, moduleHeaders +non-empty on success, single dependency appears in cmakeLists, multiple dependencies +all appear, toolName() == "whetstone_generate_project", success==false on empty name, +errors non-empty on failure, success==true on valid minimal input. + +### Step 670: `RegisterModelingTools.h` — wire `whetstone_generate_inference_job` (12 tests) + +Add `whetstone_generate_inference_job` to the same `RegisterModelingTools.h`. + +Input schema: +```json +{ + "goal": { "type": "string", "description": "Refactor goal description (required)." }, + "entropy_score": { "type": "integer", "description": "Entropy score from entropy scanner (required, ≥0)." }, + "files": { "type": "array", "description": "Source files relevant to this job.", + "items": { "type": "string" } }, + "bounty": { "type": "string", "description": "Bounty level: low|normal|high (default: normal)." } +} +``` +Required: `["goal", "entropy_score", "files"]` + +Handler: parse args, call `InferenceJobGenerator::generate(goal, entropy_score, files, bounty)`. +Return result directly (it is already a json object with `success`, `job`, etc.). + +Tests (12): goal required, entropy_score required, files required, valid inputs succeed, +negative entropy_score → success==false, empty goal → success==false, result.job.type=="refactor", +result.job.goal matches input, result.job.context.files matches input, bounty defaults to +"normal", explicit bounty "high" reflected in output, toolName()=="whetstone_generate_inference_job". + +### Step 671: Wire into `MCPServer.h` + `RegisterOnboardingAndAllTools.h` + `tools.json` (8 tests) + +**MCPServer.h:** add `#include "mcp/RegisterModelingTools.h"` after `RegisterCodegenTools.h`. + +**RegisterOnboardingAndAllTools.h:** add `registerModelingTools();` after `registerCodegenTools();`. + +**tools/claude/tools.json:** add two entries (tool count 84→86): +```json +{ + "name": "whetstone_generate_project", + "description": "Generate a C++ project skeleton from a name, description, and dependencies. Returns CMakeLists.txt content, main.cpp stub, and module header paths.", + "input_schema": { + "type": "object", + "properties": { + "name": { "type": "string", "description": "Project name." }, + "description": { "type": "string", "description": "Short project description." }, + "dependencies": { "type": "array", "items": { "type": "string" } } + }, + "required": ["name", "description"] + } +}, +{ + "name": "whetstone_generate_inference_job", + "description": "Generate a HiveMind inference job payload from an entropy observation. Returns a structured job suitable for dispatch to the nexus.", + "input_schema": { + "type": "object", + "properties": { + "goal": { "type": "string" }, + "entropy_score": { "type": "integer" }, + "files": { "type": "array", "items": { "type": "string" } }, + "bounty": { "type": "string" } + }, + "required": ["goal", "entropy_score", "files"] + } +} +``` + +Tests (8): tools.json valid JSON, whetstone_generate_project in tools.json, +whetstone_generate_inference_job in tools.json, tool count == 86, no duplicate names, +both tools have input_schema.required, MCPServer includes RegisterModelingTools.h, +registerWhetstoneTools() calls registerModelingTools(). + +### Step 672: Rebuild `whetstone_mcp` + smoke test (8 tests) + +Build: +```bash +cmake -S editor -B editor/build-native +cmake --build editor/build-native --target whetstone_mcp +``` + +Write `editor/tests/step672_test.cpp`. Test via MCP Content-Length framing protocol +(same pattern as step667_test.cpp from Sprint 41): + +1. `tools/list` → verify both new tool names present, total count == 86 +2. `whetstone_generate_project` call with `{name:"MyApp", description:"Example app", dependencies:["nlohmann_json"]}` → verify success=true, cmakeLists non-empty +3. `whetstone_generate_inference_job` call with `{goal:"extract duplicate handlers", entropy_score:3, files:["main.cpp"]}` → verify success=true, job.type=="refactor" + +Tests (8): binary builds, binary runs on --help, tools/list is valid JSON, generate_project +in tools/list, generate_inference_job in tools/list, generate_project call success=true, +generate_inference_job call success=true, binary exits cleanly. + +### Step 673: Sprint 42 Integration Summary (8 tests) + +Create `editor/src/Sprint42IntegrationSummary.h` (model on Sprint41IntegrationSummary.h). +Record: tool_count_before=84, tool_count_after=86, steps_completed=5 (669-673), +files_added=[RegisterModelingTools.h, Sprint42IntegrationSummary.h], +files_modified=[MCPServer.h, RegisterOnboardingAndAllTools.h, tools/claude/tools.json]. + +Tests (8): struct constructable, generate_project_wired, generate_inference_job_wired, +tool_count_before==84, tool_count_after==86, steps_completed==5, +files_added contains RegisterModelingTools.h, success==true. + +--- + +## Architecture Gate (All Steps) + +- Header-only for all new registration and summary files +- 600-line max per file — `RegisterModelingTools.h` will be well under +- No new vcpkg dependencies +- Build target: `whetstone_mcp` must rebuild without new warnings +- Call order in `registerWhetstoneTools()` must not rearrange existing calls +- Full sprint regression at step 673: all steps 669-673 pass diff --git a/sprint43_plan.md b/sprint43_plan.md new file mode 100644 index 0000000..e34c88d --- /dev/null +++ b/sprint43_plan.md @@ -0,0 +1,217 @@ +# Sprint 43 Plan: Cross-File Context Assembly + +## Context + +Sprint 42 completes the backlog of unregistered MCP tools (tool count 84→86). +The architect intake pipeline (Sprints 32, 36) generates annotated taskitems with +`prerequisiteOps` lists — file paths and symbol names the executing agent needs. +Currently, agents read full files to satisfy those ops, negating much of the token +budget advantage taskitems are designed to create. + +Sprint 43 adds surgical context slicing: given a taskitem's `prerequisiteOps`, +extract only the relevant declarations, function bodies, or type definitions from +each file — not the whole file. Respects a configurable token budget. Entirely +generic — no coupling to any external system. + +--- + +## Goals + +1. Index workspace files by symbol definition (fast lookup by name, not full-file scan) +2. Assemble minimal context slices for a given set of file paths + symbol names +3. Enforce a token budget — trim to fit, report what was dropped +4. Expose as `whetstone_assemble_context` MCP tool + +--- + +## Steps + +### Step 674: `WorkspaceFileIndex` — symbol definition index (12 tests) + +Create `editor/src/WorkspaceFileIndex.h`. + +Index all files under a workspace root. For each file, extract top-level +declarations (function signatures, class names, struct names, type aliases, +constants) using line-level heuristics — no full parse required for this index. + +```cpp +struct SymbolLocation { + std::string file; + int lineStart; + int lineEnd; // end of signature or short body + std::string symbolName; + std::string symbolKind; // "function" | "class" | "struct" | "type" | "const" +}; + +class WorkspaceFileIndex { +public: + static WorkspaceFileIndex build(const std::string& workspaceRoot); + std::vector find(const std::string& symbolName) const; + std::vector findInFile(const std::string& filePath) const; + int fileCount() const; + int symbolCount() const; +}; +``` + +Tests (12): empty root returns empty index, single-file workspace indexes it, +function signatures found by name, class declarations found by name, struct +declarations found by name, findInFile returns all symbols in that file, find() +returns all matches across files, symbolCount > 0 after indexing real file, +fileCount matches input file count, duplicate symbol names all returned, build() +doesn't throw on unreadable file (skips gracefully), kind field is non-empty. + +### Step 675: `ContextSliceAssembler` — extract minimal file sections (12 tests) + +Create `editor/src/ContextSliceAssembler.h`. + +Given a list of `{file, symbol_or_line_range}` requests and a workspace index, +extract only the relevant lines from each file. A "slice" is: +- For a named symbol: the lines from its declaration through the closing brace/semicolon +- For a line range: exactly those lines +- For a file with no symbol specified: first N lines (configurable, default 40) + +```cpp +struct SliceRequest { + std::string filePath; + std::string symbolName; // empty = use lineHint or head + int lineHint; // 0 = not specified + int headLines; // fallback if no symbol match; default 40 +}; + +struct ContextSlice { + std::string filePath; + std::string symbolName; + int lineStart; + int lineEnd; + std::string content; // the actual extracted lines + bool found; +}; + +class ContextSliceAssembler { +public: + static std::vector assemble( + const std::vector& requests, + const WorkspaceFileIndex& index); +}; +``` + +Tests (12): empty requests returns empty result, named symbol found returns +correct lines, unknown symbol returns head fallback, line range returns correct +lines, content non-empty on success, found==false when symbol missing, multiple +requests all processed, head fallback respects headLines parameter, function body +slice includes closing brace, class slice includes closing brace, file path +preserved in result, slice content is valid UTF-8 substring of source file. + +### Step 676: `TokenBudgetEnforcer` — trim assembled context to budget (12 tests) + +Create `editor/src/TokenBudgetEnforcer.h`. + +Given a list of `ContextSlice` objects and a max token budget, trim the list +to fit. Strategy: priority order (slices earlier in the list are higher priority), +each slice is dropped whole if it doesn't fit. Returns the slices that fit plus +a report of what was dropped. + +Token estimation: 1 token ≈ 4 characters (conservative approximation, no model +dependency). The enforcer does NOT call any model API. + +```cpp +struct BudgetReport { + int totalSlices; + int includedSlices; + int droppedSlices; + int estimatedTokensUsed; + int budgetTokens; + bool budgetExceeded; // true if any slice was dropped + std::vector droppedFiles; +}; + +class TokenBudgetEnforcer { +public: + static std::pair, BudgetReport> + enforce(const std::vector& slices, int maxTokens); + + static int estimateTokens(const std::string& text); +}; +``` + +Tests (12): empty slices returns empty + zero report, single slice under budget +kept, single slice over budget dropped, multiple slices: high-priority kept when +budget tight, droppedFiles list matches dropped slices, estimatedTokensUsed <= +budgetTokens when no drops, budgetExceeded==false when all fit, budgetExceeded==true +when any dropped, estimateTokens("") == 0, estimateTokens("abcd") == 1, +report.totalSlices == input count, report.includedSlices + report.droppedSlices +== report.totalSlices. + +### Step 677: `whetstone_assemble_context` MCP tool (8 tests) + +Add to `RegisterCodegenTools.h` (or new `RegisterContextTools.h` if file nears 600 lines). + +Input schema: +```json +{ + "files": { + "type": "array", + "description": "List of slice requests.", + "items": { + "type": "object", + "properties": { + "path": { "type": "string", "description": "File path relative to workspace." }, + "symbol": { "type": "string", "description": "Symbol name to extract (optional)." }, + "line_hint": { "type": "integer", "description": "Line number hint (optional)." }, + "head_lines": { "type": "integer", "description": "Head fallback line count (optional, default 40)." } + }, + "required": ["path"] + } + }, + "max_tokens": { "type": "integer", "description": "Token budget (default 8000)." } +} +``` + +Output: +```json +{ + "success": true, + "slices": [ + { "file": "...", "symbol": "...", "line_start": 10, "line_end": 25, "content": "..." } + ], + "budget_report": { + "total_slices": 3, + "included_slices": 2, + "dropped_slices": 1, + "estimated_tokens_used": 1840, + "budget_tokens": 2000, + "budget_exceeded": true, + "dropped_files": ["large_file.cpp"] + } +} +``` + +Wire into MCPServer.h, RegisterOnboardingAndAllTools.h, and tools.json. +Tool count 86→87. + +Tests (8): tool registered, empty files array returns success with empty slices, +valid file path returns slice, missing path arg returns error, symbol slicing +returns content, budget_report present in output, budget_exceeded==false when +all slices fit, tool count in tools.json == 87. + +### Step 678: Sprint 43 Integration Summary (8 tests) + +Create `editor/src/Sprint43IntegrationSummary.h`. +Record: tool_count_before=86, tool_count_after=87, steps_completed=5 (674-678), +files_added=[WorkspaceFileIndex.h, ContextSliceAssembler.h, TokenBudgetEnforcer.h, +Sprint43IntegrationSummary.h], success=true. + +Tests (8): struct constructable, workspace_index_builds, slice_assembler_works, +budget_enforcer_works, tool_count_before==86, tool_count_after==87, +steps_completed==5, success==true. + +--- + +## Architecture Gate (All Steps) + +- Header-only C++ for all new source files (no new .cpp except test files) +- 600-line max per file +- No new vcpkg dependencies — uses only `std::filesystem` and string operations +- `estimateTokens` must not call any external model API +- All slicing is done on raw file text — no AST dependency (fast, portable) +- Full sprint regression at step 678 diff --git a/sprint44_plan.md b/sprint44_plan.md new file mode 100644 index 0000000..2ac5bb0 --- /dev/null +++ b/sprint44_plan.md @@ -0,0 +1,225 @@ +# Sprint 44 Plan: Taskitem Self-Containment Validator + +## Context + +The architect intake pipeline (Sprints 32, 36) produces `AnnotatedTaskitem` objects +with `confidence`, `escalate`, `reasons[]`, and `prerequisiteOps[]`. The token +reduction claim — that a well-formed taskitem lets an agent execute without reading +external context — depends on those fields being accurate and complete. + +Sprint 44 adds a validation layer: given one or more taskitems, score how +self-contained they actually are, flag which `prerequisiteOps` don't resolve to +real workspace files, and produce a quality report that agents and humans can act on. + +This sprint is entirely generic editor tooling. No coupling to any external system. + +--- + +## Goals + +1. Resolve each `prerequisiteOp` string to a real file path in the workspace +2. Score each taskitem's self-containment (0–100) +3. Produce a batch quality report with actionable improvement suggestions +4. Expose validation as `whetstone_validate_taskitem` MCP tool + +--- + +## Steps + +### Step 679: `PrerequisiteOpResolver` — resolve ops to workspace paths (12 tests) + +Create `editor/src/PrerequisiteOpResolver.h`. + +A `prerequisiteOp` string is free-form, e.g.: +- `"read editor/src/SchemaToCppGenerator.h"` +- `"read editor/src/mcp/RegisterASTTools.h lines 30-60"` +- `"read editor/CMakeLists.txt"` +- `"run cmake --build editor/build-native --target step664_test"` + +Parse each op into a structured `ResolvedOp` and check whether the file referenced +(if any) exists under the workspace root. + +```cpp +enum class OpKind { ReadFile, RunCommand, Unknown }; + +struct ResolvedOp { + std::string raw; // original string + OpKind kind; + std::string filePath; // extracted path (empty for non-file ops) + bool fileExists; // false if filePath non-empty but not found + int lineStart; // 0 if not specified + int lineEnd; // 0 if not specified +}; + +class PrerequisiteOpResolver { +public: + static ResolvedOp resolve(const std::string& op, + const std::string& workspaceRoot); + static std::vector resolveAll( + const std::vector& ops, + const std::string& workspaceRoot); +}; +``` + +Tests (12): empty op string → kind==Unknown, "read file.h" → kind==ReadFile, +extracted filePath non-empty for read op, fileExists==true when file present, +fileExists==false when file absent, line range parsed ("lines 30-60" → 30,60), +no line range → lineStart==0 lineEnd==0, "run cmake ..." → kind==RunCommand, +RunCommand has empty filePath, resolveAll processes multiple ops, raw field +preserved, unknown op format → kind==Unknown gracefully. + +### Step 680: `SelfContainmentScorer` — score a taskitem 0–100 (12 tests) + +Create `editor/src/SelfContainmentScorer.h`. + +Score algorithm (points deducted from 100): +- **prerequisiteOps missing or empty:** −30 +- **Each prerequisiteOp that doesn't resolve to an existing file:** −10 (max −30) +- **reasons[] missing or empty:** −20 +- **reasons[] has < 3 entries:** −10 +- **confidence not set (0 or missing):** −10 +- **title empty:** −20 +- **dependencyTaskIds present but non-empty (has unresolved dependencies):** −5 + +Score is clamped to [0, 100]. + +```cpp +struct ScoredTaskitem { + std::string taskId; + int score; // 0-100 + bool selfContained; // score >= 80 + std::vector issues; // human-readable deduction reasons +}; + +struct TaskitemInput { + std::string taskId; + std::string title; + std::vector prerequisiteOps; + std::vector reasons; + int confidence; // 0-100 + std::vector dependencyTaskIds; +}; + +class SelfContainmentScorer { +public: + static ScoredTaskitem score(const TaskitemInput& item, + const std::string& workspaceRoot); +}; +``` + +Tests (12): fully specified taskitem scores >= 80, empty prerequisiteOps deducts 30, +missing reasons deducts 20, empty title deducts 20, unresolved file op deducts 10, +two unresolved files deducts 20, score clamped to 0 (not negative), score clamped +to 100 (not over), selfContained==true when score>=80, selfContained==false when +score<80, issues list non-empty when deductions made, issues empty when perfect score. + +### Step 681: `TaskitemQualityAuditor` — batch report (12 tests) + +Create `editor/src/TaskitemQualityAuditor.h`. + +Given a list of `TaskitemInput` objects and a workspace root, score all of them +and produce a summary report with improvement suggestions. + +```cpp +struct QualityAuditReport { + int totalTaskitems; + int selfContainedCount; // score >= 80 + int warningCount; // score 50-79 + int failingCount; // score < 50 + double averageScore; + std::vector results; // one per input item + std::vector topIssues; // most common deduction reasons +}; + +class TaskitemQualityAuditor { +public: + static QualityAuditReport audit(const std::vector& items, + const std::string& workspaceRoot); +}; +``` + +Tests (12): empty input returns valid report with 0 counts, single passing item +counted, single failing item counted, averageScore computed correctly, topIssues +non-empty when issues exist, selfContainedCount + warningCount + failingCount == +totalTaskitems, results.size() == input.size(), all results have taskId set, +mixed batch (some pass some fail) counted correctly, report is deterministic +(same input → same output), totalTaskitems matches input size, warningCount correct +for 50-79 range. + +### Step 682: `whetstone_validate_taskitem` MCP tool (8 tests) + +Add `whetstone_validate_taskitem` to `RegisterCodegenTools.h` (or a new +`RegisterValidationTools.h` if approaching 600 lines). + +Input schema: +```json +{ + "taskitems": { + "type": "array", + "description": "One or more taskitems to validate.", + "items": { + "type": "object", + "properties": { + "task_id": { "type": "string" }, + "title": { "type": "string" }, + "prerequisite_ops": { "type": "array", "items": { "type": "string" } }, + "reasons": { "type": "array", "items": { "type": "string" } }, + "confidence": { "type": "integer" }, + "dependency_task_ids": { "type": "array", "items": { "type": "string" } } + }, + "required": ["task_id"] + } + }, + "workspace": { "type": "string", "description": "Workspace root to resolve file paths against." } +} +``` + +Output: +```json +{ + "success": true, + "report": { + "total_taskitems": 3, + "self_contained_count": 2, + "warning_count": 1, + "failing_count": 0, + "average_score": 87.3, + "results": [ + { "task_id": "T1", "score": 95, "self_contained": true, "issues": [] }, + ... + ], + "top_issues": ["prerequisiteOps missing or empty", "reasons[] has fewer than 3 entries"] + } +} +``` + +Wire into MCPServer.h, RegisterOnboardingAndAllTools.h, tools.json (86→88 or 87→88 +depending on sprint 43 completion). + +Tests (8): tool registered, empty taskitems array returns valid report with 0 counts, +single valid taskitem returns score, missing task_id returns error, prerequisite_ops +resolved against workspace, report.results.length matches input length, +self_contained_count + warning_count + failing_count == total, tool count updated. + +### Step 683: Sprint 44 Integration Summary (8 tests) + +Create `editor/src/Sprint44IntegrationSummary.h`. +Record: steps_completed=5 (679-683), files_added=[PrerequisiteOpResolver.h, +SelfContainmentScorer.h, TaskitemQualityAuditor.h, Sprint44IntegrationSummary.h], +success=true. + +Tests (8): struct constructable, resolver_works, scorer_works, auditor_works, +steps_completed==5, files_added contains PrerequisiteOpResolver.h, +files_added contains SelfContainmentScorer.h, success==true. + +--- + +## Architecture Gate (All Steps) + +- Header-only C++ for all new source files +- 600-line max per file — split at `Extended.h` if approaching +- No new vcpkg dependencies — uses only std::filesystem and string operations +- Scoring algorithm is deterministic and purely computational (no model calls) +- `whetstone_validate_taskitem` must work against any workspace path, not just + the current project — no hardcoded paths +- Full sprint regression at step 683 diff --git a/sprint45_plan.md b/sprint45_plan.md new file mode 100644 index 0000000..9efcb42 --- /dev/null +++ b/sprint45_plan.md @@ -0,0 +1,252 @@ +# Sprint 45 Plan: Agent Session Metrics + A/B Harness + +## Context + +Two prior questions about Whetstone remain unanswered with real data: + +1. **Does using Whetstone tools actually reduce token consumption?** The 90%+ + reduction claim is architectural — a well-formed taskitem lets an agent skip + file-reading. But the claim has never been measured. + +2. **Is the output quality equivalent or better?** Speed without quality is useless. + +Sprint 45 builds the instrumentation to answer both. It records per-session +metrics (tool calls, estimated token cost, files read, duration) and provides +a lightweight A/B comparison framework: run the same bounded task with and without +Whetstone tools, compare results. + +This is entirely generic editor tooling. The metrics format, the comparison harness, +and the MCP tools are not coupled to any external system or project. + +--- + +## Goals + +1. Record what an agent does during a session: which tools it called, how much + context it consumed (estimated tokens), and how long each call took +2. Capture a structured task outcome (what changed, did tests pass) +3. Compare two runs of the same task (Whetstone-assisted vs baseline) +4. Expose recording control and report retrieval as MCP tools + +--- + +## Steps + +### Step 684: `AgentSessionRecorder` — per-session tool call instrumentation (12 tests) + +Create `editor/src/AgentSessionRecorder.h`. + +Records every MCP tool call within a session: tool name, input size (chars), +output size (chars), estimated tokens (input+output), duration (ms), and whether +it resulted in a file read (heuristic: tool name contains "read", "get_ast", +"assemble_context", or "workspace_list"). + +```cpp +struct ToolCallRecord { + std::string toolName; + int inputChars; + int outputChars; + int estimatedTokens; // (inputChars + outputChars) / 4 + long long durationMs; + bool wasFileRead; +}; + +struct SessionRecord { + std::string sessionId; + std::string taskDescription; + std::vector calls; + int totalToolCalls; + int totalEstimatedTokens; + int fileReadCount; + long long totalDurationMs; +}; + +class AgentSessionRecorder { +public: + void start(const std::string& sessionId, const std::string& taskDescription); + void record(const ToolCallRecord& call); + SessionRecord finish(); + static ToolCallRecord makeRecord(const std::string& toolName, + const std::string& inputJson, + const std::string& outputJson, + long long durationMs); +}; +``` + +Tests (12): start sets sessionId, record() adds to calls, finish() returns complete +record, totalToolCalls matches recorded count, totalEstimatedTokens is sum of call +tokens, fileReadCount counts only file-read calls, wasFileRead true for "read" +tools, wasFileRead false for non-file tools, estimatedTokens = (inputChars + +outputChars) / 4, durationMs preserved, taskDescription preserved in output, +empty session returns zero counts. + +### Step 685: `TaskCompletionMetrics` — capture task outcome (12 tests) + +Create `editor/src/TaskCompletionMetrics.h`. + +Captures the observable outcome of a completed task: lines of code changed +(before/after diff), whether a provided test command succeeded, and a +quality rating (0–100) supplied by the agent or reviewer. + +```cpp +struct TaskOutcome { + std::string taskId; + int linesChanged; // abs(after_lines - before_lines) + int linesAdded; + int linesRemoved; + bool testsProvided; + bool testsPassed; + int qualityRating; // 0-100, agent-supplied + std::string notes; +}; + +class TaskCompletionMetrics { +public: + static TaskOutcome diff(const std::string& taskId, + const std::string& before, + const std::string& after, + bool testsPassed, + int qualityRating, + const std::string& notes = ""); + static int countLines(const std::string& text); +}; +``` + +Tests (12): empty before and after → 0 changes, added lines counted correctly, +removed lines counted correctly, linesChanged == linesAdded + linesRemoved (net), +testsPassed preserved, qualityRating clamped to [0,100], notes preserved, taskId +preserved, countLines("") == 0, countLines("a\nb\n") == 2, countLines("a\nb") == 2, +both before and after empty → testsProvided==false when tests not given. + +### Step 686: `ABTestComparison` — compare two session records (12 tests) + +Create `editor/src/ABTestComparison.h`. + +Given two `SessionRecord` objects (baseline and whetstone-assisted) for the same +task, compute the comparison metrics. + +```cpp +struct ABComparisonResult { + std::string taskId; + + // Token comparison + int baselineTokens; + int whetstoneTokens; + double tokenReductionPct; // (baseline - whetstone) / baseline * 100 + + // File read comparison + int baselineFileReads; + int whetstoneFileReads; + double fileReadReductionPct; + + // Duration comparison + long long baselineDurationMs; + long long whetstoneDurationMs; + double durationDeltaPct; // positive = whetstone faster + + // Quality comparison (from TaskOutcome) + int baselineQuality; + int whetstoneQuality; + int qualityDelta; // whetstone - baseline (positive = better) + + bool whetstoneWon; // tokenReductionPct > 0 && qualityDelta >= 0 +}; + +class ABTestComparison { +public: + static ABComparisonResult compare( + const std::string& taskId, + const SessionRecord& baseline, + const SessionRecord& whetstone, + int baselineQuality, + int whetstoneQuality); +}; +``` + +Tests (12): tokenReductionPct computed correctly, fileReadReductionPct computed +correctly, durationDeltaPct computed correctly (positive = faster), qualityDelta = +whetstoneQuality - baselineQuality, whetstoneWon==true when tokens reduced and +quality >=, whetstoneWon==false when tokens higher, tokenReductionPct==0 when +equal tokens, tokenReductionPct negative allowed (whetstone used more tokens), +baselineTokens preserved, whetstoneTokens preserved, taskId preserved, +zero baseline tokens does not divide by zero (returns 0.0). + +### Step 687: `whetstone_start_recording` + `whetstone_get_metrics` MCP tools (8 tests) + +Add two MCP tools. Wire into MCPServer.h + RegisterOnboardingAndAllTools.h + +tools.json. + +**`whetstone_start_recording`** — begins a new recording session. +```json +{ + "session_id": { "type": "string", "description": "Unique session identifier." }, + "task_description": { "type": "string", "description": "What task this session is executing." } +} +``` +Response: `{"success": true, "session_id": "..."}` + +**`whetstone_get_metrics`** — retrieves and finalizes the current session record, +optionally comparing against a provided baseline JSON. +```json +{ + "session_id": { "type": "string" }, + "baseline": { "type": "object", "description": "Optional prior SessionRecord JSON for A/B comparison." }, + "quality_rating": { "type": "integer", "description": "Agent's self-assessment 0-100 for this session." }, + "baseline_quality":{ "type": "integer", "description": "Quality rating for baseline (required if baseline provided)." } +} +``` +Response: `{"success": true, "session": {...}, "comparison": {...} }` (comparison +only present if baseline was provided). + +MCPServer needs an `AgentSessionRecorder` member to hold in-flight session state. + +Tests (8): start_recording registers tool, get_metrics registers tool, both in +tools.json, tool count updated, start_recording returns success=true, get_metrics +returns session record, comparison present when baseline provided, comparison absent +when no baseline. + +### Step 688: Sprint 45 Integration Summary (8 tests) + +Create `editor/src/Sprint45IntegrationSummary.h`. +Record: steps_completed=5 (684-688), files_added=[AgentSessionRecorder.h, +TaskCompletionMetrics.h, ABTestComparison.h, Sprint45IntegrationSummary.h], +success=true. + +Full sprint regression: all steps 684-688 pass. + +Tests (8): struct constructable, recorder_works, metrics_works, comparison_works, +steps_completed==5, files_added contains AgentSessionRecorder.h, +files_added contains ABTestComparison.h, success==true. + +--- + +## Architecture Gate (All Steps) + +- Header-only C++ for all new source files +- 600-line limit — `AgentSessionRecorder.h` may need Extended split if state + management grows; check at step 684 +- No new vcpkg dependencies +- Token estimation must NOT call any model API — purely character-count heuristic +- `ABTestComparison::compare` must handle zero-token baseline without crashing +- The recording state in MCPServer is per-server-instance (not persisted to disk) +- Full sprint regression at step 688 + +--- + +## Using the A/B Harness + +After Sprint 45, a comparison session looks like this: + +**Baseline run (no Whetstone tools):** +1. `whetstone_start_recording` with session_id="baseline-T1" +2. Agent executes task using only native file reads (no Whetstone tools) +3. `whetstone_get_metrics` with session_id="baseline-T1" → save result as baseline JSON + +**Whetstone-assisted run:** +1. `whetstone_start_recording` with session_id="ws-T1" +2. Agent executes same task using Whetstone tools (assemble_context, validate_taskitem, etc.) +3. `whetstone_get_metrics` with session_id="ws-T1", baseline= + → returns comparison with tokenReductionPct, fileReadReductionPct, qualityDelta + +The comparison is self-contained — no external logging service, no network calls. +Results can be copied to a markdown file or fed back into architect_intake for review.