Sprint plans 42-45: review 42, write 43-45
Sprint 42 (reviewed): fixed HiveMind-specific example in smoke test step, added review notes clarifying asToolResponse usage and test file pattern. Sprint 43: Cross-File Context Assembly — WorkspaceFileIndex, ContextSliceAssembler, TokenBudgetEnforcer, whetstone_assemble_context MCP tool (Steps 674-678). Sprint 44: Taskitem Self-Containment Validator — PrerequisiteOpResolver, SelfContainmentScorer, TaskitemQualityAuditor, whetstone_validate_taskitem MCP tool (Steps 679-683). Directly addresses whether taskitem token reduction is real. Sprint 45: Agent Session Metrics + A/B Harness — AgentSessionRecorder, TaskCompletionMetrics, ABTestComparison, whetstone_start_recording + whetstone_get_metrics MCP tools (Steps 684-688). Produces the measured comparison between Whetstone-assisted and baseline agent runs. All sprints are generic editor tooling — no coupling to external systems. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
186
sprint42_plan.md
Normal file
186
sprint42_plan.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# Sprint 42 Plan: MCP Modeling Tools — Wire Unregistered Generators
|
||||
|
||||
## Context
|
||||
|
||||
Sprint 41 wired `whetstone_schema_to_cpp` and `whetstone_generate_dispatch_table`
|
||||
(Steps 664-668, tool count 82→84). During that audit, two more existing classes
|
||||
were found unregistered:
|
||||
|
||||
- `whetstone_generate_project` (`ProjectSkeletonGenerator`, Step 639)
|
||||
- `whetstone_generate_inference_job` (`InferenceJobGenerator`, Step 660)
|
||||
|
||||
Both C++ classes are complete and tested. Both have `toolName()` methods. This
|
||||
sprint is pure plumbing — wire them into the live MCP server following the Sprint 41
|
||||
pattern. No new logic is written.
|
||||
|
||||
---
|
||||
|
||||
## Existing C++ Implementations (Do Not Rewrite)
|
||||
|
||||
| Class | File | Step | Interface |
|
||||
|-------|------|------|-----------|
|
||||
| `ProjectSkeletonGenerator` | `editor/src/ProjectSkeletonGenerator.h` | 639 | `::generate(name, description, dependencies)` → `GeneratedProjectSkeleton{success, cmakeLists, mainCpp, moduleHeaders, dependencies, errors}`. Use `::asToolResponse(out)` to convert to JSON for MCP return. |
|
||||
| `InferenceJobGenerator` | `editor/src/InferenceJobGenerator.h` | 660 | `::generate(goal, entropyScore, files, bounty)` → `json` directly (no `asToolResponse` needed — already returns JSON). |
|
||||
|
||||
Both `toolName()` methods already return the correct MCP tool name strings.
|
||||
|
||||
---
|
||||
|
||||
## Review Notes (from Sprint 41 agent review)
|
||||
|
||||
- Original plan did not specify the `step672_test.cpp` file for the smoke test step.
|
||||
This sprint follows the pattern established in Sprint 41 Step 667: a test file is
|
||||
written for the rebuild+smoke step using the MCP Content-Length framing protocol.
|
||||
- `InferenceJobGenerator::generate()` returns `json` directly; no `asToolResponse`
|
||||
wrapper is needed (unlike `ProjectSkeletonGenerator`).
|
||||
- Call order in `registerWhetstoneTools()`: add `registerModelingTools()` after
|
||||
`registerCodegenTools()` — do not rearrange existing calls.
|
||||
- Architecture invariant: header-only, 600-line limit, no new vcpkg dependencies.
|
||||
|
||||
---
|
||||
|
||||
## Wiring Pattern
|
||||
|
||||
Identical to Sprint 41 (`RegisterCodegenTools.h`):
|
||||
|
||||
1. Create `editor/src/mcp/RegisterModelingTools.h`
|
||||
2. Push tool definitions to `tools_`, assign lambdas to `toolHandlers_`
|
||||
3. `#include "mcp/RegisterModelingTools.h"` in `MCPServer.h` (after RegisterCodegenTools.h)
|
||||
4. `registerModelingTools();` in `registerWhetstoneTools()` in `RegisterOnboardingAndAllTools.h`
|
||||
5. Add two entries to `tools/claude/tools.json` (tool count 84→86)
|
||||
6. Rebuild `whetstone_mcp` and smoke test both tools
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 669: `RegisterModelingTools.h` — wire `whetstone_generate_project` (12 tests)
|
||||
|
||||
Create `editor/src/mcp/RegisterModelingTools.h`. Register `whetstone_generate_project`.
|
||||
|
||||
Input schema (all args → `ProjectSkeletonGenerator::generate(name, description, dependencies)`):
|
||||
```json
|
||||
{
|
||||
"name": { "type": "string", "description": "Project name (required)." },
|
||||
"description": { "type": "string", "description": "Short project description." },
|
||||
"dependencies": { "type": "array", "description": "CMake find_package dependencies.",
|
||||
"items": { "type": "string" } }
|
||||
}
|
||||
```
|
||||
Required: `["name", "description"]`
|
||||
|
||||
Handler: call `ProjectSkeletonGenerator::generate(...)`, return
|
||||
`ProjectSkeletonGenerator::asToolResponse(out)` as the MCP response JSON.
|
||||
|
||||
Tests (12): name required (empty name → error), description required, valid inputs
|
||||
succeed, cmakeLists non-empty on success, mainCpp non-empty on success, moduleHeaders
|
||||
non-empty on success, single dependency appears in cmakeLists, multiple dependencies
|
||||
all appear, toolName() == "whetstone_generate_project", success==false on empty name,
|
||||
errors non-empty on failure, success==true on valid minimal input.
|
||||
|
||||
### Step 670: `RegisterModelingTools.h` — wire `whetstone_generate_inference_job` (12 tests)
|
||||
|
||||
Add `whetstone_generate_inference_job` to the same `RegisterModelingTools.h`.
|
||||
|
||||
Input schema:
|
||||
```json
|
||||
{
|
||||
"goal": { "type": "string", "description": "Refactor goal description (required)." },
|
||||
"entropy_score": { "type": "integer", "description": "Entropy score from entropy scanner (required, ≥0)." },
|
||||
"files": { "type": "array", "description": "Source files relevant to this job.",
|
||||
"items": { "type": "string" } },
|
||||
"bounty": { "type": "string", "description": "Bounty level: low|normal|high (default: normal)." }
|
||||
}
|
||||
```
|
||||
Required: `["goal", "entropy_score", "files"]`
|
||||
|
||||
Handler: parse args, call `InferenceJobGenerator::generate(goal, entropy_score, files, bounty)`.
|
||||
Return result directly (it is already a json object with `success`, `job`, etc.).
|
||||
|
||||
Tests (12): goal required, entropy_score required, files required, valid inputs succeed,
|
||||
negative entropy_score → success==false, empty goal → success==false, result.job.type=="refactor",
|
||||
result.job.goal matches input, result.job.context.files matches input, bounty defaults to
|
||||
"normal", explicit bounty "high" reflected in output, toolName()=="whetstone_generate_inference_job".
|
||||
|
||||
### Step 671: Wire into `MCPServer.h` + `RegisterOnboardingAndAllTools.h` + `tools.json` (8 tests)
|
||||
|
||||
**MCPServer.h:** add `#include "mcp/RegisterModelingTools.h"` after `RegisterCodegenTools.h`.
|
||||
|
||||
**RegisterOnboardingAndAllTools.h:** add `registerModelingTools();` after `registerCodegenTools();`.
|
||||
|
||||
**tools/claude/tools.json:** add two entries (tool count 84→86):
|
||||
```json
|
||||
{
|
||||
"name": "whetstone_generate_project",
|
||||
"description": "Generate a C++ project skeleton from a name, description, and dependencies. Returns CMakeLists.txt content, main.cpp stub, and module header paths.",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": { "type": "string", "description": "Project name." },
|
||||
"description": { "type": "string", "description": "Short project description." },
|
||||
"dependencies": { "type": "array", "items": { "type": "string" } }
|
||||
},
|
||||
"required": ["name", "description"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "whetstone_generate_inference_job",
|
||||
"description": "Generate a HiveMind inference job payload from an entropy observation. Returns a structured job suitable for dispatch to the nexus.",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"goal": { "type": "string" },
|
||||
"entropy_score": { "type": "integer" },
|
||||
"files": { "type": "array", "items": { "type": "string" } },
|
||||
"bounty": { "type": "string" }
|
||||
},
|
||||
"required": ["goal", "entropy_score", "files"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Tests (8): tools.json valid JSON, whetstone_generate_project in tools.json,
|
||||
whetstone_generate_inference_job in tools.json, tool count == 86, no duplicate names,
|
||||
both tools have input_schema.required, MCPServer includes RegisterModelingTools.h,
|
||||
registerWhetstoneTools() calls registerModelingTools().
|
||||
|
||||
### Step 672: Rebuild `whetstone_mcp` + smoke test (8 tests)
|
||||
|
||||
Build:
|
||||
```bash
|
||||
cmake -S editor -B editor/build-native
|
||||
cmake --build editor/build-native --target whetstone_mcp
|
||||
```
|
||||
|
||||
Write `editor/tests/step672_test.cpp`. Test via MCP Content-Length framing protocol
|
||||
(same pattern as step667_test.cpp from Sprint 41):
|
||||
|
||||
1. `tools/list` → verify both new tool names present, total count == 86
|
||||
2. `whetstone_generate_project` call with `{name:"MyApp", description:"Example app", dependencies:["nlohmann_json"]}` → verify success=true, cmakeLists non-empty
|
||||
3. `whetstone_generate_inference_job` call with `{goal:"extract duplicate handlers", entropy_score:3, files:["main.cpp"]}` → verify success=true, job.type=="refactor"
|
||||
|
||||
Tests (8): binary builds, binary runs on --help, tools/list is valid JSON, generate_project
|
||||
in tools/list, generate_inference_job in tools/list, generate_project call success=true,
|
||||
generate_inference_job call success=true, binary exits cleanly.
|
||||
|
||||
### Step 673: Sprint 42 Integration Summary (8 tests)
|
||||
|
||||
Create `editor/src/Sprint42IntegrationSummary.h` (model on Sprint41IntegrationSummary.h).
|
||||
Record: tool_count_before=84, tool_count_after=86, steps_completed=5 (669-673),
|
||||
files_added=[RegisterModelingTools.h, Sprint42IntegrationSummary.h],
|
||||
files_modified=[MCPServer.h, RegisterOnboardingAndAllTools.h, tools/claude/tools.json].
|
||||
|
||||
Tests (8): struct constructable, generate_project_wired, generate_inference_job_wired,
|
||||
tool_count_before==84, tool_count_after==86, steps_completed==5,
|
||||
files_added contains RegisterModelingTools.h, success==true.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate (All Steps)
|
||||
|
||||
- Header-only for all new registration and summary files
|
||||
- 600-line max per file — `RegisterModelingTools.h` will be well under
|
||||
- No new vcpkg dependencies
|
||||
- Build target: `whetstone_mcp` must rebuild without new warnings
|
||||
- Call order in `registerWhetstoneTools()` must not rearrange existing calls
|
||||
- Full sprint regression at step 673: all steps 669-673 pass
|
||||
217
sprint43_plan.md
Normal file
217
sprint43_plan.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Sprint 43 Plan: Cross-File Context Assembly
|
||||
|
||||
## Context
|
||||
|
||||
Sprint 42 completes the backlog of unregistered MCP tools (tool count 84→86).
|
||||
The architect intake pipeline (Sprints 32, 36) generates annotated taskitems with
|
||||
`prerequisiteOps` lists — file paths and symbol names the executing agent needs.
|
||||
Currently, agents read full files to satisfy those ops, negating much of the token
|
||||
budget advantage taskitems are designed to create.
|
||||
|
||||
Sprint 43 adds surgical context slicing: given a taskitem's `prerequisiteOps`,
|
||||
extract only the relevant declarations, function bodies, or type definitions from
|
||||
each file — not the whole file. Respects a configurable token budget. Entirely
|
||||
generic — no coupling to any external system.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Index workspace files by symbol definition (fast lookup by name, not full-file scan)
|
||||
2. Assemble minimal context slices for a given set of file paths + symbol names
|
||||
3. Enforce a token budget — trim to fit, report what was dropped
|
||||
4. Expose as `whetstone_assemble_context` MCP tool
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 674: `WorkspaceFileIndex` — symbol definition index (12 tests)
|
||||
|
||||
Create `editor/src/WorkspaceFileIndex.h`.
|
||||
|
||||
Index all files under a workspace root. For each file, extract top-level
|
||||
declarations (function signatures, class names, struct names, type aliases,
|
||||
constants) using line-level heuristics — no full parse required for this index.
|
||||
|
||||
```cpp
|
||||
struct SymbolLocation {
|
||||
std::string file;
|
||||
int lineStart;
|
||||
int lineEnd; // end of signature or short body
|
||||
std::string symbolName;
|
||||
std::string symbolKind; // "function" | "class" | "struct" | "type" | "const"
|
||||
};
|
||||
|
||||
class WorkspaceFileIndex {
|
||||
public:
|
||||
static WorkspaceFileIndex build(const std::string& workspaceRoot);
|
||||
std::vector<SymbolLocation> find(const std::string& symbolName) const;
|
||||
std::vector<SymbolLocation> findInFile(const std::string& filePath) const;
|
||||
int fileCount() const;
|
||||
int symbolCount() const;
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): empty root returns empty index, single-file workspace indexes it,
|
||||
function signatures found by name, class declarations found by name, struct
|
||||
declarations found by name, findInFile returns all symbols in that file, find()
|
||||
returns all matches across files, symbolCount > 0 after indexing real file,
|
||||
fileCount matches input file count, duplicate symbol names all returned, build()
|
||||
doesn't throw on unreadable file (skips gracefully), kind field is non-empty.
|
||||
|
||||
### Step 675: `ContextSliceAssembler` — extract minimal file sections (12 tests)
|
||||
|
||||
Create `editor/src/ContextSliceAssembler.h`.
|
||||
|
||||
Given a list of `{file, symbol_or_line_range}` requests and a workspace index,
|
||||
extract only the relevant lines from each file. A "slice" is:
|
||||
- For a named symbol: the lines from its declaration through the closing brace/semicolon
|
||||
- For a line range: exactly those lines
|
||||
- For a file with no symbol specified: first N lines (configurable, default 40)
|
||||
|
||||
```cpp
|
||||
struct SliceRequest {
|
||||
std::string filePath;
|
||||
std::string symbolName; // empty = use lineHint or head
|
||||
int lineHint; // 0 = not specified
|
||||
int headLines; // fallback if no symbol match; default 40
|
||||
};
|
||||
|
||||
struct ContextSlice {
|
||||
std::string filePath;
|
||||
std::string symbolName;
|
||||
int lineStart;
|
||||
int lineEnd;
|
||||
std::string content; // the actual extracted lines
|
||||
bool found;
|
||||
};
|
||||
|
||||
class ContextSliceAssembler {
|
||||
public:
|
||||
static std::vector<ContextSlice> assemble(
|
||||
const std::vector<SliceRequest>& requests,
|
||||
const WorkspaceFileIndex& index);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): empty requests returns empty result, named symbol found returns
|
||||
correct lines, unknown symbol returns head fallback, line range returns correct
|
||||
lines, content non-empty on success, found==false when symbol missing, multiple
|
||||
requests all processed, head fallback respects headLines parameter, function body
|
||||
slice includes closing brace, class slice includes closing brace, file path
|
||||
preserved in result, slice content is valid UTF-8 substring of source file.
|
||||
|
||||
### Step 676: `TokenBudgetEnforcer` — trim assembled context to budget (12 tests)
|
||||
|
||||
Create `editor/src/TokenBudgetEnforcer.h`.
|
||||
|
||||
Given a list of `ContextSlice` objects and a max token budget, trim the list
|
||||
to fit. Strategy: priority order (slices earlier in the list are higher priority),
|
||||
each slice is dropped whole if it doesn't fit. Returns the slices that fit plus
|
||||
a report of what was dropped.
|
||||
|
||||
Token estimation: 1 token ≈ 4 characters (conservative approximation, no model
|
||||
dependency). The enforcer does NOT call any model API.
|
||||
|
||||
```cpp
|
||||
struct BudgetReport {
|
||||
int totalSlices;
|
||||
int includedSlices;
|
||||
int droppedSlices;
|
||||
int estimatedTokensUsed;
|
||||
int budgetTokens;
|
||||
bool budgetExceeded; // true if any slice was dropped
|
||||
std::vector<std::string> droppedFiles;
|
||||
};
|
||||
|
||||
class TokenBudgetEnforcer {
|
||||
public:
|
||||
static std::pair<std::vector<ContextSlice>, BudgetReport>
|
||||
enforce(const std::vector<ContextSlice>& slices, int maxTokens);
|
||||
|
||||
static int estimateTokens(const std::string& text);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): empty slices returns empty + zero report, single slice under budget
|
||||
kept, single slice over budget dropped, multiple slices: high-priority kept when
|
||||
budget tight, droppedFiles list matches dropped slices, estimatedTokensUsed <=
|
||||
budgetTokens when no drops, budgetExceeded==false when all fit, budgetExceeded==true
|
||||
when any dropped, estimateTokens("") == 0, estimateTokens("abcd") == 1,
|
||||
report.totalSlices == input count, report.includedSlices + report.droppedSlices
|
||||
== report.totalSlices.
|
||||
|
||||
### Step 677: `whetstone_assemble_context` MCP tool (8 tests)
|
||||
|
||||
Add to `RegisterCodegenTools.h` (or new `RegisterContextTools.h` if file nears 600 lines).
|
||||
|
||||
Input schema:
|
||||
```json
|
||||
{
|
||||
"files": {
|
||||
"type": "array",
|
||||
"description": "List of slice requests.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": { "type": "string", "description": "File path relative to workspace." },
|
||||
"symbol": { "type": "string", "description": "Symbol name to extract (optional)." },
|
||||
"line_hint": { "type": "integer", "description": "Line number hint (optional)." },
|
||||
"head_lines": { "type": "integer", "description": "Head fallback line count (optional, default 40)." }
|
||||
},
|
||||
"required": ["path"]
|
||||
}
|
||||
},
|
||||
"max_tokens": { "type": "integer", "description": "Token budget (default 8000)." }
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"slices": [
|
||||
{ "file": "...", "symbol": "...", "line_start": 10, "line_end": 25, "content": "..." }
|
||||
],
|
||||
"budget_report": {
|
||||
"total_slices": 3,
|
||||
"included_slices": 2,
|
||||
"dropped_slices": 1,
|
||||
"estimated_tokens_used": 1840,
|
||||
"budget_tokens": 2000,
|
||||
"budget_exceeded": true,
|
||||
"dropped_files": ["large_file.cpp"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Wire into MCPServer.h, RegisterOnboardingAndAllTools.h, and tools.json.
|
||||
Tool count 86→87.
|
||||
|
||||
Tests (8): tool registered, empty files array returns success with empty slices,
|
||||
valid file path returns slice, missing path arg returns error, symbol slicing
|
||||
returns content, budget_report present in output, budget_exceeded==false when
|
||||
all slices fit, tool count in tools.json == 87.
|
||||
|
||||
### Step 678: Sprint 43 Integration Summary (8 tests)
|
||||
|
||||
Create `editor/src/Sprint43IntegrationSummary.h`.
|
||||
Record: tool_count_before=86, tool_count_after=87, steps_completed=5 (674-678),
|
||||
files_added=[WorkspaceFileIndex.h, ContextSliceAssembler.h, TokenBudgetEnforcer.h,
|
||||
Sprint43IntegrationSummary.h], success=true.
|
||||
|
||||
Tests (8): struct constructable, workspace_index_builds, slice_assembler_works,
|
||||
budget_enforcer_works, tool_count_before==86, tool_count_after==87,
|
||||
steps_completed==5, success==true.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate (All Steps)
|
||||
|
||||
- Header-only C++ for all new source files (no new .cpp except test files)
|
||||
- 600-line max per file
|
||||
- No new vcpkg dependencies — uses only `std::filesystem` and string operations
|
||||
- `estimateTokens` must not call any external model API
|
||||
- All slicing is done on raw file text — no AST dependency (fast, portable)
|
||||
- Full sprint regression at step 678
|
||||
225
sprint44_plan.md
Normal file
225
sprint44_plan.md
Normal file
@@ -0,0 +1,225 @@
|
||||
# Sprint 44 Plan: Taskitem Self-Containment Validator
|
||||
|
||||
## Context
|
||||
|
||||
The architect intake pipeline (Sprints 32, 36) produces `AnnotatedTaskitem` objects
|
||||
with `confidence`, `escalate`, `reasons[]`, and `prerequisiteOps[]`. The token
|
||||
reduction claim — that a well-formed taskitem lets an agent execute without reading
|
||||
external context — depends on those fields being accurate and complete.
|
||||
|
||||
Sprint 44 adds a validation layer: given one or more taskitems, score how
|
||||
self-contained they actually are, flag which `prerequisiteOps` don't resolve to
|
||||
real workspace files, and produce a quality report that agents and humans can act on.
|
||||
|
||||
This sprint is entirely generic editor tooling. No coupling to any external system.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Resolve each `prerequisiteOp` string to a real file path in the workspace
|
||||
2. Score each taskitem's self-containment (0–100)
|
||||
3. Produce a batch quality report with actionable improvement suggestions
|
||||
4. Expose validation as `whetstone_validate_taskitem` MCP tool
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 679: `PrerequisiteOpResolver` — resolve ops to workspace paths (12 tests)
|
||||
|
||||
Create `editor/src/PrerequisiteOpResolver.h`.
|
||||
|
||||
A `prerequisiteOp` string is free-form, e.g.:
|
||||
- `"read editor/src/SchemaToCppGenerator.h"`
|
||||
- `"read editor/src/mcp/RegisterASTTools.h lines 30-60"`
|
||||
- `"read editor/CMakeLists.txt"`
|
||||
- `"run cmake --build editor/build-native --target step664_test"`
|
||||
|
||||
Parse each op into a structured `ResolvedOp` and check whether the file referenced
|
||||
(if any) exists under the workspace root.
|
||||
|
||||
```cpp
|
||||
enum class OpKind { ReadFile, RunCommand, Unknown };
|
||||
|
||||
struct ResolvedOp {
|
||||
std::string raw; // original string
|
||||
OpKind kind;
|
||||
std::string filePath; // extracted path (empty for non-file ops)
|
||||
bool fileExists; // false if filePath non-empty but not found
|
||||
int lineStart; // 0 if not specified
|
||||
int lineEnd; // 0 if not specified
|
||||
};
|
||||
|
||||
class PrerequisiteOpResolver {
|
||||
public:
|
||||
static ResolvedOp resolve(const std::string& op,
|
||||
const std::string& workspaceRoot);
|
||||
static std::vector<ResolvedOp> resolveAll(
|
||||
const std::vector<std::string>& ops,
|
||||
const std::string& workspaceRoot);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): empty op string → kind==Unknown, "read file.h" → kind==ReadFile,
|
||||
extracted filePath non-empty for read op, fileExists==true when file present,
|
||||
fileExists==false when file absent, line range parsed ("lines 30-60" → 30,60),
|
||||
no line range → lineStart==0 lineEnd==0, "run cmake ..." → kind==RunCommand,
|
||||
RunCommand has empty filePath, resolveAll processes multiple ops, raw field
|
||||
preserved, unknown op format → kind==Unknown gracefully.
|
||||
|
||||
### Step 680: `SelfContainmentScorer` — score a taskitem 0–100 (12 tests)
|
||||
|
||||
Create `editor/src/SelfContainmentScorer.h`.
|
||||
|
||||
Score algorithm (points deducted from 100):
|
||||
- **prerequisiteOps missing or empty:** −30
|
||||
- **Each prerequisiteOp that doesn't resolve to an existing file:** −10 (max −30)
|
||||
- **reasons[] missing or empty:** −20
|
||||
- **reasons[] has < 3 entries:** −10
|
||||
- **confidence not set (0 or missing):** −10
|
||||
- **title empty:** −20
|
||||
- **dependencyTaskIds present but non-empty (has unresolved dependencies):** −5
|
||||
|
||||
Score is clamped to [0, 100].
|
||||
|
||||
```cpp
|
||||
struct ScoredTaskitem {
|
||||
std::string taskId;
|
||||
int score; // 0-100
|
||||
bool selfContained; // score >= 80
|
||||
std::vector<std::string> issues; // human-readable deduction reasons
|
||||
};
|
||||
|
||||
struct TaskitemInput {
|
||||
std::string taskId;
|
||||
std::string title;
|
||||
std::vector<std::string> prerequisiteOps;
|
||||
std::vector<std::string> reasons;
|
||||
int confidence; // 0-100
|
||||
std::vector<std::string> dependencyTaskIds;
|
||||
};
|
||||
|
||||
class SelfContainmentScorer {
|
||||
public:
|
||||
static ScoredTaskitem score(const TaskitemInput& item,
|
||||
const std::string& workspaceRoot);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): fully specified taskitem scores >= 80, empty prerequisiteOps deducts 30,
|
||||
missing reasons deducts 20, empty title deducts 20, unresolved file op deducts 10,
|
||||
two unresolved files deducts 20, score clamped to 0 (not negative), score clamped
|
||||
to 100 (not over), selfContained==true when score>=80, selfContained==false when
|
||||
score<80, issues list non-empty when deductions made, issues empty when perfect score.
|
||||
|
||||
### Step 681: `TaskitemQualityAuditor` — batch report (12 tests)
|
||||
|
||||
Create `editor/src/TaskitemQualityAuditor.h`.
|
||||
|
||||
Given a list of `TaskitemInput` objects and a workspace root, score all of them
|
||||
and produce a summary report with improvement suggestions.
|
||||
|
||||
```cpp
|
||||
struct QualityAuditReport {
|
||||
int totalTaskitems;
|
||||
int selfContainedCount; // score >= 80
|
||||
int warningCount; // score 50-79
|
||||
int failingCount; // score < 50
|
||||
double averageScore;
|
||||
std::vector<ScoredTaskitem> results; // one per input item
|
||||
std::vector<std::string> topIssues; // most common deduction reasons
|
||||
};
|
||||
|
||||
class TaskitemQualityAuditor {
|
||||
public:
|
||||
static QualityAuditReport audit(const std::vector<TaskitemInput>& items,
|
||||
const std::string& workspaceRoot);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): empty input returns valid report with 0 counts, single passing item
|
||||
counted, single failing item counted, averageScore computed correctly, topIssues
|
||||
non-empty when issues exist, selfContainedCount + warningCount + failingCount ==
|
||||
totalTaskitems, results.size() == input.size(), all results have taskId set,
|
||||
mixed batch (some pass some fail) counted correctly, report is deterministic
|
||||
(same input → same output), totalTaskitems matches input size, warningCount correct
|
||||
for 50-79 range.
|
||||
|
||||
### Step 682: `whetstone_validate_taskitem` MCP tool (8 tests)
|
||||
|
||||
Add `whetstone_validate_taskitem` to `RegisterCodegenTools.h` (or a new
|
||||
`RegisterValidationTools.h` if approaching 600 lines).
|
||||
|
||||
Input schema:
|
||||
```json
|
||||
{
|
||||
"taskitems": {
|
||||
"type": "array",
|
||||
"description": "One or more taskitems to validate.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"task_id": { "type": "string" },
|
||||
"title": { "type": "string" },
|
||||
"prerequisite_ops": { "type": "array", "items": { "type": "string" } },
|
||||
"reasons": { "type": "array", "items": { "type": "string" } },
|
||||
"confidence": { "type": "integer" },
|
||||
"dependency_task_ids": { "type": "array", "items": { "type": "string" } }
|
||||
},
|
||||
"required": ["task_id"]
|
||||
}
|
||||
},
|
||||
"workspace": { "type": "string", "description": "Workspace root to resolve file paths against." }
|
||||
}
|
||||
```
|
||||
|
||||
Output:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"report": {
|
||||
"total_taskitems": 3,
|
||||
"self_contained_count": 2,
|
||||
"warning_count": 1,
|
||||
"failing_count": 0,
|
||||
"average_score": 87.3,
|
||||
"results": [
|
||||
{ "task_id": "T1", "score": 95, "self_contained": true, "issues": [] },
|
||||
...
|
||||
],
|
||||
"top_issues": ["prerequisiteOps missing or empty", "reasons[] has fewer than 3 entries"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Wire into MCPServer.h, RegisterOnboardingAndAllTools.h, tools.json (86→88 or 87→88
|
||||
depending on sprint 43 completion).
|
||||
|
||||
Tests (8): tool registered, empty taskitems array returns valid report with 0 counts,
|
||||
single valid taskitem returns score, missing task_id returns error, prerequisite_ops
|
||||
resolved against workspace, report.results.length matches input length,
|
||||
self_contained_count + warning_count + failing_count == total, tool count updated.
|
||||
|
||||
### Step 683: Sprint 44 Integration Summary (8 tests)
|
||||
|
||||
Create `editor/src/Sprint44IntegrationSummary.h`.
|
||||
Record: steps_completed=5 (679-683), files_added=[PrerequisiteOpResolver.h,
|
||||
SelfContainmentScorer.h, TaskitemQualityAuditor.h, Sprint44IntegrationSummary.h],
|
||||
success=true.
|
||||
|
||||
Tests (8): struct constructable, resolver_works, scorer_works, auditor_works,
|
||||
steps_completed==5, files_added contains PrerequisiteOpResolver.h,
|
||||
files_added contains SelfContainmentScorer.h, success==true.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate (All Steps)
|
||||
|
||||
- Header-only C++ for all new source files
|
||||
- 600-line max per file — split at `Extended.h` if approaching
|
||||
- No new vcpkg dependencies — uses only std::filesystem and string operations
|
||||
- Scoring algorithm is deterministic and purely computational (no model calls)
|
||||
- `whetstone_validate_taskitem` must work against any workspace path, not just
|
||||
the current project — no hardcoded paths
|
||||
- Full sprint regression at step 683
|
||||
252
sprint45_plan.md
Normal file
252
sprint45_plan.md
Normal file
@@ -0,0 +1,252 @@
|
||||
# Sprint 45 Plan: Agent Session Metrics + A/B Harness
|
||||
|
||||
## Context
|
||||
|
||||
Two prior questions about Whetstone remain unanswered with real data:
|
||||
|
||||
1. **Does using Whetstone tools actually reduce token consumption?** The 90%+
|
||||
reduction claim is architectural — a well-formed taskitem lets an agent skip
|
||||
file-reading. But the claim has never been measured.
|
||||
|
||||
2. **Is the output quality equivalent or better?** Speed without quality is useless.
|
||||
|
||||
Sprint 45 builds the instrumentation to answer both. It records per-session
|
||||
metrics (tool calls, estimated token cost, files read, duration) and provides
|
||||
a lightweight A/B comparison framework: run the same bounded task with and without
|
||||
Whetstone tools, compare results.
|
||||
|
||||
This is entirely generic editor tooling. The metrics format, the comparison harness,
|
||||
and the MCP tools are not coupled to any external system or project.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Record what an agent does during a session: which tools it called, how much
|
||||
context it consumed (estimated tokens), and how long each call took
|
||||
2. Capture a structured task outcome (what changed, did tests pass)
|
||||
3. Compare two runs of the same task (Whetstone-assisted vs baseline)
|
||||
4. Expose recording control and report retrieval as MCP tools
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 684: `AgentSessionRecorder` — per-session tool call instrumentation (12 tests)
|
||||
|
||||
Create `editor/src/AgentSessionRecorder.h`.
|
||||
|
||||
Records every MCP tool call within a session: tool name, input size (chars),
|
||||
output size (chars), estimated tokens (input+output), duration (ms), and whether
|
||||
it resulted in a file read (heuristic: tool name contains "read", "get_ast",
|
||||
"assemble_context", or "workspace_list").
|
||||
|
||||
```cpp
|
||||
struct ToolCallRecord {
|
||||
std::string toolName;
|
||||
int inputChars;
|
||||
int outputChars;
|
||||
int estimatedTokens; // (inputChars + outputChars) / 4
|
||||
long long durationMs;
|
||||
bool wasFileRead;
|
||||
};
|
||||
|
||||
struct SessionRecord {
|
||||
std::string sessionId;
|
||||
std::string taskDescription;
|
||||
std::vector<ToolCallRecord> calls;
|
||||
int totalToolCalls;
|
||||
int totalEstimatedTokens;
|
||||
int fileReadCount;
|
||||
long long totalDurationMs;
|
||||
};
|
||||
|
||||
class AgentSessionRecorder {
|
||||
public:
|
||||
void start(const std::string& sessionId, const std::string& taskDescription);
|
||||
void record(const ToolCallRecord& call);
|
||||
SessionRecord finish();
|
||||
static ToolCallRecord makeRecord(const std::string& toolName,
|
||||
const std::string& inputJson,
|
||||
const std::string& outputJson,
|
||||
long long durationMs);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): start sets sessionId, record() adds to calls, finish() returns complete
|
||||
record, totalToolCalls matches recorded count, totalEstimatedTokens is sum of call
|
||||
tokens, fileReadCount counts only file-read calls, wasFileRead true for "read"
|
||||
tools, wasFileRead false for non-file tools, estimatedTokens = (inputChars +
|
||||
outputChars) / 4, durationMs preserved, taskDescription preserved in output,
|
||||
empty session returns zero counts.
|
||||
|
||||
### Step 685: `TaskCompletionMetrics` — capture task outcome (12 tests)
|
||||
|
||||
Create `editor/src/TaskCompletionMetrics.h`.
|
||||
|
||||
Captures the observable outcome of a completed task: lines of code changed
|
||||
(before/after diff), whether a provided test command succeeded, and a
|
||||
quality rating (0–100) supplied by the agent or reviewer.
|
||||
|
||||
```cpp
|
||||
struct TaskOutcome {
|
||||
std::string taskId;
|
||||
int linesChanged; // abs(after_lines - before_lines)
|
||||
int linesAdded;
|
||||
int linesRemoved;
|
||||
bool testsProvided;
|
||||
bool testsPassed;
|
||||
int qualityRating; // 0-100, agent-supplied
|
||||
std::string notes;
|
||||
};
|
||||
|
||||
class TaskCompletionMetrics {
|
||||
public:
|
||||
static TaskOutcome diff(const std::string& taskId,
|
||||
const std::string& before,
|
||||
const std::string& after,
|
||||
bool testsPassed,
|
||||
int qualityRating,
|
||||
const std::string& notes = "");
|
||||
static int countLines(const std::string& text);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): empty before and after → 0 changes, added lines counted correctly,
|
||||
removed lines counted correctly, linesChanged == linesAdded + linesRemoved (net),
|
||||
testsPassed preserved, qualityRating clamped to [0,100], notes preserved, taskId
|
||||
preserved, countLines("") == 0, countLines("a\nb\n") == 2, countLines("a\nb") == 2,
|
||||
both before and after empty → testsProvided==false when tests not given.
|
||||
|
||||
### Step 686: `ABTestComparison` — compare two session records (12 tests)
|
||||
|
||||
Create `editor/src/ABTestComparison.h`.
|
||||
|
||||
Given two `SessionRecord` objects (baseline and whetstone-assisted) for the same
|
||||
task, compute the comparison metrics.
|
||||
|
||||
```cpp
|
||||
struct ABComparisonResult {
|
||||
std::string taskId;
|
||||
|
||||
// Token comparison
|
||||
int baselineTokens;
|
||||
int whetstoneTokens;
|
||||
double tokenReductionPct; // (baseline - whetstone) / baseline * 100
|
||||
|
||||
// File read comparison
|
||||
int baselineFileReads;
|
||||
int whetstoneFileReads;
|
||||
double fileReadReductionPct;
|
||||
|
||||
// Duration comparison
|
||||
long long baselineDurationMs;
|
||||
long long whetstoneDurationMs;
|
||||
double durationDeltaPct; // positive = whetstone faster
|
||||
|
||||
// Quality comparison (from TaskOutcome)
|
||||
int baselineQuality;
|
||||
int whetstoneQuality;
|
||||
int qualityDelta; // whetstone - baseline (positive = better)
|
||||
|
||||
bool whetstoneWon; // tokenReductionPct > 0 && qualityDelta >= 0
|
||||
};
|
||||
|
||||
class ABTestComparison {
|
||||
public:
|
||||
static ABComparisonResult compare(
|
||||
const std::string& taskId,
|
||||
const SessionRecord& baseline,
|
||||
const SessionRecord& whetstone,
|
||||
int baselineQuality,
|
||||
int whetstoneQuality);
|
||||
};
|
||||
```
|
||||
|
||||
Tests (12): tokenReductionPct computed correctly, fileReadReductionPct computed
|
||||
correctly, durationDeltaPct computed correctly (positive = faster), qualityDelta =
|
||||
whetstoneQuality - baselineQuality, whetstoneWon==true when tokens reduced and
|
||||
quality >=, whetstoneWon==false when tokens higher, tokenReductionPct==0 when
|
||||
equal tokens, tokenReductionPct negative allowed (whetstone used more tokens),
|
||||
baselineTokens preserved, whetstoneTokens preserved, taskId preserved,
|
||||
zero baseline tokens does not divide by zero (returns 0.0).
|
||||
|
||||
### Step 687: `whetstone_start_recording` + `whetstone_get_metrics` MCP tools (8 tests)
|
||||
|
||||
Add two MCP tools. Wire into MCPServer.h + RegisterOnboardingAndAllTools.h +
|
||||
tools.json.
|
||||
|
||||
**`whetstone_start_recording`** — begins a new recording session.
|
||||
```json
|
||||
{
|
||||
"session_id": { "type": "string", "description": "Unique session identifier." },
|
||||
"task_description": { "type": "string", "description": "What task this session is executing." }
|
||||
}
|
||||
```
|
||||
Response: `{"success": true, "session_id": "..."}`
|
||||
|
||||
**`whetstone_get_metrics`** — retrieves and finalizes the current session record,
|
||||
optionally comparing against a provided baseline JSON.
|
||||
```json
|
||||
{
|
||||
"session_id": { "type": "string" },
|
||||
"baseline": { "type": "object", "description": "Optional prior SessionRecord JSON for A/B comparison." },
|
||||
"quality_rating": { "type": "integer", "description": "Agent's self-assessment 0-100 for this session." },
|
||||
"baseline_quality":{ "type": "integer", "description": "Quality rating for baseline (required if baseline provided)." }
|
||||
}
|
||||
```
|
||||
Response: `{"success": true, "session": {...}, "comparison": {...} }` (comparison
|
||||
only present if baseline was provided).
|
||||
|
||||
MCPServer needs an `AgentSessionRecorder` member to hold in-flight session state.
|
||||
|
||||
Tests (8): start_recording registers tool, get_metrics registers tool, both in
|
||||
tools.json, tool count updated, start_recording returns success=true, get_metrics
|
||||
returns session record, comparison present when baseline provided, comparison absent
|
||||
when no baseline.
|
||||
|
||||
### Step 688: Sprint 45 Integration Summary (8 tests)
|
||||
|
||||
Create `editor/src/Sprint45IntegrationSummary.h`.
|
||||
Record: steps_completed=5 (684-688), files_added=[AgentSessionRecorder.h,
|
||||
TaskCompletionMetrics.h, ABTestComparison.h, Sprint45IntegrationSummary.h],
|
||||
success=true.
|
||||
|
||||
Full sprint regression: all steps 684-688 pass.
|
||||
|
||||
Tests (8): struct constructable, recorder_works, metrics_works, comparison_works,
|
||||
steps_completed==5, files_added contains AgentSessionRecorder.h,
|
||||
files_added contains ABTestComparison.h, success==true.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate (All Steps)
|
||||
|
||||
- Header-only C++ for all new source files
|
||||
- 600-line limit — `AgentSessionRecorder.h` may need Extended split if state
|
||||
management grows; check at step 684
|
||||
- No new vcpkg dependencies
|
||||
- Token estimation must NOT call any model API — purely character-count heuristic
|
||||
- `ABTestComparison::compare` must handle zero-token baseline without crashing
|
||||
- The recording state in MCPServer is per-server-instance (not persisted to disk)
|
||||
- Full sprint regression at step 688
|
||||
|
||||
---
|
||||
|
||||
## Using the A/B Harness
|
||||
|
||||
After Sprint 45, a comparison session looks like this:
|
||||
|
||||
**Baseline run (no Whetstone tools):**
|
||||
1. `whetstone_start_recording` with session_id="baseline-T1"
|
||||
2. Agent executes task using only native file reads (no Whetstone tools)
|
||||
3. `whetstone_get_metrics` with session_id="baseline-T1" → save result as baseline JSON
|
||||
|
||||
**Whetstone-assisted run:**
|
||||
1. `whetstone_start_recording` with session_id="ws-T1"
|
||||
2. Agent executes same task using Whetstone tools (assemble_context, validate_taskitem, etc.)
|
||||
3. `whetstone_get_metrics` with session_id="ws-T1", baseline=<prior result>
|
||||
→ returns comparison with tokenReductionPct, fileReadReductionPct, qualityDelta
|
||||
|
||||
The comparison is self-contained — no external logging service, no network calls.
|
||||
Results can be copied to a markdown file or fed back into architect_intake for review.
|
||||
Reference in New Issue
Block a user