diff --git a/PROGRESS.md b/PROGRESS.md index 4d1428e..9c10308 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -23,6 +23,7 @@ Whetstone is a semantic annotation DSL (SemAnno) and structured editor for cross | Sprint 4 | 76–126 | **Complete** | Professional editor: layout presets, code editor, LSP, annotation UI, terminal, build | | Sprint 5 | 127–165 | **Complete** | Library-aware coding: package registries, constructive coding, Emacs ecosystem, full language coverage, agents | | Sprint 6 | 166–201 | **Complete** | UX & editor polish: structural refactor, themes, multi-cursor, onboarding, security, accessibility, performance | +| Sprint 7 | 202–234 | **In Progress** | MCP Bridge & agent tooling: API docs/schemas, MCP server, synthetic traces, eval harness, model-specific tools | --- @@ -345,6 +346,38 @@ All 38 steps implemented and passing. Each step has a corresponding test (`step1 --- +## Sprint 7: MCP Bridge & Agent Tooling (Steps 202–234) — IN PROGRESS + +### Phase 7a: API Documentation & Schemas (Steps 202–206) — COMPLETE +- [x] Step 202: JSON-RPC API reference document (`docs/AGENT_API.md`): 23 methods documented with schemas, examples, error codes. +- [x] Step 203: JSON Schema definitions (`schemas/`): 8 type schemas + 12 method schemas for request/response validation. +- [x] Step 204: Exposed ContextAPI and BatchMutationAPI via RPC (getInScopeSymbols, getCallHierarchy, getDependencyGraph, applyBatch). +- [x] Step 205: Exposed Pipeline operations via RPC (runPipeline, parseSource, generateFromAST, projectLanguage). +- [x] Step 206: API schema validation tests. 51/51 tests pass. + +### Phase 7b: MCP Server (Steps 207–213) — COMPLETE +- [x] Step 207: MCP server core (`MCPServer.h`): JSON-RPC 2.0 with initialize handshake, protocol version "2024-11-05". +- [x] Step 208: MCP tools — AST query and mutation (5 tools: get_ast, mutate, batch_mutate, get_scope, get_call_hierarchy). +- [x] Step 209: MCP tools — annotation and generation (5 tools: suggest_annotations, apply_annotation, generate_code, run_pipeline, project_language). +- [x] Step 210: MCP resources (5 resources: ast, diagnostics, libraries, annotations, settings). +- [x] Step 211: MCP prompts (4 prompts: annotate_module, cross_language_projection, security_audit, refactor_memory). +- [x] Step 212: MCP bridge (`MCPBridge.h`): stdio transport with Content-Length framing, embedded mode support. +- [x] Step 213: MCP server tests. 90/90 tests pass. + +### Phase 7c: Synthetic Trace Generation (Steps 214–219) — COMPLETE +- [x] Step 214: Trace data model (`TraceGenerator.h`): TraceStep (user/assistant/tool_call/tool_result), Trace with metadata. +- [x] Step 215: 6 scenario templates: ReadAndUnderstand, AddAnnotations, CrossLanguage, Refactor, SecurityAudit, MultiStepDebug. +- [x] Step 216: Built-in code corpus: 9 samples across 4 languages (Python, C++, JavaScript, Rust). +- [x] Step 217: Generator engine with deterministic seeding, batch generation, and real Pipeline.parse() integration. +- [x] Step 218: Trace export pipeline (`TraceExporter.h`): Anthropic Messages, OpenAI Chat, JSONL, Markdown formats. Filtering and statistics. +- [x] Step 219: Trace generation tests. 294/294 tests pass. + +### Phase 7d: Evaluation Harness (Steps 220–224) — PENDING +### Phase 7e: Model-Specific Tool Definitions (Steps 225–229) — PENDING +### Phase 7f: Session Recording Pipeline (Steps 230–234) — PENDING + +--- + ## Build Infrastructure Created in the most recent session: @@ -383,13 +416,14 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi ## Test Results (Last Verified) -**All 201 steps compile and pass.** 347+ test executables in `editor/build/Release/`. +**All 219 steps compile and pass.** 350+ test executables in `editor/build/Release/`. **Sprint 2 (Steps 1–38):** All pass. **Sprint 3 (Steps 39–75):** All pass. Highlights: step53 6/6, step54 10/10, step72 6/6, step74 6/6. **Sprint 4 (Steps 76–126):** All pass. Highlights: step76 10/10, step118 11/11, step125 3/3 integration. **Sprint 5 (Steps 127–165):** All pass. Highlights: step144 11/11, step145 11/11, step153 208/208 (cross-language matrix), step161 10/10. **Sprint 6 (Steps 166–201):** All pass. Highlights: step168 79/79 (split + integration), step201 integration tests. +**Sprint 7 (Steps 202–219):** All pass. Highlights: step206 51/51, step213 90/90, step219 294/294. **Architecture test:** `file_limits_test` 4/4 passes (enforces header size limits). --- @@ -499,6 +533,16 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi | `editor/src/FeatureHints.h` | Contextual tip system | | `editor/src/ShortcutReference.h` | Keyboard shortcut browser panel | +### MCP & Training Data (Sprint 7) +| File | Contents | +|------|----------| +| `editor/src/MCPServer.h` | MCP protocol server: 10 tools, 5 resources, 4 prompts, JSON-RPC 2.0 | +| `editor/src/MCPBridge.h` | MCP stdio transport bridge to Whetstone internal RPC | +| `editor/src/TraceGenerator.h` | Synthetic trace generation: 6 scenarios, code corpus, batch engine | +| `editor/src/TraceExporter.h` | Multi-format trace export (Anthropic/OpenAI/JSONL/Markdown), filtering, stats | +| `docs/AGENT_API.md` | Complete JSON-RPC API reference (23 methods) | +| `schemas/` | 20 JSON Schema files (8 types + 12 methods) | + ### Build & Infrastructure | File | Contents | |------|----------| @@ -517,13 +561,14 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi - **Generators**: AST → Python, C++, Elisp, JavaScript/TypeScript, Java, Rust, Go. All generators handle canonical memory annotations with language-appropriate mappings. - **Parsers**: Text → AST via tree-sitter for all 8 supported languages. Full CST-to-AST conversion with memory pattern detection and auto-annotation. - **Library System**: Package registry abstraction (PyPI, npm, crates.io, Maven, Go, vcpkg). PrimitivesRegistry for constructive coding. Vulnerability database (OSV). Semantic library tags. -- **Agent System**: WebSocket server with role-based permissions, library context, annotation assistant, workflow recording, agent marketplace. +- **Agent System**: WebSocket server with role-based permissions, library context, annotation assistant, workflow recording, agent marketplace. MCP server (10 tools, 5 resources, 4 prompts) for LLM integration via stdio transport. +- **Training Data**: Synthetic trace generator (6 scenario types, 4-language code corpus) with multi-format export (Anthropic Messages, OpenAI Chat, JSONL, Markdown). --- ## What's Next -**Sprint 7: MCP Bridge & Synthetic Training Data.** All 6 sprints complete (201 steps). Feature branch `001-core-ast-structure` ready for merge to `main`. Sprint 7 plan: `sprint7_plan.md`. +**Sprint 7 Phase 7d–7f remaining.** Phases 7a–7c complete (Steps 202–219, 435 combined assertions passing). Next: Phase 7d Evaluation Harness (Steps 220–224), Phase 7e Model-Specific Tool Definitions (Steps 225–229), Phase 7f Session Recording Pipeline (Steps 230–234). Full plan: `sprint7_plan.md`. --- @@ -690,4 +735,8 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi | 2026-02-10 | Codex | Step 199: Large file handling (size thresholds, text-mode fallback, large file prompt, highlight disable, memory indicator). 1/1 tests pass (step199_test). | | 2026-02-10 | Codex | Step 200: Startup/perf (LSP & highlight debounce, deferred AST sync on session restore, debug frame budget warnings). 1/1 tests pass (step200_test). | | 2026-02-10 | Codex | Step 201: Sprint 6 integration checks (panel wiring, theme switching, multicursor, first-run wizard, security badge, keyboard navigation, large file settings, notifications). 1/1 tests pass (step201_test). | +| 2026-02-10 | Codex | Step 220: Evaluation framework scaffolding (EvalHarness, task/trace loading, scoring). 1/1 tests pass (step220_test). | | 2026-02-10 | Codex | Step 166: Extract main.cpp into panel headers marked complete (already implemented). step166_test passes; file_limits_test 4/4 passes. | +| 2026-02-10 | Claude Opus 4.6 | Sprint 7 Phase 7a (Steps 202–206): API docs (AGENT_API.md, 23 methods), JSON schemas (20 files), exposed ContextAPI/BatchMutationAPI/Pipeline via RPC, permission policy updates. 51/51 tests pass. | +| 2026-02-10 | Claude Opus 4.6 | Sprint 7 Phase 7b (Steps 207–213): MCPServer.h (10 tools, 5 resources, 4 prompts, JSON-RPC 2.0 initialize handshake), MCPBridge.h (stdio transport). 90/90 tests pass. | +| 2026-02-10 | Claude Opus 4.6 | Sprint 7 Phase 7c (Steps 214–219): TraceGenerator.h (6 scenario templates, 9-sample code corpus, batch engine), TraceExporter.h (Anthropic/OpenAI/JSONL/Markdown export, filtering, statistics). 294/294 tests pass. | diff --git a/editor/CMakeLists.txt b/editor/CMakeLists.txt index efe18a8..cdbedbb 100644 --- a/editor/CMakeLists.txt +++ b/editor/CMakeLists.txt @@ -1143,6 +1143,8 @@ target_include_directories(step200_test PRIVATE src) add_executable(step201_test tests/step201_test.cpp) target_include_directories(step201_test PRIVATE src) +add_executable(step220_test tests/step220_test.cpp) +target_include_directories(step220_test PRIVATE src) add_executable(step213_test tests/step213_test.cpp) target_include_directories(step213_test PRIVATE src) target_link_libraries(step213_test PRIVATE nlohmann_json::nlohmann_json) diff --git a/editor/src/EvalHarness.h b/editor/src/EvalHarness.h new file mode 100644 index 0000000..b9aa5c7 --- /dev/null +++ b/editor/src/EvalHarness.h @@ -0,0 +1,226 @@ +#pragma once +#include +#include +#include +#include +#include +#include +#include +#include +#include "TraceGenerator.h" +#include "ast/Serialization.h" + +using json = nlohmann::json; + +struct EvalTask { + std::string id; + std::string description; + std::string language = "python"; + std::string setupAst; + json expectedOutcome = json::object(); + std::vector tools; + int maxSteps = 8; + bool allowPartial = false; +}; + +struct EvalResult { + std::string taskId; + bool passed = false; + int steps = 0; + int toolCallCount = 0; + std::vector errors; + double durationMs = 0.0; + double score = 0.0; +}; + +class EvalRunner { +public: + static EvalTask taskFromJson(const json& j) { + EvalTask task; + task.id = j.value("id", ""); + task.description = j.value("description", ""); + task.language = j.value("language", "python"); + task.setupAst = j.value("setupAst", ""); + task.expectedOutcome = j.value("expectedOutcome", json::object()); + if (j.contains("tools") && j["tools"].is_array()) { + for (const auto& t : j["tools"]) task.tools.push_back(t.get()); + } + task.maxSteps = j.value("maxSteps", 8); + task.allowPartial = j.value("allowPartial", false); + return task; + } + + static json taskToJson(const EvalTask& task) { + json j; + j["id"] = task.id; + j["description"] = task.description; + j["language"] = task.language; + j["setupAst"] = task.setupAst; + j["expectedOutcome"] = task.expectedOutcome; + j["tools"] = task.tools; + j["maxSteps"] = task.maxSteps; + j["allowPartial"] = task.allowPartial; + return j; + } + + static Trace traceFromJson(const json& j) { + Trace t; + t.id = j.value("id", ""); + t.scenario = j.value("scenario", ""); + t.difficulty = j.value("difficulty", ""); + t.language = j.value("language", ""); + t.toolCallCount = j.value("toolCallCount", 0); + t.success = j.value("success", true); + if (j.contains("toolsUsed") && j["toolsUsed"].is_array()) { + for (const auto& tool : j["toolsUsed"]) t.toolsUsed.push_back(tool.get()); + } + if (j.contains("steps") && j["steps"].is_array()) { + for (const auto& s : j["steps"]) { + TraceStep step; + step.role = s.value("role", ""); + step.content = s.value("content", ""); + step.toolName = s.value("toolName", ""); + if (s.contains("toolInput")) step.toolInput = s["toolInput"]; + if (s.contains("toolOutput")) step.toolOutput = s["toolOutput"]; + t.steps.push_back(std::move(step)); + } + } + return t; + } + + static std::vector loadTasksFromDir(const std::string& dir) { + std::vector tasks; + std::error_code ec; + for (const auto& entry : std::filesystem::directory_iterator(dir, ec)) { + if (ec) break; + if (!entry.is_regular_file()) continue; + if (entry.path().extension() != ".json") continue; + std::ifstream in(entry.path()); + if (!in.is_open()) continue; + json j; + in >> j; + tasks.push_back(taskFromJson(j)); + } + return tasks; + } + + static std::vector loadTracesJsonl(const std::string& path) { + std::vector traces; + std::ifstream in(path); + if (!in.is_open()) return traces; + std::string line; + while (std::getline(in, line)) { + if (line.empty()) continue; + json j = json::parse(line, nullptr, false); + if (j.is_discarded()) continue; + traces.push_back(traceFromJson(j)); + } + return traces; + } + + static EvalResult evaluateTraceForTask(const EvalTask& task, const Trace& trace) { + EvalResult res; + res.taskId = task.id; + res.steps = static_cast(trace.steps.size()); + res.toolCallCount = trace.toolCallCount; + + const auto start = std::chrono::high_resolution_clock::now(); + int expected = 0; + int matched = 0; + + if (task.maxSteps > 0 && res.steps > task.maxSteps) { + res.errors.push_back("Exceeded maxSteps"); + } + + if (task.expectedOutcome.contains("toolCalls")) { + for (const auto& tool : task.expectedOutcome["toolCalls"]) { + expected++; + std::string name = tool.get(); + bool found = false; + for (const auto& step : trace.steps) { + if (step.role == "tool_call" && step.toolName == name) { + found = true; + break; + } + } + if (found) matched++; + else res.errors.push_back("Missing tool call: " + name); + } + } + + if (task.expectedOutcome.contains("mustContain")) { + for (const auto& needle : task.expectedOutcome["mustContain"]) { + expected++; + std::string text = needle.get(); + bool found = false; + for (const auto& step : trace.steps) { + if (step.role == "assistant" && step.content.find(text) != std::string::npos) { + found = true; + break; + } + } + if (found) matched++; + else res.errors.push_back("Missing assistant content: " + text); + } + } + + if (expected == 0) { + res.score = trace.success ? 1.0 : 0.0; + } else { + res.score = static_cast(matched) / static_cast(expected); + } + + res.passed = res.errors.empty(); + if (task.allowPartial && res.score > 0.0) { + res.passed = res.score >= 0.5; + } + + const auto end = std::chrono::high_resolution_clock::now(); + res.durationMs = std::chrono::duration(end - start).count(); + return res; + } + + static json resultToJson(const EvalResult& res) { + json j; + j["taskId"] = res.taskId; + j["passed"] = res.passed; + j["steps"] = res.steps; + j["toolCallCount"] = res.toolCallCount; + j["errors"] = res.errors; + j["durationMs"] = res.durationMs; + j["score"] = res.score; + return j; + } + + static json run(const std::vector& tasks, + const std::vector& traces) { + json report; + json results = json::array(); + int passed = 0; + double totalScore = 0.0; + std::unordered_map traceById; + for (const auto& t : traces) traceById[t.id] = t; + + for (const auto& task : tasks) { + auto it = traceById.find(task.id); + if (it == traceById.end()) { + EvalResult missing; + missing.taskId = task.id; + missing.passed = false; + missing.errors.push_back("Missing trace"); + results.push_back(resultToJson(missing)); + continue; + } + EvalResult r = evaluateTraceForTask(task, it->second); + if (r.passed) passed++; + totalScore += r.score; + results.push_back(resultToJson(r)); + } + + report["totalTasks"] = static_cast(tasks.size()); + report["passed"] = passed; + report["avgScore"] = tasks.empty() ? 0.0 : totalScore / tasks.size(); + report["results"] = results; + return report; + } +}; diff --git a/editor/tests/step220_test.cpp b/editor/tests/step220_test.cpp new file mode 100644 index 0000000..a3039ac --- /dev/null +++ b/editor/tests/step220_test.cpp @@ -0,0 +1,26 @@ +// Step 220: Evaluation framework scaffolding. + +#include +#include +#include + +static std::string readFile(const std::string& path) { + std::ifstream f(path); + if (!f.is_open()) return {}; + return std::string((std::istreambuf_iterator(f)), + std::istreambuf_iterator()); +} + +static void assertContains(const std::string& text, const std::string& needle) { + assert(text.find(needle) != std::string::npos); +} + +int main() { + const std::string harness = readFile("src/EvalHarness.h"); + assertContains(harness, "struct EvalTask"); + assertContains(harness, "struct EvalResult"); + assertContains(harness, "class EvalRunner"); + assertContains(harness, "evaluateTraceForTask"); + printf("step220_test: all assertions passed\n"); + return 0; +} diff --git a/sprint7_plan.md b/sprint7_plan.md index c88732c..94313e1 100644 --- a/sprint7_plan.md +++ b/sprint7_plan.md @@ -292,7 +292,7 @@ the editor. Test suites that measure how accurately an LLM can use Whetstone tools. -- [ ] **Step 220: Evaluation framework** +- [x] **Step 220: Evaluation framework** Create `EvalHarness.h` with the evaluation infrastructure: - `EvalTask` struct: `{id, description, setupAST, expectedOutcome, tools, maxSteps}` - `EvalResult` struct: `{taskId, passed, steps, toolCallCount, errors, duration}`