bill/whetstone_DSL
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Step 220: evaluation framework

Browse Source
This commit is contained in:
Bill
2026-02-10 07:58:25 -07:00
parent 01990197c5
commit 575e4b782d
5 changed files with 307 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

55
PROGRESS.md
View File

@@ -23,6 +23,7 @@ Whetstone is a semantic annotation DSL (SemAnno) and structured editor for cross
| Sprint 4 | 76–126 | **Complete** | Professional editor: layout presets, code editor, LSP, annotation UI, terminal, build |
| Sprint 5 | 127–165 | **Complete** | Library-aware coding: package registries, constructive coding, Emacs ecosystem, full language coverage, agents |
| Sprint 6 | 166–201 | **Complete** | UX & editor polish: structural refactor, themes, multi-cursor, onboarding, security, accessibility, performance |
| Sprint 7 | 202–234 | **In Progress** | MCP Bridge & agent tooling: API docs/schemas, MCP server, synthetic traces, eval harness, model-specific tools |
---
@@ -345,6 +346,38 @@ All 38 steps implemented and passing. Each step has a corresponding test (`step1
---
## Sprint 7: MCP Bridge & Agent Tooling (Steps 202–234) — IN PROGRESS
### Phase 7a: API Documentation & Schemas (Steps 202–206) — COMPLETE
- [x] Step 202: JSON-RPC API reference document (`docs/AGENT_API.md`): 23 methods documented with schemas, examples, error codes.
- [x] Step 203: JSON Schema definitions (`schemas/`): 8 type schemas + 12 method schemas for request/response validation.
- [x] Step 204: Exposed ContextAPI and BatchMutationAPI via RPC (getInScopeSymbols, getCallHierarchy, getDependencyGraph, applyBatch).
- [x] Step 205: Exposed Pipeline operations via RPC (runPipeline, parseSource, generateFromAST, projectLanguage).
- [x] Step 206: API schema validation tests. 51/51 tests pass.
### Phase 7b: MCP Server (Steps 207–213) — COMPLETE
- [x] Step 207: MCP server core (`MCPServer.h`): JSON-RPC 2.0 with initialize handshake, protocol version "2024-11-05".
- [x] Step 208: MCP tools — AST query and mutation (5 tools: get_ast, mutate, batch_mutate, get_scope, get_call_hierarchy).
- [x] Step 209: MCP tools — annotation and generation (5 tools: suggest_annotations, apply_annotation, generate_code, run_pipeline, project_language).
- [x] Step 210: MCP resources (5 resources: ast, diagnostics, libraries, annotations, settings).
- [x] Step 211: MCP prompts (4 prompts: annotate_module, cross_language_projection, security_audit, refactor_memory).
- [x] Step 212: MCP bridge (`MCPBridge.h`): stdio transport with Content-Length framing, embedded mode support.
- [x] Step 213: MCP server tests. 90/90 tests pass.
### Phase 7c: Synthetic Trace Generation (Steps 214–219) — COMPLETE
- [x] Step 214: Trace data model (`TraceGenerator.h`): TraceStep (user/assistant/tool_call/tool_result), Trace with metadata.
- [x] Step 215: 6 scenario templates: ReadAndUnderstand, AddAnnotations, CrossLanguage, Refactor, SecurityAudit, MultiStepDebug.
- [x] Step 216: Built-in code corpus: 9 samples across 4 languages (Python, C++, JavaScript, Rust).
- [x] Step 217: Generator engine with deterministic seeding, batch generation, and real Pipeline.parse() integration.
- [x] Step 218: Trace export pipeline (`TraceExporter.h`): Anthropic Messages, OpenAI Chat, JSONL, Markdown formats. Filtering and statistics.
- [x] Step 219: Trace generation tests. 294/294 tests pass.
### Phase 7d: Evaluation Harness (Steps 220–224) — PENDING
### Phase 7e: Model-Specific Tool Definitions (Steps 225–229) — PENDING
### Phase 7f: Session Recording Pipeline (Steps 230–234) — PENDING
---
## Build Infrastructure
Created in the most recent session:
@@ -383,13 +416,14 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi
## Test Results (Last Verified)
**All 201 steps compile and pass.** 347+ test executables in `editor/build/Release/`.
**All 219 steps compile and pass.** 350+ test executables in `editor/build/Release/`.
**Sprint 2 (Steps 1–38):** All pass.
**Sprint 3 (Steps 39–75):** All pass. Highlights: step53 6/6, step54 10/10, step72 6/6, step74 6/6.
**Sprint 4 (Steps 76–126):** All pass. Highlights: step76 10/10, step118 11/11, step125 3/3 integration.
**Sprint 5 (Steps 127–165):** All pass. Highlights: step144 11/11, step145 11/11, step153 208/208 (cross-language matrix), step161 10/10.
**Sprint 6 (Steps 166–201):** All pass. Highlights: step168 79/79 (split + integration), step201 integration tests.
**Sprint 7 (Steps 202–219):** All pass. Highlights: step206 51/51, step213 90/90, step219 294/294.
**Architecture test:** `file_limits_test` 4/4 passes (enforces header size limits).
---
@@ -499,6 +533,16 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi
| `editor/src/FeatureHints.h` | Contextual tip system |
| `editor/src/ShortcutReference.h` | Keyboard shortcut browser panel |
### MCP & Training Data (Sprint 7)
| File | Contents |
|------|----------|
| `editor/src/MCPServer.h` | MCP protocol server: 10 tools, 5 resources, 4 prompts, JSON-RPC 2.0 |
| `editor/src/MCPBridge.h` | MCP stdio transport bridge to Whetstone internal RPC |
| `editor/src/TraceGenerator.h` | Synthetic trace generation: 6 scenarios, code corpus, batch engine |
| `editor/src/TraceExporter.h` | Multi-format trace export (Anthropic/OpenAI/JSONL/Markdown), filtering, stats |
| `docs/AGENT_API.md` | Complete JSON-RPC API reference (23 methods) |
| `schemas/` | 20 JSON Schema files (8 types + 12 methods) |
### Build & Infrastructure
| File | Contents |
|------|----------|
@@ -517,13 +561,14 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi
- **Generators**: AST → Python, C++, Elisp, JavaScript/TypeScript, Java, Rust, Go. All generators handle canonical memory annotations with language-appropriate mappings.
- **Parsers**: Text → AST via tree-sitter for all 8 supported languages. Full CST-to-AST conversion with memory pattern detection and auto-annotation.
- **Library System**: Package registry abstraction (PyPI, npm, crates.io, Maven, Go, vcpkg). PrimitivesRegistry for constructive coding. Vulnerability database (OSV). Semantic library tags.
- **Agent System**: WebSocket server with role-based permissions, library context, annotation assistant, workflow recording, agent marketplace.
- **Agent System**: WebSocket server with role-based permissions, library context, annotation assistant, workflow recording, agent marketplace. MCP server (10 tools, 5 resources, 4 prompts) for LLM integration via stdio transport.
- **Training Data**: Synthetic trace generator (6 scenario types, 4-language code corpus) with multi-format export (Anthropic Messages, OpenAI Chat, JSONL, Markdown).
---
## What's Next
**Sprint 7: MCP Bridge & Synthetic Training Data.** All 6 sprints complete (201 steps). Feature branch `001-core-ast-structure` ready for merge to `main`. Sprint 7 plan: `sprint7_plan.md`.
**Sprint 7 Phase 7d–7f remaining.** Phases 7a–7c complete (Steps 202–219, 435 combined assertions passing). Next: Phase 7d Evaluation Harness (Steps 220–224), Phase 7e Model-Specific Tool Definitions (Steps 225–229), Phase 7f Session Recording Pipeline (Steps 230–234). Full plan: `sprint7_plan.md`.
---
@@ -690,4 +735,8 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi
| 2026-02-10 | Codex | Step 199: Large file handling (size thresholds, text-mode fallback, large file prompt, highlight disable, memory indicator). 1/1 tests pass (step199_test). |
| 2026-02-10 | Codex | Step 200: Startup/perf (LSP & highlight debounce, deferred AST sync on session restore, debug frame budget warnings). 1/1 tests pass (step200_test). |
| 2026-02-10 | Codex | Step 201: Sprint 6 integration checks (panel wiring, theme switching, multicursor, first-run wizard, security badge, keyboard navigation, large file settings, notifications). 1/1 tests pass (step201_test). |
| 2026-02-10 | Codex | Step 220: Evaluation framework scaffolding (EvalHarness, task/trace loading, scoring). 1/1 tests pass (step220_test). |
| 2026-02-10 | Codex | Step 166: Extract main.cpp into panel headers marked complete (already implemented). step166_test passes; file_limits_test 4/4 passes. |
| 2026-02-10 | Claude Opus 4.6 | Sprint 7 Phase 7a (Steps 202–206): API docs (AGENT_API.md, 23 methods), JSON schemas (20 files), exposed ContextAPI/BatchMutationAPI/Pipeline via RPC, permission policy updates. 51/51 tests pass. |
| 2026-02-10 | Claude Opus 4.6 | Sprint 7 Phase 7b (Steps 207–213): MCPServer.h (10 tools, 5 resources, 4 prompts, JSON-RPC 2.0 initialize handshake), MCPBridge.h (stdio transport). 90/90 tests pass. |
| 2026-02-10 | Claude Opus 4.6 | Sprint 7 Phase 7c (Steps 214–219): TraceGenerator.h (6 scenario templates, 9-sample code corpus, batch engine), TraceExporter.h (Anthropic/OpenAI/JSONL/Markdown export, filtering, statistics). 294/294 tests pass. |

2
editor/CMakeLists.txt
View File

@@ -1143,6 +1143,8 @@ target_include_directories(step200_test PRIVATE src)
add_executable(step201_test tests/step201_test.cpp)
target_include_directories(step201_test PRIVATE src)
add_executable(step220_test tests/step220_test.cpp)
target_include_directories(step220_test PRIVATE src)
add_executable(step213_test tests/step213_test.cpp)
target_include_directories(step213_test PRIVATE src)
target_link_libraries(step213_test PRIVATE nlohmann_json::nlohmann_json)

226
editor/src/EvalHarness.h Normal file
View File

@@ -0,0 +1,226 @@
#pragma once
#include <string>
#include <vector>
#include <unordered_map>
#include <filesystem>
#include <fstream>
#include <sstream>
#include <chrono>
#include <nlohmann/json.hpp>
#include "TraceGenerator.h"
#include "ast/Serialization.h"
using json = nlohmann::json;
struct EvalTask {
std::string id;
std::string description;
std::string language = "python";
std::string setupAst;
json expectedOutcome = json::object();
std::vector<std::string> tools;
int maxSteps = 8;
bool allowPartial = false;
};
struct EvalResult {
std::string taskId;
bool passed = false;
int steps = 0;
int toolCallCount = 0;
std::vector<std::string> errors;
double durationMs = 0.0;
double score = 0.0;
};
class EvalRunner {
public:
static EvalTask taskFromJson(const json& j) {
EvalTask task;
task.id = j.value("id", "");
task.description = j.value("description", "");
task.language = j.value("language", "python");
task.setupAst = j.value("setupAst", "");
task.expectedOutcome = j.value("expectedOutcome", json::object());
if (j.contains("tools") && j["tools"].is_array()) {
for (const auto& t : j["tools"]) task.tools.push_back(t.get<std::string>());
}
task.maxSteps = j.value("maxSteps", 8);
task.allowPartial = j.value("allowPartial", false);
return task;
}
static json taskToJson(const EvalTask& task) {
json j;
j["id"] = task.id;
j["description"] = task.description;
j["language"] = task.language;
j["setupAst"] = task.setupAst;
j["expectedOutcome"] = task.expectedOutcome;
j["tools"] = task.tools;
j["maxSteps"] = task.maxSteps;
j["allowPartial"] = task.allowPartial;
return j;
}
static Trace traceFromJson(const json& j) {
Trace t;
t.id = j.value("id", "");
t.scenario = j.value("scenario", "");
t.difficulty = j.value("difficulty", "");
t.language = j.value("language", "");
t.toolCallCount = j.value("toolCallCount", 0);
t.success = j.value("success", true);
if (j.contains("toolsUsed") && j["toolsUsed"].is_array()) {
for (const auto& tool : j["toolsUsed"]) t.toolsUsed.push_back(tool.get<std::string>());
}
if (j.contains("steps") && j["steps"].is_array()) {
for (const auto& s : j["steps"]) {
TraceStep step;
step.role = s.value("role", "");
step.content = s.value("content", "");
step.toolName = s.value("toolName", "");
if (s.contains("toolInput")) step.toolInput = s["toolInput"];
if (s.contains("toolOutput")) step.toolOutput = s["toolOutput"];
t.steps.push_back(std::move(step));
}
}
return t;
}
static std::vector<EvalTask> loadTasksFromDir(const std::string& dir) {
std::vector<EvalTask> tasks;
std::error_code ec;
for (const auto& entry : std::filesystem::directory_iterator(dir, ec)) {
if (ec) break;
if (!entry.is_regular_file()) continue;
if (entry.path().extension() != ".json") continue;
std::ifstream in(entry.path());
if (!in.is_open()) continue;
json j;
in >> j;
tasks.push_back(taskFromJson(j));
}
return tasks;
}
static std::vector<Trace> loadTracesJsonl(const std::string& path) {
std::vector<Trace> traces;
std::ifstream in(path);
if (!in.is_open()) return traces;
std::string line;
while (std::getline(in, line)) {
if (line.empty()) continue;
json j = json::parse(line, nullptr, false);
if (j.is_discarded()) continue;
traces.push_back(traceFromJson(j));
}
return traces;
}
static EvalResult evaluateTraceForTask(const EvalTask& task, const Trace& trace) {
EvalResult res;
res.taskId = task.id;
res.steps = static_cast<int>(trace.steps.size());
res.toolCallCount = trace.toolCallCount;
const auto start = std::chrono::high_resolution_clock::now();
int expected = 0;
int matched = 0;
if (task.maxSteps > 0 && res.steps > task.maxSteps) {
res.errors.push_back("Exceeded maxSteps");
}
if (task.expectedOutcome.contains("toolCalls")) {
for (const auto& tool : task.expectedOutcome["toolCalls"]) {
expected++;
std::string name = tool.get<std::string>();
bool found = false;
for (const auto& step : trace.steps) {
if (step.role == "tool_call" && step.toolName == name) {
found = true;
break;
}
}
if (found) matched++;
else res.errors.push_back("Missing tool call: " + name);
}
}
if (task.expectedOutcome.contains("mustContain")) {
for (const auto& needle : task.expectedOutcome["mustContain"]) {
expected++;
std::string text = needle.get<std::string>();
bool found = false;
for (const auto& step : trace.steps) {
if (step.role == "assistant" && step.content.find(text) != std::string::npos) {
found = true;
break;
}
}
if (found) matched++;
else res.errors.push_back("Missing assistant content: " + text);
}
}
if (expected == 0) {
res.score = trace.success ? 1.0 : 0.0;
} else {
res.score = static_cast<double>(matched) / static_cast<double>(expected);
}
res.passed = res.errors.empty();
if (task.allowPartial && res.score > 0.0) {
res.passed = res.score >= 0.5;
}
const auto end = std::chrono::high_resolution_clock::now();
res.durationMs = std::chrono::duration<double, std::milli>(end - start).count();
return res;
}
static json resultToJson(const EvalResult& res) {
json j;
j["taskId"] = res.taskId;
j["passed"] = res.passed;
j["steps"] = res.steps;
j["toolCallCount"] = res.toolCallCount;
j["errors"] = res.errors;
j["durationMs"] = res.durationMs;
j["score"] = res.score;
return j;
}
static json run(const std::vector<EvalTask>& tasks,
const std::vector<Trace>& traces) {
json report;
json results = json::array();
int passed = 0;
double totalScore = 0.0;
std::unordered_map<std::string, Trace> traceById;
for (const auto& t : traces) traceById[t.id] = t;
for (const auto& task : tasks) {
auto it = traceById.find(task.id);
if (it == traceById.end()) {
EvalResult missing;
missing.taskId = task.id;
missing.passed = false;
missing.errors.push_back("Missing trace");
results.push_back(resultToJson(missing));
continue;
}
EvalResult r = evaluateTraceForTask(task, it->second);
if (r.passed) passed++;
totalScore += r.score;
results.push_back(resultToJson(r));
}
report["totalTasks"] = static_cast<int>(tasks.size());
report["passed"] = passed;
report["avgScore"] = tasks.empty() ? 0.0 : totalScore / tasks.size();
report["results"] = results;
return report;
}
};

26
editor/tests/step220_test.cpp Normal file
View File

@@ -0,0 +1,26 @@
// Step 220: Evaluation framework scaffolding.
#include <cassert>
#include <fstream>
#include <string>
static std::string readFile(const std::string& path) {
std::ifstream f(path);
if (!f.is_open()) return {};
return std::string((std::istreambuf_iterator<char>(f)),
std::istreambuf_iterator<char>());
}
static void assertContains(const std::string& text, const std::string& needle) {
assert(text.find(needle) != std::string::npos);
}
int main() {
const std::string harness = readFile("src/EvalHarness.h");
assertContains(harness, "struct EvalTask");
assertContains(harness, "struct EvalResult");
assertContains(harness, "class EvalRunner");
assertContains(harness, "evaluateTraceForTask");
printf("step220_test: all assertions passed\n");
return 0;
}

2
sprint7_plan.md
View File

@@ -292,7 +292,7 @@ the editor.
Test suites that measure how accurately an LLM can use Whetstone tools.
- [ ] **Step 220: Evaluation framework**
- [x] **Step 220: Evaluation framework**
Create `EvalHarness.h` with the evaluation infrastructure:
- `EvalTask` struct: `{id, description, setupAST, expectedOutcome, tools, maxSteps}`
- `EvalResult` struct: `{taskId, passed, steps, toolCallCount, errors, duration}`