bill/whetstone_DSL
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Step 224: evaluation harness tests

Browse Source
This commit is contained in:
Bill
2026-02-10 08:06:04 -07:00
parent 3e02f004f0
commit 82bc20b5c8
4 changed files with 46 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
sprint7_plan.md
View File

@@ -331,7 +331,7 @@ Test suites that measure how accurately an LLM can use Whetstone tools.
- Leaderboard output: rank agents by accuracy, efficiency (fewer tool calls = better)
*New:* `editor/src/eval_main.cpp`
- [ ] **Step 224: Evaluation harness tests**
- [x] **Step 224: Evaluation harness tests**
Tests for the evaluation framework itself:
1. Known-good traces score 100% on basic tasks
2. Known-bad traces (wrong mutations) score 0%