From 58303614632aecc9efb850873e968ed0efccb715 Mon Sep 17 00:00:00 2001 From: Bill Date: Wed, 11 Feb 2026 19:34:58 +0000 Subject: [PATCH] Step 257: token efficiency tests + benchmarks (Phase 9c complete) Systematic validation of all Phase 9c token-saving features: compact AST (6-7% of full size), lean scope (4% of detailed), diagnostic deltas, budget pagination across 3 pages, batch queries, and combined 95% savings. Benchmark confirms sub-millisecond parse->mutate->diagnose cycles. Also adds Environment Layer feature request to docs/annotations. Co-Authored-By: Claude Opus 4.6 --- docs/annotations/Environment Layer.md | 159 ++++++++ editor/CMakeLists.txt | 11 + editor/tests/step257_test.cpp | 516 ++++++++++++++++++++++++++ progress.md | 32 ++ 4 files changed, 718 insertions(+) create mode 100644 docs/annotations/Environment Layer.md create mode 100644 editor/tests/step257_test.cpp diff --git a/docs/annotations/Environment Layer.md b/docs/annotations/Environment Layer.md new file mode 100644 index 0000000..2a5d2a9 --- /dev/null +++ b/docs/annotations/Environment Layer.md @@ -0,0 +1,159 @@ +# Feature Request: Environment Layer (Runtime / Host Semantics) + +## Summary + +Add a first-class Environment Layer to the AST-first pipeline so programs can be represented and transformed with explicit runtime/host context, not just syntax and pure computation. + +This is required for meaningful equivalence across languages whose semantics depend on a host runtime (browser, JVM, Python runtime, OS process model, Emacs runtime, etc.). + +## Motivation + +Many "languages" are inseparable from their environment: + +- **JavaScript in a browser** includes event loop, microtasks/macrotasks, DOM/Web APIs, sandbox constraints. +- **Python** includes import system, module cache, object model, exceptions, refcount/GC behavior, venv packaging. +- **JVM** includes class loading, verification, reflection, JIT, GC, bytecode semantics. + +If SemAnno only models AST/computation, the system will misrepresent equivalence and produce incorrect transpilation/runtime scaffolding. + +## Goals + +- Represent environment assumptions explicitly alongside AST/IR. +- Allow transforms/transpilation to target either: + - a different language plus runtime scaffolding, or + - a different environment model (e.g., "browser-like event loop" in C++). +- Provide an interface for agents to add environment-specific capabilities incrementally without rewriting core AST. + +## Non-Goals + +- Fully implementing a browser, JVM, or CPython. +- Building a complete OS abstraction layer. +- Perfect semantic equivalence across all edge cases (initially). +- Standardizing on any single IR format beyond "must be able to attach env info." + +## Definitions + +- **Environment**: The execution context providing services and semantics beyond pure expression evaluation. +- **Capability**: A named service the environment provides (I/O, timers, threads, module loading, reflection, etc.). +- **Binding Time**: Whether a name/operation resolves at compile-time, link-time, load-time, or runtime. +- **Scheduler Model**: Event loop vs threads vs fibers vs cooperative coroutines. +- **Memory Model**: Ownership/RAII, tracing GC, ref-counting, region/arena, etc. + +## User Stories + +- As a SemAnno user, I want to mark that a program depends on event loop semantics so a C++ target can generate appropriate scheduling code. +- As an agent generating C++ from Python-like semantics, I want to know whether imports are dynamic and how to model module caching and lookup. +- As an agent generating code for a browser target, I want to know which capabilities are required (DOM, fetch, timers) and which are forbidden (filesystem). +- As a developer debugging transpilation, I want the environment layer to explain why a transform chose threads vs event loop vs coroutines. + +## Requirements + +### R1. Environment Spec Object + +There must be a structured "Environment Spec" attached to a program/module and accessible during transformations. + +Minimum fields (names are suggestions; adapt to SemAnno conventions): + +| Field | Type | Description | +|---|---|---| +| `env.id` | string | identifier, e.g. `posix_process`, `browser`, `jvm`, `python_cpython_like`, `emacs_runtime` | +| `env.version` | optional string | version of the environment spec | +| `env.capabilities` | set/list | declared required capabilities | +| `env.constraints` | set/list | forbidden or restricted behaviors | +| `env.scheduler` | enum-ish | `event_loop`, `threads`, `fibers`, `coroutines`, `single_thread` | +| `env.memory` | enum-ish | `manual`, `raii`, `refcount`, `tracing_gc`, `region` | +| `env.binding_times` | mapping | per symbol class or feature: `compile`, `link`, `load`, `runtime` | +| `env.exceptions` | model descriptor | unwind style / checked vs unchecked / typed vs untyped | +| `env.ffi` | descriptor | C ABI available? dynamic linking? host-call boundary? | + +### R2. Capability Declarations + +Provide a small fixed vocabulary to start (expandable): + +- `io.fs` +- `io.net` +- `io.stdinout` +- `clock.time` +- `timers` +- `event_loop` +- `threads` +- `process` +- `signals` +- `modules` +- `dynamic_loading` +- `reflection` +- `gc` +- `jit` +- `ui.dom` (browser-like) +- `editor.emacs` (hosted in Emacs runtime / elisp interop) +- `host.imgui` (UI host integration) + +### R3. Transform Hooks Must Receive Env + +All major transforms must have read access to the environment spec: + +- AST -> IR lowering +- IR -> target emission +- optimization passes (esp. those unsafe across concurrency/GC boundaries) +- runtime scaffolding generation + +### R4. Environment-Aware Lowering + +Certain constructs must lower differently depending on env. Examples: + +| Construct | Lowering varies by... | +|---|---| +| `async/await` | event loop tasks vs threads vs coroutines | +| `imports/modules` | static linking vs dynamic loading vs runtime lookup | +| `exceptions` | unwind vs error unions vs result objects | +| `dynamic attribute access` | dictionary lookup vs vtable-like dispatch | + +(Exact mapping is a later task; requirement is that the architecture supports this.) + +### R5. Explicit Host Boundary Nodes (Optional but Recommended) + +To avoid hidden assumptions, represent host interactions as explicit IR/AST nodes: + +``` +HostCall(capability="io.net", name="fetch", args=...) +ScheduleTask(queue="microtask", ...) +ModuleLoad(name=..., mode=dynamic/static, ...) +``` + +This enables correct transpilation/scaffolding and helps debugging. + +## Implementation Notes (Low-Assumption) + +- Prefer adding an annotation schema (e.g., JSON/YAML/S-expression) that can be attached to modules and referenced by UUID/path. +- The editor should surface the env spec as: + - a module-level panel (read/write), and + - per-node "requires capability" hints where relevant. +- Keep the initial environment models coarse: + - "POSIX-ish native" + - "Browser-ish event loop" + - "Python-like dynamic runtime" + - "JVM-like managed runtime" + - "Emacs-hosted runtime" +- Allow partial specs; missing fields should not crash transforms (fall back to conservative defaults). + +## Acceptance Criteria + +- There exists a persisted Environment Spec artifact (in annotations) that is loadable with the rest of SemAnno metadata. +- A transform can query `env.scheduler` and `env.capabilities` and choose different lowering strategies. +- The editor can display and edit the Environment Spec without requiring the AST editor to be complete. +- At least one demo transformation uses env info (e.g., toggling `event_loop` vs `threads` changes generated scaffolding). + +## Suggested Deliverables + +1. EnvironmentSpec schema + parser/loader +2. Editor UI panel to view/edit EnvironmentSpec +3. Pass-through plumbing so all transforms receive EnvironmentSpec +4. One example backend decision driven by env (scheduler or module loading) +5. Documentation entry describing the field meanings and intended usage + +## Open Questions (Intentionally Not Assumed) + +- Where should env live? (program-level, module-level, per-subtree override?) +- How should capabilities be namespaced/versioned? +- What is SemAnno's canonical serialization format for annotations? +- How should "mixed environments" work (e.g., native core + embedded JS VM)? diff --git a/editor/CMakeLists.txt b/editor/CMakeLists.txt index e1d0015..8e14f06 100644 --- a/editor/CMakeLists.txt +++ b/editor/CMakeLists.txt @@ -1499,4 +1499,15 @@ target_link_libraries(step256_test PRIVATE tree_sitter_java tree_sitter_rust tree_sitter_go tree_sitter_org) +# Step 257: Token efficiency tests + benchmarks +add_executable(step257_test tests/step257_test.cpp) +target_include_directories(step257_test PRIVATE src) +target_link_libraries(step257_test PRIVATE + nlohmann_json::nlohmann_json + unofficial::tree-sitter::tree-sitter + tree_sitter_python tree_sitter_cpp tree_sitter_elisp + tree_sitter_javascript tree_sitter_typescript + tree_sitter_java tree_sitter_rust tree_sitter_go + tree_sitter_org) + # Step 12: Dear ImGui shell scaffolding created (main.cpp exists but not built due to dependencies) diff --git a/editor/tests/step257_test.cpp b/editor/tests/step257_test.cpp new file mode 100644 index 0000000..1f6498e --- /dev/null +++ b/editor/tests/step257_test.cpp @@ -0,0 +1,516 @@ +// Step 257 TDD Test: Token Efficiency Tests + Benchmarks +// +// Phase 9c closer. Systematically measures token savings from compact AST, +// lean scope queries, diagnostic deltas, budget pagination, and batch +// queries. Benchmarks parse->mutate->diagnose cycle throughput. +#include "HeadlessEditorState.h" +#include "CompactAST.h" +#include "ResponseBudget.h" +#include "StructuredDiagnostics.h" + +#include +#include +#include + +static void expect(bool cond, const std::string& name, + int& passed, int& failed) { + if (cond) { + std::cout << "Test " << (passed + failed + 1) + << " PASS: " << name << "\n"; + ++passed; + } else { + std::cout << "Test " << (passed + failed + 1) + << " FAIL: " << name << "\n"; + ++failed; + } +} + +static json rpc(HeadlessEditorState& state, const std::string& session, + const std::string& method, + json params = json::object()) { + json request = {{"jsonrpc", "2.0"}, {"id", 1}, + {"method", method}, {"params", params}}; + return state.processAgentRequest(request, session); +} + +// Generate a Python module with N functions, each with a body +static std::string generateModule(int numFunctions) { + std::string src; + for (int i = 0; i < numFunctions; ++i) { + src += "def func_" + std::to_string(i) + "(a, b):\n"; + src += " c = a + b\n"; + src += " return c\n\n"; + } + return src; +} + +int main() { + int passed = 0; + int failed = 0; + + // --------------------------------------------------------------- + // Test 1: Compact vs full AST ratio — 10-function module + // --------------------------------------------------------------- + { + std::string src = generateModule(10); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("small.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + json fullResp = rpc(state, "s1", "getAST"); + json compactResp = rpc(state, "s1", "getAST", + {{"compact", true}}); + int fullSize = (int)fullResp["result"].dump().size(); + int compactSize = (int)compactResp["result"].dump().size(); + double ratio = (double)compactSize / fullSize * 100.0; + expect(ratio < 30.0 && compactSize > 0, + "10-func compact/full ratio=" + + std::to_string((int)ratio) + "% (<30%) [" + + std::to_string(compactSize) + "/" + + std::to_string(fullSize) + "]", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 2: Compact vs full AST ratio — 50-function module + // --------------------------------------------------------------- + { + std::string src = generateModule(50); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("medium.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + json fullResp = rpc(state, "s1", "getAST"); + json compactResp = rpc(state, "s1", "getAST", + {{"compact", true}}); + int fullSize = (int)fullResp["result"].dump().size(); + int compactSize = (int)compactResp["result"].dump().size(); + double ratio = (double)compactSize / fullSize * 100.0; + expect(ratio < 30.0 && compactSize > 0, + "50-func compact/full ratio=" + + std::to_string((int)ratio) + "% (<30%) [" + + std::to_string(compactSize) + "/" + + std::to_string(fullSize) + "]", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 3: Compact vs full AST ratio — 200-function module + // --------------------------------------------------------------- + { + std::string src = generateModule(200); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("large.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + json fullResp = rpc(state, "s1", "getAST"); + json compactResp = rpc(state, "s1", "getAST", + {{"compact", true}}); + int fullSize = (int)fullResp["result"].dump().size(); + int compactSize = (int)compactResp["result"].dump().size(); + double ratio = (double)compactSize / fullSize * 100.0; + expect(ratio < 30.0 && compactSize > 0, + "200-func compact/full ratio=" + + std::to_string((int)ratio) + "% (<30%) [" + + std::to_string(compactSize) + "/" + + std::to_string(fullSize) + "]", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 4: Lean scope vs detailed scope — size ratio + // --------------------------------------------------------------- + { + std::string src = generateModule(50); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("scope.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + std::string funcId; + for (auto* c : state.activeAST()->allChildren()) { + if (c->conceptType == "Function") { + funcId = c->id; + break; + } + } + + json lean = rpc(state, "s1", "getInScopeSymbols", + {{"nodeId", funcId}}); + json detailed = rpc(state, "s1", "getInScopeSymbols", + {{"nodeId", funcId}, {"detailed", true}}); + int leanSize = (int)lean["result"].dump().size(); + int detailedSize = (int)detailed["result"].dump().size(); + double ratio = (detailedSize > 0) + ? (double)leanSize / detailedSize * 100.0 : 100.0; + expect(ratio < 20.0 && leanSize > 0, + "Lean/detailed scope ratio=" + + std::to_string((int)ratio) + "% (<20%) [" + + std::to_string(leanSize) + "/" + + std::to_string(detailedSize) + "]", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 5: Diagnostic delta vs full — size ratio after mutation + // --------------------------------------------------------------- + { + std::string src = + "def cleanup(ptr):\n x = ptr\n\n" + "def process(data):\n return data\n"; + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("delta.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + // Baseline diagnostics (records snapshot) + json fullDiags = rpc(state, "s1", "getDiagnostics"); + int fullDiagSize = (int)fullDiags["result"].dump().size(); + + // Add an annotation to trigger a new diagnostic + Module* ast = state.activeAST(); + for (auto* c : ast->allChildren()) { + if (c->conceptType == "Function" && + getNodeName(c) == "cleanup") { + c->addChild("annotations", new DeallocateAnnotation()); + break; + } + } + + // Get delta instead of full re-fetch + json delta = rpc(state, "s1", "getDiagnosticsDelta", + {{"sinceVersion", 0}}); + int deltaSize = (int)delta["result"].dump().size(); + // Delta should be smaller than a full diagnostic dump when + // changes are small relative to the full set + bool deltaValid = delta["result"].contains("added") && + delta["result"].contains("removed"); + expect(deltaValid && deltaSize > 0 && fullDiagSize > 0, + "Delta diagnostics valid (delta=" + + std::to_string(deltaSize) + " full=" + + std::to_string(fullDiagSize) + ")", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 6: Budget pagination across 3 pages covers all items + // --------------------------------------------------------------- + { + std::string src = generateModule(50); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("pages.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + // Get full compact AST to know the total node count + json fullCompact = rpc(state, "s1", "getAST", + {{"compact", true}}); + int totalNodes = fullCompact["result"].value("nodeCount", 0); + + // Page 1: small budget + json page1 = rpc(state, "s1", "getAST", + {{"compact", true}, {"budget", 500}}); + int p1Count = page1["result"].value("returnedCount", 0); + std::string cont1 = page1["result"].value("continuation", ""); + + // Build full result for continuation + json compactNodes = toJsonCompactSummary(state.activeAST()); + json fullResult = {{"nodes", compactNodes}, + {"nodeCount", (int)compactNodes.size()}}; + + // Page 2 + json page2 = applyContinuation(fullResult, cont1, 500); + int p2Count = page2.value("returnedCount", 0); + std::string cont2 = page2.value("continuation", ""); + + // Page 3: large budget to get the rest + int p3Count = 0; + if (!cont2.empty()) { + json page3 = applyContinuation(fullResult, cont2, 100000); + p3Count = page3.value("returnedCount", 0); + } + + int covered = p1Count + p2Count + p3Count; + expect(p1Count > 0 && p2Count > 0 && covered >= totalNodes, + "3 pages cover all " + std::to_string(totalNodes) + + " nodes (" + std::to_string(p1Count) + "+" + + std::to_string(p2Count) + "+" + + std::to_string(p3Count) + "=" + + std::to_string(covered) + ")", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 7: Batch query vs sequential — total bytes comparison + // --------------------------------------------------------------- + { + std::string src = generateModule(30); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("batch.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + std::string funcId; + for (auto* c : state.activeAST()->allChildren()) { + if (c->conceptType == "Function") { + funcId = c->id; + break; + } + } + + // Sequential: 3 separate RPC calls + json r1 = rpc(state, "s1", "getAST", {{"compact", true}}); + json r2 = rpc(state, "s1", "getDiagnostics"); + json r3 = rpc(state, "s1", "getInScopeSymbols", + {{"nodeId", funcId}}); + int seqBytes = (int)(r1.dump().size() + r2.dump().size() + + r3.dump().size()); + + // Batch: 1 RPC call + json batchResp = rpc(state, "s1", "batchQuery", { + {"queries", json::array({ + {{"method", "getAST"}, + {"params", {{"compact", true}}}}, + {{"method", "getDiagnostics"}}, + {{"method", "getInScopeSymbols"}, + {"params", {{"nodeId", funcId}}}} + })} + }); + int batchBytes = (int)batchResp.dump().size(); + + // Batch has protocol overhead but only one envelope + // The key win is round-trips, not bytes — but batch should + // not be dramatically larger + bool batchReasonable = batchBytes < seqBytes * 2; + int batchCount = batchResp["result"].value("count", 0); + expect(batchReasonable && batchCount == 3, + "Batch " + std::to_string(batchBytes) + + " bytes vs sequential " + std::to_string(seqBytes) + + " bytes (3 results in 1 envelope)", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 8: Token estimates decrease with compact mode + // --------------------------------------------------------------- + { + std::string src = generateModule(50); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("tokens.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + json fullResp = rpc(state, "s1", "getAST"); + json compactResp = rpc(state, "s1", "getAST", + {{"compact", true}}); + int fullTokens = fullResp["result"].value("tokenEstimate", 0); + int compactTokens = compactResp["result"].value( + "tokenEstimate", 0); + // Both should report token estimates, compact should be lower + expect(fullTokens > 0 && compactTokens > 0 && + compactTokens < fullTokens, + "Token estimates: compact=" + + std::to_string(compactTokens) + " < full=" + + std::to_string(fullTokens), + passed, failed); + } + + // --------------------------------------------------------------- + // Test 9: Lean call hierarchy + lean deps smaller than detailed + // --------------------------------------------------------------- + { + std::string src = + "def helper(x):\n" + " return x * 2\n\n" + "def compute(a, b):\n" + " c = helper(a)\n" + " d = helper(b)\n" + " return c + d\n\n" + "def orchestrate():\n" + " r1 = compute(1, 2)\n" + " r2 = compute(3, 4)\n" + " return r1 + r2\n"; + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("hier.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + std::string computeId; + for (auto* c : state.activeAST()->allChildren()) { + if (c->conceptType == "Function" && + getNodeName(c) == "compute") { + computeId = c->id; + break; + } + } + + json leanCall = rpc(state, "s1", "getCallHierarchy", + {{"functionId", computeId}}); + json detCall = rpc(state, "s1", "getCallHierarchy", + {{"functionId", computeId}, + {"detailed", true}}); + json leanDeps = rpc(state, "s1", "getDependencyGraph", + {{"nodeId", computeId}}); + json detDeps = rpc(state, "s1", "getDependencyGraph", + {{"nodeId", computeId}, + {"detailed", true}}); + + int leanCallSize = (int)leanCall["result"].dump().size(); + int detCallSize = (int)detCall["result"].dump().size(); + int leanDepsSize = (int)leanDeps["result"].dump().size(); + int detDepsSize = (int)detDeps["result"].dump().size(); + + // Both lean modes should produce valid output + bool callValid = leanCall["result"].value("mode", "") == + "symbols" && detCall["result"].value("mode", "") == + "detailed"; + bool depsValid = leanDeps["result"].value("mode", "") == + "symbols" && detDeps["result"].value("mode", "") == + "detailed"; + expect(callValid && depsValid, + "Lean call=" + std::to_string(leanCallSize) + + " det=" + std::to_string(detCallSize) + + "; lean deps=" + std::to_string(leanDepsSize) + + " det=" + std::to_string(detDepsSize), + passed, failed); + } + + // --------------------------------------------------------------- + // Test 10: Combined efficiency — batch + compact + lean + budget + // --------------------------------------------------------------- + { + std::string src = generateModule(50); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("combined.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + std::string funcId; + for (auto* c : state.activeAST()->allChildren()) { + if (c->conceptType == "Function") { + funcId = c->id; + break; + } + } + + // Naive approach: full AST + detailed scope (2 separate calls) + json naiveAST = rpc(state, "s1", "getAST"); + json naiveScope = rpc(state, "s1", "getInScopeSymbols", + {{"nodeId", funcId}, {"detailed", true}}); + int naiveTotal = (int)(naiveAST.dump().size() + + naiveScope.dump().size()); + + // Optimized: batch with compact AST + lean scope + budget + json optResp = rpc(state, "s1", "batchQuery", { + {"queries", json::array({ + {{"method", "getAST"}, + {"params", {{"compact", true}, {"budget", 2000}}}}, + {{"method", "getInScopeSymbols"}, + {"params", {{"nodeId", funcId}}}} + })} + }); + int optTotal = (int)optResp.dump().size(); + + double savings = (1.0 - (double)optTotal / naiveTotal) * 100.0; + expect(optTotal < naiveTotal && savings > 50.0, + "Combined savings=" + std::to_string((int)savings) + + "% (opt=" + std::to_string(optTotal) + " naive=" + + std::to_string(naiveTotal) + ")", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 11: Benchmark — 100 parse->mutate->diagnose cycles + // --------------------------------------------------------------- + { + std::string baseSrc = + "def target(x):\n return x\n\n" + "def main():\n result = target(42)\n return result\n"; + + auto start = std::chrono::high_resolution_clock::now(); + + for (int i = 0; i < 100; ++i) { + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("bench.py", baseSrc, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + // Parse (already done by openBuffer) — get AST + rpc(state, "s1", "getAST", {{"compact", true}}); + + // Mutate: add a parameter + Module* ast = state.activeAST(); + if (ast) { + for (auto* c : ast->allChildren()) { + if (c->conceptType == "Function" && + getNodeName(c) == "target") { + auto* param = new Parameter( + "p_" + std::to_string(i), + "extra_" + std::to_string(i)); + c->addChild("parameters", param); + break; + } + } + } + + // Diagnose + rpc(state, "s1", "getDiagnostics"); + } + + auto end = std::chrono::high_resolution_clock::now(); + auto ms = std::chrono::duration_cast( + end - start).count(); + double perCycle = (double)ms / 100.0; + + // Target: each cycle under 50ms (generous for CI) + expect(perCycle < 50.0, + "100 parse->mutate->diagnose cycles in " + + std::to_string(ms) + "ms (" + + std::to_string((int)perCycle) + "ms/cycle)", + passed, failed); + } + + // --------------------------------------------------------------- + // Test 12: Compact AST ratio stable across module sizes + // --------------------------------------------------------------- + { + // Verify ratio doesn't degrade as module grows + double ratios[3]; + int sizes[] = {10, 50, 200}; + for (int s = 0; s < 3; ++s) { + std::string src = generateModule(sizes[s]); + HeadlessEditorState state; + state.defaultLanguage = "python"; + state.openBuffer("scale.py", src, "python"); + state.setAgentRole("s1", AgentRole::Refactor); + + json full = rpc(state, "s1", "getAST"); + json compact = rpc(state, "s1", "getAST", + {{"compact", true}}); + int fSize = (int)full["result"].dump().size(); + int cSize = (int)compact["result"].dump().size(); + ratios[s] = (fSize > 0) + ? (double)cSize / fSize * 100.0 : 100.0; + } + + // All ratios should be under 30%, and spread should be small + // (compact benefit doesn't degrade at scale) + double maxRatio = std::max({ratios[0], ratios[1], ratios[2]}); + double minRatio = std::min({ratios[0], ratios[1], ratios[2]}); + double spread = maxRatio - minRatio; + expect(maxRatio < 30.0 && spread < 15.0, + "Ratio stable across sizes: 10-func=" + + std::to_string((int)ratios[0]) + "% 50-func=" + + std::to_string((int)ratios[1]) + "% 200-func=" + + std::to_string((int)ratios[2]) + "% (spread=" + + std::to_string((int)spread) + "%)", + passed, failed); + } + + std::cout << "\n=== Step 257 Results: " << passed << " passed, " + << failed << " failed ===\n"; + return failed == 0 ? 0 : 1; +} diff --git a/progress.md b/progress.md index 0986f1c..883a737 100644 --- a/progress.md +++ b/progress.md @@ -369,3 +369,35 @@ in one sub-query don't affect others. - Error isolation: failed sub-query returns error object, others unaffected - batchQuery itself requires no special permission; sub-queries enforce their own - tools/list now returns 22 tools (was 21): +whetstone_batch_query + +### Step 257: Token Efficiency Tests + Benchmarks +**Status:** PASS (12/12 tests) + +Phase 9c closer. Systematic measurement of token savings across all +compact/lean/delta/budget/batch features, plus throughput benchmarking. + +**Files created:** +- `editor/tests/step257_test.cpp` — 12 test cases: + 1. Compact vs full AST ratio for 10-function module (7%) + 2. Compact vs full AST ratio for 50-function module (6%) + 3. Compact vs full AST ratio for 200-function module (6%) + 4. Lean scope vs detailed scope ratio (4%, under 20% target) + 5. Diagnostic delta vs full diagnostics validation + 6. Budget pagination across 3 pages covers all 51 nodes + 7. Batch query vs sequential bytes (single-envelope efficiency) + 8. Token estimates: compact < full (1001 vs 15039) + 9. Lean call hierarchy + lean deps both produce valid output + 10. Combined efficiency: batch + compact + lean + budget = 95% savings + 11. Benchmark: 100 parse→mutate→diagnose cycles in 8ms (0ms/cycle) + 12. Compact AST ratio stable across module sizes (spread=1%) + +**Files modified:** +- `editor/CMakeLists.txt` — step257_test target + +**Key results:** +- Phase 9c complete: all 4 steps pass (48/48 tests across steps 254–257) +- Compact AST: 6–7% of full AST size across all module sizes (93–94% savings) +- Lean scope: 4% of detailed size (96% savings) +- Combined optimized query path: 95% total token savings vs naive approach +- Compact ratio stable across 10/50/200 function modules (1% spread) +- Throughput: 0ms/cycle for parse→mutate→diagnose (sub-millisecond)