Step 257: token efficiency tests + benchmarks (Phase 9c complete)
Systematic validation of all Phase 9c token-saving features: compact AST (6-7% of full size), lean scope (4% of detailed), diagnostic deltas, budget pagination across 3 pages, batch queries, and combined 95% savings. Benchmark confirms sub-millisecond parse->mutate->diagnose cycles. Also adds Environment Layer feature request to docs/annotations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
159
docs/annotations/Environment Layer.md
Normal file
159
docs/annotations/Environment Layer.md
Normal file
@@ -0,0 +1,159 @@
|
||||
# Feature Request: Environment Layer (Runtime / Host Semantics)
|
||||
|
||||
## Summary
|
||||
|
||||
Add a first-class Environment Layer to the AST-first pipeline so programs can be represented and transformed with explicit runtime/host context, not just syntax and pure computation.
|
||||
|
||||
This is required for meaningful equivalence across languages whose semantics depend on a host runtime (browser, JVM, Python runtime, OS process model, Emacs runtime, etc.).
|
||||
|
||||
## Motivation
|
||||
|
||||
Many "languages" are inseparable from their environment:
|
||||
|
||||
- **JavaScript in a browser** includes event loop, microtasks/macrotasks, DOM/Web APIs, sandbox constraints.
|
||||
- **Python** includes import system, module cache, object model, exceptions, refcount/GC behavior, venv packaging.
|
||||
- **JVM** includes class loading, verification, reflection, JIT, GC, bytecode semantics.
|
||||
|
||||
If SemAnno only models AST/computation, the system will misrepresent equivalence and produce incorrect transpilation/runtime scaffolding.
|
||||
|
||||
## Goals
|
||||
|
||||
- Represent environment assumptions explicitly alongside AST/IR.
|
||||
- Allow transforms/transpilation to target either:
|
||||
- a different language plus runtime scaffolding, or
|
||||
- a different environment model (e.g., "browser-like event loop" in C++).
|
||||
- Provide an interface for agents to add environment-specific capabilities incrementally without rewriting core AST.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Fully implementing a browser, JVM, or CPython.
|
||||
- Building a complete OS abstraction layer.
|
||||
- Perfect semantic equivalence across all edge cases (initially).
|
||||
- Standardizing on any single IR format beyond "must be able to attach env info."
|
||||
|
||||
## Definitions
|
||||
|
||||
- **Environment**: The execution context providing services and semantics beyond pure expression evaluation.
|
||||
- **Capability**: A named service the environment provides (I/O, timers, threads, module loading, reflection, etc.).
|
||||
- **Binding Time**: Whether a name/operation resolves at compile-time, link-time, load-time, or runtime.
|
||||
- **Scheduler Model**: Event loop vs threads vs fibers vs cooperative coroutines.
|
||||
- **Memory Model**: Ownership/RAII, tracing GC, ref-counting, region/arena, etc.
|
||||
|
||||
## User Stories
|
||||
|
||||
- As a SemAnno user, I want to mark that a program depends on event loop semantics so a C++ target can generate appropriate scheduling code.
|
||||
- As an agent generating C++ from Python-like semantics, I want to know whether imports are dynamic and how to model module caching and lookup.
|
||||
- As an agent generating code for a browser target, I want to know which capabilities are required (DOM, fetch, timers) and which are forbidden (filesystem).
|
||||
- As a developer debugging transpilation, I want the environment layer to explain why a transform chose threads vs event loop vs coroutines.
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1. Environment Spec Object
|
||||
|
||||
There must be a structured "Environment Spec" attached to a program/module and accessible during transformations.
|
||||
|
||||
Minimum fields (names are suggestions; adapt to SemAnno conventions):
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `env.id` | string | identifier, e.g. `posix_process`, `browser`, `jvm`, `python_cpython_like`, `emacs_runtime` |
|
||||
| `env.version` | optional string | version of the environment spec |
|
||||
| `env.capabilities` | set/list | declared required capabilities |
|
||||
| `env.constraints` | set/list | forbidden or restricted behaviors |
|
||||
| `env.scheduler` | enum-ish | `event_loop`, `threads`, `fibers`, `coroutines`, `single_thread` |
|
||||
| `env.memory` | enum-ish | `manual`, `raii`, `refcount`, `tracing_gc`, `region` |
|
||||
| `env.binding_times` | mapping | per symbol class or feature: `compile`, `link`, `load`, `runtime` |
|
||||
| `env.exceptions` | model descriptor | unwind style / checked vs unchecked / typed vs untyped |
|
||||
| `env.ffi` | descriptor | C ABI available? dynamic linking? host-call boundary? |
|
||||
|
||||
### R2. Capability Declarations
|
||||
|
||||
Provide a small fixed vocabulary to start (expandable):
|
||||
|
||||
- `io.fs`
|
||||
- `io.net`
|
||||
- `io.stdinout`
|
||||
- `clock.time`
|
||||
- `timers`
|
||||
- `event_loop`
|
||||
- `threads`
|
||||
- `process`
|
||||
- `signals`
|
||||
- `modules`
|
||||
- `dynamic_loading`
|
||||
- `reflection`
|
||||
- `gc`
|
||||
- `jit`
|
||||
- `ui.dom` (browser-like)
|
||||
- `editor.emacs` (hosted in Emacs runtime / elisp interop)
|
||||
- `host.imgui` (UI host integration)
|
||||
|
||||
### R3. Transform Hooks Must Receive Env
|
||||
|
||||
All major transforms must have read access to the environment spec:
|
||||
|
||||
- AST -> IR lowering
|
||||
- IR -> target emission
|
||||
- optimization passes (esp. those unsafe across concurrency/GC boundaries)
|
||||
- runtime scaffolding generation
|
||||
|
||||
### R4. Environment-Aware Lowering
|
||||
|
||||
Certain constructs must lower differently depending on env. Examples:
|
||||
|
||||
| Construct | Lowering varies by... |
|
||||
|---|---|
|
||||
| `async/await` | event loop tasks vs threads vs coroutines |
|
||||
| `imports/modules` | static linking vs dynamic loading vs runtime lookup |
|
||||
| `exceptions` | unwind vs error unions vs result objects |
|
||||
| `dynamic attribute access` | dictionary lookup vs vtable-like dispatch |
|
||||
|
||||
(Exact mapping is a later task; requirement is that the architecture supports this.)
|
||||
|
||||
### R5. Explicit Host Boundary Nodes (Optional but Recommended)
|
||||
|
||||
To avoid hidden assumptions, represent host interactions as explicit IR/AST nodes:
|
||||
|
||||
```
|
||||
HostCall(capability="io.net", name="fetch", args=...)
|
||||
ScheduleTask(queue="microtask", ...)
|
||||
ModuleLoad(name=..., mode=dynamic/static, ...)
|
||||
```
|
||||
|
||||
This enables correct transpilation/scaffolding and helps debugging.
|
||||
|
||||
## Implementation Notes (Low-Assumption)
|
||||
|
||||
- Prefer adding an annotation schema (e.g., JSON/YAML/S-expression) that can be attached to modules and referenced by UUID/path.
|
||||
- The editor should surface the env spec as:
|
||||
- a module-level panel (read/write), and
|
||||
- per-node "requires capability" hints where relevant.
|
||||
- Keep the initial environment models coarse:
|
||||
- "POSIX-ish native"
|
||||
- "Browser-ish event loop"
|
||||
- "Python-like dynamic runtime"
|
||||
- "JVM-like managed runtime"
|
||||
- "Emacs-hosted runtime"
|
||||
- Allow partial specs; missing fields should not crash transforms (fall back to conservative defaults).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- There exists a persisted Environment Spec artifact (in annotations) that is loadable with the rest of SemAnno metadata.
|
||||
- A transform can query `env.scheduler` and `env.capabilities` and choose different lowering strategies.
|
||||
- The editor can display and edit the Environment Spec without requiring the AST editor to be complete.
|
||||
- At least one demo transformation uses env info (e.g., toggling `event_loop` vs `threads` changes generated scaffolding).
|
||||
|
||||
## Suggested Deliverables
|
||||
|
||||
1. EnvironmentSpec schema + parser/loader
|
||||
2. Editor UI panel to view/edit EnvironmentSpec
|
||||
3. Pass-through plumbing so all transforms receive EnvironmentSpec
|
||||
4. One example backend decision driven by env (scheduler or module loading)
|
||||
5. Documentation entry describing the field meanings and intended usage
|
||||
|
||||
## Open Questions (Intentionally Not Assumed)
|
||||
|
||||
- Where should env live? (program-level, module-level, per-subtree override?)
|
||||
- How should capabilities be namespaced/versioned?
|
||||
- What is SemAnno's canonical serialization format for annotations?
|
||||
- How should "mixed environments" work (e.g., native core + embedded JS VM)?
|
||||
@@ -1499,4 +1499,15 @@ target_link_libraries(step256_test PRIVATE
|
||||
tree_sitter_java tree_sitter_rust tree_sitter_go
|
||||
tree_sitter_org)
|
||||
|
||||
# Step 257: Token efficiency tests + benchmarks
|
||||
add_executable(step257_test tests/step257_test.cpp)
|
||||
target_include_directories(step257_test PRIVATE src)
|
||||
target_link_libraries(step257_test PRIVATE
|
||||
nlohmann_json::nlohmann_json
|
||||
unofficial::tree-sitter::tree-sitter
|
||||
tree_sitter_python tree_sitter_cpp tree_sitter_elisp
|
||||
tree_sitter_javascript tree_sitter_typescript
|
||||
tree_sitter_java tree_sitter_rust tree_sitter_go
|
||||
tree_sitter_org)
|
||||
|
||||
# Step 12: Dear ImGui shell scaffolding created (main.cpp exists but not built due to dependencies)
|
||||
|
||||
516
editor/tests/step257_test.cpp
Normal file
516
editor/tests/step257_test.cpp
Normal file
@@ -0,0 +1,516 @@
|
||||
// Step 257 TDD Test: Token Efficiency Tests + Benchmarks
|
||||
//
|
||||
// Phase 9c closer. Systematically measures token savings from compact AST,
|
||||
// lean scope queries, diagnostic deltas, budget pagination, and batch
|
||||
// queries. Benchmarks parse->mutate->diagnose cycle throughput.
|
||||
#include "HeadlessEditorState.h"
|
||||
#include "CompactAST.h"
|
||||
#include "ResponseBudget.h"
|
||||
#include "StructuredDiagnostics.h"
|
||||
|
||||
#include <iostream>
|
||||
#include <string>
|
||||
#include <chrono>
|
||||
|
||||
static void expect(bool cond, const std::string& name,
|
||||
int& passed, int& failed) {
|
||||
if (cond) {
|
||||
std::cout << "Test " << (passed + failed + 1)
|
||||
<< " PASS: " << name << "\n";
|
||||
++passed;
|
||||
} else {
|
||||
std::cout << "Test " << (passed + failed + 1)
|
||||
<< " FAIL: " << name << "\n";
|
||||
++failed;
|
||||
}
|
||||
}
|
||||
|
||||
static json rpc(HeadlessEditorState& state, const std::string& session,
|
||||
const std::string& method,
|
||||
json params = json::object()) {
|
||||
json request = {{"jsonrpc", "2.0"}, {"id", 1},
|
||||
{"method", method}, {"params", params}};
|
||||
return state.processAgentRequest(request, session);
|
||||
}
|
||||
|
||||
// Generate a Python module with N functions, each with a body
|
||||
static std::string generateModule(int numFunctions) {
|
||||
std::string src;
|
||||
for (int i = 0; i < numFunctions; ++i) {
|
||||
src += "def func_" + std::to_string(i) + "(a, b):\n";
|
||||
src += " c = a + b\n";
|
||||
src += " return c\n\n";
|
||||
}
|
||||
return src;
|
||||
}
|
||||
|
||||
int main() {
|
||||
int passed = 0;
|
||||
int failed = 0;
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 1: Compact vs full AST ratio — 10-function module
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(10);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("small.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
json fullResp = rpc(state, "s1", "getAST");
|
||||
json compactResp = rpc(state, "s1", "getAST",
|
||||
{{"compact", true}});
|
||||
int fullSize = (int)fullResp["result"].dump().size();
|
||||
int compactSize = (int)compactResp["result"].dump().size();
|
||||
double ratio = (double)compactSize / fullSize * 100.0;
|
||||
expect(ratio < 30.0 && compactSize > 0,
|
||||
"10-func compact/full ratio=" +
|
||||
std::to_string((int)ratio) + "% (<30%) [" +
|
||||
std::to_string(compactSize) + "/" +
|
||||
std::to_string(fullSize) + "]",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 2: Compact vs full AST ratio — 50-function module
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(50);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("medium.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
json fullResp = rpc(state, "s1", "getAST");
|
||||
json compactResp = rpc(state, "s1", "getAST",
|
||||
{{"compact", true}});
|
||||
int fullSize = (int)fullResp["result"].dump().size();
|
||||
int compactSize = (int)compactResp["result"].dump().size();
|
||||
double ratio = (double)compactSize / fullSize * 100.0;
|
||||
expect(ratio < 30.0 && compactSize > 0,
|
||||
"50-func compact/full ratio=" +
|
||||
std::to_string((int)ratio) + "% (<30%) [" +
|
||||
std::to_string(compactSize) + "/" +
|
||||
std::to_string(fullSize) + "]",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 3: Compact vs full AST ratio — 200-function module
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(200);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("large.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
json fullResp = rpc(state, "s1", "getAST");
|
||||
json compactResp = rpc(state, "s1", "getAST",
|
||||
{{"compact", true}});
|
||||
int fullSize = (int)fullResp["result"].dump().size();
|
||||
int compactSize = (int)compactResp["result"].dump().size();
|
||||
double ratio = (double)compactSize / fullSize * 100.0;
|
||||
expect(ratio < 30.0 && compactSize > 0,
|
||||
"200-func compact/full ratio=" +
|
||||
std::to_string((int)ratio) + "% (<30%) [" +
|
||||
std::to_string(compactSize) + "/" +
|
||||
std::to_string(fullSize) + "]",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 4: Lean scope vs detailed scope — size ratio
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(50);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("scope.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
std::string funcId;
|
||||
for (auto* c : state.activeAST()->allChildren()) {
|
||||
if (c->conceptType == "Function") {
|
||||
funcId = c->id;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
json lean = rpc(state, "s1", "getInScopeSymbols",
|
||||
{{"nodeId", funcId}});
|
||||
json detailed = rpc(state, "s1", "getInScopeSymbols",
|
||||
{{"nodeId", funcId}, {"detailed", true}});
|
||||
int leanSize = (int)lean["result"].dump().size();
|
||||
int detailedSize = (int)detailed["result"].dump().size();
|
||||
double ratio = (detailedSize > 0)
|
||||
? (double)leanSize / detailedSize * 100.0 : 100.0;
|
||||
expect(ratio < 20.0 && leanSize > 0,
|
||||
"Lean/detailed scope ratio=" +
|
||||
std::to_string((int)ratio) + "% (<20%) [" +
|
||||
std::to_string(leanSize) + "/" +
|
||||
std::to_string(detailedSize) + "]",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 5: Diagnostic delta vs full — size ratio after mutation
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src =
|
||||
"def cleanup(ptr):\n x = ptr\n\n"
|
||||
"def process(data):\n return data\n";
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("delta.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
// Baseline diagnostics (records snapshot)
|
||||
json fullDiags = rpc(state, "s1", "getDiagnostics");
|
||||
int fullDiagSize = (int)fullDiags["result"].dump().size();
|
||||
|
||||
// Add an annotation to trigger a new diagnostic
|
||||
Module* ast = state.activeAST();
|
||||
for (auto* c : ast->allChildren()) {
|
||||
if (c->conceptType == "Function" &&
|
||||
getNodeName(c) == "cleanup") {
|
||||
c->addChild("annotations", new DeallocateAnnotation());
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Get delta instead of full re-fetch
|
||||
json delta = rpc(state, "s1", "getDiagnosticsDelta",
|
||||
{{"sinceVersion", 0}});
|
||||
int deltaSize = (int)delta["result"].dump().size();
|
||||
// Delta should be smaller than a full diagnostic dump when
|
||||
// changes are small relative to the full set
|
||||
bool deltaValid = delta["result"].contains("added") &&
|
||||
delta["result"].contains("removed");
|
||||
expect(deltaValid && deltaSize > 0 && fullDiagSize > 0,
|
||||
"Delta diagnostics valid (delta=" +
|
||||
std::to_string(deltaSize) + " full=" +
|
||||
std::to_string(fullDiagSize) + ")",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 6: Budget pagination across 3 pages covers all items
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(50);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("pages.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
// Get full compact AST to know the total node count
|
||||
json fullCompact = rpc(state, "s1", "getAST",
|
||||
{{"compact", true}});
|
||||
int totalNodes = fullCompact["result"].value("nodeCount", 0);
|
||||
|
||||
// Page 1: small budget
|
||||
json page1 = rpc(state, "s1", "getAST",
|
||||
{{"compact", true}, {"budget", 500}});
|
||||
int p1Count = page1["result"].value("returnedCount", 0);
|
||||
std::string cont1 = page1["result"].value("continuation", "");
|
||||
|
||||
// Build full result for continuation
|
||||
json compactNodes = toJsonCompactSummary(state.activeAST());
|
||||
json fullResult = {{"nodes", compactNodes},
|
||||
{"nodeCount", (int)compactNodes.size()}};
|
||||
|
||||
// Page 2
|
||||
json page2 = applyContinuation(fullResult, cont1, 500);
|
||||
int p2Count = page2.value("returnedCount", 0);
|
||||
std::string cont2 = page2.value("continuation", "");
|
||||
|
||||
// Page 3: large budget to get the rest
|
||||
int p3Count = 0;
|
||||
if (!cont2.empty()) {
|
||||
json page3 = applyContinuation(fullResult, cont2, 100000);
|
||||
p3Count = page3.value("returnedCount", 0);
|
||||
}
|
||||
|
||||
int covered = p1Count + p2Count + p3Count;
|
||||
expect(p1Count > 0 && p2Count > 0 && covered >= totalNodes,
|
||||
"3 pages cover all " + std::to_string(totalNodes) +
|
||||
" nodes (" + std::to_string(p1Count) + "+" +
|
||||
std::to_string(p2Count) + "+" +
|
||||
std::to_string(p3Count) + "=" +
|
||||
std::to_string(covered) + ")",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 7: Batch query vs sequential — total bytes comparison
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(30);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("batch.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
std::string funcId;
|
||||
for (auto* c : state.activeAST()->allChildren()) {
|
||||
if (c->conceptType == "Function") {
|
||||
funcId = c->id;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Sequential: 3 separate RPC calls
|
||||
json r1 = rpc(state, "s1", "getAST", {{"compact", true}});
|
||||
json r2 = rpc(state, "s1", "getDiagnostics");
|
||||
json r3 = rpc(state, "s1", "getInScopeSymbols",
|
||||
{{"nodeId", funcId}});
|
||||
int seqBytes = (int)(r1.dump().size() + r2.dump().size() +
|
||||
r3.dump().size());
|
||||
|
||||
// Batch: 1 RPC call
|
||||
json batchResp = rpc(state, "s1", "batchQuery", {
|
||||
{"queries", json::array({
|
||||
{{"method", "getAST"},
|
||||
{"params", {{"compact", true}}}},
|
||||
{{"method", "getDiagnostics"}},
|
||||
{{"method", "getInScopeSymbols"},
|
||||
{"params", {{"nodeId", funcId}}}}
|
||||
})}
|
||||
});
|
||||
int batchBytes = (int)batchResp.dump().size();
|
||||
|
||||
// Batch has protocol overhead but only one envelope
|
||||
// The key win is round-trips, not bytes — but batch should
|
||||
// not be dramatically larger
|
||||
bool batchReasonable = batchBytes < seqBytes * 2;
|
||||
int batchCount = batchResp["result"].value("count", 0);
|
||||
expect(batchReasonable && batchCount == 3,
|
||||
"Batch " + std::to_string(batchBytes) +
|
||||
" bytes vs sequential " + std::to_string(seqBytes) +
|
||||
" bytes (3 results in 1 envelope)",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 8: Token estimates decrease with compact mode
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(50);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("tokens.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
json fullResp = rpc(state, "s1", "getAST");
|
||||
json compactResp = rpc(state, "s1", "getAST",
|
||||
{{"compact", true}});
|
||||
int fullTokens = fullResp["result"].value("tokenEstimate", 0);
|
||||
int compactTokens = compactResp["result"].value(
|
||||
"tokenEstimate", 0);
|
||||
// Both should report token estimates, compact should be lower
|
||||
expect(fullTokens > 0 && compactTokens > 0 &&
|
||||
compactTokens < fullTokens,
|
||||
"Token estimates: compact=" +
|
||||
std::to_string(compactTokens) + " < full=" +
|
||||
std::to_string(fullTokens),
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 9: Lean call hierarchy + lean deps smaller than detailed
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src =
|
||||
"def helper(x):\n"
|
||||
" return x * 2\n\n"
|
||||
"def compute(a, b):\n"
|
||||
" c = helper(a)\n"
|
||||
" d = helper(b)\n"
|
||||
" return c + d\n\n"
|
||||
"def orchestrate():\n"
|
||||
" r1 = compute(1, 2)\n"
|
||||
" r2 = compute(3, 4)\n"
|
||||
" return r1 + r2\n";
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("hier.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
std::string computeId;
|
||||
for (auto* c : state.activeAST()->allChildren()) {
|
||||
if (c->conceptType == "Function" &&
|
||||
getNodeName(c) == "compute") {
|
||||
computeId = c->id;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
json leanCall = rpc(state, "s1", "getCallHierarchy",
|
||||
{{"functionId", computeId}});
|
||||
json detCall = rpc(state, "s1", "getCallHierarchy",
|
||||
{{"functionId", computeId},
|
||||
{"detailed", true}});
|
||||
json leanDeps = rpc(state, "s1", "getDependencyGraph",
|
||||
{{"nodeId", computeId}});
|
||||
json detDeps = rpc(state, "s1", "getDependencyGraph",
|
||||
{{"nodeId", computeId},
|
||||
{"detailed", true}});
|
||||
|
||||
int leanCallSize = (int)leanCall["result"].dump().size();
|
||||
int detCallSize = (int)detCall["result"].dump().size();
|
||||
int leanDepsSize = (int)leanDeps["result"].dump().size();
|
||||
int detDepsSize = (int)detDeps["result"].dump().size();
|
||||
|
||||
// Both lean modes should produce valid output
|
||||
bool callValid = leanCall["result"].value("mode", "") ==
|
||||
"symbols" && detCall["result"].value("mode", "") ==
|
||||
"detailed";
|
||||
bool depsValid = leanDeps["result"].value("mode", "") ==
|
||||
"symbols" && detDeps["result"].value("mode", "") ==
|
||||
"detailed";
|
||||
expect(callValid && depsValid,
|
||||
"Lean call=" + std::to_string(leanCallSize) +
|
||||
" det=" + std::to_string(detCallSize) +
|
||||
"; lean deps=" + std::to_string(leanDepsSize) +
|
||||
" det=" + std::to_string(detDepsSize),
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 10: Combined efficiency — batch + compact + lean + budget
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string src = generateModule(50);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("combined.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
std::string funcId;
|
||||
for (auto* c : state.activeAST()->allChildren()) {
|
||||
if (c->conceptType == "Function") {
|
||||
funcId = c->id;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Naive approach: full AST + detailed scope (2 separate calls)
|
||||
json naiveAST = rpc(state, "s1", "getAST");
|
||||
json naiveScope = rpc(state, "s1", "getInScopeSymbols",
|
||||
{{"nodeId", funcId}, {"detailed", true}});
|
||||
int naiveTotal = (int)(naiveAST.dump().size() +
|
||||
naiveScope.dump().size());
|
||||
|
||||
// Optimized: batch with compact AST + lean scope + budget
|
||||
json optResp = rpc(state, "s1", "batchQuery", {
|
||||
{"queries", json::array({
|
||||
{{"method", "getAST"},
|
||||
{"params", {{"compact", true}, {"budget", 2000}}}},
|
||||
{{"method", "getInScopeSymbols"},
|
||||
{"params", {{"nodeId", funcId}}}}
|
||||
})}
|
||||
});
|
||||
int optTotal = (int)optResp.dump().size();
|
||||
|
||||
double savings = (1.0 - (double)optTotal / naiveTotal) * 100.0;
|
||||
expect(optTotal < naiveTotal && savings > 50.0,
|
||||
"Combined savings=" + std::to_string((int)savings) +
|
||||
"% (opt=" + std::to_string(optTotal) + " naive=" +
|
||||
std::to_string(naiveTotal) + ")",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 11: Benchmark — 100 parse->mutate->diagnose cycles
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
std::string baseSrc =
|
||||
"def target(x):\n return x\n\n"
|
||||
"def main():\n result = target(42)\n return result\n";
|
||||
|
||||
auto start = std::chrono::high_resolution_clock::now();
|
||||
|
||||
for (int i = 0; i < 100; ++i) {
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("bench.py", baseSrc, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
// Parse (already done by openBuffer) — get AST
|
||||
rpc(state, "s1", "getAST", {{"compact", true}});
|
||||
|
||||
// Mutate: add a parameter
|
||||
Module* ast = state.activeAST();
|
||||
if (ast) {
|
||||
for (auto* c : ast->allChildren()) {
|
||||
if (c->conceptType == "Function" &&
|
||||
getNodeName(c) == "target") {
|
||||
auto* param = new Parameter(
|
||||
"p_" + std::to_string(i),
|
||||
"extra_" + std::to_string(i));
|
||||
c->addChild("parameters", param);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Diagnose
|
||||
rpc(state, "s1", "getDiagnostics");
|
||||
}
|
||||
|
||||
auto end = std::chrono::high_resolution_clock::now();
|
||||
auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
end - start).count();
|
||||
double perCycle = (double)ms / 100.0;
|
||||
|
||||
// Target: each cycle under 50ms (generous for CI)
|
||||
expect(perCycle < 50.0,
|
||||
"100 parse->mutate->diagnose cycles in " +
|
||||
std::to_string(ms) + "ms (" +
|
||||
std::to_string((int)perCycle) + "ms/cycle)",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Test 12: Compact AST ratio stable across module sizes
|
||||
// ---------------------------------------------------------------
|
||||
{
|
||||
// Verify ratio doesn't degrade as module grows
|
||||
double ratios[3];
|
||||
int sizes[] = {10, 50, 200};
|
||||
for (int s = 0; s < 3; ++s) {
|
||||
std::string src = generateModule(sizes[s]);
|
||||
HeadlessEditorState state;
|
||||
state.defaultLanguage = "python";
|
||||
state.openBuffer("scale.py", src, "python");
|
||||
state.setAgentRole("s1", AgentRole::Refactor);
|
||||
|
||||
json full = rpc(state, "s1", "getAST");
|
||||
json compact = rpc(state, "s1", "getAST",
|
||||
{{"compact", true}});
|
||||
int fSize = (int)full["result"].dump().size();
|
||||
int cSize = (int)compact["result"].dump().size();
|
||||
ratios[s] = (fSize > 0)
|
||||
? (double)cSize / fSize * 100.0 : 100.0;
|
||||
}
|
||||
|
||||
// All ratios should be under 30%, and spread should be small
|
||||
// (compact benefit doesn't degrade at scale)
|
||||
double maxRatio = std::max({ratios[0], ratios[1], ratios[2]});
|
||||
double minRatio = std::min({ratios[0], ratios[1], ratios[2]});
|
||||
double spread = maxRatio - minRatio;
|
||||
expect(maxRatio < 30.0 && spread < 15.0,
|
||||
"Ratio stable across sizes: 10-func=" +
|
||||
std::to_string((int)ratios[0]) + "% 50-func=" +
|
||||
std::to_string((int)ratios[1]) + "% 200-func=" +
|
||||
std::to_string((int)ratios[2]) + "% (spread=" +
|
||||
std::to_string((int)spread) + "%)",
|
||||
passed, failed);
|
||||
}
|
||||
|
||||
std::cout << "\n=== Step 257 Results: " << passed << " passed, "
|
||||
<< failed << " failed ===\n";
|
||||
return failed == 0 ? 0 : 1;
|
||||
}
|
||||
32
progress.md
32
progress.md
@@ -369,3 +369,35 @@ in one sub-query don't affect others.
|
||||
- Error isolation: failed sub-query returns error object, others unaffected
|
||||
- batchQuery itself requires no special permission; sub-queries enforce their own
|
||||
- tools/list now returns 22 tools (was 21): +whetstone_batch_query
|
||||
|
||||
### Step 257: Token Efficiency Tests + Benchmarks
|
||||
**Status:** PASS (12/12 tests)
|
||||
|
||||
Phase 9c closer. Systematic measurement of token savings across all
|
||||
compact/lean/delta/budget/batch features, plus throughput benchmarking.
|
||||
|
||||
**Files created:**
|
||||
- `editor/tests/step257_test.cpp` — 12 test cases:
|
||||
1. Compact vs full AST ratio for 10-function module (7%)
|
||||
2. Compact vs full AST ratio for 50-function module (6%)
|
||||
3. Compact vs full AST ratio for 200-function module (6%)
|
||||
4. Lean scope vs detailed scope ratio (4%, under 20% target)
|
||||
5. Diagnostic delta vs full diagnostics validation
|
||||
6. Budget pagination across 3 pages covers all 51 nodes
|
||||
7. Batch query vs sequential bytes (single-envelope efficiency)
|
||||
8. Token estimates: compact < full (1001 vs 15039)
|
||||
9. Lean call hierarchy + lean deps both produce valid output
|
||||
10. Combined efficiency: batch + compact + lean + budget = 95% savings
|
||||
11. Benchmark: 100 parse→mutate→diagnose cycles in 8ms (0ms/cycle)
|
||||
12. Compact AST ratio stable across module sizes (spread=1%)
|
||||
|
||||
**Files modified:**
|
||||
- `editor/CMakeLists.txt` — step257_test target
|
||||
|
||||
**Key results:**
|
||||
- Phase 9c complete: all 4 steps pass (48/48 tests across steps 254–257)
|
||||
- Compact AST: 6–7% of full AST size across all module sizes (93–94% savings)
|
||||
- Lean scope: 4% of detailed size (96% savings)
|
||||
- Combined optimized query path: 95% total token savings vs naive approach
|
||||
- Compact ratio stable across 10/50/200 function modules (1% spread)
|
||||
- Throughput: 0ms/cycle for parse→mutate→diagnose (sub-millisecond)
|
||||
|
||||
Reference in New Issue
Block a user