Phase 3b: Real tree-sitter integration (Steps 45-49)

Replace TreeSitterParser stubs with real tree-sitter C bindings for
Python, C++, and Elisp parsing. Add tree-sitter core via vcpkg and
language grammars via FetchContent. Implement CST-to-AST converters
with auto-annotation, memory pattern detection, error recovery, and
ParseResult/ParseDiagnostic types. All 34 tests pass (6+7+6+8+7).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
2026-02-08 09:06:50 -07:00
parent 7c0b0c759e
commit 41166812b0
6 changed files with 1466 additions and 144 deletions

View File

@@ -106,16 +106,17 @@ All 38 steps implemented and passing. Each step has a corresponding test (`step1
> `ReclaimAnnotation`, `OwnerAnnotation`, `AllocateAnnotation`) are implemented in
> `editor/src/ast/Generator.h`.
### Phase 3b: Tree-sitter Integration (Steps 4549) — TDD STUBS ONLY
- [x] Step 45: TDD test written (compiles, passes basic assertions)
- [x] Step 46: TDD test written (compiles, passes basic assertions)
- [x] Step 47: TDD test written (compiles, passes basic assertions)
- [ ] Step 48: CST-to-AST mapping refinement — not started
- [ ] Step 49: Error recovery and diagnostics — not started
### Phase 3b: Tree-sitter Integration (Steps 4549) — COMPLETE
- [x] Step 45: Real tree-sitter Python parsing (6/6 tests pass)
- [x] Step 46: Real tree-sitter C++ parsing with memory pattern detection (7/7 tests pass)
- [x] Step 47: Real tree-sitter Elisp parsing + cross-language equivalence (6/6 tests pass)
- [x] Step 48: CST-to-AST mapping refinement — default params, if/for, multi-statement, void, multi-function, string literals (8/8 tests pass)
- [x] Step 49: Error recovery and diagnostics — ParseResult/ParseDiagnostic types, partial parse, hasErrors() (7/7 tests pass)
> Steps 4547 have TDD test stubs that compile and pass placeholder assertions.
> Real tree-sitter C bindings are NOT yet linked. The `TreeSitterParser` in
> `Parser.h` uses stub implementations from Sprint 2 Step 31.
> Tree-sitter core via vcpkg (`tree-sitter` 0.26.5). Language grammars via FetchContent:
> `tree-sitter-python` v0.23.6, `tree-sitter-cpp` v0.23.4, `tree-sitter-elisp` 1.3.0.
> `TreeSitterParser` in `Parser.h` implements full CST-to-AST conversion for all three languages
> with auto-annotation (Python/Elisp → @Reclaim(Tracing), C++ → memory pattern detection).
### Phase 3c: Classical Editing Mode (Steps 5054) — TDD STUBS ONLY
- [ ] Step 50: Text editor component — not started
@@ -132,7 +133,7 @@ All 38 steps implemented and passing. Each step has a corresponding test (`step1
### Phase 3e: Agent API Extend (Steps 5963) — TDD STUBS ONLY
- [ ] Step 59: WebSocket agent endpoint — not started
- [x] Step 60: TDD test written (does not link)
- [x] Step 60: **IMPLEMENTED** — ASTQueryAPI with getAST, findNodesByType/Property/Annotation, getSubtree (8/8 tests pass)
- [x] Step 61: TDD test written (does not link)
- [x] Step 62: TDD test written (does not link)
- [x] Step 63: TDD test written (does not link)
@@ -186,6 +187,7 @@ cd E:\Whetstone_DSL\installer\windows
- `sdl2:x64-windows`
- `imgui[docking-experimental,opengl3-binding]:x64-windows`
- `glad:x64-windows`
- `tree-sitter:x64-windows`
### Known Issue: imgui SDL2 Backend
vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exists). The SDL2 backend files are vendored locally in `editor/src/imgui_backends/` and compiled directly by CMake.
@@ -194,10 +196,12 @@ vcpkg's imgui 1.91.9 removed the `sdl2-binding` feature (only `sdl3-binding` exi
## Test Results (Last Verified)
**Steps 147:** All compile and pass (47 executables in `editor/build/Release/`)
**Steps 4875:** Either not started or TDD stubs that don't link yet
**Steps 149:** All compile and pass (49 executables in `editor/build/Release/`)
**Steps 5075:** Either not started or TDD stubs that don't link yet
The build only compiles `whetstone_editor` and `orchestrator` targets plus steps 147. Steps 51+ are TDD stubs that reference unimplemented code and will fail to link until their implementations exist.
The build compiles `whetstone_editor`, `orchestrator`, and steps 149. Steps 4549 link against
tree-sitter core + grammar static libraries. Steps 51+ are TDD stubs that reference unimplemented
code and will fail to link until their implementations exist.
---
@@ -228,12 +232,10 @@ The build only compiles `whetstone_editor` and `orchestrator` targets plus steps
## What's Next
The next logical work is **Phase 3b real implementation** (Steps 4549): replace tree-sitter stubs with real C bindings. This requires:
1. Add `tree-sitter`, `tree-sitter-python`, `tree-sitter-cpp`, `tree-sitter-elisp` as CMake FetchContent dependencies
2. Replace `TreeSitterParser` stubs in `Parser.h` with real CST-to-AST mapping
3. Ensure the existing TDD tests for steps 4547 pass with real implementations
Alternatively, jump to **Phase 3c** (Steps 5054) for classical text editing mode, or **Phase 3e** (Steps 5963) for WebSocket agent API — these are independent of tree-sitter.
Phase 3b (tree-sitter integration) is complete. Next logical work:
- **Phase 3c** (Steps 5054): Classical text editing mode (text editor component, syntax highlighting, keybindings)
- **Phase 3e** (Steps 5963): WebSocket agent API (Step 60 already done, remaining: WebSocket endpoint, agent protocol)
- **Phase 3d** (Steps 5558): Emacs integration completion (splash screen, mode-specific behavior)
---
@@ -251,3 +253,5 @@ Alternatively, jump to **Phase 3c** (Steps 5054) for classical text editing m
| 2025-latest | Claude | Vendored imgui SDL2 backend (vcpkg removed sdl2-binding feature) |
| 2025-latest | Claude | Created Windows installer (Inno Setup), verified it installs and runs |
| 2025-latest | Claude | Fixed editor hang when launched without orchestrator pipe |
| 2026-02-07 | Claude Opus 4.6 | Step 60: Implemented ASTQueryAPI (tree walk, JSON output, find by type/property/annotation). 8/8 tests pass. Added PROGRESS.md. |
| 2026-02-08 | Claude Opus 4.6 | Phase 3b: Real tree-sitter integration (Steps 4549). Replaced Parser.h stubs with real tree-sitter C bindings. Python/C++/Elisp CST-to-AST conversion, memory pattern detection, error recovery. 34/34 tests pass. |

View File

@@ -1,5 +1,5 @@
cmake_minimum_required(VERSION 3.20)
project(WhetstoneEditor LANGUAGES CXX)
project(WhetstoneEditor LANGUAGES C CXX)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
@@ -7,6 +7,79 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
# Use vcpkg-installed nlohmann_json instead of FetchContent
find_package(nlohmann_json CONFIG REQUIRED)
# Tree-sitter core (from vcpkg)
find_package(unofficial-tree-sitter CONFIG REQUIRED)
# Tree-sitter grammar libraries (not available in vcpkg, fetched from GitHub)
include(FetchContent)
FetchContent_Declare(
tree-sitter-python
GIT_REPOSITORY https://github.com/tree-sitter/tree-sitter-python.git
GIT_TAG v0.23.6
GIT_SHALLOW TRUE
)
FetchContent_Declare(
tree-sitter-cpp
GIT_REPOSITORY https://github.com/tree-sitter/tree-sitter-cpp.git
GIT_TAG v0.23.4
GIT_SHALLOW TRUE
)
FetchContent_Declare(
tree-sitter-elisp
GIT_REPOSITORY https://github.com/Wilfred/tree-sitter-elisp.git
GIT_TAG 1.3.0
GIT_SHALLOW TRUE
)
# Populate without add_subdirectory (grammars have their own CMakeLists with conflicting targets)
FetchContent_GetProperties(tree-sitter-python)
if(NOT tree-sitter-python_POPULATED)
FetchContent_Populate(tree-sitter-python)
endif()
FetchContent_GetProperties(tree-sitter-cpp)
if(NOT tree-sitter-cpp_POPULATED)
FetchContent_Populate(tree-sitter-cpp)
endif()
FetchContent_GetProperties(tree-sitter-elisp)
if(NOT tree-sitter-elisp_POPULATED)
FetchContent_Populate(tree-sitter-elisp)
endif()
# Build tree-sitter-python grammar as a static C library
add_library(tree_sitter_python STATIC
${tree-sitter-python_SOURCE_DIR}/src/parser.c
${tree-sitter-python_SOURCE_DIR}/src/scanner.c
)
target_include_directories(tree_sitter_python PUBLIC
${tree-sitter-python_SOURCE_DIR}/src
)
target_link_libraries(tree_sitter_python PUBLIC unofficial::tree-sitter::tree-sitter)
# Build tree-sitter-cpp grammar as a static C library
# Note: tree-sitter-cpp scanner is C++ (.cc)
add_library(tree_sitter_cpp STATIC
${tree-sitter-cpp_SOURCE_DIR}/src/parser.c
${tree-sitter-cpp_SOURCE_DIR}/src/scanner.c
)
target_include_directories(tree_sitter_cpp PUBLIC
${tree-sitter-cpp_SOURCE_DIR}/src
)
target_link_libraries(tree_sitter_cpp PUBLIC unofficial::tree-sitter::tree-sitter)
# Build tree-sitter-elisp grammar as a static C library
add_library(tree_sitter_elisp STATIC
${tree-sitter-elisp_SOURCE_DIR}/src/parser.c
)
target_include_directories(tree_sitter_elisp PUBLIC
${tree-sitter-elisp_SOURCE_DIR}/src
)
target_link_libraries(tree_sitter_elisp PUBLIC unofficial::tree-sitter::tree-sitter)
add_executable(step1_test tests/step1_test.cpp)
target_include_directories(step1_test PRIVATE src)
@@ -143,12 +216,23 @@ target_include_directories(step44_test PRIVATE src)
add_executable(step45_test tests/step45_test.cpp)
target_include_directories(step45_test PRIVATE src)
target_link_libraries(step45_test PRIVATE unofficial::tree-sitter::tree-sitter tree_sitter_python)
add_executable(step46_test tests/step46_test.cpp)
target_include_directories(step46_test PRIVATE src)
target_link_libraries(step46_test PRIVATE unofficial::tree-sitter::tree-sitter tree_sitter_cpp)
add_executable(step47_test tests/step47_test.cpp)
target_include_directories(step47_test PRIVATE src)
target_link_libraries(step47_test PRIVATE unofficial::tree-sitter::tree-sitter tree_sitter_python tree_sitter_elisp)
add_executable(step48_test tests/step48_test.cpp)
target_include_directories(step48_test PRIVATE src)
target_link_libraries(step48_test PRIVATE unofficial::tree-sitter::tree-sitter tree_sitter_python tree_sitter_cpp tree_sitter_elisp)
add_executable(step49_test tests/step49_test.cpp)
target_include_directories(step49_test PRIVATE src)
target_link_libraries(step49_test PRIVATE unofficial::tree-sitter::tree-sitter tree_sitter_python tree_sitter_cpp tree_sitter_elisp)
add_executable(step51_test tests/step51_test.cpp)
target_include_directories(step51_test PRIVATE src)

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,220 @@
// Step 48 TDD Test: CST-to-AST Mapping Refinement
//
// Tests more complex constructs across all three languages:
// 1. Python: function with default parameter (def f(x=10))
// 2. Python: if-statement in body
// 3. Python: for-loop in body
// 4. C++: function with multiple statements
// 5. C++: void function
// 6. Elisp: multiple functions in one source
// 7. Python: function with multiple parameters
// 8. C++: function with string literal
#include <iostream>
#include <string>
#include <cassert>
#include <memory>
#include "ast/Parser.h"
#include "ast/Annotation.h"
int main() {
int passed = 0;
int failed = 0;
// --- Test 1: Python function with default parameter ---
{
std::string source = "def f(x=10):\n return x\n";
auto module = TreeSitterParser::parsePython(source);
auto functions = module->getChildren("functions");
assert(!functions.empty() && "Should parse function");
auto* fn = static_cast<Function*>(functions[0]);
assert(fn->name == "f");
auto params = fn->getChildren("parameters");
assert(!params.empty() && "Should have parameter");
auto* param = static_cast<Parameter*>(params[0]);
assert(param->name == "x" && "Parameter name should be 'x'");
auto* defVal = param->getChild("defaultValue");
assert(defVal != nullptr && "Parameter should have a default value");
assert(defVal->conceptType == "IntegerLiteral" &&
"Default value should be IntegerLiteral");
std::cout << "Test 1 PASS: Python default parameter parsed" << std::endl;
++passed;
}
// --- Test 2: Python if-statement in body ---
{
std::string source =
"def check(x):\n"
" if x > 0:\n"
" return x\n";
auto module = TreeSitterParser::parsePython(source);
auto functions = module->getChildren("functions");
assert(!functions.empty());
auto* fn = static_cast<Function*>(functions[0]);
auto body = fn->getChildren("body");
assert(!body.empty() && "Body should not be empty");
bool hasIf = false;
for (auto* stmt : body) {
if (stmt->conceptType == "IfStatement") {
hasIf = true;
break;
}
}
assert(hasIf && "Body should contain an IfStatement");
std::cout << "Test 2 PASS: Python if-statement parsed" << std::endl;
++passed;
}
// --- Test 3: Python for-loop in body ---
{
std::string source =
"def loop(n):\n"
" for i in range(n):\n"
" x = i\n";
auto module = TreeSitterParser::parsePython(source);
auto functions = module->getChildren("functions");
assert(!functions.empty());
auto* fn = static_cast<Function*>(functions[0]);
auto body = fn->getChildren("body");
assert(!body.empty() && "Body should not be empty");
bool hasFor = false;
for (auto* stmt : body) {
if (stmt->conceptType == "ForLoop") {
hasFor = true;
break;
}
}
assert(hasFor && "Body should contain a ForLoop");
std::cout << "Test 3 PASS: Python for-loop parsed" << std::endl;
++passed;
}
// --- Test 4: C++ function with multiple statements ---
{
std::string source =
"int compute(int a, int b) {\n"
" int c = a + b;\n"
" return c;\n"
"}\n";
auto module = TreeSitterParser::parseCpp(source);
auto functions = module->getChildren("functions");
assert(!functions.empty());
auto* fn = static_cast<Function*>(functions[0]);
assert(fn->name == "compute");
auto body = fn->getChildren("body");
assert(body.size() >= 2 && "Function should have at least 2 body statements");
std::cout << "Test 4 PASS: C++ multiple statements parsed" << std::endl;
++passed;
}
// --- Test 5: C++ void function ---
{
std::string source = "void doNothing() {}\n";
auto module = TreeSitterParser::parseCpp(source);
auto functions = module->getChildren("functions");
assert(!functions.empty());
auto* fn = static_cast<Function*>(functions[0]);
assert(fn->name == "doNothing");
auto* retType = fn->getChild("returnType");
assert(retType != nullptr && "Function should have a return type");
auto* primType = static_cast<PrimitiveType*>(retType);
assert(primType->kind == "void" && "Return type should be 'void'");
std::cout << "Test 5 PASS: C++ void function parsed" << std::endl;
++passed;
}
// --- Test 6: Elisp multiple functions in one source ---
{
std::string source =
"(defun add (a b) (+ a b))\n"
"(defun sub (a b) (- a b))\n";
auto module = TreeSitterParser::parseElisp(source);
auto functions = module->getChildren("functions");
assert(functions.size() >= 2 && "Should parse at least 2 functions");
auto* fn1 = static_cast<Function*>(functions[0]);
auto* fn2 = static_cast<Function*>(functions[1]);
assert(fn1->name == "add" && "First function should be 'add'");
assert(fn2->name == "sub" && "Second function should be 'sub'");
std::cout << "Test 6 PASS: Elisp multiple functions parsed" << std::endl;
++passed;
}
// --- Test 7: Python function with multiple parameters ---
{
std::string source = "def add(a, b):\n return a + b\n";
auto module = TreeSitterParser::parsePython(source);
auto functions = module->getChildren("functions");
assert(!functions.empty());
auto* fn = static_cast<Function*>(functions[0]);
auto params = fn->getChildren("parameters");
assert(params.size() == 2 && "Function should have exactly 2 parameters");
auto* p1 = static_cast<Parameter*>(params[0]);
auto* p2 = static_cast<Parameter*>(params[1]);
assert(p1->name == "a" && "First parameter should be 'a'");
assert(p2->name == "b" && "Second parameter should be 'b'");
std::cout << "Test 7 PASS: Python multiple parameters parsed" << std::endl;
++passed;
}
// --- Test 8: C++ function with string literal ---
{
std::string source =
"void greet() {\n"
" return \"hello\";\n"
"}\n";
auto module = TreeSitterParser::parseCpp(source);
auto functions = module->getChildren("functions");
assert(!functions.empty());
auto* fn = static_cast<Function*>(functions[0]);
assert(fn->name == "greet");
auto body = fn->getChildren("body");
assert(!body.empty() && "Body should not be empty");
// Find a Return statement with a StringLiteral value
bool hasStringLit = false;
for (auto* stmt : body) {
if (stmt->conceptType == "Return") {
auto* ret = static_cast<Return*>(stmt);
auto* val = ret->getChild("value");
if (val && val->conceptType == "StringLiteral") {
hasStringLit = true;
}
}
}
assert(hasStringLit && "Should contain a Return with StringLiteral");
std::cout << "Test 8 PASS: C++ string literal parsed" << std::endl;
++passed;
}
// --- Summary ---
std::cout << "\n=== Step 48 Results: " << passed << " passed, " << failed << " failed ===" << std::endl;
return failed > 0 ? 1 : 0;
}

View File

@@ -0,0 +1,140 @@
// Step 49 TDD Test: Error Recovery and Diagnostics
//
// Tests the ParseResult and ParseDiagnostic types:
// 1. Empty source → empty module, no errors
// 2. Valid Python → module with functions, no diagnostics
// 3. Python syntax error → module + diagnostics with line/column
// 4. Partial parse (valid + broken function) → first function + diagnostics
// 5. C++ syntax error → diagnostics present
// 6. Elisp unbalanced parens → diagnostics present
// 7. ParseResult::hasErrors() returns correct boolean
#include <iostream>
#include <string>
#include <cassert>
#include <memory>
#include "ast/Parser.h"
#include "ast/Annotation.h"
int main() {
int passed = 0;
int failed = 0;
// --- Test 1: Empty source → empty module, no errors ---
{
std::string source = "";
auto result = TreeSitterParser::parsePythonWithDiagnostics(source);
assert(result.module != nullptr && "Module should not be null");
assert(result.module->targetLanguage == "python");
assert(result.module->getChildren("functions").empty() &&
"Empty source should have no functions");
assert(result.diagnostics.empty() &&
"Empty source should have no diagnostics");
assert(!result.hasErrors() && "Empty source should have no errors");
std::cout << "Test 1 PASS: Empty source → empty module, no errors" << std::endl;
++passed;
}
// --- Test 2: Valid Python → module with functions, no diagnostics ---
{
std::string source = "def f(x):\n return x + 1\n";
auto result = TreeSitterParser::parsePythonWithDiagnostics(source);
assert(result.module != nullptr);
auto functions = result.module->getChildren("functions");
assert(!functions.empty() && "Should have parsed function");
assert(result.diagnostics.empty() &&
"Valid source should have no diagnostics");
assert(!result.hasErrors());
std::cout << "Test 2 PASS: Valid Python → module with functions, no diagnostics" << std::endl;
++passed;
}
// --- Test 3: Python syntax error → diagnostics with line/column ---
{
std::string source = "def f(\n";
auto result = TreeSitterParser::parsePythonWithDiagnostics(source);
assert(result.module != nullptr && "Module should still be created");
assert(!result.diagnostics.empty() &&
"Syntax error should produce diagnostics");
assert(result.diagnostics[0].severity == "error");
assert(result.diagnostics[0].line >= 1 && "Line should be >= 1");
assert(result.diagnostics[0].column >= 1 && "Column should be >= 1");
std::cout << "Test 3 PASS: Python syntax error → diagnostics" << std::endl;
++passed;
}
// --- Test 4: Partial parse (valid + broken) → first function + diagnostics ---
{
std::string source =
"def good(x):\n"
" return x\n"
"\n"
"def bad(\n";
auto result = TreeSitterParser::parsePythonWithDiagnostics(source);
assert(result.module != nullptr);
auto functions = result.module->getChildren("functions");
assert(!functions.empty() &&
"Should have parsed at least the first valid function");
auto* fn = static_cast<Function*>(functions[0]);
assert(fn->name == "good" && "First function should be 'good'");
assert(!result.diagnostics.empty() &&
"Broken second function should produce diagnostics");
std::cout << "Test 4 PASS: Partial parse → first function + diagnostics" << std::endl;
++passed;
}
// --- Test 5: C++ syntax error → diagnostics present ---
{
std::string source = "int f(int x { return x; }";
auto result = TreeSitterParser::parseCppWithDiagnostics(source);
assert(result.module != nullptr);
assert(!result.diagnostics.empty() &&
"C++ syntax error should produce diagnostics");
std::cout << "Test 5 PASS: C++ syntax error → diagnostics present" << std::endl;
++passed;
}
// --- Test 6: Elisp unbalanced parens → diagnostics present ---
{
std::string source = "(defun f (x) (+ x 1)";
auto result = TreeSitterParser::parseElispWithDiagnostics(source);
assert(result.module != nullptr);
assert(!result.diagnostics.empty() &&
"Unbalanced parens should produce diagnostics");
std::cout << "Test 6 PASS: Elisp unbalanced parens → diagnostics present" << std::endl;
++passed;
}
// --- Test 7: ParseResult::hasErrors() returns correct boolean ---
{
// Valid source
auto goodResult = TreeSitterParser::parsePythonWithDiagnostics(
"def f(x):\n return x\n");
assert(!goodResult.hasErrors() && "Valid code should not have errors");
// Invalid source
auto badResult = TreeSitterParser::parsePythonWithDiagnostics(
"def f(\n");
assert(badResult.hasErrors() && "Invalid code should have errors");
std::cout << "Test 7 PASS: hasErrors() returns correct boolean" << std::endl;
++passed;
}
// --- Summary ---
std::cout << "\n=== Step 49 Results: " << passed << " passed, " << failed << " failed ===" << std::endl;
return failed > 0 ? 1 : 0;
}

View File

@@ -10,6 +10,7 @@
"$comment": "SDL2 backend compiled locally (sdl2-binding removed from vcpkg imgui 1.91+)"
},
"glad",
"opengl"
"opengl",
"tree-sitter"
]
}