bill/whetstone_DSL
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
41166812b03e4a29c8a82ace5db6840ba204098d
whetstone_DSL/PROGRESS.md
Bill 41166812b0 Phase 3b: Real tree-sitter integration (Steps 45-49) 
Replace TreeSitterParser stubs with real tree-sitter C bindings for
Python, C++, and Elisp parsing. Add tree-sitter core via vcpkg and
language grammars via FetchContent. Implement CST-to-AST converters
with auto-annotation, memory pattern detection, error recovery, and
ParseResult/ParseDiagnostic types. All 34 tests pass (6+7+6+8+7).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 09:06:50 -07:00

12 KiB
Raw Blame History

Whetstone DSL - Progress Tracker

Purpose: Living document for cross-session, cross-LLM continuity. Updated after each work session. The git log is the authoritative source of truth.

Project Overview

Whetstone is a semantic annotation DSL (SemAnno) and structured editor for cross-language code generation with memory strategy annotations. The core idea: annotate code with memory management intent (@Reclaim(Tracing), @Owner(Single), @Lifetime(RAII), etc.) and generate correct target-language code from a single AST representation.

Stack: C++20, Dear ImGui + SDL2 + OpenGL3, nlohmann-json, vcpkg, CMake

Sprint Summary

Sprint Steps Status Description
Sprint 1 — Complete MPS-based prototype (JetBrains MPS language plugin)
Sprint 2 1–38 Complete C++ editor stack: AST, serialization, generators, ImGui shell, orchestrator, agents
Sprint 3 39–75 In Progress Core functionality: C++ generator, tree-sitter, classical editing, optimization

Sprint 1: MPS Prototype (Complete)

Built the initial SemAnno language in JetBrains MPS:

  • 33 AST concepts (Module, Function, Variable, statements, expressions, types, annotations)
  • Editor definitions for all concepts
  • TextGen definitions for Python output
  • Phase1Example sandbox model

Key files: languages/SemAnno/, solutions/SemAnno.sandbox/

Sprint 2: C++ Editor Stack (Steps 1–38) — COMPLETE

All 38 steps implemented and passing. Each step has a corresponding test (step1_test.exe through step38_test.exe).

Phase 2a: AST in C++ (Steps 1–6)

  • Step 1: Base ASTNode class + Module concept
  • Step 2: Child links, Function and Variable
  • Step 3: All Statement, Expression, Type, Parameter concepts
  • Step 4: Annotation concepts (DerefStrategy, OptimizationLock, LangSpecific)
  • Step 5: JSON serialization (toJson)
  • Step 6: JSON deserialization + round-trip verification

Phase 2b: Validation & Generation (Steps 7–11)

  • Step 7: Schema validation (ASTSchema)
  • Step 8: Python generator base (Module/Function)
  • Step 9: Python generator — statements and expressions
  • Step 10: Python generator — remaining concepts
  • Step 11: Python generator — annotation output

Phase 2c: ImGui Editor Shell (Steps 12–15)

  • Step 12: Dear ImGui shell scaffolding (SDL2 + OpenGL3)
  • Step 13: Text viewport
  • Step 14: AST-to-text viewport
  • Step 15: Projection toggle

Phase 2d: Orchestrator (Steps 16–22)

  • Step 16: Orchestrator process
  • Step 17: JSON-RPC server
  • Step 18: AST mutation via RPC
  • Step 19: Connect ImGui to orchestrator
  • Step 20: Mutation via UI
  • Step 21: Undo/Redo journal
  • Step 22: Undo/Redo UI

Phase 2e: Emacs Integration (Steps 23–29)

  • Step 23: Emacs integration
  • Step 24: Orchestrator spawns Emacs
  • Step 25: AST-to-Elisp projection
  • Step 26: Orchestrator-to-Emacs RPC
  • Step 27: File operations via Emacs
  • Step 28: File operations via RPC
  • Step 29: AST-to-File synchronization

Phase 2f: Round-trip & Agents (Steps 30–38)

  • Step 30: Emacs-to-MPS synchronization
  • Step 31: Tree-sitter integration (stubs)
  • Step 32: Import via tree-sitter
  • Step 33: Export via generator
  • Step 34: Full round-trip / C++ generator basic output
  • Step 35: C++ generator — memory strategies
  • Step 36: Agent API
  • Step 37: Agent batch mode
  • Step 38: OptimizationLock warnings

Sprint 3: Core Functionality (Steps 39–75) — IN PROGRESS

Phase 3a: C++ Generator Implementation (Steps 39–44) — COMPLETE

  • Step 39: CppGenerator basic skeleton
  • Step 40: Statement generation (Assignment, Return, If)
  • Step 41: Expression generation (BinaryOp, literals, precedence)
  • Step 42: Type generation (PrimitiveType → C++ types, containers)
  • Step 43: Memory strategy code generation (canonical annotations)
  • Step 44: C++-specific idioms (const, references, includes)

Steps 39–44 have TDD tests that compile and pass. The CppGenerator and canonical memory annotation classes (DeallocateAnnotation, LifetimeAnnotation, ReclaimAnnotation, OwnerAnnotation, AllocateAnnotation) are implemented in editor/src/ast/Generator.h.

Phase 3b: Tree-sitter Integration (Steps 45–49) — COMPLETE

  • Step 45: Real tree-sitter Python parsing (6/6 tests pass)
  • Step 46: Real tree-sitter C++ parsing with memory pattern detection (7/7 tests pass)
  • Step 47: Real tree-sitter Elisp parsing + cross-language equivalence (6/6 tests pass)
  • Step 48: CST-to-AST mapping refinement — default params, if/for, multi-statement, void, multi-function, string literals (8/8 tests pass)
  • Step 49: Error recovery and diagnostics — ParseResult/ParseDiagnostic types, partial parse, hasErrors() (7/7 tests pass)

Tree-sitter core via vcpkg (tree-sitter 0.26.5). Language grammars via FetchContent: tree-sitter-python v0.23.6, tree-sitter-cpp v0.23.4, tree-sitter-elisp 1.3.0. TreeSitterParser in Parser.h implements full CST-to-AST conversion for all three languages with auto-annotation (Python/Elisp → @Reclaim(Tracing), C++ → memory pattern detection).

Phase 3c: Classical Editing Mode (Steps 50–54) — TDD STUBS ONLY

  • Step 50: Text editor component — not started
  • Step 51: TDD test written (does not link — depends on unimplemented code)
  • Step 52: Syntax highlighting — not started
  • Step 53: TDD test written (does not link)
  • Step 54: Emacs-style keybindings — not started

Phase 3d: Emacs Integration Complete (Steps 55–58) — TDD STUBS ONLY

  • Step 55: Emacs splash screen — not started
  • Step 56: TDD test written (does not link)
  • Step 57: TDD test written (does not link)
  • Step 58: Mode-specific behavior — not started

Phase 3e: Agent API Extend (Steps 59–63) — TDD STUBS ONLY

  • Step 59: WebSocket agent endpoint — not started
  • Step 60: IMPLEMENTED — ASTQueryAPI with getAST, findNodesByType/Property/Annotation, getSubtree (8/8 tests pass)
  • Step 61: TDD test written (does not link)
  • Step 62: TDD test written (does not link)
  • Step 63: TDD test written (does not link)

Phase 3f: Advanced Memory Management (Steps 64–67) — TDD STUBS ONLY

  • Step 64: TDD test written (does not link)
  • Step 65: TDD test written (does not link)
  • Step 66: TDD test written (does not link)
  • Step 67: TDD test written (compile errors — references undefined types: HotColdAnnotation, InlineAnnotation, etc.)

Phase 3g: Optimization Pipeline (Steps 68–71) — TDD STUBS ONLY

  • Step 68: TDD test written (does not link)
  • Step 69: TDD test written (does not link)
  • Step 70: TDD test written (does not link)
  • Step 71: TDD test written (does not link)

Phase 3h: Integration & Validation (Steps 72–75) — NOT STARTED

  • Step 72: End-to-end pipeline test
  • Step 73: Error handling and edge cases
  • Step 74: Performance profiling and benchmarks
  • Step 75: API documentation

Build Infrastructure

Created in the most recent session:

File Purpose
editor/vcpkg.json vcpkg manifest (dependencies declaration)
editor/CMakePresets.json Standardized CMake presets (Windows/Linux)
installer/windows/build.ps1 PowerShell build script (prerequisites, vcpkg, cmake, staging)
installer/windows/setup.iss Inno Setup installer script (produces .exe installer)
installer/linux/build.sh Bash build script for Linux
installer/linux/install.sh Linux install script (system deps, desktop entry)

Building on Windows

# Prerequisites: VS2022, CMake 3.20+, vcpkg at E:\vcpkg
cd E:\Whetstone_DSL\installer\windows
.\build.ps1 -Config Release -VcpkgRoot E:\vcpkg

# Then compile installer (requires Inno Setup 6):
& "C:\Users\Bill\AppData\Local\Programs\Inno Setup 6\ISCC.exe" setup.iss

vcpkg Packages Required

  • nlohmann-json:x64-windows
  • sdl2:x64-windows
  • imgui[docking-experimental,opengl3-binding]:x64-windows
  • glad:x64-windows
  • tree-sitter:x64-windows

Known Issue: imgui SDL2 Backend

vcpkg's imgui 1.91.9 removed the sdl2-binding feature (only sdl3-binding exists). The SDL2 backend files are vendored locally in editor/src/imgui_backends/ and compiled directly by CMake.

Test Results (Last Verified)

Steps 1–49: All compile and pass (49 executables in editor/build/Release/) Steps 50–75: Either not started or TDD stubs that don't link yet

The build compiles whetstone_editor, orchestrator, and steps 1–49. Steps 45–49 link against tree-sitter core + grammar static libraries. Steps 51+ are TDD stubs that reference unimplemented code and will fail to link until their implementations exist.

Key Source Files

File Contents
editor/src/ast/ASTNode.h All 33+ AST node classes, JSON serialization
editor/src/ast/Generator.h PythonGenerator, CppGenerator, ElispGenerator, all canonical annotation classes
editor/src/ast/Schema.h AST schema validation
editor/src/ast/Parser.h TreeSitterParser (stub implementations)
editor/src/Orchestrator.h Orchestrator: Emacs integration, file ops, undo/redo, agent API
editor/src/main.cpp ImGui editor shell (SDL2 + OpenGL3)
editor/src/orchestrator_main.cpp Orchestrator standalone process (JSON-RPC server)
editor/CMakeLists.txt CMake build config (vcpkg-based)

Architecture Notes

  • Editor (whetstone_editor.exe): ImGui-based GUI. Renders AST as structured editor panels with file tree, toolbar, projection toggles. Currently a Step 12 scaffold — functional but minimal.
  • Orchestrator (orchestrator.exe): Standalone JSON-RPC server. Manages AST state, undo/redo journal, file I/O, Emacs daemon integration, agent API.
  • Communication: Editor ↔ Orchestrator via JSON-RPC over stdin/stdout pipes. When launched standalone (no pipe), the editor runs in disconnected mode.
  • Generators: AST → Python, C++, Elisp text output. C++ generator handles canonical memory annotations.
  • Parsers: Text → AST via tree-sitter (currently stub implementations from Sprint 2).

What's Next

Phase 3b (tree-sitter integration) is complete. Next logical work:

  • Phase 3c (Steps 50–54): Classical text editing mode (text editor component, syntax highlighting, keybindings)
  • Phase 3e (Steps 59–63): WebSocket agent API (Step 60 already done, remaining: WebSocket endpoint, agent protocol)
  • Phase 3d (Steps 55–58): Emacs integration completion (splash screen, mode-specific behavior)

Session Log

Date Agent Work Done
Pre-Sprint 2 Multiple Sprint 1 MPS prototype, language definitions, textgen
Sprint 2 Multiple Steps 1–38 implemented sequentially
Sprint 3 start Unknown Sprint 3 plan written, TDD stubs for steps 39–71 committed
Sprint 3 Unknown Phase 3a steps 39–44 implemented (CppGenerator + canonical annotations)
Sprint 3 Unknown Steps 34–38 re-committed with different implementations (agent API, batch mode)
2025-latest Claude Build infrastructure: vcpkg.json, CMakePresets.json, build.ps1, setup.iss, Linux scripts
2025-latest Claude Fixed 7 build errors (ElispGenerator, Orchestrator, main.cpp, orchestrator_main.cpp)
2025-latest Claude Vendored imgui SDL2 backend (vcpkg removed sdl2-binding feature)
2025-latest Claude Created Windows installer (Inno Setup), verified it installs and runs
2025-latest Claude Fixed editor hang when launched without orchestrator pipe
2026-02-07 Claude Opus 4.6 Step 60: Implemented ASTQueryAPI (tree walk, JSON output, find by type/property/annotation). 8/8 tests pass. Added PROGRESS.md.
2026-02-08 Claude Opus 4.6 Phase 3b: Real tree-sitter integration (Steps 45–49). Replaced Parser.h stubs with real tree-sitter C bindings. Python/C++/Elisp CST-to-AST conversion, memory pattern detection, error recovery. 34/34 tests pass.
Reference in New Issue View Git Blame Copy Permalink