whetstone_DSL/sprint43_plan.md

Sprint 43 Plan: Cross-File Context Assembly

Context

Sprint 42 completes the backlog of unregistered MCP tools (tool count 84→86). The architect intake pipeline (Sprints 32, 36) generates annotated taskitems with prerequisiteOps lists — file paths and symbol names the executing agent needs. Currently, agents read full files to satisfy those ops, negating much of the token budget advantage taskitems are designed to create.

Sprint 43 adds surgical context slicing: given a taskitem's prerequisiteOps, extract only the relevant declarations, function bodies, or type definitions from each file — not the whole file. Respects a configurable token budget. Entirely generic — no coupling to any external system.

---

Goals

1. Index workspace files by symbol definition (fast lookup by name, not full-file scan)
2. Assemble minimal context slices for a given set of file paths + symbol names
3. Enforce a token budget — trim to fit, report what was dropped
4. Expose as whetstone_assemble_context MCP tool

---

Steps

Step 674: WorkspaceFileIndex — symbol definition index (12 tests)

Create editor/src/WorkspaceFileIndex.h.

Index all files under a workspace root. For each file, extract top-level declarations (function signatures, class names, struct names, type aliases, constants) using line-level heuristics — no full parse required for this index.

struct SymbolLocation {
    std::string file;
    int         lineStart;
    int         lineEnd;    // end of signature or short body
    std::string symbolName;
    std::string symbolKind; // "function" | "class" | "struct" | "type" | "const"
};

class WorkspaceFileIndex {
public:
    static WorkspaceFileIndex build(const std::string& workspaceRoot);
    std::vector<SymbolLocation> find(const std::string& symbolName) const;
    std::vector<SymbolLocation> findInFile(const std::string& filePath) const;
    int fileCount() const;
    int symbolCount() const;
};

Tests (12): empty root returns empty index, single-file workspace indexes it, function signatures found by name, class declarations found by name, struct declarations found by name, findInFile returns all symbols in that file, find() returns all matches across files, symbolCount > 0 after indexing real file, fileCount matches input file count, duplicate symbol names all returned, build() doesn't throw on unreadable file (skips gracefully), kind field is non-empty.

Step 675: ContextSliceAssembler — extract minimal file sections (12 tests)

Create editor/src/ContextSliceAssembler.h.

Given a list of {file, symbol_or_line_range} requests and a workspace index, extract only the relevant lines from each file. A "slice" is:

• For a named symbol: the lines from its declaration through the closing brace/semicolon
• For a line range: exactly those lines
• For a file with no symbol specified: first N lines (configurable, default 40)

struct SliceRequest {
    std::string filePath;
    std::string symbolName; // empty = use lineHint or head
    int         lineHint;   // 0 = not specified
    int         headLines;  // fallback if no symbol match; default 40
};

struct ContextSlice {
    std::string filePath;
    std::string symbolName;
    int         lineStart;
    int         lineEnd;
    std::string content;    // the actual extracted lines
    bool        found;
};

class ContextSliceAssembler {
public:
    static std::vector<ContextSlice> assemble(
        const std::vector<SliceRequest>& requests,
        const WorkspaceFileIndex& index);
};

Tests (12): empty requests returns empty result, named symbol found returns correct lines, unknown symbol returns head fallback, line range returns correct lines, content non-empty on success, found==false when symbol missing, multiple requests all processed, head fallback respects headLines parameter, function body slice includes closing brace, class slice includes closing brace, file path preserved in result, slice content is valid UTF-8 substring of source file.

Step 676: TokenBudgetEnforcer — trim assembled context to budget (12 tests)

Create editor/src/TokenBudgetEnforcer.h.

Given a list of ContextSlice objects and a max token budget, trim the list to fit. Strategy: priority order (slices earlier in the list are higher priority), each slice is dropped whole if it doesn't fit. Returns the slices that fit plus a report of what was dropped.

Token estimation: 1 token ≈ 4 characters (conservative approximation, no model dependency). The enforcer does NOT call any model API.

struct BudgetReport {
    int    totalSlices;
    int    includedSlices;
    int    droppedSlices;
    int    estimatedTokensUsed;
    int    budgetTokens;
    bool   budgetExceeded;   // true if any slice was dropped
    std::vector<std::string> droppedFiles;
};

class TokenBudgetEnforcer {
public:
    static std::pair<std::vector<ContextSlice>, BudgetReport>
        enforce(const std::vector<ContextSlice>& slices, int maxTokens);

    static int estimateTokens(const std::string& text);
};

Tests (12): empty slices returns empty + zero report, single slice under budget kept, single slice over budget dropped, multiple slices: high-priority kept when budget tight, droppedFiles list matches dropped slices, estimatedTokensUsed <= budgetTokens when no drops, budgetExceeded==false when all fit, budgetExceeded==true when any dropped, estimateTokens("") == 0, estimateTokens("abcd") == 1, report.totalSlices == input count, report.includedSlices + report.droppedSlices == report.totalSlices.

Step 677: whetstone_assemble_context MCP tool (8 tests)

Add to RegisterCodegenTools.h (or new RegisterContextTools.h if file nears 600 lines).

Input schema:

{
  "files": {
    "type": "array",
    "description": "List of slice requests.",
    "items": {
      "type": "object",
      "properties": {
        "path":        { "type": "string",  "description": "File path relative to workspace." },
        "symbol":      { "type": "string",  "description": "Symbol name to extract (optional)." },
        "line_hint":   { "type": "integer", "description": "Line number hint (optional)." },
        "head_lines":  { "type": "integer", "description": "Head fallback line count (optional, default 40)." }
      },
      "required": ["path"]
    }
  },
  "max_tokens": { "type": "integer", "description": "Token budget (default 8000)." }
}

Output:

{
  "success": true,
  "slices": [
    { "file": "...", "symbol": "...", "line_start": 10, "line_end": 25, "content": "..." }
  ],
  "budget_report": {
    "total_slices": 3,
    "included_slices": 2,
    "dropped_slices": 1,
    "estimated_tokens_used": 1840,
    "budget_tokens": 2000,
    "budget_exceeded": true,
    "dropped_files": ["large_file.cpp"]
  }
}

Wire into MCPServer.h, RegisterOnboardingAndAllTools.h, and tools.json. Tool count 86→87.

Tests (8): tool registered, empty files array returns success with empty slices, valid file path returns slice, missing path arg returns error, symbol slicing returns content, budget_report present in output, budget_exceeded==false when all slices fit, tool count in tools.json == 87.

Step 678: Sprint 43 Integration Summary (8 tests)

Create editor/src/Sprint43IntegrationSummary.h. Record: tool_count_before=86, tool_count_after=87, steps_completed=5 (674-678), files_added=[WorkspaceFileIndex.h, ContextSliceAssembler.h, TokenBudgetEnforcer.h, Sprint43IntegrationSummary.h], success=true.

Tests (8): struct constructable, workspace_index_builds, slice_assembler_works, budget_enforcer_works, tool_count_before==86, tool_count_after==87, steps_completed==5, success==true.

---

Architecture Gate (All Steps)

• Header-only C++ for all new source files (no new .cpp except test files)
• 600-line max per file
• No new vcpkg dependencies — uses only std::filesystem and string operations
• estimateTokens must not call any external model API
• All slicing is done on raw file text — no AST dependency (fast, portable)
• Full sprint regression at step 678