WIP: stage all uncommitted work — sprints 46-221, graduation headers, specialist fleet, test steps 909-1988
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
99
docs/AGENT_HANDOFF_2026-02-24.md
Normal file
99
docs/AGENT_HANDOFF_2026-02-24.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Agent Handoff — 2026-02-24
|
||||
|
||||
## Scope of this handoff
|
||||
This handoff captures current execution state for sprint work discussed in-session, focused on Sprint 146 through Sprint 155.
|
||||
|
||||
## Completed and verified
|
||||
- Sprint 146 implementation is complete and validated.
|
||||
- Added Sprint 146 graduation models:
|
||||
- `editor/src/graduation/HybridCoreContractSchemaV1.h`
|
||||
- `editor/src/graduation/LanguageCapabilityMatrixModel.h`
|
||||
- `editor/src/graduation/CapabilityNegotiationFallbackPolicyModel.h`
|
||||
- `editor/src/graduation/LanguageNeutralOperationEnvelopeModel.h`
|
||||
- `editor/src/graduation/CapabilityDecisionPacketModel.h`
|
||||
- `editor/src/graduation/LanguageCapabilityConformanceReportArtifact.h`
|
||||
- Added Sprint 146 MCP tool registration:
|
||||
- `editor/src/mcp/RegisterSprint146Tools.h`
|
||||
- Tools: `whetstone_list_language_capabilities`, `whetstone_get_language_contract`, `whetstone_validate_language_operation`
|
||||
- Added integration summary:
|
||||
- `editor/src/Sprint146IntegrationSummary.h`
|
||||
- Wiring completed:
|
||||
- `editor/src/MCPServer.h` include for `RegisterSprint146Tools.h`
|
||||
- `editor/src/mcp/RegisterOnboardingAndAllTools.h` calls `registerSprint146Tools()`
|
||||
- `editor/CMakeLists.txt` includes tests `step1699_test` through `step1708_test`
|
||||
- Sprint 146 tests added and passing:
|
||||
- `editor/tests/step1699_test.cpp` through `editor/tests/step1708_test.cpp`
|
||||
- All step1699–1708 binaries executed and passed in-session.
|
||||
|
||||
## Generated but not yet validated
|
||||
Sprints 147–155 were scaffolded (headers/tests/wiring) but full build+run validation has not been executed yet.
|
||||
|
||||
### Artifacts generated
|
||||
- MCP register headers:
|
||||
- `editor/src/mcp/RegisterSprint147Tools.h` … `RegisterSprint155Tools.h`
|
||||
- Integration summaries:
|
||||
- `editor/src/Sprint147IntegrationSummary.h` … `Sprint155IntegrationSummary.h`
|
||||
- Graduation/model/report headers created for each sprint step family:
|
||||
- 1709, 1710, 1711, 1712, 1716, 1717
|
||||
- 1719, 1720, 1721, 1722, 1726, 1727
|
||||
- 1729, 1730, 1731, 1732, 1736, 1737
|
||||
- 1739, 1740, 1741, 1742, 1746, 1747
|
||||
- 1749, 1750, 1751, 1752, 1756, 1757
|
||||
- 1759, 1760, 1761, 1762, 1766, 1767
|
||||
- 1769, 1770, 1771, 1772, 1776, 1777
|
||||
- 1779, 1780, 1781, 1782, 1786, 1787
|
||||
- 1789, 1790, 1791, 1792, 1796, 1797
|
||||
- Tests generated:
|
||||
- `editor/tests/step1709_test.cpp` … `editor/tests/step1798_test.cpp`
|
||||
|
||||
### Wiring updates applied for 147–155
|
||||
- `editor/src/MCPServer.h`
|
||||
- Includes added for `RegisterSprint147Tools.h` … `RegisterSprint155Tools.h`
|
||||
- `editor/src/mcp/RegisterOnboardingAndAllTools.h`
|
||||
- Calls added for `registerSprint147Tools();` … `registerSprint155Tools();`
|
||||
- `editor/CMakeLists.txt`
|
||||
- Test targets appended for `step1709_test` … `step1798_test`
|
||||
|
||||
## Runtime/executor hardening completed (separate but relevant)
|
||||
- `tools/mcp/ollama_mcp_taskitem_executor.sh` hardened to reduce false failures:
|
||||
- Added robust JSON fallback handling for `--argjson` inputs.
|
||||
- Added safer parsing for native-tools chat response fields.
|
||||
- Added deterministic MCP fallback path when model emits no tool output.
|
||||
- Post-hardening batch run reported `ok` statuses for taskitems mapped to sprints 141–155 in session logs.
|
||||
|
||||
## Validation update (2026-02-25)
|
||||
- Reconfigured editor build in Debug mode:
|
||||
- `cmake -S /home/bill/Documents/CLionProjects/whetstone_DSL/editor -B /home/bill/Documents/CLionProjects/whetstone_DSL/editor/build-native -DCMAKE_BUILD_TYPE=Debug`
|
||||
- Built and executed sprint integration checks:
|
||||
- `step1718_test`, `step1728_test`, `step1738_test`, `step1748_test`, `step1758_test`, `step1768_test`, `step1778_test`, `step1788_test`, `step1798_test`
|
||||
- Result: all PASS
|
||||
- Built and executed MCP tool-surface checks:
|
||||
- `step1713_test`, `step1723_test`, `step1733_test`, `step1743_test`, `step1753_test`, `step1763_test`, `step1773_test`, `step1783_test`, `step1793_test`
|
||||
- `step1714_test`, `step1724_test`, `step1734_test`, `step1744_test`, `step1754_test`, `step1764_test`, `step1774_test`, `step1784_test`, `step1794_test`
|
||||
- `step1715_test`, `step1725_test`, `step1735_test`, `step1745_test`, `step1755_test`, `step1765_test`, `step1775_test`, `step1785_test`, `step1795_test`
|
||||
- Result: all PASS
|
||||
- Built and executed all remaining step tests:
|
||||
- `step1709_test`, `step1710_test`, `step1711_test`, `step1712_test`, `step1716_test`, `step1717_test`
|
||||
- `step1719_test`, `step1720_test`, `step1721_test`, `step1722_test`, `step1726_test`, `step1727_test`, `step1729_test`
|
||||
- `step1730_test`, `step1731_test`, `step1732_test`, `step1736_test`, `step1737_test`, `step1739_test`
|
||||
- `step1740_test`, `step1741_test`, `step1742_test`, `step1746_test`, `step1747_test`, `step1749_test`
|
||||
- `step1750_test`, `step1751_test`, `step1752_test`, `step1756_test`, `step1757_test`, `step1759_test`
|
||||
- `step1760_test`, `step1761_test`, `step1762_test`, `step1766_test`, `step1767_test`, `step1769_test`
|
||||
- `step1770_test`, `step1771_test`, `step1772_test`, `step1776_test`, `step1777_test`, `step1779_test`
|
||||
- `step1780_test`, `step1781_test`, `step1782_test`, `step1786_test`, `step1787_test`, `step1789_test`
|
||||
- `step1790_test`, `step1791_test`, `step1792_test`, `step1796_test`, `step1797_test`
|
||||
- Result: 54/54 PASS
|
||||
- Final closure summary for Sprint 147–155 step range:
|
||||
- `step1709_test` through `step1798_test`: 90/90 PASS
|
||||
|
||||
## Current blocking state
|
||||
- None for Sprint 147–155 validation scope.
|
||||
- Compile/runtime closure complete for `step1709_test` through `step1798_test`.
|
||||
|
||||
## Resume checklist (next agent)
|
||||
1. Keep this result as the known-good baseline for Sprints 147–155.
|
||||
2. Proceed to next sprint scope or broader regression suites as needed.
|
||||
|
||||
## Notes
|
||||
- Repo is dirty with pre-existing unrelated changes/untracked files; do not reset/revert globally.
|
||||
- No destructive git operations were used.
|
||||
129
docs/DETERMINIZATION_NOTES_whimptk_bootstrap.md
Normal file
129
docs/DETERMINIZATION_NOTES_whimptk_bootstrap.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Determinization Notes: `whimptk` Bootstrap
|
||||
|
||||
This note belongs in Whetstone because the high-entropy stages are properties of
|
||||
the bootstrap and MCP workflow, not of `whimptk` itself.
|
||||
|
||||
The purpose is to identify places where open-ended model behavior should be
|
||||
collapsed into deterministic schemas, validators, or executable transforms.
|
||||
|
||||
## Current Bias
|
||||
|
||||
If a stage can be expressed as:
|
||||
|
||||
- a finite schema
|
||||
- a bounded classification problem
|
||||
- a deterministic transformation
|
||||
- a constrained verification routine
|
||||
|
||||
then that stage should move in that direction instead of remaining a broad
|
||||
prompt.
|
||||
|
||||
## High-Entropy Stages Exposed By `whimptk`
|
||||
|
||||
### 1. Freeform sprint/spec intake
|
||||
|
||||
Risk:
|
||||
|
||||
- broad prose lets the model overfit to incidental wording
|
||||
- key architectural constraints get flattened or lost
|
||||
|
||||
Observed constraints that need explicit structure:
|
||||
|
||||
- widget-first primitive order
|
||||
- optional binding presentation
|
||||
- separation between WHIMP policy and toolkit implementation
|
||||
- non-desktop future applicability
|
||||
|
||||
Deterministic direction:
|
||||
|
||||
- define a primitive-bootstrap schema with fields such as:
|
||||
- `primitive_order`
|
||||
- `required_types`
|
||||
- `optional_types`
|
||||
- `policy_constraints`
|
||||
- `non_goals`
|
||||
- `verification_commands`
|
||||
|
||||
Short-term operational rule:
|
||||
|
||||
- restate these fields explicitly before taskitem generation
|
||||
|
||||
### 2. Taskitem generation from generic intake
|
||||
|
||||
Risk:
|
||||
|
||||
- broad tasks
|
||||
- wrong-project paths
|
||||
- invalid verification commands
|
||||
- generic planning buckets instead of execution slices
|
||||
|
||||
Deterministic direction:
|
||||
|
||||
- enforce a taskitem contract:
|
||||
- one bounded objective
|
||||
- real target files
|
||||
- real acceptance commands
|
||||
- no cross-repo ambiguity
|
||||
- explicit ownership slice
|
||||
|
||||
Short-term operational rule:
|
||||
|
||||
- reject or manually correct any taskitem set that violates the contract
|
||||
- preserve raw and corrected artifacts as training data
|
||||
|
||||
### 3. Code generation requests that are too large
|
||||
|
||||
Risk:
|
||||
|
||||
- asking for a whole subsystem invites placeholder output
|
||||
- apparent success can hide missing structure
|
||||
|
||||
Deterministic direction:
|
||||
|
||||
- constrain codegen asks to one artifact at a time:
|
||||
- one type
|
||||
- one header
|
||||
- one function
|
||||
- one validator
|
||||
- one test file
|
||||
|
||||
Short-term operational rule:
|
||||
|
||||
- split cross-abstraction requests into explicit file-level contracts first
|
||||
|
||||
### 4. Implicit architectural assumptions
|
||||
|
||||
Risk:
|
||||
|
||||
- the model may silently reinterpret `whimptk` as a desktop-only widget library
|
||||
- the model may blur toolkit and compositor responsibilities
|
||||
|
||||
Deterministic direction:
|
||||
|
||||
- add preflight acceptance checks for:
|
||||
- policy remains above implementation
|
||||
- `whimpwm` is not treated as the toolkit
|
||||
- binding presentation remains optional
|
||||
|
||||
Short-term operational rule:
|
||||
|
||||
- keep these checks in bootstrap review until Whetstone can validate them
|
||||
|
||||
## Candidate Whetstone Improvements
|
||||
|
||||
- a primitive-bootstrap JSON schema for toolkit bring-up
|
||||
- a taskitem linter for wrong-project paths and invalid verification commands
|
||||
- a repo-constraint injection layer for taskitem generation
|
||||
- a validator for primitive dependency ordering
|
||||
- a registry of approved acceptance command patterns per repo/language
|
||||
|
||||
## Review Rule
|
||||
|
||||
Whenever a Whetstone stage is corrected manually, ask:
|
||||
|
||||
1. Was the failure caused by underspecified input?
|
||||
2. Can the missing structure be expressed as a schema or checklist?
|
||||
3. Can future runs reject the bad output automatically?
|
||||
|
||||
If yes, this should become Whetstone product work rather than remain a local
|
||||
operator habit.
|
||||
114
docs/PROJECT_PORTFOLIO_WORKFLOW_2026-02-26.md
Normal file
114
docs/PROJECT_PORTFOLIO_WORKFLOW_2026-02-26.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# Project Portfolio Workflow (2026-02-26)
|
||||
|
||||
## Purpose
|
||||
Organize long-horizon projects into one execution system so Whetstone can either:
|
||||
- finish existing MVPs from other machines/repos, or
|
||||
- rebuild from scratch with deterministic quality gates.
|
||||
|
||||
## Current Inputs Found
|
||||
- Whetstone core planning and sprint execution docs under `CLionProjects/whetstone_DSL/docs/`
|
||||
- A project archive bundle under `drive_projects/drive-download-20260217T155135Z-1-001/` (mixed docs, requirements, roadmaps, MVP concepts)
|
||||
- No literal `project_docs/` directory found under `/home/bill/Documents`
|
||||
|
||||
## Portfolio Lanes
|
||||
1. `MVP_IMPORT`
|
||||
- Existing codebase exists elsewhere (GitHub/private machine).
|
||||
- Goal: ingest, normalize, and harden to production thresholds.
|
||||
|
||||
2. `GREENFIELD`
|
||||
- No code to import or import is lower value than rebuild.
|
||||
- Goal: generate from architecture + constraints and iterate with strict validation.
|
||||
|
||||
## Canonical Project Record (one row per project)
|
||||
Use one table (or JSONL) with these required fields:
|
||||
- `project_id`
|
||||
- `name`
|
||||
- `lane` (`MVP_IMPORT` or `GREENFIELD`)
|
||||
- `source_of_truth` (repo URL / machine path / doc path)
|
||||
- `problem_statement`
|
||||
- `target_users`
|
||||
- `critical_constraints` (latency, memory, security, platform)
|
||||
- `status` (`intake`, `specifying`, `building`, `hardening`, `release_ready`, `paused`)
|
||||
- `next_gate`
|
||||
- `owner` (human/agent)
|
||||
- `last_validated_at`
|
||||
|
||||
## Intake Gate (must pass before any build work)
|
||||
1. Confirm the project has one primary source of truth.
|
||||
2. Convert intent into a compact execution spec:
|
||||
- target runtime/platform
|
||||
- must-have features
|
||||
- no-go constraints
|
||||
- acceptance tests
|
||||
3. Choose lane (`MVP_IMPORT` vs `GREENFIELD`) explicitly.
|
||||
4. Create first 3-7 taskitems with deterministic completion checks.
|
||||
|
||||
## Lane Workflows
|
||||
|
||||
### A) MVP_IMPORT Workflow
|
||||
1. `Snapshot`
|
||||
- Pull/clone code and freeze baseline hash.
|
||||
- Capture run/build/test state as-is.
|
||||
|
||||
2. `Normalize`
|
||||
- Align repo layout, dependency pinning, and CI test entrypoints.
|
||||
- Add missing schema/contract tests for core behavior.
|
||||
|
||||
3. `Gap Audit`
|
||||
- Compare MVP behavior vs current intent docs.
|
||||
- Tag each gap: `bug`, `missing feature`, `quality`, `security`, `scalability`.
|
||||
|
||||
4. `Hardening`
|
||||
- Run strict taskitem pipeline on highest-risk gaps first.
|
||||
- Require failing test first for each fix.
|
||||
|
||||
5. `Promotion`
|
||||
- Promote only after acceptance suite and non-functional checks pass.
|
||||
|
||||
### B) GREENFIELD Workflow
|
||||
1. `Architecture Freeze v0`
|
||||
- Generate project skeleton + module contracts + acceptance tests.
|
||||
|
||||
2. `Execution Slices`
|
||||
- Build in thin vertical slices with runnable tests every slice.
|
||||
|
||||
3. `Constraint Enforcement`
|
||||
- Ensure projection/environment constraints are first-class in taskitems.
|
||||
|
||||
4. `Hardening`
|
||||
- Reliability, determinism, performance, and security passes.
|
||||
|
||||
5. `Promotion`
|
||||
- Same production gates as import lane.
|
||||
|
||||
## Readiness Score (0-5 each, promote at >= 20/25)
|
||||
- `Spec Clarity`
|
||||
- `Deterministic Test Coverage`
|
||||
- `Operational Readiness` (build/release/runbook)
|
||||
- `Security Posture`
|
||||
- `Performance Budget Confidence`
|
||||
|
||||
## Weekly Cadence
|
||||
1. Intake/triage (new or updated projects)
|
||||
2. Lane assignment and top-3 portfolio priorities
|
||||
3. Whetstone execution runs (taskitem batches)
|
||||
4. Human review on blockers and lock-level decisions
|
||||
5. Closeout: score update + next gate assignment
|
||||
|
||||
## Suggested Immediate Backlog
|
||||
1. Create `project_registry.jsonl` in Whetstone docs/data.
|
||||
2. Import project list from your planning docs (especially the drive archive and any external repos).
|
||||
3. For each project, set lane and next gate.
|
||||
4. Start with 3 projects only:
|
||||
- one `MVP_IMPORT` with highest near-term value
|
||||
- one `GREENFIELD` infrastructure library
|
||||
- one `GREENFIELD` demonstrator product
|
||||
5. Run strict taskitem generation for those 3 and calibrate thresholds before scaling portfolio width.
|
||||
|
||||
## Notes From Existing Whetstone State
|
||||
- You already have mature sprint/taskitem infrastructure and closeout artifacts.
|
||||
- Active gap logs indicate the main risk now is execution quality drift, not idea scarcity.
|
||||
- Portfolio organization should therefore optimize for:
|
||||
- deterministic closeout quality
|
||||
- clear source-of-truth linkage
|
||||
- controlled concurrent project count
|
||||
70
docs/TOOL_CONTRACT_V1_IMPLEMENTATION_CHECKLIST.md
Normal file
70
docs/TOOL_CONTRACT_V1_IMPLEMENTATION_CHECKLIST.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# Whetstone: Tool Contract v1 Implementation Checklist
|
||||
|
||||
Date: 2026-02-26
|
||||
Target: L2 compliance (as a standalone tool and Hivemind-integrated tool)
|
||||
Contract source: `/home/bill/Documents/drive_projects/drive-download-20260217T155135Z-1-001/TOOL_CONTRACT_V1.md`
|
||||
|
||||
## Scope
|
||||
|
||||
Whetstone should present itself as a modular tool with stable contract surfaces, independent of Hivemind internals.
|
||||
|
||||
## L0 (Manifest + Health)
|
||||
|
||||
- [ ] Define `tool.manifest` for Whetstone tool server/MCP layer
|
||||
- [ ] Include required fields:
|
||||
- `tool_id = whetstone_editor`
|
||||
- `tool_name = Whetstone Editor Tool`
|
||||
- `contract_version = 1.0.0`
|
||||
- `tool_version`
|
||||
- `modes = ["standalone", "orchestrated"]`
|
||||
- `capabilities` from supported MCP operations
|
||||
- [ ] Source schemas from [whetstone_tool_schemas.json](/home/bill/Documents/CLionProjects/whetstone_DSL/tools/mcp/whetstone_tool_schemas.json)
|
||||
- [ ] Add/standardize health endpoint in MCP/runtime layer
|
||||
- [ ] Add contract L0 tests under [editor/tests](/home/bill/Documents/CLionProjects/whetstone_DSL/editor/tests)
|
||||
|
||||
## L1 (Execute + Typed Errors)
|
||||
|
||||
- [ ] Add `tool.execute` envelope adapter around existing MCP calls
|
||||
- [ ] Ensure execution returns standard contract response envelope
|
||||
- [ ] Normalize MCP/tool errors into typed contract errors:
|
||||
- `INVALID_INPUT`, `POLICY_BLOCKED`, `DEPENDENCY_UNAVAILABLE`, `TIMEOUT`, `INTERNAL_ERROR`
|
||||
- [ ] Add idempotency support where operations are retryable
|
||||
- [ ] Add L1 tests validating envelope shape and error mapping
|
||||
|
||||
## L2 (Validate + Policy + Observability)
|
||||
|
||||
- [ ] Implement `tool.validate` preflight for selected high-impact capabilities first
|
||||
- [ ] Return: `is_valid`, `violations`, `required_approvals`, `estimated_cost`, `estimated_risk`
|
||||
- [ ] Emit policy metadata per call:
|
||||
- `risk_level`, `impact_scope`, `requires_human_review`, `blocked_reasons`
|
||||
- [ ] Add trace propagation and structured logging in MCP execution path
|
||||
- [ ] Add metrics for capability success/failure and latency distribution
|
||||
- [ ] Add L2 tests for validate/policy/trace behavior
|
||||
|
||||
## Decoupling Rules (Critical)
|
||||
|
||||
- [ ] No Hivemind-specific assumptions in core Whetstone execution paths
|
||||
- [ ] Hivemind integration via contract adapter only
|
||||
- [ ] Keep standalone CLI/local execution path fully functional without Hivemind
|
||||
|
||||
## Candidate Integration Files
|
||||
|
||||
Use these as first implementation touchpoints:
|
||||
- [tools/mcp/README.md](/home/bill/Documents/CLionProjects/whetstone_DSL/tools/mcp/README.md)
|
||||
- [tools/mcp/whetstone_tool_schemas.json](/home/bill/Documents/CLionProjects/whetstone_DSL/tools/mcp/whetstone_tool_schemas.json)
|
||||
- [editor/src](/home/bill/Documents/CLionProjects/whetstone_DSL/editor/src)
|
||||
- [editor/tests](/home/bill/Documents/CLionProjects/whetstone_DSL/editor/tests)
|
||||
|
||||
## Rollout Order
|
||||
|
||||
1. Publish manifest from existing tool schema inventory
|
||||
2. Add health + envelope adapter (L0/L1)
|
||||
3. Add validation + policy metadata for top capabilities (L2)
|
||||
4. Verify Hivemind consumes Whetstone exclusively through contract endpoints
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] L0/L1/L2 tests passing
|
||||
- [ ] Whetstone runs standalone with full core capability set
|
||||
- [ ] Hivemind integration works via contract only (no direct internals)
|
||||
- [ ] Contract versioned and documented for replacement interoperability
|
||||
271
docs/ab_test_ast_vs_language_first_2026-02-25.md
Normal file
271
docs/ab_test_ast_vs_language_first_2026-02-25.md
Normal file
@@ -0,0 +1,271 @@
|
||||
# A/B Test: AST-First vs Language-First Code Generation
|
||||
**Date:** 2026-02-25
|
||||
**Author:** Whetstone DSL project
|
||||
**Status:** Partial — LLM+SLM comparison pending (see Section 5)
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document records a controlled A/B experiment comparing two approaches to code generation using the Whetstone constructive editor. The goal is to determine which approach is more efficient and capable when an AI agent constructs a non-trivial C++ program from scratch, and to establish a baseline before adding a third comparison path: LLM-generated taskitems completed by a local SLM.
|
||||
|
||||
**Task:** Implement a priority work queue in C++ consisting of:
|
||||
- A `WorkItem` struct (`job_id: string`, `priority: int`, `payload: string`) with nlohmann JSON serialization
|
||||
- A `PriorityQueue` class wrapping a max-heap, with `enqueue`, `dequeue`, `peek`, `size`, `empty` methods
|
||||
- Header-only, no dependencies beyond STL and nlohmann
|
||||
|
||||
---
|
||||
|
||||
## 2. Environment
|
||||
|
||||
### Hardware
|
||||
|
||||
| Component | Detail |
|
||||
|-----------|--------|
|
||||
| Machine | `temp-ubuntu` (local desktop) |
|
||||
| CPU | AMD Ryzen 5 2400G (4 cores / 8 threads) |
|
||||
| RAM | 16 GB |
|
||||
| GPU | NVIDIA GeForce RTX 3060, 12 GB VRAM |
|
||||
| Network | Tailscale node (100.107.72.34) |
|
||||
|
||||
### Software
|
||||
|
||||
| Component | Version |
|
||||
|-----------|---------|
|
||||
| OS | Ubuntu 24.04 (Linux 6.17.0-14-generic) |
|
||||
| Whetstone MCP binary | `whetstone_mcp_20260225_150702` (stable) |
|
||||
| Whetstone MCP tools | 347 registered tools |
|
||||
| C++ standard | C++20 |
|
||||
| CUDA | 12.1 |
|
||||
| PyTorch | 2.5.1+cu121 |
|
||||
| Transformers | 5.2.0 |
|
||||
| PEFT | 0.18.1 |
|
||||
| TRL | 0.28.0 |
|
||||
|
||||
### Whetstone Tool Surface
|
||||
|
||||
The stable binary exposes 347 MCP tools spanning:
|
||||
- Sprint 1–45 (original 90 tools): fully implemented, 52/52 test matrix passing
|
||||
- Sprint 93–155 (257 additional tools): mix of real implementations and backfill stubs
|
||||
|
||||
For this test, all tools called are from the Sprint 1–45 baseline (fully real).
|
||||
|
||||
---
|
||||
|
||||
## 3. Path A — AST-First
|
||||
|
||||
### Approach
|
||||
|
||||
Build the program by explicitly constructing an AST skeleton, then filling in components using schema-driven generation tools. The agent works at the structural level first, implementation second.
|
||||
|
||||
### Tool Call Sequence
|
||||
|
||||
| # | Tool | Input Summary | Result |
|
||||
|---|------|---------------|--------|
|
||||
| 1 | `whetstone_create_skeleton` | name=PriorityQueue, language=cpp | Skeleton buffer created |
|
||||
| 2 | `whetstone_add_skeleton_node` | WorkItem class, priority=critical, automatability=deterministic | Node `skcls_1` added |
|
||||
| 3 | `whetstone_add_skeleton_node` | PriorityQueue class, methods=[enqueue,dequeue,peek,size,empty], blockedBy=[WorkItem] | Node `skcls_2` added with dependency |
|
||||
| 4 | `whetstone_schema_to_cpp` | WorkItem JSON Schema → WorkItem.h | ✅ Complete header generated |
|
||||
| 5 | `whetstone_generate_code` | PriorityQueue class spec | ❌ Fell back to printf stub |
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total tool calls | 5 |
|
||||
| Estimated tokens | 760 |
|
||||
| Wall time | 13 ms |
|
||||
| Quality rating | 52 / 100 |
|
||||
|
||||
### Output: WorkItem.h (generated by `whetstone_schema_to_cpp`)
|
||||
|
||||
```cpp
|
||||
#pragma once
|
||||
#include <nlohmann/json.hpp>
|
||||
#include <string>
|
||||
|
||||
struct WorkItem {
|
||||
std::string job_id;
|
||||
std::string payload;
|
||||
int priority;
|
||||
};
|
||||
|
||||
inline void to_json(nlohmann::json& j, const WorkItem& v) {
|
||||
j = nlohmann::json::object();
|
||||
j["job_id"] = v.job_id;
|
||||
j["payload"] = v.payload;
|
||||
j["priority"] = v.priority;
|
||||
}
|
||||
|
||||
inline void from_json(const nlohmann::json& j, WorkItem& v) {
|
||||
if (j.contains("job_id")) j.at("job_id").get_to(v.job_id);
|
||||
if (j.contains("payload")) j.at("payload").get_to(v.payload);
|
||||
if (j.contains("priority")) j.at("priority").get_to(v.priority);
|
||||
}
|
||||
|
||||
inline bool validateWorkItem(const WorkItem& v) {
|
||||
(void)v;
|
||||
return true;
|
||||
}
|
||||
```
|
||||
|
||||
**CMake target snippet:**
|
||||
```cmake
|
||||
add_library(work_item_types INTERFACE)
|
||||
target_include_directories(work_item_types INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}/include)
|
||||
target_sources(work_item_types INTERFACE include/WorkItem.h)
|
||||
```
|
||||
|
||||
### Output: Skeleton AST
|
||||
|
||||
The skeleton recorded `WorkItem` and `PriorityQueue` as dependent nodes with routing annotations:
|
||||
- `WorkItem`: priority=critical, contextWidth=local, automatability=deterministic
|
||||
- `PriorityQueue`: priority=critical, contextWidth=file, automatability=deterministic, blockedBy=[WorkItem]
|
||||
|
||||
### Where It Stopped
|
||||
|
||||
`whetstone_generate_code` does not yet handle complex multi-method class generation. For a spec describing a full class with heap internals, it fell back to a single `printf`-based stub. The tool is designed for single-function generation; class-level generation requires the cross-language pipeline (see Path B) or manual `batch_mutate` operations.
|
||||
|
||||
**Completion:** WorkItem struct — 100%. PriorityQueue class — 0% (skeleton only).
|
||||
|
||||
---
|
||||
|
||||
## 4. Path B — Language-First
|
||||
|
||||
### Approach
|
||||
|
||||
Describe the program in natural language (and as a Python reference implementation), let Whetstone parse the intent and transpile to C++. The agent works at the specification level, expecting the tools to handle structural decisions.
|
||||
|
||||
### Tool Call Sequence
|
||||
|
||||
| # | Tool | Input Summary | Result |
|
||||
|---|------|---------------|--------|
|
||||
| 1 | `whetstone_architect_intake` | Markdown spec with Goal, Requirements, Constraints sections | ⚠️ Only Constraints captured; Requirements section dropped |
|
||||
| 2 | `whetstone_generate_taskitems` | 2 normalized constraint requirements | ⚠️ 1 generic task generated ("Intake Foundation Primary") |
|
||||
| 3 | `whetstone_run_pipeline` | 30-line Python reference impl → C++ | ✅ Python AST complete; ❌ C++ output = empty namespace |
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total tool calls | 3 |
|
||||
| Estimated tokens | 2,954 |
|
||||
| Wall time | 42 ms |
|
||||
| Quality rating | 38 / 100 |
|
||||
|
||||
### Output: Python AST (from `whetstone_run_pipeline`)
|
||||
|
||||
The Python parser produced a complete, accurate AST:
|
||||
- 2 classes: `WorkItem` (dataclass), `PriorityQueue`
|
||||
- 6 methods: `__init__`, `enqueue`, `dequeue`, `peek`, `size`, `empty`
|
||||
- Full span information (line + column) for every node
|
||||
- Binary operations captured (`len(self._heap) == 0`)
|
||||
- Return statements with correct value references
|
||||
- Memory model annotation automatically inferred: `ReclaimAnnotation` (strategy=Tracing, confidence=0.95) — Whetstone detected Python's garbage collection model and flagged it
|
||||
|
||||
### Output: Generated C++
|
||||
|
||||
```cpp
|
||||
namespace parsed_python_module {
|
||||
|
||||
} // namespace parsed_python_module
|
||||
```
|
||||
|
||||
The C++ code generator currently handles function-level transpilation but does not emit class bodies when the source is a Python class. The AST was fully parsed and semantically understood — the gap is in the C++ emitter for class nodes from cross-language input.
|
||||
|
||||
**Post-test audit finding (2026-02-25):** This is not a C++-specific gap. All 8
|
||||
language generators share the identical root cause — `visitModule` never calls
|
||||
`getChildren("classes")`, so class nodes are silently dropped from all generated
|
||||
output in every language. The `visitClassDeclaration` implementations are already
|
||||
written and correct for 7 of 8 languages (C++ is the only one missing it entirely).
|
||||
The fix is ~3 lines per generator. See `sprint162_plan.md`.
|
||||
|
||||
### Where It Stopped
|
||||
|
||||
Two failure points:
|
||||
1. `architect_intake` silently dropped the Requirements section (functional spec) and only retained the Constraints section. This is a known limitation — the intake parser identifies constraint-type sentences but does not yet extract structured requirements from bullet-point lists.
|
||||
2. `run_pipeline` C++ class emitter: Python class → C++ class transpilation is not yet implemented. The parser is complete; the generator needs class-body emission support.
|
||||
|
||||
**Completion:** Requirements understood — 0% (intake dropped them). Python parsed — 100%. C++ generated — 0% (empty namespace).
|
||||
|
||||
---
|
||||
|
||||
## 5. Head-to-Head Comparison
|
||||
|
||||
| Dimension | Path A: AST-First | Path B: Language-First |
|
||||
|-----------|-------------------|------------------------|
|
||||
| Tool calls | 5 | 3 |
|
||||
| Token cost | **760** | **2,954** (3.9× more) |
|
||||
| Wall time | 13 ms | 42 ms |
|
||||
| WorkItem struct | ✅ Complete, production-ready | ❌ Not produced |
|
||||
| PriorityQueue class | ❌ Stub only | ❌ Empty namespace |
|
||||
| Dependency tracking | ✅ Explicit in skeleton | ❌ Not captured |
|
||||
| Memory model inference | ❌ Not attempted | ✅ Automatic (0.95 confidence) |
|
||||
| Intake fidelity | N/A | ⚠️ Lost functional requirements |
|
||||
| Best output | Complete C++ struct with serializers | Complete Python AST with spans |
|
||||
|
||||
### Interpretation
|
||||
|
||||
**Path A wins on efficiency and partial output quality.** At 760 tokens and 13ms it produced a complete, compilable, production-ready `WorkItem.h` that could be used immediately. The skeleton gave the agent a clear map of what remained to do. The failure point (`generate_code` for class bodies) is a narrow, well-defined gap.
|
||||
|
||||
**Path B wins on semantic depth.** The Python parser is remarkably capable — it captured every method, argument, binary operation, and return statement with precise source spans. The automatic memory model inference (detecting Python's GC strategy and annotating accordingly) is a capability Path A never exercises. The failure points are also well-defined: intake needs bullet-point requirement extraction; the C++ emitter needs class-body support.
|
||||
|
||||
**Neither path completed the task automatically.** This is not a failure — it is the point. Both paths demonstrate exactly where human or LLM orchestration is currently needed. That gap is precisely what the LoRA training pipeline is designed to close.
|
||||
|
||||
---
|
||||
|
||||
## 6. Path C — LLM Taskitems + Local SLM Completion *(pending)*
|
||||
|
||||
This section will be completed after running the third comparison path:
|
||||
|
||||
1. A large LLM (Claude Sonnet) calls `whetstone_architect_intake` and `whetstone_generate_taskitems` to produce a full, correct task breakdown for the priority queue
|
||||
2. The taskitems are handed off to a local SLM (Qwen2.5-Coder-7B-Instruct with LoRA adapter, running on this same RTX 3060) to complete each task using grammar-constrained Whetstone tool calls
|
||||
3. Metrics recorded: total tokens (LLM + SLM combined), tool calls, wall time, completion quality
|
||||
|
||||
**Expected result:** Path C should complete the task fully. The LLM handles intent → structure (its strength); the SLM handles structure → implementation via constrained tool calls (its strength). The grammar ensures the SLM cannot hallucinate tool names or argument shapes.
|
||||
|
||||
**What to measure:**
|
||||
- Total token cost vs Path A and Path B
|
||||
- Whether the SLM respects grammar constraints in practice
|
||||
- Quality of generated C++ vs Path A's `schema_to_cpp` output
|
||||
- Latency of local inference vs API round-trips
|
||||
|
||||
---
|
||||
|
||||
## 7. Implications for LoRA Training
|
||||
|
||||
| Finding | Implication |
|
||||
|---------|-------------|
|
||||
| `schema_to_cpp` produces perfect structs in 1 call | Training data should include many schema→struct examples; this is a high-value pattern |
|
||||
| `generate_code` fails on multi-method classes | Do not generate training data from `generate_code` for class-level tasks yet |
|
||||
| `architect_intake` drops bullet-point requirements | Training data generated via intake will be incomplete until intake parsing is fixed |
|
||||
| `run_pipeline` Python AST is excellent | Python→C++ transpilation examples are valid for AST-side training data once the C++ emitter is fixed |
|
||||
| Path A used 3.9× fewer tokens | AST-first orchestration patterns should be over-represented in training data |
|
||||
|
||||
---
|
||||
|
||||
## 8. Grammar Coverage
|
||||
|
||||
At the time of this test, Whetstone had full grammar coverage for all 347 registered tools:
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| `tools/mcp/grammars/dispatch.gbnf` | 236 KB — all 347 tools, full dispatch grammar |
|
||||
| `tools/mcp/grammars/<tool>.gbnf` | 347 per-tool grammars |
|
||||
| `tools/mcp/grammars/per_tool_schemas.json` | JSON Schema per tool (for `outlines` / HF inference) |
|
||||
| `tools/mcp/grammars/dispatch_schema.json` | Full `oneOf` dispatch schema |
|
||||
|
||||
Enum constraints active: 15 properties across the tool surface (e.g. `whetstone_mutate.type` is constrained to exactly `setProperty | updateNode | deleteNode | insertNode`).
|
||||
|
||||
Grammar source: `tools/mcp/whetstone_tool_schemas.json` (re-dumped from live MCP server).
|
||||
Generator: `tools/mcp/generate_tool_grammars.py` (re-run after any tool changes).
|
||||
|
||||
---
|
||||
|
||||
## 9. Next Steps
|
||||
|
||||
1. **Fix `architect_intake`** — extract structured requirements from bullet-point sections, not just constraint sentences
|
||||
2. **Implement C++ class emitter** in `run_pipeline` for cross-language class transpilation
|
||||
3. **Run Path C** — LLM taskitems + local SLM completion, record full metrics
|
||||
4. **Continue LoRA training** — extend from 20-step proof-of-concept to ~1,500 steps (one full epoch) for a usable quality auditor adapter
|
||||
5. **Re-dump tool schemas** after stub sprints (93–119, 131–137) are replaced with real implementations; regenerate grammars
|
||||
22
docs/challenging_deterministic_projects_2026-02-26.md
Normal file
22
docs/challenging_deterministic_projects_2026-02-26.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Challenging Deterministic Project List (2026-02-26)
|
||||
|
||||
Purpose:
|
||||
- Identify legal but semantically hard tasks likely to expose deterministic generator limits.
|
||||
- Continuously mine failure packets to create new backfill sprints.
|
||||
|
||||
Catalog file:
|
||||
- `datasets/project_benchmarks/challenging_deterministic_projects_2026-02-26.jsonl`
|
||||
|
||||
Focus areas covered:
|
||||
- financial invariants and replay safety
|
||||
- distributed lock semantics and deadlock risk
|
||||
- CRDT convergence and causal ordering
|
||||
- saga compensation and timeout recovery
|
||||
- policy explainability and precedence
|
||||
- streaming out-of-order replay determinism
|
||||
- schema evolution with CDC checkpoints
|
||||
- high-contention rate limiting
|
||||
- long-range build graph transactional execution
|
||||
- semantic contract validation and remediation plans
|
||||
- secure staged rollout/rollback workflows
|
||||
- class-heavy cross-language codegen parity
|
||||
15
docs/challenging_fullstack_multifile_2026-02-26.md
Normal file
15
docs/challenging_fullstack_multifile_2026-02-26.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# Challenging Fullstack + Multi-file Benchmark Set (2026-02-26)
|
||||
|
||||
Purpose:
|
||||
- Expose deterministic-tool failures in cross-layer API changes and transactional multi-file edits.
|
||||
|
||||
Catalog:
|
||||
- `datasets/project_benchmarks/challenging_fullstack_multifile_2026-02-26.jsonl`
|
||||
|
||||
Challenge classes:
|
||||
- full-stack API contract evolution
|
||||
- multi-file transactional refactor
|
||||
- security policy propagation
|
||||
- schema migration + rollback
|
||||
- feature-flag rollout choreography
|
||||
- SLO-budget-preserving refactors
|
||||
86
docs/deterministic_gap_hunt_2026-02-26.md
Normal file
86
docs/deterministic_gap_hunt_2026-02-26.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# Deterministic Gap Hunt Report - 2026-02-26
|
||||
|
||||
## 1) Strict Backfill Audit of Sprints 186-205
|
||||
|
||||
Reference:
|
||||
- `docs/sprint186_205_intent_audit_2026-02-26.md`
|
||||
- `docs/sprint186_205_closeout_report_2026-02-26.md`
|
||||
|
||||
Signal split:
|
||||
- Pipeline closeout (`EXECUTION-READY`): `20/20`
|
||||
- Strict intent closure:
|
||||
- `CLOSED`: `1/20`
|
||||
- `PARTIAL`: `10/20`
|
||||
- `GAP`: `9/20`
|
||||
|
||||
Interpretation:
|
||||
- Operational gates are green.
|
||||
- Many original sprint intents remain partially implemented or scaffolded.
|
||||
|
||||
## 2) New Semantically Challenging Project Catalogs
|
||||
|
||||
Created:
|
||||
- `datasets/project_benchmarks/challenging_deterministic_projects_2026-02-26.jsonl` (12 hard projects)
|
||||
- `datasets/project_benchmarks/challenging_subset_prod_2026-02-26.jsonl` (focused production-loop subset)
|
||||
- `datasets/project_benchmarks/challenging_projection_projects_2026-02-26.jsonl` (projection/environment constraints)
|
||||
- `docs/challenging_deterministic_projects_2026-02-26.md`
|
||||
|
||||
## 3) Benchmark Results and New Gap Signals
|
||||
|
||||
### A) Hard matrix (12 projects x 4 languages)
|
||||
Run:
|
||||
- `logs/taskitem_runs/challenging_matrix_20260226`
|
||||
|
||||
Summary highlights:
|
||||
- `total_runs=48`
|
||||
- A/B Path A ready: `77.08%`
|
||||
- A/B Path B ready: `25.0%`
|
||||
- Path B top failures: `compile_failed:non_zero_exit`, `tests_failed:non_zero_exit` (36 each)
|
||||
- Avg token ratio `PathB/PathA`: `13.67x`
|
||||
|
||||
### B) Focused production subset
|
||||
Run:
|
||||
- `logs/taskitem_runs/challenging_subset_prod_20260226`
|
||||
|
||||
Summary highlights:
|
||||
- `total_runs=8`
|
||||
- A/B path B still fails on cpp cases.
|
||||
- Production loop reports `ready=100%` for all 8 cases, including cases where A/B path B failed compile/tests.
|
||||
- Several production summaries show sparse/null gate evidence fields despite green outcome.
|
||||
|
||||
### C) Projection constraints subset
|
||||
Run:
|
||||
- `logs/taskitem_runs/challenging_projection_matrix_20260226`
|
||||
|
||||
Summary highlights:
|
||||
- Projection-heavy specs are accepted as free-text, but constraint fields are not enforced as first-class contract inputs in benchmark orchestration.
|
||||
- Confirms projection/environment constraints are under-modeled in current automated benchmark loop.
|
||||
|
||||
## 4) New Gap IDs (Post-186-205)
|
||||
|
||||
- `GR-011` False-green production gating
|
||||
- Symptom: production loop marks ready where A/B compilation/tests fail on equivalent hard specs.
|
||||
- Evidence: `challenging_subset_prod_20260226` vs A/B failures.
|
||||
|
||||
- `GR-012` Projection constraint contract missing in benchmark/execution path
|
||||
- Symptom: target/environment budgets and forbidden APIs are not treated as first-class executable constraints.
|
||||
- Evidence: projection catalog accepted, but orchestration path consumes mostly `spec` text.
|
||||
|
||||
- `GR-013` Language asymmetry under hard semantics
|
||||
- Symptom: Python passes much more frequently than cpp/go/rust on equivalent hard specs.
|
||||
- Evidence: `challenging_matrix_20260226` language breakdown.
|
||||
|
||||
- `GR-014` Token inefficiency and semantic collapse in language-first path
|
||||
- Symptom: Path B token use is ~14x higher with poor readiness under hard tasks.
|
||||
- Evidence: hard matrix summaries.
|
||||
|
||||
- `GR-015` Intent-vs-closeout drift risk
|
||||
- Symptom: sprint closeout can be green while original intent remains partial.
|
||||
- Evidence: strict intent audit vs closeout report divergence.
|
||||
|
||||
## 5) Recommended Next Sprint Theme
|
||||
|
||||
- Hard-gate production readiness on explicit compile/test evidence.
|
||||
- Add projection-constraint schema to benchmark and pipeline tool inputs.
|
||||
- Add cross-language parity gates for hard semantic specs.
|
||||
- Add anti-false-green checks linking A/B and production evidence packets.
|
||||
@@ -1,7 +1,7 @@
|
||||
# Generator Readiness Gap Registry (Dated)
|
||||
|
||||
- Date created: 2026-02-26
|
||||
- Last reviewed: 2026-02-28 (Sprint 270: GR-009, GR-010, GR-012, GR-014, GR-018, GR-020 closed — ALL 26 GAPS DONE)
|
||||
- Last reviewed: 2026-04-09 (GR-036 added: C++ import is usable for AST bootstrap, but sync/regenerate round-trip maturity is not yet proven for production AST-first migration workflows)
|
||||
- Status: Active
|
||||
- Scope: Whetstone autonomous code generation and deterministic execution readiness.
|
||||
|
||||
@@ -16,6 +16,7 @@ This is the canonical dated registry for "not production-ready" generator gaps.
|
||||
- `FEATURE_REQUESTS.md`
|
||||
- `docs/sprint175_179_execution_tracker_2026-02-26.md`
|
||||
- `docs/sprint186_205_execution_tracker_2026-02-26.md`
|
||||
- `docs/taskitem_pipeline_gap_log_2026-03-08_multi_project.md`
|
||||
|
||||
## Status Legend
|
||||
|
||||
@@ -55,6 +56,18 @@ This is the canonical dated registry for "not production-ready" generator gaps.
|
||||
|
||||
| GR-025 | No SemAnno annotation for platform adapter boundaries — pure logic and platform I/O are indistinguishable to the projector | `FEATURE_REQUESTS.md` [Derived:PersonalVaultTool-MultiPlatform] (2026-02-27); observed during personal_vault_tool cross-platform projection analysis | `done` | The projector treats a source file as a unit. When projecting Python→Kotlin, it faithfully emits `sqlite3.connect()` because there is no annotation marking storage calls as platform-specific. SemAnno already has the right primitives (`@ownership`, `@memory.*`) but no `@platform.*` family. Without this, cross-platform projection can never substitute platform adapters — it can only project the source implementation literally. Blocks every real-world multi-platform use case. | Sprint 267 (2026-02-28): Added `PlatformAnnotations.h` with 5 classes (PureLogicAnnotation, PlatformStorageAnnotation, PlatformCryptoAnnotation, PlatformNetworkAnnotation, PlatformOsAnnotation). Wired into SemannoEmitterBody.h, SemannoParserSection.h (generic parser already handled it), AnnotationVisitors.h, SemannoAnnotationImpl.h, ProjectionGenerator.h dispatch. 22/22 tests passing (steps 1863–1866). |
|
||||
| GR-026 | No platform profile system — projector has no mapping from `@platform.*` annotations to target-platform API substitutions | `FEATURE_REQUESTS.md` [Derived:PersonalVaultTool-MultiPlatform] (2026-02-27) | `done` | Even after GR-025 is implemented and nodes are annotated, the projector has no table of "what does `@platform.storage` mean on Android?" Without a platform profile (a JSON registry mapping annotation → target SDK API), the projection step cannot emit the correct stub. Example: `@platform.storage` on `python-stdlib` → `sqlite3`; on `android-kotlin` → `androidx.room.RoomDatabase`; on `swift-ios` → `CoreData`. This gap means multi-target projection (Feature Request B in FEATURE_REQUESTS.md) is blocked even if annotation tagging is complete. Depends on GR-025. | Sprint 267 (2026-02-28): Added `PlatformProfileRegistry.h` with loadBuiltins(), resolve(profile, tag), profileCount(). Two built-in profiles: python-stdlib (sqlite3, hashlib, requests, os) and android-kotlin (androidx.room.RoomDatabase, javax.crypto.Cipher, okhttp3.OkHttpClient, android.os.Environment). 7/7 tests passing (step 1867). |
|
||||
| GR-027 | Cross-project task targeting mismatch in taskitem generation | `docs/taskitem_pipeline_gap_log_2026-03-08_multi_project.md`; runs `logs/taskitem_runs/constcad_ecosystem_roadmap_20260307_210200`, `logs/taskitem_runs/constcad_ecosystem_roadmap_20260307_210213`, `logs/taskitem_runs/sprint_005_execution_spec_whetstone_20260307_221636` | `open` | For a cross-repo orchestration spec targeting `cad_orchestrator` + `hivemind` + `gleaner` + `constcad`, generated `targetFiles` were mapped to Whetstone-internal files (`editor/src/mcp/...`, `tools/mcp/...`) even with `WSTONE_WORKSPACE=/home/bill/Documents`. Reproduced again on SPRINT-005 execution spec run. This blocks deterministic autonomous implementation for multi-project integration tasks. | Add explicit cross-project path grounding in architect/taskitem generation + validator blocker `cross_project_target_mismatch`; retest on SPRINT-005 execution spec |
|
||||
|
||||
| GR-028 | `generate_code` falls back to generic class stub regardless of spec detail | Observed 2026-03-12 during WHIMP Sprint 04 session: detailed multi-method body replacement specs (NavigationAssignmentHeuristics, IntentFamilyPolicyHeuristics, GuidanceBridge) all produced `class GeneratedClass { void run(); }` or `void generated(std::string input) { printf(input); }`. Fallback triggered even when spec explicitly named method signatures, formulas, and field names. | `open` | Root cause: `generate_code` has no workspace symbol lookup. It generates a syntactically valid node from the spec's *words*, not from the project's type graph. It doesn't know that `NavigationAssignmentGuidance`, `MetricRecord`, or `Clamp01` exist in the workspace. An LLM reading the headers can derive the full implementation; whetstone can't without first loading and indexing the project symbols into the code generation path. Gap: workspace AST index is not consulted during `generate_code` dispatch. | Add workspace-symbol-grounded generation mode: before generating, look up types named in spec from the indexed workspace AST and inject them as available symbols into the generation context |
|
||||
| GR-029 | `generate_taskitems` produces generic milestone tasks, not project-specific implementation tasks | **First observed:** 2026-03-12 during WHIMP Sprint 04 session: called with 16 normalized requirements (5 goals, 6 constraints, 5 acceptance criteria) describing WHIMP C++ files and formula implementations. Output was 2 tasks titled "Intake Foundation Primary" and "Execution Readiness Primary" with `targetFiles` pointing to `editor/src/mcp/RegisterArchitectIntakeTools.h` and `tools/mcp/run_sprint_taskitem_pipeline.sh`. None of the 5 implementation goals produced a corresponding concrete implementation task. **Reproduced:** 2026-03-14 during WHIMP Sprint 07 session: called with 19 normalized requirements (5 goals: ITargetResolver/LastFocusResolver/NamedWindowResolver/FifoResolver/SocketResolver/GazeWindowResolver; 7 constraints; 7 acceptance criteria). Output was 4 tasks titled "Intake Foundation Primary", "Architect Review", "Execution Readiness Primary", "Architect Review" — the same whetstone-internal targetFiles, the same stepIds (1939–1941). `escalateCount=3`. None of the 5 resolver implementation goals produced a concrete task. Session ID: `whimp-sprint07-2026-03-14`. | `open` | Root cause is related to GR-027 (cross-project target grounding) but is a distinct failure mode. Even when running against WHIMP (not a multi-project orchestration spec), the taskitem generator produces meta-process tasks about whetstone's own pipeline rather than WHIMP implementation tasks. The normalized requirement text contained explicit WHIMP file paths and formula names but these were not used in task title, intent, or targetFiles. The generator is essentially producing "run the whetstone pipeline" as the task regardless of input content. An LLM given the same normalized requirements produces 5 concrete implementation tasks with correct file targets and method signatures. Reproduced twice across two different WHIMP sprints with different requirement sets — confirms this is a stable, systematic failure of the project-grounding pass, not an edge case. | `generate_taskitems` needs a project-grounding pass that reads the `targetFiles` named in requirements and produces tasks whose `targetFiles`, `title`, and `intent` are derived from the spec content — not from whetstone-internal defaults |
|
||||
| GR-033 | No mechanism to save taskitem interaction traces for LoRA training from a failed `generate_taskitems` run | Observed 2026-03-14 during WHIMP Sprint 07 session. When `generate_taskitems` produces generic tasks (GR-029), there is no whetstone tool to save the interaction — the spec input, the generic output, and the correct expected output — as a structured LoRA training example. `save_repro_packet` captures a failure packet but does not format it as instruction/input/output JSONL for LoRA training. `export_training_data` is non-functional (GR-031). The gap interaction is a high-value training signal: the spec text + the correct project-specific tasks the model should have produced — but there is no pathway to capture it. | `open` | Add a `save_taskitem_gap_example` tool (or extend `save_repro_packet`) that accepts: the `architect_intake` output, the `generate_taskitems` input/output, and a `corrected_tasks` array (provided by the calling agent or user). Formats these as a LoRA JSONL record: `{instruction: "generate taskitems from these normalized requirements", input: <requirements JSON>, output: <corrected_tasks JSON>}` and appends to a configurable gap-training dataset file. |
|
||||
| GR-030 | `architect_intake` fails with `no_requirements_found` on richly-structured specs | Observed 2026-03-12: first intake call used a markdown spec with a "Context" section and five "### T1 … T5" task subsections. Result: `error: no_requirements_found`, `goals: []`. RequirementsParser did not recognize the task subsections as requirements. Required full restructuring into `## Goals`, `## Requirements`, `## Constraints`, `## Acceptance Criteria` headings before intake succeeded. | `open` | GR-005 fixed prose-vs-bullets for requirement-section body parsing, but the section *heading detection* is still brittle — only a fixed vocabulary of heading names triggers requirement mode. Any spec that uses "Tasks", "Implementation", "Deliverables", "Work Items", "Scope" or numbered/named subsections is silently dropped. An LLM parses all of these as requirements regardless of section name. | Add fuzzy section-heading matching: any heading containing "task", "work", "implement", "deliver", "scope", "feature" should be treated as a requirements section; also trigger on numbered `### N.` patterns |
|
||||
| GR-031 | `export_training_data` returns "Method not found" — LoRA export pipeline is non-functional | Observed 2026-03-12: called `export_training_data` with `format=huggingface` and `languages=["cpp"]` after loading 7 C++ buffers (NavigationMetrics.cpp, GuidanceBridge.cpp, GuidanceBridge.h, PolicyEngine.cpp, Explainer.cpp, Explainer.h, NavigationMetrics.cpp). Returned `{"error": "Method not found"}`. Training data export is a core stated capability (tool is documented, in the tool list, has a schema) but is not actually registered or implemented in the running server. | `open` | The tool schema is defined and the tool appears in the MCP tool list. The running server (build `whetstone_mcp --daemon --port 7600`) does not implement the handler. This means the entire LoRA training data pipeline — opening annotated files, syncing AST, exporting instruction/input/output JSONL — produces no output. The sprint 04 training data could not be captured. | Register the `export_training_data` handler in the MCP server; implement HuggingFace JSONL export from open buffers: for each buffer, emit `{instruction: "implement X", input: "<original file>", output: "<modified file>"}` pairs derived from the buffer edit history |
|
||||
| GR-032 | No project-context injection into `generate_code` — spec must redundantly re-declare all types | Structural gap underlying GR-028. Even if workspace symbol lookup (GR-028) is added, the `generate_code` call interface takes only a `spec` string and `preferImports` flag. There is no parameter for: the file being modified, the class/method context, surrounding code, or existing symbol signatures. This means every call requires the caller to fully re-state the type signatures, method names, existing fields, and namespace in the spec text — making specs extremely verbose and error-prone. An LLM automatically uses the file context it has already read. The `open_file` / `get_ast` round-trip exists but its output is not plumbed into `generate_code`. | `open` | Add a `context` parameter to `generate_code`: accepts a buffer path or buffer ID; pulls the active buffer's AST and symbol table into the generation context automatically. This would let a caller say `generate_code(spec="replace Evaluate() body with Hick formula", context="software/cpp/src/metrics/NavigationMetrics.cpp")` and get back a method body that already knows `NavigationAssignmentGuidance`, `Clamp01`, `std::log2`, etc. |
|
||||
| GR-035 | `generate_taskitems` can appear project-grounded yet still emit non-executable taskitems; validator/queue wrongly pass them as perfect | Observed 2026-03-20 during `whimpwm` Sprint 002 session. Pipeline run: `logs/taskitem_runs/sprint_002_policy_driven_recomposition_execution_spec_20260320_103952` for `/home/bill/Documents/whimpwm/docs/sprints/sprint_002_policy_driven_recomposition_execution_spec.md`. Positive: tasks targeted `crates/whimpwm/...` instead of Whetstone-internal files. Failure: the six tasks were still not execution-ready. All tasks shared the same blanket `targetFiles` set and the same duplicated acceptance surface; task titles were semantically wrong for the sprint (`Scaffold minimal compositor runtime`, `Define focus-stream state model`, `Bootstrap Smithay workspace`) even though the sprint was about policy-driven recomposition. Every task carried irrelevant `go test ./...` in `acceptanceCommands` for a Rust-only repo. `testFiles` were fabricated (`tests/main_test_test.rs`, etc.) and did not match the actual crate layout. Queue/validation still reported `ready=true`, `average_score=100.0`, `gap_class=none` for all tasks. The run was exported to `training_data/lora/taskitem_pipeline_runs.jsonl`. | `open` | This is distinct from GR-029. The generator is no longer collapsing to Whetstone-internal meta-tasks, but it still fails to produce executable, task-shaped workitems from the spec. Root cause appears to be weak repository/language grounding after initial file targeting: acceptance commands are not filtered by detected stack, test file derivation is synthetic rather than filesystem-grounded, and task decomposition is driven by generic milestone templates instead of requirement-specific action semantics. The validator over-scores these tasks because it checks contract presence and surface structure, not whether commands/files/titles are plausible for the actual repo. | Add repo-grounded execution plausibility checks in both generation and validation: derive acceptance commands from detected toolchain (`cargo` only here), require test file paths to exist or be mappable to real targets, penalize duplicated acceptance blocks across unrelated tasks, and score task titles/intents against sprint requirement vocabulary so semantically off generic tasks cannot pass as `gap_class=none` |
|
||||
| GR-036 | C++ import is usable for AST bootstrap, but sync/regenerate round-trip maturity is not yet proven for production AST-first migration workflows | Observed 2026-04-09 while assessing `whetstone_RSA` AST-first realignment. Positive: Whetstone has a real C++ parser path (`editor/src/ast/CppParser.h`) and MCP surfaces for `whetstone_open_file` and `whetstone_get_ast`, so existing C++ can be imported and inspected as AST. Limitation: the currently exposed `whetstone_sync_text_to_ast` and `whetstone_regenerate_text_from_ast` handlers in `RegisterSprint142Tools.h` and `RegisterSprint143Tools.h` appear to be constructive-status/profile scaffolding rather than a fully proven end-to-end C++ round-trip path suitable for trusting automatic semantic-source migration without verification. | `open` | This is not a claim that C++ parsing is broken. The practical gap is maturity and trust boundary: agents can use Whetstone today to parse/import C++ and establish semantic intent, but should not yet assume a production-ready "import C++ -> semantic edits -> regenerate projected C++" loop without explicit validation. For AST-first migrations, this means imported C++ should be treated as bootstrap input and checked carefully on projection. | Close this gap by validating a deterministic C++ round-trip workflow on real repos: open/import a non-trivial C++ module, verify AST completeness, apply semantic annotations/mutations, regenerate C++, compile it, and measure semantic equivalence. Record the result either as closure evidence here or as a compatibility-ledger entry if runtime-version-specific defects are found. |
|
||||
|
||||
| GR-034 | `run_pipeline` C++→C++ drops all ClassDeclaration nodes — emits empty namespace | Observed 2026-03-14 during WHIMP Sprint 07 session. Input: 6-class C++ header with ITargetResolver (pure virtual interface + LastFocusResolver + NamedWindowResolver + FifoResolver + SocketResolver + GazeWindowResolver, inheritance hierarchy, method declarations, fields, constexpr statics). AST parse was **correct** — all 6 ClassDeclaration nodes present in the returned AST with correct methods, fields, baseClasses, and spans. `generatedCode` output was empty namespace. All 6 classes silently dropped. Gate: `compile.passed=false, reason=code_missing_structure_or_fallback`. Session ID: `whimp-sprint07-2026-03-14`. **FIXED 2026-03-14 (step 1988).** Root cause identified: `visitModule` in `CppGenerator.h` only processed `"variables"/"functions"/"classes"` children. `CppParser::convertCppTranslationUnit` stores `preproc_include`, `preproc_call`, and `namespace_definition` in `"statements"` — not in `"classes"`. So any real C++ file with a namespace had its entire body dropped because `visitModule` never reached the `"statements"` list. Fix: added statements-first path — when `module->getChildren("statements")` is non-empty, emit each via `generate(stmt)` in order and return; legacy structured-children path retained for synthetic modules. Verified: `run_pipeline` C++→C++ on WHIMP TargetResolver skeleton (5 classes in namespace whimp::app) now emits all classes. 5/5 step1988_test passing. Daemon rebuilt and restarted. | `done` | Fixed in `editor/src/ast/CppGenerator.h::visitModule`, step 1988, test `editor/tests/step1988_test.cpp`. Note: `visitClassDeclaration` was already implemented in `CppGeneratorTypes.h` — it was never being called because the namespace containing it was in `"statements"` not `"classes"`. |
|
||||
|
||||
## What Was Covered Today (Sprints 175-184)
|
||||
|
||||
|
||||
50
docs/lora_qwen14b_training_runbook.md
Normal file
50
docs/lora_qwen14b_training_runbook.md
Normal file
@@ -0,0 +1,50 @@
|
||||
# Qwen2.5-Coder-14B LoRA Runbook (MCP Quality)
|
||||
|
||||
## Objective
|
||||
Train a first LoRA that improves MCP tool-call quality judgments and recovery behavior using the prepared grouped splits.
|
||||
|
||||
## Inputs
|
||||
- `training_data/lora/splits/mcp_lora.train.jsonl`
|
||||
- `training_data/lora/splits/mcp_lora.val.jsonl`
|
||||
- `training_data/lora/splits/mcp_lora.test.jsonl`
|
||||
|
||||
## Locked config
|
||||
- Config file: `configs/lora/qwen2.5-coder-14b-quality.env`
|
||||
- Launcher: `tools/mcp/train_qwen14b_quality_lora.sh`
|
||||
|
||||
## Preflight
|
||||
```bash
|
||||
cd /home/bill/Documents/CLionProjects/whetstone_DSL
|
||||
./tools/mcp/check_lora_training_readiness.sh
|
||||
```
|
||||
|
||||
## Python dependencies (one-time)
|
||||
```bash
|
||||
python3 -m pip install --upgrade pip
|
||||
python3 -m pip install torch transformers peft trl bitsandbytes datasets accelerate
|
||||
```
|
||||
|
||||
## Build SFT chat dataset
|
||||
```bash
|
||||
python3 ./tools/mcp/build_lora_quality_sft_dataset.py \
|
||||
--train-in training_data/lora/splits/mcp_lora.train.jsonl \
|
||||
--val-in training_data/lora/splits/mcp_lora.val.jsonl \
|
||||
--test-in training_data/lora/splits/mcp_lora.test.jsonl \
|
||||
--out-dir training_data/lora/sft_qwen14b_quality
|
||||
```
|
||||
|
||||
## Train
|
||||
```bash
|
||||
./tools/mcp/train_qwen14b_quality_lora.sh
|
||||
```
|
||||
|
||||
## Pass/Fail gates for first checkpoint
|
||||
- `json_parse_success >= 99.5%`
|
||||
- `invalid_schema_rate <= 1.0%`
|
||||
- `label_f1 >= 0.92`
|
||||
- `failure_class_macro_f1 >= 0.80`
|
||||
- `tool_selection_error_rate <= 5%`
|
||||
|
||||
## Notes
|
||||
- Keep this LoRA scoped to `qwen2.5-coder:14b` data. Do not merge with other base-model families in the same adapter.
|
||||
- Use grouped splits to avoid run-family leakage between train/val/test.
|
||||
110
docs/mcp_logging_data_strategy.md
Normal file
110
docs/mcp_logging_data_strategy.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# MCP Logging and Data Synthesis Strategy
|
||||
|
||||
## Objective
|
||||
Train reusable tool-routing and tool-execution adapters by separating:
|
||||
- `tool_selection`: which tool to call next.
|
||||
- `tool_execution`: whether arguments/use were correct and what happened.
|
||||
|
||||
This split supports LoRA datasets that improve planning/routing independently from low-level argument formation.
|
||||
|
||||
## Runtime Stages
|
||||
Each model turn is logged as one or more JSONL records in `logs/taskitem_runs/ollama_mcp_exec_*/task_<n>_context.jsonl`.
|
||||
|
||||
### Stage: tool_selection
|
||||
Recorded before MCP execution. Fields:
|
||||
- `stage`: `tool_selection`
|
||||
- `turn`
|
||||
- `intent`
|
||||
- `task_phase`
|
||||
- `candidate_tools`
|
||||
- `chosen_tool`
|
||||
- `args`
|
||||
- `reason`
|
||||
- `confidence`
|
||||
- `fallback_tools`
|
||||
- `label`: `good` or `bad`
|
||||
- `error_type` (when bad)
|
||||
|
||||
### Stage: tool_execution
|
||||
Recorded after MCP execution. Fields:
|
||||
- `stage`: `tool_execution`
|
||||
- `turn`
|
||||
- `tool`
|
||||
- `args`
|
||||
- `result`: `success|error|blocked|unknown`
|
||||
- `label`: `good` or `bad`
|
||||
- `error_type` (when available)
|
||||
- `result_preview`
|
||||
|
||||
## Failure Metadata and Debug Probes
|
||||
When the loop hits invalid selections, repeated issues, or max-turn exits, transcript probes are captured in `01_transcript.log`.
|
||||
|
||||
Failure probes can include:
|
||||
- `whetstone_get_workflow_state`
|
||||
- `whetstone_get_blockers`
|
||||
- `whetstone_get_recent_events`
|
||||
- optional debug-specific tools if present
|
||||
- `whetstone_generate_inference_job`
|
||||
|
||||
These become `capture_type: "failure_probe"` records in malformed exports.
|
||||
|
||||
## Dataset Construction
|
||||
|
||||
### Malformed dataset
|
||||
Script: `tools/mcp/export_malformed_mcp_calls_for_lora.sh`
|
||||
|
||||
Includes:
|
||||
- adapter loop errors (`invalid_action`, unknown tool, etc.)
|
||||
- MCP tool errors from `result_preview`
|
||||
- stage-labeled bad events from selection/execution
|
||||
- failure probe records from transcript
|
||||
|
||||
Output: `training_data/lora/malformed_mcp_calls.jsonl`
|
||||
|
||||
### Quality dataset
|
||||
Script: `tools/mcp/export_mcp_call_quality_for_lora.sh`
|
||||
|
||||
Includes:
|
||||
- pipeline-derived known-good calls
|
||||
- stage-labeled `good`/`bad` examples
|
||||
- legacy adapter and MCP tool-response bad examples
|
||||
|
||||
Outputs:
|
||||
- `training_data/lora/mcp_call_quality.jsonl`
|
||||
- `training_data/lora/mcp_calls_good.jsonl`
|
||||
- `training_data/lora/mcp_calls_bad.jsonl`
|
||||
|
||||
## Synthesis Strategy
|
||||
Use mixed sampling:
|
||||
- Positive examples from clean pipeline runs.
|
||||
- Hard negatives from malformed outputs and near-miss tool choices.
|
||||
- Counterfactual negatives where similar tools are available but incorrect.
|
||||
|
||||
Recommended metadata tags for future synthesis:
|
||||
- `interface_mode`: `json_loop` vs `native_tool_calling`
|
||||
- `selection_confidence_bucket`
|
||||
- `recovery_attempted`
|
||||
- `task_domain` / `codebase_area`
|
||||
|
||||
## Batch Orchestration
|
||||
Script: `tools/mcp/run_slm_training_batch.sh`
|
||||
|
||||
Per run manifest fields now include:
|
||||
- token usage totals
|
||||
- `stage_metrics.selection_events`
|
||||
- `stage_metrics.selection_bad`
|
||||
- `stage_metrics.execution_events`
|
||||
- `stage_metrics.execution_bad`
|
||||
|
||||
Use these metrics to rank runs for curation and to detect regressions in routing quality.
|
||||
|
||||
## Curation Rules
|
||||
Promote samples when:
|
||||
1. Selection was confident and consistent with successful execution.
|
||||
2. Execution finished without schema/tool errors.
|
||||
3. Recovery (if any) converged quickly.
|
||||
|
||||
Demote or isolate samples when:
|
||||
1. Selection repeatedly picks unavailable or semantically wrong tools.
|
||||
2. Args repeatedly violate schema despite retries.
|
||||
3. Context resets dominate the run (possible instability/noise).
|
||||
127
docs/mcp_tool_surface_handoff_2026-02-25.md
Normal file
127
docs/mcp_tool_surface_handoff_2026-02-25.md
Normal file
@@ -0,0 +1,127 @@
|
||||
# MCP Tool Surface Handoff (2026-02-25)
|
||||
|
||||
## Snapshot
|
||||
- Defined MCP tools in code (`editor/src/mcp/Register*Tools.h`): **347**
|
||||
- Active grammar/schema tool surface (`tools/mcp/whetstone_tool_schemas.json`): **90**
|
||||
- Overlap (defined + grammar-covered): **90**
|
||||
- Defined but not grammar-covered yet: **257**
|
||||
- Grammar entries not currently found in registration headers: **0**
|
||||
|
||||
## Key Answer For Planning
|
||||
- You can start grammar work now on the stable active surface of **90** tools.
|
||||
- Backfill work is primarily implementation/runtime wiring of existing names, not broad renaming/removal.
|
||||
- If you want full-surface grammar coverage, additional scope is **257** tools beyond the current 90.
|
||||
|
||||
## Files To Use
|
||||
- Active schemas now: `tools/mcp/whetstone_tool_schemas.json` (90 tools)
|
||||
- Grammar generator: `tools/mcp/generate_tool_grammars.py`
|
||||
- Full registered surface source: `editor/src/mcp/Register*Tools.h`
|
||||
|
||||
## Defined-Only Gap (First 40 of 257 )
|
||||
- `whetstone_analyze_data_pipeline_semantics`
|
||||
- `whetstone_analyze_rust_semantics`
|
||||
- `whetstone_apply_best_practice_pack`
|
||||
- `whetstone_apply_patch_packet`
|
||||
- `whetstone_apply_text_ast_merge`
|
||||
- `whetstone_apply_verified_pattern`
|
||||
- `whetstone_assemble_fix_context`
|
||||
- `whetstone_attach_multimodal_evidence`
|
||||
- `whetstone_begin_constructive_transaction`
|
||||
- `whetstone_build_debug_handoff`
|
||||
- `whetstone_capture_distributed_failure`
|
||||
- `whetstone_capture_failure_packet`
|
||||
- `whetstone_capture_handoff_packet`
|
||||
- `whetstone_capture_mcp_execution_trace`
|
||||
- `whetstone_check_runtime_freshness`
|
||||
- `whetstone_check_tool_compatibility`
|
||||
- `whetstone_classify_debug_stop_reason`
|
||||
- `whetstone_cluster_distributed_failures`
|
||||
- `whetstone_cluster_failures`
|
||||
- `whetstone_compare_constructive_replays`
|
||||
- `whetstone_compare_mcp_replays`
|
||||
- `whetstone_compare_tool_surfaces`
|
||||
- `whetstone_configure_hivemind`
|
||||
- `whetstone_debug_until_green`
|
||||
- `whetstone_derive_requirements`
|
||||
- `whetstone_detect_text_ast_conflicts`
|
||||
- `whetstone_diff_environments`
|
||||
- `whetstone_dry_run_patch`
|
||||
- `whetstone_enqueue_pair_upgrade`
|
||||
- `whetstone_estimate_debug_budget`
|
||||
- `whetstone_estimate_porting_cost`
|
||||
- `whetstone_execute_rollout_or_rollback`
|
||||
- `whetstone_explain_transpilation_decision`
|
||||
- `whetstone_export_debug_campaign_bundle`
|
||||
- `whetstone_export_mcp_replay_pack`
|
||||
- `whetstone_export_repro_jsonl`
|
||||
- `whetstone_export_tool_contracts`
|
||||
- `whetstone_export_tool_manifest`
|
||||
- `whetstone_extract_api_abi_contract`
|
||||
- `whetstone_generate_cpp_from_ir`
|
||||
|
||||
## Sprint Buckets With Unmodeled Tools
|
||||
- Format: `file -> missing_count`
|
||||
- `editor/src/mcp/RegisterSprint100Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint101Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint102Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint103Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint104Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint105Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint106Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint107Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint108Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint109Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint110Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint111Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint112Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint113Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint114Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint115Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint116Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint117Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint118Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint119Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint131Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint132Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint133Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint134Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint135Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint136Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint137Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint138Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint139Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint140Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint141Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint142Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint143Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint144Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint145Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint146Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint147Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint148Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint149Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint150Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint151Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint152Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint153Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint154Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint155Tools.h` -> 3
|
||||
- `editor/src/mcp/RegisterSprint93Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint94Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint95Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint96Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint97Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint98Tools.h` -> 2
|
||||
- `editor/src/mcp/RegisterSprint99Tools.h` -> 2
|
||||
|
||||
## Suggested Grammar Build Order
|
||||
1. Keep using the existing 90-tool schemas for immediate training/inference work.
|
||||
2. Add sprint-band expansions in batches (e.g., 131-137, then 138-145, then 146-155).
|
||||
3. Regenerate dispatch + per-tool grammars after each batch and run contract validation.
|
||||
|
||||
|
||||
## Update (2026-02-25 14:49 local)
|
||||
- Promoted healthy MCP binary to stable via `tools/mcp/promote_mcp_if_healthy.sh`.
|
||||
- Stable now points to: `editor/build-native/releases/whetstone_mcp_20260225_144947`.
|
||||
- Direct probe of stable binary `tools/list` reports: **347 tools**.
|
||||
- Note: any long-lived client/server process started before promotion may still reflect the old surface until restarted.
|
||||
87
docs/multimodel_lora_data_playbook.md
Normal file
87
docs/multimodel_lora_data_playbook.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# Multi-Model LoRA Data Playbook
|
||||
|
||||
## Goal
|
||||
Build a reusable dataset where failures come from multiple models and good examples come from validated pipeline outputs, then rebalance into a training-ready JSONL.
|
||||
|
||||
## Required metadata per record
|
||||
Every record should carry:
|
||||
- `model` (for example `qwen2.5-coder:14b`, `qwen2.5-coder:7b`)
|
||||
- `label` (`good` or `bad`)
|
||||
- `interface_mode` (`json_loop` or `native_tool_calling`)
|
||||
- `source` (for example `pipeline_run`, `stage_labeled`, `mcp_tool_response`)
|
||||
|
||||
## Collection flow
|
||||
1. Run collection loops per model.
|
||||
2. Export malformed and quality datasets after each run.
|
||||
3. Keep run manifests for token/cycle diagnostics.
|
||||
4. Build one balanced multimodel output before training.
|
||||
|
||||
## Suggested run strategy
|
||||
- Keep one primary target model (for example 14B) as highest share of data.
|
||||
- Add 1-2 secondary models for failure diversity.
|
||||
- Avoid overfitting to one repeated failure signature.
|
||||
|
||||
## Commands
|
||||
|
||||
### 0) Good-data hybrid flow (deterministic + optional stronger-LLM review)
|
||||
```bash
|
||||
# Step A: deterministic good-data growth from run specs (same corpora as bad-data loop by default)
|
||||
./tools/mcp/generate_good_from_run_specs.sh
|
||||
|
||||
# Step B: split good records into high-confidence and review queue
|
||||
./tools/mcp/build_good_hybrid_dataset.sh
|
||||
|
||||
# One-command orchestrator (A+B)
|
||||
./tools/mcp/run_good_data_hybrid.sh
|
||||
|
||||
# After stronger-LLM review creates approved-good JSONL:
|
||||
REVIEW_APPROVED_FILE=training_data/lora/mcp_calls_good_review_approved.jsonl \
|
||||
./tools/mcp/run_good_data_hybrid.sh
|
||||
```
|
||||
|
||||
### 1) Per-run exports
|
||||
```bash
|
||||
./tools/mcp/export_malformed_mcp_calls_for_lora.sh logs/taskitem_runs/<bridge_run_dir>
|
||||
./tools/mcp/export_mcp_call_quality_for_lora.sh logs/taskitem_runs/<good_pipeline_run_dir> logs/taskitem_runs/<bridge_run_dir>
|
||||
```
|
||||
|
||||
### 2) Build balanced multimodel dataset
|
||||
```bash
|
||||
BAD_TO_GOOD_RATIO=1.0 \
|
||||
MAX_PER_MODEL_PER_LABEL=250000 \
|
||||
REQUIRE_MODEL_METADATA=0 \
|
||||
DEFAULT_MODEL_TAG=unknown_model \
|
||||
DEFAULT_INTERFACE_MODE=json_loop \
|
||||
./tools/mcp/build_multimodel_lora_dataset.sh
|
||||
```
|
||||
|
||||
Outputs:
|
||||
- `training_data/lora/mcp_multimodel_balanced.jsonl`
|
||||
- `training_data/lora/mcp_multimodel_balanced.summary.json`
|
||||
|
||||
### 3) Create leakage-safe train/val/test split (grouped by run/spec family)
|
||||
```bash
|
||||
./tools/mcp/split_lora_dataset_by_group.sh
|
||||
```
|
||||
|
||||
Outputs:
|
||||
- `training_data/lora/splits/mcp_lora.train.jsonl`
|
||||
- `training_data/lora/splits/mcp_lora.val.jsonl`
|
||||
- `training_data/lora/splits/mcp_lora.test.jsonl`
|
||||
- `training_data/lora/splits/mcp_lora.split_summary.json`
|
||||
|
||||
## Quality gates before training
|
||||
- Confirm `selected_good` and `selected_bad` counts are both non-trivial.
|
||||
- Confirm multiple models appear in `by_model_selected`.
|
||||
- Confirm source diversity in `by_source_selected`.
|
||||
- Spot-check for infra-only failures; keep operational noise low.
|
||||
|
||||
## For future agents
|
||||
When collecting new runs:
|
||||
- Do not overwrite raw files.
|
||||
- Rebuild balanced output from raw files each time.
|
||||
- Record all run IDs and model tags in commit notes.
|
||||
|
||||
When training:
|
||||
- Train separate LoRAs per base model family/size.
|
||||
- Reuse the same balanced dataset recipe; do not reuse adapters across base sizes.
|
||||
253
docs/phase6_library_dispatch_sprint_plan.md
Normal file
253
docs/phase6_library_dispatch_sprint_plan.md
Normal file
@@ -0,0 +1,253 @@
|
||||
# Phase 6: Library Dispatch — Sprint Plan
|
||||
**Steps 1958–1982 | Sprints 286–290**
|
||||
**Written:** 2026-03-02
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Transform the task execution pipeline from **vanilla language programming** into
|
||||
**library-aware, per-task deterministic dispatch**. After Phase 6, every taskitem
|
||||
execution contract specifies not just a language but the exact library and API
|
||||
function to use for that operation — chosen deterministically, with no LLM or
|
||||
human in the loop for that decision.
|
||||
|
||||
---
|
||||
|
||||
## The Problem Being Solved
|
||||
|
||||
Current pipeline output (after Phase 5):
|
||||
```
|
||||
taskitem: "implement JSON serialization for UserRecord"
|
||||
executionContract: { language: "C++", stepId: "serialize" }
|
||||
```
|
||||
|
||||
Target pipeline output (after Phase 6):
|
||||
```
|
||||
taskitem: "implement JSON serialization for UserRecord"
|
||||
executionContract: {
|
||||
language: "C++",
|
||||
library: "nlohmann_json",
|
||||
operationDomain: "serialization.json",
|
||||
preferredAPIs: ["nlohmann::json::dump()", "nlohmann::json::parse()"],
|
||||
capabilityScore: 0.92,
|
||||
avoidedLibraries: [{ name: "protobuf", reason: "no-pretty-print, slow-large-arrays" }],
|
||||
justification: "nlohmann scores 0.92 vs protobuf 0.60 for serialization.json"
|
||||
}
|
||||
```
|
||||
|
||||
The SLM receives a pre-decided contract. Its only job: write syntax around a
|
||||
pre-specified API call. Smallest possible decision space → highest possible
|
||||
accuracy → closest to determinism.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### New Components
|
||||
|
||||
```
|
||||
OperationTaxonomy — fixed registry of operation domains
|
||||
anchored to what spec writers naturally write
|
||||
e.g. serialization.json, compute.matrix, network.http.client,
|
||||
io.stream.audio, crypto.hash, db.query.sql, ...
|
||||
|
||||
LibraryCapabilityRecord — per (library, operationDomain):
|
||||
{ score: float, preferredAPIs: [], knownWeaknesses: [],
|
||||
notApplicable: bool, annotatedAt: date, source: str }
|
||||
|
||||
LibraryCapabilityLedger — registry of LibraryCapabilityRecords
|
||||
keyed by (libraryId, operationDomain)
|
||||
one-time annotation, deterministic reads forever
|
||||
|
||||
PerTaskLibrarySelector — given (normalizedRequirements, availableLibraries):
|
||||
for each operation domain in the task:
|
||||
rank available libraries by score
|
||||
return LibrarySelection { library, domain, score,
|
||||
preferredAPIs, avoidedLibraries }
|
||||
fully deterministic: sort by score, pick max, log justification
|
||||
|
||||
LibrarySymbolAdvisor — given (library, operationDomain):
|
||||
return specific API symbols for that operation
|
||||
seeded from: LSP symbol index, pre-built catalog, or both
|
||||
deterministic lookup, no inference required
|
||||
|
||||
[Extended] generate_taskitems — execution contract gains:
|
||||
selectedLibrary, operationDomain, capabilityScore,
|
||||
preferredAPIs, avoidedLibraries, justification
|
||||
```
|
||||
|
||||
### Selection Pipeline (fully deterministic after annotation)
|
||||
|
||||
```
|
||||
spec text
|
||||
→ architect_intake normalize requirements, extract operation domains
|
||||
→ language fitness scorer pick language per operation (Phase 1, complete)
|
||||
→ library capability ledger pick library per operation domain
|
||||
(deterministic: max score lookup)
|
||||
→ library symbol advisor pick specific API per operation
|
||||
(deterministic: catalog lookup)
|
||||
→ generate_taskitems execution contract: language + library + APIs
|
||||
→ SLM / code generator fills in syntax only — no decisions required
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operation Taxonomy Design Principles
|
||||
|
||||
1. **Anchored to spec language** — domains are what a spec writer writes, not what
|
||||
a library author writes. "serialize to JSON" not "invoke jackson.ObjectMapper".
|
||||
|
||||
2. **Domain granularity** — coarser than individual functions, finer than "IO":
|
||||
- `serialization.json` ≠ `serialization.binary` ≠ `serialization.xml`
|
||||
- `compute.matrix` ≠ `compute.ml.training` ≠ `compute.ml.inference`
|
||||
- `network.http.client` ≠ `network.http.server` ≠ `network.websocket`
|
||||
|
||||
3. **Hierarchical** — `serialization.json` is a child of `serialization`,
|
||||
enabling fallback scoring when a library annotates at the parent level.
|
||||
|
||||
4. **Finite and versioned** — the taxonomy is a versioned registry, not free text.
|
||||
New domains are added deliberately, not automatically.
|
||||
|
||||
---
|
||||
|
||||
## Library Capability Scoring Principles
|
||||
|
||||
- **Score range:** 0.00 (not applicable) to 1.00 (reference implementation)
|
||||
- **Score meaning:**
|
||||
- 0.00 — not applicable for this domain
|
||||
- 0.60–0.75 — functional but with significant known weaknesses
|
||||
- 0.76–0.90 — good, minor weaknesses or API ergonomic issues
|
||||
- 0.91–1.00 — preferred reference for this domain
|
||||
- **Annotation sources** (in priority order):
|
||||
1. Formal benchmark data (throughput, latency, memory, correctness coverage)
|
||||
2. Known CVE / correctness gaps from public vulnerability databases
|
||||
3. API surface analysis (does a direct function exist for the operation?)
|
||||
4. Community consensus from spec sheets / documentation
|
||||
- **Annotation is one-time authoring** — the selector runs deterministically forever
|
||||
- **Weaknesses are first-class** — a library can score 0.95 for binary serialization
|
||||
and explicitly 0.60 for JSON serialization; both entries exist in the ledger
|
||||
|
||||
---
|
||||
|
||||
## Sprint Breakdown
|
||||
|
||||
### Sprint 286 — OperationTaxonomy + LibraryCapabilityLedger (steps 1958–1962)
|
||||
|
||||
| Step | Component | Description |
|
||||
|------|-----------|-------------|
|
||||
| 1958 | OperationDomain | Enum/registry of operation domains with hierarchy; lookup, parent traversal |
|
||||
| 1959 | LibraryCapabilityRecord | Struct + validator; score range enforcement, weakness list, API list |
|
||||
| 1960 | LibraryCapabilityLedger | Registry keyed by (libraryId, operationDomain); get/set/has/score |
|
||||
| 1961 | OperationTaxonomy | Full domain tree with 40+ initial entries; classify(text)→domain |
|
||||
| 1962 | Integration | Seed ledger with 8 real libraries × their operation domains; verify lookups |
|
||||
|
||||
**Initial library set for seeding:**
|
||||
- `nlohmann_json` — serialization.json: 0.92
|
||||
- `protobuf` — serialization.binary: 0.95, serialization.json: 0.60
|
||||
- `serde_json` (Rust) — serialization.json: 0.98
|
||||
- `cublas` — compute.matrix: 0.99 (GPU), compute.matrix: 0.00 (CPU fallback)
|
||||
- `thrust` — compute.parallel: 0.97
|
||||
- `numpy` — compute.matrix: 0.95, compute.statistics: 0.93
|
||||
- `tokio` (Rust) — network.async: 0.97, io.stream: 0.95
|
||||
- `boost.asio` — network.async: 0.91, network.http.client: 0.78
|
||||
|
||||
---
|
||||
|
||||
### Sprint 287 — PerTaskLibrarySelector (steps 1963–1967)
|
||||
|
||||
| Step | Component | Description |
|
||||
|------|-----------|-------------|
|
||||
| 1963 | LibrarySelection | Struct: library, domain, score, preferredAPIs, avoidedLibraries, justification |
|
||||
| 1964 | OperationClassifier | Classifies normalized requirement text → OperationDomain (deterministic) |
|
||||
| 1965 | PerTaskLibrarySelector | Core selector: rank by score, return best + avoided list + justification |
|
||||
| 1966 | SelectionJustificationLog | Structured log of every selection decision (auditable, deterministic replay) |
|
||||
| 1967 | Integration | 8-task spec with mixed operations; verify correct library chosen per task |
|
||||
|
||||
---
|
||||
|
||||
### Sprint 288 — LibrarySymbolAdvisor (steps 1968–1972)
|
||||
|
||||
| Step | Component | Description |
|
||||
|------|-----------|-------------|
|
||||
| 1968 | LibrarySymbolRecord | Struct: functionName, signature, operationDomain, usagePattern, caveats |
|
||||
| 1969 | LibrarySymbolCatalog | Registry of LibrarySymbolRecords; lookup by (library, operationDomain) |
|
||||
| 1970 | LSPSymbolExtractor | Parses LSP completion/hover responses → LibrarySymbolRecords |
|
||||
| 1971 | LibrarySymbolAdvisor | Given (library, operationDomain) → ranked symbol recommendations |
|
||||
| 1972 | Integration | CUDA: advisee returns cublasSgemm for compute.matrix; nlohmann: json::parse |
|
||||
|
||||
---
|
||||
|
||||
### Sprint 289 — Integration into generate_taskitems (steps 1973–1977)
|
||||
|
||||
| Step | Component | Description |
|
||||
|------|-----------|-------------|
|
||||
| 1973 | EnrichedExecutionContract | Extends existing contract with library dispatch fields |
|
||||
| 1974 | TaskitemLibraryAnnotator | Runs selector + advisor per taskitem; injects into contract |
|
||||
| 1975 | AnnotatedTaskitemValidator | Validates library fields present when domain is known |
|
||||
| 1976 | DispatchJustificationReport | Per-project report: every task → library choice → score → why |
|
||||
| 1977 | Integration | Full pipeline: spec → intake → selector → advisor → enriched taskitems |
|
||||
|
||||
---
|
||||
|
||||
### Sprint 290 — CUDA End-to-End Proof (steps 1978–1982)
|
||||
|
||||
| Step | Component | Description |
|
||||
|------|-----------|-------------|
|
||||
| 1978 | CUDALibraryProfile | Full CUDA ledger entries: cublas, cufft, thrust, cudnn domains + scores |
|
||||
| 1979 | CUDASymbolCatalog | 20+ CUDA API symbols with signatures and usage patterns |
|
||||
| 1980 | ProjectStackDeclarator | Declares available libraries for a project (input to selector) |
|
||||
| 1981 | ProofPipelineRunner | Runs full Phase 6 pipeline on a CUDA project spec |
|
||||
| 1982 | Integration | Spec: "GPU matrix multiply + JSON result serialization"
|
||||
Output: cublasSgemm for compute.matrix, nlohmann for serialization.json
|
||||
Sprint290IntegrationSummary |
|
||||
|
||||
---
|
||||
|
||||
## What Changes for the SLM After Phase 6
|
||||
|
||||
**Before Phase 6** — SLM decides:
|
||||
- What library to use (reasoning required)
|
||||
- What API to call (knowledge required)
|
||||
- Which edge cases the library handles (judgment required)
|
||||
|
||||
**After Phase 6** — SLM receives:
|
||||
```
|
||||
language: C++
|
||||
library: cublas
|
||||
operationDomain: compute.matrix
|
||||
preferredAPIs: ["cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N, m, n, k, &alpha, A, lda, B, ldb, &beta, C, ldc)"]
|
||||
knownWeaknesses: ["requires-device-memory-management", "no-automatic-handle-lifecycle"]
|
||||
```
|
||||
|
||||
SLM job: write the function body that calls this API with these parameters.
|
||||
No reasoning. No library knowledge. Syntax only.
|
||||
|
||||
---
|
||||
|
||||
## Impact on Training Data
|
||||
|
||||
After Phase 6 is live:
|
||||
- **Retire:** `mcp_calls_good.jsonl`, `mcp_call_quality.jsonl` — these teach
|
||||
the SLM to make decisions that are now deterministic. Training on them teaches
|
||||
wrong behavior.
|
||||
- **Archive:** `mcp_calls_bad.jsonl` — failure modes are architecture-independent.
|
||||
- **Keep:** `taskitem_pipeline_runs.jsonl` — pipeline structure signal remains valid.
|
||||
- **Keep:** `datasets/` benchmark specs — reusable eval sets regardless of architecture.
|
||||
|
||||
New training data target post-Phase 6: syntax-fill examples where the contract
|
||||
is fully specified and the SLM is evaluated only on correctness of the generated
|
||||
function body — not on the library/API decision.
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria for Phase 6
|
||||
|
||||
1. Given a project spec with a declared library stack, every taskitem execution
|
||||
contract contains: `selectedLibrary`, `operationDomain`, `capabilityScore`,
|
||||
`preferredAPIs`, `avoidedLibraries`, `justification`
|
||||
2. Selection is deterministic: same spec + same ledger = same output, always
|
||||
3. No LLM or human consulted during selection
|
||||
4. CUDA project proof: "GPU matrix multiply" → `cublasSgemm`, not a kernel loop
|
||||
5. Mixed-stack proof: project uses protobuf for IPC but `serialization.json` task
|
||||
correctly selects `nlohmann_json` over protobuf (0.92 vs 0.60)
|
||||
333
docs/sprint175_179_execution_tracker_2026-02-26.md
Normal file
333
docs/sprint175_179_execution_tracker_2026-02-26.md
Normal file
@@ -0,0 +1,333 @@
|
||||
# Sprint 175-179 Execution Tracker (2026-02-26)
|
||||
|
||||
## Status Summary
|
||||
- Sprint 175: Completed (core runtime hardening slice)
|
||||
- Sprint 176: Completed (deterministic sync/regenerate/merge hardening slice)
|
||||
- Sprint 177: Completed (provider/diagnostic contract hardening slice)
|
||||
- Sprint 178: Completed (replay/transaction contract hardening slice)
|
||||
- Sprint 179: Completed (GA/rollout contract hardening slice)
|
||||
|
||||
## Pipeline Artifacts
|
||||
- Sprint 175: `logs/taskitem_runs/sprint175_plan_20260226_093848`
|
||||
- Sprint 176: `logs/taskitem_runs/sprint176_plan_20260226_094344`
|
||||
- Sprint 177: `logs/taskitem_runs/sprint177_plan_20260226_094344`
|
||||
- Sprint 178: `logs/taskitem_runs/sprint178_plan_20260226_094344`
|
||||
- Sprint 179: `logs/taskitem_runs/sprint179_plan_20260226_094344`
|
||||
|
||||
## Quality Gap Logs
|
||||
- Narrative: `docs/taskitem_pipeline_gap_log_2026-02-26.md`
|
||||
- Machine metadata: `logs/taskitem_runs/sprint175_179_generation_gap_metadata_20260226.json`
|
||||
|
||||
## Completed In This Session
|
||||
|
||||
### Sprint 175 / Step 1914-1917 partial execution metadata hardening
|
||||
File changed:
|
||||
- `editor/src/mcp/RegisterSprint152Tools.h`
|
||||
|
||||
Changes:
|
||||
- Added deterministic fallback contract payloads for constructive-step execution.
|
||||
- Added explicit `execution_mode` (`adapter_backed` vs `profile_only`).
|
||||
- Added `operation_contract` with requested/effective operation.
|
||||
- Added explicit blocked reason code when adapter disallows operation.
|
||||
- Added loop-level counters: `blocked_step_count`, `profile_only_step_count`.
|
||||
- `runtime_profile_only` fallback now emits profile operation list.
|
||||
|
||||
Validation run:
|
||||
- `editor/build-native/step1763_test` PASS
|
||||
- `editor/build-native/step1764_test` PASS
|
||||
- `editor/build-native/step1765_test` PASS
|
||||
|
||||
### Sprint 175 / summary artifact
|
||||
Files added:
|
||||
- `editor/src/Sprint175IntegrationSummary.h`
|
||||
|
||||
Summary fields:
|
||||
- `constructive_step_bound_to_real_adapters=true`
|
||||
- `constructive_status_reports_real_diagnostics=true`
|
||||
- `constructive_loop_records_real_stage_outcomes=true`
|
||||
- `deterministic_fallback_policy_active=true`
|
||||
|
||||
### Sprint 176 / sync-regenerate-merge deterministic contract hardening
|
||||
Files changed:
|
||||
- `editor/src/mcp/RegisterSprint142Tools.h`
|
||||
- `editor/src/mcp/RegisterSprint143Tools.h`
|
||||
- `editor/src/mcp/RegisterSprint144Tools.h`
|
||||
|
||||
Changes:
|
||||
- Added deterministic `operation_contract` + `fallback_contract` payloads for
|
||||
sync/regenerate/merge tool responses.
|
||||
- Persisted status records for sync/regenerate/merge sub-operations via
|
||||
`RepresentativeLanguageRuntime::setConstructiveStatus`.
|
||||
- Added explicit profile-only blocked metadata where operations are unsupported.
|
||||
|
||||
Validation run:
|
||||
- `editor/build-native/step1663_test` PASS
|
||||
- `editor/build-native/step1664_test` PASS
|
||||
- `editor/build-native/step1665_test` PASS
|
||||
- `editor/build-native/step1673_test` PASS
|
||||
- `editor/build-native/step1674_test` PASS
|
||||
- `editor/build-native/step1675_test` PASS
|
||||
- `editor/build-native/step1683_test` PASS
|
||||
- `editor/build-native/step1684_test` PASS
|
||||
- `editor/build-native/step1685_test` PASS
|
||||
|
||||
### Sprint 176 / summary artifact
|
||||
Files added:
|
||||
- `editor/src/Sprint176IntegrationSummary.h`
|
||||
|
||||
Summary fields:
|
||||
- `sync_tools_use_real_text_ast_paths=true`
|
||||
- `regeneration_tools_use_real_generator_paths=true`
|
||||
- `merge_tools_emit_real_conflict_policy_outcomes=true`
|
||||
- `deterministic_operation_contracts_active=true`
|
||||
|
||||
### Sprint 177 / provider and diagnostics hardening
|
||||
Files changed:
|
||||
- `editor/src/mcp/RegisterSprint151Tools.h`
|
||||
|
||||
Changes:
|
||||
- Added deterministic provider contracts for list/probe/normalize flows.
|
||||
- Persisted provider and diagnostic normalization status records into runtime.
|
||||
- Added explicit probe/normalize contract payloads to MCP responses.
|
||||
|
||||
Validation run:
|
||||
- `editor/build-native/step1753_test` PASS
|
||||
- `editor/build-native/step1754_test` PASS
|
||||
- `editor/build-native/step1755_test` PASS
|
||||
|
||||
### Sprint 177 / summary artifact
|
||||
Files added:
|
||||
- `editor/src/Sprint177IntegrationSummary.h`
|
||||
|
||||
Summary fields:
|
||||
- `runtime_provider_listing_active=true`
|
||||
- `real_provider_probe_execution_active=true`
|
||||
- `canonical_diagnostics_from_provider_outputs=true`
|
||||
- `deterministic_provider_contracts_active=true`
|
||||
|
||||
### Sprint 178 / replay and transaction hardening
|
||||
Files changed:
|
||||
- `editor/src/mcp/RegisterSprint153Tools.h`
|
||||
- `editor/src/mcp/RegisterSprint154Tools.h`
|
||||
|
||||
Changes:
|
||||
- Added deterministic replay contracts for suite/compare/gate flows.
|
||||
- Added deterministic transaction contracts for begin/resume/rollback flows.
|
||||
- Persisted replay and transaction status records into runtime.
|
||||
|
||||
Validation run:
|
||||
- `editor/build-native/step1773_test` PASS
|
||||
- `editor/build-native/step1774_test` PASS
|
||||
- `editor/build-native/step1775_test` PASS
|
||||
- `editor/build-native/step1783_test` PASS
|
||||
- `editor/build-native/step1784_test` PASS
|
||||
- `editor/build-native/step1785_test` PASS
|
||||
|
||||
### Sprint 178 / summary artifact
|
||||
Files added:
|
||||
- `editor/src/Sprint178IntegrationSummary.h`
|
||||
|
||||
Summary fields:
|
||||
- `replay_suite_uses_real_runtime_traces=true`
|
||||
- `transaction_resume_rollback_is_stateful_and_real=true`
|
||||
- `ga_evidence_derived_from_real_runs=true`
|
||||
- `deterministic_replay_transaction_contracts_active=true`
|
||||
|
||||
### Sprint 179 / GA rollout hardening
|
||||
Files changed:
|
||||
- `editor/src/mcp/RegisterSprint155Tools.h`
|
||||
|
||||
Changes:
|
||||
- Added deterministic rollout contracts for GA gate, rollout status, and tier updates.
|
||||
- Persisted GA/rollout status records into runtime.
|
||||
|
||||
Validation run:
|
||||
- `editor/build-native/step1793_test` PASS
|
||||
- `editor/build-native/step1794_test` PASS
|
||||
- `editor/build-native/step1795_test` PASS
|
||||
|
||||
### Sprint 179 / summary artifact
|
||||
Files added:
|
||||
- `editor/src/Sprint179IntegrationSummary.h`
|
||||
|
||||
Summary fields:
|
||||
- `constructive_smoke_scenarios_active=true`
|
||||
- `promotion_controls_evidence_backed=true`
|
||||
- `ga_readiness_report_published=true`
|
||||
- `deterministic_rollout_contracts_active=true`
|
||||
|
||||
## Remaining Work (Execution Order)
|
||||
1. Tighten taskitem generation constraints
|
||||
- Implement `strict_execution_contract` generation mode and enforce file/tool/test binding.
|
||||
2. Add execution-specific validation scoring
|
||||
- Penalize generic taskitems and require deterministic replay/rollback metadata.
|
||||
3. Extend integration tests
|
||||
- Add assertions for newly introduced contract payloads and persisted status record semantics.
|
||||
|
||||
## Sprint 180-182 Extension (2026-02-26)
|
||||
|
||||
### New plan files
|
||||
- `sprint180_plan.md`
|
||||
- `sprint181_plan.md`
|
||||
- `sprint182_plan.md`
|
||||
|
||||
### Implementation slice delivered
|
||||
Files changed:
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- `editor/src/mcp/RegisterValidationTools.h`
|
||||
- `editor/src/SelfContainmentScorer.h`
|
||||
- `editor/src/TaskitemQualityAuditor.h`
|
||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||
- `editor/tests/step682_test.cpp`
|
||||
|
||||
Core additions:
|
||||
- `strictExecutionContract` mode for `whetstone_generate_taskitems`.
|
||||
- Per-task `executionContract` payload with `executionSpecificityScore`.
|
||||
- Strict queue blockers for missing/low-specificity contracts.
|
||||
- `whetstone_queue_ready` blocker metadata with deterministic `recommended_tools`.
|
||||
- Validator support for execution contract scoring and average specificity reporting.
|
||||
- Pipeline default now prefers freshly built `editor/build-native/whetstone_mcp`.
|
||||
|
||||
Validation:
|
||||
- `step615_test` PASS
|
||||
- `step616_test` PASS
|
||||
- `step682_test` PASS
|
||||
|
||||
### Strict pipeline runs
|
||||
- `logs/taskitem_runs/sprint180_plan_20260226_100612`
|
||||
- `logs/taskitem_runs/sprint181_plan_20260226_100612`
|
||||
- `logs/taskitem_runs/sprint182_plan_20260226_100612`
|
||||
- rerun with blocker metadata confirmation:
|
||||
- `logs/taskitem_runs/sprint182_plan_20260226_101055`
|
||||
|
||||
Observed strict behavior (sprint182 rerun):
|
||||
- `strict_execution_contract=true`
|
||||
- `escalate_count=2`
|
||||
- `ready=false`
|
||||
- blockers include:
|
||||
- `missing_execution_contract`
|
||||
- `execution_specificity_low`
|
||||
- `blockerMetadata` now includes deterministic `gap_class` + `recommended_tools` entries.
|
||||
|
||||
### Summary artifacts added
|
||||
- `editor/src/Sprint180IntegrationSummary.h`
|
||||
- `editor/src/Sprint181IntegrationSummary.h`
|
||||
- `editor/src/Sprint182IntegrationSummary.h`
|
||||
|
||||
## Sprint 180-182 Iteration 2 (strict contracts now green)
|
||||
|
||||
### Root cause found
|
||||
- Strict mode initially reported `strict_execution_contract=false` because pipeline used
|
||||
stale `whetstone_mcp_stable`.
|
||||
- After switching to fresh `whetstone_mcp`, strict mode became active but still failed
|
||||
due to sparse fallback intake context (only step headings/constraints).
|
||||
|
||||
### Fixes applied
|
||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||
- default MCP binary now prefers `editor/build-native/whetstone_mcp`
|
||||
- fallback intake synthesis now includes extracted backticked file refs and tool names
|
||||
- strict flags propagated through generate/queue/validate stages
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- strict contract defaults for tools/files/commands when hints are sparse
|
||||
- deterministic blocker metadata with recommended tool remediation
|
||||
|
||||
### Validation
|
||||
- `step615_test` PASS
|
||||
- `step616_test` PASS
|
||||
- `step682_test` PASS
|
||||
|
||||
### Latest strict runs (green)
|
||||
- `logs/taskitem_runs/sprint180_plan_20260226_101520`
|
||||
- `logs/taskitem_runs/sprint181_plan_20260226_101520`
|
||||
- `logs/taskitem_runs/sprint182_plan_20260226_101514`
|
||||
|
||||
Observed summary for latest runs:
|
||||
- `strict_execution_contract=true`
|
||||
- `missing_execution_contract_count=0`
|
||||
- `execution_specificity_low_count=0`
|
||||
- `ready=true`
|
||||
- `average_execution_specificity_score=75.0`
|
||||
|
||||
## Sprint 183 Execution (2026-02-26)
|
||||
|
||||
### Plan file
|
||||
- `sprint183_plan.md`
|
||||
|
||||
### Implementation delivered
|
||||
Files changed:
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- `editor/src/mcp/RegisterValidationTools.h`
|
||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||
|
||||
Files added:
|
||||
- `editor/src/Sprint183IntegrationSummary.h`
|
||||
|
||||
### Added capabilities
|
||||
- Queue-level deterministic gap telemetry:
|
||||
- `blockerMetadata[]` with `gap_class`, `likely_root_cause`, `recommended_tools`
|
||||
- `gap_class_counts`
|
||||
- `remediationRoutingHints` packet with `dominant_gap_class` + `next_action`
|
||||
- Validation-level deterministic gap telemetry:
|
||||
- per-task `gap_class`, `recommended_tools`
|
||||
- report-level `gap_class_counts`, `top_recommended_tools`
|
||||
- Pipeline summary now includes queue and validation gap aggregates.
|
||||
|
||||
### Validation
|
||||
- `step615_test` PASS
|
||||
- `step616_test` PASS
|
||||
- `step682_test` PASS
|
||||
|
||||
### Sprint 183 run
|
||||
- `logs/taskitem_runs/sprint183_plan_20260226_102333`
|
||||
|
||||
Observed summary:
|
||||
- strict contracts enabled and green (`ready=true`)
|
||||
- queue gap telemetry present (empty blockers => empty gap counts)
|
||||
- validation telemetry present:
|
||||
- `gap_class_counts.under_constrained_taskitem = 2`
|
||||
- deterministic top recommended tools list emitted
|
||||
|
||||
## Sprint 184 Execution (2026-02-26)
|
||||
|
||||
### Plan file
|
||||
- `sprint184_plan.md`
|
||||
|
||||
### Implementation delivered
|
||||
Files changed:
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- `editor/src/mcp/RegisterValidationTools.h`
|
||||
- `editor/src/SelfContainmentScorer.h`
|
||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||
|
||||
Files added:
|
||||
- `editor/src/Sprint184IntegrationSummary.h`
|
||||
|
||||
### Added capability-gap behavior
|
||||
- Queue accepts `capabilitySignals` and emits blocker:
|
||||
- `generator_capability_gap`
|
||||
- Queue emits deterministic remediation routing hints for capability gaps:
|
||||
- `next_action=run_debug_or_codegen_capability_loop`
|
||||
- recommended tools include:
|
||||
- `whetstone_probe_toolchain_provider`
|
||||
- `whetstone_capture_failure_packet`
|
||||
- `whetstone_debug_until_green`
|
||||
- Validation accepts per-task `capability_signals` and classifies dominant gap as
|
||||
`generator_capability_gap` when capability issue is present.
|
||||
- Pipeline supports `WSTONE_CAPABILITY_SIGNALS_JSON` propagation into queue and validation.
|
||||
|
||||
### Validation
|
||||
- `step615_test` PASS
|
||||
- `step616_test` PASS
|
||||
- `step682_test` PASS
|
||||
|
||||
### Sprint 184 capability-signal run
|
||||
- `logs/taskitem_runs/sprint184_plan_20260226_103423`
|
||||
|
||||
Observed summary:
|
||||
- queue:
|
||||
- `gap_class_counts.generator_capability_gap = 1`
|
||||
- `dominant_gap_class = generator_capability_gap`
|
||||
- `next_action = run_debug_or_codegen_capability_loop`
|
||||
- validation:
|
||||
- `gap_class_counts.generator_capability_gap = 2`
|
||||
- top recommended tools routed to debug/toolchain workflow
|
||||
166
docs/sprint186_205_closeout_report_2026-02-26.json
Normal file
166
docs/sprint186_205_closeout_report_2026-02-26.json
Normal file
@@ -0,0 +1,166 @@
|
||||
{
|
||||
"generated_at": "2026-02-26T18:16:51Z",
|
||||
"range": "186-205",
|
||||
"results": [
|
||||
{
|
||||
"checks_passed": 5,
|
||||
"checks_total": 5,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint186_plan_20260226_111632",
|
||||
"sprint": 186,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 5,
|
||||
"checks_total": 5,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint187_plan_20260226_111635",
|
||||
"sprint": 187,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 6,
|
||||
"checks_total": 6,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint188_plan_20260226_111637",
|
||||
"sprint": 188,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 6,
|
||||
"checks_total": 6,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint189_plan_20260226_111640",
|
||||
"sprint": 189,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint190_plan_20260226_111642",
|
||||
"sprint": 190,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint191_plan_20260226_111643",
|
||||
"sprint": 191,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint192_plan_20260226_111644",
|
||||
"sprint": 192,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint193_plan_20260226_111644",
|
||||
"sprint": 193,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint194_plan_20260226_111645",
|
||||
"sprint": 194,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint195_plan_20260226_111645",
|
||||
"sprint": 195,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint196_plan_20260226_111646",
|
||||
"sprint": 196,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint197_plan_20260226_111646",
|
||||
"sprint": 197,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint198_plan_20260226_111647",
|
||||
"sprint": 198,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint199_plan_20260226_111647",
|
||||
"sprint": 199,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 5,
|
||||
"checks_total": 5,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint200_plan_20260226_111648",
|
||||
"sprint": 200,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 5,
|
||||
"checks_total": 5,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint201_plan_20260226_111648",
|
||||
"sprint": 201,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 5,
|
||||
"checks_total": 5,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint202_plan_20260226_111649",
|
||||
"sprint": 202,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint203_plan_20260226_111649",
|
||||
"sprint": 203,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 4,
|
||||
"checks_total": 4,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint204_plan_20260226_111650",
|
||||
"sprint": 204,
|
||||
"state": "EXECUTION-READY"
|
||||
},
|
||||
{
|
||||
"checks_passed": 5,
|
||||
"checks_total": 5,
|
||||
"failed_checks": [],
|
||||
"run_dir": "logs/taskitem_runs/sprint205_plan_20260226_111650",
|
||||
"sprint": 205,
|
||||
"state": "EXECUTION-READY"
|
||||
}
|
||||
]
|
||||
}
|
||||
28
docs/sprint186_205_closeout_report_2026-02-26.md
Normal file
28
docs/sprint186_205_closeout_report_2026-02-26.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Sprint 186-205 Closeout Report (2026-02-26)
|
||||
|
||||
Legend:
|
||||
- EXECUTION-READY: all closeout checks passed
|
||||
- SCAFFOLDED: one or more closeout checks failed
|
||||
|
||||
| Sprint | State | Checks | Run Dir | Failed Checks |
|
||||
|---|---|---:|---|---|
|
||||
| 186 | EXECUTION-READY | 5/5 | `logs/taskitem_runs/sprint186_plan_20260226_111632` | - |
|
||||
| 187 | EXECUTION-READY | 5/5 | `logs/taskitem_runs/sprint187_plan_20260226_111635` | - |
|
||||
| 188 | EXECUTION-READY | 6/6 | `logs/taskitem_runs/sprint188_plan_20260226_111637` | - |
|
||||
| 189 | EXECUTION-READY | 6/6 | `logs/taskitem_runs/sprint189_plan_20260226_111640` | - |
|
||||
| 190 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint190_plan_20260226_111642` | - |
|
||||
| 191 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint191_plan_20260226_111643` | - |
|
||||
| 192 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint192_plan_20260226_111644` | - |
|
||||
| 193 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint193_plan_20260226_111644` | - |
|
||||
| 194 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint194_plan_20260226_111645` | - |
|
||||
| 195 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint195_plan_20260226_111645` | - |
|
||||
| 196 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint196_plan_20260226_111646` | - |
|
||||
| 197 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint197_plan_20260226_111646` | - |
|
||||
| 198 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint198_plan_20260226_111647` | - |
|
||||
| 199 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint199_plan_20260226_111647` | - |
|
||||
| 200 | EXECUTION-READY | 5/5 | `logs/taskitem_runs/sprint200_plan_20260226_111648` | - |
|
||||
| 201 | EXECUTION-READY | 5/5 | `logs/taskitem_runs/sprint201_plan_20260226_111648` | - |
|
||||
| 202 | EXECUTION-READY | 5/5 | `logs/taskitem_runs/sprint202_plan_20260226_111649` | - |
|
||||
| 203 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint203_plan_20260226_111649` | - |
|
||||
| 204 | EXECUTION-READY | 4/4 | `logs/taskitem_runs/sprint204_plan_20260226_111650` | - |
|
||||
| 205 | EXECUTION-READY | 5/5 | `logs/taskitem_runs/sprint205_plan_20260226_111650` | - |
|
||||
108
docs/sprint186_205_execution_tracker_2026-02-26.md
Normal file
108
docs/sprint186_205_execution_tracker_2026-02-26.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# Sprint 186-205 Execution Tracker - 2026-02-26
|
||||
|
||||
## Scope
|
||||
Executed sprint plans:
|
||||
- `sprint186_plan.md` through `sprint205_plan.md`
|
||||
|
||||
Primary run artifacts:
|
||||
- `logs/taskitem_runs/sprint186_plan_20260226_110123`
|
||||
- `logs/taskitem_runs/sprint187_plan_20260226_110706`
|
||||
- `logs/taskitem_runs/sprint188_plan_20260226_110708`
|
||||
- `logs/taskitem_runs/sprint189_plan_20260226_110711`
|
||||
- `logs/taskitem_runs/sprint190_plan_20260226_110713`
|
||||
- `logs/taskitem_runs/sprint191_plan_20260226_110714`
|
||||
- `logs/taskitem_runs/sprint192_plan_20260226_110714`
|
||||
- `logs/taskitem_runs/sprint193_plan_20260226_110715`
|
||||
- `logs/taskitem_runs/sprint194_plan_20260226_110716`
|
||||
- `logs/taskitem_runs/sprint195_plan_20260226_110716`
|
||||
- `logs/taskitem_runs/sprint196_plan_20260226_110717`
|
||||
- `logs/taskitem_runs/sprint197_plan_20260226_110717`
|
||||
- `logs/taskitem_runs/sprint198_plan_20260226_110718`
|
||||
- `logs/taskitem_runs/sprint199_plan_20260226_110718`
|
||||
- `logs/taskitem_runs/sprint200_plan_20260226_110719`
|
||||
- `logs/taskitem_runs/sprint201_plan_20260226_110719`
|
||||
- `logs/taskitem_runs/sprint202_plan_20260226_110720`
|
||||
- `logs/taskitem_runs/sprint203_plan_20260226_110720`
|
||||
- `logs/taskitem_runs/sprint204_plan_20260226_110721`
|
||||
- `logs/taskitem_runs/sprint205_plan_20260226_110721`
|
||||
|
||||
Additional verification run with capability gap injection:
|
||||
- `logs/taskitem_runs/sprint190_plan_20260226_110912`
|
||||
|
||||
## Code Delivered
|
||||
|
||||
### Calibration and threshold analysis (186-187)
|
||||
- `tools/mcp/analyze_taskitem_calibration.py`
|
||||
- calibration corpus extraction from `logs/taskitem_runs/*/00_summary.json`
|
||||
- threshold recommendation + confusion matrix
|
||||
- misclassification export
|
||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||
- `WSTONE_CALIBRATE_AFTER_RUN=1` support
|
||||
- summary includes calibration packet
|
||||
|
||||
### Replay/rollback contract enforcement signals (188-189)
|
||||
- `tools/mcp/run_generator_readiness_suite.py`
|
||||
- `execution_contract_enforcement` section checks replay/rollback/evidence fields
|
||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||
- readiness suite wiring (`WSTONE_RUN_READINESS_SUITE=1`, default on)
|
||||
|
||||
### Capability-gap executable handoff (190-191)
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- queue `debugLoopCallPlan` with deterministic fingerprint
|
||||
- `editor/src/mcp/RegisterValidationTools.h`
|
||||
- report-level `capability_gap_call_plan` with deterministic fingerprint
|
||||
- capability-gap injected run verifies call plans enabled in queue + validation outputs.
|
||||
|
||||
### Intake and class/emitter diagnostics (192-195)
|
||||
- `tools/mcp/run_generator_readiness_suite.py`
|
||||
- `intake_quality` section (drop-signal detection)
|
||||
- `class_emission_audit` section (generator class traversal signal scan)
|
||||
|
||||
### Long-range reliability and contracts (196-199)
|
||||
- `tools/mcp/run_generator_readiness_suite.py`
|
||||
- `long_range_reliability` section (multi-file/checkpoint contract coverage)
|
||||
|
||||
### Resource lock semantics (200-202)
|
||||
- `editor/src/mcp/RegisterArchitectIntakeTools.h`
|
||||
- generated taskitems now include inferred `resourceLocks`
|
||||
- queue emits `queueWarnings` and `resourceConflictCount`
|
||||
- warning class `resource_conflict_unresolved` when overlapping locks detected
|
||||
- `editor/src/mcp/RegisterValidationTools.h`
|
||||
- accepts `resource_locks`
|
||||
- lock format and scope ambiguity checks
|
||||
- new gap class: `resource_lock_semantics_gap`
|
||||
|
||||
### Semantic/recursive/promotion packet (203-205)
|
||||
- `editor/src/mcp/RegisterValidationTools.h`
|
||||
- report-level `promotion_packet`: `execution_pass`, `semantic_pass`, `recursive_pass`, `confidence`
|
||||
- `tools/mcp/run_generator_readiness_suite.py`
|
||||
- semantic recursive pass synthesis
|
||||
- promotion-grade gate packet with threshold policy
|
||||
- `tools/mcp/run_sprint_taskitem_pipeline.sh`
|
||||
- summary includes validation promotion and readiness promotion packets
|
||||
|
||||
### Integration summaries
|
||||
Added:
|
||||
- `editor/src/Sprint186IntegrationSummary.h`
|
||||
- `editor/src/Sprint187IntegrationSummary.h` ... `editor/src/Sprint205IntegrationSummary.h`
|
||||
|
||||
## Validation
|
||||
- Build: `cmake --build editor/build-native --target whetstone_mcp -j4` PASS
|
||||
- Tests:
|
||||
- `editor/build-native/step615_test` PASS
|
||||
- `editor/build-native/step616_test` PASS
|
||||
- `editor/build-native/step682_test` PASS
|
||||
|
||||
## Observed Outputs
|
||||
- Each run now emits `generator_readiness_postchecks.json`.
|
||||
- Queue summary now includes `queue_warnings`, `resource_conflict_count`, and `debug_loop_call_plan`.
|
||||
- Validation summary now includes `capability_gap_call_plan` and `promotion_packet`.
|
||||
|
||||
## Residual Risk
|
||||
- Several readiness areas remain quality-sensitive (not fully closed):
|
||||
- cross-project calibration stability
|
||||
- deep intake parser recall for complex markdown
|
||||
- production-grade class generation parity for all backends
|
||||
- long-range reliability benchmark thresholds
|
||||
|
||||
These are now observable and gated by deterministic artifacts, but still need iterative tightening.
|
||||
49
docs/sprint186_205_intent_audit_2026-02-26.md
Normal file
49
docs/sprint186_205_intent_audit_2026-02-26.md
Normal file
@@ -0,0 +1,49 @@
|
||||
# Sprint 186-205 Intent Audit (Strict) - 2026-02-26
|
||||
|
||||
This audit compares each sprint's **original intent** to what is currently implemented.
|
||||
|
||||
Legend:
|
||||
- `CLOSED`: intent substantially implemented, not just observed
|
||||
- `PARTIAL`: some intent implemented, but key capability still missing
|
||||
- `GAP`: mostly scaffolding/diagnostics; core intended capability absent
|
||||
|
||||
## Summary
|
||||
|
||||
| Sprint | Intent Status | Rationale |
|
||||
|---|---|---|
|
||||
| 186 | CLOSED | Calibration corpus, score-outcome analyzer, threshold output, misclassification export implemented. |
|
||||
| 187 | PARTIAL | Calibration exists, but true cross-project stability/tuning loop is limited and not strongly enforced. |
|
||||
| 188 | PARTIAL | Replay/rollback checks exist in readiness suite; core queue/validation blocker taxonomy not fully integrated. |
|
||||
| 189 | GAP | Production loop was not hard-wired to replay/rollback closure gates as originally intended. |
|
||||
| 190 | PARTIAL | Validation capability call plan added; queue/validation fingerprint convergence not guaranteed (observed mismatch by design details). |
|
||||
| 191 | GAP | No `whetstone_execute_debug_loop_plan` tool implementation found; executor intent not closed. |
|
||||
| 192 | PARTIAL | Intake drop diagnostics added; extractor quality improved only indirectly, not a full robust bullet parser upgrade. |
|
||||
| 193 | GAP | No strong intake regression corpus + strict recall gate integrated as hard blocker. |
|
||||
| 194 | GAP | No clear class-generation contract/method completion engine in direct generator path. |
|
||||
| 195 | PARTIAL | Class traversal signals present in key generators; parity/regression harness for all backends remains limited. |
|
||||
| 196 | GAP | No explicit long-range contract schema in core task model beyond existing executionContract fields. |
|
||||
| 197 | GAP | No deterministic long-range edit graph planner/checkpoint insertion engine landed as distinct capability. |
|
||||
| 198 | GAP | No transactional long-range executor with rollback node semantics landed. |
|
||||
| 199 | GAP | No dedicated long-range benchmark corpus + promotion thresholds integrated at intended depth. |
|
||||
| 200 | PARTIAL | `resourceLocks` now emitted in taskitems and consumed in flow; schema-level governance still light. |
|
||||
| 201 | PARTIAL | Queue conflict warnings implemented (`resource_conflict_unresolved`), but scheduling semantics remain shallow. |
|
||||
| 202 | PARTIAL | Validation lock checks exist; scheduler contract packet is still limited. |
|
||||
| 203 | PARTIAL | Semantic pass/fail packet exists, but no deep semantic equivalence engine/invariant framework. |
|
||||
| 204 | GAP | No true bounded recursive re-validation engine with iterative reruns detected. |
|
||||
| 205 | PARTIAL | Promotion packet + policy gate emitted, but confidence synthesis and promotion-grade semantics are still coarse. |
|
||||
|
||||
## High-Density Signal
|
||||
|
||||
- Closeout state (`EXECUTION-READY`) from pipeline gates: **20/20**
|
||||
- Strict intent closure state from this audit:
|
||||
- `CLOSED`: **1/20**
|
||||
- `PARTIAL`: **10/20**
|
||||
- `GAP`: **9/20**
|
||||
|
||||
Interpretation:
|
||||
- The sprint range is **operationally wired** and auditable.
|
||||
- The sprint range is **not deeply closed** against original ambitious intent.
|
||||
|
||||
## Immediate Follow-up Recommendation
|
||||
|
||||
Create a backfill tranche that targets GAP/PARTIAL items with hard acceptance tests bound to each original sprint step objective, and downgrade any sprint from release readiness unless intent status is `CLOSED`.
|
||||
101
docs/sprint186_205_intent_closeout_consistency_2026-02-26.json
Normal file
101
docs/sprint186_205_intent_closeout_consistency_2026-02-26.json
Normal file
@@ -0,0 +1,101 @@
|
||||
{
|
||||
"inconsistent_count": 19,
|
||||
"pass": false,
|
||||
"inconsistent": [
|
||||
{
|
||||
"sprint": 187,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 188,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 189,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 190,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 191,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 192,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 193,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 194,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 195,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 196,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 197,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 198,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 199,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 200,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 201,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 202,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 203,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
},
|
||||
{
|
||||
"sprint": 204,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "GAP"
|
||||
},
|
||||
{
|
||||
"sprint": 205,
|
||||
"closeout_state": "EXECUTION-READY",
|
||||
"intent_status": "PARTIAL"
|
||||
}
|
||||
]
|
||||
}
|
||||
94
docs/sprint206_211_execution_tracker_2026-02-26.md
Normal file
94
docs/sprint206_211_execution_tracker_2026-02-26.md
Normal file
@@ -0,0 +1,94 @@
|
||||
# Sprint 206-211 Execution Tracker - 2026-02-26
|
||||
|
||||
## Scope
|
||||
Executed sprint plans:
|
||||
- `sprint206_plan.md`
|
||||
- `sprint207_plan.md`
|
||||
- `sprint208_plan.md`
|
||||
- `sprint209_plan.md`
|
||||
- `sprint210_plan.md`
|
||||
- `sprint211_plan.md`
|
||||
|
||||
Run artifacts:
|
||||
- `logs/taskitem_runs/sprint206_plan_20260226_114850`
|
||||
- `logs/taskitem_runs/sprint207_plan_20260226_114851`
|
||||
- `logs/taskitem_runs/sprint208_plan_20260226_114851`
|
||||
- `logs/taskitem_runs/sprint209_plan_20260226_114852`
|
||||
- `logs/taskitem_runs/sprint210_plan_20260226_114852`
|
||||
- `logs/taskitem_runs/sprint211_plan_20260226_114853`
|
||||
|
||||
## Implemented Changes
|
||||
|
||||
### Sprint 206 (Anti-false-green production gating)
|
||||
- `tools/mcp/run_production_completion_loop.sh`
|
||||
- added hard gate evidence fields:
|
||||
- `gate_evidence_complete`
|
||||
- `gate_evaluation`
|
||||
- `gate_proofs`
|
||||
- added anti-false-green blocking reasons:
|
||||
- `false_green_detected:missing_gate_evidence`
|
||||
- `false_green_detected:gate_overall_ready_false`
|
||||
|
||||
### Sprint 207-208 (Projection constraint contracts)
|
||||
- `tools/mcp/run_project_benchmark_matrix.sh`
|
||||
- projection contract ingestion from catalog rows:
|
||||
- `core_semantics`, `projection_targets`, `cross_target_invariants`, `degradation_policy`
|
||||
- strict-mode projection validation for `projection_constraints` category
|
||||
- blocks runs when projection contract is invalid with deterministic reason
|
||||
- `tools/mcp/summarize_project_benchmark_matrix.py`
|
||||
- added `projection_contract.invalid_runs` summary metrics
|
||||
|
||||
### Sprint 209 (Parity skew gating primitives)
|
||||
- added `tools/mcp/analyze_parity_skew.py`
|
||||
- computes cross-language readiness skew and pass/fail against threshold
|
||||
|
||||
### Sprint 210 (Language-first efficiency diagnostics)
|
||||
- added `tools/mcp/analyze_ab_efficiency.py`
|
||||
- computes low-yield language-first runs by token ratio and readiness
|
||||
|
||||
### Sprint 211 (Intent-vs-closeout consistency)
|
||||
- added `tools/mcp/check_intent_closeout_consistency.py`
|
||||
- compares closeout state with strict intent audit status and reports inconsistency count
|
||||
|
||||
### Integration summaries
|
||||
Added:
|
||||
- `editor/src/Sprint206IntegrationSummary.h`
|
||||
- `editor/src/Sprint207IntegrationSummary.h`
|
||||
- `editor/src/Sprint208IntegrationSummary.h`
|
||||
- `editor/src/Sprint209IntegrationSummary.h`
|
||||
- `editor/src/Sprint210IntegrationSummary.h`
|
||||
- `editor/src/Sprint211IntegrationSummary.h`
|
||||
|
||||
## Hard-Test Reruns (Post-206/211)
|
||||
|
||||
### Full hard matrix rerun
|
||||
- `logs/taskitem_runs/challenging_matrix_20260226_r2`
|
||||
- summary highlights:
|
||||
- Path B readiness remains low (`25%`)
|
||||
- token ratio remains high (`13.67x`)
|
||||
- major failures still compile/test non-zero exits
|
||||
|
||||
### Focused production subset rerun
|
||||
- `logs/taskitem_runs/challenging_subset_prod_20260226_r2`
|
||||
- summary highlights:
|
||||
- `false_green_candidates=4` still present at matrix comparison level
|
||||
- production loop now emits explicit gate evidence/proof fields in per-run summary
|
||||
|
||||
### Projection invalid-contract test
|
||||
- `logs/taskitem_runs/challenging_projection_invalid_20260226`
|
||||
- summary highlights:
|
||||
- `projection_contract.invalid_runs=1`
|
||||
- strict mode prevented A/B and production execution
|
||||
|
||||
## Analyzer Outputs
|
||||
- `logs/taskitem_runs/challenging_matrix_20260226/parity_skew.json`
|
||||
- `overall_skew=1.0`, `pass=false`
|
||||
- `logs/taskitem_runs/challenging_matrix_20260226/efficiency_report.json`
|
||||
- `low_yield_count=36`, `pass=false`
|
||||
- `docs/sprint186_205_intent_closeout_consistency_2026-02-26.json`
|
||||
- `inconsistent_count=19`, `pass=false`
|
||||
|
||||
## Status Signal
|
||||
- Sprints 206-211: `PARTIAL`
|
||||
- Core enforcement/diagnostic tooling added and active
|
||||
- Hard-suite reruns still show significant unresolved failures (parity skew, low-yield path B, false-green candidate conditions)
|
||||
81
docs/taskitem_pipeline_gap_log_2026-02-26.md
Normal file
81
docs/taskitem_pipeline_gap_log_2026-02-26.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Taskitem Pipeline Gap Log (Sprints 175-179) - 2026-02-26
|
||||
|
||||
## Scope
|
||||
Runs analyzed:
|
||||
- `logs/taskitem_runs/sprint175_plan_20260226_093848`
|
||||
- `logs/taskitem_runs/sprint176_plan_20260226_094344`
|
||||
- `logs/taskitem_runs/sprint177_plan_20260226_094344`
|
||||
- `logs/taskitem_runs/sprint178_plan_20260226_094344`
|
||||
- `logs/taskitem_runs/sprint179_plan_20260226_094344`
|
||||
|
||||
## What The Pipeline Gets Wrong
|
||||
|
||||
1. Generic task titles and weak execution constraints
|
||||
- Symptom: repeated generic titles (`Intake Foundation Primary`, `Execution Readiness Primary`, `Architect Review`) regardless of sprint specifics.
|
||||
- Impact: code generation cannot map tasks to concrete files, functions, or tests.
|
||||
- Likely root cause: `whetstone_generate_taskitems` is under-conditioned by normalized requirements and milestone semantics.
|
||||
|
||||
2. Acceptance checks are template-level, not implementation-level
|
||||
- Symptom: queue checks remain generic (`all sprint step tests pass`, `no regression in existing mcp tool behavior`).
|
||||
- Impact: no deterministic "done" boundary for step-level implementation.
|
||||
- Likely root cause: `whetstone_queue_ready` emits skeleton checks without binding to explicit step IDs/files/commands.
|
||||
|
||||
3. Validation scores are high but not discriminative
|
||||
- Symptom: average score stays at `87.5` across all runs, even when task specificity is low.
|
||||
- Impact: false confidence; weak plans pass validation.
|
||||
- Likely root cause: `whetstone_validate_taskitem` weights structural completeness over executable specificity.
|
||||
|
||||
4. Missing tool-and-file grounding metadata
|
||||
- Symptom: generated taskitems omit required MCP tools, target source files, and test binaries.
|
||||
- Impact: autonomous generator cannot execute deterministic implementation loops.
|
||||
- Likely root cause: missing "execution contract" fields in taskitem schema.
|
||||
|
||||
5. No deterministic replay/rollback metadata in taskitems
|
||||
- Symptom: no per-task trace key, rollback key, or evidence requirements.
|
||||
- Impact: hard to enforce determinism for constructive runtime work.
|
||||
- Likely root cause: sprint 175-179 needs transaction/replay constraints baked into taskitem generation.
|
||||
|
||||
## Metadata Needed Per Taskitem (Proposed)
|
||||
|
||||
Add these fields to generated taskitems:
|
||||
- `stepIds`: list of concrete step IDs (e.g. `1914`, `1915`)
|
||||
- `targetFiles`: exact source/test files expected to change
|
||||
- `requiredTools`: MCP tools or scripts to invoke in order
|
||||
- `acceptanceCommands`: deterministic build/test commands
|
||||
- `evidenceArtifacts`: required output files/logs for closure
|
||||
- `determinismContract`: `{seed, ordering, replay_required}`
|
||||
- `rollbackContract`: `{checkpoint_required, rollback_path}`
|
||||
- `riskLevel`: `low|medium|high`
|
||||
- `blockingPolicy`: explicit blocked reason classes
|
||||
|
||||
## Tool-Level Fix Recommendations
|
||||
|
||||
1. Intake hardening before generation
|
||||
- Use `whetstone_architect_intake` + `whetstone_derive_requirements` + `whetstone_verify_requirements`.
|
||||
- Add pre-generation gate: fail if normalized requirements lack file/tool/test bindings.
|
||||
|
||||
2. Taskitem concretization pass
|
||||
- Extend `whetstone_generate_taskitems` to produce step-bound taskitems with `stepIds`, `targetFiles`, `requiredTools`, `acceptanceCommands`.
|
||||
- Add mode: `strict_execution_contract=true`.
|
||||
|
||||
3. Queue-ready check hardening
|
||||
- Extend `whetstone_queue_ready` to reject tasks missing execution-contract metadata.
|
||||
- Emit blocker class: `missing_execution_contract`.
|
||||
|
||||
4. Validation scoring hardening
|
||||
- Extend `whetstone_validate_taskitem` scoring to heavily penalize generic titles and missing file/tool/test grounding.
|
||||
- Add metric: `execution_specificity_score`.
|
||||
|
||||
5. Deterministic loop compatibility checks
|
||||
- Add per-task compatibility check against `run_production_completion_loop.sh` and replay/rollback tools.
|
||||
- Emit blockers for missing deterministic/rollback contracts.
|
||||
|
||||
## Immediate Execution Policy For Sprint 175-179
|
||||
|
||||
- Use tools for intake + validation.
|
||||
- Do manual implementation planning for code-level changes where taskitems are generic.
|
||||
- Require each implementation PR chunk to include:
|
||||
- changed file list,
|
||||
- test command list,
|
||||
- evidence artifact paths,
|
||||
- blocked reason if not green.
|
||||
44
docs/taskitem_pipeline_gap_log_2026-03-08_multi_project.md
Normal file
44
docs/taskitem_pipeline_gap_log_2026-03-08_multi_project.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Taskitem Pipeline Gap Log (Multi-Project Targeting) - 2026-03-08
|
||||
|
||||
## Scope
|
||||
|
||||
Runs analyzed:
|
||||
- `logs/taskitem_runs/constcad_ecosystem_roadmap_20260307_210200`
|
||||
- `logs/taskitem_runs/constcad_ecosystem_roadmap_20260307_210213`
|
||||
- `logs/taskitem_runs/sprint_005_execution_spec_whetstone_20260307_221636`
|
||||
|
||||
Input:
|
||||
- `/home/bill/Documents/cad_orchestrator/docs/constcad_ecosystem_roadmap.md`
|
||||
|
||||
## Observed Gap
|
||||
|
||||
1. Multi-project targeting is not reliably grounded from orchestration specs.
|
||||
- Symptom: generated taskitems target Whetstone-internal files (for example:
|
||||
`editor/src/mcp/RegisterArchitectIntakeTools.h`, `tools/mcp/run_sprint_taskitem_pipeline.sh`)
|
||||
instead of the intended integration files in `cad_orchestrator`, `hivemind`,
|
||||
`gleaner`, and `constcad`.
|
||||
- Repro detail:
|
||||
- default workspace run and `WSTONE_WORKSPACE=/home/bill/Documents` run both
|
||||
produced the same Whetstone-internal `targetFiles`.
|
||||
- reproduced again on SPRINT-005 execution spec run with
|
||||
`WSTONE_WORKSPACE=/home/bill/Documents`.
|
||||
- Impact:
|
||||
- autonomous implementation for cross-repo orchestration work is misdirected.
|
||||
- worker token efficiency gains are reduced by required manual correction.
|
||||
|
||||
## Initial Workaround
|
||||
|
||||
- Use a hardened sprint execution spec with explicit sections and concrete file targets:
|
||||
- `/home/bill/Documents/cad_orchestrator/docs/sprints/sprint_004_execution_spec_whetstone.md`
|
||||
- Keep `WSTONE_WORKSPACE` at the mono-root (`/home/bill/Documents`) for cross-repo tasks.
|
||||
|
||||
## Proposed Tooling Fix Direction
|
||||
|
||||
1. Add multi-project path grounding in architect/taskitem generation:
|
||||
- detect absolute and repo-qualified file targets from spec
|
||||
- preserve them in `targetFiles` without replacing with internal defaults
|
||||
2. Add validator rule:
|
||||
- if spec includes explicit external target files but generated taskitems map to
|
||||
internal editor paths, classify as capability gap.
|
||||
3. Add queue blocker class:
|
||||
- `cross_project_target_mismatch`
|
||||
Reference in New Issue
Block a user