Add Sprints 111-117: closed-loop iteration epoch
Sprint 111: IterationSession context primitive Sprint 112: EnvironmentSnapshot and diff Sprint 113: Build output feedback loop Sprint 114: Test failure feedback loop Sprint 115: Requirements derivation from AST Sprint 116: FeedbackLoopOrchestrator (wires 111-115) Sprint 117: HiveMind integration (iteration steps as distributed jobs) These sprints begin Epoch 3 — closed-loop determinism. Sprints 1-110 made code generation open-loop deterministic. Sprints 111+ make the full dev workflow (sense→act→observe→repeat) structurally deterministic, with HiveMind as the execution layer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
175
sprint111_plan.md
Normal file
175
sprint111_plan.md
Normal file
@@ -0,0 +1,175 @@
|
||||
# Sprint 111 Plan: Iteration Session Context
|
||||
|
||||
## Context
|
||||
|
||||
Sprints 1-110 established Whetstone as a deterministic *open-loop* generator:
|
||||
schema → struct, spec → taskitems, source → transpiled output. The output is
|
||||
fully determined by the input, with no environmental state involved.
|
||||
|
||||
Sprint 111 begins a new capability tier: **closed-loop iteration**. Real
|
||||
development is not open-loop. It involves running code, observing what breaks,
|
||||
forming a hypothesis, patching, and repeating until a goal is met. This sprint
|
||||
introduces the foundational primitive for that loop: the `IterationSession`.
|
||||
|
||||
An `IterationSession` is a structured record of one attempt to achieve a
|
||||
convergence goal (all tests pass, build succeeds, environment is ready). It
|
||||
tracks: the goal, each attempt taken, each observation made, and the current
|
||||
distance from the goal. Every subsequent sprint in this epoch builds on it.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Define the `IterationSession` schema with goal, attempt history, and state
|
||||
2. Define `AttemptRecord` and `ObservationRecord` as structured sub-types
|
||||
3. Define `GapModel` to measure distance from the convergence goal
|
||||
4. Implement a state machine: pending → active → converged / escalated / failed
|
||||
5. Persist sessions to sidecar files alongside annotated ASTs
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1339: `IterationSession` core schema (12 tests)
|
||||
|
||||
Create `editor/src/IterationSession.h`.
|
||||
|
||||
Fields: `session_id`, `goal` (string), `goal_type` (enum: build_success,
|
||||
tests_pass, environment_ready, custom), `status` (enum: pending, active,
|
||||
converged, escalated, failed), `created_at`, `updated_at`, `attempts[]`,
|
||||
`max_attempts` (default 10).
|
||||
|
||||
Tests (12): constructable with defaults, goal stored, goal_type round-trips,
|
||||
status defaults to pending, session_id is non-empty on construction,
|
||||
attempt list starts empty, max_attempts enforced, JSON serialization,
|
||||
JSON deserialization, unknown status survives passthrough,
|
||||
deterministic serialization order, schema version field present.
|
||||
|
||||
### Step 1340: `AttemptRecord` schema (10 tests)
|
||||
|
||||
Create `editor/src/AttemptRecord.h`.
|
||||
|
||||
Fields: `attempt_id`, `session_id`, `action_type` (enum: build, test, patch,
|
||||
probe, escalate), `action_description`, `started_at`, `completed_at`,
|
||||
`outcome` (enum: success, failure, partial, error), `observation_id`.
|
||||
|
||||
Tests (10): constructable, action_type round-trips, outcome defaults to error,
|
||||
duration computable from timestamps, JSON round-trip, unknown action_type
|
||||
passthrough, attempt_id non-empty, links to session_id, links to observation_id,
|
||||
deterministic serialization.
|
||||
|
||||
### Step 1341: `ObservationRecord` schema (10 tests)
|
||||
|
||||
Create `editor/src/ObservationRecord.h`.
|
||||
|
||||
Fields: `observation_id`, `source` (enum: build_output, test_output,
|
||||
process_stdout, process_stderr, file_read, env_probe), `raw_text`,
|
||||
`exit_code`, `duration_ms`, `structured_data` (JSON blob for parsed output),
|
||||
`truncated` (bool — raw_text may be capped at 16KB).
|
||||
|
||||
Tests (10): constructable, source enum round-trips, exit_code defaults to -1,
|
||||
truncated flag set when raw_text exceeds limit, structured_data round-trips,
|
||||
JSON serialization, deserialization, unknown source passthrough,
|
||||
observation_id non-empty, deterministic output.
|
||||
|
||||
### Step 1342: `GapModel` — distance from convergence goal (10 tests)
|
||||
|
||||
Create `editor/src/GapModel.h`.
|
||||
|
||||
Fields: `metric_type` (enum: error_count, test_failure_count, missing_deps,
|
||||
custom), `current_value`, `target_value`, `delta`, `converged` (bool —
|
||||
true when current_value == target_value), `trend` (enum: improving,
|
||||
stagnant, worsening — derived from last two values).
|
||||
|
||||
Tests (10): converged true when delta is zero, trend improving when delta
|
||||
decreasing, trend worsening when delta increasing, stagnant after two equal
|
||||
values, JSON round-trip, negative delta flagged as worsening, custom metric
|
||||
type passthrough, deterministic output, delta auto-computed, schema constructable.
|
||||
|
||||
### Step 1343: Session state machine transitions (10 tests)
|
||||
|
||||
Add `SessionStateMachine` to `IterationSession.h` (or small companion header).
|
||||
|
||||
Valid transitions:
|
||||
- pending → active (on first attempt recorded)
|
||||
- active → converged (on gap.converged == true)
|
||||
- active → escalated (on max_attempts reached without convergence)
|
||||
- active → failed (on attempt with outcome=error and no recovery path)
|
||||
- Any terminal state is immutable.
|
||||
|
||||
Tests (10): pending→active on first attempt, active→converged when gap closes,
|
||||
active→escalated at max_attempts, active→failed on terminal error,
|
||||
illegal transition rejected, terminal states immutable, state serialized,
|
||||
state deserialized, history preserved through transitions, deterministic
|
||||
transition log.
|
||||
|
||||
### Step 1344: Session persistence to sidecar file (10 tests)
|
||||
|
||||
Add `SessionStore` to a new `editor/src/SessionStore.h`.
|
||||
|
||||
Sidecar path convention: `.whetstone/sessions/<session_id>.json` relative
|
||||
to workspace root. `SessionStore::save(session)` writes atomically (write
|
||||
to `.tmp` then rename). `SessionStore::load(session_id)` reads and
|
||||
deserializes. `SessionStore::list()` returns all session IDs in the store.
|
||||
|
||||
Tests (10): save creates file, load round-trips session, list returns saved ids,
|
||||
overwrite preserves new state, missing session returns nullopt, atomic write
|
||||
(partial write does not corrupt), path derivation from workspace root,
|
||||
concurrent save does not corrupt (file lock), deterministic file layout,
|
||||
sidecar dir created on first save.
|
||||
|
||||
### Step 1345: `whetstone_start_iteration_session` MCP tool (8 tests)
|
||||
|
||||
Create `editor/src/mcp/RegisterIterationSessionTools.h`.
|
||||
|
||||
Input: `{ "goal": string, "goal_type": string, "max_attempts": int? }`
|
||||
Output: `{ "session_id": string, "status": "pending" }`
|
||||
|
||||
Persists session to SessionStore immediately on creation.
|
||||
|
||||
Tests (8): tool registered, schema validates, session_id returned,
|
||||
status is pending, session persisted to disk, missing goal_type defaults
|
||||
to "custom", max_attempts defaults to 10, deterministic output shape.
|
||||
|
||||
### Step 1346: `whetstone_record_attempt` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string, "action_type": string,
|
||||
"action_description": string, "observation": { ... } }`
|
||||
Output: `{ "attempt_id": string, "session_status": string, "gap": { ... }? }`
|
||||
|
||||
Appends AttemptRecord + ObservationRecord to session. Returns updated
|
||||
session status after state machine step.
|
||||
|
||||
Tests (8): tool registered, attempt appended, session_status updated,
|
||||
unknown session_id returns error, illegal transition returns error,
|
||||
gap field present when gap model updated, deterministic output,
|
||||
session persisted after record.
|
||||
|
||||
### Step 1347: `whetstone_get_session_state` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string }`
|
||||
Output: full session JSON with attempts[], gap, and current status.
|
||||
|
||||
Tests (8): tool registered, full session returned, attempts included,
|
||||
gap included when present, terminal sessions readable, missing session
|
||||
returns structured error, deterministic serialization, attempt count correct.
|
||||
|
||||
### Step 1348: Sprint 111 integration summary + regression (8 tests)
|
||||
|
||||
Create `editor/src/Sprint111IntegrationSummary.h`.
|
||||
|
||||
Tests (8): all new headers constructable, tools registered in initialize
|
||||
response, session round-trip end-to-end (start → record × 2 → converge →
|
||||
read), state machine covers all transitions, sidecar created on disk,
|
||||
no header exceeds 600 lines, deterministic snapshots, full prior regression
|
||||
passes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate
|
||||
|
||||
- Max 600 lines per header — split `IterationSessionExtended.h` if needed
|
||||
- No runtime dependencies beyond nlohmann-json and std::filesystem
|
||||
- Session IDs use UUIDv4 (reuse existing UUID utility if present, else add one)
|
||||
- All MCP tool outputs are stable and machine-readable
|
||||
- Sidecar format is versioned (`"schema_version": 1`)
|
||||
168
sprint112_plan.md
Normal file
168
sprint112_plan.md
Normal file
@@ -0,0 +1,168 @@
|
||||
# Sprint 112 Plan: Environment Snapshot and Diff
|
||||
|
||||
## Context
|
||||
|
||||
Before iterating on a failing environment, you must read it deterministically.
|
||||
"Environment" means the intersection of: installed packages, active config files,
|
||||
running processes, and file system state that your project depends on. Without a
|
||||
structured snapshot, the iteration session has no baseline to compare against and
|
||||
no way to tell whether a change actually helped.
|
||||
|
||||
Sprint 112 adds `EnvironmentSnapshot` — a structured, serializable record of the
|
||||
environment at a point in time — and `EnvironmentDiff` — the delta between two
|
||||
snapshots. These are the observability primitives that the build, test, and setup
|
||||
feedback loops in later sprints will use.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Define `EnvironmentSnapshot` schema covering deps, configs, processes, and paths
|
||||
2. Implement probes that read each domain from the actual system
|
||||
3. Define `EnvironmentDiff` as a typed delta between two snapshots
|
||||
4. Expose snapshot and diff as MCP tools
|
||||
5. Produce a setup verification report (what is present vs what is needed)
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1349: `EnvironmentSnapshot` core schema (12 tests)
|
||||
|
||||
Create `editor/src/EnvironmentSnapshot.h`.
|
||||
|
||||
Fields: `snapshot_id`, `captured_at`, `workspace_root`, `platform`
|
||||
(os, arch, shell), `packages[]` (name, version, source: apt/pip/npm/cargo/etc.),
|
||||
`config_files[]` (path, hash, key_values[] for known formats),
|
||||
`processes[]` (name, pid, port, status), `env_vars[]` (key, value — filtered
|
||||
to relevant set, never secrets).
|
||||
|
||||
Tests (12): constructable with defaults, snapshot_id non-empty, platform fields
|
||||
populated, packages list starts empty, config list starts empty, process list
|
||||
starts empty, env_vars list starts empty, JSON serialization, JSON deserialization,
|
||||
unknown package source passthrough, schema version present, deterministic output.
|
||||
|
||||
### Step 1350: Package state probe (10 tests)
|
||||
|
||||
Create `editor/src/PackageProbe.h`.
|
||||
|
||||
Probes installed packages from available managers on the current platform.
|
||||
Strategy: try each manager (dpkg-query, pip list, npm list -g, cargo --list)
|
||||
via subprocess, parse stdout into PackageRecord rows, tag with source.
|
||||
Failures are non-fatal — missing manager produces empty list for that source.
|
||||
|
||||
Tests (10): probe runs without crashing, result is parseable, known-installed
|
||||
package appears in results, unknown package absent, source tag correct,
|
||||
failed probe returns empty not error, deterministic sort order, version field
|
||||
populated, JSON round-trip, timeout respected (default 5s).
|
||||
|
||||
### Step 1351: Config file probe (10 tests)
|
||||
|
||||
Create `editor/src/ConfigFileProbe.h`.
|
||||
|
||||
Scans workspace for known config files: `.env`, `*.yaml`, `*.toml`, `*.json`
|
||||
(at workspace root depth only — no deep recursion). Reads content, computes
|
||||
SHA-256 hash, extracts key_values for .env and simple YAML/TOML maps.
|
||||
Sensitive keys (PASSWORD, SECRET, KEY, TOKEN) are redacted to `"[REDACTED]"`.
|
||||
|
||||
Tests (10): .env file parsed into key_values, sensitive keys redacted,
|
||||
hash computed and stable, yaml keys extracted, toml keys extracted,
|
||||
json keys extracted (top-level only), missing file produces absent entry,
|
||||
hash changes on content change, deterministic key ordering, depth limit
|
||||
respected.
|
||||
|
||||
### Step 1352: Process state probe (10 tests)
|
||||
|
||||
Create `editor/src/ProcessProbe.h`.
|
||||
|
||||
Lists running processes filtered to a known-relevant set (configurable list of
|
||||
process name substrings). Uses `/proc` on Linux, `ps` subprocess on macOS.
|
||||
Captures name, pid, and listening ports (from `ss -tlnp` or `netstat`).
|
||||
|
||||
Tests (10): probe runs without crashing, result parseable, known process
|
||||
appears when running, absent process not in list, port captured when listening,
|
||||
no port when not listening, deterministic sort, name filter respected,
|
||||
JSON round-trip, timeout respected.
|
||||
|
||||
### Step 1353: `EnvironmentDiff` schema and computation (10 tests)
|
||||
|
||||
Create `editor/src/EnvironmentDiff.h`.
|
||||
|
||||
Compares two `EnvironmentSnapshot` instances. Fields: `added_packages[]`,
|
||||
`removed_packages[]`, `changed_packages[]` (name + old_version + new_version),
|
||||
`added_configs[]`, `removed_configs[]`, `changed_configs[]` (path + old_hash
|
||||
+ new_hash), `added_processes[]`, `removed_processes[]`, `unchanged` (bool —
|
||||
true when all diff lists are empty).
|
||||
|
||||
Tests (10): empty diff on identical snapshots, unchanged true on identical,
|
||||
added package detected, removed package detected, changed package detected,
|
||||
added config detected, changed config hash detected, process added detected,
|
||||
JSON round-trip, deterministic diff ordering.
|
||||
|
||||
### Step 1354: Snapshot persistence (sidecar integration) (8 tests)
|
||||
|
||||
Extend `SessionStore.h` (or add `SnapshotStore.h`) to persist
|
||||
`EnvironmentSnapshot` alongside iteration sessions.
|
||||
|
||||
Convention: `.whetstone/snapshots/<snapshot_id>.json`
|
||||
|
||||
Tests (8): save creates file, load round-trips, list returns ids, overwrite
|
||||
preserves new data, missing snapshot returns nullopt, atomic write, path
|
||||
derivation correct, sidecar dir created on first save.
|
||||
|
||||
### Step 1355: Setup verification report (8 tests)
|
||||
|
||||
Create `editor/src/SetupVerificationReport.h`.
|
||||
|
||||
Given a `RequiredEnvironment` spec (a list of expected packages, config files,
|
||||
and processes) and a current `EnvironmentSnapshot`, produces a structured report:
|
||||
`satisfied[]`, `missing[]`, `version_mismatch[]`, `ready` (bool — true when
|
||||
missing and version_mismatch are both empty).
|
||||
|
||||
Tests (8): ready true when all present, missing package detected, version
|
||||
mismatch detected, correct version passes, empty required produces ready=true,
|
||||
JSON round-trip, deterministic output, report serializable.
|
||||
|
||||
### Step 1356: `whetstone_snapshot_environment` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "workspace": string, "process_filter": string[]? }`
|
||||
Output: full `EnvironmentSnapshot` JSON + `snapshot_id`.
|
||||
|
||||
Runs all three probes (packages, configs, processes) and persists result.
|
||||
|
||||
Tests (8): tool registered, snapshot_id returned, packages field present,
|
||||
configs field present, processes field present, snapshot persisted to disk,
|
||||
invalid workspace returns structured error, deterministic output shape.
|
||||
|
||||
### Step 1357: `whetstone_diff_environments` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "before_id": string, "after_id": string }`
|
||||
Output: `EnvironmentDiff` JSON + `unchanged` bool.
|
||||
|
||||
Loads both snapshots from store, computes diff, returns result.
|
||||
|
||||
Tests (8): tool registered, unchanged true on same snapshot, added package
|
||||
appears in diff, removed package appears, changed package appears, missing
|
||||
snapshot_id returns error, diff persisted to session if session_id provided,
|
||||
deterministic output.
|
||||
|
||||
### Step 1358: Sprint 112 integration summary + regression (8 tests)
|
||||
|
||||
Create `editor/src/Sprint112IntegrationSummary.h`.
|
||||
|
||||
End-to-end test: capture snapshot → mutate environment (add a dummy config
|
||||
file) → capture second snapshot → diff → verify change detected.
|
||||
|
||||
Tests (8): headers constructable, tools registered, end-to-end diff detects
|
||||
change, verification report correct, sidecar files created, no header exceeds
|
||||
600 lines, deterministic snapshots, full prior regression passes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate
|
||||
|
||||
- Probes are non-fatal: subprocess failure returns empty list, not panic
|
||||
- No new dependencies — subprocess via std::system or popen, parsing in C++
|
||||
- Sensitive data never stored in snapshots (redaction in ConfigFileProbe)
|
||||
- Max 600 lines per header
|
||||
- All MCP outputs stable and machine-readable
|
||||
196
sprint113_plan.md
Normal file
196
sprint113_plan.md
Normal file
@@ -0,0 +1,196 @@
|
||||
# Sprint 113 Plan: Build Output Feedback Loop
|
||||
|
||||
## Context
|
||||
|
||||
Build failures are the most common reason a development iteration stalls.
|
||||
They are also highly structured: compilers emit errors in machine-parseable
|
||||
formats (file, line, col, error code, message). That structure is currently
|
||||
discarded — the developer reads the raw output and decides what to fix.
|
||||
|
||||
Sprint 113 makes build output a first-class Whetstone input. A build failure
|
||||
becomes a structured `BuildErrorRecord`, which is located in the AST to
|
||||
identify the responsible node, classified by error type, and used to generate
|
||||
a ranked list of fix candidates. Each candidate is a concrete AST mutation
|
||||
with a confidence score. The result integrates with the IterationSession from
|
||||
Sprint 111 — each build attempt becomes a recorded attempt with a
|
||||
structured observation.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Parse compiler output (GCC, Clang, CMake) into structured `BuildErrorRecord`s
|
||||
2. Locate each error in the AST by file + line + col
|
||||
3. Classify error types (missing dep, type error, undefined symbol, etc.)
|
||||
4. Generate ranked fix candidates as AST mutations
|
||||
5. Integrate with IterationSession (build attempt → AttemptRecord)
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1359: `BuildErrorRecord` schema (12 tests)
|
||||
|
||||
Create `editor/src/BuildErrorRecord.h`.
|
||||
|
||||
Fields: `error_id`, `tool` (enum: gcc, clang, clang_tidy, cmake, linker,
|
||||
unknown), `severity` (enum: error, warning, note), `file_path`, `line`,
|
||||
`col`, `error_code` (e.g. "E0137", "-Wunused"), `message`, `context_lines[]`
|
||||
(surrounding source lines captured from file), `ast_node_id` (populated by
|
||||
locator, initially empty).
|
||||
|
||||
Tests (12): constructable with defaults, tool enum round-trips, severity
|
||||
defaults to error, file_path stored, line/col stored, error_code stored,
|
||||
message stored, context_lines starts empty, ast_node_id starts empty,
|
||||
JSON serialization, JSON deserialization, deterministic output.
|
||||
|
||||
### Step 1360: Build output parser — GCC/Clang format (10 tests)
|
||||
|
||||
Create `editor/src/BuildOutputParser.h`.
|
||||
|
||||
Parses the stdout/stderr of a build command into `BuildErrorRecord` list.
|
||||
GCC/Clang format: `<file>:<line>:<col>: <severity>: <message>`.
|
||||
CMake format: `CMake Error at <file>:<line>`.
|
||||
Linker format: `undefined reference to '<symbol>'`.
|
||||
Unknown lines captured as `tool=unknown` with raw message.
|
||||
|
||||
Tests (10): GCC error parsed correctly, Clang error parsed, CMake error
|
||||
parsed, linker error parsed, warning parsed with severity=warning, note
|
||||
parsed with severity=note, multi-line error grouped, empty input returns
|
||||
empty list, malformed line returns unknown record, deterministic parse order.
|
||||
|
||||
### Step 1361: AST node locator from build error (10 tests)
|
||||
|
||||
Create `editor/src/BuildErrorLocator.h`.
|
||||
|
||||
Given a `BuildErrorRecord` (file_path, line, col) and an open AST, finds
|
||||
the AST node responsible. Strategy: find the node whose span contains
|
||||
(line, col). If exact match fails, find the nearest enclosing function/class.
|
||||
Populates `ast_node_id` on the record.
|
||||
|
||||
Tests (10): exact line/col match returns node, enclosing function found when
|
||||
exact fails, file mismatch returns nullopt, line 0 returns nullopt, col 0
|
||||
accepted (start of line), multiple errors located in one pass, deterministic
|
||||
node selection, linker error (no file) returns nullopt gracefully, node_id
|
||||
written to record, works across all supported languages.
|
||||
|
||||
### Step 1362: Build error classifier (10 tests)
|
||||
|
||||
Create `editor/src/BuildErrorClassifier.h`.
|
||||
|
||||
Classifies `BuildErrorRecord` into semantic error types:
|
||||
- `missing_include` — #include not found
|
||||
- `undefined_symbol` — undeclared identifier or linker undefined ref
|
||||
- `type_mismatch` — incompatible types in assignment/call
|
||||
- `missing_dependency` — cmake target/package not found
|
||||
- `syntax_error` — parse-level failure
|
||||
- `resource_limit` — OOM, stack overflow
|
||||
- `unknown` — catch-all
|
||||
|
||||
Classification uses error_code patterns + message substring matching.
|
||||
|
||||
Tests (10): missing_include classified, undefined_symbol classified,
|
||||
type_mismatch classified, cmake dependency failure classified, syntax_error
|
||||
classified, linker undefined ref → undefined_symbol, warning not
|
||||
misclassified as error type, unknown catch-all, deterministic classification,
|
||||
JSON round-trip of classified record.
|
||||
|
||||
### Step 1363: Fix candidate generator (10 tests)
|
||||
|
||||
Create `editor/src/BuildFixCandidateGenerator.h`.
|
||||
|
||||
Given a classified `BuildErrorRecord` and the open AST, generates a ranked
|
||||
list of `FixCandidate` objects. Each candidate has: `candidate_id`,
|
||||
`description`, `confidence` (0.0–1.0), `mutation_type` (addImport,
|
||||
changeType, addDependency, deleteNode, insertNode, manual), `mutation`
|
||||
(JSON blob suitable for passing to `whetstone_mutate`).
|
||||
|
||||
Classification → candidate strategy:
|
||||
- `missing_include` → addImport candidate for the likely header
|
||||
- `undefined_symbol` → suggest declaration or import fix
|
||||
- `type_mismatch` → suggest cast or type change
|
||||
- `missing_dependency` → suggest vcpkg/apt install command (manual mutation)
|
||||
- `syntax_error` → low-confidence deleteNode or manual
|
||||
- `unknown` → manual with description only
|
||||
|
||||
Tests (10): missing_include produces addImport candidate, confidence > 0.5
|
||||
for known patterns, confidence < 0.3 for unknown, mutation JSON round-trips,
|
||||
manual candidate always produced as fallback, candidates ordered by confidence
|
||||
desc, candidate_id non-empty, description human-readable, no candidates on
|
||||
empty error list, deterministic output.
|
||||
|
||||
### Step 1364: Build iteration session integration (8 tests)
|
||||
|
||||
Create `editor/src/BuildIterationAdapter.h`.
|
||||
|
||||
Wraps a build attempt as an `AttemptRecord` + `ObservationRecord`:
|
||||
- Runs build command via subprocess, captures stdout+stderr+exit_code
|
||||
- Parses output into `BuildErrorRecord` list
|
||||
- Computes `GapModel` (metric_type=error_count, current=errors.size(),
|
||||
target=0)
|
||||
- Records attempt in provided `IterationSession`
|
||||
- Returns list of `FixCandidate` for all errors
|
||||
|
||||
Tests (8): attempt appended to session, observation raw_text captured,
|
||||
gap metric_type is error_count, gap converged when build succeeds,
|
||||
session status transitions to converged on success, fix candidates returned,
|
||||
attempt outcome=success on exit_code 0, attempt outcome=failure on non-zero.
|
||||
|
||||
### Step 1365: `whetstone_parse_build_output` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "output": string, "tool": string? }`
|
||||
Output: `{ "errors": BuildErrorRecord[], "warning_count": int, "error_count": int }`
|
||||
|
||||
Parses raw build output string, classifies each record, returns structured list.
|
||||
|
||||
Tests (8): tool registered, error_count correct, warning_count correct,
|
||||
tool field respected, empty output returns empty errors, malformed input
|
||||
returns structured error, classification applied to all records,
|
||||
deterministic output shape.
|
||||
|
||||
### Step 1366: `whetstone_suggest_build_fix` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string?, "build_output": string,
|
||||
"file": string?, "line": int?, "col": int? }`
|
||||
Output: `{ "candidates": FixCandidate[], "error_count": int }`
|
||||
|
||||
Parses output, locates in active AST buffer, classifies, generates candidates.
|
||||
|
||||
Tests (8): tool registered, candidates returned, confidence present on all,
|
||||
mutation field present, manual fallback always present, missing AST graceful,
|
||||
candidates sorted by confidence, deterministic output.
|
||||
|
||||
### Step 1367: `whetstone_run_build_iteration` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string, "build_command": string,
|
||||
"workspace": string }`
|
||||
Output: `{ "attempt_id": string, "session_status": string,
|
||||
"error_count": int, "candidates": FixCandidate[] }`
|
||||
|
||||
Runs the build, records the attempt in the session, returns fix candidates.
|
||||
|
||||
Tests (8): tool registered, attempt_id returned, session_status updated,
|
||||
error_count correct, candidates present on failure, session converged on
|
||||
success, build timeout enforced (default 120s), deterministic output shape.
|
||||
|
||||
### Step 1368: Sprint 113 integration summary + regression (8 tests)
|
||||
|
||||
Create `editor/src/Sprint113IntegrationSummary.h`.
|
||||
|
||||
End-to-end: start session → run_build_iteration (failing build) → verify
|
||||
attempt recorded + candidates returned → apply top candidate → run again →
|
||||
verify error count reduced.
|
||||
|
||||
Tests (8): headers constructable, tools registered, end-to-end flow works,
|
||||
gap model improves after fix applied, session transitions correctly, no header
|
||||
exceeds 600 lines, deterministic snapshots, full prior regression passes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate
|
||||
|
||||
- Build command execution via popen/std::system — not a shell injection vector
|
||||
(command is passed as-is from MCP caller; caller must sanitize inputs)
|
||||
- Confidence scores are heuristic — never presented as guarantees
|
||||
- Manual fallback candidate always generated so the loop is never stuck
|
||||
- Max 600 lines per header
|
||||
186
sprint114_plan.md
Normal file
186
sprint114_plan.md
Normal file
@@ -0,0 +1,186 @@
|
||||
# Sprint 114 Plan: Test Failure Feedback Loop
|
||||
|
||||
## Context
|
||||
|
||||
Build success is only the first gate. Tests are the second gate, and they carry
|
||||
richer semantic signal than compiler errors: a failing assertion tells you not
|
||||
just *where* something went wrong but *what* the code was supposed to do versus
|
||||
what it actually did. That gap between expected and actual is exactly the kind of
|
||||
structured signal Whetstone can act on deterministically.
|
||||
|
||||
Sprint 114 adds test output parsing, failure classification, test-to-source
|
||||
location mapping, and fix candidate generation for test failures. It mirrors
|
||||
the architecture of Sprint 113 (Build Feedback Loop) but operates on test
|
||||
output formats (Google Test, pytest, Jest, TAP). The result integrates with
|
||||
IterationSession: each test run becomes an AttemptRecord with structured
|
||||
observations and a GapModel keyed on failing test count.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Parse test runner output (gtest, pytest, Jest, TAP) into `TestFailureRecord`s
|
||||
2. Classify failures by type: assertion, exception, timeout, infra, wrong output
|
||||
3. Map each failure to a source location and AST node
|
||||
4. Generate ranked fix candidates from test failures
|
||||
5. Integrate with IterationSession (test run → AttemptRecord)
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1369: `TestFailureRecord` schema (12 tests)
|
||||
|
||||
Create `editor/src/TestFailureRecord.h`.
|
||||
|
||||
Fields: `failure_id`, `runner` (enum: gtest, pytest, jest, tap, unknown),
|
||||
`test_suite`, `test_name`, `failure_type` (enum: assertion, exception,
|
||||
timeout, missing_output, wrong_output, infra, unknown),
|
||||
`expected_value` (string, may be empty), `actual_value` (string, may be empty),
|
||||
`file_path`, `line`, `message`, `ast_node_id` (populated by locator).
|
||||
|
||||
Tests (12): constructable, runner enum round-trips, failure_type defaults to
|
||||
unknown, expected/actual stored, file_path stored, line stored, message stored,
|
||||
ast_node_id starts empty, JSON serialization, JSON deserialization, unknown
|
||||
runner passthrough, deterministic output.
|
||||
|
||||
### Step 1370: Test output parser — gtest format (10 tests)
|
||||
|
||||
Create `editor/src/TestOutputParser.h`.
|
||||
|
||||
Parses Google Test stdout format:
|
||||
- `[ FAILED ] SuiteName.TestName (Nms)` → failure header
|
||||
- `Expected: <value>` / ` Actual: <value>` → assertion fields
|
||||
- `<file>:<line>: Failure` → source location
|
||||
|
||||
Also parses pytest (`FAILED test_file.py::test_name - AssertionError: ...`)
|
||||
and basic TAP (`not ok N - description`). Jest support for `✕ test name`
|
||||
format. Unknown lines attached to current test as raw message.
|
||||
|
||||
Tests (10): gtest failure parsed, gtest assertion expected/actual extracted,
|
||||
gtest file/line extracted, pytest failure parsed, jest failure parsed,
|
||||
TAP not-ok parsed, pass lines ignored, multi-failure batch parsed,
|
||||
empty input returns empty list, deterministic parse order.
|
||||
|
||||
### Step 1371: Test failure classifier (10 tests)
|
||||
|
||||
Create `editor/src/TestFailureClassifier.h`.
|
||||
|
||||
Classifies `TestFailureRecord` into semantic types using message patterns:
|
||||
- `assertion` — Expect/Assert macro failure (expected != actual)
|
||||
- `exception` — unhandled exception or signal in test
|
||||
- `timeout` — test exceeded time limit
|
||||
- `wrong_output` — stdout/output mismatch (snapshot tests)
|
||||
- `missing_output` — expected output file not produced
|
||||
- `infra` — test setup/teardown failure, missing fixture
|
||||
- `unknown` — catch-all
|
||||
|
||||
Tests (10): assertion classified, unhandled exception classified, timeout
|
||||
classified, snapshot mismatch → wrong_output, missing file → missing_output,
|
||||
fixture failure → infra, unknown catch-all, deterministic classification,
|
||||
classification applied to batch, JSON round-trip of classified record.
|
||||
|
||||
### Step 1372: Test-to-source locator (10 tests)
|
||||
|
||||
Create `editor/src/TestSourceLocator.h`.
|
||||
|
||||
Maps a `TestFailureRecord` to the implementation code under test, not just
|
||||
the test file. Strategy:
|
||||
1. If file_path is a test file (matches `*test*`, `*spec*`), find the
|
||||
corresponding implementation file by naming convention.
|
||||
2. Locate the function-under-test by test_name substring match against
|
||||
AST function names.
|
||||
3. Fall back to the test file's AST node if implementation not found.
|
||||
|
||||
Populates `ast_node_id` on the failure record.
|
||||
|
||||
Tests (10): test file maps to implementation file, function-under-test found
|
||||
by name, fallback to test file node, no match returns nullopt gracefully,
|
||||
gtest suite.test → function match, pytest test_name → function match,
|
||||
deterministic selection, works across Python and C++, batch locating works,
|
||||
ast_node_id written to record.
|
||||
|
||||
### Step 1373: Fix candidate generator for test failures (10 tests)
|
||||
|
||||
Create `editor/src/TestFixCandidateGenerator.h`.
|
||||
|
||||
Generates `FixCandidate` list from classified `TestFailureRecord`:
|
||||
|
||||
- `assertion` (expected != actual): suggest value correction or logic change
|
||||
at the implementation AST node. Confidence = 0.4–0.7 (semantic, heuristic).
|
||||
- `exception`: suggest try/catch addition or precondition guard. Confidence 0.3.
|
||||
- `timeout`: suggest async/await addition or algorithm complexity reduction hint.
|
||||
Confidence 0.2 (manual candidate prominent).
|
||||
- `wrong_output` / `missing_output`: suggest regenerating snapshot or checking
|
||||
file write path. Manual candidate.
|
||||
- `infra`: suggest fixture setup. Manual.
|
||||
- `unknown`: manual only.
|
||||
|
||||
Tests (10): assertion produces value-correction candidate, confidence in range,
|
||||
exception produces guard candidate, timeout produces manual prominent,
|
||||
wrong_output produces snapshot candidate, batch processing works, candidates
|
||||
ordered by confidence, manual fallback always present, deterministic output,
|
||||
candidate mutations JSON round-trip.
|
||||
|
||||
### Step 1374: Test iteration session integration (8 tests)
|
||||
|
||||
Create `editor/src/TestIterationAdapter.h`.
|
||||
|
||||
Mirrors `BuildIterationAdapter`. Runs test command via subprocess, parses
|
||||
output, computes GapModel (metric_type=test_failure_count), records attempt
|
||||
in IterationSession, returns FixCandidates.
|
||||
|
||||
Tests (8): attempt appended to session, observation raw_text captured,
|
||||
gap metric_type is test_failure_count, gap converged when all tests pass,
|
||||
session converges on zero failures, fix candidates returned on failures,
|
||||
attempt outcome=success on zero failures, attempt outcome=failure on any failure.
|
||||
|
||||
### Step 1375: `whetstone_parse_test_output` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "output": string, "runner": string? }`
|
||||
Output: `{ "failures": TestFailureRecord[], "pass_count": int, "fail_count": int }`
|
||||
|
||||
Tests (8): tool registered, fail_count correct, pass_count correct, runner
|
||||
field respected, empty output returns empty failures, malformed input handled,
|
||||
classification applied, deterministic output.
|
||||
|
||||
### Step 1376: `whetstone_suggest_test_fix` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string?, "test_output": string, "runner": string? }`
|
||||
Output: `{ "candidates": FixCandidate[], "fail_count": int }`
|
||||
|
||||
Tests (8): tool registered, candidates returned, confidence present,
|
||||
mutation field present, manual fallback always present, missing AST graceful,
|
||||
candidates sorted by confidence, deterministic output.
|
||||
|
||||
### Step 1377: `whetstone_run_test_iteration` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string, "test_command": string, "workspace": string }`
|
||||
Output: `{ "attempt_id": string, "session_status": string,
|
||||
"fail_count": int, "candidates": FixCandidate[] }`
|
||||
|
||||
Tests (8): tool registered, attempt_id returned, session_status updated,
|
||||
fail_count correct, candidates on failure, session converged on all-pass,
|
||||
timeout enforced (default 120s), deterministic output shape.
|
||||
|
||||
### Step 1378: Sprint 114 integration summary + regression (8 tests)
|
||||
|
||||
Create `editor/src/Sprint114IntegrationSummary.h`.
|
||||
|
||||
End-to-end: start session → run_test_iteration (failing tests) → verify
|
||||
attempt recorded + candidates returned → apply top candidate → run again →
|
||||
verify fail count reduced.
|
||||
|
||||
Tests (8): headers constructable, tools registered, end-to-end flow works,
|
||||
gap model improves after fix, session transitions correctly, no header exceeds
|
||||
600 lines, deterministic snapshots, full prior regression passes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate
|
||||
|
||||
- Assertion parsing is heuristic — expected/actual extraction may be empty
|
||||
for non-standard formats; that is acceptable
|
||||
- Fix candidates are suggestions, not guaranteed correct fixes
|
||||
- Manual fallback always generated so the iteration loop is never stuck
|
||||
- Max 600 lines per header
|
||||
196
sprint115_plan.md
Normal file
196
sprint115_plan.md
Normal file
@@ -0,0 +1,196 @@
|
||||
# Sprint 115 Plan: Requirements Derivation from AST
|
||||
|
||||
## Context
|
||||
|
||||
Environment setup is currently manual and implicit. A developer knows their
|
||||
project needs Python 3.11, numpy, and a running PostgreSQL instance because
|
||||
they remember building it — not because the system can read that from the code.
|
||||
|
||||
Sprint 115 makes requirements derivation deterministic: given an AST, walk all
|
||||
import/include/dependency nodes and produce a structured `RequirementsManifest`
|
||||
— a language-agnostic list of what the project needs to run. From that manifest,
|
||||
generate an idempotent setup script in the appropriate package manager format
|
||||
(apt, pip, npm, cargo, cmake/vcpkg). Cross-reference the manifest against an
|
||||
`EnvironmentSnapshot` (Sprint 112) to produce a gap report: what's missing,
|
||||
what's the wrong version, what's ready.
|
||||
|
||||
This is the "environment as code" step: the project's needs become a
|
||||
machine-readable artifact that any agent can check, compare, and act on.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Define `RequirementsManifest` schema derived from the AST
|
||||
2. Walk all dependency nodes across supported languages
|
||||
3. Emit setup scripts per package manager (apt, pip, npm, cargo, vcpkg)
|
||||
4. Cross-reference manifest against an EnvironmentSnapshot to get a gap report
|
||||
5. Expose derivation and gap report as MCP tools
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1379: `RequirementsManifest` schema (12 tests)
|
||||
|
||||
Create `editor/src/RequirementsManifest.h`.
|
||||
|
||||
Fields: `manifest_id`, `source_file` (optional — if derived from single file),
|
||||
`workspace` (optional — if derived from full project), `language_runtimes[]`
|
||||
(name, min_version, source), `packages[]` (name, version_constraint,
|
||||
package_manager: apt/pip/npm/cargo/vcpkg/brew/unknown),
|
||||
`system_services[]` (name, port — e.g. postgres:5432, redis:6379),
|
||||
`env_vars_required[]` (key, description, has_default: bool),
|
||||
`derived_at`, `schema_version`.
|
||||
|
||||
Tests (12): constructable with defaults, manifest_id non-empty, packages
|
||||
list starts empty, runtimes list starts empty, services list starts empty,
|
||||
env_vars list starts empty, JSON serialization, JSON deserialization,
|
||||
package_manager enum round-trips, unknown manager passthrough, schema_version
|
||||
present, deterministic output.
|
||||
|
||||
### Step 1380: Import/include dependency walker (10 tests)
|
||||
|
||||
Create `editor/src/DependencyWalker.h`.
|
||||
|
||||
Walks an open AST, finds all import/include/require nodes, and extracts
|
||||
package names. Language-specific strategies:
|
||||
- Python: `import X`, `from X import Y` → package name is top-level module
|
||||
- C/C++: `#include <X>` → system header; `#include "X"` → local (skip)
|
||||
- JavaScript/TypeScript: `require('X')`, `import ... from 'X'` → npm package
|
||||
- Rust: `extern crate X`, `use X::` → cargo crate
|
||||
- Java: `import X.Y.Z` → Maven group
|
||||
|
||||
Stdlib modules filtered via a known-stdlib list per language (extensible).
|
||||
|
||||
Tests (10): Python import extracted, Python stdlib filtered, C++ system header
|
||||
extracted, C++ local include skipped, JS require extracted, TS import extracted,
|
||||
Rust extern crate extracted, Java import extracted, batch walk across multiple
|
||||
files, deterministic output order.
|
||||
|
||||
### Step 1381: Runtime version inferrer (10 tests)
|
||||
|
||||
Create `editor/src/RuntimeVersionInferrer.h`.
|
||||
|
||||
Infers language runtime requirements from project configuration files:
|
||||
- Python: `pyproject.toml` `requires-python`, `.python-version`, `setup.py`
|
||||
- Node.js: `package.json` `engines.node`, `.nvmrc`
|
||||
- Rust: `rust-toolchain.toml`
|
||||
- C++: CMake `CMAKE_CXX_STANDARD`
|
||||
- Java: `pom.xml` `maven.compiler.source`
|
||||
|
||||
Falls back to "any recent version" if no constraint found (min_version = "").
|
||||
|
||||
Tests (10): pyproject.toml version read, .python-version read, package.json
|
||||
engines read, .nvmrc read, rust-toolchain.toml read, cmake standard read,
|
||||
pom.xml source read, missing file returns no-constraint, multiple constraints
|
||||
take strictest, deterministic output.
|
||||
|
||||
### Step 1382: Service dependency detector (10 tests)
|
||||
|
||||
Create `editor/src/ServiceDependencyDetector.h`.
|
||||
|
||||
Detects service dependencies from config files and connection string patterns
|
||||
in source. Scans `.env`, `config.yaml`, `appsettings.json` for patterns like
|
||||
`POSTGRES_HOST`, `REDIS_URL`, `MONGO_URI`, `DATABASE_URL`. Maps to
|
||||
`SystemServiceRecord` (name, port).
|
||||
|
||||
Also detects from source: `psycopg2.connect(...)`, `mongoose.connect(...)`,
|
||||
`redis.Redis(...)` call patterns in the AST.
|
||||
|
||||
Tests (10): postgres from .env POSTGRES_HOST, redis from REDIS_URL, mongo
|
||||
from MONGO_URI, mysql from DATABASE_URL pattern, psycopg2 call in AST,
|
||||
mongoose call in AST, redis.Redis call in AST, no false positives on
|
||||
unrelated config, deterministic output, unknown service captured as unknown.
|
||||
|
||||
### Step 1383: Setup script generator (10 tests)
|
||||
|
||||
Create `editor/src/SetupScriptGenerator.h`.
|
||||
|
||||
Given a `RequirementsManifest`, generates an idempotent shell script per
|
||||
requested package manager. Scripts use guard patterns (apt: check dpkg,
|
||||
pip: check importlib, npm: check node_modules, cargo: check cargo metadata).
|
||||
|
||||
Output: `SetupScript` with fields `package_manager`, `script_text`,
|
||||
`estimated_packages` (count), `requires_sudo` (bool).
|
||||
|
||||
One `whetstone_generate_setup_script` call may return multiple scripts
|
||||
(one per relevant package manager detected from the manifest).
|
||||
|
||||
Tests (10): apt script generated for apt packages, pip script generated,
|
||||
npm script generated, cargo script generated, idempotent guard present in
|
||||
all scripts, sudo flag set for apt, not set for pip/npm/cargo, empty package
|
||||
list produces no-op script, deterministic output, script text is valid shell.
|
||||
|
||||
### Step 1384: Requirements gap report (8 tests)
|
||||
|
||||
Create `editor/src/RequirementsGapReport.h`.
|
||||
|
||||
Cross-references `RequirementsManifest` against `EnvironmentSnapshot`.
|
||||
|
||||
Fields: `ready` (bool), `satisfied[]` (package name),
|
||||
`missing[]` (name + required_version + package_manager),
|
||||
`version_mismatch[]` (name + required + installed),
|
||||
`unverifiable[]` (packages where installed version can't be confirmed),
|
||||
`service_gaps[]` (services in manifest not found in snapshot processes).
|
||||
|
||||
Tests (8): ready true when all satisfied, missing package detected, version
|
||||
mismatch detected, correct version passes, service gap detected, unverifiable
|
||||
when probe can't confirm, JSON round-trip, deterministic output.
|
||||
|
||||
### Step 1385: `whetstone_derive_requirements` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "workspace": string?, "files": string[]? }`
|
||||
Output: `RequirementsManifest` JSON.
|
||||
|
||||
Walks all open buffers (or specified files), runs DependencyWalker +
|
||||
RuntimeVersionInferrer + ServiceDependencyDetector, merges results.
|
||||
|
||||
Tests (8): tool registered, packages field present, runtimes field present,
|
||||
services field present, empty workspace returns empty manifest, single file
|
||||
mode works, manifest_id returned, deterministic output.
|
||||
|
||||
### Step 1386: `whetstone_generate_setup_script` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "manifest_id": string?, "manifest": RequirementsManifest? }`
|
||||
Output: `{ "scripts": SetupScript[] }`
|
||||
|
||||
Accepts either a previously derived manifest by ID or an inline manifest.
|
||||
|
||||
Tests (8): tool registered, scripts returned, apt script present for C++
|
||||
project, pip script present for Python project, idempotent guards in output,
|
||||
empty manifest returns empty scripts, manifest_id lookup works, deterministic.
|
||||
|
||||
### Step 1387: `whetstone_verify_requirements` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "manifest_id": string, "snapshot_id": string }`
|
||||
Output: `RequirementsGapReport` JSON.
|
||||
|
||||
Loads both from store, computes gap report.
|
||||
|
||||
Tests (8): tool registered, ready true when all installed, missing reported,
|
||||
mismatch reported, service gap reported, missing manifest_id error, missing
|
||||
snapshot_id error, deterministic output.
|
||||
|
||||
### Step 1388: Sprint 115 integration summary + regression (8 tests)
|
||||
|
||||
Create `editor/src/Sprint115IntegrationSummary.h`.
|
||||
|
||||
End-to-end: open Python project → derive_requirements → snapshot_environment →
|
||||
verify_requirements → generate_setup_script → verify gap report matches missing
|
||||
packages in snapshot.
|
||||
|
||||
Tests (8): headers constructable, tools registered, end-to-end flow works,
|
||||
gap report accurate, setup script produced, no header exceeds 600 lines,
|
||||
deterministic snapshots, full prior regression passes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate
|
||||
|
||||
- Dependency walker uses AST node traversal only — no regex on raw source text
|
||||
- Service detection from config files uses ConfigFileProbe (Sprint 112) —
|
||||
no duplicated parsing logic
|
||||
- Setup scripts are never executed by Whetstone — generation only, execution
|
||||
is the caller's responsibility
|
||||
- Max 600 lines per header
|
||||
205
sprint116_plan.md
Normal file
205
sprint116_plan.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# Sprint 116 Plan: Feedback Loop Orchestrator
|
||||
|
||||
## Context
|
||||
|
||||
Sprints 111-115 built the primitives: iteration sessions, environment snapshots,
|
||||
build feedback, test feedback, and requirements derivation. Each works
|
||||
independently. Sprint 116 wires them into a single orchestrated loop.
|
||||
|
||||
The `FeedbackLoopOrchestrator` is a state machine that drives an
|
||||
`IterationSession` from start to convergence (or escalation). It decides the
|
||||
next action based on the current session state: if the environment is not
|
||||
verified, run `verify_requirements` first; if the build is failing, run
|
||||
`run_build_iteration`; if tests are failing, run `run_test_iteration`. After
|
||||
each action it selects the highest-confidence fix candidate, applies it, and
|
||||
loops. When max_attempts is reached without convergence, it escalates to human
|
||||
with a structured summary of what was tried and what the current gap is.
|
||||
|
||||
This is the "inner loop" of deterministic development: goal → sense → act →
|
||||
observe → repeat.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Define the orchestrator's decision policy (what to do given current state)
|
||||
2. Implement automatic candidate selection and application
|
||||
3. Handle escalation with a structured "what was tried" summary
|
||||
4. Expose the orchestrator as a single MCP entry point
|
||||
5. Ensure the loop is inspectable at every step (full audit trail)
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1389: `OrchestratorPolicy` decision model (12 tests)
|
||||
|
||||
Create `editor/src/OrchestratorPolicy.h`.
|
||||
|
||||
The policy maps (session_status, gap_model, last_attempt_outcome) →
|
||||
next action type:
|
||||
- If environment not verified: probe environment
|
||||
- If requirements gap present: generate setup script → escalate (can't
|
||||
auto-install; human must run the script)
|
||||
- If build failing: run build iteration, apply top candidate
|
||||
- If tests failing: run test iteration, apply top candidate
|
||||
- If gap = 0: mark converged
|
||||
- If max_attempts reached: escalate
|
||||
- If last attempt made gap worse (worsening trend, 2+ consecutive): escalate
|
||||
|
||||
Each decision includes: `action_type`, `rationale` string, `confidence` float.
|
||||
|
||||
Tests (12): environment probe first when no snapshot, requirements gap → escalate,
|
||||
build failure → build iteration, test failure → test iteration, zero gap →
|
||||
converge, max_attempts → escalate, worsening trend → escalate, consecutive
|
||||
no-progress → escalate, decision includes rationale, confidence in range,
|
||||
JSON round-trip of decision, deterministic decision for same inputs.
|
||||
|
||||
### Step 1390: Candidate auto-selector (10 tests)
|
||||
|
||||
Create `editor/src/CandidateAutoSelector.h`.
|
||||
|
||||
Given a list of `FixCandidate`s and a policy, selects the candidate to apply.
|
||||
Rules:
|
||||
- Select highest confidence candidate
|
||||
- Never re-apply a candidate that was already attempted in this session
|
||||
(tracked by candidate description hash)
|
||||
- If all non-manual candidates were already tried: escalate
|
||||
- If top candidate is manual (mutation_type=manual): escalate immediately
|
||||
(cannot apply manually)
|
||||
|
||||
Tests (10): highest confidence selected, already-tried skipped, all tried →
|
||||
escalate decision, manual type → escalate, empty list → escalate, confidence
|
||||
tie broken deterministically, attempt history consulted, selection logged
|
||||
to session, JSON round-trip of selection record, deterministic output.
|
||||
|
||||
### Step 1391: Escalation report composer (10 tests)
|
||||
|
||||
Create `editor/src/EscalationReport.h`.
|
||||
|
||||
Produces a human-readable + machine-readable summary when the loop escalates:
|
||||
- `session_id`, `goal`, `goal_type`
|
||||
- `attempts_made` (count)
|
||||
- `final_gap` (GapModel)
|
||||
- `attempts_summary[]`: each attempt's action, outcome, gap_before, gap_after
|
||||
- `top_remaining_candidates[]`: the best manual candidates not yet tried
|
||||
- `escalation_reason` (enum: max_attempts, worsening_trend, all_auto_tried,
|
||||
manual_required, infra_gap)
|
||||
- `recommended_next_step` (human-readable string)
|
||||
|
||||
Tests (10): report constructable, session_id present, attempts_summary accurate,
|
||||
top candidates included, escalation_reason populated, recommended_next_step
|
||||
human-readable, JSON round-trip, empty attempts handled, gap accurate,
|
||||
deterministic output.
|
||||
|
||||
### Step 1392: `FeedbackLoopOrchestrator` core loop (10 tests)
|
||||
|
||||
Create `editor/src/FeedbackLoopOrchestrator.h`.
|
||||
|
||||
The orchestrator drives one step of the loop at a time (not a blocking loop —
|
||||
caller steps it). Interface:
|
||||
- `step(session_id)` → `OrchestratorStepResult` (action taken, new session
|
||||
status, gap after, escalation_report if escalated)
|
||||
|
||||
Internally:
|
||||
1. Load session from SessionStore
|
||||
2. Run OrchestratorPolicy to get next action decision
|
||||
3. Execute the action (build iteration / test iteration / env probe)
|
||||
4. Run CandidateAutoSelector on result
|
||||
5. Apply selected mutation to AST (if non-manual)
|
||||
6. Record attempt, update session, persist
|
||||
7. Return result
|
||||
|
||||
Tests (10): step runs without crashing, action matches policy, mutation applied
|
||||
to AST, attempt recorded in session, session converges when gap=0, session
|
||||
escalates at max_attempts, worsening trend triggers escalation, escalation
|
||||
report present when escalated, step result includes gap, deterministic output.
|
||||
|
||||
### Step 1393: Full loop driver (end-to-end stepping) (10 tests)
|
||||
|
||||
Add `FeedbackLoopOrchestrator::run_to_completion(session_id, max_wall_s)`.
|
||||
|
||||
Calls `step()` in a loop until session reaches a terminal state or wall time
|
||||
expires. Each step result appended to an audit log. Returns the final session
|
||||
state + full audit log.
|
||||
|
||||
Tests (10): run_to_completion returns on convergence, returns on escalation,
|
||||
respects wall time limit, audit log has one entry per step, step count ≤
|
||||
max_attempts, gap trend visible in audit log, escalation report in last entry
|
||||
if escalated, deterministic for deterministic inputs, session persisted at end,
|
||||
no infinite loop possible.
|
||||
|
||||
### Step 1394: Audit trail and inspection (8 tests)
|
||||
|
||||
Create `editor/src/LoopAuditLog.h`.
|
||||
|
||||
A serializable list of `AuditEntry` records: one per orchestrator step.
|
||||
Fields: `step_index`, `action_type`, `action_description`, `candidate_applied`
|
||||
(optional), `gap_before`, `gap_after`, `session_status_after`, `duration_ms`.
|
||||
|
||||
Persist to `.whetstone/audits/<session_id>.jsonl` (newline-delimited JSON,
|
||||
one entry per line, appendable).
|
||||
|
||||
Tests (8): audit entry constructable, appended per step, persisted to JSONL,
|
||||
JSONL format valid, gap_before/after present, candidate_applied present when
|
||||
applied, duration_ms populated, deterministic output.
|
||||
|
||||
### Step 1395: `whetstone_start_feedback_loop` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "goal": string, "goal_type": string, "build_command": string?,
|
||||
"test_command": string?, "workspace": string, "max_attempts": int? }`
|
||||
Output: `{ "session_id": string, "status": "pending" }`
|
||||
|
||||
Creates the session and registers build/test commands for the orchestrator.
|
||||
Does not start the loop — caller steps it.
|
||||
|
||||
Tests (8): tool registered, session_id returned, status is pending, session
|
||||
persisted, build_command stored, test_command stored, missing workspace error,
|
||||
deterministic output.
|
||||
|
||||
### Step 1396: `whetstone_step_feedback_loop` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string }`
|
||||
Output: `{ "action_taken": string, "session_status": string, "gap": GapModel,
|
||||
"candidates": FixCandidate[]?, "escalation": EscalationReport? }`
|
||||
|
||||
Calls orchestrator.step() once and returns the result.
|
||||
|
||||
Tests (8): tool registered, action_taken populated, session_status updated,
|
||||
gap present, candidates present on failure, escalation present when escalated,
|
||||
terminal session returns error, deterministic output.
|
||||
|
||||
### Step 1397: `whetstone_run_feedback_loop` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string, "max_wall_seconds": int? }`
|
||||
Output: `{ "final_status": string, "steps_taken": int, "final_gap": GapModel,
|
||||
"audit_log": AuditEntry[]?, "escalation": EscalationReport? }`
|
||||
|
||||
Calls run_to_completion and returns the full result.
|
||||
|
||||
Tests (8): tool registered, final_status correct, steps_taken accurate,
|
||||
final_gap present, audit_log included when requested, escalation present when
|
||||
escalated, wall time respected, deterministic output shape.
|
||||
|
||||
### Step 1398: Sprint 116 integration summary + regression (8 tests)
|
||||
|
||||
Create `editor/src/Sprint116IntegrationSummary.h`.
|
||||
|
||||
End-to-end: start_feedback_loop → run_feedback_loop on a project with a
|
||||
known fixable error → verify converged within max_attempts.
|
||||
|
||||
Tests (8): headers constructable, tools registered, end-to-end convergence
|
||||
demonstrated, audit log produced, no header exceeds 600 lines,
|
||||
deterministic snapshots, escalation path tested, full prior regression passes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate
|
||||
|
||||
- Orchestrator steps are non-blocking (one step per call) — no background
|
||||
threads, no blocking I/O in the orchestrator itself
|
||||
- The loop is always inspectable: every step is logged to the audit trail
|
||||
before returning
|
||||
- Manual candidates never auto-applied — escalation is the only path for
|
||||
mutation_type=manual
|
||||
- Max 600 lines per header
|
||||
182
sprint117_plan.md
Normal file
182
sprint117_plan.md
Normal file
@@ -0,0 +1,182 @@
|
||||
# Sprint 117 Plan: HiveMind Integration — Feedback Loop as Distributed Jobs
|
||||
|
||||
## Context
|
||||
|
||||
Sprints 111-116 implement the feedback loop as a local, in-process operation:
|
||||
the orchestrator runs build/test commands on the same machine as Whetstone.
|
||||
That works for single-node development but misses HiveMind's core value: the
|
||||
right work runs on the right hardware node.
|
||||
|
||||
Sprint 117 integrates the feedback loop with HiveMind's job dispatch model.
|
||||
Instead of running `cmake --build` locally, the orchestrator can emit a
|
||||
`build_job` to HiveMind and wait for the result. Instead of running `pytest`
|
||||
locally, it emits a `test_job`. The orchestrator becomes a HiveMind
|
||||
client — it submits iteration steps as jobs, polls for completion, and feeds
|
||||
the results back into the session as `ObservationRecord`s.
|
||||
|
||||
This is the natural boundary between the two systems: Whetstone owns the
|
||||
*reasoning* (what to do next, what does the result mean, what to try next);
|
||||
HiveMind owns the *execution* (run this on the GPU node, run that on the
|
||||
NAS, parallelize across the fleet).
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
1. Define a HiveMind job adapter for build, test, and probe iteration steps
|
||||
2. Submit iteration steps as HiveMind jobs via the nexus HTTP API
|
||||
3. Poll for job completion and map the result to ObservationRecord
|
||||
4. Integrate the distributed path with the FeedbackLoopOrchestrator policy
|
||||
5. Allow the orchestrator to run locally or distributed (config per session)
|
||||
|
||||
---
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1399: `HiveMindJobAdapter` schema and config (12 tests)
|
||||
|
||||
Create `editor/src/HiveMindJobAdapter.h`.
|
||||
|
||||
Config fields: `nexus_url` (e.g. http://localhost:8765), `capabilities[]`
|
||||
(required capabilities for the job — forwarded to HiveMind requirements),
|
||||
`bounty` (low/normal/high_priority), `poll_interval_s` (default 3),
|
||||
`timeout_s` (default 300).
|
||||
|
||||
Maps orchestrator action types to HiveMind job types:
|
||||
- `build` → `"build_job"` with payload `{ "command": string, "workspace": string }`
|
||||
- `test` → `"test_job"` with payload `{ "command": string, "workspace": string }`
|
||||
- `probe` → `"probe_job"` with payload `{ "workspace": string }`
|
||||
|
||||
Tests (12): config constructable, nexus_url stored, capabilities stored,
|
||||
bounty defaults to normal, poll_interval defaults to 3, timeout defaults
|
||||
to 300, build action maps to build_job, test action maps to test_job,
|
||||
probe action maps to probe_job, payload fields populated, JSON round-trip,
|
||||
deterministic output.
|
||||
|
||||
### Step 1400: Job submission via nexus HTTP API (10 tests)
|
||||
|
||||
Create `editor/src/NexusJobSubmitter.h`.
|
||||
|
||||
Submits a HiveMind job via `POST {nexus_url}/jobs`. Uses libcurl (available
|
||||
from Sprint 113 build subprocess pattern, or via HTTP client in stdlibs).
|
||||
Returns `job_id` string on success. Handles: 200 OK (parse id), 4xx (error),
|
||||
timeout (fail fast), network error (fail fast).
|
||||
|
||||
Tests (10): submit produces job_id, invalid nexus_url returns error, 4xx
|
||||
response returns structured error, timeout returns error, payload correctly
|
||||
serialized, capabilities forwarded, bounty forwarded, job_type forwarded,
|
||||
non-blocking submit (returns immediately after HTTP response),
|
||||
deterministic request format.
|
||||
|
||||
### Step 1401: Job result poller (10 tests)
|
||||
|
||||
Create `editor/src/NexusJobPoller.h`.
|
||||
|
||||
Polls `GET {nexus_url}/jobs/{job_id}` until status reaches `done` or `failed`.
|
||||
Returns `JobPollResult`: `job_id`, `status` (done/failed/pending/claimed),
|
||||
`output` (parsed from result payload — `output_file` content or inline),
|
||||
`exit_code`, `duration_ms`.
|
||||
|
||||
Respects poll_interval_s and timeout_s from HiveMindJobAdapter config.
|
||||
|
||||
Tests (10): poll returns done on success, poll returns failed on failure,
|
||||
poll respects timeout, poll respects interval, output field populated,
|
||||
exit_code populated, pending status keeps polling, claimed status keeps
|
||||
polling, network error returns error, deterministic result shape.
|
||||
|
||||
### Step 1402: Result-to-ObservationRecord mapper (10 tests)
|
||||
|
||||
Create `editor/src/JobResultMapper.h`.
|
||||
|
||||
Maps a `JobPollResult` to an `ObservationRecord`:
|
||||
- `source` = build_output / test_output / env_probe (from action type)
|
||||
- `raw_text` = output field (truncated at 16KB)
|
||||
- `exit_code` = from JobPollResult
|
||||
- `duration_ms` = from JobPollResult
|
||||
- `structured_data` = parsed by BuildOutputParser or TestOutputParser
|
||||
depending on action type
|
||||
|
||||
Tests (10): build result maps to ObservationRecord with source=build_output,
|
||||
test result maps with source=test_output, probe result maps with
|
||||
source=env_probe, raw_text truncated at limit, exit_code mapped, duration_ms
|
||||
mapped, structured_data populated by parser, failed job maps to exit_code!=0,
|
||||
JSON round-trip of mapped record, deterministic output.
|
||||
|
||||
### Step 1403: Distributed path in FeedbackLoopOrchestrator (10 tests)
|
||||
|
||||
Extend `FeedbackLoopOrchestrator.h` to support a `distributed` mode
|
||||
(set per session via config).
|
||||
|
||||
When distributed=true:
|
||||
- `step()` submits a HiveMind job instead of running subprocess locally
|
||||
- Returns immediately with `session_status=active, action=submitted`
|
||||
(the loop is async — caller must step again to poll)
|
||||
- On next step: if a pending job exists for the session, poll it;
|
||||
if done, map result and continue; if timeout, escalate
|
||||
|
||||
When distributed=false (default): existing local subprocess behavior unchanged.
|
||||
|
||||
Tests (10): distributed mode submits job not subprocess, immediate return
|
||||
on submit, subsequent step polls existing job, done result processed correctly,
|
||||
failed result processed, timeout escalates, local mode still works, config
|
||||
persisted in session, mode switch between sessions works, deterministic policy.
|
||||
|
||||
### Step 1404: `whetstone_configure_hivemind` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "nexus_url": string, "capabilities": string[]?,
|
||||
"bounty": string?, "timeout_s": int? }`
|
||||
Output: `{ "configured": true, "nexus_url": string }`
|
||||
|
||||
Persists HiveMind config to `.whetstone/hivemind_config.json` in workspace.
|
||||
All sessions created after this call use distributed mode by default.
|
||||
|
||||
Tests (8): tool registered, config persisted, nexus_url stored, capabilities
|
||||
stored, bounty defaulted to normal, timeout defaulted to 300, overwrite works,
|
||||
deterministic output.
|
||||
|
||||
### Step 1405: `whetstone_submit_iteration_job` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string, "action_type": string,
|
||||
"command": string, "workspace": string }`
|
||||
Output: `{ "job_id": string, "attempt_id": string, "status": "submitted" }`
|
||||
|
||||
Submits one iteration step as a HiveMind job and records the pending attempt.
|
||||
|
||||
Tests (8): tool registered, job_id returned, attempt_id returned, status
|
||||
is submitted, attempt recorded in session, nexus_url from config used,
|
||||
missing config falls back to local, deterministic output.
|
||||
|
||||
### Step 1406: `whetstone_poll_iteration_job` MCP tool (8 tests)
|
||||
|
||||
Input: `{ "session_id": string, "attempt_id": string }`
|
||||
Output: `{ "job_status": string, "session_status": string,
|
||||
"gap": GapModel?, "candidates": FixCandidate[]? }`
|
||||
|
||||
Polls the HiveMind job associated with the attempt, processes result if done.
|
||||
|
||||
Tests (8): tool registered, pending returns job_status=pending, done processes
|
||||
result, failed processes failure, session_status updated on done, gap present
|
||||
after processing, candidates present on failure, deterministic output.
|
||||
|
||||
### Step 1407: Sprint 117 integration summary + regression (8 tests)
|
||||
|
||||
Create `editor/src/Sprint117IntegrationSummary.h`.
|
||||
|
||||
End-to-end distributed test (against mock nexus): start session with
|
||||
distributed=true → submit_iteration_job → poll_iteration_job (mock returns
|
||||
done) → verify attempt recorded + candidates returned.
|
||||
|
||||
Tests (8): headers constructable, tools registered, distributed end-to-end
|
||||
mock test passes, local mode still works, config round-trip, no header exceeds
|
||||
600 lines, deterministic snapshots, full prior regression passes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gate
|
||||
|
||||
- HiveMind nexus URL is never hardcoded — always from config or MCP input
|
||||
- Distributed mode does not block the MCP server — jobs are async by design
|
||||
- Local mode (distributed=false) remains the default until explicitly configured
|
||||
- No HiveMind-specific types leak into the core orchestrator headers —
|
||||
the adapter pattern keeps the boundary clean
|
||||
- Max 600 lines per header
|
||||
Reference in New Issue
Block a user