whetstone_DSL/sprint116_plan.md

Sprint 116 Plan: Feedback Loop Orchestrator

Context

Sprints 111-115 built the primitives: iteration sessions, environment snapshots, build feedback, test feedback, and requirements derivation. Each works independently. Sprint 116 wires them into a single orchestrated loop.

The FeedbackLoopOrchestrator is a state machine that drives an IterationSession from start to convergence (or escalation). It decides the next action based on the current session state: if the environment is not verified, run verify_requirements first; if the build is failing, run run_build_iteration; if tests are failing, run run_test_iteration. After each action it selects the highest-confidence fix candidate, applies it, and loops. When max_attempts is reached without convergence, it escalates to human with a structured summary of what was tried and what the current gap is.

This is the "inner loop" of deterministic development: goal → sense → act → observe → repeat.

---

Goals

1. Define the orchestrator's decision policy (what to do given current state)
2. Implement automatic candidate selection and application
3. Handle escalation with a structured "what was tried" summary
4. Expose the orchestrator as a single MCP entry point
5. Ensure the loop is inspectable at every step (full audit trail)

---

Steps

Step 1389: OrchestratorPolicy decision model (12 tests)

Create editor/src/OrchestratorPolicy.h.

The policy maps (session_status, gap_model, last_attempt_outcome) → next action type:

• If environment not verified: probe environment
• If requirements gap present: generate setup script → escalate (can't auto-install; human must run the script)
• If build failing: run build iteration, apply top candidate
• If tests failing: run test iteration, apply top candidate
• If gap = 0: mark converged
• If max_attempts reached: escalate
• If last attempt made gap worse (worsening trend, 2+ consecutive): escalate

Each decision includes: action_type, rationale string, confidence float.

Tests (12): environment probe first when no snapshot, requirements gap → escalate, build failure → build iteration, test failure → test iteration, zero gap → converge, max_attempts → escalate, worsening trend → escalate, consecutive no-progress → escalate, decision includes rationale, confidence in range, JSON round-trip of decision, deterministic decision for same inputs.

Step 1390: Candidate auto-selector (10 tests)

Create editor/src/CandidateAutoSelector.h.

Given a list of FixCandidates and a policy, selects the candidate to apply. Rules:

• Select highest confidence candidate
• Never re-apply a candidate that was already attempted in this session (tracked by candidate description hash)
• If all non-manual candidates were already tried: escalate
• If top candidate is manual (mutation_type=manual): escalate immediately (cannot apply manually)

Tests (10): highest confidence selected, already-tried skipped, all tried → escalate decision, manual type → escalate, empty list → escalate, confidence tie broken deterministically, attempt history consulted, selection logged to session, JSON round-trip of selection record, deterministic output.

Step 1391: Escalation report composer (10 tests)

Create editor/src/EscalationReport.h.

Produces a human-readable + machine-readable summary when the loop escalates:

• session_id, goal, goal_type
• attempts_made (count)
• final_gap (GapModel)
• attempts_summary[]: each attempt's action, outcome, gap_before, gap_after
• top_remaining_candidates[]: the best manual candidates not yet tried
• escalation_reason (enum: max_attempts, worsening_trend, all_auto_tried, manual_required, infra_gap)
• recommended_next_step (human-readable string)

Tests (10): report constructable, session_id present, attempts_summary accurate, top candidates included, escalation_reason populated, recommended_next_step human-readable, JSON round-trip, empty attempts handled, gap accurate, deterministic output.

Step 1392: FeedbackLoopOrchestrator core loop (10 tests)

Create editor/src/FeedbackLoopOrchestrator.h.

The orchestrator drives one step of the loop at a time (not a blocking loop — caller steps it). Interface:

• step(session_id) → OrchestratorStepResult (action taken, new session status, gap after, escalation_report if escalated)

Internally:

1. Load session from SessionStore
2. Run OrchestratorPolicy to get next action decision
3. Execute the action (build iteration / test iteration / env probe)
4. Run CandidateAutoSelector on result
5. Apply selected mutation to AST (if non-manual)
6. Record attempt, update session, persist
7. Return result

Tests (10): step runs without crashing, action matches policy, mutation applied to AST, attempt recorded in session, session converges when gap=0, session escalates at max_attempts, worsening trend triggers escalation, escalation report present when escalated, step result includes gap, deterministic output.

Step 1393: Full loop driver (end-to-end stepping) (10 tests)

Add FeedbackLoopOrchestrator::run_to_completion(session_id, max_wall_s).

Calls step() in a loop until session reaches a terminal state or wall time expires. Each step result appended to an audit log. Returns the final session state + full audit log.

Tests (10): run_to_completion returns on convergence, returns on escalation, respects wall time limit, audit log has one entry per step, step count ≤ max_attempts, gap trend visible in audit log, escalation report in last entry if escalated, deterministic for deterministic inputs, session persisted at end, no infinite loop possible.

Step 1394: Audit trail and inspection (8 tests)

Create editor/src/LoopAuditLog.h.

A serializable list of AuditEntry records: one per orchestrator step. Fields: step_index, action_type, action_description, candidate_applied (optional), gap_before, gap_after, session_status_after, duration_ms.

Persist to .whetstone/audits/<session_id>.jsonl (newline-delimited JSON, one entry per line, appendable).

Tests (8): audit entry constructable, appended per step, persisted to JSONL, JSONL format valid, gap_before/after present, candidate_applied present when applied, duration_ms populated, deterministic output.

Step 1395: whetstone_start_feedback_loop MCP tool (8 tests)

Input: { "goal": string, "goal_type": string, "build_command": string?, "test_command": string?, "workspace": string, "max_attempts": int? } Output: { "session_id": string, "status": "pending" }

Creates the session and registers build/test commands for the orchestrator. Does not start the loop — caller steps it.

Tests (8): tool registered, session_id returned, status is pending, session persisted, build_command stored, test_command stored, missing workspace error, deterministic output.

Step 1396: whetstone_step_feedback_loop MCP tool (8 tests)

Input: { "session_id": string } Output: { "action_taken": string, "session_status": string, "gap": GapModel, "candidates": FixCandidate[]?, "escalation": EscalationReport? }

Calls orchestrator.step() once and returns the result.

Tests (8): tool registered, action_taken populated, session_status updated, gap present, candidates present on failure, escalation present when escalated, terminal session returns error, deterministic output.

Step 1397: whetstone_run_feedback_loop MCP tool (8 tests)

Input: { "session_id": string, "max_wall_seconds": int? } Output: { "final_status": string, "steps_taken": int, "final_gap": GapModel, "audit_log": AuditEntry[]?, "escalation": EscalationReport? }

Calls run_to_completion and returns the full result.

Tests (8): tool registered, final_status correct, steps_taken accurate, final_gap present, audit_log included when requested, escalation present when escalated, wall time respected, deterministic output shape.

Step 1398: Sprint 116 integration summary + regression (8 tests)

Create editor/src/Sprint116IntegrationSummary.h.

End-to-end: start_feedback_loop → run_feedback_loop on a project with a known fixable error → verify converged within max_attempts.

Tests (8): headers constructable, tools registered, end-to-end convergence demonstrated, audit log produced, no header exceeds 600 lines, deterministic snapshots, escalation path tested, full prior regression passes.

---

Architecture Gate

• Orchestrator steps are non-blocking (one step per call) — no background threads, no blocking I/O in the orchestrator itself
• The loop is always inspectable: every step is logged to the audit trail before returning
• Manual candidates never auto-applied — escalation is the only path for mutation_type=manual
• Max 600 lines per header