# MCP Logging and Data Synthesis Strategy

## Objective
Train reusable tool-routing and tool-execution adapters by separating:
- `tool_selection`: which tool to call next.
- `tool_execution`: whether arguments/use were correct and what happened.

This split supports LoRA datasets that improve planning/routing independently from low-level argument formation.

## Runtime Stages
Each model turn is logged as one or more JSONL records in `logs/taskitem_runs/ollama_mcp_exec_*/task_<n>_context.jsonl`.

### Stage: tool_selection
Recorded before MCP execution. Fields:
- `stage`: `tool_selection`
- `turn`
- `intent`
- `task_phase`
- `candidate_tools`
- `chosen_tool`
- `args`
- `reason`
- `confidence`
- `fallback_tools`
- `label`: `good` or `bad`
- `error_type` (when bad)

### Stage: tool_execution
Recorded after MCP execution. Fields:
- `stage`: `tool_execution`
- `turn`
- `tool`
- `args`
- `result`: `success|error|blocked|unknown`
- `label`: `good` or `bad`
- `error_type` (when available)
- `result_preview`

## Failure Metadata and Debug Probes
When the loop hits invalid selections, repeated issues, or max-turn exits, transcript probes are captured in `01_transcript.log`.

Failure probes can include:
- `whetstone_get_workflow_state`
- `whetstone_get_blockers`
- `whetstone_get_recent_events`
- optional debug-specific tools if present
- `whetstone_generate_inference_job`

These become `capture_type: "failure_probe"` records in malformed exports.

## Dataset Construction

### Malformed dataset
Script: `tools/mcp/export_malformed_mcp_calls_for_lora.sh`

Includes:
- adapter loop errors (`invalid_action`, unknown tool, etc.)
- MCP tool errors from `result_preview`
- stage-labeled bad events from selection/execution
- failure probe records from transcript

Output: `training_data/lora/malformed_mcp_calls.jsonl`

### Quality dataset
Script: `tools/mcp/export_mcp_call_quality_for_lora.sh`

Includes:
- pipeline-derived known-good calls
- stage-labeled `good`/`bad` examples
- legacy adapter and MCP tool-response bad examples

Outputs:
- `training_data/lora/mcp_call_quality.jsonl`
- `training_data/lora/mcp_calls_good.jsonl`
- `training_data/lora/mcp_calls_bad.jsonl`

## Synthesis Strategy
Use mixed sampling:
- Positive examples from clean pipeline runs.
- Hard negatives from malformed outputs and near-miss tool choices.
- Counterfactual negatives where similar tools are available but incorrect.

Recommended metadata tags for future synthesis:
- `interface_mode`: `json_loop` vs `native_tool_calling`
- `selection_confidence_bucket`
- `recovery_attempted`
- `task_domain` / `codebase_area`

## Batch Orchestration
Script: `tools/mcp/run_slm_training_batch.sh`

Per run manifest fields now include:
- token usage totals
- `stage_metrics.selection_events`
- `stage_metrics.selection_bad`
- `stage_metrics.execution_events`
- `stage_metrics.execution_bad`

Use these metrics to rank runs for curation and to detect regressions in routing quality.

## Curation Rules
Promote samples when:
1. Selection was confident and consistent with successful execution.
2. Execution finished without schema/tool errors.
3. Recovery (if any) converged quickly.

Demote or isolate samples when:
1. Selection repeatedly picks unavailable or semantically wrong tools.
2. Args repeatedly violate schema despite retries.
3. Context resets dominate the run (possible instability/noise).