whetstone_DSL/docs/mcp_logging_data_strategy.md

MCP Logging and Data Synthesis Strategy

Objective

Train reusable tool-routing and tool-execution adapters by separating:

• tool_selection: which tool to call next.
• tool_execution: whether arguments/use were correct and what happened.

This split supports LoRA datasets that improve planning/routing independently from low-level argument formation.

Runtime Stages

Each model turn is logged as one or more JSONL records in logs/taskitem_runs/ollama_mcp_exec_*/task_<n>_context.jsonl.

Stage: tool_selection

Recorded before MCP execution. Fields:

• stage: tool_selection
• turn
• intent
• task_phase
• candidate_tools
• chosen_tool
• args
• reason
• confidence
• fallback_tools
• label: good or bad
• error_type (when bad)

Stage: tool_execution

Recorded after MCP execution. Fields:

• stage: tool_execution
• turn
• tool
• args
• result: success|error|blocked|unknown
• label: good or bad
• error_type (when available)
• result_preview

Failure Metadata and Debug Probes

When the loop hits invalid selections, repeated issues, or max-turn exits, transcript probes are captured in 01_transcript.log.

Failure probes can include:

• whetstone_get_workflow_state
• whetstone_get_blockers
• whetstone_get_recent_events
• optional debug-specific tools if present
• whetstone_generate_inference_job

These become capture_type: "failure_probe" records in malformed exports.

Dataset Construction

Malformed dataset

Script: tools/mcp/export_malformed_mcp_calls_for_lora.sh

Includes:

• adapter loop errors (invalid_action, unknown tool, etc.)
• MCP tool errors from result_preview
• stage-labeled bad events from selection/execution
• failure probe records from transcript

Output: training_data/lora/malformed_mcp_calls.jsonl

Quality dataset

Script: tools/mcp/export_mcp_call_quality_for_lora.sh

Includes:

• pipeline-derived known-good calls
• stage-labeled good/bad examples
• legacy adapter and MCP tool-response bad examples

Outputs:

• training_data/lora/mcp_call_quality.jsonl
• training_data/lora/mcp_calls_good.jsonl
• training_data/lora/mcp_calls_bad.jsonl

Synthesis Strategy

Use mixed sampling:

• Positive examples from clean pipeline runs.
• Hard negatives from malformed outputs and near-miss tool choices.
• Counterfactual negatives where similar tools are available but incorrect.

Recommended metadata tags for future synthesis:

• interface_mode: json_loop vs native_tool_calling
• selection_confidence_bucket
• recovery_attempted
• task_domain / codebase_area

Batch Orchestration

Script: tools/mcp/run_slm_training_batch.sh

Per run manifest fields now include:

• token usage totals
• stage_metrics.selection_events
• stage_metrics.selection_bad
• stage_metrics.execution_events
• stage_metrics.execution_bad

Use these metrics to rank runs for curation and to detect regressions in routing quality.

Curation Rules

Promote samples when:

1. Selection was confident and consistent with successful execution.
2. Execution finished without schema/tool errors.
3. Recovery (if any) converged quickly.

Demote or isolate samples when:

1. Selection repeatedly picks unavailable or semantically wrong tools.
2. Args repeatedly violate schema despite retries.
3. Context resets dominate the run (possible instability/noise).