111 lines
3.3 KiB
Markdown
111 lines
3.3 KiB
Markdown
# MCP Logging and Data Synthesis Strategy
|
|
|
|
## Objective
|
|
Train reusable tool-routing and tool-execution adapters by separating:
|
|
- `tool_selection`: which tool to call next.
|
|
- `tool_execution`: whether arguments/use were correct and what happened.
|
|
|
|
This split supports LoRA datasets that improve planning/routing independently from low-level argument formation.
|
|
|
|
## Runtime Stages
|
|
Each model turn is logged as one or more JSONL records in `logs/taskitem_runs/ollama_mcp_exec_*/task_<n>_context.jsonl`.
|
|
|
|
### Stage: tool_selection
|
|
Recorded before MCP execution. Fields:
|
|
- `stage`: `tool_selection`
|
|
- `turn`
|
|
- `intent`
|
|
- `task_phase`
|
|
- `candidate_tools`
|
|
- `chosen_tool`
|
|
- `args`
|
|
- `reason`
|
|
- `confidence`
|
|
- `fallback_tools`
|
|
- `label`: `good` or `bad`
|
|
- `error_type` (when bad)
|
|
|
|
### Stage: tool_execution
|
|
Recorded after MCP execution. Fields:
|
|
- `stage`: `tool_execution`
|
|
- `turn`
|
|
- `tool`
|
|
- `args`
|
|
- `result`: `success|error|blocked|unknown`
|
|
- `label`: `good` or `bad`
|
|
- `error_type` (when available)
|
|
- `result_preview`
|
|
|
|
## Failure Metadata and Debug Probes
|
|
When the loop hits invalid selections, repeated issues, or max-turn exits, transcript probes are captured in `01_transcript.log`.
|
|
|
|
Failure probes can include:
|
|
- `whetstone_get_workflow_state`
|
|
- `whetstone_get_blockers`
|
|
- `whetstone_get_recent_events`
|
|
- optional debug-specific tools if present
|
|
- `whetstone_generate_inference_job`
|
|
|
|
These become `capture_type: "failure_probe"` records in malformed exports.
|
|
|
|
## Dataset Construction
|
|
|
|
### Malformed dataset
|
|
Script: `tools/mcp/export_malformed_mcp_calls_for_lora.sh`
|
|
|
|
Includes:
|
|
- adapter loop errors (`invalid_action`, unknown tool, etc.)
|
|
- MCP tool errors from `result_preview`
|
|
- stage-labeled bad events from selection/execution
|
|
- failure probe records from transcript
|
|
|
|
Output: `training_data/lora/malformed_mcp_calls.jsonl`
|
|
|
|
### Quality dataset
|
|
Script: `tools/mcp/export_mcp_call_quality_for_lora.sh`
|
|
|
|
Includes:
|
|
- pipeline-derived known-good calls
|
|
- stage-labeled `good`/`bad` examples
|
|
- legacy adapter and MCP tool-response bad examples
|
|
|
|
Outputs:
|
|
- `training_data/lora/mcp_call_quality.jsonl`
|
|
- `training_data/lora/mcp_calls_good.jsonl`
|
|
- `training_data/lora/mcp_calls_bad.jsonl`
|
|
|
|
## Synthesis Strategy
|
|
Use mixed sampling:
|
|
- Positive examples from clean pipeline runs.
|
|
- Hard negatives from malformed outputs and near-miss tool choices.
|
|
- Counterfactual negatives where similar tools are available but incorrect.
|
|
|
|
Recommended metadata tags for future synthesis:
|
|
- `interface_mode`: `json_loop` vs `native_tool_calling`
|
|
- `selection_confidence_bucket`
|
|
- `recovery_attempted`
|
|
- `task_domain` / `codebase_area`
|
|
|
|
## Batch Orchestration
|
|
Script: `tools/mcp/run_slm_training_batch.sh`
|
|
|
|
Per run manifest fields now include:
|
|
- token usage totals
|
|
- `stage_metrics.selection_events`
|
|
- `stage_metrics.selection_bad`
|
|
- `stage_metrics.execution_events`
|
|
- `stage_metrics.execution_bad`
|
|
|
|
Use these metrics to rank runs for curation and to detect regressions in routing quality.
|
|
|
|
## Curation Rules
|
|
Promote samples when:
|
|
1. Selection was confident and consistent with successful execution.
|
|
2. Execution finished without schema/tool errors.
|
|
3. Recovery (if any) converged quickly.
|
|
|
|
Demote or isolate samples when:
|
|
1. Selection repeatedly picks unavailable or semantically wrong tools.
|
|
2. Args repeatedly violate schema despite retries.
|
|
3. Context resets dominate the run (possible instability/noise).
|