# Multi-Model LoRA Data Playbook

## Goal
Build a reusable dataset where failures come from multiple models and good examples come from validated pipeline outputs, then rebalance into a training-ready JSONL.

## Required metadata per record
Every record should carry:
- `model` (for example `qwen2.5-coder:14b`, `qwen2.5-coder:7b`)
- `label` (`good` or `bad`)
- `interface_mode` (`json_loop` or `native_tool_calling`)
- `source` (for example `pipeline_run`, `stage_labeled`, `mcp_tool_response`)

## Collection flow
1. Run collection loops per model.
2. Export malformed and quality datasets after each run.
3. Keep run manifests for token/cycle diagnostics.
4. Build one balanced multimodel output before training.

## Suggested run strategy
- Keep one primary target model (for example 14B) as highest share of data.
- Add 1-2 secondary models for failure diversity.
- Avoid overfitting to one repeated failure signature.

## Commands

### 0) Good-data hybrid flow (deterministic + optional stronger-LLM review)
```bash
# Step A: deterministic good-data growth from run specs (same corpora as bad-data loop by default)
./tools/mcp/generate_good_from_run_specs.sh

# Step B: split good records into high-confidence and review queue
./tools/mcp/build_good_hybrid_dataset.sh

# One-command orchestrator (A+B)
./tools/mcp/run_good_data_hybrid.sh

# After stronger-LLM review creates approved-good JSONL:
REVIEW_APPROVED_FILE=training_data/lora/mcp_calls_good_review_approved.jsonl \
  ./tools/mcp/run_good_data_hybrid.sh
```

### 1) Per-run exports
```bash
./tools/mcp/export_malformed_mcp_calls_for_lora.sh logs/taskitem_runs/<bridge_run_dir>
./tools/mcp/export_mcp_call_quality_for_lora.sh logs/taskitem_runs/<good_pipeline_run_dir> logs/taskitem_runs/<bridge_run_dir>
```

### 2) Build balanced multimodel dataset
```bash
BAD_TO_GOOD_RATIO=1.0 \
MAX_PER_MODEL_PER_LABEL=250000 \
REQUIRE_MODEL_METADATA=0 \
DEFAULT_MODEL_TAG=unknown_model \
DEFAULT_INTERFACE_MODE=json_loop \
./tools/mcp/build_multimodel_lora_dataset.sh
```

Outputs:
- `training_data/lora/mcp_multimodel_balanced.jsonl`
- `training_data/lora/mcp_multimodel_balanced.summary.json`

### 3) Create leakage-safe train/val/test split (grouped by run/spec family)
```bash
./tools/mcp/split_lora_dataset_by_group.sh
```

Outputs:
- `training_data/lora/splits/mcp_lora.train.jsonl`
- `training_data/lora/splits/mcp_lora.val.jsonl`
- `training_data/lora/splits/mcp_lora.test.jsonl`
- `training_data/lora/splits/mcp_lora.split_summary.json`

## Quality gates before training
- Confirm `selected_good` and `selected_bad` counts are both non-trivial.
- Confirm multiple models appear in `by_model_selected`.
- Confirm source diversity in `by_source_selected`.
- Spot-check for infra-only failures; keep operational noise low.

## For future agents
When collecting new runs:
- Do not overwrite raw files.
- Rebuild balanced output from raw files each time.
- Record all run IDs and model tags in commit notes.

When training:
- Train separate LoRAs per base model family/size.
- Reuse the same balanced dataset recipe; do not reuse adapters across base sizes.