WIP: stage all uncommitted work — sprints 46-221, graduation headers, specialist fleet, test steps 909-1988

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs/multimodel_lora_data_playbook.md

@@ -0,0 +1,87 @@
+# Multi-Model LoRA Data Playbook
+
+## Goal
+Build a reusable dataset where failures come from multiple models and good examples come from validated pipeline outputs, then rebalance into a training-ready JSONL.
+
+## Required metadata per record
+Every record should carry:
+- `model` (for example `qwen2.5-coder:14b`, `qwen2.5-coder:7b`)
+- `label` (`good` or `bad`)
+- `interface_mode` (`json_loop` or `native_tool_calling`)
+- `source` (for example `pipeline_run`, `stage_labeled`, `mcp_tool_response`)
+
+## Collection flow
+1. Run collection loops per model.
+2. Export malformed and quality datasets after each run.
+3. Keep run manifests for token/cycle diagnostics.
+4. Build one balanced multimodel output before training.
+
+## Suggested run strategy
+- Keep one primary target model (for example 14B) as highest share of data.
+- Add 1-2 secondary models for failure diversity.
+- Avoid overfitting to one repeated failure signature.
+
+## Commands
+
+### 0) Good-data hybrid flow (deterministic + optional stronger-LLM review)
+```bash
+# Step A: deterministic good-data growth from run specs (same corpora as bad-data loop by default)
+./tools/mcp/generate_good_from_run_specs.sh
+
+# Step B: split good records into high-confidence and review queue
+./tools/mcp/build_good_hybrid_dataset.sh
+
+# One-command orchestrator (A+B)
+./tools/mcp/run_good_data_hybrid.sh
+
+# After stronger-LLM review creates approved-good JSONL:
+REVIEW_APPROVED_FILE=training_data/lora/mcp_calls_good_review_approved.jsonl \
+  ./tools/mcp/run_good_data_hybrid.sh
+```
+
+### 1) Per-run exports
+```bash
+./tools/mcp/export_malformed_mcp_calls_for_lora.sh logs/taskitem_runs/<bridge_run_dir>
+./tools/mcp/export_mcp_call_quality_for_lora.sh logs/taskitem_runs/<good_pipeline_run_dir> logs/taskitem_runs/<bridge_run_dir>
+```
+
+### 2) Build balanced multimodel dataset
+```bash
+BAD_TO_GOOD_RATIO=1.0 \
+MAX_PER_MODEL_PER_LABEL=250000 \
+REQUIRE_MODEL_METADATA=0 \
+DEFAULT_MODEL_TAG=unknown_model \
+DEFAULT_INTERFACE_MODE=json_loop \
+./tools/mcp/build_multimodel_lora_dataset.sh
+```
+
+Outputs:
+- `training_data/lora/mcp_multimodel_balanced.jsonl`
+- `training_data/lora/mcp_multimodel_balanced.summary.json`
+
+### 3) Create leakage-safe train/val/test split (grouped by run/spec family)
+```bash
+./tools/mcp/split_lora_dataset_by_group.sh
+```
+
+Outputs:
+- `training_data/lora/splits/mcp_lora.train.jsonl`
+- `training_data/lora/splits/mcp_lora.val.jsonl`
+- `training_data/lora/splits/mcp_lora.test.jsonl`
+- `training_data/lora/splits/mcp_lora.split_summary.json`
+
+## Quality gates before training
+- Confirm `selected_good` and `selected_bad` counts are both non-trivial.
+- Confirm multiple models appear in `by_model_selected`.
+- Confirm source diversity in `by_source_selected`.
+- Spot-check for infra-only failures; keep operational noise low.
+
+## For future agents
+When collecting new runs:
+- Do not overwrite raw files.
+- Rebuild balanced output from raw files each time.
+- Record all run IDs and model tags in commit notes.
+
+When training:
+- Train separate LoRAs per base model family/size.
+- Reuse the same balanced dataset recipe; do not reuse adapters across base sizes.