whetstone_RSA/presentations/slide_deck_brief.md

# Slide Deck Brief — whetstone_RSA: Tiny Models for Bounded Decisions

**Audience:** Whetstone_RSA, ex data science instructor. Technical depth expected.
Wants to understand the actual transformer training approach, not just the concept.

**Format:** Slide deck, dark theme preferred. One idea per slide, minimal text.
Visuals are in the same directory as this file (HTML files — embed or screenshot them).

**Length:** ~30–35 slides across two acts.

---

## Act I — The Problem and the Pattern (Slides 1–17)

These slides are already written in `rsa_tiny_models_talk.md`. Keep them as-is.
They establish the motivation: fuzzy input → bounded output → deterministic execution.
Key points already covered:
- Why regex is brittle
- Why LLMs are oversized for bounded tasks
- What RSA does (contract layer between user and tool)
- The core pipeline: input → RSA gate → typed contract → deterministic software
- Where this pattern shows up (command palettes, search bars, workflow routing)

**Visual for Act I:** `visual_rsa_pipeline.html`

---

## Act II — The Actual Training Work (Slides 18–35)

This is the new material. It covers the real implementation, dataset, and results.

---

### Slide 18: The Case Study

**Title:** First Real Target — whetstone_DSL

**Content:**
- whetstone_DSL is a C++ IDE with an AST-first codegen pipeline
- It uses an LLM to make many bounded decisions bundled together
- Goal: replace those bundles with one tiny specialist per decision gate
- First gate selected: `prereq_op_selector`
  - Decides which prerequisite operations a taskitem needs before it can execute

**Speaker notes:** This is not a toy example. These gates run on real pipeline decisions.

---

### Slide 19: What a Gate Is

**Title:** The Gate Contract

**Content:**
```
input:   task title · requirements · acceptance criteria · constraints
output:  needs_resolve_dependencies: bool
         needs_architect_review: bool
         needs_validate_intake: bool  (always true — dropped)
         confidence: float
         abstain: bool
```

**Visual idea:** Simple box diagram: structured text in, typed struct out.

**Speaker notes:** The output is not generated text. It is a typed struct with known fields.

---

### Slide 20: The Training Pipeline Overview

**Title:** From Run Artifacts to Specialist

**Visual:** `visual_training_pipeline.html` — use full slide.

**Speaker notes:** Six-step pipeline. Real data from real pipeline runs. The training
framework (Fabricate) lives on the GPU desktop — can demo it live there.

---

### Slide 21: Step 1 — The Corpus

**Title:** Source Data: 1000 Pipeline Runs

**Content:**
- whetstone_DSL was run 1000 times on real and synthetic projects
- Each run produces JSON log artifacts recording every taskitem decision
- `extract_gate_rows.py` parses these into per-gate supervision rows

**Visual idea:** Stack of JSON files on left → extraction tool → rows on right.

---

### Slide 22: Step 2 — Gate Extraction

**Title:** Isolating One Decision at a Time

**Content:**
- Each row contains: decision_context · decision_label · decision_provenance
- Only the information available at decision time is allowed in the context
- Rows with schema-drift labels are rejected (24 of 1321 total)
- Result: 1297 clean rows for prereq_op_selector

**Key extraction rule:** Never leak downstream fields into the context.
A gate must only see what was available when it would actually fire.

---

### Slide 23: The Discovery — Gate Decomposition

**Title:** needs_validate_intake Is Always True

**Content:**
- Analysis of 1297 rows: needs_validate_intake = True in all 1297 of them
- A label with zero variance cannot be learned — it is a constant, not a gate
- Real gate decomposes into two independent binary classifiers:
  - needs_resolve_dependencies (76.9% positive rate)
  - needs_architect_review (19.8% positive rate)

**Visual:** `visual_gate_decomposition.html`

**Speaker notes:** This is a real finding, not a design choice made upfront. The data
told us the gate shape was wrong. That's the process working correctly.

---

### Slide 24: Dataset Anatomy

**Title:** What 1297 Rows Actually Looks Like

**Visual:** `visual_dataset_anatomy.html` — use full slide.

**Speaker notes:** The ~17 unique templates is the important number. High row count
does not mean high diversity. This is the honest accounting.

---

### Slide 25: The TSV Format

**Title:** Fabricate Training Format

**Content:**
```
label <TAB> hop <TAB> text
```
- label: 0 (no) or 1 (yes) — one file per binary gate
- hop: always 0 (unused routing field in Fabricate)
- text: composed from title · reasons · acceptance criteria · constraints

**Four files produced:**
- prereq_resolve_train.tsv (1102 rows)
- prereq_resolve_eval.tsv (195 rows)
- prereq_architect_train.tsv (1102 rows)
- prereq_architect_eval.tsv (195 rows)

**Split:** 85% train / 15% eval, random seed 42.

---

### Slide 26: The Model — Tiny Transformer

**Title:** ~213K Parameters. ~800KB on Disk.

**Content:**
- Framework: Fabricate (custom tiny transformer training tool)
- Architecture: transformer encoder with binary classification head
- ~213K parameters per specialist
- Serialized checkpoint: ~800KB
- Training: GPU desktop, 4000 steps per specialist
- Inference: CPU-friendly, local, no network dependency

**Visual idea:** Size comparison — model checkpoint vs. a JPEG photo (~800KB each).

**Speaker notes:** This is the whole point. A model smaller than an average image file,
making a bounded yes/no decision. Can demo inference live on the GPU desktop.

---

### Slide 27: Training Results

**Title:** Baseline Results @ 4000 Steps

**Content:**

| Specialist | Accuracy | Steps | Positive rate |
|---|---|---|---|
| prereq_resolve (needs_resolve_dependencies) | 69.4% | 4000 | 76.9% |
| prereq_architect (needs_architect_review) | 75.0% | 4000 | 19.8% |

**Important context:**
- A majority classifier on needs_resolve would score ~77% (model is below naive baseline)
- A majority classifier on needs_architect would score ~80% (model is learning minority class)
- These are weak baselines — expected given the 17-template corpus

**Visual idea:** Bar chart with majority-classifier baseline marked as a reference line.

---

### Slide 28: Why the Baselines Are Weak (And Why That's Fine)

**Title:** The Corpus Was Not Designed for This

**Content:**
- The taskitem schema was designed to record execution outputs, not discriminative inputs
- 1297 rows, but only ~17 unique text patterns — the model has almost no signal variation
- The labels are correct; the features are not rich enough yet

**The fix is not more data from the same corpus.**
The fix is adding RSA-native fields to the taskitem schema upstream.

**Speaker notes:** This is an architectural finding. The pipeline methodology is validated.
Signal quality is a schema problem, not a model problem.

---

### Slide 29: What Would Make the Signal Better

**Title:** Better Input = Better Gate

**Content:**
Each taskitem needs to carry:
- `architectural_surface` — what part of the system is touched
- `cross_component_deps` — explicit dependency flags
- `security_sensitive` — flag for security-relevant changes
- `uncertainty_score` — explicit uncertainty signal

With these fields, some gates that currently need ML may become deterministic rules.
The RSA entropy threshold shifts as the contract gets richer.

**Visual idea:** Taskitem schema before vs. after adding RSA fields.

---

### Slide 30: The Entropy → Model Size Hypothesis

**Title:** Match the Model to the Decision Surface

**Visual:** `visual_model_tiers.html`

**Speaker notes:** The hypothesis is measurable and falsifiable. The experiment is:
train each tier on the same gate, record the smallest one that clears the target.

---

### Slide 31: Gate Entropy Factors

**Title:** What Makes a Gate Hard?

**Content:**
Factors that increase required model tier:
- output_cardinality (more labels = harder)
- class_imbalance
- lexical_variation in inputs
- context_width_required
- label_boundary_fuzziness
- world_knowledge_dependence
- slot_interdependence

**Target deliverable:**
```
gate_id | entropy_score | smallest_passing_tier | accuracy | latency_ms
```

**Speaker notes:** We don't have this table yet. It's the next experiment.

---

### Slide 32: What's Next — SPRINT-004

**Title:** Pipeline Decision Audit

**Content:**
Before training more gates, map every decision in the full whetstone_DSL pipeline:
- Start at RegisterArchitectIntakeTools.h (pipeline entry point)
- At every stage: what decision is being made?
- What inputs are available?
- What is the optimal method: deterministic / RSA / SLM / LLM / human?

`generate_taskitems` alone bundles 10+ separable decisions.

**Output:** `docs/case_studies/whetstone_dsl_pipeline_decision_map.md`

**Speaker notes:** Train data collection strategy follows from the decision map.
Don't collect data for the wrong gate shape.

---

### Slide 33: What Is Already Deterministic

**Title:** Most of Whetstone Is Already Right

**Content:**
Once the AST and target language are fixed, most current code generation is deterministic:
- Target generator dispatch — deterministic
- AST concept dispatch — deterministic
- Environment gating — deterministic
- C++ raising profile validation — deterministic

RSA is highest value in:
- taskitem generation
- execution contract shaping
- routing decisions
- pre-codegen profile decisions

**Speaker notes:** This is a feature, not a gap. Don't replace deterministic software
with models. Find the actual ambiguity boundary.

---

### Slide 34: The C++ Runtime

**Title:** whetstone_RSA Runtime Scaffold

**Content:**
A C++ library wrapping the gate contracts:
- `GateDefinition` · `GateEvidence` · `GateDiagnosis` · `GateSuitability`
- `ProbeRegistry` + `DiagnosisEngine`
- First concrete probe: `confidence_threshold_probe`
- Builds with CMake, demo executable runs

This is the deployment layer — gates are loaded here at runtime.

**Visual idea:** Stack diagram: Fabricate specialist → RSA runtime → whetstone_DSL pipeline.

---

### Slide 35: The Summary

**Title:** What This Is

**Content:**
1. Real, running pipeline decisions in whetstone_DSL
2. Real extracted training data (1297 rows, two binary gates)
3. Real trained specialists (~800KB each, ~213K params)
4. Honest baselines (weak corpus, clear reason, clear fix)
5. A measurable hypothesis: entropy → tier selection
6. A C++ runtime that loads and runs them

**Not a chatbot. Not a search engine.**
A bounded decision contract layer, with real data and real results.

---

## Appendix Slides (optional, for Q&A)

- Rejection taxonomy (schema drift, sibling-uniform labels)
- Full gate inventory for whetstone_DSL
- The decision_contract_ontology (input_structure, output_topology, policy_stability, etc.)
- Backend selection taxonomy (transformer vs. gradient boosting vs. Bayes — kept separate from gate identity)
- The abstain/escalate policy design

---

## Files in this directory

| File | Description |
|---|---|
| `rsa_tiny_models_talk.md` | Act I source (31 slides, already complete) |
| `slide_deck_brief.md` | This file |
| `visual_rsa_pipeline.html` | The core RSA pattern — fuzzy in, contract out |
| `visual_training_pipeline.html` | Six-step pipeline from run artifacts to trained specialist |
| `visual_gate_decomposition.html` | prereq_op_selector decomposed into two binary gates |
| `visual_dataset_anatomy.html` | Dataset stats, class balance, TSV format sample |
| `visual_model_tiers.html` | Entropy → model tier sizing diagram |