diff --git a/presentations/rsa_tiny_models_talk.md b/presentations/rsa_tiny_models_talk.md index f06f402..5398c70 100644 --- a/presentations/rsa_tiny_models_talk.md +++ b/presentations/rsa_tiny_models_talk.md @@ -225,186 +225,247 @@ Huge input cloud mapped into a small action menu Speaker notes: The model does not need to generate arbitrary text. -## Slide 18: This Is Closer To Classical NLP +## Slide 18: First Real Target -- intent classification -- entity extraction -- slot filling -- query interpretation +`whetstone_DSL` + +AST-first codegen pipeline Visual idea: -Old-school NLP pipeline icons +WhetstoneDSL box feeding RSA gates Speaker notes: -This is often closer to smart search than to modern chatbot behavior. +This is not a toy example. These are real pipeline decisions. -## Slide 19: Search Bar Example +## Slide 19: The Initial Gate -Make a search/help bar feel intelligent. +`prereq_op_selector` -Stay local. +Decides prerequisite operations -Stay bounded. +before a taskitem executes. Visual idea: -Application search bar with local chip or small-device icon +Taskitem -> prereq gate -> operation checklist Speaker notes: -This is one of the clearest examples for programmers. +The gate decides what must happen before execution. -## Slide 20: Search Bar Inputs +## Slide 20: Gate Input -Users type: +Input is structured task context: -- `where do I change export settings` -- `find the failed import from last week` -- `how do I reset my API token` +- title +- requirements +- acceptance criteria +- constraints Visual idea: -Three stacked queries in a fake app UI +Four input cards into one gate Speaker notes: -These are messy, but the outputs are still bounded. +The model sees the compact decision context, not the whole world. -## Slide 21: Search Bar Outputs +## Slide 21: Gate Output -RSA decides: +Output was: -- mode -- entity -- filters -- confidence +- `needs_resolve_dependencies` +- `needs_architect_review` +- `needs_validate_intake` Visual idea: -Four labeled output chips +Three boolean switches Speaker notes: -It interprets what subsystem should run and how. +This is the bounded contract surface. -## Slide 22: Search Bar Execution +## Slide 22: Source Corpus -Deterministic systems then: +1000 pipeline runs -- open settings -- run filtered search -- show help docs +JSON artifacts + +real decision logs Visual idea: -Three deterministic destination panels +Stack of JSON files Speaker notes: -RSA interprets the request and fills the structured call. +The point is extracting supervision from actual pipeline behavior. -## Slide 23: Where Else This Fits +## Slide 23: Extraction -- command palettes -- workflow routing -- support triage -- tool selection -- template selection +`extract_gate_rows.py` + +turns run artifacts + +into per-gate rows. Visual idea: -Grid of small app examples +JSON stack -> extraction script -> row table Speaker notes: -Same shape, different domain. +Each row isolates one decision. -## Slide 24: WhetstoneDSL / WHIMP +## Slide 24: Clean Rows -This fits: +1321 total rows -- `whetstone_DSL` -- `WHIMP` +24 schema-drift rows rejected + +1297 clean rows Visual idea: -Two case-study boxes with arrows into RSA +Row counter with a small rejected pile Speaker notes: -Those systems have many bounded decisions that are painful to express as pure rules. +Bad rows get rejected instead of quietly poisoning the gate. -## Slide 25: A Lot Of “LLM Workflows” Are Really This +## Slide 25: The Surprise -- classification -- routing -- extraction -- slot filling +`needs_validate_intake` + +was always true. + +1297 / 1297 Visual idea: -Big prompt window collapsing into four small tasks +One switch permanently locked on Speaker notes: -A lot of LLM spend is really being used as fuzzy routing over deterministic software. +A label with zero variance is not learnable. It is a constant. -## Slide 26: The Enterprise Angle +## Slide 26: Gate Decomposition -Many teams use giant prompts and context windows +The real gate became two gates: -for tasks that are actually bounded. +- resolve dependencies +- architect review Visual idea: -Huge prompt box compared to small contract box +One big gate splitting into two smaller gates Speaker notes: -That is the efficiency argument. +The data changed the gate shape. That is the process working. -## Slide 27: The Careful Claim +## Slide 27: Tiny Specialists -RSA does **not** replace all LLM reasoning. +One specialist per gate. + +About 213K parameters. + +About 800KB on disk. + +Visual idea: +Tiny model file next to a normal image file + +Speaker notes: +The size is small because the decision surface is small. + +## Slide 28: First Results + +Baseline accuracy: + +- resolve: 69.4% +- architect: 75.0% + +Visual idea: +Two simple bars + +Speaker notes: +These are not victory-lap numbers. They are diagnostic numbers. + +## Slide 29: Honest Diagnosis + +1297 rows + +but only about 17 unique text patterns. + +Visual idea: +Lots of rows collapsing into a few templates + +Speaker notes: +More rows from the same shape do not create richer signal. + +## Slide 30: The Fix + +Better input fields: + +- architecture surface +- cross-component deps +- security sensitivity +- uncertainty score + +Visual idea: +Taskitem schema before and after + +Speaker notes: +The fix is improving the contract upstream, not pretending the weak model is good. + +## Slide 31: Model Sizing + +Match model size + +to gate entropy. + +Visual idea: +Gate difficulty -> model tier + +Speaker notes: +The goal is the smallest model that clears the quality bar. + +## Slide 32: What Makes A Gate Hard? + +- more labels +- class imbalance +- wider context +- fuzzy boundaries +- slot interactions + +Visual idea: +Difficulty knobs + +Speaker notes: +These are the factors that should predict model tier. + +## Slide 33: Runtime Scaffold + +C++ library: + +- gate definitions +- evidence +- diagnosis +- probes + +Visual idea: +Fabricate specialist -> RSA runtime -> WhetstoneDSL + +Speaker notes: +The repo is not just notes. There is a runtime scaffold. + +## Slide 34: Careful Claim + +RSA does not replace all LLM reasoning. RSA replaces bounded semantic decisions. Visual idea: -Two circles: `bounded decisions` inside scope, `open-ended reasoning` outside scope +Bounded decisions inside scope, open-ended reasoning outside Speaker notes: -Keep the claim narrow and defensible. +Keep the claim narrow. That makes it harder to knock down. -## Slide 28: Large Context Caveat +## Slide 35: Closing -RSA is best when the decision can use: +Fuzzy input in. -- compact context -- retrieved context -- distilled context +Typed contract out. + +Deterministic software after that. Visual idea: -Large document pile reduced to a few selected cards +Final clean pipeline graphic Speaker notes: -For large-context tasks, pair RSA with retrieval or preprocessing. - -## Slide 29: Another Example Near The Edge - -Script generation can sometimes be: - -- template selection -- slot filling -- deterministic rendering - -Visual idea: -Prompt -> template -> filled script - -Speaker notes: -Some “code generation” is really hidden template filling. - -## Slide 30: The Main Idea - -Use a tiny model only where deterministic software still needs help. - -Visual idea: -Small model block at the routing boundary only - -Speaker notes: -That is the whole philosophy. - -## Slide 31: Closing - -`whetstone_RSA` makes software semantically smarter - -without turning everything into a chatbot. - -Visual idea: -Final pipeline graphic, clean and minimal - -Speaker notes: -That is the punchline I want people to leave with. +Do not make the model own the whole task when it only needs to own the ambiguous boundary. diff --git a/presentations/whetstoneRSA_fast_deck.html b/presentations/whetstoneRSA_fast_deck.html new file mode 100644 index 0000000..3cacf96 --- /dev/null +++ b/presentations/whetstoneRSA_fast_deck.html @@ -0,0 +1,814 @@ + + + + + +WhetstoneRSA - Tiny Models For Bounded NLP Decisions + + + +
+ +
+
whetstone_RSA
+
+

Tiny Models For Bounded NLP Decisions

+

Turning messy user requests into deterministic tool calls

+
+
1RSA: contract layer, not chatbot
+
+ +
+
A common shape
+
+

Users type messy things.

+

Software needs clean actions.

+
+
2messy input -> clean action
+
+ +
+
The gap
+
+
    +
  • fuzzy input
    • +
  • bounded output
    • +
  • deterministic execution
    • +
+
+
3the shape RSA targets
+
+ +
+
Today's options
+
+
+
cheap
regex
+
inspectable
rules
+
flexible
LLM prompt
+
+
+
4brittle vs oversized
+
+ +
+
Regex is cheap
+
+
    +
  • fast
    • +
  • local
    • +
  • inspectable
    • +
+
+
5rules have real advantages
+
+ +
+
Regex is also annoying
+
+
    +
  • brittle
    • +
  • tedious
    • +
  • bad at paraphrases
    • +
+
+
6variation breaks brittle rules
+
+ +
+
LLMs feel great
+
+
    +
  • flexible
    • +
  • natural-language friendly
    • +
  • good with paraphrases
    • +
+
+
7large models are useful
+
+ +
+
LLMs are often overkill
+
+
    +
  • expensive
    • +
  • heavy
    • +
  • hard to control
    • +
+
+
8bounded tasks do not need open-ended machinery
+
+ +
+
The missing middle
+
+
regex -> RSA -> LLM
+
+
9a middle layer for bounded semantics
+
+ +
+
What RSA is
+
+

A contract layer for bounded semantic decisions.

+
+
10contract, not conversation
+
+ +
+
What RSA takes
+
+
    +
  • messy input
    • +
  • bounded schema
    • +
  • optional context
    • +
+
+
11schema narrows the task
+
+ +
+
What RSA returns
+
+
    +
  • action
    • +
  • slots
    • +
  • confidence
    • +
  • abstain or escalate
    • +
+
+
12structured result
+
+ +
+
The core pattern
+
+
+
1
Natural language
+
2
Semantic interpretation
+
3
Bounded contract
+
4
Deterministic software
+
+
+
13model at the boundary only
+
+ +
+
Example query
+
+

show me the failed builds from yesterday

+
+
14messy but bounded
+
+ +
+
Example contract
+
+ 
action:     search_builds
+status:     failed
+date:       yesterday
+confidence: 0.93
+
+
15typed shape, not prose
+
+ +
+
Example execution
+
+
    +
  • run the query
    • +
  • render the results
    • +
  • log the action
    • +
+
+
16ordinary software after the contract
+
+ +
+
Why this can stay tiny
+
+

The output is bounded.

+
+
17small output surface, small policy
+
+ +
+
First real target
+
+

whetstone_DSL

+

AST-first codegen pipeline

+
+
18the case study
+
+ +
+
The initial gate
+
+

prereq_op_selector

+

Decides prerequisite operations before a taskitem executes.

+
+
19one bounded pipeline decision
+
+ +
+
Gate input
+
+
    +
  • title
    • +
  • requirements
    • +
  • acceptance criteria
    • +
  • constraints
    • +
+
+
20compact context
+
+ +
+
Gate output
+
+
    +
  • needs_resolve_dependencies
    • +
  • needs_architect_review
    • +
  • needs_validate_intake
    • +
+
+
21three booleans
+
+ +
+
Source corpus
+
+
1000pipeline runs
+

JSON artifacts with real decision logs

+
+
22real pipeline artifacts
+
+ +
+
Extraction
+
+

extract_gate_rows.py

+

run artifacts -> per-gate rows

+
+
23isolate supervision by gate
+
+ +
+
Clean rows
+
+
+
1321total rows
+
24schema drift rejected
+
1297clean rows
+
+
+
24reject bad evidence
+
+ +
+
The surprise
+
+

needs_validate_intake

+

was always true: 1297 / 1297

+
+
25constant, not gate
+
+ +
+
Gate decomposition
+
+
+
Gate 1
resolve dependencies
+
Gate 2
architect review
+
+
+
26one bundled gate became two gates
+
+ +
+
Tiny specialists
+
+
+
213Kparameters
+
800KBon disk
+
+

one specialist per gate

+
+
27small model for small decision surface
+
+ +
+
First results
+
+
+
+
resolve dependencies69.4%
+
+
+
+
architect review75.0%
+
+
+
+
+
28baseline accuracy at 4000 steps
+
+ +
+
Honest diagnosis
+
+

1297 rows

+

but only about 17 unique text patterns

+
+
29row count is not diversity
+
+ +
+
The fix
+
+
    +
  • architecture surface
    • +
  • cross-component deps
    • +
  • security sensitivity
    • +
  • uncertainty score
    • +
+
+
30better input, better gate
+
+ +
+
Model sizing
+
+

Match model size to gate entropy.

+
+
31smallest passing tier
+
+ +
+
What makes a gate hard?
+
+
    +
  • more labels
    • +
  • class imbalance
    • +
  • wider context
    • +
  • fuzzy boundaries
    • +
  • slot interactions
    • +
+
+
32entropy factors
+
+ +
+
Runtime scaffold
+
+
    +
  • gate definitions
    • +
  • evidence
    • +
  • diagnosis
    • +
  • probes
    • +
+
+
33C++ deployment layer
+
+ +
+
Careful claim
+
+

RSA does not replace all LLM reasoning.

+

RSA replaces bounded semantic decisions.

+
+
34narrow and defensible
+
+ +
+
Closing
+
+
+
Fuzzy input in.
+
Typed contract out.
+
Deterministic software after that.
+
+
+
35the whole point
+
+ +
+ +
Click / arrows: navigate · N: notes · F: fullscreen
+
+
+ + + +