whetstone_RSA

Tiny Models For Bounded NLP Decisions

Turning messy user requests into deterministic tool calls

1RSA: contract layer, not chatbot
A common shape

Users type messy things.

Software needs clean actions.

2messy input -> clean action
The gap
  • fuzzy input
  • bounded output
  • deterministic execution
3the shape RSA targets
Today's options
cheap
regex
inspectable
rules
flexible
LLM prompt
4brittle vs oversized
Regex is cheap
  • fast
  • local
  • inspectable
5rules have real advantages
Regex is also annoying
  • brittle
  • tedious
  • bad at paraphrases
6variation breaks brittle rules
LLMs feel great
  • flexible
  • natural-language friendly
  • good with paraphrases
7large models are useful
LLMs are often overkill
  • expensive
  • heavy
  • hard to control
8bounded tasks do not need open-ended machinery
The missing middle
regex -> RSA -> LLM
9a middle layer for bounded semantics
What RSA is

A contract layer for bounded semantic decisions.

10contract, not conversation
What RSA takes
  • messy input
  • bounded schema
  • optional context
11schema narrows the task
What RSA returns
  • action
  • slots
  • confidence
  • abstain or escalate
12structured result
The core pattern
1
Natural language
2
Semantic interpretation
3
Bounded contract
4
Deterministic software
13model at the boundary only
Example query

show me the failed builds from yesterday

14messy but bounded
Example contract
action:     search_builds
status:     failed
date:       yesterday
confidence: 0.93
15typed shape, not prose
Example execution
  • run the query
  • render the results
  • log the action
16ordinary software after the contract
Why this can stay tiny

The output is bounded.

17small output surface, small policy
First real target

whetstone_DSL

AST-first codegen pipeline

18the case study
The initial gate

prereq_op_selector

Decides prerequisite operations before a taskitem executes.

19one bounded pipeline decision
Gate input
  • title
  • requirements
  • acceptance criteria
  • constraints
20compact context
Gate output
  • needs_resolve_dependencies
  • needs_architect_review
  • needs_validate_intake
21three booleans
Source corpus
1000pipeline runs

JSON artifacts with real decision logs

22real pipeline artifacts
Extraction

extract_gate_rows.py

run artifacts -> per-gate rows

23isolate supervision by gate
Clean rows
1321total rows
24schema drift rejected
1297clean rows
24reject bad evidence
The surprise

needs_validate_intake

was always true: 1297 / 1297

25constant, not gate
Gate decomposition
Gate 1
resolve dependencies
Gate 2
architect review
26one bundled gate became two gates
Tiny specialists
213Kparameters
800KBon disk

one specialist per gate

27small model for small decision surface
First results
resolve dependencies69.4%
architect review75.0%
28baseline accuracy at 4000 steps
Honest diagnosis

1297 rows

but only about 17 unique text patterns

29row count is not diversity
The fix
  • architecture surface
  • cross-component deps
  • security sensitivity
  • uncertainty score
30better input, better gate
Model sizing

Match model size to gate entropy.

31smallest passing tier
What makes a gate hard?
  • more labels
  • class imbalance
  • wider context
  • fuzzy boundaries
  • slot interactions
32entropy factors
Runtime scaffold
  • gate definitions
  • evidence
  • diagnosis
  • probes
33C++ deployment layer
Careful claim

RSA does not replace all LLM reasoning.

RSA replaces bounded semantic decisions.

34narrow and defensible
Closing
Fuzzy input in.
Typed contract out.
Deterministic software after that.
35the whole point
Click / arrows: navigate · N: notes · F: fullscreen