WhetstoneRSA

When LLMs Are Overkill

Messy input, small menu, local transformer gate

The starting point

LLMs are amazing.

But many software decisions are much smaller than an LLM.

The shape

The input can be huge.

The output is small.

Example request

`show me the failed builds from yesterday`

Same intent, many phrasings

what broke in CI yesterday
pull up yesterday's red builds
did the pipeline fail last night
which build jobs failed since yesterday

What the app actually needs

action

search_builds

status

failed

date

yesterday

Output scale

5 actions x 4 statuses x 4 date buckets
= about 80 structured outputs

Classical code starts well

if text has "failed" and "build":
    action = search_builds

if text has "yesterday":
    date = yesterday

Then regex pays the human tax

You enumerate the input side.

Every new phrase becomes another pattern, exception, or bug.

Rule explosion

5 actions
x 10 common phrasings
x 4 statuses
x 8 time phrasings
= 1,600-ish rule cases

Regex tradeoff

excellent output control
tiny runtime cost
input coverage gets expensive

LLMs feel easy

They understand the messy input.

One prompt can replace a pile of brittle phrase rules.

But the scale is wrong

A giant general model

choosing from a tiny application menu.

API runtime scale

Model	Input	Output
GPT-4.1	$2 / 1M tokens	$8 / 1M tokens
GPT-4.1 mini	$0.40 / 1M tokens	$1.60 / 1M tokens
GPT-4.1 nano	$0.10 / 1M tokens	$0.40 / 1M tokens

Fine-tuning scale

Fine-tuned model	Training	Runtime input	Runtime output
GPT-4.1 mini	$5 / 1M tokens	$0.80 / 1M	$3.20 / 1M
GPT-4.1 nano	$1.50 / 1M tokens	$0.20 / 1M	$0.80 / 1M

My original plan

Use LoRA on a local SLM.

Adapt a small language model to WhetstoneDSL decisions.

LoRA still carries the base model

1B / 3B / 7B parameter base
adapter is small
runtime model is still big

The better question

Was the task ever large enough to deserve that model?

Where this came from

WhetstoneDSL

AST-first programming and universal transpiling

What a taskitem is

a small implementation work packet
title, requirements, constraints
acceptance criteria and dependencies

Taskitems need prep decisions

What has to happen before this task runs?

The prep checklist

validate the intake
resolve dependencies
send to architect review

In the repo

`prereq_op_selector`

picks the prerequisite operations for a taskitem

Tiny transformer gate

Learn the messy input.

Keep the output menu fixed.

Actual training cost

3 to 5minutes

3060RTX GPU

per transformer gate

Actual model scale

213Kparameters

800KBcheckpoint

one specialist per small gate

Training data source

LLM as scaffolding.

Use it to bootstrap examples, not to sit in every runtime decision.

Synthetic data can help

generate taskitem variants
assign known labels
oversample edge cases
train the tiny gate directly

First extracted dataset

1297clean rows

1102train

195eval

The data caught a design mistake

`needs_validate_intake`

was always true: 1297 / 1297

Split the gate

Gate 1

resolve dependencies?

Gate 2

architect review?

First results

Gate	Accuracy	Training
resolve dependencies	69.4%	3-5 min on 3060
architect review	75.0%	3-5 min on 3060

What we learned

1297 rows did not mean 1297 different examples.

There were only about 17 unique text templates.

The comparison

Approach	Input handling	Runtime	Output control
Regex	manual enumeration	tiny	excellent
LLM / SLM + LoRA	easy	base model still runs	needs guardrails
Tiny gate transformer	learned	~800KB specialist	fixed output head

Final claim

LoRA adapts a large model to the task.

WhetstoneRSA asks whether the task deserved the large model.