- split_prereq_op_dataset.py: extract accepted/rejected/schema-drift rows per gate - gen_prereq_op_rsa_data.py: convert accepted rows to Fabricate TSV format - 1000-run extracted dataset: 1297 accepted prereq_op rows, 24 schema-drift - Key finding: needs_validate_intake always True; effective gate is binary (needs_resolve_dependencies 77%, needs_architect_review 20%) - Key finding: run corpus text is template-driven (17 unique inputs) — not suitable for text-based training; RSA exists before ideal input is available - ROADMAP: gate lifecycle framing — goal is choice of backend, not forced determinism; RSA operates on imperfect/mixed/incomplete input by design - training_data_strategy.md: taskitem-as-self-contained-work-unit constraint - SPRINT-003: updated with 1000-run results, text audit findings, design constraints - SPRINT-004: pipeline decision audit — manual walkthrough of every decision from project description to generated artifact; execute next session Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
whetstone_RSA
whetstone_RSA is a library-first project for bounded natural-language-to-decision
systems.
The core premise is simple:
- use learned routing only at the ambiguity boundary
- keep the output space explicitly enumerable
- hand execution to deterministic software as early as possible
This project grows out of two existing lines of work:
WhetstoneAI_Fabricate: routed-specialist architecture and tiny transformer experimentsCLionProjects/whetstone_DSL: AST-first IDE and MCP pipeline with bounded decisions
Problem Statement
LLMs are often being used for tasks that are not truly open-ended:
- choose one workflow from a small set
- fill a known form schema
- classify a request into a bounded enum
- pick an AST action from a valid operation set
- route a request to a deterministic tool
Those tasks do not need internet-scale latent knowledge. They need:
- a bounded decision vocabulary
- lightweight contextual inference
- validation and abstention
- deterministic downstream execution
whetstone_RSA exists to turn that into a reusable runtime rather than a one-off
experiment.
Current Hypothesis
There is a measurable relationship between task entropy and the smallest model that can fill a decision gate reliably.
The practical goal is not "make the model bigger until it works." The goal is:
- estimate gate entropy
- choose the smallest model family that clears the quality target
- escalate to a larger model only when the entropy budget requires it
This should let downstream applications keep model scope aligned with the actual decision surface they are filling.
Initial Scope
The first target is a reusable library that supports three modes:
classify: pick one label from a fixed enumfill: populate a bounded structured schemaroute: choose a deterministic tool/template/action and return typed arguments
Each run should produce:
- selected action or label
- filled slots
- confidence
- abstain or escalate decision
- trace metadata suitable for evaluation
First Case Study
The first case study is the WhetstoneDSL specialist fleet in:
/home/bill/Documents/CLionProjects/whetstone_DSL/specialists
That work already established:
- ~800KB transformer specialists are viable for bounded editor decisions
- some tasks fit comfortably in that footprint
- other gates appear too entropic for the smallest tier and need larger variants
This project turns that observation into a formal sizing framework instead of handling it ad hoc.
Project Layout
docs/architecture.md: library boundaries and runtime modeldocs/case_studies/whetstone_dsl.md: current grounding case studydocs/entropy_model_sizing.md: baseline research plan for entropy-to-model sizing
Near-Term Deliverables
- Define the core decision schema and runtime API.
- Inventory WhetstoneDSL gates by output vocabulary, ambiguity, and failure mode.
- Build a baseline entropy score for each gate.
- Train and compare specialist sizes against those gates.
- Turn the result into a library other local projects can call.
Design Rule
Learned behavior should live only where deterministic tooling is not yet enough.
Everything after a bounded decision should be deterministic unless there is a clear reason not to do that.