whetstone_RSA/docs/gate_rubric.md

Gate Rubric

Purpose

This document gives a provisional deployment rubric for the current WhetstoneDSL specialist gates.

The goal is not to rank models by benchmark score. The goal is to decide, per gate:

• deploy now
• deploy with guardrails
• wait for larger-tier comparison
• redesign the gate before scaling

Rubric Buckets

Deploy Now

Use when:

• the gate is effectively deterministic or near-deterministic
• wrong decisions are easy to catch or reverse
• the current model already clears the practical quality bar

Deploy With Guardrails

Use when:

• the gate is useful at the current size
• raw accuracy is not enough for blind auto-accept
• confidence thresholding, retry, or deterministic checks can contain risk

Wait For Large Run

Use when:

• the current tiny-tier result is too weak for safe deployment
• the gate definition appears defensible
• larger-tier quality data is actively being generated

Redesign Before Scaling

Use when:

• the label space appears structurally muddy
• the gate may really be multiple smaller gates
• important input context is missing
• bigger models would mostly compensate for a bad formulation

Current WhetstoneDSL Gates

confidence_tier

• Tiny-tier result: 100%
• Provisional bucket: Deploy Now
• Rationale:
  • this is very close to a formula task
  • it should be treated as a deterministic baseline whenever possible
  • specialist use is acceptable, but only as a convenience layer over a known rule

verification_type

• Tiny-tier result: about 84.9%
• Provisional bucket: Deploy With Guardrails
• Rationale:
  • bounded 5-way task with real signal at the tiny tier
  • likely useful if accepted predictions have high precision
  • reversible in many workflows because verification type can be validated or corrected downstream
• Recommended next step:
  • evaluate confidence-thresholded precision
  • inspect confusion concentration, especially adjacent classes

worker_type

• Tiny-tier result: about 88.5%
• Provisional bucket: Deploy With Guardrails
• Rationale:
  • strong enough to be operationally useful if low-confidence cases abstain
  • wrong worker selection is often recoverable
  • likely a good candidate for first-pass routing plus retry
• Recommended next step:
  • measure accepted precision and abstain coverage
  • add a cheap retry path with richer context

prereq_op

• Tiny-tier result: about 69.4%, flat across steps 1000-4000
• Provisional bucket: Redesign Before Scaling
• Rationale:
  • the current 4-way label is really a composition of two latent yes/no decisions: needs architect review? and needs manual approval?
  • the current generator maps those two axes into one of: standard, needs_review, needs_approval, full_gates
  • that means the model is being asked to infer a product label from short text, even though the natural decision structure is factorized
  • labels are heuristic and keyword-driven, which likely adds noise
  • scaling the model may help, but it may also just memorize fuzzy boundaries
• Why it is hard:
  • short descriptions often under-specify whether a task is architectural, production-sensitive, both, or neither
  • many cues are policy-level rather than lexical
  • some examples mix "phase/integration/framework" signals with release/security signals, making the combined class boundary weak
  • the downstream action is itself compositional, which suggests the predictor should also be compositional
• Recommended decomposition:
  • gate A: requires_architect_review → yes/no
  • gate B: requires_manual_approval → yes/no
  • deterministic combiner:
    • 00 -> standard
    • 10 -> needs_review
    • 01 -> needs_approval
    • 11 -> full_gates
• Why decomposition is attractive:
  • each model solves a cleaner binary problem
  • outputs align directly with the policy semantics
  • confidence can be calibrated independently per axis
  • one axis may be deterministic or near-deterministic in some contexts
• What to test before scaling:
  • binary performance for each axis
  • whether richer structured inputs outperform plain step text
  • whether some approval decisions should be deterministic policy checks instead

automatability

• Tiny-tier result: about 69.6%, flat across steps 1000-4000
• Provisional bucket: Wait For Large Run and Redesign Before Scaling
• Rationale:
  • this gate is conceptually important, but the label boundaries are likely moving targets because they depend on what tools and templates the system already has
  • adjacent classes are semantically close
  • the cost of some confusions is very asymmetric
• Why it is hard:
  • the classes are not purely descriptive; they are partly roadmap-dependent
  • a node can move from llm to specialist or template as tooling improves
  • the difference between template, specialist, slm, and llm is partly an operational routing policy, not just an intrinsic property of the node
• Recommended next step:
  • wait for current large runs to finish
  • inspect confusion matrix, not just headline accuracy
  • consider a hierarchical formulation
• Possible decomposition:
  • gate A: is deterministic/template sufficient?
  • gate B: if learned, is output bounded or open-vocabulary?
  • gate C: if open-vocabulary, does small-model routing suffice or is full LLM needed?

Immediate Operational Policy

Until the larger runs complete, use this posture:

• confidence_tier: deterministic or auto-deploy
• verification_type: candidate for guarded deployment
• worker_type: candidate for guarded deployment
• prereq_op: do not blindly scale; test decomposed formulation first
• automatability: keep training and collect richer evidence before committing

Main Lesson

When a bounded gate performs poorly, the first question should not be:

"How much bigger should the model be?"

The first questions should be:

• is the gate actually one gate?
• is the label space stable?
• is the input missing policy-critical structure?
• should part of the logic be deterministic?

That is the point of this rubric.