diff --git a/specialists/HANDOFF-2026-04-18.md b/specialists/HANDOFF-2026-04-18.md new file mode 100644 index 0000000..b246ede --- /dev/null +++ b/specialists/HANDOFF-2026-04-18.md @@ -0,0 +1,86 @@ +# Specialist Fleet Handoff — 2026-04-18 + +## Status at Session End + +Sweep was killed mid-run to conserve solar battery. State: + +| Gate | Tier | Status | Accuracy | +|------|------|--------|----------| +| verification_type | small_plus | COMPLETE | 81.4% — verdict: `scale_more` | +| verification_type | medium | KILLED | Partial checkpoint, do not trust | +| worker_type | — | NOT STARTED | — | +| prereq_op | — | NOT STARTED | — | +| automatability | — | NOT STARTED | — | + +## First Task Next Session + +From the whetstone_DSL root, run the full pilot sweep fresh: + +```bash +cd /home/bill/Documents/CLionProjects/whetstone_DSL +bash specialists/scripts/sweep_all_gates.sh --pilot-only 2>&1 | tee specialists/eval/results/sweep_run.log +``` + +The `--reset` flag inside `capacity_sweep.py` ensures the broken `verification_type` medium +checkpoint from this session doesn't interfere. All 4 gates run from scratch. + +Expected runtime: ~25–35 minutes total (small_plus + medium × 4 gates, 3000 steps each). + +Results land in: +- `specialists/eval/results/_sweep.json` — per-gate sweep table +- `specialists/eval/results/summary.json` — combined best-tier per gate + +## Fixes Applied This Session (All Saved to Disk) + +1. **Venv path** — `sweep_all_gates.sh` now uses whetstone DSL venv: + `PYTHON="/home/bill/Documents/CLionProjects/whetstone_DSL/.venv/bin/python3"` + +2. **Venv path** — `capacity_sweep.py` + `prereq_op_binary_split.py` now use `sys.executable` + instead of the deleted `WhetstoneAI_Fabricate/.venv` path. + +3. **Eval JSON parse** — `eval_specialist_pt.py` now prints single-line JSON as last line: + `print(json.dumps(result)) # single line — capacity_sweep.py reads last line as JSON` + +4. **Large tier removed** — `sweep_all_gates.sh` defaults to `TIERS="small_plus,medium"`. + Large (512/4/8, 13M params) takes ~247s/block — impractical for a pilot. + +5. **Nested tensor slowpath** — `train_specialist_pt.py` has `enable_nested_tensor=False` + on TransformerEncoder. This eliminated ~40% overhead AND improved accuracy. + +6. **Inductor compile** — `torch.compile(mode="reduce-overhead")` enabled with TF32. + `python3.12-dev` is installed. Actual speedup: ~62s → ~45s/block (~25%). + Note: `torch.randint` dynamic sampling prevents full CUDA graph replay; partial + re-recording still happens. Pre-generated static batch pool would fix this but + wasn't implemented (one-time training cost doesn't justify complexity). + +## Baseline Accuracy (from prior session, before this session's fixes) + +| Gate | Prior Best | Labels | +|------|-----------|--------| +| worker_type | 88.5% | implementer, reviewer, architect, qa | +| verification_type | 84.9% | unit, integration, schema, smoke, docs | +| prereq_op | 69.4% | standard, needs_review, needs_approval, full_gates | +| automatability | 69.6% | deterministic, template, specialist, slm, llm, human | + +Target: 98-99% (deploy_direct verdict) or 95%+ at ≤0.30 abstain rate (deploy_with_guardrails). + +## After the Sweep + +Check `specialists/eval/results/summary.json`. For any gate with verdict `scale_more`: +- Try full 15k steps: edit capacity_sweep.py `MAX_STEPS` or run the gate individually +- Or generate more training data — this is the higher-leverage move for low-accuracy gates + +Gates most likely to need more data: `prereq_op` (4-way, hard to separate) and `automatability` +(6-way, some classes are similar). Consider a data gen script that uses an LLM to generate +examples per label given a description. + +## Model Architecture (for reference) + +Small_plus tier: hidden=256, layers=2, heads=4 (~1.2M params) +Medium tier: hidden=384, layers=3, heads=6 (~3.0M params) + +Encoder-only transformer, mean-pool over non-padding positions, linear classification head. +Tokenizer: whitespace split, lowercase. Seq len: 96 tokens. +Checkpoint format: `checkpoint.pt` — contains model, optimizer, vocab, config, labels, step. + +Checkpoints live in: `/mnt/storage/fabricate_runs/whetstone__/checkpoint.pt` diff --git a/specialists/data/generated/automatability_binary_eval.tsv b/specialists/data/generated/automatability_binary_eval.tsv new file mode 100644 index 0000000..5af28f3 --- /dev/null +++ b/specialists/data/generated/automatability_binary_eval.tsv @@ -0,0 +1,46 @@ +1 0 name=LibrarySymbolAdvisor +1 0 name=SelfHostTranspileAudit +0 0 name=TrainingDataGenerator +1 0 name=IntentTranslator +1 0 name=TypeSystemTranslator +0 0 name=ContextBundleMinimizer +1 0 name=SecurityPreservingTranslation +1 0 name=IncrementalOptimizer +0 0 name=BufferOpsLifecycle +0 0 name=MigrationAcceptanceContract +1 0 name=ApprovalEscalationPlanner +0 0 name=Sprint163IntegrationSummary +1 0 name=ArchitectReviewInterface +1 0 name=MemoryModelTranslator +1 0 name=AgentCodeGen +0 0 name=Sprint238IntegrationSummary +0 0 name=CppJSBindingEmitter +1 0 name=OperationClassifier +0 0 name=Sprint36IntegrationSummary +0 0 name=SelfHostHarness +1 0 name=SymbolSelectorAPI +1 0 name=ArchitectReviewSurface +0 0 name=Sprint111IntegrationSummary +1 0 name=LibrarySymbolAdvisor +1 0 name=TaskitemLibraryAnnotator +0 0 name=SyntaxHighlighterElisp +0 0 name=SelfHostCodebaseAudit +1 0 name=PerTaskLibrarySelector +0 0 name=SyntaxHighlighterGo +1 0 name=AgentRegistry +1 0 name=OperationClassifier +0 0 name=WorkflowTemplates +1 0 name=AgentMutationApproval +1 0 name=AgentMutationPreview +0 0 name=RustPythonBindingEmitter +1 0 name=TransformEngine +1 0 name=OperationClassifier +0 0 name=CompilerPhaseSpec +1 0 name=ArchitectReviewSurface +1 0 name=OperationSelectorAPI +1 0 name=AgentMutationPreview +1 0 name=AgentMutationApproval +1 0 name=AgentTaskStatusOverlay +1 0 name=ScopeMilestoneDecomposer +1 0 name=SecurityExceptionReviewBoard +0 0 name=CompositionBuilder diff --git a/specialists/data/generated/automatability_binary_train.tsv b/specialists/data/generated/automatability_binary_train.tsv new file mode 100644 index 0000000..31d4bd5 --- /dev/null +++ b/specialists/data/generated/automatability_binary_train.tsv @@ -0,0 +1,266 @@ +1 0 name=AgentChatContextInjector +1 0 name=ApprovalEscalationPlanner +1 0 name=StrategyAwareOptimizer +0 0 name=SettingsManager +1 0 name=LibrarySymbolAdvisor +1 0 name=TransformEngineExtended +1 0 name=StrategyAwareOptimizer +1 0 name=SymbolSelectorAPI +1 0 name=ApprovalEscalationPlanner +0 0 name=PolyApiProject +1 0 name=MemoryModelTranslator +1 0 name=AgentChatSessionPersistence +1 0 name=AgentSessionRecorder +1 0 name=IncrementalOptimizer +1 0 name=TypeSystemTranslator +0 0 name=MigrationTestGenerator +0 0 name=SCIPEmitter +1 0 name=ArchitectReviewInterface +0 0 name=CapabilityDeclarationGenerator +0 0 name=Sprint190IntegrationSummary +1 0 name=SecurityExceptionReviewBoard +0 0 name=SecurityTestSkeletonGenerator +0 0 name=SchemaToCppGenerator +0 0 name=JobDispatchTableGenerator +0 0 name=Sprint85IntegrationSummary +1 0 name=SymbolSelectorAPI +0 0 name=ProjectAstCMakeGenerator +0 0 name=BatchMutationAPI +1 0 name=CanaryPromotionJudge +1 0 name=ArchitectReviewSurface +1 0 name=TypeAwareMappings +1 0 name=SecurityPreservingTranslation +1 0 name=TransformEngineExtended +1 0 name=ArchitectReviewSurface +0 0 name=Sprint222IntegrationSummary +1 0 name=ArchitectReviewInterface +1 0 name=ArchitectReviewSurface +1 0 name=PolyglotSuiteOrchestrator +0 0 name=Sprint265IntegrationSummary +0 0 name=SyntaxHighlighterOrg +1 0 name=TransformEngineExtended +1 0 name=TypeAwareMappings +1 0 name=AgentChatPanelModel +1 0 name=ArchitectModuleDecomposer +0 0 name=ContextSliceAssembler +0 0 name=SyntaxHighlighterJavaScript +1 0 name=TranspilationConfidence +1 0 name=ArchitectTechStackSelector +1 0 name=PerTaskLibrarySelector +1 0 name=ArchitectReviewInterface +1 0 name=SecurityExceptionReviewBoard +0 0 name=Sprint106IntegrationSummary +1 0 name=OperationClassifier +0 0 name=SQLiteDataLayerGenerator +1 0 name=SelfHostTranspileAudit +0 0 name=Sprint27IntegrationSummary +1 0 name=IncrementalOptimizer +1 0 name=AgentMutationApproval +0 0 name=Sprint98IntegrationSummary +1 0 name=OperationSelectorAPI +1 0 name=SelfHostTranspileAudit +1 0 name=OperationSelectorAPI +0 0 name=AccessibilityBaseline +1 0 name=AgentMutationApproval +0 0 name=TraceGenerator +0 0 name=CostEstimator +1 0 name=AgentPermissionPolicy +1 0 name=AgentMutationPreview +1 0 name=ConcurrencyTranslator +1 0 name=SelfHostTranspileAudit +1 0 name=ContextWindowOptimizer +1 0 name=AgentMutationPreview +1 0 name=IntentTranslator +1 0 name=AgentLibraryPolicy +0 0 name=SyntaxHighlighterPython +0 0 name=JobDispatchTableGenerator +0 0 name=EditorModePolicy +1 0 name=LibrarySymbolAdvisor +1 0 name=AgentMarketplace +1 0 name=TranspilationConfidence +0 0 name=SwarmStatusPanelModel +1 0 name=ApprovalEscalationPlanner +1 0 name=TransformEngine +0 0 name=SyntaxHighlighter +1 0 name=ArchitectIntakeProcessor +1 0 name=PolyglotOrchestrator +1 0 name=AgentTaskStatusOverlay +1 0 name=ArchitectReviewInterface +1 0 name=OperationSelectorAPI +1 0 name=LibrarySymbolAdvisor +0 0 name=Sprint184IntegrationSummary +1 0 name=SecurityExceptionReviewBoard +0 0 name=Sprint36aIntegration +1 0 name=ArchitectReviewInterface +0 0 name=SyntaxHighlighterPython +0 0 name=SearchUtils +0 0 name=Diagnostics +0 0 name=TestPlanGenerator +1 0 name=SecurityExceptionReviewBoard +0 0 name=InferenceJobGenerator +1 0 name=ArchitectProblemParser +0 0 name=LSPSymbolExtractor +1 0 name=SymbolSelectorAPI +1 0 name=ArchitectProjectPipeline +1 0 name=PerTaskLibrarySelector +1 0 name=AgentToolCallVisualization +0 0 name=ProjectSkeletonGenerator +1 0 name=TaskitemLibraryAnnotator +1 0 name=CanaryPromotionJudge +1 0 name=SecurityExceptionReviewBoard +1 0 name=OperationSelectorAPI +0 0 name=LanguageGeneratorPluginManifest +1 0 name=TransformEngine +1 0 name=OnCallCoveragePlanner +1 0 name=TaskitemLibraryAnnotator +1 0 name=TaskitemLibraryAnnotator +1 0 name=AgentMutationPreview +1 0 name=AgentChatSessionPersistence +0 0 name=CHeaderEmitter +0 0 name=Sprint49IntegrationSummary +1 0 name=ContextWindowOptimizer +1 0 name=PerTaskLibrarySelector +1 0 name=ArchitectMultiLanguageOrchestrator +0 0 name=MCPWorkflowPrompts +1 0 name=TranspilationRPC +1 0 name=ContextWindowOptimizer +1 0 name=OperationSelectorAPI +0 0 name=Sprint198IntegrationSummary +1 0 name=ArchitectScaffoldGenerator +0 0 name=GoCppBindingEmitter +0 0 name=AnimationUtils +0 0 name=SelfHostAnnotationAudit +1 0 name=ArchitectIntakeProcessor +1 0 name=AgentMarketplace +1 0 name=TranspilationConfidence +1 0 name=ArchitectReviewSurface +0 0 name=SecureByDefaultGenerator +1 0 name=OperationSelectorAPI +1 0 name=CanaryPromotionJudge +1 0 name=TaskitemLibraryAnnotator +1 0 name=AgentChatPanelModel +1 0 name=LibrarySymbolAdvisor +1 0 name=SecurityPreservingTranslation +0 0 name=APIBoundaryPreserver +1 0 name=ApprovalEscalationPlanner +1 0 name=TypeSystemTranslator +1 0 name=StrategyAwareOptimizer +1 0 name=TransformEngineExtended +0 0 name=SyntaxHighlighterRust +1 0 name=AgentSessionRecorder +0 0 name=Sprint136IntegrationSummary +0 0 name=OperationTaxonomy +1 0 name=ArchitectTechStackSelector +0 0 name=AnnotationConflict +1 0 name=CanaryPromotionJudge +1 0 name=ApprovalEscalationPlanner +0 0 name=Sprint286IntegrationSummary +0 0 name=MigrationPlanGenerator +0 0 name=Sprint261IntegrationSummary +0 0 name=TrainingDataGenerator +0 0 name=RustPythonBindingEmitter +1 0 name=AgentTaskSlots +1 0 name=AgentCodeGen +0 0 name=Sprint74IntegrationSummary +0 0 name=ArchitectTemplates +1 0 name=SymbolSelectorAPI +1 0 name=TaskitemLibraryAnnotator +1 0 name=ApprovalEscalationPlanner +1 0 name=CanaryPromotionJudge +1 0 name=AgentMutationApproval +0 0 name=SyntaxHighlighterJava +1 0 name=AgentMutationPreview +1 0 name=AgentMutationPreview +0 0 name=SQLiteDataLayerGenerator +1 0 name=AgentTaskSlots +1 0 name=AgentRPCHandler +1 0 name=MemoryModelTranslator +1 0 name=ConcurrencyTranslator +1 0 name=ArchitectReviewSurface +1 0 name=AgentToolCallVisualization +1 0 name=SecurityExceptionReviewBoard +0 0 name=RustGoBindingEmitter +0 0 name=Sprint58IntegrationSummary +1 0 name=IncrementalOptimizer +0 0 name=Sprint239IntegrationSummary +0 0 name=SyntaxHighlighterElisp +0 0 name=PlatformCMakeTargetGenerator +1 0 name=ApprovalEscalationPlanner +0 0 name=MqttBoilerplateGenerator +1 0 name=PerTaskLibrarySelector +1 0 name=IntentTranslator +1 0 name=SecurityExceptionReviewBoard +0 0 name=KeySymbolRenderer +1 0 name=OperationSelectorAPI +1 0 name=MemoryModelTranslator +1 0 name=ArchitectReviewInterface +0 0 name=ArchitectSkeletonGenerator +1 0 name=OperationClassifier +1 0 name=ApprovalEscalationPlanner +1 0 name=OperationClassifier +0 0 name=PackageScriptGenerator +1 0 name=TaskitemLibraryAnnotator +1 0 name=TypeAwareMappings +1 0 name=ArchitectReviewInterface +0 0 name=GoCppBindingEmitter +1 0 name=AgentMutationApproval +0 0 name=CHeaderEmitter +0 0 name=SyntaxHighlighterOrg +0 0 name=TaskitemGeneratorV2 +0 0 name=Sprint96IntegrationSummary +0 0 name=ScoreClampUtil +1 0 name=AgentRPCHandler +1 0 name=ArchitectScaffoldGenerator +1 0 name=ScopeMilestoneDecomposer +1 0 name=AgentMutationApproval +1 0 name=AgentMutationPreview +1 0 name=AgentPermissionPolicy +1 0 name=PerTaskLibrarySelector +0 0 name=ConstrainedRoutingRulesetExtension +0 0 name=LanguageGeneratorPluginManifest +1 0 name=ArchitectReviewSurface +0 0 name=WorkItem +1 0 name=TaskitemLibraryAnnotator +1 0 name=CanaryPromotionJudge +1 0 name=AgentMutationApproval +1 0 name=LibrarySymbolAdvisor +1 0 name=ConcurrencyTranslator +1 0 name=CanaryPromotionJudge +1 0 name=WorkflowOrchestrator +0 0 name=InferenceJobGenerator +1 0 name=TranspilationRPC +1 0 name=AgentLibraryPolicy +0 0 name=Sprint272IntegrationSummary +1 0 name=TransformEngine +1 0 name=TranspilationRPC +1 0 name=OnCallCoveragePlanner +1 0 name=SymbolSelectorAPI +1 0 name=PolyglotOrchestrator +1 0 name=OperationClassifier +0 0 name=RefactorActions +1 0 name=LibrarySymbolAdvisor +1 0 name=IntentTranslator +1 0 name=PerTaskLibrarySelector +1 0 name=WorkflowOrchestrator +1 0 name=AgentChatContextInjector +1 0 name=AgentAnnotationAssistant +1 0 name=AgentMutationApproval +1 0 name=SymbolSelectorAPI +1 0 name=ArchitectReviewInterface +0 0 name=SyntaxHighlighterCpp +1 0 name=ArchitectMultiLanguageOrchestrator +0 0 name=APIDocGenerator +1 0 name=ArchitectProblemParser +1 0 name=StrategyAwareOptimizer +1 0 name=AgentAnnotationAssistant +0 0 name=PackageScriptGenerator +1 0 name=AgentRegistry +1 0 name=ArchitectModuleDecomposer +0 0 name=Sprint249IntegrationSummary +1 0 name=SecurityPreservingTranslation +1 0 name=ArchitectProjectPipeline +0 0 name=WorkerDispatchValidator +1 0 name=ContextWindowOptimizer +1 0 name=PolyglotSuiteOrchestrator +1 0 name=AgentMutationPreview +1 0 name=TypeSystemTranslator diff --git a/specialists/data/generated/automatability_routed_eval.tsv b/specialists/data/generated/automatability_routed_eval.tsv new file mode 100644 index 0000000..77d002a --- /dev/null +++ b/specialists/data/generated/automatability_routed_eval.tsv @@ -0,0 +1,29 @@ +0 0 name=LibrarySymbolAdvisor +1 0 name=SelfHostTranspileAudit +1 0 name=IntentTranslator +1 0 name=TypeSystemTranslator +1 0 name=SecurityPreservingTranslation +1 0 name=IncrementalOptimizer +3 0 name=ApprovalEscalationPlanner +3 0 name=ArchitectReviewInterface +1 0 name=MemoryModelTranslator +2 0 name=AgentCodeGen +0 0 name=OperationClassifier +0 0 name=SymbolSelectorAPI +3 0 name=ArchitectReviewSurface +0 0 name=LibrarySymbolAdvisor +0 0 name=TaskitemLibraryAnnotator +0 0 name=PerTaskLibrarySelector +2 0 name=AgentRegistry +0 0 name=OperationClassifier +3 0 name=AgentMutationApproval +3 0 name=AgentMutationPreview +1 0 name=TransformEngine +0 0 name=OperationClassifier +3 0 name=ArchitectReviewSurface +0 0 name=OperationSelectorAPI +3 0 name=AgentMutationPreview +3 0 name=AgentMutationApproval +2 0 name=AgentTaskStatusOverlay +2 0 name=ScopeMilestoneDecomposer +3 0 name=SecurityExceptionReviewBoard diff --git a/specialists/data/generated/automatability_routed_train.tsv b/specialists/data/generated/automatability_routed_train.tsv new file mode 100644 index 0000000..cfed1fa --- /dev/null +++ b/specialists/data/generated/automatability_routed_train.tsv @@ -0,0 +1,179 @@ +2 0 name=AgentChatContextInjector +3 0 name=ApprovalEscalationPlanner +1 0 name=StrategyAwareOptimizer +0 0 name=LibrarySymbolAdvisor +1 0 name=TransformEngineExtended +1 0 name=StrategyAwareOptimizer +0 0 name=SymbolSelectorAPI +3 0 name=ApprovalEscalationPlanner +1 0 name=MemoryModelTranslator +2 0 name=AgentChatSessionPersistence +2 0 name=AgentSessionRecorder +1 0 name=IncrementalOptimizer +1 0 name=TypeSystemTranslator +3 0 name=ArchitectReviewInterface +3 0 name=SecurityExceptionReviewBoard +0 0 name=SymbolSelectorAPI +0 0 name=CanaryPromotionJudge +3 0 name=ArchitectReviewSurface +1 0 name=TypeAwareMappings +1 0 name=SecurityPreservingTranslation +1 0 name=TransformEngineExtended +3 0 name=ArchitectReviewSurface +3 0 name=ArchitectReviewInterface +3 0 name=ArchitectReviewSurface +2 0 name=PolyglotSuiteOrchestrator +1 0 name=TransformEngineExtended +1 0 name=TypeAwareMappings +2 0 name=AgentChatPanelModel +2 0 name=ArchitectModuleDecomposer +1 0 name=TranspilationConfidence +2 0 name=ArchitectTechStackSelector +0 0 name=PerTaskLibrarySelector +3 0 name=ArchitectReviewInterface +3 0 name=SecurityExceptionReviewBoard +0 0 name=OperationClassifier +1 0 name=SelfHostTranspileAudit +1 0 name=IncrementalOptimizer +3 0 name=AgentMutationApproval +0 0 name=OperationSelectorAPI +1 0 name=SelfHostTranspileAudit +0 0 name=OperationSelectorAPI +3 0 name=AgentMutationApproval +2 0 name=AgentPermissionPolicy +3 0 name=AgentMutationPreview +1 0 name=ConcurrencyTranslator +1 0 name=SelfHostTranspileAudit +1 0 name=ContextWindowOptimizer +3 0 name=AgentMutationPreview +1 0 name=IntentTranslator +2 0 name=AgentLibraryPolicy +0 0 name=LibrarySymbolAdvisor +2 0 name=AgentMarketplace +1 0 name=TranspilationConfidence +3 0 name=ApprovalEscalationPlanner +1 0 name=TransformEngine +2 0 name=ArchitectIntakeProcessor +2 0 name=PolyglotOrchestrator +2 0 name=AgentTaskStatusOverlay +3 0 name=ArchitectReviewInterface +0 0 name=OperationSelectorAPI +0 0 name=LibrarySymbolAdvisor +3 0 name=SecurityExceptionReviewBoard +3 0 name=ArchitectReviewInterface +3 0 name=SecurityExceptionReviewBoard +2 0 name=ArchitectProblemParser +0 0 name=SymbolSelectorAPI +2 0 name=ArchitectProjectPipeline +0 0 name=PerTaskLibrarySelector +2 0 name=AgentToolCallVisualization +0 0 name=TaskitemLibraryAnnotator +0 0 name=CanaryPromotionJudge +3 0 name=SecurityExceptionReviewBoard +0 0 name=OperationSelectorAPI +1 0 name=TransformEngine +2 0 name=OnCallCoveragePlanner +0 0 name=TaskitemLibraryAnnotator +0 0 name=TaskitemLibraryAnnotator +3 0 name=AgentMutationPreview +2 0 name=AgentChatSessionPersistence +1 0 name=ContextWindowOptimizer +0 0 name=PerTaskLibrarySelector +2 0 name=ArchitectMultiLanguageOrchestrator +1 0 name=TranspilationRPC +1 0 name=ContextWindowOptimizer +0 0 name=OperationSelectorAPI +2 0 name=ArchitectScaffoldGenerator +2 0 name=ArchitectIntakeProcessor +2 0 name=AgentMarketplace +1 0 name=TranspilationConfidence +3 0 name=ArchitectReviewSurface +0 0 name=OperationSelectorAPI +0 0 name=CanaryPromotionJudge +0 0 name=TaskitemLibraryAnnotator +2 0 name=AgentChatPanelModel +0 0 name=LibrarySymbolAdvisor +1 0 name=SecurityPreservingTranslation +3 0 name=ApprovalEscalationPlanner +1 0 name=TypeSystemTranslator +1 0 name=StrategyAwareOptimizer +1 0 name=TransformEngineExtended +2 0 name=AgentSessionRecorder +2 0 name=ArchitectTechStackSelector +0 0 name=CanaryPromotionJudge +3 0 name=ApprovalEscalationPlanner +2 0 name=AgentTaskSlots +2 0 name=AgentCodeGen +0 0 name=SymbolSelectorAPI +0 0 name=TaskitemLibraryAnnotator +3 0 name=ApprovalEscalationPlanner +0 0 name=CanaryPromotionJudge +3 0 name=AgentMutationApproval +3 0 name=AgentMutationPreview +3 0 name=AgentMutationPreview +2 0 name=AgentTaskSlots +2 0 name=AgentRPCHandler +1 0 name=MemoryModelTranslator +1 0 name=ConcurrencyTranslator +3 0 name=ArchitectReviewSurface +2 0 name=AgentToolCallVisualization +3 0 name=SecurityExceptionReviewBoard +1 0 name=IncrementalOptimizer +3 0 name=ApprovalEscalationPlanner +0 0 name=PerTaskLibrarySelector +1 0 name=IntentTranslator +3 0 name=SecurityExceptionReviewBoard +0 0 name=OperationSelectorAPI +1 0 name=MemoryModelTranslator +3 0 name=ArchitectReviewInterface +0 0 name=OperationClassifier +3 0 name=ApprovalEscalationPlanner +0 0 name=OperationClassifier +0 0 name=TaskitemLibraryAnnotator +1 0 name=TypeAwareMappings +3 0 name=ArchitectReviewInterface +3 0 name=AgentMutationApproval +2 0 name=AgentRPCHandler +2 0 name=ArchitectScaffoldGenerator +2 0 name=ScopeMilestoneDecomposer +3 0 name=AgentMutationApproval +3 0 name=AgentMutationPreview +2 0 name=AgentPermissionPolicy +0 0 name=PerTaskLibrarySelector +3 0 name=ArchitectReviewSurface +0 0 name=TaskitemLibraryAnnotator +0 0 name=CanaryPromotionJudge +3 0 name=AgentMutationApproval +0 0 name=LibrarySymbolAdvisor +1 0 name=ConcurrencyTranslator +0 0 name=CanaryPromotionJudge +2 0 name=WorkflowOrchestrator +1 0 name=TranspilationRPC +2 0 name=AgentLibraryPolicy +1 0 name=TransformEngine +1 0 name=TranspilationRPC +2 0 name=OnCallCoveragePlanner +0 0 name=SymbolSelectorAPI +2 0 name=PolyglotOrchestrator +0 0 name=OperationClassifier +0 0 name=LibrarySymbolAdvisor +1 0 name=IntentTranslator +0 0 name=PerTaskLibrarySelector +2 0 name=WorkflowOrchestrator +2 0 name=AgentChatContextInjector +2 0 name=AgentAnnotationAssistant +3 0 name=AgentMutationApproval +0 0 name=SymbolSelectorAPI +3 0 name=ArchitectReviewInterface +2 0 name=ArchitectMultiLanguageOrchestrator +2 0 name=ArchitectProblemParser +1 0 name=StrategyAwareOptimizer +2 0 name=AgentAnnotationAssistant +2 0 name=AgentRegistry +2 0 name=ArchitectModuleDecomposer +1 0 name=SecurityPreservingTranslation +2 0 name=ArchitectProjectPipeline +1 0 name=ContextWindowOptimizer +2 0 name=PolyglotSuiteOrchestrator +3 0 name=AgentMutationPreview +1 0 name=TypeSystemTranslator diff --git a/specialists/data/generated/worker_type_combined_eval.tsv b/specialists/data/generated/worker_type_combined_eval.tsv new file mode 100644 index 0000000..ca732b3 --- /dev/null +++ b/specialists/data/generated/worker_type_combined_eval.tsv @@ -0,0 +1,115 @@ +0 0 Snapshot persistence (sidecar integration) +1 0 `whetstone_preview_regeneration_diff_generic` MCP tool +2 0 Phase 25a Integration +3 0 Sprint 116 integration summary + regression +0 0 Debug assist packet bundle model +1 0 `whetstone_score_failure_triage` MCP tool +3 0 Regression watchlist for promoted pairs +2 0 Architecture Templates +3 0 Phase 10d Integration Tests +2 0 Phase 21a Integration +2 0 Result Acceptance Protocol +2 0 Guided transpilation exercise framework +0 0 Patch proposal schema and provenance model +2 0 `whetstone_publish_roadmap_epoch` MCP tool +0 0 Compliance policy binding engine +2 0 Workflow Session Protocol +3 0 Sprint 181 Integration Summary +2 0 Language-specific method body scaffolding for core patterns +0 0 Sprint 27 Integration + Summary +3 0 Phase 15b Integration — Full Protocol Test +0 0 Long-range edit graph planner +3 0 Sprint 120 integration summary + regression +0 0 Federated governance report generator +3 0 Sprint 66 integration summary + regression +3 0 Sprint 204 Integration Summary +2 0 Phase 27a Integration +1 0 Review Gates +3 0 Sprint 55 integration summary + regression +0 0 LTS baseline criteria schema +0 0 Replay comparison and divergence classification runtime model +3 0 Sprint 101 integration summary + regression +2 0 Retry/Escalation Protocol for Constraint Failures +0 0 Compatibility policy profiles by interface criticality +0 0 Language support tiers + gates +2 0 Phase 13d Integration + Sprint 13 Summary +1 0 Deterministic triage queue planner model +1 0 Unmappable semantic blocklist + mandatory review gate +2 0 Phase 23a Integration +1 0 `whetstone_preview_text_ast_merge_generic` MCP tool +3 0 Sprint 132 integration summary + regression +2 0 Phase 27a Integration +3 0 Sprint 71 integration summary + regression +2 0 Phase 14b Integration +0 0 Epoch-2 workstream schema and ownership map +3 0 Sprint 141 integration summary + regression +3 0 AST-native projection benchmark suite +0 0 Certification evidence archive format v2 +2 0 Phase 29a Integration +0 0 Tier promotion engine (beta->stable) +1 0 Quarterly lock/review cadence policy +1 0 Diagnostic normalization runtime implementation model +2 0 Phase 22a Integration +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Plan Build a streaming join worker correlating click and | Build a streaming join worker correlating click and conversion events with watermark logic. +0 0 Define verification and readiness surface | Build a live notifications hub with subscription filters, fanout, and reconnect replay support. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build a secrets rotation scheduler with lease tracking, retry plans, and notifications. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +0 0 Define verification and readiness surface | Build kanban API with boards, columns, cards, drag reorder, and activity history. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Plan Build an IoT telemetry gateway supporting MQTT ingestion | Build an IoT telemetry gateway supporting MQTT ingestion, validation, and stream forwarding. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a CLI todo manager with add/list/done/delete commands and JSON file storage. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a whiteboard sync backend with operational ordering and conflict-safe patch application. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +0 0 Define verification and readiness surface | Build an image conversion CLI with resize, format conversion, and batch mode support. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Plan Split monolithic endpoint into read/write paths while preserving | Split monolithic endpoint into read/write paths while preserving latency and memory budgets and updating observability d +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +3 0 Define verification and readiness surface | Build portfolio tracker with holdings, transactions, PnL snapshots, and benchmark comparison. +0 0 Define verification and readiness surface | Build checkout backend coordinating address, shipping options, totals, and payment intent creation. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +0 0 Define verification and readiness surface | Build chat bot backend with slash commands, permission checks, and scheduled digests. +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build a contacts service with import/export, deduplication, and fuzzy name search. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Define verification and readiness surface | Build realtime polling backend with vote dedupe, room join, and result broadcast. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +0 0 Plan Upgrade API contract from v1 to v2 backend | Upgrade API contract from v1 to v2: backend handlers, OpenAPI schema, generated client SDK, frontend integration, and co +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +0 0 Plan Build a streaming detector with keyed state windows | Build a streaming detector with keyed state windows, out-of-order event handling, watermark policy, and deterministic re +0 0 Define verification and readiness surface | Build a release notes generator from commits, PR labels, and issue references. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +0 0 Define verification and readiness surface | Build a change-data-capture processor that applies upserts/deletes and checkpoints offsets. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Define verification and readiness surface | Build a model registry API supporting stages, approvals, rollback, and signed artifact links. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua diff --git a/specialists/data/generated/worker_type_combined_train.tsv b/specialists/data/generated/worker_type_combined_train.tsv new file mode 100644 index 0000000..6e61f58 --- /dev/null +++ b/specialists/data/generated/worker_type_combined_train.tsv @@ -0,0 +1,661 @@ +1 0 `whetstone_get_sync_diagnostics` MCP tool +3 0 Regression fixtures for under-constrained vs capability-limited cases +3 0 Sprint 83 integration summary + regression +1 0 Audit trail and inspection +3 0 Sprint 99 integration summary + regression +3 0 Security Testing Skeleton Generation +2 0 Phase 25b Integration +3 0 Sprint 115 integration summary + regression +0 0 Feedback Loop — Rejection Re-Routing +2 0 Long-horizon roadmap planning engine v2 +2 0 Retry/Escalation Protocol for Constraint Failures +0 0 `BuildErrorRecord` schema +2 0 Phase 26a Integration +3 0 Sprint 179 Integration Summary +0 0 Policy-compliance auto-checkpoint integration +2 0 Refactor grammar generator to use normalized schema IR +3 0 Intake-to-Queue Simulation Harness +3 0 Sprint 50 integration summary + regression +1 0 Human review queue generator from ambiguities +2 0 Scaffold File Generation +3 0 Sprint 85 integration summary + regression +0 0 `whetstone_get_swarm_maintenance_status` MCP tool +3 0 Sprint 157 integration summary + regression +3 0 Full historical regression + post-release checklist +1 0 `whetstone_normalize_diagnostics` runtime implementation +3 0 End-to-end — Python PriorityQueue → C++, Rust, Go +2 0 Phase 12d Integration + Sprint 12 Summary +1 0 False-green diagnostic class +3 0 Sprint 86 integration summary + regression +3 0 Sprint 64 integration summary + regression +3 0 Sprint 142 integration summary + regression +1 0 Priority-focused review queue shaping engine +1 0 Efficiency diagnostics and trend report +0 0 Context slice canonicalization for handoffs +0 0 Context Bundle Format +3 0 Reference implementation harness for IR/evidence specs +1 0 Constraint completeness diagnostics +1 0 Porting review board model +0 0 Maintenance demand estimator +0 0 `whetstone_get_program_kpis` MCP tool +1 0 Tool-surface diagnostics report artifact +3 0 Rust runner adapter for harness +2 0 `whetstone_verify_architecture_consistency` MCP tool +1 0 Security scan orchestrator for generated targets +1 0 Strictness audit utility and baseline report +0 0 Sprint pipeline summary enrichment +0 0 C Parser — Functions, Structs, Enums +1 0 Call Stack and Frame Inspector +3 0 Sprint 74 integration summary + regression +2 0 Phase 25b Integration +0 0 Binary/interface diff engine integration +3 0 Sprint 125 integration summary + regression +1 0 Reviewer quorum policy model for critical approvals +1 0 Conflict preview and impact estimator model +3 0 Test failure classifier +3 0 Benchmark ingestion support for projection schema +0 0 Artifact quality scoring and ranking engine +3 0 Sprint 47 integration summary + regression +0 0 `whetstone_assemble_context` MCP tool +0 0 Legacy modernization dossier generator +0 0 Step Engine +1 0 Agent role contract schema (`analyzer`, `patcher`, `verifier`, `reviewer`) +0 0 `whetstone_trigger_porting_incident_drill` MCP tool +1 0 Lint diagnostics normalization +1 0 AST mutation preview +2 0 Burn-down planning optimizer +1 0 KPI delta analysis engine for block review +2 0 Long-horizon roadmap planning engine v2 +0 0 `whetstone_rollback_constructive_transaction` MCP tool +2 0 Deterministic blocker taxonomy +1 0 Docking Reliability Audit + Deterministic Layout Rules +0 0 `whetstone_run_build_iteration` MCP tool +1 0 Divergence triage model for interop failures +2 0 Phase 36a Integration +0 0 Language capability conformance report artifact +3 0 `whetstone_run_pair_benchmark` MCP tool +1 0 `whetstone_optimize_review_queue` MCP tool +3 0 Sprint 205 Integration Summary +1 0 Loop summary contract diagnostics +2 0 Phase 13a Integration — Layout Persistence +3 0 Sprint 189 Integration Summary +2 0 Continuity drill framework and scoring +2 0 Deterministic blocker taxonomy extension +3 0 Cross-team benchmark comparator +0 0 VB.NET lowering/raising adapters +1 0 Review-Gate Policy Refinement +2 0 Phase 14d Integration + Sprint 14 Summary +0 0 Multi-language compile gate executor +1 0 Dependency and supply-chain audit packet +1 0 Reviewer-focused semantic diff summarizer +1 0 Architect Review Surface for Intake +2 0 Phase 24a Integration +2 0 Long-range blocker taxonomy +2 0 Result Acceptance Protocol +1 0 Patch proposal review report artifact +1 0 Parity diagnostics export +3 0 Debug benchmark harness for fixture suites +1 0 Triage packet bundle model +1 0 Incremental sync diagnostics model +2 0 Swarm maintenance job decomposition model +0 0 Epoch kickoff report artifact +3 0 Sprint 96 integration summary + regression +2 0 Phase 32a Integration +2 0 Self-Hosting Progress Report + Phase Integration +2 0 Phase 13a Integration — Layout Persistence +0 0 Sprint 28 Integration + Summary +2 0 Migration Plan Generator +2 0 Phase 20a Integration +3 0 End-to-end replay + rollback scenario suite +3 0 C++ runner adapter for harness +1 0 Safety Audit via Annotations +2 0 Phase 33a Integration +3 0 Sprint 79 integration summary + regression +0 0 Lisp lowering/raising adapters +0 0 `whetstone_run_porting_gates` MCP tool +1 0 Explainability compression model for rapid review +2 0 Phase 32a Integration +3 0 Sprint 51 integration summary + regression +3 0 Sprint 175 Integration Summary +3 0 Sprint 146 integration summary + regression +2 0 Deterministic blocker taxonomy extension +3 0 Sprint 139 integration summary + regression +3 0 Sprint 90 integration summary + full-program regression +3 0 Benchmark publication bundle +2 0 Transpilation failure taxonomy schema +3 0 Sprint 138 integration summary + regression +1 0 Architect Review Interface +1 0 Real adapter diagnostics in status tool +0 0 Kotlin Parser +1 0 Remediation router from gate diagnostics to tool actions +0 0 Family promotion checks (experimental->beta) +2 0 Phase 16a Integration +3 0 Sprint 73 integration summary + regression +2 0 Phase 13d Integration + Sprint 13 Summary +0 0 AnnotationValidator — Subject 5-8 Rules +0 0 Deterministic replay pack composer +1 0 Assurance review dossier template +3 0 Auto-tuning proposal generator from benchmark deltas +1 0 `whetstone_get_distributed_triage_queue` MCP tool +1 0 KPI anomaly triage model +2 0 Phase 36a Integration +2 0 Module Decomposition Engine +1 0 Scope ambiguity diagnostics +2 0 Phase 38a Integration +1 0 Governance and audit report bundle +2 0 Phase 13b Integration — Visual Consistency +3 0 CI integration in build/test flow +0 0 Zero-trust policy model for execution surfaces +1 0 Patch execution audit bundle +2 0 Phase 30a Integration +1 0 Drift audit report artifact +0 0 `whetstone_marketplace_search` MCP tool +3 0 Cross-language class emission parity tests +3 0 Test output parser — gtest format +0 0 `whetstone_get_debug_campaign_status` MCP tool +1 0 Dynamic family acceptance report + review queue wiring +3 0 Sprint 191 Integration Summary +2 0 Generator Updates + Phase 11c Integration +1 0 Review effort estimator from ambiguity/risk packets +3 0 Sprint 152 integration summary + regression +2 0 Candidate generation engine from failure taxonomy +0 0 Ambiguity packet model (`assumed`, `unknown`, `conflict`) +3 0 Sprint 198 Integration Summary +1 0 Memory Inspector UI Model +0 0 Argument/result canonicalizer model +3 0 Sprint 89 integration summary + regression +1 0 Divergence classification and triage model +0 0 `whetstone_run_constructive_loop` MCP tool +1 0 Dependency Audit Annotations +3 0 Benchmark runner and metrics export +3 0 Sprint 62 integration summary + regression +1 0 Review Comparison View +1 0 Review SLA policy integration +0 0 Failure injection and safety validation model +1 0 Intent drift diagnostics export +3 0 Sprint 186 Integration Summary +0 0 Auto Type Deduction +2 0 Scope and Milestone Decomposer +2 0 Phase 25a Integration +1 0 Constraint Violation Diagnostics +2 0 Refactor pattern schema and metadata model +1 0 `whetstone_normalize_diagnostics` MCP tool +0 0 Replay/transaction reliability report artifact +1 0 Reviewer effectiveness report artifact +1 0 Post-incident continuity audit report model +1 0 Entropy scanner — editor-side +2 0 Phase 13c Integration — Keyboard Shortcuts Help Panel +3 0 Test gate runner and minimal harness injection +2 0 Backend framework adapter hooks +0 0 Enhanced evidence requirements engine +1 0 `whetstone_preview_regenerated_diff` MCP tool +1 0 Hint safety guardrails and audit fields +2 0 Refactor intent annotation model +3 0 Sprint 137 integration summary + regression +1 0 Reporting templates for C++ review teams +0 0 Generic/monomorphization intent model +0 0 `whetstone_compare_mcp_replays` MCP tool +0 0 Memory Model Translation +0 0 Schema evolution compatibility checker +0 0 Closeout consistency gate +0 0 Real-time constraint model (`deadline`, `jitter`, `period`) +2 0 Workflow Session Protocol +0 0 Include/import ordering normalization model +2 0 Deprecation/succession lifecycle framework +0 0 Trace Timeline Model +1 0 Decision consistency analyzer across reviewers +1 0 Profile compliance auditor integration +0 0 `whetstone_plan_rollout_stage` MCP tool +0 0 Top-level strictness gate +0 0 Strict queue gate for execution contracts +3 0 Sprint 58 integration summary + regression +3 0 Sprint 48 integration summary + regression +0 0 `whetstone_detect_text_ast_conflicts_generic` MCP tool +2 0 Phase 19a Integration +1 0 `whetstone_get_review_load_status` MCP tool +3 0 Phase 19b Integration + Sprint Summary +3 0 Sprint 200 Integration Summary +0 0 Async/await state intent lowering +1 0 Failure triage score model +0 0 Plan fingerprinting and replay checks +1 0 MCP replay diagnostics artifact +0 0 Plugin compatibility resolver (core/API/version) +1 0 Diagnostic normalization canonical model +3 0 Test-to-source locator +0 0 Debug action budget model +2 0 Scaffold File Generation +1 0 Canonical diagnostic normalization from provider/LSP output +2 0 Backend framework adapter hooks +3 0 Sprint 162 Integration Summary +2 0 Phase 39a Integration +0 0 ADT + pattern matching canonical lowering +2 0 Verification gate integration for architecture artifacts +0 0 Modernization RPC + MCP +2 0 Phase 19a Integration +3 0 Convention Extractor + Validator (tests: unit, negative, regression) +1 0 `whetstone_preview_text_ast_merge` MCP tool +0 0 ABI gate bindings to tier promotions +2 0 Architecture claim linkage engine +0 0 Permission conformance report artifact +2 0 Phase 20a Integration +2 0 Governance model refresh framework +1 0 Sync diagnostics normalization model +3 0 Sprint 63 integration summary + regression +0 0 `whetstone_suggest_build_fix` MCP tool +3 0 A/B + benchmark validation artifact update +0 0 `whetstone_plan_tool_migrations` MCP tool +3 0 Full regression + architecture gate +0 0 C interop adapter deepening +3 0 Phase 10e Integration Tests +2 0 Phase 29a Integration +2 0 Scope and Milestone Decomposer +0 0 Multi-check gate orchestrator model +1 0 Lint executor integration +3 0 Sprint 118 integration summary + regression +1 0 Dropped-content diagnostics packet +0 0 Execution trace and recovery packet export +2 0 Phase 18a Integration +3 0 Sprint 45 Integration Summary +2 0 Deterministic blocker taxonomy +2 0 Refactor pattern schema and metadata model +1 0 Reviewer decision ledger integration +2 0 Phase 28a Integration +2 0 Phase 37a Integration +3 0 Sprint 114 integration summary + regression +2 0 `whetstone_publish_roadmap_epoch` MCP tool +1 0 `whetstone_review_porting_decision` MCP tool +2 0 Language-specific method body scaffolding for core patterns +1 0 Audit trail completeness checker +1 0 C++ include fixer from gate diagnostics +3 0 End-to-end capability inventory and gap scan +1 0 Static registry scanner for `register*Tools` blocks +2 0 Phase 31a Integration +1 0 `whetstone_rank_failure_triage_actions` MCP tool +3 0 Timing regression runner integration +2 0 Phase 40a Integration +1 0 Reviewer load and fatigue model +2 0 Swarm maintenance job decomposition model +2 0 Target projection blocker taxonomy +0 0 Annotate Whetstone's Own Code +2 0 Phase 15c Integration + Sprint 15 Summary +0 0 Epoch block health re-baseline model +3 0 Sprint 159 integration summary + regression +1 0 Human Review Interface via MCP +0 0 Transpile Whetstone Modules +0 0 Capability matrix model +2 0 Phase 14d Integration + Sprint 14 Summary +3 0 Prior-Step Context Injector (tests: unit, integration) +0 0 Unsafe-practice detection and quarantine +3 0 Sprint 127 integration summary + regression +1 0 `TaskitemQualityAuditor` — batch report +3 0 Systemic regression detector +0 0 Debug campaign spec model +3 0 `whetstone_get_execution_attestation` MCP tool +3 0 Sprint 72 integration summary + regression +2 0 Self-Hosting Progress Report + Phase Integration +0 0 C# Parser +1 0 Ambiguity triage UI model and API +0 0 F# Parser +0 0 Result-to-ObservationRecord mapper +0 0 Self-service escalation handoff hooks +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build job queue service with delayed jobs, retries, visibility timeout, and metrics. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Define verification and readiness surface | Build a CLI password vault with encrypted local storage and master-key unlock. +0 0 Define verification and readiness surface | Build order management with status machine, cancellation rules, and shipment events. +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build a registry mirror service with digest verification, cache pruning, and pull-through. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +0 0 Define verification and readiness surface | Build a websocket chat server with rooms, presence, typing indicators, and message history. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +0 0 Define verification and readiness surface | Build subscription billing engine with proration, trials, coupons, and failed-payment retries. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +0 0 Define verification and readiness surface | Build invoice generator with tax calculation, due schedules, and PDF export metadata. +0 0 Define verification and readiness surface | Build a webhook dispatcher with retries, signature verification, dead-letter queue, and backoff. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Plan Build a ticker stream service with topic subscriptions | Build a ticker stream service with topic subscriptions and throttled update delivery. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a CLI log tailer with multi-file follow, regex filters, and output highlighting. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a CSV analyzer CLI with schema inference, summary stats, and filter expressions. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build returns API with RMA creation, restock policy, and refund eligibility checks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a service monitor that runs health checks, tracks SLO windows, and pages on breaches. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build expense backend with receipt OCR placeholders, category budgets, and recurring expenses. +0 0 Define verification and readiness surface | Build a harness that transpiles class-heavy source modules across multiple target languages and validates class/method p +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +0 0 Define verification and readiness surface | Build matchmaking service with queue buckets, skill-based pairing, and timeout fallback. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +0 0 Plan Build an incremental snapshot builder that compacts event | Build an incremental snapshot builder that compacts event logs into query-optimized state. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +0 0 Define verification and readiness surface | Build a REST Todo API with CRUD, pagination, filtering by status, and SQLite persistence. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +0 0 Plan Add normalized address schema migrate legacy rows with | Add normalized address schema, migrate legacy rows with backfill job, dual-read/write transition, and rollback script. +0 0 Define verification and readiness surface | Build a log enrichment batch job joining geo data and user agents with partitioned output. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +0 0 Define verification and readiness surface | Build a URL shortener API with custom aliases, redirect tracking, and expiration support. +0 0 Define verification and readiness surface | Build meeting notes API with templates, attendee action items, and follow-up reminders. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build cart service with item merge, discount application, and inventory reservation integration. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build session store service with token revocation, idle timeout, and concurrent login policy. +0 0 Define verification and readiness surface | Build a secret rotation controller with staged rollout, canary verification, rollback triggers, and explicit blast-radiu +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Plan shared sync core plus Android/Desktop/Web projections with platform-specific | Implement shared sync core plus Android/Desktop/Web projections with platform-specific persistence and transport adapter +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Define verification and readiness surface | Build LLM guardrail service for policy checks, PII filtering, and response classification. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a multi-tenant limiter with token bucket + sliding window hybrid policy, burst fairness guarantees, and lock-free +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a Kubernetes manifest linter enforcing probes, limits, and namespace policy. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +0 0 Define verification and readiness surface | Build an API for file metadata indexing, search by attributes, and soft delete recovery. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build experiment tracking backend storing params, metrics, artifacts, and compare queries. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build user profile service with avatars, privacy settings, and profile completeness scoring. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +0 0 Define verification and readiness surface | Build batch inference pipeline reading partitions, loading model, and writing scored outputs. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build a JWT auth service with register, login, refresh token rotation, and password hashing. +0 0 Define verification and readiness surface | Build notification preferences API with channel rules, quiet hours, and topic subscriptions. +0 0 Plan Build a CI orchestrator service triggering jobs tracking | Build a CI orchestrator service triggering jobs, tracking states, and emitting summaries. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build a harness that transpiles class-heavy source modules across multiple target languages and validates class/method p +0 0 Plan telemetry agent for linux arduino targets using shared | Implement telemetry agent for linux + arduino targets using shared serialization core and target-specific runtime wrappe +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build prompt evaluation runner with rubric scoring, aggregate metrics, and reproducible reports. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a feature store service with online/offline reads and point-in-time correctness. +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build wiki backend with page versions, backlinks, and markdown rendering pipeline hooks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build drift detection worker comparing live feature distributions against reference baselines. +0 0 Define verification and readiness surface | Build form builder backend with schema validation, submissions, and analytics counters. +0 0 Define verification and readiness surface | Build email campaign manager with lists, segmentation, scheduling, and bounce tracking. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Define verification and readiness surface | Build an access gateway that evaluates hierarchical allow/deny policies with shadow rules, policy versioning, and explai +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a presence tracker with heartbeat timeout, status transitions, and shard-safe updates. +0 0 Define verification and readiness surface | Build payments API with idempotency keys, ledger-safe transfers, and webhook callbacks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Plan Build a task runner CLI that reads YAML | Build a task runner CLI that reads YAML tasks and executes dependency-aware command graphs. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build configuration service with versioned configs, staged rollout, and validation checks. +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +0 0 Define verification and readiness surface | Build warehouse picking service with wave assignment, pick confirmation, and shortage handling. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build shipping label worker integrating carrier rates, label generation, and tracking updates. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a metrics aggregation worker with tumbling windows, late data handling, and rollups. +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build backup retention manager with policy tiers, legal hold support, and deletion safety checks. +0 0 Plan Build reconciliation service matching ledger entries to bank | Build reconciliation service matching ledger entries to bank statements with mismatch reporting. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build an ETL pipeline that ingests orders CSV, validates schema, and loads curated tables. +0 0 Define verification and readiness surface | Build search suggest API with prefix index, typo tolerance, and popularity boosting. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build a feature flag API with environments, targeting rules, and rollout percentages. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +0 0 Plan Replace search ranking pipeline behind feature flag with | Replace search ranking pipeline behind feature flag with staged rollout, telemetry checks, and automatic abort trigger. +0 0 Define verification and readiness surface | Build a training scheduler with queue priorities, resource quotas, and retry policies. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +0 0 Define verification and readiness surface | Build a gateway that validates incoming payloads against versioned contracts, performs semantic constraint checks, and e +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a backup CLI with include/exclude rules, incremental snapshots, and restore command. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +0 0 Define verification and readiness surface | Build pricing engine with currency handling, promotions, and time-bound campaign rules. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +0 0 Define verification and readiness surface | Build an event deduplicator using idempotency keys, time windows, and replay-safe writes. +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Define verification and readiness surface | Build dataset version management with lineage metadata, immutable snapshots, and diff operations. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +0 0 Define verification and readiness surface | Build tax report backend aggregating gains/losses and generating filing-year summaries. +0 0 Define verification and readiness surface | Build distributed rate limiter with fixed/sliding window modes and policy APIs. +0 0 Define verification and readiness surface | Build telemetry collector with trace/span ingestion, sampling, and export adapters. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | Build a CRDT-based collaborative notes backend with causal metadata, merge convergence proofs, tombstone compaction, and +0 0 Define verification and readiness surface | Build a loader from object storage to warehouse with schema evolution and retry checkpoints. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +0 0 Define verification and readiness surface | Build access control service with RBAC roles, permissions, and resource-scoped grants. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +3 0 Define verification and readiness surface | Refactor billing pipeline to introduce idempotency keys and rollback checkpoints across ledger service, queue worker, re +0 0 Define verification and readiness surface | Build a CDC replication worker that handles schema evolution, backfill merges, checkpoint consistency, and deterministic +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +0 0 Plan Build drift detection comparing desired and observed infra | Build drift detection comparing desired and observed infra states with actionable diff reports. +1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and +0 0 Define verification and readiness surface | Build product catalog API with variants, facets, search indexing hooks, and soft delete. +0 0 Define verification and readiness surface | Build a realtime log viewer backend with tail sessions, regex filters, and backpressure. +1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies. +1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics. +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a realtime cursor service broadcasting positions with rate limiting and stale cleanup. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +0 0 Define verification and readiness surface | Build a text diff CLI with unified output, word-level mode, and ignore-whitespace option. +0 0 Define verification and readiness surface | Build calendar backend with events, recurrence rules, reminders, and timezone normalization. +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build inventory endpoints with stock reservation, release, and low-stock alerts. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +0 0 Define verification and readiness surface | Build an inference API with model loading, request validation, and batch prediction endpoint. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +0 0 Define verification and readiness surface | Build recommendation API combining collaborative and rule-based fallback candidate generation. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build an order saga orchestrator with compensating actions, timeout escalation, exactly-once message handling, and deter +2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua +2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks. +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build a .env manager CLI with profile switch, validation, and secure redaction in output. +1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails. +0 0 Define verification and readiness surface | Build a CMS backend with posts, tags, draft/publish workflow, and role-based access control. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +3 0 Define verification and readiness surface | All core features implemented Tests passing +1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints. +0 0 Define verification and readiness surface | Build a log shipping agent with offset persistence, multiline parsing, and retries. +0 0 Define verification and readiness surface | Build a data quality checker with null, range, uniqueness, and referential integrity rules. +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +3 0 Define verification and readiness surface | All core features implemented Tests passing +0 0 Define verification and readiness surface | Build payout scheduler with settlement windows, minimum thresholds, and retry-safe processing. +3 0 Define verification and readiness surface | All core features implemented Tests passing +2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues. +1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics +2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features. +3 0 Define verification and readiness surface | Build a CRDT-based collaborative notes backend with causal metadata, merge convergence proofs, tombstone compaction, and +0 0 Define verification and readiness surface | Build a notes sync API with optimistic concurrency, conflict detection, and change feed endpoints. +0 0 Define verification and readiness surface | Build OCR queue processor with job retries, dead-letter handling, and extraction status API. +0 0 Define verification and readiness surface | Build shared core and projections but omit projection contract fields intentionally. +1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging. +3 0 Define verification and readiness surface | All core features implemented Tests passing diff --git a/specialists/eval/results/automatability_decompose.json b/specialists/eval/results/automatability_decompose.json new file mode 100644 index 0000000..434beaa --- /dev/null +++ b/specialists/eval/results/automatability_decompose.json @@ -0,0 +1,165 @@ +{ + "gate": "automatability", + "flat_6way_baseline": { + "small_plus": 69.6, + "medium": 78.3 + }, + "decomposed": { + "stage1_binary": { + "overall_accuracy": 100.0, + "n_eval": 46, + "per_class": { + "low_entropy": { + "acc": 100.0, + "n": 17 + }, + "needs_routing": { + "acc": 100.0, + "n": 29 + } + }, + "confusion_summary": [], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.6, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.7, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.8, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.9, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + } + ], + "guardrail_threshold": 0.5, + "guardrail_abstain_rate": 0.0, + "eval_latency_ms_per_example": 5.294, + "model_config": { + "hidden_dim": 256, + "layers": 2, + "heads": 4, + "n_labels": 2, + "seq_len": 96, + "vocab_size": 133 + }, + "verdict": "deploy_direct" + }, + "stage2_4way": { + "overall_accuracy": 100.0, + "n_eval": 29, + "per_class": { + "specialist": { + "acc": 100.0, + "n": 9 + }, + "slm": { + "acc": 100.0, + "n": 7 + }, + "llm": { + "acc": 100.0, + "n": 4 + }, + "human": { + "acc": 100.0, + "n": 9 + } + }, + "confusion_summary": [], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 29, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.6, + "accepted": 29, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.7, + "accepted": 29, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.8, + "accepted": 29, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + }, + { + "threshold": 0.9, + "accepted": 29, + "abstain_rate": 0.0, + "accepted_accuracy": 100.0 + } + ], + "guardrail_threshold": 0.5, + "guardrail_abstain_rate": 0.0, + "eval_latency_ms_per_example": 8.796, + "model_config": { + "hidden_dim": 256, + "layers": 2, + "heads": 4, + "n_labels": 4, + "seq_len": 96, + "vocab_size": 55 + }, + "verdict": "deploy_direct" + }, + "combined": { + "method": "2stage_decomposed", + "routing_accuracy": 100.0, + "accuracy_5way": 100.0, + "n_eval": 46, + "per_class_5way": { + "low_entropy": { + "acc": 100.0, + "n": 17 + }, + "specialist": { + "acc": 100.0, + "n": 9 + }, + "slm": { + "acc": 100.0, + "n": 7 + }, + "llm": { + "acc": 100.0, + "n": 4 + }, + "human": { + "acc": 100.0, + "n": 9 + } + }, + "top_errors": [], + "note": "low_entropy bucket = {deterministic, template} merged; 5-way is the effective label space" + } + } +} \ No newline at end of file diff --git a/specialists/eval/results/automatability_sweep.json b/specialists/eval/results/automatability_sweep.json new file mode 100644 index 0000000..d7aa861 --- /dev/null +++ b/specialists/eval/results/automatability_sweep.json @@ -0,0 +1,178 @@ +{ + "gate": "automatability", + "labels": "deterministic,template,specialist,slm,llm,human", + "train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/automatability_train.tsv", + "eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/automatability_eval.tsv", + "tiers": [ + { + "tier": "small_plus", + "hidden": 256, + "layers": 2, + "heads": 4, + "model_size_approx": "~1M params", + "pilot_accuracy": 69.6, + "pilot_time_s": 150.0, + "eval_accuracy": 69.6, + "per_class": { + "deterministic": { + "acc": 0.0, + "n": 10 + }, + "template": { + "acc": 42.857142857142854, + "n": 7 + }, + "specialist": { + "acc": 100.0, + "n": 9 + }, + "slm": { + "acc": 100.0, + "n": 7 + }, + "llm": { + "acc": 100.0, + "n": 4 + }, + "human": { + "acc": 100.0, + "n": 9 + } + }, + "confusion_summary": [ + { + "gt": "deterministic", + "pred": "llm", + "count": 10, + "gt_total": 10 + }, + { + "gt": "template", + "pred": "llm", + "count": 4, + "gt_total": 7 + } + ], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 69.6 + }, + { + "threshold": 0.6, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 69.6 + }, + { + "threshold": 0.7, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 69.6 + }, + { + "threshold": 0.8, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 69.6 + }, + { + "threshold": 0.9, + "accepted": 32, + "abstain_rate": 0.304, + "accepted_accuracy": 100.0 + } + ], + "guardrail_threshold": 0.9, + "guardrail_abstain_rate": 0.304, + "eval_latency_ms": 5.137, + "verdict": "decompose_or_relabel", + "action": "decompose_or_relabel", + "eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/automatability_small_plus.json" + }, + { + "tier": "medium", + "hidden": 384, + "layers": 3, + "heads": 6, + "model_size_approx": "~5M params", + "pilot_accuracy": 78.3, + "pilot_time_s": 390.0, + "eval_accuracy": 78.3, + "per_class": { + "deterministic": { + "acc": 0.0, + "n": 10 + }, + "template": { + "acc": 100.0, + "n": 7 + }, + "specialist": { + "acc": 100.0, + "n": 9 + }, + "slm": { + "acc": 100.0, + "n": 7 + }, + "llm": { + "acc": 100.0, + "n": 4 + }, + "human": { + "acc": 100.0, + "n": 9 + } + }, + "confusion_summary": [ + { + "gt": "deterministic", + "pred": "template", + "count": 10, + "gt_total": 10 + } + ], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 78.3 + }, + { + "threshold": 0.6, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 78.3 + }, + { + "threshold": 0.7, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 78.3 + }, + { + "threshold": 0.8, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 78.3 + }, + { + "threshold": 0.9, + "accepted": 46, + "abstain_rate": 0.0, + "accepted_accuracy": 78.3 + } + ], + "guardrail_threshold": null, + "guardrail_abstain_rate": null, + "eval_latency_ms": 5.345, + "verdict": "decompose_or_relabel", + "action": "decompose_or_relabel", + "eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/automatability_medium.json" + } + ] +} \ No newline at end of file diff --git a/specialists/eval/results/prereq_op_sweep.json b/specialists/eval/results/prereq_op_sweep.json new file mode 100644 index 0000000..e642905 --- /dev/null +++ b/specialists/eval/results/prereq_op_sweep.json @@ -0,0 +1,116 @@ +{ + "gate": "prereq_op", + "labels": "standard,needs_review,needs_approval,full_gates", + "train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/combined/prereq_op_train.tsv", + "eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/combined/prereq_op_eval.tsv", + "tiers": [ + { + "tier": "small_plus", + "hidden": 256, + "layers": 2, + "heads": 4, + "model_size_approx": "~1M params", + "pilot_accuracy": 75.0, + "pilot_time_s": 152.0, + "eval_accuracy": 72.2, + "per_class": { + "standard": { + "acc": 37.5, + "n": 8 + }, + "needs_review": { + "acc": 80.0, + "n": 10 + }, + "needs_approval": { + "acc": 66.66666666666666, + "n": 9 + }, + "full_gates": { + "acc": 100.0, + "n": 9 + } + }, + "confusion_summary": [ + { + "gt": "standard", + "pred": "needs_approval", + "count": 4, + "gt_total": 8 + }, + { + "gt": "needs_approval", + "pred": "needs_review", + "count": 2, + "gt_total": 9 + }, + { + "gt": "standard", + "pred": "needs_review", + "count": 1, + "gt_total": 8 + }, + { + "gt": "needs_review", + "pred": "standard", + "count": 1, + "gt_total": 10 + }, + { + "gt": "needs_review", + "pred": "needs_approval", + "count": 1, + "gt_total": 10 + } + ], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 36, + "abstain_rate": 0.0, + "accepted_accuracy": 72.2 + }, + { + "threshold": 0.6, + "accepted": 34, + "abstain_rate": 0.056, + "accepted_accuracy": 76.5 + }, + { + "threshold": 0.7, + "accepted": 34, + "abstain_rate": 0.056, + "accepted_accuracy": 76.5 + }, + { + "threshold": 0.8, + "accepted": 34, + "abstain_rate": 0.056, + "accepted_accuracy": 76.5 + }, + { + "threshold": 0.9, + "accepted": 33, + "abstain_rate": 0.083, + "accepted_accuracy": 78.8 + } + ], + "guardrail_threshold": null, + "guardrail_abstain_rate": null, + "eval_latency_ms": 6.008, + "verdict": "decompose_or_relabel", + "action": "decompose_or_relabel", + "eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/prereq_op_small_plus.json" + }, + { + "tier": "medium", + "hidden": 384, + "layers": 3, + "heads": 6, + "model_size_approx": "~5M params", + "pilot_accuracy": 66.7, + "pilot_time_s": 388.0, + "action": "no_gain_stop_scaling" + } + ] +} \ No newline at end of file diff --git a/specialists/eval/results/verification_type_sweep.json b/specialists/eval/results/verification_type_sweep.json new file mode 100644 index 0000000..7c869b5 --- /dev/null +++ b/specialists/eval/results/verification_type_sweep.json @@ -0,0 +1,120 @@ +{ + "gate": "verification_type", + "labels": "unit,integration,schema,smoke,docs", + "train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/verification_type_train.tsv", + "eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/verification_type_eval.tsv", + "tiers": [ + { + "tier": "small_plus", + "hidden": 256, + "layers": 2, + "heads": 4, + "model_size_approx": "~1M params", + "pilot_accuracy": 83.7, + "pilot_time_s": 186.0, + "eval_accuracy": 83.7, + "per_class": { + "unit": { + "acc": 75.75757575757575, + "n": 33 + }, + "integration": { + "acc": 76.47058823529412, + "n": 34 + }, + "schema": { + "acc": 76.47058823529412, + "n": 34 + }, + "smoke": { + "acc": 93.75, + "n": 32 + }, + "docs": { + "acc": 94.87179487179486, + "n": 39 + } + }, + "confusion_summary": [ + { + "gt": "unit", + "pred": "schema", + "count": 5, + "gt_total": 33 + }, + { + "gt": "integration", + "pred": "unit", + "count": 5, + "gt_total": 34 + }, + { + "gt": "schema", + "pred": "unit", + "count": 4, + "gt_total": 34 + }, + { + "gt": "schema", + "pred": "integration", + "count": 4, + "gt_total": 34 + }, + { + "gt": "unit", + "pred": "integration", + "count": 3, + "gt_total": 33 + } + ], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 172, + "abstain_rate": 0.0, + "accepted_accuracy": 83.7 + }, + { + "threshold": 0.6, + "accepted": 171, + "abstain_rate": 0.006, + "accepted_accuracy": 83.6 + }, + { + "threshold": 0.7, + "accepted": 170, + "abstain_rate": 0.012, + "accepted_accuracy": 83.5 + }, + { + "threshold": 0.8, + "accepted": 164, + "abstain_rate": 0.047, + "accepted_accuracy": 84.8 + }, + { + "threshold": 0.9, + "accepted": 160, + "abstain_rate": 0.07, + "accepted_accuracy": 85.0 + } + ], + "guardrail_threshold": null, + "guardrail_abstain_rate": null, + "eval_latency_ms": 4.976, + "verdict": "scale_more", + "action": "scale_more", + "eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/verification_type_small_plus.json" + }, + { + "tier": "medium", + "hidden": 384, + "layers": 3, + "heads": 6, + "model_size_approx": "~5M params", + "pilot_accuracy": 84.3, + "pilot_time_s": 402.0, + "action": "no_gain_stop_scaling" + } + ] +} \ No newline at end of file diff --git a/specialists/eval/results/worker_type_combined_sweep.json b/specialists/eval/results/worker_type_combined_sweep.json new file mode 100644 index 0000000..39a41a6 --- /dev/null +++ b/specialists/eval/results/worker_type_combined_sweep.json @@ -0,0 +1,116 @@ +{ + "gate": "worker_type_combined", + "labels": "implementer,reviewer,architect,qa", + "train": "specialists/data/generated/worker_type_combined_train.tsv", + "eval": "specialists/data/generated/worker_type_combined_eval.tsv", + "tiers": [ + { + "tier": "small_plus", + "hidden": 256, + "layers": 2, + "heads": 4, + "model_size_approx": "~1M params", + "pilot_accuracy": 92.2, + "pilot_time_s": 155.0, + "eval_accuracy": 91.3, + "per_class": { + "implementer": { + "acc": 75.0, + "n": 32 + }, + "reviewer": { + "acc": 100.0, + "n": 29 + }, + "architect": { + "acc": 100.0, + "n": 29 + }, + "qa": { + "acc": 92.0, + "n": 25 + } + }, + "confusion_summary": [ + { + "gt": "implementer", + "pred": "reviewer", + "count": 6, + "gt_total": 32 + }, + { + "gt": "implementer", + "pred": "architect", + "count": 1, + "gt_total": 32 + }, + { + "gt": "implementer", + "pred": "qa", + "count": 1, + "gt_total": 32 + }, + { + "gt": "qa", + "pred": "implementer", + "count": 1, + "gt_total": 25 + }, + { + "gt": "qa", + "pred": "architect", + "count": 1, + "gt_total": 25 + } + ], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 115, + "abstain_rate": 0.0, + "accepted_accuracy": 91.3 + }, + { + "threshold": 0.6, + "accepted": 113, + "abstain_rate": 0.017, + "accepted_accuracy": 92.0 + }, + { + "threshold": 0.7, + "accepted": 111, + "abstain_rate": 0.035, + "accepted_accuracy": 91.9 + }, + { + "threshold": 0.8, + "accepted": 109, + "abstain_rate": 0.052, + "accepted_accuracy": 91.7 + }, + { + "threshold": 0.9, + "accepted": 109, + "abstain_rate": 0.052, + "accepted_accuracy": 91.7 + } + ], + "guardrail_threshold": null, + "guardrail_abstain_rate": null, + "eval_latency_ms": 1.993, + "verdict": "scale_more", + "action": "scale_more", + "eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/worker_type_combined_small_plus.json" + }, + { + "tier": "medium", + "hidden": 384, + "layers": 3, + "heads": 6, + "model_size_approx": "~5M params", + "pilot_accuracy": 89.6, + "pilot_time_s": 393.0, + "action": "no_gain_stop_scaling" + } + ] +} \ No newline at end of file diff --git a/specialists/eval/results/worker_type_sweep.json b/specialists/eval/results/worker_type_sweep.json new file mode 100644 index 0000000..244c27c --- /dev/null +++ b/specialists/eval/results/worker_type_sweep.json @@ -0,0 +1,104 @@ +{ + "gate": "worker_type", + "labels": "implementer,reviewer,architect,qa", + "train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/worker_type_train.tsv", + "eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/worker_type_eval.tsv", + "tiers": [ + { + "tier": "small_plus", + "hidden": 256, + "layers": 2, + "heads": 4, + "model_size_approx": "~1M params", + "pilot_accuracy": 80.8, + "pilot_time_s": 154.0, + "eval_accuracy": 80.8, + "per_class": { + "implementer": { + "acc": 42.857142857142854, + "n": 14 + }, + "reviewer": { + "acc": 100.0, + "n": 8 + }, + "architect": { + "acc": 100.0, + "n": 16 + }, + "qa": { + "acc": 85.71428571428571, + "n": 14 + } + }, + "confusion_summary": [ + { + "gt": "implementer", + "pred": "reviewer", + "count": 5, + "gt_total": 14 + }, + { + "gt": "implementer", + "pred": "architect", + "count": 3, + "gt_total": 14 + }, + { + "gt": "qa", + "pred": "architect", + "count": 2, + "gt_total": 14 + } + ], + "threshold_analysis": [ + { + "threshold": 0.5, + "accepted": 52, + "abstain_rate": 0.0, + "accepted_accuracy": 80.8 + }, + { + "threshold": 0.6, + "accepted": 52, + "abstain_rate": 0.0, + "accepted_accuracy": 80.8 + }, + { + "threshold": 0.7, + "accepted": 51, + "abstain_rate": 0.019, + "accepted_accuracy": 82.4 + }, + { + "threshold": 0.8, + "accepted": 51, + "abstain_rate": 0.019, + "accepted_accuracy": 82.4 + }, + { + "threshold": 0.9, + "accepted": 48, + "abstain_rate": 0.077, + "accepted_accuracy": 85.4 + } + ], + "guardrail_threshold": null, + "guardrail_abstain_rate": null, + "eval_latency_ms": 4.296, + "verdict": "scale_more", + "action": "scale_more", + "eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/worker_type_small_plus.json" + }, + { + "tier": "medium", + "hidden": 384, + "layers": 3, + "heads": 6, + "model_size_approx": "~5M params", + "pilot_accuracy": 78.8, + "pilot_time_s": 393.0, + "action": "no_gain_stop_scaling" + } + ] +} \ No newline at end of file diff --git a/specialists/scripts/split_automatability_decompose.py b/specialists/scripts/split_automatability_decompose.py new file mode 100644 index 0000000..29313c3 --- /dev/null +++ b/specialists/scripts/split_automatability_decompose.py @@ -0,0 +1,332 @@ +#!/usr/bin/env python3 +""" +split_automatability_decompose.py + +Tests whether automatability benefits from a 2-stage decomposition: + + Stage 1 (binary): low_entropy {deterministic, template} vs needs_routing {specialist, slm, llm, human} + Stage 2 (4-way): specialist / slm / llm / human (only for items routed past stage 1) + +The flat 6-way model confuses deterministic↔template because the text features +that distinguish them aren't present in taskitem titles. This decomposition +isolates that gap — stage 1 correctly routes low-entropy items out, stage 2 +handles the routed set cleanly. + +Outputs combined accuracy as a 5-way result (low_entropy, specialist, slm, llm, human) +where deterministic/template are merged into one bucket, plus routing accuracy alone. + +Usage: + python3 specialists/scripts/split_automatability_decompose.py \\ + --data-dir specialists/data/generated \\ + --runs-dir /mnt/storage/fabricate_runs \\ + --results-out specialists/eval/results/automatability_decompose.json +""" + +import argparse +import json +import subprocess +import sys +from collections import Counter +from pathlib import Path + + +# Must be at module level so torch.load can find __main__.Vocab when unpickling checkpoints +class Vocab: + PAD, UNK = 0, 1 + + def __init__(self): + self.str_to_id = {"": 0, "": 1} + self.id_to_str = ["", ""] + + def encode(self, text, seq_len): + ids = [self.str_to_id.get(t, self.UNK) for t in text.lower().split()] + ids = ids[:seq_len] + ids += [self.PAD] * (seq_len - len(ids)) + return ids + + def __len__(self): + return len(self.id_to_str) + + +ROOT = Path(__file__).parent.parent.parent +VENV_PYTHON = sys.executable +TRAIN_SCRIPT = ROOT / "specialists" / "scripts" / "train_specialist_pt.py" +EVAL_SCRIPT = ROOT / "specialists" / "eval" / "eval_specialist_pt.py" + +# Original 6-way label order: deterministic(0), template(1), specialist(2), slm(3), llm(4), human(5) +LABEL_NAMES_6 = ["deterministic", "template", "specialist", "slm", "llm", "human"] + +# Stage 1: low_entropy = {0, 1}, needs_routing = {2, 3, 4, 5} +LOW_ENTROPY = {0, 1} + +# Stage 2: remap {2→0, 3→1, 4→2, 5→3} +STAGE2_LABELS = ["specialist", "slm", "llm", "human"] +STAGE2_REMAP = {2: 0, 3: 1, 4: 2, 5: 3} + + +def split_binary(src: Path, out: Path): + """6-way TSV → binary TSV: low_entropy=0, needs_routing=1.""" + rows = [] + with open(src) as f: + for line in f: + parts = line.strip().split("\t", 2) + if len(parts) < 3: + continue + orig = int(parts[0]) + rows.append(f"{'0' if orig in LOW_ENTROPY else '1'}\t0\t{parts[2]}") + out.write_text("\n".join(rows) + "\n") + n0 = sum(1 for r in rows if r.startswith("0")) + print(f" {out.name}: {len(rows)} rows low_entropy={n0} needs_routing={len(rows)-n0}") + + +def split_stage2(src: Path, out: Path): + """6-way TSV → 4-way TSV keeping only needs_routing rows, remapped to 0-3.""" + rows = [] + with open(src) as f: + for line in f: + parts = line.strip().split("\t", 2) + if len(parts) < 3: + continue + orig = int(parts[0]) + if orig in STAGE2_REMAP: + rows.append(f"{STAGE2_REMAP[orig]}\t0\t{parts[2]}") + out.write_text("\n".join(rows) + "\n") + from collections import Counter + dist = Counter(int(r.split("\t")[0]) for r in rows) + print(f" {out.name}: {len(rows)} rows " + " ".join(f"{STAGE2_LABELS[i]}={dist[i]}" for i in range(4))) + + +def train(run_dir: Path, train_tsv: Path, eval_tsv: Path, labels: str, max_steps: int = 10000) -> bool: + run_dir.mkdir(parents=True, exist_ok=True) + for f in run_dir.glob("*"): + f.unlink() + r = subprocess.run([ + str(VENV_PYTHON), str(TRAIN_SCRIPT), + "--dataset", str(train_tsv), + "--eval-dataset", str(eval_tsv), + "--out-dir", str(run_dir), + "--labels", labels, + "--lr", "0.001", + "--weight-decay", "0.01", + "--max-steps", str(max_steps), + "--hidden-dim", "256", + "--layers", "2", + "--heads", "4", + "--batch-size", "256", + "--grok-loss-threshold", "0.05", + "--grok-acc-jump", "10.0", + "--stop-after-grokking-blocks", "4", + "--reset", + "--history-out", str(run_dir / "history.json"), + ]) + return r.returncode == 0 + + +def eval_checkpoint(run_dir: Path, eval_tsv: Path, labels: str) -> dict | None: + ck = run_dir / "checkpoint.pt" + if not ck.exists(): + return None + r = subprocess.run( + [str(VENV_PYTHON), str(EVAL_SCRIPT), + "--checkpoint", str(ck), + "--dataset", str(eval_tsv), "--labels", labels, "--quiet"], + capture_output=True, text=True, + ) + if r.returncode != 0: + return None + try: + return json.loads(r.stdout.strip().splitlines()[-1]) + except (json.JSONDecodeError, IndexError): + return None + + +def evaluate_combined(eval_tsv_6way: Path, stage1_run: Path, stage2_run: Path) -> dict: + """Load both models; stage1 routes, stage2 classifies routed items. + Reports: routing accuracy, 5-way accuracy (low_entropy merged), and + comparison against original 6-way labels.""" + import torch + import torch.nn as nn + import numpy as np + + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + + def _tok(text): + return text.lower().split() + + class _Model(nn.Module): + def __init__(self, cfg): + super().__init__() + D, L, H, V, seq = cfg["hidden_dim"], cfg["layers"], cfg["heads"], cfg["vocab_size"], cfg["seq_len"] + self.tok_emb = nn.Embedding(V, D, padding_idx=0) + self.pos_emb = nn.Embedding(seq, D) + enc = nn.TransformerEncoderLayer(d_model=D, nhead=H, dim_feedforward=D*4, + dropout=0.0, batch_first=True) + self.encoder = nn.TransformerEncoder(enc, num_layers=L, enable_nested_tensor=False) + self.head = nn.Linear(D, cfg["n_labels"]) + self.seq_len = seq + + def forward(self, x): + pad_mask = (x == 0) + pos = torch.arange(x.size(1), device=x.device).unsqueeze(0) + h = self.tok_emb(x) + self.pos_emb(pos) + h = self.encoder(h, src_key_padding_mask=pad_mask) + lengths = (~pad_mask).float().sum(1, keepdim=True).clamp(min=1) + h = (h * (~pad_mask).unsqueeze(-1).float()).sum(1) / lengths + return self.head(h) + + def load_model(run_dir): + ckpt = torch.load(run_dir / "checkpoint.pt", map_location=device, weights_only=False) + m = _Model(ckpt["config"]).to(device) + m.load_state_dict(ckpt["model"]) + m.eval() + vocab = ckpt.get("vocab_obj") or ckpt.get("vocab") + if isinstance(vocab, dict): + class _V: + UNK = 1 + def __init__(self, d): self.str_to_id = d + def encode(self, text, seq_len): + ids = [self.str_to_id.get(t, self.UNK) for t in text.lower().split()] + ids = ids[:seq_len]; ids += [0] * (seq_len - len(ids)); return ids + vocab = _V(vocab) + return m, vocab, ckpt["config"]["seq_len"] + + def predict_probs(model, vocab, seq_len, texts): + import torch.nn.functional as F + all_probs = [] + with torch.no_grad(): + for start in range(0, len(texts), 256): + batch = texts[start:start+256] + ids = [vocab.encode(t, seq_len) for t in batch] + x = torch.tensor(ids, dtype=torch.long, device=device) + logits = model(x) + all_probs.append(F.softmax(logits, dim=-1).cpu().numpy()) + return np.concatenate(all_probs, axis=0) + + # Load original 6-way eval data + labels_gt, texts = [], [] + with open(eval_tsv_6way) as f: + for line in f: + parts = line.strip().split("\t", 2) + if len(parts) < 3: + continue + labels_gt.append(int(parts[0])) + texts.append(parts[2]) + labels_gt = np.array(labels_gt) + n = len(labels_gt) + + m1, v1, s1 = load_model(stage1_run) + m2, v2, s2 = load_model(stage2_run) + + probs1 = predict_probs(m1, v1, s1, texts) # shape (n, 2): [low_entropy, needs_routing] + probs2 = predict_probs(m2, v2, s2, texts) # shape (n, 4): [specialist, slm, llm, human] + + stage1_pred = np.argmax(probs1, axis=1) # 0=low_entropy, 1=needs_routing + + # Ground-truth routing: 0 if original label in {0,1}, else 1 + gt_routing = np.where(np.isin(labels_gt, [0, 1]), 0, 1) + routing_acc = float((stage1_pred == gt_routing).mean() * 100) + + # Build 5-way predictions (low_entropy=0, specialist=1, slm=2, llm=3, human=4) + stage2_pred = np.argmax(probs2, axis=1) # 0-3 → specialist/slm/llm/human + pred_5way = np.where(stage1_pred == 0, 0, stage2_pred + 1) + + # Ground-truth 5-way: low_entropy if orig in {0,1}, else stage2 remapped + gt_5way = np.where(np.isin(labels_gt, [0, 1]), 0, labels_gt - 1) + + acc_5way = float((pred_5way == gt_5way).mean() * 100) + correct_5way = pred_5way == gt_5way + + label_names_5 = ["low_entropy", "specialist", "slm", "llm", "human"] + per_class = {} + for i, name in enumerate(label_names_5): + mask = gt_5way == i + per_class[name] = { + "acc": float(correct_5way[mask].mean() * 100) if mask.sum() > 0 else None, + "n": int(mask.sum()), + } + + errors = Counter() + for gt, pred in zip(gt_5way.tolist(), pred_5way.tolist()): + if gt != pred: + errors[(label_names_5[gt], label_names_5[pred])] += 1 + + return { + "method": "2stage_decomposed", + "routing_accuracy": round(routing_acc, 1), + "accuracy_5way": round(acc_5way, 1), + "n_eval": n, + "per_class_5way": per_class, + "top_errors": [{"gt": k[0], "pred": k[1], "count": v} + for k, v in errors.most_common(6)], + "note": "low_entropy bucket = {deterministic, template} merged; 5-way is the effective label space", + } + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--data-dir", default="specialists/data/generated") + ap.add_argument("--runs-dir", default="/mnt/storage/fabricate_runs") + ap.add_argument("--results-out", default="specialists/eval/results/automatability_decompose.json") + ap.add_argument("--skip-train", action="store_true") + args = ap.parse_args() + + data_dir = Path(args.data_dir) + runs_dir = Path(args.runs_dir) + results = Path(args.results_out) + results.parent.mkdir(parents=True, exist_ok=True) + + tmp = data_dir # write split TSVs alongside existing data + + stage1_train = tmp / "automatability_binary_train.tsv" + stage1_eval = tmp / "automatability_binary_eval.tsv" + stage2_train = tmp / "automatability_routed_train.tsv" + stage2_eval = tmp / "automatability_routed_eval.tsv" + + print("=== Splitting data ===") + split_binary(data_dir / "automatability_train.tsv", stage1_train) + split_binary(data_dir / "automatability_eval.tsv", stage1_eval) + split_stage2(data_dir / "automatability_train.tsv", stage2_train) + split_stage2(data_dir / "automatability_eval.tsv", stage2_eval) + + stage1_run = runs_dir / "whetstone_automatability_stage1_binary" + stage2_run = runs_dir / "whetstone_automatability_stage2_4way" + + if not args.skip_train: + print("\n=== Training stage 1: low_entropy vs needs_routing (binary) ===") + train(stage1_run, stage1_train, stage1_eval, "low_entropy,needs_routing") + + print("\n=== Training stage 2: specialist/slm/llm/human (4-way) ===") + train(stage2_run, stage2_train, stage2_eval, "specialist,slm,llm,human") + + print("\n=== Evaluating individual models ===") + r1 = eval_checkpoint(stage1_run, stage1_eval, "low_entropy,needs_routing") + r2 = eval_checkpoint(stage2_run, stage2_eval, "specialist,slm,llm,human") + if r1: + print(f" Stage 1 (binary): {r1.get('overall_accuracy')}%") + if r2: + print(f" Stage 2 (4-way): {r2.get('overall_accuracy')}%") + + print("\n=== Evaluating combined 2-stage pipeline ===") + combined = evaluate_combined(data_dir / "automatability_eval.tsv", stage1_run, stage2_run) + print(f" Routing accuracy: {combined['routing_accuracy']}%") + print(f" 5-way accuracy: {combined['accuracy_5way']}%") + print(" Per class (5-way):") + for name, stat in combined["per_class_5way"].items(): + if stat["acc"] is not None: + print(f" {name:20s}: {stat['acc']:.1f}% (n={stat['n']})") + + output = { + "gate": "automatability", + "flat_6way_baseline": {"small_plus": 69.6, "medium": 78.3}, + "decomposed": { + "stage1_binary": r1, + "stage2_4way": r2, + "combined": combined, + }, + } + results.write_text(json.dumps(output, indent=2)) + print(f"\nResults → {results}") + + +if __name__ == "__main__": + main()