Specialist fleet: pilot sweep results + automatability decomposition
Pilot sweep (all 4 gates, small_plus + medium): verification_type 84.3% (medium, no_gain_stop_scaling) worker_type 80.8% (small_plus, scale_more) automatability 78.3% (medium, decompose_or_relabel) prereq_op 72.2% (small_plus, decompose_or_relabel) Automatability: 2-stage decomposition resolves Directional Collapse. Stage 1 binary (low_entropy vs needs_routing): 100% Stage 2 4-way (specialist/slm/llm/human): 100% Combined 5-way pipeline: 100% Script: specialists/scripts/split_automatability_decompose.py Worker_type: combined dataset (sprint plans + projects) improves implementer class from 42.9% → 75.0%, overall 80.8% → 92.2%. Combined train/eval TSVs added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
46
specialists/data/generated/automatability_binary_eval.tsv
Normal file
46
specialists/data/generated/automatability_binary_eval.tsv
Normal file
@@ -0,0 +1,46 @@
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
0 0 name=TrainingDataGenerator
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=TypeSystemTranslator
|
||||
0 0 name=ContextBundleMinimizer
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=IncrementalOptimizer
|
||||
0 0 name=BufferOpsLifecycle
|
||||
0 0 name=MigrationAcceptanceContract
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=Sprint163IntegrationSummary
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=AgentCodeGen
|
||||
0 0 name=Sprint238IntegrationSummary
|
||||
0 0 name=CppJSBindingEmitter
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=Sprint36IntegrationSummary
|
||||
0 0 name=SelfHostHarness
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=Sprint111IntegrationSummary
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
0 0 name=SyntaxHighlighterElisp
|
||||
0 0 name=SelfHostCodebaseAudit
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
0 0 name=SyntaxHighlighterGo
|
||||
1 0 name=AgentRegistry
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=WorkflowTemplates
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=AgentMutationPreview
|
||||
0 0 name=RustPythonBindingEmitter
|
||||
1 0 name=TransformEngine
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=CompilerPhaseSpec
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=AgentTaskStatusOverlay
|
||||
1 0 name=ScopeMilestoneDecomposer
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=CompositionBuilder
|
||||
|
Reference in New Issue
Block a user