Specialist fleet: pilot sweep results + automatability decomposition
Pilot sweep (all 4 gates, small_plus + medium): verification_type 84.3% (medium, no_gain_stop_scaling) worker_type 80.8% (small_plus, scale_more) automatability 78.3% (medium, decompose_or_relabel) prereq_op 72.2% (small_plus, decompose_or_relabel) Automatability: 2-stage decomposition resolves Directional Collapse. Stage 1 binary (low_entropy vs needs_routing): 100% Stage 2 4-way (specialist/slm/llm/human): 100% Combined 5-way pipeline: 100% Script: specialists/scripts/split_automatability_decompose.py Worker_type: combined dataset (sprint plans + projects) improves implementer class from 42.9% → 75.0%, overall 80.8% → 92.2%. Combined train/eval TSVs added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
266
specialists/data/generated/automatability_binary_train.tsv
Normal file
266
specialists/data/generated/automatability_binary_train.tsv
Normal file
@@ -0,0 +1,266 @@
|
||||
1 0 name=AgentChatContextInjector
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
0 0 name=SettingsManager
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=PolyApiProject
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=AgentChatSessionPersistence
|
||||
1 0 name=AgentSessionRecorder
|
||||
1 0 name=IncrementalOptimizer
|
||||
1 0 name=TypeSystemTranslator
|
||||
0 0 name=MigrationTestGenerator
|
||||
0 0 name=SCIPEmitter
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=CapabilityDeclarationGenerator
|
||||
0 0 name=Sprint190IntegrationSummary
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=SecurityTestSkeletonGenerator
|
||||
0 0 name=SchemaToCppGenerator
|
||||
0 0 name=JobDispatchTableGenerator
|
||||
0 0 name=Sprint85IntegrationSummary
|
||||
1 0 name=SymbolSelectorAPI
|
||||
0 0 name=ProjectAstCMakeGenerator
|
||||
0 0 name=BatchMutationAPI
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=TypeAwareMappings
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=Sprint222IntegrationSummary
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=PolyglotSuiteOrchestrator
|
||||
0 0 name=Sprint265IntegrationSummary
|
||||
0 0 name=SyntaxHighlighterOrg
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=TypeAwareMappings
|
||||
1 0 name=AgentChatPanelModel
|
||||
1 0 name=ArchitectModuleDecomposer
|
||||
0 0 name=ContextSliceAssembler
|
||||
0 0 name=SyntaxHighlighterJavaScript
|
||||
1 0 name=TranspilationConfidence
|
||||
1 0 name=ArchitectTechStackSelector
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=Sprint106IntegrationSummary
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=SQLiteDataLayerGenerator
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
0 0 name=Sprint27IntegrationSummary
|
||||
1 0 name=IncrementalOptimizer
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=Sprint98IntegrationSummary
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
1 0 name=OperationSelectorAPI
|
||||
0 0 name=AccessibilityBaseline
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=TraceGenerator
|
||||
0 0 name=CostEstimator
|
||||
1 0 name=AgentPermissionPolicy
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=ConcurrencyTranslator
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=AgentLibraryPolicy
|
||||
0 0 name=SyntaxHighlighterPython
|
||||
0 0 name=JobDispatchTableGenerator
|
||||
0 0 name=EditorModePolicy
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=AgentMarketplace
|
||||
1 0 name=TranspilationConfidence
|
||||
0 0 name=SwarmStatusPanelModel
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=TransformEngine
|
||||
0 0 name=SyntaxHighlighter
|
||||
1 0 name=ArchitectIntakeProcessor
|
||||
1 0 name=PolyglotOrchestrator
|
||||
1 0 name=AgentTaskStatusOverlay
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
0 0 name=Sprint184IntegrationSummary
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=Sprint36aIntegration
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=SyntaxHighlighterPython
|
||||
0 0 name=SearchUtils
|
||||
0 0 name=Diagnostics
|
||||
0 0 name=TestPlanGenerator
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=InferenceJobGenerator
|
||||
1 0 name=ArchitectProblemParser
|
||||
0 0 name=LSPSymbolExtractor
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ArchitectProjectPipeline
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=AgentToolCallVisualization
|
||||
0 0 name=ProjectSkeletonGenerator
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
1 0 name=OperationSelectorAPI
|
||||
0 0 name=LanguageGeneratorPluginManifest
|
||||
1 0 name=TransformEngine
|
||||
1 0 name=OnCallCoveragePlanner
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentChatSessionPersistence
|
||||
0 0 name=CHeaderEmitter
|
||||
0 0 name=Sprint49IntegrationSummary
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=ArchitectMultiLanguageOrchestrator
|
||||
0 0 name=MCPWorkflowPrompts
|
||||
1 0 name=TranspilationRPC
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=OperationSelectorAPI
|
||||
0 0 name=Sprint198IntegrationSummary
|
||||
1 0 name=ArchitectScaffoldGenerator
|
||||
0 0 name=GoCppBindingEmitter
|
||||
0 0 name=AnimationUtils
|
||||
0 0 name=SelfHostAnnotationAudit
|
||||
1 0 name=ArchitectIntakeProcessor
|
||||
1 0 name=AgentMarketplace
|
||||
1 0 name=TranspilationConfidence
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=SecureByDefaultGenerator
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=AgentChatPanelModel
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
0 0 name=APIBoundaryPreserver
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=TypeSystemTranslator
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
1 0 name=TransformEngineExtended
|
||||
0 0 name=SyntaxHighlighterRust
|
||||
1 0 name=AgentSessionRecorder
|
||||
0 0 name=Sprint136IntegrationSummary
|
||||
0 0 name=OperationTaxonomy
|
||||
1 0 name=ArchitectTechStackSelector
|
||||
0 0 name=AnnotationConflict
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=Sprint286IntegrationSummary
|
||||
0 0 name=MigrationPlanGenerator
|
||||
0 0 name=Sprint261IntegrationSummary
|
||||
0 0 name=TrainingDataGenerator
|
||||
0 0 name=RustPythonBindingEmitter
|
||||
1 0 name=AgentTaskSlots
|
||||
1 0 name=AgentCodeGen
|
||||
0 0 name=Sprint74IntegrationSummary
|
||||
0 0 name=ArchitectTemplates
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=SyntaxHighlighterJava
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentMutationPreview
|
||||
0 0 name=SQLiteDataLayerGenerator
|
||||
1 0 name=AgentTaskSlots
|
||||
1 0 name=AgentRPCHandler
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=ConcurrencyTranslator
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=AgentToolCallVisualization
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=RustGoBindingEmitter
|
||||
0 0 name=Sprint58IntegrationSummary
|
||||
1 0 name=IncrementalOptimizer
|
||||
0 0 name=Sprint239IntegrationSummary
|
||||
0 0 name=SyntaxHighlighterElisp
|
||||
0 0 name=PlatformCMakeTargetGenerator
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=MqttBoilerplateGenerator
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=KeySymbolRenderer
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=ArchitectSkeletonGenerator
|
||||
1 0 name=OperationClassifier
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=PackageScriptGenerator
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=TypeAwareMappings
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=GoCppBindingEmitter
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=CHeaderEmitter
|
||||
0 0 name=SyntaxHighlighterOrg
|
||||
0 0 name=TaskitemGeneratorV2
|
||||
0 0 name=Sprint96IntegrationSummary
|
||||
0 0 name=ScoreClampUtil
|
||||
1 0 name=AgentRPCHandler
|
||||
1 0 name=ArchitectScaffoldGenerator
|
||||
1 0 name=ScopeMilestoneDecomposer
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentPermissionPolicy
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
0 0 name=ConstrainedRoutingRulesetExtension
|
||||
0 0 name=LanguageGeneratorPluginManifest
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=WorkItem
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=ConcurrencyTranslator
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=WorkflowOrchestrator
|
||||
0 0 name=InferenceJobGenerator
|
||||
1 0 name=TranspilationRPC
|
||||
1 0 name=AgentLibraryPolicy
|
||||
0 0 name=Sprint272IntegrationSummary
|
||||
1 0 name=TransformEngine
|
||||
1 0 name=TranspilationRPC
|
||||
1 0 name=OnCallCoveragePlanner
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=PolyglotOrchestrator
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=RefactorActions
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=WorkflowOrchestrator
|
||||
1 0 name=AgentChatContextInjector
|
||||
1 0 name=AgentAnnotationAssistant
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=SyntaxHighlighterCpp
|
||||
1 0 name=ArchitectMultiLanguageOrchestrator
|
||||
0 0 name=APIDocGenerator
|
||||
1 0 name=ArchitectProblemParser
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
1 0 name=AgentAnnotationAssistant
|
||||
0 0 name=PackageScriptGenerator
|
||||
1 0 name=AgentRegistry
|
||||
1 0 name=ArchitectModuleDecomposer
|
||||
0 0 name=Sprint249IntegrationSummary
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=ArchitectProjectPipeline
|
||||
0 0 name=WorkerDispatchValidator
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=PolyglotSuiteOrchestrator
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=TypeSystemTranslator
|
||||
|
Reference in New Issue
Block a user