Specialist fleet: pilot sweep results + automatability decomposition
Pilot sweep (all 4 gates, small_plus + medium): verification_type 84.3% (medium, no_gain_stop_scaling) worker_type 80.8% (small_plus, scale_more) automatability 78.3% (medium, decompose_or_relabel) prereq_op 72.2% (small_plus, decompose_or_relabel) Automatability: 2-stage decomposition resolves Directional Collapse. Stage 1 binary (low_entropy vs needs_routing): 100% Stage 2 4-way (specialist/slm/llm/human): 100% Combined 5-way pipeline: 100% Script: specialists/scripts/split_automatability_decompose.py Worker_type: combined dataset (sprint plans + projects) improves implementer class from 42.9% → 75.0%, overall 80.8% → 92.2%. Combined train/eval TSVs added. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill
86
specialists/HANDOFF-2026-04-18.md
Normal file
86
specialists/HANDOFF-2026-04-18.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# Specialist Fleet Handoff — 2026-04-18
|
||||
|
||||
## Status at Session End
|
||||
|
||||
Sweep was killed mid-run to conserve solar battery. State:
|
||||
|
||||
| Gate | Tier | Status | Accuracy |
|
||||
|------|------|--------|----------|
|
||||
| verification_type | small_plus | COMPLETE | 81.4% — verdict: `scale_more` |
|
||||
| verification_type | medium | KILLED | Partial checkpoint, do not trust |
|
||||
| worker_type | — | NOT STARTED | — |
|
||||
| prereq_op | — | NOT STARTED | — |
|
||||
| automatability | — | NOT STARTED | — |
|
||||
|
||||
## First Task Next Session
|
||||
|
||||
From the whetstone_DSL root, run the full pilot sweep fresh:
|
||||
|
||||
```bash
|
||||
cd /home/bill/Documents/CLionProjects/whetstone_DSL
|
||||
bash specialists/scripts/sweep_all_gates.sh --pilot-only 2>&1 | tee specialists/eval/results/sweep_run.log
|
||||
```
|
||||
|
||||
The `--reset` flag inside `capacity_sweep.py` ensures the broken `verification_type` medium
|
||||
checkpoint from this session doesn't interfere. All 4 gates run from scratch.
|
||||
|
||||
Expected runtime: ~25–35 minutes total (small_plus + medium × 4 gates, 3000 steps each).
|
||||
|
||||
Results land in:
|
||||
- `specialists/eval/results/<gate>_sweep.json` — per-gate sweep table
|
||||
- `specialists/eval/results/summary.json` — combined best-tier per gate
|
||||
|
||||
## Fixes Applied This Session (All Saved to Disk)
|
||||
|
||||
1. **Venv path** — `sweep_all_gates.sh` now uses whetstone DSL venv:
|
||||
`PYTHON="/home/bill/Documents/CLionProjects/whetstone_DSL/.venv/bin/python3"`
|
||||
|
||||
2. **Venv path** — `capacity_sweep.py` + `prereq_op_binary_split.py` now use `sys.executable`
|
||||
instead of the deleted `WhetstoneAI_Fabricate/.venv` path.
|
||||
|
||||
3. **Eval JSON parse** — `eval_specialist_pt.py` now prints single-line JSON as last line:
|
||||
`print(json.dumps(result)) # single line — capacity_sweep.py reads last line as JSON`
|
||||
|
||||
4. **Large tier removed** — `sweep_all_gates.sh` defaults to `TIERS="small_plus,medium"`.
|
||||
Large (512/4/8, 13M params) takes ~247s/block — impractical for a pilot.
|
||||
|
||||
5. **Nested tensor slowpath** — `train_specialist_pt.py` has `enable_nested_tensor=False`
|
||||
on TransformerEncoder. This eliminated ~40% overhead AND improved accuracy.
|
||||
|
||||
6. **Inductor compile** — `torch.compile(mode="reduce-overhead")` enabled with TF32.
|
||||
`python3.12-dev` is installed. Actual speedup: ~62s → ~45s/block (~25%).
|
||||
Note: `torch.randint` dynamic sampling prevents full CUDA graph replay; partial
|
||||
re-recording still happens. Pre-generated static batch pool would fix this but
|
||||
wasn't implemented (one-time training cost doesn't justify complexity).
|
||||
|
||||
## Baseline Accuracy (from prior session, before this session's fixes)
|
||||
|
||||
| Gate | Prior Best | Labels |
|
||||
|------|-----------|--------|
|
||||
| worker_type | 88.5% | implementer, reviewer, architect, qa |
|
||||
| verification_type | 84.9% | unit, integration, schema, smoke, docs |
|
||||
| prereq_op | 69.4% | standard, needs_review, needs_approval, full_gates |
|
||||
| automatability | 69.6% | deterministic, template, specialist, slm, llm, human |
|
||||
|
||||
Target: 98-99% (deploy_direct verdict) or 95%+ at ≤0.30 abstain rate (deploy_with_guardrails).
|
||||
|
||||
## After the Sweep
|
||||
|
||||
Check `specialists/eval/results/summary.json`. For any gate with verdict `scale_more`:
|
||||
- Try full 15k steps: edit capacity_sweep.py `MAX_STEPS` or run the gate individually
|
||||
- Or generate more training data — this is the higher-leverage move for low-accuracy gates
|
||||
|
||||
Gates most likely to need more data: `prereq_op` (4-way, hard to separate) and `automatability`
|
||||
(6-way, some classes are similar). Consider a data gen script that uses an LLM to generate
|
||||
examples per label given a description.
|
||||
|
||||
## Model Architecture (for reference)
|
||||
|
||||
Small_plus tier: hidden=256, layers=2, heads=4 (~1.2M params)
|
||||
Medium tier: hidden=384, layers=3, heads=6 (~3.0M params)
|
||||
|
||||
Encoder-only transformer, mean-pool over non-padding positions, linear classification head.
|
||||
Tokenizer: whitespace split, lowercase. Seq len: 96 tokens.
|
||||
Checkpoint format: `checkpoint.pt` — contains model, optimizer, vocab, config, labels, step.
|
||||
|
||||
Checkpoints live in: `/mnt/storage/fabricate_runs/whetstone_<gate>_<tier>/checkpoint.pt`
|
||||
46
specialists/data/generated/automatability_binary_eval.tsv
Normal file
46
specialists/data/generated/automatability_binary_eval.tsv
Normal file
@@ -0,0 +1,46 @@
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
0 0 name=TrainingDataGenerator
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=TypeSystemTranslator
|
||||
0 0 name=ContextBundleMinimizer
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=IncrementalOptimizer
|
||||
0 0 name=BufferOpsLifecycle
|
||||
0 0 name=MigrationAcceptanceContract
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=Sprint163IntegrationSummary
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=AgentCodeGen
|
||||
0 0 name=Sprint238IntegrationSummary
|
||||
0 0 name=CppJSBindingEmitter
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=Sprint36IntegrationSummary
|
||||
0 0 name=SelfHostHarness
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=Sprint111IntegrationSummary
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
0 0 name=SyntaxHighlighterElisp
|
||||
0 0 name=SelfHostCodebaseAudit
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
0 0 name=SyntaxHighlighterGo
|
||||
1 0 name=AgentRegistry
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=WorkflowTemplates
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=AgentMutationPreview
|
||||
0 0 name=RustPythonBindingEmitter
|
||||
1 0 name=TransformEngine
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=CompilerPhaseSpec
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=AgentTaskStatusOverlay
|
||||
1 0 name=ScopeMilestoneDecomposer
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=CompositionBuilder
|
||||
|
266
specialists/data/generated/automatability_binary_train.tsv
Normal file
266
specialists/data/generated/automatability_binary_train.tsv
Normal file
@@ -0,0 +1,266 @@
|
||||
1 0 name=AgentChatContextInjector
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
0 0 name=SettingsManager
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=PolyApiProject
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=AgentChatSessionPersistence
|
||||
1 0 name=AgentSessionRecorder
|
||||
1 0 name=IncrementalOptimizer
|
||||
1 0 name=TypeSystemTranslator
|
||||
0 0 name=MigrationTestGenerator
|
||||
0 0 name=SCIPEmitter
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=CapabilityDeclarationGenerator
|
||||
0 0 name=Sprint190IntegrationSummary
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=SecurityTestSkeletonGenerator
|
||||
0 0 name=SchemaToCppGenerator
|
||||
0 0 name=JobDispatchTableGenerator
|
||||
0 0 name=Sprint85IntegrationSummary
|
||||
1 0 name=SymbolSelectorAPI
|
||||
0 0 name=ProjectAstCMakeGenerator
|
||||
0 0 name=BatchMutationAPI
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=TypeAwareMappings
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=Sprint222IntegrationSummary
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=PolyglotSuiteOrchestrator
|
||||
0 0 name=Sprint265IntegrationSummary
|
||||
0 0 name=SyntaxHighlighterOrg
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=TypeAwareMappings
|
||||
1 0 name=AgentChatPanelModel
|
||||
1 0 name=ArchitectModuleDecomposer
|
||||
0 0 name=ContextSliceAssembler
|
||||
0 0 name=SyntaxHighlighterJavaScript
|
||||
1 0 name=TranspilationConfidence
|
||||
1 0 name=ArchitectTechStackSelector
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=Sprint106IntegrationSummary
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=SQLiteDataLayerGenerator
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
0 0 name=Sprint27IntegrationSummary
|
||||
1 0 name=IncrementalOptimizer
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=Sprint98IntegrationSummary
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
1 0 name=OperationSelectorAPI
|
||||
0 0 name=AccessibilityBaseline
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=TraceGenerator
|
||||
0 0 name=CostEstimator
|
||||
1 0 name=AgentPermissionPolicy
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=ConcurrencyTranslator
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=AgentLibraryPolicy
|
||||
0 0 name=SyntaxHighlighterPython
|
||||
0 0 name=JobDispatchTableGenerator
|
||||
0 0 name=EditorModePolicy
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=AgentMarketplace
|
||||
1 0 name=TranspilationConfidence
|
||||
0 0 name=SwarmStatusPanelModel
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=TransformEngine
|
||||
0 0 name=SyntaxHighlighter
|
||||
1 0 name=ArchitectIntakeProcessor
|
||||
1 0 name=PolyglotOrchestrator
|
||||
1 0 name=AgentTaskStatusOverlay
|
||||
1 0 name=ArchitectReviewInterface
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
0 0 name=Sprint184IntegrationSummary
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=Sprint36aIntegration
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=SyntaxHighlighterPython
|
||||
0 0 name=SearchUtils
|
||||
0 0 name=Diagnostics
|
||||
0 0 name=TestPlanGenerator
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=InferenceJobGenerator
|
||||
1 0 name=ArchitectProblemParser
|
||||
0 0 name=LSPSymbolExtractor
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ArchitectProjectPipeline
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=AgentToolCallVisualization
|
||||
0 0 name=ProjectSkeletonGenerator
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
1 0 name=OperationSelectorAPI
|
||||
0 0 name=LanguageGeneratorPluginManifest
|
||||
1 0 name=TransformEngine
|
||||
1 0 name=OnCallCoveragePlanner
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentChatSessionPersistence
|
||||
0 0 name=CHeaderEmitter
|
||||
0 0 name=Sprint49IntegrationSummary
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=ArchitectMultiLanguageOrchestrator
|
||||
0 0 name=MCPWorkflowPrompts
|
||||
1 0 name=TranspilationRPC
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=OperationSelectorAPI
|
||||
0 0 name=Sprint198IntegrationSummary
|
||||
1 0 name=ArchitectScaffoldGenerator
|
||||
0 0 name=GoCppBindingEmitter
|
||||
0 0 name=AnimationUtils
|
||||
0 0 name=SelfHostAnnotationAudit
|
||||
1 0 name=ArchitectIntakeProcessor
|
||||
1 0 name=AgentMarketplace
|
||||
1 0 name=TranspilationConfidence
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=SecureByDefaultGenerator
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=AgentChatPanelModel
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
0 0 name=APIBoundaryPreserver
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=TypeSystemTranslator
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
1 0 name=TransformEngineExtended
|
||||
0 0 name=SyntaxHighlighterRust
|
||||
1 0 name=AgentSessionRecorder
|
||||
0 0 name=Sprint136IntegrationSummary
|
||||
0 0 name=OperationTaxonomy
|
||||
1 0 name=ArchitectTechStackSelector
|
||||
0 0 name=AnnotationConflict
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=Sprint286IntegrationSummary
|
||||
0 0 name=MigrationPlanGenerator
|
||||
0 0 name=Sprint261IntegrationSummary
|
||||
0 0 name=TrainingDataGenerator
|
||||
0 0 name=RustPythonBindingEmitter
|
||||
1 0 name=AgentTaskSlots
|
||||
1 0 name=AgentCodeGen
|
||||
0 0 name=Sprint74IntegrationSummary
|
||||
0 0 name=ArchitectTemplates
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=SyntaxHighlighterJava
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentMutationPreview
|
||||
0 0 name=SQLiteDataLayerGenerator
|
||||
1 0 name=AgentTaskSlots
|
||||
1 0 name=AgentRPCHandler
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=ConcurrencyTranslator
|
||||
1 0 name=ArchitectReviewSurface
|
||||
1 0 name=AgentToolCallVisualization
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=RustGoBindingEmitter
|
||||
0 0 name=Sprint58IntegrationSummary
|
||||
1 0 name=IncrementalOptimizer
|
||||
0 0 name=Sprint239IntegrationSummary
|
||||
0 0 name=SyntaxHighlighterElisp
|
||||
0 0 name=PlatformCMakeTargetGenerator
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=MqttBoilerplateGenerator
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=KeySymbolRenderer
|
||||
1 0 name=OperationSelectorAPI
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=ArchitectSkeletonGenerator
|
||||
1 0 name=OperationClassifier
|
||||
1 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=PackageScriptGenerator
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=TypeAwareMappings
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=GoCppBindingEmitter
|
||||
1 0 name=AgentMutationApproval
|
||||
0 0 name=CHeaderEmitter
|
||||
0 0 name=SyntaxHighlighterOrg
|
||||
0 0 name=TaskitemGeneratorV2
|
||||
0 0 name=Sprint96IntegrationSummary
|
||||
0 0 name=ScoreClampUtil
|
||||
1 0 name=AgentRPCHandler
|
||||
1 0 name=ArchitectScaffoldGenerator
|
||||
1 0 name=ScopeMilestoneDecomposer
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=AgentPermissionPolicy
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
0 0 name=ConstrainedRoutingRulesetExtension
|
||||
0 0 name=LanguageGeneratorPluginManifest
|
||||
1 0 name=ArchitectReviewSurface
|
||||
0 0 name=WorkItem
|
||||
1 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=ConcurrencyTranslator
|
||||
1 0 name=CanaryPromotionJudge
|
||||
1 0 name=WorkflowOrchestrator
|
||||
0 0 name=InferenceJobGenerator
|
||||
1 0 name=TranspilationRPC
|
||||
1 0 name=AgentLibraryPolicy
|
||||
0 0 name=Sprint272IntegrationSummary
|
||||
1 0 name=TransformEngine
|
||||
1 0 name=TranspilationRPC
|
||||
1 0 name=OnCallCoveragePlanner
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=PolyglotOrchestrator
|
||||
1 0 name=OperationClassifier
|
||||
0 0 name=RefactorActions
|
||||
1 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=PerTaskLibrarySelector
|
||||
1 0 name=WorkflowOrchestrator
|
||||
1 0 name=AgentChatContextInjector
|
||||
1 0 name=AgentAnnotationAssistant
|
||||
1 0 name=AgentMutationApproval
|
||||
1 0 name=SymbolSelectorAPI
|
||||
1 0 name=ArchitectReviewInterface
|
||||
0 0 name=SyntaxHighlighterCpp
|
||||
1 0 name=ArchitectMultiLanguageOrchestrator
|
||||
0 0 name=APIDocGenerator
|
||||
1 0 name=ArchitectProblemParser
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
1 0 name=AgentAnnotationAssistant
|
||||
0 0 name=PackageScriptGenerator
|
||||
1 0 name=AgentRegistry
|
||||
1 0 name=ArchitectModuleDecomposer
|
||||
0 0 name=Sprint249IntegrationSummary
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=ArchitectProjectPipeline
|
||||
0 0 name=WorkerDispatchValidator
|
||||
1 0 name=ContextWindowOptimizer
|
||||
1 0 name=PolyglotSuiteOrchestrator
|
||||
1 0 name=AgentMutationPreview
|
||||
1 0 name=TypeSystemTranslator
|
||||
|
29
specialists/data/generated/automatability_routed_eval.tsv
Normal file
29
specialists/data/generated/automatability_routed_eval.tsv
Normal file
@@ -0,0 +1,29 @@
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
1 0 name=IntentTranslator
|
||||
1 0 name=TypeSystemTranslator
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=IncrementalOptimizer
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
3 0 name=ArchitectReviewInterface
|
||||
1 0 name=MemoryModelTranslator
|
||||
2 0 name=AgentCodeGen
|
||||
0 0 name=OperationClassifier
|
||||
0 0 name=SymbolSelectorAPI
|
||||
3 0 name=ArchitectReviewSurface
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
0 0 name=PerTaskLibrarySelector
|
||||
2 0 name=AgentRegistry
|
||||
0 0 name=OperationClassifier
|
||||
3 0 name=AgentMutationApproval
|
||||
3 0 name=AgentMutationPreview
|
||||
1 0 name=TransformEngine
|
||||
0 0 name=OperationClassifier
|
||||
3 0 name=ArchitectReviewSurface
|
||||
0 0 name=OperationSelectorAPI
|
||||
3 0 name=AgentMutationPreview
|
||||
3 0 name=AgentMutationApproval
|
||||
2 0 name=AgentTaskStatusOverlay
|
||||
2 0 name=ScopeMilestoneDecomposer
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
|
179
specialists/data/generated/automatability_routed_train.tsv
Normal file
179
specialists/data/generated/automatability_routed_train.tsv
Normal file
@@ -0,0 +1,179 @@
|
||||
2 0 name=AgentChatContextInjector
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
0 0 name=SymbolSelectorAPI
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=MemoryModelTranslator
|
||||
2 0 name=AgentChatSessionPersistence
|
||||
2 0 name=AgentSessionRecorder
|
||||
1 0 name=IncrementalOptimizer
|
||||
1 0 name=TypeSystemTranslator
|
||||
3 0 name=ArchitectReviewInterface
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=SymbolSelectorAPI
|
||||
0 0 name=CanaryPromotionJudge
|
||||
3 0 name=ArchitectReviewSurface
|
||||
1 0 name=TypeAwareMappings
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
1 0 name=TransformEngineExtended
|
||||
3 0 name=ArchitectReviewSurface
|
||||
3 0 name=ArchitectReviewInterface
|
||||
3 0 name=ArchitectReviewSurface
|
||||
2 0 name=PolyglotSuiteOrchestrator
|
||||
1 0 name=TransformEngineExtended
|
||||
1 0 name=TypeAwareMappings
|
||||
2 0 name=AgentChatPanelModel
|
||||
2 0 name=ArchitectModuleDecomposer
|
||||
1 0 name=TranspilationConfidence
|
||||
2 0 name=ArchitectTechStackSelector
|
||||
0 0 name=PerTaskLibrarySelector
|
||||
3 0 name=ArchitectReviewInterface
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=OperationClassifier
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
1 0 name=IncrementalOptimizer
|
||||
3 0 name=AgentMutationApproval
|
||||
0 0 name=OperationSelectorAPI
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
0 0 name=OperationSelectorAPI
|
||||
3 0 name=AgentMutationApproval
|
||||
2 0 name=AgentPermissionPolicy
|
||||
3 0 name=AgentMutationPreview
|
||||
1 0 name=ConcurrencyTranslator
|
||||
1 0 name=SelfHostTranspileAudit
|
||||
1 0 name=ContextWindowOptimizer
|
||||
3 0 name=AgentMutationPreview
|
||||
1 0 name=IntentTranslator
|
||||
2 0 name=AgentLibraryPolicy
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
2 0 name=AgentMarketplace
|
||||
1 0 name=TranspilationConfidence
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=TransformEngine
|
||||
2 0 name=ArchitectIntakeProcessor
|
||||
2 0 name=PolyglotOrchestrator
|
||||
2 0 name=AgentTaskStatusOverlay
|
||||
3 0 name=ArchitectReviewInterface
|
||||
0 0 name=OperationSelectorAPI
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
3 0 name=ArchitectReviewInterface
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
2 0 name=ArchitectProblemParser
|
||||
0 0 name=SymbolSelectorAPI
|
||||
2 0 name=ArchitectProjectPipeline
|
||||
0 0 name=PerTaskLibrarySelector
|
||||
2 0 name=AgentToolCallVisualization
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
0 0 name=CanaryPromotionJudge
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=OperationSelectorAPI
|
||||
1 0 name=TransformEngine
|
||||
2 0 name=OnCallCoveragePlanner
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
3 0 name=AgentMutationPreview
|
||||
2 0 name=AgentChatSessionPersistence
|
||||
1 0 name=ContextWindowOptimizer
|
||||
0 0 name=PerTaskLibrarySelector
|
||||
2 0 name=ArchitectMultiLanguageOrchestrator
|
||||
1 0 name=TranspilationRPC
|
||||
1 0 name=ContextWindowOptimizer
|
||||
0 0 name=OperationSelectorAPI
|
||||
2 0 name=ArchitectScaffoldGenerator
|
||||
2 0 name=ArchitectIntakeProcessor
|
||||
2 0 name=AgentMarketplace
|
||||
1 0 name=TranspilationConfidence
|
||||
3 0 name=ArchitectReviewSurface
|
||||
0 0 name=OperationSelectorAPI
|
||||
0 0 name=CanaryPromotionJudge
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
2 0 name=AgentChatPanelModel
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
1 0 name=TypeSystemTranslator
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
1 0 name=TransformEngineExtended
|
||||
2 0 name=AgentSessionRecorder
|
||||
2 0 name=ArchitectTechStackSelector
|
||||
0 0 name=CanaryPromotionJudge
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
2 0 name=AgentTaskSlots
|
||||
2 0 name=AgentCodeGen
|
||||
0 0 name=SymbolSelectorAPI
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=CanaryPromotionJudge
|
||||
3 0 name=AgentMutationApproval
|
||||
3 0 name=AgentMutationPreview
|
||||
3 0 name=AgentMutationPreview
|
||||
2 0 name=AgentTaskSlots
|
||||
2 0 name=AgentRPCHandler
|
||||
1 0 name=MemoryModelTranslator
|
||||
1 0 name=ConcurrencyTranslator
|
||||
3 0 name=ArchitectReviewSurface
|
||||
2 0 name=AgentToolCallVisualization
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
1 0 name=IncrementalOptimizer
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=PerTaskLibrarySelector
|
||||
1 0 name=IntentTranslator
|
||||
3 0 name=SecurityExceptionReviewBoard
|
||||
0 0 name=OperationSelectorAPI
|
||||
1 0 name=MemoryModelTranslator
|
||||
3 0 name=ArchitectReviewInterface
|
||||
0 0 name=OperationClassifier
|
||||
3 0 name=ApprovalEscalationPlanner
|
||||
0 0 name=OperationClassifier
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
1 0 name=TypeAwareMappings
|
||||
3 0 name=ArchitectReviewInterface
|
||||
3 0 name=AgentMutationApproval
|
||||
2 0 name=AgentRPCHandler
|
||||
2 0 name=ArchitectScaffoldGenerator
|
||||
2 0 name=ScopeMilestoneDecomposer
|
||||
3 0 name=AgentMutationApproval
|
||||
3 0 name=AgentMutationPreview
|
||||
2 0 name=AgentPermissionPolicy
|
||||
0 0 name=PerTaskLibrarySelector
|
||||
3 0 name=ArchitectReviewSurface
|
||||
0 0 name=TaskitemLibraryAnnotator
|
||||
0 0 name=CanaryPromotionJudge
|
||||
3 0 name=AgentMutationApproval
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=ConcurrencyTranslator
|
||||
0 0 name=CanaryPromotionJudge
|
||||
2 0 name=WorkflowOrchestrator
|
||||
1 0 name=TranspilationRPC
|
||||
2 0 name=AgentLibraryPolicy
|
||||
1 0 name=TransformEngine
|
||||
1 0 name=TranspilationRPC
|
||||
2 0 name=OnCallCoveragePlanner
|
||||
0 0 name=SymbolSelectorAPI
|
||||
2 0 name=PolyglotOrchestrator
|
||||
0 0 name=OperationClassifier
|
||||
0 0 name=LibrarySymbolAdvisor
|
||||
1 0 name=IntentTranslator
|
||||
0 0 name=PerTaskLibrarySelector
|
||||
2 0 name=WorkflowOrchestrator
|
||||
2 0 name=AgentChatContextInjector
|
||||
2 0 name=AgentAnnotationAssistant
|
||||
3 0 name=AgentMutationApproval
|
||||
0 0 name=SymbolSelectorAPI
|
||||
3 0 name=ArchitectReviewInterface
|
||||
2 0 name=ArchitectMultiLanguageOrchestrator
|
||||
2 0 name=ArchitectProblemParser
|
||||
1 0 name=StrategyAwareOptimizer
|
||||
2 0 name=AgentAnnotationAssistant
|
||||
2 0 name=AgentRegistry
|
||||
2 0 name=ArchitectModuleDecomposer
|
||||
1 0 name=SecurityPreservingTranslation
|
||||
2 0 name=ArchitectProjectPipeline
|
||||
1 0 name=ContextWindowOptimizer
|
||||
2 0 name=PolyglotSuiteOrchestrator
|
||||
3 0 name=AgentMutationPreview
|
||||
1 0 name=TypeSystemTranslator
|
||||
|
115
specialists/data/generated/worker_type_combined_eval.tsv
Normal file
115
specialists/data/generated/worker_type_combined_eval.tsv
Normal file
@@ -0,0 +1,115 @@
|
||||
0 0 Snapshot persistence (sidecar integration)
|
||||
1 0 `whetstone_preview_regeneration_diff_generic` MCP tool
|
||||
2 0 Phase 25a Integration
|
||||
3 0 Sprint 116 integration summary + regression
|
||||
0 0 Debug assist packet bundle model
|
||||
1 0 `whetstone_score_failure_triage` MCP tool
|
||||
3 0 Regression watchlist for promoted pairs
|
||||
2 0 Architecture Templates
|
||||
3 0 Phase 10d Integration Tests
|
||||
2 0 Phase 21a Integration
|
||||
2 0 Result Acceptance Protocol
|
||||
2 0 Guided transpilation exercise framework
|
||||
0 0 Patch proposal schema and provenance model
|
||||
2 0 `whetstone_publish_roadmap_epoch` MCP tool
|
||||
0 0 Compliance policy binding engine
|
||||
2 0 Workflow Session Protocol
|
||||
3 0 Sprint 181 Integration Summary
|
||||
2 0 Language-specific method body scaffolding for core patterns
|
||||
0 0 Sprint 27 Integration + Summary
|
||||
3 0 Phase 15b Integration — Full Protocol Test
|
||||
0 0 Long-range edit graph planner
|
||||
3 0 Sprint 120 integration summary + regression
|
||||
0 0 Federated governance report generator
|
||||
3 0 Sprint 66 integration summary + regression
|
||||
3 0 Sprint 204 Integration Summary
|
||||
2 0 Phase 27a Integration
|
||||
1 0 Review Gates
|
||||
3 0 Sprint 55 integration summary + regression
|
||||
0 0 LTS baseline criteria schema
|
||||
0 0 Replay comparison and divergence classification runtime model
|
||||
3 0 Sprint 101 integration summary + regression
|
||||
2 0 Retry/Escalation Protocol for Constraint Failures
|
||||
0 0 Compatibility policy profiles by interface criticality
|
||||
0 0 Language support tiers + gates
|
||||
2 0 Phase 13d Integration + Sprint 13 Summary
|
||||
1 0 Deterministic triage queue planner model
|
||||
1 0 Unmappable semantic blocklist + mandatory review gate
|
||||
2 0 Phase 23a Integration
|
||||
1 0 `whetstone_preview_text_ast_merge_generic` MCP tool
|
||||
3 0 Sprint 132 integration summary + regression
|
||||
2 0 Phase 27a Integration
|
||||
3 0 Sprint 71 integration summary + regression
|
||||
2 0 Phase 14b Integration
|
||||
0 0 Epoch-2 workstream schema and ownership map
|
||||
3 0 Sprint 141 integration summary + regression
|
||||
3 0 AST-native projection benchmark suite
|
||||
0 0 Certification evidence archive format v2
|
||||
2 0 Phase 29a Integration
|
||||
0 0 Tier promotion engine (beta->stable)
|
||||
1 0 Quarterly lock/review cadence policy
|
||||
1 0 Diagnostic normalization runtime implementation model
|
||||
2 0 Phase 22a Integration
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Plan Build a streaming join worker correlating click and | Build a streaming join worker correlating click and conversion events with watermark logic.
|
||||
0 0 Define verification and readiness surface | Build a live notifications hub with subscription filters, fanout, and reconnect replay support.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build a secrets rotation scheduler with lease tracking, retry plans, and notifications.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
0 0 Define verification and readiness surface | Build kanban API with boards, columns, cards, drag reorder, and activity history.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Plan Build an IoT telemetry gateway supporting MQTT ingestion | Build an IoT telemetry gateway supporting MQTT ingestion, validation, and stream forwarding.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a CLI todo manager with add/list/done/delete commands and JSON file storage.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a whiteboard sync backend with operational ordering and conflict-safe patch application.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
0 0 Define verification and readiness surface | Build an image conversion CLI with resize, format conversion, and batch mode support.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Plan Split monolithic endpoint into read/write paths while preserving | Split monolithic endpoint into read/write paths while preserving latency and memory budgets and updating observability d
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
3 0 Define verification and readiness surface | Build portfolio tracker with holdings, transactions, PnL snapshots, and benchmark comparison.
|
||||
0 0 Define verification and readiness surface | Build checkout backend coordinating address, shipping options, totals, and payment intent creation.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
0 0 Define verification and readiness surface | Build chat bot backend with slash commands, permission checks, and scheduled digests.
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build a contacts service with import/export, deduplication, and fuzzy name search.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Define verification and readiness surface | Build realtime polling backend with vote dedupe, room join, and result broadcast.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
0 0 Plan Upgrade API contract from v1 to v2 backend | Upgrade API contract from v1 to v2: backend handlers, OpenAPI schema, generated client SDK, frontend integration, and co
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
0 0 Plan Build a streaming detector with keyed state windows | Build a streaming detector with keyed state windows, out-of-order event handling, watermark policy, and deterministic re
|
||||
0 0 Define verification and readiness surface | Build a release notes generator from commits, PR labels, and issue references.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
0 0 Define verification and readiness surface | Build a change-data-capture processor that applies upserts/deletes and checkpoints offsets.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Define verification and readiness surface | Build a model registry API supporting stages, approvals, rollback, and signed artifact links.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
|
661
specialists/data/generated/worker_type_combined_train.tsv
Normal file
661
specialists/data/generated/worker_type_combined_train.tsv
Normal file
@@ -0,0 +1,661 @@
|
||||
1 0 `whetstone_get_sync_diagnostics` MCP tool
|
||||
3 0 Regression fixtures for under-constrained vs capability-limited cases
|
||||
3 0 Sprint 83 integration summary + regression
|
||||
1 0 Audit trail and inspection
|
||||
3 0 Sprint 99 integration summary + regression
|
||||
3 0 Security Testing Skeleton Generation
|
||||
2 0 Phase 25b Integration
|
||||
3 0 Sprint 115 integration summary + regression
|
||||
0 0 Feedback Loop — Rejection Re-Routing
|
||||
2 0 Long-horizon roadmap planning engine v2
|
||||
2 0 Retry/Escalation Protocol for Constraint Failures
|
||||
0 0 `BuildErrorRecord` schema
|
||||
2 0 Phase 26a Integration
|
||||
3 0 Sprint 179 Integration Summary
|
||||
0 0 Policy-compliance auto-checkpoint integration
|
||||
2 0 Refactor grammar generator to use normalized schema IR
|
||||
3 0 Intake-to-Queue Simulation Harness
|
||||
3 0 Sprint 50 integration summary + regression
|
||||
1 0 Human review queue generator from ambiguities
|
||||
2 0 Scaffold File Generation
|
||||
3 0 Sprint 85 integration summary + regression
|
||||
0 0 `whetstone_get_swarm_maintenance_status` MCP tool
|
||||
3 0 Sprint 157 integration summary + regression
|
||||
3 0 Full historical regression + post-release checklist
|
||||
1 0 `whetstone_normalize_diagnostics` runtime implementation
|
||||
3 0 End-to-end — Python PriorityQueue → C++, Rust, Go
|
||||
2 0 Phase 12d Integration + Sprint 12 Summary
|
||||
1 0 False-green diagnostic class
|
||||
3 0 Sprint 86 integration summary + regression
|
||||
3 0 Sprint 64 integration summary + regression
|
||||
3 0 Sprint 142 integration summary + regression
|
||||
1 0 Priority-focused review queue shaping engine
|
||||
1 0 Efficiency diagnostics and trend report
|
||||
0 0 Context slice canonicalization for handoffs
|
||||
0 0 Context Bundle Format
|
||||
3 0 Reference implementation harness for IR/evidence specs
|
||||
1 0 Constraint completeness diagnostics
|
||||
1 0 Porting review board model
|
||||
0 0 Maintenance demand estimator
|
||||
0 0 `whetstone_get_program_kpis` MCP tool
|
||||
1 0 Tool-surface diagnostics report artifact
|
||||
3 0 Rust runner adapter for harness
|
||||
2 0 `whetstone_verify_architecture_consistency` MCP tool
|
||||
1 0 Security scan orchestrator for generated targets
|
||||
1 0 Strictness audit utility and baseline report
|
||||
0 0 Sprint pipeline summary enrichment
|
||||
0 0 C Parser — Functions, Structs, Enums
|
||||
1 0 Call Stack and Frame Inspector
|
||||
3 0 Sprint 74 integration summary + regression
|
||||
2 0 Phase 25b Integration
|
||||
0 0 Binary/interface diff engine integration
|
||||
3 0 Sprint 125 integration summary + regression
|
||||
1 0 Reviewer quorum policy model for critical approvals
|
||||
1 0 Conflict preview and impact estimator model
|
||||
3 0 Test failure classifier
|
||||
3 0 Benchmark ingestion support for projection schema
|
||||
0 0 Artifact quality scoring and ranking engine
|
||||
3 0 Sprint 47 integration summary + regression
|
||||
0 0 `whetstone_assemble_context` MCP tool
|
||||
0 0 Legacy modernization dossier generator
|
||||
0 0 Step Engine
|
||||
1 0 Agent role contract schema (`analyzer`, `patcher`, `verifier`, `reviewer`)
|
||||
0 0 `whetstone_trigger_porting_incident_drill` MCP tool
|
||||
1 0 Lint diagnostics normalization
|
||||
1 0 AST mutation preview
|
||||
2 0 Burn-down planning optimizer
|
||||
1 0 KPI delta analysis engine for block review
|
||||
2 0 Long-horizon roadmap planning engine v2
|
||||
0 0 `whetstone_rollback_constructive_transaction` MCP tool
|
||||
2 0 Deterministic blocker taxonomy
|
||||
1 0 Docking Reliability Audit + Deterministic Layout Rules
|
||||
0 0 `whetstone_run_build_iteration` MCP tool
|
||||
1 0 Divergence triage model for interop failures
|
||||
2 0 Phase 36a Integration
|
||||
0 0 Language capability conformance report artifact
|
||||
3 0 `whetstone_run_pair_benchmark` MCP tool
|
||||
1 0 `whetstone_optimize_review_queue` MCP tool
|
||||
3 0 Sprint 205 Integration Summary
|
||||
1 0 Loop summary contract diagnostics
|
||||
2 0 Phase 13a Integration — Layout Persistence
|
||||
3 0 Sprint 189 Integration Summary
|
||||
2 0 Continuity drill framework and scoring
|
||||
2 0 Deterministic blocker taxonomy extension
|
||||
3 0 Cross-team benchmark comparator
|
||||
0 0 VB.NET lowering/raising adapters
|
||||
1 0 Review-Gate Policy Refinement
|
||||
2 0 Phase 14d Integration + Sprint 14 Summary
|
||||
0 0 Multi-language compile gate executor
|
||||
1 0 Dependency and supply-chain audit packet
|
||||
1 0 Reviewer-focused semantic diff summarizer
|
||||
1 0 Architect Review Surface for Intake
|
||||
2 0 Phase 24a Integration
|
||||
2 0 Long-range blocker taxonomy
|
||||
2 0 Result Acceptance Protocol
|
||||
1 0 Patch proposal review report artifact
|
||||
1 0 Parity diagnostics export
|
||||
3 0 Debug benchmark harness for fixture suites
|
||||
1 0 Triage packet bundle model
|
||||
1 0 Incremental sync diagnostics model
|
||||
2 0 Swarm maintenance job decomposition model
|
||||
0 0 Epoch kickoff report artifact
|
||||
3 0 Sprint 96 integration summary + regression
|
||||
2 0 Phase 32a Integration
|
||||
2 0 Self-Hosting Progress Report + Phase Integration
|
||||
2 0 Phase 13a Integration — Layout Persistence
|
||||
0 0 Sprint 28 Integration + Summary
|
||||
2 0 Migration Plan Generator
|
||||
2 0 Phase 20a Integration
|
||||
3 0 End-to-end replay + rollback scenario suite
|
||||
3 0 C++ runner adapter for harness
|
||||
1 0 Safety Audit via Annotations
|
||||
2 0 Phase 33a Integration
|
||||
3 0 Sprint 79 integration summary + regression
|
||||
0 0 Lisp lowering/raising adapters
|
||||
0 0 `whetstone_run_porting_gates` MCP tool
|
||||
1 0 Explainability compression model for rapid review
|
||||
2 0 Phase 32a Integration
|
||||
3 0 Sprint 51 integration summary + regression
|
||||
3 0 Sprint 175 Integration Summary
|
||||
3 0 Sprint 146 integration summary + regression
|
||||
2 0 Deterministic blocker taxonomy extension
|
||||
3 0 Sprint 139 integration summary + regression
|
||||
3 0 Sprint 90 integration summary + full-program regression
|
||||
3 0 Benchmark publication bundle
|
||||
2 0 Transpilation failure taxonomy schema
|
||||
3 0 Sprint 138 integration summary + regression
|
||||
1 0 Architect Review Interface
|
||||
1 0 Real adapter diagnostics in status tool
|
||||
0 0 Kotlin Parser
|
||||
1 0 Remediation router from gate diagnostics to tool actions
|
||||
0 0 Family promotion checks (experimental->beta)
|
||||
2 0 Phase 16a Integration
|
||||
3 0 Sprint 73 integration summary + regression
|
||||
2 0 Phase 13d Integration + Sprint 13 Summary
|
||||
0 0 AnnotationValidator — Subject 5-8 Rules
|
||||
0 0 Deterministic replay pack composer
|
||||
1 0 Assurance review dossier template
|
||||
3 0 Auto-tuning proposal generator from benchmark deltas
|
||||
1 0 `whetstone_get_distributed_triage_queue` MCP tool
|
||||
1 0 KPI anomaly triage model
|
||||
2 0 Phase 36a Integration
|
||||
2 0 Module Decomposition Engine
|
||||
1 0 Scope ambiguity diagnostics
|
||||
2 0 Phase 38a Integration
|
||||
1 0 Governance and audit report bundle
|
||||
2 0 Phase 13b Integration — Visual Consistency
|
||||
3 0 CI integration in build/test flow
|
||||
0 0 Zero-trust policy model for execution surfaces
|
||||
1 0 Patch execution audit bundle
|
||||
2 0 Phase 30a Integration
|
||||
1 0 Drift audit report artifact
|
||||
0 0 `whetstone_marketplace_search` MCP tool
|
||||
3 0 Cross-language class emission parity tests
|
||||
3 0 Test output parser — gtest format
|
||||
0 0 `whetstone_get_debug_campaign_status` MCP tool
|
||||
1 0 Dynamic family acceptance report + review queue wiring
|
||||
3 0 Sprint 191 Integration Summary
|
||||
2 0 Generator Updates + Phase 11c Integration
|
||||
1 0 Review effort estimator from ambiguity/risk packets
|
||||
3 0 Sprint 152 integration summary + regression
|
||||
2 0 Candidate generation engine from failure taxonomy
|
||||
0 0 Ambiguity packet model (`assumed`, `unknown`, `conflict`)
|
||||
3 0 Sprint 198 Integration Summary
|
||||
1 0 Memory Inspector UI Model
|
||||
0 0 Argument/result canonicalizer model
|
||||
3 0 Sprint 89 integration summary + regression
|
||||
1 0 Divergence classification and triage model
|
||||
0 0 `whetstone_run_constructive_loop` MCP tool
|
||||
1 0 Dependency Audit Annotations
|
||||
3 0 Benchmark runner and metrics export
|
||||
3 0 Sprint 62 integration summary + regression
|
||||
1 0 Review Comparison View
|
||||
1 0 Review SLA policy integration
|
||||
0 0 Failure injection and safety validation model
|
||||
1 0 Intent drift diagnostics export
|
||||
3 0 Sprint 186 Integration Summary
|
||||
0 0 Auto Type Deduction
|
||||
2 0 Scope and Milestone Decomposer
|
||||
2 0 Phase 25a Integration
|
||||
1 0 Constraint Violation Diagnostics
|
||||
2 0 Refactor pattern schema and metadata model
|
||||
1 0 `whetstone_normalize_diagnostics` MCP tool
|
||||
0 0 Replay/transaction reliability report artifact
|
||||
1 0 Reviewer effectiveness report artifact
|
||||
1 0 Post-incident continuity audit report model
|
||||
1 0 Entropy scanner — editor-side
|
||||
2 0 Phase 13c Integration — Keyboard Shortcuts Help Panel
|
||||
3 0 Test gate runner and minimal harness injection
|
||||
2 0 Backend framework adapter hooks
|
||||
0 0 Enhanced evidence requirements engine
|
||||
1 0 `whetstone_preview_regenerated_diff` MCP tool
|
||||
1 0 Hint safety guardrails and audit fields
|
||||
2 0 Refactor intent annotation model
|
||||
3 0 Sprint 137 integration summary + regression
|
||||
1 0 Reporting templates for C++ review teams
|
||||
0 0 Generic/monomorphization intent model
|
||||
0 0 `whetstone_compare_mcp_replays` MCP tool
|
||||
0 0 Memory Model Translation
|
||||
0 0 Schema evolution compatibility checker
|
||||
0 0 Closeout consistency gate
|
||||
0 0 Real-time constraint model (`deadline`, `jitter`, `period`)
|
||||
2 0 Workflow Session Protocol
|
||||
0 0 Include/import ordering normalization model
|
||||
2 0 Deprecation/succession lifecycle framework
|
||||
0 0 Trace Timeline Model
|
||||
1 0 Decision consistency analyzer across reviewers
|
||||
1 0 Profile compliance auditor integration
|
||||
0 0 `whetstone_plan_rollout_stage` MCP tool
|
||||
0 0 Top-level strictness gate
|
||||
0 0 Strict queue gate for execution contracts
|
||||
3 0 Sprint 58 integration summary + regression
|
||||
3 0 Sprint 48 integration summary + regression
|
||||
0 0 `whetstone_detect_text_ast_conflicts_generic` MCP tool
|
||||
2 0 Phase 19a Integration
|
||||
1 0 `whetstone_get_review_load_status` MCP tool
|
||||
3 0 Phase 19b Integration + Sprint Summary
|
||||
3 0 Sprint 200 Integration Summary
|
||||
0 0 Async/await state intent lowering
|
||||
1 0 Failure triage score model
|
||||
0 0 Plan fingerprinting and replay checks
|
||||
1 0 MCP replay diagnostics artifact
|
||||
0 0 Plugin compatibility resolver (core/API/version)
|
||||
1 0 Diagnostic normalization canonical model
|
||||
3 0 Test-to-source locator
|
||||
0 0 Debug action budget model
|
||||
2 0 Scaffold File Generation
|
||||
1 0 Canonical diagnostic normalization from provider/LSP output
|
||||
2 0 Backend framework adapter hooks
|
||||
3 0 Sprint 162 Integration Summary
|
||||
2 0 Phase 39a Integration
|
||||
0 0 ADT + pattern matching canonical lowering
|
||||
2 0 Verification gate integration for architecture artifacts
|
||||
0 0 Modernization RPC + MCP
|
||||
2 0 Phase 19a Integration
|
||||
3 0 Convention Extractor + Validator (tests: unit, negative, regression)
|
||||
1 0 `whetstone_preview_text_ast_merge` MCP tool
|
||||
0 0 ABI gate bindings to tier promotions
|
||||
2 0 Architecture claim linkage engine
|
||||
0 0 Permission conformance report artifact
|
||||
2 0 Phase 20a Integration
|
||||
2 0 Governance model refresh framework
|
||||
1 0 Sync diagnostics normalization model
|
||||
3 0 Sprint 63 integration summary + regression
|
||||
0 0 `whetstone_suggest_build_fix` MCP tool
|
||||
3 0 A/B + benchmark validation artifact update
|
||||
0 0 `whetstone_plan_tool_migrations` MCP tool
|
||||
3 0 Full regression + architecture gate
|
||||
0 0 C interop adapter deepening
|
||||
3 0 Phase 10e Integration Tests
|
||||
2 0 Phase 29a Integration
|
||||
2 0 Scope and Milestone Decomposer
|
||||
0 0 Multi-check gate orchestrator model
|
||||
1 0 Lint executor integration
|
||||
3 0 Sprint 118 integration summary + regression
|
||||
1 0 Dropped-content diagnostics packet
|
||||
0 0 Execution trace and recovery packet export
|
||||
2 0 Phase 18a Integration
|
||||
3 0 Sprint 45 Integration Summary
|
||||
2 0 Deterministic blocker taxonomy
|
||||
2 0 Refactor pattern schema and metadata model
|
||||
1 0 Reviewer decision ledger integration
|
||||
2 0 Phase 28a Integration
|
||||
2 0 Phase 37a Integration
|
||||
3 0 Sprint 114 integration summary + regression
|
||||
2 0 `whetstone_publish_roadmap_epoch` MCP tool
|
||||
1 0 `whetstone_review_porting_decision` MCP tool
|
||||
2 0 Language-specific method body scaffolding for core patterns
|
||||
1 0 Audit trail completeness checker
|
||||
1 0 C++ include fixer from gate diagnostics
|
||||
3 0 End-to-end capability inventory and gap scan
|
||||
1 0 Static registry scanner for `register*Tools` blocks
|
||||
2 0 Phase 31a Integration
|
||||
1 0 `whetstone_rank_failure_triage_actions` MCP tool
|
||||
3 0 Timing regression runner integration
|
||||
2 0 Phase 40a Integration
|
||||
1 0 Reviewer load and fatigue model
|
||||
2 0 Swarm maintenance job decomposition model
|
||||
2 0 Target projection blocker taxonomy
|
||||
0 0 Annotate Whetstone's Own Code
|
||||
2 0 Phase 15c Integration + Sprint 15 Summary
|
||||
0 0 Epoch block health re-baseline model
|
||||
3 0 Sprint 159 integration summary + regression
|
||||
1 0 Human Review Interface via MCP
|
||||
0 0 Transpile Whetstone Modules
|
||||
0 0 Capability matrix model
|
||||
2 0 Phase 14d Integration + Sprint 14 Summary
|
||||
3 0 Prior-Step Context Injector (tests: unit, integration)
|
||||
0 0 Unsafe-practice detection and quarantine
|
||||
3 0 Sprint 127 integration summary + regression
|
||||
1 0 `TaskitemQualityAuditor` — batch report
|
||||
3 0 Systemic regression detector
|
||||
0 0 Debug campaign spec model
|
||||
3 0 `whetstone_get_execution_attestation` MCP tool
|
||||
3 0 Sprint 72 integration summary + regression
|
||||
2 0 Self-Hosting Progress Report + Phase Integration
|
||||
0 0 C# Parser
|
||||
1 0 Ambiguity triage UI model and API
|
||||
0 0 F# Parser
|
||||
0 0 Result-to-ObservationRecord mapper
|
||||
0 0 Self-service escalation handoff hooks
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build job queue service with delayed jobs, retries, visibility timeout, and metrics.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Define verification and readiness surface | Build a CLI password vault with encrypted local storage and master-key unlock.
|
||||
0 0 Define verification and readiness surface | Build order management with status machine, cancellation rules, and shipment events.
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build a registry mirror service with digest verification, cache pruning, and pull-through.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
0 0 Define verification and readiness surface | Build a websocket chat server with rooms, presence, typing indicators, and message history.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
0 0 Define verification and readiness surface | Build subscription billing engine with proration, trials, coupons, and failed-payment retries.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
0 0 Define verification and readiness surface | Build invoice generator with tax calculation, due schedules, and PDF export metadata.
|
||||
0 0 Define verification and readiness surface | Build a webhook dispatcher with retries, signature verification, dead-letter queue, and backoff.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Plan Build a ticker stream service with topic subscriptions | Build a ticker stream service with topic subscriptions and throttled update delivery.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a CLI log tailer with multi-file follow, regex filters, and output highlighting.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a CSV analyzer CLI with schema inference, summary stats, and filter expressions.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build returns API with RMA creation, restock policy, and refund eligibility checks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a service monitor that runs health checks, tracks SLO windows, and pages on breaches.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build expense backend with receipt OCR placeholders, category budgets, and recurring expenses.
|
||||
0 0 Define verification and readiness surface | Build a harness that transpiles class-heavy source modules across multiple target languages and validates class/method p
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
0 0 Define verification and readiness surface | Build matchmaking service with queue buckets, skill-based pairing, and timeout fallback.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
0 0 Plan Build an incremental snapshot builder that compacts event | Build an incremental snapshot builder that compacts event logs into query-optimized state.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
0 0 Define verification and readiness surface | Build a REST Todo API with CRUD, pagination, filtering by status, and SQLite persistence.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
0 0 Plan Add normalized address schema migrate legacy rows with | Add normalized address schema, migrate legacy rows with backfill job, dual-read/write transition, and rollback script.
|
||||
0 0 Define verification and readiness surface | Build a log enrichment batch job joining geo data and user agents with partitioned output.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
0 0 Define verification and readiness surface | Build a URL shortener API with custom aliases, redirect tracking, and expiration support.
|
||||
0 0 Define verification and readiness surface | Build meeting notes API with templates, attendee action items, and follow-up reminders.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build cart service with item merge, discount application, and inventory reservation integration.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build session store service with token revocation, idle timeout, and concurrent login policy.
|
||||
0 0 Define verification and readiness surface | Build a secret rotation controller with staged rollout, canary verification, rollback triggers, and explicit blast-radiu
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Plan shared sync core plus Android/Desktop/Web projections with platform-specific | Implement shared sync core plus Android/Desktop/Web projections with platform-specific persistence and transport adapter
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Define verification and readiness surface | Build LLM guardrail service for policy checks, PII filtering, and response classification.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a multi-tenant limiter with token bucket + sliding window hybrid policy, burst fairness guarantees, and lock-free
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a Kubernetes manifest linter enforcing probes, limits, and namespace policy.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
0 0 Define verification and readiness surface | Build an API for file metadata indexing, search by attributes, and soft delete recovery.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build experiment tracking backend storing params, metrics, artifacts, and compare queries.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build user profile service with avatars, privacy settings, and profile completeness scoring.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
0 0 Define verification and readiness surface | Build batch inference pipeline reading partitions, loading model, and writing scored outputs.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build a JWT auth service with register, login, refresh token rotation, and password hashing.
|
||||
0 0 Define verification and readiness surface | Build notification preferences API with channel rules, quiet hours, and topic subscriptions.
|
||||
0 0 Plan Build a CI orchestrator service triggering jobs tracking | Build a CI orchestrator service triggering jobs, tracking states, and emitting summaries.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build a harness that transpiles class-heavy source modules across multiple target languages and validates class/method p
|
||||
0 0 Plan telemetry agent for linux arduino targets using shared | Implement telemetry agent for linux + arduino targets using shared serialization core and target-specific runtime wrappe
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build prompt evaluation runner with rubric scoring, aggregate metrics, and reproducible reports.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a feature store service with online/offline reads and point-in-time correctness.
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build wiki backend with page versions, backlinks, and markdown rendering pipeline hooks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build drift detection worker comparing live feature distributions against reference baselines.
|
||||
0 0 Define verification and readiness surface | Build form builder backend with schema validation, submissions, and analytics counters.
|
||||
0 0 Define verification and readiness surface | Build email campaign manager with lists, segmentation, scheduling, and bounce tracking.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Define verification and readiness surface | Build an access gateway that evaluates hierarchical allow/deny policies with shadow rules, policy versioning, and explai
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a presence tracker with heartbeat timeout, status transitions, and shard-safe updates.
|
||||
0 0 Define verification and readiness surface | Build payments API with idempotency keys, ledger-safe transfers, and webhook callbacks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Plan Build a task runner CLI that reads YAML | Build a task runner CLI that reads YAML tasks and executes dependency-aware command graphs.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build configuration service with versioned configs, staged rollout, and validation checks.
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
0 0 Define verification and readiness surface | Build warehouse picking service with wave assignment, pick confirmation, and shortage handling.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build shipping label worker integrating carrier rates, label generation, and tracking updates.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a metrics aggregation worker with tumbling windows, late data handling, and rollups.
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build backup retention manager with policy tiers, legal hold support, and deletion safety checks.
|
||||
0 0 Plan Build reconciliation service matching ledger entries to bank | Build reconciliation service matching ledger entries to bank statements with mismatch reporting.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build an ETL pipeline that ingests orders CSV, validates schema, and loads curated tables.
|
||||
0 0 Define verification and readiness surface | Build search suggest API with prefix index, typo tolerance, and popularity boosting.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build a feature flag API with environments, targeting rules, and rollout percentages.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
0 0 Plan Replace search ranking pipeline behind feature flag with | Replace search ranking pipeline behind feature flag with staged rollout, telemetry checks, and automatic abort trigger.
|
||||
0 0 Define verification and readiness surface | Build a training scheduler with queue priorities, resource quotas, and retry policies.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
0 0 Define verification and readiness surface | Build a gateway that validates incoming payloads against versioned contracts, performs semantic constraint checks, and e
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a backup CLI with include/exclude rules, incremental snapshots, and restore command.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
0 0 Define verification and readiness surface | Build pricing engine with currency handling, promotions, and time-bound campaign rules.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
0 0 Define verification and readiness surface | Build an event deduplicator using idempotency keys, time windows, and replay-safe writes.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Define verification and readiness surface | Build dataset version management with lineage metadata, immutable snapshots, and diff operations.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
0 0 Define verification and readiness surface | Build tax report backend aggregating gains/losses and generating filing-year summaries.
|
||||
0 0 Define verification and readiness surface | Build distributed rate limiter with fixed/sliding window modes and policy APIs.
|
||||
0 0 Define verification and readiness surface | Build telemetry collector with trace/span ingestion, sampling, and export adapters.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | Build a CRDT-based collaborative notes backend with causal metadata, merge convergence proofs, tombstone compaction, and
|
||||
0 0 Define verification and readiness surface | Build a loader from object storage to warehouse with schema evolution and retry checkpoints.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
0 0 Define verification and readiness surface | Build access control service with RBAC roles, permissions, and resource-scoped grants.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
3 0 Define verification and readiness surface | Refactor billing pipeline to introduce idempotency keys and rollback checkpoints across ledger service, queue worker, re
|
||||
0 0 Define verification and readiness surface | Build a CDC replication worker that handles schema evolution, backfill merges, checkpoint consistency, and deterministic
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
0 0 Plan Build drift detection comparing desired and observed infra | Build drift detection comparing desired and observed infra states with actionable diff reports.
|
||||
1 0 Define verification and readiness surface | Build a ledger reconciliation service with strict double-entry invariants, idempotent replay, rollback checkpoints, and
|
||||
0 0 Define verification and readiness surface | Build product catalog API with variants, facets, search indexing hooks, and soft delete.
|
||||
0 0 Define verification and readiness surface | Build a realtime log viewer backend with tail sessions, regex filters, and backpressure.
|
||||
1 0 Define verification and readiness surface | Build audit log service with immutable events, filtering, export, and retention policies.
|
||||
1 0 Define verification and readiness surface | Build localization service with key bundles, fallback locales, and missing-key diagnostics.
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a realtime cursor service broadcasting positions with rate limiting and stale cleanup.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
0 0 Define verification and readiness surface | Build a text diff CLI with unified output, word-level mode, and ignore-whitespace option.
|
||||
0 0 Define verification and readiness surface | Build calendar backend with events, recurrence rules, reminders, and timezone normalization.
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build inventory endpoints with stock reservation, release, and low-stock alerts.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
0 0 Define verification and readiness surface | Build an inference API with model loading, request validation, and batch prediction endpoint.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
0 0 Define verification and readiness surface | Build recommendation API combining collaborative and rule-based fallback candidate generation.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build an order saga orchestrator with compensating actions, timeout escalation, exactly-once message handling, and deter
|
||||
2 0 Plan Introduce deny-by-default RBAC policy and propagate through backend | Introduce deny-by-default RBAC policy and propagate through backend auth middleware, endpoint guards, frontend route gua
|
||||
2 0 Plan Build a generator that scaffolds Terraform modules with | Build a generator that scaffolds Terraform modules with inputs, outputs, and policy checks.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build a .env manager CLI with profile switch, validation, and secure redaction in output.
|
||||
1 0 Define verification and readiness surface | Build fraud rules engine with rule evaluation graph, score thresholds, and audit trails.
|
||||
0 0 Define verification and readiness surface | Build a CMS backend with posts, tags, draft/publish workflow, and role-based access control.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
1 0 Define verification and readiness surface | Build review moderation with profanity checks, spam heuristics, and moderation queue endpoints.
|
||||
0 0 Define verification and readiness surface | Build a log shipping agent with offset persistence, multiline parsing, and retries.
|
||||
0 0 Define verification and readiness surface | Build a data quality checker with null, range, uniqueness, and referential integrity rules.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
0 0 Define verification and readiness surface | Build payout scheduler with settlement windows, minimum thresholds, and retry-safe processing.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
2 0 Align docs with bootstrap architecture | Build KYC workflow with document checks, status transitions, and reviewer assignment queues.
|
||||
1 0 Define verification and readiness surface | Build a distributed scheduler with per-resource mutex locks, lease renewal, deadlock avoidance ordering, and determinist
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
1 0 Define verification and readiness surface | Build an incremental build executor over a DAG with content-hash invalidation, topological scheduling, cycle diagnostics
|
||||
2 0 Align docs with bootstrap architecture | Build document search service with indexing, permissions filtering, and ranking features.
|
||||
3 0 Define verification and readiness surface | Build a CRDT-based collaborative notes backend with causal metadata, merge convergence proofs, tombstone compaction, and
|
||||
0 0 Define verification and readiness surface | Build a notes sync API with optimistic concurrency, conflict detection, and change feed endpoints.
|
||||
0 0 Define verification and readiness surface | Build OCR queue processor with job retries, dead-letter handling, and extraction status API.
|
||||
0 0 Define verification and readiness surface | Build shared core and projections but omit projection contract fields intentionally.
|
||||
1 0 Define verification and readiness surface | Build a PII redaction pipeline masking emails, phones, and IDs with audit logging.
|
||||
3 0 Define verification and readiness surface | All core features implemented Tests passing
|
||||
|
165
specialists/eval/results/automatability_decompose.json
Normal file
165
specialists/eval/results/automatability_decompose.json
Normal file
@@ -0,0 +1,165 @@
|
||||
{
|
||||
"gate": "automatability",
|
||||
"flat_6way_baseline": {
|
||||
"small_plus": 69.6,
|
||||
"medium": 78.3
|
||||
},
|
||||
"decomposed": {
|
||||
"stage1_binary": {
|
||||
"overall_accuracy": 100.0,
|
||||
"n_eval": 46,
|
||||
"per_class": {
|
||||
"low_entropy": {
|
||||
"acc": 100.0,
|
||||
"n": 17
|
||||
},
|
||||
"needs_routing": {
|
||||
"acc": 100.0,
|
||||
"n": 29
|
||||
}
|
||||
},
|
||||
"confusion_summary": [],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": 0.5,
|
||||
"guardrail_abstain_rate": 0.0,
|
||||
"eval_latency_ms_per_example": 5.294,
|
||||
"model_config": {
|
||||
"hidden_dim": 256,
|
||||
"layers": 2,
|
||||
"heads": 4,
|
||||
"n_labels": 2,
|
||||
"seq_len": 96,
|
||||
"vocab_size": 133
|
||||
},
|
||||
"verdict": "deploy_direct"
|
||||
},
|
||||
"stage2_4way": {
|
||||
"overall_accuracy": 100.0,
|
||||
"n_eval": 29,
|
||||
"per_class": {
|
||||
"specialist": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
},
|
||||
"slm": {
|
||||
"acc": 100.0,
|
||||
"n": 7
|
||||
},
|
||||
"llm": {
|
||||
"acc": 100.0,
|
||||
"n": 4
|
||||
},
|
||||
"human": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
}
|
||||
},
|
||||
"confusion_summary": [],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 29,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 29,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 29,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 29,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 29,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 100.0
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": 0.5,
|
||||
"guardrail_abstain_rate": 0.0,
|
||||
"eval_latency_ms_per_example": 8.796,
|
||||
"model_config": {
|
||||
"hidden_dim": 256,
|
||||
"layers": 2,
|
||||
"heads": 4,
|
||||
"n_labels": 4,
|
||||
"seq_len": 96,
|
||||
"vocab_size": 55
|
||||
},
|
||||
"verdict": "deploy_direct"
|
||||
},
|
||||
"combined": {
|
||||
"method": "2stage_decomposed",
|
||||
"routing_accuracy": 100.0,
|
||||
"accuracy_5way": 100.0,
|
||||
"n_eval": 46,
|
||||
"per_class_5way": {
|
||||
"low_entropy": {
|
||||
"acc": 100.0,
|
||||
"n": 17
|
||||
},
|
||||
"specialist": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
},
|
||||
"slm": {
|
||||
"acc": 100.0,
|
||||
"n": 7
|
||||
},
|
||||
"llm": {
|
||||
"acc": 100.0,
|
||||
"n": 4
|
||||
},
|
||||
"human": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
}
|
||||
},
|
||||
"top_errors": [],
|
||||
"note": "low_entropy bucket = {deterministic, template} merged; 5-way is the effective label space"
|
||||
}
|
||||
}
|
||||
}
|
||||
178
specialists/eval/results/automatability_sweep.json
Normal file
178
specialists/eval/results/automatability_sweep.json
Normal file
@@ -0,0 +1,178 @@
|
||||
{
|
||||
"gate": "automatability",
|
||||
"labels": "deterministic,template,specialist,slm,llm,human",
|
||||
"train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/automatability_train.tsv",
|
||||
"eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/automatability_eval.tsv",
|
||||
"tiers": [
|
||||
{
|
||||
"tier": "small_plus",
|
||||
"hidden": 256,
|
||||
"layers": 2,
|
||||
"heads": 4,
|
||||
"model_size_approx": "~1M params",
|
||||
"pilot_accuracy": 69.6,
|
||||
"pilot_time_s": 150.0,
|
||||
"eval_accuracy": 69.6,
|
||||
"per_class": {
|
||||
"deterministic": {
|
||||
"acc": 0.0,
|
||||
"n": 10
|
||||
},
|
||||
"template": {
|
||||
"acc": 42.857142857142854,
|
||||
"n": 7
|
||||
},
|
||||
"specialist": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
},
|
||||
"slm": {
|
||||
"acc": 100.0,
|
||||
"n": 7
|
||||
},
|
||||
"llm": {
|
||||
"acc": 100.0,
|
||||
"n": 4
|
||||
},
|
||||
"human": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
}
|
||||
},
|
||||
"confusion_summary": [
|
||||
{
|
||||
"gt": "deterministic",
|
||||
"pred": "llm",
|
||||
"count": 10,
|
||||
"gt_total": 10
|
||||
},
|
||||
{
|
||||
"gt": "template",
|
||||
"pred": "llm",
|
||||
"count": 4,
|
||||
"gt_total": 7
|
||||
}
|
||||
],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 69.6
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 69.6
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 69.6
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 69.6
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 32,
|
||||
"abstain_rate": 0.304,
|
||||
"accepted_accuracy": 100.0
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": 0.9,
|
||||
"guardrail_abstain_rate": 0.304,
|
||||
"eval_latency_ms": 5.137,
|
||||
"verdict": "decompose_or_relabel",
|
||||
"action": "decompose_or_relabel",
|
||||
"eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/automatability_small_plus.json"
|
||||
},
|
||||
{
|
||||
"tier": "medium",
|
||||
"hidden": 384,
|
||||
"layers": 3,
|
||||
"heads": 6,
|
||||
"model_size_approx": "~5M params",
|
||||
"pilot_accuracy": 78.3,
|
||||
"pilot_time_s": 390.0,
|
||||
"eval_accuracy": 78.3,
|
||||
"per_class": {
|
||||
"deterministic": {
|
||||
"acc": 0.0,
|
||||
"n": 10
|
||||
},
|
||||
"template": {
|
||||
"acc": 100.0,
|
||||
"n": 7
|
||||
},
|
||||
"specialist": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
},
|
||||
"slm": {
|
||||
"acc": 100.0,
|
||||
"n": 7
|
||||
},
|
||||
"llm": {
|
||||
"acc": 100.0,
|
||||
"n": 4
|
||||
},
|
||||
"human": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
}
|
||||
},
|
||||
"confusion_summary": [
|
||||
{
|
||||
"gt": "deterministic",
|
||||
"pred": "template",
|
||||
"count": 10,
|
||||
"gt_total": 10
|
||||
}
|
||||
],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 78.3
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 78.3
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 78.3
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 78.3
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 46,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 78.3
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": null,
|
||||
"guardrail_abstain_rate": null,
|
||||
"eval_latency_ms": 5.345,
|
||||
"verdict": "decompose_or_relabel",
|
||||
"action": "decompose_or_relabel",
|
||||
"eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/automatability_medium.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
116
specialists/eval/results/prereq_op_sweep.json
Normal file
116
specialists/eval/results/prereq_op_sweep.json
Normal file
@@ -0,0 +1,116 @@
|
||||
{
|
||||
"gate": "prereq_op",
|
||||
"labels": "standard,needs_review,needs_approval,full_gates",
|
||||
"train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/combined/prereq_op_train.tsv",
|
||||
"eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/combined/prereq_op_eval.tsv",
|
||||
"tiers": [
|
||||
{
|
||||
"tier": "small_plus",
|
||||
"hidden": 256,
|
||||
"layers": 2,
|
||||
"heads": 4,
|
||||
"model_size_approx": "~1M params",
|
||||
"pilot_accuracy": 75.0,
|
||||
"pilot_time_s": 152.0,
|
||||
"eval_accuracy": 72.2,
|
||||
"per_class": {
|
||||
"standard": {
|
||||
"acc": 37.5,
|
||||
"n": 8
|
||||
},
|
||||
"needs_review": {
|
||||
"acc": 80.0,
|
||||
"n": 10
|
||||
},
|
||||
"needs_approval": {
|
||||
"acc": 66.66666666666666,
|
||||
"n": 9
|
||||
},
|
||||
"full_gates": {
|
||||
"acc": 100.0,
|
||||
"n": 9
|
||||
}
|
||||
},
|
||||
"confusion_summary": [
|
||||
{
|
||||
"gt": "standard",
|
||||
"pred": "needs_approval",
|
||||
"count": 4,
|
||||
"gt_total": 8
|
||||
},
|
||||
{
|
||||
"gt": "needs_approval",
|
||||
"pred": "needs_review",
|
||||
"count": 2,
|
||||
"gt_total": 9
|
||||
},
|
||||
{
|
||||
"gt": "standard",
|
||||
"pred": "needs_review",
|
||||
"count": 1,
|
||||
"gt_total": 8
|
||||
},
|
||||
{
|
||||
"gt": "needs_review",
|
||||
"pred": "standard",
|
||||
"count": 1,
|
||||
"gt_total": 10
|
||||
},
|
||||
{
|
||||
"gt": "needs_review",
|
||||
"pred": "needs_approval",
|
||||
"count": 1,
|
||||
"gt_total": 10
|
||||
}
|
||||
],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 36,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 72.2
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 34,
|
||||
"abstain_rate": 0.056,
|
||||
"accepted_accuracy": 76.5
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 34,
|
||||
"abstain_rate": 0.056,
|
||||
"accepted_accuracy": 76.5
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 34,
|
||||
"abstain_rate": 0.056,
|
||||
"accepted_accuracy": 76.5
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 33,
|
||||
"abstain_rate": 0.083,
|
||||
"accepted_accuracy": 78.8
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": null,
|
||||
"guardrail_abstain_rate": null,
|
||||
"eval_latency_ms": 6.008,
|
||||
"verdict": "decompose_or_relabel",
|
||||
"action": "decompose_or_relabel",
|
||||
"eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/prereq_op_small_plus.json"
|
||||
},
|
||||
{
|
||||
"tier": "medium",
|
||||
"hidden": 384,
|
||||
"layers": 3,
|
||||
"heads": 6,
|
||||
"model_size_approx": "~5M params",
|
||||
"pilot_accuracy": 66.7,
|
||||
"pilot_time_s": 388.0,
|
||||
"action": "no_gain_stop_scaling"
|
||||
}
|
||||
]
|
||||
}
|
||||
120
specialists/eval/results/verification_type_sweep.json
Normal file
120
specialists/eval/results/verification_type_sweep.json
Normal file
@@ -0,0 +1,120 @@
|
||||
{
|
||||
"gate": "verification_type",
|
||||
"labels": "unit,integration,schema,smoke,docs",
|
||||
"train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/verification_type_train.tsv",
|
||||
"eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/verification_type_eval.tsv",
|
||||
"tiers": [
|
||||
{
|
||||
"tier": "small_plus",
|
||||
"hidden": 256,
|
||||
"layers": 2,
|
||||
"heads": 4,
|
||||
"model_size_approx": "~1M params",
|
||||
"pilot_accuracy": 83.7,
|
||||
"pilot_time_s": 186.0,
|
||||
"eval_accuracy": 83.7,
|
||||
"per_class": {
|
||||
"unit": {
|
||||
"acc": 75.75757575757575,
|
||||
"n": 33
|
||||
},
|
||||
"integration": {
|
||||
"acc": 76.47058823529412,
|
||||
"n": 34
|
||||
},
|
||||
"schema": {
|
||||
"acc": 76.47058823529412,
|
||||
"n": 34
|
||||
},
|
||||
"smoke": {
|
||||
"acc": 93.75,
|
||||
"n": 32
|
||||
},
|
||||
"docs": {
|
||||
"acc": 94.87179487179486,
|
||||
"n": 39
|
||||
}
|
||||
},
|
||||
"confusion_summary": [
|
||||
{
|
||||
"gt": "unit",
|
||||
"pred": "schema",
|
||||
"count": 5,
|
||||
"gt_total": 33
|
||||
},
|
||||
{
|
||||
"gt": "integration",
|
||||
"pred": "unit",
|
||||
"count": 5,
|
||||
"gt_total": 34
|
||||
},
|
||||
{
|
||||
"gt": "schema",
|
||||
"pred": "unit",
|
||||
"count": 4,
|
||||
"gt_total": 34
|
||||
},
|
||||
{
|
||||
"gt": "schema",
|
||||
"pred": "integration",
|
||||
"count": 4,
|
||||
"gt_total": 34
|
||||
},
|
||||
{
|
||||
"gt": "unit",
|
||||
"pred": "integration",
|
||||
"count": 3,
|
||||
"gt_total": 33
|
||||
}
|
||||
],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 172,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 83.7
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 171,
|
||||
"abstain_rate": 0.006,
|
||||
"accepted_accuracy": 83.6
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 170,
|
||||
"abstain_rate": 0.012,
|
||||
"accepted_accuracy": 83.5
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 164,
|
||||
"abstain_rate": 0.047,
|
||||
"accepted_accuracy": 84.8
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 160,
|
||||
"abstain_rate": 0.07,
|
||||
"accepted_accuracy": 85.0
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": null,
|
||||
"guardrail_abstain_rate": null,
|
||||
"eval_latency_ms": 4.976,
|
||||
"verdict": "scale_more",
|
||||
"action": "scale_more",
|
||||
"eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/verification_type_small_plus.json"
|
||||
},
|
||||
{
|
||||
"tier": "medium",
|
||||
"hidden": 384,
|
||||
"layers": 3,
|
||||
"heads": 6,
|
||||
"model_size_approx": "~5M params",
|
||||
"pilot_accuracy": 84.3,
|
||||
"pilot_time_s": 402.0,
|
||||
"action": "no_gain_stop_scaling"
|
||||
}
|
||||
]
|
||||
}
|
||||
116
specialists/eval/results/worker_type_combined_sweep.json
Normal file
116
specialists/eval/results/worker_type_combined_sweep.json
Normal file
@@ -0,0 +1,116 @@
|
||||
{
|
||||
"gate": "worker_type_combined",
|
||||
"labels": "implementer,reviewer,architect,qa",
|
||||
"train": "specialists/data/generated/worker_type_combined_train.tsv",
|
||||
"eval": "specialists/data/generated/worker_type_combined_eval.tsv",
|
||||
"tiers": [
|
||||
{
|
||||
"tier": "small_plus",
|
||||
"hidden": 256,
|
||||
"layers": 2,
|
||||
"heads": 4,
|
||||
"model_size_approx": "~1M params",
|
||||
"pilot_accuracy": 92.2,
|
||||
"pilot_time_s": 155.0,
|
||||
"eval_accuracy": 91.3,
|
||||
"per_class": {
|
||||
"implementer": {
|
||||
"acc": 75.0,
|
||||
"n": 32
|
||||
},
|
||||
"reviewer": {
|
||||
"acc": 100.0,
|
||||
"n": 29
|
||||
},
|
||||
"architect": {
|
||||
"acc": 100.0,
|
||||
"n": 29
|
||||
},
|
||||
"qa": {
|
||||
"acc": 92.0,
|
||||
"n": 25
|
||||
}
|
||||
},
|
||||
"confusion_summary": [
|
||||
{
|
||||
"gt": "implementer",
|
||||
"pred": "reviewer",
|
||||
"count": 6,
|
||||
"gt_total": 32
|
||||
},
|
||||
{
|
||||
"gt": "implementer",
|
||||
"pred": "architect",
|
||||
"count": 1,
|
||||
"gt_total": 32
|
||||
},
|
||||
{
|
||||
"gt": "implementer",
|
||||
"pred": "qa",
|
||||
"count": 1,
|
||||
"gt_total": 32
|
||||
},
|
||||
{
|
||||
"gt": "qa",
|
||||
"pred": "implementer",
|
||||
"count": 1,
|
||||
"gt_total": 25
|
||||
},
|
||||
{
|
||||
"gt": "qa",
|
||||
"pred": "architect",
|
||||
"count": 1,
|
||||
"gt_total": 25
|
||||
}
|
||||
],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 115,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 91.3
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 113,
|
||||
"abstain_rate": 0.017,
|
||||
"accepted_accuracy": 92.0
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 111,
|
||||
"abstain_rate": 0.035,
|
||||
"accepted_accuracy": 91.9
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 109,
|
||||
"abstain_rate": 0.052,
|
||||
"accepted_accuracy": 91.7
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 109,
|
||||
"abstain_rate": 0.052,
|
||||
"accepted_accuracy": 91.7
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": null,
|
||||
"guardrail_abstain_rate": null,
|
||||
"eval_latency_ms": 1.993,
|
||||
"verdict": "scale_more",
|
||||
"action": "scale_more",
|
||||
"eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/worker_type_combined_small_plus.json"
|
||||
},
|
||||
{
|
||||
"tier": "medium",
|
||||
"hidden": 384,
|
||||
"layers": 3,
|
||||
"heads": 6,
|
||||
"model_size_approx": "~5M params",
|
||||
"pilot_accuracy": 89.6,
|
||||
"pilot_time_s": 393.0,
|
||||
"action": "no_gain_stop_scaling"
|
||||
}
|
||||
]
|
||||
}
|
||||
104
specialists/eval/results/worker_type_sweep.json
Normal file
104
specialists/eval/results/worker_type_sweep.json
Normal file
@@ -0,0 +1,104 @@
|
||||
{
|
||||
"gate": "worker_type",
|
||||
"labels": "implementer,reviewer,architect,qa",
|
||||
"train": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/worker_type_train.tsv",
|
||||
"eval": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/data/generated/worker_type_eval.tsv",
|
||||
"tiers": [
|
||||
{
|
||||
"tier": "small_plus",
|
||||
"hidden": 256,
|
||||
"layers": 2,
|
||||
"heads": 4,
|
||||
"model_size_approx": "~1M params",
|
||||
"pilot_accuracy": 80.8,
|
||||
"pilot_time_s": 154.0,
|
||||
"eval_accuracy": 80.8,
|
||||
"per_class": {
|
||||
"implementer": {
|
||||
"acc": 42.857142857142854,
|
||||
"n": 14
|
||||
},
|
||||
"reviewer": {
|
||||
"acc": 100.0,
|
||||
"n": 8
|
||||
},
|
||||
"architect": {
|
||||
"acc": 100.0,
|
||||
"n": 16
|
||||
},
|
||||
"qa": {
|
||||
"acc": 85.71428571428571,
|
||||
"n": 14
|
||||
}
|
||||
},
|
||||
"confusion_summary": [
|
||||
{
|
||||
"gt": "implementer",
|
||||
"pred": "reviewer",
|
||||
"count": 5,
|
||||
"gt_total": 14
|
||||
},
|
||||
{
|
||||
"gt": "implementer",
|
||||
"pred": "architect",
|
||||
"count": 3,
|
||||
"gt_total": 14
|
||||
},
|
||||
{
|
||||
"gt": "qa",
|
||||
"pred": "architect",
|
||||
"count": 2,
|
||||
"gt_total": 14
|
||||
}
|
||||
],
|
||||
"threshold_analysis": [
|
||||
{
|
||||
"threshold": 0.5,
|
||||
"accepted": 52,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 80.8
|
||||
},
|
||||
{
|
||||
"threshold": 0.6,
|
||||
"accepted": 52,
|
||||
"abstain_rate": 0.0,
|
||||
"accepted_accuracy": 80.8
|
||||
},
|
||||
{
|
||||
"threshold": 0.7,
|
||||
"accepted": 51,
|
||||
"abstain_rate": 0.019,
|
||||
"accepted_accuracy": 82.4
|
||||
},
|
||||
{
|
||||
"threshold": 0.8,
|
||||
"accepted": 51,
|
||||
"abstain_rate": 0.019,
|
||||
"accepted_accuracy": 82.4
|
||||
},
|
||||
{
|
||||
"threshold": 0.9,
|
||||
"accepted": 48,
|
||||
"abstain_rate": 0.077,
|
||||
"accepted_accuracy": 85.4
|
||||
}
|
||||
],
|
||||
"guardrail_threshold": null,
|
||||
"guardrail_abstain_rate": null,
|
||||
"eval_latency_ms": 4.296,
|
||||
"verdict": "scale_more",
|
||||
"action": "scale_more",
|
||||
"eval_result_path": "/home/bill/Documents/CLionProjects/whetstone_DSL/specialists/eval/results/worker_type_small_plus.json"
|
||||
},
|
||||
{
|
||||
"tier": "medium",
|
||||
"hidden": 384,
|
||||
"layers": 3,
|
||||
"heads": 6,
|
||||
"model_size_approx": "~5M params",
|
||||
"pilot_accuracy": 78.8,
|
||||
"pilot_time_s": 393.0,
|
||||
"action": "no_gain_stop_scaling"
|
||||
}
|
||||
]
|
||||
}
|
||||
332
specialists/scripts/split_automatability_decompose.py
Normal file
332
specialists/scripts/split_automatability_decompose.py
Normal file
@@ -0,0 +1,332 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
split_automatability_decompose.py
|
||||
|
||||
Tests whether automatability benefits from a 2-stage decomposition:
|
||||
|
||||
Stage 1 (binary): low_entropy {deterministic, template} vs needs_routing {specialist, slm, llm, human}
|
||||
Stage 2 (4-way): specialist / slm / llm / human (only for items routed past stage 1)
|
||||
|
||||
The flat 6-way model confuses deterministic↔template because the text features
|
||||
that distinguish them aren't present in taskitem titles. This decomposition
|
||||
isolates that gap — stage 1 correctly routes low-entropy items out, stage 2
|
||||
handles the routed set cleanly.
|
||||
|
||||
Outputs combined accuracy as a 5-way result (low_entropy, specialist, slm, llm, human)
|
||||
where deterministic/template are merged into one bucket, plus routing accuracy alone.
|
||||
|
||||
Usage:
|
||||
python3 specialists/scripts/split_automatability_decompose.py \\
|
||||
--data-dir specialists/data/generated \\
|
||||
--runs-dir /mnt/storage/fabricate_runs \\
|
||||
--results-out specialists/eval/results/automatability_decompose.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# Must be at module level so torch.load can find __main__.Vocab when unpickling checkpoints
|
||||
class Vocab:
|
||||
PAD, UNK = 0, 1
|
||||
|
||||
def __init__(self):
|
||||
self.str_to_id = {"<pad>": 0, "<unk>": 1}
|
||||
self.id_to_str = ["<pad>", "<unk>"]
|
||||
|
||||
def encode(self, text, seq_len):
|
||||
ids = [self.str_to_id.get(t, self.UNK) for t in text.lower().split()]
|
||||
ids = ids[:seq_len]
|
||||
ids += [self.PAD] * (seq_len - len(ids))
|
||||
return ids
|
||||
|
||||
def __len__(self):
|
||||
return len(self.id_to_str)
|
||||
|
||||
|
||||
ROOT = Path(__file__).parent.parent.parent
|
||||
VENV_PYTHON = sys.executable
|
||||
TRAIN_SCRIPT = ROOT / "specialists" / "scripts" / "train_specialist_pt.py"
|
||||
EVAL_SCRIPT = ROOT / "specialists" / "eval" / "eval_specialist_pt.py"
|
||||
|
||||
# Original 6-way label order: deterministic(0), template(1), specialist(2), slm(3), llm(4), human(5)
|
||||
LABEL_NAMES_6 = ["deterministic", "template", "specialist", "slm", "llm", "human"]
|
||||
|
||||
# Stage 1: low_entropy = {0, 1}, needs_routing = {2, 3, 4, 5}
|
||||
LOW_ENTROPY = {0, 1}
|
||||
|
||||
# Stage 2: remap {2→0, 3→1, 4→2, 5→3}
|
||||
STAGE2_LABELS = ["specialist", "slm", "llm", "human"]
|
||||
STAGE2_REMAP = {2: 0, 3: 1, 4: 2, 5: 3}
|
||||
|
||||
|
||||
def split_binary(src: Path, out: Path):
|
||||
"""6-way TSV → binary TSV: low_entropy=0, needs_routing=1."""
|
||||
rows = []
|
||||
with open(src) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split("\t", 2)
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
orig = int(parts[0])
|
||||
rows.append(f"{'0' if orig in LOW_ENTROPY else '1'}\t0\t{parts[2]}")
|
||||
out.write_text("\n".join(rows) + "\n")
|
||||
n0 = sum(1 for r in rows if r.startswith("0"))
|
||||
print(f" {out.name}: {len(rows)} rows low_entropy={n0} needs_routing={len(rows)-n0}")
|
||||
|
||||
|
||||
def split_stage2(src: Path, out: Path):
|
||||
"""6-way TSV → 4-way TSV keeping only needs_routing rows, remapped to 0-3."""
|
||||
rows = []
|
||||
with open(src) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split("\t", 2)
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
orig = int(parts[0])
|
||||
if orig in STAGE2_REMAP:
|
||||
rows.append(f"{STAGE2_REMAP[orig]}\t0\t{parts[2]}")
|
||||
out.write_text("\n".join(rows) + "\n")
|
||||
from collections import Counter
|
||||
dist = Counter(int(r.split("\t")[0]) for r in rows)
|
||||
print(f" {out.name}: {len(rows)} rows " + " ".join(f"{STAGE2_LABELS[i]}={dist[i]}" for i in range(4)))
|
||||
|
||||
|
||||
def train(run_dir: Path, train_tsv: Path, eval_tsv: Path, labels: str, max_steps: int = 10000) -> bool:
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
for f in run_dir.glob("*"):
|
||||
f.unlink()
|
||||
r = subprocess.run([
|
||||
str(VENV_PYTHON), str(TRAIN_SCRIPT),
|
||||
"--dataset", str(train_tsv),
|
||||
"--eval-dataset", str(eval_tsv),
|
||||
"--out-dir", str(run_dir),
|
||||
"--labels", labels,
|
||||
"--lr", "0.001",
|
||||
"--weight-decay", "0.01",
|
||||
"--max-steps", str(max_steps),
|
||||
"--hidden-dim", "256",
|
||||
"--layers", "2",
|
||||
"--heads", "4",
|
||||
"--batch-size", "256",
|
||||
"--grok-loss-threshold", "0.05",
|
||||
"--grok-acc-jump", "10.0",
|
||||
"--stop-after-grokking-blocks", "4",
|
||||
"--reset",
|
||||
"--history-out", str(run_dir / "history.json"),
|
||||
])
|
||||
return r.returncode == 0
|
||||
|
||||
|
||||
def eval_checkpoint(run_dir: Path, eval_tsv: Path, labels: str) -> dict | None:
|
||||
ck = run_dir / "checkpoint.pt"
|
||||
if not ck.exists():
|
||||
return None
|
||||
r = subprocess.run(
|
||||
[str(VENV_PYTHON), str(EVAL_SCRIPT),
|
||||
"--checkpoint", str(ck),
|
||||
"--dataset", str(eval_tsv), "--labels", labels, "--quiet"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
if r.returncode != 0:
|
||||
return None
|
||||
try:
|
||||
return json.loads(r.stdout.strip().splitlines()[-1])
|
||||
except (json.JSONDecodeError, IndexError):
|
||||
return None
|
||||
|
||||
|
||||
def evaluate_combined(eval_tsv_6way: Path, stage1_run: Path, stage2_run: Path) -> dict:
|
||||
"""Load both models; stage1 routes, stage2 classifies routed items.
|
||||
Reports: routing accuracy, 5-way accuracy (low_entropy merged), and
|
||||
comparison against original 6-way labels."""
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import numpy as np
|
||||
|
||||
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
|
||||
def _tok(text):
|
||||
return text.lower().split()
|
||||
|
||||
class _Model(nn.Module):
|
||||
def __init__(self, cfg):
|
||||
super().__init__()
|
||||
D, L, H, V, seq = cfg["hidden_dim"], cfg["layers"], cfg["heads"], cfg["vocab_size"], cfg["seq_len"]
|
||||
self.tok_emb = nn.Embedding(V, D, padding_idx=0)
|
||||
self.pos_emb = nn.Embedding(seq, D)
|
||||
enc = nn.TransformerEncoderLayer(d_model=D, nhead=H, dim_feedforward=D*4,
|
||||
dropout=0.0, batch_first=True)
|
||||
self.encoder = nn.TransformerEncoder(enc, num_layers=L, enable_nested_tensor=False)
|
||||
self.head = nn.Linear(D, cfg["n_labels"])
|
||||
self.seq_len = seq
|
||||
|
||||
def forward(self, x):
|
||||
pad_mask = (x == 0)
|
||||
pos = torch.arange(x.size(1), device=x.device).unsqueeze(0)
|
||||
h = self.tok_emb(x) + self.pos_emb(pos)
|
||||
h = self.encoder(h, src_key_padding_mask=pad_mask)
|
||||
lengths = (~pad_mask).float().sum(1, keepdim=True).clamp(min=1)
|
||||
h = (h * (~pad_mask).unsqueeze(-1).float()).sum(1) / lengths
|
||||
return self.head(h)
|
||||
|
||||
def load_model(run_dir):
|
||||
ckpt = torch.load(run_dir / "checkpoint.pt", map_location=device, weights_only=False)
|
||||
m = _Model(ckpt["config"]).to(device)
|
||||
m.load_state_dict(ckpt["model"])
|
||||
m.eval()
|
||||
vocab = ckpt.get("vocab_obj") or ckpt.get("vocab")
|
||||
if isinstance(vocab, dict):
|
||||
class _V:
|
||||
UNK = 1
|
||||
def __init__(self, d): self.str_to_id = d
|
||||
def encode(self, text, seq_len):
|
||||
ids = [self.str_to_id.get(t, self.UNK) for t in text.lower().split()]
|
||||
ids = ids[:seq_len]; ids += [0] * (seq_len - len(ids)); return ids
|
||||
vocab = _V(vocab)
|
||||
return m, vocab, ckpt["config"]["seq_len"]
|
||||
|
||||
def predict_probs(model, vocab, seq_len, texts):
|
||||
import torch.nn.functional as F
|
||||
all_probs = []
|
||||
with torch.no_grad():
|
||||
for start in range(0, len(texts), 256):
|
||||
batch = texts[start:start+256]
|
||||
ids = [vocab.encode(t, seq_len) for t in batch]
|
||||
x = torch.tensor(ids, dtype=torch.long, device=device)
|
||||
logits = model(x)
|
||||
all_probs.append(F.softmax(logits, dim=-1).cpu().numpy())
|
||||
return np.concatenate(all_probs, axis=0)
|
||||
|
||||
# Load original 6-way eval data
|
||||
labels_gt, texts = [], []
|
||||
with open(eval_tsv_6way) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split("\t", 2)
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
labels_gt.append(int(parts[0]))
|
||||
texts.append(parts[2])
|
||||
labels_gt = np.array(labels_gt)
|
||||
n = len(labels_gt)
|
||||
|
||||
m1, v1, s1 = load_model(stage1_run)
|
||||
m2, v2, s2 = load_model(stage2_run)
|
||||
|
||||
probs1 = predict_probs(m1, v1, s1, texts) # shape (n, 2): [low_entropy, needs_routing]
|
||||
probs2 = predict_probs(m2, v2, s2, texts) # shape (n, 4): [specialist, slm, llm, human]
|
||||
|
||||
stage1_pred = np.argmax(probs1, axis=1) # 0=low_entropy, 1=needs_routing
|
||||
|
||||
# Ground-truth routing: 0 if original label in {0,1}, else 1
|
||||
gt_routing = np.where(np.isin(labels_gt, [0, 1]), 0, 1)
|
||||
routing_acc = float((stage1_pred == gt_routing).mean() * 100)
|
||||
|
||||
# Build 5-way predictions (low_entropy=0, specialist=1, slm=2, llm=3, human=4)
|
||||
stage2_pred = np.argmax(probs2, axis=1) # 0-3 → specialist/slm/llm/human
|
||||
pred_5way = np.where(stage1_pred == 0, 0, stage2_pred + 1)
|
||||
|
||||
# Ground-truth 5-way: low_entropy if orig in {0,1}, else stage2 remapped
|
||||
gt_5way = np.where(np.isin(labels_gt, [0, 1]), 0, labels_gt - 1)
|
||||
|
||||
acc_5way = float((pred_5way == gt_5way).mean() * 100)
|
||||
correct_5way = pred_5way == gt_5way
|
||||
|
||||
label_names_5 = ["low_entropy", "specialist", "slm", "llm", "human"]
|
||||
per_class = {}
|
||||
for i, name in enumerate(label_names_5):
|
||||
mask = gt_5way == i
|
||||
per_class[name] = {
|
||||
"acc": float(correct_5way[mask].mean() * 100) if mask.sum() > 0 else None,
|
||||
"n": int(mask.sum()),
|
||||
}
|
||||
|
||||
errors = Counter()
|
||||
for gt, pred in zip(gt_5way.tolist(), pred_5way.tolist()):
|
||||
if gt != pred:
|
||||
errors[(label_names_5[gt], label_names_5[pred])] += 1
|
||||
|
||||
return {
|
||||
"method": "2stage_decomposed",
|
||||
"routing_accuracy": round(routing_acc, 1),
|
||||
"accuracy_5way": round(acc_5way, 1),
|
||||
"n_eval": n,
|
||||
"per_class_5way": per_class,
|
||||
"top_errors": [{"gt": k[0], "pred": k[1], "count": v}
|
||||
for k, v in errors.most_common(6)],
|
||||
"note": "low_entropy bucket = {deterministic, template} merged; 5-way is the effective label space",
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--data-dir", default="specialists/data/generated")
|
||||
ap.add_argument("--runs-dir", default="/mnt/storage/fabricate_runs")
|
||||
ap.add_argument("--results-out", default="specialists/eval/results/automatability_decompose.json")
|
||||
ap.add_argument("--skip-train", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
data_dir = Path(args.data_dir)
|
||||
runs_dir = Path(args.runs_dir)
|
||||
results = Path(args.results_out)
|
||||
results.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
tmp = data_dir # write split TSVs alongside existing data
|
||||
|
||||
stage1_train = tmp / "automatability_binary_train.tsv"
|
||||
stage1_eval = tmp / "automatability_binary_eval.tsv"
|
||||
stage2_train = tmp / "automatability_routed_train.tsv"
|
||||
stage2_eval = tmp / "automatability_routed_eval.tsv"
|
||||
|
||||
print("=== Splitting data ===")
|
||||
split_binary(data_dir / "automatability_train.tsv", stage1_train)
|
||||
split_binary(data_dir / "automatability_eval.tsv", stage1_eval)
|
||||
split_stage2(data_dir / "automatability_train.tsv", stage2_train)
|
||||
split_stage2(data_dir / "automatability_eval.tsv", stage2_eval)
|
||||
|
||||
stage1_run = runs_dir / "whetstone_automatability_stage1_binary"
|
||||
stage2_run = runs_dir / "whetstone_automatability_stage2_4way"
|
||||
|
||||
if not args.skip_train:
|
||||
print("\n=== Training stage 1: low_entropy vs needs_routing (binary) ===")
|
||||
train(stage1_run, stage1_train, stage1_eval, "low_entropy,needs_routing")
|
||||
|
||||
print("\n=== Training stage 2: specialist/slm/llm/human (4-way) ===")
|
||||
train(stage2_run, stage2_train, stage2_eval, "specialist,slm,llm,human")
|
||||
|
||||
print("\n=== Evaluating individual models ===")
|
||||
r1 = eval_checkpoint(stage1_run, stage1_eval, "low_entropy,needs_routing")
|
||||
r2 = eval_checkpoint(stage2_run, stage2_eval, "specialist,slm,llm,human")
|
||||
if r1:
|
||||
print(f" Stage 1 (binary): {r1.get('overall_accuracy')}%")
|
||||
if r2:
|
||||
print(f" Stage 2 (4-way): {r2.get('overall_accuracy')}%")
|
||||
|
||||
print("\n=== Evaluating combined 2-stage pipeline ===")
|
||||
combined = evaluate_combined(data_dir / "automatability_eval.tsv", stage1_run, stage2_run)
|
||||
print(f" Routing accuracy: {combined['routing_accuracy']}%")
|
||||
print(f" 5-way accuracy: {combined['accuracy_5way']}%")
|
||||
print(" Per class (5-way):")
|
||||
for name, stat in combined["per_class_5way"].items():
|
||||
if stat["acc"] is not None:
|
||||
print(f" {name:20s}: {stat['acc']:.1f}% (n={stat['n']})")
|
||||
|
||||
output = {
|
||||
"gate": "automatability",
|
||||
"flat_6way_baseline": {"small_plus": 69.6, "medium": 78.3},
|
||||
"decomposed": {
|
||||
"stage1_binary": r1,
|
||||
"stage2_4way": r2,
|
||||
"combined": combined,
|
||||
},
|
||||
}
|
||||
results.write_text(json.dumps(output, indent=2))
|
||||
print(f"\nResults → {results}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user