Sprint 1: project skeleton, type system, and all architecture specs
- src/types.hpp: complete UCWM type system in C++20 — 19 enums, 11 facet data types, all core structs (CanonicalObject, Constraint, Facet, GateSignal, WorldState, etc.) with full JSON round-trip serialization - src/main.cpp: smoke test — constructs apple-problem WorldState by hand, serializes to JSON - tests/test_types.cpp: 19 tests, 123 assertions, all passing - CMakeLists.txt: CMake + CPM build with nlohmann/json, spdlog, Catch2 - schemas/: JSON Schema contracts for all UCWM data types - gates/, specialists/, resolver/, synthesis/: language-agnostic interface contracts and domain specs for all pipeline layers - docs/: architecture, vocabulary, decision matrices, roadmap (6 phases, 28 sprints), sprint_001, implementation_constraints Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
119
tests/contracts/gate_contract_tests.md
Normal file
119
tests/contracts/gate_contract_tests.md
Normal file
@@ -0,0 +1,119 @@
|
||||
# Gate Contract Tests
|
||||
|
||||
Template for testing any gate implementation. These tests must pass regardless of implementation method (regex, classifier, neural, hybrid).
|
||||
|
||||
---
|
||||
|
||||
## Test format
|
||||
|
||||
```
|
||||
GateContractTest {
|
||||
test_id: string
|
||||
gate_id: string
|
||||
input_text: string
|
||||
expected: {
|
||||
activated: boolean
|
||||
min_confidence: float -- activated=true: confidence must be >= this
|
||||
max_confidence: float -- activated=false: confidence must be <= this
|
||||
}
|
||||
test_class: "positive_recall" | "true_negative" | "boundary" | "idempotency"
|
||||
rationale: string
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `is_math` gate tests
|
||||
|
||||
| test_id | input | expected activated | min_conf | test_class |
|
||||
|---|---|---|---|---|
|
||||
| G_MATH_001 | "Alice had 5 apples. She gave Bob 2." | true | 0.80 | positive_recall |
|
||||
| G_MATH_002 | "What is 3 plus 7?" | true | 0.90 | positive_recall |
|
||||
| G_MATH_003 | "How many apples does she have left?" | true | 0.70 | positive_recall |
|
||||
| G_MATH_004 | "The cat sat on the mat." | false | — | true_negative |
|
||||
| G_MATH_005 | "Alice went to the store." | false | — | true_negative |
|
||||
| G_MATH_006 | "" | false | — | boundary |
|
||||
| G_MATH_007 | "42" | true | 0.70 | boundary |
|
||||
|
||||
---
|
||||
|
||||
## `has_entity_reference` gate tests
|
||||
|
||||
| test_id | input | expected activated | min_conf | test_class |
|
||||
|---|---|---|---|---|
|
||||
| G_ENT_001 | "Alice gave Bob 2 apples." | true | 0.90 | positive_recall |
|
||||
| G_ENT_002 | "She gave him the book." | true | 0.70 | positive_recall |
|
||||
| G_ENT_003 | "Dr. Smith visited the clinic." | true | 0.90 | positive_recall |
|
||||
| G_ENT_004 | "2 plus 3 equals 5." | false | — | true_negative |
|
||||
| G_ENT_005 | "run the program" | false | — | true_negative |
|
||||
| G_ENT_006 | "" | false | — | boundary |
|
||||
|
||||
---
|
||||
|
||||
## `has_ownership_transfer` gate tests
|
||||
|
||||
| test_id | input | expected activated | min_conf | test_class |
|
||||
|---|---|---|---|---|
|
||||
| G_OWN_001 | "Alice gave Bob 2 apples." | true | 0.90 | positive_recall |
|
||||
| G_OWN_002 | "She sold the car to him." | true | 0.90 | positive_recall |
|
||||
| G_OWN_003 | "He received a package." | true | 0.85 | positive_recall |
|
||||
| G_OWN_004 | "The sun rose at 6am." | false | — | true_negative |
|
||||
| G_OWN_005 | "Alice is happy." | false | — | true_negative |
|
||||
| G_OWN_006 | "gave" | true | 0.70 | boundary |
|
||||
|
||||
---
|
||||
|
||||
## `has_temporal_relation` gate tests
|
||||
|
||||
| test_id | input | expected activated | min_conf | test_class |
|
||||
|---|---|---|---|---|
|
||||
| G_TMP_001 | "She arrived after the meeting started." | true | 0.90 | positive_recall |
|
||||
| G_TMP_002 | "Before noon, Alice left." | true | 0.90 | positive_recall |
|
||||
| G_TMP_003 | "Step 1: open the file. Step 2: read it." | true | 0.80 | positive_recall |
|
||||
| G_TMP_004 | "Alice is 5 feet tall." | false | — | true_negative |
|
||||
| G_TMP_005 | "x equals y." | false | — | true_negative |
|
||||
|
||||
---
|
||||
|
||||
## `has_logical_negation` gate tests
|
||||
|
||||
| test_id | input | expected activated | min_conf | test_class |
|
||||
|---|---|---|---|---|
|
||||
| G_NEG_001 | "Alice did not give Bob any apples." | true | 0.90 | positive_recall |
|
||||
| G_NEG_002 | "She never arrived." | true | 0.90 | positive_recall |
|
||||
| G_NEG_003 | "Unless it rains, the picnic continues." | true | 0.85 | positive_recall |
|
||||
| G_NEG_004 | "Alice gave Bob 2 apples." | false | — | true_negative |
|
||||
|
||||
---
|
||||
|
||||
## Idempotency tests
|
||||
|
||||
Every gate must pass these for every gate_id:
|
||||
|
||||
| test_id | procedure | expected |
|
||||
|---|---|---|
|
||||
| G_IDEM_001 | Call gate(X) twice with identical input | Both calls return identical GateSignal |
|
||||
| G_IDEM_002 | Call gate(X) after unrelated text processed | Same result as fresh call |
|
||||
|
||||
---
|
||||
|
||||
## Boundary tests (all gates)
|
||||
|
||||
| test_id | input | expected |
|
||||
|---|---|---|
|
||||
| G_BOUND_001 | "" (empty string) | Valid GateSignal with activated=false, confidence > 0 |
|
||||
| G_BOUND_002 | Single character "a" | Valid GateSignal, no crash |
|
||||
| G_BOUND_003 | 10,000 character string | Valid GateSignal, no crash, returns within 100ms |
|
||||
| G_BOUND_004 | String with only symbols "!@#$%^&*()" | Valid GateSignal, no crash |
|
||||
|
||||
---
|
||||
|
||||
## Recall vs. precision tolerance
|
||||
|
||||
Gates are high-recall by design. The following tolerances apply to implementation evaluation:
|
||||
|
||||
- Recall must be >= 0.90 on positive test cases
|
||||
- Precision on clearly negative test cases must be >= 0.80
|
||||
- Precision on ambiguous test cases: no threshold (false positives are acceptable)
|
||||
|
||||
A gate that achieves 0.95 recall with 0.70 precision is preferred over one with 0.85 recall and 0.95 precision.
|
||||
Reference in New Issue
Block a user