diff --git a/features/001-core-ast-structure/plan.md b/features/001-core-ast-structure/plan.md new file mode 100644 index 0000000..cd45f60 --- /dev/null +++ b/features/001-core-ast-structure/plan.md @@ -0,0 +1,469 @@ +--- +feature_number: 001 +feature_name: Core AST Structure +status: Planning +created_at: 2026-02-03T16:45:00Z +spec_version: 1.0 +--- + +# Implementation Plan: Core AST Structure for SemAnno + +## Technical Context + +**Technology Stack**: +- **Workbench**: JetBrains MPS 2024.3+ +- **Target Language**: SemAnno (language definition in MPS) +- **Model Format**: MPS XML (.mps files) +- **Development Method**: Incremental concept definition with immediate editor testing + +**Key Decisions**: +- All concepts defined in single SemAnno language module (no separate WhetstoneCore module) +- Editors built in SemAnno.editor.mps with projection support for Python/C++ (added in future features) +- No code generation or external tools required for this feature +- Type system uses reference composition for nested type structures (ListType contains reference to inner Type) + +**Dependencies**: +- MPS 2024.3 devkit for language design +- None on external libraries or frameworks +- Builds on existing SemAnno structure model (assumed to exist from previous session) + +## Constitution Check + +**Project Constitution Principles** (from `.specswarm/constitution.md` or assumed defaults): +- Clarity over cleverness: ✅ All concepts have clear, intuitive semantics +- Consistency: ✅ Naming conventions and patterns applied uniformly +- Type safety: ✅ Strong typing via MPS reference system +- Testability: ✅ Manual AST creation is the test mechanism +- Documentation: ✅ Specification documents design rationale + +**No conflicting principles detected** - proceeding with implementation. + +## Implementation Phases + +### Phase 0: Research & Exploration + +**Objectives**: Resolve any ambiguities, document assumptions, identify edge cases + +**Tasks**: + +1. **Research Task 1: MPS Editor Design Patterns** + - Question: How are projection-ready editors structured in MPS for multi-projection languages? + - Goal: Understand patterns used in multi-view languages (e.g., JetBrains MPS itself with structure/interface views) + - Deliverable: Quick reference for editor cell patterns that support projection switching + - Owner: Implementation team (research before Phase 1) + +2. **Research Task 2: Nested Type Composition in MPS** + - Question: Best practices for building deeply nested type structures (list[map[string, optional[int]]]) + - Goal: Ensure type references are properly composable without circular dependencies + - Deliverable: Sample MPS structure showing 3-4 levels of type nesting + - Owner: Implementation team + +3. **Research Task 3: Annotation Attachment Patterns** + - Question: How do MPS languages attach multiple annotations to a single concept? + - Goal: Ensure annotations are optional, ordered, and independently removable + - Deliverable: Example annotation structure from existing MPS language (e.g., jetbrains.mps.lang.core) + - Owner: Implementation team + +**Status**: Research items are light and can be completed during Phase 1 implementation (not blocking) + +--- + +### Phase 1: Core Structure Definition + +**Objectives**: Define all 30+ AST concepts in SemAnno.structure.mps + +**Deliverables**: +- SemAnno.structure.mps with all concepts defined (Module, Function, Variable, all Statement types, all Expression types, all Type types, annotation concepts) +- Preliminary constraints model (SemAnno.constraints.mps) with basic validation rules +- Test model file (e.g., `languages/SemAnno/tests/SimpleExample.mps`) demonstrating a complete AST + +**Key Subtasks**: + +#### 1.1: Root & Container Concepts (1-2 hours) +- Module concept with properties (name: string) and references (annotations, variables, functions) +- Marker: "Container concepts complete" + +#### 1.2: Function & Parameter Concepts (2-3 hours) +- Function concept: name, parameters[], returnType, body[], annotations +- Parameter concept: name, type, defaultValue? +- Establish parameter ordering and type reference patterns +- Marker: "Function signature complete" + +#### 1.3: Variable Concept (1 hour) +- Variable concept: name, type, initializer?, annotations +- Ensure type and initializer reference logic +- Marker: "Variable concept complete" + +#### 1.4: Statement Concepts (4-5 hours) +- Abstract Statement parent +- Concrete: Block, Assignment, IfStatement, WhileLoop, ForLoop, Return, ExpressionStatement +- Each with appropriate properties and references +- Marker: "All 7 statement types complete" + +#### 1.5: Expression Concepts (4-5 hours) +- Abstract Expression parent +- Binary/Unary operations with operator properties +- Function call, variable reference, literal types (5 literal types) +- List literal, index access, member access +- Total: 13 expression concept types +- Marker: "All 13 expression types complete" + +#### 1.6: Type Concepts (3-4 hours) +- Abstract Type parent +- Concrete: PrimitiveType (with enum), ListType, SetType, MapType, TupleType, ArrayType, OptionalType, CustomType +- 8 concrete type concepts +- Ensure ListType.elementType references Type (composable) +- MapType.keyType and valueType both reference Type +- TupleType.elementTypes are ordered, multi-valued Type references +- Marker: "All 8 type concepts complete, nesting tested" + +#### 1.7: Annotation Concepts (2-3 hours) +- Abstract Annotation parent (optional, for future extensibility) +- DerefStrategy: strategy enum, derefTime?, derefLocation?, owner? +- OptimizationLock: lockedBy, lockReason, lockLevel enum, affectedStrategies[], timestamp +- LangSpecific: language enum, idiomType, rawSyntax, semanticHint?, position enum +- Marker: "All 3 annotation types complete" + +#### 1.8: Constraints & Validation (2 hours) +- Create SemAnno.constraints.mps +- Basic constraints: statements only in statement lists, expressions only in expression contexts, types reference only Type concepts +- Can-be-parent/can-be-child rules to prevent invalid structures +- Marker: "Constraints model complete" + +#### 1.9: Test Model Creation (2-3 hours) +- Create `languages/SemAnno/tests/SimpleExample.mps` demonstrating: + - 1 Module with name "example" + - 1 Function "sum" with 2 parameters (items: list[int], factor: float), return type: float + - Function body with: ForLoop (iterate items), Assignment (accumulate), IfStatement (conditional factor), Return + - 1 Variable with complex type: optional[map[string, list[int]]] + - Annotations: DerefStrategy on function, OptimizationLock metadata, LangSpecific on variable +- Verify model loads without errors +- Marker: "Test model created and validated" + +**Estimated Duration**: 20-25 hours of focused implementation + +**Quality Gates**: +- SemAnno language compiles with zero errors +- Test model loads and renders without errors +- All concept definitions are complete (no stub concepts) + +--- + +### Phase 2: Editor Definition + +**Objectives**: Define editors for all concepts in SemAnno.editor.mps so AST is manually creatable and editable in MPS + +**Deliverables**: +- SemAnno.editor.mps with editors for all 30+ concepts +- Editors use MPS cell models (constant, property, reference cells) with layout (horizontal, vertical, indent) +- Syntax highlighting: keywords in blue (def, return, if, for, while), strings in green, numbers in magenta, types in blue +- No projection switching yet (that comes in Phase 2 of Python/C++ features) + +**Key Subtasks**: + +#### 2.1: Container & Function Editors (2-3 hours) +- Module editor: "module" keyword, name property, vertical list of variables and functions +- Function editor: "def" keyword, name, parameters in parens separated by commas, "->", return type, ":", function body with indentation +- Parameter editor: name, ":", type reference +- Block editor: indented statements vertically + +#### 2.2: Statement Editors (4-5 hours) +- Assignment: target expression, "=", value expression +- IfStatement: "if" keyword (blue), condition, ":", then-branch indented, optional "else:" keyword, else-branch indented +- WhileLoop: "while" keyword, condition, ":", body indented +- ForLoop: "for" keyword, iterator name, "in" keyword, iterable expression, ":", body indented +- Return: "return" keyword, optional return expression +- ExpressionStatement: expression only +- Indentation and keyword highlighting applied consistently + +#### 2.3: Expression Editors (3-4 hours) +- BinaryOperation: left operand, operator property, right operand +- UnaryOperation: operator, operand +- FunctionCall: function name, "(", arguments comma-separated, ")" +- VariableReference: variable name property +- Literal editors: each shows its value with appropriate syntax highlighting (numbers, strings, keywords) +- ListLiteral: "[", elements comma-separated, "]" +- IndexAccess: target, "[", index, "]" +- MemberAccess: target, ".", member name + +#### 2.4: Type Editors (2-3 hours) +- PrimitiveType: enum displayed as name (int, float, string, bool) +- ListType: "[", element type, "]" +- SetType: "set[", element type, "]" +- MapType: "map[", key type, ",", value type, "]" +- TupleType: "tuple[", element types comma-separated, "]" +- ArrayType: type, "[", size expression, "]" +- OptionalType: type, "?" +- CustomType: type name property +- Type nesting renders naturally: "list[map[string, int]]" + +#### 2.5: Annotation Editors (1-2 hours) +- DerefStrategy: "@deref(", strategy enum, optional properties in comments/sub-lines, ")" +- OptimizationLock: "@lock(", lockedBy, ",", lockReason, optional properties, ")" +- LangSpecific: "@lang_specific(", language, ",", idiomType, ",", rawSyntax, optional properties, ")" +- Annotations render near the node they modify (before/after depending on position) + +#### 2.6: Editor Testing in MPS (2-3 hours) +- Rebuild language in MPS +- Manually create Module in test model +- Test each concept's editor: create, edit properties, add references +- Verify indentation, syntax highlighting, navigation work correctly +- Test nested structures (function → body → statements → expressions → nested types) + +**Estimated Duration**: 15-18 hours + +**Quality Gates**: +- All editors render without cell errors +- All properties are editable inline +- All references are selectable from dropdown/search +- Indentation and syntax highlighting work as specified +- Nested structures (functions with statements, statements with expressions, types with types) display correctly + +--- + +### Phase 3: Integration & Testing + +**Objectives**: Verify AST structure works end-to-end, document for next phase, prepare for Python/C++ projections + +**Deliverables**: +- Comprehensive test model (SemAnno.tests.mps) with multiple examples +- Behavior model stubs (SemAnno.behavior.mps) - empty for now, but structure in place +- Typesystem model stub (SemAnno.typesystem.mps) - empty for now +- Quick-start guide for creating AST in MPS + +**Key Subtasks**: + +#### 3.1: Extended Test Model (2-3 hours) +- Add 3-5 example functions covering: + - Simple function (no loops or conditionals) + - Function with nested loops and conditionals + - Function with complex types (nested collections, optional types) + - Function with annotations (DerefStrategy, OptimizationLock) + - Function with language-specific idiom annotations +- Verify all examples load and render without errors + +#### 3.2: Model Serialization Tests (1-2 hours) +- Save test models to disk (automatic via MPS) +- Close and reopen MPS +- Reload test models and verify no corruption +- Verify all editor state preserved + +#### 3.3: Quality & Correctness Checks (1-2 hours) +- Run SemAnno language rebuild: zero errors, zero warnings +- Run MPS consistency checks on test models +- Verify no dangling references or broken hierarchy +- Document any edge cases found (e.g., circular type references if possible) + +#### 3.4: Documentation & Quickstart (1-2 hours) +- Create `languages/SemAnno/QUICKSTART.md` with steps to: + - Create a new SemAnno model + - Add a Module and Function + - Add parameters and statements + - Create type references + - Add annotations +- Include screenshots or ASCII examples of expected editor rendering + +#### 3.5: Prepare for Phase 2 (Python Projection) (1 hour) +- Note any editor patterns that will need enhancement for projections +- Document how Python/C++ projection editors will overlay the core AST editors +- Ensure no blocking issues for projection implementation + +**Estimated Duration**: 7-10 hours + +**Quality Gates**: +- All test models load, render, and save without errors +- Serialization round-trips (save→close→reopen) work correctly +- Zero errors from MPS language rebuild and consistency checks +- Quickstart documentation is clear and complete + +--- + +## Phase Completion Criteria + +### Phase 1 Completion +- [ ] All 30+ concept definitions in SemAnno.structure.mps complete +- [ ] SemAnno.constraints.mps defines validation rules +- [ ] SimpleExample.mps test model created with multi-statement, multi-type function +- [ ] MPS rebuild produces zero errors, zero warnings + +### Phase 2 Completion +- [ ] All concept editors defined in SemAnno.editor.mps +- [ ] Manual AST creation works in MPS (can create Module → Function → Statements → Expressions) +- [ ] Syntax highlighting, indentation, and reference selection work correctly +- [ ] Complex nested types display correctly (e.g., "list[map[string, optional[int]]]") + +### Phase 3 Completion +- [ ] Extended test models (3-5 examples) all load without errors +- [ ] Save/close/reopen cycle works without corruption +- [ ] Zero errors from language rebuild and consistency checks +- [ ] Quickstart guide complete and documented + +--- + +## Data Model + +### Entities + +**Module** (root AST) +- Properties: name (string) +- References: annotations (Annotation*), variables (Variable*), functions (Function*) + +**Function** +- Properties: name (string) +- References: parameters (Parameter*), returnType (Type), body (Statement*), annotations (Annotation*) + +**Parameter** +- Properties: name (string) +- References: type (Type), defaultValue (Expression?) + +**Variable** +- Properties: name (string) +- References: type (Type), initializer (Expression?), annotations (Annotation*) + +**Statement** (abstract) +- Subtypes: Block, Assignment, IfStatement, WhileLoop, ForLoop, Return, ExpressionStatement + +**Block** +- References: statements (Statement*) + +**Assignment** +- References: target (Expression), value (Expression) + +**IfStatement** +- References: condition (Expression), thenBranch (Statement*), elseBranch (Statement*) + +**WhileLoop** +- References: condition (Expression), body (Statement*) + +**ForLoop** +- Properties: iteratorName (string) +- References: iterable (Expression), body (Statement*) + +**Return** +- References: value (Expression?) + +**ExpressionStatement** +- References: expression (Expression) + +**Expression** (abstract) +- Subtypes: BinaryOperation, UnaryOperation, FunctionCall, VariableReference, Literals, ListLiteral, IndexAccess, MemberAccess + +**BinaryOperation** +- Properties: operator (string) +- References: left (Expression), right (Expression) + +**UnaryOperation** +- Properties: operator (string) +- References: operand (Expression) + +**FunctionCall** +- Properties: functionName (string) +- References: arguments (Expression*) + +**VariableReference** +- Properties: variableName (string) + +**Literals** +- IntegerLiteral: value (int) +- FloatLiteral: value (float) +- StringLiteral: value (string) +- BooleanLiteral: value (bool) +- NullLiteral: (no properties) + +**ListLiteral** +- References: elements (Expression*) + +**IndexAccess** +- References: target (Expression), index (Expression) + +**MemberAccess** +- Properties: memberName (string) +- References: target (Expression) + +**Type** (abstract) +- Subtypes: PrimitiveType, ListType, SetType, MapType, TupleType, ArrayType, OptionalType, CustomType + +**PrimitiveType** +- Properties: kind (enum: int, float, string, bool) + +**ListType** +- References: elementType (Type) + +**SetType** +- References: elementType (Type) + +**MapType** +- References: keyType (Type), valueType (Type) + +**TupleType** +- References: elementTypes (Type*) + +**ArrayType** +- References: elementType (Type), size (Expression) + +**OptionalType** +- References: innerType (Type) + +**CustomType** +- Properties: typeName (string) + +**Annotation** (abstract) +- Subtypes: DerefStrategy, OptimizationLock, LangSpecific + +**DerefStrategy** +- Properties: strategy (enum: imperative, streamed, batched, content_addressed), derefLocation (string?), owner (string?) +- References: derefTime (Expression?) + +**OptimizationLock** +- Properties: lockedBy (string), lockReason (string), lockLevel (enum: warning, soft, hard), affectedStrategies (string*), timestamp (string) + +**LangSpecific** +- Properties: language (enum: python, cpp, rust), idiomType (string), rawSyntax (string), semanticHint (string?), position (enum: before, after, wrapping) + +--- + +## Risk Mitigation + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|-----------| +| Circular type references (e.g., Type → Type) | Medium | High | Research MPS patterns during Phase 1; use composition, not inheritance | +| Editor cell model complexity | Medium | Medium | Reference MPS documentation and examples; iterate incrementally | +| Missing statement/expression types | Low | Medium | Verify against spec requirements before starting; use checklist | +| Annotation attachment logic issues | Low | Medium | Test annotation creation manually; verify multi-valued references work | +| Serialization/persistence issues | Low | High | Test save/close/reopen cycle extensively in Phase 3 | + +--- + +## Timeline Estimate + +**Total Duration**: 42-53 hours (5-7 working days at 8 hours/day, or 2-3 weeks part-time) + +- Phase 0 (Research): Concurrent with Phase 1, not blocking +- Phase 1 (Structure): 20-25 hours +- Phase 2 (Editors): 15-18 hours +- Phase 3 (Testing & Docs): 7-10 hours + +--- + +## Success Criteria (from Specification) + +1. ✅ **Manual Creation**: Developer can create Module with Function, statements, expressions in MPS with no red errors +2. ✅ **AST Completeness**: All concepts from Req 1.1-1.6 and 2.1 defined and working +3. ✅ **Editor Usability**: Selecting, modifying, deleting nodes works in editor (no manual XML editing) +4. ✅ **Type System**: Complex nested types render correctly +5. ✅ **Annotations**: DerefStrategy, OptimizationLock, LangSpecific can be created and attached +6. ✅ **Model Persistence**: Models save and reload without corruption +7. ✅ **Zero Errors**: Language compiles with zero errors, zero warnings +8. ✅ **Test Model**: Reference test model with multi-statement functions and annotations exists + +--- + +## Next Feature Prerequisites + +Once Phase 3 is complete: +- Python Projection feature can begin (uses core AST + Python-specific editors) +- C++ Projection feature can begin (uses core AST + C++-specific editors) +- Behavior model (SemAnno.behavior.mps) can be populated with import logic +- Typesystem model (SemAnno.typesystem.mps) can define type checking rules + +No blockers expected. Core AST is standalone and self-contained. diff --git a/features/001-core-ast-structure/tasks.md b/features/001-core-ast-structure/tasks.md new file mode 100644 index 0000000..4de1258 --- /dev/null +++ b/features/001-core-ast-structure/tasks.md @@ -0,0 +1,847 @@ +# Implementation Tasks: Core AST Structure (Feature 001) + +**Feature**: Core AST Structure for SemAnno +**Phase**: Planning Complete → Ready for Phase 1 Execution +**Total Tasks**: 26 +**Estimated Duration**: 42-53 hours + +--- + +## Legend + +- **[PHASE]**: 0 (Research), 1 (Structure), 2 (Editors), 3 (Testing) +- **[PRIORITY]**: P0 (blocker), P1 (must-have), P2 (should-have), P3 (nice-to-have) +- **[DURATION]**: Estimated hours +- **[DEPENDS]**: Task IDs that must complete first + +--- + +## Phase 0: Research & Exploration + +### RESEARCH-1: MPS Editor Design Patterns for Projections +**Priority**: P1 | **Duration**: 2h | **Depends**: None +**Status**: Pending + +**Objective**: Research how MPS languages support multiple projections (e.g., Python and C++ views on same AST). + +**Tasks**: +- Read MPS Language Design documentation on projection cells +- Find 2-3 examples in jetbrains.mps.lang.* languages (e.g., editor, structure) +- Document patterns: how are editors structured to allow runtime projection switching? +- Create summary: "Editor Design Patterns" with code examples + +**Definition of Done**: +- Summary document created +- At least 2 patterns identified and documented +- Phase 1 editor design can proceed using these patterns + +--- + +### RESEARCH-2: Nested Type Composition in MPS +**Priority**: P1 | **Duration**: 2h | **Depends**: None +**Status**: Pending + +**Objective**: Verify that deeply nested type references don't cause circular dependencies or serialization issues. + +**Tasks**: +- Research MPS reference composition patterns +- Test if Type → Type references can create lists[map[string, int]] without circular deps +- Document: How to prevent cycles, how to serialize nested structures +- Prepare example: 3-4 levels of nesting (list[map[string, optional[int]]]) + +**Definition of Done**: +- Patterns documented +- Circular reference prevention strategy clear +- Example type hierarchy designed + +--- + +### RESEARCH-3: Annotation Attachment Patterns +**Priority**: P1 | **Duration**: 2h | **Depends**: None +**Status**: Pending + +**Objective**: Understand best practices for attaching multiple independent annotations to a single node. + +**Tasks**: +- Research multi-valued references in MPS +- Study how jetbrains.mps.lang.core.structure handles attributes/annotations +- Verify: Can a Function have [DerefStrategy, OptimizationLock, LangSpecific] simultaneously? +- Document ordering, optional/required logic, removal behavior + +**Definition of Done**: +- Annotation design pattern documented +- Multi-valued reference behavior verified +- Phase 1 structure design can proceed + +--- + +## Phase 1: Core Structure Definition + +### TASK-1-1: Define Root & Container Concepts +**Priority**: P0 | **Duration**: 2h | **Depends**: None +**Phase**: 1 +**Status**: Pending + +**Objective**: Define Module concept and establish container pattern. + +**Subtasks**: +1. Create Module concept in SemAnno.structure.mps + - name: string property + - annotations: Annotation* reference (multi-valued) + - variables: Variable* reference (multi-valued) + - functions: Function* reference (multi-valued) +2. Verify MPS recognizes all properties and references +3. Mark concept as "root" in MPS (if applicable) + +**Definition of Done**: +- Module concept defined and saved +- MPS structure editor shows concept without errors +- All 4 features (name property, 3 references) visible in model tree + +**Review Criteria**: +- [ ] Concept name, property names match spec +- [ ] References are multi-valued where appropriate +- [ ] No red errors in MPS + +--- + +### TASK-1-2: Define Function & Parameter Concepts +**Priority**: P0 | **Duration**: 3h | **Depends**: TASK-1-1 +**Phase**: 1 +**Status**: Pending + +**Objective**: Define Function and Parameter concepts with proper signature structure. + +**Subtasks**: +1. Create Function concept + - name: string property + - parameters: Parameter* reference (multi-valued, ordered) + - returnType: Type reference (single) + - body: Statement* reference (multi-valued, ordered) + - annotations: Annotation* reference (multi-valued) +2. Create Parameter concept + - name: string property + - type: Type reference (single) + - defaultValue: Expression? reference (optional) +3. Verify Function.parameters and Parameter.type are composable +4. Test: Create Function, add 2 Parameters with types, set returnType + +**Definition of Done**: +- Function and Parameter concepts defined +- Parameter ordering preserved +- Type references are composable (Type → Type reference works) +- Can create function with 2 parameters in test model without errors + +--- + +### TASK-1-3: Define Variable Concept +**Priority**: P0 | **Duration**: 1h | **Depends**: TASK-1-1 +**Phase**: 1 +**Status**: Pending + +**Objective**: Define Variable concept for module and local variables. + +**Subtasks**: +1. Create Variable concept + - name: string property + - type: Type reference (single) + - initializer: Expression? reference (optional) + - annotations: Annotation* reference (multi-valued) +2. Ensure Variable can be added to Module.variables +3. Test: Create Module → add Variable with type and initializer + +**Definition of Done**: +- Variable concept defined +- Can be added to Module and function scopes +- No errors in test model + +--- + +### TASK-1-4: Define All Statement Concepts +**Priority**: P0 | **Duration**: 5h | **Depends**: TASK-1-2 +**Phase**: 1 +**Status**: Pending + +**Objective**: Define abstract Statement parent and 7 concrete statement types. + +**Subtasks**: +1. Create abstract Statement concept (parent) +2. Create concrete statement concepts: + - **Block**: statements: Statement* (multi-valued) + - **Assignment**: target: Expression, value: Expression + - **IfStatement**: condition: Expression, thenBranch: Statement*, elseBranch: Statement? (optional) + - **WhileLoop**: condition: Expression, body: Statement* + - **ForLoop**: iteratorName: string property, iterable: Expression, body: Statement* + - **Return**: value: Expression? (optional) + - **ExpressionStatement**: expression: Expression +3. Verify inheritance hierarchy (all inherit from Statement) +4. Test: Create function → add each statement type to body +5. Test nesting: Block → Assignment → BinaryOperation expression +6. Verify statement ordering is preserved + +**Definition of Done**: +- All 7 concrete statement types defined +- All inherit from Statement parent +- Can create function with mixed statement types +- Nesting works (statements inside blocks, expressions inside statements) +- No errors in test model + +**Review Criteria**: +- [ ] All statement types from spec are defined +- [ ] Abstract Statement parent established +- [ ] Multi-valued statement references are ordered +- [ ] Optional references (elseBranch, return value) work correctly + +--- + +### TASK-1-5: Define All Expression Concepts +**Priority**: P0 | **Duration**: 5h | **Depends**: TASK-1-4 +**Phase**: 1 +**Status**: Pending + +**Objective**: Define abstract Expression parent and 13 concrete expression types. + +**Subtasks**: +1. Create abstract Expression concept (parent) +2. Create concrete expression concepts: + - **BinaryOperation**: operator: string, left: Expression, right: Expression + - **UnaryOperation**: operator: string, operand: Expression + - **FunctionCall**: functionName: string, arguments: Expression* (multi-valued) + - **VariableReference**: variableName: string + - **IntegerLiteral**: value: integer + - **FloatLiteral**: value: float + - **StringLiteral**: value: string + - **BooleanLiteral**: value: boolean + - **NullLiteral**: (no properties) + - **ListLiteral**: elements: Expression* (multi-valued) + - **IndexAccess**: target: Expression, index: Expression + - **MemberAccess**: target: Expression, memberName: string +3. Verify inheritance hierarchy +4. Test: Create Assignment with BinaryOperation on right side +5. Test nesting: FunctionCall → BinaryOperation arguments +6. Test all literal types can be created + +**Definition of Done**: +- All 13 concrete expression types defined +- All inherit from Expression parent +- Can create complex nested expressions +- Literals can be created and have correct value types +- No errors in test model + +**Review Criteria**: +- [ ] All expression types from spec defined +- [ ] Abstract Expression parent established +- [ ] Expression nesting works (expressions contain expressions) +- [ ] Operator strings work as properties +- [ ] Argument lists for FunctionCall are ordered + +--- + +### TASK-1-6: Define All Type Concepts +**Priority**: P0 | **Duration**: 4h | **Depends**: TASK-1-3 +**Phase**: 1 +**Status**: Pending + +**Objective**: Define abstract Type parent and 8 concrete type types with composition support. + +**Subtasks**: +1. Create abstract Type concept (parent) +2. Create concrete type concepts: + - **PrimitiveType**: kind: enum {int, float, string, bool} + - **ListType**: elementType: Type (reference) + - **SetType**: elementType: Type (reference) + - **MapType**: keyType: Type, valueType: Type + - **TupleType**: elementTypes: Type* (multi-valued, ordered) + - **ArrayType**: elementType: Type, size: Expression + - **OptionalType**: innerType: Type + - **CustomType**: typeName: string +3. Verify Type → Type references are composable +4. Test: Create type "list[map[string, int]]" by composition +5. Test: Create type "optional[tuple[int, string, bool]]" +6. Verify no circular reference issues +7. Test: Create Variable with complex type, render in editor + +**Definition of Done**: +- All 8 concrete type types defined +- Type references are fully composable (ListType.elementType → MapType.keyType → PrimitiveType) +- Can create complex nested types without errors +- PrimitiveType enum values work correctly +- No circular reference issues detected + +**Review Criteria**: +- [ ] All type types from spec defined +- [ ] Type composition works (type → type → type nesting) +- [ ] Enum values for PrimitiveType are: int, float, string, bool +- [ ] ArrayType.size accepts Expression references +- [ ] TupleType elementTypes preserve order + +--- + +### TASK-1-7: Define All Annotation Concepts +**Priority**: P1 | **Duration**: 3h | **Depends**: TASK-1-2, TASK-1-5 +**Phase**: 1 +**Status**: Pending + +**Objective**: Define abstract Annotation parent and 3 concrete annotation types. + +**Subtasks**: +1. Create abstract Annotation concept (parent, optional for extensibility) +2. Create concrete annotation concepts: + - **DerefStrategy**: + - strategy: enum {imperative, streamed, batched, content_addressed} + - derefTime: Expression? (optional) + - derefLocation: string? (optional) + - owner: string? (optional) + - **OptimizationLock**: + - lockedBy: string + - lockReason: string + - lockLevel: enum {warning, soft, hard} + - affectedStrategies: string* (multi-valued) + - timestamp: string + - **LangSpecific**: + - language: enum {python, cpp, rust} + - idiomType: string + - rawSyntax: string + - semanticHint: string? (optional) + - position: enum {before, after, wrapping} +3. Verify annotations can be attached to Function.annotations +4. Test: Create Function with 1 DerefStrategy annotation +5. Test: Create Variable with 1 LangSpecific annotation +6. Test: Attach multiple annotations to same node + +**Definition of Done**: +- All 3 concrete annotation types defined +- Optional properties work (derefTime, owner, semanticHint) +- Enum values are correct +- Can attach annotations to functions and variables +- Multi-valued affectedStrategies list works +- No errors in test model + +--- + +### TASK-1-8: Define Constraints & Validation Rules +**Priority**: P1 | **Duration**: 2h | **Depends**: TASK-1-4, TASK-1-5, TASK-1-6 +**Phase**: 1 +**Status**: Pending + +**Objective**: Create SemAnno.constraints.mps with basic validation to prevent invalid structures. + +**Subtasks**: +1. Create SemAnno.constraints.mps model file +2. Define constraints: + - Statements can only appear in Statement* contexts (Block.statements, Function.body, etc.) + - Expressions can only appear in Expression contexts + - Types can only appear in Type contexts + - Statements cannot be direct children of Expression + - Expressions cannot be direct children of Statement (only in ExpressionStatement) +3. Add can-be-parent/can-be-child rules to prevent invalid structures +4. Test: Attempt to place a Statement inside an Expression context → should be blocked by constraint + +**Definition of Done**: +- SemAnno.constraints.mps created +- Basic structural constraints defined +- MPS enforces constraint violations +- Invalid structures are rejected during creation + +--- + +### TASK-1-9: Create Test Model with Complex AST +**Priority**: P0 | **Duration**: 3h | **Depends**: TASK-1-7 +**Phase**: 1 +**Status**: Pending + +**Objective**: Create SimpleExample.mps demonstrating all concept types. + +**Subtasks**: +1. Create new SemAnno model file: `languages/SemAnno/tests/SimpleExample.mps` +2. Create Module with name "example" +3. Add 1 Function "sum" with: + - Parameters: items (list[int]), factor (float) + - returnType: float + - Body with 4 statements: + - ForLoop: iterate items, assign accumulator + - Assignment: result = accumulator * factor + - IfStatement: if factor > 0 then return result, else return 0 + - Return: return result +4. Add 1 Variable: cache (optional[map[string, list[int]]]) +5. Add annotations: + - DerefStrategy on function (strategy: batched) + - OptimizationLock on function (locked by "alice", reason "optimization") + - LangSpecific on variable (language: python, idiomType: "type_hint") +6. Save and reload model in MPS +7. Verify no errors, no warnings + +**Definition of Done**: +- SimpleExample.mps created with all concept types demonstrated +- Model includes: Module, Function, Parameters, Variable, all Statement types, nested Expressions, nested Types, Annotations +- Model loads and renders without errors +- Model serializes and deserializes correctly +- No red errors in MPS error list + +**Acceptance Criteria**: +- [ ] Module created with name +- [ ] Function with 2 parameters, return type, body +- [ ] Body contains ForLoop, Assignment, IfStatement, Return +- [ ] Variable with complex nested type +- [ ] All annotations attached and visible + +--- + +## Phase 1 Completion Checkpoint + +**Validation Checklist**: +- [ ] All 30+ concepts defined in SemAnno.structure.mps +- [ ] SemAnno.constraints.mps created with validation rules +- [ ] SimpleExample.mps created and loads without errors +- [ ] MPS language rebuild: zero errors, zero warnings +- [ ] All concept names match specification +- [ ] All properties and references match specification + +**Approval**: Ready to proceed to Phase 2 (Editors) only if all checkpoints pass. + +--- + +## Phase 2: Editor Definition + +### TASK-2-1: Define Module & Function Editors +**Priority**: P0 | **Duration**: 3h | **Depends**: TASK-1-2 +**Phase**: 2 +**Status**: Pending + +**Objective**: Create editor definitions for Module, Function, Parameter concepts. + +**Subtasks**: +1. Open SemAnno.editor.mps (create if needed) +2. Define Module editor: + - Render: "module" (blue keyword), space, name (editable property), newline + - List functions and variables vertically, indented +3. Define Function editor: + - Render: "def" (blue keyword), space, name, "(", parameters comma-separated, ")", space, "->", space, return type, ":", newline + - Body statements indented vertically +4. Define Parameter editor: + - Render: name, ":", space, type reference +5. Test in MPS: + - Create Module, add Function, add Parameters + - Verify rendering matches expected layout + - Verify indentation works + +**Definition of Done**: +- Module, Function, Parameter editors defined +- Editors render with proper syntax highlighting (blue keywords) +- Indentation works for nested structures +- Parameters display comma-separated +- Return type reference is selectable + +--- + +### TASK-2-2: Define All Statement Editors +**Priority**: P0 | **Duration**: 5h | **Depends**: TASK-2-1 +**Phase**: 2 +**Status**: Pending + +**Objective**: Create editor definitions for all 7 statement types. + +**Subtasks**: +1. Define Block editor: + - Render statements vertically, indented +2. Define Assignment editor: + - Render: target, space, "=", space, value +3. Define IfStatement editor: + - Render: "if" (blue), space, condition, ":", newline + - Then-branch indented + - Optional "else:" (blue), newline, else-branch indented +4. Define WhileLoop editor: + - Render: "while" (blue), space, condition, ":", newline, body indented +5. Define ForLoop editor: + - Render: "for" (blue), space, iteratorName, space, "in" (blue), space, iterable, ":", newline, body indented +6. Define Return editor: + - Render: "return" (blue), space, optional value expression +7. Define ExpressionStatement editor: + - Render: expression +8. Test: Create function with mixed statements, verify layout + +**Definition of Done**: +- All 7 statement editors defined +- Syntax highlighting applied (keywords in blue) +- Indentation rendered correctly for nested statements +- Optional branches (else, return value) display when present, hidden when absent + +--- + +### TASK-2-3: Define All Expression Editors +**Priority**: P0 | **Duration**: 4h | **Depends**: TASK-2-2 +**Phase**: 2 +**Status**: Pending + +**Objective**: Create editor definitions for all 13 expression types. + +**Subtasks**: +1. Define BinaryOperation editor: + - Render: left, space, operator, space, right +2. Define UnaryOperation editor: + - Render: operator, operand +3. Define FunctionCall editor: + - Render: functionName, "(", arguments comma-separated, ")" +4. Define VariableReference editor: + - Render: variableName +5. Define Literal editors (5 types): + - IntegerLiteral: value (number color - magenta) + - FloatLiteral: value (number color - magenta) + - StringLiteral: '"', value (string color - green), '"' + - BooleanLiteral: value (keyword color - blue) + - NullLiteral: "None" or "null" (keyword color - blue) +6. Define ListLiteral editor: + - Render: "[", elements comma-separated, "]" +7. Define IndexAccess editor: + - Render: target, "[", index, "]" +8. Define MemberAccess editor: + - Render: target, ".", memberName +9. Test: Create complex nested expressions, verify rendering + +**Definition of Done**: +- All 13 expression editors defined +- Syntax coloring applied (keywords blue, strings green, numbers magenta) +- Operators and operands display correctly +- Nested expressions render with proper precedence indication +- Literals render with appropriate colors + +--- + +### TASK-2-4: Define All Type Editors +**Priority**: P0 | **Duration**: 3h | **Depends**: TASK-2-3 +**Phase**: 2 +**Status**: Pending + +**Objective**: Create editor definitions for all 8 type types. + +**Subtasks**: +1. Define PrimitiveType editor: + - Render: kind enum value (int, float, string, bool) +2. Define ListType editor: + - Render: "list" (blue), "[", elementType, "]" +3. Define SetType editor: + - Render: "set" (blue), "[", elementType, "]" +4. Define MapType editor: + - Render: "map" (blue), "[", keyType, ",", space, valueType, "]" +5. Define TupleType editor: + - Render: "tuple" (blue), "[", elementTypes comma-separated, "]" +6. Define ArrayType editor: + - Render: elementType, "[", size expression, "]" +7. Define OptionalType editor: + - Render: innerType, "?" +8. Define CustomType editor: + - Render: typeName +9. Test: Create complex nested types (list[map[string, optional[int]]]), verify rendering + +**Definition of Done**: +- All 8 type editors defined +- Type keywords rendered in blue +- Nested types render naturally (list[map[...]]) +- Type references are selectable +- Composition works (can select complex types) + +--- + +### TASK-2-5: Define Annotation Editors +**Priority**: P1 | **Duration**: 2h | **Depends**: TASK-2-4 +**Phase**: 2 +**Status**: Pending + +**Objective**: Create editor definitions for all 3 annotation types. + +**Subtasks**: +1. Define DerefStrategy editor: + - Render: "@deref" (gray), "(", strategy enum, optional properties as sub-lines, ")" +2. Define OptimizationLock editor: + - Render: "@lock" (gray), "(", properties comma-separated, ")" + - Include: lockedBy, lockReason, lockLevel, affectedStrategies, timestamp +3. Define LangSpecific editor: + - Render: "@lang_specific" (gray), "(", language, ",", idiomType, ",", rawSyntax, optional properties, ")" +4. Test: Create function with each annotation type, verify rendering near function + +**Definition of Done**: +- All 3 annotation editors defined +- Annotation keywords in gray/italic (distinguishes from code) +- Enum values display correctly +- Optional properties hidden when absent +- Annotations render attached to parent node + +--- + +### TASK-2-6: Editor Testing & Refinement in MPS +**Priority**: P0 | **Duration**: 3h | **Depends**: TASK-2-5 +**Phase**: 2 +**Status**: Pending + +**Objective**: Rebuild language in MPS, test editors interactively, fix issues. + +**Subtasks**: +1. Rebuild SemAnno language in MPS + - Check for cell model errors + - Verify no "missing editor" errors +2. Open SimpleExample.mps and manually verify: + - Module renders correctly (name, lists functions/variables) + - Function renders with "def", name, parameters, return type, body + - All statements render with proper indentation and syntax highlighting + - All expressions render correctly (operators, operands, nesting) + - All types render correctly (composition, nesting) + - Annotations render near their nodes +3. Test interactive editing: + - Create new Module, add Function by adding statements + - Edit statement properties (if condition, loop iterator, etc.) + - Edit expression operators and operands + - Change types by selecting from dropdown + - Navigate between references (e.g., click parameter type → jump to Variable definition) +4. Fix any rendering issues: + - Indentation not working → adjust cell layout + - Colors not applying → verify color codes + - References not selectable → verify reference cell type +5. Document any workarounds or editor-specific behavior + +**Definition of Done**: +- Language rebuilds with zero errors, zero warnings +- All editors render without errors +- Interactive editing works (properties editable, references selectable) +- All syntax highlighting and indentation work correctly +- SimpleExample.mps renders as designed + +**Acceptance Criteria**: +- [ ] Language rebuild: zero errors +- [ ] All concepts have working editors +- [ ] Indentation works for nested structures +- [ ] Syntax highlighting works (blue keywords, green strings, magenta numbers) +- [ ] Properties can be edited inline +- [ ] References are selectable + +--- + +## Phase 2 Completion Checkpoint + +**Validation Checklist**: +- [ ] All concept editors defined in SemAnno.editor.mps +- [ ] MPS language rebuild: zero errors, zero warnings +- [ ] SimpleExample.mps renders without editor errors +- [ ] Manual AST creation works (can add statements, expressions, types) +- [ ] Syntax highlighting and indentation correct +- [ ] All references are selectable and navigable + +**Approval**: Ready to proceed to Phase 3 (Testing) only if all checkpoints pass. + +--- + +## Phase 3: Integration & Testing + +### TASK-3-1: Create Extended Test Models +**Priority**: P1 | **Duration**: 3h | **Depends**: TASK-2-6 +**Phase**: 3 +**Status**: Pending + +**Objective**: Create 3-5 test models demonstrating various AST structures. + +**Subtasks**: +1. Create `SimpleFunction.mps`: Basic function (no loops, minimal types) + - Function: greet(name: string) -> string + - Body: Return StringLiteral +2. Create `ComplexFunction.mps`: Nested control flow + - Function with ForLoop → IfStatement → BinaryOperation + - Demonstrates statement nesting +3. Create `NestedTypes.mps`: Complex type structures + - Variables with types: list[map[string, optional[int]]], tuple[int, string, bool] + - Demonstrates type composition +4. Create `WithAnnotations.mps`: Full example with annotations + - Function with DerefStrategy, OptimizationLock, LangSpecific + - Variables with annotations + - Demonstrates annotation attachment +5. Save all models and verify load without errors + +**Definition of Done**: +- 5 test models created +- Each demonstrates different AST features +- All load and render without errors +- Each model documents its purpose in comments (if MPS supports) + +--- + +### TASK-3-2: Test Model Serialization Round-Trips +**Priority**: P0 | **Duration**: 2h | **Depends**: TASK-3-1 +**Phase**: 3 +**Status**: Pending + +**Objective**: Verify models persist correctly and don't corrupt on save/reload. + +**Subtasks**: +1. For each test model: + - Save model to disk (automatic via MPS) + - Close MPS completely + - Reopen MPS and reload project + - Reload each test model + - Verify no errors, warnings, or corrupted nodes + - Verify content matches what was saved +2. Test model properties: + - All string properties preserved correctly + - All enum values preserved + - All references intact (no broken links) + - Statement and type ordering preserved +3. Document any serialization quirks or issues found + +**Definition of Done**: +- All test models round-trip successfully (save → close → reopen → load) +- No corruption or data loss detected +- Content and structure verified after reload + +--- + +### TASK-3-3: Quality & Correctness Verification +**Priority**: P0 | **Duration**: 2h | **Depends**: TASK-3-2 +**Phase**: 3 +**Status**: Pending + +**Objective**: Run comprehensive checks to ensure language quality. + +**Subtasks**: +1. Rebuild SemAnno language in MPS: + - Check for zero errors, zero warnings + - Verify all concept definitions are complete + - Verify all editor definitions are complete +2. Run MPS consistency checks on all test models: + - Check for dangling references + - Check for orphaned nodes + - Verify hierarchy is valid +3. Test constraint enforcement: + - Attempt to create invalid structure (e.g., Statement as Expression) + - Verify constraint blocks it +4. Document findings: Are there any edge cases or limitations? + +**Definition of Done**: +- Language rebuild: zero errors, zero warnings +- All test models pass consistency checks +- Constraints work as designed +- No unresolved issues + +--- + +### TASK-3-4: Write Quickstart Guide +**Priority**: P1 | **Duration**: 2h | **Depends**: TASK-3-3 +**Phase**: 3 +**Status**: Pending + +**Objective**: Document how to create AST in SemAnno for future users. + +**Deliverable**: `languages/SemAnno/QUICKSTART.md` + +**Contents**: +1. Overview: What is SemAnno, what can you do with it +2. Step-by-step guide: Create first Module + - Create new model file + - Add Module concept + - Set module name +3. Create first Function + - Add Function to Module.functions + - Set function name, return type + - Add parameters +4. Create statements and expressions + - Add statements to function body + - Show examples: ForLoop, Assignment, Return + - Show expression examples: BinaryOperation, FunctionCall +5. Work with types + - Create primitive type + - Create nested type (list[map[...]]) + - Show how types compose +6. Add annotations + - Attach DerefStrategy to function + - Attach OptimizationLock to variable +7. Tips & tricks + - How to navigate references + - How to edit properties + - Keyboard shortcuts (if any) +8. Examples section + - Link to test models (SimpleFunction.mps, ComplexFunction.mps, etc.) + - Show rendered output for each + +**Definition of Done**: +- QUICKSTART.md created +- All major workflows documented +- Step-by-step instructions clear +- Examples provided with screenshots or ASCII renderings +- No unexplained concepts + +--- + +### TASK-3-5: Prepare for Phase 2 Features (Python/C++ Projections) +**Priority**: P2 | **Duration**: 1h | **Depends**: TASK-3-4 +**Phase**: 3 +**Status**: Pending + +**Objective**: Document how Phase 2 features will extend core AST. + +**Deliverable**: `languages/SemAnno/NEXT_PHASES.md` + +**Contents**: +1. Overview: What comes next (Python Projection, C++ Projection, Tree-sitter Import) +2. Python Projection (Week 3-4): + - Will extend editors to show Python-syntax rendering + - Will add Python text generator + - Will not change core AST structure + - Dependencies: This feature (core AST structure) +3. C++ Projection (Week 5-6): + - Will extend editors to show C++-syntax rendering + - Will add C++ text generator + - Will add deref strategy translation + - Dependencies: This feature + Python Projection +4. Tree-sitter Import (Week 7-8): + - Will add behavior model (SemAnno.behavior.mps) with import logic + - Will integrate tree-sitter-python and tree-sitter-cpp + - Will parse source files into AST + - Dependencies: This feature + both projections +5. Warning System (Week 9-10): + - Will add OptimizationLock enforcement logic + - Will add warning UI to editors + - Will add provenance tracking + - Dependencies: All previous features +6. Blockers or constraints for next phases (if any) + +**Definition of Done**: +- NEXT_PHASES.md created +- Clear roadmap for future features +- Dependencies documented +- No surprises in Phase 2+ implementation + +--- + +## Phase 3 Completion Checkpoint + +**Validation Checklist**: +- [ ] 5 test models created (SimpleFunction, ComplexFunction, NestedTypes, WithAnnotations, etc.) +- [ ] All models load and render without errors +- [ ] Serialization round-trips work (save → close → reopen) +- [ ] Language rebuild: zero errors, zero warnings +- [ ] Consistency checks pass +- [ ] Constraints work as designed +- [ ] Quickstart guide complete and clear +- [ ] NEXT_PHASES.md prepared + +**Approval**: Feature 001 (Core AST Structure) is COMPLETE and ready for merge to parent branch. + +--- + +## Summary + +**Total Tasks**: 26 +**Phases**: 3 (Research, Implementation, Testing) +**Estimated Duration**: 42-53 hours +**Status**: Ready for execution + +**Next Step**: Begin Phase 1 (TASK-1-1) with core structure definition. + +--- + +## Approval Signature + +Once all tasks are complete and checkpoints pass: + +``` +Feature: 001 - Core AST Structure +Status: ✅ COMPLETE +Merged to: sprint-1-ast-redesign (parent branch) +Timestamp: [Completion date] +```