whetstone_DSL/features/001-core-ast-structure/plan.md

---
feature_number: 001
feature_name: Core AST Structure
status: Planning
created_at: 2026-02-03T16:45:00Z
spec_version: 1.0
---

# Implementation Plan: Core AST Structure for SemAnno

## Technical Context

**Technology Stack**:
- **Workbench**: JetBrains MPS 2024.3+
- **Target Language**: SemAnno (language definition in MPS)
- **Model Format**: MPS XML (.mps files)
- **Development Method**: Incremental concept definition with immediate editor testing

**Key Decisions**:
- All concepts defined in single SemAnno language module (no separate WhetstoneCore module)
- Editors built in SemAnno.editor.mps with projection support for Python/C++ (added in future features)
- No code generation or external tools required for this feature
- Type system uses reference composition for nested type structures (ListType contains reference to inner Type)

**Dependencies**:
- MPS 2024.3 devkit for language design
- None on external libraries or frameworks
- Builds on existing SemAnno structure model (assumed to exist from previous session)

## Constitution Check

**Project Constitution Principles** (from `.specswarm/constitution.md` or assumed defaults):
- Clarity over cleverness: ✅ All concepts have clear, intuitive semantics
- Consistency: ✅ Naming conventions and patterns applied uniformly
- Type safety: ✅ Strong typing via MPS reference system
- Testability: ✅ Manual AST creation is the test mechanism
- Documentation: ✅ Specification documents design rationale

**No conflicting principles detected** - proceeding with implementation.

## Implementation Phases

### Phase 0: Research & Exploration

**Objectives**: Resolve any ambiguities, document assumptions, identify edge cases

**Tasks**:

1. **Research Task 1: MPS Editor Design Patterns**
   - Question: How are projection-ready editors structured in MPS for multi-projection languages?
   - Goal: Understand patterns used in multi-view languages (e.g., JetBrains MPS itself with structure/interface views)
   - Deliverable: Quick reference for editor cell patterns that support projection switching
   - Owner: Implementation team (research before Phase 1)

2. **Research Task 2: Nested Type Composition in MPS**
   - Question: Best practices for building deeply nested type structures (list[map[string, optional[int]]])
   - Goal: Ensure type references are properly composable without circular dependencies
   - Deliverable: Sample MPS structure showing 3-4 levels of type nesting
   - Owner: Implementation team

3. **Research Task 3: Annotation Attachment Patterns**
   - Question: How do MPS languages attach multiple annotations to a single concept?
   - Goal: Ensure annotations are optional, ordered, and independently removable
   - Deliverable: Example annotation structure from existing MPS language (e.g., jetbrains.mps.lang.core)
   - Owner: Implementation team

**Status**: Research items are light and can be completed during Phase 1 implementation (not blocking)

---

### Phase 1: Core Structure Definition

**Objectives**: Define all 30+ AST concepts in SemAnno.structure.mps

**Deliverables**:
- SemAnno.structure.mps with all concepts defined (Module, Function, Variable, all Statement types, all Expression types, all Type types, annotation concepts)
- Preliminary constraints model (SemAnno.constraints.mps) with basic validation rules
- Test model file (e.g., `languages/SemAnno/tests/SimpleExample.mps`) demonstrating a complete AST

**Key Subtasks**:

#### 1.1: Root & Container Concepts (1-2 hours)
- Module concept with properties (name: string) and references (annotations, variables, functions)
- Marker: "Container concepts complete"

#### 1.2: Function & Parameter Concepts (2-3 hours)
- Function concept: name, parameters[], returnType, body[], annotations
- Parameter concept: name, type, defaultValue?
- Establish parameter ordering and type reference patterns
- Marker: "Function signature complete"

#### 1.3: Variable Concept (1 hour)
- Variable concept: name, type, initializer?, annotations
- Ensure type and initializer reference logic
- Marker: "Variable concept complete"

#### 1.4: Statement Concepts (4-5 hours)
- Abstract Statement parent
- Concrete: Block, Assignment, IfStatement, WhileLoop, ForLoop, Return, ExpressionStatement
- Each with appropriate properties and references
- Marker: "All 7 statement types complete"

#### 1.5: Expression Concepts (4-5 hours)
- Abstract Expression parent
- Binary/Unary operations with operator properties
- Function call, variable reference, literal types (5 literal types)
- List literal, index access, member access
- Total: 13 expression concept types
- Marker: "All 13 expression types complete"

#### 1.6: Type Concepts (3-4 hours)
- Abstract Type parent
- Concrete: PrimitiveType (with enum), ListType, SetType, MapType, TupleType, ArrayType, OptionalType, CustomType
- 8 concrete type concepts
- Ensure ListType.elementType references Type (composable)
- MapType.keyType and valueType both reference Type
- TupleType.elementTypes are ordered, multi-valued Type references
- Marker: "All 8 type concepts complete, nesting tested"

#### 1.7: Annotation Concepts (2-3 hours)
- Abstract Annotation parent (optional, for future extensibility)
- DerefStrategy: strategy enum, derefTime?, derefLocation?, owner?
- OptimizationLock: lockedBy, lockReason, lockLevel enum, affectedStrategies[], timestamp
- LangSpecific: language enum, idiomType, rawSyntax, semanticHint?, position enum
- Marker: "All 3 annotation types complete"

#### 1.8: Constraints & Validation (2 hours)
- Create SemAnno.constraints.mps
- Basic constraints: statements only in statement lists, expressions only in expression contexts, types reference only Type concepts
- Can-be-parent/can-be-child rules to prevent invalid structures
- Marker: "Constraints model complete"

#### 1.9: Test Model Creation (2-3 hours)
- Create `languages/SemAnno/tests/SimpleExample.mps` demonstrating:
  - 1 Module with name "example"
  - 1 Function "sum" with 2 parameters (items: list[int], factor: float), return type: float
  - Function body with: ForLoop (iterate items), Assignment (accumulate), IfStatement (conditional factor), Return
  - 1 Variable with complex type: optional[map[string, list[int]]]
  - Annotations: DerefStrategy on function, OptimizationLock metadata, LangSpecific on variable
- Verify model loads without errors
- Marker: "Test model created and validated"

**Estimated Duration**: 20-25 hours of focused implementation

**Quality Gates**:
- SemAnno language compiles with zero errors
- Test model loads and renders without errors
- All concept definitions are complete (no stub concepts)

---

### Phase 2: Editor Definition

**Objectives**: Define editors for all concepts in SemAnno.editor.mps so AST is manually creatable and editable in MPS

**Deliverables**:
- SemAnno.editor.mps with editors for all 30+ concepts
- Editors use MPS cell models (constant, property, reference cells) with layout (horizontal, vertical, indent)
- Syntax highlighting: keywords in blue (def, return, if, for, while), strings in green, numbers in magenta, types in blue
- No projection switching yet (that comes in Phase 2 of Python/C++ features)

**Key Subtasks**:

#### 2.1: Container & Function Editors (2-3 hours)
- Module editor: "module" keyword, name property, vertical list of variables and functions
- Function editor: "def" keyword, name, parameters in parens separated by commas, "->", return type, ":", function body with indentation
- Parameter editor: name, ":", type reference
- Block editor: indented statements vertically

#### 2.2: Statement Editors (4-5 hours)
- Assignment: target expression, "=", value expression
- IfStatement: "if" keyword (blue), condition, ":", then-branch indented, optional "else:" keyword, else-branch indented
- WhileLoop: "while" keyword, condition, ":", body indented
- ForLoop: "for" keyword, iterator name, "in" keyword, iterable expression, ":", body indented
- Return: "return" keyword, optional return expression
- ExpressionStatement: expression only
- Indentation and keyword highlighting applied consistently

#### 2.3: Expression Editors (3-4 hours)
- BinaryOperation: left operand, operator property, right operand
- UnaryOperation: operator, operand
- FunctionCall: function name, "(", arguments comma-separated, ")"
- VariableReference: variable name property
- Literal editors: each shows its value with appropriate syntax highlighting (numbers, strings, keywords)
- ListLiteral: "[", elements comma-separated, "]"
- IndexAccess: target, "[", index, "]"
- MemberAccess: target, ".", member name

#### 2.4: Type Editors (2-3 hours)
- PrimitiveType: enum displayed as name (int, float, string, bool)
- ListType: "[", element type, "]"
- SetType: "set[", element type, "]"
- MapType: "map[", key type, ",", value type, "]"
- TupleType: "tuple[", element types comma-separated, "]"
- ArrayType: type, "[", size expression, "]"
- OptionalType: type, "?"
- CustomType: type name property
- Type nesting renders naturally: "list[map[string, int]]"

#### 2.5: Annotation Editors (1-2 hours)
- DerefStrategy: "@deref(", strategy enum, optional properties in comments/sub-lines, ")"
- OptimizationLock: "@lock(", lockedBy, ",", lockReason, optional properties, ")"
- LangSpecific: "@lang_specific(", language, ",", idiomType, ",", rawSyntax, optional properties, ")"
- Annotations render near the node they modify (before/after depending on position)

#### 2.6: Editor Testing in MPS (2-3 hours)
- Rebuild language in MPS
- Manually create Module in test model
- Test each concept's editor: create, edit properties, add references
- Verify indentation, syntax highlighting, navigation work correctly
- Test nested structures (function → body → statements → expressions → nested types)

**Estimated Duration**: 15-18 hours

**Quality Gates**:
- All editors render without cell errors
- All properties are editable inline
- All references are selectable from dropdown/search
- Indentation and syntax highlighting work as specified
- Nested structures (functions with statements, statements with expressions, types with types) display correctly

---

### Phase 3: Integration & Testing

**Objectives**: Verify AST structure works end-to-end, document for next phase, prepare for Python/C++ projections

**Deliverables**:
- Comprehensive test model (SemAnno.tests.mps) with multiple examples
- Behavior model stubs (SemAnno.behavior.mps) - empty for now, but structure in place
- Typesystem model stub (SemAnno.typesystem.mps) - empty for now
- Quick-start guide for creating AST in MPS

**Key Subtasks**:

#### 3.1: Extended Test Model (2-3 hours)
- Add 3-5 example functions covering:
  - Simple function (no loops or conditionals)
  - Function with nested loops and conditionals
  - Function with complex types (nested collections, optional types)
  - Function with annotations (DerefStrategy, OptimizationLock)
  - Function with language-specific idiom annotations
- Verify all examples load and render without errors

#### 3.2: Model Serialization Tests (1-2 hours)
- Save test models to disk (automatic via MPS)
- Close and reopen MPS
- Reload test models and verify no corruption
- Verify all editor state preserved

#### 3.3: Quality & Correctness Checks (1-2 hours)
- Run SemAnno language rebuild: zero errors, zero warnings
- Run MPS consistency checks on test models
- Verify no dangling references or broken hierarchy
- Document any edge cases found (e.g., circular type references if possible)

#### 3.4: Documentation & Quickstart (1-2 hours)
- Create `languages/SemAnno/QUICKSTART.md` with steps to:
  - Create a new SemAnno model
  - Add a Module and Function
  - Add parameters and statements
  - Create type references
  - Add annotations
- Include screenshots or ASCII examples of expected editor rendering

#### 3.5: Prepare for Phase 2 (Python Projection) (1 hour)
- Note any editor patterns that will need enhancement for projections
- Document how Python/C++ projection editors will overlay the core AST editors
- Ensure no blocking issues for projection implementation

**Estimated Duration**: 7-10 hours

**Quality Gates**:
- All test models load, render, and save without errors
- Serialization round-trips (save→close→reopen) work correctly
- Zero errors from MPS language rebuild and consistency checks
- Quickstart documentation is clear and complete

---

## Phase Completion Criteria

### Phase 1 Completion
- [ ] All 30+ concept definitions in SemAnno.structure.mps complete
- [ ] SemAnno.constraints.mps defines validation rules
- [ ] SimpleExample.mps test model created with multi-statement, multi-type function
- [ ] MPS rebuild produces zero errors, zero warnings

### Phase 2 Completion
- [ ] All concept editors defined in SemAnno.editor.mps
- [ ] Manual AST creation works in MPS (can create Module → Function → Statements → Expressions)
- [ ] Syntax highlighting, indentation, and reference selection work correctly
- [ ] Complex nested types display correctly (e.g., "list[map[string, optional[int]]]")

### Phase 3 Completion
- [ ] Extended test models (3-5 examples) all load without errors
- [ ] Save/close/reopen cycle works without corruption
- [ ] Zero errors from language rebuild and consistency checks
- [ ] Quickstart guide complete and documented

---

## Data Model

### Entities

**Module** (root AST)
- Properties: name (string)
- References: annotations (Annotation*), variables (Variable*), functions (Function*)

**Function**
- Properties: name (string)
- References: parameters (Parameter*), returnType (Type), body (Statement*), annotations (Annotation*)

**Parameter**
- Properties: name (string)
- References: type (Type), defaultValue (Expression?)

**Variable**
- Properties: name (string)
- References: type (Type), initializer (Expression?), annotations (Annotation*)

**Statement** (abstract)
- Subtypes: Block, Assignment, IfStatement, WhileLoop, ForLoop, Return, ExpressionStatement

**Block**
- References: statements (Statement*)

**Assignment**
- References: target (Expression), value (Expression)

**IfStatement**
- References: condition (Expression), thenBranch (Statement*), elseBranch (Statement*)

**WhileLoop**
- References: condition (Expression), body (Statement*)

**ForLoop**
- Properties: iteratorName (string)
- References: iterable (Expression), body (Statement*)

**Return**
- References: value (Expression?)

**ExpressionStatement**
- References: expression (Expression)

**Expression** (abstract)
- Subtypes: BinaryOperation, UnaryOperation, FunctionCall, VariableReference, Literals, ListLiteral, IndexAccess, MemberAccess

**BinaryOperation**
- Properties: operator (string)
- References: left (Expression), right (Expression)

**UnaryOperation**
- Properties: operator (string)
- References: operand (Expression)

**FunctionCall**
- Properties: functionName (string)
- References: arguments (Expression*)

**VariableReference**
- Properties: variableName (string)

**Literals**
- IntegerLiteral: value (int)
- FloatLiteral: value (float)
- StringLiteral: value (string)
- BooleanLiteral: value (bool)
- NullLiteral: (no properties)

**ListLiteral**
- References: elements (Expression*)

**IndexAccess**
- References: target (Expression), index (Expression)

**MemberAccess**
- Properties: memberName (string)
- References: target (Expression)

**Type** (abstract)
- Subtypes: PrimitiveType, ListType, SetType, MapType, TupleType, ArrayType, OptionalType, CustomType

**PrimitiveType**
- Properties: kind (enum: int, float, string, bool)

**ListType**
- References: elementType (Type)

**SetType**
- References: elementType (Type)

**MapType**
- References: keyType (Type), valueType (Type)

**TupleType**
- References: elementTypes (Type*)

**ArrayType**
- References: elementType (Type), size (Expression)

**OptionalType**
- References: innerType (Type)

**CustomType**
- Properties: typeName (string)

**Annotation** (abstract)
- Subtypes: DerefStrategy, OptimizationLock, LangSpecific

**DerefStrategy**
- Properties: strategy (enum: imperative, streamed, batched, content_addressed), derefLocation (string?), owner (string?)
- References: derefTime (Expression?)

**OptimizationLock**
- Properties: lockedBy (string), lockReason (string), lockLevel (enum: warning, soft, hard), affectedStrategies (string*), timestamp (string)

**LangSpecific**
- Properties: language (enum: python, cpp, rust), idiomType (string), rawSyntax (string), semanticHint (string?), position (enum: before, after, wrapping)

---

## Risk Mitigation

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|-----------|
| Circular type references (e.g., Type → Type) | Medium | High | Research MPS patterns during Phase 1; use composition, not inheritance |
| Editor cell model complexity | Medium | Medium | Reference MPS documentation and examples; iterate incrementally |
| Missing statement/expression types | Low | Medium | Verify against spec requirements before starting; use checklist |
| Annotation attachment logic issues | Low | Medium | Test annotation creation manually; verify multi-valued references work |
| Serialization/persistence issues | Low | High | Test save/close/reopen cycle extensively in Phase 3 |

---

## Timeline Estimate

**Total Duration**: 42-53 hours (5-7 working days at 8 hours/day, or 2-3 weeks part-time)

- Phase 0 (Research): Concurrent with Phase 1, not blocking
- Phase 1 (Structure): 20-25 hours
- Phase 2 (Editors): 15-18 hours
- Phase 3 (Testing & Docs): 7-10 hours

---

## Success Criteria (from Specification)

1. ✅ **Manual Creation**: Developer can create Module with Function, statements, expressions in MPS with no red errors
2. ✅ **AST Completeness**: All concepts from Req 1.1-1.6 and 2.1 defined and working
3. ✅ **Editor Usability**: Selecting, modifying, deleting nodes works in editor (no manual XML editing)
4. ✅ **Type System**: Complex nested types render correctly
5. ✅ **Annotations**: DerefStrategy, OptimizationLock, LangSpecific can be created and attached
6. ✅ **Model Persistence**: Models save and reload without corruption
7. ✅ **Zero Errors**: Language compiles with zero errors, zero warnings
8. ✅ **Test Model**: Reference test model with multi-statement functions and annotations exists

---

## Next Feature Prerequisites

Once Phase 3 is complete:
- Python Projection feature can begin (uses core AST + Python-specific editors)
- C++ Projection feature can begin (uses core AST + C++-specific editors)
- Behavior model (SemAnno.behavior.mps) can be populated with import logic
- Typesystem model (SemAnno.typesystem.mps) can define type checking rules

No blockers expected. Core AST is standalone and self-contained.