docs: Add specification for feature 001 - Core AST Structure

features/001-core-ast-structure/spec.md

@@ -0,0 +1,235 @@
+---
+parent_branch: sprint-1-ast-redesign
+feature_number: 001
+status: In Progress
+created_at: 2026-02-03T16:31:00Z
+---
+
+# Feature: Core AST Structure for SemAnno
+
+## Overview
+
+Enable developers to manually create and manipulate abstract syntax tree (AST) structures in MPS using the SemAnno language. This establishes the foundational data model for representing Python and C++ code as a unified, language-agnostic AST. The core AST captures the essential elements of both languages: functions, variables, statements, expressions, types, and annotations.
+
+This feature is the prerequisite for all downstream features (Python projection, C++ projection, tree-sitter import, and warning system). Without a well-defined and properly-edited AST structure, the projections and generators cannot function.
+
+## User Scenarios
+
+### Scenario 1: Junior Developer Manually Creates a Function
+
+**Actor**: Junior Python developer learning Whetstone DSL
+
+**Flow**:
+1. Developer opens MPS and creates a new SemAnno Module
+2. Creates a Function concept with name "calculate_total"
+3. Adds two Parameters: "items" (list of numbers) and "discount" (float)
+4. Sets return type to "float"
+5. Adds statements to the function body: a for loop iterating over items, an assignment to accumulate a total, an if statement to apply discount, and a return statement
+6. Sees the function rendered in the MPS editor with proper syntax highlighting and indentation
+7. Manually verifies the AST structure is correct by examining the model tree
+
+**Outcome**: A complete, editable function AST exists in the model. The developer understands how to create and navigate AST nodes.
+
+### Scenario 2: Developer Inspects Complex Type Structures
+
+**Actor**: Developer working with nested data types
+
+**Flow**:
+1. Developer creates a Variable with type "list[map[string, widget]]"
+2. Expands the type hierarchy and sees the nested structure: ListType → MapType → PrimitiveType
+3. Adds another variable with type "optional[tuple[int, string, bool]]"
+4. Verifies that tuple elements are correctly ordered and typed
+5. Uses the editor's navigation to jump between type definitions and usages
+
+**Outcome**: Complex nested types are correctly represented and navigable in the editor.
+
+### Scenario 3: Developer Adds Annotations to AST Nodes
+
+**Actor**: Senior developer preparing for optimization
+
+**Flow**:
+1. Developer creates a Function and manually attaches a DerefStrategy annotation
+2. Sets the strategy to "batched"
+3. Views the annotation in the editor as "@deref(batched)"
+4. Later switches the strategy to "streamed" and sees the annotation update
+5. Adds an OptimizationLock annotation with metadata: locked by "senior_dev_alice", reason "SIMD vectorization", warning level
+
+**Outcome**: Annotations are properly attached to nodes, visible in the editor, and can be modified.
+
+## Functional Requirements
+
+### Core AST Nodes (Structure)
+
+#### Requirement 1.1: Module Concept
+A Module is the root container for all code. It must support:
+- **name** property (string): identifier for the module
+- **annotations** reference (multi-valued): collection of annotation nodes
+- **variables** reference (multi-valued): module-level variables
+- **functions** reference (multi-valued): top-level function definitions
+
+#### Requirement 1.2: Function Concept
+A Function represents a callable unit of code. It must support:
+- **name** property (string): function identifier
+- **parameters** reference (multi-valued): ordered list of Parameter nodes
+- **returnType** reference (single): type reference for return value
+- **body** reference (multi-valued): ordered list of Statement nodes
+- **annotations** reference (multi-valued): annotation nodes attached to the function
+
+#### Requirement 1.3: Parameter Concept
+A Parameter represents a function input. It must support:
+- **name** property (string): parameter identifier
+- **type** reference (single): type reference for the parameter
+- **defaultValue** reference (single, optional): optional default expression
+
+#### Requirement 1.4: Variable Concept
+A Variable represents a named data container. It must support:
+- **name** property (string): variable identifier
+- **type** reference (single): type reference for the variable
+- **initializer** reference (single, optional): optional expression for initialization
+- **annotations** reference (multi-valued): annotation nodes
+
+#### Requirement 1.5: Abstract Statement Concept
+An abstract Statement parent concept that all statement types inherit from. Concrete statement types:
+- **Block**: container for multiple statements
+- **Assignment**: assign an expression to a target (variable reference or member access)
+- **IfStatement**: conditional execution with condition, thenBranch, and optional elseBranch
+- **WhileLoop**: loop with condition and body statements
+- **ForLoop**: iteration loop with iterator name, iterable expression, and body
+- **Return**: return from function with optional expression
+- **ExpressionStatement**: standalone expression (function call, etc.)
+
+#### Requirement 1.6: Abstract Expression Concept
+An abstract Expression parent concept. Concrete expression types:
+- **BinaryOperation**: left operand, operator (string), right operand
+- **UnaryOperation**: operator (string), operand
+- **FunctionCall**: function name, ordered list of argument expressions
+- **VariableReference**: variable name
+- **Literal Types**: IntegerLiteral (value: integer), FloatLiteral (value: decimal), StringLiteral (value: string), BooleanLiteral (value: boolean), NullLiteral (no value)
+- **ListLiteral**: ordered list of element expressions
+- **IndexAccess**: target expression, index expression
+- **MemberAccess**: target expression, member name (string)
+
+### Type System (Structure)
+
+#### Requirement 2.1: Abstract Type Concept
+An abstract Type parent concept. Concrete type types:
+- **PrimitiveType**: kind property (enum: int, float, string, bool)
+- **ListType**: elementType reference (single)
+- **SetType**: elementType reference (single)
+- **MapType**: keyType reference (single), valueType reference (single)
+- **TupleType**: elementTypes reference (multi-valued, ordered)
+- **ArrayType**: elementType reference (single), size reference (single, expression for dimension)
+- **OptionalType**: innerType reference (single)
+- **CustomType**: typeName property (string, for user-defined types)
+
+### Annotation Structures (Structure)
+
+#### Requirement 3.1: DerefStrategy Annotation Concept
+Represents a memory dereferencing strategy. Must support:
+- **strategy** property (enum: imperative, streamed, batched, content_addressed)
+- **derefTime** reference (single, optional): expression specifying when deref occurs (only for imperative)
+- **derefLocation** property (string, optional): where deref happens (only for imperative)
+- **owner** property (string, optional): agent reference
+
+#### Requirement 3.2: OptimizationLock Annotation Concept
+Represents a lock on optimization changes. Must support:
+- **lockedBy** property (string): identifier of the agent who locked it
+- **lockReason** property (string): explanation of why it was locked
+- **lockLevel** property (enum: warning, soft, hard) – Sprint 1 uses "warning" only
+- **affectedStrategies** property (string, multi-valued): names of affected deref strategies
+- **timestamp** property (string): ISO datetime when lock was applied
+
+#### Requirement 3.3: LangSpecific Annotation Concept
+Preserves language-specific idioms during import. Must support:
+- **language** property (enum: python, cpp, rust)
+- **idiomType** property (string): "decorator", "template", "attribute", "pragma", etc.
+- **rawSyntax** property (string): the original syntax as written
+- **semanticHint** property (string, optional): "memoization", "generic", "parallelization", etc.
+- **position** property (enum: before, after, wrapping): where it attaches to the node
+
+### Editor Integration (Editor Model)
+
+#### Requirement 4.1: Editor Rendering for AST Nodes
+Each concept must have an editor definition that:
+- Renders the node's structure readably
+- Uses syntax highlighting for keywords (if, else, while, for, def, return)
+- Uses color coding for different element types (types in blue, keywords in blue, strings in green, numbers in magenta)
+- Supports inline property editing (names, operators, keywords)
+- Supports reference selection (picking target concepts)
+- Indents nested structures (blocks, function bodies)
+
+#### Requirement 4.2: Projection-Ready Editors
+Editors must be built to support dual projection (Python and C++ renderings will be added in later features):
+- Module editor shows module name and lists functions/variables
+- Function editor shows signature and body with clear statement structure
+- Statement editors show statement type-specific syntax (if/else indentation, loop structure, etc.)
+- Expression editors render operators and operands clearly
+- Type editors show type hierarchy (list[map[...]] nesting)
+
+### Testability
+
+#### Requirement 5.1: Manual AST Creation Tests
+The language must allow users to:
+- Create a Module concept manually in MPS
+- Add Functions with proper parameters and return types
+- Add Statements to function bodies and have them parse correctly
+- Add Expressions with operators and operands
+- Add Type references and verify nesting
+- All created structures must be serializable to the .mps model file and loadable without errors
+
+#### Requirement 5.2: Property and Reference Integrity
+- All properties must accept and save their defined types (strings, enums, etc.)
+- All references must accept only the correct concept types
+- Deletion of a referenced node must trigger appropriate error or cleanup
+- The editor must prevent creating invalid structures (e.g., statements outside of statement lists, expressions in the wrong context)
+
+## Success Criteria
+
+1. **Manual Creation**: A developer can manually create a Module with at least one Function containing statements and expressions in MPS, with no red error marks in the error list.
+
+2. **AST Completeness**: The created AST models all required concepts from Req 1.1-1.6 and 2.1 with proper hierarchy and reference structure.
+
+3. **Editor Usability**: The editor rendering supports selecting, modifying, and deleting AST nodes without requiring manual XML editing.
+
+4. **Type System**: Complex nested types (e.g., list[map[string, optional[int]]]) can be created and rendered correctly in the editor.
+
+5. **Annotations**: DerefStrategy, OptimizationLock, and LangSpecific annotations can be created and attached to appropriate nodes (functions, variables, statements).
+
+6. **Model Persistence**: Created AST models save to .mps files without corruption and reload without errors.
+
+7. **Zero Errors**: The SemAnno language compiles with zero errors and zero warnings when building and rebuilding.
+
+8. **Test Model**: A test model exists in the project demonstrating a non-trivial AST (function with multiple statements, nested types, annotations) that serves as a reference for downstream features.
+
+## Key Entities
+
+- **Module**: Root AST container
+- **Function**: Callable unit with parameters, return type, and body
+- **Parameter**: Named input to a function with a type
+- **Variable**: Named data container with type and optional initializer
+- **Statement**: Abstract parent for Block, Assignment, IfStatement, WhileLoop, ForLoop, Return, ExpressionStatement
+- **Expression**: Abstract parent for operations, calls, literals, references, and access patterns
+- **Type**: Abstract parent for primitive types, collections (list, set, map, tuple, array), optional types, and custom types
+- **Annotation**: Metadata attached to AST nodes (DerefStrategy, OptimizationLock, LangSpecific)
+
+## Assumptions
+
+1. **MPS Version**: JetBrains MPS 2024.3 or later is available and properly configured.
+
+2. **Single Language Module**: All AST concepts, editors, and supporting models are defined in the SemAnno language module. No separate modules (e.g., "WhetstoneCore") are created during this phase.
+
+3. **Manual Creation**: No automatic code generation or tree-sitter import is performed in this feature. AST creation is entirely manual via the MPS editor.
+
+4. **Stateless Annotations**: Annotations are passive metadata. No annotation-specific behavior (e.g., warning logic) is implemented until the Warning System feature (Week 9-10).
+
+5. **Default Type Bindings**: When a function or variable is created without an explicit type, the editor provides a default (e.g., OptionalType or a reasonable primitive).
+
+6. **English-Based Names**: All concept names, properties, and references use English identifiers (no localization).
+
+7. **Linear Statement Ordering**: Statements are always ordered (Block contains an ordered list), preserving the logical flow of code.
+
+8. **Reference Resolution**: MPS's built-in reference resolution is used for all concept-to-concept references. No custom resolution logic is needed.
+
+9. **No Cross-Module References**: AST nodes do not reference concepts outside the SemAnno language during this feature.
+
+10. **Single Root Module**: Each .mps model file contains exactly one Module instance (the root of the AST).