diff --git a/features/001-core-ast-structure/checklists/requirements.md b/features/001-core-ast-structure/checklists/requirements.md new file mode 100644 index 0000000..e1c9ea9 --- /dev/null +++ b/features/001-core-ast-structure/checklists/requirements.md @@ -0,0 +1,42 @@ +# Specification Quality Checklist: Core AST Structure + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-02-03 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Validation Results + +✅ **All items PASS** - Specification is complete and ready for planning phase. + +## Notes + +- Specification derives directly from SPRINT_1_REQUIREMENTS.md (TR-1: Core AST Nodes) +- Annotation concepts (DerefStrategy, OptimizationLock, LangSpecific) are included in structure but their behavior is deferred to later features +- Editor rendering requirements are detailed enough to guide implementation without prescribing specific MPS cell models +- Type system includes all required types from requirements plus CustomType for extensibility +- Assumptions document necessary context for downstream features diff --git a/features/001-core-ast-structure/spec.md b/features/001-core-ast-structure/spec.md new file mode 100644 index 0000000..6376f69 --- /dev/null +++ b/features/001-core-ast-structure/spec.md @@ -0,0 +1,235 @@ +--- +parent_branch: sprint-1-ast-redesign +feature_number: 001 +status: In Progress +created_at: 2026-02-03T16:31:00Z +--- + +# Feature: Core AST Structure for SemAnno + +## Overview + +Enable developers to manually create and manipulate abstract syntax tree (AST) structures in MPS using the SemAnno language. This establishes the foundational data model for representing Python and C++ code as a unified, language-agnostic AST. The core AST captures the essential elements of both languages: functions, variables, statements, expressions, types, and annotations. + +This feature is the prerequisite for all downstream features (Python projection, C++ projection, tree-sitter import, and warning system). Without a well-defined and properly-edited AST structure, the projections and generators cannot function. + +## User Scenarios + +### Scenario 1: Junior Developer Manually Creates a Function + +**Actor**: Junior Python developer learning Whetstone DSL + +**Flow**: +1. Developer opens MPS and creates a new SemAnno Module +2. Creates a Function concept with name "calculate_total" +3. Adds two Parameters: "items" (list of numbers) and "discount" (float) +4. Sets return type to "float" +5. Adds statements to the function body: a for loop iterating over items, an assignment to accumulate a total, an if statement to apply discount, and a return statement +6. Sees the function rendered in the MPS editor with proper syntax highlighting and indentation +7. Manually verifies the AST structure is correct by examining the model tree + +**Outcome**: A complete, editable function AST exists in the model. The developer understands how to create and navigate AST nodes. + +### Scenario 2: Developer Inspects Complex Type Structures + +**Actor**: Developer working with nested data types + +**Flow**: +1. Developer creates a Variable with type "list[map[string, widget]]" +2. Expands the type hierarchy and sees the nested structure: ListType → MapType → PrimitiveType +3. Adds another variable with type "optional[tuple[int, string, bool]]" +4. Verifies that tuple elements are correctly ordered and typed +5. Uses the editor's navigation to jump between type definitions and usages + +**Outcome**: Complex nested types are correctly represented and navigable in the editor. + +### Scenario 3: Developer Adds Annotations to AST Nodes + +**Actor**: Senior developer preparing for optimization + +**Flow**: +1. Developer creates a Function and manually attaches a DerefStrategy annotation +2. Sets the strategy to "batched" +3. Views the annotation in the editor as "@deref(batched)" +4. Later switches the strategy to "streamed" and sees the annotation update +5. Adds an OptimizationLock annotation with metadata: locked by "senior_dev_alice", reason "SIMD vectorization", warning level + +**Outcome**: Annotations are properly attached to nodes, visible in the editor, and can be modified. + +## Functional Requirements + +### Core AST Nodes (Structure) + +#### Requirement 1.1: Module Concept +A Module is the root container for all code. It must support: +- **name** property (string): identifier for the module +- **annotations** reference (multi-valued): collection of annotation nodes +- **variables** reference (multi-valued): module-level variables +- **functions** reference (multi-valued): top-level function definitions + +#### Requirement 1.2: Function Concept +A Function represents a callable unit of code. It must support: +- **name** property (string): function identifier +- **parameters** reference (multi-valued): ordered list of Parameter nodes +- **returnType** reference (single): type reference for return value +- **body** reference (multi-valued): ordered list of Statement nodes +- **annotations** reference (multi-valued): annotation nodes attached to the function + +#### Requirement 1.3: Parameter Concept +A Parameter represents a function input. It must support: +- **name** property (string): parameter identifier +- **type** reference (single): type reference for the parameter +- **defaultValue** reference (single, optional): optional default expression + +#### Requirement 1.4: Variable Concept +A Variable represents a named data container. It must support: +- **name** property (string): variable identifier +- **type** reference (single): type reference for the variable +- **initializer** reference (single, optional): optional expression for initialization +- **annotations** reference (multi-valued): annotation nodes + +#### Requirement 1.5: Abstract Statement Concept +An abstract Statement parent concept that all statement types inherit from. Concrete statement types: +- **Block**: container for multiple statements +- **Assignment**: assign an expression to a target (variable reference or member access) +- **IfStatement**: conditional execution with condition, thenBranch, and optional elseBranch +- **WhileLoop**: loop with condition and body statements +- **ForLoop**: iteration loop with iterator name, iterable expression, and body +- **Return**: return from function with optional expression +- **ExpressionStatement**: standalone expression (function call, etc.) + +#### Requirement 1.6: Abstract Expression Concept +An abstract Expression parent concept. Concrete expression types: +- **BinaryOperation**: left operand, operator (string), right operand +- **UnaryOperation**: operator (string), operand +- **FunctionCall**: function name, ordered list of argument expressions +- **VariableReference**: variable name +- **Literal Types**: IntegerLiteral (value: integer), FloatLiteral (value: decimal), StringLiteral (value: string), BooleanLiteral (value: boolean), NullLiteral (no value) +- **ListLiteral**: ordered list of element expressions +- **IndexAccess**: target expression, index expression +- **MemberAccess**: target expression, member name (string) + +### Type System (Structure) + +#### Requirement 2.1: Abstract Type Concept +An abstract Type parent concept. Concrete type types: +- **PrimitiveType**: kind property (enum: int, float, string, bool) +- **ListType**: elementType reference (single) +- **SetType**: elementType reference (single) +- **MapType**: keyType reference (single), valueType reference (single) +- **TupleType**: elementTypes reference (multi-valued, ordered) +- **ArrayType**: elementType reference (single), size reference (single, expression for dimension) +- **OptionalType**: innerType reference (single) +- **CustomType**: typeName property (string, for user-defined types) + +### Annotation Structures (Structure) + +#### Requirement 3.1: DerefStrategy Annotation Concept +Represents a memory dereferencing strategy. Must support: +- **strategy** property (enum: imperative, streamed, batched, content_addressed) +- **derefTime** reference (single, optional): expression specifying when deref occurs (only for imperative) +- **derefLocation** property (string, optional): where deref happens (only for imperative) +- **owner** property (string, optional): agent reference + +#### Requirement 3.2: OptimizationLock Annotation Concept +Represents a lock on optimization changes. Must support: +- **lockedBy** property (string): identifier of the agent who locked it +- **lockReason** property (string): explanation of why it was locked +- **lockLevel** property (enum: warning, soft, hard) – Sprint 1 uses "warning" only +- **affectedStrategies** property (string, multi-valued): names of affected deref strategies +- **timestamp** property (string): ISO datetime when lock was applied + +#### Requirement 3.3: LangSpecific Annotation Concept +Preserves language-specific idioms during import. Must support: +- **language** property (enum: python, cpp, rust) +- **idiomType** property (string): "decorator", "template", "attribute", "pragma", etc. +- **rawSyntax** property (string): the original syntax as written +- **semanticHint** property (string, optional): "memoization", "generic", "parallelization", etc. +- **position** property (enum: before, after, wrapping): where it attaches to the node + +### Editor Integration (Editor Model) + +#### Requirement 4.1: Editor Rendering for AST Nodes +Each concept must have an editor definition that: +- Renders the node's structure readably +- Uses syntax highlighting for keywords (if, else, while, for, def, return) +- Uses color coding for different element types (types in blue, keywords in blue, strings in green, numbers in magenta) +- Supports inline property editing (names, operators, keywords) +- Supports reference selection (picking target concepts) +- Indents nested structures (blocks, function bodies) + +#### Requirement 4.2: Projection-Ready Editors +Editors must be built to support dual projection (Python and C++ renderings will be added in later features): +- Module editor shows module name and lists functions/variables +- Function editor shows signature and body with clear statement structure +- Statement editors show statement type-specific syntax (if/else indentation, loop structure, etc.) +- Expression editors render operators and operands clearly +- Type editors show type hierarchy (list[map[...]] nesting) + +### Testability + +#### Requirement 5.1: Manual AST Creation Tests +The language must allow users to: +- Create a Module concept manually in MPS +- Add Functions with proper parameters and return types +- Add Statements to function bodies and have them parse correctly +- Add Expressions with operators and operands +- Add Type references and verify nesting +- All created structures must be serializable to the .mps model file and loadable without errors + +#### Requirement 5.2: Property and Reference Integrity +- All properties must accept and save their defined types (strings, enums, etc.) +- All references must accept only the correct concept types +- Deletion of a referenced node must trigger appropriate error or cleanup +- The editor must prevent creating invalid structures (e.g., statements outside of statement lists, expressions in the wrong context) + +## Success Criteria + +1. **Manual Creation**: A developer can manually create a Module with at least one Function containing statements and expressions in MPS, with no red error marks in the error list. + +2. **AST Completeness**: The created AST models all required concepts from Req 1.1-1.6 and 2.1 with proper hierarchy and reference structure. + +3. **Editor Usability**: The editor rendering supports selecting, modifying, and deleting AST nodes without requiring manual XML editing. + +4. **Type System**: Complex nested types (e.g., list[map[string, optional[int]]]) can be created and rendered correctly in the editor. + +5. **Annotations**: DerefStrategy, OptimizationLock, and LangSpecific annotations can be created and attached to appropriate nodes (functions, variables, statements). + +6. **Model Persistence**: Created AST models save to .mps files without corruption and reload without errors. + +7. **Zero Errors**: The SemAnno language compiles with zero errors and zero warnings when building and rebuilding. + +8. **Test Model**: A test model exists in the project demonstrating a non-trivial AST (function with multiple statements, nested types, annotations) that serves as a reference for downstream features. + +## Key Entities + +- **Module**: Root AST container +- **Function**: Callable unit with parameters, return type, and body +- **Parameter**: Named input to a function with a type +- **Variable**: Named data container with type and optional initializer +- **Statement**: Abstract parent for Block, Assignment, IfStatement, WhileLoop, ForLoop, Return, ExpressionStatement +- **Expression**: Abstract parent for operations, calls, literals, references, and access patterns +- **Type**: Abstract parent for primitive types, collections (list, set, map, tuple, array), optional types, and custom types +- **Annotation**: Metadata attached to AST nodes (DerefStrategy, OptimizationLock, LangSpecific) + +## Assumptions + +1. **MPS Version**: JetBrains MPS 2024.3 or later is available and properly configured. + +2. **Single Language Module**: All AST concepts, editors, and supporting models are defined in the SemAnno language module. No separate modules (e.g., "WhetstoneCore") are created during this phase. + +3. **Manual Creation**: No automatic code generation or tree-sitter import is performed in this feature. AST creation is entirely manual via the MPS editor. + +4. **Stateless Annotations**: Annotations are passive metadata. No annotation-specific behavior (e.g., warning logic) is implemented until the Warning System feature (Week 9-10). + +5. **Default Type Bindings**: When a function or variable is created without an explicit type, the editor provides a default (e.g., OptionalType or a reasonable primitive). + +6. **English-Based Names**: All concept names, properties, and references use English identifiers (no localization). + +7. **Linear Statement Ordering**: Statements are always ordered (Block contains an ordered list), preserving the logical flow of code. + +8. **Reference Resolution**: MPS's built-in reference resolution is used for all concept-to-concept references. No custom resolution logic is needed. + +9. **No Cross-Module References**: AST nodes do not reference concepts outside the SemAnno language during this feature. + +10. **Single Root Module**: Each .mps model file contains exactly one Module instance (the root of the AST).