whetstone_DSL/features/001-core-ast-structure/spec.md

parent_branch, feature_number, status, created_at

parent_branch	feature_number	status	created_at
sprint-1-ast-redesign	001	In Progress	2026-02-03T16:31:00Z

Feature: Core AST Structure for SemAnno

Overview

Enable developers to manually create and manipulate abstract syntax tree (AST) structures in MPS using the SemAnno language. This establishes the foundational data model for representing Python and C++ code as a unified, language-agnostic AST. The core AST captures the essential elements of both languages: functions, variables, statements, expressions, types, and annotations.

This feature is the prerequisite for all downstream features (Python projection, C++ projection, tree-sitter import, and warning system). Without a well-defined and properly-edited AST structure, the projections and generators cannot function.

User Scenarios

Scenario 1: Junior Developer Manually Creates a Function

Actor: Junior Python developer learning Whetstone DSL

Flow:

1. Developer opens MPS and creates a new SemAnno Module
2. Creates a Function concept with name "calculate_total"
3. Adds two Parameters: "items" (list of numbers) and "discount" (float)
4. Sets return type to "float"
5. Adds statements to the function body: a for loop iterating over items, an assignment to accumulate a total, an if statement to apply discount, and a return statement
6. Sees the function rendered in the MPS editor with proper syntax highlighting and indentation
7. Manually verifies the AST structure is correct by examining the model tree

Outcome: A complete, editable function AST exists in the model. The developer understands how to create and navigate AST nodes.

Scenario 2: Developer Inspects Complex Type Structures

Actor: Developer working with nested data types

Flow:

1. Developer creates a Variable with type "list[map[string, widget]]"
2. Expands the type hierarchy and sees the nested structure: ListType → MapType → PrimitiveType
3. Adds another variable with type "optional[tuple[int, string, bool]]"
4. Verifies that tuple elements are correctly ordered and typed
5. Uses the editor's navigation to jump between type definitions and usages

Outcome: Complex nested types are correctly represented and navigable in the editor.

Scenario 3: Developer Adds Annotations to AST Nodes

Actor: Senior developer preparing for optimization

Flow:

1. Developer creates a Function and manually attaches a DerefStrategy annotation
2. Sets the strategy to "batched"
3. Views the annotation in the editor as "@deref(batched)"
4. Later switches the strategy to "streamed" and sees the annotation update
5. Adds an OptimizationLock annotation with metadata: locked by "senior_dev_alice", reason "SIMD vectorization", warning level

Outcome: Annotations are properly attached to nodes, visible in the editor, and can be modified.

Functional Requirements

Core AST Nodes (Structure)

Requirement 1.1: Module Concept

A Module is the root container for all code. It must support:

• name property (string): identifier for the module
• annotations reference (multi-valued): collection of annotation nodes
• variables reference (multi-valued): module-level variables
• functions reference (multi-valued): top-level function definitions

Requirement 1.2: Function Concept

A Function represents a callable unit of code. It must support:

• name property (string): function identifier
• parameters reference (multi-valued): ordered list of Parameter nodes
• returnType reference (single): type reference for return value
• body reference (multi-valued): ordered list of Statement nodes
• annotations reference (multi-valued): annotation nodes attached to the function

Requirement 1.3: Parameter Concept

A Parameter represents a function input. It must support:

• name property (string): parameter identifier
• type reference (single): type reference for the parameter
• defaultValue reference (single, optional): optional default expression

Requirement 1.4: Variable Concept

A Variable represents a named data container. It must support:

• name property (string): variable identifier
• type reference (single): type reference for the variable
• initializer reference (single, optional): optional expression for initialization
• annotations reference (multi-valued): annotation nodes

Requirement 1.5: Abstract Statement Concept

An abstract Statement parent concept that all statement types inherit from. Concrete statement types:

• Block: container for multiple statements
• Assignment: assign an expression to a target (variable reference or member access)
• IfStatement: conditional execution with condition, thenBranch, and optional elseBranch
• WhileLoop: loop with condition and body statements
• ForLoop: iteration loop with iterator name, iterable expression, and body
• Return: return from function with optional expression
• ExpressionStatement: standalone expression (function call, etc.)

Requirement 1.6: Abstract Expression Concept

An abstract Expression parent concept. Concrete expression types:

• BinaryOperation: left operand, operator (string), right operand
• UnaryOperation: operator (string), operand
• FunctionCall: function name, ordered list of argument expressions
• VariableReference: variable name
• Literal Types: IntegerLiteral (value: integer), FloatLiteral (value: decimal), StringLiteral (value: string), BooleanLiteral (value: boolean), NullLiteral (no value)
• ListLiteral: ordered list of element expressions
• IndexAccess: target expression, index expression
• MemberAccess: target expression, member name (string)

Type System (Structure)

Requirement 2.1: Abstract Type Concept

An abstract Type parent concept. Concrete type types:

• PrimitiveType: kind property (enum: int, float, string, bool)
• ListType: elementType reference (single)
• SetType: elementType reference (single)
• MapType: keyType reference (single), valueType reference (single)
• TupleType: elementTypes reference (multi-valued, ordered)
• ArrayType: elementType reference (single), size reference (single, expression for dimension)
• OptionalType: innerType reference (single)
• CustomType: typeName property (string, for user-defined types)

Annotation Structures (Structure)

Requirement 3.1: DerefStrategy Annotation Concept

Represents a memory dereferencing strategy. Must support:

• strategy property (enum: imperative, streamed, batched, content_addressed)
• derefTime reference (single, optional): expression specifying when deref occurs (only for imperative)
• derefLocation property (string, optional): where deref happens (only for imperative)
• owner property (string, optional): agent reference

Requirement 3.2: OptimizationLock Annotation Concept

Represents a lock on optimization changes. Must support:

• lockedBy property (string): identifier of the agent who locked it
• lockReason property (string): explanation of why it was locked
• lockLevel property (enum: warning, soft, hard) – Sprint 1 uses "warning" only
• affectedStrategies property (string, multi-valued): names of affected deref strategies
• timestamp property (string): ISO datetime when lock was applied

Requirement 3.3: LangSpecific Annotation Concept

Preserves language-specific idioms during import. Must support:

• language property (enum: python, cpp, rust)
• idiomType property (string): "decorator", "template", "attribute", "pragma", etc.
• rawSyntax property (string): the original syntax as written
• semanticHint property (string, optional): "memoization", "generic", "parallelization", etc.
• position property (enum: before, after, wrapping): where it attaches to the node

Editor Integration (Editor Model)

Requirement 4.1: Editor Rendering for AST Nodes

Each concept must have an editor definition that:

• Renders the node's structure readably
• Uses syntax highlighting for keywords (if, else, while, for, def, return)
• Uses color coding for different element types (types in blue, keywords in blue, strings in green, numbers in magenta)
• Supports inline property editing (names, operators, keywords)
• Supports reference selection (picking target concepts)
• Indents nested structures (blocks, function bodies)

Requirement 4.2: Projection-Ready Editors

Editors must be built to support dual projection (Python and C++ renderings will be added in later features):

• Module editor shows module name and lists functions/variables
• Function editor shows signature and body with clear statement structure
• Statement editors show statement type-specific syntax (if/else indentation, loop structure, etc.)
• Expression editors render operators and operands clearly
• Type editors show type hierarchy (list[map[...]] nesting)

Testability

Requirement 5.1: Manual AST Creation Tests

The language must allow users to:

• Create a Module concept manually in MPS
• Add Functions with proper parameters and return types
• Add Statements to function bodies and have them parse correctly
• Add Expressions with operators and operands
• Add Type references and verify nesting
• All created structures must be serializable to the .mps model file and loadable without errors

Requirement 5.2: Property and Reference Integrity

• All properties must accept and save their defined types (strings, enums, etc.)
• All references must accept only the correct concept types
• Deletion of a referenced node must trigger appropriate error or cleanup
• The editor must prevent creating invalid structures (e.g., statements outside of statement lists, expressions in the wrong context)

Success Criteria

1. Manual Creation: A developer can manually create a Module with at least one Function containing statements and expressions in MPS, with no red error marks in the error list.

2. AST Completeness: The created AST models all required concepts from Req 1.1-1.6 and 2.1 with proper hierarchy and reference structure.

3. Editor Usability: The editor rendering supports selecting, modifying, and deleting AST nodes without requiring manual XML editing.

4. Type System: Complex nested types (e.g., list[map[string, optional[int]]]) can be created and rendered correctly in the editor.

5. Annotations: DerefStrategy, OptimizationLock, and LangSpecific annotations can be created and attached to appropriate nodes (functions, variables, statements).

6. Model Persistence: Created AST models save to .mps files without corruption and reload without errors.

7. Zero Errors: The SemAnno language compiles with zero errors and zero warnings when building and rebuilding.

8. Test Model: A test model exists in the project demonstrating a non-trivial AST (function with multiple statements, nested types, annotations) that serves as a reference for downstream features.

Key Entities

• Module: Root AST container
• Function: Callable unit with parameters, return type, and body
• Parameter: Named input to a function with a type
• Variable: Named data container with type and optional initializer
• Statement: Abstract parent for Block, Assignment, IfStatement, WhileLoop, ForLoop, Return, ExpressionStatement
• Expression: Abstract parent for operations, calls, literals, references, and access patterns
• Type: Abstract parent for primitive types, collections (list, set, map, tuple, array), optional types, and custom types
• Annotation: Metadata attached to AST nodes (DerefStrategy, OptimizationLock, LangSpecific)

Assumptions

1. MPS Version: JetBrains MPS 2024.3 or later is available and properly configured.

2. Single Language Module: All AST concepts, editors, and supporting models are defined in the SemAnno language module. No separate modules (e.g., "WhetstoneCore") are created during this phase.

3. Manual Creation: No automatic code generation or tree-sitter import is performed in this feature. AST creation is entirely manual via the MPS editor.

4. Stateless Annotations: Annotations are passive metadata. No annotation-specific behavior (e.g., warning logic) is implemented until the Warning System feature (Week 9-10).

5. Default Type Bindings: When a function or variable is created without an explicit type, the editor provides a default (e.g., OptionalType or a reasonable primitive).

6. English-Based Names: All concept names, properties, and references use English identifiers (no localization).

7. Linear Statement Ordering: Statements are always ordered (Block contains an ordered list), preserving the logical flow of code.

8. Reference Resolution: MPS's built-in reference resolution is used for all concept-to-concept references. No custom resolution logic is needed.

9. No Cross-Module References: AST nodes do not reference concepts outside the SemAnno language during this feature.

10. Single Root Module: Each .mps model file contains exactly one Module instance (the root of the AST).