whetstone_DSL/CODEGEN_BUG_REPORT.md

# generate_code Quality Gap Report
**Date**: 2026-03-01
**Observed by**: Claude Code session (Fabricate project, Task 1–2)
**Binary**: `editor/build-native/whetstone_mcp` (PID 3955, started from /home/bill/Documents)
**Mode**: `--language cpp`

---

## Bug 1: Non-C++ spec returns wrong fallback, gates mislead

**Tool**: `whetstone_generate_code`
**Input spec** (verbatim):

```
Generate a CMakeLists.txt for a CUDA C++17 project called WhetstoneAI_Fabricate targeting
Linux (Ubuntu 24.04), CUDA 13.1, GCC 13, CMake 3.28, GPU architecture sm_86 (RTX 3060).

Requirements:
- cmake_minimum_required(VERSION 3.28)
- project(WhetstoneAI_Fabricate LANGUAGES CXX CUDA)
[... full CMake spec with find_package, targets, link libraries ...]
```

**Returned `generatedCode`**:
```cpp
void generated(std::string input) {
    printf(input);
}
```

**Returned quality**:
```json
{
  "gates": {
    "compile": { "passed": false, "reason": "code_missing_structure_or_fallback" },
    "failure_reasons": ["compile_estimate_failed", "fallback_generation_detected"],
    "overall_ready": false
  },
  "quality": {
    "compilable_estimate": false,
    "fallback_used": true,
    "warnings": ["compile_estimate_failed", "fallback_generation_detected"]
  }
}
```

**What happened**: The tool recognized it couldn't handle CMake (fallback_generation_detected,
overall_ready: false) but instead of returning an error or empty result, it returned a
completely unrelated C++ printf stub. The fallback detection is correct — the response
format is wrong. A caller that checks `overall_ready` will discard it, but a caller that
reads `generatedCode` directly will get garbage.

**Expected behavior**: Either return an error message explaining CMake is not a supported
language in cpp mode, or return empty `generatedCode` with a clear `unsupported_language`
failure reason.

---

## Bug 2 (more serious): Detailed C++ spec returns generic stub, gates report success

**Tool**: `whetstone_generate_code`
**Input spec** (verbatim):

```
Generate a C++17 header file: include/trajectory_writer.h

This is the TrajectoryWriter class for a CUDA ML training logger. It writes a binary
trajectory file in the locked FABR format.

Dependencies: #include "reflective_tensor.h" (provides WeightMorphism struct: alignas(16),
fields uint64_t source_node_id, float sensitivity, uint32_t logic_axiom_id),
#include <cstdint>, #include <string>, #include <vector>, #include <fstream>

Class TrajectoryWriter:
- Constructor: TrajectoryWriter(const std::string& path, uint64_t num_params)
  Opens file at path for binary write. Writes the file header immediately:
    magic:      4 bytes  "FABR"
    version:    1 byte   0x01
    num_params: uint64_t (little-endian native)
    reserved:   7 bytes  all zero
  Throws std::runtime_error if file cannot be opened.
- void write_initial_weights(const float* weights)
  Writes raw float32[num_params] immediately after the header. Must be called exactly
  once before write_step.
- void write_step(uint64_t step_idx, const float* deltas,
                  const std::vector<WeightMorphism>& morphisms)
  Writes one per-step record: [step_index, morphism_count, morphism_data,
  compressed_size, compressed_data]. Byte-shuffle + Zstd compress.
- Destructor: closes the file.
- Private members: std::ofstream file_, uint64_t num_params_
- No copy constructor or assignment operator (delete both).

Also declare a free function:
  bool verify_trajectory(const std::string& path, uint64_t expected_steps)
  [description of reader behavior]
```

**Returned `generatedCode`**:
```cpp
#pragma once
#include <string>

namespace generated_module {

class GeneratedClass {
public:
    void run();
};

}  // namespace generated_module
```

**Returned quality**:
```json
{
  "gates": {
    "compile": { "passed": true, "reason": "ok" },
    "failure_reasons": [],
    "overall_ready": true,
    "placeholder": { "passed": true, "reason": "ok" },
    "tests": { "passed": true, "reason": "ok" }
  },
  "quality": {
    "compilable_estimate": true,
    "fallback_used": false,
    "placeholder_count": 0,
    "todo_count": 0,
    "warnings": []
  },
  "usedSymbols": []
}
```

**What happened**: The spec described a class with a specific name (`TrajectoryWriter`),
4 methods with specific signatures, specific includes, a deleted copy constructor,
and a free function (`verify_trajectory`). The generated code has:
- Wrong class name (`GeneratedClass` vs `TrajectoryWriter`)
- Wrong namespace (`generated_module` — not in spec)
- 1 method (`run()`) vs 4 specified methods
- None of the specified includes
- No free function
- No deleted copy constructor

**Critically**: `overall_ready: true`, `fallback_used: false`, `compilable_estimate: true`.
The gates all passed. The code compiles, but it is completely wrong. There is zero
correspondence between the spec and the output beyond "it is a class in a header file."

This is the dangerous case: a caller trusting the quality gates would accept this output.

---

## Hypothesis

The code generator appears to be pattern-matching on structural keywords in the spec
("class", "header") and emitting a minimal valid skeleton, but is not:
1. Extracting method signatures from the spec text
2. Extracting field names or types
3. Using the specified class name
4. Including specified dependencies
5. Generating any method bodies or stubs

The quality gates are measuring compilability of the generated code in isolation,
not correspondence to the spec. A generic `class Foo { void run(); }` will always
pass compile/placeholder/test gates.

---

## Suggested Fix Directions

**For Bug 1**: Add an `unsupported_language` quality gate failure that fires when
the spec appears to describe a non-C++ artifact (CMakeLists, shell script, JSON schema,
etc.) when running in `--language cpp` mode. Return empty `generatedCode` with a clear
error, not a C++ stub.

**For Bug 2**: The spec→AST extraction pipeline is not pulling method signatures.
The class name from the spec should become the AST ClassDeclaration name. Method
signatures described in the spec (name, parameters, return type) should each become
MethodDeclaration nodes. At minimum, a "spec fidelity" gate should fire if the
generated class name does not match any name mentioned in the spec.

---

## Reproducibility

- Mode: `--language cpp`
- Tool: `whetstone_generate_code`
- Both bugs reproduce on the exact spec strings above, sent as the `spec` parameter.
- No prior `start_recording` was called in this session (no LoRA data captured).
- Session started fresh from `/home/bill/Documents` (single PID confirmed: 3955).