190 lines
6.5 KiB
Markdown
190 lines
6.5 KiB
Markdown
# generate_code Quality Gap Report
|
||
**Date**: 2026-03-01
|
||
**Observed by**: Claude Code session (Fabricate project, Task 1–2)
|
||
**Binary**: `editor/build-native/whetstone_mcp` (PID 3955, started from /home/bill/Documents)
|
||
**Mode**: `--language cpp`
|
||
|
||
---
|
||
|
||
## Bug 1: Non-C++ spec returns wrong fallback, gates mislead
|
||
|
||
**Tool**: `whetstone_generate_code`
|
||
**Input spec** (verbatim):
|
||
|
||
```
|
||
Generate a CMakeLists.txt for a CUDA C++17 project called WhetstoneAI_Fabricate targeting
|
||
Linux (Ubuntu 24.04), CUDA 13.1, GCC 13, CMake 3.28, GPU architecture sm_86 (RTX 3060).
|
||
|
||
Requirements:
|
||
- cmake_minimum_required(VERSION 3.28)
|
||
- project(WhetstoneAI_Fabricate LANGUAGES CXX CUDA)
|
||
[... full CMake spec with find_package, targets, link libraries ...]
|
||
```
|
||
|
||
**Returned `generatedCode`**:
|
||
```cpp
|
||
void generated(std::string input) {
|
||
printf(input);
|
||
}
|
||
```
|
||
|
||
**Returned quality**:
|
||
```json
|
||
{
|
||
"gates": {
|
||
"compile": { "passed": false, "reason": "code_missing_structure_or_fallback" },
|
||
"failure_reasons": ["compile_estimate_failed", "fallback_generation_detected"],
|
||
"overall_ready": false
|
||
},
|
||
"quality": {
|
||
"compilable_estimate": false,
|
||
"fallback_used": true,
|
||
"warnings": ["compile_estimate_failed", "fallback_generation_detected"]
|
||
}
|
||
}
|
||
```
|
||
|
||
**What happened**: The tool recognized it couldn't handle CMake (fallback_generation_detected,
|
||
overall_ready: false) but instead of returning an error or empty result, it returned a
|
||
completely unrelated C++ printf stub. The fallback detection is correct — the response
|
||
format is wrong. A caller that checks `overall_ready` will discard it, but a caller that
|
||
reads `generatedCode` directly will get garbage.
|
||
|
||
**Expected behavior**: Either return an error message explaining CMake is not a supported
|
||
language in cpp mode, or return empty `generatedCode` with a clear `unsupported_language`
|
||
failure reason.
|
||
|
||
---
|
||
|
||
## Bug 2 (more serious): Detailed C++ spec returns generic stub, gates report success
|
||
|
||
**Tool**: `whetstone_generate_code`
|
||
**Input spec** (verbatim):
|
||
|
||
```
|
||
Generate a C++17 header file: include/trajectory_writer.h
|
||
|
||
This is the TrajectoryWriter class for a CUDA ML training logger. It writes a binary
|
||
trajectory file in the locked FABR format.
|
||
|
||
Dependencies: #include "reflective_tensor.h" (provides WeightMorphism struct: alignas(16),
|
||
fields uint64_t source_node_id, float sensitivity, uint32_t logic_axiom_id),
|
||
#include <cstdint>, #include <string>, #include <vector>, #include <fstream>
|
||
|
||
Class TrajectoryWriter:
|
||
- Constructor: TrajectoryWriter(const std::string& path, uint64_t num_params)
|
||
Opens file at path for binary write. Writes the file header immediately:
|
||
magic: 4 bytes "FABR"
|
||
version: 1 byte 0x01
|
||
num_params: uint64_t (little-endian native)
|
||
reserved: 7 bytes all zero
|
||
Throws std::runtime_error if file cannot be opened.
|
||
- void write_initial_weights(const float* weights)
|
||
Writes raw float32[num_params] immediately after the header. Must be called exactly
|
||
once before write_step.
|
||
- void write_step(uint64_t step_idx, const float* deltas,
|
||
const std::vector<WeightMorphism>& morphisms)
|
||
Writes one per-step record: [step_index, morphism_count, morphism_data,
|
||
compressed_size, compressed_data]. Byte-shuffle + Zstd compress.
|
||
- Destructor: closes the file.
|
||
- Private members: std::ofstream file_, uint64_t num_params_
|
||
- No copy constructor or assignment operator (delete both).
|
||
|
||
Also declare a free function:
|
||
bool verify_trajectory(const std::string& path, uint64_t expected_steps)
|
||
[description of reader behavior]
|
||
```
|
||
|
||
**Returned `generatedCode`**:
|
||
```cpp
|
||
#pragma once
|
||
#include <string>
|
||
|
||
namespace generated_module {
|
||
|
||
class GeneratedClass {
|
||
public:
|
||
void run();
|
||
};
|
||
|
||
} // namespace generated_module
|
||
```
|
||
|
||
**Returned quality**:
|
||
```json
|
||
{
|
||
"gates": {
|
||
"compile": { "passed": true, "reason": "ok" },
|
||
"failure_reasons": [],
|
||
"overall_ready": true,
|
||
"placeholder": { "passed": true, "reason": "ok" },
|
||
"tests": { "passed": true, "reason": "ok" }
|
||
},
|
||
"quality": {
|
||
"compilable_estimate": true,
|
||
"fallback_used": false,
|
||
"placeholder_count": 0,
|
||
"todo_count": 0,
|
||
"warnings": []
|
||
},
|
||
"usedSymbols": []
|
||
}
|
||
```
|
||
|
||
**What happened**: The spec described a class with a specific name (`TrajectoryWriter`),
|
||
4 methods with specific signatures, specific includes, a deleted copy constructor,
|
||
and a free function (`verify_trajectory`). The generated code has:
|
||
- Wrong class name (`GeneratedClass` vs `TrajectoryWriter`)
|
||
- Wrong namespace (`generated_module` — not in spec)
|
||
- 1 method (`run()`) vs 4 specified methods
|
||
- None of the specified includes
|
||
- No free function
|
||
- No deleted copy constructor
|
||
|
||
**Critically**: `overall_ready: true`, `fallback_used: false`, `compilable_estimate: true`.
|
||
The gates all passed. The code compiles, but it is completely wrong. There is zero
|
||
correspondence between the spec and the output beyond "it is a class in a header file."
|
||
|
||
This is the dangerous case: a caller trusting the quality gates would accept this output.
|
||
|
||
---
|
||
|
||
## Hypothesis
|
||
|
||
The code generator appears to be pattern-matching on structural keywords in the spec
|
||
("class", "header") and emitting a minimal valid skeleton, but is not:
|
||
1. Extracting method signatures from the spec text
|
||
2. Extracting field names or types
|
||
3. Using the specified class name
|
||
4. Including specified dependencies
|
||
5. Generating any method bodies or stubs
|
||
|
||
The quality gates are measuring compilability of the generated code in isolation,
|
||
not correspondence to the spec. A generic `class Foo { void run(); }` will always
|
||
pass compile/placeholder/test gates.
|
||
|
||
---
|
||
|
||
## Suggested Fix Directions
|
||
|
||
**For Bug 1**: Add an `unsupported_language` quality gate failure that fires when
|
||
the spec appears to describe a non-C++ artifact (CMakeLists, shell script, JSON schema,
|
||
etc.) when running in `--language cpp` mode. Return empty `generatedCode` with a clear
|
||
error, not a C++ stub.
|
||
|
||
**For Bug 2**: The spec→AST extraction pipeline is not pulling method signatures.
|
||
The class name from the spec should become the AST ClassDeclaration name. Method
|
||
signatures described in the spec (name, parameters, return type) should each become
|
||
MethodDeclaration nodes. At minimum, a "spec fidelity" gate should fire if the
|
||
generated class name does not match any name mentioned in the spec.
|
||
|
||
---
|
||
|
||
## Reproducibility
|
||
|
||
- Mode: `--language cpp`
|
||
- Tool: `whetstone_generate_code`
|
||
- Both bugs reproduce on the exact spec strings above, sent as the `spec` parameter.
|
||
- No prior `start_recording` was called in this session (no LoRA data captured).
|
||
- Session started fresh from `/home/bill/Documents` (single PID confirmed: 3955).
|