unchecked-io/HANDOFF-2026-04-21.md

# Handoff — 2026-04-21

## What Was Done This Session

Fixed all four Phase 1 correctness bugs. Code compiles clean on temp-ubuntu with CUDA 13.1.

---

## Phase 1 Bugs — ALL FIXED

### 1.1 Dynamic buffer allocation — DONE
**Old behavior:** `max_elements = 4096 * 512` hardcoded in `PinnedBatcher::new`. Any batch exceeding ~2M tokens returned an error.
**Fix:** Removed `gpu_buffer`, `host_buffer`, and `max_elements` from the `PinnedBatcher` struct entirely. Both buffers are now allocated per-call based on actual `batch_size * seq_len`.

### 1.2 Buffer isolation (DLPack overwrite) — DONE
**Old behavior:** Every call overwrote the same `CudaSlice`. Any tensor from call N got corrupted by call N+1.
**Fix:** Each call allocates its own device buffer and wraps it in a `DLPackContext` struct stored in `DLManagedTensor.manager_ctx`. The deleter frees both the device memory (by dropping `CudaSlice<i64>`) and the shape array when PyTorch releases the tensor. Each returned tensor now owns its allocation.

### 1.3 Thread safety — DONE
**Old behavior:** `batch_encode_to_gpu` took `&mut self`. `TokenizerEngine.encode_batch` took `&mut self`. Not usable from multiple Python threads.
**Fix:** Since `PinnedBatcher` no longer has mutable shared state (buffers are per-call), `batch_encode_to_gpu` now takes `&self`. `TokenizerEngine` wraps `Arc<PinnedBatcher>` instead of owning it directly. `encode_batch` takes `&self`. Concurrent calls from Python threads are safe.

### 1.4 Variable sequence length + padding — DONE
**Old behavior:** `seq_len = encodings[0].len()` — assumed uniform length. Variable-length batches produced wrong tensor shape.
**Fix:** `seq_len = encodings.iter().map(|e| e.len()).max()`. Host buffer prefilled with `pad_id` (from `tokenizer.get_padding()`, fallback to 0). Rayon fill loop only writes actual tokens; shorter sequences retain the pad value.

---

## Build Issue Discovered and Resolved

`cudarc 0.11.9` panics at build time on CUDA 13.1 (only knows 12.5 and below). This machine has CUDA 13.1.

**Fix:** Changed the `gpu` feature in `Cargo.toml` from `cudarc/cuda-version-from-build-system` to `cudarc/cuda-12050`. This hard-pins to CUDA 12.5 headers and skips the `nvcc` version check. CUDA 13.x is backward-compatible with 12.5 driver API at runtime. If you ever upgrade cudarc to 0.12+, revert this change and re-test.

---

## Files Changed

| File | What Changed |
|------|-------------|
| `src/llm.rs` | Full rewrite of `PinnedBatcher` — removed static buffers, per-call allocation, `DLPackContext` for ownership, `&self` throughout |
| `src/lib.rs` | `TokenizerEngine.inner` changed to `Arc<PinnedBatcher>`, `encode_batch` takes `&self`, added `use std::sync::Arc` |
| `Cargo.toml` | `gpu` feature: `cuda-version-from-build-system` → `cuda-12050` |

---

## This Week — What's Left (in order)

### Today / Tomorrow (Tuesday 2026-04-22)
- [ ] Rent H100 on Lambda Labs (~$2 for 30 min)
- [ ] Run `benchmark_gpu.py` under `nsys profile --stats=true python benchmark_gpu.py`
- [ ] Capture Nsight `.nsys-rep` file and download it
- [ ] Note the speedup number — this is the headline for Thursday's talk

**Still need:** H100 setup script (a single paste-into-terminal script for the cloud instance). Needs: Rust install, maturin build with `--features gpu`, nsys profile run, output capture. This was not written last session.

### Wednesday (2026-04-23)
- [ ] Update `docs/talk-10min.md` Slide 6 with the H100 result
- [ ] Open Nsight Systems GUI locally, load the `.nsys-rep` file, screenshot the timeline showing both pipelines

### Thursday (2026-04-24)
- [ ] Talk to IT/security group (10 min)
- [ ] If someone gives you a contact: do not show GPU source code without NDA signed first; first call is discovery

---

## Next Session Priorities (after Thursday)

1. **Write H100 setup script** — single paste-into-terminal script (see above)
2. **vLLM integration** — Phase 2.1, this is the demo that sells the concept
3. **NM anonymous LLC** — Wyoming, single-member, before any contract is signed
4. **C++ version scoping** — after Phase 1 solid, assess Whetstone transpilation of `llm.rs`

---

## Key Technical Context

- Repo: `/home/bill/Documents/unchecked-io/`, on `gpu-zero` branch
- Build: `maturin develop --release --features gpu` (maturin not installed on temp-ubuntu — install with `pip install maturin` first)
- `cargo check --features gpu` confirms clean compile
- The four Phase 1 bugs are fixed and verified to compile
- `src/llm.rs` is the GPU core. `src/lib.rs` is the Python-facing wrapper.
- `chrono-tz` pinned at `=0.10.4` — do not change
- `gpu-zero` branch is private; `main` is open source Apache 2.0