Fix all four Phase 1 correctness bugs (dynamic buffer, DLPack ownership, thread safety, padding)

- 1.1: Remove hardcoded 4096*512 buffer from PinnedBatcher; allocate host+device per-call
- 1.2: Wrap CudaSlice in DLPackContext owned by DLManagedTensor.manager_ctx; deleter frees
  device memory when PyTorch releases tensor — no more silent overwrite across calls
- 1.3: batch_encode_to_gpu now takes &self (no mutable shared state); TokenizerEngine wraps
  Arc<PinnedBatcher> and encode_batch takes &self — safe for concurrent Python threads
- 1.4: seq_len = max across all encodings; host buffer prefilled with pad_id before token write

Also: switch cudarc gpu feature from cuda-version-from-build-system to cuda-12050 to fix
build on machines with CUDA 13.x (cudarc 0.11.9 only knows up to 12.5)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Bill Holcombe
2026-04-21 14:10:31 -06:00
parent 6f8b6440aa
commit dc8494bb2a
4 changed files with 149 additions and 63 deletions

View File

@@ -15,7 +15,7 @@ gpu = [
"tokenizers",
"cudarc",
"pyo3-dlpack",
"cudarc/cuda-version-from-build-system",
"cudarc/cuda-12050",
"dep:cuda-driver-sys"
]
profiling = ["dep:tracing", "dep:tracing-subscriber", "dep:tracing-tracy"]
@@ -52,4 +52,5 @@ tracing-tracy = { version = "=0.11.2", optional = true }
tracing = {version ="0.1.41", optional = true }
[target.'cfg(not(target_env = "msvc"))'.dependencies]
mimalloc = { version = "0.1.39" }
mimalloc = { version = "0.1.39" }