111 lines
3.8 KiB
Markdown
111 lines
3.8 KiB
Markdown
|
# UncheckedIO — 10-Minute Technical Talk
|
|||
|
|
**Audience:** IT/Cybersecurity group with programmers and ML engineers
|
|||
|
|
**Goal:** Plant the seed, demonstrate the mechanism, invite connections to LLM infrastructure teams
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Slide 1 — Open with the hidden thing (0:00–0:45)
|
|||
|
|
|
|||
|
|
Lead with an Nsight Systems timeline screenshot. Don't explain it yet.
|
|||
|
|
|
|||
|
|
> "This is what happens inside your GPU when a language model processes your request.
|
|||
|
|
> The orange is waiting. The GPU costs $30,000 and it's waiting for data."
|
|||
|
|
|
|||
|
|
Let the image do the work. This audience will immediately want to know what they're looking at.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Slide 2 — The problem (0:45–2:00)
|
|||
|
|
|
|||
|
|
**GPU starvation.** Define it in dollar terms.
|
|||
|
|
|
|||
|
|
- Real LLM providers run at 30–50% actual GPU utilization on inference
|
|||
|
|
- An 8-GPU H100 cluster costs ~$20/hr to rent
|
|||
|
|
- At 40% utilization, $12/hr is wasted — the GPUs are idle waiting for data
|
|||
|
|
- LLM providers run hundreds of clusters
|
|||
|
|
- The waste is structural and nobody has fixed the root cause
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Slides 3–4 — The mechanism (2:00–4:00)
|
|||
|
|
|
|||
|
|
Two diagrams, side by side.
|
|||
|
|
|
|||
|
|
**Standard pipeline:**
|
|||
|
|
```
|
|||
|
|
CPU tokenizes → Python tensor (pageable memory) → .to("cuda") → kernel buffer → GPU
|
|||
|
|
^--- two OS-mediated copies
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**UncheckedIO pipeline:**
|
|||
|
|
```
|
|||
|
|
Rust tokenizes in parallel → pinned memory (DMA-accessible) → direct DMA → DLPack → GPU
|
|||
|
|
^--- one copy, no kernel involvement
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The second diagram has fewer boxes and fewer arrows. That is the entire pitch.
|
|||
|
|
|
|||
|
|
Key terms to explain briefly:
|
|||
|
|
- **Pinned memory** — memory registered with the OS so the GPU's DMA engine can read it directly
|
|||
|
|
- **DMA** — the GPU pulls the data itself, no CPU involvement after the transfer starts
|
|||
|
|
- **DLPack** — a standard zero-copy tensor exchange protocol; PyTorch accepts it natively with `torch.from_dlpack()`
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Slide 5 — The Nsight demo (4:00–7:00)
|
|||
|
|
|
|||
|
|
Show the recorded Nsight Systems timeline from the H100 benchmark run.
|
|||
|
|
|
|||
|
|
Both pipelines visible side by side:
|
|||
|
|
- Point out the idle gaps (orange) in the standard pipeline
|
|||
|
|
- Point out the absence of idle gaps in the UncheckedIO pipeline
|
|||
|
|
- Show the DMA transfer lane — this is the "hidden thing" made visible
|
|||
|
|
|
|||
|
|
This is the centerpiece of the talk. Expect questions here. That's good.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Slide 6 — The number (7:00–8:00)
|
|||
|
|
|
|||
|
|
One slide. No prose.
|
|||
|
|
|
|||
|
|
| Hardware | Speedup |
|
|||
|
|
|----------|---------|
|
|||
|
|
| RTX 3060 (local) | 4.7x |
|
|||
|
|
| H100 SXM (cloud) | [Wednesday result] |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Slide 7 — Market fit (8:00–9:00)
|
|||
|
|
|
|||
|
|
- This is infrastructure every LLM provider needs
|
|||
|
|
- Built as an open-core library: PostgreSQL→Arrow is Apache 2.0 and already on PyPI
|
|||
|
|
- GPU ingestion pipeline is the commercial layer
|
|||
|
|
- AMD ROCm support (MI300X) is next — providers are actively evaluating MI300X to escape H100 pricing
|
|||
|
|
|
|||
|
|
Don't pitch deal specifics. This slide plants the seed.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Slide 8 — The close (9:00–10:00)
|
|||
|
|
|
|||
|
|
> "I'm looking for connections to ML infrastructure teams and LLM providers.
|
|||
|
|
> If you know someone building at that layer, I'd appreciate an introduction."
|
|||
|
|
|
|||
|
|
Direct. Not desperate. Leave room for questions.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Notes
|
|||
|
|
|
|||
|
|
**What NOT to do:**
|
|||
|
|
- Don't go deep on the Rust implementation — the visual is the hook, not the code
|
|||
|
|
- Don't mention the PostgreSQL→Arrow piece — it's a distraction in this context
|
|||
|
|
- Don't volunteer that it's unfinished — if asked, "it's in active development and the core mechanism is proven"
|
|||
|
|
|
|||
|
|
**For the well-connected engineer (separate talk, two weeks out):**
|
|||
|
|
Don't reuse this version. Prep a deeper, tailored pitch after seeing what questions come up here.
|
|||
|
|
|
|||
|
|
**For the NVIDIA contact:**
|
|||
|
|
Wait until after the H100 benchmark. Send the Nsight trace + one paragraph. Leading with a measurement is different from leading with a claim.
|