111 lines
3.8 KiB
Markdown
111 lines
3.8 KiB
Markdown
# UncheckedIO — 10-Minute Technical Talk
|
||
**Audience:** IT/Cybersecurity group with programmers and ML engineers
|
||
**Goal:** Plant the seed, demonstrate the mechanism, invite connections to LLM infrastructure teams
|
||
|
||
---
|
||
|
||
## Slide 1 — Open with the hidden thing (0:00–0:45)
|
||
|
||
Lead with an Nsight Systems timeline screenshot. Don't explain it yet.
|
||
|
||
> "This is what happens inside your GPU when a language model processes your request.
|
||
> The orange is waiting. The GPU costs $30,000 and it's waiting for data."
|
||
|
||
Let the image do the work. This audience will immediately want to know what they're looking at.
|
||
|
||
---
|
||
|
||
## Slide 2 — The problem (0:45–2:00)
|
||
|
||
**GPU starvation.** Define it in dollar terms.
|
||
|
||
- Real LLM providers run at 30–50% actual GPU utilization on inference
|
||
- An 8-GPU H100 cluster costs ~$20/hr to rent
|
||
- At 40% utilization, $12/hr is wasted — the GPUs are idle waiting for data
|
||
- LLM providers run hundreds of clusters
|
||
- The waste is structural and nobody has fixed the root cause
|
||
|
||
---
|
||
|
||
## Slides 3–4 — The mechanism (2:00–4:00)
|
||
|
||
Two diagrams, side by side.
|
||
|
||
**Standard pipeline:**
|
||
```
|
||
CPU tokenizes → Python tensor (pageable memory) → .to("cuda") → kernel buffer → GPU
|
||
^--- two OS-mediated copies
|
||
```
|
||
|
||
**UncheckedIO pipeline:**
|
||
```
|
||
Rust tokenizes in parallel → pinned memory (DMA-accessible) → direct DMA → DLPack → GPU
|
||
^--- one copy, no kernel involvement
|
||
```
|
||
|
||
The second diagram has fewer boxes and fewer arrows. That is the entire pitch.
|
||
|
||
Key terms to explain briefly:
|
||
- **Pinned memory** — memory registered with the OS so the GPU's DMA engine can read it directly
|
||
- **DMA** — the GPU pulls the data itself, no CPU involvement after the transfer starts
|
||
- **DLPack** — a standard zero-copy tensor exchange protocol; PyTorch accepts it natively with `torch.from_dlpack()`
|
||
|
||
---
|
||
|
||
## Slide 5 — The Nsight demo (4:00–7:00)
|
||
|
||
Show the recorded Nsight Systems timeline from the H100 benchmark run.
|
||
|
||
Both pipelines visible side by side:
|
||
- Point out the idle gaps (orange) in the standard pipeline
|
||
- Point out the absence of idle gaps in the UncheckedIO pipeline
|
||
- Show the DMA transfer lane — this is the "hidden thing" made visible
|
||
|
||
This is the centerpiece of the talk. Expect questions here. That's good.
|
||
|
||
---
|
||
|
||
## Slide 6 — The number (7:00–8:00)
|
||
|
||
One slide. No prose.
|
||
|
||
| Hardware | Speedup |
|
||
|----------|---------|
|
||
| RTX 3060 (local) | 4.7x |
|
||
| H100 SXM (cloud) | [Wednesday result] |
|
||
|
||
---
|
||
|
||
## Slide 7 — Market fit (8:00–9:00)
|
||
|
||
- This is infrastructure every LLM provider needs
|
||
- Built as an open-core library: PostgreSQL→Arrow is Apache 2.0 and already on PyPI
|
||
- GPU ingestion pipeline is the commercial layer
|
||
- AMD ROCm support (MI300X) is next — providers are actively evaluating MI300X to escape H100 pricing
|
||
|
||
Don't pitch deal specifics. This slide plants the seed.
|
||
|
||
---
|
||
|
||
## Slide 8 — The close (9:00–10:00)
|
||
|
||
> "I'm looking for connections to ML infrastructure teams and LLM providers.
|
||
> If you know someone building at that layer, I'd appreciate an introduction."
|
||
|
||
Direct. Not desperate. Leave room for questions.
|
||
|
||
---
|
||
|
||
## Notes
|
||
|
||
**What NOT to do:**
|
||
- Don't go deep on the Rust implementation — the visual is the hook, not the code
|
||
- Don't mention the PostgreSQL→Arrow piece — it's a distraction in this context
|
||
- Don't volunteer that it's unfinished — if asked, "it's in active development and the core mechanism is proven"
|
||
|
||
**For the well-connected engineer (separate talk, two weeks out):**
|
||
Don't reuse this version. Prep a deeper, tailored pitch after seeing what questions come up here.
|
||
|
||
**For the NVIDIA contact:**
|
||
Wait until after the H100 benchmark. Send the Nsight trace + one paragraph. Leading with a measurement is different from leading with a claim.
|