Add roadmap, commercialization strategy, talk outline, and handoff notes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
67
HANDOFF-2026-04-20.md
Normal file
67
HANDOFF-2026-04-20.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Handoff — 2026-04-20
|
||||
|
||||
## What Was Decided This Session
|
||||
|
||||
- **Commercialization path:** Licensing first, IP sale after traction. Wyoming LLC before any contract.
|
||||
- **Language strategy:** Rust is source of truth. C++ generated via Whetstone + manual library substitution as enterprise deliverable. Both versions is a deliberate differentiator.
|
||||
- **Primary buyers:** GPU cloud providers and LLM inference companies (CoreWeave, Lambda Labs, Together AI, Fireworks AI, Modal). Not PyTorch/Meta.
|
||||
- **Hugging Face** is a secondary buyer worth a conversation — they own the `tokenizers` crate this code already uses.
|
||||
- **NVIDIA contact:** Approach after H100 benchmark. Send Nsight trace + one paragraph. Not before.
|
||||
- **NM funding:** Apply for NMSBA (Sandia/LANL hardware access). Skip money grants for now.
|
||||
- **Outsourcing:** Friend's software business for non-core work. 60-70% of contract value for defined scope. IP assignment clause required in writing first.
|
||||
|
||||
## Documents Created This Session
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `ROADMAP.md` | Full technical development plan, Phases 1–5 |
|
||||
| `docs/talk-10min.md` | 10-minute presentation structure for Wednesday |
|
||||
| `docs/commercialization-strategy.md` | Pricing, B2B sales process, structure, outsourcing |
|
||||
| `HANDOFF-2026-04-20.md` | This file |
|
||||
|
||||
---
|
||||
|
||||
## This Week — Time-Sensitive (in order)
|
||||
|
||||
### Before Wednesday
|
||||
- [ ] Fix Phase 1.1 — dynamic buffer allocation (`src/llm.rs`, `PinnedBatcher::new`)
|
||||
- [ ] Fix Phase 1.2 — buffer isolation / DLPack overwrite bug
|
||||
- [ ] Fix Phase 1.3 — thread-safe engine pool (`TokenizerEngine` takes `&mut self`)
|
||||
- [ ] Fix Phase 1.4 — padding for variable-length sequences
|
||||
|
||||
### Wednesday
|
||||
- [ ] Rent H100 on Lambda Labs (~$2 for 30 min)
|
||||
- [ ] Run `benchmark_gpu.py` under `nsys profile --stats=true python benchmark_gpu.py`
|
||||
- [ ] Capture Nsight `.nsys-rep` file and download it
|
||||
- [ ] Note the speedup number — this is the headline for the talk
|
||||
|
||||
### Wednesday/Thursday
|
||||
- [ ] Update `docs/talk-10min.md` Slide 6 with the H100 result
|
||||
- [ ] Open Nsight Systems GUI locally, load the `.nsys-rep` file, screenshot the timeline showing both pipelines
|
||||
|
||||
### Thursday (if someone gives you a contact after the talk)
|
||||
- [ ] Do not show GPU source code without NDA signed first
|
||||
- [ ] First call is discovery — ask about their infrastructure before pitching
|
||||
- [ ] Pricing range if pushed: $50K–$150K annually depending on scale
|
||||
|
||||
---
|
||||
|
||||
## Next Session Priorities (after this week)
|
||||
|
||||
1. **H100 setup script** — a single paste-into-terminal script for the cloud instance (was going to write this, ran out of session). Needs: Rust install, maturin build with `--features gpu`, nsys profile run, output capture.
|
||||
2. **Phase 1 fixes** — if not completed before Wednesday, finish these
|
||||
3. **vLLM integration** — Phase 2.1, this is the demo that sells the concept
|
||||
4. **NM anonymous LLC** — straightforward to form locally, no member names in public filings, no annual report requirement
|
||||
5. **C++ version scoping** — after Phase 1 is solid, assess Whetstone transpilation of `llm.rs`
|
||||
|
||||
---
|
||||
|
||||
## Key Technical Context for Next Session
|
||||
|
||||
- Repo is cloned at `/home/bill/Documents/unchecked-io/`, currently on `gpu-zero` branch
|
||||
- GPU feature is compiled with `maturin develop --release --features gpu`
|
||||
- The four Phase 1 bugs are documented in detail in `ROADMAP.md` Phase 1 section
|
||||
- `src/llm.rs` is the file to fix — `PinnedBatcher` struct, `batch_encode_to_gpu` method
|
||||
- `src/lib.rs` `TokenizerEngine` wraps `PinnedBatcher` and is the Python-facing class
|
||||
- chrono-tz is pinned at `=0.10.4` — do not change this, it was hard-won
|
||||
- `gpu-zero` branch is private; `main` branch is open source Apache 2.0
|
||||
160
ROADMAP.md
Normal file
160
ROADMAP.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# UncheckedIO — Development Roadmap
|
||||
|
||||
**Goal:** Production-grade, multi-GPU zero-copy ingestion pipeline for LLM providers.
|
||||
**Current state:** Working proof-of-concept on CUDA (gpu-zero branch). PostgreSQL→Arrow open source on main.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Fix Correctness (gpu-zero branch)
|
||||
|
||||
The GPU feature works in benchmark conditions but has four bugs that would surface immediately in production. Fix these before anything else.
|
||||
|
||||
### 1.1 Dynamic buffer allocation
|
||||
**Problem:** `max_elements = 4096 * 512` is hardcoded at init time. Modern LLMs have 32K–128K context windows. A single request at 32K tokens + any real batch size hits this limit.
|
||||
**Fix:** Allocate `gpu_buffer` and `host_buffer` per-call based on actual `batch_size * max_seq_len`, or use a growable strategy that reallocates when the current batch exceeds the existing buffer size.
|
||||
|
||||
### 1.2 Buffer isolation (silent data corruption bug)
|
||||
**Problem:** Every call to `encode_batch` overwrites the same `CudaSlice`. The DLPack capsule hands PyTorch a pointer to that same memory. A caller holding a tensor from call N gets its data silently overwritten by call N+1.
|
||||
**Fix:** Double-buffer (ping-pong between two allocations) or allocate a fresh device buffer per call and transfer ownership to the DLPack capsule's deleter so CUDA frees it when PyTorch releases the tensor.
|
||||
|
||||
### 1.3 Thread safety
|
||||
**Problem:** `encode_batch` takes `&mut self`. The `TokenizerEngine` Python class cannot be called from multiple threads simultaneously. Every production serving engine (vLLM, TGI) is multithreaded.
|
||||
**Fix:** Wrap `PinnedBatcher` in an `Arc<Mutex<>>` or, better, create a `EnginePool` that holds N instances and checks one out per call. Pool size = CPU core count is a reasonable default.
|
||||
|
||||
### 1.4 Variable sequence length + padding
|
||||
**Problem:** `seq_len = encodings[0].len()` assumes all sequences in the batch have identical length. The `tokenizers` crate doesn't apply padding by default, so real variable-length inputs produce an incorrect 2D tensor shape.
|
||||
**Fix:** Apply padding to `max(encodings[i].len())` within the batch, fill with the tokenizer's `pad_id`, and use the actual padded length as `seq_len`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Production Features
|
||||
|
||||
### 2.1 vLLM integration
|
||||
vLLM is the dominant open-source LLM serving engine. Make `unchecked_io` a drop-in tokenization backend for it.
|
||||
- Implement the `TokenizerGroup` protocol vLLM expects
|
||||
- The continuous batching scheduler sends variable-length requests asynchronously — this is where Phase 1 fixes must be solid
|
||||
- Target: a 3-line config change in vLLM to switch to the UncheckedIO backend
|
||||
- Deliverable: `examples/vllm_integration.py` showing the swap
|
||||
|
||||
### 2.2 Streaming / chunked output
|
||||
**Current:** The entire batch is tokenized and DMA'd in one blocking call.
|
||||
**Target:** Yield Arrow RecordBatches or DLPack tensors incrementally as tokens are ready, so the model can begin prefill before the full batch is staged.
|
||||
|
||||
### 2.3 Attention mask export
|
||||
LLM inference requires both `input_ids` and `attention_mask`. Currently only token IDs are exported. Add a second DLPack capsule for the mask, or pack both into a named dict capsule.
|
||||
|
||||
### 2.4 Configurable max context
|
||||
Replace the hardcoded constants with constructor arguments:
|
||||
```python
|
||||
engine = TokenizerEngine(
|
||||
tokenizer_path="tokenizer.json",
|
||||
max_seq_len=32768,
|
||||
max_batch_size=64,
|
||||
pool_size=8
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Multi-GPU Support
|
||||
|
||||
### 3.1 ROCm / HIP (AMD) — highest priority
|
||||
AMD's MI300X has 192GB memory vs the H100's 80GB and providers are actively evaluating it to escape NVIDIA pricing. HIP is the path of least resistance — function names are nearly 1:1 with CUDA (`hipMemHostRegister`, `hipMemcpyHtoD`, `hipMemHostUnregister`).
|
||||
|
||||
- Add `rocm` feature flag in `Cargo.toml`
|
||||
- Create `src/backends/rocm.rs` implementing the same `GpuBackend` trait
|
||||
- Use AMD's `hip-sys` or `rocm-sys` Rust crate
|
||||
- The DLPack export layer stays identical — device_type `10` for ROCm vs `2` for CUDA
|
||||
|
||||
**Refactor needed first:** Extract a `GpuBackend` trait from the current CUDA-specific code:
|
||||
```rust
|
||||
trait GpuBackend {
|
||||
fn allocate_pinned(capacity: usize) -> Result<PinnedBuffer>;
|
||||
fn copy_to_device(src: &[i64], dst: &mut DeviceBuffer) -> Result<()>;
|
||||
fn device_ptr(buf: &DeviceBuffer) -> u64;
|
||||
fn dlpack_device_type() -> i32;
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 OpenCL (broad coverage)
|
||||
Covers AMD, Intel, older NVIDIA, Apple (via Metal-OpenCL bridge). Lower peak performance than native CUDA/HIP but maximum hardware coverage. Use the `opencl3` Rust crate.
|
||||
- Add `opencl` feature flag
|
||||
- Useful for providers running mixed-GPU clusters
|
||||
|
||||
### 3.3 Intel Gaudi / Level Zero
|
||||
Intel's Gaudi 2/3 accelerators are gaining traction for training and are available on AWS and Azure. Intel Level Zero is the low-level API equivalent to CUDA driver API. Lower priority than ROCm but worth tracking.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Benchmark Suite
|
||||
|
||||
The current `benchmark_gpu.py` is good for what it tests but synthetic. A buyer will run their own benchmarks — make it easy and cover real scenarios.
|
||||
|
||||
### 4.1 Scenarios to cover
|
||||
- Short sequences: 128, 256 tokens (BERT-style)
|
||||
- Medium sequences: 2048, 4096 tokens (typical chat)
|
||||
- Long sequences: 16K, 32K tokens (document/RAG workloads)
|
||||
- Variable length batches (realistic distribution, not identical strings)
|
||||
- Concurrent throughput: simulate N async callers
|
||||
|
||||
### 4.2 Tokenizers to cover
|
||||
- BERT (`bert-base-uncased`) — already done
|
||||
- LLaMA 3 tokenizer (BPE, more representative of modern LLMs)
|
||||
- GPT-2 tokenizer (common baseline)
|
||||
|
||||
### 4.3 Hardware targets
|
||||
- Local RTX 3060 (done, 4.7x)
|
||||
- Cloud H100 SXM (~$2 on Lambda Labs, 30 min run) — **do this first, it changes the pitch**
|
||||
- AMD MI300X (after Phase 3.1)
|
||||
|
||||
### 4.4 What to measure
|
||||
- Tokens/second throughput
|
||||
- Latency per batch (p50, p95, p99)
|
||||
- GPU utilization % (the thing providers care about)
|
||||
- Memory bandwidth saturation
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Business Infrastructure
|
||||
|
||||
### 5.1 Wyoming LLC formation
|
||||
- Single-member LLC, Wyoming (strongest privacy, no public member disclosure)
|
||||
- LLC owns all IP
|
||||
- Registered agent handles public address
|
||||
- Done before any contract is signed
|
||||
|
||||
### 5.2 License structure
|
||||
- Open core: PostgreSQL→Arrow stays Apache 2.0
|
||||
- GPU features: commercial license
|
||||
- Pricing model: annual per-cluster license, priced against GPU utilization savings
|
||||
- License enforcement: honor-system enterprise contracts initially, not technical DRM
|
||||
|
||||
### 5.3 Sales collateral
|
||||
- 1-page technical brief (problem / mechanism / numbers / what's included)
|
||||
- Self-contained benchmark the prospect can run on their own hardware
|
||||
- Reference architecture diagram showing where this sits in a vLLM serving stack
|
||||
|
||||
---
|
||||
|
||||
## Branch Strategy
|
||||
|
||||
| Branch | Purpose |
|
||||
|--------|---------|
|
||||
| `main` | Open source PostgreSQL→Arrow, Apache 2.0, public |
|
||||
| `gpu-zero` | GPU feature development, private |
|
||||
| `rocm` | AMD port (Phase 3.1) |
|
||||
| `release/x.y` | Tagged releases for licensing |
|
||||
|
||||
The `gpu` feature is compiled in only with `--features gpu`. The open source build never includes GPU code.
|
||||
|
||||
---
|
||||
|
||||
## Immediate Next Steps (in order)
|
||||
|
||||
1. Fix 1.1 (dynamic buffer) — unblocks everything else
|
||||
2. Fix 1.2 (buffer isolation) — correctness before benchmarking
|
||||
3. Fix 1.3 (thread safety) — required for any real serving test
|
||||
4. Fix 1.4 (padding) — correctness
|
||||
5. Spend $2, run benchmark on cloud H100 — this one number changes the pitch
|
||||
6. Phase 2.1 vLLM integration — this is the demo that sells the concept
|
||||
7. Phase 3.1 ROCm — this is the differentiator
|
||||
206
docs/commercialization-strategy.md
Normal file
206
docs/commercialization-strategy.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# UncheckedIO — Commercialization Strategy
|
||||
**Session date:** 2026-04-20
|
||||
**Status:** Active planning
|
||||
|
||||
---
|
||||
|
||||
## Product Summary
|
||||
|
||||
Two components with different commercial treatment:
|
||||
|
||||
| Component | Language | License | Status |
|
||||
|---|---|---|---|
|
||||
| PostgreSQL → Arrow pipeline | Rust | Apache 2.0 (open source) | On PyPI, working |
|
||||
| GPU zero-copy ingestion pipeline | Rust + CUDA | Commercial | Proof-of-concept, needs Phase 1 fixes |
|
||||
|
||||
The GPU pipeline is the commercial product. The Postgres piece is the credibility signal and open-source loss leader.
|
||||
|
||||
---
|
||||
|
||||
## The Problem Being Solved
|
||||
|
||||
GPU starvation — LLM providers run 30-50% actual GPU utilization on inference. The bottleneck is the data ingestion pipeline: CPU tokenizes, creates a Python tensor in pageable memory, `.to("cuda")` performs two OS-mediated copies (pageable → kernel buffer → GPU).
|
||||
|
||||
UncheckedIO replaces this with: Rust parallel tokenization → pinned memory (DMA-accessible) → single direct DMA → DLPack zero-copy handover to PyTorch.
|
||||
|
||||
**Current benchmark:** 4.7x faster than standard HuggingFace pipeline on RTX 3060.
|
||||
**Pending:** H100 SXM benchmark (Wednesday, ~$2 on Lambda Labs). This number changes the pitch.
|
||||
|
||||
---
|
||||
|
||||
## Business Structure
|
||||
|
||||
**Entity:** New Mexico single-member LLC
|
||||
- NM does not require member names in Articles of Organization
|
||||
- No annual report requirement — less administrative overhead than most states
|
||||
- LLC owns all IP and signs all contracts
|
||||
- Registered agent handles public address
|
||||
- Form this before any contract is signed
|
||||
- Note: if the IP is eventually sold, the buyer's anonymity is their own concern — the LLC structure only needs to protect the original author
|
||||
|
||||
**Operating model:** Anonymous principal — LLC is the public face. The identity of the technical author is not disclosed to buyers or the public. Any investor or partner arrangement must include explicit confidentiality provisions.
|
||||
|
||||
---
|
||||
|
||||
## Go-to-Market Strategy
|
||||
|
||||
### Phase 1: Licensing (current goal)
|
||||
License the GPU pipeline commercially. Annual per-cluster licenses. Open-source Postgres piece stays Apache 2.0 as a credibility and discovery channel.
|
||||
|
||||
**Why licensing before IP sale:**
|
||||
A library generating $50K+/month in licenses is a business acquisition, not a proof-of-concept. The IP sale price increases significantly and the negotiating position improves. Licensing revenue demonstrates market validation.
|
||||
|
||||
### Phase 2: IP Sale (after traction)
|
||||
Once licensing revenue establishes market proof, approach GPU cloud providers and LLM infrastructure companies for outright IP purchase. Source code + short-term handoff contract. Clean exit.
|
||||
|
||||
---
|
||||
|
||||
## Pricing
|
||||
|
||||
The ROI math for a GPU provider: 100 H100s at $175K/year each = $17.5M in GPU costs. A 10% utilization improvement = $1.75M saved. A $50K license is a 35x ROI.
|
||||
|
||||
**Do not anchor at $50K. That is the floor for small deployments.**
|
||||
|
||||
| Deployment size | Annual license | One-time source sale |
|
||||
|---|---|---|
|
||||
| Small (50–200 GPUs) | $25K–$75K | $100K–$200K |
|
||||
| Medium (200–1,000 GPUs) | $75K–$200K | $300K–$600K |
|
||||
| Large (1,000+ GPUs) | $200K–$500K | $750K–$1.5M |
|
||||
|
||||
**First client:** Accept below market rate in exchange for a private reference and permission to say "deployed at [company]" to the next prospect. That reference is worth more than the price difference.
|
||||
|
||||
---
|
||||
|
||||
## Target Buyers
|
||||
|
||||
**Primary (write the check):**
|
||||
GPU cloud providers and LLM inference companies feel GPU starvation directly in their margins. They sell GPU utilization as a product. Any measurable improvement is direct revenue.
|
||||
|
||||
- CoreWeave, Lambda Labs — GPU cloud, sell utilization by the hour
|
||||
- Together AI, Fireworks AI, Modal — LLM inference providers
|
||||
- Anyscale — Ray-based ML infrastructure
|
||||
|
||||
**Secondary (may buy, slower process):**
|
||||
- Hugging Face — owns the `tokenizers` Rust library this code uses, runs TGI serving stack. More plausible buyer than PyTorch/Meta.
|
||||
- Databricks (MosaicML) — LLM training infrastructure
|
||||
|
||||
**Do not pitch:**
|
||||
- PyTorch/Meta — open source project, they'd implement it themselves
|
||||
- Hyperscalers (AWS, GCP, Azure) — too slow, too much process
|
||||
|
||||
**NVIDIA contact:**
|
||||
Wait until H100 benchmark is complete. Send the Nsight trace + one paragraph. They may be interested as an ecosystem/marketing play ("runs X% faster on H100"). Do not approach before having the measurement.
|
||||
|
||||
---
|
||||
|
||||
## B2B Sales Process
|
||||
|
||||
### First contact (Zoom call)
|
||||
The first call is discovery, not pitch. Goal: understand their specific infrastructure before presenting a solution.
|
||||
|
||||
**Ask first:**
|
||||
- What GPU hardware? (H100, A100, MI300X — how many?)
|
||||
- What serving stack? (vLLM, TGI, custom?)
|
||||
- Current GPU utilization rate, if known?
|
||||
- What caught their attention?
|
||||
|
||||
**Then:** "Based on what you're describing, here's specifically where this helps you..."
|
||||
|
||||
**On pricing:** Don't volunteer it first. If pressed: *"For a deployment your size I'd expect $50K–$150K annually. I want to understand your infrastructure better before giving you a specific number."*
|
||||
|
||||
**End of first call:** Set up follow-up with their infrastructure lead. Do not sign anything on the first call.
|
||||
|
||||
### NDA
|
||||
Get an NDA signed before showing the GPU source code on any call. The open-source Postgres piece is fine to show. The GPU pipeline (`src/llm.rs`, `gpu-zero` branch) is not.
|
||||
|
||||
### Solo developer objection
|
||||
Enterprise buyers will ask: *"What happens if you're unavailable?"*
|
||||
|
||||
**Answers:**
|
||||
1. Source code sale model answers it directly — they own the code
|
||||
2. Technical partner (friend's software business) provides continuity
|
||||
3. Source code escrow clause: code held by third party, released if unavailable for 90 days — this is standard and removes most of the objection
|
||||
|
||||
---
|
||||
|
||||
## Maintenance and Support
|
||||
|
||||
**Reality for 1–2 enterprise clients:**
|
||||
|
||||
| Phase | Time commitment |
|
||||
|---|---|
|
||||
| Active development (first 3–6 months) | 20–40 hrs/week |
|
||||
| Steady state | 5–10 hrs/week per client |
|
||||
| Major CUDA/PyTorch version update | 2–4 days, happens 2–3x/year |
|
||||
| Production incident | Drop everything, 1–3 days |
|
||||
|
||||
**Scope the support in the contract:** Cover current PyTorch LTS + one prior release. Anything newer is a paid upgrade. This prevents unbounded compatibility obligations.
|
||||
|
||||
One person can support 2–3 enterprise clients at steady state.
|
||||
|
||||
---
|
||||
|
||||
## Outsourcing
|
||||
|
||||
A friend with a software business can handle work that is not the core differentiator:
|
||||
- Support ticket triage and first-response
|
||||
- Documentation and integration guides
|
||||
- Testing infrastructure (benchmark runs against new PyTorch/CUDA versions)
|
||||
- Non-GPU features (Postgres pipeline, Python API polish)
|
||||
|
||||
**Do not outsource:** The GPU pipeline code, benchmarking methodology, CUDA/ROCm work — these are the differentiator.
|
||||
|
||||
**Rate:** Pass 60–70% of contract value for defined scope. On a $100K contract where the friend does 30% of the work: $18K–$21K to them. You take the larger share because you carry the business development risk and hold the client relationship.
|
||||
|
||||
**Critical:** Any work they do on the core library must be IP-assigned to the LLC in writing before the first engagement.
|
||||
|
||||
---
|
||||
|
||||
## Multi-Language Strategy (Rust + C++)
|
||||
|
||||
**Why both:**
|
||||
- Rust is the development language — memory safety makes GPU pipeline bugs easier to catch
|
||||
- C++ is the enterprise deliverable — the entire ML infrastructure ecosystem (cuDNN, TensorRT, vLLM C++ backend) is C++
|
||||
- Having both is a differentiator from vibe-coded projects; it signals deep systems understanding
|
||||
|
||||
**How:**
|
||||
- WhetstoneAI (separate project) can transpile language primitives automatically
|
||||
- Library dependencies require manual substitution (see below)
|
||||
- Rust is source of truth; C++ is a generated + manually doctored artifact per release
|
||||
|
||||
**Library substitutions for C++ version:**
|
||||
|
||||
| Rust dependency | C++ equivalent | Complexity |
|
||||
|---|---|---|
|
||||
| `cudarc` | Direct CUDA C++ calls (actually simpler) | Low |
|
||||
| `rayon` | OpenMP `#pragma omp parallel for` | Low |
|
||||
| `tokenizers` | SentencePiece (LLaMA) + tiktoken (GPT) | Medium — different API |
|
||||
| `pyo3` | pybind11 | Medium — parallel structure |
|
||||
| `tokio` | Not needed for GPU-only C++ version | N/A |
|
||||
|
||||
`parser.rs` (Postgres pipeline) is not worth transpiling — stays Rust only.
|
||||
|
||||
**Estimated work for C++ `llm.rs`:** 2–3 days after Whetstone handles primitives.
|
||||
|
||||
---
|
||||
|
||||
## NM State Funding Assessment
|
||||
|
||||
**Worth pursuing:**
|
||||
- **NM Small Business Assistance Program (NMSBA)** — pairs small businesses with Sandia/LANL. The value is free access to HPC hardware (H100/A100 clusters), not the money. Solves the enterprise benchmark problem. Apply for this.
|
||||
|
||||
**Not worth pursuing (for now):**
|
||||
- NMEDD money grants, angel tax credits — require registered business, matching funds, 3–6 month timelines. Time cost doesn't pencil out against a direct licensing deal.
|
||||
|
||||
**Key point:** Most state grant programs want an ongoing business, not a one-time IP sale. If the goal shifts toward a licensing business, revisit.
|
||||
|
||||
---
|
||||
|
||||
## Immediate Priorities
|
||||
|
||||
1. Fix Phase 1 correctness issues (see ROADMAP.md) — nothing else matters until these are done
|
||||
2. Wednesday: H100 benchmark on Lambda Labs (~$2), run under `nsys profile`
|
||||
3. Thursday/Friday: Incorporate H100 numbers into talk slides
|
||||
4. Talk: 10-minute presentation to IT/Cybersecurity group (see docs/talk-10min.md)
|
||||
5. Form Wyoming LLC before any serious sales conversation
|
||||
6. After first client: evaluate C++ version via Whetstone transpilation
|
||||
110
docs/talk-10min.md
Normal file
110
docs/talk-10min.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# UncheckedIO — 10-Minute Technical Talk
|
||||
**Audience:** IT/Cybersecurity group with programmers and ML engineers
|
||||
**Goal:** Plant the seed, demonstrate the mechanism, invite connections to LLM infrastructure teams
|
||||
|
||||
---
|
||||
|
||||
## Slide 1 — Open with the hidden thing (0:00–0:45)
|
||||
|
||||
Lead with an Nsight Systems timeline screenshot. Don't explain it yet.
|
||||
|
||||
> "This is what happens inside your GPU when a language model processes your request.
|
||||
> The orange is waiting. The GPU costs $30,000 and it's waiting for data."
|
||||
|
||||
Let the image do the work. This audience will immediately want to know what they're looking at.
|
||||
|
||||
---
|
||||
|
||||
## Slide 2 — The problem (0:45–2:00)
|
||||
|
||||
**GPU starvation.** Define it in dollar terms.
|
||||
|
||||
- Real LLM providers run at 30–50% actual GPU utilization on inference
|
||||
- An 8-GPU H100 cluster costs ~$20/hr to rent
|
||||
- At 40% utilization, $12/hr is wasted — the GPUs are idle waiting for data
|
||||
- LLM providers run hundreds of clusters
|
||||
- The waste is structural and nobody has fixed the root cause
|
||||
|
||||
---
|
||||
|
||||
## Slides 3–4 — The mechanism (2:00–4:00)
|
||||
|
||||
Two diagrams, side by side.
|
||||
|
||||
**Standard pipeline:**
|
||||
```
|
||||
CPU tokenizes → Python tensor (pageable memory) → .to("cuda") → kernel buffer → GPU
|
||||
^--- two OS-mediated copies
|
||||
```
|
||||
|
||||
**UncheckedIO pipeline:**
|
||||
```
|
||||
Rust tokenizes in parallel → pinned memory (DMA-accessible) → direct DMA → DLPack → GPU
|
||||
^--- one copy, no kernel involvement
|
||||
```
|
||||
|
||||
The second diagram has fewer boxes and fewer arrows. That is the entire pitch.
|
||||
|
||||
Key terms to explain briefly:
|
||||
- **Pinned memory** — memory registered with the OS so the GPU's DMA engine can read it directly
|
||||
- **DMA** — the GPU pulls the data itself, no CPU involvement after the transfer starts
|
||||
- **DLPack** — a standard zero-copy tensor exchange protocol; PyTorch accepts it natively with `torch.from_dlpack()`
|
||||
|
||||
---
|
||||
|
||||
## Slide 5 — The Nsight demo (4:00–7:00)
|
||||
|
||||
Show the recorded Nsight Systems timeline from the H100 benchmark run.
|
||||
|
||||
Both pipelines visible side by side:
|
||||
- Point out the idle gaps (orange) in the standard pipeline
|
||||
- Point out the absence of idle gaps in the UncheckedIO pipeline
|
||||
- Show the DMA transfer lane — this is the "hidden thing" made visible
|
||||
|
||||
This is the centerpiece of the talk. Expect questions here. That's good.
|
||||
|
||||
---
|
||||
|
||||
## Slide 6 — The number (7:00–8:00)
|
||||
|
||||
One slide. No prose.
|
||||
|
||||
| Hardware | Speedup |
|
||||
|----------|---------|
|
||||
| RTX 3060 (local) | 4.7x |
|
||||
| H100 SXM (cloud) | [Wednesday result] |
|
||||
|
||||
---
|
||||
|
||||
## Slide 7 — Market fit (8:00–9:00)
|
||||
|
||||
- This is infrastructure every LLM provider needs
|
||||
- Built as an open-core library: PostgreSQL→Arrow is Apache 2.0 and already on PyPI
|
||||
- GPU ingestion pipeline is the commercial layer
|
||||
- AMD ROCm support (MI300X) is next — providers are actively evaluating MI300X to escape H100 pricing
|
||||
|
||||
Don't pitch deal specifics. This slide plants the seed.
|
||||
|
||||
---
|
||||
|
||||
## Slide 8 — The close (9:00–10:00)
|
||||
|
||||
> "I'm looking for connections to ML infrastructure teams and LLM providers.
|
||||
> If you know someone building at that layer, I'd appreciate an introduction."
|
||||
|
||||
Direct. Not desperate. Leave room for questions.
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
**What NOT to do:**
|
||||
- Don't go deep on the Rust implementation — the visual is the hook, not the code
|
||||
- Don't mention the PostgreSQL→Arrow piece — it's a distraction in this context
|
||||
- Don't volunteer that it's unfinished — if asked, "it's in active development and the core mechanism is proven"
|
||||
|
||||
**For the well-connected engineer (separate talk, two weeks out):**
|
||||
Don't reuse this version. Prep a deeper, tailored pitch after seeing what questions come up here.
|
||||
|
||||
**For the NVIDIA contact:**
|
||||
Wait until after the H100 benchmark. Send the Nsight trace + one paragraph. Leading with a measurement is different from leading with a claim.
|
||||
Reference in New Issue
Block a user