83 lines
4.1 KiB
Markdown
83 lines
4.1 KiB
Markdown
|
Here is the technical summary of our debugging session, formatted as a Markdown file for your documentation.
|
||
|
|
|
||
|
|
Markdown
|
||
|
|
|
||
|
|
# Technical Post-Mortem: The "Thrashing" & Performance Optimization Log
|
||
|
|
|
||
|
|
**Date:** 2025-11-21
|
||
|
|
**Project:** UncheckedIO
|
||
|
|
**Topic:** Debugging "Astronomical" Task Switching Delays & Laptop vs. Cloud Discrepancies
|
||
|
|
|
||
|
|
## 1. The Symptoms
|
||
|
|
* **The "Astronomical" Delay:** Increasing partition count (e.g., from 16 to 32) caused disproportionate slowdowns. Switching between tasks took seconds, not nanoseconds.
|
||
|
|
* **The "Tracy" Paradox:** The profiler showed "CPU Data Starved" and frequently crashed the runtime. When it crashed/detached, the library actually ran faster.
|
||
|
|
* **Platform Discrepancy:** The library performed significantly better on Google Colab (Server CPU) than on a high-spec Laptop (Consumer CPU) relative to competitors, despite the laptop having faster single-core speed.
|
||
|
|
|
||
|
|
## 2. Root Cause Analysis
|
||
|
|
|
||
|
|
### A. The Database Villain: "The Missing Index"
|
||
|
|
* **The Issue:** The `setup_db.sql` script created the table but **failed to index the `id` column**.
|
||
|
|
* **The Consequence:** The "Parallel Coordinator" splits the job into partitions (e.g., `WHERE id BETWEEN 0 AND 125000`). Without an index, Postgres performed a **Sequential Scan** (Full Table Scan) for *every single partition*.
|
||
|
|
* **The Impact:** Running 16 parallel workers triggered **16 simultaneous Full Table Scans**. This saturated memory bandwidth instantly, explaining the "CPU Data Starved" metric in Tracy.
|
||
|
|
|
||
|
|
### B. The Concurrency Villain: "Pool Contention & Thrashing"
|
||
|
|
* **The Issue:** The connection pool was hardcoded to `max_size(20)` in `src/parser.rs`.
|
||
|
|
* **The "Alive" Misconception:** We assumed keeping connections "alive" allowed infinite parallelism. However, a connection is a single-lane bridge; it can only transport one query at a time.
|
||
|
|
* **The Bottleneck:** When spawning 800 tasks (partitions) against 20 connections:
|
||
|
|
* **20 Tasks** ran immediately.
|
||
|
|
* **780 Tasks** sat frozen in a RAM queue.
|
||
|
|
* **The "Thrashing":** The CPU spent ~90% of its time managing this queue—saving/restoring task states (Context Switching) and flushing L1/L2 caches—rather than parsing data. This overhead destroyed performance on the laptop (which has smaller caches than the Colab server).
|
||
|
|
|
||
|
|
### C. The Red Herring: `io_uring`
|
||
|
|
* **Hypothesis:** We initially thought switching to `io_uring` (Linux asynchronous I/O) would solve the "waiting."
|
||
|
|
* **Verdict:** **Incorrect.** The bottleneck was not syscall overhead (which `io_uring` solves); it was **Logical Contention** (waiting for a DB connection) and **Physical Latency** (waiting for RAM/Network).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 3. The Solutions
|
||
|
|
|
||
|
|
### Fix #1: Create the Index (Database Side)
|
||
|
|
Change the operation from a Sequential Scan (reading 20M rows x 16 times) to an **Index Scan** (jumping directly to the data).
|
||
|
|
|
||
|
|
**File:** `setup_db.sql`
|
||
|
|
```sql
|
||
|
|
-- OLD
|
||
|
|
ANALYZE benchmark_table;
|
||
|
|
|
||
|
|
-- NEW
|
||
|
|
CREATE INDEX idx_benchmark_id ON benchmark_table(id);
|
||
|
|
ANALYZE benchmark_table;
|
||
|
|
Fix #2: Dynamic Pool Sizing (Rust Side)
|
||
|
|
Remove the artificial speed limit. Match the pool size to the hardware capabilities so tasks never wait in a queue.
|
||
|
|
|
||
|
|
File: src/parser.rs
|
||
|
|
|
||
|
|
Rust
|
||
|
|
|
||
|
|
// OLD
|
||
|
|
.max_size(20)
|
||
|
|
|
||
|
|
// NEW
|
||
|
|
use num_cpus; // Add to Cargo.toml
|
||
|
|
let num_threads = num_cpus::get();
|
||
|
|
.max_size(num_threads) // 1:1 mapping of Threads to Connections
|
||
|
|
Fix #3: Auto-Tuned "Blast Radius"
|
||
|
|
Instead of a hardcoded partition size or user guess, calculate the optimal partition count to balance
|
||
|
|
load without causing overhead.
|
||
|
|
|
||
|
|
Strategy: 4 * CPU Cores partitions.
|
||
|
|
|
||
|
|
Why: Enough chunks to allow "Work Stealing" (fast threads take more work), but few enough to prevent
|
||
|
|
connection setup overhead.
|
||
|
|
|
||
|
|
Rust
|
||
|
|
|
||
|
|
pub fn calculate_optimal_blast_radius(min_id: i64, max_id: i64) -> i64 {
|
||
|
|
let total_rows = max_id - min_id;
|
||
|
|
let target_partitions = num_cpus::get() * 4;
|
||
|
|
total_rows / (target_partitions as i64)
|
||
|
|
}
|
||
|
|
4. Conclusion
|
||
|
|
The library's "Schema-on-Read" optimization (trusting the config) was working perfectly.
|
||
|
|
The performance loss was due to infrastructure bottlenecks (DB Index and Connection Pool limits)
|
||
|
|
that forced the CPU to manage queues instead of parsing data.
|