diff --git a/docs/project-start-log b/docs/project-start-log new file mode 100644 index 0000000..dc69787 --- /dev/null +++ b/docs/project-start-log @@ -0,0 +1,127 @@ +# Project: UncheckedIO (f.k.a. BlitzPipe, YOLO-Pipe) + +Date: 2025-11-17 + +Status: MVP functional, performance-tuning phase. + +## 1. Project Philosophy & Goal + +The primary goal of `UncheckedIO` is to create the fastest possible data connector for bulk-loading data from a PostgreSQL database directly into a Python/Arrow environment. + +The core philosophy is the **"Configured Opinion"**: + +1. **The Problem:** Generic connectors (like `pandas.read_sql` or even `ConnectorX`) must be flexible. They spend significant CPU time at runtime discovering the database schema, validating data types, and converting them (a "schema-on-read" approach). + +2. **Our Solution:** We trade this flexibility for raw speed. Our library is **opinionated**. We _require_ the user to provide the exact query and schema in a `config.yaml` file. + +3. **The "Unchecked" Joke:** The name `UncheckedIO` (a reference to Java's `UncheckedIOException`) is a tongue-in-cheek warning. We are "unchecked" because we **trust the user's config completely** and skip the expensive runtime validation, allowing us to build a hyper-optimized binary parser. The Rust compiler provides the _memory_ safety; the user provides the _schema_ safety. + + +## 2. Core Architectural Decisions + +### Stack + +- **Core Language:** Rust. Chosen for its C-level performance, memory safety (Borrow Checker), and powerful async capabilities. + +- **Interface:** Python. We are building a native Python module so data scientists can get this speed without learning Rust. + +- **Python Bridge:** `PyO3` (for bindings) and `Maturin` (for building/packaging). + +- **Async Runtime:** `Tokio`. The industry standard for high-performance, asynchronous I/O in Rust. + +- **Data Format:** `Apache Arrow`. This is the standard for zero-copy, columnar data. We parse directly from the database stream into Arrow `RecordBatch` objects. + +- **Error Handling:** `anyhow`. Chosen for its simplicity in application-level error handling, especially for wrapping and propagating errors across thread boundaries (`Send + 'static`). + + +### Project Structure (Idiomatic Rust) + +The project was refactored from a single `lib.rs` file into a modular structure: + +- `src/lib.rs`: The "front door." Contains only the `#[pyfunction]` and `#[pymodule]` definitions. This is the FFI (Foreign Function Interface) layer. + +- `src/config.rs`: Handles all logic for parsing and validating the user's `config.yaml` file. Defines the `ConnectorConfig` struct. + +- `src/parser.rs`: The "engine." Contains all core logic: connecting to Postgres, running the query, and parsing the binary stream. + + +### Data Flow (The "Fast Path") + +1. **User Config:** The user provides a `config.yaml` with their `connection_string`, an ordered `schema` (as a `Vec`), and the exact `query`. + +2. **Query Type:** The `query` _must_ be a PostgreSQL `COPY ... TO STDOUT (FORMAT binary)` command. This bypasses the slow row-by-row SQL protocol and uses Postgres's high-speed bulk export stream. + +3. **Stream Handling:** We use `tokio-postgres`'s `client.copy_out()` function to get the raw `CopyOutStream`. + +4. **Parser:** The `handle_binary_copy` function iterates this stream, parses the binary format (header, row-by-row field data, and trailer), and appends values directly into the correct Arrow `ArrayBuilder` based on the user's config. + + +## 3. Key Challenges & Debugging History (Critical Context) + +This section is vital, as it documents the non-obvious fixes required to make the project compile. + +### FFI (Python <-> Rust) Hell + +We spent ~15 iterations debugging the FFI (Foreign Function Interface) to return the `RecordBatch` to Python. + +- **Problem:** Mismatched versions of `pyo3` (v0.27.1) and `pyo3-arrow` (v0.15.0) led to confusing compiler errors. + +- **Failed Attempts:** `ArrowIntoPy` was unresolved. `PyArrowConvert` was unresolved. `PyRecordBatch::new` had the wrong signature. `m.add_function` had constant E0277 (trait bound) and E0061 (argument count) errors. + +- **Final Working Syntax (The "Magic Incantation"):** + + - **`src/lib.rs` (Return):** The function _must_ return a `Bound` reference: + + ``` + fn load_data_from_config<'py>(py: Python<'py>, ...) -> PyResult> + ``` + + - **`src/lib.rs` (Conversion):** The `RecordBatch` must be wrapped in `PyRecordBatch` and converted using the `into_pyarrow(py)` method: + + ``` + let py_record_batch = PyRecordBatch::new(record_batch); + py_record_batch.into_pyarrow(py) + ``` + + - **`src/lib.rs` (Module):** The `wrap_pyfunction!` macro _must_ take the module `m` as a second argument to resolve the trait bounds: + + ``` + m.add_function(wrap_pyfunction!(load_data_from_config, m)?)?; + ``` + + +## 4. Current Status & Next Steps + +- **Status:** **MVP Complete.** The library successfully: + + 1. Installs in Colab from GitHub. + + 2. Installs a local Postgres server. + + 3. Populates a 1,000,000 row, 10-column table. + + 4. Runs a benchmark via `timeit`. + +- **Benchmark Results (1M Rows):** + + - `Pandas`: ~7800 ms + + - `ConnectorX`: ~3400 ms + + - `UncheckedIO`: ~3450 ms + +- **Diagnosis:** We are **tying** the #1 competitor, not beating them. This is because our parser is **inefficient**. + +- **The Bottleneck:** `handle_binary_copy` aggregates the _entire_ 130MB+ byte stream into a single `Vec` in memory _before_ it starts parsing. This is a **batch processor**. ConnectorX is almost certainly a **streaming processor**. + +- Next Step (The Real Optimization): + + Refactor handle_binary_copy to be a true streaming parser. We must move the parsing logic (the Cursor) inside the while let Some(segment) = stream.next().await loop. This will involve: + + 1. Parsing the small `segment` (e.g., 64kB) chunk. + + 2. Handling partial rows (when a row's data is split between two chunks). + + 3. Appending to the Arrow builders in real-time. + + This will dramatically reduce memory overhead and should make us significantly faster than ConnectorX. \ No newline at end of file diff --git a/setup_db.sql b/setup_db.sql index 729f09c..8dff7e5 100644 --- a/setup_db.sql +++ b/setup_db.sql @@ -1,7 +1,5 @@ --- 1. Drop the old table if it exists DROP TABLE IF EXISTS benchmark_table; --- 2. Create the new 10-column table CREATE TABLE benchmark_table ( id BIGINT, uuid TEXT, @@ -15,7 +13,6 @@ CREATE TABLE benchmark_table ( rating FLOAT8 ); --- 3. Use generate_series to insert 1,000,000 rows of data INSERT INTO benchmark_table ( id, uuid, @@ -43,54 +40,4 @@ FROM generate_series(1, 1000000) s(i); -- 4. Analyze the table for better query planning (good practice) ANALYZE benchmark_table; -``` -### Action 2: Commit and Push the Change - -You must commit and push this new `setup_db.sql` to your GitHub repo so Colab can pull it. - -```bash -# In your local terminal -git add setup_db.sql -git commit -m "Create 1M row benchmark table" -git push -``` - -### Action 3: Update Your Colab Notebook - -Now, in Colab, **replace your old Cell 2 and Cell 3** with these two new cells. - -**New Cell 2 (Install & Setup):** This cell now does *all* setup in one go to fix the connection error. - -```python -# 2. Install Rust, Tools, and Clone Repo -!curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y -import os -os.environ['PATH'] += ":/root/.cargo/bin" -!pip install maturin wheel - -# Go to root, clean up, and clone fresh -%cd /content/ -!rm -rf unchecked-io -!git clone https://github.com/BillTheMaker/unchecked-io.git -%cd unchecked-io - -# Build and install UncheckedIO -!pip install . -``` - -**New Cell 3 (Start & Populate DB):** This cell installs Postgres, robustly starts it, and runs your *new* 1M row script. - -```python -# 3. Install, Run, and Populate PostgreSQL Server -print("Installing PostgreSQL...") -!apt-get -y -qq install postgresql postgresql-client > /dev/null - -# Force the service to start and set password -!service postgresql start -!sudo -u postgres psql -c "ALTER USER postgres WITH PASSWORD 'mysecretpassword';" - -print("Running 1M row setup_db.sql script (this may take a moment)...") -# Run our *new* 1M row SQL script -!psql "postgresql://postgres:mysecretpassword@localhost:5432/postgres" -f setup_db.sql -print("Database setup complete.") \ No newline at end of file