From 11ddd184f43448479c44c681fc51ebb383d85bcc Mon Sep 17 00:00:00 2001 From: Bill Date: Mon, 24 Nov 2025 20:17:03 -0700 Subject: [PATCH] version update to publish to Pypi 7 --- docs/refactor-prompt | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/refactor-prompt diff --git a/docs/refactor-prompt b/docs/refactor-prompt new file mode 100644 index 0000000..c61078f --- /dev/null +++ b/docs/refactor-prompt @@ -0,0 +1 @@ +I need to refactor src/parser.rs to implement a "Worker Pool" pattern that strictly limits active database queries to the number of CPU cores.The Goal:Currently, the code spawns a Tokio task for every partition immediately. This floods the scheduler and causes database thrashing (too many concurrent COPY commands).I want to spawn a fixed number of workers (e.g., 16) that consume tasks from a queue. This ensures that if I have 1,600 partitions, only 16 COPY commands are active on the database at any given time.Requirements:Dependencies: Use async-channel (assume it's in Cargo.toml).Concurrency Limit:let num_workers = num_cpus::get();Set deadpool_postgres max_size to num_workers.The Queue:Create a channel: let (tx, rx) = async_channel::bounded(num_workers * 2); (Backpressure is good).Spawn a separate "Distributor" task that iterates partitions and sends them into tx.The Workers:Spawn exactly num_workers tasks.Each worker loops: while let Ok(task) = rx.recv().await.Inside the loop:Get a connection from the pool.Run copy_out(task.query).Parse the result.Store the resulting RecordBatch.Output:Collect all RecordBatch results from all workers.Flatten and concatenate them using concat_batches.Why: This ensures Postgres never sees a queue of queries. It only sees active workers. The "queue" lives entirely in Rust memory. \ No newline at end of file