Merge pull request #9 from BillTheMaker/feature

add doc file
2025-11-24 20:18:09 -07:00
parent 0b0f3677b1 11ddd184f4
commit 9cacb95b0f
1 changed files with 1 additions and 0 deletions
--- a/docs/refactor-prompt
+++ b/docs/refactor-prompt
@@ -0,0 +1 @@
 I need to refactor src/parser.rs to implement a "Worker Pool" pattern that strictly limits active database queries to the number of CPU cores.The Goal:Currently, the code spawns a Tokio task for every partition immediately. This floods the scheduler and causes database thrashing (too many concurrent COPY commands).I want to spawn a fixed number of workers (e.g., 16) that consume tasks from a queue. This ensures that if I have 1,600 partitions, only 16 COPY commands are active on the database at any given time.Requirements:Dependencies: Use async-channel (assume it's in Cargo.toml).Concurrency Limit:let num_workers = num_cpus::get();Set deadpool_postgres max_size to num_workers.The Queue:Create a channel: let (tx, rx) = async_channel::bounded(num_workers * 2); (Backpressure is good).Spawn a separate "Distributor" task that iterates partitions and sends them into tx.The Workers:Spawn exactly num_workers tasks.Each worker loops: while let Ok(task) = rx.recv().await.Inside the loop:Get a connection from the pool.Run copy_out(task.query).Parse the result.Store the resulting RecordBatch.Output:Collect all RecordBatch results from all workers.Flatten and concatenate them using concat_batches.Why: This ensures Postgres never sees a queue of queries. It only sees active workers. The "queue" lives entirely in Rust memory.
		`@@ -0,0 +1 @@`
							I need to refactor src/parser.rs to implement a "Worker Pool" pattern that strictly limits active database queries to the number of CPU cores.The Goal:Currently, the code spawns a Tokio task for every partition immediately. This floods the scheduler and causes database thrashing (too many concurrent COPY commands).I want to spawn a fixed number of workers (e.g., 16) that consume tasks from a queue. This ensures that if I have 1,600 partitions, only 16 COPY commands are active on the database at any given time.Requirements:Dependencies: Use async-channel (assume it's in Cargo.toml).Concurrency Limit:let num_workers = num_cpus::get();Set deadpool_postgres max_size to num_workers.The Queue:Create a channel: let (tx, rx) = async_channel::bounded(num_workers * 2); (Backpressure is good).Spawn a separate "Distributor" task that iterates partitions and sends them into tx.The Workers:Spawn exactly num_workers tasks.Each worker loops: while let Ok(task) = rx.recv().await.Inside the loop:Get a connection from the pool.Run copy_out(task.query).Parse the result.Store the resulting RecordBatch.Output:Collect all RecordBatch results from all workers.Flatten and concatenate them using concat_batches.Why: This ensures Postgres never sees a queue of queries. It only sees active workers. The "queue" lives entirely in Rust memory.