performance Questions
2 months ago
Python: fastest way to read and process a large CSV file (10M+ rows)
I need to process a CSV file with ~12 million rows. Each row has ~20 columns. Currently using pandas:
df = pd.read_csv('huge_file.csv')
result = df.groupby('category').sum()This loads everything into memory and crashes on my 8GB machine. What are the best alternatives? I've heard about chunking and Polars but not sure which to start with.
C
Carlos Ribeiro
asked 2 months ago
1
Answers
116
Upvotes
11200
Views2 months ago
SQL: optimizing a slow JOIN query with millions of records
I have a query that's taking 30+ seconds on a PostgreSQL database:
SELECT u.name, COUNT(o.id) as order_count, SUM(o.total) as revenue
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE o.created_at > '2024-01-01'
GROUP BY u.id, u.name
ORDER BY revenue DESC;Tables: 500K users, 8M orders. I've added indexes on user_id and created_at but it's still slow. What am I missing?
2 months ago
Python: asyncio vs threading vs multiprocessing — which to use?
I'm building a web scraper that needs to handle 1000+ URLs concurrently. I've read about three approaches:
- asyncio: single-threaded event loop
- threading: for I/O-bound tasks
- multiprocessing: for CPU-bound tasks
My task is HTTP requests (I/O-bound). Is asyncio always the right choice? What's the overhead comparison?