python Questions
2 months ago
Python: fastest way to read and process a large CSV file (10M+ rows)
I need to process a CSV file with ~12 million rows. Each row has ~20 columns. Currently using pandas:
df = pd.read_csv('huge_file.csv')
result = df.groupby('category').sum()This loads everything into memory and crashes on my 8GB machine. What are the best alternatives? I've heard about chunking and Polars but not sure which to start with.
C
Carlos Ribeiro
asked 2 months ago
1
Answers
116
Upvotes
11200
Views2 months ago
Python: asyncio vs threading vs multiprocessing — which to use?
I'm building a web scraper that needs to handle 1000+ URLs concurrently. I've read about three approaches:
- asyncio: single-threaded event loop
- threading: for I/O-bound tasks
- multiprocessing: for CPU-bound tasks
My task is HTTP requests (I/O-bound). Is asyncio always the right choice? What's the overhead comparison?