Python: fastest way to read and process a large CSV file (10M+ rows)

Top Questions

CSS: how to center a div — the definitive 2024 guide?

CSS Grid vs Flexbox — when to use which?

JWT authentication — where should I store the token?

Git: how to cleanly undo a pushed commit without rewriting history?

TypeScript: difference between `type` and `interface` — practical guide

Popular Tags

Python: fastest way to read and process a large CSV file (10M+ rows) | Dev Overflow

Carlos Ribeiro

116

Python: fastest way to read and process a large CSV file (10M+ rows)

asked 2 months ago

11.2K

I need to process a CSV file with ~12 million rows. Each row has ~20 columns. Currently using pandas:

df = pd.read_csv('huge_file.csv')
result = df.groupby('category').sum()

This loads everything into memory and crashes on my 8GB machine. What are the best alternatives? I've heard about chunking and Polars but not sure which to start with.

python

performance

1 Answer

Priya Nair

• answered 2 months ago

Great question! I've run into this before. The key thing to keep in mind is to follow the official documentation closely and test your implementation with edge cases. Happy to elaborate if you need more specifics.