Your Pandas code works fine on 100K rows. Then the dataset hits 10M, the kernel dies, and you're rewriting everything in SQL. There's a better path: keep the Python workflow, swap the engine.
Why Polars Expressions Change the Game
Pandas evaluates eagerly. Every filter, groupby, and join materializes a new DataFrame in memory. Polars expressions are lazy by default. You describe the computation as a query plan; the engine optimizes and executes it in a single pass over columnar Arrow arrays. No intermediate copies. No Python loop overhead.
The Expression API: Composable, Not Chained
Polars expressions return expressions. You can combine them algebraically before execution. This enables predicate pushdown, projection pushdown, and automatic parallelism across all cores.
LazyFrames: Out-of-Core by Default
scan_csv and scan_parquet return LazyFrames. They don't read data until collect(). This means you can query 500GB on a 16GB laptop. The engine streams chunks, applies filters early, and only materializes the final result.
💡
TipUse sink_parquet() or sink_csv() to write results directly to disk without collecting to memory.
Migration Cheat Sheet
| Pandas | Polars Expression | Note |
|---|---|---|
| df[col] | pl.col('col') | Column reference |
| df[df['a'] > 5] | .filter(pl.col('a') > 5) | Predicate pushdown |
| df.groupby('g').agg({'v': 'mean'}) | .group_by('g').agg(pl.col('v').mean()) | Multi-agg in one pass |
| df.merge(other, on='key') | .join(other, on='key', how='inner') | Hash join, parallel |
| df.apply(func) | pl.col('c').map_elements(func) | Avoid; use native expressions |
| df.iterrows() | Never | Use expressions or .to_numpy() |
Real Benchmark: 10M Rows, GroupBy + Filter
Dataset: 10M rows, 8 columns (int, float, string, datetime). Machine: M2 MacBook Pro, 16GB RAM.
| Operation | Pandas (s) | Polars Lazy (s) | Speedup |
|---|---|---|---|
| Read CSV | 4.2 | 0.8 | 5.3x |
| Filter + GroupBy + Agg | 3.7 | 0.12 | 31x |
| Sort + Limit 10 | 1.1 | 0.03 | 37x |
| Total (end-to-end) | 9.0 | 0.95 | 9.5x |
"The query optimizer does in milliseconds what takes me hours to hand-tune in Pandas.
— Staff Data Engineer, Fintech Unicorn
Common Pitfalls to Avoid
Don't call collect() inside a loop. Don't convert to Pandas mid-pipeline unless you're plotting. Don't use map_elements for row-wise logic — rewrite as vectorized expressions. The engine can't optimize what it can't see.
⚠️
WarningPolars raises on nulls in comparisons by default. Use .fill_null() or .is_not_null() explicitly.
Your Next Step
Pick your slowest Pandas notebook. Wrap the read in pl.scan_parquet (convert once). Rewrite the chain as a single LazyFrame expression. Call collect() at the end. Measure. You'll likely cut runtime 10x and memory 5x without leaving Python.