Why is apply slower than vectorised operations in pandas?
apply runs a Python function once per row or element, paying Python overhead each time. Vectorised operations such as column arithmetic or str methods run in compiled code on whole arrays, and are often tens of times faster.
Explain it simply
A vectorised operation is a printing press: set it once and the whole page is done. apply is a clerk copying each line by hand. Both give the same answer, but one is far faster on large data.
Example
import pandas as pd
x = pd.Series([1, 2, 3, 4])
print((x * 2).tolist()) # [2, 4, 6, 8]
print(x.apply(lambda v: v * 2).tolist()) # [2, 4, 6, 8]Same output, very different speed once you have a million rows. Better tools than apply for common jobs:
| Task | Vectorised choice |
|---|---|
| Arithmetic on columns | df["a"] + df["b"] |
| If-else on a column | np.where(df["x"] > 0, "pos", "neg") |
| Several conditions | np.select(conditions, choices) |
| Text cleanup | df["name"].str.strip().str.lower() |
| Dates | df["d"].dt.month |
Use apply when the logic is genuinely complex or needs several columns in a way that has no vectorised form, and accept the cost.
What to say in the interview
"apply loops in Python, so it scales poorly. I prefer built-in column operations, np.where, np.select, the str and dt accessors, or map for dictionary lookups. I reach for apply only as a last resort and avoid axis=1 apply on large frames."
Common mistakes
- Writing a row-wise apply for something a single column expression can do.