How do you handle missing values in a dataset?
First find how much is missing and why. Then either drop rows or columns (when little is missing or a column is mostly empty) or fill the gaps, using the median for skewed numbers, the mode for categories, forward fill for time series, or a model-based method. Often add a flag column that records which values were missing.
Explain it simply
Imagine a class register where some students' marks were never entered. You can ignore those students, guess their marks from the rest of the class, or note down "absent" as its own piece of information. Which choice is right depends on how many are missing and why they are missing. If only the weakest students skipped the test, simply ignoring them will make the class look better than it is.
Example
import pandas as pd
df = pd.DataFrame({
"city": ["Pune", None, "Delhi", "Pune"],
"salary": [6.5, 4.0, None, 5.0], # lakh per year
"age": [23, None, 25, 24],
})
print(df.isna().sum()) # missing count per column: 1 each here
print(df.isna().mean()) # fraction missing per column
# 1. Drop rows that have any missing value (loses 2 of 4 rows here, too costly)
dropped = df.dropna()
# 2. Fill numbers with the median and categories with the mode
df["salary_missing"] = df["salary"].isna() # keep a record first
df["salary"] = df["salary"].fillna(df["salary"].median()) # median is 5.0
df["city"] = df["city"].fillna(df["city"].mode()[0]) # "Pune"
# 3. For time series, carry the last known value forward
# prices = prices.ffill()How it works
Step 1: measure. Use df.isna().sum() to see counts per column. A column that is 80% empty is a different problem from one that is 2% empty.
Step 2: ask why it is missing. Statisticians use three labels:
- MCAR (missing completely at random): the gap has nothing to do with any value, like a form lost in the post. Dropping rows is fairly safe.
- MAR (missing at random): the gap depends on other columns you can see, like older users skipping the "app rating" question. Filling using those other columns works well.
- MNAR (missing not at random): the gap depends on the missing value itself, like high earners refusing to disclose salary. Any simple fill will be biased; the missingness itself carries information.
Step 3: choose a strategy.
| Situation | Common choice |
|---|---|
| Very few rows missing, large dataset | Drop those rows |
| Column mostly empty and not important | Drop the column |
| Numeric, roughly symmetric | Fill with mean |
| Numeric, skewed or with outliers | Fill with median |
| Categorical | Fill with mode, or a new category like "Unknown" |
| Time series | Forward fill or interpolate |
| Missingness may be meaningful | Fill, and add a "was_missing" flag column |
| Relationships between columns matter | KNN or model-based imputation |
Real-world use
- Loan application data where income is blank for self-employed applicants.
- Sensor readings from a factory where a device went offline for an hour.
- Survey data where people skip uncomfortable questions.
- E-commerce data where "delivery rating" is empty for orders that were cancelled.
Common mistakes
- Calling
dropna()on the whole dataset without checking how many rows disappear. - Computing the fill value (mean, median) on the full dataset before the train-test split. This leaks information from the test set. Compute it on the training data only and reuse it for test data.
- Filling with 0 when 0 is a real, meaningful value, such as salary or temperature.
- Missing placeholders that are not
NaN, like"NA","-",-1or999. Convert them first. - Filling MNAR data with the mean and losing the signal that "this person did not answer".
What to say in the interview
"I start by measuring how much is missing per column and thinking about why it is missing. If very little is missing and it looks random, I drop those rows. If a column is mostly empty, I consider dropping it. Otherwise I impute: median for skewed numeric columns, mode or an 'Unknown' category for categorical ones, forward fill for time series, and KNN or model-based imputation when columns are related. I fit the imputer on training data only to avoid leakage, and I often add a flag column so the model knows the value was missing."
Follow-up questions
- What is the difference between MCAR, MAR and MNAR?
- Why should imputation be fitted only on the training set?
- When would you prefer median over mean imputation?
- Which models can handle missing values without imputation?
Go deeper
Read these once the basics above feel easy.
In scikit-learn, put imputation inside a Pipeline with SimpleImputer(strategy="median") so it is fitted only on the training folds during cross-validation. SimpleImputer(add_indicator=True) adds the missing-value flag columns for you.
KNNImputer fills a value using the average of the k most similar rows. IterativeImputer models each column with missing values as a function of the others, and repeats. Both preserve relationships between columns better than a single median, but they are slower and can still be biased under MNAR. Some gradient boosting libraries, such as XGBoost and LightGBM, handle missing values natively by learning which branch to send them down.