Why do we split data into train, validation and test sets, and what is k-fold cross-validation?

Think of preparing for a university exam.

• Training set: the textbook exercises you practise on.
• Validation set: the mock tests you use to decide how to study. Should you spend more time on calculus? Change your method? You adjust based on mock scores.
• Test set: the final exam. You see it once, and the score tells you how well you really learnt.

If you had seen the final paper while studying, your score would look great but mean nothing. Models are the same. A score is only honest if it comes from data the model, and you, did not use to make decisions.

You have 10,000 loan applications. A common split is 70/15/15:

With 5-fold cross-validation, you take the training data (say 8,500 rows after keeping 1,500 aside for test) and cut it into 5 folds of 1,700 rows. You train 5 times. Each time, 4 folds (6,800 rows) are used for training and 1 fold for validation. You average the 5 scores. If the scores are 0.81, 0.79, 0.83, 0.80 and 0.82, the mean is 0.81, and the small spread tells you the result is stable.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.15, stratify=y, random_state=42
)

# The scaler is inside the pipeline, so it is fitted only on each training fold
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=cv, scoring="f1")
print(scores.mean(), scores.std())

• Parameters (like regression weights) are learnt from the training set.
• Hyperparameters (like learning rate or number of trees) are chosen by comparing validation scores.
• Every time you look at a score and change something, that data influences the model a little. That is why the test set must be used once, at the end.

Cross-validation uses data more efficiently: every row is used for validation exactly once and for training k−1 times. It costs k times the training time, so k = 5 or 10 is typical. Stratified k-fold keeps the class ratio the same in every fold, which matters for imbalanced data like fraud detection.

• Time series (sales, crop prices): never shuffle. Train on the past and validate on the future, using a forward-moving window.
• Grouped data (several X-rays per patient): keep all rows from one patient in the same fold using group k-fold, or the model will recognise the patient instead of the disease.

Data leakage is the big one: information from validation or test data sneaks into training. Typical causes:

• Scaling or filling missing values on the full dataset before splitting. The mean then includes test rows. Fit preprocessing on training data only, ideally inside a pipeline.
• Features that are only known after the outcome, such as "number of recovery calls made" when predicting loan default.
• Duplicate rows that land in both train and test.
• Random splits on time-based data, so the model peeks at the future.

Leakage gives excellent offline scores and a model that fails in production.

• What is data leakage and how have you prevented it?
• When would you use leave-one-out cross-validation?
• How do you cross-validate a time series model?
• What is nested cross-validation?