A fraud model shows 99% accuracy. Why might it still be useless?
If only 1% of transactions are fraud, a model that always says 'not fraud' scores 99% accuracy while catching zero fraud. On imbalanced data, look at recall, precision, F1 or PR-AUC instead.
Explain it simply
Imagine a security guard at a mall gate who lets everyone in, every day. If only 1 in 100 visitors is a thief, he is "right" 99% of the time. He has also never stopped a single thief.
Accuracy counts all correct answers equally. When one class is rare, getting the common class right is easy and tells you nothing.
Example
10,000 UPI transactions, 100 of them fraud (1%). The model predicts "legit" for everything.
Correct predictions : 9,900 legit
Accuracy : 9,900 / 10,000 = 99%
Confusion matrix: predicted fraud predicted legit
actual fraud 0 100
actual legit 0 9,900
Recall for fraud = 0 / (0 + 100) = 0%Recall is zero: not one fraud is caught. A better model might have 90% accuracy overall yet catch 85 of 100 frauds. That model is far more useful, even though its accuracy is lower.
Better tools here: recall and precision of the minority class, F1, the precision-recall curve, and a cost-based view (what does a missed fraud cost versus a blocked genuine payment?).
What to say in the interview
"Accuracy is misleading when classes are imbalanced because a trivial majority-class predictor scores high. I check the confusion matrix, recall and precision for the rare class, PR-AUC, and choose the threshold using the business cost of each error type."
Common mistakes
- Reporting accuracy alone for rare-event problems.
- Balancing the test set; only the training data may be resampled.