Interview Q&A / Data Science

1. What are variance and standard deviation, and why do we need them?

   Variance is the average of the squared distances from the mean, and standard deviation is its square root. They measure how spread out the data is. Two datasets can have the same mean but very different spread, and standard deviation is easier to read because it is in the same units as the data.

   BeginnerIn-depth3 min read
2. What is the difference between correlation and causation?

   Correlation means two variables move together. Causation means a change in one actually produces a change in the other. Correlation can come from a hidden third factor, from chance, or from cause running the other way, so proving causation usually needs a controlled experiment such as an A/B test.

   BeginnerIn-depth3 min read
3. What is the difference between mean, median and mode, and when is the median better?

   The mean is the sum divided by the count, the median is the middle value after sorting, and the mode is the most frequent value. Use the median when the data is skewed or has outliers, like salaries, because one very large value pulls the mean up but barely moves the median.

   BeginnerIn-depth4 min read
4. What is the difference between merge, join and concat in pandas?

   merge combines two DataFrames on matching column values, like a SQL join. join does the same but matches on the index by default. concat simply stacks DataFrames on top of each other (rows) or side by side (columns) without matching any keys.

   BeginnerIn-depth3 min read
5. What is the difference between normalisation and standardisation?

   Normalisation (min-max scaling) squeezes values into a fixed range, usually 0 to 1, using (x - min) / (max - min). Standardisation (z-score scaling) shifts values to mean 0 and standard deviation 1, using (x - mean) / std. Standardisation handles outliers better, and both should be fitted on training data only.

   BeginnerIn-depth4 min read
6. How do you detect outliers using the IQR and z-score methods?

   With the IQR method, a value is an outlier if it lies below Q1 minus 1.5 times the IQR or above Q3 plus 1.5 times the IQR. With the z-score method, a value is flagged if it is more than about 3 standard deviations from the mean. IQR is more robust for skewed data and small samples, because outliers inflate the mean and standard deviation.

   IntermediateIn-depth4 min read
7. How do you handle missing values in a dataset?

   First find how much is missing and why. Then either drop rows or columns (when little is missing or a column is mostly empty) or fill the gaps, using the median for skewed numbers, the mode for categories, forward fill for time series, or a model-based method. Often add a flag column that records which values were missing.

   IntermediateIn-depth4 min read
8. What is hypothesis testing, and what does a p-value actually mean?

   Hypothesis testing checks whether data is surprising under a default assumption called the null hypothesis. The p-value is the probability of seeing results at least as extreme as yours if the null hypothesis were true. A small p-value is evidence against the null. It is not the probability that the null is true.

   AdvancedIn-depth5 min read