Practical Pitfalls

Question 1

What is data leakage in ML and statistical modeling?

Answer 1

Using information during training or feature creation that would not be available at prediction time, leading to overly optimistic evaluation and poor deployment performance.

Question 2

What is target leakage specifically?

Answer 2

A type of data leakage where features directly or indirectly encode the target variable using information from the future or post-outcome events.

Question 3

Why is target leakage particularly dangerous?

Answer 3

It can make models appear extremely accurate in offline evaluation, only to collapse when deployed because the leaked information is absent.

Question 4

What is an example of target leakage in a credit risk model?

Answer 4

Including a feature like ‘loan written off’ or ‘days delinquent after default date’ when predicting default at application time.

Question 5

How can cross-validation be misused to create leakage?

Answer 5

By computing feature transformations, scaling, or imputations on the full dataset before splitting, so information from validation folds influences training.

Question 6

What is look-ahead bias in time-series modeling?

Answer 6

Using data from the future in training or evaluation when simulating predictions that would have been made in the past.

Question 7

How do you avoid look-ahead bias in time-series evaluation?

Answer 7

Use chronological splits where training uses only past data and validation/test use future periods.

Question 8

Why is using the test set repeatedly for model tuning a pitfall?

Answer 8

It effectively turns the test set into another validation set, causing optimistic bias in reported performance.

Question 9

What is the correct role of the test set in ML experiments?

Answer 9

To provide a final, unbiased estimate of performance after all model selection and tuning decisions are complete.

Question 10

What is selection bias in datasets?

Answer 10

Bias introduced when the observed data are not a random sample from the target population, often due to the way data are collected or filtered.

Question 11

How can selection bias affect model performance in production?

Answer 11

Models trained on biased samples may perform poorly when deployed to a broader or different population.

Question 12

What is covariate shift?

Answer 12

A change between training and deployment in the distribution of input features while the conditional distribution of outputs given inputs remains the same.

Question 13

What is label shift?

Answer 13

A change in the distribution of labels across environments with relatively stable conditional distribution of inputs given labels.

Question 14

Why is ignoring distribution shift a pitfall?

Answer 14

Models evaluated only under the training distribution may fail when real-world conditions change, leading to unexpected degradation.

Question 15

What is class imbalance and why is it problematic?

Answer 15

When one class is much more frequent than others; naive models can achieve high accuracy by predicting the majority class while failing on the minority.

Question 16

What metric-related mistake is common with imbalanced data?

Answer 16

Relying on accuracy instead of metrics like precision, recall, F1, or PR-AUC that focus on the minority class.

Question 17

What is overfitting in the context of model complexity?

Answer 17

When a model fits noise and idiosyncrasies in the training data, achieving low training error but high test error.

Question 18

Why is evaluating a very flexible model on a tiny validation set a pitfall?

Answer 18

Random noise in the small validation set can mislead model selection, making unstable models look best by chance.

Question 19

What is the danger of p-hacking in statistical analysis?

Answer 19

Testing many hypotheses or analysis variants and only reporting significant ones inflates the false positive rate and undermines trust in results.

Question 20

Why is ‘p<0.05’ not adequate evidence on its own?

Answer 20

It ignores effect size, uncertainty, prior plausibility, multiple testing, and costs/benefits; context is essential.

Question 21

What is a common misinterpretation of a 95% confidence interval?

Answer 21

Believing there is a 95% probability that the true parameter lies in this specific interval, rather than understanding it as a long-run coverage property.

Question 22

Why is extrapolating far beyond the range of training data risky?

Answer 22

Model relationships that hold within the observed range may not hold outside it, leading to wildly inaccurate predictions.

Question 23

What is label noise?

Answer 23

Errors or inconsistencies in target labels, such as misclassifications or ambiguous outcomes.

Question 24

How can heavy label noise affect model performance and evaluation?

Answer 24

It can cap achievable accuracy, cause models to overfit spurious patterns, and distort metrics if not accounted for.

Question 25

What pitfall arises from ignoring missing data mechanisms?

Answer 25

Assuming missingness is random when it depends on unobserved factors can bias estimates and model predictions.

Question 26

What is MCAR vs MAR vs MNAR in missing data?

Answer 26

Missing Completely At Random, Missing At Random (depends only on observed data), and Missing Not At Random (depends on unobserved values).

Question 27

Why is deleting all rows with missing values often a bad idea?

Answer 27

It can waste data and introduce bias if missingness is related to the outcome or covariates.

Question 28

What is data snooping in feature engineering?

Answer 28

Looking at test or validation labels while creating features or selecting variables, thereby bleeding information from evaluation sets into training.

Question 29

Why is scaling features on the whole dataset before splitting a subtle pitfall?

Answer 29

It uses statistics from test/validation data inside training transformations, introducing mild leakage and optimistic estimates.

Question 30

What is multiple comparisons in ML model search?

Answer 30

Training many models or evaluating many metrics and then cherry-picking the best result without accounting for the number of tried alternatives.

Question 31

How can ensembling be misused as a pitfall?

Answer 31

Blindly stacking many models without understanding their diversity and error correlation can lead to opaque systems that are hard to debug or govern.

Question 32

Why is ignoring uncertainty in metrics a practical pitfall?

Answer 32

Treating noisy metric differences as real improvements may cause unnecessary model churn and degraded performance in production.

Question 33

What is a train/test contamination through temporal ordering?

Answer 33

Randomly splitting time-dependent data so that future events appear in the training set or leak into feature calculations for past test instances.

Question 34

Why is ignoring business constraints a statistical pitfall?

Answer 34

A model that optimizes metrics but violates latency, fairness, or operational constraints will not succeed in real deployment.

Question 35

What is survivorship bias in datasets?

Answer 35

Bias that occurs when only entities that 'survive' some process are observed, missing those that dropped out or failed earlier.

Question 36

How can survivorship bias distort models?

Answer 36

They may learn from only the successes or survivors, misestimating risk or performance for new cases.

Question 37

Why is blindly trusting default library settings a pitfall?

Answer 37

Defaults may not suit your data’s scale, distribution, or problem type, leading to suboptimal or misleading results if not examined.

Question 38

What is a key sign that a result may be 'too good to be true'?

Answer 38

Extremely high metrics compared to baseline or domain expectations, especially if accompanied by complex pipelines and minimal scrutiny for leakage.

Question 39

In one sentence, what is the overarching lesson about practical statistical pitfalls?

Answer 39

Be suspicious of surprisingly good results, keep evaluation data sacred, respect time and data-generating processes, and always ask how your data and setup could be lying to you.