Linear Regression as a Statistical Model

Question 1

What is linear regression at a high level?

Answer 1

A model that assumes the expected value of a response variable Y is a linear combination of input features X plus an error term.

Question 2

What is the standard form of a multiple linear regression model?

Answer 2

Y = β₀ + β₁X₁ + ··· + β_pX_p + ε, where ε is an error term with mean zero.

Question 3

In linear regression, what do the coefficients β represent?

Answer 3

They quantify the expected change in the response Y associated with a one-unit change in each predictor X_j , holding other predictors constant.

Question 4

What is the role of the error term ε in the linear regression model?

Answer 4

It captures unexplained variability in Y due to measurement noise, omitted variables, and inherent randomness.

Question 5

What is the main goal when fitting a linear regression model?

Answer 5

To estimate coefficients β that best explain the relationship between X and Y according to some optimality criterion, usually minimizing squared error.

Question 6

What is ordinary least squares (OLS)?

Answer 6

A method of estimating β by minimizing the sum of squared residuals between observed Y and model predictions.

Question 7

What is a residual in regression?

Answer 7

The difference between the observed value of Y and the predicted value Ŷ for a given observation.

Question 8

Why is the sum of squared residuals used in OLS?

Answer 8

It penalizes larger errors more heavily, leads to convenient analytic solutions, and corresponds to maximum likelihood under Gaussian noise.

Question 9

Under what distributional assumption does OLS coincide with maximum likelihood estimation?

Answer 9

When the errors ε are assumed i.i.d. Normal with mean zero and constant variance.

Question 10

What does ‘linear in parameters’ mean?

Answer 10

The model is linear with respect to the coefficients β, even if it includes nonlinear transformations of inputs (e.g., X², log X).

Question 11

How can we model nonlinear relationships within linear regression?

Answer 11

By including engineered features such as polynomial terms, interactions, or transformations of the original inputs.

Question 12

What is the design matrix X in linear regression?

Answer 12

An n×(p+1) matrix whose rows are observations and columns are predictors (including a column of ones for the intercept).

Question 13

What is the closed-form OLS solution in matrix notation (when XᵀX is invertible)?

Answer 13

β̂ = (XᵀX)⁻¹ Xᵀ y.

Question 14

What are the common assumptions of the classical linear regression model?

Answer 14

Linearity in parameters, independence of errors, errors with mean zero, constant error variance (homoscedasticity), and often Normality of errors for inference.

Question 15

What does homoscedasticity mean?

Answer 15

That the variance of the errors is constant across all levels of the predictors.

Question 16

What is heteroscedasticity?

Answer 16

A situation where error variance changes with the level of predictors, violating the homoscedasticity assumption.

Question 17

Why is heteroscedasticity problematic for classical inference?

Answer 17

It can make standard errors and confidence intervals from basic formulas invalid, even if OLS estimates remain unbiased under certain conditions.

Question 18

What is multicollinearity in linear regression?

Answer 18

A condition where some predictors are highly correlated with each other, making coefficient estimates unstable and difficult to interpret.

Question 19

How does severe multicollinearity affect OLS estimates?

Answer 19

Small changes in data can cause large swings in estimated coefficients, and standard errors become large.

Question 20

Does multicollinearity necessarily harm predictive performance?

Answer 20

Not always; predictions can still be good, but interpretability of individual coefficients suffers.

Question 21

What is R² (coefficient of determination)?

Answer 21

The proportion of variability in Y explained by the regression model relative to a baseline that predicts the mean of Y.

Question 22

How is R² interpreted?

Answer 22

R²=1 means the model explains all variability in Y; R²=0 means it does no better than predicting the mean for all observations.

Question 23

Why can R² be misleading when comparing models with different numbers of predictors?

Answer 23

R² never decreases when new predictors are added, even if they offer no real explanatory power, encouraging overfitting.

Question 24

What is adjusted R²?

Answer 24

A modified version of R² that penalizes additional predictors, providing a more balanced comparison across models.

Question 25

What is the Gauss–Markov theorem (intuitively)?

Answer 25

Under certain assumptions, OLS gives the best linear unbiased estimator (BLUE) of β, meaning the smallest variance among all linear unbiased estimators.

Question 26

What does 'best linear unbiased estimator' not guarantee?

Answer 26

It does not guarantee best possible estimator among all nonlinear or biased estimators, only among linear unbiased ones.

Question 27

What is the difference between in-sample and out-of-sample performance?

Answer 27

In-sample performance measures fit on the training data, while out-of-sample performance evaluates generalization to new data.

Question 28

Why should model selection be based on out-of-sample performance rather than just R² on the training set?

Answer 28

Because R² on training data can be inflated by overfitting, while out-of-sample performance reflects generalization ability.

Question 29

What is leverage in linear regression diagnostics?

Answer 29

A measure of how extreme an observation's predictor values are relative to others; high leverage points can strongly influence the fitted line.

Question 30

What is an influential point?

Answer 30

An observation that, if removed, would significantly change the fitted regression; often identified using leverage and residual diagnostics.

Question 31

Why are diagnostic plots important for regression?

Answer 31

They help detect violations of assumptions (nonlinearity, heteroscedasticity), outliers, and influential points that may distort results.

Question 32

What is the difference between interpolation and extrapolation in regression?

Answer 32

Interpolation predicts within the range of observed X values; extrapolation predicts outside that range, which is often risky and unreliable.

Question 33

Why can a linear regression with a high R² still be a poor model?

Answer 33

It might violate assumptions, be driven by outliers, capture spurious relationships, or be mis-specified for the question at hand.

Question 34

How does linear regression relate to least-squares linear classification?

Answer 34

If you threshold continuous predictions to classify, you effectively get a linear decision boundary, though logistic regression is usually more appropriate for classification.

Question 35

In ML practice, why is linear regression still widely used despite more complex models?

Answer 35

It is simple, fast, interpretable, and can perform competitively when paired with good feature engineering and regularization.

Question 36

In one sentence, what is the core mental model for linear regression as a statistical model?

Answer 36

It assumes that the average response is a linear function of features plus noise and estimates the coefficients by minimizing squared error under assumptions that justify inference and interpretation.