The OLS Engine Slides The OLS Engine
Theoretical Foundations of Bivariate Linear Models
Lesson 1: Derivation & Matrix Algebra
The Minimization Problem
Why do we choose the "least squares" criterion?
"Can you prove mathematically why the mean is the minimizer of the sum of squared errors, and extend that logic to a regression line?"
Consider the loss function: \[ L(\theta) = \sum_{i=1}^n (y_i - \theta)^2 \]
Start thinking: What is the derivative of $L$ with respect to $\theta$?
Matrix Representation
For $n$ observations, we express our bivariate model as:
\[ \mathbf{y} = \mathbf{X}\beta + \epsilon \]
y: $n \times 1$ vector of responses
X: $n \times 2$ design matrix (ones and $x_i$)
\(\beta\): $2 \times 1$ vector of parameters
\[ \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} + \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix} \]
The RSS Function
We seek to minimize the Residual Sum of Squares (RSS):
\[ S(\beta) = (\mathbf{y} - \mathbf{X}\beta)^T (\mathbf{y} - \mathbf{X}\beta) \]
1. Expand
Multiply the transpose through the parentheses.
2. Differentiate
Take the gradient with respect to the vector $\beta$.
3. Solve
Set equal to zero (Normal Equations).
Derivation Step
Expanding $S(\beta)$: \[ S(\beta) = \mathbf{y}^T\mathbf{y} - \beta^T\mathbf{X}^T\mathbf{y} - \mathbf{y}^T\mathbf{X}\beta + \beta^T\mathbf{X}^T\mathbf{X}\beta \] Since $\beta^T\mathbf{X}^T\mathbf{y}$ is a scalar, it equals its transpose $\mathbf{y}^T\mathbf{X}\beta$: \[ S(\beta) = \mathbf{y}^T\mathbf{y} - 2\beta^T\mathbf{X}^T\mathbf{y} + \beta^T\mathbf{X}^T\mathbf{X}\beta \]
The Gradient:
\[ \frac{\partial S}{\partial \beta} = -2\mathbf{X}^T\mathbf{y} + 2\mathbf{X}^T\mathbf{X}\beta \]
The Solution
Setting the gradient to zero gives the Normal Equations:
\[ \mathbf{X}^T\mathbf{X}\hat{\beta} = \mathbf{X}^T\mathbf{y} \]
The OLS Estimator
\[ \hat{\beta} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \]
Requirement: $(\mathbf{X}^T\mathbf{X})$ must be non-singular (full column rank).
Orthogonal Projection
The vector of fitted values $\hat{\mathbf{y}}$ is the orthogonal projection of $\mathbf{y}$ onto the column space of $\mathbf{X}$.
\[ \hat{\mathbf{y}} = \mathbf{X}\hat{\beta} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \]
We define the Hat Matrix ($H$):
\[ \mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T \]
y
Col(X)
ŷ
ϵ
Residuals are orthogonal to the prediction space.
Calculus of Least Squares Worksheet The Calculus of Least Squares
Bivariate Regression Derivation • Graduate Statistics
NAME: ___________________________
DATE: ___________________________
Problem Statement
In this workshop, you will derive the analytical solutions for the bivariate regression coefficients \(\beta_0\) (intercept) and \(\beta_1\) (slope). While computer packages solve these instantly, understanding the optimization landscape is critical for diagnosing model failures.
1. The Objective Function
Given a set of $n$ observations $(x_i, y_i)$, we define our model as \( y_i = \beta_0 + \beta_1 x_i + \epsilon_i \). We seek to minimize the Sum of Squared Residuals (SSR):
\[ S(\beta_0, \beta_1) = \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i)^2 \]
Task 1: Calculate the partial derivatives of $S$ with respect to $\beta_0$ and $\beta_1$.
PARTIAL WITH RESPECT TO \(\beta_0\):
PARTIAL WITH RESPECT TO \(\beta_1\):
2. Solving the System
By setting the partial derivatives to zero, we obtain the Normal Equations . Rearrange your derivatives above to match this system:
\[ n\hat{\beta}_0 + \hat{\beta}_1\sum x_i = \sum y_i \]
\[ \hat{\beta}_0\sum x_i + \hat{\beta}_1\sum x_i^2 = \sum x_i y_i \]
Task 2: Solve the first equation for \(\hat{\beta}_0\) in terms of \(\hat{\beta}_1, \bar{y},\) and \(\bar{x}\).
3. The Slope Estimator
Substitute your expression for \(\hat{\beta}_0\) into the second Normal Equation and isolate \(\hat{\beta}_1\). Show that the estimator can be expressed as the ratio of sample covariance to sample variance of $x$.
Final Result Verification:
\[ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} \]
4. Matrix Geometry (Challenge)
Let \(\mathbf{y}\) be the vector of observations and \(\mathbf{X}\) be the design matrix. We define the hat matrix as \(\mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\).
A. Prove that $\mathbf{H}$ is idempotent (i.e., $\mathbf{H}^2 = \mathbf{H}$).
B. Explain the geometric meaning of idempotency in the context of projection.
Real Estate Residuals Case Study Real Estate Residuals
Case Study: Violating the Constant Variance Assumption
NAME: ___________________________
DATE: ___________________________
The Dataset: Metro City Housing
You are analyzing the relationship between Square Footage (Predictor, $X$) and Sale Price (Response, $Y$) for 1,500 residential properties. Initial OLS regression yields an $R^2$ of 0.82. However, your supervisor is concerned about the reliability of the standard errors for high-value properties.
1. The Residual Scatter Plot
Below is a conceptual representation of the residuals vs. fitted values. Observe the "fan" shape.
Residuals (\(e_i\)) Fitted Values (\(\hat{y}_i\))
A. Describe the pattern shown above. What does this suggest about the relationship between the variance of the errors and the magnitude of the predicted price?
B. Why is this a problem for OLS? Specifically, discuss the impact on the BLUE property (Best Linear Unbiased Estimator).
2. The Breusch-Pagan Test
To move beyond visual suspicion, we perform a formal test. The Breusch-Pagan test regresses the squared residuals against the predictor: \( e_i^2 = \gamma_0 + \gamma_1 x_i + \nu_i \).
Test Statistics:
Null Hypothesis ($H_0$): Homoscedasticity exists.LM Statistic: 42.18Degrees of Freedom: 1p-value: < 0.0001
Interpretation Task:
Based on the results above, what is your conclusion at the \(\alpha = 0.05\) level? Does the model require remedial measures?
3. Standard Error Inflation
In the presence of heteroscedasticity, OLS estimators remain unbiased, but they are no longer efficient.
Explain how failing to account for this non-constant variance affects the confidence intervals for the price of a \$2M mansion versus a \$200k condo.
If we cannot transform the data, what alternative estimation method might we use to provide "Robust Standard Errors"?
Next Steps
In Lesson 4, we will apply the Box-Cox Transformation to address this specific real estate dataset. Be prepared to discuss why a log-transformation might stabilize the variance here.
Real Estate Residuals Key Answer Key & Teaching Guide
Real Estate Residuals Case Study • Instructor Copy
VERSION: 1.0.A
RECOVERY: DIAGNOSTICS
1. Visual Inspection Solutions
Part A: Pattern Description
The plot shows heteroscedasticity , specifically a "fanning" or "megaphone" pattern. As the fitted values (predicted price) increase, the spread (variance) of the residuals also increases. This suggests that the model is more precise for lower-valued homes and significantly less precise for high-valued homes.
Part B: Problem for OLS
Violating the homoscedasticity assumption means the OLS estimator is no longer BLUE (Best Linear Unbiased Estimator). While the coefficients \(\hat{\beta}\) remain unbiased (meaning they aren't systematically wrong on average), they are no longer efficient (they don't have the minimum possible variance). Most importantly, the standard errors computed by standard OLS formulas will be biased , leading to invalid hypothesis tests and confidence intervals.
2. Breusch-Pagan Interpretation
Interpretation Task
With a p-value < 0.0001, we reject the null hypothesis (\(H_0\)) of homoscedasticity. There is strong evidence that the error variance is not constant across levels of the predictor. Remedial measures (transformations or WLS) are required before drawing valid inferences.
3. Implications & Alternatives
Impact on Confidence Intervals
For \$200k condos, the OLS confidence interval might be too wide (conservative), while for \$2M mansions, the OLS interval will be dangerously narrow, leading to overconfidence in predictions where the model actually has the most error.
Alternative Method
Weighted Least Squares (WLS) or using Huber-White Robust Standard Errors (Sandwich estimators). These methods adjust the variance-covariance matrix to account for the non-constant variance without necessarily changing the coefficient estimates.
Instructional Notes
Common Misconception: Students often think heteroscedasticity makes the slope estimate "wrong." Emphasize that it affects the certainty (inference), not the average prediction (unbiasedness).Visual Cue: Remind students that the "fan" can go both ways, or be "bowed." Any non-random pattern in the spread is cause for concern.The "Why": In real estate, a 5% error on a \$200k house is \$10k. A 5% error on a \$2M house is \$100k. The absolute variance naturally scales with the price. Normality Diagnostics Slides The Normality Niche
Diagnosing the Gaussian Assumption in Residuals
Lesson 3
dist ~ N(0, σ²)
Why do we care?
The Gauss-Markov Theorem does NOT require normality for OLS to be BLUE.
The Catch:
We need normality for our t-tests , F-tests , and Confidence Intervals to be exact.
Simulation Insight
"Small samples with heavy-tailed errors lead to inflated Type I error rates."
Standard errors are underestimated
P-values become unreliable
Quantile-Quantile (Q-Q) Plots
Theoretical Quantiles Sample Quantiles
Heavy Tails
Points curve away from the line at both ends (S-shape).
Skewed Data
Points curve only at one end (C-shape).
Shapiro-Wilk Test
How it works
Tests the null hypothesis that a sample $x_1, \dots, x_n$ came from a normally distributed population.
W = (Σ a_i x_(i))² / Σ (x_i - x̄)²
p < α
Reject $H_0$: Evidence of Non-Normality
p > α
Fail to reject $H_0$: Assume Normality
Note: For large $n$, even tiny deviations become significant. Use visual tools!
Diagnostic Decision Tree Reference Diagnostic Decision Tree
Bivariate Regression Assumption Validation • Teacher Reference
Reference ID: STAT-DIAG-003
Assumption: Normality of Errors (\(\epsilon \sim N(0, \sigma^2)\))
This cheat sheet guides the student (and instructor) through the evaluation process for bivariate residuals.
1. Visual Check (The Q-Q Plot)
Straight Diagonal:
Ideal. Normality assumption is satisfied.
"S" Shape (Bowed at ends):
Heavy or Light Tails (Kurtosis issues). Potential outliers.
"C" Shape (Bowed in center):
Skewness. May require a variable transformation (Lesson 4).
2. Numerical Check
Shapiro-Wilk Test
Best for small to moderate samples (\(n < 2000\)).
If \(p < 0.05\): Violation found.
Kolmogorov-Smirnov
Alternative for larger datasets.
Sensitive to the center of the distribution.
3. Impact of Violations
Sample Size Priority Level Implication Small (\(n < 30\)) Critical Standard errors and p-values are untrustworthy. Tests are not exact. Moderate (\(30 < n < 100\)) Moderate CLT starts to help, but heavy skewness still biases inference. Large (\(n > 100\)) Lower Central Limit Theorem ensures that \(\hat{\beta}\) is approximately normal regardless of error distribution.
Remedial Strategies (Preview of Lesson 4)
Log Transformation: Addresses right-skewed residuals.
Box-Cox: Iterative search for optimal power \(\lambda\).
Transformation Translator Reference Sheet The Transformation Translator
Interpreting Coefficients in Non-Linear Bivariate Models
GRADUATE LEVEL
UNIT: BIVARIATE DATA
When we transform variables to meet OLS assumptions, we fundamentally change the interpretation of our coefficients (\(\beta_1\)). This guide provides the mathematical "dictionary" for the four most common model specifications.
1. Level - Level (Standard) y = \(\beta_0\) + \(\beta_1\)x + \(\epsilon\)
Visual Pattern:
Straight line.
Interpretation:
A 1-unit increase in \(x\) is associated with a \(\beta_1\) unit increase in \(y\).
2. Log - Level (Growth) ln(y) = \(\beta_0\) + \(\beta_1\)x + \(\epsilon\)
Visual Pattern:
Exponential growth/decay in raw data.
Interpretation:
A 1-unit increase in \(x\) is associated with a \((100 \times \beta_1)\)% change in \(y\).
3. Level - Log (Diminishing) y = \(\beta_0\) + \(\beta_1\)ln(x) + \(\epsilon\)
Visual Pattern:
Rapid initial growth that levels off.
Interpretation:
A 1% increase in \(x\) is associated with a \((\beta_1 / 100)\) unit change in \(y\).
4. Log - Log (Elasticity) ln(y) = \(\beta_0\) + \(\beta_1\)ln(x) + \(\epsilon\)
Visual Pattern:
Curved relationship; used for "constant elasticity."
Interpretation:
A 1% increase in \(x\) is associated with a \(\beta_1\)% change in \(y\).
Pro-Tip: The Box-Cox Transformation
Don't guess the transformation. Use the Box-Cox procedure to find the optimal power \(\lambda\). \[ y^{(\lambda)} = \begin{cases} \frac{y^\lambda - 1}{\lambda} & \text{if } \lambda \neq 0 \\ \ln(y) & \text{if } \lambda = 0 \end{cases} \] R/Python will optimize \(\lambda\) to maximize the likelihood function. Common values: \(\lambda=0.5\) (square root), \(\lambda=0\) (log), \(\lambda=-1\) (inverse).
Exponential Expansion Lab Activity Exponential Expansion Lab
Modeling Bacterial Growth & Log-Linearization
STUDENT NAME: ____________________
LAB SESSION: ____________________
Biological Scenario
You are monitoring a culture of E. coli . Every hour, the population size is measured. You suspect the growth follows an exponential model: \( y = \alpha e^{\beta x} \), where \(y\) is population and \(x\) is time in hours. To use OLS, you must linearize this relationship.
1. Mathematical Prep
Convert the exponential model \( y = \alpha e^{\beta x} \) into a linear regression form by taking the natural log of both sides. Show your work.
How does the intercept (\(\beta_0\)) in your linear model relate to the original \(\alpha\)?
2. Growth Data Table
Time (h) Pop. Size (y) ln(y) 1 148 4.997 2 403 6.000 3 1097 7.000 4 2981 8.000
Task: Estimate \(\hat{\beta}_1\)
Looking at the ln(y) column, estimate the slope of the linearized model. What is the hourly growth rate?
3. Impact Analysis
If the estimated slope is \(\hat{\beta}_1 = 1.0\), provide a sentence interpreting this result for a non-statistician biologist.
Prediction Pitfall Validation Project The Prediction Pitfall
FINAL VALIDATION WORKSHOP • BIVARIATE MODELS
PROJECT: CV-BIV-5
OBJECTIVE: ROBUSTNESS
"A model that fits perfectly but predicts poorly is a sophisticated way of lying to yourself about the future."
In this final project, you will move beyond in-sample metrics like \(R^2\). You will validate your model's ability to generalize to unseen data using Leave-One-Out Cross-Validation (LOOCV).
Beware of Overfitting
1. Defining Performance
Standard OLS minimizes \( \sum (y_i - \hat{y}_i)^2 \). However, for validation, we care about the Mean Squared Prediction Error (MSPE) . Calculate MSPE based on the provided cross-validation results:
Fold (k)
Observed \(y\)
Predicted \(\hat{y}_{(-k)}\)
1
14.2
13.8
2
22.1
25.4
3
18.5
19.0
4
31.0
28.2
Calculate the CV error for each point and find the total MSPE:
2. Synthesis
Explain why a model with a lower training \(R^2\) might actually have a lower cross-validation error than a more "complex" model (e.g., adding high-degree polynomial terms to a bivariate plot).
Validation Mastery Rubric
Criteria Developing (1-2) Proficient (3-4) Mastery (5) Diagnostic Rigor Identifies basic violations but misses subtle patterns. Correctly interprets BP and SW tests. Synthesizes visual and numerical evidence perfectly. Validation Logic Confuses training error with predictive error. Calculates MSPE correctly using hold-out sets. Articulates the bias-variance tradeoff in context. Interpretative Clarity Interpretation of transformed coefficients is literal. Uses log-log/log-level phrasing accurately. Explains model limitations and scope of inference.
Final Reflection Prompt
Reflect on the essential question: "How do we validate the mathematical assumptions underlying bivariate linear models to ensure statistical inference is robust and reliable?" Describe one specific diagnostic technique you found most insightful and how it changes your approach to data modeling.
Beyond R-Squared Slides Beyond R-Squared
Model Validation & The Future of Generalization
The Problem with In-Sample Fit
A high Coefficient of Determination ($R^2$) tells us how well our model explains the current data.
The Danger:
Adding terms always increases $R^2$, even if they are noise.
Visualizing Overfitting
Capturing Noise, Not Signal
LOOCV Framework
Leave-One-Out Cross-Validation (LOOCV):
1
Exclude observation $i$.
2
Fit model on $n-1$ points.
3
Predict excluded $y_i$.
4
Calculate error $e_{(i)}$.
CV_{(n)} = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_{(i)})^2
The Robust Statistician
Your Diagnostic Checklist:
Derive estimators from first principles.
Audit residuals for heteroscedasticity.
Validate normality using Q-Q plots.
Linearize via transformations when appropriate.
Always cross-validate your predictive claims.
