The OLS Engine

Theoretical Foundations of Bivariate Linear Models

Lesson 1: Derivation & Matrix Algebra

The Minimization Problem

Why do we choose the "least squares" criterion?

"Can you prove mathematically why the mean is the minimizer of the sum of squared errors, and extend that logic to a regression line?"

Consider the loss function: \[ L(\theta) = \sum_{i=1}^n (y_i - \theta)^2 \]

Start thinking: What is the derivative of $L$ with respect to $\theta$?

Matrix Representation

For $n$ observations, we express our bivariate model as:

\[ \mathbf{y} = \mathbf{X}\beta + \epsilon \]

y: $n \times 1$ vector of responses
X: $n \times 2$ design matrix (ones and $x_i$)
\(\beta\): $2 \times 1$ vector of parameters

\[ \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix} \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} + \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix} \]

The RSS Function

We seek to minimize the Residual Sum of Squares (RSS):

\[ S(\beta) = (\mathbf{y} - \mathbf{X}\beta)^T (\mathbf{y} - \mathbf{X}\beta) \]

1. Expand

Multiply the transpose through the parentheses.

2. Differentiate

Take the gradient with respect to the vector $\beta$.

3. Solve

Set equal to zero (Normal Equations).

Derivation Step

Expanding $S(\beta)$: \[ S(\beta) = \mathbf{y}^T\mathbf{y} - \beta^T\mathbf{X}^T\mathbf{y} - \mathbf{y}^T\mathbf{X}\beta + \beta^T\mathbf{X}^T\mathbf{X}\beta \] Since $\beta^T\mathbf{X}^T\mathbf{y}$ is a scalar, it equals its transpose $\mathbf{y}^T\mathbf{X}\beta$: \[ S(\beta) = \mathbf{y}^T\mathbf{y} - 2\beta^T\mathbf{X}^T\mathbf{y} + \beta^T\mathbf{X}^T\mathbf{X}\beta \]

The Gradient:

\[ \frac{\partial S}{\partial \beta} = -2\mathbf{X}^T\mathbf{y} + 2\mathbf{X}^T\mathbf{X}\beta \]

The Solution

Setting the gradient to zero gives the Normal Equations:

\[ \mathbf{X}^T\mathbf{X}\hat{\beta} = \mathbf{X}^T\mathbf{y} \]

The OLS Estimator

\[ \hat{\beta} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \]

Requirement: $(\mathbf{X}^T\mathbf{X})$ must be non-singular (full column rank).

Orthogonal Projection

The vector of fitted values $\hat{\mathbf{y}}$ is the orthogonal projection of $\mathbf{y}$ onto the column space of $\mathbf{X}$.

\[ \hat{\mathbf{y}} = \mathbf{X}\hat{\beta} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \]

We define the Hat Matrix ($H$):

\[ \mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T \]

y

Col(X)

ŷ

ϵ

Residuals are orthogonal to the prediction space.

Regression Rigor

The OLS Engine

Residual Analysis and Heteroscedasticity Drop-off

The Normality Niche

Linearizing the World

Beyond the R-Squared Mirror

The OLS Engine Slides

Calculus of Least Squares Worksheet

Real Estate Residuals Case Study

Real Estate Residuals Key

Normality Diagnostics Slides

Diagnostic Decision Tree Reference

Transformation Translator Reference Sheet

Exponential Expansion Lab Activity

Prediction Pitfall Validation Project

Beyond R-Squared Slides

Similar Sequences

Slice Lab Fraction Division

dfredericks

Ratio Lab Intervention

dfredericks

Number Scouts Sequence

dfredericks

Statistiques Exploratoires

Hugo Pierrard

Division in Daily Life

Saiko Otake

Radical Relationships

Calculus Foundations

Transformation Surgeons

Sigma Summit

Solution Sleuths

Period Power Remediation

GCF Visual Remediation

Angle Avengers

Integer Investigation

Big Batch Bakery

Discount Deal Detectives

Algebraic Foundations

Polynomial Powers

Fraction Mastery

Proportional Powerhouse

Integer Mastery

Number Line Mastery

Modern Math for Families

Math Night Masters

Exponential Strategy Masterclass

Estimation Essentials