Ridge Regression and Shrinkage

Lesson 1: L2 Regularization and the Bias-Variance Tradeoff

The Challenge of \( p > n \)

In modern datasets, the number of predictors \( p \) often exceeds the number of observations \( n \).

Standard OLS fails: The matrix \( \mathbf{X}^T\mathbf{X} \) is singular and non-invertible.
Overfitting: Even if \( p < n \), high dimensionality leads to high variance in coefficient estimates.

Geometric Intuition

"In high dimensions, the data points are sparsely distributed. A standard hyperplane will 'wiggle' to pass through noise, leading to extreme coefficient values."

Introducing L2 Regularization

Instead of minimizing just the Residual Sum of Squares (RSS), we add a shrinkage penalty:

\[ \min_{\beta} \left\{ \sum_{i=1}^{n} (y_i - \beta_0 - \sum_{j=1}^{p} x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \right\} \]

RSS Term

Measures goodness of fit to the training data.

Shrinkage Penalty

The L2 norm \( \lambda ||\beta||_2^2 \) penalizes large coefficients.

Tuning Parameter

\( \lambda \geq 0 \) controls the strength of the penalty.

Mathematical Stability

One major advantage of Ridge over Lasso is the existence of a unique, closed-form solution:

\[ \hat{\beta}_{ridge} = (\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T\mathbf{y} \]

By adding \( \lambda \) to the diagonal of \( \mathbf{X}^T\mathbf{X} \)...

We guarantee the matrix is non-singular and invertible, even if \( p > n \).

The Effect of \(\lambda\)

As \(\lambda \to 0\):

The penalty loses influence. The coefficients approach the OLS estimates.

As \(\lambda \to \infty\):

The impact of the shrinkage penalty grows. Coefficients approach (but never exactly hit) zero.

Key Observation

Ridge regression performs **shrinkage** but not **selection**. All variables remain in the model.

Visual: Coefficient Path Graph showing coefficients decaying toward zero as \(\log(\lambda)\) increases.

Fig 1.1: Typical Ridge Coefficient Paths

The Tradeoff

Why use Ridge?

We accept a small amount of **bias** in exchange for a significant reduction in **variance**.

OLS

Unbiased, but can have massive variance in high dimensions.

Ridge

Biased, but stable and better for prediction on new data.

Graphic: U-Shaped Test Error Curve vs. Flexiblity

Ridge finds the "sweet spot" by reducing model complexity just enough to minimize total Mean Squared Error.

Regularization and Continuous Feature Selection

Ridge Regression and Shrinkage

The Lasso and Sparsity

Tuning Lambda via Cross Validation

Elastic Net and Correlation

Regularization in GLMs

Ridge Shrinkage Slides

Ridge Shrinkage Worksheet

Ridge Shrinkage Teacher Guide

Lasso Sparsity Slides

Lasso Geometry Worksheet

Lasso Teacher Guide

Tuning Lambda Slides

Tuning Path Worksheet

Tuning Teacher Guide

Elastic Net Slides

Elastic Net Activity

Elastic Net Teacher Guide

GLM Regularization Slides

GLM Case Study Worksheet

GLM Teacher Guide

Similar Sequences

The Addicted Brain

Neurobiology of Mental Illness

Flow Force

Teri Minami

Le Pouvoir Solaire

dimitri0681

Metabolic Engines

dimitri0681

Human Origins

Scientific Method Mystery

Human Nature and Conflict

Molecular Survival Strategies

Body Systems Blueprint

Neighbor Planets

Entropy and Energy Flow

Gravity Paradox

Thermal Trackers

Picnic Particles Chemistry

Cosmic Voyage

Green Grid Future

Symbiotic Systems

Ice Age Explorers

Habitat Detectives

Root Cause Investigation

Strategic Hazard Assessment and Risk Control

OSHA Standards and Compliance

Agitation Anatomy

Access Keys: Screen Reader Navigation and Context

Mouseless Mastery Sequence

System Architect Navigation

Keyboard Navigators

Focus Jumpers

Cursor Quest Navigation

Ethical Design Standards and Intellectual Property

Responsive Layouts and Mobile-First Development

Code Canvas

Portfolio Launchpad

Constructing Semantic HTML Structure

CSS Layout Mastery

Professional Web Portfolio Design