Error Decomposition Slides Advanced Model Selection • Lesson 1
Decomposing Error
The Mathematical Foundation of the Bias-Variance Tradeoff
The Generalization Problem
Our goal isn't to fit the training data; it's to predict accurately on unseen data.
We define the statistical model: \[ Y = f(X) + \epsilon \] Where:
\( f(X) \) is the true relationship.
\( \epsilon \) is irreducible noise: \( E[\epsilon] = 0, Var(\epsilon) = \sigma^2 \).
Expected Prediction Error (EPE)
\( E[(Y - \hat{f}(x))^2] \)
The Three Pillars of Error
Variance
The amount by which \( \hat{f} \) would change if we estimated it using a different training data set.
\( E[(\hat{f}(x) - E[\hat{f}(x)])^2] \)
Bias\(^2\)
The error introduced by approximating a real-life problem with a much simpler model.
\( (E[\hat{f}(x)] - f(x))^2 \)
Noise
The variance of the error terms \( \epsilon \). It cannot be reduced regardless of the model chosen.
\( Var(\epsilon) = \sigma^2 \)
The Complexity Curve
Synthesis
Variance (Increases with complexity)
Bias (Decreases with complexity)
Total Error (U-Shaped Curve)
Model Complexity →
Prediction Error
Finding the Sweet Spot
As complexity increases, we see:
Low Complexity: High Bias, Low Variance (Underfitting)
High Complexity: Low Bias, High Variance (Overfitting)
Optimal: The point where total error is minimized.
The Derivation
We will now proceed to mathematically prove this decomposition starting from the definition of Mean Squared Error.
Required Proof
Show that EPE can be expressed as the sum of three distinct non-negative components.
Key Concept
The "Sweet Spot" is where the derivative of the error curve equals zero.
Error Decomposition Worksheet Error Decomposition Derivation
Advanced Model Selection • Lesson 1
Student:
Date:
1. The Statistical Framework
Suppose we have a target variable \( Y \) and a vector of inputs \( X \), related by: \[ Y = f(X) + \epsilon \] where \( E[\epsilon] = 0 \) and \( Var(\epsilon) = \sigma^2 \). Let \( \hat{f}(x) \) be our estimate of \( f(x) \) obtained from a training set \( \mathcal{T} \). We wish to decompose the Expected Prediction Error (EPE) at a point \( x \): \[ EPE(x) = E_{\mathcal{T}, \epsilon} \left[ (Y - \hat{f}(x))^2 \right] \]
2. The Derivation
Complete the steps below to derive the bias-variance decomposition. Show all intermediate steps including the addition and subtraction of \( E[\hat{f}(x)] \).
Step A: Expand the squared term using \( Y = f(x) + \epsilon \).
Step B: Introduce \( E[\hat{f}(x)] \) into the expression and group terms for Bias and Variance.
Step C: Use the properties of expectations to show that the cross-product terms go to zero.
3. Synthesis & Application
A. Define the final three components based on your derivation:
Bias^2:
Variance:
Irreducible:
B. Scenario Analysis:
You are training a 50th-degree polynomial to fit 20 data points generated from a simple linear process with high noise. Describe the likely state of the bias and variance, and predict the model's performance on a large test set.
4. Computational Simulation
Briefly outline a simulation experiment to visualize the B-V tradeoff using R or Python (pseudocode is acceptable).
Asymptotic Penalties Slides Advanced Model Selection • Lesson 2
Asymptotic Penalties
Deriving and Implementing AIC & BIC in Statistical Inference
The Parsimony Principle
Likelihood always increases with complexity. If we only maximize \(\mathcal{L}(\theta)\), we will always choose the most complex model.
"Everything should be made as simple as possible, but not simpler."
Information Criteria Logic
1
Fit the data (Goodness of Fit)
-
2
Penalize parameters (Complexity)
Akaike Information Criterion (AIC)
Origin: KL Divergence
AIC aims to minimize the Kullback-Leibler divergence between the true distribution \(f\) and the model \(g\). \[ D_{KL}(f || g) = E_f[\log f(x)] - E_f[\log g(x|\theta)] \]
The Metric
AIC = -2\log(\hat{L}) + 2k
Where \(k\) is the number of parameters.
Properties
Asymptotically efficient.
Minimizes out-of-sample error.
Inconsistent: It may overfit as \(n \to \infty\).
Bayesian Information Criterion (BIC)
Origin: Bayes Factor
BIC approximates the log of the posterior probability of a model. It stems from the Laplace approximation of the integral over the parameter space.
The Metric
BIC = -2\log(\hat{L}) + k\log(n)
Where \(n\) is the number of observations.
Properties
Asymptotically consistent.
Selects the true model if it's in the set.
Harsher penalty for large datasets compared to AIC.
AIC vs. BIC: The Comparison
Feature AIC BIC Penalty Fixed at \(2k\) Sample dependent: \(k\log(n)\) Philosophy Predictive Accuracy Model Identification Consistency Inconsistent Consistent Small \(n\) Bias Overfits Underfits
Criterion Analysis Problem Set Criterion Analysis Problem Set
Advanced Model Selection • Lesson 2
Student:
Date:
1. Kullback-Leibler and the AIC Origin
AIC is an asymptotic estimate of the expected relative KL divergence. Given the true density \( f \) and a candidate model \( g(x|\theta) \), the divergence is: \[ D_{KL}(f || g) = E_f[\log f(x)] - E_f[\log g(x|\theta)] \]
Q1.1 Maximizing the Likelihood:
Explain why minimizing \( D_{KL} \) is equivalent to maximizing the expected log-likelihood \( E_f[\log g(x|\theta)] \). Why is this quantity difficult to estimate directly from a training sample?
Q1.2 The Bias Correction:
The training log-likelihood \( \log \hat{L} \) is a biased estimate of the expected log-likelihood. Under what specific asymptotic conditions is the bias exactly equal to the number of parameters \( k \)?
2. The AIC/BIC Crossover Point
Consider a scenario where you are comparing two nested models: \( M_1 \) with \( k_1 \) parameters and \( M_2 \) with \( k_2 \) parameters, where \( k_2 = k_1 + 1 \).
Derivation Challenge:
Find the sample size \( n \) at which BIC begins to penalize the additional parameter more harshly than AIC. Prove this mathematically starting from the penalty terms: \( 2k \) vs \( k\log(n) \).
Interpretive Question:
What are the implications of this crossover point for model selection in Big Data contexts (\( n > 1,000,000 \))? Which criterion is more likely to satisfy the principle of parsimony as \( n \to \infty \)?
3. Quantitative Selection
A researcher fits two models to a dataset with \( n = 50 \):
Model A: \( \log \hat{L} = -120 \), \( k = 4 \)
Model B: \( \log \hat{L} = -115 \), \( k = 12 \)
Calculate AIC for both:
AIC_A: AIC_B:
Calculate BIC for both:
BIC_A: BIC_B:
Final Decision: Justify which model should be selected and why the two criteria might disagree here.
Resampling Architectures Slides Advanced Model Selection • Lesson 3
Resampling Strategies
Architectures for Out-of-Sample Error Estimation
The Validation Imperative
When \( n \) is small, a single train-test split is inefficient. Information criteria (AIC/BIC) rely on asymptotic assumptions.
Resampling methods provide a non-parametric estimate of prediction error by reusing the data.
The Validation Hierarchy
Hold-out Method
becomes
K-Fold Cross-Validation
approaches
Leave-One-Out (LOOCV)
K-Fold Architecture
The Algorithm
Split data into \( K \) equal subsets (folds).
For each fold \( i \in \{1, ..., K\} \):
Train on all folds except fold \( i \).
Calculate error on fold \( i \).
Average the \( K \) error estimates.
VAL
VAL
VAL
Repeat K times...
The Trade-off Within the Method
Leave-One-Out (K=n)
High Variance: The \( n \) training sets are nearly identical; estimates are highly correlated.
Low Bias: Uses nearly all data (\( n-1 \)) for each fit.
K-Fold (K=5 or 10)
Low Variance: More distinct training sets lead to more stable average estimates.
Higher Bias: Training on less data (e.g., \( 0.8n \)) may overestimate error.
When to use what?
Massive Data
Hold-out or small \( K \). Computational cost is the bottleneck.
Standard Regression
10-Fold CV. The industry standard for balancing bias and variance.
Small Samples
LOOCV or K=n. Every data point is precious for the training set.
Resampling Algorithms Activity Resampling Algorithms Activity
Advanced Model Selection • Lesson 3
Group:
Part 1: Algorithmic Logic
In this section, you will logic out the computational steps for different cross-validation architectures. Cross-validation is not just a statistical method; it is a computational loop.
Pseudocode Challenge: K-Fold CV
Write the loop structure for a generic K-Fold CV. Ensure you track the error for each fold and return the average.
# Input: dataset D, model M, integer K
1. Split D into K equal folds F1...FK
Part 2: Analyzing the Tradeoff
Complete the following comparisons for LOOCV vs. 5-Fold CV.
Training Correlation
Which method has training sets that are more correlated with each other? How does this impact the variance of the error estimate?
Training Size Bias
In which method is the training size furthest from the full dataset \( n \)? How does this impact the bias of the error estimate?
Part 3: Stratification Analysis
Scenario: Rare Class Classification
You are building a model to detect a rare genetic mutation that occurs in only 2% of your 1,000 samples. You decide to use 10-Fold Cross-Validation.
Q3.1: Random Splitting Risk
What is the probability that at least one fold contains zero positive cases if you use simple random splitting? Why is this a problem for the validation loop?
Q3.2: Stratification Solution
Define "Stratified K-Fold CV" and explain how it modifies the fold generation step (Step 1 of the algorithm) to solve this issue.
Part 4: Big-O Complexity
Calculate the computational complexity (Big-O) for the following validation tasks:
LOOCV
Training a model with \( n \) samples and \( p \) predictors takes \( O(p^2n + p^3) \).
10-Fold CV
Training the same model on \( 90\% \) of the data 10 times.
Dimension Dilemmas Case Study Dimension Dilemmas: The $p \gg n$ Challenge
Case Study: Genomic Predictors of Treatment Response
Advanced Model Selection • Lesson 4
Ref: GEN-2026-X
The Dataset Context
You are provided with gene expression profiles from 100 oncology patients (\(n=100\)). For each patient, you have measurements for 20,000 genes (\(p=20,000\)). The goal is to select a subset of genes that best predict the quantitative drug response \(Y\). Traditional OLS fails here because \(p > n\), making the matrix \(X^TX\) singular.
Phase 1: Metric Disagreement
During your initial screening, you run three selection algorithms. The results for the "best" model identified by each are as follows:
AIC Optimized
Model Size: \(k=45\)
Adj. R²: 0.88
BIC Optimized
Model Size: \(k=8\)
Adj. R²: 0.62
10-Fold CV Min
Model Size: \(k=22\)
MSE: 12.4
Q1. Theoretical Diagnosis:
Why does BIC select a significantly smaller model than AIC in this high-dimensional setting? Refer specifically to the penalty terms derived in Lesson 2.
Q2. CV vs. Information Criteria:
The 10-Fold CV minimum sits between AIC and BIC. If the true underlying signal depends on 50 genes (mostly weak effects), which metric are you likely to trust and why?
Phase 2: Ridge vs. Lasso Diagnostics
You transition from discrete subset selection to continuous regularization. You apply Lasso (\(L_1\)) and Ridge (\(L_2\)) regression using cross-validation to tune the penalty parameter \(\lambda\).
Observation A:
Lasso sets 19,950 coefficients exactly to zero. The CV error curve is sharp and has a clear global minimum.
Space for Coefficient Path Sketch (L1)
Observation B:
Ridge keeps all 20,000 coefficients non-zero but shrinks them toward zero. The CV error is 15% lower than Lasso's minimum.
Space for Coefficient Path Sketch (L2)
The Diagnostic Challenge
Based on the results above (Ridge outperforming Lasso in CV), what can you infer about the "true" nature of the genomic signal? Is the response driven by a small number of large effects or a large number of small effects? Justify your choice using the geometry of the constraint regions.
High-Dimensional Analysis • Graduate Statistics Workshop • Case GEN-2026-X
High Dimensionality Slides Advanced Model Selection • Lesson 4
Dimension Dilemmas
Model Selection and Regularization in High-Dimensional Space
The Curse of Dimensionality
"In high dimensions, the volume of the space increases so fast that the available data become sparse."
OLS Failure
When \( p > n \), the least squares solution is not unique. \( X^TX \) is singular, meaning there are infinitely many coefficients \( \hat{\beta} \) that give zero training error.
p » n
Statistical Red Zone
Overfitting is guaranteed without regularization.
Standard error estimates become meaningless.
Feature selection is mandatory.
Ridge vs. Lasso: The Geometry of Selection
Lasso (\( L_1 \))
Variable Selection
Uses a diamond-shaped constraint region. It is likely to hit the axes at "corners," setting coefficients exactly to zero.
RSS + \(\lambda \sum_{j=1}^p |\beta_j|\)
Ridge (\( L_2 \))
Shrinkage Only
Uses a circular constraint region. It shrinks coefficients toward zero but almost never exactly to it.
RSS + \(\lambda \sum_{j=1}^p \beta_j^2\)
The Metric Conflict
In high dimensions, AIC and BIC diverge rapidly.
AIC Preference
Tends to select larger models to minimize predictive error. Higher risk of false positives.
BIC Preference
Extremely conservative. May miss small but true effects in favor of simplicity.
The Consistency Debate
If the "true" model is sparse, BIC is consistent as \( n \to \infty \). But if the truth is dense (many small effects), BIC may be too sparse to be useful.
Warning
Standard CV error estimates have high variance in high dimensions. One split is never enough.
Genomic Case Study
We will now apply these concepts to a real genomic dataset (\( p = 20,000, n = 100 \)). You must decide: which regularizer captures the biology best?
Step 1: Metric Selection
Step 2: Lambda Tuning
Step 3: Justification
The Final Choice Slides Advanced Model Selection • Lesson 5
The Final Choice
Synthesizing Conflicting Evidence to Defend the Optimal Model
The Synthesis Framework
Selection is rarely unanimous. In complex datasets, metrics often point in different directions.
Defense requires articulating why a specific trade-off was accepted.
Statistical Fit
Predictive Power
Theoretical Logic
Operational Cost
Conflict Resolution
Scenario: CV error vs. BIC
CV suggests \(k=20\), BIC suggests \(k=5\). Is the predictive gain worth the loss of parsimony?
Scenario: High Variance Predictions
The "best" model has unstable coefficients across folds. Do we shrink or drop features?
Beyond the Numbers: Residual Analysis
The "Smell Test"
A metric-perfect model is invalid if residuals exhibit structure.
Heteroscedasticity: Is the model failing at specific ranges of the target?
Autocorrelation: Did we miss a temporal/spatial dependency?
Influential Points: Is our selection driven by 3 outliers in a 1,000-point set?
Selection Red Flag #1
Minimum CV error occurs at the highest possible complexity. (Check for leakage or overfitting).
Selection Red Flag #2
BIC and AIC disagree by orders of magnitude in model size. (Investigate n vs. signal strength).
Selection Red Flag #3
Model selection changes drastically with a 1% change in data. (Address high variance/instability).
Structure of a Professional Defense
The selected model is not the "truth," it is the most useful approximation.
01
Global Performance
Present the CV error curves and Information Criteria rankings.
02
Sensitivity Analysis
Show how the selection holds up under slight data perturbations.
03
Operational Feasibility
Discuss parameter count vs. interpretability and data requirements.
04
Risk Acknowledgement
State clearly where the model is likely to fail or over-simplify.
The Final Case
Apply these frameworks to the noise-saturated clinical trial dataset. You must defend one model—and one model only—against a board of stakeholders who value different outcomes.
Commence Synthesis
Model Defense Portfolio Guide Model Defense Portfolio
Advanced Model Selection • Culminating Project
Student:
The Final Synthesis
You have been provided with the "Clinical Noise Dataset" (Ref: CLN-2026). Your task is to select and defend a single statistical model. You must address the trade-off between predictive accuracy and model parsimony, justifying why your chosen model is the most robust for future deployment.
Phase 1: Evidence Aggregation
Complete the matrix below for your top three candidate models.
Model Specification AIC / BIC Rank CV Error (Mean ± SD) Parsimony (k) Candidate 1: Candidate 2: Candidate 3:
Phase 2: The Logic of Selection
Primary Justification:
Which metric served as your primary selection criterion? Justify this based on the goals of the "Clinical Noise" project (e.g., were we seeking structural discovery or pure prediction?).
Addressing Conflict:
If BIC and CV-Error disagreed on the optimal model size, how did you resolve this? What specific diagnostics (e.g., residual plots, coefficient stability) broke the tie?
Phase 3: Executive Defense
In the space below, draft a 3-paragraph executive summary defending your "Best Model."
Paragraph 1: Model Performance & Stability
Discuss cross-validation results and error distribution...
Paragraph 2: Information Criteria & Complexity
Discuss AIC/BIC rankings and the parsimony argument...
Paragraph 3: Operational Limitations
Acknowledge where the model might fail and provide recommendations for future data collection...
Capstone Portfolio • Graduate Model Selection Series • Ref CLN-2026
