The Overfitting Trap Worksheet The Overfitting Trap
Analyzing the Bias-Variance Trade-Off
Subject
Advanced Functions / Statistics
Student:
Date:
The "Perfect" Fit Challenge
Below is a dataset representing the growth of a rare plant over 6 days. Your goal: Draw a smooth curve that passes exactly through every single data point.
Day (x)
Height (cm)
Predict Day 7?
Reflection Question:
If your curve hits every point, it must be highly complex (high degree). Does that complexity make you MORE or LESS confident in predicting Day 7? Why?
High Bias (Underfitting)
The model makes simplistic assumptions about the data. It misses key patterns (high error on both training and testing data).
Linear fit on curved data
High Variance (Overfitting)
The model captures "noise" instead of "signal". It performs perfectly on training data but fails on new data.
High-degree polynomial fit
Model Comparison Exercise
1. The Mean Squared Error (MSE) Problem
Imagine we use a 10th-degree polynomial to model the height of a bounce. Our Training MSE is 0.001 (very low), but our Testing MSE on new data is 45.2 (very high).
Analysis:
Why is Training Error low?
Why is Testing Error high?
2. The Complexity Curve
Sketch the general relationship between Model Complexity (x-axis) and Prediction Error (y-axis). Draw two lines: one for Training Error and one for Testing Error.
Total Error
Model Complexity
Legend:
• Training Error (usually goes down)
• Testing Error (U-shaped)
Where on your graph is the "Optimal Model" located? Mark it with an 'X'.
Critical Thinking
In your own words, explain why a model that fits 100% of the data points in a lab experiment might be "less scientific" than one that captures the general trend but ignores small deviations.
Overfit Overkill Slides Overfit Overkill
Mastering the Bias-Variance Trade-Off
12th Grade Math
Lesson 1
The Perfection Paradox
Is a "Perfect" Fit actually perfect?
Scenario
You create a model that hits every single data point in your training set.
The Problem:
You've modeled the Noise , not the Signal .
Real-world data is messy; perfection is often a lie.
The next data point will almost certainly fail.
Total Overfit!
Bias vs. Variance
High Bias
The model is too simple. It "underfits" because it ignores the actual complexity of the data.
Example: Using a Straight Line to model a Parabola.
High Variance
The model is too complex. It "overfits" because it is extremely sensitive to small fluctuations in the training set.
Example: Using a 10th-degree Polynomial for 5 data points.
The "Sweet Spot"
Total Error = Bias² + Variance + Irreducible Noise
Training Error Testing Error
Model Complexity →
Optimal
The Takeaway
Simpler is Safer
Unless the data provides strong evidence for complexity, favor the simpler model (Parsimony).
Split Your Data
Always evaluate on data the model hasn't seen. Training performance is a biased metric.
Generalization
The goal of modeling isn't to explain the past, but to predict the future.
Bias Variance Facilitation Guide Facilitation Guide
Lesson 1: The Bias-Variance Trade-Off
DURATION
60-90 MIN
Learning Objectives
1
Define Bias and Variance in the context of predictive modeling.
2
Differentiate between Training Error and Testing/Generalization Error .
3
Identify signs of Overfitting in high-degree polynomial models.
Materials Needed
Overfit Overkill Slides
The Overfitting Trap Worksheet
Graphing Calculators / Desmos
Instructional Sequence
1. The Hook: The Perfection Paradox (10 min)
Hand out the worksheet immediately. Challenge students to draw the "perfect" curve. Key Question: "If I told you this plant was growing according to a physical law, do you think it would wiggle that much? Or is it just 'messy' data?"
2. Concept Direct: Bias vs. Variance (20 min)
Use the slides to define the trade-off. Use the Dartboard Analogy :
3. Guided Simulation: Desmos Polynomials (25 min)
Have students use Desmos to plot a small dataset of 6 points. Compare a Linear Regression vs. a 5th Degree Polynomial . Ask: "Which one looks more 'realistic'? Which one would you use to predict values between the points?"
4. Independent Analysis (20 min)
Students complete the complexity curve and critical thinking sections of the worksheet. Monitor for the "Training vs. Testing" distinction—ensure they understand that training error always decreases with complexity, but that's not the goal.
Common Misconceptions
"Lower error is always better": Students often think minimizing training error is the finish line. Reinforce that 0 error is usually a sign of a failed (overfit) model.Complexity = Intelligence: Some students assume more complex equations are more "advanced." Clarify that elegance and parsimony are the true goals of modeling.
Differentiation
Support: Provide a pre-drawn graph of a high-degree polynomial and ask them to identify where it "goes wrong" relative to a linear trend. Penalty Points Slides Penalty Points
Measuring Model Efficiency with AIC & BIC
Lesson 2 Model Selection
The Cost of Complexity
Trade-Off Formula
Information Criteria ask one simple question:
"Is the extra accuracy worth the extra complexity?"
In statistics, we want the most accurate model with the fewest parameters.
The Generic Equation
Metric = Goodness-of-Fit + Penalty Term
Lower value = Better model
AIC
Akaike Information Criterion
Formula (SSE-based)
\[ AIC = n \ln\left(\frac{SSE}{n}\right) + 2k \]
n
Number of data points (sample size)
SSE
Sum of Squared Errors (Residuals)
2k
The Penalty.k = number of estimated parameters (coefficients).
AIC favors slightly more complex models compared to BIC.
BIC
Bayesian Information Criterion
Formula (SSE-based)
\[ BIC = n \ln\left(\frac{SSE}{n}\right) + k \ln(n) \]
The Key Difference
Notice the penalty: \( k \ln(n) \). As your sample size (\(n\)) grows, BIC penalizes complexity much harder than AIC.
Select BIC when...
You have a large dataset.
You want a very simple model.
Overfitting is a major concern.
Compare!
Model A: Linear
\( n = 50 \)
\( k = 2 \) (slope + intercept)
\( SSE = 450 \)
Model B: Quadratic
\( n = 50 \)
\( k = 3 \) (\(ax^2 + bx + c\))
\( SSE = 410 \)
Calculation Challenge: Which model has the lower AIC?
AIC_A = 114.9
AIC_B = 112.2
Winner: Model B (Despite the extra penalty!)
Complexity Cost Worksheet Complexity Cost
Calculating Information Criteria (AIC & BIC)
Penalty Level
Log-Likelihood Approximation
Worksheet ID: MAT-12-MOD-02
Analyst:
Date:
Akaike (AIC)
\( n \ln(SSE/n) + 2k \)
Bayesian (BIC)
\( n \ln(SSE/n) + k \ln(n) \)
Task 1: The Simple Comparison
A scientist is modeling the decay of a substance over \( n = 40 \) observations. She tests two models:Model 1: Linear (\( k = 2 \)). \( SSE = 320 \).Model 2: Cubic (\( k = 4 \)). \( SSE = 285 \).
Model 1 Calculations
Calculate \( AIC_1 \):
Calculate \( BIC_1 \):
Model 2 Calculations
Calculate \( AIC_2 \):
Calculate \( BIC_2 \):
Final Selection:
Based on your results, which model should the scientist select? Justify your answer using the values above.
Task 2: The Sensitivity of n
Complete the table below to see how the BIC penalty changes as your sample size (\( n \)) increases for a model with 5 parameters (\( k = 5 \)).
Sample Size (n) AIC Penalty (2k) BIC Penalty (k * ln(n)) n = 10 10 11.51 n = 100 n = 1000 n = 10,000
Reflection Question:
As the amount of data increases, which metric (AIC or BIC) becomes more reluctant to add new variables? Why do you think this is a helpful property for large datasets?
AIC BIC Cheat Sheet AIC vs BIC
The Information Criteria Cheat Sheet
Akaike (AIC)
\( n \ln(SSE/n) + 2k \)
Best For:
Small sample sizes
Predicting new data points
Selecting "good" candidate models
AIC tends to be more "forgiving" of complexity. It focuses on minimizing the distance between the model and the "true" function.
Bayesian (BIC)
\( n \ln(SSE/n) + k \ln(n) \)
Best For:
Large datasets
Identifying the "true" model
When simplicity is the highest priority
BIC is more "strict." As \(n\) grows, the penalty becomes much harsher. It tries to find the most probable "true" model.
Variable Key
n
Sample Size
The total number of observed data points.
k
Parameter Count
Number of coefficients (\( \beta \)) in the model. (e.g., \( y = mx + b \) has \( k = 2 \)).
SSE
Sum of Squared Errors
Measure of fit accuracy. \( \sum(y_i - \hat{y}_i)^2 \).
Model Selection Rule
LOWER IS BETTER
Between two models, always choose the one with the smaller AIC or BIC value. A difference of > 2 is considered statistically significant.
Pro-Tip
R-squared always goes up when you add variables. AIC/BIC can go up OR down because they include that penalty term. This is why we use them for selection and not R-squared!
Example Calculation (n=100, k=3, SSE=200)
AIC:
\( 100 \ln(200/100) + 2(3) \)
\( 100 \ln(2) + 6 \)
\( 69.31 + 6 = \mathbf{75.31} \)
BIC:
\( 100 \ln(200/100) + 3 \ln(100) \)
\( 69.31 + 3(4.605) \)
\( 69.31 + 13.81 = \mathbf{83.12} \)
Fold and Fit Slides Fold & Fit
Empirical Model Selection via K-Fold Cross-Validation
The Taste Test
Why Training Data is a Biased Judge
Imagine a chef cooks a signature dish and tastes it themselves. They love it! (Training Performance)
The Problem:
The chef knows what to expect. They are biased. To know if the food is truly good, we need new customers who didn't watch it being made.
Validation Set
"The customers who haven't tasted the dish yet."
The K-Fold Logic
1. Split
Split your data into k equal-sized groups (folds).
2. Hide
Pick one fold to be the Validation Set . Hide it from the model.
3. Train
Train the model on the remaining k-1 folds.
4. Test
Test the model on the hidden fold. Record the error.
Repeat k times! Every fold gets a turn being the judge.
The Cycle of Validation
Test
Training Data
Test
Test
Test
Test
Step 5
Average the Errors
CV_Error = (E1 + E2 + E3 + E4 + E5) / 5
Why is this Superior?
Efficiency
Every single data point gets to be used for both training and testing exactly once. No data is wasted.
Robustness
It accounts for unlucky data splits. One weird data point won't ruin your error estimate if you average across 5 or 10 folds.
Golden Rule: A model that performs well across ALL folds is truly generalizable.
Validation Shuffle Activity Validation Shuffle
Simulating 3-Fold Cross-Validation
Method
Empirical Error Estimation
Researcher:
Date:
Scenario
We have a dataset of 6 points. We want to estimate the generalization error of a simple Linear Model . Instead of just calculating the error on the training set, we will use 3-Fold Cross-Validation .
Step 1: The Partition
Fold A
(1, 2) | (2, 4)
Fold B
(3, 5) | (4, 8)
Fold C
(5, 9) | (6, 13)
Your Mission:
1. Train the model on 2 folds.
2. Calculate Residual Squared Error on the 3rd (hidden) fold.
3. Repeat for all three combinations.
Round 1
Training: B + C Validation: Fold A
Model trained on B+C: \( \hat{y} = 2.1x - 0.5 \)
Point in Fold A
Predicted \( \hat{y} \)
Squared Error \( (y - \hat{y})^2 \)
(1, 2)
\( 2.1(1)-0.5 = 1.6 \)
\( (2-1.6)^2 = 0.16 \)
(2, 4)
Total Error (E1)
Round 2
Training: A + C Validation: Fold B
Model trained on A+C: \( \hat{y} = 2.2x - 0.2 \)
Point in Fold B
Predicted \( \hat{y} \)
Squared Error
(3, 5)
(4, 8)
Total Error (E2)
Round 3
Training: A + B Validation: Fold C
Model trained on A+B: \( \hat{y} = 2.0x - 0.1 \)
Work area for E3
Total Error (E3)
Final Aggregation
Cross-Validation Error Formula
\[ CV_{MSE} = \frac{E_1 + E_2 + E_3}{3} \]
Calculate final CV Error here...
Analysis:
If the 3 errors were very different (e.g. 0.1, 5.2, 0.4), what would that tell you about the stability of your model?
Competing Theories Slides Competing Theories
Evaluating Non-Nested Model Selection
The Comparison Challenge
Nested Models
One model is just a "simplified" version of the other.
M1: \( y = \beta_0 + \beta_1 x \)
Non-Nested Models
The models have completely different functional forms.
M1: \( y = a e^{bx} \) (Exponential)
How do we choose when models don't "match"?
The Selection Toolkit
AIC / BIC
Quantitative penalties for complexity. Best for comparing across any function type.
Math-Led
Cross-Validation
Empirical testing. Which model predicts the future better regardless of the equation?
Data-Led
First Principles
Does the equation make physical sense? Does it have asymptotes where the real world has limits?
Theory-Led
Case Study: Rumor Spread
The Conflict
Theory A: Viral Growth
"Rumors grow exponentially forever because every person tells 2 more."
\( y = P e^{rt} \)
No Limit
Theory B: Saturation
"Rumors grow until everyone knows. Then the growth stops."
\( y = \frac{L}{1+Ae^{-kt}} \)
Carrying Capacity
Evaluation Question:
If Theory B has a slightly higher AIC but makes more 'logical' sense, which do you pick?
The Final Call
Statistical metrics (AIC/BIC) are tools, not absolute laws. In 12th-grade modeling, you must justify your selection with a blend of:
1. Parsimony
"I chose Model A because it achieved similar accuracy with 3 fewer variables."
2. Generalizability
"Model B was superior in 10-fold Cross Validation, suggesting it will perform better on future data."
The Viral Rumor Case Study The Viral Rumor
Non-Nested Model Selection Case Study
Lead Investigator:
The Incident
"A rumor starts on Day 0 in a high school of 2,000 students. By Day 5, the administration is panicking. Two data scientists have proposed different models for how this rumor will continue to spread. Your job is to decide which theory to trust before the assembly on Day 10."
1
Theory: Viral Cascade
"Rumors spread like a biological virus. The more people who know, the more people they infect. Growth is purely exponential."
Model Specs
\( y = 12.5 e^{0.85t} \)
SSE
412.5
k (params)
2
2
Theory: Logistic Satiation
"Eventually, you run out of people to tell. The growth must slow down as it approaches the total student body population."
Model Specs
\( y = \frac{2000}{1 + 159 e^{-0.8t}} \)
SSE
458.2
k (params)
3
Phase 1: Quantitative Analysis
Calculate the AIC for both models using \( n = 20 \) historical observations. Use the formula: \( AIC = n \ln(SSE/n) + 2k \).
Model 1 AIC
Model 2 AIC
Phase 2: Contextual Justification
The Extrapolation Test
Predict the number of "infected" students for Day 10 for both models. Recall: The school only has 2,000 students.
Model 1 Predict (t=10)
Model 2 Predict (t=10)
The Final Board Meeting
The AIC likely suggests Model 1 is "better" due to lower complexity and slightly lower error on historical data. However, based on your Day 10 prediction and the physical reality of the school, which model would you actually recommend to the principal? Write a 3-sentence justification.
Reference both statistical metrics and theoretical limits in your response.
Data Consultant Lab Brief Data Consultant Lab
Project: Predictive Performance Audit
Client Confidential
Lab Reference: #DS-L5-FINAL
The Brief
You have been hired as a Lead Data Scientist for Nexus Logistics. The company has a dataset of 50 delivery routes and is trying to predict fuel consumption (L) based on cargo weight (kg) . Three different research teams have submitted competing models.
The Ultimate Test:
Your chosen model will be tested against a hidden validation dataset (Day 51-60) at the end of this lab. If your model overfits, your client loses money. Choose wisely.
Candidate Models
Team Alpha: Linear
\( y = 0.45x + 12 \)
Params (k): 2
Train SSE: 850.2
"Keep it simple. Fuel use is proportional to weight."
Team Beta: Quadratic
\( y = 0.02x^2 + 0.1x + 8 \)
Params (k): 3
Train SSE: 620.4
"Engines become less efficient at heavy loads."
Team Gamma: Poly-5
\( y = \sum_{i=0}^5 a_i x^i \)
Params (k): 6
Train SSE: 310.1
"The data is complex. We need to hit every point."
Performance Audit
Model Penalty Term (2k) Fit Term (n * ln(SSE/n)) Total AIC Alpha Beta Gamma
5-Fold Cross-Validation Report
A separate 5-fold CV was run on these models. Model Gamma showed high variance (Error ranged from 120 to 1,500 across folds). Model Beta was the most stable.
Interpret how Stability affects your choice...
The Recommendation
Based on the AIC scores and the CV stability report, which model should Nexus Logistics implement for their nationwide fleet?
TEAM ALPHA (Linear)
TEAM BETA (Quadratic)
TEAM GAMMA (Poly-5)
Technical Brief (Explain Your Logic) Hidden Dataset Key Validation Key
The "Hidden" Truth for Lesson 5
AIC Comparison results (n=50)
Model Alpha
Fit: \( 50 \ln(850.2/50) = 141.67 \)
Penalty: \( 2(2) = 4 \)
Total AIC: 145.67
Model Beta (Winner)
Fit: \( 50 \ln(620.4/50) = 125.88 \)
Penalty: \( 2(3) = 6 \)
Total AIC: 131.88
Model Gamma
Fit: \( 50 \ln(310.1/50) = 91.25 \)
Penalty: \( 2(6) = 12 \)
Total AIC: 103.25
*Note: While Gamma has the lowest AIC, the CV stability report (Variance) should warn students away.
The Hidden Validation Set
At the end of class, reveal these Actual Errors on the hidden dataset (Day 51-60). This proves that the complex model was just "memorizing" noise.
Alpha Performance
MSE = 18.5
(Stable but inaccurate)
Beta Performance
MSE = 14.2
Optimal Balance
Gamma Performance
MSE = 84.9
Massive Overfit!
Grading Criteria: Gamma Selection
Students who select Model Gamma because "it has the lowest AIC" should receive partial credit for calculation, but lose points for Generalization Justification . They ignored the CV stability warning!
Grading Criteria: Beta Selection
Students who select Model Beta should be rewarded for explaining the trade-off: "While Gamma had a lower AIC, its CV variance was too high, making it a risky choice for real-world deployment."
