Complexity Curse Slides Lesson 01 // Model Performance
THE COMPLEXITY CURSE
Why "More Accurate" isn't always "Better" in Mathematical Modeling.
The $100\%$ Prediction Myth
I can create a model that predicts your grade on the next math test with 100% accuracy right now.
The Secret Equation:
Grade = (Age × 2) + (Shoe Size / 5) - (Cereal Eaten × 0.1) + ...
Is this a good model? Why or why not?
WARNING: OVERFITTING
Fitting the noise instead of the signal.
The $R^2$ Illusion
$R^2$ = Coefficient of Determination
How it works
$R^2$ measures what percentage of the variance in the output is explained by the input variables.
0.0 to 1.0
The Fatal Flaw
As you add any variable—no matter how useless—$R^2$ will always stay the same or increase. It never goes down.
If we only look at $R^2$...
We will always choose the most complex model.
Case Study: House Prices
Model A: Simple
Variables: Square Footage
$R^2 = 0.82$
Model B: Moderate
Variables: Sq Ft + # of Bedrooms
$R^2 = 0.85$
Model C: "Nonsense"
Variables: Sq Ft + Bedrooms + Owner's Lucky Number
$R^2 = 0.851$
The Problem of "Nonsense Gains"
Model C has the highest $R^2$. By this metric alone, it is the "best" model.
But we know the "Lucky Number" is irrelevant. It just happened to align slightly with the data in this one sample.
"A model with 100 parameters can fit an elephant. A model with 101 can make his trunk wiggle."
The Solution?
We need a metric that penalizes complexity.
Good Fit
Too Many Variables
The Winner
Coming up in Lesson 2: AIC and BIC
Nonsense Variable Worksheet The Nonsense Variable Experiment
Lesson 01 // Identifying the Limitations of $R^2$
Name:
Date:
The Challenge
In statistics, $R^2$ (the Coefficient of Determination) is often used to measure how well a model fits the data. However, $R^2$ has a significant flaw: it will always stay the same or increase as you add more variables, regardless of whether they actually matter. In this activity, we will analyze data from a study on "Coffee Shop Sales" and see how $R^2$ behaves as we add "Nonsense" variables.
Part 1: The Base Models
Target Variable: Daily Revenue ($)
Model ID Independent Variables Included $R^2$ Score Interpretation Model 1 Daily Temperature (°F) 0.64 Moderately strong relationship. Model 2 Temp + Foot Traffic (Count) 0.82 Significant improvement. Better fit. Model 3 Temp + Foot Traffic + Average Height of Baristas 0.824 Wait... it went up?
1. Does the height of a barista actually influence how much coffee people buy? Explain your reasoning.
2. Why did the $R^2$ score increase from Model 2 to Model 3, even though the new variable is clearly "nonsense"?
Part 2: Visualizing the Trap
Imagine we added 100 more random variables (e.g., "Number of birds seen outside," "Current price of titanium in China," etc.) to Model 3. Below is a sketch of what would happen to $R^2$.
Number of Variables
$R^2$ Score
Approaching 1.0 (Perfect Fit)
Critical Thinking Question
If a model with 100 random variables has an $R^2$ of 0.99, but a model with 2 meaningful variables has an $R^2$ of 0.82, which model would you trust more to predict tomorrow's sales? Why?
The Concept of Degrees of Freedom
Every variable you add uses up a "degree of freedom"—essentially a chance to capture the specific weirdness of the current data sample. If you have enough variables, you can eventually "connect the dots" perfectly for any set of data, but that doesn't mean the resulting equation describes the actual reality of the world. It just describes that specific dataset.
Data Lab Blueprint // Model Performance Lab Series
Complexity Curse Teacher Guide Teacher Guide // L01
The Complexity Curse
Breaking the $R^2$ dependency through inquiry.
Learning Objectives
Explain why $R^2$ is a non-decreasing function of the number of parameters.
Differentiate between "signal" (meaningful data) and "noise" (random variation).
Critique models that prioritize fit over parsimony (simplicity).
Facilitation Guide
The Hook (5-10 mins)
"I have a secret formula for predicting your future. It's 100% accurate on the past 10 years of your life. It involves the temperature on your 5th birthday and the color of the third car you ever saw."
Ask: Why does this feel wrong? (Students usually identify that these factors aren't causal). Transition to: "In math, our calculators don't know 'why'. They only know 'how close'. Today we find the danger in that."
The Investigation (20 mins)
Distribute Nonsense Variable Worksheet. Have students work in pairs. Monitor responses to Question 2.
Key Misconception: Students often think $R^2$ only increases if the variable is "good". Emphasize that adding a column of random numbers will likely result in a non-zero (even if tiny) correlation, which mathematically must add to the total explained variance.
The Deep Dive (15 mins)
Use the Slides to visualize Overfitting. Compare a model that follows every zig-zag of data points (overfit) vs. a smooth line that captures the trend (robust).
Key Concepts
Overfitting
Modeling random noise rather than the underlying process.
Parsimony
The principle that the simplest explanation (or model) is usually best.
Noise vs Signal
Signal is the truth; noise is the measurement error and randomness.
Discussion Prompts
"Is a 99% accuracy rate always better than an 80% accuracy rate?"
"If we can't trust $R^2$ to select the best model, what should we look for instead?"
"How does a 'nonsense' variable steal credit in a mathematical formula?"
Information Tax Slides Lesson 02 // The Information Tax
AIC & BIC DECODED
Learning to pay the "Complexity Tax" for more robust models.
The Model Performance Equation
Generic Information Criterion =
BADNESS OF FIT
(Mismatch with data)
COMPLEXITY PENALTY
("Tax" per variable)
GOAL: MINIMIZE the total score!
AIC
Akaike Information Criterion
The Formula (Simplified):
\[ AIC = n \ln\left(\frac{RSS}{n}\right) + 2k \]
n Sample size (Number of data points)
k Number of parameters (Variables)
RSS Residual Sum of Squares (Error)
AIC Characteristics
Penalizes each variable by a fixed amount ($2k$).
Focuses on predictive accuracy.
More "lenient" with complex models than BIC.
Best when you want the model that predicts best on new data.
BIC
Bayesian Information Criterion
The Formula (Simplified):
\[ BIC = n \ln\left(\frac{RSS}{n}\right) + k \ln(n) \]
PENALTY: $k \ln(n)$
Note how the penalty grows larger as the sample size ($n$) increases!
BIC Characteristics
Much stricter than AIC.
Penalizes complexity more heavily as you have more data.
Favors parsimony (simplicity).
Best when you want to find the "true" underlying model.
Which one to choose?
AIC
Choose AIC if your primary goal is prediction . It's okay with a slightly more complex model if it means better accuracy on future data.
The Optimist
BIC
Choose BIC if your primary goal is explanation . It forces you to justify every variable you add with substantial evidence.
The Skeptic
Complexity Tax Calculator Worksheet The Complexity Tax Calculator
Lesson 02 // Calculating Information Criteria
Lab Tech:
System Date:
Standard Operating Formulas
Akaike Information Criterion (AIC)
\( AIC = n \ln(\frac{RSS}{n}) + 2k \)
Bayesian Information Criterion (BIC)
\( BIC = n \ln(\frac{RSS}{n}) + k \ln(n) \)
n = Sample Size k = Number of Parameters RSS = Residual Sum of Squares (Error)
1
Scenario: Yield Prediction
A scientist is predicting crop yield based on rainfall and soil nutrients. She has a dataset of n = 100 observations.
Model A (Simple)
Parameters: k = 2
Error: RSS = 450
Model B (Complex)
Parameters: k = 8
Error: RSS = 410
Calculate scores for Model A
AIC:
BIC:
Calculate scores for Model B
AIC:
BIC:
2
Analysis & Selection
Based on your calculations above, which model would AIC select? Which model would BIC select? Why do they differ?
Expert Insight
Remember: BIC has a "variable tax" of \( \ln(n) \). When \( n = 100 \), \( \ln(100) \approx 4.6 \). Compare this to the AIC fixed tax of 2. In this case, BIC is penalizing each variable more than twice as much as AIC!
Tool: TI-84 ln() function required Model Performance Series // Module 2
Complexity Tax Answer Key Official Key
Answer Key
Material // Complexity Tax Calculator
Scenario 1: Yield Prediction
Model A (Simple)
AIC Calculation:
\( 100 \cdot \ln(450/100) + 2(2) \)
\( 100 \cdot 1.504 + 4 \)
154.40
BIC Calculation:
\( 100 \cdot \ln(4.5) + 2 \cdot \ln(100) \)
\( 150.40 + 2 \cdot 4.605 \)
159.61
Model B (Complex)
AIC Calculation:
\( 100 \cdot \ln(410/100) + 2(8) \)
\( 100 \cdot 1.411 + 16 \)
157.10
BIC Calculation:
\( 100 \cdot \ln(4.1) + 8 \cdot \ln(100) \)
\( 141.10 + 8 \cdot 4.605 \)
177.94
Selection Analysis
AIC Selection: Model A (154.40 is lower than 157.10).
BIC Selection: Model A (159.61 is lower than 177.94).
Why they differ:
AIC and BIC both selected Model A here, but BIC selected it with a much larger "margin of victory". If Model B's RSS had been slightly lower (e.g., 395), AIC might have switched to Model B, while BIC would likely still have stayed with Model A.
Selection Showdown Slides Lesson 03 // The Selection Showdown
RANKING MODELS
Navigating scenarios where AIC and BIC disagree.
3
1
2
The "Relative" Nature of IC
AIC and BIC scores mean nothing on their own.
A score of 1,240 is not "bad" and a score of 12.4 is not "good."
They are only useful when compared to other models using the same dataset.
Rule 1
Lower is always better.
Rule 2
Only compare models trained on the exact same data.
Rule 3
If differences are small (< 2), models are effectively equal.
The Disagreement Zone
Sometimes, AIC will choose Model X (more complex), while BIC will choose Model Y (simpler).
Why?
BIC penalizes complexity more heavily as the sample size $n$ grows. AIC remains fixed in its penalty approach.
What to do?
Ask yourself:
Prediction vs. Parsimony?
Case: Stock Price Prediction
Candidate Model Parameters ($k$) AIC Score BIC Score Model 1: Linear 2 452.1 456.4 Model 2: Quadratic 3 448.2 454.8 Model 3: Cubic 4 449.8 458.2
AIC Winner: Model 2
BIC Winner: Model 1
Showdown Protocol
01
Calculate both AIC and BIC.
02
Identify the minimum score for each.
03
Defend your choice based on project goal.
Model Selection Matrix Worksheet Model Selection Matrix
Activity 03 // Comparative Ranking & Analysis
Assigned Investigator
Mission Brief
Your tech startup is developing a prediction model for app engagement. We have tested five different models using the same dataset of n = 250 users. Your job is to rank these models and recommend the one we should put into production.
Model ID Parameters (k) AIC Score BIC Score M1: User Age only 2 1,450.2 1,457.1 M2: Age + Phone Type 3 1,422.5 1,433.0 M3: Age + Phone + Region 5 1,418.8 1,436.2 M4: All Features (Complex) 12 1,417.2 1,459.5 M5: Age + Session Time 3 1,421.1 1,431.6
AIC Ranking (Top 3)
1
Model ID #
2
3
BIC Ranking (Top 3)
1
Model ID #
2
3
The Winning Recommendation
Look at Model 4 . It has the absolute lowest AIC score, but the second worst BIC score. Why is there such a massive disagreement for this specific model? Which model should we actually use for production?
Your Defense:
Data Lab Blueprint // Model Performance Lab Series // Activity 03
Model Selection Answer Key Verified Key
Answer Key
Material // Model Selection Matrix
AIC Ranking
1
Model 4
Score: 1,417.2
2
Model 3
Score: 1,418.8
3
Model 5
Score: 1,421.1
BIC Ranking
1
Model 5
Score: 1,431.6
2
Model 2
Score: 1,433.0
3
Model 3
Score: 1,436.2
Master Analysis
The M4 Disagreement: Model 4 has many parameters (k=12). AIC only penalizes it 24 points ($2 \times 12$). However, BIC penalizes it significantly more based on $n=250$ ($\ln(250) \approx 5.5$). BIC's penalty for M4 is approximately 66 points. This extra 42-point "tax" is why BIC ranks it nearly last.
The Recommendation: Model 5 is likely the winner. It is in the Top 3 for both metrics and is the absolute #1 for BIC. It offers the best balance of simplicity (k=3) and performance.
Note: M2 and M5 are extremely close in AIC score. A difference of 1.4 suggests they are statistically indistinguishable, but M5 has a better BIC score, making it the safer choice.
Razors Edge Slides Lesson 04 // The Razor's Edge
PARSIMONY
Why Simplicity is a Mathematical Superpower
Occam's Razor
"Pluralitas non est ponenda sine necessitate."
(Entities should not be multiplied beyond necessity.)
In math, if two models explain the data equally well, we choose the simpler one.
TRUTH LIES IN SIMPLICITY
The 2008 Housing Crisis Model
Before 2008, financial institutions used incredibly complex models to predict mortgage defaults. These models had hundreds of variables .
They had near-perfect $R^2$ scores on historical data.
They looked "too smart to fail."
They captured every tiny ripple in the market's past.
The Failure
Because the models were too complex, they were "brittle." When the market changed slightly, the models didn't just bend—they broke.
Simpler models actually predicted the crash better than the "Genius" models.
Parsimony's 3 Strengths
Generalizability
Simpler models work better on new, unseen data because they haven't memorized the "flukes" of the old data.
Explainability
You can explain a 3-variable model to a client or boss. You can't explain a 300-variable black box.
Robustness
Simple models are harder to "break" with outliers or small measurement errors.
The Challenge
"Is the goal of math to create a perfect map of what happened yesterday...
...or a useful guide for what will happen tomorrow?"
Overfit Disaster Case Study Case Study // The Overfit Disaster
Doc ID: CS-04-PARSIMONY
Subject: The LTCM Collapse (1998)
"When Genius Failed"
Long-Term Capital Management (LTCM) was a hedge fund led by Nobel Prize-winning economists and PhD mathematicians. They developed a model for trading bonds that was incredibly complex, using advanced variables that the rest of the market didn't even understand.
Their models were mathematically "perfect" when applied to the previous 20 years of data. They had found the exact coefficients to explain every historical market movement. Based on these models, the probability of the fund ever going bankrupt was calculated to be once in every trillion years .
The Reality:
In 1998, just four years after starting, the fund lost $4.6 billion in less than four months and required a massive bailout by the Federal Reserve to prevent a global financial collapse.
1. Identification of the Error
The LTCM models included hundreds of variables that captured specific historical "glitches" in the market. Why would capturing these tiny historical details make the model less effective at predicting the future?
2. The BIC Perspective
If LTCM had used BIC (Bayesian Information Criterion) to evaluate their models instead of just looking at historical fit, what would the "Complexity Tax" ($k \ln n$) have likely told them about their hundred-variable model?
3. Philosophical Application
Occam's Razor suggests that "entities should not be multiplied beyond necessity." In the context of the LTCM disaster, how do we define "necessity" when building a model for the stock market?
Model Performance Lab // L04 Case Study // Page 01 of 01
Ultimate Fit Challenge Worksheet The Ultimate Fit
FINAL MODEL SELECTION CHALLENGE
Time Limit
60 MIN
ANALYST:
DATE:
1
The Briefing
"Our logistics company is trying to predict fuel consumption for our fleet of 500 trucks (n=500 ). Our Senior Engineer wants to use Model Delta because it has the highest accuracy ever recorded in our history. However, the CEO is worried the model is too complex and won't work in the real world. Your job: Review the data, calculate the metrics, and make a final recommendation."
2
The Lab Data
Model ID Parameters (k) RSS (Error) AIC Score BIC Score Model Alpha 3 1,250 Model Beta 6 1,180 Model Gamma 10 1,155 Model Delta 45 1,020
Note: Use formulas \( AIC = n \ln(RSS/n) + 2k \) and \( BIC = n \ln(RSS/n) + k \ln(n) \). Use ln(500) ≈ 6.21
3
The Executive Defense
1. The Verdict:
Identify which model is the "best" according to AIC and which is "best" according to BIC.
2. The Rebuttal:
Write a brief defense explaining why Model Delta is a dangerous choice for the business, despite having the lowest error rate.
3. Final Recommendation:
"If you were the CEO, which model would you put into production tomorrow morning? Defend your choice using the concept of parsimony."
Ultimate Fit Rubric Key Assessment Rubric
Ultimate Fit Performance Challenge
Criteria: Calculation Accuracy / 10 Points
<table class="w-full text-sm"><tbody><tr class="border-b border-slate-100"><td class="p-4 w-1/4 font-bold bg-slate-50">Exceeds (9-10)</td><td class="p-4">All 8 AIC/BIC scores calculated with perfect accuracy. Natural logs handled correctly.</td></tr><tr class="border-b border-slate-100"><td class="p-4 w-1/4 font-bold bg-slate-50">Meets (7-8)</td><td class="p-4">Minor calculation errors in 1-2 scores, but the process and substitution into formulas were correct.</td></tr><tr><td class="p-4 w-1/4 font-bold bg-slate-50">Approaching (1-6)</td><td class="p-4">Significant errors in formula application or log rules. Several scores incorrect or missing.</td></tr></tbody></table>
Criteria: Interpretation & Selection / 10 Points
<table class="w-full text-sm"><tbody><tr class="border-b border-slate-100"><td class="p-4 w-1/4 font-bold bg-slate-50">Exceeds (9-10)</td><td class="p-4">Correctly identifies the "winners" for both metrics. Explains the "disagreement" between AIC/BIC for Model Delta with technical precision.</td></tr><tr class="border-b border-slate-100"><td class="p-4 w-1/4 font-bold bg-slate-50">Meets (7-8)</td><td class="p-4">Identifies winners correctly. Provides a general explanation of why complexity hurts a model's score.</td></tr><tr><td class="p-4 w-1/4 font-bold bg-slate-50">Approaching (1-6)</td><td class="p-4">Struggles to identify the lowest scores as winners. Misinterprets the relationship between error and penalty.</td></tr></tbody></table>
Criteria: Parsimony Defense / 10 Points
<table class="w-full text-sm"><tbody><tr class="border-b border-slate-100"><td class="p-4 w-1/4 font-bold bg-slate-50">Exceeds (9-10)</td><td class="p-4">Persuasive argument using concepts like "Overfitting," "Signal vs Noise," and "Robustness." Uses specific numbers from the table to support the choice.</td></tr><tr class="border-b border-slate-100"><td class="p-4 w-1/4 font-bold bg-slate-50">Meets (7-8)</td><td class="p-4">Clear recommendation made. Mentions parsimony or simplicity but may lack deep technical connections to the logistics context.</td></tr><tr><td class="p-4 w-1/4 font-bold bg-slate-50">Approaching (1-6)</td><td class="p-4">Vague recommendation. Argument relies on "gut feeling" rather than mathematical evidence from the Information Criteria.</td></tr></tbody></table>
Quick Solution Key
Model Alpha (k=3):
AIC: 461.35 // BIC: 473.99
Model Beta (k=6):
AIC: 442.20 // BIC: 467.49
Model Gamma (k=10):
AIC: 439.52 // BIC: 481.68
Model Delta (k=45):
AIC: 446.50 // BIC: 635.95
Winner: Model Beta is the most balanced choice for BIC, while Gamma wins AIC. Delta is clearly overfitted.
