The Split Strategy Slides Module 01: Data Partitioning
The Split
Strategy
How can we trust a model to predict the future if it already knows the answers?
11th Grade Mathematics: Functions & Modeling
The "Exam" Analogy
Scenario A
Your teacher gives you the exact same problems for the homework and the final exam.
Result: Memorization
Scenario B
Your teacher gives you practice problems to learn from, but the exam uses new, similar problems.
Result: Real Learning
"If a model is tested on the same data it was trained on, it isn't predicting—it's just remembering."
The Core Concept
Training Set (80%)
Testing Set (20%)
Training Data
Used to build the model. The model sees these data points to find patterns and determine coefficients.
Testing Data
The "Unseen" data. Held back until the model is finished. Used to verify if the model actually works.
The Trap: Overfitting
Overfitting occurs when a model follows the "noise" or random fluctuations in the data rather than the underlying trend.
High accuracy on Training data.
Poor accuracy on Testing data.
Visualizing "The Wiggle"
(Model hugging points too closely)
The Golden Rules of Splitting
01
Randomize
Don't just take the first 80%. Randomly shuffle the data so the test set is representative.
02
No Peeking
The model must never see the test data during training. No exceptions!
03
Ratio Matters
80/20 is standard. 70/30 or 75/25 also work. You need enough data to learn AND enough to verify.
Next Step: Manual Data Partitioning Challenge
The Split Challenge Worksheet The Split Challenge
Module 01: Data Partitioning Worksheet
Name:
Date:
Context: Urban Gardening
An urban farm is trying to model the relationship between Light Hours (x) and Tomato Yield (y) in kilograms. To build a trustworthy model, we must split our 10 data observations into a Training Set (80%) and a Testing Set (20%) .
ID
Light (hrs)
Yield (kg)
ID
Light (hrs)
A
4.2
1.5
F
5.1
B
6.5
3.2
G
8.8
C
3.1
0.8
H
7.2
D
9.5
4.8
I
2.5
E
10.2
5.1
J
5.5
2.4
4.3
3.6
0.6
2.6
1. The Selection Strategy
If you simply take the first 8 IDs (A-H) for training, what bias might you introduce? Look closely at the "Light (hrs)" values in order.
2. Execute the Split
Choose 2 IDs to be your Testing Set . Try to pick values that are spread across the range of the data (not just the extremes).
Testing IDs (Pick 2):
Explain why you chose these two IDs specifically:
Training IDs (List remaining 8):
3. The Generalization Test
Imagine you built a model \( y = 0.5x - 0.2 \) using only your Training Set . Use that model to predict the yield for your Testing Set IDs below.
Test ID Light Hours (x) Model Prediction (y) Actual Yield
Self-Reflection:
How much "trust" would you place in this model if the prediction and the actual yield were off by more than 1.0 kg? Why?
Split Strategy Teacher Guide Teacher Guidance
Lesson 01: The Train-Test Split Method
Objective
Students will understand the mathematical necessity of separating data into training and testing sets to evaluate model reliability and prevent overfitting.
Pacing
10m: The "Exam" Hook & Discussion15m: Core Concepts Presentation20m: Split Challenge Worksheet10m: Debrief & Discussion
The Hook: Discussion Prompts
"Imagine you spend 4 years in high school only ever seeing the exact same 100 questions on every quiz. Then, for the SAT, they give you those same 100 questions. Did you learn math, or did you learn a list of answers?"
Guide students to realize that modeling is learning . If the model sees the test data, it's just looking up the answer key.
Common Misconceptions
"More data is always better"
Students often want to use 100% of data for training. Explain that 100% training results in 0% ability to prove it works.
Non-random splits
If data is sorted by time or size, taking the "last 20%" might mean your test set is fundamentally different from the training set.
Worksheet Key Points
Part 1: Students should notice that the IDs A-E generally have higher values than I-J. A sequential split would bias the model toward "large" or "small" values.Part 2: Ideal test IDs would be something like B (6.5) and H (7.2) or a spread like C (3.1) and G (8.8) .Part 3: Ensure students apply the provided linear function correctly to their chosen 'x' values. The goal isn't the right answer, but seeing how the error manifests.
MOD01 // TRAIN-TEST-SPLIT TECHNICAL WORKSHOP SERIES 2026
The Error Penalty Slides Module 02: Performance Metrics
The Error
Penalty
Quantifying mistakes: How much does "off by a little" matter compared to "off by a lot"?
Mean Squared Error & Root Mean Squared Error
Penalty Kick Analysis
Model A is off by 2 units five times.
Model B is off by 10 units once, but perfect the other four times.
The Question
Which model is worse? Do we prefer many small mistakes or one catastrophic failure?
MSE is the math that decides.
Defining the Metric
Mean Squared Error (MSE)
\[ MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \]
Take the residuals (Actual - Predicted), square them, and find the average .
Why Square?
It turns negative errors positive (so they don't cancel out) and heavily penalizes large errors.
Why Mean?
Gives us the average "cost" of a prediction across the whole dataset.
Making it Real: RMSE
If we are predicting dollars, the MSE unit is squared dollars.
The Solution: RMSE
\[ RMSE = \sqrt{MSE} \]
Returns the error to the original units (e.g., dollars), making it easier for humans to interpret.
The Interpretive Scale
Lower is always better.
0 means a perfect fit.
RMSE tells you: "On average, my model is off by X units."
Calculation Steps
01
Subtract
Find difference between actual and predicted.
02
Square
Raise each difference to the power of 2.
03
Average
Add them up and divide by the count.
04
Root
Square root the result for interpretability.
GET READY FOR THE CALCULATOR GAUNTLET
The MSE Mastery Worksheet The MSE Gauntlet
Module 02: Calculation Workshop
Operator:
Timestamp:
Below is a dataset for a model predicting Energy Consumption (kW) based on Temperature (Celsius) . You must calculate the error for two different models to determine which one is statistically superior.
1
Model A: Linear Approach
Temp (x) Actual (y) Pred (ŷ) Residual (y - ŷ) Squared Error 15 40 38 20 52 55 25 75 72 30 98 89
Sum of Squared Errors (SSE)
Final MSE for Model A
2
Model B: Power Curve Approach
Model B has already been calculated for you: Sum of Squares = 112
Comparative Analysis
1. Which model has the lower MSE? (Show your comparison work below)
2. Calculate the RMSE for your chosen "Winner" model. Why is this number more useful than the MSE?
3
The Quadratic Penalty
"Model A missed the 30°C reading by 9 units. Explain how the squaring process in MSE 'punished' this specific prediction more than a simple average would have."
The MSE Answer Key Answer Key
The MSE Gauntlet // Teacher Reference
Confidential
1 Model A: Calculation Verification
Temp Actual Pred Residual (y - ŷ) Sq. Error 15 40 38 2 4 20 52 55 -3 9 25 75 72 3 9 30 98 89 9 81 Sum of Squares (SSE) 103
Final MSE (Model A): 25.75
Formula: \( 103 \div 4 = 25.75 \)
2 Comparative Analysis Results
Winner: Model A
Model A MSE = 25.75 112 / 4 = 28.0
RMSE Calculation:
\( \sqrt{25.75} \approx \mathbf{5.074} \)Interpretation: On average, Model A's predictions are off by about 5.07 kW. This is easier to understand than "25.75 squared kilowatts."
3 The Quadratic Penalty Guide
"Look for students to identify that squaring makes large errors disproportionately impactful."
An error of 1 becomes 1 (\( 1^2 \))
An error of 3 becomes 9 (\( 3^2 \)) — triple the error, 9x the penalty
An error of 9 becomes 81 (\( 9^2 \)) — 9x the error, 81x the penalty
This ensures the model prioritizes avoiding "big misses" over being perfectly aligned with every minor data point.
The Folding Cycle Slides Module 03: Advanced Validation
The Folding
Cycle
Don't leave any data behind. Using every point for training and testing.
K-Fold Cross-Validation Mechanics
The "Losing Lottery" Risk
What if your random 20% Testing Set happened to contain only the hardest cases or only extreme outliers ?
The Danger
Your error metric (MSE) would be artificially high, making you think a great model is actually terrible.
"A single split is a snapshot. Cross-validation is a panoramic video."
The K-Fold Solution
Divide your data into k equal groups (folds). Rotate which group is the "Test" set.
TEST
TRAIN
TRAIN
TRAIN
TRAIN
TRAIN
TEST
TRAIN
TRAIN
TRAIN
... Repeat until every fold has been the test set ...
The Metric: CV-Score
The Average Wins
You calculate the MSE for each iteration. Your final performance score is the AVERAGE of all these MSEs.
Avg MSE = \(\frac{\sum MSE_k}{k}\)
Stability
Reduces the impact of a weird testing split. Gives a more reliable estimate of real-world error.
Confidence
If all k-folds have similar MSEs, your model is stable. If they vary wildly, your model is erratic.
Industry Standards
K=5
Most common for medium datasets. Balances computation time with statistical reliability.
K=10
The "Gold Standard". Provides high confidence, though requires 10 separate training cycles.
Next: The Folding Simulation Challenge
The Folding Logic Activity The Folding Simulation
Module 03: Iterative Validation Logic
Team:
Date:
Dataset: The 10-Point Survey
We have 10 data points labeled P1 through P10 . We are performing K=5 Cross-Validation . This means we must divide the 10 points into 5 equal folds (2 points per fold).
Fold 1
P1, P2
Fold 2
P3, P4
Fold 3
P5, P6
Fold 4
P7, P8
Fold 5
P9, P10
Part 1: The Validation Cycle
For each iteration, identify which fold is used for Testing and which are used for Training .
Iteration Testing Fold Training Folds (Remaining) Total Points Trained 01 Fold 1 2, 3, 4, 5 8 02 Fold 2 8 03 8 04 8 05 8
Part 2: Calculating the CV-Score
Your data science team ran the simulation and returned the MSE for each of the 5 iterations. Use this data to calculate the final Cross-Validation Score (Average MSE) .
Iteration 1 MSE: 14.2
Iteration 2 MSE: 12.8
Iteration 3 MSE: 22.4
Iteration 4 MSE: 13.5
Iteration 5 MSE: 15.1
Final CV-Score Calculation
MSE = ________
Show the sum of MSEs divided by k=5 below:
The Stability Check
Iteration 3 had a significantly higher MSE (22.4) than the others. If you had only done a single train-test split and gotten Fold 3 as your test set, what wrong conclusion might you have drawn about the model?
The Validation Reference Sheet Cross-Validation Reference
Protocol & Step-by-Step Procedure
Terminology
Fold (k): A subset of data of equal size used for training or testing.Iteration: One complete cycle of training the model and testing it on a specific fold.Out-of-Sample Error: The error (MSE) measured on the testing fold (unseen data).CV-Score: The average error metric across all k-iterations.
Why use it?
Maximizes data usage (every point is used for testing exactly once).
Reduces bias from "lucky" or "unlucky" random splits.
Evaluates the stability and reliability of the model structure.
The K-Fold Protocol
1
Shuffle
Randomly order all data points.
2
Partition
Split data into k equal folds.
3
Loop
Train k times, holding out one fold each time.
4
Average
Calculate the mean of all iteration MSEs.
How to Interpret Results
High Average MSE
The model structure is likely a poor fit for the data (underfitting) or the features are not predictive.
High Variance (Inconsistency)
If MSEs vary widely between iterations, your model is likely overfitting to specific clusters of data.
Data Science Toolbelt // Version 1.0.4 // K-Fold Cross-Validation Standard
Model Faceoff Slides Module 04: Model Comparison
Model
Faceoff
The battle between the simple and the complex. Which structure truly predicts the future?
Linear vs. Non-Linear Validation
The Spectrum of Fit
Underfit
Too simple. Misses the trend in both training and testing.
Straight line / Curved data
Ideal
Robust
Captures the trend but ignores the noise. Consistent across all folds.
"The Sweet Spot"
Overfit
Too complex. Perfect on training, but crashes on test data.
"Zig-Zag Wiggle"
The Battle of Structures
We are modeling Ice Cream Sales based on Temperature .
Candidate 1: Linear
Sales = 5.2(Temp) + 10
Candidate 2: Quadratic
Sales = -0.8(Temp)² + 40(Temp) - 150
The Validation Verdict
Quadratic wins if CV-Score is lower.
Linear wins if it peaks later than the data range.
Never trust a "High R²" on the training set. Trust the Average Test MSE.
The Principle of Parsimony
"If two models have similar cross-validation scores,choose the simpler one. "
Ease of Use
Simpler models are easier to explain to non-technical stakeholders.
Robustness
Simple models are less likely to break when presented with extreme outlier data.
The Model Selector's Oath
01
I will not judge a model by its Training R-Squared.
02
I will trust the Average Test MSE from K-Fold Validation.
03
I will favor simplicity when error scores are tied.
Faceoff Analysis Begins Now
The Comparative Audit Worksheet The Comparative Audit
Module 04: Linear vs. Quadratic Duel
Auditor:
Status:
Scenario: Smartphone Demand
A tech company is modeling demand for their new flagship phone. They are comparing a Linear Model (Model A) and a Quadratic Model (Model B). High Training accuracy has made the board lean toward Model B, but you must audit the Cross-Validation results.
Model A (Linear)
Demand increases steadily over time.
CV-Score (Avg MSE): 245.2
Model B (Quadratic)
Demand peaks at month 6 then drops.
CV-Score (Avg MSE): 210.8
1. Stability Audit
Look at the individual MSEs for each fold of Model B . Does this look like a stable model? Why or why not?
Fold 1
120.4
Fold 2
115.2
Fold 3
450.8
Fold 4
128.5
Fold 5
239.1
Analyze the variance across these folds...
2. The Executive Recommendation
Decision Logic
Even though Model B has a lower average MSE (210.8 vs 245.2), it failed significantly on Fold 3. This indicates it might be overfitting to specific seasonal trends.
Your Recommendation:
Recommend Model A (Linear)
Recommend Model B (Quadratic)
Justify your choice based on "Reliability" and "Parsimony":
3. Error Visualizer
Sketch what the residual plot (Error vs. Time) might look like for a model that is Underfitting vs Overfitting .
Underfit Sketch
Overfit Sketch
The Model Debate Guide Debrief Guide
Lesson 04: The Faceoff Discussion
Phase 1: The Complexity Trap
Ask: "If Model B had a 100% R-squared on the training data, why did it perform so poorly on Fold 3?"
Key takeaway: Training accuracy measures history . Cross-validation measures future potential .
Phase 2: Occam's Razor
Ask: "Imagine the average MSE for both models was exactly 215. Why would the CEO still prefer the Linear model?"
Fewer variables = Fewer things to break.
Easier to communicate "Growth of $X per month" vs "Growth of $X times the square of the month plus..."
Less sensitive to small changes in initial data.
The Trust Question
"Can we trust a model that is perfect on the past but fails on even a single slice of the future?"
The Skeptic's View
If it fails once (Fold 3), it might fail at the exact moment we need it most (e.g., a market crash).
The Optimizer's View
Lower average error overall means we save more money on average, even if we lose sometimes.
Guide students to realize there is no "correct" answer—only a justified one.
Mod04 // Discussion Script
The Selection Challenge Packet Selection
Challenge
Capstone Assessment Packet
Project Lead:
Deadline:
The Mission: Credit Risk Analysis
Apex Bank is launching a new micro-loan product. They have three candidate models to predict Default Probability based on Savings Balance . Your task is to use K-Fold Cross-Validation (K=4) to determine which model the bank should deploy.
Data Integrity: Verified // Security Level: 5
The Candidate Profiles
Model Alpha
Linear Regression
"Simple and consistent. Expects a straight-line drop in risk as savings increase."
Model Beta
Quadratic Curve
"Complex. Suggests risk drops fast at first, then levels off at high savings."
Model Gamma
High-Degree Poly
"Hyper-detailed. Perfectly tracks every minor variation in the pilot data."
Validation Report: Raw MSE Scores
Below are the MSE results from a 4-fold cross-validation experiment for each model.
Iteration Alpha (MSE) Beta (MSE) Gamma (MSE) Iter 1 0.082 0.041 0.002 Iter 2 0.079 0.045 0.850 Iter 3 0.085 0.038 0.005 Iter 4 0.081 0.042 0.910 AVG Score
Submission Requirements
Calculate the average CV-Score (Mean MSE) for all three models.
Identify which model is Underfitting and which is Overfitting .
Select the winning model and justify your choice in the Model Decision Report .
The Model Decision Report Model Decision Report
Apex Bank Risk Management Division
1. Summary of Performance Metrics
Model Alpha CV-Score
Model Beta CV-Score
Model Gamma CV-Score
2. Fit Characterization
Which model exhibits Underfitting? Explain why.
Identify the simple model with high consistent error...
Which model exhibits Overfitting? Explain why.
Identify the erratic model with low training but high test error...
3. Final Selection & Technical Justification
Recommended Model for Deployment:
Justify your recommendation using the following criteria: (1) Predictive Stability, (2) Error Minimization, and (3) Risk Assessment.
Consider the average MSE vs the variance of individual folds...
Why is a consistent model better for a bank's security?
How does the principle of parsimony apply here?
Chief Data Scientist Signature
The Selection Challenge Rubric Mastery Rubric
Model Selection Challenge
Scorecard // 2026
Criteria Advanced (4) Proficient (3) Basic (2-1) CV-Score Calculation All average MSEs are calculated perfectly for Alpha, Beta, and Gamma. Most averages are correct; minor arithmetic errors present. Significant errors in calculating means across folds. Structural Analysis Clearly identifies Alpha as underfit and Gamma as overfit with specific fold-level evidence. Correctly identifies overfitting/underfitting but evidence is vague. Unable to distinguish between overfit and underfit models. Justification Uses "Stability" and "Parsimony" concepts to explain why Model Beta is the optimal choice despite errors. Selects Model Beta but focuses only on the average score rather than stability. Recommendation is based on intuition rather than data metrics. Report Precision Report is formatted professionally; technical language is used correctly throughout. Report is complete and legible; most technical terms are used correctly. Report is disorganized or missing key sections.
The Verdict Key
15-16 pts: Lead Data Scientist12-14 pts: Junior Model Auditor8-11 pts: Technical Apprentice< 8 pts: Revision Required
Evaluator Comments
Academic Standards // Validating Model Accuracy // Final Module
