Selection Strategy Slides Selection Strategy
Comparative Analysis Capstone | Lesson 1
The Metric Trap
You have a dataset with 99% class imbalance and significant outliers.
"Standard accuracy will lie to you. RMSE will be dominated by a few points. How do you design a strategy that won't fail in production?"
Consider:
Cost of False Positives vs. Negatives
Scale of errors (Squared vs. Absolute)
Validation set leakage risks
Beyond Simple CV
Stratified K-Fold
Preserves class distributions across folds. Essential for imbalanced data.
Nested Cross-Validation
Inner loop for hyperparameter tuning; Outer loop for performance estimation. Prevents "tuning leakage."
Nested CV Structure
Ensures model selection is independent of final evaluation.
Choosing Your Compass
Metric Best For... The "Gotcha" RMSE Heavy-tailed errors where big misses are catastrophic. Extremely sensitive to single outliers. MAE Robustness; when all errors scale linearly. Not differentiable at zero (optimization issue). Log-Loss Probabilistic outputs; penalizing confidence in being wrong. Hard to interpret for non-technical stakeholders.
Your Mission
Analyze your assigned dataset. Propose a complete validation strategy, justify your metric choice, and document potential pitfalls before training a single model.
Ready for "Defining the Strategy Proposal"?
Strategy Proposal Worksheet Strategy Proposal
Comparative Analysis Capstone | Lesson 1
Lab Document: #L1-STRAT
Level: Graduate Technical
Researcher Name
Date of Submission
1
Dataset Characterization
Analyze your assigned dataset. Identify specific mathematical characteristics that will influence your selection strategy.
Class Distribution & Imbalance Ratio:
Feature Dimensionality & Cardinality Concerns:
Observation on Noise and Outliers:
2
Validation Framework Design
Propose the exact structure of your validation pipeline (e.g., Nested K-Fold, Time-Series Split).
Proposed Validation Methodology:
Technical Justification (Why this choice? How does it prevent leakage/overfitting?):
3
Metric Selection Logic
Identify your primary optimization metric and secondary monitoring metrics.
Primary Performance Metric:
Secondary Monitoring Metrics:
Scientific/Contextual Justification (Link the metric choice to the real-world impact of prediction errors):
4
Risk Mitigation
What factors could lead to a biased selection? How will you account for them?
Identification of Potential Bias Sources:
Proposed Mitigation Steps:
Cap-An-01-SP Computational Lab Notebook Series Confidential Research
Strategic Planning Guide Strategic Planning Guide
Lesson 1: Defining the Selection Strategy
90 Minutes
Lesson Overview
This lesson shifts students from "fitting a model" to "designing a selection system." Graduate students often focus on algorithm implementation; this session forces them to consider the mathematical and contextual validity of their evaluation pipeline.
Key Objectives
• Identify class imbalance and outlier impact on metrics.
• Design a Nested Cross-Validation (NCV) framework.
• Justify metric selection based on business/scientific cost.
Advanced Concepts
• Leakage via hyperparameter tuning.
• Non-differentiable loss functions (MAE).
• Heteroscedasticity in error distributions.
The "Metric Trap" Discussion
The Provocation:
"In a fraud detection dataset where 0.1% of transactions are fraudulent, a model that predicts 'Not Fraud' 100% of the time has 99.9% accuracy. Why is this model statistically useless, and which metric would expose its failure?"
Facilitation Notes:
Guide students toward Precision-Recall Area Under Curve (PR-AUC) rather than ROC-AUC for imbalanced sets.
Discuss the cost matrix : Is the cost of a missed fraud higher than the cost of a false alert?
Instructional Focus: Nested CV
# Why Nested Cross-Validation?
Simple K-Fold is biased when hyperparameters are tuned. If you use the same CV to tune and evaluate, your performance estimate is "optimistically biased."
Outer Loop: Estimates generalization error of the entire pipeline.
Inner Loop: Optimizes hyperparameters for the specific model choice.
Strategy Proposal Review Checklist
Metric Justification: Does the student link RMSE/MAE/Log-Loss to the actual distribution of the residuals?
Leakage Prevention: Does the proposed framework explicitly separate tuning from evaluation?
Contextual Awareness: Is the "No Free Lunch" theorem mentioned or implied regarding the model's inductive bias?
Candidate Generation Slides Candidate Generation
Building a Diverse Model Portfolio
The Power of Diversity
Linear Models
Assume additivity and linearity. Strong global bias, low variance. Great for small datasets.
Tree Ensembles
Assume local piece-wise constants. Capture complex non-linear interactions and hierarchies.
Regularized Architectures
Lasso (L1) and Ridge (L2). Intentionally introduce bias to reduce variance in high-dimensional space.
"If all your models share the same inductive bias, your selection is just noise."
Early Screening Protocols
How do we filter 50 candidates down to 5 without over-optimizing for the test set?
Screening Visualization
Model A (Tree): High Performance
Model B (Linear): Baseline Only
Model C (Complex): Underperforming
Preventing Selection Bias
"If you torture the data enough, it will confess to anything."
DO:
• Decide on candidate architectures BEFORE seeing test results.
• Use a dedicated "Hold-out" that remains untouched during screening.
DON'T:
• Add/Remove features based on final test error.
• Iterate on the same test set more than 2-3 times.
Launch: Model Generation
1. Open your Model Portfolio Tracker.
2. Implement at least 3 distinct mathematical frameworks.
3. Apply the Initial Screening Protocol to select your top 5 candidates for statistical testing.
Lesson 2 | Computational Portfolio Phase
Model Portfolio Tracker Model Portfolio Tracker
Comparative Analysis Capstone | Phase 2: Generation & Screening
Document Ref: #L2-PORTFOLIO
Researcher
Target Metric
Instructions
Document every candidate model family generated. For each, specify the intended Inductive Bias. Only promote models to the "Candidate Pool" if they pass the Initial Screening Protocol (Performance > Baseline + Complexity Penalty).
Model Class / Inductive Bias Key Hyperparameters Val. Score (Outer) Status Notes / Screening Logic 1. Linear / Lasso
Bias: Sparsity, Feature Selection
| alpha, max_iter |
|
|
|
|
2. Random Forest
Bias: Local Piecewise, Ensembling
| n_estimators, max_depth |
|
|
|
|
3. XGBoost / Gradient Boost
Bias: Residual Reduction, Boosting
| learning_rate, subsample |
|
|
|
|
4. Support Vector Machine
Bias: Maximum Margin, Kernel Trick
| C, kernel, gamma |
|
|
|
|
|
|
|
|
|
Candidate Pool Summary
Final 3-5 Models for Statistical Testing:
Screening Rational (Why these and not others?):
EXPERIMENTAL DATA LOG // DO NOT DUPLICATE WITHOUT AUTHORIZATION
Workshop Facilitation Guide Workshop Facilitation Guide
FAC-L2-GEN
Lesson 2: Candidate Model Generation
This guide outlines the facilitation of the Model Generation Workshop. The goal is to move students from "trying everything" to "strategically sampling inductive biases."
The "Roaming" Protocol
Observation Targets
A. Are students using diverse frameworks? (Ensure they aren't just tuning 5 different Random Forests).
B. Are they logging every failure? Failure documentation is critical for the final defense.
Key Questions to Ask
"What mathematical assumption does this model make about your data distribution?"
"If this model fails, what does that tell us about the underlying function?"
Troubleshooting Failure Modes
Lasso Underfits
Likely high non-linearity or interaction effects. Suggest a Polynomial expansion or a Spline model.
Trees Overfit
Suggest aggressive pruning (max_depth) or increasing min_samples_leaf. Check if the validation set is too small.
Convergence Issues
Ensure features are scaled (StandardScaler). SVMs and Linear Models are highly sensitive to scale.
Screening Rubric (Internal)
When students ask which models to "keep," guide them toward this logic:
Metric Action Justification Score < Baseline Discard Model assumptions are actively harmful. Score ≈ Baseline (Complex) Discard Occam’s Razor: Don't use a forest for a linear job. Score >> Baseline (Complex) Candidate Pool Performance justifies the complexity.
Final 15 Minutes: The "Bias Audit"
Have students pair up and swap Portfolio Trackers. Each reviewer must answer:
"Can you identify at least 3 distinct mathematical assumptions being tested here? If all these models fail, what kind of model is missing?"
Statistical Comparison Slides Statistical Comparison
Separating Signal from Noise in Model Selection
Lesson 3: Hypothesis Testing & Intervals
The 0.5% Dilemma
Model A yields an RMSE of 0.245.
Is Model A actually better, or did it just get lucky with the specific split of the data?
The Statistical Solution
Paired t-test on fold results.
McNemar's test for error correlation.
Bootstrap confidence intervals.
Paired t-test for Folds
When comparing two models across \( K \) folds, we don't just look at the mean. We look at the paired difference in each fold.
\[ t = \frac{\bar{d}}{\sigma_d / \sqrt{K}} \]
*Note: This assumes differences are normally distributed. In small samples, use a non-parametric Wilcoxon Signed-Rank test.
Fold Comparisons
Fold 1:
+0.02
Fold 2:
+0.01
Fold 3:
-0.01
The Bootstrap Approach
"What if we can't assume normality?"
Resample with replacement from your test set 1,000+ times. Calculate the metric each time. The distribution of these results provides the Confidence Interval.
If the 95% CI of the difference contains 0, the models are statistically equivalent.
Resample #1
Resample #1000
Statistical Rigor Lab
Download your candidate results.
Run the Paired Difference analysis.
Identify the One True Winner (or the set of candidates that are indistinguishable).
Signal > Noise // Lesson 3
Statistical Benchmarking Lab Statistical Benchmarking Lab
Differentiating Signal from Noise in Selection
NAME: __________________________
DATE: __________________________
Objective
"Numerical superiority does not equal statistical significance." In this lab, you will apply rigorous hypothesis testing to your top candidate models to determine if observed performance gains are reproducible or merely artifacts of data partitioning.
01
Paired-Difference (Fold-wise)
Extract the validation scores for your top 2 models across all \( K \) folds. Calculate the difference for each fold: \( d_i = M_{1,i} - M_{2,i} \).
Fold Model 1 Score Model 2 Score Difference (\( d_i \)) 1 2 3 4 5
Mean Difference (\( \bar{d} \)):
Standard Deviation of Differences (\( s_d \)):
// Calculate t-statistic
t = (mean_diff) / (std_diff / sqrt(K))
RESULT T = ____________________
02
Bootstrap Distribution
Visualize the results of your bootstrap resampling. Sketch the distribution of the performance difference below.
Performance Difference (\( \Delta \))
95% Confidence Interval for \( \Delta \):
[ _________ , _________ ]
Does the interval contain 0? If so, the difference is statistically negligible at \( \alpha = 0.05 \).
03
Synthesis & Selection
Selection Logic:
One Final Model Candidate:
MODEL_ID: ________________
REF-STAT-LAB-03 COMPUTATIONAL RIGOR PHASE SIGNIFICANCE LEVEL α = 0.05
Benchmarking Answer Key Benchmarking Guide
Instructor Protocol // Lesson 3
Ver: 1.0 // Auth: Lenny
Statistical Significance in Selection
Students often mistake the empirical mean difference for a generalizable gain. Your role is to enforce the rigor of the "Null Hypothesis": that Model A and Model B have identical performance distributions, and any observed difference is due to the stochasticity of the data splits.
Test Selection Matrix
Scenario Recommended Test Critical Assumption Regression (MSE/MAE) Paired t-test Fold differences are \(\sim N(\mu, \sigma)\) Classification (Accuracy) McNemar’s Test Contingency of errors (1 vs 0) Small/Non-Normal Data Wilcoxon Signed-Rank Rank-based, no normality required Complex Metrics (PR-AUC) Bootstrap CI Independent test observations
Interpreting the Lab Results
The "Indistinguishable" Result (\( p > 0.05 \))
If the t-test fails to reject the null, students must not simply pick the one with the higher mean. They should revert to Parsimony : select the simpler model (fewer features, less compute) as it is likely to be more stable.
The "Significant" Result (\( p < 0.05 \))
Even if significant, check the Effect Size . A significant difference of 0.0001 in RMSE may be mathematically real but practically irrelevant to the business case.
The Multiple Testing Trap
Warning: If a student compares 10 models (45 pairs), the chance of a "significant" result occurring by random chance is very high.
// Suggest Bonferroni Correction:
Adjusted Alpha = \(\alpha / N\)
If comparing 5 models to a baseline: \(\alpha_{adj} = 0.05 / 5 = 0.01\)
Teaching models is science. Selecting them is philosophy.
Stability Analysis Slides Stress Testing & Stability
Diagnosing the Fragility of Selection
The Winner's Curse
Your top model won by a statistically significant margin.
"But would it still win if we changed just 10% of the data? Or if we added a small amount of Gaussian noise to the sensors?"
Selection Bias: When the selection process itself overfits the training data.
Internal Stability
Do the same features get selected across different sub-samples?
Predictive Stability
Does the error variance remain constant under data perturbation?
Mechanism: Stability Selection
1. Subsampling
Repeat the selection process on hundreds of randomly chosen sub-samples (usually 50% of the data).
2. Frequency Thresholding
Calculate how often a model/feature is chosen. Truly robust signals should appear in >75% of subsamples.
Feature 01
92%
Feature 02
45%
Feature 03
08%
Only the 92% signal is trustworthy.
Sensitivity Analysis Matrix
Stress Test Method What it Exposes Data Dropout Remove 10-20% random rows. Dependence on specific observations (High Variance). Noise Injection Add random \(\epsilon\) to features. Fragility of high-degree polynomial fits. Label Perturbation Flip 5% of target labels. Sensitivity to measurement error/noise in Ground Truth.
Break Your Model
Select your "Winning" model from Lesson 3.
Run the Sensitivity Analysis protocol.
If performance drops by >15%, your model is too fragile for production.
"Reliability is more important than a low p-value."
Stress Test Analysis Sheet Stress Test Analysis
Reliability Audit // Phase 4
Document Code
#STR-04-STRESS
Principal Investigator
Model Architecture Under Test
Perturbation Response Matrix
Stress Type Baseline Score Stressed Score Degradation Ratio (\( \delta \)) Data Dropout (20%)
Randomly remove 20% of training observations.
|
|
|
|
|
Noise Injection
Add Gaussian noise (\( \sigma=0.1 \)) to continuous features.
|
|
|
|
|
Target Perturbation
Flip/Shift 5% of target values to simulate entry error.
|
|
|
|
\( \delta = (\text{Baseline} - \text{Stressed}) / \text{Baseline} \). A \( \delta > 0.15 \) indicates severe fragility.
Qualitative Impact Assessment
Feature Importance Shifts:
Did the rank-order of top features change significantly under stress?
Observed Error Patterns:
Are the new errors clustered in specific sub-segments of the data?
Final Stability Verdict
Verdict Status
GO
NO
Mitigation Required if "NO":
REF-STRESS-LOG-V2 COMPUTATIONAL ROBUSTNESS PROTOCOL CONFIDENTIAL LABORATORY RECORD
Final Defense Slides The Technical Defense
Capstone Synthesis & Model Reporting
Defending the Selection
"The client doesn't care about your p-values. They care about confidence."
Your task is to synthesize 4 weeks of mathematical benchmarking into a single, cohesive recommendation.
The Synthesis Triad
Performance
Stability
Interpretability
Reporting Trade-offs
Case A: The Racer
High accuracy but high sensitivity to noise.
Recommendation:
"Only if data quality is guaranteed."
Case B: The Tank
Lower accuracy, but performance is identical under all stress tests.
Recommendation:
"Safe for noisy production environments."
Case C: The Glass Box
Simplest model, indistinguishable performance from complex ones.
Recommendation:
"Optimal choice per Occam's Razor."
Structure of the Defense
01
The Strategic Context
Remind the audience of the metric choice and the "Metric Trap" you avoided.
02
The Statistical Evidence
Present the p-values and confidence intervals from your benchmarking.
03
The Robustness Audit
Show how the model survived (or didn't) the stress tests.
04
The Deployment Verdict
The final 'Why'.
Submit & Defend
"You have defined, generated, tested, and stressed. Now, prove your choice."
Ready for "Final Technical Report Template"?
