Significance Signal Slides Significance Signal
Uncovering Truth in the Noise: Hypothesis Testing & Statistical Significance
Lesson 01 | Predictive Power Sequence
Is it Rigged?
You flip a coin 10 times. It comes up Heads 8 times.
"That's suspicious, but it could be luck."
You flip it 1,000 times. It comes up Heads 800 times.
"Something is definitely wrong."
The Core Question:
At what point does variance (luck) become significance (a real signal)?
The Null Hypothesis \(H_0\)
The Default Assumption
"There is no effect. Any observed difference is just due to random chance."
The Burden of Proof
In statistics, we never "prove" something is true. We only collect enough evidence to reject the idea that it's random.
H₀: μ₁ = μ₂ (No difference)
Hₐ: μ₁ ≠ μ₂ (The signal exists)
Decoding the P-Value
The probability of observing your results (or more extreme ones) if the null hypothesis is true.
p > 0.05
Weak evidence against \(H_0\). Likely noise.
p < 0.05
"Statistically Significant"
Reject the Null.
p < 0.01
Very strong evidence. Highly significant.
Confidence Intervals (CI)
Instead of a single point estimate, we provide a range where we expect the true population parameter to fall.
"We are 95% confident that the true conversion rate is between 12% and 18%."
Lower Bound Point Estimate Upper Bound
Width of the interval depends on sample size and variance.
The Workhorse: T-Tests
1-Sample T-Test
Does my sample mean differ from a specific value?
2-Sample (Independent) T-Test
Are the means of Group A and Group B different? (A/B Testing)
# Scipy Implementation
from scipy import stats
T-test for independent samples
t_stat, p_val = stats.ttest_ind(group_a, group_b)
print(f"T-statistic: {t_stat:.4f}")
print(f"P-value: {p_val:.4f}")
Into the Lab
You have a dataset of conversion rates for two different website layouts. Your mission: determine if the new design is actually better.
Start Simulation
Signal Detection Lab Worksheet Signal Detection Lab
Lesson 01: Hypothesis Testing & A/B Analysis
Name: ________________________________
Date: ________________________________
Objective: Formulate statistical hypotheses, interpret p-values from simulated datasets, and determine significance in an A/B testing scenario.
1 The Foundation: Defining Hypotheses
A digital marketing firm is testing a new "One-Click Checkout" button. The current checkout rate is 4.2%. They hope the new button increases this rate.
Null Hypothesis (\(H_0\))
Example: The new button has no effect on the checkout rate.
Alternative Hypothesis (\(H_a\))
Example: The new button increases the checkout rate.
Scenario: Algorithmic Trading
You've developed a new trading bot. You want to know if its average daily return is higher than the S&P 500's daily average of 0.04%.
State the Null Hypothesis (\(H_0\)) mathematically:
State the Alternative Hypothesis (\(H_a\)) mathematically:
2 Interpreting the Output
You run a 2-sample T-test comparing User Engagement Time for two app versions.
# Python Output (Scipy)
T-statistic: 2.145
P-value: 0.034
95% CI: [0.12, 1.85] seconds
A. Using an alpha level of \(\alpha = 0.05\), do you reject the null hypothesis? Why or why not?
B. If you changed your confidence level to 99% (\(\alpha = 0.01\)), would your conclusion change? Explain.
C. Thinking Critically: Type I vs Type II Errors
If the new app version is actually not better, but your test said it was significant, what type of error did you commit? What is the probability of this happening in the test above?
3 A/B Test Design: The "Social Proof" Experiment
A streaming service wants to add a "Trending Now" badge to movies. They split their users into two groups:
Analysis Plan
Describe the steps you would take to validate this experiment. Which test would you use, and what data points are essential?
The "Fair Coin" Check
Short Answer
Why is having an equal sample size (500 vs 500) important in this context? What happens to the power of the test if Group B only had 50 users?
Visualizing Distribution
Sketch
Roughly sketch two normal distributions that would lead to a statistically significant result. Label the means.
Variable Value
Predictive Power | Module 01 Formulation & Logic © 2026 Statistical Inference Sequence
Significance Teacher Guide Significance Signal
Teacher Facilitation Guide | Lesson 01
Lesson Overview
This lesson transitions students from simple data description to statistical inference . The goal is to move beyond "the average is different" to "is the difference meaningful?" using the framework of hypothesis testing. We use a simulation-first approach to build intuition before diving into the formal math and code.
Key Learning Objectives
Define Null (\(H_0\)) and Alternative (\(H_a\)) hypotheses for real-world scenarios.
Interpret p-values relative to significance thresholds (\(\alpha\)).
Explain the relationship between sample size, variance, and confidence intervals.
Implement and interpret a 2-sample T-test using Python's Scipy library.
Pacing
Hook: Coin Toss 10m
Core Concepts Slides 25m
Signal Detection Lab 40m
Debrief & Discussion 15m
Materials Needed
• Significance Signal Slides
• Signal Detection Lab (1 per student)
• Laptop with Python/Jupyter
• (Optional) A physical deck of cards or coins
The Hook: The "Rigged Coin" Simulation
Facilitation Strategy:
Ask for a volunteer. Use a random number generator (0 or 1) to simulate 10 coin flips. If you get 7 or 8 heads, ask the class: "Is this coin fair?" Most will say "probably."
Then, simulate 100 flips. If the ratio remains 80/20, ask again. The shift in confidence illustrates the power of Sample Size in reducing standard error.
Discussion Prompt:
"Why are you more certain with 100 flips than 10, even though the percentage of heads is exactly the same? What are we actually measuring when we say we're 'sure'?"
Common Student Misconceptions
"P = 0.05 means there's a 95% chance I'm right."
Correction: A p-value is NOT the probability that the hypothesis is true. It is the probability of seeing that data if the null is true. It measures the strength of evidence against the null, not the probability of the truth.
"High p-value means the Null is true."
Correction: A high p-value simply means we fail to reject the Null. It's like a "Not Guilty" verdict in court; we didn't prove innocence, we just lacked enough evidence to prove guilt.
Lab Facilitation & Key Answers
Section 1: Hypothesis Formulation
Students often struggle with "equal to" vs "not equal to." Remind them that \(H_0\) always contains the equality sign.
Linear Logic Slides Linear Logic
The Bedrock of Predictive Modeling: Simple Linear Regression
Lesson 02 | Predictive Power Sequence
Can Study Hours Predict Scores?
Hours (\(X\)) Score (\(Y\)) 2 65 5 82 8 94 1 ?
If you know the relationship between input and output, you can predict the unknown.
"For every extra hour I study, how much higher will my score be?"
The Linear Model
\(\hat{y} = \beta_0 + \beta_1x + \epsilon\)
\(\hat{y}\)
Predicted Output (Target)
\(\beta_0\)
Intercept (Starting Point)
\(\beta_1\)
Slope (The Relationship)
\(\epsilon\)
Error/Residual (Noise)
How do we find the "Best" line?
Ordinary Least Squares (OLS)
We want to find the line that minimizes the sum of the squared differences (residuals) between our predictions and actual points.
Why Square?
Squaring penalizes large errors more heavily and ensures all differences are positive values.
Input (X)
Target (Y)
The Ghosts of Errors
Residuals
Residual = \(y_{actual} - y_{predicted}\)
If your model is good, your residuals should look like white noise. No patterns, no clusters, no "fanning out."
GOOD: Random
The model captured all the "logic."
BAD: Patterned
You missed something important!
Implementation
from sklearn.linear_model import LinearRegression
# 1. Initialize model
model = LinearRegression()
# 2. Fit the logic
model.fit(X_train, y_train)
# 3. Predict the future
predictions = model.predict(X_new)
OLS Workshop Worksheet OLS Workshop
Manual Fitting & Interpretation
Name: ________________________________
Lab Section: _________________________
The Dataset
The following table shows the relationship between Square Footage (X) in thousands and Monthly Rent (Y) in hundreds for five apartments.
SqFt (\(X\)) Rent (\(Y\)) 0.8 12 1.1 15 1.4 19 1.8 24 2.2 30
The Model Output
After fitting an OLS model, the computer yields the following equation:
\(\hat{y} = 1.45 + 12.8x\)
1. Interpret the slope (\(\beta_1 = 12.8\)) in the context of this problem:
2. Interpret the intercept (\(\beta_0 = 1.45\)). Does this value make physical sense in this context?
Residual Calculation
Calculate the predicted rent and the residual for the following two points from the dataset above using the equation: \(\hat{y} = 1.45 + 12.8x\)
Point A: \(x = 0.8, y = 12\)
Predicted \(\hat{y}\): _____________________
Residual (\(y - \hat{y}\)): __________________
Point B: \(x = 1.8, y = 24\)
Predicted \(\hat{y}\): _____________________
Residual (\(y - \hat{y}\)): __________________
Visualization Challenge
Below is a residual plot (Residuals vs Predicted Values). Based on the pattern, would you trust this linear model for high-value predictions? Explain why.
Predicted Value
Write your analysis here...
Predictive Limits
An investor wants to use your model to predict the rent of a mega-mansion with 15,000 SqFt (\(x = 15.0\)).
Calculation
Predicted Rent: __________
Risk Alert: Extrapolation
"Is a 15,000 sq ft mansion likely to follow the same linear pattern as 1,000 sq ft apartments?"
Conceptual Summary
In your own words, describe what OLS is trying to "minimize" and why we don't just use the absolute difference (\(|y - \hat{y}|\)).
Bonus: Matrix Notation
If \(X\) is a matrix of features and \(y\) is a vector of targets, the OLS solution for the weights \(\beta\) is given by the Normal Equation. Fill in the missing symbol:
\(\hat{\beta} = (X^T X)^{-1} X^T \quad \text{____}\)
Regression Reference Sheet Regression Reference Sheet
Simple Linear Regression & OLS Fundamentals
The Mathematical Model
\(\hat{y} = \beta_0 + \beta_1x\)
The "Line of Best Fit" calculated by minimizing the sum of squared residuals.
\(\beta_0\)
Intercept
The value of \(\hat{y}\) when \(x = 0\). Often a baseline or starting point.
\(\beta_1\)
Slope (Coefficient)
The average change in \(\hat{y}\) for every one-unit increase in \(x\).
Interpretation Guide
SLOPE (\(\beta_1\)) Template
"For every additional [Unit of X], we predict the [Target Variable] will increase/decrease by [\(\beta_1\)] [Units of Y], on average."
INTERCEPT (\(\beta_0\)) Template
"When [Variable X] is zero, the predicted [Target Variable] is [\(\beta_0\)] [Units of Y]."
Diagnostics Checklist
Residual Formula
\(e = y - \hat{y}\)
The vertical distance from an actual data point to the regression line.
What to look for in Residual Plots:
Random Scatter
No discernible patterns. Indicates a linear model is appropriate.
Curvature
Indicates a non-linear relationship (try polynomial regression).
"Fanning" (Heteroscedasticity)
Variance changes as X increases. Violation of OLS assumptions.
Outliers
Single points with very high residuals. Can disproportionately pull the line.
Normal Equation
\(\beta = (X^T X)^{-1} X^T y\)
The analytical solution for multivariate weights
Sum of Squares
"The shortest total 'distance' squared."
Predictive Power Reference Card // Module 02: Simple Linear Regression
Multivariate Models Slides Multivariate Models
Expanding the Universe: Multiple Regression & Feature Selection
Lesson 03 | Predictive Power Sequence
One Predictor is Rarely Enough
What determines the price of a house?
Sq. Footage
Zip Code
Year Built
School Rank
If we only use Sq. Footage, we ignore 90% of the story.
The Multivariate Equation:
\(\hat{y} = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_nx_n\)
The "All Else Being Equal" Rule
In Multiple Regression, \(\beta_1\) is the effect of \(x_1\) on \(y\) while holding all other variables constant.
Example:
"For two houses with the same zip code and age, the one with 100 more sq ft costs $15k more."
Adjusted \(R^2\)
Adding variables always increases \(R^2\), even if the variables are garbage.
Adjusted \(R^2\) penalizes you for adding irrelevant features.
The Problem: Multicollinearity
When two predictors are highly correlated (e.g., Number of Bathrooms and Number of Toilets).
• Coefficients become unstable
• Hard to tell which variable is doing the work
• Variance Inflation Factor (VIF) is our diagnostic tool
Redundant Info!
"The model can't distinguish between the signals."
Selecting the "A-Team"
Forward Selection
Start with zero features. Add the best one at each step until no significant improvement remains.
Backward Elimination
Start with ALL features. Remove the least significant one until only the strong survive.
LASSO (L1)
Regularization that pushes coefficients of weak variables exactly to zero. Automatic selection!
More is Not Better
Great models are built on meaningful signals, not a mountain of noisy features.
Start Case Study
Check VIF Scores
Housing Price Case Study Worksheet The "A-Team" Case Study
Lesson 03: Multivariate Models & Feature Selection
Student ID: __________________________
Section: _____________________________
Scenario: Global Realty Analytics
Global Realty wants to build a model to predict house sales prices. They've given you a model with 5 variables. Your job is to determine if all 5 are necessary, or if "noise" is clouding the model.
Regression Summary Output
Variable Coefficient (\(\beta\)) P-value VIF Intercept 15,400 0.001 — Living Area (sq ft) 210 0.000 2.4 # of Bedrooms -1,200 0.542 12.5 # of Bathrooms 5,800 0.031 1.8 Lot Size (acres) 3,100 0.485 1.2 Garage Capacity 8,200 0.015 1.5
Model Metrics:
\(R^2\): 0.884
Adjusted \(R^2\): 0.812
F-statistic P-value: 0.000
Refresher: VIF Scale
• 1.0 = No correlation with other features
• 1.0 - 5.0 = Moderate correlation (usually OK)
• > 5.0 - 10.0 = High correlation (suspicious)
• > 10.0 = Severe multicollinearity (REDUNDANT)
? Expert Analysis
1. The "Bedroom Paradox":
Look at the coefficient for # of Bedrooms. It is negative (-1,200), suggesting more bedrooms decrease home value. However, look at its VIF and P-value . Explain why you cannot trust this negative coefficient.
2. Feature Selection Strategy:
Based on the P-values and VIF scores, which two variables would you consider removing from the model first? Justify your choice for each.
Candidate 1:
Candidate 2:
3. The Final Prediction:
Assume you kept only Living Area (\(\beta=210\)) and Garage Capacity (\(\beta=8,200\)), plus the Intercept (\(\beta=15,400\)). Calculate the predicted price for a house with 2,000 sq ft and a 2-car garage.
\(\hat{y} = \) __________________________________________
Predictive Power | Module 03 Multivariate Selection © 2026 Statistical Inference Sequence
Diagnostic Drilldown Slides Diagnostic Drilldown
Is Your Model Honest? Evaluation Metrics & Assumption Testing
Lesson 04 | Predictive Power Sequence
Which Model Wins?
MODEL A
"The high performer"
\(R^2 = 0.98\)
...but it's wildly overfitted.
MODEL B
"The reliable worker"
\(R^2 = 0.75\)
...stable, interpretable, and generalized.
A high \(R^2\) doesn't always mean a better model.
We need to look at Error Metrics and Statistical Assumptions.
Quantifying Failure
MAE
Mean Absolute Error
\(\frac{1}{n}\sum|y - \hat{y}|\)
Treats all errors equally. Very robust to outliers. The "average mistake" in natural units.
RMSE
Root Mean Squared Error
\(\sqrt{\frac{1}{n}\sum(y - \hat{y})^2}\)
Penalizes large errors. If your business hates big mistakes, optimize for this!
The 4 Commandments (LINE)
L
Linearity
The relationship between X and Y is a straight line.
I
Independence
Observations are not correlated with each other.
N
Normality
Residuals (errors) follow a normal distribution.
E
Equal Variance
Homoscedasticity: Error spreads evenly across all X values.
The "Fan" Problem
Homoscedasticity (Good)
Consistent uncertainty across all ranges.
Heteroscedasticity (Bad)
Errors grow with the input. You can't trust high-value predictions!
Model Physical
Today, we play Data Doctor. You'll run diagnostics on multiple models and decide which ones are healthy enough to go into production.
Start Diagnostic Lab
Model Evaluation Lab Worksheet Model Diagnostic Lab
"Is your model healthy enough for production?"
Reference: Predictive Power Lesson 04
Lab ID: DX-402-B
Patient: E-commerce Sales Model
Vitals: \(R^2\), RMSE, MAE, Residuals
1 Quantitative Vitals
You are comparing two models that predict Total Sales ($). One uses a linear approach, the other uses a complex polynomial.
Model 1 (Simple)
\(R^2\) 0.82
MAE $450
RMSE $480
Model 2 (Complex)
\(R^2\) 0.88
MAE $420
RMSE $950
Analysis A: The RMSE Anomaly
Model 2 has a higher \(R^2\) and a lower MAE, but its RMSE is double that of Model 1. What does this tell you about Model 2's errors?
Analysis B: The "Better" Model
If the cost of a single large error is devastating (e.g., warehouse overstocking), which model would you recommend? Why?
2 Visual Diagnostics (Residual Plots)
Inspect the three residual plots below. Identify which OLS assumption (LINE) is being violated in each case.
Plot A
Violation:
Remedy:
Plot B
Violation:
Remedy:
Plot C
Violation:
Remedy:
Final Prognosis
You run a Normality test (Q-Q Plot) and find that the errors are heavily skewed. Which metric (RMSE or MAE) is likely to be a more stable representation of error in this case? Why?
Predictive Power | Module 04 Model Diagnostics © 2026 Statistical Inference Sequence
Classification Crisis Slides Classification Crisis
Beyond Continuous Numbers: Logistic Regression & Probabilities
Lesson 05 | Predictive Power Sequence
When the Answer is Yes or No
Standard regression predicts quantity (e.g., $100k price).
But what if you need to predict category?
Spam or Not?
Fraud or Valid?
Disease or Healthy?
Churn or Stay?
The Problem:
"If I use linear regression, I might predict a 150% chance of rain. That doesn't make sense."
The Sigmoid Curve
We use the Logit Link Function to "squash" any real-valued number into a range between 0 and 1.
\(P(y=1) = \frac{1}{1 + e^{-z}}\)
Where \(z = \beta_0 + \beta_1x\)
0.5 Threshold
When to pull the trigger?
Default (0.5)
If prob > 50%, predict "Spam." Balanced approach.
Conservative (0.9)
Only predict "Cancer" if we are 90% sure. High Precision, Low Recall.
Aggressive (0.1)
Flag "Fraud" if there's even a 10% chance. High Recall, Low Precision.
The Confusion Matrix
<table class="w-full border-collapse text-2xl text-center"><tbody><tr><td></td><td class="pb-4 font-bold">Predicted +</td><td class="pb-4 font-bold">Predicted -</td></tr><tr><td class="pr-4 font-bold text-right">Actual +</td><td class="p-8 bg-indigo-500 text-white border border-indigo-600 rounded-tl-3xl">True Pos (TP)</td><td class="p-8 bg-slate-800 border border-slate-700">False Neg (FN)</td></tr><tr><td class="pr-4 font-bold text-right">Actual -</td><td class="p-8 bg-slate-800 border border-slate-700">False Pos (FP)</td><td class="p-8 bg-indigo-500 text-white border border-indigo-600 rounded-br-3xl">True Neg (TN)</td></tr></tbody></table>
Precision
"Of all flagged as spam, how many actually were?"
Recall (Sensitivity)
"Of all actual spam, how many did we catch?"
Classification Quest
Your final mission: Build a Spam Filter. You'll decide the threshold and analyze the fallout of false positives.
Launch Spam Lab
Spam Filter Project Guide Project: Zero-Spam Protocol
Lesson 05: Logistic Regression & Classification
Lab Access: ALPHA-6
Status: Modeling Phase
Mission Overview
You are building a binary classifier to detect Spam Emails. Your model uses word frequency, sender reputation, and attachment metadata to calculate the probability that an email is malicious.
Critical Constraint
A "False Positive" (legitimate email sent to spam) is 10x more costly than a "False Negative" (spam reaching the inbox). Adjust your strategy accordingly.
The Logic
Logit(p) = -3.2 + 2.5(Urgent) + 1.8(LinkCount) - 4.1(KnownSender)
Where "p" is the probability of SPAM.
1. Setting the Decision Boundary
Choose a probability threshold (\(\tau\)) for your filter. If \(P(Spam) > \tau\), the email is blocked.
Proposed Threshold
0. ____
Justification:
"Why did you pick this number? Consider the trade-off between Precision and Recall based on the mission constraint above."
2. Performance Audit
After running your model on a test set of 1,000 emails, you get the following confusion matrix:
Pred: SPAM Pred: SAFE Actual: SPAM 180 20 Actual: SAFE 45 755
Calculate Accuracy:
____ %
Calculate Recall:
(\(TP / (TP + FN)\))
____ %
3. Optimization Protocol
Refining the Model
You find 45 "False Positives"—important business emails that were blocked. What specific action will you take to fix this? (e.g., Change threshold, add specific feature, collect more data?)
Conceptual Check: The Log-Odds
In logistic regression, the coefficient for "Known Sender" is -4.1. Explain what this negative number means for the probability of an email being marked as spam.
Writing space...
Bonus: ROC Curve Sketch
Sketch a perfect classifier vs a random guess (baseline) on this ROC space.
True Pos Rate
False Pos Rate
Final Summary
"A model is only as good as the metrics you choose to optimize. For this project, which metric was the most important 'north star'?"
Predictive Power | Module 05 Binary Classification Lab © 2026 Statistical Inference Sequence
Classification Metrics Card The Metric Match
Final Assessment | Lesson 05
Name: ____________________
Case: Fraud Detection
Your algorithm flagged 100 transactions as fraud. Of those, 80 were actually fraud. There were 20 actual fraud cases that the model missed.
Pred + Pred - Act + 80 20 Act - 20 880
1. Precision:
2. Recall:
3. F1-Score:
4. The Trade-Off Paradox
If you lower your decision threshold from 0.5 to 0.1, what will happen to your Recall ? Why?
5. The "Medical Scan" Dilemma
In a cancer screening model, which error is deadlier: a False Positive (False Alarm) or a False Negative (Missed Diagnosis)? Explain your reasoning.
Sequence Synthesis
Thinking back to Lesson 2 (Linear Regression) vs Lesson 5 (Logistic Regression): When should you use Linear and when should you use Logistic modeling?
End of Sequence // Predictive Power // Module 05 exit
