Model Match Slides Model Match
Blueprint for Comparison
AGENCY_ID: DD-INF-L1
The Consultant's Dilemma
You are handed a messy dataset from a global firm. Your boss needs to know if the difference between two groups is real or just noise.
"We have two groups. We have measurements. Which formula should we use?"
Core Questions:
What type of data is it? (Categorical or Quantitative)
How was the data collected ? (Independent or Paired)
How many populations are involved?
Pathway: Categorical
2-Prop Z-Test
Scenario Indicators
• "Success" or "Failure" rates
• Percentages (%) of a population
• Yes/No survey responses
The Logic
Comparing the proportion of successes (\(p_1\)) against another (\(p_2\)).
Case Example
"Does a higher percentage of high school seniors (Group 1) plan to vote than high school juniors (Group 2)?"
Pathway: Quantitative
2-Sample T-Test
Scenario Indicators
• Comparing Means (\(\mu_1\) vs \(\mu_2\))
• Measurements (Time, Height, Score)
• Independent random samples
The Logic
The two groups have no relationship to one another. They are distinct entities.
Case Example
"Is there a difference in the average test scores of students in the morning class vs. students in the afternoon class?"
Pathway: Paired
Paired T-Test
Scenario Indicators
• "Before and After" studies
• Matching pairs (Twins, Husbands/Wives)
• Two measurements on the same subject
The Logic
We analyze the difference within each pair first, then find the mean of those differences (\(\mu_d\)).
Case Example
"Did the average blood pressure of patients drop after taking the medication compared to their baseline before the study?"
The Decision Tree Filter
Variable Type?
Categorical Quantitative
Relationship?
Independent Paired
Populations?
One Two
"Choose the right tool. Build the right model. Get the right answer."
Inference Blueprint Worksheet Inference Blueprint
Internal Memo: Decision Doctors Agency
File Ref: DD-INF-L1-W
Name
Date
CONSULTANT OBJECTIVE: Your client needs to compare two populations. Your task is to build a reliable decision tree (blueprint) to select the correct statistical test and then apply it to the following messy case files. Precision is non-negotiable.
PART 1: THE DECISION ARCHITECTURE
Fill in the branches of the decision tree to differentiate between a 2-Sample T-Test , a Paired T-Test , and a 2-Prop Z-Test .
IS THE DATA CATEGORICAL OR QUANTITATIVE?
CATEGORICAL
Identify Test:
QUANTITATIVE
ARE THE SAMPLES INDEPENDENT OR PAIRED?
INDEPENDENT
Test:
PAIRED / MATCHED
Test:
PART 2: CASE FILE APPLICATIONS
Review each client scenario. Identify the test and provide a brief justification based on your blueprint.
Case ID: 402-A (Education) Priority
"A researcher wants to see if a new SAT prep course actually works. She records the scores of 50 students before they take the course and then after they complete it."
Recommended Test
Reasoning
Case ID: 511-C (Healthcare)
"A hospital wants to know if the proportion of smokers who develop lung disease is significantly higher than the proportion of non-smokers who develop the disease."
Recommended Test
Reasoning
Case ID: 622-R (E-Commerce)
"An online retailer compares the average amount spent per transaction between customers in the UK and customers in the US to decide where to focus their marketing budget."
Recommended Test
Reasoning
Internal Justification Protocol
When presenting your choice to a client, what is the single most important factor that differentiates an independent samples design from a paired design? Why can't you use them interchangeably?
Inference Navigator Teacher Guide Teaching Manual
Inference Navigator
Lesson 1: Blueprint for Comparison
L1
Module Unit
Pacing Guide
10m
The Hook: Use Slide 2 to frame students as "Consultants" dealing with messy data.
20m
Direct Instruction: Walk through Pathways (Slides 3-5). Emphasize study design over mechanics.
25m
Workshop: Students complete Part 1 & 2 of the Inference Blueprint Worksheet in pairs.
05m
Debrief: Rapid-fire case file answers to check for consensus.
Key Objectives
Differentiate categorical and quantitative variables in context.
Distinguish between independent samples and paired/matched designs.
Select the correct 2-population test with 100% accuracy.
COMMON PITFALLS & MISCONCEPTIONS
The "Two Means vs. Two Groups" Confusion
Students often think if there are two groups, it must be a 2-sample t-test. Remind them to look at the variable . If they are recording "Yes/No" or "Success/Failure", it's categorical (Proportions), not means.
Paired vs. Independent
This is the toughest distinction. Teach students to ask: "If I change the order of Group A, does it break the connection to Group B?" If yes, it's paired. If no, it's independent.
WORKSHEET KEY: CASE FILES
Case 402-A (SAT) Paired T-Test
Justification: Quantitative data (scores). Paired because the same students are measured twice (Before/After).
Case 511-C (Smoking) 2-Prop Z-Test
Justification: Categorical data (Disease: Yes/No). Comparing two independent groups of people.
Case 622-R (E-Commerce) 2-Sample T-Test
Justification: Quantitative data (Spent Amount). Independent groups (UK vs. US customers).
Discussion Briefing: "The Boardroom Questions"
Question 1
"Why does it matter if we use a paired test vs. an independent test if the data looks the same?"
Target Answer: Paired tests account for individual variation (they have more power). Using an independent test on paired data is wasteful and often inaccurate.
Question 2
"Can you ever use a T-test for proportions?"
Target Answer: Technically, no. T-tests are for means of continuous data. Proportions use Z-tests based on the normal approximation of the binomial distribution.
High Stakes Errors Slides High Stakes Errors
The Cost of Being Wrong
The Binary Burden
In statistics, we make a binary choice: Reject \(H_0\) or Fail to Reject \(H_0\).
But what if the truth is different from our decision?
Decision: Keep \(H_0\) Truth: \(H_0\) is True
Type I Error False Alarm
Type II Error Missed Opportunity
Decision: Reject \(H_0\) Truth: \(H_0\) is False
Type I: The False Alarm
\(\alpha\)
The Definition
Rejecting the Null Hypothesis when the Null is actually True .
In a Comparison:
Finding a "significant difference" between two groups when none exists.
The Real-World Sting
"A startup claims their new coffee formula makes people 20% faster at typing. They release it, but it's just regular coffee. They've wasted millions on marketing a lie."
Type II: The Miss
\(\beta\)
The Definition
Failing to reject the Null Hypothesis when the Null is actually False .
In a Comparison:
Concluding there is "no significant difference" when a real difference exists.
The Real-World Sting
"A medical lab develops a cure for a rare disease. Because their sample size was too small, their test shows 'no difference'. They stop production on a life-saving drug."
The Tug of War: \(\alpha\) vs \(\beta\)
Type I
Type II
DECREASE \(\alpha\)
Harder to reject the Null. You become more conservative.
INCREASE \(\beta\)
INCREASE SAMPLE SIZE (\(n\))
The only way to decrease both errors simultaneously.
INCREASE POWER
The Boardroom Debate
Scenario: New Safety Braking System
We are comparing a new AI brake system to the standard mechanical brake.
Null: No difference in stopping distance.Alt: AI brakes stop significantly faster.
Type I Risk:
We market "faster brakes" that aren't actually faster. Potential lawsuits/recalls.
Type II Risk:
We scrap a safer braking system because the data was slightly unclear. People die.
Which error is more dangerous here?
Risk Assessment Case Studies Risk Management Dossier
High Stakes Analysis
Topic: Type I and Type II Errors in Comparison
Consultant
Date
Warning: Statistical significance is not the same as truth. As a decision doctor, you must evaluate the consequences of being wrong in both directions. Your goal is to advise clients on which risk (\(\alpha\) or \(\beta\)) is more acceptable in their specific industry.
Part 1: The Decision Matrix
TRUTH: \(H_0\) is TRUE
TRUTH: \(H_0\) is FALSE
Action: REJECT \(H_0\)
Type I Error
Describe in context:
Correct Decision
Action: FAIL TO REJECT \(H_0\)
Correct Decision
Type II Error
Describe in context:
Part 2: Case Study Analysis
SCENARIO A: Pharmaceutical Comparison High Risk
A lab is testing a new heart medication. They compare the survival rates of patients on the new drug to those on the standard drug.\(H_0\): Survival rate for New Drug \(\le\) survival rate for Standard Drug.\(H_a\): Survival rate for New Drug \(>\) survival rate for Standard Drug.
Define Type I Error in this context:
Define Type II Error in this context:
Consultant Judgment:
Which error is worse for the patients ? Why?
SCENARIO B: Tech Startup A/B Testing Financial Risk
A social media app is testing a new "Addict-Mode" algorithm. They want to see if it increases the average time spent on the app compared to the current algorithm. If it does, they will spend $5M to implement it.\(H_0\): \(\mu_{new} \le \mu_{old}\)\(H_a\): \(\mu_{new} > \mu_{old}\)
Define Type I Error in this context:
Define Type II Error in this context:
Consultant Judgment:
Which error is worse for the company's bottom line ? Why?
Risk Assessment Answer Key Answer Key: Risk Assessment
Decision Doctors Internal Resource
Confidential
Part 1: The Decision Matrix (Suggested Text)
Type I Error (False Alarm)
The study finds a statistically significant difference between the two populations, but in reality, the populations are the same (no difference). You claimed a "win" that wasn't there.
Type II Error (Miss)
The study fails to find a significant difference, but a real difference actually exists in the population. You missed a "win" because the signal was too weak or the noise was too high.
Part 2: Case Study Guidance
Scenario A: Pharmaceutical Comparison
Type I (Alpha)
"The lab concludes the new heart med is better when it's actually no better (or worse) than the standard."
Type II (Beta)
"The lab concludes there's no difference when the new drug is actually a life-saver."
Teaching Note: Patient Ethics
Which is worse? This is debatable, but usually, Type I is considered dangerous in medicine because you are giving people a drug that doesn't work (false hope, potential side effects). However, for a fatal disease, Type II (missing a cure) is equally catastrophic. Encourage students to argue both sides.
Scenario B: Tech Startup A/B Testing
Type I (Alpha)
"The company spends $5M implementing 'Addict-Mode' only to find it doesn't actually increase user time."
Type II (Beta)
"The company sticks with the old algo and misses out on a massive revenue boost from increased user engagement."
Teaching Note: Financial Logic
Which is worse? For a startup with limited cash, Type I is often deadly because of the $5M sunk cost. For a giant like Meta or Google, Type II might be worse because "stagnation is death" in tech.
Key Takeaway for Students
"Statistics isn't just about math; it's about consequence management. Choosing an \(\alpha\) level (like 0.05 vs 0.01) is a business decision that reflects how much risk you are willing to take of being a 'False Alarm' vs. a 'Miss'."
Power Surge Slides Power Surge
The Ability to Detect Truth
The Microscope Lens
Statistical Power is the probability that your test will correctly reject the null hypothesis when there is a real difference.
Power = \(1 - \beta\)
The power to find the signal in the noise.
The Radar Analogy
High Power = A sensitive radar that spots every incoming plane.
Low Power = A fuzzy radar that misses half the targets.
The Three Levers of Power
Sample Size (\(n\))
More data reduces variability and makes the picture clearer.
INCREASE \(n\)
Effect Size
It's easier to see a giant difference than a tiny one.
BIGGER GAP
Alpha (\(\alpha\))
If you are less strict about false alarms, you detect more real things.
LARGER \(\alpha\)
Visualizing the "Gap"
Scenario A: Tiny Effect
The distributions overlap almost entirely. Very hard to tell apart.
Scenario B: Large Effect
Clear daylight between groups. High probability of rejection.
The Power Tip
"If the client wants to detect a tiny difference, they must increase the sample size. There is no other way to keep the false alarm rate low."
The Business of Samples
Data isn't free. Every survey respondent, every lab rat, and every clinical trial participant costs money.
The Consultant's Job:
"Find the minimum sample size needed to achieve a power of 0.80 (80%)."
Why 80%?
It's the industry standard trade-off between detecting a difference and saving money.
We accept a 20% chance of a Type II error (\(\beta = 0.20\)).
Detection Challenge
You are testing two website designs. Design A has a 5% click rate. Design B has a 5.1% click rate.
The Problem
Effect size is tiny (0.1%).
The Solution
You need a massive sample size (thousands of users) to have any power to detect it.
Is it worth the money to detect that 0.1%?
Detection Lab Worksheet DETECTION LAB
Statistical Power Analysis v1.0
Junior Analyst
Lab Date
Phase 1: The Null vs. The Alternative
In the space below, sketch two normal distributions. One representing the Null Hypothesis (\(\mu_1 = \mu_2\)) and one representing a specific Alternative Hypothesis (\(\mu_1 \neq \mu_2\)). Shade the area that represents Power .
Parameter Value (\(\mu\))
Annotation Check:
Where is the rejection region (\(\alpha\)) on the Null curve? Where is \(\beta\) on the Alternative curve?
Reflective Note:
Why does "Power" only exist if the Null is actually false?
Phase 2: Parameter Sensitivity Analysis
Predict how each change will affect the Power of a 2-sample t-test. Explain the why.
Increase Sample Size
Example: \(n=30 \rightarrow n=100\)
Effect on Power
UP
DN
Reasoning (Conceptual)
Decrease Significance Level
Example: \(\alpha=0.05 \rightarrow \alpha=0.01\)
Effect on Power
UP
DN
Reasoning (Conceptual)
Increase Effect Size
Example: Detecting a 5 lb diff vs 10 lb diff.
Effect on Power
UP
DN
Reasoning (Conceptual)
Final Consultant Advisory
A client is terrified of a Type I error (False Alarm) so they demand you use \(\alpha = 0.001\). What is the consequence for their ability to detect a real difference, and how must you adjust your study design (budget/sample size) to compensate?
Power Play Reference Guide Teacher Reference
Power Play Reference
Ref: DD-PWR-T3
The Power Dynamics Matrix
Sample Size (\(n\)) \(\uparrow\)
Standard Error \(\downarrow\)
Power \(\uparrow\)
Alpha (\(\alpha\)) \(\downarrow\)
Rejection Region \(\downarrow\)
Power \(\downarrow\) / Beta (\(\beta\)) \(\uparrow\)
Effect Size \(\uparrow\)
Distance between curves \(\uparrow\)
Power \(\uparrow\)
Variability (\(\sigma\)) \(\downarrow\)
Overlap between curves \(\downarrow\)
Power \(\uparrow\)
Detection Lab Key Guidance
Sketching Phase (Phase 1)
Ensure students draw the alternative distribution shifted to the right (for a 1-sided test) or both ways (for 2-sided). Power is the area under the Alternative curve that falls inside the rejection region defined by the Null curve. Type II error (\(\beta\)) is the area under the Alternative curve that falls in the fail-to-reject region.
Parameter Analysis Key (Phase 2)
Change Power Conceptual Reasoning Increase \(n\) UP Larger sample reduces standard error (spread), making the distributions narrower and easier to distinguish. Decrease \(\alpha\) DOWN Making it harder to reject the null (moving the goalposts) naturally makes it harder to detect a real effect. Increase Effect UP A bigger "gap" between population means means less overlap between the two curves.
The "Analyst Advisory" (Final Challenge Key)
If a client demands \(\alpha = 0.001\), they are prioritizing avoiding Type I errors (False Alarms). The consequence is a massive drop in Power. To maintain high Power (say, 80%), the consultant must significantly increase the sample size. This results in a much higher budget cost for the study.
Consultant Brief: The Bistro Battle Project Alpha-Bistro: Top Secret
BISTRO BATTLE
Consultant Case File: #882-B
Security Clearance
Tier 1
Client Info
Le Petit Bistro Global
142 Chains Worldwide
HQ: Lyon, France
The Situation
"Our revenue is dipping. We've introduced a new 'Speed-Menu' designed to get customers in and out in under 45 minutes. We've rolled this out in half of our locations (Group B), while the other half (Group A) kept the 'Classic-Menu'. The CEO is breathing down my neck—she wants to know if the Speed-Menu actually leads to a statistically significant increase in the proportion of customers who give us a 5-star rating on their survey."
Raw Data Specs
Location A (Classic): 400 surveys collected. 240 were 5-star ratings. 10 were invalid (incomplete).Location B (Speed): 450 surveys collected. 315 were 5-star ratings. 20 were invalid (incomplete).Note: Surveys were collected randomly over a 30-day period. These samples represent thousands of monthly customers.
Consultant Mission
You must determine if the Speed-Menu is a success. This is not just about the numbers; it's about the validity of your conclusion.
Clean the messy data.
Verify ALL statistical conditions.
State your Hypotheses and \(\alpha\).
Director's Note:
"Don't just give me a p-value. I need to know if we can generalize this to all Le Petit Bistro locations. If you miss a condition, your entire report is junk. Get to work."
Data Prep & Protocol Sheet Consultant Protocol
Form DD-SIM-4A | Project Alpha-Bistro
Analyst ID:
Case Status:
In Progress
1 DATA SCRUBBING & AGGREGATION
"Remove invalid survey entries from the total count and calculate your success counts for each group."
GROUP A: CLASSIC MENU
Total Surveys Handed In: 400
Invalid Surveys (Discard): - 10
Final Sample Size (\(n_1\)):
Success Count (\(x_1\)):
GROUP B: SPEED MENU
Total Surveys Handed In: 450
Invalid Surveys (Discard): - 20
Final Sample Size (\(n_2\)):
Success Count (\(x_2\)):
2 INFERENCE PROTOCOL SETUP
State Your Hypotheses (Symbolic Form)
\(H_0:\)
\(H_a:\)
Significance Level (\(\alpha\))
Selected Test
"Explain why you chose this test over others."
3 CONDITION VERIFICATION LOG
RANDOM
Check the brief. How was data collected? Is this satisfied?
INDEPENDENCE (10%)
Are the two groups independent? Are samples small enough relative to population?
NORMALITY / COUNTS
Show the Success/Failure check for BOTH groups.
Verification Complete?
Proceed to Phase 2 (Analysis & Boardroom Briefing) only if conditions are met.
Not Authorized
Bistro Battle Master Key Simulation Master Key
Project: Alpha-Bistro (Phase 1 & 2)
Teacher Reference
Phase 1: Data Cleaning & Protocol
Group A (Classic)
Handed In: 400Invalid: 10\(n_1\): 390\(x_1\): 240\(\hat{p}_1\): 0.615
Group B (Speed)
Handed In: 450Invalid: 20\(n_2\): 430\(x_2\): 315\(\hat{p}_2\): 0.733
Protocol Verification
Test Choice: Two-proportion Z-test. (Categorical data: 5-star or not).
Alternative: \(H_a: p_B > p_A\) (Speed menu is better).
Phase 1: Condition Check Key
Random: Satisfied. Brief states "surveys were collected randomly over a 30-day period."
Independence (10%): Satisfied. Samples (390/430) are less than 10% of "thousands of monthly customers."
Large Counts (Normal): Satisfied.
Phase 2: Mathematical Execution
Pooled Proportion (\(\hat{p}_c\)) = 555 / 820 \(\approx\) 0.677
SE = \(\sqrt{0.677(0.323)(1/390 + 1/430)}\) \(\approx\) 0.0327
Z = (0.733 - 0.615) / 0.0327 \(\approx\) 3.61
P-Value (Z > 3.61) \(\approx\) 0.0001
Pedagogical Note: Students may forget to "pool" the proportions for the Standard Error. Remind them that for a 2-prop Z-test assuming \(p_1 = p_2\) (the null), pooling provides the best estimate of that shared population proportion.
Boardroom Briefing Guide Final Presentation Template
Boardroom Briefing
Project Alpha-Bistro: Executive Summary
1. THE MATH
Standard Error for Two Proportions
\(SE = \sqrt{\hat{p}_c(1-\hat{p}_c)(\frac{1}{n_1} + \frac{1}{n_2})}\)
Test Statistic (Z)
P-Value
Decision Protocol
Compare P-value to \(\alpha\). Circle your action:
REJECT \(H_0\)
FAIL TO REJECT \(H_0\)
2. THE CONCLUSION
Formal Statistical Conclusion
"Based on our p-value of... we [reject/fail to reject]..."
Executive Recommendation
Translate your findings for the CEO. Should we roll out the Speed-Menu worldwide? What is the risk of being wrong?
Boardroom Defense (Peer Review Prep)
A fellow consultant challenges your choice of test. They argue you should have used a paired t-test because both groups are from the same restaurant brand. Why are they wrong?
Potential Type I Error
If we reject the null but the Speed-Menu is actually a failure, what happens to the brand?
Potential Type II Error
If we fail to reject but the Speed-Menu is actually a success, what happens to our revenue?
Consultant Audit Rubric CONSULTANT AUDIT
Quality Assurance Performance Review
Final Score:
/ 25
Competency Superior (5) Satisfactory (3) Deficient (1) MODEL SELECTION & PROTOCOL Test chosen (2-Prop Z) is correct. Protocol is detailed and logical. Test is correct but justification is vague or slightly flawed. Incorrect test selected (e.g., T-test used for proportions). CONDITION VERIFICATION Random, 10%, and Large Counts verified with explicit math for both groups. Conditions mentioned but math is missing or one condition is missed. Significant conditions skipped or failed without notice. STATISTICAL COMPUTATION Pooled proportion, SE, Z-stat, and P-value are all calculated correctly. Calculation present with minor rounding or formula entry error. P-value is mathematically impossible or missing entirely. INTERPRETATION & DECISION Action is correct based on P vs \(\alpha\). Conclusion is in context of the Bistro. Action is correct but conclusion lacks context or is too robotic. Conclusion contradicts the p-value result or is absent. EXECUTIVE RECOMMENDATION Clear advice for the CEO. Analysis of Type I/II risks is profound and accurate. Advice given but risk analysis is shallow or generic. No recommendation or recommendation ignores statistical findings.
Auditor Comments
Internal Certification
By signing this audit, the reviewer confirms that the statistical findings presented are robust and the model choice has been independently verified.
Auditor Signature
Boardroom Report Key Simulation Answer Key
Project: Alpha-Bistro (Lesson 5)
Authorized Only
1. Data Scrubbing Results
Group A (Classic)
\(n_1\): 390 (400 total - 10 invalid)\(x_1\): 240 (5-star ratings)\(\hat{p}_1\): 0.615 (61.5%)
Group B (Speed)
\(n_2\): 430 (450 total - 20 invalid)\(x_2\): 315 (5-star ratings)\(\hat{p}_2\): 0.733 (73.3%)
2. Inference Calculation Trace
Pooled Proportion (\(\hat{p}_c\))
555 / 820 = 0.677
Standard Error (\(SE\))
SE \(\approx\) 0.0327
Test Statistic (\(Z\))
Z \(\approx\) 3.61
P-Value
P \(\approx\) 0.0001
3. Decision & Recommendations
Final Conclusion
"Because our P-value (0.0001) is less than \(\alpha\) (0.05), we reject the null hypothesis . We have convincing evidence that the 'Speed-Menu' leads to a significantly higher proportion of 5-star ratings than the 'Classic-Menu'."
Type I Error Risk
We roll out the Speed-Menu globally, spending millions, only to realize that the 5-star spike was due to 'novelty bias' and the menu doesn't actually improve satisfaction long-term.
The "Paired" Defense Note
Students should argue that even though they are the same brand, the customers are different individuals in different locations. There is no natural link (like before/after) between a diner in Group A and a diner in Group B.
Grading Priority
Focus on the Conditions Verification . Most students can plug numbers into a calculator; the true consultant skill is knowing when the numbers can't be trusted (e.g., if the 10% condition or Large Counts weren't met).
