Simulation Showdown Slides Simulating the Difference
When is a difference actually DIFFERENT?
Statistics & Probability
12th Grade
The Taste Test Challenge
In a blind taste test, 18 out of 30 people prefer Soda A.
In a different group, 12 out of 30 prefer Soda B.
"Is Soda A actually more popular, or is this 20% difference just lucky sampling?"
The Problem
Samples vary naturally.
How do we know if a difference is "real"?
Essential Question
How can we distinguish between genuine differences and variations caused by random sampling?
Simulation Logic
1
Assume No Difference
We assume both groups are exactly the same (the Null Hypothesis).
2
Re-shuffle the Data
If there's no difference, we can mix all people together and randomly assign them to groups.
3
Measure Results
How often does shuffling create a difference as big as the one we saw?
Today's Mission: The Shuffle
The Deck
60 cards representing our 60 subjects.
The Shuffle
Randomly assign to Group A and Group B.
The Difference
Calculate \(\hat{p}_1 - \hat{p}_2\) for every trial.
Outcome Dot Plot
.
-0.4 -0.2 0 0.2 0.4
This is our "Sampling Distribution"
Shuffle Showdown Activity Shuffle Showdown
Inquiry Activity: Simulating Differences in Proportions
Name:
Date:
The Marketing Mystery
A digital marketing company is testing two different "Buy Now" button designs to see which results in more sales. They ran a pilot test with 60 randomly assigned users:
Design A (Original)
Sample Size (\(n_1\)): 30 users
Sales (\(x_1\)): 18 users
Sample Proportion (\(\hat{p}_1\)): 0.60
Design B (New)
Sample Size (\(n_2\)): 30 users
Sales (\(x_2\)): 12 users
Sample Proportion (\(\hat{p}_2\)): 0.40
Observed Difference (\(\hat{p}_1 - \hat{p}_2\)): 0.20
1. The Research Question
Is the 20% difference in sales between Design A and Design B strong enough evidence to say Design A is truly better, or could this difference just be due to random chance (sampling variability)?
2. The Simulation Setup
To find out, we will assume there is no difference between the designs. If they were the same, we could combine all 60 users and 30 "sales" into one big pile and shuffle them.
1
Combine the Data:
Take 60 index cards. 30 cards say "SALE" and 30 cards say "NO SALE". Shuffle the deck thoroughly.
2
Deal the Groups:
Deal 30 cards into "Group 1" and 30 cards into "Group 2".
3
Calculate the Difference:
Find \(\hat{p}_1\) (Sales in Group 1 / 30) and \(\hat{p}_2\) (Sales in Group 2 / 30). Subtract them: \(\hat{p}_1 - \hat{p}_2\).
3. Data Collection
Perform 5 trials of the simulation and record your results below.
Trial Sales in Group 1 (\(x_1\)) \(\hat{p}_1\) (\(x_1 / 30\)) Sales in Group 2 (\(x_2\)) \(\hat{p}_2\) (\(x_2 / 30\)) Difference (\(\hat{p}_1 - \hat{p}_2\)) 1 2 3 4 5
4. Class Dot Plot
Contribute your results to the class dot plot. Sketch the final class distribution below.
-0.40 -0.30 -0.20 -0.10 0.00 0.10 0.20 0.30 0.40
Difference in Proportions (\(\hat{p}_1 - \hat{p}_2\))
5. Analysis
A. How many of the class's simulated trials resulted in a difference of 0.20 or greater?
B. Based on your answer above, what is the approximate probability (p-value) of getting a difference of 0.20 or more purely by chance?
C. Final Verdict: Do we have strong evidence that Design A is better? Explain why or why not.
Simulation Facilitator Guide Facilitator Guide
Lesson 1: Simulating the Difference
Teacher Resource
Learning Objective
Students will build a conceptual understanding of the sampling distribution of \(\hat{p}_1 - \hat{p}_2\) and estimate a p-value through randomization (shuffling).
Materials
60 index cards per group
Red/Black markers (for labeling)
Chart paper (for class dot plot)
Student Activity Sheet
Facilitation Timeline
01
The Hook (10 mins)
Use the Simulation Showdown Slides to present the Soda Taste Test. Ask: "Is a 20% difference impressive?" Get students to predict if it could happen by chance. Don't give them the answer yet!
02
The "Why" of Shuffling (5 mins)
Explain that if the button designs don't matter (the Null), then whether a person "buys" or "doesn't buy" is fixed. The only thing that changes is which group we happen to assign them to. Shuffling simulates this random assignment.
03
The Activity (20 mins)
Distribute index cards. Have students label 30 as "SALE" and 30 as "NO SALE".difference , not just the raw counts.
04
Class Discussion (15 mins)
Aggregate results on a large dot plot. Most dots should be near 0. If 0.20 appears in roughly 10-15% of trials, it's "plausible" by chance. If it's rare (<5%), it's "statistically significant."
Critical Discussion Prompts
Question:
"Where is the center of our class dot plot? Why does that make sense?"
Expected Response:
"The center is 0. This makes sense because we assumed there was no difference, so on average, the groups should be equal."
Question:
"What would the plot look like if we had 1,000 users instead of 60?"
Expected Response:
"The dots would be much more tightly packed around 0. Larger samples have less variability."
Next Steps
In Lesson 2, we will move from "shuffling cards" to "shuffling math." We'll introduce the Standard Error formula, which mathematically predicts the spread of the dot plot we made today.
Interval Insight Slides The Confidence Gap
Estimating the True Difference
Confidence Intervals
Two Samples
The Newsroom Dilemma
Poll in City A: 45% support the park tax.
Poll in City B: 52% support the park tax.
"The headline says City B is more supportive. But with a margin of error of 5%, is there actually a difference?"
Today's Goal
Instead of just "yes/no," we want to estimate how much of a difference there really is.
Construct 2-Sample z-Intervals
The Anatomy of the Interval
\[ (\hat{p}_1 - \hat{p}_2) \pm z^* \times SE_{\text{diff}} \]
Point Estimate
The difference between our two samples.
Critical Value
Determined by confidence level (e.g., 1.96 for 95%).
Standard Error
How much we expect the difference to vary.
The 2-Sample SE Formula
To get the standard error of a difference, we add the variances of the two separate samples.
\[ SE_{\text{diff}} = \sqrt{\frac{\hat{p}_1(1-\hat{p}_1)}{n_1} + \frac{\hat{p}_2(1-\hat{p}_2)}{n_2}} \]
Variance is additive! Even if we subtract proportions, we add the uncertainty.
Large samples (\(n\)) decrease the SE, making our interval narrower and more precise.
The "Can I Use This?" Checklist
Random
Data must come from independent random samples or random assignment.
Independent
10% condition: Sample size \(n\) must be less than 10% of the population.
Large Counts
\(n\hat{p} \ge 10\) and \(n(1-\hat{p}) \ge 10\) for both samples.
If these fail, our Normal-based formula is "broken."
Precision Practice Worksheet Precision Practice
2-Sample z-Intervals for Proportions
Name:
Date:
Formula Quick Reference
\[ (\hat{p}_1 - \hat{p}_2) \pm z^* \sqrt{\frac{\hat{p}_1(1-\hat{p}_1)}{n_1} + \frac{\hat{p}_2(1-\hat{p}_2)}{n_2}} \]
Confidence Levels (\(z^*\)):
90%: 1.645 | 95%: 1.96 | 99%: 2.576
Conditions: Random, Independent, Large Counts (\(n\hat{p} \ge 10\), \(n(1-\hat{p}) \ge 10\)).
1
The Social Media Divide
A researcher wants to estimate the difference in the proportion of teens (13-17) and adults (18+) who use TikTok daily. A random sample of 200 teens found 140 daily users. A separate random sample of 300 adults found 150 daily users.
A. Calculate the point estimate (\(\hat{p}_1 - \hat{p}_2\)) and the Standard Error (\(SE_{\text{diff}}\)) for this data.
B. Verify the "Large Counts" condition for both groups.
C. Construct a 95% confidence interval for the difference in proportions.
D. Interpret the interval in the context of the problem.
2
Remote Work Productivity
A company CEO surveys two departments to see who prefers remote work. In Marketing, 32 out of 40 employees prefer remote. In Engineering, 45 out of 60 prefer remote.
A. Check the conditions for inference. Is it appropriate to construct a 2-sample z-interval here? Explain.
B. Construct a 90% confidence interval for the difference (Marketing - Engineering).
C. Does the interval provide convincing evidence that Marketing has a higher preference for remote work than Engineering? Why or why not?
Think Deeply
If a 95% confidence interval for the difference between two proportions is (-0.04, 0.12), what does the fact that zero is inside the interval tell us about the population parameters?
Precision Practice Answer Key Answer Key
Precision Practice: 2-Sample Intervals
1. Social Media Divide (TikTok)
A. Estimates & SE:
\(\hat{p}_1 (\text{Teens}) = 140/200 = 0.70\)
\(\hat{p}_2 (\text{Adults}) = 150/300 = 0.50\)
\(\hat{p}_1 - \hat{p}_2 = 0.70 - 0.50 = 0.20\)
\(SE = \sqrt{\frac{0.7(0.3)}{200} + \frac{0.5(0.5)}{300}} = \sqrt{0.00105 + 0.000833} \approx 0.0434\)
B. Large Counts Condition:
Teens: \(200(0.7) = 140 \ge 10\); \(200(0.3) = 60 \ge 10\). (Met)
Adults: \(300(0.5) = 150 \ge 10\); \(300(0.5) = 150 \ge 10\). (Met)
C. 95% Confidence Interval:
\(0.20 \pm 1.96(0.0434) = 0.20 \pm 0.0851\)
(0.1149, 0.2851)
D. Interpretation:
We are 95% confident that the true difference in the proportion of teens and adults who use TikTok daily is between 11.5% and 28.5%.
2. Remote Work
A. Condition Check:
Random: Not explicitly stated as random samples, but could be a census of the departments. If results are meant to generalize, randomization is needed.
Large Counts: Marketing Successes (32) and Failures (8). Condition Fails for Marketing failures (8 < 10). A z-interval may not be accurate.
B. 90% Confidence Interval (Proceeding despite failure):
\(\hat{p}_1 = 0.80\), \(\hat{p}_2 = 0.75\). Diff = 0.05.
\(SE = \sqrt{\frac{0.8(0.2)}{40} + \frac{0.75(0.25)}{60}} = \sqrt{0.004 + 0.003125} \approx 0.0844\)
\(0.05 \pm 1.645(0.0844) = 0.05 \pm 0.1388 \rightarrow \mathbf{(-0.0888, 0.1888)}\)
C. Evidence:
No. Since the interval contains 0, it is plausible that there is no difference in preference between the two departments.
Think Deeply Solution
If zero is in the interval, it means that \(p_1 = p_2\) is a plausible value for the population difference. Therefore, we do not have enough evidence to claim that the two population proportions are truly different.
Significance Sprint Slides Significance Sprint
The 2-Sample z-Test for Proportions
Hypothesis Testing
The Pooled Proportion
Proof of Impact
A health app developer claims their new push notifications increased user activity from 22% to 28%.
"The board of directors wants proof. Is this 6% increase statistically significant, or is it just 'noise'?"
Today's Protocol
We use a formal process to calculate the exact probability (p-value) of seeing this difference if the app actually did nothing.
The "Pooled" Logic
Confidence Intervals
We don't know the truth, so we use each sample's individual proportion (\(\hat{p}_1, \hat{p}_2\)).
Hypothesis Tests
We assume \(H_0: p_1 = p_2\). If they are equal, why use two different estimates? We combine them!
Pooled Proportion (\(\hat{p}_c\))
\[ \hat{p}_c = \frac{x_1 + x_2}{n_1 + n_2} \]
"Total successes divided by total subjects."
The z-Statistic Formula
\[ z = \frac{(\hat{p}_1 - \hat{p}_2) - 0}{\sqrt{\hat{p}_c(1-\hat{p}_c)(\frac{1}{n_1} + \frac{1}{n_2})}} \]
This counts how many standard deviations our observed difference is from the null hypothesis of Zero.
The Inference Routine
1. State
Hypotheses (\(H_0, H_a\)), parameters (\(p_1, p_2\)), and significance level (\(\alpha\)).
2. Plan
Identify the test (2-sample z-test) and check conditions (Random, 10%, Large Counts).
3. Do
Calculate \(\hat{p}_c\), the z-statistic, and the p-value using technology or table.
4. Conclude
Compare p-value to \(\alpha\). Reject or fail to reject \(H_0\). Write in context.
Pool Party Worksheet Pool Party
Mastering the 2-Sample z-Test
Name:
Date:
Testing Tools
\[ \hat{p}_c = \frac{x_1 + x_2}{n_1 + n_2} \]
\[ z = \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}_c(1-\hat{p}_c)(\frac{1}{n_1} + \frac{1}{n_2})}} \]
Remember: We only use the "Pooled" proportion for Hypothesis Tests where \(H_0: p_1 = p_2\).
The Spirit Challenge
The student council wants to know if offering a "Free Pizza" prize increases student participation in Dress-Up Days. They randomly assign two homeroom floors to different conditions:
Floor A (Pizza Prize)
Sample Size (\(n_1\)): 100 students
Participated (\(x_1\)): 65 students
Sample Proportion (\(\hat{p}_1\)): 0.65
Floor B (No Prize)
Sample Size (\(n_2\)): 120 students
Participated (\(x_2\)): 48 students
Sample Proportion (\(\hat{p}_2\)): 0.40
Perform a 2-sample z-test at the \(\alpha = 0.05\) level to determine if the prize works.
Step 1: State
Identify your parameters, hypotheses (\(H_0\) and \(H_a\)), and significance level.
Step 2: Plan
Name the test and check the Random, 10%, and Large Counts conditions.
Step 3: Do
Calculate \(\hat{p}_c\), the z-statistic, and the p-value.
Step 4: Conclude
Make a decision about \(H_0\) and write your conclusion in context.
Critical Thinking
Suppose the student council found a p-value of 0.07. Would their conclusion change? Why does the choice of \(\alpha\) (significance level) matter so much in real-world decision-making?
Pool Party Answer Key Answer Key
Pool Party: 2-Sample z-Test
Step 1: State
Parameters: \(p_1\) = True proportion of students on Floor A who participate; \(p_2\) = Floor B.
Hypotheses: \(H_0: p_1 = p_2\) (or \(p_1 - p_2 = 0\)); \(H_a: p_1 > p_2\) (or \(p_1 - p_2 > 0\)).
Significance Level: \(\alpha = 0.05\).
Step 2: Plan
Name: 2-Sample z-test for the difference of proportions.
Random: Stated random assignment of floors.
10%: 100 < 10% of all Floor A students; 120 < 10% of all Floor B students (assumed).
Large Counts: Group 1: 65, 35; Group 2: 48, 72. All are \(\ge 10\). Condition Met.
Step 3: Do
Pooled Proportion: \(\hat{p}_c = \frac{65+48}{100+120} = \frac{113}{220} \approx 0.5136\)
Test Statistic:
\[ z = \frac{0.65 - 0.40}{\sqrt{0.5136(0.4864)(\frac{1}{100} + \frac{1}{120})}} = \frac{0.25}{0.0677} \approx 3.69 \]
p-value: \(P(z > 3.69) \approx 0.0001\)
Step 4: Conclude
Since the p-value (0.0001) is less than \(\alpha = 0.05\), we reject the null hypothesis .
There is convincing evidence that offering a "Free Pizza" prize increases the proportion of students who participate in Dress-Up Days.
Thinking Deeply Solution
Yes, the conclusion would change. If the p-value (0.07) is greater than \(\alpha\) (0.05), we would fail to reject \(H_0\). Even though the sample difference is 25%, a p-value of 0.07 means such a difference could happen by chance about 7% of the time, which is not considered "statistically significant" at the 5% level.
Click Combat Slides Click Combat
Statistics in the Tech World: A/B Testing
A/B Testing
Business Analytics
The Modern Lab
Tech giants like Google, Amazon, and Netflix run thousands of experiments every day.
The Method:
Randomly split incoming traffic. Version A gets the "Control." Version B gets the "Variant."
The Metric:
Proportions!
Conversion Rate = (Orders / Visitors)
A
Standard UI
B
New UI
Compare proportions to find the winner.
Beyond the Formula
In the real world, you rarely calculate $z$ by hand. You read software output.
Test: Two Proportions Alpha: 0.05
Sample 1 (Control): \(n=4500\), \(x=450\)
Sample 2 (Variant): \(n=4500\), \(x=504\)
Estimate for difference: -0.012
z-statistic: -2.01
p-value: 0.044
Pooled proportion: 0.106
Identify the key values.
Compare p-value to \(\alpha\).
Make the call.
Big Samples, Tiny Differences
With millions of users, a difference of 0.001% can be statistically significant (p < 0.05).
The Million Dollar Question:
Is it Practically Significant? Is the gain worth the cost of the change?
Marketing Mastery Case Study Marketing Mastery
Case Studies in Digital A/B Testing
Team:
The Analyst's Desk
You are a Data Analyst for StreamLine E-Commerce. Your job is to look at the experimental output from the engineering team and decide which features should be deployed to millions of users.
Scenario 1
The Checkout Button
The team tested a Red Button against the current Blue Button. They want to see if Red increases the "Click-to-Checkout" rate.
Sample Data
Version Blue (Control): \(n=12,500\), \(x=1,250\)
Version Red (Variant): \(n=12,500\), \(x=1,335\)
Test Statistics
95% CI for Difference: (-0.014, -0.0002)
z-statistic: -2.21
p-value: 0.027
1. Based on the p-value, is the difference statistically significant at \(\alpha = 0.05\)?
2. Does the confidence interval include 0? What does this mean for the "winner"?
Scenario 2
Search Algorithm Update
The engineers updated the search algorithm to be faster. They want to know if it increases the "Search-to-Purchase" conversion rate.
Sample Data
Algorithm A (Old): \(n=45,000\), \(x=1,800\)
Algorithm B (New): \(n=45,000\), \(x=1,890\)
Test Statistics
95% CI for Difference: (-0.0044, 0.0004)
z-statistic: -1.61
p-value: 0.108
3. The CEO says: "B had 90 more sales than A! Switch to B immediately!" As an analyst, how do you respond using the p-value?
4. If we switched to B, would we be making a certain type of error? Explain.
The "Bottom Line" Discussion
Scenario 3: A massive test (\(n = 1 \text{ million per group}\)) shows a new logo increases click-through rate from 5.0% to 5.05%. This result is statistically significant (p < 0.01).
Why might the company decide NOT to change the logo despite the significance?
Marketing Mastery Answer Key Answer Key
Marketing Mastery: A/B Testing Analysis
Scenario 1: Checkout Button
1. Statistical Significance:
Yes. The p-value (0.027) is less than \(\alpha = 0.05\). This means the difference is statistically significant.
2. Confidence Interval & Winner:
The interval (-0.014, -0.0002) does not include 0. Since both values are negative, it indicates that \(\hat{p}_{Blue} - \hat{p}_{Red}\) is likely negative, meaning \(\hat{p}_{Red} > \hat{p}_{Blue}\). The Red button is the winner.
Scenario 2: Search Algorithm
3. CEO Response:
"While B had more sales in our sample, the p-value of 0.108 tells us that there is a 10.8% chance of seeing a difference this large purely due to random variation. This is above our 5% threshold, so we cannot conclude that B is truly better than A."
4. Type of Error:
If we switch to B, we would be making a Type I Error (False Positive): claiming there is an effect when there isn't one.
Scenario 3: Bottom Line Discussion
Even though the logo change is statistically significant, the practical significance is tiny (a 0.05% gain).
Reasons to stay with the old logo:
Cost of rebranding (updating all physical and digital assets).
Loss of brand recognition.
Customer confusion.
The "ROI" (Return on Investment) isn't high enough to justify the engineering/design work.
Truth or Trap Slides Truth or Trap
Errors, Ethics, and Interpretation
Statistical Errors
Interpretation
The Decision Matrix
TYPE I ERROR
The False Positive
We reject \(H_0\) and claim there's a difference, but in reality, there isn't one.
"Convicting an innocent person."
TYPE II ERROR
The False Negative
We fail to reject \(H_0\) and say there's no evidence, but in reality, there is a difference.
"Letting a guilty person go free."
Which Trap is Deadlier?
I
"We approve a drug that actually does nothing, wasting billions and potentially causing side effects."
II
"We reject a life-saving treatment because our sample wasn't large enough to detect the benefit."
Analyst Tip:
Before running a test, ask: "What is the cost of being wrong in either direction?" This determines our \(\alpha\) level.
Speaking Human
Nobody in the boardroom wants to hear about "fail to reject nulls." Use the Conclusion Formula:
"Since our p-value of [P] is [less/greater] than \(\alpha\), we [do/do not] have convincing evidence that [context of Ha]."
"The null is true."
"We don't have enough evidence."
Unit Summary
"We can never be 100% certain, but we can measure our uncertainty."
Simulation
Intervals
Hypothesis Tests
Decision Making
Decision Danger Exit Ticket Decision Danger
Exit Ticket: Error Analysis & Interpretation
Name:
1
Identify the Error
A researcher is testing a new fertilizer to see if it increases the proportion of seeds that sprout. In reality, the fertilizer does increase the sprout rate. However, the researcher's sample was small, and they failed to reject the null hypothesis.
Type I Error
Type II Error
Explain the consequence of this error in this specific context:
2
Write the Conclusion
A study compared the proportion of students who passed an exam using two different study guides. The result of the 2-sample z-test was a p-value of 0.082. The significance level was \(\alpha = 0.05\).
Write a formal conclusion in non-technical language:
3
Significance vs. Reality
A software update for a popular social media app is found to increase the proportion of users who stay on the app for more than 10 minutes from 30.0% to 30.1%. The p-value is 0.00001.
Is this result statistically significant? Is it likely to be practically significant for the company? Explain.
"Data is only as good as the decisions we make with it."
Decision Danger Answer Key Answer Key
Decision Danger: Exit Ticket Solutions
1. Identify the Error
Error Type: Type II Error (False Negative)
Consequence:
The company will miss out on a fertilizer that actually works. Farmers will not benefit from higher sprout rates, and the company loses potential revenue from a successful product.
2. Write the Conclusion
Sample Conclusion:
"Since our p-value of 0.082 is greater than the significance level of 0.05, we fail to reject the null hypothesis. We do not have convincing evidence that there is a difference in the proportion of students who pass the exam between the two different study guides."
3. Significance vs. Reality
Statistical Significance: Yes. Because the p-value (0.00001) is extremely low, the results are highly statistically significant.
Practical Significance: No. A 0.1% increase in time spent (from 30.0% to 30.1% of users) is likely too small to impact the company's bottom line or user experience in a meaningful way, despite being "mathematically" real.
End of Unit: Proportion Power Play
