Sampling Logic Slides Sampling Logic
Foundations of SRS & Estimator Properties
The Fundamental Goal
In statistical inference, we aim to characterize a population parameter \(\theta\) using only a subset \(S \subset P\).
"The sample is a window, but how clear is the glass?"
Essential Question:
How do mathematical properties of estimators influence the validity of our inferences?
1
Definition of Simple Random Sampling (SRS)
2
Mathematical Expectation & Unbiasedness
3
Asymptotic Properties: Consistency
Formalizing SRS
A sampling design is Simple Random Sampling if every subset of size \(n\) has the same probability of being selected:
\[ P(S) = \frac{1}{\binom{N}{n}} \]
Equal probability for all elements
Independence between selections (SRSWR)
Unbiasedness of the Mean
Does the average of our averages hit the mark?
\(E[\bar{X}] = E\left[\frac{1}{n} \sum_{i=1}^n X_i\right]\)
Through the Linearity of Expectation , we derive: \[ E[\bar{X}] = \frac{1}{n} \sum_{i=1}^n E[X_i] = \frac{1}{n} \cdot n\mu = \mu \]
The Reality Check
Unbiasedness is a property of the procedure , not a single sample. A single sample will almost certainly differ from \(\mu\). Bias is the systemic deviation of the sampling distribution's mean from the parameter.
Asymptotics & Simulation
As \(n \to \infty\), the sampling distribution \(\bar{X}_n\) converges to a Normal distribution, regardless of the population's shape.
Monte Carlo Simulation Logic
Define a skewed population (e.g., Exponential).
Draw 10,000 samples of size \(n\).
Calculate \(\bar{x}\) for each sample.
Plot the distribution of \(\bar{x}\).
Increase \(n\) and observe the variance decrease.
Estimator Properties Worksheet Estimator Properties
Lesson 1: Sampling Foundations & Mathematical Logic
Name:
Date:
Part 1: Proof of Unbiasedness
Let \(X_1, X_2, \dots, X_n\) be a simple random sample with replacement (SRSWR) from a population with mean \(\mu\) and variance \(\sigma^2\). Define the sample mean estimator as \(\bar{X} = \frac{1}{n} \sum_{i=1}^n X_i\).
1.1 Mathematically derive the expected value \(E[\bar{X}]\) and show that it is an unbiased estimator of \(\mu\). State each property of expectation you utilize.
1.2 Derive the variance of the estimator \(Var(\bar{X})\). How does this derivation change if the sampling is performed without replacement (SRSWOR) in a finite population of size \(N\)? Briefly explain the Finite Population Correction (FPC).
Part 2: Consistency & Convergence
An estimator \(\hat{\theta}_n\) is consistent if it converges in probability to \(\theta\) as \(n \to \infty\).
2.1 Use Chebyshev's Inequality to argue that \(\bar{X}\) is a consistent estimator of \(\mu\). What condition must the variance \(\sigma^2\) satisfy for this proof to hold?
Part 3: Monte Carlo Pseudocode
You are tasked with validating the Central Limit Theorem for a Chi-Square population (\(df=1\)), which is highly skewed.
3.1 Write the pseudocode for a Monte Carlo simulation that generates a sampling distribution of the mean for \(n=5, 30, 100\). Specify how you would visualize the "narrowing" of the distribution and the approach to normality.
# Pseudocode / Logic steps here...
Estimator Properties Answer Key Answer Key
Lesson 1: Estimator Properties & Logic
Teacher Resource
Part 1: Theoretical Derivation
1.1 Proof of Unbiasedness
\[ E[\bar{X}] = E\left[\frac{1}{n} \sum_{i=1}^n X_i\right] \]
By linearity of expectation , we pull the constant out and distribute the expectation over the sum:
\[ = \frac{1}{n} \sum_{i=1}^n E[X_i] \]
Since each \(X_i\) is drawn from the same population, \(E[X_i] = \mu\) for all \(i\):
\[ = \frac{1}{n} (n\mu) = \mu \]
Note: This holds for both SRSWR and SRSWOR because the linearity of expectation does not require independence.
1.2 Variance Derivation
For SRSWR (Independence):
\[ Var(\bar{X}) = Var\left(\frac{1}{n} \sum X_i\right) = \frac{1}{n^2} \sum Var(X_i) = \frac{1}{n^2} (n\sigma^2) = \frac{\sigma^2}{n} \]
For SRSWOR (Dependence):
In finite populations, selections are not independent. The variance includes a covariance term, resulting in:
\[ Var(\bar{X})_{WOR} = \frac{\sigma^2}{n} \left( \frac{N-n}{N-1} \right) \]
The term \(\frac{N-n}{N-1}\) is the Finite Population Correction (FPC) . As \(N \to \infty\), FPC \(\to 1\), and the variance converges to the SRSWR case.
Part 2: Consistency
By Chebyshev's Inequality: \(P(|\bar{X} - \mu| \geq \epsilon) \leq \frac{Var(\bar{X})}{\epsilon^2}\)
Substitute \(Var(\bar{X}) = \frac{\sigma^2}{n}\):
\[ P(|\bar{X} - \mu| \geq \epsilon) \leq \frac{\sigma^2}{n\epsilon^2} \]
As \(n \to \infty\), the right side goes to 0 for any \(\epsilon > 0\). Thus, \(\bar{X} \xrightarrow{p} \mu\). Condition: The population variance \(\sigma^2\) must be finite.
Part 3: Simulation Logic
# Logical steps for student code:
Selection Bias Slides Bias Mechanics
Deconstructing Selection Bias & Proofs
The Definition of Bias
Bias is defined as the difference between the expected value of an estimator and the true parameter value:
\[ Bias(\hat{\theta}) = E[\hat{\theta}] - \theta \]
Selection Bias: Occurs when the probability of inclusion is correlated with the outcome variable.
Note: Increasing sample size \(n\) does not reduce bias in non-probability designs. It only narrows the confidence interval around a wrong value.
A Classic Failure: 1936
The Poll: 2.4 million responses predicted Alf Landon would beat FDR by 57% to 43%.
The Result: FDR won with 62% of the vote.
Where did the math fail?
Sampling frame: Car registrations and phone books (wealthy bias).
Non-response bias: Enthusiastic opponents of FDR responded at higher rates.
"The Big Data Fallacy"
More data \(\neq\) Better inference
Selection Probabilities
Let \(I_i\) be an indicator variable where \(I_i = 1\) if element \(i\) is selected. For an unbiased estimator, we require:
\[ P(I_i = 1) = \pi_i > 0 \quad \forall i \in P \]
In non-probability sampling, \(\pi_i\) is often unknown and varies with \(Y_i\). The expected value of the sample mean becomes: \[ E[\bar{y}_{sample}] = \frac{\sum \pi_i Y_i}{\sum \pi_i} \neq \mu \]
The Critical Takeaway
"Mathematical rigor allows us to quantify exactly how much our non-random choices distort our view of reality."
Bias
A property of the center of the distribution. Cannot be fixed by larger \(n\).
Variance
A property of the spread. Decreases with \(\sqrt{n}\).
Bias Proofs Worksheet Bias Proofs
Lesson 2: Selection Mechanisms & Formal Logic
Name:
Date:
Part 1: The Distorted Expectation
Consider a population \(P\) of size \(N\). Let \(Y_i\) be the value for individual \(i\). Let \(I_i \in \{0, 1\}\) be the indicator that \(i\) is selected for the sample, with \(P(I_i = 1) = \pi_i\).
1.1 Show that for a non-probability sample, the expected value of the sample mean \(\bar{y}_s = \frac{\sum I_i Y_i}{\sum I_i}\) can be approximated by \(E[\bar{y}_s] \approx \frac{\sum \pi_i Y_i}{\sum \pi_i}\).
Part 2: Bias and Covariance
"Bias exists when those we hear from are fundamentally different from those we don't."
2.1 Using the approximation from 1.1, prove that the bias of the sample mean is proportional to the covariance between the selection probabilities \(\pi_i\) and the outcome values \(Y_i\).
Hint: Use the identity \(Cov(\pi, Y) = E[\pi Y] - E[\pi]E[Y]\).
Part 3: 1936 Breakdown
In the 1936 poll, \(\pi_i\) was significantly higher for individuals who owned telephones and cars. Let \(Y_i = 1\) if voting for FDR, and \(Y_i = 0\) otherwise.
3.1 Based on the proof in Part 2, explain why the Literary Digest poll was biased downward for FDR. Explicitly state the sign of \(Cov(\pi, Y)\) and justify it based on the socioeconomic conditions of the Great Depression.
Bias Proofs Answer Key Answer Key
Lesson 2: Bias Proofs & Mechanisms
Teacher Resource
Part 1: Distorted Expectation
1.1 Proof Sketch:
We use the first-order Taylor expansion (Delta Method) for the ratio of two random variables:
\[ E\left[\frac{\sum I_i Y_i}{\sum I_i}\right] \approx \frac{E[\sum I_i Y_i]}{E[\sum I_i]} \]
By linearity of expectation: \[ \frac{\sum E[I_i] Y_i}{\sum E[I_i]} = \frac{\sum \pi_i Y_i}{\sum \pi_i} \]
This is the 'Selection-Adjusted Mean'. It only equals \(\mu = \frac{1}{N}\sum Y_i\) if \(\pi_i\) is constant or uncorrelated with \(Y_i\).
Part 2: Covariance Proof
Let \(\bar{\pi} = \frac{1}{N} \sum \pi_i\). The bias is \(E[\bar{y}_s] - \mu\).
\[ E[\bar{y}_s] = \frac{\frac{1}{N}\sum \pi_i Y_i}{\bar{\pi}} = \frac{E[\pi Y]}{\bar{\pi}} \]
Using the covariance identity \(Cov(\pi, Y) = E[\pi Y] - E[\pi]E[Y]\):
\[ E[\pi Y] = Cov(\pi, Y) + \bar{\pi}\mu \]
Substitute back into the mean equation:
\[ E[\bar{y}_s] = \frac{Cov(\pi, Y) + \bar{\pi}\mu}{\bar{\pi}} = \mu + \frac{Cov(\pi, Y)}{\bar{\pi}} \]
Result: \(Bias = \frac{Cov(\pi, Y)}{\bar{\pi}}\)
Part 3: 1936 Analysis
3.1 Sociosocioeconomic Explanation
During the Great Depression, owning a car or telephone was a strong proxy for wealth. Wealthier voters (high \(\pi_i\)) were significantly more likely to vote for the Republican Alf Landon (\(Y_i = 0\)). Poorer voters (low \(\pi_i\)) were more likely to vote for the Democrat FDR (\(Y_i = 1\)).
Since \(\pi_i\) is high when \(Y_i\) is low, and \(\pi_i\) is low when \(Y_i\) is high, the covariance \(Cov(\pi, Y)\) is negative .
Conclusion: A negative covariance leads to a negative bias, mathematically explaining why the poll underestimated FDR's support.
Frame Error Slides Frame Fidelity
Target Populations vs. Sampling Frames
The Coverage Problem
The Target Population is the set of elements we wish to study. The Sampling Frame is the list or mechanism used to select them.
Coverage Error
Exists when the frame does not perfectly match the target population.
TARGET
FRAME
Intersection
Mathematical Impact
Let \(W\) be the proportion of the population covered by the frame. The total population mean \(\mu\) is:
\[ \mu = W\mu_C + (1-W)\mu_{NC} \]
C:
Covered elements (reachable by frame)
NC:
Non-covered elements (systemically excluded)
Bias = \((1-W)(\mu_C - \mu_{NC})\). The magnitude of bias depends on the size of the non-covered group and the difference in their means.
Modern Challenges
Cell-Only Households
Traditional RDD (Random Digit Dialing) frames exclude households without landlines—often younger, urban, or lower-income.
Digital Divide
Online panels exclude those without reliable internet access, skewing estimates on technology use or political engagement.
Unlisted Elements
Incomplete lists (e.g., membership rosters) create undercoverage ; duplicate entries create overcoverage .
Mitigation Strategies
Dual-Frame Designs: Combining RDD and Address-Based Sampling (ABS).Post-Stratification: Re-weighting the sample to match known population benchmarks.In-Field Enumeration: Creating a frame manually for a specific cluster. Frame Design Activity Frame Design
Lesson 3: Mitigating Coverage Error in Practice
Group Name:
The Scenario: The Gig Economy Audit
The City Council wants to estimate the average hourly earnings of app-based delivery drivers (e.g., DoorDash, UberEats) in the metropolitan area. This population is highly mobile, has high turnover, and many drivers work for multiple platforms simultaneously.
Target Population
All active app-based delivery workers residing in the city.
Constraint
Individual companies refuse to share their full driver lists due to privacy concerns and proprietary data.
1 Defining Potential Frames
Identify two distinct potential sampling frames (e.g., social media groups, physical wait zones, tax records). For each, describe the Selection Probability (\(\pi_i\)) mechanism.
Frame A
Frame B
2 Coverage Error Analysis
For Frame A above, identify the primary source of Undercoverage . Who is excluded, and how might their hourly earnings systematically differ from the covered group?
3 Proposed Mitigation Design
Design a Dual-Frame or Multi-Stage sampling design to reach the target population more accurately. How will you handle Overcoverage (e.g., drivers who appear in multiple frames)?
Mathematical Constraint Check:
"If you use two frames, Frame 1 and Frame 2, how do you adjust the weights of individuals who belong to the intersection \(F_1 \cap F_2\)?"
Big Data Slides Big Data Paradox
Precision without Accuracy
Quantity \(\neq\) Quality
The Big Data Paradox refers to the phenomenon where a larger sample size leads to more confidence in an incorrect result.
"As \(n\) increases, the standard error shrinks, but the selection bias remains constant—or even increases."
Why now?
Administrative data, web-scraping, and social media datasets often lack a controlled sampling mechanism, making them "large non-probability samples."
Mean Squared Error (MSE)
For any estimator \(\hat{\theta}\):
\[ MSE(\hat{\theta}) = Var(\hat{\theta}) + [Bias(\hat{\theta})]^2 \]
In Probability Samples:
Bias \(\approx 0\), so MSE \(\approx Var\). Variance \(\to 0\) as \(1/n\).
In Big Data (Non-Prob):
Variance \(\to 0\) rapidly, but Bias\(^2\) dominates the error.
Effective Sample Size
Xiao-Li Meng (2018) showed that the actual "useful" information in a large non-probability sample is equivalent to a much smaller random sample.
\[ n_{eff} \approx \frac{1}{\rho_{R,Y}^2} \]
Where \(\rho_{R,Y}\) is the correlation between response and outcome.
If \(\rho = 0.05\) (a tiny correlation!), a sample of 1 million people is as informative as a random sample of only 400 people.
The Big Data Warning
"It is better to have 1,000 observations you can trust than 100 million you don't understand."
Rigorous Design
Naïve Aggregation
Paradox Math Worksheet Paradox Math
Lesson 4: Quantifying the Big Data Paradox
Name:
Date:
Part 1: MSE Decomposition
You are comparing two surveys intended to estimate the population mean \(\mu\) of a variable with \(\sigma^2 = 1\).Survey A (Prob): SRS of size \(n = 1,000\). Unbiased.Survey B (Big Data): Non-probability sample of size \(n = 1,000,000\). Selection bias \(B = 0.05\).
1.1 Calculate the Mean Squared Error (MSE) for both surveys. Which survey provides a more accurate estimate of \(\mu\)? Show all calculations.
Part 2: Meng's n-eff
Meng's identity for the "data defect correlation" \(\rho_{R,Y}\) states that the relative error increase due to non-random sampling is proportional to \(\sqrt{N}\), where \(N\) is the total population size.
2.1 In a population of \(N = 250,000,000\), you have a non-probability sample that covers 1% of the population. If the correlation between response and outcome is a mere \(\rho = 0.01\), calculate the effective sample size \(n_{eff}\). Interpret the result in terms of "trustworthiness" of big data.
Part 3: The Use Case
3.1 Under what specific mathematical or structural conditions might the Big Data sample (Survey B) become preferable to the small probability sample (Survey A)? Consider scenarios involving extremely rare events or high-dimensional interactions.
Paradox Math Answer Key Answer Key
Lesson 4: Big Data Paradox Math
Teacher Resource
Part 1: MSE Comparison
Survey A (SRS, n=1,000):
\[ MSE_A = Var_A + Bias^2 = \frac{\sigma^2}{n} + 0^2 = \frac{1}{1,000} = 0.001 \]
Survey B (Big Data, n=1,000,000):
\[ MSE_B = Var_B + Bias^2 = \frac{\sigma^2}{n} + (0.05)^2 = \frac{1}{1,000,000} + 0.0025 = 0.000001 + 0.0025 = 0.002501 \]
Conclusion: Survey A is more accurate (\(MSE_A < MSE_B\)). Even with 1,000x more data, the small bias in Survey B makes it 2.5 times worse than the small random sample.
Part 2: Meng's n-eff
Using the simplified Meng identity: \(n_{eff} \approx \frac{1}{\rho^2}\)
For \(\rho = 0.01\):
\[ n_{eff} = \frac{1}{(0.01)^2} = \frac{1}{0.0001} = 10,000 \]
Interpretation: In a population of 250 million, a 1% non-probability sample (2.5 million people) with a tiny correlation of 0.01 is only as useful as a perfect random sample of 10,000 people. We have "thrown away" 99.6% of the data's potential due to selection bias.
Part 3: The Use Case
Big Data is preferable when:
The parameter of interest is a rare event (e.g., 0.01% occurrence) where a sample of 1,000 would yield zero observations.
We need high granularity (e.g., estimates for every single zip code), where the random sample would be too sparse to provide local estimates.
The goal is internal validity (e.g., comparing group A vs B within the sample) rather than external validity (generalizing to the population).
Simulation Project Slides Monte Carlo Mastery
Stress-Testing Sampling Designs
What is a Monte Carlo Study?
In theory, \(n \to \infty\). In reality, \(n = 30\).
A Monte Carlo study uses repeated sampling from a simulated population to evaluate how an estimator behaves under finite constraints or "messy" conditions.
Why Simulate?
Testing robustness to non-normality.
Comparing two competing estimators.
Validating bias under specific frame errors.
The Simulation Loop
1. Define Pop
Generate \(N=10^5\) values with known \(\mu, \sigma\).
2. Sample
Draw random sample of size \(n\).
3. Estimate
Calculate \(\hat{\theta}\) and store result.
4. Iterate
Repeat \(M=10,000\) times.
Your Mission
Design a study to answer one of the following:
Performance of the Median vs. Mean in a Log-Normal population.
Bias of the Ratio Estimator in small samples (\(n < 20\)).
Impact of Heteroscedasticity on sampling variance.
Deliverables
1. Clean, documented code (R/Python).
2. Histograms of sampling distributions.
3. Table of Bias, Variance, and MSE.
4. A 5-minute lightning talk.
Break the Theory
"Asymptotic theory is a map; simulation is the terrain. Go find where the map fails."
Good Luck, Data Architects
Monte Carlo Project Monte Carlo Project
Lesson 5: Culminating Simulation Study
Final Assessment
Objective
Students will design and execute a Monte Carlo simulation study to evaluate the properties of a specific estimator or sampling design. The goal is to move beyond asymptotic proofs and observe how statistical theory performs in finite, non-ideal samples.
Phase 1: Design Parameters
1. The Population
Choose a population distribution that challenges standard assumptions:
Skewed: Lognormal(\(\mu=0, \sigma=1\))Heavy-Tailed: Cauchy or Student-t with low dfBimodal: Mixture of two Normals
2. The Estimators
Compare at least two estimators for the same parameter:
Sample Mean vs. Sample Median
Standard Variance vs. Unbiased Variance (\(n\) vs \(n-1\))
Ratio Estimator vs. Difference Estimator
Phase 2: Technical Execution
Your simulation loop must include:
# Parameters
N_iterations = 10,000
sample_sizes = [10, 30, 100, 500]
# Execution
For each n in sample_sizes:
For i in 1 to N_iterations:
Draw sample; Calculate Estimates; Store Results
Phase 3: Required Metrics
For each sample size \(n\), calculate and report:
Empirical Bias
\( \text{Mean}(\hat{\theta}) - \theta \)
Empirical Variance
\( \text{Var}(\hat{\theta}) \)
Root MSE
\( \sqrt{Bias^2 + Var} \)
Deliverables
1. Technical Brief (3-4 pages)
Detailed methodology, results tables, and interpretation of findings.
2. Visualization Suite
Overlaid histograms of sampling distributions across different \(n\).
Simulation Assessment Rubric Project Rubric
Monte Carlo Simulation Assessment
Evaluation Criteria
Criterion Exemplary (4) Proficient (3) Developing (1-2) Technical Design Simulation loop is efficient, documented, and correctly implements sampling from the specified population. Iterations (\(M \geq 10,000\)) ensure stability. Logic is correct but code may be inefficient or sparsely documented. Iterations are sufficient to show general trends. Technical errors in sampling logic or estimator calculation. Insufficient iterations for stable results. Statistical Metrics All required metrics (Bias, Var, MSE, Relative Bias) are correctly calculated for all sample sizes. Theoretical values are provided for comparison. Most metrics are correct. Some minor errors in calculation or missing theoretical benchmarks. Major errors in calculating metrics. Fails to distinguish between variance and bias. Visualization High-quality plots with clear labeling, scales, and overlays (e.g., normal density). Effectively illustrates the convergence to normality or persistent bias. Clear plots but lacking professional polish or helpful overlays. Labels are present but may be generic. Poorly scaled or unlabeled plots. Visualization fails to communicate the sampling distribution properties. Theoretical Synthesis Insightfully links simulation results to classroom theory (CLT, Bias proofs, Big Data Paradox). Explains why the estimator behaved as it did. Correctly identifies results as matching or diverging from theory, but depth of explanation is limited. Superficial analysis. Does not connect numerical results to theoretical sampling properties.
Instructor Comments
Total Score:
/ 16
Pass
Revise
