Breaking the Line Slides Breaking the Line
The Limitations of Pearson's Correlation
Advanced Statistics & Probability Graduate Seminar
The Linearity Trap
Pearson's product-moment correlation coefficient (\(r\)) is the gold standard for measuring relationship strength.
"It measures the strength and direction of the linear relationship between two variables."
Assumption: Bivariate Normality
Assumption: No influential outliers
The Equation
\[ r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}} \]
Note how the covariance is normalized by individual variances.
Pathological Datasets: Anscombe's Quartet
Four datasets with identical summary statistics:
I: Simple Linear
II: Non-linear (Parabolic)
III: Linear with Outlier
IV: Leverage Point Effects
Mean of X / Y
9.0 / 7.5
Sample Variance
11.0 / 4.12
Correlation (\(r\))
0.816
Association vs. Linearity
A correlation of 0 does not mean there is no relationship.
Pearson only detects linear trends.
Non-monotonic relationships (circles, parabolas, sine waves) yield \(r \approx 0\).
Pearson is highly sensitive to outliers.
Visualization is Mandatory
Summary statistics can be manipulated to hide structural patterns. Statistical rigor requires:
Scatterplot inspection
Residual analysis
Check for monotonicity
The Sequence Map
1
Pearson Limitations
2
Rank Methods
3
Polynomials & Splines
4
Local Regression
5
Information Theory
Lesson 01: The Limitations of Pearson's Correlation
Graduate Statistics & Probability
Statistical Pathologies Teacher Guide Statistical Pathologies
Teacher Guide | Lesson 01
Sequence: Non-Linear Association
Instructional Overview
This lesson serves as the "anti-introduction" to bivariate analysis. While undergraduate courses emphasize the power of Pearson's \(r\), this graduate session highlights its frailty. The goal is to instill a deep skepticism of summary statistics when divorced from visualization.
Key Objectives
Identify the mathematical assumptions of Pearson's correlation.
Analyze Anscombe's Quartet to identify leverage and non-linearity.
Differentiate between "correlation" and "general association".
Prerequisite Knowledge
Basic variance and covariance calculations.
Familiarity with OLS regression residuals.
Normal distribution properties.
The Hook: Summary Stat Trap
Prompt 1: The Blind Analyst
"If a research paper reports a correlation of 0.82 with a significant p-value, can we conclude a strong relationship exists? Why or why not?"
Expected Insight: Students should mention outliers or non-linear structures (like in Anscombe's Quartet III or IV) that can artificially inflate \(r\) even if the underlying bulk of data shows a different pattern.
Prompt 2: The Symmetrical Lie
"Describe a relationship where the association is perfect but the Pearson correlation is exactly zero."
Expected Insight: Any symmetrical relationship where the covariance cancels out, such as \(y = x^2\) centered at zero or a full period of a sine wave.
Technical Breakdown: Anscombe's Quartet
Dataset Nature of the Pathology Remediation Strategy I (Linear) The "ideal" case. Assumptions of normality and homoscedasticity roughly hold. Pearson's \(r\) is appropriate here. II (Curved) Deterministic non-linear relationship. Correlation captures only the average trend. Polynomial regression or non-parametric correlation. III (Outlier) A single extreme outlier significantly alters the slope and correlation. Robust regression (M-estimators) or outlier removal. IV (Leverage) Vertical cluster with one distant leverage point creating a "fake" trend. Investigation of data collection errors; resampling.
Activity: The Quartet Replication
Group Challenge (15 mins)
Distribute the "Association Detectives" worksheet. Students must manually verify summary statistics for two "mystery" datasets. One is truly linear, one is pathological. They must calculate means and then visualize to find the truth.
Handout: Mystery Data Tools: Graph Paper / Calculator
© 2026 Lenny Educational Resources | Advanced Statistics Series
Association Detectives Worksheet Association Detectives
Bivariate Analysis Laboratory | Case No. 001
Researcher:
Date:
Part 1: The Numerical Mirage
Below are two datasets. Perform the summary statistics below and observe the results. Do not plot the data until instructed.
Dataset A
X Y 10 8.04 8 6.95 13 7.58 9 8.81 11 8.33 14 9.96
Dataset B
X Y 10 9.14 8 8.14 13 8.74 9 8.77 11 9.26 14 8.10
Workspace: Calculate Means and \(r\)
Dataset A \(\bar{x}\): _________________
Dataset A \(\bar{y}\): _________________
Dataset A \(r\): __________________
Dataset B \(\bar{x}\): _________________
Dataset B \(\bar{y}\): _________________
Dataset B \(r\): __________________
Part 2: The Visual Verdict
Sketch a rough scatterplot for each dataset in the grids below. Use the scales provided.
0 15 12
Dataset A Visualization
0 15 12
Dataset B Visualization
Conclusion & Analysis
Based on your visualizations, which dataset is better described by Pearson's \(r\)? Justify your answer using the concept of linearity and monotonicity .
Ranking the World Slides Ranking the World
Spearman’s \(\rho\) and Kendall’s \(\tau\)
Non-Parametric Bivariate Analysis Lesson 02
When Pearson Fails
Standard correlation assumes Interval or Ratio data and bivariate normality.
The Non-Parametric Solution
Use rank-based methods when:
Data is Ordinal (e.g., Likert scales, leaderboards).
Relationship is Monotonic but not linear.
Outliers are present and meaningful.
Monotonicity vs. Linearity
Linear
Monotonic
"As X increases, Y always increases (or always decreases), but not necessarily at a constant rate."
Spearman’s Rank Correlation (\(\rho\))
Essentially Pearson's \(r\) applied to ranks instead of raw values.
Simplified Formula (for no ties):
\[ \rho = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)} \]
Where \(d_i\) is the difference between the ranks of each observation.
The Algorithm
Rank observations for X.
Rank observations for Y.
Calculate difference \(d_i\) for each pair.
Square \(d_i\), sum them, and plug into formula.
Kendall’s Tau (\(\tau\))
Based on Concordant and Discordant pairs.
Concordant: Order of X matches order of Y.
Discordant: Order of X is opposite to order of Y.
\[ \tau = \frac{C - D}{\frac{1}{2} n (n - 1)} \]
Rho vs. Tau
\(\rho\) is better for "near-linear" data.
\(\tau\) is more robust to small samples and ties.
\(\tau\) has a better statistical interpretation in terms of probability.
Summary
Pearson
Parametric, Linear, Interval/Ratio
Spearman
Non-Parametric, Monotonic, Ordinal
Kendall
Non-Parametric, Concordance, Robust
Beyond Normality Teacher Guide Beyond Normality
Teacher Guide | Lesson 02
Sequence: Non-Linear Association
Core Concepts
Transitioning students from parametric to non-parametric statistics requires a shift in how they view "information." In Pearson's world, the specific distance between values matters. In the non-parametric world, only the relative position (rank) matters. This lesson builds intuition for that transition.
Lesson Flow
10 mins
The Leaderboard Hook
Present a scenario: Two judges rank 5 wines. Judge A gives scores [98, 97, 96, 95, 94]. Judge B gives [80, 79, 78, 77, 76]. Pearson would show high correlation, but the "intensity" is different. Does the absolute score matter, or just the order? Use this to introduce Ordinal data.
20 mins
Rho vs. Tau Deep Dive
Walk through the Spearman formula. Show how it's just the Pearson calculation on the ranks. Then, introduce Kendall's Tau as a more sophisticated "combinatorial" approach. Emphasize that Tau is often preferred in modern research for its robustness.
30 mins
Workshop: Ordinal Odds
Students work through the hand-calculation of Spearman's Rho. This is a crucial "aha" moment where they see how ties and outliers lose their power when ranked.
Common Misconceptions
"Spearman is always better than Pearson." Correction: Spearman has less statistical power than Pearson when data truly is normally distributed."Kendall's Tau and Spearman's Rho should yield the same value." Correction: They are on different scales. Tau is usually smaller in magnitude than Rho.
Ordinal Odds Answer Key
Task 1: Spearman Calculation
Data: [X: 1, 5, 10, 20] [Y: 2, 4, 15, 12]
Ranks: [RX: 1, 2, 3, 4] [RY: 1, 2, 4, 3]
Differences (\(d\)): [0, 0, -1, 1]
Result: \(\rho = 1 - \frac{6(2)}{4(15)} = 1 - \frac{12}{60} = 0.8\)
Task 2: Concordance Analysis
Pairs: (1,2), (5,4), (10,15), (20,12)
Concordant: 5 | Discordant: 1
Result: \(\tau = \frac{5-1}{6} = 0.67\)
© 2026 Lenny Educational Resources | Advanced Statistics Series
Ordinal Odds Worksheet Ordinal Odds
Rank-Based Correlation Workshop
Researcher:
Date:
Part 1: The Spearman Method
Calculate the Spearman Rank Correlation Coefficient (\(\rho\)) for the following dataset. Note that this data contains a significant outlier that would heavily bias a Pearson calculation.
X (Raw) Y (Raw) 1.2 15.5 3.5 12.2 4.1 35.8 12.5 48.2 95.0 (Outlier) 112.5
Calculation Space
\[ \rho = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)} \]
\(\sum d_i^2 = \) _________________
\(n = \) _________________
Final Spearman Coefficient (\(\rho\)): ________________________
Part 2: Concordance & Combinatorics
In Kendall's Tau, we examine every possible pair of data points to see if they are concordant (ordered the same way).
"A pair is concordant if \(x_i < x_j\) and \(y_i < y_j\) (or both greater)."
Identify the relationship for Point Pair (A, B):
Point A: (5, 10)
Point B: (8, 7)
Concordant
Discordant
Analytical Comparison
If you were analyzing clinical trial data with many tied ranks (e.g., pain scores from 1-5), why might you choose Kendall's Tau over Spearman's Rho?
Bending the Model Slides Bending the Model
Polynomials, Splines, and the Bias-Variance Tradeoff
Curvature in Bivariate Data Lesson 03
Linear vs. Polynomial
When scatterplots reveal curvature, we expand the feature space.
Polynomial Basis
\[ \hat{y} = \beta_0 + \beta_1 x + \beta_2 x^2 + ... + \beta_d x^d \]
This is still linear in parameters , allowing us to use Ordinary Least Squares (OLS) for estimation.
A cubic polynomial (\(d=3\)) capturing non-linear fluctuations.
Runge's Phenomenon
Higher degree (\(d\)) \(\neq\) Better model.
Runge's Phenomenon
High-degree polynomial interpolation oscillates wildly at the edges of the interval. The model "memorizes" noise instead of capturing the signal.
Bias-Variance Tradeoff
L
Low Degree (High Bias):
Underfits. Misses the true curvature.
H
High Degree (High Variance):
Overfits. Sensitive to minor fluctuations in training data.
The Spline Alternative
Instead of one global polynomial, fit several local polynomials joined at "knots."
Cubic Splines
Polynomials of degree 3 that are continuous and have continuous 1st and 2nd derivatives at the knots.
Knot 1 \(x=10\)
Knot 2 \(x=20\)
Knot 3 \(x=30\)
Why Splines?
More flexible than global poly.
Less oscillating at edges.
Intuitive "localized" modeling.
Model Selection
"Complexity should be as low as possible, but no lower."
Validation
Use Cross-Validation to find the optimal degree \(d\) or knot count.
Regularization
Penalize complexity (Ridge/Lasso) to manage high-variance models.
Flexibility Frontier Teacher Guide Flexibility Frontier
Teacher Guide | Lesson 03
Sequence: Non-Linear Association
Lesson Summary
This lesson introduces the first parametric attempt at non-linearity: Polynomial Regression. We then quickly pivot to why this approach is dangerous (Runge's Phenomenon) and how Splines offer a more stable, piece-wise alternative.
Implementation Guide
The Hook: Curve Fitting Challenge
Start by showing a plot of a complex wave (like a damped oscillation). Ask: "If you only had straight lines, how many would you need to model this perfectly?" This motivates the move to \(x^2\), \(x^3\), etc.
Demonstrating Runge's Phenomenon
Use a live simulation if possible, or show static slides of a 10th-degree polynomial. Point out the wild swings at the edges of the data range. Explain that polynomials are global —changing a single data point on the left can wildly change the curve on the right.
The "Spline" Logic
Use the analogy of a physical "spline"—a flexible strip of wood held down by weights (knots) used by shipbuilders. Splines are local . This is the conceptual bridge to the next lesson on LOESS.
Polynomial Pitfalls Answer Key
Section 1: Identifying Overfit
Scenario: Degree 8 polynomial for 10 data points.Diagnosis: Extreme overfit / High Variance. The model will have an \(R^2\) near 1.0 but perform poorly on out-of-sample data.
Section 2: Spline Constraints
Students must list the 3 requirements for a Cubic Spline at the knots:
Continuity: The pieces must touch at the knot (\(f_1(k) = f_2(k)\)).Smoothness (1st Deriv): The slope must be the same (\(f'_1(k) = f'_2(k)\)).Smoothness (2nd Deriv): The rate of change of slope must be the same (\(f''_1(k) = f''_2(k)\)).
Technical Note: Orthogonal Polynomials
Remind graduate students that raw polynomials (\(x, x^2, x^3\)) are often highly correlated with each other, leading to multicollinearity in regression. In practice, we use orthogonal polynomials (like Legendre polynomials) to stabilize the coefficients.
© 2026 Lenny Educational Resources | Advanced Statistics Series
Polynomial Pitfalls Worksheet Polynomial Pitfalls
Modeling Curvature and Flexibility
Researcher:
Date:
Part 1: The Bias-Variance Scan
Examine the three regression models below. For each, identify the Model Type (Linear, Quadratic, or High-Degree Poly) and the Risk Factor (Underfitting, Good Fit, or Overfitting).
Model: _________________
Risk: __________________
Model: _________________
Risk: __________________
Model: _________________
Risk: __________________
Part 2: Designing the Knots
Imagine you are modeling the population growth of a bacteria colony that faces a sudden nutrient shortage at Hour 12.
Where would you place a knot in your regression spline? Why?
A colleague suggests using a 7th-degree polynomial instead of a spline. What is the biggest danger they face in their predictions beyond the observed time range?
Workspace: Residual Sketch
Sketch what a residual plot looks like for an underfit linear model on parabolic data.
Continuity Check
For a cubic spline with two pieces (\(f_1\) and \(f_2\)) joined at knot \(k\), which of the following is NOT a required condition for smoothness at the knot?
\(f_1(k) = f_2(k)\)
\(f'_1(k) = f'_2(k)\)
\(f'''_1(k) = f'''_2(k)\)
\(f''_1(k) = f''_2(k)\)
Smoothing the Noise Slides Smoothing the Noise
LOESS and Locally Weighted Regression
Non-Parametric Exploratory Analysis Lesson 04
The Case for LOESS
What if you don't want to assume a global function (linear, cubic, etc.)?
Locally Estimated Scatterplot Smoothing
A method for fitting a smooth curve through a scatterplot without requiring a pre-specified functional form.
Ideal for Exploratory Data Analysis (EDA) where the relationship is unknown or highly irregular.
The core idea:
"To estimate the value at point \(x_0\), use only the data points near \(x_0\)."
How it Works
For every target point \(x_i\):
Select a window of neighbors (the Span ).
Assign weights using a kernel (e.g., Tricube ). Points further away get less weight.
Perform a weighted linear or quadratic regression.
The predicted value becomes the curve point.
Weighting Kernel
\[ w_i = (1 - |d|^3)^3 \]
Where \(d\) is the relative distance from the target point. This ensures a "smooth" drop-off in influence.
Tuning the "Span" (\(\alpha\))
The Span controls the width of the local neighborhood.
Small Span (0.1)
Very wiggly. Captures local noise. High variance.
Large Span (0.8)
Very smooth. May miss local peaks. High bias.
Span Comparison
--- Small Span Large Span
When to use LOESS?
Climate Data
Smoothing temperature anomalies over 100 years to find the decadal trend.
Biometrics
Cleaning high-frequency heart rate data to identify underlying stress levels.
The Smoothing Strategy Teacher Guide Smoothing Strategy
Teacher Guide | Lesson 04
Sequence: Non-Linear Association
Pedagogical Intent
LOESS is the "honest" way to look at data. It doesn't force a structure. The goal for students in this lesson is to understand that hyperparameter tuning (the Span) is a subjective choice that balances truth-to-data vs. clarity-of-signal.
Facilitation Notes
1. The "Locality" Concept
Ask students: "If you want to predict the weather in London, do you look at the historical data for the whole world, or just the data for Western Europe?" LOESS codifies this intuition. We weight the local information higher than the distant information.
2. Computation Cost
Point out that LOESS is computationally expensive . Unlike OLS (one calculation for the whole dataset), LOESS performs a regression for every target point. For massive datasets, this is inefficient.
3. Sensitivity to Outliers
Standard LOESS is sensitive to outliers. Discuss Robust LOESS (R-LOESS) which uses an iterative weighting scheme to down-weight points with large residuals from the initial smooth.
Workshop: Trend Hunters Guide
Scenario: Environmental Noise
The worksheet provides a dataset representing CO2 levels (y) over time (x) with heavy seasonal noise.
Goal 1: Under-smoothing
Students sketch a line that follows every seasonal peak. They should realize this is useless for predicting the 50-year trend.
Goal 2: Balanced Smoothing
Students sketch a line that "cuts through" the middle of the noise. This is the "Goldilocks" span (\(\approx 0.3\)).
Assessment Benchmarks
Task Exemplary Student Response Span Tuning Reflection Explains that as span increases, the bias increases (model is too simple) but variance decreases (model is less jumpy). The Kernel Choice Identifies that the Tricube kernel allows for a smooth, continuous transition between neighborhoods, unlike a hard-cutoff window.
© 2026 Lenny Educational Resources | Advanced Statistics Series
Trend Hunters Worksheet Trend Hunters
LOESS Tuning and Exploratory Smoothing
Researcher:
Date:
Part 1: The Smoothing Spectrum
Below is a scatterplot representing average global temperature anomalies over time. The data is "noisy" due to annual variations. Your task is to mentally (and physically) perform a LOESS smooth.
Time (Years) Anomaly (C)
Task A: The Low-Span Smooth
Using a dashed line, sketch a LOESS curve with a span of \(\alpha = 0.1\). Follow the noise closely.
Task B: The High-Span Smooth
Using a solid line, sketch a LOESS curve with a span of \(\alpha = 0.8\). Aim for the long-term trend.
Part 2: The Bias-Variance Tradeoff
1. Sensitivity Analysis
Suppose a single outlier is introduced at the far right of the dataset (a massive spike in temperature). Which model—the Low-Span or the High-Span—will be more affected in its overall shape? Explain your reasoning using the concept of local weighting.
2. The "Over-smoothing" Trap
If you increase the span until it covers the entire dataset (\(\alpha = 1.0\)), what does the LOESS curve essentially become? (Hint: Think about what weighted linear regression looks like when the weights are near-identical across the range).
Part 3: Algorithmic Logic
Match the LOESS component to its function in the algorithm:
A. Span (\(\alpha\))
[ ] Calculates the final y-value for the target point.
B. Tricube Kernel
[ ] Defines the proportion of data used for each local fit.
C. Weighted OLS
[ ] Ensures smooth transitions between neighboring windows.
Beyond Correlation Slides The Information Interface
Mutual Information and Entropy in Bivariate Analysis
The Frontier of Association Lesson 05
The Sine Wave Challenge
Imagine a relationship that is perfect and deterministic , yet yields no correlation.
Case Study: \(y = \sin(x)\)
Over a full period \([0, 2\pi]\):
Pearson's \(r \approx 0\)
Spearman's \(\rho \approx 0\)
Kendall's \(\tau \approx 0\)
"There is zero linear or monotonic trend, yet X tells us EVERYTHING about Y."
Entropy (\(H\))
Entropy measures uncertainty or surprise in a variable.
\[ H(X) = - \sum p(x_i) \log p(x_i) \]
High Entropy = High Uncertainty / Uniformity
Low Entropy = Predictable / Skewed
The Intuition
If I tell you the outcome of a fair coin flip, I give you 1 bit of information.
If the coin is double-headed, I give you 0 bits (no surprise).
Mutual Information (\(I\))
MI measures how much information is shared between two variables.
\[ I(X;Y) = H(X) + H(Y) - H(X,Y) \]
"How much does knowing X reduce my uncertainty about Y?"
MI Key Features:
Always \(\ge 0\).
Symmetric: \(I(X;Y) = I(Y;X)\).
Captures ANY dependency.
Choosing the Right Tool
Relationship Pearson \(r\) Spearman \(\rho\) Mutual Info \(I\) Perfect Linear 1.0 1.0 High Monotonic Curve 0.8 1.0 High Full Sine Wave 0.0 0.0 High
Beyond Correlation Teacher Guide Beyond Correlation
Teacher Guide | Lesson 05
Sequence: Non-Linear Association
The Information Theoretic Lens
This final lesson transitions students from "variance-based" thinking to "information-based" thinking. In graduate research, especially in high-dimensional biology or finance, relationships are rarely linear or even monotonic. Mutual Information (MI) is the ultimate catch-all tool for dependency detection.
Case Study: The Sine Wave
The Setup
Present a scatterplot of \(y = \sin(x)\) from \(0\) to \(2\pi\). Ask the class to guess the correlation coefficient. Many will guess near 0. Then ask: "Is there no relationship?"
The Revelation
Explain that because the relationship is perfectly symmetrical, the covariance (and thus \(r\)) is zero. However, knowing \(x\) allows you to predict \(y\) perfectly. This "predictive power" is exactly what MI measures.
Entropy vs. Variance
Variance
Measures spread around a mean. Units are square of the variable. Assumes a specific distribution (usually Gaussian).
Entropy
Measures "uncertainty" in bits. Distribution-agnostic. Works on categorical and continuous data equally.
Information Impact Exit Ticket Guide
Question 1: The Zero-MI Scenario
The only time \(I(X;Y) = 0\) is when X and Y are statistically independent . This is a much stronger statement than saying \(r=0\).
Question 2: Normalization (MIC)
MI is not bounded between 0 and 1 (it depends on the entropy of the variables). Mention the Maximal Information Coefficient (MIC) as a modern way to normalize MI for comparison across different datasets.
Sequence Debrief
By the end of this lesson, students should be able to look at any bivariate plot and select the right tool:
Straight line? Pearson.
Steady curve? Spearman/Kendall.
Complex trend with noise? LOESS.
Pure chaos or complex loops? Mutual Information.
© 2026 Lenny Educational Resources | Advanced Statistics Series
Information Impact Worksheet Information Impact
Mutual Information and Complex Dependency
Researcher:
Date:
Part 1: The Entropy Calculation
Recall that Entropy (\(H\)) measures the uncertainty of a random variable. Given a categorical variable \(X\) representing weather types in a desert:
Weather Probabilities
Sunny: 0.90
Cloudy: 0.08
Rainy: 0.02
Analytical Question:
Without performing the full log calculation, would you expect the Entropy of this desert weather (\(H(X)\)) to be High or Low ? Justify your answer using the concept of "surprise."
Part 2: The Non-Monotonic Challenge
Consider the three bivariate relationships shown in the table below. For each, state whether Pearson's \(r\) or Mutual Information (\(I\)) would be more effective at capturing the dependency.
Relationship Structure Primary Metric Selection Justification Global linear trend with Gaussian noise. _________________ ________________________________ Cyclical sine-wave with period \(2\pi\). _________________ ________________________________ Two variables joined in a circle (\(x^2 + y^2 = 1\)). _________________ ________________________________
Theoretical Proof
Prove conceptually why \(I(X;Y) = 0\) implies that \(X\) and \(Y\) are independent. How does this contrast with the implication of \(r = 0\)?
Part 3: The Toolkit Summary
You are a lead data scientist. Your junior analyst presents a report stating: "We found a Spearman correlation of 0.2, so there is no meaningful relationship between these biological variables." Using what you have learned in this sequence, draft a one-paragraph correction to their statement.
