Lebesgue Expectation Worksheet Lebesgue Expectation
Advanced Probability & Measure Theory
Name: ________________________________
Date: ________________
1. The Riemann Failure
Consider the Dirichlet function \( f: [0, 1] \to \{0, 1\} \) defined by: \[ f(x) = \begin{cases} 1 & \text{if } x \in \mathbb{Q} \cap [0, 1] \\ 0 & \text{if } x \notin \mathbb{Q} \cap [0, 1] \end{cases} \]
A. Prove that \( f \) is not Riemann integrable on [0, 1].
B. Using the Lebesgue measure \( \lambda \), calculate the Lebesgue integral \( \int_{[0, 1]} f \, d\lambda \). Show your measure-theoretic reasoning.
2. Simple Function Decomposition
Let \( X \) be a non-negative random variable. We define the expectation \( E[X] \) as the supremum of the integrals of simple functions \( \phi \) such that \( 0 \le \phi \le X \).
Express the expectation of a simple function \( \phi = \sum_{i=1}^n a_i \mathbf{1}_{A_i} \) in terms of the probability measure \( P \). Then, prove that this definition is independent of the representation of \( \phi \).
3. Constructing the Integral
Let \( X \ge 0 \) be a random variable. Define the sequence of simple functions: \[ X_n(\omega) = \sum_{k=1}^{n 2^n} \frac{k-1}{2^n} \mathbf{1}_{\{ \frac{k-1}{2^n} \le X(\omega) < \frac{k}{2^n} \}} + n \mathbf{1}_{\{ X(\omega) \ge n \}} \]
A. Show that \( X_n \uparrow X \) pointwise as \( n \to \infty \).
B. Argue why \( E[X_n] \) is a non-decreasing sequence and define \( E[X] \) in terms of this limit.
4. Fundamental Properties
Prove Monotonicity:
If \( X \le Y \) a.s., then \( E[X] \le E[Y] \).
Prove Linearity (Scalar):
\( E[cX] = cE[X] \) for \( c \ge 0 \).
Expectation Foundations Slides Measure Theory & Expectation
From Riemann's Limits to Lebesgue's Measures
The Riemann Failure
Where classical calculus breaks down
Riemann integration partitions the domain.
Consider the Dirichlet function:
Upper Sum: Always 1
Lower Sum: Always 0
Result: Not Integrable
"We need a way to measure 'how many' points are in the set of interest, rather than just partitioning the x-axis."
The Lebesgue Strategy
Partition the Range
Instead of vertical slices, we use horizontal slices. We ask: "For which \( x \) does \( f(x) \) fall in this interval?"
Measure Theory
\[ E[X] = \int_{\Omega} X(\omega) \, dP(\omega) \] Unifies discrete and continuous cases under one notation.
Expectation is just an integral over a probability measure.
Building Block: Simple Functions
\( \phi(\omega) = \sum_{i=1}^n a_i \mathbf{1}_{A_i}(\omega) \)
Finiteness
\( \phi \) takes only a finite number of values.
Measurability
Each set \( A_i \) must be in the \( \sigma \)-algebra \( \mathcal{F} \).
Integral
\( E[\phi] = \sum_{i=1}^n a_i P(A_i) \)
Expectation Inequalities Workshop Inequality Workshop
Student Name: _________________________
Course: Advanced Statistical Theory
Objectives
In this workshop, you will apply the core inequalities of probability theory to derive bounds for tail probabilities and moments. Graduate research often requires these tools to prove consistency or convergence of estimators.
1
The "Black Swan" Tail
"A financial analyst observes a return distribution with mean \( \mu = 0 \) and variance \( \sigma^2 = 1 \). No other information is known about the shape of the distribution."
A. Use Chebyshev’s Inequality to find an upper bound for the probability of a 5-sigma event: \( P(|X| \ge 5) \).
B. If we further assume \( X \) is non-negative, use Markov’s Inequality to bound \( P(X \ge 10) \). Compare this to the Chebyshev bound if applicable.
2
Convexity and Entropy
Let \( X \) be a positive random variable. Use Jensen's Inequality to compare \( E[\ln X] \) and \( \ln(E[X]) \). Which is larger? Provide a rigorous proof for your claim.
3
Hölder and Minkowski
A. Let \( X, Y \) be random variables such that \( X \in L^3 \) and \( Y \in L^{3/2} \). Use Hölder’s Inequality to bound \( E[|XY|] \).
B. Use Minkowski’s Inequality to show that the \( L^p \) norm \( \| \cdot \|_p \) satisfies the triangle inequality for \( p = 2 \).
Inequality Foundations Slides Bounding Reality
Fundamental Inequalities in Probability
Markov Jensen Hölder
Tail Bounds: Markov & Chebyshev
Markov's Inequality
For \( X \ge 0 \) and \( a > 0 \):
\( P(X \ge a) \le \frac{E[X]}{a} \)
"The simplest bound. Requires only the first moment."
Chebyshev's Inequality
For \( \mu = E[X] \) and \( \sigma^2 = Var(X) \):
\( P(|X - \mu| \ge k\sigma) \le \frac{1}{k^2} \)
"Uses the second moment to sharpen the tail bound."
The Power of Convexity
Jensen's Inequality
If \( \phi \) is a convex function, then:
\( \phi(E[X]) \le E[\phi(X)] \)
"The average of the function is greater than or equal to the function of the average."
Common Applications
AM-GM Inequality
Log-Sum Inequality (Information Theory)
Kullback-Leibler Divergence non-negativity
Hölder & Minkowski
Hölder's Inequality
If \( \frac{1}{p} + \frac{1}{q} = 1 \), then \( \|XY\|_1 \le \|X\|_p \|Y\|_q \)
Minkowski's Inequality
\( \|X + Y\|_p \le \|X\|_p + \|Y\|_p \)
Essential for proving that \( L^p \) spaces are normed vector spaces.
Convergence Analysis Worksheet Convergence Analysis
Monotone, Fatou, and Dominated Convergence
The Counterexample Hook
Consider \( f_n(x) = n \mathbf{1}_{(0, 1/n)}(x) \). Observe that \( f_n(x) \to 0 \) pointwise for all \( x \), but \( \int f_n \, dx = 1 \) for all \( n \). Thus, \( \lim \int f_n \neq \int \lim f_n \).Goal: Identify the missing conditions that allow the limit and integral to commute.
1. Monotone Convergence Theorem
State the requirements for MCT and apply it to the following:
Let \( X_n \) be a sequence of non-negative random variables such that \( X_n \uparrow X \) a.s. Prove that \( E[X_n] \uparrow E[X] \).
2. Fatou's Lemma
Why is Fatou's Lemma often called the "one-sided" convergence result?
Given non-negative \( X_n \), prove the inequality: \( E[\liminf X_n] \le \liminf E[X_n] \).
Exercise: Construct a sequence where the strict inequality holds.
3. Dominated Convergence Theorem
The most powerful tool for practitioners. Identify the "dominator" in the following problem:
Evaluate \( \lim_{n \to \infty} E[\frac{n \sin(X/n)}{X}] \) where \( X \) is a random variable with \( E[|X|] < \infty \) and \( X > 0 \).
Convergence Theorems Slides Limits & Expectations
Monotone, Fatou, and Dominated Convergence
When can we swap?
\( \lim_{n \to \infty} E[X_n] \stackrel{?}{=} E[\lim_{n \to \infty} X_n] \)
In graduate probability, "swapping the limit and integral" is not guaranteed. We require specific conditions to ensure mathematical validity.
Monotone Convergence
Requirements
1. \( X_n \ge 0 \)
2. \( X_n \uparrow X \) almost surely
Conclusion
\( E[X_n] \uparrow E[X] \)
"If the sequence is increasing and non-negative, the limit of the expectations is the expectation of the limit. No extra domination is needed because monotonicity provides control."
Fatou's Lemma: The Safety Net
\( E[\liminf_{n \to \infty} X_n] \le \liminf_{n \to \infty} E[X_n] \)
Applies to any non-negative sequence \( X_n \). It gives us a lower bound on the limit of expectations.
Minimal Conditions
Only requires \( X_n \ge 0 \).
Strict Inequality
Occurs when mass 'escapes' to infinity.
Utility
Used extensively in the proof of DCT.
The Dominated Convergence Theorem
If \( X_n \to X \) a.s. and there exists a random variable \( Y \) such that \( |X_n| \le Y \) for all \( n \) with \( E[Y] < \infty \), then:
\( \lim_{n \to \infty} E[X_n] = E[X] \)
Why it matters:
This is the workhorse of integration theory. It allows us to differentiate under the integral sign and prove the consistency of many statistical estimators.
Conditional Expectation Worksheet Handout Conditional Expectation
Sub-Sigma-Algebras and Projections
Advanced Probability Theory
Handout 4.1
Formal Definition
Let \( X \) be an integrable random variable on \( (\Omega, \mathcal{F}, P) \). Let \( \mathcal{G} \subset \mathcal{F} \) be a sub-sigma-algebra. The conditional expectation of \( X \) given \( \mathcal{G} \), denoted \( E[X|\mathcal{G}] \), is a random variable \( Y \) such that:
\( Y \) is \( \mathcal{G} \)-measurable.
For every set \( G \in \mathcal{G} \), \( \int_G X \, dP = \int_G Y \, dP \).
1. Radon-Nikodym Perspective
Briefly outline how the Radon-Nikodym Theorem ensures the existence of \( E[X|\mathcal{G}] \). Why is it unique only up to a set of measure zero?
2. Partition Sigma-Algebras
Suppose \( \Omega = \{1, 2, 3, 4, 5, 6\} \) with uniform probability. Let \( \mathcal{G} = \sigma(\{1, 2\}, \{3, 4\}, \{5, 6\}) \). Let \( X(\omega) = \omega \).
Calculate \( E[X|\mathcal{G}](\omega) \) for \( \omega = 1 \):
Calculate \( E[X|\mathcal{G}](\omega) \) for \( \omega = 5 \):
3. The Projection Property
If \( X \in L^2(\Omega, \mathcal{F}, P) \), then \( E[X|\mathcal{G}] \) is the orthogonal projection of \( X \) onto the subspace \( L^2(\Omega, \mathcal{G}, P) \).
Prove the Orthogonality Principle:
Show that \( E[(X - E[X|\mathcal{G}])Z] = 0 \) for any \( \mathcal{G} \)-measurable random variable \( Z \in L^2 \).
4. Tower Property
If \( \mathcal{H} \subset \mathcal{G} \subset \mathcal{F} \), show that \( E[E[X|\mathcal{G}] | \mathcal{H}] = E[X|\mathcal{H}] \). Interpret this result in terms of "information loss."
Conditional Expectation Slides Deck The Information Filter
Defining Conditional Expectation via Sigma-Algebras
Beyond \( P(A|B) = \frac{P(A \cap B)}{P(B)} \)
The Problem with the Ratio
Fails if \( P(B) = 0 \) (e.g., continuous random variables).
Does not easily handle multiple pieces of information or "partial" knowledge.
Not easily generalized to abstract spaces.
The Solution
Define expectation as a Random Variable measurable with respect to the information available.
Sigma-Algebras = Information
No Information
\( \mathcal{G} = \{ \emptyset, \Omega \} \)
\( E[X|\mathcal{G}] = E[X] \)
(A constant)
Partial Information
\( \mathcal{G} \subset \mathcal{F} \)
\( E[X|\mathcal{G}] \)
A "smoothed" version of \( X \).
Perfect Information
\( \mathcal{G} = \mathcal{F} \)
\( E[X|\mathcal{G}] = X \)
The variable itself.
The Geometry of Expectation
In \( L^2(\Omega, \mathcal{F}, P) \), the conditional expectation \( E[X|\mathcal{G}] \) is the best approximation of \( X \) using only the information in \( \mathcal{G} \).
"It is the orthogonal projection onto the subspace of G-measurable functions."
\( L^2(\mathcal{G}) \) subspace
\( X \)
\( E[X|\mathcal{G}] \)
Martingale Theory Worksheet Martingale Theory
Fair Games, Stopping Times, and Gambler's Ruin
1. The Martingale Property
"A stochastic process \( \{M_n\}_{n \ge 0} \) is a martingale with respect to a filtration \( \{\mathcal{F}_n\} \) if \( E[|M_n|] < \infty \) and \( E[M_{n+1} | \mathcal{F}_n] = M_n \) a.s."
Problem A: Symmetric Random Walk
Let \( X_i \) be i.i.d. with \( P(X_i = 1) = P(X_i = -1) = 1/2 \). Let \( S_n = \sum_{i=1}^n X_i \). Show that \( S_n \) is a martingale.
Problem B: Squaring the Walk
For the same random walk, show that \( M_n = S_n^2 - n \) is a martingale.
2. Optional Stopping Theorem (OST)
The OST states that if \( M_n \) is a martingale and \( \tau \) is a stopping time, then under certain conditions, \( E[M_\tau] = E[M_0] \).
Case Study: The Gambler's Ruin
A gambler starts with \( \$a \) and plays a fair game (win/loss \( \$1 \) with equal prob) until they reach \( \$N \) or \( \$0 \). Let \( \tau \) be the stopping time when the game ends.
1. Use the martingale \( S_n \) and OST to find the probability of reaching \( \$N \) before \( \$0 \).
2. Use the martingale \( S_n^2 - n \) and OST to find the expected duration of the game, \( E[\tau] \).
3. Reflection
Why is the condition that \( \tau \) is a stopping time (i.e., \( \{ \tau = n \} \in \mathcal{F}_n \)) critical for these theorems to hold? What happens if the gambler can see into the future?
Martingale Theory Slides Deck Fairness & Flow
Martingales and the Optional Stopping Theorem
The Fair Game Concept
Definition
\( E[X_{n+1} | \mathcal{F}_n] = X_n \)
"The expected future value, given everything we know now, is exactly what we have now."
Super-martingale
\( E[X_{n+1} | \mathcal{F}_n] \le X_n \)
"Unfavorable game" (Expect to lose money)
Sub-martingale
\( E[X_{n+1} | \mathcal{F}_n] \ge X_n \)
"Favorable game" (Expect to win money)
The Power of Stopping
Optional Stopping Theorem (OST)
For a Martingale \( M_n \) and a Stopping Time \( \tau \):
\( E[M_\tau] = E[M_0] \)
(Conditions apply: e.g., \( \tau \) is bounded, or the martingale is uniformly integrable.)
Case Study: Gambler's Ruin
We want to know the probability of hitting boundary \( A \) before boundary \( B \).
Martingale Approach:
1. Identity the martingale (often the walk itself).
2. Apply OST: \( E[S_\tau] = S_0 \).
3. \( p \cdot A + (1-p) \cdot B = S_0 \).
4. Solve for \( p \).
"No Strategy Matters"
The Martingale property proves that in a fair game, you cannot change your expected winnings by choosing when to stop.
