Loss Dimensions Discussion Guide Dimensions & Loss
Technical Discussion Guide • Lesson 1
Topic
Optimization in ML
The Objective Function
In machine learning, we seek to find the parameter vector \(\theta \in \mathbb{R}^d\) that minimizes a loss function \(L(\theta)\). In deep learning, \(d\) can range from \(10^6\) to \(10^{11}\). This discussion explores the transition from undergraduate 3D calculus to the high-dimensional realities of data science.
1. Parameter vs. Feature Space
Discuss the distinction between the space of inputs (features) and the space of model variables (parameters).
Key Point: Students often confuse the dimensionality of the data with the dimensionality of the optimization landscape. A 1D regression can have a 2D parameter space (\(y = mx+b\)).
2. The Curse of Dimensionality in Optimization
How does the volume of the search space grow as we move from 3 to 100,000 dimensions? What happens to the "distance" between points?
Key Point: In high dimensions, "most" of the volume of a sphere is near the surface. Random initialization becomes critical because the "middle" of the landscape is vast and sparsely populated with minima.
3. Convexity vs. Reality
Why is convexity rarely found in deep learning loss functions? Discuss how symmetry (e.g., node permutation) creates multiple equivalent local minima.
Visualization Challenge
Visual 1: The 3D intuition
A smooth convex bowl with a single global minimum.
Visual 2: The \(d\)-dim Reality
A rugged, non-convex landscape with high-dimensional "valleys".
Instructional Notes
Remind students that gradient descent follows the steepest descent locally, but has no global "vision."
Define Loss Surface as the graph of \(L(\theta)\) over the parameter space.
Loss Landscapes Slides Loss Landscapes
Optimization in High Dimensions
The Objective
Find the parameter vector \(\theta^*\) that minimizes the empirical risk:
\[ \theta^* = \arg\min_{\theta} \frac{1}{n} \sum_{i=1}^n L(f(x_i; \theta), y_i) \]
Feature Space
The dimensionality of your data \(x\).
Parameter Space
The dimensionality of \(\theta\).
"To visualize 10 dimensions, imagine 3 dimensions and say 'ten' very loudly."
3D (Undergrad)
• Easy to plot
• Single global minima are common in textbooks
• Intuition: A simple bowl
1M Dimensions
• Impossible to plot
• Hessian has \(10^{12}\) entries
• Intuition: Massive "Swiss cheese" manifolds
The Problem
• Symmetry & Redundancy
• Plateaus (flat regions)
• Saddle Points (next lessons!)
Steepest Descent
The First Tool
The Update Rule
\[ \theta_{t+1} = \theta_t - \eta \nabla L(\theta_t) \]
The gradient \(\nabla L\) points in the direction of steepest ascent. We step in the opposite direction.
Learning Rate (\(\eta\))
Controls the step size. Too large \(\rightarrow\) overshoot. Too small \(\rightarrow\) glacial speed.
Local Information
The gradient only knows about the current "patch" of the landscape. It is blind to the global optimum.
Loss Space Geometry Worksheet Loss Space Geometry
Course: Stochastic Optimization • Worksheet 01
Name:
Date:
1. Feature Space vs. Parameter Space
Consider a simple neural network with one hidden layer. It takes an input vector \(x \in \mathbb{R}^d\) and produces an output \(y \in \mathbb{R}\). The architecture is: \[ \hat{y} = \sigma(w_2^T \sigma(W_1 x + b_1) + b_2) \] where \(W_1\) is a \(h \times d\) matrix and \(w_2\) is a \(h \times 1\) vector.
A) If \(d = 784\) (e.g., MNIST) and \(h = 128\), what is the dimensionality of the parameter space? Show your calculation.
B) Why does calculating the Hessian matrix \(\nabla^2 L(\theta)\) become computationally prohibitive in this scenario?
2. Symmetry and the Multiplicity of Minima
In deep learning, the loss function is typically non-convex. One reason is weight-space symmetry .
Suppose you have two neurons in a hidden layer. If you swap their incoming and outgoing weights, does the output of the network change? How does this affect the number of "equivalent" global minima in the loss landscape?
3. Gradient Field Visualization
Sketch the gradient vector field for the convex loss function \(L(\theta_1, \theta_2) = \theta_1^2 + 4\theta_2^2\). Draw several trajectories for Gradient Descent starting from different points.
Sketch Area
Analysis:
Describe how the gradient "behaves" differently along the \(\theta_1\) and \(\theta_2\) axes. How would a constant learning rate \(\eta\) affect convergence on each axis?
SGD Introduction Slides 0.103 0.942 0.441 0.223 0.119 0.456 0.778 0.001 0.334 0.982 0.121 0.443 0.556 0.223 0.111 0.998 0.445 0.332...
The Power of Noise
Stochastic Gradient Descent
The Computational Wall
Full Gradient
\[ \nabla L(\theta) = \frac{1}{n} \sum_{i=1}^n \nabla \ell(x_i, y_i; \theta) \]
If \(n = 1,000,000,000\) (Big Data), calculating one single update takes forever.
Batch Gradient Descent is too slow for modern scale.
The SGD Solution
Update Rule
\[ \theta_{t+1} = \theta_t - \eta \nabla \ell(x_j, y_j; \theta_t) \]
where \(j\) is a randomly sampled index.
1
Unbiased estimate of the full gradient.
2
Computational cost is independent of \(n\).
3
Introduces "Exploration Noise."
Finding the Balance
Size: 1
Pure SGD
High variance, hardware inefficient, but fast exploration.
Size: 32 - 512
Minibatch
Reduced noise, vectorization-friendly. The gold standard.
Size: N
Batch GD
Deterministic, stable, but computationally infeasible.
SGD Simulation Guide SGD Path Analysis
Comparing deterministic and stochastic optimization paths in a non-convex field.
Case Study: The "Banana" Landscape
We are optimizing the Rosenbrock function, a classic non-convex test case for optimization algorithms: \[ f(x, y) = (a - x)^2 + b(y - x^2)^2 \] The global minimum is at \((a, a^2)\). In this simulation, we compare the path taken by Batch GD (calculating the gradient across the entire surface) vs. SGD (calculating the gradient based on noisy, partial samples).
Batch Path (Deterministic)
Follows the exact negative gradient direction.
Smooth, monotonic descent in loss.
Highly susceptible to getting trapped in local minima.
SGD Path (Stochastic)
Follows a "zig-zag" path around the true gradient.
"Noise" acts as a form of exploration.
Can "jump" out of shallow local minima or flat plateaus.
Observational Debrief
1. The Convergence Jitter:
As SGD approaches the global minimum, why does it never truly "rest" at the exact bottom like Batch GD does?
2. Learning Rate Decay:
Given the jitter observed above, suggest a strategy for the learning rate \(\eta\) that would help SGD settle into the minimum in the final epochs.
Algorithmic Analysis Worksheet Algorithmic Noise
Course: Stochastic Optimization • SGD Statistical Analysis
Candidate ID:
Date:
01 The Unbiased Estimator
Let \(L(\theta) = \frac{1}{n} \sum_{i=1}^n \ell_i(\theta)\) be the full empirical risk. In SGD, we sample a single index \(j\) uniformly at random from \(\{1, \dots, n\}\).
A) Prove that the stochastic gradient \(\nabla \ell_j(\theta)\) is an unbiased estimator of the full gradient \(\nabla L(\theta)\). Show every step of the expectation calculation.
Expectation Derivation Space
02 Variance Reduction
Consider a minibatch \(B\) of size \(m\), where indices are sampled with replacement. The minibatch gradient is \(G_B = \frac{1}{m} \sum_{j \in B} \nabla \ell_j(\theta)\).
If the variance of a single sample gradient is \(\sigma^2\), what is the variance of the minibatch gradient \(G_B\)?
Discuss the trade-off between minibatch size \(m\) and the computational speed of a single update.
03 Convergence Requirements
Robbins-Monro Conditions: For SGD to converge almost surely to a local minimum, the learning rates \(\eta_t\) must satisfy:
\[ \sum_{t=1}^\infty \eta_t = \infty \quad \text{and} \quad \sum_{t=1}^\infty \eta_t^2 < \infty \]
Explain the intuition behind these two conditions. Why must the sum diverge but the sum of squares converge?
Regularization Slides Bound to Succeed
Regularization & Constraints
The Overfitting Trap
In high dimensions, a model can "memorize" noise in the training data, leading to explosive parameter weights and poor generalization.
Symptoms:
Low training error, high test error, large \(\|\theta\|\).
Visualization: Rugged Fit
One Concept, Two Perspectives
The Penalized View
Add a penalty term to the loss function:
\[ L(\theta) + \lambda \Omega(\theta) \]
"Soft" constraint via the loss.
The Constrained View
Minimize loss subject to a budget:
\[ \min L(\theta) \text{ s.t. } \Omega(\theta) \le C \]
"Hard" constraint on parameter space.
The Mathematical Bridge
The Lagrangian connects these views. The optimal point occurs where the gradients of the objective and the constraint are aligned:
\[ \nabla L(\theta) = -\lambda \nabla \Omega(\theta) \]
L2
Ridge
Hypersphere constraint. Shrinks all weights towards zero.
L1
Lasso
Hyperdiamond constraint. Promotes sparsity (feature selection).
Regularization Practice Worksheet Constraint Geometry
Regularization Practice • Lesson 03
Collaborator:
Date:
"Regularization is not just a penalty; it is a structural constraint on the model's capacity."
1. The Dual Formulation
Consider minimizing the Mean Squared Error loss \(L(\theta)\) subject to an L2 constraint: \(\|\theta\|_2^2 \le C\).
A) Write the Lagrangian function \(\mathcal{L}(\theta, \lambda)\) for this constrained problem.
B) Solve for the optimal \(\theta^*\) in terms of \(\lambda\) for the specific case where \(L(\theta) = \frac{1}{2}\|\theta - y\|_2^2\).
2. Why Lasso is Sparse
Sketch the feasible regions for the L1 constraint (\(|\theta_1| + |\theta_2| \le 1\)) and the L2 constraint (\(\theta_1^2 + \theta_2^2 \le 1\)) in the 2D plane below.
Sketch Area
Geometric Analysis:
Explain why the "corners" of the L1 ball lead to sparse solutions (where some \(\theta_i = 0\)) while the L2 ball does not. Use the concept of contour lines of the loss function \(L(\theta)\) touching the constraint boundary.
3. Weight Decay in SGD
In many deep learning libraries, "weight decay" is implemented by adding \(\lambda \theta_t\) to the gradient. Prove that for a simple SGD update, this is mathematically equivalent to multiplying the current weights by a factor \((1 - \eta\lambda)\) before the gradient update.
Adaptive Rates Slides Navigating the Slope
Adaptive Optimizers
The "Ravine" Problem
In deep learning, gradients are often very large in some directions and very small in others (poorly conditioned Hessian).
Vanilla SGD Failures:
• Oscillates wildly across the steep walls.
• Moves at a snail's pace along the flat floor.
Oscillation Visualization
The Physics of Optimization
Momentum
Instead of just using the current gradient, we maintain a "velocity" that accumulates past gradients.
\[ v_t = \gamma v_{t-1} + \eta \nabla L(\theta_t) \] \[ \theta_{t+1} = \theta_t - v_t \]
Dampens oscillations in high-curvature directions.
Accelerates descent along consistent directions.
The King of Optimizers: Adam
Ada ptive M oment Estimation combines two ideas:
1. First Moment
Moving average of the gradient (Momentum).
\[ m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t \]
2. Second Moment
Moving average of squared gradients (Scaling).
\[ v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2 \]
Individual learning rates for every single parameter!
Optimizer Comparison Sheet Optimizer Field Guide
Quick Reference • Stochastic Optimization
Optimizer Key Update Logic Advantage Best For... Vanilla SGD \[ \theta - \eta g \] Simplicity, memory efficiency. Large datasets with redundant info. SGD + Momentum \[ v = \gamma v + \eta g \] Dampens oscillations, navigates ravines. High curvature landscapes. RMSprop \[ \theta - \frac{\eta}{\sqrt{E[g^2]}} g \] Adapts rate per parameter; solves scaling. RNNs, non-stationary objectives. Adam RMSprop + Momentum The "Gold Standard" for Deep Learning. Almost everything.
The Components of Adam
Alpha (\(\alpha\))
Base learning rate. Usually 0.001. Still needs tuning!
Beta 1 (\(\beta_1\))
Decay for the 1st moment (momentum). Usually 0.9.
Beta 2 (\(\beta_2\))
Decay for the 2nd moment (scaling). Usually 0.999.
Pro Tip: Bias Correction
Because \(m_t\) and \(v_t\) are initialized at zero, they are biased towards zero in early steps. Adam uses \(\hat{m}_t = m_t / (1-\beta_1^t)\) to correct this initialization artifact.
Momentum Calculus Workshop The Heavy Ball
Calculus of Momentum • Workshop 04
Engineer:
Session:
1 From Discrete to Continuous
Polyaks's "Heavy Ball" method can be viewed as the discretization of a second-order Ordinary Differential Equation (ODE). Imagine a ball of mass \(m=1\) rolling on the loss surface \(L(\theta)\) with friction coefficient \(\mu\).
\[ \ddot{\theta}(t) + \mu \dot{\theta}(t) + \nabla L(\theta(t)) = 0 \]
A) Identify the physical meaning of each term in the ODE above (Acceleration, Friction, Gravity/Gradient Force).
B) Why does the second-order term \(\ddot{\theta}\) prevent the optimizer from instantly changing direction when it hits a steep wall in a ravine?
2 Damping and Overshoot
Consider a 1D quadratic loss \(L(\theta) = \frac{1}{2}k\theta^2\). The ODE becomes:
\[ \ddot{\theta} + \mu \dot{\theta} + k\theta = 0 \]
This is the equation for a damped harmonic oscillator.
What value of \(\mu\) results in "Critical Damping"? (i.e., the fastest convergence without oscillation).
Sketching Phase Space:
Sketch the trajectory in the \((\theta, \dot{\theta})\) plane for an underdamped system (small \(\mu\)).
Phase Space Sketch
Discussion: In high-dimensional deep learning, we often use a friction (momentum) coefficient of \(\gamma \approx 0.9\). How does this "physical" intuition help you explain why momentum helps find flatter minima?
Saddle Point Slides The Saddle Trap
High-Dimensional Landscapes
The "Big Lie"
In 3D, we fear local minima. But in 1,000,000 dimensions, the probability of being at a local minimum is exponentially small compared to a saddle point .
"To be a local minimum, all 1 million eigenvalues of the Hessian must be positive. To be a saddle, just one needs to be negative."
Low Prob: \((1/2)^d\)
Spectral Signature
Minima
All \(\lambda_i > 0\)
Curvature is positive in all directions.
Saddle
Mix of \(\lambda_i > 0, \lambda_i < 0\)
Curvature is positive in some directions, negative in others.
Maxima
All \(\lambda_i < 0\)
Curvature is negative in all directions.
Escape Trajectories
Gradients become vanishingly small near saddle points. Standard GD gets stuck.
How to escape?
Follow the eigenvector corresponding to the most negative eigenvalue.
Noisy SGD to the rescue!
Stochastic noise naturally provides the "kick" needed to fall down the negative curvature slope.
Hessian Analysis Lab Hessian Analysis Lab
Course: Stochastic Optimization • Lab 05
Researcher:
Date:
The Spectral Profile
In this lab, we analyze the curvature of a high-dimensional loss surface near a critical point. Our goal is to determine if the point is a local minimum, maximum, or saddle point using the eigenvalues of the Hessian matrix \(\mathbf{H}\).
1. Characterizing the Critical Point
Suppose you reach a critical point where \(\nabla L(\theta^*) = \mathbf{0}\). You calculate the Hessian matrix at this point and find its eigenvalues \(\lambda_i\).
Set A
\{2.1, 0.5, 4.3\}
Classification:
Set B
\{1.2, -0.4, 2.8\}
Classification:
Set C
\{-0.1, -1.2, -0.9\}
Classification:
2. Finding the Escape Route
Consider the 2D function \(L(x, y) = x^2 - y^2\).
A) Compute the Hessian \(\mathbf{H}\) for this function.
B) Identify the eigenvector corresponding to the negative eigenvalue. Along which axis should the optimizer move to reduce the loss?
3. The Combinatorial Trap
If each eigenvalue of a Hessian in \(d\) dimensions has an independent \(50\%\) chance of being positive or negative (a highly simplified model), what is the probability that a random critical point is a local minimum?
\( P(\text{min}) = (1/2)^d \)
Reflect:
As \(d \rightarrow 10^6\), what does this tell us about the prevalence of saddle points vs. local minima in neural networks?
Saddle Point Research Summary Landscape Synthesis
Research Seminar • Final Review
The Core Hypothesis
The prevailing theory in modern deep learning optimization, supported by works such as Dauphin et al. (2014), suggests that the primary difficulty in training neural networks is not local minima, but rather non-convexity in the form of saddle points.
Why it matters:
In high-dimensional spaces, a local minimum requires every single eigenvalue of the Hessian to be positive. This is statistically improbable. Saddle points, however, only require a mix of positive and negative eigenvalues, which is the "default" state of a high-dimensional random landscape.
The "Flatness" Conclusion
Recent research also suggests that flat minima (regions where the loss is low and the Hessian has many near-zero eigenvalues) generalize better than "sharp" minima. Flatness implies that small perturbations in the parameters (due to noise or new data) won't cause a massive spike in loss.
Generalization \(\propto\) 1 / Curvature
Seminar Debrief
1. Critical Reflection on SGD Noise
How does the "noise" we analyzed in Lesson 2 act as a natural mechanism for escaping the "saddle point trap" we identified in Lesson 5?
2. The Lagrange Connection
If regularization restricts the parameter space (Lesson 3), how might this change the "density" of saddle points in the feasible region?
End of Sequence • Stochastic Optimization in Data Science
