Optimization Algorithm Cheat Sheet

Core Algorithms for Gradient-Based Minimization

Vanilla Gradient Descent

\[ \theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta_t) \]

Pros: Simple to implement, computationally cheap per step.
Cons: Stuck in local minima/saddles, slow on plateaus, sensitive to learning rate.

Momentum

\[ v_t = \gamma v_{t-1} + \eta \nabla_\theta J(\theta_t) \] \[ \theta_{t+1} = \theta_t - v_t \]

Pros: Accelerates in relevant directions, dampens oscillations in "valleys".
Cons: Adds an extra hyperparameter (\( \gamma \), usually 0.9).

Nesterov (NAG)

\[ v_t = \gamma v_{t-1} + \eta \nabla_\theta J(\theta_t - \gamma v_{t-1}) \] \[ \theta_{t+1} = \theta_t - v_t \]

Insight: Calculates the gradient "ahead" based on momentum, providing a corrective look-ahead.

AdaGrad

\[ G_t = G_{t-1} + g_t^2 \] \[ \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t + \epsilon}} g_t \]

Pros: Adaptive learning rate per parameter. Great for sparse data.
Cons: Learning rate eventually shrinks to zero.

Troubleshooting Convergence

Symptom	Likely Cause	Fix
Loss suddenly goes to NaN/Inf	Learning rate too high (Exploding Gradient)	Decrease \( \eta \) by factor of 10.
Loss plateaus very early	Learning rate too low or Saddle point	Increase \( \eta \) or add Momentum.
Loss oscillates wildly	Learning rate slightly too high	Apply Decay or use Adaptive Opt (Adam).

Gradient Vector Optimization

Interpreting the Gradient Vector

Analyzing Curvature with Hessians

Implementing Gradient Descent

Constrained Lagrange Optimization

Stochastic Gradients in High Dimensions

Ascent Vector Slides

Steepest Ascent Worksheet

Gradient Geometry Guide

Curvature Landscape Slides

Point Classification Lab Worksheet

Hessian Analysis Key

Descent Dynamics Slides

Convergence Code Challenge

Optimization Algorithm Cheat Sheet

Boundary Path Slides

Constrained Extremes Worksheet

Lagrangian Insight Guide

Stochastic Storm Slides

Modern ML Debates

Neural Optimization Check Exit Ticket

Similar Sequences

Slice Lab Fraction Division

dfredericks

Ratio Lab Intervention

dfredericks

Number Scouts Sequence

dfredericks

Statistiques Exploratoires

Hugo Pierrard

Division in Daily Life

Saiko Otake

Radical Relationships

Calculus Foundations

Transformation Surgeons

Sigma Summit

Solution Sleuths

Period Power Remediation

GCF Visual Remediation

Angle Avengers

Integer Investigation

Big Batch Bakery

Discount Deal Detectives

Algebraic Foundations