What Is Gradient Descent?

Gradient descent updates parameters along the negative gradient of the loss to reduce error step by step.

The learning rate controls how large each step is.

It is the foundational optimization engine powering linear models, neural networks, and modern transformer architectures.

The Mountain Descent Analogy

Imagine standing in a dense fog on a steep mountain with zero visibility. To find the valley (global minimum of loss), you feel the slope of the ground under your feet. The direction of steepest ascent is the gradient vector ∇L(θ). By stepping in the exact opposite direction -∇L(θ), you descend toward lower elevation.

The Parameter Update Rule

Mathematically, parameters θ are updated iteratively across iterations t according to:

θ_(t+1) = θ_t - η * ∇L(θ_t)

where η (eta) represents the learning rate.

The Learning Rate Dilemma

  • Too large (η >> 0): The algorithm overshoots the minimum, oscillating wildly and potentially diverging to infinity.
  • Too small (η << 1): Convergence becomes excruciatingly slow, requiring excessive compute and getting trapped in shallow local plateaus.

Variants: Batch, SGD, and Mini-Batch

Standard Batch Gradient Descent computes gradients across the entire dataset per step, which is computationally expensive on big data. Stochastic Gradient Descent (SGD) updates parameters using a single random sample per iteration, yielding fast but noisy trajectories. Modern deep learning utilizes Mini-Batch Gradient Descent (typically batch sizes of 32 to 512) paired with adaptive optimizers like Adam and RMSprop.