Gradient Descent: Definition and Examples

Gradient Descent is the fundamental algorithm that allows artificial intelligence models to learn. Its principle is remarkably intuitive: imagine you are lost in a thick fog at the top of a mountain and you are trying to descend into the valley. At each step, you feel the ground around you and move in the direction where the slope descends most steeply. This is exactly what Gradient Descent does with a model's parameters.

Specifically, the algorithm computes the gradient (the partial derivative) of the cost function with respect to each parameter of the model. This gradient indicates the direction in which the error increases most rapidly. By moving in the opposite direction, the error is gradually reduced. The learning rate controls the size of each step: too large, you risk overshooting the minimum; too small, training will be extremely slow.

There are several variants of this algorithm. Batch Gradient Descent uses the entire dataset to compute each update, which is precise but computationally expensive. Stochastic Gradient Descent (SGD) uses only one example at a time, making it faster but noisier. Mini-batch Gradient Descent, the most commonly used in practice, strikes a balance by using small batches of data. Modern optimizers like Adam, RMSProp, or AdaGrad add adaptive mechanisms to automatically adjust the learning rate.

Gradient Descent is at the heart of training all neural networks, including large language models (LLMs) like GPT or Claude. Without this algorithm, it would be impossible to adjust the billions of parameters that enable these models to understand and generate text. Understanding how it works helps to better grasp why some models converge with difficulty, why fine-tuning works, and how hyperparameters influence the final quality of a model.