What is AdaGrad
An adaptive optimization algorithm that adjusts the learning step separately for each model parameter.
Definition
AdaGrad reduces the step more often for parameters that have already been updated a lot, and preserves the influence of rare features. This is useful for sparse data, such as text where many words are rare. The algorithm shows how training history can influence future model updates.
Example
In a text classifier, a rare but important word can receive sufficient weight, and frequently occurring function words will not be overly dominant.
Why it matters
Understanding AdaGrad helps you understand why different data types require different optimizers and why the learning rate cannot always be set to a single constant.
How it works
The algorithm accumulates the squares of past gradients for each parameter and divides the new update by the amount of this accumulated history.
Where it is used
- word processing
- training on sparse data
- experiments with optimizers
Limitations
The main disadvantage is that the training step can become too small over time, and the model almost stops learning.