AdaGrad

Definição

AdaGrad reduces the step more often for parameters that have already been updated a lot, and preserves the influence of rare features. This is useful for sparse data, such as text where many words are rare. The algorithm shows how training history can influence future model updates.

Exemplo

In a text classifier, a rare but important word can receive sufficient weight, and frequently occurring function words will not be overly dominant.

Por que importa

Understanding AdaGrad helps you understand why different data types require different optimizers and why the learning rate cannot always be set to a single constant.

Como funciona

The algorithm accumulates the squares of past gradients for each parameter and divides the new update by the amount of this accumulated history.

Onde é usado

• word processing
• training on sparse data
• experiments with optimizers

Limitações

The main disadvantage is that the training step can become too small over time, and the model almost stops learning.

FAQ