AdaGrad

Definition

AdaGrad reduces the step more often for parameters that have already been updated a lot, and preserves the influence of rare features. This is useful for sparse data, such as text where many words are rare. The algorithm shows how training history can influence future model updates.

Beispiel

In a text classifier, a rare but important word can receive sufficient weight, and frequently occurring function words will not be overly dominant.

Warum es wichtig ist

Understanding AdaGrad helps you understand why different data types require different optimizers and why the learning rate cannot always be set to a single constant.

So funktioniert es

The algorithm accumulates the squares of past gradients for each parameter and divides the new update by the amount of this accumulated history.

Wo es genutzt wird

• word processing
• training on sparse data
• experiments with optimizers

Einschränkungen

The main disadvantage is that the training step can become too small over time, and the model almost stops learning.

FAQ