Définition
AdaDelta is an adaptive optimizer: different model parameters receive different update steps depending on the history of the gradients. The algorithm was proposed as a way to make learning more robust without constantly adjusting the learning rate manually. It is now more often found in teaching materials and experiments than as a standard choice for new large models.
Exemple
If the neural network learns too quickly and the quality fluctuates, the adaptive optimizer can change the amount of updates for different parameters.
Pourquoi c'est important
The term helps to understand that training models is not only about the architecture and data, but also about the way the weights are updated.
Fonctionnement
During training, AdaDelta stores moving averages of gradients and updates. Based on these, it scales the next step so as not to make it too big or too small.
Où c'est utilisé
- neural network training
- experiments with optimizers
- deep learning training projects
Limites
The algorithm is not universally best. In modern problems, several optimizers are usually compared, including Adam and stochastic gradient descent.
FAQ
Why is “AdaDelta” useful to know?
The term helps to understand that training models is not only about the architecture and data, but also about the way the weights are updated.