Definição
AdaDelta is an adaptive optimizer: different model parameters receive different update steps depending on the history of the gradients. The algorithm was proposed as a way to make learning more robust without constantly adjusting the learning rate manually. It is now more often found in teaching materials and experiments than as a standard choice for new large models.
Exemplo
If the neural network learns too quickly and the quality fluctuates, the adaptive optimizer can change the amount of updates for different parameters.
Por que importa
The term helps to understand that training models is not only about the architecture and data, but also about the way the weights are updated.
Como funciona
During training, AdaDelta stores moving averages of gradients and updates. Based on these, it scales the next step so as not to make it too big or too small.
Onde é usado
- neural network training
- experiments with optimizers
- deep learning training projects
Limitações
The algorithm is not universally best. In modern problems, several optimizers are usually compared, including Adam and stochastic gradient descent.
FAQ
Why is “AdaDelta” useful to know?
The term helps to understand that training models is not only about the architecture and data, but also about the way the weights are updated.