What is AdaDelta
An optimization algorithm that automatically adjusts the neural network training step and reduces dependence on manual selection of the learning rate.
Definition
AdaDelta is an adaptive optimizer: different model parameters receive different update steps depending on the history of the gradients. The algorithm was proposed as a way to make learning more robust without constantly adjusting the learning rate manually. It is now more often found in teaching materials and experiments than as a standard choice for new large models.
Example
If the neural network learns too quickly and the quality fluctuates, the adaptive optimizer can change the amount of updates for different parameters.
Why it matters
The term helps to understand that training models is not only about the architecture and data, but also about the way the weights are updated.
How it works
During training, AdaDelta stores moving averages of gradients and updates. Based on these, it scales the next step so as not to make it too big or too small.
Where it is used
- neural network training
- experiments with optimizers
- deep learning training projects
Limitations
The algorithm is not universally best. In modern problems, several optimizers are usually compared, including Adam and stochastic gradient descent.