What is Activation Functions
Mathematical functions in a neural network that help it find complex nonlinear dependencies.
Definition
Activation functions decide which signal to pass from one layer of the neural network to the next. Without them, the neural network would be reduced to a simple linear model and would poorly describe complex patterns in text, images, sound, or user behavior. Common functions include ReLU, sigmoid, tanh and variations thereof.
Example
In an image recognition model, the activation function helps to identify increasingly complex features: first contours, then shapes, then parts of an object.
Why it matters
The term is useful for those who want to understand why neural networks can solve problems where the usual linear formula is not enough.
How it works
Each neuron receives numbers, applies a function to them, and passes the result on. The choice of function affects the learning speed, the stability of the model, and the quality of the final predictions.
Where it is used
- neural network training
- deep learning
- selection of model architecture
Limitations
Some features may cause gradient decay, unstable learning, or a large number of inactive neurons. In practice, the choice depends on the architecture and task.