What is Autoregressive Models
Models that predict the next element in a sequence based on previous elements.
Definition
The autoregressive approach is widely used in language models: the model generates text one fragment at a time, each time taking into account the already created context. Such models are also used in time series, audio, and other serial data.
Example
The language model takes the beginning of the phrase “Today I want to...” and predicts the likely continuation, then uses the new fragment for the next step.
Why it matters
The term is important for understanding generative AI: many responses are created not by one instantaneous act, but by successive predictions.
How it works
The model estimates the probability of the next token or value based on the past sequence. Once the next element is selected, it is added to the context.
Where it is used
- text generation
- time series forecast
- audio generation
Limitations
Sequential generation can accumulate errors. Also, the model is limited to the context window and may lose long-distance connections.