Autoregressive Models

A language model answers one fundamental question: given everything that came before, what comes next?

Autoregressive models make this concrete by breaking the joint probability of an entire sequence into a chain of conditional probabilities — each token predicted from all previous tokens. This simple factorization, combined with the Transformer architecture, powers GPT, LLaMA, Mistral, and virtually every major modern language model.

1. The Chain Rule Factorization

The probability of a sequence $\mathbf{x} = (x_1, x_2, \dots, x_T)$ can always be decomposed exactly using the chain rule of probability:

This is not an approximation — it is an identity. What makes it useful is that each factor $P(x_t \mid x_{<t})$ can be modelled by a neural network that takes the prefix $x_{<t}$ as input and outputs a probability distribution over the vocabulary.

2. Vocabulary and Token Distributions

Before diving into how the model is trained, it helps to understand what the model actually outputs.

Modern language models operate over a fixed vocabulary $\mathcal{V}$ — typically 32 000 to 128 000 subword tokens (produced by BPE or SentencePiece tokenisation). At each step, the model outputs a vector of logits $\mathbf{z} \in \mathbb{R}^{|\mathcal{V}|}$, one per vocabulary entry. A softmax converts these into a valid probability distribution:

3. Training: Next-Token Prediction

The training objective for an autoregressive model is cross-entropy loss summed over every position in the sequence:

Minimising this loss is equivalent to maximising the log-likelihood of the training corpus. Crucially, a single sequence provides $T$ training signal pairs — one per position — making the objective extremely data-efficient.

What the Model Sees During Training

At position $t$, the model receives the prefix $(x_1, \dots, x_{t-1})$ and must assign high probability to $x_t$ (the ground-truth next token). The gradient of the loss with respect to $\theta$ nudges the model to be more confident about correct continuations.

This is known as teacher forcing: during training, the ground-truth prefix is always fed in, regardless of what the model would have predicted.

4. Causal Masking: Preventing Future Leakage

Transformers process all positions in parallel during training, which creates a problem: a naive attention mechanism would let token $t$ attend to token $t+1$, allowing the model to "cheat" by reading the answer before predicting it.

Causal masking (also called a look-ahead mask or autoregressive mask) solves this by setting all attention weights for positions $j > i$ to $-\infty$ before the softmax, effectively zeroing them out:

where the mask $M$ is:

The result is a lower-triangular attention matrix: each token can attend only to itself and all tokens to its left.

5. Generation: The Autoregressive Loop

At inference time, the model generates a sequence one token at a time. This is the autoregressive loop:

1. Start with a prompt (the context tokens $x_1, \dots, x_k$).
2. Forward pass — feed the current sequence into the model to get logits $\mathbf{z}$ at the last position.
3. Sample — draw the next token $\hat{x}_{k+1}$ from the distribution $P(x \mid x_{\leq k})$.
4. Append $\hat{x}_{k+1}$ to the sequence.
5. Repeat from step 2 until an end-of-sequence token is sampled or a length limit is reached.

Because each new token depends on all previous ones, generation is inherently sequential — unlike training, it cannot be parallelised across time steps.

6. Sampling Strategies

How we draw from $P(x_t \mid x_{<t})$ at inference time dramatically affects the quality and diversity of generated text. The choice is a trade-off between coherence and creativity.

Greedy Decoding

Always select the most probable token:

Simple and deterministic, but prone to repetitive, degenerate outputs since it commits fully to the local maximum at every step.

Temperature Scaling

Before the softmax, divide the logits by a temperature parameter $\tau > 0$:

Top-$k$ Sampling

Restrict sampling to the $k$ most probable tokens, renormalising over them:

This cuts off the long tail of unlikely tokens. Common values: $k = 40$ to $k = 100$.

Nucleus (Top-$p$) Sampling

Rather than a fixed $k$, keep the smallest set of tokens whose cumulative probability exceeds $p$:

The cutoff adapts dynamically: when the distribution is peaked, only a handful of tokens are included; when it is flat, more tokens enter the nucleus.

7. Exposure Bias: Training vs. Inference Gap

A subtle but important mismatch exists between training and generation.

During training (teacher forcing), the model always receives the ground-truth prefix. During inference, it conditions on its own previous predictions. If the model makes an error at step $t$, that error becomes part of the context for step $t+1$, and mistakes can compound — a problem called exposure bias.

Approaches to mitigate this include scheduled sampling (gradually replacing ground-truth tokens with predicted ones during training) and RLHF fine-tuning, which allows the model to experience the consequences of its own generations.

8. Why Autoregressive Models Scale

Autoregressive modelling with Transformers has a remarkable property: the training objective requires no labelled data. Any raw text corpus is a valid training set — the supervision signal comes entirely from predicting the next token in the corpus itself. This is self-supervised learning.

Combined with the Transformer's parallelism across sequence positions (enabled by causal masking), this means:

• Data: Any text on the internet is usable.
• Compute: Each GPU processes all $T$ positions simultaneously during training.
• Signal: A single document of $T$ tokens generates $T$ gradient-informative predictions.

The result is a training paradigm that scales predictably with model size, data, and compute — the empirical observation captured by the scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020).

9. Beyond Text: Autoregressive Generation in Other Domains

The autoregressive factorisation is not limited to text. The same principle applies to any domain where data can be serialised into a discrete sequence:

In each case, the model learns $P(x_t \mid x_{<t})$ and generates by sampling token-by-token in the appropriate discrete space.

10. From API Call to Token Stream: The SDK Pipeline

When you call a chat API, the request you send bears no resemblance to what the model actually processes. Your JSON message objects go through three silent transformation steps before a single forward pass occurs.

Step 1 — JSON input. The SDK accepts a structured list of { role, content } objects — a developer-friendly abstraction.

Step 2 — Chat template. The SDK serialises those objects into a single raw string using a model-specific format (e.g. ChatML, Llama-chat, Gemma). Special boundary tokens like <|im_start|> and <|im_end|> mark speaker turns. The model was trained to expect exactly this format.

Step 3 — Tokenisation. The formatted string is split into integer token IDs. These IDs — not strings — are the model's actual input.

Step 4 — Autoregressive generation. The model runs the loop described in §5, sampling one token at a time until it emits a stop token. The SDK reassembles the output IDs back into text for you.

This pipeline is invisible in typical API usage, but understanding it clarifies why prompt format matters, why different models require different templates, and why "tokens" are the natural unit of cost and context length.

Summary