Transformers in conversation

We now know the original transformer’s components. Let’s connect them to the large language models we use today.

A decoder-based language model: GPT

The original transformer used an encoder and decoder for translation. Researchers also applied transformers to language modeling, including predicting the next token from preceding tokens. Repeating that prediction generates new text.

A prominent decoder-only family is GPT: Generative Pre-trained Transformer. It is pretrained on large amounts of text and uses that learning to generate continuations.

Beginning with OpenAI’s first GPT in 2018, GPT-2 and GPT-3 increased scale and capabilities. Many later generative LLMs share the decoder-only, next-token-prediction foundation, despite differences in their details.

We will use GPT-2 as a concrete architectural example. The left of the diagram shows the complete flow; the right expands a repeated block.

Full GPT architecture, CC0

There is no separate encoder and therefore no encoder-decoder cross-attention. But GPT certainly has inputs: the text prefix goes directly into its decoder stack. User text and generated text become part of the continuing token sequence, with role or boundary tokens where the model’s format requires them.

For example, Every morning I drink might generate coffee. Append that token, then predict the next one. Generating each new token from the prefix and earlier generated tokens is autoregressive generation.

GPT’s transformer blocks

GPT blocks resemble the original decoder’s masked-attention and FFN components, with several important details.

Dropout

Dropout also appeared in the original transformer implementation, even where the architecture drawing did not show it explicitly.

A model can fit training-specific patterns too closely and perform worse on unseen examples. This is overfitting.

Dropout randomly sets some intermediate outputs to 0 during training, discouraging dependence on a particular set of units. Think of a soccer player practicing with their weaker foot instead of always relying on their stronger one. The analogy illustrates varied practice, not a literal model of learning. Dropout is normally disabled during inference.

Pre-LN

LayerNorm is the layer normalization operation we saw in Add & Norm. GPT-2 changes its placement.

The original transformer normalizes after adding a sublayer output to its input. GPT-2 normally applies LayerNorm before attention or the FFN, then adds the sublayer output to the original input. This is pre-layer normalization (Pre-LN).

The FFN’s Linear → GELU → Linear sequence follows the same pattern. A direct residual path lets the original representation bypass normalization and the sublayer.

During backpropagation, this path also supports relatively direct gradient flow toward earlier layers, helping deeper stacks train stably.

MatMul

MatMul means matrix multiplication. Q, K, and V projections are examples. Packing multiple token vectors into a matrix lets hardware calculate many operations together rather than processing each vector separately.

GELU

GPT-2 uses GELU (Gaussian Error Linear Unit) rather than the original transformer’s ReLU. GELU smoothly modulates inputs instead of cutting all negative values to zero. Some negative outputs remain, and the function changes smoothly.

ChatGPT

Pretraining teaches GPT to continue text. Making it follow instructions and hold useful conversations takes additional training.

InstructGPT adapted GPT-3 using supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF).

SFT supplies examples of desirable responses. For instance, pair “What is the capital of France?” with “The capital of France is Paris.” Training on instruction-answer pairs teaches the model response patterns.

But many answers can be acceptable, and some are more useful than others. RLHF incorporates human preferences between responses.

People compare model-generated answers. Their rankings train a reward model that predicts preferred responses. Reinforcement learning then adjusts the language model toward higher reward, usually with constraints discouraging excessive departure from the starting model.

SFT says, “Here is an example of a good response.” Preference-based training adds, “This response is better than that one.” Neither guarantees that every output is correct.

Released in 2022, early ChatGPT used a GPT-3.5-series model, dialogue demonstrations written by human trainers, and RLHF. This adapted next-token prediction to following requests and using conversation context.

GPT and early ChatGPT’s underlying models therefore share the decoder-only foundation. Additional training makes the model more suitable for dialogue; it does not replace autoregressive generation.

How LLMs emerged

LLMs did not appear in one sudden invention. They combine advances in architecture, training, large datasets, and computation.

Recall the three ingredients: data, a model, and a learning method. Transformers supplied a powerful architecture; pretraining, fine-tuning, and preference-based learning supplied training methods; large text collections supplied data.

Their interaction helped produce today’s language models.

Exercises

Decide whether each statement is True or False.

1. GPT adds generated tokens to its prefix when predicting subsequent tokens.
2. Standard decoder-only GPT uses encoder-decoder cross-attention.
3. Pre-LN applies normalization before attention or the FFN.
4. SFT can use human-written instruction-answer pairs.
5. RLHF incorporates feedback about which answers people prefer.