Inside the transformer: the encoder

Self-attention gives each token a contextual representation by consulting other tokens. A full transformer combines it with several other components. Let’s examine their roles.

Transformer architecture

Vaswani et al., *Attention Is All You Need* (2017), Figure 1

The original transformer has an encoder on the left and a decoder on the right. The encoder builds contextual representations of the source tokens. The decoder uses them and the target tokens seen so far to predict the next target token.

The boxed sections are encoder blocks and decoder blocks. N× means that each block is repeated N times. The original paper evaluated this design on English–German and English–French translation.

We will start with the encoder.

Input embeddings

The model has a vocabulary of tokens, each assigned a unique token ID. An ID identifies a token; its numerical size does not express its meaning.

An embedding table maps each ID to a learned vector. In the original English–German model, a vocabulary of approximately 37,000 tokens used 512-dimensional embeddings.

Thus, even a token such as apple is represented by 512 numbers:

Converting input tokens into these embedding vectors is the input embedding step.

Input representations

A token embedding alone does not specify the token’s position. Self-attention needs positional information to distinguish ordering.

The original transformer adds a positional encoding to each embedding. “Dog bites man” and “Man bites dog” use the same words but mean different things. Position information helps the model distinguish them.

The embedding plus positional encoding forms the input representation passed to the encoder blocks.

Multi-head attention

Multi-head attention performs several attention calculations in parallel, each using its own learned Q, K, and V projections. A head can learn different relationships: grammatical structure, reference, or long-range associations, for example. These are possibilities, not fixed roles assigned by the designer.

The attention output then reaches Add & Norm. A bypass path carries the sublayer’s original input there too: the residual connection.

A residual connection adds a sublayer’s input to its output. It preserves a direct route for information and gradients, helping deeper networks train effectively.

In the original transformer, Add & Norm first performs this addition, then applies layer normalization. For each token, normalization centers and scales its feature values using their mean and variance, with a small numerical stabilizer. Learned scale and shift parameters then adjust them. The operation helps stabilize training.

Feed-forward networks

The next component is a feed-forward network (FFN), a small multilayer network.

Attention exchanges information between tokens. The FFN then transforms each contextual token vector individually, using the same learned network at every position. It does not consult other positions again in that sublayer.

The FFN applies a nonlinear transformation, allowing richer representations than a stack of linear maps alone. The original model used ReLU, which maps negative values to 0 and passes nonnegative ones through.

Without nonlinear activations, any number of linear layers collapses into one linear map. Including biases gives one affine map. In one dimension, a linear map is a line through the origin; adding a bias can shift that line.

For inputs 4 and 2, expanding the calculations illustrates this collapse:

The coefficient ① w₁w₅ + w₂w₆ multiplies 4, so call it W₁. Similarly, ② w₃w₅ + w₄w₆ multiplies 2, so call it W₂. The entire computation becomes y = 4W₁ + 2W₂.

Extra linear layers alone do not create a more expressive nonlinear function. Activations such as ReLU prevent this collapse.

An individual unit can define a simple straight boundary. Combining several units through nonlinear activations can create boundaries no single line could represent. The FFN uses this principle to further transform information gathered by attention.

The FFN output passes through another Add & Norm: add the FFN input through its residual path, then normalize.

Multi-head self-attention, the FFN, and their residual and normalization operations make one encoder block. The original transformer stacked six such blocks.

Exercises

Decide whether each statement is True or False.

1. An embedding’s dimension is the number of components representing one token.
2. Positional encoding helps distinguish sequences with the same tokens in different orders.
3. All attention heads share the same Q, K, and V projection weights.
4. A residual connection adds the original input to the transformed result.
5. Stacking linear layers without activations still produces one linear map.