Connecting context with transformers

In 2017, researchers at Google published Attention Is All You Need, introducing the transformer. Its attention-centered architecture helped overcome limitations of earlier sequence models and became a foundation for modern LLMs.

Reading in sequence: RNNs

Before transformers, recurrent neural networks (RNNs) were widely used to analyze and generate text. A recurrent connection carries information from one processing step to the next.

At each step, the network combines the current token with information from previous steps, creates a new hidden state, and passes it onward.

We index steps by t. The current input x⁽ᵗ⁾ and previous hidden state h⁽ᵗ⁻¹⁾ determine the new state h⁽ᵗ⁾, which summarizes information read so far.

For a simple translation example, take the German sentence “Ich trinke Kaffee” (“I drink coffee”). Treating words as tokens for illustration, the RNN updates its state three times:

Early translation systems often paired two RNNs. The encoder read the source and compressed it into a final context vector. The decoder used that summary and previously generated words to produce a translation.

<sos> and <eos> mark the start and end of a sequence. Tokens above the decoder are its inputs; those below are its predicted outputs.

Using the context vector and <sos>, the decoder predicts I. Feeding I into the next step leads to drink, then coffee, and finally <eos>. It generates the translation sequentially, using information carried from the source.

RNN limitations

Three limitations matter here:

• Long-term dependencies: repeated state updates can weaken information from much earlier tokens, making distant relationships difficult to learn.
• A vector bottleneck: compressing an entire source sentence into one fixed-size summary can lose details, especially for long inputs.
• Limited parallelism within a sequence: a step depends on the previous hidden state, so it cannot simply be calculated simultaneously with all other steps. Large-scale training becomes slower and more expensive.

Attention Is All You Need

The transformer paper did not invent attention. In 2014, Dzmitry Bahdanau and colleagues proposed attention for neural translation in Neural Machine Translation by Jointly Learning to Align and Translate.

The transformer’s key contribution was to remove recurrence and organize the network around attention.

Attention

Attention calculates which available information is most relevant at a particular step and combines it accordingly.

An intuitive analogy is translating a sentence: you focus on different source words as you produce different parts of the translation. Bahdanau has described language-learning experience as an inspiration for this perspective.

Instead of relying on one fixed source summary, an attention-equipped decoder consults different encoder states with different weights at each output step.

When generating coffee, for example, the decoder can emphasize the encoder state for Kaffee. It scores the encoder states against its current state, then normalizes the scores into weights summing to 1.

With illustrative weights 0.1, 0.7, and 0.2, the middle source token trinke contributes most to that particular weighted combination. The numbers illustrate the calculation rather than a required translation alignment. In a real trained model, the weights depend on the learned representations and current output step.

Self-attention

Attention relieved the fixed-vector bottleneck, but an RNN encoder still processed its input sequentially.

What if tokens within the same sequence could consult one another directly? Self-attention does exactly that, including allowing a token to consult itself.

Each token gathers information from other tokens according to their relevance and forms a new, contextual representation. Distant tokens can interact directly rather than through many recurrent updates. Multiple positions can also be processed in parallel when their inputs are available.

Queries, keys, and values

To understand this calculation, we need queries, keys, and values (Q, K, V).

Consider bank in “I withdrew money from the bank” and “We sat on the river bank.” The surrounding words distinguish a financial institution from the edge of a river.

Self-attention decides which information to consult and how much of it to gather. It transforms each input vector into three representations:

Multiplication by three learned matrices, W_Q, W_K, and W_V, creates the query, key, and value. One token plays all three roles, using different vectors for each role.

For each query, compute dot products with the keys. Divide by √dₖ, where dₖ is key dimension, to moderate score scale. Apply softmax to obtain attention weights summing to 1.

The T in QKᵀ means transpose, exchanging rows and columns. Matrix multiplication calculates the query-key comparisons for all token pairs together.

Consider “Every morning before work I drink coffee.” To update the representation of drink, compare its query with all token keys. In the illustration, coffee receives the highest weight, while the other tokens contribute smaller amounts.

Multiply each value vector by its attention weight and add them. The result is a new representation of drink enriched by relevant context. This context-dependent transformation is a defining feature of transformers.

Full self-attention compares n tokens against n tokens, creating n² pairwise scores. Computation and memory therefore grow rapidly with sequence length. Models impose a context window, the maximum amount of context they can process in a request under their supported configuration.

A larger window can include more information, but it also has costs. Window sizes and attention implementations vary by model, so practical systems balance context needs with resource use.

We now have the core ideas needed to examine how the transformer’s components fit together.

Exercises

Decide whether each statement is True or False.

1. Attention Is All You Need first introduced attention itself.
2. An RNN depends on previous hidden states and therefore processes positions sequentially.
3. Self-attention can include a token’s own information.
4. A larger query-key dot product produces a larger relevance score before normalization.
5. Self-attention can directly compare distant tokens within its allowed context.