The Transformer Architecture, Explained
Break down self-attention and feed-forward layers, the two building blocks behind every modern language model.
The architecture behind almost every modern model
Since the 2017 paper "Attention Is All You Need," the transformer has become the default architecture for language models. GPT, Llama, Mistral, and Claude are all, at their core, stacks of the same two building blocks repeated many times.
Self-attention: who should I pay attention to?
Self-attention lets every token look at every other token in the input and decide how relevant each one is. In the sentence "The trophy didn't fit in the suitcase because it was too big," attention is what lets the model figure out that "it" refers to the trophy, not the suitcase, by weighing the relationship between those words.
attention_score(token_a, token_b) = how relevant token_b is when
interpreting token_a
Multiple heads, multiple perspectives
A single attention calculation only captures one kind of relationship. Transformers run several attention "heads" in parallel, each free to focus on a different pattern — one head might track grammatical subject-object relationships, another might track topic continuity across a paragraph.
The feed-forward layer
After attention gathers context, a feed-forward layer processes each token's representation independently, refining it based on patterns learned during training. Attention mixes information across positions; the feed-forward layer transforms each position on its own. A transformer block alternates between these two steps.
Why stacking works
A single transformer block captures fairly simple relationships. Stacking dozens of them lets the model build up increasingly abstract understanding — early layers might capture syntax, while later layers capture meaning, tone, and long-range structure across a document.
Key takeaways
A transformer block is just self-attention followed by a feed-forward layer, repeated many times. Attention lets tokens exchange information; the feed-forward layer refines each token on its own. Depth is what turns that simple loop into genuine language understanding.