Type a phrase and follow it through a transformer. Text becomes tokens, tokens become vectors, attention heads let each position read from earlier ones, and the final vector is turned back into a probability for every token in the vocabulary. This model is tiny (29 dimensions, 3 heads, 1 layer) and its weights are set by hand so you can read every number. A trained model learns weights like these from data, at a much larger scale.
A tokenizer splits text into pieces from a fixed vocabulary and replaces each with its ID number. Common words are one token; rarer words split into subword pieces such as cat + s. The ▁ marks a piece that starts a new word. Real vocabularies hold 50,000 to 200,000 pieces; this one has 60.
Click any token to follow it through every stage below. Try a word the vocabulary doesn't know, such as "giraffe", to see it split into chunks.
Each token ID selects one row of an embedding table. That row encodes what the token is: here, its part of speech and a few topic features. A positional encoding is added so the model knows where each token sits, since attention by itself ignores order. Together they form the starting point of the token's vector, called the residual stream.
Real models use 768 to 12,288 dimensions per token, all learned. Their dimensions rarely mean one clean thing; the labels here exist only because this model is hand-built.
Every attention head projects each token's vector into a query (what am I looking for?), a key (what do I offer?) and a value (what do I pass along?). A token's query is compared with the keys of itself and all earlier tokens; softmax turns the scores into weights that sum to 1; the head returns the weighted sum of values. Later tokens are masked out, because the model must predict the future without seeing it.
Striped cells are masked: a token cannot look at tokens after it. Each row sums to 1.
| Key | q·k / √d | Weight |
|---|
Each head's output is projected and added to the token's vector, so information accumulates rather than being overwritten. Here each head writes into its own slots. A feed-forward network (MLP) then reads the combined vector and writes new features, such as "a place noun is coming". The result is a context-aware latent embedding. Real models stack dozens of these attention + MLP layers.
Orange cells are positive and blue cells are negative. Compare the same word in different contexts: "the" after "on" carries different latent features than "the" at the start.
Each neuron is a weighted sum of residual features minus a threshold, passed through ReLU (negative values become 0). This is what lets the model combine facts, like "I am a determiner" AND "the token before me was a preposition".
The final vector is multiplied by an unembedding matrix that has one row per vocabulary token. Each dot product is a logit: how well that token fits. Softmax turns the logits into probabilities, temperature controls how peaked they are, and one token is chosen and appended. Then the whole pass runs again with one more token. That loop is how text is generated.
Generate past the first sentence and the text starts to loop. Small models decoded greedily repeat themselves; switching to sampling at a temperature near 1 is one reason real systems produce more varied text.
The logit is the dot product of the final vector with that token's unembedding row. Grouping the terms shows which parts of the vector pushed it up or down. Predictions are made at every position; select an earlier token to see what the model would have predicted there.