Professor Thinh Nguyen's AI & Legal Technology Laboratory

Every token gets to look back

A transformer reads a whole sequence at once. In each layer, every token sends out a query ("what am I looking for?"), offers a key ("what do I contain?") and a value ("what will I pass along?"). Matching queries to keys decides how much of each value flows to each token. Stack that attention step with a small feed-forward network, add information about word order, repeat, and predict the next token. The second half of this page is a real transformer training in your browser right now.

Step 1 · Attention by hand

What does “it” refer to?

Attention head

Score scale 4.0

Only look back

Tap a word to make it the one asking. Each bar shows how much attention that word gets from the asking word.

Gray arrows are keys; the orange arrow is the query. Drag it, or focus the plot and use the arrow keys. A key pointing the same way as the query gets a high score.

Key word	q · k	score	attention weight

What “it” carries forward after attention

Step 2 · Live training

Watch attention organize itself

This transformer has never seen these puzzles. It gets 8 random digits, then the → marker, and must produce the answer one digit at a time. Nobody tells it where to look. It finds out by trial, error and backpropagation.

Updates0

Loss–

Digits right–

Whole answers right–

Puzzle

Position info

Layers

Heads per layer

Learning rate

Pace

Example · input → the model's answer (right / wrong)

Attention maps for this example · row = token asking, column = token looked at · tap a map to follow that head below

Cross-entropy loss, log scale

Training batchesFixed test puzzles

On Reverse, watch a bright anti-diagonal form in the lower-right of a map: answer position 1 looks at input position 8, and so on.
Set Position info to None. Attention by itself ignores order, so reversing stalls near half the digits right.
Sort is harder: no fixed position holds the answer, so the model must compare digits. It takes longer.
Switching the puzzle keeps the current weights, so you can watch it unlearn one pattern and learn another.

Step 3 · Generation

One token at a time

Your 8 digits

Show attention of

The model only sees tokens to its left (the causal mask). At each step it attends to earlier tokens (blue arcs), turns the result into a score for every possible next token, and appends the most likely one. That new token becomes part of the input for the next step.

The residual stream for the current token

positivenegativeeach row scaled to its own range · 32 numbers per vector

Reading the stream top to bottom

The token is looked up in an embedding table, and a position vector is added so the model knows where it sits.
That sum is the residual stream. Each block reads it and adds its own result back instead of replacing it.
Attention adds information gathered from other tokens. The MLP (feed-forward network) then processes each token on its own.
After the last layer, the stream is compared with every vocabulary entry to produce next-token scores. Softmax turns those into the probabilities shown in the bars.