Cross-Entropy Lab
Cross-Entropy Lab
Units
Interactive lab · Information theory → classification

Surprise, Entropy & Cross-Entropy

In 1948 Claude Shannon proposed that the information in an event is measured by how surprised you are to see it. Today almost every neural network that sorts inputs into categories is trained by minimizing that same quantity. These five experiments build the idea from a single coin flip to a classifier learning in front of you.

Experiment 1

Surprise is the logarithm of improbability

Shannon's self-information of an outcome x with probability p(x) is

I(x) = −log p(x)

A certain event (p = 1) tells you nothing. A rare event tells you a lot.

Why a logarithm? Independent events multiply their probabilities, and we want their information to add. A 1-in-4 event followed by a 1-in-2 event has probability 1/8 and surprise 2 + 1 = 3 bits.

With base-2 logs the unit is the bit: the surprise of a fair coin landing heads. Machine-learning libraries use the natural log, which gives nats (1 nat ≈ 1.443 bits). Switch units in the bar at the top.

Surprise
Equivalent to this many fair coins all landing heads
Experiment 2

Entropy is average surprise

A source that emits symbols with distribution p has entropy

H(p) = Σ p(x) · (−log p(x))

It is the surprise of each outcome, weighted by how often that outcome happens.

Shannon proved this is the smallest average number of bits per symbol that any code can achieve. A predictable source compresses well. A uniform source cannot be compressed at all.

Reshape tomorrow's forecast with the sliders. Entropy peaks when all four outcomes are equally likely and falls to zero when one is certain. The stream draws real samples from p, and its running average of surprise settles onto H(p).

Entropy, built from each outcome's share
0 (certain)
surprise of each samplerunning averageH(p)
Experiment 3

Cross-entropy: surprise measured with the wrong beliefs

Now the world still follows p from Experiment 2, but you hold beliefs q. Each time x happens you feel −log q(x) of surprise. Averaged over what actually happens:

H(p, q) = −Σ p(x) · log q(x)

In coding terms, this is the bits per symbol you pay when your code was built for q but the data come from p.

Gibbs' inequality guarantees H(p, q) ≥ H(p), with equality only when q = p. The excess is the Kullback–Leibler divergence:

H(p, q) = H(p) + DKL(p ‖ q)

Changing q can only shrink the KL term. H(p) is a floor set by the world. Try giving a likely outcome a tiny q and watch one confident mistake spike the stream.

world p (set in Experiment 2)model beliefs q (drag)
H(p)
DKL(p ‖ q)
Cross-entropy H(p, q)
0
surprise under p → H(p)surprise under q → H(p, q)
Experiment 4

The loss on a single labeled example

A classifier turns raw scores, called logits z, into a distribution with the softmax:

qk = ezk / Σj ezj

For a labeled example, the true distribution y is one-hot: all its probability sits on the correct class. Then H(y) = 0, and cross-entropy collapses to a single term:

L = H(y, q) = −log qtrue

The loss is the model's surprise at the right answer. Its gradient with respect to each logit is simply qk − yk. The correct logit rises by the probability it lacks; each wrong logit falls by the probability it took.

Label smoothing spreads a little of y onto the other classes. Then H(y) > 0, the loss has a floor, and the model gains nothing from unlimited confidence.

Training example #0417 · true label
classlogit zsoftmax q · target y (dashed)∂L/∂z
Loss H(y, q)
Floor H(y)
DKL(y ‖ q)
Experiment 5

Training a classifier shrinks average surprise

Each dot belongs to one of three classes. A softmax-regression model learns its weights by gradient descent on the mean cross-entropy over all 150 points:

L = (1/N) Σi −log q(yi | xi)

The background shows the model's predicted distribution: saturated where it is confident, muddy where it hedges. The amber ring around each dot grows with the model's surprise at that dot's true label. Watch the rings shrink as the loss falls.

Then widen the spread so the clusters overlap. The loss levels off well above zero. Some points are genuinely ambiguous, and the leftover loss reflects uncertainty in the data itself (its conditional entropy), the classifier's version of the H(p) floor.

Tap or hover a dot to inspect it.

Mean loss
Accuracy
Step
No point selected.
Class AClass BClass Csurprise −log q(true)
Bench summary

One quantity, four names

IdeaFormulaWhat it measures
Surprise−log p(x)Information in one outcome you believed had probability p.
EntropyEx∼p[−log p(x)]Average surprise of a source judged by its own true odds. The compression limit.
Cross-entropyEx∼p[−log q(x)]Average surprise when the world is p and your beliefs are q. Equals H(p) + KL.
Classification loss−log q(ytrue | x)Cross-entropy with a one-hot target: the model's surprise at the correct label.