A loss function turns "how wrong is this prediction?" into a single number. Training searches for the parameters that make that number small. The choice of loss decides which mistakes count most, and the shape of the surface it creates decides whether the search finds the best answer or settles in a nearby dip.
For a line ŷ = m·x + c, each point's residual is r = y − ŷ. MSE averages the squared residuals. On the left, each square's area is that point's squared error, so the loss is the average area of the red squares.
The right panel maps the loss for every possible slope and intercept. For a linear model, MSE makes a smooth bowl with exactly one bottom, which is a convex surface. Drag points, click the map to move the line, or press Fit to roll downhill. Then add an outlier: squaring makes one big miss dominate the average, while MAE barely moves.
A classifier outputs p = σ(z), its probability that the label is 1. Binary cross-entropy charges −log p when the truth is 1 and −log(1 − p) when it is 0. Squared error charges (p − y)², which can never exceed 1.
The gradients show why classification uses BCE. Measured against the score z, the BCE gradient is simply p − y: largest exactly when the model is most wrong. The squared-error gradient gets multiplied by the sigmoid's slope p(1 − p), which nearly vanishes when the model is confidently wrong. Start both from a confident mistake and race them.
With K classes the target is one-hot and the prediction is a softmax distribution. Categorical cross-entropy is −log of the probability given to the true class. Squared error over the whole vector (the Brier score) also counts how the wrong probability is spread among the other classes.
Try the two "same p(true)" presets. Cross-entropy gives them an identical score; Brier does not. The bars below show where each loss comes from, class by class.
The curve shows each loss as p(true) varies, with the remaining probability split evenly among the wrong classes. The dot is the current prediction.
Gradient descent only sees the slope where it stands. On a convex bowl like Plate 1 every downhill path ends at the same bottom. A neural network's loss surface is not convex. It has many valleys, and a run can settle in a local minimum that is lower than everything around it yet higher than the global minimum.
Click either surface to drop a ball. The colored strip under the curve shows each starting point's basin of attraction: where plain gradient descent from there ends up. Momentum (β) lets a ball coast through shallow dips. Noise imitates the randomness of mini-batch SGD and can shake a ball out of a poor valley.
| Loss | Formula | Use it for | Behavior |
|---|---|---|---|
| Mean squared error | mean (y − ŷ)² | Regression | Smooth and convex for linear models. Large errors dominate, so outliers pull hard. |
| Mean absolute error | mean |y − ŷ| | Regression with outliers | Every unit of error counts the same. Robust, but the gradient does not shrink near the answer. |
| Binary cross-entropy | −[y log p + (1−y) log(1−p)] | Two-class or multi-label classification | Unbounded for confident mistakes. With a sigmoid, the gradient is simply p − y. |
| Categorical cross-entropy | −log ptrue | One-of-K classification | Depends only on the true class's probability. With softmax, the gradient is p − y. |