A single accuracy number hides two different questions. When a model flags something, how often is it right? Of everything it should have flagged, how much did it find? And a model that aces its training data may still fail on new data. These three benches let you trade precision against recall, push a model from underfitting to overfitting, and use regularization to pull it back.
A classifier scores 160 documents for how likely each is to be responsive to a discovery request. Everything at or above the threshold gets flagged for production. Responsive documents sit above the axis, non-responsive ones below.
Precision asks how much of what you flagged is truly responsive. Recall asks how much of the responsive material you found. Drag the threshold. Lowering it finds more (recall up) but sweeps in more junk (precision down). Only a better model, one that separates the two groups more cleanly, improves both at once.
The blue points are training data drawn from a smooth curve plus noise. A polynomial of degree d is fit to them by least squares. The coral points are test data from the same process, which the model never sees during fitting.
Underfitting (low degree): the model cannot bend enough, so error is high on both sets. Overfitting (high degree): the curve threads through every training point, training error falls toward zero, and test error climbs. Turn on the purple ghost fits to see the same degree fit to ten other samples of the data: a flexible model changes wildly from sample to sample. That spread is variance; the stiffness of a simple model is bias.
Regularization adds a penalty on the size of the weights to the training loss. The model can still bend, but bending sharply now costs something, so it bends only where the data clearly calls for it. The data here is the same as in Bench 2, fit with a degree-12 polynomial.
At tiny λ the model overfits. At huge λ every weight is crushed and the model underfits. In between, test error dips to its best. L2 shrinks all coefficients smoothly; L1 pushes many of them to exactly zero, which also selects features. Press Sweep λ to watch the coefficients shrink.
| Symptom | Diagnosis | Remedies |
|---|---|---|
| Many false negatives, recall low | Threshold too strict for the task | Lower the threshold when misses are costly (screening, review for production); a better model helps both metrics. |
| Many false positives, precision low | Threshold too loose | Raise the threshold when false alarms are costly (spam filtering, alerts). |
| High training and test error | Underfitting (high bias) | More capacity or better features, less regularization. |
| Low training error, much higher test error | Overfitting (high variance) | More data, regularization (L2, L1, dropout, early stopping), or a simpler model. |