An agent learns to keep a pole upright on a moving cart. It starts with no idea what to do, earns +1 for every 20 ms the pole stays up, and slowly turns trial and error into a policy. Switch to Max speed to watch a few thousand attempts fly by, then Test the result and poke the pole.
Tap or click left or right of the pole to give it a shove.
The four numbers on the right are sorted into buckets (3 × 4 × 8 × 10 = 960 states), so the agent can keep a table.
With probability ε it pushes at random to explore. Otherwise it picks the push with the higher Q-value.
Physics advances 20 ms. Staying up earns +1. Falling or leaving the track ends the episode with nothing.
The table entry moves a fraction α toward the reward plus the discounted best value of the next state.
Why the curve is noisy. Bucketing throws away detail, so two quite different situations can share a table cell. The agent also keeps a little randomness (ε never drops below 0.01), and a single bad update can briefly undo progress. Deep RL replaces the table with a neural network to smooth this out.
Things to try. Set γ near 0.9 and the agent becomes short-sighted. Set starting ε to 0 and it never explores, which often leaves it stuck. In You drive, use ← → or the buttons and see how long you last at 1× speed.