An agent must cross a frozen lake from the start to the gift without falling through a hole. The ice is slippery: it moves the way it meant to only one time in three, otherwise it slides sideways. The only reward is +1 for reaching the gift, so it has to learn which routes are safe from almost no feedback. Try Max speed, then switch to Test.
Shading in each square shows the agent's Q-value for moving that way. Arrows show its current best guess.
The state is just which square the agent is on: 16 states on the small lake, 64 on the large one.
With probability ε it picks a random direction. Otherwise it picks the direction with the highest Q-value.
On slippery ice the move goes as intended 1/3 of the time and 90° to either side 1/3 each. Holes and the gift end the episode.
Reward is 1 at the gift and 0 everywhere else. Q-learning nudges the estimate toward that reward plus the discounted value of where it landed.
Sparse reward. At first nothing is learned at all, because a random walk rarely reaches the gift. Once it does, value spreads backward one square per visit, which is why the shading grows outward from the gift. The discount γ makes shorter routes worth more.
Learning versus planning. The Optimal overlay comes from value iteration, which knows the slip probabilities in advance. Q-learning never sees those rules; it only learns from experience. On slippery ice the best policy often points away from holes rather than toward the goal, so it can look odd.
Things to try. Turn off slippery ice and watch it learn in a few dozen episodes. Drop γ to 0.6 and the far side of the 8 × 8 lake stays dark. In Edit lake, tap squares to add or remove holes.
You play. Use ← ↑ ↓ →, the buttons, or tap a neighboring square. The ice is just as slippery for you.