Reinforcement Learning Lab
Reinforcement learning · trial, error and reward

Learning what to do from consequences

In reinforcement learning nobody labels the right answer. An agent acts, the world responds with a new situation and a number called a reward, and the agent adjusts so that, over time, it collects as much reward as it can. These two labs show the core ideas: balancing exploring against exploiting, and learning the long-run value of actions from experience.

AgentThe learner and decision-maker.
EnvironmentEverything outside the agent that responds to its actions.
State sWhat the agent observes about its situation.
Action aA choice the agent can make in a state.
Reward rA number that says how good the last step was.
Policy πThe agent's rule for choosing actions.
Value Q(s,a)Expected total future reward from taking a in s.

1Explore or exploit? The multi-armed bandit

Five slot machines each pay a coin with a different hidden probability. Every pull either exploits (plays the machine that looks best so far) or explores (tries another to learn more). A purely greedy player can lock onto a mediocre machine forever; a little randomness, controlled by ε (epsilon), keeps it learning. Pull the levers yourself, or let the ε-greedy agent play.

0.10
0pulls
–average reward
0%pulls that explored

The orange bar is the agent's estimate: the average payout it has seen from that machine. With "Reveal" on, the black tick marks the true probability. Estimates from few pulls are noisy, which is exactly why exploring is necessary.

Average reward so far

This run
Compare strategies over 2,000 random bandits

Each line is the share of pulls that chose the best machine, averaged over many fresh sets of machines.

2Learning values: Q-learning in a gridworld

The robot starts at S and wants the +10 goal while avoiding −10 pits. Each move costs 0.1. It keeps a table Q(s,a): its estimate of the total future reward for each move from each square. After every step it nudges that estimate toward what actually happened, the reward it got plus the discounted value of where it landed. Repeat over many episodes and good values spread backward from the goal, one square at a time.

High value Low value Wall → Best action so far

The agent–environment loop, live

Agentε-greedy on Q Environmentthe grid action aₜ– next state sₜ₊₁ · reward rₜ₊₁–
Last update (temporal-difference learning)
Q(s,a) ← Q(s,a) + α [ r + γ · maxa′ Q(s′,a′) − Q(s,a) ]
Press Step or Train to see the numbers.
0.50
0.90
0.20
0%
Drive it yourself
Q-values for the robot's square

Your moves update Q too: you are the policy, and the table still learns from what happens. Click any square to inspect it. Arrow keys work when the grid is focused.

Return per episode

Each episodeAverage of last 20
0episodes
0steps taken
–last return

Why discount the future? γ = 0.90

Things to try

ε = 0 on the banditRun it a few times with new machines. The greedy agent often gets stuck on whichever machine paid first.
Optimistic startBegin every estimate at 1.0. Disappointment drives exploration even with ε = 0.
γ = 0.3 in the gridReset and train. The agent becomes short-sighted, and the goal's value barely spreads beyond its neighbors.
Cliff + slipperinessPick the Cliff map and set slipperiness to 20%. The learned path moves away from the edge: risk is priced into value.
Draw your own worldTurn on Edit map, paint walls and pits, and watch values flow around them.