Notes / Reinforcement Learning / Paper 2 of 2

Continuous control with deep reinforcement learning

How an agent learns to move a paddle at any speed.

Robots are moved by amounts, not by picks from a list: how hard to turn a joint, how fast to spin a wheel. Their actions are real numbers. The methods of the earlier notes choose among a handful of moves, and they do not carry over when a move can be any number. In 2016, Lillicrap and seven colleagues at DeepMind published a method that does, DDPG, and trained it on more than 20 simulated physics tasks. This note runs it on a small game of its own: a paddle that can move at any speed. The game plays below and stops now and then to show what the agent is doing. Press Continue to go on.

This note covers the two networks DDPG trains, how each one learns, and the two things that keep their learning steady: a memory of past steps to learn from, and slow copies of both networks to set their targets.

Throughout this page, blue ink is the critic’s estimates, red ink what they aim for (a target, or a speed’s exact value), yellow the speed just chosen, and green a value being updated.

1The actor and the critic

DDPG trains two networks: an actor, which picks a speed for each frame, and a critic, which judges how good that speed is.

2Teaching the critic

The critic learns from steps the agent has already played. For each one it builds a target, and the target asks the actor which speed would come next. Below, training plays ten times as fast, and the chart redraws the critic’s values for one remembered frame as it learns.

The critic’s target is Algorithm 1’s

y_i = r_i + γQ′(s′_i, μ′(s′_i | θ^μ′) | θ^Q′)

where s′_i is the frame that followed step i. Where Q-learning takes the largest value over the next moves, DDPG asks the actor for its next speed and values only that one, so no search over speeds is needed. The paper’s equation 5 writes the same target with the critic and the actor themselves; Algorithm 1 uses the slow copies of section 3.

The critic’s loss is the mean squared gap over a minibatch of N remembered steps,

L = 1N Σ i(y_i − Q(s_i, a_i | θ^Q))²

and its gradient moves the critic’s weights θ^Q. The rally draws N = 64, as the paper does for low-dimensional inputs; its memory keeps up to 100,000 steps, the paper’s 10⁶.

Without the cost of moving, every speed that returns the ball would be worth the same, and only the critic’s smooth curve would give the actor a direction. Trained that way, 2 of 10 seeds learn to play every serve, against 8 of 10 with the cost.

A slip in the paperAlgorithm 1 sets y_i = r_i + γQ′(…) for every step, with no case for the last step of an episode. In the rally a miss ends the episode, and the runs of this note give that step the reward alone as its target.

3Targets that drift slowly

The target of section 2 used Q′ and μ′, slow copies of the two networks. Without them, the critic would chase targets that it moves itself.

Target networks:

The slow copies make each target change only slowly, so the critic chases something that holds still long enough to be caught. The paper calls these soft updates: rather than copying the weights now and then, it moves the copies a little at every step, and it needs both copies, Q′ and μ′, to keep the critic from diverging. The cost is speed, since a value learned now reaches the targets only gradually. The paper uses τ = 0.001; the rally uses 0.005, a copy that follows five times faster.

On the rally the copies mattered only once the critic learned fast. That is what the ten seeds above show, and it is all this note claims about them.

4Exploring with noise

To find better speeds, the agent in training adds noise to the actor’s speed. DDPG’s noise drifts rather than jumps: each frame’s value stays close to the one before. Its spread, the standard deviation of its values, sets how far it strays.

Noise spread:

Because the critic’s targets use the actor’s own next speed, not the speed that was played, DDPG can explore with any noise it likes: how a step was chosen does not enter its targets. The paper picks noise that drifts, an Ornstein–Uhlenbeck process with θ = 0.15 and σ = 0.2, for bodies with inertia, where a push that lasts a few frames goes further than one that flips every frame. The rally tried both kinds at two spreads, counting the seeds of 10 that end playing every serve perfectly after 20,000 steps:

Spread	Drifting noise	Jumping noise
0.38	8 of 10	9 of 10
0.2	5 of 10	2 of 10

In the rally the spread matters more than the kind of noise. At the wider spread, drifting and jumping noise do about equally well; at half of it both do worse, the jumping kind more so. Ten seeds are too few to say more.

The jumping noise is Gaussian, drawn fresh each frame. The drifting one follows 𝒩_t = 0.85 𝒩_t−1 + 0.2 n_t, with n_t a standard normal number. The paper gives θ and σ but no step, so this update is the rally’s; its spread is 0.2 / √(1 − 0.85²) ≈ 0.38.

The paper writes the noisy policy as μ′(s_t) = μ(s_t | θμt) + 𝒩, its equation 7.

A slip in the paperTwo symbols mean two things here: μ′ is both the target actor and the noisy policy, and θ is both the networks’ weights and the noise’s pull back to 0. This note writes the noisy speed a_t and gives the pull as a number.

5Batch normalization

Batch normalization rescales each input of a layer by the mean and spread it has across the minibatch, so every layer sees numbers of a similar size.

Batch normalization:

The paper applies it to the state input, to every layer of the actor, and to the critic’s layers before the action joins. Its reason is the inputs: a body’s position and its speed come in different units, with ranges that change from task to task, and normalizing saves rescaling each one by hand.

The paperNormalizing replaces scaling the inputs by hand, which inputs in different units would otherwise need.

In the rallyThe inputs are pixels and ±1. Scaling them by hand to similar ranges, without batch normalization, did not help: none of 10 seeds was ever perfect at a check, against 2 of 10 with the raw inputs. Here it helps for another reason, which these runs do not pin down.

These runs without it keep the learning rates and settings chosen with it, which were not retuned.

6The whole algorithm

Algorithm 1 of the paper puts all of this together. Each frame the agent plays, it stores the step, teaches the critic, moves the actor uphill and moves the copies.

Training:

The paper ran this loop with the same networks and settings on every task. Its settings for tasks with low-dimensional inputs: two hidden layers of 400 and 300 units, learning rates of 10⁻⁴ for the actor and 10⁻³ for the critic, γ = 0.99, τ = 0.001, minibatches of 64 and a memory of 10⁶ steps. The rally’s networks are smaller, 32 and 32 units, for a game of five numbers.

7Summary

DDPG in six lines: the noisy speed, the noise, the critic’s target and loss, the actor’s gradient, and the slow copies. In the code:

mu and q are the actor and the critic, and mu_target and q_target their slow copies.
s, a, r, s_next and done hold the N steps drawn from the memory; done is 1 when the ball got past the paddle (a miss), a case the paper does not have.
dq_da holds each step’s slope of the critic at the actor’s speed, and dmu_dtheta how the actor’s weights move its speed, as rows, which backpropagation computes.

Noisy speed

a_t = μ(s_t | θ^μ) + 𝒩_t, clipped to the speeds the paddle has

a = np.clip(mu(s) + noise, -1, 1)

The rally’s noise

𝒩_t = 0.85 𝒩_t−1 + 0.2 n_t

noise = 0.85 * noise + 0.2 * rng.standard_normal()

Critic’s target

y_i = r_i + γQ′(s′_i, μ′(s′_i | θ^μ′) | θ^Q′)

y = r + gamma * q_target(s_next, mu_target(s_next)) * (1 - done)

Critic’s loss

L = 1N Σ i(y_i − Q(s_i, a_i | θ^Q))²

loss = np.mean((y - q(s, a)) ** 2)

Actor’s gradient

∇_θ^μJ ≈ 1N Σ i ∇_aQ(s, a | θ^Q)|_{s = s_i, a = μ(s_i)} ∇_θ^μμ(s | θ^μ)|_{s_i}

grad = (dq_da[:, None] * dmu_dtheta).mean(axis=0)

Slow copies

θ′ ← τθ + (1 − τ)θ′, for the critic and for the actor

theta_target = tau * theta + (1 - tau) * theta_target