Distributed Distributional Actor–Critic (D4PG): A Deep Dive into Modern Reinforcement Learning

The Distributed Distributional Actor–Critic (D4PG) training process separates experience collection from learning, enabling stable and scalable reinforcement learning. Below is a clear, step-by-step explanation of how D4PG is trained.

Step 1: Initialize Networks and Buffers

• Initialize:

  • Actor network

    $\pi_\theta$

    ​

  • Distributional critic

    $Z_\phi$

    ​

  • Target actor

    $\pi_{\theta’}$

    ​

  • Target critic

    $Z_{\phi’}$

    ​

• Create a centralized replay buffer.

• Define:

  • Number of atoms,

    $V_{\min}, V_{\max}$

    ​

  • Discount factor

    $\gamma$

     

  • N-step return length

    $N$

     

Step 2: Launch Distributed Actors

• Spawn multiple actors running in parallel.

• Each actor:

  • Receives the latest actor parameters from the learner

  • Interacts with its own environment

  • Selects actions using:

     

    $$a_t = \pi_\theta(s_t) + \text{exploration noise}$$

     

Step 3: Collect N-Step Transitions

• Each actor records:

   

  $$(s_t, a_t, r_t, \dots, r_{t+N-1}, s_{t+N})$$

   

• Compute N-step returns.

• Store transitions in the shared replay buffer.

Step 4: Sample a Mini-Batch

• The learner samples a batch of N-step transitions from the replay buffer.

• Sampling is off-policy, improving sample efficiency.

Step 5: Compute Target Actions

• Use the target actor to compute next actions:

$$a_{t+N} = \pi_{\theta’}(s_{t+N})$$

Step 6: Apply Distributional Bellman Update

• Shift and discount the return distribution:

$$\mathcal{T}^{(N)} Z(s_t,a_t) = \sum_{k=0}^{N-1} \gamma^k r_{t+k} + \gamma^N Z_{\phi’}(s_{t+N}, a_{t+N})$$

Step 7: Project onto Fixed Support

• The target distribution is projected onto the fixed categorical support

  $[V_{\min}, V_{\max}]$

  .

• This ensures the output matches the critic’s atom structure.

Step 8: Update the Distributional Critic

• Minimize cross-entropy loss between predicted and target distributions:

$$\mathcal{L} = – \sum_i p_i^{\text{target}} \log p_i^{\text{pred}}$$

​

Step 9: Update the Actor Network

• Use deterministic policy gradients:

$$\nabla_\theta J(\theta) = \mathbb{E}\left[\nabla_a Q(s,a)\nabla_\theta \pi_\theta(s)\right]$$

• The expected Q-value is computed from the critic’s distribution.

Step 10: Soft Update Target Networks

• Apply Polyak averaging:

$$\theta’ \leftarrow \tau \theta + (1 – \tau)\theta’ \\ \phi’ \leftarrow \tau \phi + (1 – \tau)\phi’$$

Step 11: Synchronize Actors

• Learner periodically sends updated actor parameters to all actors.

• Actors continue collecting experience with the improved policy.

Step 12: Repeat Until Convergence

• Steps 3–11 repeat continuously.

• Training ends when:

  • Performance converges

  • Maximum steps are reached