Here is the complete workflow, explained step-by-step:

1. Initialize Actor and Critic Networks

• • Actor (πθ) → predicts action probabilities.

  • Critic (Vφ) → estimates value function

    $V(s)$

    .

  • Both networks start with random parameters.

2. Collect Trajectories from the Environment

• • Run the current policy πθ in the environment.

  • Collect sequences of:

  • States

    $s_t$

    ​

  • Actions

    $a_t$

    ​

  • Rewards

    $r_t$

    ​

  • Next states

    $s_{t+1}$

    ​

  • Stop when enough data (batch size) is collected.

3. Compute Returns (Total Future Reward)

• • For every timestep

    $t$

    t, compute:

     

    $$R_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \dots$$

     

  • This gives the total discounted reward from that state onward.

4. Use Critic to Estimate Value Function

• • The critic predicts:

     

    $$V(s_t)$$

     

  • This gives an approximation of the expected return from each state.

5. Compute Advantage Estimates

Advantage shows how good an action was compared to expectations.

$$A(s_t, a_t) = R_t – V(s_t)$$

Or using GAE for more stability (common in TRPO).

Why this step matters:

• • • • Tells TRPO which actions to increase or decrease in probability.

      • Reduces variance and stabilizes policy updates.

6. Construct the Surrogate Objective

TRPO does not maximize the raw reward.
Instead, it constructs a surrogate loss:

$$L(\theta) = \mathbb{E}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{old}}(a|s)} A(s,a)\right]$$

This evaluates how much the policy would improve if changed in direction θ.

7. Apply the Trust-Region Constraint (KL Divergence)

TRPO must keep the new policy close to the old policy:

$$D_{KL}(\pi_{\theta_{old}} || \pi_\theta) < \delta$$

This prevents:

• • • • Too large updates

      • Performance collapse

      • Unstable learning

8. Compute Natural Gradient Using Conjugate Gradient (CG)

• • Compute policy gradient ∇L.

  • Estimate Fisher Information Matrix (FIM).

  • Use Conjugate Gradient to solve:

     

    $$F x = \nabla L$$

     

  • This gives the natural gradient direction.

TRPO uses natural gradient because:

• • It’s geometry-aware

  • More stable and efficient than normal gradients

  • Improves the policy in the safest direction

9. Perform Line Search to Find Safe Step Size

Even with the natural gradient, TRPO still checks:

• • Does the step improve the surrogate loss?

  • Does it satisfy the KL constraint?

Line search reduces the step size until both conditions are satisfied.

If not satisfied:
→ TRPO rejects the update.

This guarantees safety and stability.

10. Update the Policy (Actor Network)

After line search finds the acceptable step:

• • • • Update the actor parameters:

         

        $$\theta \leftarrow \theta + \alpha x$$

         

Where:

• • • •  

        $x$

        = natural gradient

      •  

        $\alpha$

        = step size from line search

This completes one stable policy update.

11. Train the Critic (Value Function Update)

• • • The critic is updated by minimizing:

       

      $$(V(s_t) – R_t)^2$$

       

    • This makes the critic better at predicting future returns.

A good critic improves advantage estimates → improves updates → more stable TRPO performance.