Actor-Critic Algorithm in Reinforcement Learning

In the field of machine learning known as reinforcement learning (RL), an agent picks up knowledge by interacting with its surroundings. The purpose of rewards and penalties is to maximize behavior. The effectiveness of the Actor-Critic (AC) method in striking a balance between value estimate and policy optimization makes it stand out among the other RL systems. An extensive explanation of the Actor-Critic algorithm’s operation, benefits, and practical uses is offered by this blog.

What is the Actor-Critic Algorithm?

Two essential elements are combined in the Actor-Critic algorithm, a hybrid reinforcement learning technique:

Actor: In charge of choosing courses of action according to a learned policy.
Critic: Assesses how successful the action was and offers input in the form of a value function.

By lowering variance and enhancing learning stability, this dual-approach improves on conventional policy gradient techniques.

Key Components of Reinforcement Learning

Prior to exploring the actor-critic approach, it is essential to comprehend the basic elements of reinforcement learning (RL):

Agent: The thing that interacts with the surroundings and makes decisions.
Environment: the external system that the agent interacts with is referred to as the environment.
State: An illustration of the existing circumstance or arrangement.
Action: The choice or action the agent takes.
Reward: The agent’s feedback derived from its actions.
Policy: The approach or collection of guidelines that direct the agent’s choices.

Key Terms in Actor Critic Algorithm

Two essential terms are:

Policy (Actor):

The likelihood of acting in state s is represented by the policy, which is represented as π(a∣s) π(a∣s).

By optimizing this policy, the actor aims to maximize the expected return.

The actor network models the policy, while θ stands for its parameters.

Critical Value Function:

Beginning with state s, the value function, represented as V(s) V(s), calculates the predicted cumulative reward.

The critic network represents the value function, and w stands for its parameters.

How the Actor-Critic Algorithm Works

The following is how the Actor-Critic framework works:

The agent chooses an action based on a policy (such as a stochastic policy) after receiving the current state of the environment.
In response to this behavior, the environment changes states and offers a reward.
By calculating a value function or an advantage estimate, the critic assesses how effective the activity was.
Based on the critic’s comments, the actor modifies its policy settings, honing its subsequent choices.
To increase the accuracy of subsequent feedback, the critic modifies the parameters of its value function.

Update Rule

Actor (Policy Network): Adjusts the policy to favor high-reward actions.
Critic (Value Network): Improves value estimates to better predict rewards.

1. Actor Update Rule

The Actor updates its policy parameters $θ$ to maximize expected rewards using the policy gradient:

$θ \leftarrow θ + α \cdot \nabla_{θ} log π (a ∣ s; θ) \cdot A (s, a)$

Key Terms:

$α$ : Actor’s learning rate (e.g., 0.001).
$\nabla_{θ} log π (a ∣ s; θ)$ : Gradient of the log-probability of taking action $a$ .
$A (s, a)$ : Advantage function (how much better $a$ is than average in state $s$ ).

Intuition:

Increase the probability of actions with $A (s, a) > 0$ (better than average).
Decrease the probability of actions with $A (s, a) < 0$ .

2. Critic Update Rule

The Critic updates its value parameters $w$ to minimize prediction error using TD error:

$w \leftarrow w - β \cdot \nabla_{w} (δ^{2})$

Key Terms:

$β$ : Critic’s learning rate (e.g., 0.002).
$δ$ : TD error = $r + γV (s^{'}; w) - V (s; w)$ .
$γ$ : Discount factor (e.g., 0.99).

Intuition:

Make $V (s; w)$ better approximate the true value of state $s$ .

Example: Defining Actor and Critic Networks


# Define the actor and critic networks
import tensorflow as tf

# Actor network
actor = tf.keras.Sequential([
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(env.action_space.n, activation="softmax")
])

# Critic network
critic = tf.keras.Sequential([
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(1)
])

Tagline: This example demonstrates how to define the actor and critic neural networks, essential components of the Actor-Critic algorithm for reinforcement learning.

Methametical Representation

1. Core Components

Actor (Policy Network)

Policy Function:
$π (a ∣ s; θ) = Probability of action a in state s (parameterized by θ) .$
- Example: In CartPole, $π (left ∣ s; θ) = 0.7$ , $π (right ∣ s; θ) = 0.3$ .

Critic (Value Network)

State-Value Function:
$V (s; w) = Expected cumulative reward from state s (parameterized by w) .$
- Example: $V$ predicts 50 future reward points.

2. Key Equations

Temporal Difference (TD) Error

$δ = r + γV (s^{'}; w) - V (s; w)$

Interpretation:
- $r$ : Immediate reward.
- $γ$ : Discount factor (e.g., 0.99).
- $V (s^{'}; w)$ : Critic’s value estimate of the next state.
- TD Error ( $δ$ ) quantifies how “surprised” the Critic is by the reward.

Advantage Function

$A (s, a) = δ = r + γV (s^{'}; w) - V (s; w)$

Purpose: Measures if an action $a$ is better/worse than average in state $s$ .
Example: If $δ = + 5$ , the action was 5 units better than the Critic’s prediction.

3. Actor Update (Policy Gradient)

The Actor adjusts $θ$ to maximize expected rewards:

$\nabla_{θ} J (θ) = E [\nabla_{θ} log π (a ∣ s; θ) \cdot δ]$

Update Rule:
$θ \leftarrow θ + α \cdot \nabla_{θ} J (θ)$
- $α$ : Actor’s learning rate (e.g., 0.001).
- Intuition: Increase the probability of actions with positive $δ$ .

Example Calculation

Suppose $π (left ∣ s; θ) = 0.7$ , $δ = + 5$ :
$\nabla_{θ} J (θ) = \nabla_{θ} log (0.7) \cdot 5$
- The gradient pushes $π (left ∣ s; θ)$ closer to 1.

4. Critic Update (Value Function)

The Critic adjusts $w$ to minimize prediction error:

$L (w) = E [δ^{2}]$

Update Rule:
$w \leftarrow w - β \cdot \nabla_{w} L (w)$
- $β$ : Critic’s learning rate (e.g., 0.002).
- Intuition: Make $V (s; w)$ align better with observed rewards.

Example Calculation

If $V (s; w) = 50$ , $r = 1$ ,
$γ = 0.99$
, $V (s^{'}; w) = 49$ :
$δ = 1 + 0.99 \cdot 49 - 50 = 1 + 48.51 - 50 = - 0.49$ $\nabla_{w} L (w) = 2 \cdot (- 0.49) \cdot \nabla_{w} V (s; w)$

5. Algorithm Pseudocode

Initialize θ (Actor),w (Critic), α, β, γ for episode in 1, 2, ...: state = env.reset() while not done: # Actor selects action action_probs = π(state; θ) action = sample(action_probs) # Perform action next_state, reward, done = env.step(action) # Critic computes TD error V_current = V(state; w) V_next = V(next_state; w) δ = reward + γ * V_next - V_current # Actor update log_prob = log(action_probs[action]) ∇θ_J = log_prob * δ θ += α * ∇θ_J # Critic update ∇w_ℒ = 2 * δ * ∇w V(state; w) w -= β * ∇w_ℒ state = next_state

Advantages of the Actor-Critic Algorithm

A particular version of the Actor-Critic algorithm that presents the idea of the advantage function is called Advantage Actor-Critic. This function calculates an action’s superiority over the state’s average action. A2C concentrates the learning process on activities that are substantially more valuable than the usual action performed in that condition by utilizing this benefit knowledge.

Variance Reduction: Actor-Critic uses value function approximation to stabilize learning, in contrast to pure policy gradient approaches.
Sample Efficiency: It learns more quickly than standard policy gradient algorithms by utilizing both policy and value functions.
Balanced Exploration and Exploitation: The policy’s stochastic character promotes exploration, but the critic makes sure that decisions are made with greater knowledge.
Scalability: Actor-Critic techniques scale well in high-dimensional, complicated settings.

Advantage Actor Critic (A2C) vs. Asynchronous Advantage Actor Critic (A3C)

Feature	A2C (Advantage Actor-Critic)	A3C (Asynchronous A2C)
Parallelism	Single agent	Multiple agents async.
Speed	Moderate	Faster (parallel workers)
Complexity	Easier to implement	Needs distributed systems
Use Case	Small-scale tasks (CartPole)	Large environments (Atari)

When to Choose A2C: Startups with limited compute.
When to Choose A3C: Tech giants training on massive datasets.

Applications of the Actor-Critic Algorithm

Robotics: Used for training robots in adaptive control and real-time decision-making.
Finance: Applied in stock trading strategies and portfolio optimization.
Game AI: Enhances the performance of AI agents in games like AlphaGo, OpenAI Five, and Dota 2.
Autonomous Vehicles: Aids in self-driving car navigation and intelligent decision-making.
Healthcare: Helps in optimizing treatment strategies and medical diagnosis models.

Conclusion

The Actor-Critic algorithm is a fundamental reinforcement learning technique that bridges the gap between policy-based and value-based approaches. Its ability to optimize decision-making in dynamic and high-dimensional environments makes it a crucial tool in AI research and real-world applications. As reinforcement learning continues to advance, Actor-Critic methods will remain central to intelligent learning systems, contributing to the development of more efficient and adaptable AI models.