Now let’s dive into the real backbone of the algorithm.

1. Maximum Entropy Reinforcement Learning

Traditional RL optimization is:

$$\pi^\* = \arg\max_\pi \mathbb{E}\left[\sum_t r(s_t, a_t)\right]$$

SAC modifies this objective by adding entropy:

$$\pi^\* = \arg\max_\pi \mathbb{E}\left[\sum_t r(s_t, a_t) + \alpha \cdot \mathcal{H}(\pi(\cdot|s_t))\right]$$

Where entropy:

$$\mathcal{H}(\pi(\cdot|s)) = -\mathbb{E}_{a\sim\pi} [\log \pi(a|s)]$$

Intuition

The agent is rewarded not only for performing well, but also for staying unpredictable.
This unpredictability prevents local optima and improves exploration.

2. SAC Architecture Overview

The algorithm uses:

Two Q-networks:

$$Q_{\theta_1}, Q_{\theta_2}$$

​​

One Value Network:

$$V_\psi(s)$$

One Policy Network (Actor):

$$\pi_\phi(a|s)$$

Replay Buffer

Stores experience tuples:

$$(s, a, r, s’)$$

This setup allows SAC to learn using older transition data, improving sample efficiency.

3. Policy Network (Actor)

The actor outputs a Gaussian distribution:

$$a \sim \pi_\phi(a|s) = \mathcal{N}(\mu_\phi(s), \sigma_\phi(s))$$

To keep actions bounded (e.g., −1 to 1), SAC uses a tanh squashing function:

$$a = \tanh(\mu + \sigma\epsilon), \quad \epsilon \sim \mathcal{N}(0,1)$$

4. Soft Q-Function Update

The target for the Q-value is:

$$y = r + \gamma V_{\bar\psi}(s’)$$

The loss:

$$J_Q(\theta) = \mathbb{E}\left[(Q_\theta(s,a) – y)^2\right]$$

Two Q-networks reduce overestimation:

$$Q_{\min} = \min(Q_{\theta_1}, Q_{\theta_2})$$

5. Soft Value Function Update

The value network is updated as:

$$V(s) = \mathbb{E}_{a\sim\pi} \left[Q(s,a) – \alpha \log \pi(a|s)\right]$$

The loss:

$$J_V(\psi) = \mathbb{E}\left[\left(V_\psi(s) – (Q_{\min}(s,a) – \alpha \log \pi(a|s))\right)^2\right]$$

6. Actor Update (Policy Improvement)

The policy maximizes:

$$J_\pi(\phi) = \mathbb{E}_{s\sim D}\left[\alpha \log \pi_\phi(a|s) – Q_{\min}(s,a)\right]$$

Here:

• maximizing Q = choose better actions

• maximizing log π = stay random

A balance controlled by temperature parameter α.

7. Automatic Entropy Temperature Adjustment

Instead of fixing α manually:

$$J(\alpha) = -\mathbb{E}[\alpha \cdot (\log \pi(a|s) + \mathcal{H}_{target})]$$

This ensures the policy maintains a target entropy level (not too random, not too deterministic).