Vanilla Policy Gradient : A Complete Theoretical Foundation

In Reinforcement Learning (RL), a policy defines the agent’s decision-making mechanism—that is, how actions are selected based on observed states. When actions are chosen probabilistically rather than deterministically, the policy is referred to as a stochastic policy. Stochastic policies are foundational to policy-gradient methods such as Vanilla Policy Gradient , because they enable differentiable optimization and principled exploration.

1. Policy in the Markov Decision Process Framework

Formally, reinforcement learning problems are modeled as a Markov Decision Process (MDP), defined by the tuple:

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma)$$

where:

• S

  is the state space

• A

  is the action space

• P

  $P(s’ \mid s, a)$

   is the state transition probability

• R

  $R(s, a)$

   is the reward function

• γ∈(0,1] is the discount factor

Within this framework, a policy governs the agent’s interaction with the environment.

2. Formal Definition of a Stochastic Policy

A stochastic policy is defined as a conditional probability distribution over actions given a state:

$$\pi(a \mid s) = \mathbb{P}(A_t = a \mid S_t = s)$$

where:

• s∈S is the current state

• a∈A is a possible action

• π(a∣s) denotes the probability of selecting action

  $a$

  a in state s

Key Properties

For every state

$s \in \mathcal{S}$

, the policy satisfies:

1. Non-negativity

$$\pi(a \mid s) \ge 0 \quad \forall a \in \mathcal{A}$$

2. Normalization

$$\sum_{a \in \mathcal{A}} \pi(a \mid s) = 1$$

(or

$\int_{\mathcal{A}} \pi(a \mid s)\, da = 1$

 continuous action spaces)

These conditions ensure that

$\pi(\cdot \mid s)$

 is a valid probability distribution.

3. Parameterized Stochastic Policies

In policy-gradient methods, policies are typically parameterized by a vector

$\theta \in \mathbb{R}^d$

, yielding a family of policies:

$$\pi_\theta(a \mid s)$$

The objective of learning is to adjust

$\theta$

so as to maximize expected cumulative reward.

Examples of Parameterizations

• Discrete actions: Softmax policy

$$\pi_\theta(a \mid s) = \frac{\exp(f_\theta(s,a))}{\sum_{a’} \exp(f_\theta(s,a’))}$$

• Continuous actions: Gaussian policy

$$\pi_\theta(a \mid s) = \mathcal{N}(a \mid \mu_\theta(s), \Sigma_\theta(s))$$

These parameterizations are chosen to be smooth and differentiable with respect to

$\theta$

, a critical requirement for gradient-based optimization.

4. Stochastic Policy as a Randomized Decision Rule

From a theoretical standpoint, a stochastic policy can be interpreted as a randomized decision rule:

$$\pi: \mathcal{S} \rightarrow \Delta(\mathcal{A})$$

where

$\Delta(\mathcal{A})$

denotes the probability simplex over the action space.

This formulation highlights an important conceptual point:

The policy does not choose an action directly—it defines a distribution from which actions are sampled.

This probabilistic structure induces randomness in the agent’s behavior, even when the environment itself is deterministic.

5. Why Stochastic Policies Are Essential in Policy Gradient Theory

Stochastic policies play a central theoretical role in VPG for several reasons:

5.1 Differentiability of the Objective

The expected return objective:

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)]$$

depends on

$\theta$

 only through the policy distribution. The likelihood-ratio (log-derivative) trick used in policy gradients:

$$\nabla_\theta \log \pi_\theta(a \mid s)$$

is well-defined only when the policy assigns non-zero probability mass smoothly across actions.

5.2 Exploration Guarantee

Unlike deterministic policies, stochastic policies naturally ensure exploration:

$$\pi(a \mid s) > 0 \Rightarrow \text{action } a \text{ is eventually explored}$$

This is critical for convergence guarantees in theoretical analyses of policy-gradient algorithms.

5.3 Avoiding Non-Differentiability

Deterministic policies often lead to non-differentiable mappings from parameters to actions. In contrast, stochastic policies maintain a smooth dependence of action probabilities on

$\theta$

, enabling unbiased gradient estimation.

6. Relationship to Deterministic Policies

A deterministic policy

$\mu(s)$

can be viewed as a degenerate stochastic policy:

$$\pi(a \mid s) = \begin{cases} 1, & a = \mu(s) \\ 0, & \text{otherwise} \end{cases}$$

​

However, such policies lack differentiability almost everywhere, which is why classical VPG relies on stochastic rather than deterministic formulations.

7. Theoretical Role in Trajectory Distributions

A stochastic policy induces a probability distribution over trajectories:

$$p_\theta(\tau) = \rho_0(s_0)\prod_{t=0}^{T-1} \pi_\theta(a_t \mid s_t)\, P(s_{t+1} \mid s_t, a_t)$$

Here, the policy is the only component dependent on

$\theta$

. This factorization is fundamental to deriving the Policy Gradient Theorem, as it allows gradients of expected return to be expressed entirely in terms of

$\nabla_\theta \log \pi_\theta(a \mid s)$

.

8. Summary (Theoretical Perspective)

From a theoretical standpoint, a stochastic policy is:

• A probability distribution over actions conditioned on states

• A differentiable, parameterized object enabling gradient-based optimization

• The core mechanism through which randomness, exploration, and learning are introduced in VPG

In Vanilla Policy Gradient, the stochastic policy is not merely a design choice—it is the mathematical object that makes policy optimization tractable, analyzable, and theoretically sound.