Temporal decomposition

Let’s Decompose Path into States and Actions

The policy gradient from 6.2 contains ∇_θ log P(τ⁽ⁱ⁾; θ). We can compute this trajectory log-probability gradient by adding contributions from the individual actions along the path.

The superscript (i) identifies the sampled trajectory; t identifies a time step within it. At step t, the agent observes s_t⁽ⁱ⁾, takes action u_t⁽ⁱ⁾, and reaches s_t+1⁽ⁱ⁾. Here each path includes the next state after its last action.

• Factor the trajectory probability

  $${\nabla }_{\theta }logP({\tau }^{\left(i\right)};\theta )={\nabla }_{\theta }log\left[\prod _{t=0}^H\underset{\begin{array}{c}\uparrow \\ \text{Dynamics model}\end{array}}{P\left(s_{t+1}^{\left(i\right)}\right|s_t^{\left(i\right)},u_t^{\left(i\right)})}·\underset{\begin{array}{c}\uparrow \\ \text{Policy}\end{array}}{{\pi }_{\theta }\left(u_t^{\left(i\right)}\right|s_t^{\left(i\right)})}\right]$$

  With Markov dynamics and a policy that depends on the current state, a path’s probability factors into the probabilities of its actions and state transitions, multiplied across time.

  • Dynamics model: P(s_t+1⁽ⁱ⁾ | s_t⁽ⁱ⁾, u_t⁽ⁱ⁾) is the probability of reaching the observed next state after taking that action.
  • Policy: π_θ(u_t⁽ⁱ⁾ | s_t⁽ⁱ⁾) is the probability that the policy assigns to the action actually taken in that state.

  The initial-state probability is omitted here. If the initial-state distribution is fixed with respect to θ, its log-probability contributes zero to the gradient.

• Turn the product into sums of logs

  $$={\nabla }_{\theta }[\underset{\begin{array}{c}\uparrow \\ \text{Log transition probabilities}\end{array}}{\sum _{t=0}^{H}logP\left(s_{t+1}^{\left(i\right)}\right|s_t^{\left(i\right)},u_t^{\left(i\right)})}+\underset{\begin{array}{c}\uparrow \\ \text{Log action probabilities}\end{array}}{\sum _{t=0}^{H}log{\pi }_{\theta }\left(u_t^{\left(i\right)}\right|s_t^{\left(i\right)})}]$$

  Apply log(ab) = log a + log b. The product over time becomes a sum, which we separate into log transition probabilities and log action probabilities. This is an exact algebraic rewrite.

• Remove the dynamics gradient

  $$={\nabla }_{\theta }\underset{\begin{array}{c}\uparrow \\ \text{Only the policy depends on θ}\end{array}}{\sum _{t=0}^{H}log{\pi }_{\theta }\left(u_t^{\left(i\right)}\right|s_t^{\left(i\right)})}$$

  The environment dynamics do not depend on the policy parameters θ. For the fixed states and actions in this sampled path, each transition log-probability therefore has zero derivative with respect to θ. Only the policy terms remain.

  Likelihood ratio changes probabilities of experienced paths. When differentiating the likelihood of one observed path, we hold its sampled states and actions fixed. Changing θ changes the probability assigned to that path and therefore how likely it is to occur in future runs.

  A path derivative follows how actions and subsequent states change with θ, and differentiates the return through those dependencies when the required derivatives are available. The likelihood ratio calculation instead differentiates the log-probability of a fixed sampled path; it requires no derivatives through the environment or reward function.

• Differentiate each action log-probability

  $$=\sum _{t=0}^H\underset{\begin{array}{c}\uparrow \\ \text{No dynamics model required}\end{array}}{{\nabla }_{\theta }log{\pi }_{\theta }\left(u_t^{\left(i\right)}\right|s_t^{\left(i\right)})}$$

  Differentiation is linear, so the gradient of the finite sum equals the sum of the gradients. For every observed state-action pair, compute ∇_θ log π_θ(u_t⁽ⁱ⁾ | s_t⁽ⁱ⁾) using the policy network, then add these vectors over time.

  No dynamics model required: we need sampled experience and the policy’s action log-probabilities. We can obtain the gradient without knowing the transition probabilities or differentiating through the environment. The dynamics still determine the sampled paths; their θ derivatives are what disappear from this calculation.