kissin42 commited on
Commit
db92c42
·
verified ·
1 Parent(s): d90f43f

docs: show the environment transition in the rollout picture

Browse files

The environment read as a pure source, so the generated a_t+1 looked like the cause of the incoming s_t+1. Every action in a token now goes up into the environment, labelled with the glyphs the circles use. An action is *for* the state it sits beside, not *from* it. Alt text updated to match.

README.md CHANGED
@@ -21,7 +21,7 @@ token → next token (LLM generation)
21
  (state, action) → (next state from env, next action) (RL rollout)
22
  ```
23
 
24
- ![A rollout unrolled — the state arrives from the environment at every step, while the action is produced by the model and fed back in as part of the next token](assets/action-generation-in-a-rollout.svg)
25
 
26
  Causal GPT-RL policies act stably under their own rollouts — long-horizon control without the drift that has historically kept transformers from being usable as RL agents.
27
 
 
21
  (state, action) → (next state from env, next action) (RL rollout)
22
  ```
23
 
24
+ ![A rollout unrolled — each token pairs the current state with the action for it, that action moves the environment, and the environment returns the next state; meanwhile the model generates the next action from completed state-action pairs, not from the state it just received](assets/action-generation-in-a-rollout.svg)
25
 
26
  Causal GPT-RL policies act stably under their own rollouts — long-horizon control without the drift that has historically kept transformers from being usable as RL agents.
27
 
assets/action-generation-in-a-rollout.svg CHANGED