Add frozen-critic PPO and show objective gradients directly on tokens

Use the pinned ORZ 0.5B value critic with per-token GAE and editable values. Replace toy probability-change chips with exact objective gradients, keep collected probabilities fixed, and explain the optimizer/forward-pass distinction. Preserve monochrome RL modes and fix mobile navigation overflow.

.dockerignore

@@ -4,6 +4,8 @@
 !runtime.py
 !server.py
 !grpo.py
+!ppo.py
+!critic.py
 !preload_models.py
 !static/
 !static/**

Dockerfile

@@ -12,14 +12,14 @@ RUN python -c 'import subprocess, sys, tomllib; deps = tomllib.load(open("/tmp/p
 
 RUN useradd --create-home --uid 1000 user
 WORKDIR /app
-COPY --chown=user:user runtime.py preload_models.py ./
+COPY --chown=user:user runtime.py critic.py preload_models.py ./
 USER user
 
 # Cache only the pinned inference files, excluding training checkpoints.
 RUN python preload_models.py
 ENV HF_HUB_OFFLINE=1
 
-COPY --chown=user:user server.py grpo.py ./
+COPY --chown=user:user server.py grpo.py ppo.py ./
 COPY --chown=user:user static ./static
 
 EXPOSE 7860

README.md

@@ -19,7 +19,7 @@ tags:
 
 # Layer lab — Qwen & Gemma
 
-A website for inspecting **Qwen3-0.6B** and **Gemma 3 270M TinyGSM reasoning**: next-token predictions at every position after every decoder layer, attention from every query head, and a frozen-model GRPO sandbox. Run it locally or on Hugging Face Spaces. Qwen remains the default model.
+A website for inspecting **Qwen3-0.6B** and **Gemma 3 270M TinyGSM reasoning**: next-token predictions at every position after every decoder layer, attention from every query head, and frozen-model GRPO/PPO sandboxes. PPO uses a separate pretrained ORZ 0.5B value critic. Run it locally or on Hugging Face Spaces. Qwen remains the default actor.
 
 | Checkpoint | Layers | Query / KV heads per layer | Reasoning |
 | --- | ---: | ---: | --- |
@@ -33,7 +33,7 @@ cd qwen-lab
 uv run --python 3.11 python server.py --port 8043
 ```
 
-Open [http://localhost:8043](http://localhost:8043). If the dependencies are already installed, `python3.11 server.py` is sufficient. The first launch downloads both checkpoints and their tokenizers from Hugging Face; subsequent launches use the standard Hugging Face cache. No API key is required. Qwen loads first, then Gemma, in the background. Each model has its own loading status and remains available after loading. The initial example runs automatically when the model is ready.
+Open [http://localhost:8043](http://localhost:8043). If the dependencies are already installed, `python3.11 server.py` is sufficient. The first launch downloads both actor checkpoints and the value critic from Hugging Face; subsequent launches use the standard Hugging Face cache. No API key is required. Qwen, Gemma, and the critic load in that order in the background. Each actor becomes available after loading; PPO also waits for the critic. The layer explorer's initial example runs automatically when the model is ready.
 
 The local server binds to `127.0.0.1` by default. Computation runs on the server hosting the website: your computer for a local launch, or the Space for a hosted launch. Prompts are not sent to an inference API. On first local launch, the library downloads checkpoint files from Hugging Face. Compatible Macs use MPS automatically; other systems use CPU. To force CPU, run `QWEN_DEVICE=cpu uv run --python 3.11 python server.py`. Computation uses float32 from the original BF16 checkpoint, without quantization.
 
@@ -41,7 +41,7 @@ The local server binds to `127.0.0.1` by default. Computation runs on the server
 
 The repository is ready for a **Docker Space** on **CPU Basic** (2 vCPU, 16 GB RAM). CPU Basic has no hourly hardware charge; creating Docker Spaces currently requires PRO for personal accounts or Team/Enterprise for organizations. CPU inference is slower than MPS, especially for long contexts.
 
-Create a Docker Space and upload `README.md`, `Dockerfile`, `.dockerignore`, `pyproject.toml`, `preload_models.py`, `runtime.py`, `grpo.py`, `server.py`, and `static/` to the repository root. The Docker image installs CPU-only PyTorch and downloads both pinned checkpoints during the build, excluding training checkpoints and optimizer state. Runtime loads the cached files offline. No model token or external inference service is required. The server listens on `0.0.0.0:7860` as required by Spaces.
+Create a Docker Space and upload `README.md`, `Dockerfile`, `.dockerignore`, `pyproject.toml`, `preload_models.py`, `runtime.py`, `critic.py`, `grpo.py`, `ppo.py`, `server.py`, and `static/` to the repository root. The Docker image installs CPU-only PyTorch and downloads both pinned actor checkpoints and the pinned critic during the build, excluding training checkpoints and optimizer state. Runtime loads the cached files offline. No model token or external inference service is required. The server listens on `0.0.0.0:7860` as required by Spaces.
 
 The app uses a single Python process. Results live in memory, with the two most recent analyses retained per model across all visitors. A Space restart or sleep clears them; an evicted result needs a new analysis. CPU Basic can sleep after inactivity and wake when visited. No persistent storage is required.
 
@@ -68,12 +68,28 @@ The notation follows the maximized objective **𝒥_GRPO**: `r_i` is a rollout r
 
 At the collection policy, `ρ_i,t = 1`, `K_i,t = 0`, and neither clipping nor KL changes the first gradient. **Candidate drift** ranges from **−3 to +3**, with zero in the middle, and sets `logit(π_θ) = logit(π_old) + d Â_i`. Positive drift moves with the advantage; negative drift moves against it. Clipping still allows gradients that recover from unfavorable policy moves even outside the clip interval. Here `logit(π)` means log-odds of the sampled token against all other vocabulary entries. This drift is hypothetical, not another model forward pass.
 
-**Preview one step** differentiates this objective with respect to each sampled-token occurrence's logit, holds all other vocabulary logits fixed, takes an ascent step `z_preview = z_θ + η ∂𝒥_GRPO/∂z`, and exactly renormalizes the categorical distribution. Chips show collection probability → preview probability, with signed changes in **percentage points**. Selecting a token shows the candidate probability, step-only change, ρ_i,t, clipping branch, K_i,t, and ascent-gradient contributions. A large step can overshoot; clipping removes a policy-gradient incentive and does not clamp final probabilities. JSON export includes the actual samples, manual rewards, objective settings, notation, and preview calculations. The API also retains `loss = -objective` and its negative gradients for compatibility; the interface consistently displays the maximized objective.
+Every chip automatically shows its collected probability and **`g_i,t = ∂𝒥_GRPO/∂log π_θ(y_i,t | h_i,t)`**, including group/token averaging. Gradients recalculate immediately after editing rewards or objective settings. Positive g locally favors increasing this log-probability; negative g favors decreasing it. Selecting a token shows its unaveraged objective/loss, policy and KL gradient contributions, candidate probability, ratio, and clipping branch. Both sandboxes use flat monochrome styling.
 
-**No model parameter gradients, optimizer updates, or trained-model probabilities are computed.** This is an independent-logit educational preview; shared parameters and repeated prefixes do not couple token occurrences. The interface states this distinction explicitly. Both checkpoints stay unchanged. Up to two completed rollout groups per model are retained alongside an active group; older groups and restarts require resampling. Generation shares the model's existing computation lock with the layer explorer.
+**A gradient is not a probability change.** Actual training computes `∇θ𝒥 = Σ g_i,t ∇θ log π_θ`, updates shared weights with an optimizer, and runs another forward pass to obtain new probabilities. Shared parameters couple tokens; the optimizer and its state affect the result. This sandbox computes no parameter gradients, optimizer updates, or updated-model probabilities. Clipping removes a policy-gradient incentive; it does not clamp probabilities. JSON exports contain samples, rewards, settings, objective values, and local derivatives. For API compatibility, `loss = -objective`, loss derivatives, and legacy independent-logit toy `after`/`delta` fields remain available. The website requests a zero toy step and never displays those fields as measured or predicted training changes.
+
+Both checkpoints stay unchanged. Up to two completed groups per actor and algorithm are retained alongside an active group; older groups and restarts require resampling. Generation shares the actor's existing computation lock with the layer explorer.
 
 The API uses `POST /api/grpo/runs` to start sampling, `GET /api/grpo/runs/{id}` for progress and samples, `POST /api/grpo/runs/{id}/cancel` to stop, and `POST /api/grpo/runs/{id}/preview` for fast reward/objective recalculation. Sampling is asynchronous to keep long CPU requests out of the web proxy's request timeout.
 
+## PPO sandbox
+
+Open `/ppo`. The same real Qwen/Gemma sampling, templates, reasoning controls, rewards, clipping, KL, signed illustrative drift, and exports are available. Five rollouts remain the default. Each token additionally shows **V(history before the token)** and its **GAE advantage**, alongside the collected probability and actor-objective gradient g. The inspector exposes the immediate reward, next V, TD residual, raw/normalized advantage, detached λ-return, and value loss. Values can be overridden per state, reset to checkpoint predictions, or ablated to zero.
+
+The critic is [Open-Reasoner-Zero/Open-Reasoner-Zero-Critic-0.5B](https://huggingface.co/Open-Reasoner-Zero/Open-Reasoner-Zero-Critic-0.5B), pinned to `8c1c681b55b641811b88a41d9a81017f90ac6369` (MIT). It contains a trained Qwen2.5 backbone and scalar `value_head.weight`, loaded strictly using local Transformers classes without executing remote code. Its optional mean/std normalization follows the [original critic implementation](https://github.com/Open-Reasoner-Zero/Open-Reasoner-Zero/blob/main/orz/ppo/models.py). No random head or reward-model substitution is used. A scalar value model has no sampling temperature.
+
+**The critic was trained on mathematical reasoning with another policy and reward distribution.** Its estimates are illustrative, not calibrated values for Qwen3/Gemma or arbitrary custom targets. Changing an answer or reward does not retrain V. The interface explains this mismatch and permits manual value experiments. Both actor and critic stay frozen.
+
+Critic histories are the exact decoded actor prefixes before each action, including control tokens, independently encoded with the critic's own tokenizer. Future text is never used to find a token boundary. Incremental inference crops the critic's KV cache to the longest unchanged token prefix when BPE retokenization changes a suffix. This also handles Gemma's different tokenizer and partial Unicode pieces. A critic history exceeding 4096 critic tokens is rejected without truncation.
+
+EOS **and the token budget** terminate the finite-horizon episode. The final action receives the rollout score; earlier rewards and terminal V are zero. A token-budget completion is scored as displayed, with no infinite-horizon bootstrap. For each rollout, `δ_t = r_t + γ V(h_{t+1}) − V(h_t)` and `Â_t = δ_t + γλ Â_{t+1}` are evaluated backward. Defaults are γ = 1 and λ = 0.95. Optional actor-advantage normalization uses all generated tokens, sample SD + 10⁻⁸, and is off by default. The unnormalized detached λ-return is `R̂_t = Â_t + V(h_t)`; critic loss is `½(V(h_t) − stopgrad(R̂_t))²`. The critic loss and actor objective both average each rollout's token mean, then average the group. KL is an actor-objective penalty, not an additional reward in GAE. Actor and critic parameter gradients are not computed.
+
+PPO routes mirror GRPO under `/api/ppo/runs`. A run progresses through `sampling`, `valuing`, and `ready`; cancellation works in both computation phases. The preview accepts `gamma`, `gae_lambda`, `critic`, and optional per-token `values` overrides in addition to the shared objective settings. Overrides and reward edits reuse fixed samples and cached critic predictions. `/api/status` also reports critic loading status and availability.
+
 ## Interface
 
 - **Model**: switch between Qwen and Gemma. Shapes, token IDs, templates, GQA mapping, method notes and checkpoint links follow the selected model. A switch clears previous results while preserving the input. Both models stay loaded; switching does not reload weights.
@@ -127,7 +143,7 @@ The interface limit is **512 tokens including the continuation**, independently
 ## Validation
 
 ```sh
-uv run --python 3.11 --group dev python -m pytest tests/test_model.py tests/test_grpo.py -q
+uv run --python 3.11 --group dev python -m pytest tests/test_model.py tests/test_grpo.py tests/test_ppo.py -q
 
 # Keep the Python server running in another terminal:
 npm install
@@ -140,4 +156,6 @@ Numerical integration checks use both actual pinned checkpoints: the official re
 
 GRPO checks compare analytic logit gradients with finite differences of the complete objective, verify softmax renormalization and clipping/KL behavior, compare real sampled-token probabilities against an independent teacher-forced forward pass in both checkpoints, check seeded sampling and cancellation, and verify unchanged parameter versions, probe values, absent parameter gradients, and evaluation mode. Browser checks cover both models, editable rewards, zero-advantage groups, ablations, prefixes and thinking, export, cancellation, and responsive layouts.
 
+PPO checks cover GAE/TD recursion, discounted returns, finite-horizon terminal values, detached λ-return targets, value overrides and ablations, global advantage normalization, actor and critic derivatives against finite differences, and real actor/critic jobs with both actor tokenizers. Cached critic values are compared to separate full-history forward passes, including BPE boundary changes and incomplete Unicode pieces, and both networks remain unchanged. `npm run test:ppo` checks the real browser flow, immediate token gradients/V/GAE, value edits, ablations, JSON export, monochrome colors, and responsive 320–1440px layouts.
+
 Browser checks cover the append column, predicted and manual additions in both models, tokenizer previews, limit errors, unchanged prefix IDs, token sources and the active attention color mode after appending. They also cover the reversed layer stack, the sticky bottom input row, native resizing, rendered colors for every head, exact weights and masks at first, intermediate and last query positions, automatic column visibility, peak navigation, linear and logarithmic colors, model switching, templates, control tokens, reasoning, exports, and the absence of page overflow at widths from 320 to 1440 px. Screenshots are saved in `test-results/`.

critic.py

@@ -0,0 +1,105 @@
+"""Pinned, frozen ORZ state-value critic; no remote Python code is executed."""
+from __future__ import annotations
+
+import math
+import os
+import threading
+
+import torch
+from torch import nn
+from transformers import AutoConfig, AutoTokenizer, Qwen2Model, Qwen2PreTrainedModel
+
+CRITIC_ID = "Open-Reasoner-Zero/Open-Reasoner-Zero-Critic-0.5B"
+CRITIC_REVISION = "8c1c681b55b641811b88a41d9a81017f90ac6369"
+CRITIC_MAX_TOKENS = 4096
+
+
+class ValueModel(Qwen2PreTrainedModel):
+    """The checkpoint's Qwen2 backbone and trained scalar value_head."""
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = Qwen2Model(config)
+        self.value_head = nn.Linear(config.hidden_size, 1, bias=False)
+        self.post_init()
+
+
+class CriticRuntime:
+    def __init__(self):
+        self.phase = "loading"
+        self.message = "Loading the ORZ 0.5B value critic…"
+        self.model = self.tokenizer = None
+        self.lock = threading.Lock()
+        self.device = "cpu"
+
+    def status(self):
+        return dict(phase=self.phase, message=self.message, model=CRITIC_ID,
+                    revision=CRITIC_REVISION, device=self.device, busy=self.lock.locked())
+
+    def load(self):
+        try:
+            config = AutoConfig.from_pretrained(CRITIC_ID, revision=CRITIC_REVISION)
+            self.tokenizer = AutoTokenizer.from_pretrained(CRITIC_ID, revision=CRITIC_REVISION)
+            self.model, info = ValueModel.from_pretrained(
+                CRITIC_ID, revision=CRITIC_REVISION, config=config, dtype=torch.float32,
+                attn_implementation="eager", output_loading_info=True)
+            if any(info.get(key) for key in ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")):
+                raise ValueError(f"The value checkpoint did not load exactly: {info}")
+            requested = os.environ.get("QWEN_DEVICE", "auto")
+            self.device = ("mps" if torch.backends.mps.is_available() else "cpu") if requested == "auto" else requested
+            self.model.to(self.device).eval().requires_grad_(False)
+            self.mean = float(getattr(config, "mean", 0.0))
+            self.std = float(getattr(config, "std", 1.0))
+            if not math.isfinite(self.std) or self.std <= 0:
+                raise ValueError("The critic normalization standard deviation must be positive.")
+            self.normalize = bool(getattr(config, "normalize_reward", False))
+            self.phase = "ready"
+            self.message = f"ORZ 0.5B · trained value-head · {self.device.upper()} · frozen"
+        except Exception as exc:
+            import traceback
+            traceback.print_exc()
+            self.phase = "error"
+            self.message = f"Could not load the value critic: {exc}"
+
+    @torch.inference_mode()
+    def values(self, policy, prefix_ids, tokens, progress, cancelled):
+        """Evaluate V(history BEFORE each action), using the critic's tokenizer.
+
+        Retokenize each exact policy history independently. Its final BPE pieces
+        can change as a character is appended, so crop KV cache to the longest
+        unchanged prefix. Never align a state using future text or actor IDs.
+        """
+        result, previous, previous_values, cache = [], [], [], None
+        sampled = [token["id"] for token in tokens]
+        for position in range(len(tokens)):
+            if cancelled():
+                return None
+            progress(position)
+            history = policy.tokenizer.decode(prefix_ids + sampled[:position],
+                skip_special_tokens=False, clean_up_tokenization_spaces=False)
+            ids = self.tokenizer.encode(history, add_special_tokens=False)
+            if not ids or len(ids) > CRITIC_MAX_TOKENS:
+                raise ValueError(f"A critic history must contain 1–{CRITIC_MAX_TOKENS} critic tokens. No text was truncated.")
+            common = 0
+            for old, new in zip(previous, ids):
+                if old != new:
+                    break
+                common += 1
+            if cache is not None:
+                cache.crop(common)
+            state_values = previous_values[:common]
+            if common < len(ids):
+                output = self.model.model(torch.tensor([ids[common:]], device=self.device),
+                    past_key_values=cache, use_cache=True, output_attentions=False, output_hidden_states=False)
+                cache = output.past_key_values
+                scores = self.model.value_head(output.last_hidden_state).squeeze(-1).float()
+                if self.normalize:
+                    scores = (scores - self.mean) / self.std
+                state_values += scores[0].cpu().tolist()
+                del output, scores
+            value = state_values[-1]
+            if not math.isfinite(value):
+                raise ValueError("The critic returned a nonfinite value.")
+            result.append(value)
+            previous, previous_values = ids, state_values
+        return result

grpo.py

@@ -1,4 +1,4 @@
-"""Frozen-policy rollouts and an explicit, independent-logit GRPO sandbox."""
+"""Frozen-policy rollouts and exact local GRPO objective derivatives."""
 from __future__ import annotations
 
 import copy
@@ -39,6 +39,46 @@ def sigmoid(x):
     return e / (1 + e)
 
 
+def token_objective(token, advantage, weight, payload):
+    """The clipped actor objective and its exact local derivatives.
+
+    Advantages, reference probabilities, and collection probabilities are
+    detached. The derivative in log-probability coordinates is a coefficient
+    for backpropagation, not a prediction of a parameter update's effect.
+    Legacy independent-logit step fields remain an explicitly toy calculation.
+    """
+    old = min(max(token["probability"], 1e-12), 1 - 1e-12)
+    old_logit = math.log(old) - math.log1p(-old)
+    current = old if payload.drift == 0 else min(max(sigmoid(old_logit + payload.drift * advantage), 1e-12), 1 - 1e-12)
+    ratio = current / old
+    clipped_ratio = min(max(ratio, 1 - payload.epsilon), 1 + payload.epsilon)
+    clipped = payload.clipping and ((advantage > 0 and ratio > 1 + payload.epsilon)
+                                   or (advantage < 0 and ratio < 1 - payload.epsilon))
+    surrogate = min(ratio * advantage, clipped_ratio * advantage) if payload.clipping else ratio * advantage
+    reference_ratio = old / current
+    kl = max(0.0, reference_ratio - math.log(reference_ratio) - 1)
+    beta = payload.beta if payload.kl else 0.0
+    policy_gradient = 0.0 if clipped else -advantage * ratio
+    kl_gradient = beta * (1 - reference_ratio)
+    logp_gradient = policy_gradient + kl_gradient
+    logit_gradient = weight * logp_gradient * (1 - current)
+    after = sigmoid(math.log(current) - math.log1p(-current) - payload.step_size * logit_gradient)
+    loss = -surrogate + beta * kl
+    return dict(old=old, reference=old, current=current, after=after,
+                delta=after-old, step_delta=after-current,
+                ratio=ratio, clipped_ratio=clipped_ratio, clipped=clipped,
+                kl=kl, policy_gradient=policy_gradient, kl_gradient=kl_gradient,
+                logp_gradient=logp_gradient, logit_gradient=logit_gradient,
+                weight=weight, surrogate=surrogate, loss=loss,
+                advantage=advantage, objective=-loss,
+                objective_contribution=-weight * loss,
+                objective_policy_gradient=-policy_gradient,
+                objective_kl_gradient=-kl_gradient,
+                objective_logp_gradient=-logp_gradient,
+                objective_weighted_logp_gradient=-weight * logp_gradient,
+                objective_logit_gradient=-logit_gradient)
+
+
 def preview_step(rollouts, payload):
     """Differentiate the GRPO loss, then step each occurrence's sampled-token logit.
 
@@ -68,49 +108,25 @@ def preview_step(rollouts, payload):
             raise ValueError("A rollout has no sampled tokens.")
         weight = 1 / (count * length)
         for token in row["tokens"]:
-            old = min(max(token["probability"], 1e-12), 1 - 1e-12)
-            old_logit = math.log(old) - math.log1p(-old)
-            current = old if payload.drift == 0 else min(max(sigmoid(old_logit + payload.drift * advantage), 1e-12), 1 - 1e-12)
-            ratio = current / old
-            clipped_ratio = min(max(ratio, 1 - payload.epsilon), 1 + payload.epsilon)
-            # At the boundary, use the unclipped branch's one-sided derivative.
-            clipped = payload.clipping and ((advantage > 0 and ratio > 1 + payload.epsilon)
-                                           or (advantage < 0 and ratio < 1 - payload.epsilon))
-            surrogate = min(ratio * advantage, clipped_ratio * advantage) if payload.clipping else ratio * advantage
-            reference_ratio = old / current
-            kl = max(0.0, reference_ratio - math.log(reference_ratio) - 1)
-            beta = payload.beta if payload.kl else 0.0
-            # d loss / d log(pi(a|s)); KL uses the sampled k3 estimator.
-            policy_gradient = 0.0 if clipped else -advantage * ratio
-            kl_gradient = beta * (1 - reference_ratio)
-            logp_gradient = policy_gradient + kl_gradient
-            logit_gradient = weight * logp_gradient * (1 - current)
-            after = sigmoid(math.log(current) - math.log1p(-current) - payload.step_size * logit_gradient)
-            loss = -surrogate + beta * kl
-            tokens.append(dict(old=old, reference=old, current=current, after=after,
-                               delta=after-old, step_delta=after-current,
-                               ratio=ratio, clipped_ratio=clipped_ratio, clipped=clipped,
-                               kl=kl, policy_gradient=policy_gradient, kl_gradient=kl_gradient,
-                               logp_gradient=logp_gradient, logit_gradient=logit_gradient,
-                               weight=weight, surrogate=surrogate, loss=loss,
-                               objective=-loss, objective_policy_gradient=-policy_gradient,
-                               objective_kl_gradient=-kl_gradient,
-                               objective_logp_gradient=-logp_gradient,
-                               objective_logit_gradient=-logit_gradient))
-            total_loss += weight * loss
-            total_kl += weight * kl
+            detail = token_objective(token, advantage, weight, payload)
+            tokens.append(detail)
+            total_loss += weight * detail["loss"]
+            total_kl += weight * detail["kl"]
             token_count += 1
-            clipped_count += int(clipped)
+            clipped_count += int(detail["clipped"])
         rows.append(dict(reward=reward, manual=override is not None, advantage=advantage,
                          scored_answer=scored_answer(row["text"]), tokens=tokens))
     # J_GRPO is maximized; retain the equivalent loss L = -J for API compatibility.
     return dict(mean=mean, std=std, objective=-total_loss, loss=total_loss, mean_kl=total_kl,
                 clipped_count=clipped_count, token_count=token_count,
                 all_equal=std == 0, weights_updated=False,
+                parameter_gradients_computed=False, updated_probabilities_computed=False,
                 probability_space="independent sampled-token logits", rollouts=rows)
 
 
 class GRPOJobs:
+    algorithm = "grpo"
+
     def __init__(self):
         self.jobs = OrderedDict()
         self.lock = threading.RLock()
@@ -140,6 +156,7 @@ class GRPOJobs:
             raise BlockingIOError("This model is already computing. Try again when it finishes.")
         sid = uuid.uuid4().hex
         job = dict(id=sid, phase="sampling", message="Starting the rollout group…",
+                   algorithm=self.algorithm,
                    context=context, prefix_tokens=[runtime.token(t) for t in ids],
                    config=payload.model_dump(), rollouts=[], elapsed=0.0,
                    weights_updated=False, **runtime.metadata(), _cancel=threading.Event())
@@ -147,7 +164,7 @@ class GRPOJobs:
             with self.lock:
                 self.jobs[sid] = job
                 completed = [key for key, value in self.jobs.items()
-                             if value["model"] == runtime.model_id and value["phase"] != "sampling"]
+                             if value["model"] == runtime.model_id and value["phase"] not in ("sampling", "valuing")]
                 for key in completed[:-2]:
                     del self.jobs[key]
             threading.Thread(target=self._sample, args=(runtime, job, ids), daemon=True).start()
@@ -156,12 +173,18 @@ class GRPOJobs:
             raise
         return self.snapshot(sid)
 
+    def after_sampling(self, runtime, job, prefix_ids):
+        return not job["_cancel"].is_set()
+
+    def after_finished(self):
+        pass
+
     @torch.inference_mode()
     def _sample(self, runtime, job, prefix_ids):
         started = time.perf_counter()
         config = job["config"]
         phase = "ready"
-        final_message = "Group complete. Assign rewards and inspect the preview."
+        final_message = "Group complete. Assign rewards and inspect the objective gradients."
         try:
             generator = torch.Generator(device="cpu").manual_seed(config["seed"])
             eos = runtime.model.generation_config.eos_token_id
@@ -179,7 +202,7 @@ class GRPOJobs:
                         final_message = "Sampling stopped. Model weights are unchanged."
                         return
                     message = f"Rollout {index+1} / {config['group_size']} · token {position+1} / {config['max_new_tokens']}"
-                    runtime.active = f"GRPO · {message}"
+                    runtime.active = f"{self.algorithm.upper()} · {message}"
                     with self.lock:
                         job.update(message=message, elapsed=round(time.perf_counter()-started, 2))
                     output = runtime.model(ids, past_key_values=cache, use_cache=True,
@@ -202,17 +225,21 @@ class GRPOJobs:
                 del cache
                 with self.lock:
                     row.update(finished=True, stop_reason="eos" if sampled[-1] in eos else "length")
+            if not self.after_sampling(runtime, job, prefix_ids):
+                phase = "cancelled"
+                final_message = "Computation stopped. Model weights are unchanged."
         except Exception as exc:
             import traceback
             traceback.print_exc()
             phase = "error"
-            final_message = f"Sampling failed: {exc}"
+            final_message = f"Computation failed: {exc}"
         finally:
             with self.lock:
                 job.update(phase=phase, message=final_message, elapsed=round(time.perf_counter()-started, 2))
                 runtime.active = None
                 runtime.lock.release()
+                self.after_finished()
                 completed = [key for key, value in self.jobs.items()
-                             if value["model"] == runtime.model_id and value["phase"] != "sampling"]
+                             if value["model"] == runtime.model_id and value["phase"] not in ("sampling", "valuing")]
                 for key in completed[:-2]:
                     del self.jobs[key]

ppo.py

@@ -0,0 +1,126 @@
+"""Frozen-critic PPO: terminal rewards, per-token GAE, actor and critic losses."""
+from __future__ import annotations
+
+import math
+import time
+
+from grpo import GRPOJobs, automatic_reward, scored_answer, token_objective
+
+
+def generalized_advantages(values, reward, gamma, gae_lambda):
+    """Finite-horizon episodes: EOS AND the token budget terminate, V_T = 0."""
+    advantages, residuals = [0.0] * len(values), [0.0] * len(values)
+    running = 0.0
+    for t in reversed(range(len(values))):
+        next_value = values[t + 1] if t + 1 < len(values) else 0.0
+        delta = (reward if t == len(values)-1 else 0.0) + gamma * next_value - values[t]
+        running = delta + gamma * gae_lambda * running
+        residuals[t], advantages[t] = delta, running
+    return advantages, residuals
+
+
+def ppo_objective(rollouts, payload):
+    count = len(rollouts)
+    if count < 2 or len(payload.rewards) != count:
+        raise ValueError("Provide one reward per rollout, with at least two rollouts.")
+    if payload.values is not None and len(payload.values) != count:
+        raise ValueError("Provide one value-override array per rollout.")
+    rewards, data = [], []
+    for index, (row, override) in enumerate(zip(rollouts, payload.rewards)):
+        reward = automatic_reward(row["text"], payload.answer, payload.reward_rule) if override is None else float(override)
+        values = list(row["values"])
+        if not values or len(values) != len(row["tokens"]):
+            raise ValueError("Each generated token needs one critic value.")
+        edits = payload.values[index] if payload.values is not None else None
+        if edits is not None:
+            if len(edits) != len(values):
+                raise ValueError("Value overrides must match the rollout's token count.")
+            values = [value if edit is None else float(edit) for value, edit in zip(values, edits)]
+        if not payload.critic:
+            values = [0.0] * len(values)
+        if not all(math.isfinite(x) for x in [reward, *values]):
+            raise ValueError("Rewards and critic values must be finite numbers.")
+        advantages, residuals = generalized_advantages(values, reward, payload.gamma, payload.gae_lambda)
+        rewards.append(reward)
+        data.append((values, advantages, residuals))
+    flat = [a for _, advantages, _ in data for a in advantages]
+    a_mean = sum(flat) / len(flat)
+    a_std = math.sqrt(sum((a-a_mean)**2 for a in flat) / max(1, len(flat)-1))
+    rows = []
+    objective = mean_kl = critic_loss = 0.0
+    clipped_count = 0
+    for row, reward, override, (values, advantages, residuals) in zip(rollouts, rewards, payload.rewards, data):
+        tokens = []
+        weight = 1 / (count * len(values))
+        for t, (token, value, raw_advantage, residual) in enumerate(zip(row["tokens"], values, advantages, residuals)):
+            advantage = (raw_advantage - a_mean) / (a_std + 1e-8) if payload.normalize else raw_advantage
+            detail = token_objective(token, advantage, weight, payload)
+            target = raw_advantage + value  # Detached lambda-return target.
+            value_loss = 0.5 * (value - target)**2
+            detail.update(value=value, original_value=row["values"][t],
+                          next_value=values[t+1] if t+1 < len(values) else 0.0,
+                          reward=reward if t == len(values)-1 else 0.0,
+                          terminal=t == len(values)-1, td_residual=residual,
+                          raw_advantage=raw_advantage, return_target=target,
+                          value_loss=value_loss, value_gradient=weight * (value-target))
+            tokens.append(detail)
+            objective += detail["objective_contribution"]
+            mean_kl += weight * detail["kl"]
+            critic_loss += weight * value_loss
+            clipped_count += int(detail["clipped"])
+        rows.append(dict(reward=reward, manual=override is not None,
+                         scored_answer=scored_answer(row["text"]), tokens=tokens))
+    mean = sum(rewards)/count
+    std = math.sqrt(sum((r-mean)**2 for r in rewards)/(count-1))
+    return dict(mean=mean, std=std, objective=objective, loss=-objective,
+                critic_loss=critic_loss, mean_kl=mean_kl, clipped_count=clipped_count,
+                token_count=len(flat), all_equal=std == 0, advantage_mean=a_mean,
+                advantage_std=a_std, weights_updated=False, parameter_gradients_computed=False,
+                updated_probabilities_computed=False,
+                terminal_convention="EOS or token budget ends the episode; final reward at last action; V_T=0",
+                rollouts=rows)
+
+
+class PPOJobs(GRPOJobs):
+    algorithm = "ppo"
+
+    def __init__(self, critic):
+        super().__init__()
+        self.critic = critic
+
+    def start(self, runtime, payload, context, ids):
+        if self.critic.phase != "ready":
+            raise ValueError(self.critic.message)
+        if not self.critic.lock.acquire(blocking=False):
+            raise BlockingIOError("The value critic is already computing. Try again when it finishes.")
+        try:
+            return super().start(runtime, payload, context, ids)
+        except Exception:
+            self.critic.lock.release()
+            raise
+
+    def after_sampling(self, runtime, job, prefix_ids):
+        with self.lock:
+            job.update(phase="valuing", critic=self.critic.status())
+        started = time.perf_counter() - job["elapsed"]
+        for index, row in enumerate(job["rollouts"]):
+            def progress(position):
+                message = f"Critic · rollout {index+1} / {len(job['rollouts'])} · state {position+1} / {len(row['tokens'])}"
+                runtime.active = f"PPO · {message}"
+                with self.lock:
+                    job.update(message=message, elapsed=round(time.perf_counter()-started, 2))
+            values = self.critic.values(runtime, prefix_ids, row["tokens"], progress, job["_cancel"].is_set)
+            if values is None:
+                return False
+            with self.lock:
+                row["values"] = values
+        return not job["_cancel"].is_set()
+
+    def after_finished(self):
+        self.critic.lock.release()
+
+    def preview(self, sid, payload):
+        job = self.snapshot(sid)
+        if job["phase"] != "ready":
+            raise ValueError("Wait for the rollouts and critic values to finish.")
+        return ppo_objective(job["rollouts"], payload)

preload_models.py

@@ -2,10 +2,12 @@
 from huggingface_hub import snapshot_download
 
 from runtime import MODELS
+from critic import CRITIC_ID, CRITIC_REVISION
 
 
 if __name__ == "__main__":
-    for model_id, spec in MODELS.items():
+    checkpoints = {**MODELS, CRITIC_ID: dict(revision=CRITIC_REVISION)}
+    for model_id, spec in checkpoints.items():
         snapshot_download(
             repo_id=model_id,
             revision=spec["revision"],

server.py

@@ -15,11 +15,15 @@ from pydantic import BaseModel, ConfigDict, Field
 
 from runtime import Runtime, MAX_TOKENS, MODEL_ID, GEMMA_ID, MODELS
 from grpo import GRPOJobs
+from critic import CriticRuntime
+from ppo import PPOJobs
 
 ROOT = Path(__file__).parent
 ModelChoice = Literal[MODEL_ID, GEMMA_ID]
 runtimes = {model_id: Runtime(model_id) for model_id in MODELS}
 grpo_jobs = GRPOJobs()
+critic = CriticRuntime()
+ppo_jobs = PPOJobs(critic)
 
 
 class Message(BaseModel):
@@ -70,11 +74,21 @@ class PreviewInput(BaseModel):
     drift: float = Field(default=0, ge=-3, le=3)
 
 
+class PPOPreviewInput(PreviewInput):
+    normalize: bool = False
+    step_size: float = Field(default=0, ge=0, le=100)
+    gamma: float = Field(default=1, ge=0, le=1)
+    gae_lambda: float = Field(default=0.95, ge=0, le=1)
+    critic: bool = True
+    values: list[Annotated[list[Annotated[float, Field(ge=-1000000, le=1000000)] | None], Field(max_length=64)] | None] | None = Field(default=None, max_length=12)
+
+
 @asynccontextmanager
 async def lifespan(app):
     def load_models():
         for runtime in runtimes.values():
             runtime.load()
+        critic.load()
     threading.Thread(target=load_models, daemon=True).start()
     yield
 
@@ -82,21 +96,30 @@ async def lifespan(app):
 app = FastAPI(lifespan=lifespan)
 
 
-@app.post("/api/grpo/runs")
-def start_grpo(payload: RolloutInput):
+def start_rollouts(payload, jobs):
     runtime = runtimes[payload.model]
     try:
         context, ids = runtime.prepare(Input(model=payload.model, mode=payload.mode, text=payload.prefix,
             messages=[Message(role="user", content=payload.prefix)], thinking=payload.thinking))
         if payload.mode == "chat" and payload.assistant_prefix:
             context, ids = runtime.prepare(Input(model=payload.model, mode="raw", text=context+payload.assistant_prefix))
-        return grpo_jobs.start(runtime, payload, context, ids)
+        return jobs.start(runtime, payload, context, ids)
     except BlockingIOError as exc:
         raise HTTPException(409, str(exc)) from exc
     except ValueError as exc:
         raise HTTPException(422, str(exc)) from exc
 
 
+@app.post("/api/grpo/runs")
+def start_grpo(payload: RolloutInput):
+    return start_rollouts(payload, grpo_jobs)
+
+
+@app.post("/api/ppo/runs")
+def start_ppo(payload: RolloutInput):
+    return start_rollouts(payload, ppo_jobs)
+
+
 @app.get("/api/grpo/runs/{sid}")
 def get_grpo(sid: str):
     try:
@@ -123,9 +146,35 @@ def preview_grpo(sid: str, payload: PreviewInput):
         raise HTTPException(422, str(exc)) from exc
 
 
+@app.get("/api/ppo/runs/{sid}")
+def get_ppo(sid: str):
+    try:
+        return ppo_jobs.snapshot(sid)
+    except KeyError as exc:
+        raise HTTPException(404, "This PPO group has expired. Sample a new group.") from exc
+
+
+@app.post("/api/ppo/runs/{sid}/cancel")
+def cancel_ppo(sid: str):
+    try:
+        return ppo_jobs.cancel(sid)
+    except KeyError as exc:
+        raise HTTPException(404, "This PPO group has expired.") from exc
+
+
+@app.post("/api/ppo/runs/{sid}/preview")
+def preview_ppo(sid: str, payload: PPOPreviewInput):
+    try:
+        return ppo_jobs.preview(sid, payload)
+    except KeyError as exc:
+        raise HTTPException(404, "This PPO group has expired. Sample a new group.") from exc
+    except ValueError as exc:
+        raise HTTPException(422, str(exc)) from exc
+
+
 @app.get("/api/status")
 def status(model: ModelChoice = MODEL_ID):
-    return dict(**runtimes[model].status(), models=[dict(id=mid, **spec) for mid, spec in MODELS.items()])
+    return dict(**runtimes[model].status(), models=[dict(id=mid, **spec) for mid, spec in MODELS.items()], critic=critic.status())
 
 
 @app.post("/api/prepare")
@@ -235,6 +284,11 @@ def grpo_index():
     return FileResponse(ROOT / "static" / "grpo.html")
 
 
+@app.get("/ppo")
+def ppo_index():
+    return FileResponse(ROOT / "static" / "ppo.html")
+
+
 app.mount("/", StaticFiles(directory=ROOT / "static"), name="static")
 
 if __name__ == "__main__":

static/grpo.css

@@ -35,7 +35,8 @@
 .g-ablation{padding:10px;border:1px solid var(--line);background:#fff;border-radius:5px;cursor:pointer;min-width:0}
 .g-ablation>span{display:flex;align-items:center;gap:8px;font-size:12px;font-weight:600}
 .g-ablation small{display:block;color:var(--muted);font-size:11px;line-height:1.5;margin:6px 0 0 23px}
-.g-slider-grid{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:20px;margin-top:19px}
+.g-ppo-ablations{grid-template-columns:repeat(2,minmax(0,1fr))}
+.g-slider-grid{display:grid;grid-template-columns:repeat(3,minmax(0,1fr));gap:20px;margin-top:19px}
 .g-slider-grid label{display:block;font-size:11px;color:var(--muted);min-width:0}
 .g-slider-grid output{float:right;color:var(--ink);font:12px var(--mono);margin-left:4px}
 .g-slider-grid input{display:block;width:100%;margin:12px 0 0;accent-color:var(--accent);height:16px}
@@ -73,9 +74,15 @@
 .g-token:hover{border-color:var(--accent)}.g-token.active{outline:2px solid var(--accent);outline-offset:1px}
 .g-token-token{font:600 12px var(--mono);max-width:100%;overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
 .g-token-prob{font:10px var(--mono);color:#555;white-space:nowrap}
-.g-token-delta{font:600 11px var(--mono);white-space:nowrap}
+.g-token-gradient{font:600 11px var(--mono);white-space:nowrap}
+.g-token-value{font:10px var(--mono);color:#555;white-space:nowrap}
+.g-value-editor{display:flex;align-items:end;gap:14px;flex-wrap:wrap;margin:15px 0 8px}
+.g-value-editor label{font-size:12px;color:var(--muted)}
+.g-value-editor input[type=number]{max-width:180px}
+.g-critic-note{border-left:2px solid #999;padding-left:12px}
+.g-ppo-summary{grid-template-columns:repeat(4,minmax(0,1fr))}
 .g-token.up{background:#fff;border-color:#aaa}.g-token.down{background:#fff;border-color:#aaa}
-.g-token.up .g-token-delta{color:#222}.g-token.down .g-token-delta{color:#222}
+.g-token.up .g-token-gradient{color:#222}.g-token.down .g-token-gradient{color:#222}
 .g-token.is-clipped{border-style:dashed;border-color:#555}
 .g-completion{margin-top:12px;color:var(--muted);font-size:11px}.g-completion summary{cursor:pointer}.g-completion pre{white-space:pre-wrap;overflow-wrap:anywhere;max-height:180px;overflow:auto;padding:10px;background:var(--soft);font:12px/1.7 var(--mono)}
 .g-prefix-details{margin:18px 0;color:var(--muted);font-size:12px}.g-prefix-details summary{cursor:pointer}.g-prefix-details pre{white-space:pre-wrap;overflow-wrap:anywhere;font:12px/1.7 var(--mono);max-height:240px;overflow:auto;padding:12px;background:var(--soft);border-radius:4px}
@@ -91,6 +98,7 @@
 .g-equation-scroll{min-width:0;max-width:100%;overflow-x:auto;padding:12px 4px;scrollbar-width:thin}
 .g-equation-scroll math{font-size:21px;width:max-content;margin:0 auto}
 @media(max-width:600px){.g-equation-scroll math{font-size:18px}}
+@media(max-width:820px){.g-ppo-summary{grid-template-columns:repeat(2,minmax(0,1fr));gap:18px}.g-ppo-summary article:nth-child(3){border:0;padding-left:0}}
 .grpo-page .site-footer{gap:15px;flex-wrap:wrap}
 @media(min-width:1500px){.g-workspace{grid-template-columns:310px minmax(0,1fr);gap:38px}}
 @media(max-width:1100px){.g-workspace{grid-template-columns:265px minmax(0,1fr);gap:22px}.g-slider-grid{grid-template-columns:repeat(2,minmax(0,1fr))}.g-ablation-grid{grid-template-columns:1fr}.g-ablation{display:flex;justify-content:space-between;align-items:center;gap:12px}.g-ablation small{margin:0;text-align:right}.g-token-metrics{grid-template-columns:repeat(2,minmax(0,1fr))}.g-process{display:none}}

static/grpo.html

@@ -13,12 +13,12 @@
 <body class="grpo-page">
   <header class="site-header">
     <a class="brand" href="/"><span class="brand-mark">L</span> Layer lab <span class="brand-light">/</span><span class="brand-caption">GRPO sandbox</span></a>
-    <nav><a class="mode-link" href="/">Layers & attention</a><a class="mode-link active" href="/grpo" aria-current="page">GRPO</a><span class="model-pill">Weights frozen</span></nav>
+    <nav><a class="mode-link" href="/">Layers & attention</a><a class="mode-link active" href="/grpo" aria-current="page">GRPO</a><a class="mode-link" href="/ppo">PPO</a><span class="model-pill">Weights frozen</span></nav>
   </header>
   <main>
     <div class="intro">
-      <div><p class="eyebrow">GROUP RELATIVE POLICY OPTIMIZATION</p><h1>One group. One step.</h1><p class="intro-description">Sample real completions, assign rewards, and follow the gradient.<br>Explore a step in logit space while the model stays fixed.</p></div>
-      <div class="g-process" aria-label="Experiment steps"><span><b>01</b> Sample</span><i>→</i><span><b>02</b> Reward</span><i>→</i><span><b>03</b> Preview</span></div>
+      <div><p class="eyebrow">GROUP RELATIVE POLICY OPTIMIZATION</p><h1>One group. Follow the gradient.</h1><p class="intro-description">Sample real completions, assign rewards, and inspect each token's learning signal.<br>Exact objective derivatives. Frozen model weights.</p></div>
+      <div class="g-process" aria-label="Experiment steps"><span><b>01</b> Sample</span><i>→</i><span><b>02</b> Reward</span><i>→</i><span><b>03</b> Gradient</span></div>
     </div>
     <div class="g-workspace">
       <aside class="controls g-controls" aria-label="Rollout settings">
@@ -64,7 +64,7 @@
       </aside>
       <div class="g-results">
         <section class="g-step-panel" aria-labelledby="g-step-title">
-          <div class="g-section-heading"><div><p class="eyebrow">03 / THE OBJECTIVE</p><h2 id="g-step-title">Build the step</h2></div><button id="g-export" class="text-button" disabled>Export JSON ↓</button></div>
+          <div class="g-section-heading"><div><p class="eyebrow">03 / THE OBJECTIVE</p><h2 id="g-step-title">Inspect the objective</h2></div><button id="g-export" class="text-button" disabled>Export JSON ↓</button></div>
           <div class="g-ablation-grid">
             <label class="g-ablation"><span><input id="g-normalize" type="checkbox" checked> Scale advantages Âᵢ</span><small>(rᵢ − mean(r)) / (std(r) + δ)</small></label>
             <label class="g-ablation"><span><input id="g-clipping" type="checkbox" checked> Clip the ratio ρᵢ,ₜ</span><small>ρᶜˡⁱᵖᵢ,ₜ = clip(ρᵢ,ₜ, 1−ε, 1+ε)</small></label>
@@ -73,11 +73,10 @@
           <div class="g-slider-grid">
             <label>Clip ε <output id="g-epsilon-value">0.20</output><input id="g-epsilon" type="range" min="0.01" max="0.5" step="0.01" value="0.2"></label>
             <label>KL β <output id="g-beta-value">0.04</output><input id="g-beta" type="range" min="0" max="1" step="0.01" value="0.04"></label>
-            <label>Preview step size η <output id="g-step-value">10.0</output><input id="g-step" type="range" min="0" max="100" step="0.5" value="10"></label>
             <label>Candidate drift d <output id="g-drift-value">0.00</output><input id="g-drift" type="range" min="-3" max="3" step="0.05" value="0"></label>
           </div>
           <p id="g-origin-note" class="g-explanation">At d = 0, πθ = πold = πref: ρᵢ,ₜ = 1 and Kᵢ,ₜ = 0. Clipping and KL do not change this first gradient. Move drift in either direction to explore their effects.</p>
-          <div class="g-preview-action"><button id="g-preview" class="primary-button" disabled><span>Preview one step</span><span>→</span></button><button id="g-hide-preview" class="text-button hidden">Hide probability changes</button><span id="g-preview-status" class="field-note" role="status">Sample a group first.</span></div>
+          <p id="g-preview-status" class="field-note" role="status">Sample a group to see gradients on every token automatically.</p>
         </section>
         <div class="g-summary" aria-live="polite">
           <article><span>Mean reward · mean(r)</span><strong id="g-mean">—</strong><small id="g-std">Group baseline</small></article>
@@ -89,13 +88,13 @@
           <div id="g-progress" class="g-progress hidden" role="status"><span class="spinner"></span><span id="g-progress-text">Sampling…</span></div>
           <p id="g-group-note" class="g-explanation">Choose a prefix and target, then sample a group. No completions are prewritten.</p>
           <p id="g-equal" class="g-warning hidden">All rewards rᵢ are equal → every advantage Âᵢ is zero. Edit a reward or resample to see a learning signal.</p>
-          <div class="g-token-legend"><span id="g-token-legend">Each token shows its collection probability.</span><span><i class="g-swatch" aria-hidden="true">+</i> Increase <i class="g-swatch negative" aria-hidden="true">−</i> Decrease <i class="g-swatch clipped"></i> Clipped</span></div>
+          <div class="g-token-legend"><span id="g-token-legend">Each token shows its collection probability and objective gradient.</span><span><i class="g-swatch" aria-hidden="true">+</i> Positive g <i class="g-swatch negative" aria-hidden="true">−</i> Negative g <i class="g-swatch clipped"></i> Clipped</span></div>
           <div id="g-rollouts" class="g-rollouts"><div class="g-empty"><span>G</span><h3>A group makes the comparison.</h3><p>Five real rollouts by default.<br>Rewards become advantages relative to this group.</p></div></div>
           <details id="g-prefix-details" class="g-prefix-details hidden"><summary>Inspect the exact collection prefix</summary><pre id="g-formatted-prefix"></pre><div id="g-prefix-tokens" class="g-prefix-tokens"></div></details>
         </section>
         <section id="g-inspector" class="g-inspector hidden" aria-labelledby="g-inspector-title"><div class="g-section-heading"><div><p class="eyebrow">SELECTED TOKEN</p><h2 id="g-inspector-title">Follow the gradient</h2></div></div><div id="g-token-detail"></div></section>
         <section class="g-method" id="g-method">
-          <p class="eyebrow">WHAT THIS PREVIEW COMPUTES</p><h2>A real objective. An illustrative probability step.</h2>
+          <p class="eyebrow">WHAT THIS SANDBOX COMPUTES</p><h2>Real samples. Exact objective derivatives.</h2>
           <p>Rollouts and collection probabilities come from the selected checkpoint. The reference stays equal to that frozen collection policy. Rewards are assigned to complete rollouts; their group-relative advantages apply to every generated token, including the stop token. The prefix receives no reward gradient.</p>
           <div class="g-equations" aria-label="GRPO objective and notation">
             <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="J GRPO is the expectation of the group and token averaged minimum of rho times A hat and clipped rho times A hat, minus beta K."><mrow><msub><mi>𝒥</mi><mtext>GRPO</mtext></msub><mo>=</mo><mi>𝔼</mi><mo>[</mo><mfrac><mi>1</mi><mi>G</mi></mfrac><munder><mo>∑</mo><mi>i</mi></munder><mfrac><mi>1</mi><mrow><mo>|</mo><msub><mi>y</mi><mi>i</mi></msub><mo>|</mo></mrow></mfrac><munder><mo>∑</mo><mi>t</mi></munder><mrow><mo>(</mo><mrow><mrow><mi mathvariant="normal">min</mi><mrow><mo>(</mo><mrow><msub><mi>ρ</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>,</mo><msubsup><mi>ρ</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow><mtext>clip</mtext></msubsup><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>−</mo><mi>β</mi><msub><mi>K</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub></mrow><mo>)</mo></mrow><mo>]</mo></mrow></math></div>
@@ -104,10 +103,11 @@
             <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="A hat i is reward r i minus the group mean reward, divided by the reward standard deviation plus delta."><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mrow><mi mathvariant="normal">mean</mi><mrow><mo>(</mo><mi mathvariant="bold">r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mrow><mi mathvariant="normal">std</mi><mrow><mo>(</mo><mi mathvariant="bold">r</mi><mo>)</mo></mrow></mrow><mo>+</mo><mi>δ</mi></mrow></mfrac></mrow></math></div>
           </div>
           <p>Here G is the number of rollouts, yᵢ is a generated completion, and hᵢ,ₜ is the prefix plus its earlier tokens. Reward rᵢ belongs to the whole rollout. πθ denotes the candidate policy and πold the collection policy. The frozen reference πref equals πold. Kᵢ,ₜ is the sampled KL estimate <code>πref/πθ − log(πref/πθ) − 1</code> for the same token and history. We use sample standard deviation and δ = 10⁻⁸; disabling scaling leaves <code>Âᵢ = rᵢ − mean(r)</code>. The displayed 𝒥GRPO is one group's estimate of the expectation.</p>
-          <p>Candidate drift sets <code>logit(πθ) = logit(πold) + dÂᵢ</code>, with d from −3 to +3. Positive drift moves with the advantage; negative drift moves against it. The preview takes one exact gradient-ascent step on 𝒥GRPO for each occurrence's sampled-token logit, holds all other vocabulary logits fixed, then renormalizes. Step size η belongs to this illustration and is not a model learning rate.</p>
-          <p><strong>No model parameter gradients or optimizer steps are computed.</strong> Token occurrences are independent in this sandbox; it does not simulate how shared network parameters couple tokens or repeated prefixes. The displayed probabilities are a local logit-space preview, not predictions from a trained model. Neither ratios nor final probabilities are forcibly clamped by GRPO clipping.</p>
+          <p>Each chip shows the collection probability and <code>gᵢ,ₜ = ∂𝒥GRPO/∂log πθ(yᵢ,ₜ | hᵢ,ₜ)</code>, including group/token averaging. Positive g locally favors increasing this log-probability; negative g favors decreasing it. Selecting a token also shows its objective contribution and equivalent loss. The loss is the negative of the maximized objective.</p>
+          <p>Candidate drift sets <code>logit(πθ) = logit(πold) + dÂᵢ</code>, with d from −3 to +3. This is a synthetic candidate for exploring clipping and KL, not a model forward pass or a measured training change. Positive drift moves with the advantage; negative drift moves against it. At zero drift the candidate is exactly the collection policy.</p>
+          <p><strong>A gradient is not a probability change.</strong> Actual training backpropagates <code>∇θ𝒥 = Σ gᵢ,ₜ ∇θ log πθ</code>, updates shared weights with an optimizer, then computes new probabilities with another forward pass. Shared parameters couple tokens, and optimizer state affects the result. This sandbox computes no parameter gradients or optimizer steps and reports no updated-model probabilities. GRPO clipping removes an incentive; it does not forcibly clamp probabilities.</p>
         </section>
-        <footer class="site-footer"><span>Layer lab <i>/</i> GRPO sandbox</span><span>One group · one hypothetical step · frozen weights</span></footer>
+        <footer class="site-footer"><span>Layer lab <i>/</i> GRPO sandbox</span><span>Samples · rewards · objective gradients · frozen weights</span></footer>
       </div>
     </div>
   </main>

static/grpo.js

@@ -4,7 +4,11 @@ const visible = text => String(text).replace(/ /g,'·').replace(/\n/g,'↵').rep
 const pct = p => `${(p*100).toFixed(p>=.1?1:p>=.001?2:3)}%`;
 const number = n => Number(n).toFixed(4);
 const signed = (n, digits=2) => `${n<0?'−':'+'}${Math.abs(n).toFixed(digits)}`;
-const state = {job:null, preview:null, rewards:[], showStep:false, selected:null, status:null, busy:false, previewBusy:false};
+const signal = n => n===0?'0':`${n<0?'−':'+'}${Math.abs(n).toPrecision(3)}`;
+const PPO = location.pathname==='/ppo';
+const algorithm = PPO?'PPO':'GRPO', apiRoot=`/api/${algorithm.toLowerCase()}/runs`;
+const computing = job => ['sampling','valuing'].includes(job?.phase);
+const state = {job:null, preview:null, rewards:[], values:[], selected:null, status:null, busy:false, previewBusy:false};
 let statusTimer, jobTimer, previewTimer, previewVersion=0, editor=null;
 
 async function api(path, body) {
@@ -23,34 +27,37 @@ function collectionPayload(){
 function objectivePayload(){
   return {answer:$('#g-answer').value, reward_rule:$('#g-rule').value, rewards:state.rewards,
     normalize:$('#g-normalize').checked, clipping:$('#g-clipping').checked, epsilon:Number($('#g-epsilon').value),
-    kl:$('#g-kl').checked, beta:Number($('#g-beta').value), step_size:Number($('#g-step').value), drift:Number($('#g-drift').value)};
+    kl:$('#g-kl').checked, beta:Number($('#g-beta').value), step_size:0, drift:Number($('#g-drift').value),
+    ...(PPO?{gamma:Number($('#g-gamma').value), gae_lambda:Number($('#g-lambda').value), critic:$('#g-critic').checked, values:state.values}: {})};
 }
 function updateControls(){
-  const sampling=state.busy||state.job?.phase==='sampling', ready=state.job?.phase==='ready';
+  const sampling=state.busy||computing(state.job), ready=state.job?.phase==='ready';
   document.querySelectorAll('[data-sampling]').forEach(el=>{el.disabled=sampling;});
   $('#g-example').disabled=sampling;
   const raw=$('#g-mode').value==='raw', native=$('#g-model').value==='Qwen/Qwen3-0.6B';
   $('#g-assistant-field').classList.toggle('hidden',raw);
   $('#g-thinking').disabled=sampling||raw||!native;
   $('#g-thinking-note').textContent=!native?'Gemma learns Reasoning / Solution text; no native thinking switch.':raw?'Reasoning is controlled by the exact Raw prefix.':"Qwen's native thinking template.";
-  $('#g-sample').disabled=sampling||state.status?.phase!=='ready'||Boolean(state.status?.active);
+  $('#g-sample').disabled=sampling||state.status?.phase!=='ready'||Boolean(state.status?.active)||(PPO&&(state.status?.critic?.phase!=='ready'||state.status?.critic?.busy));
   $('#g-sample span:first-child').textContent=sampling?'Sampling…':`Sample ${$('#g-count').value||5} rollouts`;
-  $('#g-cancel').classList.toggle('hidden',state.job?.phase!=='sampling');
-  $('#g-preview').disabled=sampling||!ready||state.previewBusy||!state.preview;
-  $('#g-preview span:first-child').textContent=state.showStep?'Preview shown':'Preview one step';
-  $('#g-hide-preview').classList.toggle('hidden',!state.showStep);
+  $('#g-cancel').classList.toggle('hidden',!computing(state.job));
+  $('#g-cancel').textContent=state.job?.phase==='valuing'?'Stop critic evaluation':'Stop sampling';
   $('#g-reset-rewards').disabled=sampling||!ready;
   $('#g-export').disabled=sampling||!ready||!state.preview||state.previewBusy;
   document.querySelectorAll('[data-reward]').forEach(el=>{el.disabled=sampling||!ready;});
   $('#g-epsilon').disabled=!$('#g-clipping').checked;
   $('#g-beta').disabled=!$('#g-kl').checked;
-  $('#g-token-legend').textContent=state.showStep?'Collected πold → preview πpreview · Δ in percentage points vs collected':'Each token shows πold(yᵢ,ₜ | hᵢ,ₜ), its collection probability.';
-  for(const [id,digits] of [['epsilon',2],['beta',2],['step',1]])$('#g-'+id+'-value').textContent=Number($('#g-'+id).value).toFixed(digits);
+  $('#g-token-legend').textContent='πold = collected probability · g = ∂𝒥/∂log πθ, including averaging · g is not a probability change';
+  for(const [id,digits] of [['epsilon',2],['beta',2],...(PPO?[['gamma',2],['lambda',2]]:[])])$('#g-'+id+'-value').textContent=Number($('#g-'+id).value).toFixed(digits);
   const drift=Number($('#g-drift').value);
   $('#g-drift-value').textContent=drift===0?'0.00':signed(drift);
   $('#g-origin-note').textContent=Number($('#g-drift').value)===0?
     'At d = 0, πθ = πold = πref: ρᵢ,ₜ = 1 and Kᵢ,ₜ = 0. Clipping and KL do not change this first gradient. Move drift in either direction to explore their effects.':
-    'Candidate drift shifts each sampled-token logit by d × Âᵢ: positive d moves with the advantage; negative d moves against it. Displayed Δ includes both drift and gradient ascent on 𝒥GRPO. Click a token to separate them.';
+    `Illustrative candidate drift shifts each sampled-token logit by d × ${PPO?'Âᵢ,ₜ':'Âᵢ'}. Positive d moves with the advantage; negative d moves against it. This synthetic candidate helps explore clipping and KL; it is not a model update.`;
+  if(PPO){
+    $('#g-critic-status').textContent=state.status?.critic?.message||'Connecting to the value critic…';
+    $('#g-reset-values').disabled=sampling||!ready;
+  }
 }
 function samplingChanged(){
   if(state.job)$('#g-stale').classList.remove('hidden');
@@ -70,23 +77,23 @@ async function pollStatus(){
     $('#g-status').textContent=result.active||result.message;
     renderSpecial(result);updateControls();
   }catch(exc){if(model===$('#g-model').value){state.status=null;$('#g-status').textContent=`Cannot reach the server: ${exc.message}`;updateControls();}}
-  if(model===$('#g-model').value)statusTimer=setTimeout(pollStatus,state.job?.phase==='sampling'||state.status?.phase!=='ready'?1800:5000);
+  if(model===$('#g-model').value)statusTimer=setTimeout(pollStatus,computing(state.job)||state.status?.phase!=='ready'||(PPO&&state.status?.critic?.phase!=='ready')?1800:5000);
 }
 
 function tokenMarkup(token, ri, ti){
-  const detail=state.preview?.rollouts[ri]?.tokens[ti], show=state.showStep&&detail;
-  const change=show?detail.delta:0;
+  const detail=state.preview?.rollouts[ri]?.tokens[ti], show=Boolean(detail);
+  const change=detail?.objective_weighted_logp_gradient||0;
   const direction=change>1e-10?'up':change< -1e-10?'down':'';
-  const title=`Token ${ti} · ID ${token.id} · BPE ${token.piece}\nCollected πold: ${pct(token.probability)}${detail?`\nCurrent πθ: ${pct(detail.current)}\nPreview πpreview: ${pct(detail.after)}\nTotal change: ${signed(detail.delta*100,6)} pp\nStep alone: ${signed(detail.step_delta*100,6)} pp\nRatio ρᵢ,ₜ: ${number(detail.ratio)} · ${detail.clipped?'policy gradient clipped':'policy gradient active'}`:''}`;
-  return `<button type="button" class="g-token ${show?direction:''} ${show&&detail.clipped?'is-clipped':''} ${state.selected?.[0]===ri&&state.selected?.[1]===ti?'active':''}" data-token="${ri}:${ti}" data-old="${token.probability}" ${detail?`data-after="${detail.after}" data-delta="${detail.delta}" data-clipped="${detail.clipped}"`:''} title="${escape(title)}" aria-label="${escape(title)}">${show?`<span class="g-token-delta">${signed(change*100)} pp</span>`:''}<span class="g-token-token">${escape(visible(token.text))}</span><span class="g-token-prob">${pct(token.probability)}${show?` → ${pct(detail.after)}`:''}</span></button>`;
+  const title=`Token ${ti} · ID ${token.id} · BPE ${token.piece}\nCollected πold: ${pct(token.probability)}${detail?`\nGradient g = ∂𝒥/∂log πθ: ${change}\nToken objective (before averaging): ${detail.objective}\nIllustrative candidate πθ: ${pct(detail.current)}\nRatio ρᵢ,ₜ: ${number(detail.ratio)} · ${detail.clipped?'policy gradient clipped':'policy gradient active'}${PPO?`\nV(h): ${detail.value}\nGAE advantage: ${detail.advantage}\nTD residual: ${detail.td_residual}`:''}\nNo probability update or model-weight gradient is computed.`:''}`;
+  return `<button type="button" class="g-token ${show?direction:''} ${show&&detail.clipped?'is-clipped':''} ${state.selected?.[0]===ri&&state.selected?.[1]===ti?'active':''}" data-token="${ri}:${ti}" data-old="${token.probability}" ${detail?`data-gradient="${change}" data-clipped="${detail.clipped}"`:''} title="${escape(title)}" aria-label="${escape(title)}">${show?`<span class="g-token-gradient">g ${signal(change)}</span>`:''}<span class="g-token-token">${escape(visible(token.text))}</span><span class="g-token-prob">πold ${pct(token.probability)}</span>${PPO&&show?`<span class="g-token-value">V ${number(detail.value)}</span><span class="g-token-value">Â ${signal(detail.advantage)}</span>`:''}</button>`;
 }
 function renderCards(){
   const job=state.job;if(!job)return;
-  $('#g-rollouts').innerHTML=job.rollouts.map((row,index)=>`<article class="g-rollout" data-rollout="${index}"><div class="g-rollout-head"><div class="g-rollout-id">Rollout ${String(index+1).padStart(2,'0')} <small>${row.tokens.length} tokens · ${row.finished?(row.stop_reason==='eos'?'stop token':'token limit'):'sampling…'}</small>${row.stop_reason==='length'?'<small class="g-truncated">Truncated</small>':''}</div><div class="g-reward-fields"><label><span>Reward r<sub>${index+1}</sub></span> <input aria-label="Reward for rollout ${index+1}" type="number" step="any" min="-1000000" max="1000000" value="0" data-reward="${index}" ${job.phase==='ready'?'':'disabled'}></label><span class="g-advantage" data-advantage="${index}" title="Group-relative advantage Âᵢ">Âᵢ —</span></div></div><div class="g-rollout-body"><p class="g-answer-line" data-answer="${index}">Reward assigned after the group completes.</p><div class="g-token-strip" data-strip="${index}">${row.tokens.map((token,ti)=>tokenMarkup(token,index,ti)).join('')}</div><details class="g-completion"><summary>Completion text</summary><pre>${escape(row.text||'(No visible text yet)')}</pre></details></div></article>`).join('');
+  $('#g-rollouts').innerHTML=job.rollouts.map((row,index)=>`<article class="g-rollout" data-rollout="${index}"><div class="g-rollout-head"><div class="g-rollout-id">Rollout ${String(index+1).padStart(2,'0')} <small>${row.tokens.length} tokens · ${row.finished?(row.stop_reason==='eos'?'stop token':'token limit'):'sampling…'}</small>${row.stop_reason==='length'?'<small class="g-truncated">Truncated</small>':''}</div><div class="g-reward-fields"><label><span>Reward r<sub>${index+1}</sub></span> <input aria-label="Reward for rollout ${index+1}" type="number" step="any" min="-1000000" max="1000000" value="0" data-reward="${index}" ${job.phase==='ready'?'':'disabled'}></label><span class="g-advantage" data-advantage="${index}" title="${PPO?'Per-token state values and GAE advantages':'Group-relative advantage Âᵢ'}">${PPO?'V / Â pending':'Âᵢ —'}</span></div></div><div class="g-rollout-body"><p class="g-answer-line" data-answer="${index}">Reward assigned after the group completes.</p><div class="g-token-strip" data-strip="${index}">${row.tokens.map((token,ti)=>tokenMarkup(token,index,ti)).join('')}</div><details class="g-completion"><summary>Completion text</summary><pre>${escape(row.text||'(No visible text yet)')}</pre></details></div></article>`).join('');
   if(state.preview)renderPreview();
 }
 function renderJob(){
-  const job=state.job, sampling=job.phase==='sampling';
+  const job=state.job, sampling=computing(job);
   $('#g-progress').classList.toggle('hidden',!sampling);
   $('#g-progress-text').textContent=`${job.message} · ${job.elapsed.toFixed(1)} s`;
   $('#g-group-note').textContent=`${job.name} · G = ${job.config.group_size} rollouts · seed ${job.config.seed} · temperature ${job.config.temperature} · ${job.elapsed.toFixed(1)} s. ${sampling?'Sampling from the fixed checkpoint.':job.phase==='ready'?'Rewards can be changed without sampling again.':job.message}`;
@@ -102,16 +109,18 @@ function renderPreview(){
   $('#g-loss').textContent=number(preview.objective);
   $('#g-clipped-count').textContent=`${preview.clipped_count} / ${preview.token_count}`;
   $('#g-kl-metric').textContent=`Mean Kᵢ,ₜ: ${number(preview.mean_kl)}${$('#g-kl').checked?'':' · penalty off'}`;
-  $('#g-equal').classList.toggle('hidden',!preview.all_equal);
+  $('#g-equal').classList.toggle('hidden',PPO||!preview.all_equal);
   preview.rollouts.forEach((row,index)=>{
     const input=$(`[data-reward="${index}"]`);
     if(input&&document.activeElement!==input)input.value=row.reward;
     const adv=$(`[data-advantage="${index}"]`);
-    adv.innerHTML=`Â<sub>${index+1}</sub> ${signed(row.advantage,3)}`;adv.className=`g-advantage ${row.advantage>0?'g-positive':row.advantage<0?'g-negative':''}`;
+    adv.innerHTML=PPO?'Per-token V & Â below':`Â<sub>${index+1}</sub> ${signed(row.advantage,3)}`;adv.className=`g-advantage ${row.advantage>0?'g-positive':row.advantage<0?'g-negative':''}`;
+    if(PPO)adv.title='PPO uses a separate state value and GAE advantage at every token.';
     $(`[data-answer="${index}"]`).innerHTML=`Scored answer: <strong>${escape(row.scored_answer||'(empty / unfinished reasoning)')}</strong>${row.manual?'<span class="g-manual-tag">MANUAL REWARD</span>':` · ${$('#g-rule').value==='exact'?'exact match':'contains answer'}`}`;
     $(`[data-strip="${index}"]`).innerHTML=state.job.rollouts[index].tokens.map((token,ti)=>tokenMarkup(token,index,ti)).join('');
   });
-  $('#g-preview-status').textContent=state.showStep?'Logit-space preview · weights unchanged':'Rewards ready. Preview the hypothetical step.';
+  $('#g-preview-status').textContent='Objective and token gradients update automatically. Probabilities and weights stay fixed.';
+  if(PPO)$('#g-critic-loss').textContent=number(preview.critic_loss);
   renderInspector();updateControls();
 }
 function renderInspector(){
@@ -120,8 +129,10 @@ function renderInspector(){
   if(!detail)return;
   const token=state.job.rollouts[selected[0]].tokens[selected[1]], row=state.preview.rollouts[selected[0]];
   $('#g-inspector-title').textContent=`Rollout ${selected[0]+1} · token ${selected[1]} · ${visible(token.text)}`;
-  const metrics=[['Collection πold',pct(detail.old)],['Candidate πθ',pct(detail.current)],['Preview πpreview',pct(detail.after)],['Step alone',signed(detail.step_delta*100,4)+' pp'],['Advantage Âᵢ',signed(row.advantage,4)],['Policy ratio ρᵢ,ₜ',number(detail.ratio)],['Sampled KL Kᵢ,ₜ',number(detail.kl)],['Policy branch',detail.clipped?'Clipped · zero slope':'Unclipped']];
-  $('#g-token-detail').innerHTML=`<div class="g-token-metrics">${metrics.map(([label,value])=>`<div><span>${label}</span><strong>${escape(value)}</strong></div>`).join('')}</div><p class="g-gradient-line">Token objective, before averaging: policy contribution to the log-probability gradient <code>${number(detail.objective_policy_gradient)}</code> · KL contribution <code>${number(detail.objective_kl_gradient)}</code>.<br>After the softmax derivative and group/token averaging: <code>∂𝒥GRPO/∂z = ${number(detail.objective_logit_gradient)}</code> (averaging weight <code>${detail.weight.toPrecision(4)}</code>).<br>Gradient ascent: <code>z_preview = z_θ + η × ∂𝒥GRPO/∂z</code>. Other vocabulary logits stay fixed; softmax renormalizes the entire distribution.${detail.clipped?' The policy term has zero slope here. An enabled KL term can still move the token.':''}</p>`;
+  const metrics=[['Collection πold',pct(detail.old)],['Illustrative candidate πθ',pct(detail.current)],['Token objective jᵢ,ₜ',number(detail.objective)],['Token loss −jᵢ,ₜ',number(detail.loss)],[PPO?'GAE advantage Âᵢ,ₜ':'Advantage Âᵢ',signed(detail.advantage,4)],['Policy ratio ρᵢ,ₜ',number(detail.ratio)],['Sampled KL Kᵢ,ₜ',number(detail.kl)],['Policy branch',detail.clipped?'Clipped · zero slope':'Unclipped'],['Gradient g = ∂𝒥/∂log πθ',signal(detail.objective_weighted_logp_gradient)],...(PPO?[['Value V(hᵢ,ₜ)',number(detail.value)],['Next value V(hᵢ,ₜ₊₁)',number(detail.next_value)],['Immediate reward',number(detail.reward)],['TD residual δᵢ,ₜ',number(detail.td_residual)],['Raw GAE advantage',number(detail.raw_advantage)],['λ-return target',number(detail.return_target)],['Value loss ½(V − R̂)²',number(detail.value_loss)]]:[])];
+  const editing=document.activeElement?.id==='g-value-edit';
+  $('#g-token-detail').innerHTML=`<div class="g-token-metrics">${metrics.map(([label,value])=>`<div><span>${label}</span><strong>${escape(value)}</strong></div>`).join('')}</div><p class="g-gradient-line">Before averaging: policy contribution to <code>∂j/∂log πθ = ${number(detail.objective_policy_gradient)}</code> · KL contribution <code>${number(detail.objective_kl_gradient)}</code>.<br>The chip shows <code>g = ∂𝒥${algorithm}/∂log πθ = ${signal(detail.objective_weighted_logp_gradient)}</code>, including averaging weight <code>${detail.weight.toPrecision(4)}</code>.<br>A positive g locally favors increasing this log-probability; a negative g favors decreasing it. Actual training uses <code>∇θ𝒥 = Σ gᵢ,ₜ ∇θ log πθ</code>, an optimizer step, and a new forward pass. Shared model weights couple all tokens, so g does not predict Δ probability.${detail.clipped?' The clipped policy term has zero slope here; an enabled KL term can still contribute.':''}</p>${PPO?`<div class="g-value-editor"><label for="g-value-edit">Override this state's V <input id="g-value-edit" type="number" step="any" min="-1000000" max="1000000" value="${detail.value}" ${$('#g-critic').checked?'':'disabled'}></label><button type="button" id="g-reset-value" class="text-button">Use checkpoint value (${number(detail.original_value)})</button></div><p class="field-note">Value is evaluated before this token. ${detail.terminal?'This action ends the finite-horizon episode, so next V = 0.':'Next V comes from the history after this token.'} Critic loss uses a detached λ-return; neither network is updated.</p>`:''}`;
+  if(editing)$('#g-value-edit')?.focus();
 }
 
 function schedulePreview(){
@@ -132,37 +143,37 @@ function schedulePreview(){
   previewTimer=setTimeout(async()=>{
     try{
       if(!$('#g-answer').value.trim())throw new Error('Enter a correct answer to calculate automatic rewards.');
-      const result=await api(`/api/grpo/runs/${sid}/preview`,objectivePayload());
+      const result=await api(`${apiRoot}/${sid}/preview`,objectivePayload());
       if(version!==previewVersion||state.job?.id!==sid)return;
       state.preview=result;error();renderPreview();
-    }catch(exc){if(version===previewVersion&&state.job?.id===sid){state.preview=null;error(exc.message);$('#g-preview-status').textContent='Preview unavailable. Check the reward settings.';}}
+    }catch(exc){if(version===previewVersion&&state.job?.id===sid){state.preview=null;renderCards();renderInspector();error(exc.message);$('#g-preview-status').textContent='Objective unavailable. Check the reward settings.';}}
     finally{if(version===previewVersion){state.previewBusy=false;$('#g-rollouts').setAttribute('aria-busy','false');updateControls();}}
   },180);
 }
 async function pollJob(sid){
   clearTimeout(jobTimer);
   try{
-    const job=await api(`/api/grpo/runs/${sid}`);
+    const job=await api(`${apiRoot}/${sid}`);
     if(state.job?.id!==sid)return;
     state.job=job;renderJob();
-    if(job.phase==='sampling')jobTimer=setTimeout(()=>pollJob(sid),800);
+    if(computing(job))jobTimer=setTimeout(()=>pollJob(sid),800);
     else{
       state.status={...state.status,active:null};updateControls();
-      if(job.phase==='ready'){state.rewards=job.rollouts.map(()=>null);schedulePreview();}
+      if(job.phase==='ready'){state.rewards=job.rollouts.map(()=>null);state.values=job.rollouts.map(row=>row.tokens.map(()=>null));schedulePreview();}
       else if(job.phase==='error')error(job.message);
     }
   }catch(exc){if(state.job?.id===sid){error(`Could not check the rollout group: ${exc.message}`);jobTimer=setTimeout(()=>pollJob(sid),2500);}}
 }
 $('#g-form').addEventListener('submit',async event=>{
-  event.preventDefault();if(state.busy||state.job?.phase==='sampling')return;
+  event.preventDefault();if(state.busy||computing(state.job))return;
   if(!$('#g-form').reportValidity())return;
   const payload=collectionPayload();
   error();state.busy=true;updateControls();
   try{
-    const job=await api('/api/grpo/runs',payload);
+    const job=await api(apiRoot,payload);
     clearTimeout(jobTimer);clearTimeout(previewTimer);previewVersion++;
-    state.job=job;state.preview=null;state.rewards=[];state.selected=null;state.showStep=false;state.previewBusy=false;
-    for(const id of ['g-mean','g-loss','g-clipped-count'])$('#'+id).textContent='—';
+    state.job=job;state.preview=null;state.rewards=[];state.values=[];state.selected=null;state.previewBusy=false;
+    for(const id of ['g-mean','g-loss','g-clipped-count',...(PPO?['g-critic-loss']:[])])$('#'+id).textContent='—';
     $('#g-std').textContent='Group baseline';$('#g-kl-metric').textContent='Mean Kᵢ,ₜ: —';
     $('#g-stale').classList.add('hidden');$('#g-equal').classList.add('hidden');$('#g-inspector').classList.add('hidden');
     $('#g-preview-status').textContent='Waiting for the rollout group…';renderJob();pollJob(job.id);
@@ -171,11 +182,9 @@ $('#g-form').addEventListener('submit',async event=>{
 });
 $('#g-cancel').addEventListener('click',async()=>{
   if(!state.job)return;$('#g-cancel').disabled=true;
-  try{await api(`/api/grpo/runs/${state.job.id}/cancel`,{});}catch(exc){error(exc.message);}
+  try{await api(`${apiRoot}/${state.job.id}/cancel`,{});}catch(exc){error(exc.message);}
   finally{$('#g-cancel').disabled=false;}
 });
-$('#g-preview').addEventListener('click',()=>{state.showStep=true;renderPreview();});
-$('#g-hide-preview').addEventListener('click',()=>{state.showStep=false;renderPreview();});
 $('#g-reset-rewards').addEventListener('click',()=>{state.rewards=state.job.rollouts.map(()=>null);schedulePreview();});
 $('#g-rollouts').addEventListener('input',event=>{
   const input=event.target.closest('[data-reward]');if(!input)return;
@@ -188,27 +197,38 @@ $('#g-rollouts').addEventListener('click',event=>{
   state.selected=token.dataset.token.split(':').map(Number);
   document.querySelectorAll('.g-token.active').forEach(el=>el.classList.remove('active'));token.classList.add('active');renderInspector();
 });
-for(const id of ['g-answer','g-rule','g-normalize','g-clipping','g-kl','g-epsilon','g-beta','g-step','g-drift'])$('#'+id).addEventListener('input',()=>{updateControls();schedulePreview();});
+for(const id of ['g-answer','g-rule','g-normalize','g-clipping','g-kl','g-epsilon','g-beta','g-drift',...(PPO?['g-gamma','g-lambda','g-critic']:[])])$('#'+id).addEventListener('input',()=>{updateControls();schedulePreview();});
 document.querySelectorAll('[data-sampling]').forEach(el=>el.addEventListener('input',samplingChanged));
 $('#g-model').addEventListener('change',()=>{state.status=null;$('#g-thinking').checked=false;updateControls();pollStatus();});
 for(const id of ['g-prefix','g-assistant'])$('#'+id).addEventListener('focus',event=>{editor=event.target;});
 $('#g-special-tokens').addEventListener('click',event=>{
-  const button=event.target.closest('[data-insert]');if(!button||state.job?.phase==='sampling')return;
+  const button=event.target.closest('[data-insert]');if(!button||computing(state.job))return;
   const target=$('#g-mode').value==='raw'||!editor||editor.disabled?$('#g-prefix'):editor;
   target.focus();target.setRangeText(button.dataset.insert,target.selectionStart,target.selectionEnd,'end');target.dispatchEvent(new Event('input',{bubbles:true}));
 });
 $('#g-example').addEventListener('click',()=>{
-  $('#g-mode').value='chat';$('#g-prefix').value='Name one primary color. Reply with one word.';$('#g-assistant').value='';$('#g-answer').value='red';$('#g-rule').value='exact';
+  $('#g-mode').value='chat';$('#g-prefix').value=PPO?'What is 2 + 3? Reply with only the answer.':'Name one primary color. Reply with one word.';$('#g-assistant').value='';$('#g-answer').value=PPO?'5':'red';$('#g-rule').value='exact';
   $('#g-count').value='5';$('#g-length').value='24';$('#g-temperature').value='1';$('#g-seed').value='42';$('#g-thinking').checked=false;
   samplingChanged();schedulePreview();
 });
 $('#g-export').addEventListener('click',()=>{
   if(!state.job||!state.preview)return;
-  const data={kind:'grpo-logit-space-preview',weights_updated:false,collection:state.job,objective:objectivePayload(),preview:state.preview,preview_visible:state.showStep,
-    notation:{objective:'J_GRPO (maximized)',reward:'r_i',advantage:'A_hat_i',ratio:'rho_i_t',clipped_ratio:'rho_i_t^clip',kl:'K_i_t',current:'pi_theta(y_i_t | h_i_t)',old:'pi_old(y_i_t | h_i_t)',reference:'pi_ref(y_i_t | h_i_t)',normalization_epsilon:'delta = 1e-8',loss:'L = -J_GRPO (minimized)'},
-    note:'Real frozen-policy samples. Independent sampled-token logit ascent on J_GRPO; no model parameter gradients, optimizer updates, or parameter-coupled predictions.'};
+  const data={kind:`${algorithm.toLowerCase()}-objective-gradients`,weights_updated:false,collection:state.job,objective:objectivePayload(),preview:state.preview,
+    notation:{objective:`J_${algorithm} (maximized)`,reward:'r_i',advantage:PPO?'A_hat_i_t (GAE)':'A_hat_i',gradient:'g_i_t = d J / d log pi_theta (including averaging)',ratio:'rho_i_t',clipped_ratio:'rho_i_t^clip',kl:'K_i_t',current:'illustrative pi_theta(y_i_t | h_i_t)',old:'pi_old(y_i_t | h_i_t)',reference:'pi_ref(y_i_t | h_i_t)',normalization_epsilon:'delta = 1e-8',loss:`L = -J_${algorithm} (minimized)`},
+    note:'Real frozen-policy samples and objective derivatives. No model parameter gradients, optimizer updates, or updated-model probabilities. Legacy after/delta fields describe an independent-logit toy calculation only; the UI displays g, never a predicted probability change.'};
   const url=URL.createObjectURL(new Blob([JSON.stringify(data,null,2)],{type:'application/json'}));
-  const a=document.createElement('a');a.href=url;a.download=`grpo-${state.job.id.slice(0,8)}.json`;a.click();setTimeout(()=>URL.revokeObjectURL(url),1000);
+  const a=document.createElement('a');a.href=url;a.download=`${algorithm.toLowerCase()}-${state.job.id.slice(0,8)}.json`;a.click();setTimeout(()=>URL.revokeObjectURL(url),1000);
 });
+if(PPO){
+  $('#g-reset-values').addEventListener('click',()=>{state.values=state.job.rollouts.map(row=>row.tokens.map(()=>null));schedulePreview();});
+  $('#g-token-detail').addEventListener('input',event=>{
+    if(event.target.id!=='g-value-edit'||!event.target.value.trim()||!event.target.validity.valid)return;
+    const [ri,ti]=state.selected;state.values[ri][ti]=Number(event.target.value);schedulePreview();
+  });
+  $('#g-token-detail').addEventListener('click',event=>{
+    if(event.target.id!=='g-reset-value')return;
+    const [ri,ti]=state.selected;state.values[ri][ti]=null;schedulePreview();
+  });
+}
 document.addEventListener('keydown',event=>{if((event.metaKey||event.ctrlKey)&&event.key==='Enter'){event.preventDefault();if(!$('#g-sample').disabled)$('#g-form').requestSubmit();}});
 updateControls();pollStatus();

static/index.html

@@ -12,7 +12,7 @@
 <body>
   <header class="site-header">
     <a class="brand" href="/"><span class="brand-mark">L</span> Layer lab <span class="brand-light">/</span><span class="brand-caption">Layers & attention</span></a>
-    <nav><a href="/grpo" class="grpo-nav" title="Open the GRPO sandbox">GRPO</a><a href="#lens-section">Layer explorer</a><a href="#attention-section">Attention</a><a href="#method">Method</a><span id="model-pill" class="model-pill">Qwen3-0.6B</span></nav>
+    <nav><a href="/grpo" class="grpo-nav" title="Open the GRPO sandbox">GRPO</a><a href="/ppo" class="grpo-nav" title="Open the PPO sandbox">PPO</a><a href="#lens-section">Layer explorer</a><a href="#attention-section">Attention</a><a href="#method">Method</a><span id="model-pill" class="model-pill">Qwen3-0.6B</span></nav>
   </header>
   <main>
     <div class="intro">

static/ppo.html

@@ -0,0 +1,126 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  <meta name="theme-color" content="#ffffff">
+  <title>PPO sandbox — Layer lab</title>
+  <link rel="icon" href="data:,">
+  <link rel="stylesheet" href="/styles.css">
+  <link rel="stylesheet" href="/grpo.css">
+  <script type="module" src="/grpo.js"></script>
+</head>
+<body class="grpo-page">
+  <header class="site-header">
+    <a class="brand" href="/"><span class="brand-mark">L</span> Layer lab <span class="brand-light">/</span><span class="brand-caption">PPO sandbox</span></a>
+    <nav><a class="mode-link" href="/">Layers & attention</a><a class="mode-link" href="/grpo">GRPO</a><a class="mode-link active" href="/ppo" aria-current="page">PPO</a><span class="model-pill">Weights frozen</span></nav>
+  </header>
+  <main>
+    <div class="intro">
+      <div><p class="eyebrow">PROXIMAL POLICY OPTIMIZATION</p><h1>A value for every state.</h1><p class="intro-description">Sample with a frozen actor. Evaluate each state with a trained value critic.<br>Follow rewards → values → GAE advantages → token gradients.</p></div>
+      <div class="g-process" aria-label="Experiment steps"><span><b>01</b> Sample</span><i>→</i><span><b>02</b> Reward</span><i>→</i><span><b>03</b> Gradient</span></div>
+    </div>
+    <div class="g-workspace">
+      <aside class="controls g-controls" aria-label="Rollout settings">
+        <div class="sidebar-heading"><h2>Experiment <span class="small-tag">INPUT</span></h2><button id="g-example" class="text-button" type="button">Reset example</button></div>
+        <form id="g-form">
+          <section class="control-section">
+            <label class="field-label" for="g-model">Model</label>
+            <select id="g-model" data-sampling><option value="Qwen/Qwen3-0.6B">Qwen3-0.6B</option><option value="ThomasTheMaker/gm3-270m-TinyGSM-reasoning">Gemma 3 270M · TinyGSM reasoning</option></select>
+            <p id="g-status" class="field-note" role="status">Connecting to the model…</p>
+          </section>
+          <section class="control-section">
+            <label class="field-label" for="g-mode"><span class="step">01</span> Prefix format</label>
+            <select id="g-mode" data-sampling><option value="chat">Chat · official model template</option><option value="raw">Raw · exact tokenized prefix</option></select>
+            <label class="field-label g-spaced" for="g-prefix">Prompt / prefix</label>
+            <textarea id="g-prefix" rows="4" required maxlength="20000" spellcheck="false" data-sampling>What is 2 + 3? Reply with only the answer.</textarea>
+            <div id="g-assistant-field"><label class="field-label g-spaced" for="g-assistant">Assistant prefix <span class="small-tag">OPTIONAL</span></label><textarea id="g-assistant" rows="2" maxlength="5000" spellcheck="false" placeholder="Start the model's answer here…" data-sampling></textarea></div>
+            <label class="g-check g-spaced"><input type="checkbox" id="g-thinking" data-sampling> Enable reasoning</label>
+            <p id="g-thinking-note" class="field-note">Qwen's native thinking template.</p>
+            <details class="g-special"><summary>Insert a control token</summary><div id="g-special-tokens" class="special-tokens"></div></details>
+          </section>
+          <section class="control-section">
+            <label class="field-label" for="g-answer"><span class="step">02</span> Correct answer</label>
+            <textarea id="g-answer" rows="2" required maxlength="5000" spellcheck="false">5</textarea>
+            <label class="field-label g-spaced" for="g-rule">Automatic reward</label>
+            <select id="g-rule"><option value="exact">Exact match · 1 or 0</option><option value="contains">Contains answer · 1 or 0</option></select>
+            <p class="field-note">Ignores case and repeated whitespace. Scores the answer after reasoning. Every rollout's reward is editable.</p>
+          </section>
+          <section class="control-section">
+            <div class="g-input-grid">
+              <label>Rollouts G<input id="g-count" type="number" value="5" min="2" max="12" step="1" required data-sampling></label>
+              <label>Max new tokens<input id="g-length" type="number" value="24" min="1" max="64" step="1" required data-sampling></label>
+              <label>Temperature<input id="g-temperature" type="number" value="1" min="0.1" max="2" step="0.1" required data-sampling></label>
+              <label>Seed<input id="g-seed" type="number" value="42" min="0" max="4294967295" step="1" required data-sampling></label>
+            </div>
+            <p class="field-note">Full-vocabulary sampling. Probabilities include temperature; no top-k or top-p filtering.</p>
+            <button id="g-sample" class="primary-button" type="submit" disabled><span>Sample 5 rollouts</span><span>↗</span></button>
+            <button id="g-cancel" class="g-stop hidden" type="button">Stop sampling</button>
+            <p id="g-error" class="g-error hidden" role="alert"></p>
+            <p id="g-stale" class="g-warning hidden">Sampling settings changed. Sample a new group to use them.</p>
+          </section>
+        </form>
+        <p class="sidebar-footnote">Real checkpoint samples · no optimizer<br>No model weights are updated.</p>
+      </aside>
+      <div class="g-results">
+        <section class="g-step-panel" aria-labelledby="g-step-title">
+          <div class="g-section-heading"><div><p class="eyebrow">03 / THE OBJECTIVE</p><h2 id="g-step-title">Inspect the objective</h2></div><button id="g-export" class="text-button" disabled>Export JSON ↓</button></div>
+          <div class="g-ablation-grid g-ppo-ablations">
+            <label class="g-ablation"><span><input id="g-normalize" type="checkbox"> Normalize GAE advantages</span><small>Center and scale across all generated tokens</small></label>
+            <label class="g-ablation"><span><input id="g-critic" type="checkbox" checked> Use value critic V</span><small>Disable to set all state values to zero</small></label>
+            <label class="g-ablation"><span><input id="g-clipping" type="checkbox" checked> Clip the ratio ρᵢ,ₜ</span><small>ρᶜˡⁱᵖᵢ,ₜ = clip(ρᵢ,ₜ, 1−ε, 1+ε)</small></label>
+            <label class="g-ablation"><span><input id="g-kl" type="checkbox" checked> KL penalty βKᵢ,ₜ</span><small>Frozen πref = collection πold</small></label>
+          </div>
+          <p class="g-explanation g-critic-note"><a href="https://huggingface.co/Open-Reasoner-Zero/Open-Reasoner-Zero-Critic-0.5B" target="_blank" rel="noopener noreferrer">Open Reasoner Zero Critic 0.5B</a> · trained scalar value-head. This math critic was trained with a different policy and reward distribution. Its values are illustrative estimates for your prompt, not calibrated success probabilities for the selected actor or custom target. Editing the target does not retrain V. Click a token to override its value.</p>
+          <p id="g-critic-status" class="field-note" role="status">Connecting to the value critic…</p>
+          <button id="g-reset-values" class="text-button" disabled>Reset all V to checkpoint values</button>
+          <div class="g-slider-grid">
+            <label>Clip ε <output id="g-epsilon-value">0.20</output><input id="g-epsilon" type="range" min="0.01" max="0.5" step="0.01" value="0.2"></label>
+            <label>KL β <output id="g-beta-value">0.04</output><input id="g-beta" type="range" min="0" max="1" step="0.01" value="0.04"></label>
+            <label>Discount γ <output id="g-gamma-value">1.00</output><input id="g-gamma" type="range" min="0" max="1" step="0.01" value="1"></label>
+            <label>GAE λ <output id="g-lambda-value">0.95</output><input id="g-lambda" type="range" min="0" max="1" step="0.01" value="0.95"></label>
+            <label>Candidate drift d <output id="g-drift-value">0.00</output><input id="g-drift" type="range" min="-3" max="3" step="0.05" value="0"></label>
+          </div>
+          <p id="g-origin-note" class="g-explanation">At d = 0, πθ = πold = πref: ρᵢ,ₜ = 1 and Kᵢ,ₜ = 0. Clipping and KL do not change this first gradient. Move drift in either direction to explore their effects.</p>
+          <p id="g-preview-status" class="field-note" role="status">Sample a group to see gradients on every token automatically.</p>
+        </section>
+        <div class="g-summary g-ppo-summary" aria-live="polite">
+          <article><span>Mean reward · mean(r)</span><strong id="g-mean">—</strong><small id="g-std">Terminal reward statistics</small></article>
+          <article title="The maximized clipped actor objective, with GAE advantages treated as constants."><span>Objective 𝒥<sub>PPO</sub> ↑</span><strong id="g-loss">—</strong><small>Maximize · advantages from GAE</small></article>
+          <article><span>Critic loss L<sub>V</sub> ↓</span><strong id="g-critic-loss">—</strong><small>½(V − detached λ-return)²</small></article>
+          <article><span>Clipped tokens</span><strong id="g-clipped-count">—</strong><small id="g-kl-metric">Mean Kᵢ,ₜ: —</small></article>
+        </div>
+        <section aria-labelledby="g-rollouts-heading">
+          <div class="g-section-heading"><div><p class="eyebrow">01 → 02 / SAMPLES & REWARDS</p><h2 id="g-rollouts-heading">The rollout group</h2></div><button id="g-reset-rewards" class="text-button" disabled>Reset auto rewards</button></div>
+          <div id="g-progress" class="g-progress hidden" role="status"><span class="spinner"></span><span id="g-progress-text">Sampling…</span></div>
+          <p id="g-group-note" class="g-explanation">Choose a prefix and target, then sample a group. No completions are prewritten.</p>
+          <p id="g-equal" class="g-warning hidden">Equal rollout rewards do not imply equal PPO advantages; each state has its own value estimate.</p>
+          <div class="g-token-legend"><span id="g-token-legend">Each token shows its collection probability and objective gradient.</span><span><i class="g-swatch" aria-hidden="true">+</i> Positive g <i class="g-swatch negative" aria-hidden="true">−</i> Negative g <i class="g-swatch clipped"></i> Clipped</span></div>
+          <div id="g-rollouts" class="g-rollouts"><div class="g-empty"><span>G</span><h3>A group makes the comparison.</h3><p>Five real rollouts by default.<br>Every token gets a value, a GAE advantage, and an actor gradient.</p></div></div>
+          <details id="g-prefix-details" class="g-prefix-details hidden"><summary>Inspect the exact collection prefix</summary><pre id="g-formatted-prefix"></pre><div id="g-prefix-tokens" class="g-prefix-tokens"></div></details>
+        </section>
+        <section id="g-inspector" class="g-inspector hidden" aria-labelledby="g-inspector-title"><div class="g-section-heading"><div><p class="eyebrow">SELECTED TOKEN</p><h2 id="g-inspector-title">Follow the gradient</h2></div></div><div id="g-token-detail"></div></section>
+        <section class="g-method" id="g-method">
+          <p class="eyebrow">WHAT THIS SANDBOX COMPUTES</p><h2>Real samples. Exact objective derivatives.</h2>
+          <p>Rollouts and collection probabilities come from the selected actor. A separate frozen ORZ 0.5B critic estimates V(h) before every generated token, using its own tokenizer. Exact histories are retokenized independently; future text is never included. No random value-head, reward-model substitution, or temperature-based pseudo-critic is used.</p>
+          <div class="g-equations" aria-label="PPO objective and notation">
+            <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="J PPO is the expectation of the group and token averaged minimum of rho times A hat and clipped rho times A hat, minus beta K."><mrow><msub><mi>𝒥</mi><mtext>PPO</mtext></msub><mo>=</mo><mi>𝔼</mi><mo>[</mo><mfrac><mi>1</mi><mi>G</mi></mfrac><munder><mo>∑</mo><mi>i</mi></munder><mfrac><mi>1</mi><mrow><mo>|</mo><msub><mi>y</mi><mi>i</mi></msub><mo>|</mo></mrow></mfrac><munder><mo>∑</mo><mi>t</mi></munder><mrow><mo>(</mo><mrow><mrow><mi mathvariant="normal">min</mi><mrow><mo>(</mo><mrow><msub><mi>ρ</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub><mo>,</mo><msubsup><mi>ρ</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow><mtext>clip</mtext></msubsup><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub></mrow><mo>)</mo></mrow></mrow><mo>−</mo><mi>β</mi><msub><mi>K</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub></mrow><mo>)</mo></mrow><mo>]</mo></mrow></math></div>
+            <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="rho i t is pi theta of y i t given h i t divided by pi old of the same token given the same history."><mrow><msub><mi>ρ</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub><mo>=</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mrow><mo>(</mo><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub><mo>|</mo><msub><mi>h</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub></mrow><mo>)</mo></mrow></mrow><mrow><msub><mi>π</mi><mtext>old</mtext></msub><mrow><mo>(</mo><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub><mo>|</mo><msub><mi>h</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub></mrow><mo>)</mo></mrow></mrow></mfrac></mrow></math></div>
+            <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="Clipped rho i t is rho i t clipped between one minus epsilon and one plus epsilon."><mrow><msubsup><mi>ρ</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow><mtext>clip</mtext></msubsup><mo>=</mo><mrow><mi mathvariant="normal">clip</mi><mrow><mo>(</mo><mrow><msub><mi>ρ</mi><mrow><mi>i</mi><mo>,</mo><mi>t</mi></mrow></msub><mo>,</mo><mi>1</mi><mo>−</mo><mi>ε</mi><mo>,</mo><mi>1</mi><mo>+</mo><mi>ε</mi></mrow><mo>)</mo></mrow></mrow></mrow></math></div>
+            <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="TD residual equals immediate reward plus discounted next value minus current value."><mrow><msub><mi>δ</mi><mi>t</mi></msub><mo>=</mo><msub><mi>r</mi><mi>t</mi></msub><mo>+</mo><mi>γ</mi><mi>V</mi><mo>(</mo><msub><mi>h</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>)</mo><mo>−</mo><mi>V</mi><mo>(</mo><msub><mi>h</mi><mi>t</mi></msub><mo>)</mo></mrow></math></div>
+            <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="GAE advantage equals TD residual plus gamma lambda times next advantage."><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo>=</mo><msub><mi>δ</mi><mi>t</mi></msub><mo>+</mo><mi>γ</mi><mi>λ</mi><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo>,</mo><mspace width="1em"/><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>t</mi></msub><mo>=</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo>+</mo><mi>V</mi><mo>(</mo><msub><mi>h</mi><mi>t</mi></msub><mo>)</mo></mrow></math></div>
+            <div class="g-equation-scroll"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block" aria-label="Critic loss is the mean half squared error to a detached lambda-return."><mrow><msub><mi>L</mi><mi>V</mi></msub><mo>=</mo><mtext>mean</mtext><mo>[</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><msup><mrow><mo>(</mo><mi>V</mi><mo>(</mo><msub><mi>h</mi><mi>t</mi></msub><mo>)</mo><mo>−</mo><mtext>stopgrad</mtext><mo>(</mo><msub><mover accent="true"><mi>R</mi><mo>^</mo></mover><mi>t</mi></msub><mo>)</mo><mo>)</mo></mrow><mn>2</mn></msup><mo>]</mo></mrow></math></div>
+          </div>
+          <p>Both EOS and the maximum token budget end a finite-horizon episode. The final generated action receives the rollout score rᵢ, all earlier rewards are zero, and terminal V = 0. A completion cut off by the token budget is scored as displayed; it is not bootstrapped as an unfinished infinite-horizon trajectory. Stop tokens participate in the objective; prompt tokens do not.</p>
+          <p>GAE is computed backward with γ and λ. The actor uses detached advantages; optional normalization centers and scales them across all generated tokens with sample SD + 10⁻⁸. The critic target is the unnormalized λ-return  + V, treated as a constant for its squared-error loss. Both losses use the same group mean of each rollout's token mean. The critic is frozen and can be overridden per state or disabled (V = 0). A scalar value model has no sampling temperature.</p>
+          <p>The actor objective uses the clipped ratio ρ = πθ/πold. The optional KL penalty uses frozen πref = πold and K = πref/πθ − log(πref/πθ) − 1. This sandbox adds KL to the actor objective; it does not also add KL reward shaping to GAE.</p>
+          <p>Each chip shows the collection probability and <code>gᵢ,ₜ = ∂𝒥PPO/∂log πθ(yᵢ,ₜ | hᵢ,ₜ)</code>, including group/token averaging. Positive g locally favors increasing this log-probability; negative g favors decreasing it. Selecting a token also shows its objective contribution and equivalent loss. The loss is the negative of the maximized objective.</p>
+          <p>Candidate drift sets <code>logit(πθ) = logit(πold) + dÂᵢ,ₜ</code>, with d from −3 to +3. This is a synthetic candidate for exploring clipping and KL, not a model forward pass or a measured training change. Positive drift moves with the advantage; negative drift moves against it. At zero drift the candidate is exactly the collection policy.</p>
+          <p><strong>A gradient is not a probability change.</strong> Actual training backpropagates <code>∇θ𝒥 = Σ gᵢ,ₜ ∇θ log πθ</code>, updates shared weights with an optimizer, then computes new probabilities with another forward pass. Shared parameters couple tokens, and optimizer state affects the result. This sandbox computes no parameter gradients or optimizer steps and reports no updated-model probabilities. Policy clipping removes an incentive; it does not forcibly clamp probabilities.</p>
+        </section>
+        <footer class="site-footer"><span>Layer lab <i>/</i> PPO sandbox</span><span>Samples · rewards · objective gradients · frozen weights</span></footer>
+      </div>
+    </div>
+  </main>
+</body>
+</html>

static/styles.css

@@ -276,6 +276,7 @@ h3{font-size:13px;font-weight:500}
 
 @media(max-width:600px){.site-header{padding:0 16px}
 .site-header nav a{display:none}
+.site-header .brand-light{display:none}
 main{padding:0 16px}
 .workspace{display:block}
 .controls{position:relative;top:0;max-height:none;margin-bottom:24px}
@@ -315,6 +316,7 @@ main{padding:0 16px}
 }
 
 @media(max-width:390px){.metrics article:last-child{display:none}
+.site-header .model-pill{display:none}
 .metrics{grid-template-columns:1fr 1fr}
 .heads-grid{grid-template-columns:repeat(2,minmax(0,1fr))}
 .matrix-layout{grid-template-columns:1fr}