paper-bpco-bounded-critic-1.5b

Recipe: recipes/papers/bpco-bounded-critic · Collection: Papers, replicated

PPO-style single-rollout training on GSM8K with a bounded critic against the standard clipped-value one. The recipe arm bounds the value head's output, trains it toward the Monte Carlo return with 15 critic-only warm-up steps, and uses the DPPO ratio range. value_head.pt and curves.json ship with each arm.

Result

Arm	pass@1	95% CI	pass@k	Steps	GPU min
Base, no training	0.39	[0.33, 0.46]	0.60	0	0
Baseline (standard PPO critic recipe)	0.43	[0.36, 0.50]	0.66	40	25.1
Recipe (the BPCO bundle)	0.41	[0.35, 0.48]	0.73	15 warm-up + 40	26.3

Recipe vs baseline: -0.017 [-0.069, +0.037] over 120 paired tasks. Verdict: unresolved. At this step size the ratio never left 1 by more than 0.0014, so neither clip ever fired and the comparison reduces to the bounded critic against the unbounded one, which the curves show and pass@1 does not.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
.	recipe arm: bounded critic bundle, 2026-09-21 run
baseline	baseline arm: clipped-value PPO critic, 2026-09-21 run

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
model = PeftModel.from_pretrained(base, "while-ai/paper-bpco-bounded-critic-1.5b")  # the headline arm
model = PeftModel.from_pretrained(base, "while-ai/paper-bpco-bounded-critic-1.5b", subfolder="baseline")  # another arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/papers/bpco-bounded-critic
python recipe.py

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.