\documentclass[11pt]{article} \usepackage{common} \begin{document} \notehead{0109}{KL regularisation estimators $k_1,k_2,k_3$; why recent verifier-RL sets $\beta=0$}{Policy-optimisation lineage, batch M1. Companion notebook: \texttt{notebooks/0109-kl-estimators.ipynb}.} \begin{problem} \question{What does each KL estimator estimate as a value, what does it optimise as a loss, and should the penalty be there at all for an exact verifier?} RLHF maximises $\E[R]-\beta\,\KL(\pi_\theta\|\pi_{\rm ref})$ so that the policy stays where a learned reward model is accurate. The KL over a space of sequences cannot be summed, so it is estimated from samples, and one of three Monte-Carlo estimators (Schulman's $k_1,k_2,k_3$) is used, either as a number or as a loss to differentiate. Three separate questions must not be conflated: (i) as a \emph{value}, how biased and how noisy is each estimator? (ii) as a \emph{loss}, what does differentiating it actually optimise? (iii) for RL with an exact verifier (theorem proving), should the penalty be there at all? Recent recipes answer (iii) with $\beta=0$; we say why, and what that choice costs. \end{problem} \section{Notation} \begin{center}\small\begin{tabular}{@{}ll@{}}\toprule symbol & meaning\\\midrule $q=\pi_\theta$ (sampling), $p=\pi_{\rm ref}$ & policy and frozen reference; samples $x\sim q$\\ $r=p(x)/q(x)$, $\ell=\log r$ & ratio, log-ratio; $\E_q[r]=1$ if $\mathrm{supp}\,p\subseteq\mathrm{supp}\,q$\\ $\KL(q\|p)=\E_q[-\ell]$ & reverse KL (the regulariser); $\KL(p\|q)$ the forward KL\\ $k_1,\,k_2,\,k_3$ & $-\log r,\ \tfrac12(\log r)^2,\ (r-1)-\log r$ (Schulman's note)\\ $\sigma^2$ & for Gaussians $q=\mathcal N(0,1)$, $p=\mathcal N(\sigma,1)$: $\KL=\sigma^2/2$\\ $\lambda=-\ell=\log(q/p)$ & log-ratio in the \emph{policy} direction; $k_1=\lambda$, $k_2=\tfrac12\lambda^2$\\ $s_t,\ \lambda_t$ & token score and token log-ratio $\lambda_t=\log(\pi_\theta/\pi_{\rm ref})(o_t|s_t)$\\\bottomrule\end{tabular}\end{center} \textbf{Sign convention.} Secs.~2.1--2.2 (values) use $\ell=\log(p/q)=-\lambda$; Secs.~2.3--2.4 (gradients, sequences) use $\lambda=\log(q/p)$, so that $\nabla\lambda=+\nabla\log q$; in the boxed result the first line (a value) has $\ell$, the last line (gradients) has $\lambda$. \section{Derivation} \subsection{Values: unbiasedness} \begin{itemize} \item $k_1$: $\E_q[-\log r]=\sum q\log\frac qp=\KL(q\|p)$. Unbiased by definition. \item $k_3$: $\E_q[r-1]=\sum p-1=0$ (a control variate with coefficient $c=1$), so $\E_q[k_3]=\KL$. It is $\ge0$ pointwise since $\log x\le x-1$ (the vertical distance between $\log$ and its tangent at $1$, a Bregman divergence). \item $k_2$: $\E_q[\tfrac12\ell^2]$ is the $f$-divergence with $f=\tfrac12\log^2$, and $f''(1)=1$ equals KL's, so $k_2$ and KL agree to second order near $p\approx q$ but not beyond. The exact bias: $\E_q[e^\ell]=\E_q[r]=1$ gives $\sum_{n\ge1}\E[\ell^n]/n!=0$, i.e. \begin{equation}\KL(q\|p)=-\E[\ell]=\sum_{n\ge2}\frac{\E_q[\ell^n]}{n!},\qquad\E_q[k_2]-\KL=-\sum_{n\ge3}\frac{\E_q[\ell^n]}{n!}.\label{eq:k2bias}\end{equation} \end{itemize} \ver{on three random categorical pairs ($K=6$): $\E[k_1]=\E[k_3]=\KL$ to $\le2.9\times10^{-16}$; the series in Eq.~\eqref{eq:k2bias} (40 terms) reproduces both KL and the bias of $k_2$ to $\le2.8\times10^{-16}$; the sign of $k_2$'s bias is not fixed ($-0.019,\ +0.014,\ +0.155$ for KL $=0.106,\ 0.286,\ 0.612$); $k_3,k_2\ge0$ on a $10^5$-point log-grid of $r\in[e^{-12},e^{12}]$ while $k_1<0$ on half of it.} \subsection{Values: variance, in closed form for Gaussians} Let $q=\mathcal N(0,1)$, $p=\mathcal N(\sigma,1)$. Then $\ell=m+\sigma Z$ with $m=-\sigma^2/2$, $Z\sim\mathcal N(0,1)$ under $q$, and $\KL=\sigma^2/2$. \begin{align*} \Var(k_1)&=\Var(\ell)=\sigma^2=2\KL.\\ \E[k_2]&=\tfrac12\E\ell^2=\tfrac12(\sigma^2+\tfrac{\sigma^4}4),\ \text{so}\ \text{bias}(k_2)=\tfrac{\sigma^4}8=\tfrac12\KL^2.\\ \Var(k_2)&=\tfrac14\Var(\ell^2)=\tfrac14(4m^2\sigma^2+2\sigma^4)=\tfrac{\sigma^6}4+\tfrac{\sigma^4}2.\\ \Var(k_3)&=\Var(r)+\Var(\ell)-2\Cov(r,\ell)=(e^{\sigma^2}-1)+\sigma^2-2\sigma^2=e^{\sigma^2}-1-\sigma^2, \end{align*} using $\E_q[r^2]=e^{\sigma^2}$ and $\Cov_q(r,\ell)=\E_p[\ell]-\E_q[\ell]=\sigma^2$. Hence, relative to the true value, \[\frac{\mathrm{std}(k_1)}{\KL}=\sqrt{\tfrac{2}{\KL}},\qquad\frac{\mathrm{std}(k_2)}{\KL}=\sqrt{2+2\KL},\qquad\frac{\mathrm{std}(k_3)}{\KL}=\frac{\sqrt{e^{2\KL}-1-2\KL}}{\KL},\qquad\frac{\text{bias}(k_2)}{\KL}=\frac{\KL}2.\] For small KL, $k_2$ and $k_3\approx\ell^2/2$ have standard deviation $\approx\sqrt2\,\KL$ while $k_1$ has $\sqrt{2\KL}\gg\KL$: that is why $k_1$ is unusable as a value when the policy is near the reference. \ver{Gauss--Hermite quadrature (120 nodes) matches every closed form: variances to $\le2.2\times10^{-12}$ (relative), biases to $\le1.2\times10^{-16}$ at $\KL=0.005,\,0.05,\,0.5$. At $\KL=0.005$: std/KL $=20.0,\,1.418,\,1.417$ for $k_1,k_2,k_3$ and bias$(k_2)/\KL=0.0025$; at $\KL=0.5$: $2.0,\,1.732,\,1.695$ and $0.25$ (Schulman's table gives $20,\,1.42,\,1.42$ and $2,\,1.73,\,1.7$, $0.25$).} \paragraph{Heavy tails of $k_3$.} $k_3\approx r$ for large $r$, so $\Var(k_3)\sim\E_q[r^2]=1+\chi^2(p\|q)$: infinite when the policy gives vanishing mass to something the reference likes. Example: $q=(0.001,0.999)$, $p=(0.3,0.7)$: $\KL=0.350$; std of $k_1=0.19$, $k_2=0.51$, $k_3=9.27$ ($r=300$ on the rare token, $\chi^2=89.5$). So $k_3$'s ``low variance'' holds near $q\approx p$ and fails exactly when the policy has drifted away from reference support. \ver{$\E_q[r^2]-1=\chi^2(p\|q)=89.5$ exactly; stds as above.} \subsection{Gradients: what each estimator optimises when differentiated} With $p$ frozen, $\nabla\lambda=\nabla\log q$ (note the sign: $\ell=\log p-\log q$ would give $\nabla\ell=-\nabla\log q$). Samples $x\sim q_\theta$ are drawn \emph{on-policy and detached}; the gradient flows through $q_\theta$ inside $r$ (``$k_n$ as loss''), or the estimator is used as a detached coefficient on the score (``$k_n$ in reward'' as in an RL reward penalty). \begin{align*} \nabla k_1&=\nabla\log q\ \Rightarrow\ \E_q[\nabla k_1]=0\quad\text{(Lemma 1 of 0102: no regularising gradient)}\\ \nabla k_2&=\lambda\,\nabla\log q\ \Rightarrow\ \E_q[\nabla k_2]=\E_q[\lambda\nabla\log q]=\nabla\KL(q\|p)\\ \nabla k_3&=(1-r)\nabla\log q\ \Rightarrow\ \E_q[\nabla k_3]=0-\textstyle\sum_xp\,\nabla\log q=\nabla\KL(p\|q) \end{align*} Derivations: $\nabla\KL(q\|p)=\sum\nabla q\,\lambda+\sum q\nabla\lambda=\E_q[\lambda\nabla\log q]+\E_q[\nabla\log q]=\E_q[\lambda\nabla\log q]$. For $k_3$: $\nabla r=-r\nabla\log q$, so $\nabla k_3=\nabla r\,(1-1/r)=(1-r)\nabla\log q$, and $\E_q[r\nabla\log q]=\sum p\nabla\log q$, while $\nabla\KL(p\|q)=-\sum p\nabla\log q$. \textbf{Conclusions} (Liu et al., arXiv:2510.01555, framework in Sec.~4, gradient analysis in Secs.~5.1--5.3, off-policy correction in App.~G): (a) $k_1$ is an unbiased \emph{value} (noisy near $\pi_{\rm ref}$, Sec.~2.2) and a useless \emph{loss}; (b) $k_2$ as a loss and $k_1$ as a detached coefficient (reward penalty $\beta\,\mathrm{sg}(\log q/p)$) both give exactly the gradient of the reverse KL; (c) $k_3$ as a loss (GRPO's choice, DeepSeekMath Eqs.~3--4) has the gradient of the \emph{forward} KL $\KL(p\|q)$, not of the quantity it is an unbiased estimate of. The coefficient $1-r$ is the first-order Taylor expansion of the principled $-\log r$ at $r=1$, so the two gradients agree to first order and differ at second. \ver{exact on-policy enumeration: $\E[\nabla k_1]=5.6\times10^{-17}$; $\E[\nabla k_2]$ and the detached-coefficient form equal $\nabla\KL(q\|p)$ to $5.6\times10^{-17}$; $\E[\nabla k_3]$ equals $\nabla\KL(p\|q)$ to $8.3\times10^{-17}$; the two gradients differ by $21.5\%$ in relative norm at one distance, and the gap halves as $q\to p$: $15.3\%,\,9.2\%,\,5.1\%,\,2.6\%,\,1.35\%$ for distance $0.4,\,0.2,\,0.1,\,0.05,\,0.025$ (first-order agreement).} Off-policy (several updates per batch) all ``as loss'' forms additionally need the importance weight (0103); Liu et al.\ App.~G give the corrected form. The same gradient lens applies to the \emph{sequence} view below. \subsection{Sequences: per-token estimators ignore how earlier tokens move later contexts} The sequence KL gradient is $\nabla\KL_{\rm seq}=\E[(\sum_t\lambda_t)\,s(o)]=\E\big[\sum_t(\sum_{t'\ge t}\lambda_{t'})\,s_t\big]$ (0102: reward-to-go; ``$k_1$ in reward''). A per-token loss $\sum_tk_2(\lambda_t)$ instead gives $\E[\sum_t\lambda_ts_t]$, which drops the dependence of the token-$t'$ log-ratio on tokens $t0$ the optimum of the regularised objective is $\pi^\star\propto\pi_{\rm ref}\,e^{R/\beta}$, which for $R$ the pass indicator keeps probability on every reference proof, including wrong ones. \paragraph{What it costs, honestly.} Nothing in the objective then keeps the policy near its initialisation; drift, entropy collapse and repetition are controlled only by the other mechanisms in 0106--0108. ProRL (Liu et al., cited in arXiv:2510.01555 Sec.~2) is reported to keep a $k_2$-as-loss KL penalty and periodically reset the reference model, and to find KL important for long-run stability: so $\beta=0$ is a design choice with evidence on both sides, not a theorem. And a KL penalty is \emph{not} a defence against a flaw in the verifier: a verifier hole reachable by small steps from $\pi_{\rm ref}$ is still reachable under $\beta>0$. \section{Boxed result} \begin{result} \begin{align*} &\E_q[k_1]=\E_q[k_3]=\KL(q\|p),\qquad\E_q[k_2]-\KL=-\textstyle\sum_{n\ge3}\E_q[\ell^n]/n!\quad(=\tfrac12\KL^2\text{ for Gaussians})\\ &\text{std}/\KL\ \text{(Gaussian)}:\ \ k_1:\sqrt{2/\KL},\quad k_2:\sqrt{2+2\KL},\quad k_3:\sqrt{e^{2\KL}-1-2\KL}/\KL\\ &\E_q[\nabla k_1]=0,\quad\E_q[\nabla k_2]=\nabla\KL(q\|p),\quad\E_q[\nabla k_3]=\nabla\KL(p\|q)\\ &\E_q[\mathrm{sg}(\lambda)\nabla\log q]=\nabla\KL(q\|p),\qquad\nabla\KL_{\rm seq}-\E\textstyle\sum_t\lambda_ts_t=\sum_t\E[\KL_t(s_t)\nabla\log\pi(o_{