Score Function Centering: Your Stale Rollout Does Not Have to Be Importance Sampled and Stabilize Your Async RL Training with Trainer-Generator KL

You can always find my latest articles (also the Chinese version if you are a native Chinese speaker) in my personal blog website, which has better rendering and interactive experience and you can also leave comments there! It astonishes me that I need to pay for a pro subscription in order to post a community article in HuggingFace. I will probably stop my subscription some day and you know where to find me ⬆️

This is the second article in the Score Function Centering series. I expect that after the first article you already have a first understanding of what a score function is, why we do Score Function Centering, and what it achieves. In this article I want to give you a completely new perspective on Score Function Centering, on what it solves in async RL and, more importantly, on what it does not solve yet. I also want to break one belief, the belief that a stale rollout has to be corrected onto the current policy with importance sampling, and to offer some new ways of dealing with it.

This article has two halves. The first half, sections 1 to 5, is the derivation. In the second half, from section 6 on, I go through the experiments in full, on models from Qwen3-0.6B up to GLM-4.5-Air, to see what the derivation looks like in real asynchronous training, and to answer some questions that the derivation leaves open.

Open source: the experiment configurations, training curves and code used in this article are all open-sourced at github.com/NormalUhr/score-function-centering-trainer-generator-kl.

A note on authorship and novelty: this article is NOT a summary of existing work or existing understanding. It is the author's own new perspective, understanding and a new method for Score Function Centering. Like the first article, the derivation in the first half was written entirely by hand by the author, not by AI. Of course, if you are reading the English version, I am sorry, that one was translated by AI. The author also strongly recommends reading the first article on Score Function Centering first, so that the terms used here and the positions taken here are clear.

1. Starting from what Score Function Centering has not solved

In this section we talk about Score Function Centering first, about what it repairs and what it does not repair yet.

Let us do a quick recap. The previous article started from the score function itself and ended at a very concrete problem in async RL: the rollout is produced by the generator $q_{\varphi }q\_\backslash phi$, but the trainer that actually does the backward pass has already become $p_{\theta }p\_\backslash theta$.

The standard score is still a completely correct log-probability gradient:

$$s_{\theta }(a)={\mathrm{\nabla }}_{\theta }\log p_{\theta }(a)=J_{\theta }(a)-{\mathbb{E}}_{v\sim p_{\theta }}[J_{\theta }(v)]s\_\backslash theta(a)=\backslash nabla\_\backslash theta\backslash log\; p\_\backslash theta(a)=J\_\backslash theta(a)-\backslash mathbb\; E\_\{v\backslash sim\; p\_\backslash theta\}[J\_\backslash theta(v)]$$

but it is no longer zero-mean under the distribution that actually generated the data, $q_{\varphi }q\_\backslash phi$:

$${\mathbb{E}}_{a\sim q_{\varphi }}[s_{\theta }(a)]={\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }]\mathrm{\ne }0.\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}[s\_\backslash theta(a)]=\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]\backslash neq\; 0.$$

What Score Function Centering does is this: since the action is sampled from $q_{\varphi }q\_\backslash phi$, stop centering with $p_{\theta }p\_\backslash theta$ and center with

$${\tilde{s}}_{q,\theta }(a)=J_{\theta }(a)-{\mathbb{E}}_{v\sim q_{\varphi }}[J_{\theta }(v)]\mathrm{.}\backslash tilde\; s\_\{q,\backslash theta\}(a)=J\_\backslash theta(a)-\backslash mathbb\; E\_\{v\backslash sim\; q\_\backslash phi\}[J\_\backslash theta(v)].$$

So that:

$${\mathbb{E}}_{a\sim q_{\varphi }}[{\tilde{s}}_{q,\theta }(a)]=0.\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}[\backslash tilde\; s\_\{q,\backslash theta\}(a)]=0.$$

At this point the central problem of the previous article is solved. We have the zero-mean property back, and we have baseline invariance back.

But there is in fact a larger problem that is still not solved:

We did center the score correctly, but what we are learning is still the learning signal under the generator $q_{\varphi }q\_\backslash phi$. The trainer is already $p_{\theta }p\_\backslash theta$. Is that really fine?

Let us first separate the two problems that generator staleness brings.

What Score Function Centering repairs is score drift. The standard score had a nonzero mean under $q_{\varphi }q\_\backslash phi$:

$$d_{q,\theta }:={\mathbb{E}}_{q_{\varphi }}[s_{\theta }]={\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }]\mathrm{.}d\_\{q,\backslash theta\}:=\backslash mathbb\; E\_\{q\_\backslash phi\}[s\_\backslash theta]=\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta].$$

So a reward update that has not been centered properly:

$${\mathbb{E}}_{q_{\varphi }}[A(a)s_{\theta }(a)]\backslash mathbb\; E\_\{q\_\backslash phi\}[A(a)s\_\backslash theta(a)]$$

can be split into:

$${\mathbb{E}}_{q_{\varphi }}[A(a)s_{\theta }(a)]={\mathrm{Cov}}_{q_{\varphi }}(A,J_{\theta })+{\bar{A}}_q{d}_{q,\theta },{\bar{A}}_q:={\mathbb{E}}_{q_{\varphi }}[A]\mathrm{.}\backslash mathbb\; E\_\{q\_\backslash phi\}[A(a)s\_\backslash theta(a)]=\backslash operatorname\{Cov\}\_\{q\_\backslash phi\}(A,J\_\backslash theta)+\backslash bar\; A\_q\backslash ,d\_\{q,\backslash theta\},\backslash qquad\; \backslash bar\; A\_q:=\backslash mathbb\; E\_\{q\_\backslash phi\}[A].$$

The problem with the second term is that ${\bar{A}}_q\backslash bar\; A\_q$ is only an average advantage. It says nothing about which token is better and which is worse, and yet it directly controls the update along the trainer-generator mismatch direction. The Score Function Centering of the previous article is, in essence, removing exactly this term:

$$g_q:={\mathbb{E}}_{q_{\varphi }}[A(a){\tilde{s}}_{q,\theta }(a)]={\mathbb{E}}_{q_{\varphi }}[(A(a)-{\bar{A}}_q)J_{\theta }(a)]={\mathrm{Cov}}_{q_{\varphi }}(A,J_{\theta })\mathrm{.}g\_q:=\backslash mathbb\; E\_\{q\_\backslash phi\}[A(a)\backslash tilde\; s\_\{q,\backslash theta\}(a)]=\backslash mathbb\; E\_\{q\_\backslash phi\}[(A(a)-\backslash bar\; A\_q)J\_\backslash theta(a)]=\backslash operatorname\{Cov\}\_\{q\_\backslash phi\}(A,J\_\backslash theta).$$

This formula matters a lot. It tells us that the update left after SFC cares about one thing only: under the sampler $q_{\varphi }q\_\backslash phi$ that actually generated the data, which actions have the relatively higher advantage, and what their logit direction is.

But note that it is still:

$$g_q={\mathbb{E}}_{q_{\varphi }}[(A-{\bar{A}}_q)s_{\theta }]\mathrm{.}g\_q=\backslash mathbb\; E\_\{q\_\backslash phi\}[(A-\backslash bar\; A\_q)s\_\backslash theta].$$

It is not the one under the current trainer:

$$g_p={\mathbb{E}}_{p_{\theta }}[A{s}_{\theta }]\mathrm{.}g\_p=\backslash mathbb\; E\_\{p\_\backslash theta\}[A\; s\_\backslash theta].$$

So SFC cannot turn $q_{\varphi }q\_\backslash phi$ into $p_{\theta }p\_\backslash theta$ out of nothing. All it says is this: since the data was generated by $q_{\varphi }q\_\backslash phi$, let us at least define the learning signal under that distribution cleanly.

2. Importance Sampling: it has always been done this way, but is it always right?

2.1 Where importance sampling comes from

The problem that is still not solved is this:

Should we correct this update under $q_{\varphi }q\_\backslash phi$ into an update under $p_{\theta }p\_\backslash theta$?

What does that question mean? Take a somewhat extreme example. Suppose the generator is running a version of the trainer from 10,000 steps ago, which is to say the current trainer has already graduated from university while the generator is still in primary school. For the same question, say the embarrassment of language models, which is bigger, 0.11 or 0.9, the generator writes a pile of analysis, argues that 11 is bigger than 9, and ends at 0.11 > 0.9. To the current trainer that answer is ridiculous, so the probability that the trainer itself samples this answer, the one the generator was most confident about, is close to 0. If you look at the gradient at this moment, you will find that the answer is wildly wrong and its reward, and therefore its advantage, is very low. Normally the gradient from such a sample should bring a fairly large correction, but it may not only be meaningless to the trainer, it may also overcorrect. Applying that gradient to the trainer directly is like forcing a university student to spend a night studying a wrong exercise that says 1+1=3. The student will probably never make that mistake again, but they may also go mad. This is why importance sampling exists. It tells you how important the sample drawn back then really is to the current version of the model, whatever score it received.

If you have studied a little probability, your first reaction after reading the problem above is most likely importance sampling. Suppose what we really want to compute is:

$${\mathbb{E}}_{a\sim p_{\theta }}[f(a)]\mathrm{.}\backslash mathbb\; E\_\{a\backslash sim\; p\_\backslash theta\}[f(a)].$$

Write the expectation out:

$${\mathbb{E}}_{a\sim p_{\theta }}[f(a)]=\sum _a{p}_{\theta }(a)f(a)\mathrm{.}\backslash mathbb\; E\_\{a\backslash sim\; p\_\backslash theta\}[f(a)]=\backslash sum\_a\; p\_\backslash theta(a)f(a).$$

But all the samples we have come from the generator:

$$a\sim q_{\varphi }\mathrm{.}a\backslash sim\; q\_\backslash phi.$$

What do we do? A very simple multiplication and division:

$${\mathbb{E}}_{a\sim p_{\theta }}[f(a)]=\sum _a{q}_{\varphi }(a)\frac{p_{\theta }(a)}{q_{\varphi }(a)}f(a)={\mathbb{E}}_{a\sim q_{\varphi }}[\frac{p_{\theta }(a)}{q_{\varphi }(a)}f(a)]\mathrm{.}\backslash mathbb\; E\_\{a\backslash sim\; p\_\backslash theta\}[f(a)]=\backslash sum\_a\; q\_\backslash phi(a)\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}f(a)=\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}\backslash left[\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}f(a)\backslash right].$$

So we define:

$$r(a)=\frac{p_{\theta }(a)}{q_{\varphi }(a)}\mathrm{.}r(a)=\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}.$$

This is the importance ratio. The essence of importance sampling is in fact very plain: I can only sample from $q_{\varphi }q\_\backslash phi$, but if I multiply every sample by $p_{\theta }\mathrm{/}{q}_{\varphi }p\_\backslash theta/q\_\backslash phi$, in expectation I get the result under $p_{\theta }p\_\backslash theta$ back.

For the example above, if the answer "0.11 > 0.9" has probability 0.99 under the generator but probability 0.01 under the current version of the trainer, then no matter how positive or how negative the advantage of that rollout is, it is multiplied by a very small weight of 0.01/0.99 and stops being important.

Putting this back into the policy gradient works the same way. The current-policy update is:

$$g_p={\mathbb{E}}_{a\sim p_{\theta }}[A(a)s_{\theta }(a)]\mathrm{.}g\_p=\backslash mathbb\; E\_\{a\backslash sim\; p\_\backslash theta\}[A(a)s\_\backslash theta(a)].$$

If the data comes from $q_{\varphi }q\_\backslash phi$, full importance weighting gives:

$$g_p={\mathbb{E}}_{a\sim q_{\varphi }}[\frac{p_{\theta }(a)}{q_{\varphi }(a)}A(a)s_{\theta }(a)]\mathrm{.}g\_p=\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}\backslash left[\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}A(a)s\_\backslash theta(a)\backslash right].$$

In pure theory this is of course the most standard answer. The data comes from an old policy, but through $p\mathrm{/}qp/q$ I reinterpret it as an expectation under the current policy. The ratio everyone knows from PPO, $r_t(\theta )={\pi }_{\theta }(a_t\mathrm{\mid }{s}_t)\mathrm{/}{\pi }_{\mathrm{o}\mathrm{l}\mathrm{d}}(a_t\mathrm{\mid }{s}_t)r\_t(\backslash theta)=\backslash pi\_\backslash theta(a\_t|s\_t)/\backslash pi\_\{\backslash mathrm\{old\}\}(a\_t|s\_t)$, is the same thing once you trace it back. ${\pi }_{\mathrm{o}\mathrm{l}\mathrm{d}}\backslash pi\_\{\backslash mathrm\{old\}\}$ produces the rollout, ${\pi }_{\theta }\backslash pi\_\backslash theta$ is the current policy, and the job of the ratio is distribution correction.

So if our goal is clearly I have to recover the current-policy update, importance sampling is entirely reasonable. The only problem is that once we are in async LLM RL, the technique may become a lot less reasonable.

2.2 Why did we start to dislike the importance ratio?

The first problem is very practical: the ratio easily becomes very large. If some token has probability $q_{\varphi }(a)={10}^{-6}q\_\backslash phi(a)=10^\{-6\}$ under the old generator while the trainer has already learned it to $p_{\theta }(a)={10}^{-3}p\_\backslash theta(a)=10^\{-3\}$, then:

$$r(a)=\frac{p_{\theta }(a)}{q_{\varphi }(a)}=1000.r(a)=\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}=1000.$$

This means the token is almost never sampled by the generator, but the one time it does appear, its single gradient can outweigh a thousand others. The expectation derived in theory is fine. It is the variance of the Monte Carlo estimator that becomes very large. A long rollout makes this more visible. The probability ratio of a full trajectory is:

Even if the two policies differ only slightly at every token, the product of these ratios can become extreme. Why PPO clips the ratio comes down to the same point: exact importance weighting is theoretically correct, but used directly for stochastic optimization its variance can be large enough to be useless. So methods in practice usually truncate, for example:

$$w_{\tau }(a)=\min \{\frac{p_{\theta }(a)}{q_{\varphi }(a)},\tau \}\mathrm{.}w\_\backslash tau(a)=\backslash min\backslash left\backslash \{\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\},\backslash tau\backslash right\backslash \}.$$

This is of course a lot more stable, but once you truncate you are no longer doing exact importance sampling. Take a token with $q(v)=0.01q(v)=0.01$ and $p_{\theta }(v)=0.10p\_\backslash theta(v)=0.10$. With $\tau =2\backslash tau=2$, its probability weight under full importance weighting is $0.100.10$, and after truncation it is only $0.01\times 2=0.020.01\backslash times\; 2=0.02$. That is, only 20% of the update contribution this token should have had is kept.

One case here deserves particular attention: the trainer has already learned some effective behavior, so the probability of that token has risen under $p_{\theta }p\_\backslash theta$, while the old generator $q_{\varphi }q\_\backslash phi$ has not caught up. That token then easily lands in the high-ratio region. Clipping may at the same time lower the variance and weaken genuinely valuable new behavior.

Note that I said "may". A large ratio does not automatically mean the token is important, and it certainly does not imply that clipping must hurt training. It only tells us this: truncation picks which updates get weakened by distribution mismatch, and whether task value happens to be positively correlated with that mismatch has to be verified separately. This distinction matters.

Besides the variance, importance sampling has another conflict that is more directly with Score Function Centering itself.

2.3 Importance sampling and Score Function Centering are not good buddies

What Score Function Centering builds is a score centered specifically under $q_{\varphi }q\_\backslash phi$:

$${\tilde{s}}_{q,\theta }(a)=J_{\theta }(a)-{\mathbb{E}}_{q_{\varphi }}[J_{\theta }]\mathrm{.}\backslash tilde\; s\_\{q,\backslash theta\}(a)=J\_\backslash theta(a)-\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta].$$

It satisfies:

$${\mathbb{E}}_{a\sim q_{\varphi }}[{\tilde{s}}_{q,\theta }(a)]=0.\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}[\backslash tilde\; s\_\{q,\backslash theta\}(a)]=0.$$

If at this point we change nothing and simply multiply by an importance ratio:

$${\mathbb{E}}_{a\sim q_{\varphi }}[\frac{p_{\theta }(a)}{q_{\varphi }(a)}{\tilde{s}}_{q,\theta }(a)],\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}\backslash left[\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}\backslash tilde\; s\_\{q,\backslash theta\}(a)\backslash right],$$

then by the importance sampling identity this expectation becomes:

$${\mathbb{E}}_{a\sim q_{\varphi }}[\frac{p_{\theta }(a)}{q_{\varphi }(a)}{\tilde{s}}_{q,\theta }(a)]={\mathbb{E}}_{a\sim p_{\theta }}[{\tilde{s}}_{q,\theta }(a)]={\mathbb{E}}_{p_{\theta }}[J_{\theta }]-{\mathbb{E}}_{q_{\varphi }}[J_{\theta }]\mathrm{.}\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}\backslash left[\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}\backslash tilde\; s\_\{q,\backslash theta\}(a)\backslash right]=\backslash mathbb\; E\_\{a\backslash sim\; p\_\backslash theta\}[\backslash tilde\; s\_\{q,\backslash theta\}(a)]=\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]-\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta].$$

In general:

$${\mathbb{E}}_{p_{\theta }}[J_{\theta }]-{\mathbb{E}}_{q_{\varphi }}[J_{\theta }]\mathrm{\ne }0.\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]-\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]\backslash neq\; 0.$$

In other words, you just spent all that effort centering the score under $q_{\varphi }q\_\backslash phi$, and one $p\mathrm{/}qp/q$ switches the expectation measure from $q_{\varphi }q\_\backslash phi$ back to $p_{\theta }p\_\backslash theta$.

The zero-mean property you went to such lengths to recover through Score Function Centering no longer holds.

So the thing you really cannot do here is this:

You cannot take a score that is already centered under $q_{\varphi }q\_\backslash phi$ and multiply it by $p_{\theta }\mathrm{/}{q}_{\varphi }p\_\backslash theta/q\_\backslash phi$ with nothing else changed.

This does not mean that importance sampling and score centering can never be combined. You can of course derive an importance-weighted centering so that the weighted score is zero-mean under $q_{\varphi }q\_\backslash phi$ again. Some work does exactly this kind of thing, but to buy a self-consistent estimator this way is more like making a compromise in the theory, and the result ends up not accurate enough on either side.

So at this point two different routes have appeared.

The first is the importance sampling route:

$$a\sim q_{\varphi }+r(a)=\frac{p_{\theta }(a)}{q_{\varphi }(a)}+s_{\theta }(a)⟹\text{current-policy update under }{p}_{\theta }\mathrm{.}a\backslash sim\; q\_\backslash phi\backslash quad+\backslash quad\; r(a)=\backslash frac\{p\_\backslash theta(a)\}\{q\_\backslash phi(a)\}\backslash quad+\backslash quad\; s\_\backslash theta(a)\backslash quad\backslash Longrightarrow\backslash quad\; \backslash text\{current-policy\; update\; under\; \}\; p\_\backslash theta.$$

The intuition behind it is: I want to correct the probability of the old data into the probability of now.

The second is the Score Function Centering route:

$$a\sim q_{\varphi }+{\tilde{s}}_{q,\theta }(a)=J_{\theta }(a)-{\mathbb{E}}_{q_{\varphi }}[J_{\theta }]⟹\text{sampler-relative update under }{q}_{\varphi }\mathrm{.}a\backslash sim\; q\_\backslash phi\backslash quad+\backslash quad\; \backslash tilde\; s\_\{q,\backslash theta\}(a)=J\_\backslash theta(a)-\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]\backslash quad\backslash Longrightarrow\backslash quad\; \backslash text\{sampler-relative\; update\; under\; \}\; q\_\backslash phi.$$

I accept that the data was produced by the past $q_{\varphi }q\_\backslash phi$, and I first keep the relative reward signal it expresses under its own distribution.

Next, let us talk about how to walk the second route.

3. If I do not do importance sampling, how can I trust this update?

The question is now very concrete. We have decided to keep the sampler-relative update:

$$g_q={\mathbb{E}}_{q_{\varphi }}[(A-{\bar{A}}_q)s_{\theta }],{\bar{A}}_q={\mathbb{E}}_{q_{\varphi }}[A]\mathrm{.}g\_q=\backslash mathbb\; E\_\{q\_\backslash phi\}[(A-\backslash bar\; A\_q)s\_\backslash theta],\backslash qquad\; \backslash bar\; A\_q=\backslash mathbb\; E\_\{q\_\backslash phi\}[A].$$

But the trainer we actually care about is already $p_{\theta }p\_\backslash theta$. The local update of the current policy is:

$$g_p={\mathbb{E}}_{p_{\theta }}[A{s}_{\theta }]\mathrm{.}g\_p=\backslash mathbb\; E\_\{p\_\backslash theta\}[A\; s\_\backslash theta].$$

So how far apart can $g_{q}g\_q$ and $g_{p}g\_p$ be? We first use the zero-mean property of the standard score:

$${\mathbb{E}}_{p_{\theta }}[s_{\theta }]=0.\backslash mathbb\; E\_\{p\_\backslash theta\}[s\_\backslash theta]=0.$$

So we can subtract any baseline that does not depend on the action inside $g_{p}g\_p$. To line it up with $g_{q}g\_q$, we deliberately subtract the same ${\bar{A}}_q\backslash bar\; A\_q$:

$$g_p={\mathbb{E}}_{p_{\theta }}[A{s}_{\theta }]={\mathbb{E}}_{p_{\theta }}[(A-{\bar{A}}_q)s_{\theta }]\mathrm{.}g\_p=\backslash mathbb\; E\_\{p\_\backslash theta\}[A\; s\_\backslash theta]=\backslash mathbb\; E\_\{p\_\backslash theta\}[(A-\backslash bar\; A\_q)s\_\backslash theta].$$

Now you can see that the two are already very close:

$$g_q={\mathbb{E}}_{q_{\varphi }}[(A-{\bar{A}}_q)s_{\theta }],g_p={\mathbb{E}}_{p_{\theta }}[(A-{\bar{A}}_q)s_{\theta }]\mathrm{.}g\_q=\backslash mathbb\; E\_\{q\_\backslash phi\}[(A-\backslash bar\; A\_q)s\_\backslash theta],\backslash qquad\; g\_p=\backslash mathbb\; E\_\{p\_\backslash theta\}[(A-\backslash bar\; A\_q)s\_\backslash theta].$$

Their integrands are identical. The only difference is that one takes the expectation under $q_{\varphi }q\_\backslash phi$ and the other under $p_{\theta }p\_\backslash theta$. Subtract them:

$$g_q-g_p=\sum _{v\in \mathcal{V}}(q_{\varphi }(v)-p_{\theta }(v))(A(v)-{\bar{A}}_q)s_{\theta }(v)\mathrm{.}g\_q-g\_p=\backslash sum\_\{v\backslash in\backslash mathcal\; V\}\backslash left(q\_\backslash phi(v)-p\_\backslash theta(v)\backslash right)\backslash left(A(v)-\backslash bar\; A\_q\backslash right)s\_\backslash theta(v).$$

Every term inside the sum, $(q_{\varphi }(v)-p_{\theta }(v))\times (A(v)-{\bar{A}}_q)s_{\theta }(v)(q\_\backslash phi(v)-p\_\backslash theta(v))\backslash times(A(v)-\backslash bar\; A\_q)s\_\backslash theta(v)$, is one token's contribution to the update mismatch. The first factor tells you how far apart the two policies are on that token, and the second tells you how much that token affects the learning direction.

Suppose that in the local region we care about there is a uniform $BB$ such that:

$$\parallel (A(v)-{\bar{A}}_q)s_{\theta }(v)\parallel \le B,\mathrm{\forall }v\in \mathcal{V}\mathrm{.}\backslash left\backslash |\backslash left(A(v)-\backslash bar\; A\_q\backslash right)s\_\backslash theta(v)\backslash right\backslash |\backslash le\; B,\backslash qquad\; \backslash forall\; v\backslash in\backslash mathcal\; V.$$

Then by the triangle inequality:

$$\mathrm{\parallel }{g}_q-g_p\mathrm{\parallel }\le \sum _{v\in \mathcal{V}}\mathrm{\mid }{q}_{\varphi }(v)-p_{\theta }(v)\mathrm{\mid }\parallel (A(v)-{\bar{A}}_q)s_{\theta }(v)\parallel \le B\sum _{v\in \mathcal{V}}\mathrm{\mid }{q}_{\varphi }(v)-p_{\theta }(v)\mathrm{\mid }\mathrm{.}\backslash |g\_q-g\_p\backslash |\backslash le\backslash sum\_\{v\backslash in\backslash mathcal\; V\}|q\_\backslash phi(v)-p\_\backslash theta(v)|\backslash left\backslash |\backslash left(A(v)-\backslash bar\; A\_q\backslash right)s\_\backslash theta(v)\backslash right\backslash |\backslash le\; B\backslash sum\_\{v\backslash in\backslash mathcal\; V\}|q\_\backslash phi(v)-p\_\backslash theta(v)|.$$

At this point, controlling the gradient mismatch has become controlling the $L_{1}L\_1$ distance between two distributions. Then by Pinsker's inequality:

$$\sum _{v\in \mathcal{V}}\mathrm{\mid }{q}_{\varphi }(v)-p_{\theta }(v)\mathrm{\mid }\le \sqrt{2D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })}\mathrm{.}\backslash sum\_\{v\backslash in\backslash mathcal\; V\}|q\_\backslash phi(v)-p\_\backslash theta(v)|\backslash le\backslash sqrt\{2D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash |p\_\backslash theta)\}.$$

So we get:

$$\mathrm{\parallel }{g}_q-g_p\mathrm{\parallel }\le B\sqrt{2D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })}\mathrm{.}\backslash |g\_q-g\_p\backslash |\backslash le\; B\backslash sqrt\{2D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash |p\_\backslash theta)\}.$$

This formula gives us a very important conclusion: we did not force the update under $q_{\varphi }q\_\backslash phi$ back into the update under $p_{\theta }p\_\backslash theta$ with an importance ratio, but as long as we control $D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash Vert\; p\_\backslash theta)$ we can control the error between the sampler-relative local update and the current-policy local update.

For example, if we can maintain:

$$D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })\le ϵ,D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash |p\_\backslash theta)\backslash le\backslash epsilon,$$

then we get:

$$\mathrm{\parallel }{g}_q-g_p\mathrm{\parallel }\le B\sqrt{2ϵ}\mathrm{.}\backslash |g\_q-g\_p\backslash |\backslash le\; B\backslash sqrt\{2\backslash epsilon\}.$$

So this route thinks quite differently from importance sampling. Importance sampling says: I apply a correction to the update directly, hoping to recover $g_{p}g\_p$. Sampler-relative learning plus a trainer-generator KL says: I keep $g_{q}g\_q$, but I control the distance between $g_{q}g\_q$ and $g_{p}g\_p$.

One repairs at the level of the estimator, the other limits, at the level of the distribution, the error that comes from not repairing anything.

This bound should not be read too far either. It is a fixed-prefix local analysis. It holds the current prefix and the conditional advantage function fixed, and it does not handle the fact that the visitation distribution of future prefixes also changes once the policy changes. So the $g_{p}g\_p$ here is not the full on-policy trajectory gradient. Besides, if $BB$ itself is very large the bound can be loose. What it gives us is a clear design motivation, not a theorem or a guarantee that long training will converge.

4. Wait, is the KL not pulling the current model back into the past?

It is. If what we add is:

$$D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta }),D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash |p\_\backslash theta),$$

where $q_{\varphi }q\_\backslash phi$ is the old generator and $p_{\theta }p\_\backslash theta$ is the current trainer, then minimizing this KL does produce a force pulling $p_{\theta }p\_\backslash theta$ toward $q_{\varphi }q\_\backslash phi$.

But let us pause here. A KL has a direction. What we write here is $D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash Vert\; p\_\backslash theta)$, which takes the generator $q_{\varphi }q\_\backslash phi$ as the reference and constrains the current trainer $p_{\theta }p\_\backslash theta$. That direction is not chosen at random. It is exactly the same thing as the mismatch direction that Score Function Centering removed earlier.

Because $q_{\varphi }q\_\backslash phi$ is fixed while the trainer updates:

$$D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })={\mathbb{E}}_{a\sim q_{\varphi }}[\log q_{\varphi }(a)-\log p_{\theta }(a)]\mathrm{.}D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash |p\_\backslash theta)=\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}\backslash left[\backslash log\; q\_\backslash phi(a)-\backslash log\; p\_\backslash theta(a)\backslash right].$$

Taking the gradient with respect to $\theta \backslash theta$:

$${\mathrm{\nabla }}_{\theta }{D}_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })=-{\mathbb{E}}_{a\sim q_{\varphi }}[{\mathrm{\nabla }}_{\theta }\log p_{\theta }(a)]=-{\mathbb{E}}_{q_{\varphi }}[s_{\theta }]\mathrm{.}\backslash nabla\_\backslash theta\; D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash |p\_\backslash theta)=-\backslash mathbb\; E\_\{a\backslash sim\; q\_\backslash phi\}\backslash left[\backslash nabla\_\backslash theta\backslash log\; p\_\backslash theta(a)\backslash right]=-\backslash mathbb\; E\_\{q\_\backslash phi\}[s\_\backslash theta].$$

And from the previous article we already know:

$${\mathbb{E}}_{q_{\varphi }}[s_{\theta }]={\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }]\mathrm{.}\backslash mathbb\; E\_\{q\_\backslash phi\}[s\_\backslash theta]=\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta].$$

So:

$${\mathrm{\nabla }}_{\theta }{D}_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })={\mathbb{E}}_{p_{\theta }}[J_{\theta }]-{\mathbb{E}}_{q_{\varphi }}[J_{\theta }]\mathrm{.}\backslash nabla\_\backslash theta\; D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash |p\_\backslash theta)=\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]-\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta].$$

This is interesting. Before Score Function Centering, the extra term in the reward update was exactly:

$${\bar{A}}_q({\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }])\mathrm{.}\backslash bar\; A\_q\backslash left(\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]\backslash right).$$

And when we minimize $D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash Vert\; p\_\backslash theta)$, the update direction it provides is also exactly:

$${\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }]\mathrm{.}\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta].$$

The only difference is that this time the thing controlling that direction is no longer the inexplicable average advantage ${\bar{A}}_q\backslash bar\; A\_q$, but a KL coefficient $\beta \backslash beta$ that we decide ourselves.

So using $D_{\mathrm{K}\mathrm{L}}(q_{\varphi }\mathrm{\parallel }{p}_{\theta })D\_\{\backslash mathrm\{KL\}\}(q\_\backslash phi\backslash Vert\; p\_\backslash theta)$ here has a very natural reading: SFC first takes the uncontrolled trainer-generator mismatch term out of the reward update, and then the KL adds the same direction back as an independent regularization term.

This is why the SFC plus KL decomposition is so neat. It is not simply "add a KL so the trainer and the generator do not differ too much". It decouples the two things that used to be mixed together inside the reward gradient:

$${\mathbb{E}}_{q_{\varphi }}[A{s}_{\theta }]={\mathrm{Cov}}_{q_{\varphi }}(A,J_{\theta })+{\bar{A}}_q({\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }])\mathrm{.}\backslash mathbb\; E\_\{q\_\backslash phi\}[A\; s\_\backslash theta]=\backslash operatorname\{Cov\}\_\{q\_\backslash phi\}(A,J\_\backslash theta)+\backslash bar\; A\_q\backslash left(\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]\backslash right).$$

SFC keeps the first term:

$$g_{\mathrm{r}\mathrm{e}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d}}={\mathrm{Cov}}_{q_{\varphi }}(A,J_{\theta }),g\_\{\backslash mathrm\{reward\}\}=\backslash operatorname\{Cov\}\_\{q\_\backslash phi\}(A,J\_\backslash theta),$$

while the KL takes care of the second direction on its own:

$$g_{\mathrm{K}\mathrm{L}}=\beta ({\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }])\mathrm{.}g\_\{\backslash mathrm\{KL\}\}=\backslash beta\backslash left(\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]\backslash right).$$

So in the end:

$$g_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}={\mathrm{Cov}}_{q_{\varphi }}(A,J_{\theta })+\beta ({\mathbb{E}}_{q_{\varphi }}[J_{\theta }]-{\mathbb{E}}_{p_{\theta }}[J_{\theta }])\mathrm{.}g\_\{\backslash mathrm\{total\}\}=\backslash operatorname\{Cov\}\_\{q\_\backslash phi\}(A,J\_\backslash theta)+\backslash beta\backslash left(\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]-\backslash mathbb\; E\_\{p\_\backslash theta\}[J\_\backslash theta]\backslash right).$$

Now "what to learn" is decided by the reward, and "how far the trainer is allowed to be from the sampler" is decided by $\beta \backslash beta$.

5. One last question: what if this sampler is already very old?

If the KL is pulling toward the generator in the first place, then the older the generator is, the further back into the past it pulls the model. Is that not a problem?

It is.

So when you really do async RL, the tokens should not share one fixed KL strength. A token that was just generated and a token generated a few dozen optimizer steps ago clearly do not correspond to sampler distributions of the same quality as a reference. Suppose token $tt$ was produced at sampler step ${\sigma }_t\backslash sigma\_t$ and the current trainer step is $\tau \backslash tau$. Define its age:

$$e_t=\tau -{\sigma }_t\mathrm{.}e\_t=\backslash tau-\backslash sigma\_t.$$

Here I propose one rule that can be used in practice:

That is, the staler the token, the weaker its KL constraint. The intuition behind this design is that a fresh sampler is still close to the current trainer, so it is a fairly trustworthy local anchor, while a sampler that is already old represents a past policy. We can of course still use the relative reward information it left behind when it produced the rollout, but there is no need to force the current trainer to stay tightly attached to it.

So two things have to be separated here: "which $q_{t}q\_t$ each token uses as its reference" and "how strong the KL weight on that reference should be". These are two different questions. The first is decided by the version that actually generated the token. The second is decided by how stale it is.

Setting ${\beta }_t=0\backslash beta\_t=0$ for a fresh token has a simple reason too: if the generator and the trainer are the same version in the first place, there is no need to treat the tiny distribution difference that comes from numerics, the inference kernel or the precision as an extra imitation target.

But one thing has to be stated clearly: $1\mathrm{/}\sqrt{e_t}1/\backslash sqrt\{e\_t\}$ is not an answer derived from the KL error bound above. The theory tells us that the smaller $D_{\mathrm{K}\mathrm{L}}(q_t\mathrm{\parallel }{p}_{\theta })D\_\{\backslash mathrm\{KL\}\}(q\_t\backslash Vert\; p\_\backslash theta)$ is, the smaller the error bound between the sampler-relative local update and the current-policy local update. It does not tell us that ${\beta }_t\backslash beta\_t$ has to equal ${\beta }_0\mathrm{/}\sqrt{e_t}\backslash beta\_0/\backslash sqrt\{e\_t\}$. The latter is only a design choice that I recommend, and you are free to study a constant coefficient, a batch-level coefficient, other age schedules, or even an adaptive adjustment driven by the observed KL.

There is one more place that easily causes confusion. We said above that the reward gradient no longer uses importance weighting, but you may still see $p_{\theta }(v)\mathrm{/}{q}_{\varphi }(v)p\_\backslash theta(v)/q\_\backslash phi(v)$ inside the sparse reverse-KL estimator. There is no contradiction. The role of the importance ratio in the reward gradient is:

$${\mathbb{E}}_q\left[\frac{p}{q}f\right]={\mathbb{E}}_p[f],\backslash mathbb\; E\_q\backslash left[\backslash frac\; pq\; f\backslash right]=\backslash mathbb\; E\_p[f],$$

which changes the distribution under which we learn the reward signal. The ratio inside the sparse KL is there to estimate a completely different target:

$${\mathrm{\nabla }}_{\theta }{D}_{\mathrm{K}\mathrm{L}}(q\mathrm{\parallel }{p}_{\theta })\mathrm{.}\backslash nabla\_\backslash theta\; D\_\{\backslash mathrm\{KL\}\}(q\backslash |p\_\backslash theta).$$

For the logit $z_{j}z\_j$, for example:

$$\frac{\mathrm{\partial }{D}_{\mathrm{K}\mathrm{L}}(q\mathrm{\parallel }{p}_{\theta })}{\mathrm{\partial }{z}_j}=p_{\theta }(j)-q(j)\mathrm{.}\backslash frac\{\backslash partial\; D\_\{\backslash mathrm\{KL\}\}(q\backslash |p\_\backslash theta)\}\{\backslash partial\; z\_j\}=p\_\backslash theta(j)-q(j).$$

If we only sample one $v\sim qv\backslash sim\; q$, we can use:

$${\widehat{g}}_j(v)=(\frac{p_{\theta }(v)}{q(v)}-1)\mathbf{1}[j=v],\backslash hat\; g\_j(v)=\backslash left(\backslash frac\{p\_\backslash theta(v)\}\{q(v)\}-1\backslash right)\backslash mathbf\; 1[j=v],$$

because:

$${\mathbb{E}}_{v\sim q}[{\widehat{g}}_j(v)]=q(j)(\frac{p_{\theta }(j)}{q(j)}-1)=p_{\theta }(j)-q(j)\mathrm{.}\backslash mathbb\; E\_\{v\backslash sim\; q\}[\backslash hat\; g\_j(v)]=q(j)\backslash left(\backslash frac\{p\_\backslash theta(j)\}\{q(j)\}-1\backslash right)=p\_\backslash theta(j)-q(j).$$

So the $p\mathrm{/}qp/q$ here is only a coefficient inside a sparse Monte Carlo gradient estimator. It does not change the reward update from the $qq$ measure to the $pp$ measure.

6. So does this method actually work?

Everything above is intuition and derivation. To check that this method really works, I ran a large number of experiments, and the baselines include SFC and several of its known variants. This section describes the setup and the overall result, and the sections after it go through the details.

All experiments run on a real asynchronous pipeline. The generator keeps producing rollouts, the trainer keeps updating, and every token records the version of the sampler that generated it, so the staleness of every token is measured.

The tasks are Countdown and GSM8K, and the reward of both is only 0 or 1. Countdown gives the model a few numbers and a target, and the model has to write an arithmetic expression over these numbers whose result equals the target, 1 point when it is right and 0 when it is wrong, so the reward below can be read directly as accuracy. Most experiments use Qwen3-0.6B on Countdown, and the larger models are Qwen3-4B, Qwen3-30B-A3B and GLM-4.5-Air, where GLM-4.5-Air has 106B total parameters and 12B active parameters. Every run trains for 3,200 steps, the context length is at most 4,096 tokens, and the optimizer is SGD.

The size of the staleness is set by the granularity at which the trainer consumes rollouts. The generator produces the rollouts of 64 prompts at a time, the trainer cuts this batch of rollouts into a number of slices, and each slice takes one optimizer step. When each slice has 4 prompts, we call it a 4-prompt slice, and the mean staleness of the tokens is then about 50 steps. When each slice has only 2 prompts, the same batch of rollouts takes twice as many steps to consume, and the mean staleness becomes about 100 steps. The three larger models use a 1-prompt slice.

Below I mainly compare four methods built on SFC. SFC does centering only. SFC+TIS multiplies the centered score by a truncated importance ratio $\min (p_{\theta }\mathrm{/}{q}_{\varphi },2)\backslash min(p\_\backslash theta/q\_\backslash phi,\backslash \; 2)$, which is what truncated importance sampling does. SFC+MIS keeps only the tokens whose ratio falls inside $[0.5,5][0.5,\backslash \; 5]$ and masks out all the other tokens, an approach that comes from IcePop. Combining these two ratio corrections with SFC is what the work mentioned earlier does. The last one is SFC+KL, the method derived above, with ${\beta }_0=0.3\backslash beta\_0=0.3$.

When comparing methods, I look at the best held-out evaluation reward that each run reaches over the whole training, called the best score below. In real training, the checkpoint we keep and use in the end is usually the one with the best evaluation anyway. This way, even if a run collapses later, its best score before the collapse still counts, so besides comparing best scores, I also look separately at whether each run collapsed.

The figure below gives the best scores on seven settings, and each setting is labelled with the mean staleness that was actually measured for it, from 46 steps all the way to 367. SFC+KL is the only method that trained to the end on all seven settings, and in five of them its best score is also the highest of the four methods.

But the figure only gives the results. The sections below explain what happened to each method during training, and answer these questions:

• Is SFC alone enough?
• What goes wrong when importance sampling is added on top of SFC?
• Can importance sampling be dropped altogether?
• How much information does SFC need the generator to send back?
• Do these conclusions still hold with a different optimizer, dataset and model?
• What happens if the difference between the trainer and the generator is not caused by staleness?

At the end there is a bonus section that compares SFC+KL with PPO, DAPO, GSPO, DPPO and TOPR, and also checks whether they can be combined. If you only want the conclusions, you can jump straight to the takeaways at the end.

7. Is SFC alone enough?

This question asks whether score function centering by itself can stabilize async RL training. The answer is no. On the two Countdown settings, the runs that do only SFC basically did not learn anything.

The two figures below show the training reward and the held-out evaluation reward of the four methods on Countdown, with the 4-prompt slice on the left and the 2-prompt slice on the right. This section and the next two sections all use these two figures.

Look at SFC first. In both settings, its training reward stayed around 0.1 for the first one or two hundred steps, then dropped to 0 and never came back. These two runs were stopped at step 400 and step 640, and their best scores are the evaluations from before training started, 0.080 and 0.094.

This agrees with the analysis in section 1. What SFC repairs is the score drift, but it does not limit the distance between the trainer and the sampler. With the sampler 50 to 100 steps older than the trainer on average, training with SFC alone collapsed very quickly. The KL term from sections 3 and 4 is exactly what fills this gap.

8. What happens when importance sampling is added on top of SFC?

On the 4-prompt slice, SFC+TIS and SFC+MIS can both learn for a while, but neither makes it to the end. On the 2-prompt slice, where the staleness is larger, they did not learn from the start. I also saw something in them that SFC+KL does not have: the responses they generate are especially long. SFC+TIS is the clearest case. It makes the responses longer and longer during training, and the share of truncated responses is much higher as a result.

Look at SFC+MIS first. On the 4-prompt slice it learned well at first: its evaluation at step 800 was 0.646, while SFC+KL at the same point was 0.725. But from then on it slowly went down, and it finally collapsed at step 3,168. On the 2-prompt slice, where the staleness is larger, it did not learn from the start and was stopped at step 400.

SFC+TIS is a bit different. On the 4-prompt slice, its training reward rose to around 0.7 at one point within the first 1,300 steps, not far from SFC+KL, only with much larger fluctuations. But its evaluation stayed low, only 0.389 at step 800. Around step 1,350 its training reward suddenly fell and soon reached 0, and it did not recover all the way to step 3,200, with the evaluation also staying at 0. On the 2-prompt slice it never learned anything from beginning to end, and its best score is the 0.111 from the start of training.

Now the response lengths. Over the last 200 steps of training on the 4-prompt slice, the share of responses truncated for exceeding the 4,096-token context length was 99% for SFC+TIS and 49% for SFC+MIS, but only 19% for SFC+KL. For the three runs on the 2-prompt slice that did not learn, this share was also between 89% and 97%.

Sections 2.2 and 2.3 discussed two problems that ratio correction can bring. One is the bias from truncation. The other is that the ratio is incompatible with SFC: multiplying a centered score by the ratio takes the zero-mean property away again. MIS is similar to TIS, except that it masks out entire tokens whose ratio falls outside the interval. For SFC+TIS, I went back and looked carefully at the training logs and also spot-checked what the model generated, and section 8.2 explains step by step how it broke down.

Seeing so many truncated responses, the first possibility that came to my mind was that a context length of 4,096 tokens is too short. If SFC+TIS needs longer responses to solve the problems in the first place, it makes sense that the reward drops once the length is capped. The next subsection looks at exactly this.

8.1 Did SFC+TIS collapse because the context length cap was too small?

No. After raising the context length cap to 8,192 and 16,384, far fewer responses were truncated, but SFC+TIS still collapsed.

This group of experiments stays on the Countdown 4-prompt slice with SFC+TIS, and only raises the context length cap to two times and four times the original.

The table below gives, for these runs, the share of truncated responses over the last 200 steps before training ended and their best scores, with SFC+KL at 4,096 as a reference.

With a higher cap, the truncated share did come down, and at 16,384 it is even lower than SFC+KL at 4,096. The extra length also did let SFC+TIS learn a bit better: its best score rose from 0.389 to around 0.65, though it is still below the 0.725 of SFC+KL at 4,096.

But they still collapsed in the end, and at about the same time as the 4,096 run.

So the length cap is not the cause of the SFC+TIS collapse. With a higher cap it can learn a little more before collapsing, but the collapse itself is not avoided. Why the responses get longer and longer is what the next subsection looks at.

8.2 Why do the responses of SFC+TIS get longer and longer?

In short, these very long responses are a search that becomes more and more aimless and finally turns into meaningless loops, so no amount of extra length is ever enough.

First, let us see what a very long response on Countdown actually contains. I took the model of the 16,384 run at step 319, before the collapse started, and sampled it on three held-out problems with the length cap set to 2,048 tokens. The easier problem was to make 67 from 28, 10, 33 and 16. The model got it right both times, using 595 and 790 tokens, by trying one piece at a time:

What about:
33 - 16 = 17
28 - 10 = 18
18 + 17 = 35. No.
...
How about:
(16 + 33) + (28 - 10) = 49 + 18 = 67. Yes!

On the other two, harder problems, the model kept writing until it reached the 2,048-token cap, with Let me think, What about and Let me try appearing over and over, and the same few partial sums recomputed each time. A normally trained SFC+KL behaves the same way on these three problems: it tries first and then solves the easy one, and it also keeps writing until the cap on the two hard ones. So on Countdown, a very long response is really a brute-force search that has not converged, and how long it gets depends on how aimless that search is.

The problem with SFC+TIS is that training makes this search more and more aimless. From the logs, its collapse goes through three stages, and they are the same under both the 4,096 and the 8,192 caps.

The first stage is a steady rise in entropy. The per-token entropy of the model kept rising for several hundred steps in a row, while the KL between the trainer and the sampler stayed very small. Under the 8,192 cap, the entropy rose from 0.52 in the first 200 steps to 0.83 at steps 576 to 639, and then to 1.4 at steps 768 to 895. The mean response length rose along with it, from 381 tokens to 4,896 tokens, while the training reward could still stay between 0.5 and 0.75 at this point. Under the 4,096 cap, the entropy rose from 0.40 to 0.98 between steps 1,200 and 1,399. A length of 4,896 tokens is already beyond the 4,096 cap, so under the 4,096 cap these responses get truncated. Add to that the fact that after the collapse the model keeps repeating itself up to the cap, and that is why SFC+TIS has a much higher truncated share than SFC+KL.

The second stage is the KL getting out of control. Within one or two windows of 64 steps, the KL between the trainer and the sampler suddenly rose by two orders of magnitude, to 8.4 for the 8,192 run and 5.0 for the 4,096 run. At the same time the gradient norm went down and the reward fell to 0. My understanding is this: the farther the trainer is from the sampler, the lower the probability that the trainer assigns to the tokens the sampler generated, so their ratios are either close to 0 or truncated, the sampler's data almost stops having any effect, and nothing is left that can pull the trainer back. In addition, the tokens truncated by $\min (\cdot ,2)\backslash min(\backslash cdot,\backslash \; 2)$ were only 1.5% to 3% from beginning to end, so truncation itself does not look like the main cause.

The third stage is training coming to a complete stop. All the responses in a batch are wrong, the advantages after group centering are all 0, the gradient norm is exactly 0, and the model can no longer move at all. This is also why the 4,096 run never recovered all the way to step 3,200 after its reward fell to 0. What the model outputs at this point depends on the length cap. Under the 4,096 cap, the model repeats the same content almost deterministically until it hits the cap, and those are the 99% truncated responses. Under the 8,192 cap, the model rambles for about 2,300 tokens with an entropy of about 0.45, and no parseable answer can be found in it. So the extra length just ends up filled with more meaningless content.

SFC+KL on the same setting did not go down this road at all. Over the full 3,200 steps, its entropy stayed between 0.23 and 0.35, and its KL to the sampler stayed between 0.003 and 0.004. I think the reason is that the KL constrains the complete distribution at every position, including how spread out this distribution is, and also the probability of the token that ends the sequence. An importance ratio acts only on the token that was sampled, so it cannot do this.

9. Then can importance sampling be dropped altogether?

Yes. SFC+KL uses no importance ratio at all in the reward gradient. In the two figures of section 7, it is the only method that trains stably to the end on both settings, with best scores of 0.725 and 0.670.

Across all the settings in this article, SFC+KL in its default configuration never collapsed even once, and that includes the experiments later on with a different optimizer, a different dataset, a different model, and a quantized sampler. In the same settings, SFC, SFC+TIS and SFC+MIS each collapsed more than once.

This is also the one thing the first half of this article most wanted to show. A stale rollout does not necessarily have to be reweighted into data under the current policy. We can keep its learning signal under $q_{\varphi }q\_\backslash phi$ and use KL to control how far the trainer is from the sampler, and training can be just as stable.

10. How much information does SFC need the generator to send back?

On Countdown, the composition of the payload has little effect on the best score, and sending back 16 tokens per position is enough. The difference lies in stability: the run that sends back only the top-16 collapsed later, while the configurations that also use 16 tokens but take part of them from tail samples did not.

Let me first say what is being compared here. To compute ${\mathbb{E}}_{q_{\varphi }}[J_{\theta }]\backslash mathbb\; E\_\{q\_\backslash phi\}[J\_\backslash theta]$, the generator has to send part of its distribution back to the trainer along with every token it generates, for example the few tokens with the highest probabilities, plus a few tokens sampled from the remaining tail. I call this information the payload. The default payload used in all the experiments above is top-128: at every position the generator sends back only the 128 tokens with the highest probabilities and their logprobs, and the tail is not sampled. On the trainer side, the total probability of the tail is set to the value the sampler gives, and how much probability each token in the tail gets is allocated according to the trainer's own distribution. The other approach is the head plus tail described in section 3.5 of the first article. The head still sends back the top-k, and for the tail, a few extra tokens are sampled from the sampler and sent back, which I call tail samples below. The width of the payload is the total number of tokens in these two parts.

This group of experiments is all on the Countdown 4-prompt slice with SFC+KL, and only the composition of the payload changes.

The best scores of the nine configurations are all between 0.68 and 0.73, so from the best score alone I cannot tell which of these configurations is better.

The difference is in stability. The run that sends back only the top-16 scored 0.713 at step 2,000, as good as the other configurations, but it collapsed at step 2,992. With the same 16 tokens, top-8 plus 8 tail samples and top-4 plus 12 tail samples both trained stably to the end.

My understanding is this. When only the top-k is sent back, the tail is filled in with the trainer's own distribution, and how accurate that is depends on how much probability the tail holds and on how much the trainer and the sampler differ on the tail. The narrower the top-k, the more probability is left to the tail, and the larger the error of this approximation. Tail samples come directly from the sampler's own distribution, so the estimate of the tail does not have this bias, at the cost of some extra variance.

This result also shows that a collapse can sometimes come very late. If we had trained for only 2,000 steps, we would probably have happily concluded that "sending back only the top-16 is enough".

The width of the payload also affects speed. I tested each payload separately for 100 steps, without evaluation and with the same seed.

The trainer's compute time per step is almost unaffected by the payload and stays around 1.2 seconds. What changes most is the rollout file. As the width goes from 128 down to 16, the rollout file goes from 2,074 MB down to 282 MB, and the time to read and upload it goes from 1.4 to 1.5 seconds down to about 0.2 seconds. The cost of tail samples falls on the generator: at the same width, the more tokens are sampled, the slower the decoding. At a width of 128, 96 tail samples made the decode time 18% longer, while at a width of 16 they made it only 2% to 3% longer.

If I wanted to save bandwidth, I would pick a configuration like top-8 plus 8 tail samples. Its rollout file is about one seventh of the default configuration's, its generator decode time is about the same as the default configuration's, and its best score is 0.705, on par with the other configurations.

11. What about a different optimizer, dataset or model?

On these five settings, SFC+KL trained to the end without collapsing. But looking at the best score, it is not the highest every time: on Qwen3-4B and GLM-4.5-Air, SFC+MIS has a higher best score.

There are five settings here, which are the five settings in the figure of section 6 other than the first two. The first one changes the optimizer to AdamW and is otherwise the same as the Countdown 4-prompt slice. The second one changes the dataset to GSM8K, still with Qwen3-0.6B and the 4-prompt slice. The remaining three change to larger models, Qwen3-4B, Qwen3-30B-A3B and GLM-4.5-Air, where GLM-4.5-Air has 106B total parameters and 12B active parameters. These three models use the 1-prompt slice, and their mean staleness is between about 90 and 370 steps.

The two figures below are the training reward and held-out evaluation reward curves on these five settings.

The optimizer first. With AdamW, SFC+KL, SFC+TIS and the SFC-only run all got above 0.7 at step 800, at 0.738, 0.725 and 0.715 respectively, while SFC+MIS had a lower best score of 0.652. The difference shows up later. SFC+TIS and SFC+MIS both collapsed later on. The SFC-only run did not collapse, but its evaluation kept falling, down to 0.527. Only SFC+KL stayed above 0.7 the whole time. This is also the only time that SFC alone did not collapse. My guess is that AdamW normalizes the update size of every parameter, which by itself acts as a kind of constraint, but I have not verified this guess.

Then the dataset. On GSM8K, SFC+TIS trained to the end with a best score of 0.785, and SFC+KL had 0.803, so the two are about the same. This is the only one of these five settings where SFC+TIS behaved normally. Its responses were still very long, with a truncated share of 97% against 43% for SFC+KL. It is just that on GSM8K this did not stop it from solving the problems. SFC alone and SFC+MIS both collapsed on GSM8K.

Finally, the three larger models. The best scores of SFC+KL are 0.744, 0.877 and 0.750. SFC+TIS collapsed on Qwen3-4B and GLM-4.5-Air. On Qwen3-30B-A3B it did not collapse, but it never went above the 0.680 from the start of training. The SFC-only run was run only on Qwen3-4B, and it also collapsed.

SFC+MIS on these three models is quite interesting. It collapsed on none of them, and its best was 0.871 on Qwen3-4B and 0.777 on GLM-4.5-Air, both higher than SFC+KL. But its scores swung a lot and dropped after the peak, and its last evaluations were only 0.668 and 0.574. The score of SFC+KL on GLM-4.5-Air first dropped a bit at step 400 and then climbed slowly, and its best score came at the end of training.

There is one thing to point out here. On Qwen3-4B and Qwen3-30B-A3B, SFC+KL was evaluated only once at the start and once at the end of training, while SFC+MIS was evaluated every 400 steps. So on these two models, the best score of SFC+KL is in fact its last score, and we do not know whether it reached higher somewhere in the middle of training.

There is also one thing I have not figured out. SFC+MIS collapsed on all four settings with Qwen3-0.6B, but on these three larger models, where the staleness did not get smaller, it held up on every one of them.

12. What if the difference between the trainer and the generator is not caused by staleness?

Then KL has no clear advantage. In this group of quantized-sampler experiments, SFC+TIS and SFC+KL performed almost the same.

The discussion so far has assumed that the difference between the trainer and the generator comes from staleness. But in real systems there is another common source. To run inference faster, the generator uses a quantized model while the trainer uses an unquantized one, so even when both sides use the same version of the parameters, the distributions they give are not exactly the same.

This group of experiments uses Qwen3-30B-A3B-Base and the math problems in INTELLECT-2, with the generator using three kinds of quantization, and each run trains for 800 steps. Unlike before, ${\beta }_0\backslash beta\_0$ of SFC+KL is 0.03 in this group.

The gaps in all three groups are within 0.012, and I would treat them as the same. On this group of experiments, TIS works as well as KL. For reference, these runs all scored between 0.33 and 0.35 at the start of training. In the INT8 plus INT4 KV cache group, both methods improved by only about 0.01 after 800 steps of training, so quantized to this degree, neither method really learned anything.

I think the reason is that the whole argument in the first half of this article is built around staleness. The older the sampler, the larger its distance to the trainer, and that distance is exactly what the KL is meant to control. The difference that quantization brings is a different situation: both sides use the same version of the parameters, and the distributions differ only numerically, so there is not much for the KL to constrain in the first place. On the TIS side, the ratios should mostly be close to 1, few tokens get truncated, and the bias from truncation is small. Section 5 sets ${\beta }_t=0\backslash beta\_t=0$ for fresh tokens for the same reason: a small difference caused by numerical precision under the same version does not need to be treated as a target to imitate.

13. Bonus: next to other methods, and whether they can be combined

The sections above only compare the four SFC-based methods with each other. This section brings in PPO, DAPO, GSPO, DPPO and TOPR as well. For PPO you can read this article, and for DAPO and GSPO this one. DPPO replaces PPO's ratio clipping with a constraint on the policy divergence. TOPR truncates the importance sampling differently for samples with positive and with negative rewards.

13.1 Next to other methods

Compared on its own, SFC+KL is not the best on every setting. The table below gives the best score of each method on six settings, and the first column is the average score of these runs at the start of training.

SFC+KL is the highest on three settings, and TOPR on the other three. On the Countdown 2-prompt slice the two differ by only 0.006, which can be seen as a tie. On GSM8K, TOPR and GSPO are both more than 0.035 higher than SFC+KL. On Qwen3-4B, TOPR is 0.018 higher than SFC+KL, though as mentioned before, SFC+KL was evaluated only at the end on this model. In terms of stability, TOPR trained to the end on all six settings, just like SFC+KL. PPO, DAPO and DPPO each failed on more than one setting: some collapsed midway, and some never got above their score at the start of training.

13.2 Can they be combined?

Yes, and adding it rarely makes things worse. I added SFC+KL on top of TOPR and GSPO, that is, I added SFC's centering and the KL term on top of each of them, which gives 12 comparisons over the six settings.

In 10 of the 12 comparisons the best score went up, though quite a few of them by less than 0.02, which is not much of a difference. Five comparisons changed by more than 0.02. Among the four that improved, the most obvious one is Qwen3-4B. TOPR went from 0.762 up to 0.822. GSPO on its own reached 0.557 at step 400 and then dropped to nearly 0, and with SFC+KL added it trained to the end with a best score of 0.822. The other two are TOPR on the Countdown 2-prompt slice and GSPO on Qwen3-30B-A3B. The one that got worse is TOPR on GSM8K, lower by 0.022.

So SFC+KL can also be seen as a component that can be added on top of other methods, and not necessarily as something you have to choose instead of them. Adding it rarely makes things worse, and on a setting like Qwen3-4B it can help a lot.

14. A few takeaways

Let me summarize what these experiments tell us:

• In asynchronous training with large staleness, SFC alone is not enough. It repairs the score drift but does not limit how far the trainer is from the sampler, and with SGD it collapses very quickly.
• Adding TIS or MIS on top of SFC is very unstable. On Qwen3-0.6B, they collapse or fail to learn most of the time. SFC+MIS did not collapse on two of the larger models, and its best score even beat SFC+KL there, but it dropped hard after the peak.
• SFC+TIS makes the responses longer and longer, and the truncated share is much higher as a result. Its entropy keeps rising during training and the search becomes more and more aimless. In the end the KL between the trainer and the sampler suddenly rises by two orders of magnitude, and training comes to a complete stop. Raising the context length cap fourfold did not prevent the collapse either, and the extra length was just filled with meaningless content.
• SFC+KL removes importance sampling and uses KL to control the distance between the trainer and the sampler instead, and it never collapsed in any setting of this article. In five of the seven asynchronous settings, its best score was the highest among these SFC methods.
• On Countdown, the composition of the payload has little effect on the best score, and compressing it to 16 tokens per position is enough, but part of it should go to tail samples. The run that sent back only the top-16 collapsed late in training.
• If the difference between the trainer and the generator comes mainly from quantization rather than from staleness, TIS works as well as KL.
• SFC+KL can be added on top of methods like TOPR and GSPO. In 10 of the 12 comparisons the best score went up, and Qwen3-4B improved the most.

15. Closing words

Back to the question this article started from: when you learn from a stale rollout, do you have to "correct it into the present" before you can use it?

The answer is that you do not have to. The first half of this article explained why, and in the second half, SFC+KL, which uses no importance ratio in the reward gradient, was the only one of the four SFC methods that trained to the end on all seven asynchronous settings.

Importance sampling answers: the data comes from $q_{\varphi }q\_\backslash phi$, but I will try to recover the current-policy expectation through $p_{\theta }\mathrm{/}{q}_{\varphi }p\_\backslash theta/q\_\backslash phi$. Sampler-relative learning answers: since the data was produced by $q_{\varphi }q\_\backslash phi$, I will first keep the relative reward signal it expresses under its own distribution, and as for how far it has already drifted from the current trainer, I control that with a separate KL constraint.

Put simply, the method proposed in this article drops importance sampling, and it can be seen as the combination of Score Function Centering, a sparse KL and staleness scaling, each with its own job:

• SFC: I accept that the data comes from $q_{\varphi }q\_\backslash phi$, so I center under $q_{\varphi }q\_\backslash phi$.
• KL: but I do not allow $q_{\varphi }q\_\backslash phi$ and $p_{\theta }p\_\backslash theta$ to drift too far apart.
• Staleness scaling: and the older $q_{\varphi }q\_\backslash phi$ is, the less I think its reference value counts.

I think this is a new way of understanding the trainer-generator mismatch in async RL: do not assume that stale data has to be reweighted into current-policy data first. You can also keep the learning signal it carries, and then split "what to learn" and "how far from the current policy" into two separate questions.

Limited by computing resources, these experiments did not do a systematic hyperparameter sweep, and they did not run multiple seeds for each configuration, so the results in this article are for reference only. The link to the code and data is at the beginning of the article, and you are welcome to keep building on it. Thank you for reading this far.