MOPD: how frontier labs merge RL experts into one model and how to do it using TRL

TRL now supports MOPD. AsyncDistillationTrainer routes each prompt to its own expert teacher, and support for the synchronous DistillationTrainer is on its way in #7215. If you already know what MOPD is and are here for the TRL example, jump to MOPD in TRL.

One of the post-training techniques that keeps showing up in this year's tech reports is Multi-Teacher On-Policy Distillation (MOPD). Xiaomi named it in January for MiMo-V2-Flash, and since then DeepSeek, Moonshot, NVIDIA, Liquid AI and Mistral, among others, have used it to build their models.

Getting a single model to be good at many domains at once, such as math, code, tool use or following instructions, with one RL run is hard. During training the domains pull the model in different directions, each one gets a thin slice of every batch, and whatever improves one can quietly hurt another. With MOPD, labs get around this by training one expert per domain and then merging the experts back into a single model.

We covered the broader picture of how labs use distillation in Distillation in 2026 (so far).

What is MOPD?

Distillation trains a model, the student, to imitate another one, the teacher. In MOPD there are several teachers, one per domain, and the student learns from all of them. Strictly, MOPD is the last stage of a three-stage post-training recipe, although the name is often used for the whole recipe:

1. General SFT: one model learns the basics of every domain with regular supervised fine-tuning on mixed data. This is the shared starting point.
2. Domain experts: starting from that checkpoint, a separate model is trained with RL for each domain, for example one for math, one for code, one for instruction following and one for agentic tasks. Each expert only has to be really good at one thing.
3. Fusion: the student, usually the SFT checkpoint again, answers prompts from every domain. The expert of each prompt's domain acts as its teacher and scores the answer token by token, and the student is trained to move toward that teacher's distribution.

The MOPD recipe: one SFT checkpoint, one RL expert per domain, then a fusion loop where every prompt is scored by its domain's expert.

MOPD is on-policy distillation with several teachers. Three properties make it work, the first two from on-policy distillation itself and the third from having several teachers:

• The student learns from its own samples. The teacher grades what the student actually wrote. With plain SFT on teacher outputs, the student only ever trains on text the teacher wrote, but at inference it has to continue its own text, and one early mistake takes it somewhere it never saw in training. Training on its own samples avoids that, and this is the "on-policy" part.
• The signal is dense. Where RL gives one reward per answer, here every token gets feedback from the teacher, and the loss is usually the reverse KL between student and teacher on the student's tokens, that is, how much the teacher disagrees with each token the student chose. The teacher never writes anything itself, it only reads the student's answer. Labs differ in how much of the teacher's distribution they use at each token. Nemotron 3 Ultra and MiMo-V2-Flash only use the teacher's probability for the token the student wrote. Open-MOPD compares the 16 most likely tokens. DeepSeek-V4 uses the full distribution over the vocabulary, because the single-token version gave them high-variance gradients and unstable training. Mistral Large 4 also uses the full distribution.
• Each prompt goes to one teacher. A math prompt is graded only by the math expert, a code prompt only by the code expert.

In these recipes the teachers usually have the same size as the student and start from the same checkpoint. They are useful as teachers because each one was trained on a single domain.

Who uses it

Liquid AI's recipe is the simplest to read. A generalist lane goes from SFT to MOPD, and a lane of specialists, each one trained with SFT and RL, feeds into it.

LFM2.5-2.6B training recipe. Source: Liquid AI.

MiMo-V2-Flash, the paper that named the technique, draws the fusion step in more detail. The student rolls out, and each domain teacher returns a token-level reward.

Figure 3 of the MiMo-V2-Flash technical report.

NVIDIA shows what MOPD looks like at scale. For Nemotron 3 Ultra they use more than ten teachers and run MOPD twice. After the first round, they train new teachers starting from the merged model and run MOPD again.

Figure 10 of the Nemotron 3 Ultra technical report.

NVIDIA also released five of those teachers on the Hub, for STEM, competition coding, instruction following, general reasoning and chat. Most labs only release the final model, so expert checkpoints are still rare, and we expect more of them in the coming months.

These are the main releases whose own reports or model cards describe MOPD:

MOPD builds on regular on-policy distillation, with a single teacher, which Thinking Machines explained well in late 2025. On the research side, the MOPD paper studies the method on its own, and Open-MOPD releases the full recipe with its checkpoints, which is what we use in the example below.

MOPD in TRL

In TRL, MOPD is supported in AsyncDistillationTrainer, which is what the example below uses. You serve each expert with vLLM, give the trainer one URL per teacher, and add a teacher_id column that routes each prompt to its expert. Generation, teacher scoring and training run asynchronously, so the student keeps generating while it trains.

We built an example to show the feature with real experts, the ones released with Open-MOPD. The student is their SFT checkpoint of SmolLM3-3B, and the teachers are three experts trained from it with GRPO, for math, code and instruction following. The released prompt mix already labels each prompt with its domain.

from datasets import interleave_datasets, load_dataset
from trl.experimental.async_distillation import AsyncDistillationConfig, AsyncDistillationTrainer

dataset = load_dataset("BytedTsinghua-SIA/Open-MOPD-Data", data_files="rl_prompt_mix/train.parquet", split="train")
dataset = dataset.select_columns(["prompt", "domain"]).rename_column("domain", "teacher_id").shuffle(seed=42)
# About half the prompts are instruction following, so sample math, code and IF 40/40/20
dataset = interleave_datasets(
    [dataset.filter(lambda row, d=domain: row["teacher_id"] == d) for domain in ("math", "code", "if")],
    probabilities=[0.4, 0.4, 0.2],
    seed=42,
)

config = AsyncDistillationConfig(
    teacher_server_urls={
        "math": "http://localhost:8001",
        "code": "http://localhost:8002",
        "if": "http://localhost:8003",
    },
    beta=1.0,  # reverse KL against the routed teacher
    teacher_top_k=1,  # with beta=1.0 only the sampled token's teacher logprob is used
    max_completion_length=16384,
    request_timeout=3600,
)
trainer = AsyncDistillationTrainer(
    model="BytedTsinghua-SIA/Open-MOPD-SmolLM3-3B-MixSFT",
    args=config,
    train_dataset=dataset,
)
trainer.train()

With reverse KL (beta=1.0), TRL only uses the teacher's probability for the token the student sampled, like Nemotron 3 Ultra and MiMo-V2-Flash, so the example does not follow Open-MOPD's top-16 comparison. teacher_top_k=1 keeps each teacher's response small. The top-k only matters with forward KL (beta=0.0), where the loss compares the teacher's k most likely tokens plus one bucket for the rest of the probability mass. Comparing the teacher and the student token by token is also why every teacher must share the student's tokenizer.

The full script is async_distillation_mopd.py in the TRL examples.

Running it on HF Jobs

MOPD needs five processes running at once: a vLLM server for the student, which generates the answers and receives the updated weights after every step, one vLLM server per teacher (three in this example), which only score, and the training process. This is how we fit them on a single machine with 4 H200s:

A job on HF Jobs gives you one container and one command, so the example ships a launcher.py that starts the four servers, waits until they answer, and then runs the training script. The whole run is one command:

hf jobs uv run \
    --flavor h200x4 \
    --image huggingface/trl \
    --secrets HF_TOKEN \
    --timeout 6h \
    https://raw.githubusercontent.com/huggingface/trl/main/examples/async_distillation_math/launcher.py

If you run it on your own machine, copy the vllm serve commands from the launcher. Every teacher gets a larger --max-model-len than the student, because scoring sends the student's whole answer plus one token, and the two teachers that share a GPU start one after the other, because vLLM sizes its memory from what is free when it starts.

This layout is the one we tested, and it depends on the hardware you have. With 8 GPUs (h200x8), for example, you can give each teacher its own GPU, run the student's server on two GPUs to generate faster, and train on the remaining three.

Watching the token imbalance

The trainer logs metrics per teacher, so you can see how the student is learning from each one. teacher_jsd/<id> shows how far the student is from each teacher (lower means closer), and teacher_token_frac/<id> shows what share of each training step went to that teacher.

teacher_token_frac is the one to keep an eye on. The loss is averaged over tokens, so each domain weighs as much as the tokens it produces, not as much as the prompts it has. A math or code answer is a long chain of reasoning, thousands of tokens. An instruction-following answer, like "write two paragraphs separated by asterisks", is a few hundred. So instruction following can have 20% of the prompts and still get 1 or 2% of the tokens, and the student barely learns from that teacher. The Open-MOPD authors spotted this and explain it in depth in their paper, together with a fix.

We ran a short experiment with the setup above to check that the TRL implementation works and that these metrics show what is going on. It is a short run, 60 steps or about 1% of the training the paper did. The imbalance shows up in the first few steps. Instruction following starts with a fair share of tokens, but once math and code answers get long it falls to a few percent. In the plot below, the student gets steadily closer to the math and code teachers, while its distance to the instruction-following teacher barely moves.

Metrics logged by TRL in a 60-step run with the Open-MOPD experts and prompts split 40/40/20. Top, the share of trained tokens per teacher (teacher_token_frac/<id>): instruction following has 20% of the prompts and gets 1 to 3% of the tokens. Bottom, the divergence to each teacher (teacher_jsd/<id>, 5-step moving average over the raw values): math and code get closer to their teachers, instruction following barely moves.

It also shows up in the evaluation. We stopped the run after 60 steps and evaluated it with the Open-MOPD evaluation sets and scorers, next to the starting checkpoint and the paper's final model, which was trained with the full recipe, including its fix for this imbalance:

AIME: 4 samples per problem. IFEval: strict prompt-level accuracy. We did not evaluate code.

The student improves on both domains in 60 steps, enough to show that the example works. On math it lands where the paper's final model does (with 4 samples per problem, a few points either way is noise). Instruction following is where the gap is, the domain that got the fewest tokens.

The checkpoint, the Trackio dashboard with every metric of the run, and the Open-MOPD models and data are in this collection.

The example uses math, code and instruction following because those are the experts Open-MOPD released, but the idea is that you plug in your own. If you have RL experts for the domains you care about, trained from the same base model so they share a tokenizer, you only need to serve them with vLLM and label each prompt with its domain. From there you can train longer or give the instruction-following prompts more weight in the data, and the per-teacher metrics will tell you if it helps.

Takeaway

MOPD lets a single model keep what several RL experts learned, without one RL run having to balance every domain at once. Labs are already using it at scale, and with the experts some of them release, open data and AsyncDistillationTrainer, you can now try it yourself. If you want to go further, the AsyncDistillationTrainer docs cover the rest of its options and metrics, and the DistillationTrainer docs the synchronous version.