DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Paper • 2402.03300 • Published • 152
Papers implemented in TRL
Every paper in the TRL paper index, in the same order: https://huggingface.co/docs/trl/paper_index
Note Group Relative Policy Optimization
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Paper • 2501.12948 • Published • 458Note Group Relative Policy Optimization
Group Sequence Policy Optimization
Paper • 2507.18071 • Published • 319Note Group Relative Policy Optimization
Geometric-Mean Policy Optimization
Paper • 2507.20673 • Published • 30Note Group Relative Policy Optimization
DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Paper • 2503.14476 • Published • 148Note Group Relative Policy Optimization
Demystifying Long Chain-of-Thought Reasoning in LLMs
Paper • 2502.03373 • Published • 57Note Group Relative Policy Optimization
INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning
Paper • 2505.07291 • Published • 16Note Group Relative Policy Optimization
Skywork Open Reasoner 1 Technical Report
Paper • 2505.22312 • Published • 56Note Group Relative Policy Optimization
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
Paper • 2506.01939 • Published • 189Note Group Relative Policy Optimization
Understanding R1-Zero-Like Training: A Critical Perspective
Paper • 2503.20783 • Published • 60Note Group Relative Policy Optimization
Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
Paper • 2508.08221 • Published • 50Note Group Relative Policy Optimization
Safe and Efficient Off-Policy Reinforcement Learning
Paper • 1606.02647 • PublishedNote Group Relative Policy Optimization
Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model
Paper • 2510.18855 • Published • 74Note Group Relative Policy Optimization
The Art of Scaling Reinforcement Learning Compute for LLMs
Paper • 2510.13786 • Published • 34Note Group Relative Policy Optimization
It Takes Two: Your GRPO Is Secretly DPO
Paper • 2510.00977 • Published • 31Note Group Relative Policy Optimization
Soft Adaptive Policy Optimization
Paper • 2511.20347 • Published • 44Note Group Relative Policy Optimization
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
Paper • 2512.02556 • Published • 275Note Group Relative Policy Optimization
GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
Paper • 2601.05242 • Published • 234Note Group Relative Policy Optimization
Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
Paper • 2602.05261 • Published • 38Note Group Relative Policy Optimization
VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training
Paper • 2602.10693 • Published • 38Note Group Relative Policy Optimization
Score Centering Stabilizes Off-policy Reinforcement Learning
Paper • 2609.20807 • Published • 10Note Group Relative Policy Optimization
Accelerating RL for LLM Reasoning with Optimal Advantage Regression
Paper • 2505.20686 • Published • 3Note Optimal Advantage Regression
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Paper • 2305.18290 • Published • 73Note Direct Preference Optimization
SLiC-HF: Sequence Likelihood Calibration with Human Feedback
Paper • 2305.10425 • Published • 7Note Direct Preference Optimization
Statistical Rejection Sampling Improves Preference Optimization
Paper • 2309.06657 • Published • 15Note Direct Preference Optimization
Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
Paper • 2309.16240 • PublishedNote Direct Preference Optimization
A General Theoretical Paradigm to Understand Learning from Human Preferences
Paper • 2310.12036 • Published • 20Note Direct Preference Optimization
Towards Efficient and Exact Optimization of Language Model Alignment
Paper • 2402.00856 • Published • 2Note Direct Preference Optimization
Noise Contrastive Alignment of Language Models with Explicit Rewards
Paper • 2402.05369 • Published • 2Note Direct Preference Optimization
Provably Robust DPO: Aligning Language Models with Noisy Feedback
Paper • 2403.00409 • Published • 2Note Direct Preference Optimization
Binary Classifier Optimization for Large Language Model Alignment
Paper • 2404.04656 • Published • 2Note Direct Preference Optimization
Learn Your Reference Model for Real Good Alignment
Paper • 2404.09656 • Published • 91Note Direct Preference Optimization
Iterative Reasoning Preference Optimization
Paper • 2404.19733 • Published • 49Note Direct Preference Optimization
Self-Play Preference Optimization for Language Model Alignment
Paper • 2405.00675 • Published • 28Note Direct Preference Optimization
Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
Paper • 2405.16436 • Published • 1Note Direct Preference Optimization
Distributional Preference Alignment of LLMs via Optimal Transport
Paper • 2406.05882 • Published • 2Note Direct Preference Optimization
Discovering Preference Optimization Algorithms with and for Large Language Models
Paper • 2406.08414 • Published • 14Note Direct Preference Optimization
WPO: Enhancing RLHF with Weighted Preference Optimization
Paper • 2406.11827 • Published • 17Note Direct Preference Optimization
Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
Paper • 2408.06266 • Published • 10Note Direct Preference Optimization
Length Desensitization in Direct Preference Optimization
Paper • 2409.06411 • PublishedNote Direct Preference Optimization
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
Paper • 2411.10442 • Published • 84Note Direct Preference Optimization
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training
Paper • 2411.15124 • Published • 67Note Direct Preference Optimization
SimPO: Simple Preference Optimization with a Reference-Free Reward
Paper • 2405.14734 • Published • 13Note Direct Preference Optimization
Unbiased Alignment for Large Language Models with Noisy Preferences
Paper • 2607.03248 • Published • 1Note Direct Preference Optimization
KTO: Model Alignment as Prospect Theoretic Optimization
Paper • 2402.01306 • Published • 24Note Kahneman–Tversky Optimization
TailSFT: Filtered Fine-Tuning Improves Post-Training Performance
Paper • 2608.25756 • PublishedNote Supervised Fine-Tuning
EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes
Paper • 2508.00180 • Published • 1Note Supervised Fine-Tuning
On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
Paper • 2508.05629 • Published • 182Note Supervised Fine-Tuning
Fewer Truncations Improve Language Modeling
Paper • 2404.10830 • Published • 5Note Supervised Fine-Tuning
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Paper • 1910.10683 • Published • 20Note Supervised Fine-Tuning
Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
Paper • 2605.22765 • Published • 4Note Supervised Fine-Tuning
Reducing Activation Recomputation in Large Transformer Models
Paper • 2205.05198 • PublishedNote Supervised Fine-Tuning
LoRA: Low-Rank Adaptation of Large Language Models
Paper • 2106.09685 • Published • 67Note Parameter-Efficient Fine-Tuning (PEFT)
DoRA: Weight-Decomposed Low-Rank Adaptation
Paper • 2402.09353 • Published • 33Note Parameter-Efficient Fine-Tuning (PEFT)
Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
Paper • 2402.14740 • Published • 18Note Reinforce Leave-One-Out
REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models
Paper • 2501.03262 • Published • 97Note Reinforce Leave-One-Out
ORPO: Monolithic Preference Optimization without Reference Model
Paper • 2403.07691 • Published • 75Note Odds Ratio Preference Optimization
Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
Paper • 2401.08417 • Published • 37Note Contrastive Preference Optimization
AlphaPO -- Reward shape matters for LLM alignment
Paper • 2501.03884 • Published • 2Note Contrastive Preference Optimization
Triple Preference Optimization: Achieving Better Alignment with Less Data in a Single Step Optimization
Paper • 2405.16681 • Published • 4Note Triple Preference Optimization
Nash Learning from Human Feedback
Paper • 2312.00886 • Published • 17Note Nash Learning from Human Feedback
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
Paper • 2312.09244 • Published • 8Note Reward Modeling
Llama 2: Open Foundation and Fine-Tuned Chat Models
Paper • 2307.09288 • Published • 252Note Reward Modeling
Direct Language Model Alignment from Online AI Feedback
Paper • 2402.04792 • Published • 36Note Online Direct Preference Optimization
Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
Paper • 2405.21046 • Published • 4Note Online Direct Preference Optimization
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
Paper • 2306.13649 • Published • 38Note Distillation
MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training
Paper • 2606.30406 • Published • 29Note Distillation
On the Position Bias of On-Policy Distillation
Paper • 2606.22600 • Published • 3Note Distillation
X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation
Paper • 2605.21699 • Published • 4Note Distillation
Knowledge Distillation of Large Language Models
Paper • 2306.08543 • Published • 25Note Distillation
Reinforcement Learning via Self-Distillation
Paper • 2601.20802 • Published • 52Note Distillation
Self-Distillation Enables Continual Learning
Paper • 2601.19897 • Published • 39Note Distillation
Embarrassingly Simple Self-Distillation Improves Code Generation
Paper • 2604.01193 • Published • 55Note Distillation
A Fix for Softmax and Cross-Entropy
Paper • 2610.31642 • Published • 5Note Distillation
Entropy-Preserving Reinforcement Learning
Paper • 2603.11682 • PublishedNote Distillation
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Paper • 1910.02054 • Published • 11Note Distributed Training