course_notes/offline.md

---

comments: True

description: Offline (batch) reinforcement learning concepts, dataset shift challenges, and practical algorithms such as BCQ, BRAC, CQL, IQL, and TD3+BC for stable learning without environment interaction.

---

Offline RL

Offline RL learns policies without further environment interaction, using a fixed dataset (\mathcal{D} = {(s,a,r,s')}) collected by some behavior policy (\mu). This setting is powerful when exploration is unsafe or expensive, but introduces severe distribution shift: learned policies may query out-of-distribution (OOD) actions/states where value estimates are unreliable.

Key Challenges

• Extrapolation error: Q-functions overestimate values for actions rarely or never seen in (\mathcal{D}).
• Distribution shift: Policy improvement moves away from (\mu), compounding error in bootstrapped targets.
• Coverage: If (\mathcal{D}) lacks support for optimal actions, performance is upper-bounded by the best in-data behavior.

Core Approaches

Conservative / Penalized Q-Learning

• CQL: Adds a conservative term to push Q-values down on OOD actions: [ \mathcal{L}{\text{CQL}} = \mathcal{L}{\text{TD}} + \alpha \Big(\mathbb{E}{a \sim \pi}[Q(s,a)] - \mathbb{E}{a \sim \mu}[Q(s,a)]\Big), ] approximated via sampled actions. Reduces overestimation and prevents greedy policies from exploiting spurious high Q.
• IQL: Avoids explicit behavior cloning by regressing Q-values toward expectile values of returns, then distilling a policy via advantage-weighted regression. Stable and performant without importance sampling.

Behavior-Regularized Policy Improvement

• BCQ: Trains a generative model (VAE) of behavior actions; policy selects near-behavior actions and uses a perturbation model for refinement.
• BRAC: Penalizes divergence between (\pi) and (\mu) (KL or MMD) during actor updates.
• TD3+BC / AWR / AWAC: Adds behavior-cloning or advantage-weighted regression terms to the actor loss to keep actions close to the dataset support while still improving return.

Model-Based Offline RL

• Learn a dynamics model, generate imagined rollouts only near the data distribution (short horizons, uncertainty penalties).
• Use conservative value estimation on model rollouts (e.g., MOPO/MOReL) by adding penalties for model-uncertain states.

Practical Tips

• Start with TD3+BC or IQL for continuous control baselines; they balance simplicity and robustness.
• Monitor divergence: action likelihood under the behavior model or KL to (\mu).
• Calibrate uncertainty: ensembles for Q-functions or dynamics; penalize high-variance predictions.
• Limit policy updates (small step sizes, target networks) and prefer short rollout horizons if using models.
• Normalize rewards and observations; ensure replay buffers are frozen to avoid on-policy contamination.
• Evaluate with multiple seeds; offline metrics (e.g., behavior cloning loss, estimated return) are weak—deploy cautiously.

When to Use Offline RL vs. IL

• If returns are available in the dataset and coverage is adequate, offline RL can outperform pure imitation by improving over suboptimal experts.
• If demonstrations are narrow and high quality, BC or DAgger-style IL may be safer; combine with conservative RL to further improve.

References

• Fujimoto et al. (2019) BCQ; Kumar et al. (2019) BRAC; Kumar et al. (2020) CQL; Kostrikov et al. (2021) IQL; Fujimoto & Gu (2021) TD3+BC; Yu et al. (2020) MOPO; Kidambi et al. (2020) MOReL.