huggingface-projects/Deep-RL-Course-Certification
Updated • 244 • 18
AI & ML interests
None defined yet.
Recent Activity
View all activity
sergiopaniego
updated a
dataset about 2 hours ago
sergiopaniego
posted an update about 19 hours ago
Post
678
LFM2.5-2.6B just dropped!
and the @liquidai blog comes with some nice details about the training procedure, so let's analyze it.
basically, a full agent training pipeline but compressed into 2.6B
base model → SFT → specialized teachers per domain (SFT + RLVR) → on-policy distillation back into one student → agentic RL
the two most interesting stages
→ MOPD: the student generates, each prompt routes to its domain teacher for token-level feedback. teachers branch from the same SFT checkpoint, so their signal stays close to the student's distribution
→ agentic RL: multi-turn GRPO inside real harnesses (OpenClaw, Hermes Agent), one sandbox per rollout, a proxy captures token-level trajectories while the harness stays a black box
this makes a 2.6B that beats much larger models on instruction following and tool use
SFT, distillation, RL, RL envs: exactly what we're covering in our Training Agents livestream series (next one coming soon!)
→ model: LiquidAI/LFM2.5-2.6B
→ blog: https://www.liquid.ai/blog/lfm2-5-2-6b
→ live series: https://www.youtube.com/playlist?list=PLo2EIpI_JMQvQZm-kVlz4wY1vWF0LBcf5
and the @liquidai blog comes with some nice details about the training procedure, so let's analyze it.
basically, a full agent training pipeline but compressed into 2.6B
base model → SFT → specialized teachers per domain (SFT + RLVR) → on-policy distillation back into one student → agentic RL
the two most interesting stages
→ MOPD: the student generates, each prompt routes to its domain teacher for token-level feedback. teachers branch from the same SFT checkpoint, so their signal stays close to the student's distribution
→ agentic RL: multi-turn GRPO inside real harnesses (OpenClaw, Hermes Agent), one sandbox per rollout, a proxy captures token-level trajectories while the harness stays a black box
this makes a 2.6B that beats much larger models on instruction following and tool use
SFT, distillation, RL, RL envs: exactly what we're covering in our Training Agents livestream series (next one coming soon!)
→ model: LiquidAI/LFM2.5-2.6B
→ blog: https://www.liquid.ai/blog/lfm2-5-2-6b
→ live series: https://www.youtube.com/playlist?list=PLo2EIpI_JMQvQZm-kVlz4wY1vWF0LBcf5
- 2 replies
sergiopaniego
posted an update 6 days ago
Post
2569
Simon Willison (@simonw ) has asked every new model to draw a pelican riding a bicycle for some time now
you look at the drawing and you know. but there is no number, so nothing can train against it, no?
I turned this idea into an rl env in OpenEnv. now, you can eval any model against it, and train against it with TRL
read the details!🤓
https://huggingface.co/blog/sergiopaniego/pelican-env-openenv
you look at the drawing and you know. but there is no number, so nothing can train against it, no?
I turned this idea into an rl env in OpenEnv. now, you can eval any model against it, and train against it with TRL
read the details!🤓
https://huggingface.co/blog/sergiopaniego/pelican-env-openenv
- 2 replies
sergiopaniego
posted an update 7 days ago
Post
164
yesterday we had Class 3 of the Training Agents live series
we went through how GRPO works in depth and applied it to real experiments with TRL
sharing the resources in case you want to dig in, enjoy!
🎥 session recording: https://www.youtube.com/watch?v=ztdTed5egrM
📄 slides with links: https://docs.google.com/presentation/d/19v5_HR5B-1CPHuoZ-RXjhgBrGFXGnBNd6TfLhsXWT1c/edit?usp=sharing
we went through how GRPO works in depth and applied it to real experiments with TRL
sharing the resources in case you want to dig in, enjoy!
🎥 session recording: https://www.youtube.com/watch?v=ztdTed5egrM
📄 slides with links: https://docs.google.com/presentation/d/19v5_HR5B-1CPHuoZ-RXjhgBrGFXGnBNd6TfLhsXWT1c/edit?usp=sharing
sergiopaniego
posted an update 9 days ago
Post
2870
quick reminder! 🚨
tomorrow (Tuesday, July 28), we're back with Class 3 of the Training Agents live series
🧠 what: reinforcement learning for training agents (GRPO): how it works, how to implement it in TRL, and end-to-end examples
🗓️ when: Tuesday, July 28 - 🕔 5:00 PM CEST / 8:30 PM IST
📍 where: Live on @huggingface 's X, YouTube, and LinkedIn
live: https://www.youtube.com/watch?v=ztdTed5egrM
class 1: https://x.com/SergioPaniego/status/2069382207618379813
class 2: https://x.com/SergioPaniego/status/2075180665184686187
tomorrow (Tuesday, July 28), we're back with Class 3 of the Training Agents live series
🧠 what: reinforcement learning for training agents (GRPO): how it works, how to implement it in TRL, and end-to-end examples
🗓️ when: Tuesday, July 28 - 🕔 5:00 PM CEST / 8:30 PM IST
📍 where: Live on @huggingface 's X, YouTube, and LinkedIn
live: https://www.youtube.com/watch?v=ztdTed5egrM
class 1: https://x.com/SergioPaniego/status/2069382207618379813
class 2: https://x.com/SergioPaniego/status/2075180665184686187
- 1 reply
sergiopaniego
posted an update 12 days ago
Post
199
you can now train your own coding agents with trl + openenv, starting with opencode
we just added end-to-end support for training agent harnesses:
> TRL: a loop-owning training path (AsyncGRPOTrainer + HarnessRolloutWorker) that launches the agent in an OpenEnv session, reads back its trace, reconstructs the training samples, and trains with AsyncGRPO
> OpenEnv: the OpenCode harness environment plus a transparent proxy that forwards the agent's model calls and records each turn's token ids and logprobs
you train the actual opencode agent as is, it runs its own loop and tools and the policy learns from the exact tokens it produced
we're shipping a self-contained example: local subprocess sandbox, DeepCoder problems, validated on Qwen3-8B.
> example: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/opencode.py
> docs: https://huggingface.co/docs/trl/main/openenv
and we're working actively on both sides so expect more 🤓
we just added end-to-end support for training agent harnesses:
> TRL: a loop-owning training path (AsyncGRPOTrainer + HarnessRolloutWorker) that launches the agent in an OpenEnv session, reads back its trace, reconstructs the training samples, and trains with AsyncGRPO
> OpenEnv: the OpenCode harness environment plus a transparent proxy that forwards the agent's model calls and records each turn's token ids and logprobs
you train the actual opencode agent as is, it runs its own loop and tools and the policy learns from the exact tokens it produced
we're shipping a self-contained example: local subprocess sandbox, DeepCoder problems, validated on Qwen3-8B.
> example: https://github.com/huggingface/trl/blob/main/examples/scripts/openenv/opencode.py
> docs: https://huggingface.co/docs/trl/main/openenv
and we're working actively on both sides so expect more 🤓
- 1 reply
sergiopaniego
posted an update 13 days ago
Post
1511
you can train DiffusionGemma (a block-diffusion LLM) in TRL! and we're sharing an example for it
TRL trainers are made to be easily extended and adapted to different real-world use cases.
in this one, with a single method overridden in SFTTrainer (compute_loss), you can train this model
> example: https://github.com/huggingface/trl/blob/main/examples/scripts/sft_diffusion_gemma.py
TRL trainers are made to be easily extended and adapted to different real-world use cases.
in this one, with a single method overridden in SFTTrainer (compute_loss), you can train this model
> example: https://github.com/huggingface/trl/blob/main/examples/scripts/sft_diffusion_gemma.py
sergiopaniego
posted an update 14 days ago
Post
230
join us next Tuesday, July 28, for Class 3 of the Training Agents live series!
we'll dive into reinforcement learning for agent training, covering the intuition behind GRPO, how it works, and how to implement it in TRL with practical, e2e examples
see you there 🤠
live: https://www.youtube.com/live/ztdTed5egrM
> in case you missed class 1:
https://x.com/SergioPaniego/status/2069382207618379813
> and in case you missed class 2: https://x.com/SergioPaniego/status/2075180665184686187
we'll dive into reinforcement learning for agent training, covering the intuition behind GRPO, how it works, and how to implement it in TRL with practical, e2e examples
see you there 🤠
live: https://www.youtube.com/live/ztdTed5egrM
> in case you missed class 1:
https://x.com/SergioPaniego/status/2069382207618379813
> and in case you missed class 2: https://x.com/SergioPaniego/status/2075180665184686187
sayakpaul
authored a
paper 16 days ago
[Admin maintenance] Support new ZeroGPU hardware
🚀 1
#9 opened 22 days ago
by 
multimodalart
sayakpaul
authored 3
papers 26 days ago
sayakpaul
submitted a
paper to Daily Papers 26 days ago
sergiopaniego
posted an update 28 days ago
Post
7737
Frontier models use distillation as a step of their post-training pipelines.
In 2026 it has three jobs: compress a big model into a small one, merge RL experts into a single model, and let a model teach itself.
I wrote up which frontier models use each one and how: https://huggingface.co/blog/sergiopaniego/distillation-2026
It pairs with Class 2 of the Training an Agent series Ben and I are doing, where we teach these techniques hands-on with TRL!
In 2026 it has three jobs: compress a big model into a small one, merge RL experts into a single model, and let a model teach itself.
I wrote up which frontier models use each one and how: https://huggingface.co/blog/sergiopaniego/distillation-2026
It pairs with Class 2 of the Training an Agent series Ben and I are doing, where we teach these techniques hands-on with TRL!
- 3 replies
AdinaY
submitted a
paper to Daily Papers about 1 month ago
hysts
updated 4
Spaces about 1 month ago
Gemma 4 12B IT
⚡
138
Chat with an AI using text, images, audio, or video
Gemma 4 31B It
🚀
36
Chat with a multimodal AI using text, images, or video
Gemma 4 26B-A4B It
🚀
10
Chat with an AI that understands text, images, and video
Gemma 4 E4B It
🚀
30
Chat with a multimodal AI using text, images, audio, or video