| --- |
| library_name: transformers |
| tags: |
| - verl |
| - reinforcement-learning |
| - agent |
| - alfworld |
| --- |
| |
| # opd-rl checkpoints |
|
|
| Checkpoints are stored four levels deep: |
|
|
| ``` |
| <environment>/<model>/<training-method>/<checkpoint>/ |
| ``` |
|
|
| Each checkpoint directory holds the verl FSDP actor state: |
|
|
| ``` |
| <checkpoint>/actor/model_world_size_<N>_rank_<r>.pt sharded policy weights |
| <checkpoint>/actor/optim_world_size_<N>_rank_<r>.pt optimizer state (full uploads only) |
| <checkpoint>/actor/extra_state_world_size_<N>_rank_<r>.pt |
| <checkpoint>/actor/config.json, tokenizer files |
| <checkpoint>/data.pt dataloader state |
| <checkpoint>/hf/ merged HuggingFace weights (when present) |
| ``` |
|
|
| Merge the shards back into a loadable HuggingFace model with verl: |
|
|
| ```sh |
| python scripts/model_merger.py merge --backend fsdp \ |
| --local_dir <checkpoint>/actor --target_dir ./merged |
| ``` |
|
|
| ## Contents |
|
|
| | environment | model | training method | checkpoint | |
| |---|---|---|---| |
| | `alfworld` | `qwen2.5-3b` | `grpo` | `global_step_50` | |
| | `alfworld` | `qwen2.5-3b` | `rlsd_warmdown_100` | `global_step_150` | |
| | `alfworld` | `qwen2.5-3b` | `sdar_coef0.01_beta5.0_skillallfalse` | `global_step_150` | |
| | `alfworld` | `qwen3-1.7b` | `grpo_2gpu` | `global_step_100` | |
| | `alfworld` | `qwen3-1.7b` | `grpo_2gpu` | `global_step_150` | |
| | `alfworld` | `qwen3-1.7b` | `grpo_2gpu` | `global_step_50` | |
| | `alfworld` | `qwen3-1.7b` | `rlsd_warmdown_100` | `global_step_100` | |
| | `alfworld` | `qwen3-1.7b` | `rlsd_warmdown_100` | `global_step_150` | |
| | `alfworld` | `qwen3-1.7b` | `rlsd_warmdown_100` | `global_step_50` | |
| | `alfworld` | `qwen3-1.7b` | `sdar_coef0.01_beta5.0_skillallfalse` | `global_step_100` | |
| | `alfworld` | `qwen3-1.7b` | `sdar_coef0.01_beta5.0_skillallfalse` | `global_step_150` | |
| | `alfworld` | `qwen3-1.7b` | `sdar_coef0.01_beta5.0_skillallfalse` | `global_step_50` | |
|
|
| Uploaded with `scripts/upload_checkpoints_to_hf.py` (`--content full`). |
|
|
