# Submit your final project!

It's time to submit your DPO-aligned model! This unit uses the same leaderboard-based submission system as Unit 1. Here's the plan:

1. Read the written guide for the chapter ✅
2. Train a model using what you learned in the chapter.
3. Push the model to the Hugging Face Hub.
4. Evaluate the model using `hf jobs`.
5. Open a pull request on the leaderboard.

On this page we will go through each step.

## 1. Read the written guide for the chapter and 2. Train a model using what you learned in the chapter.

For Unit 3's submission, you should read all the materials in the unit and train a preference-aligned model using DPO. The training code is provided in:

- [DPO Training Exercise](./4) - Complete DPO training guide with SmolLM3

You'll need to combine this with [Training with Hugging Face Jobs](../unit1/5) techniques from Unit 1.

## 3. Push the model to the Hugging Face Hub

Once you've trained your DPO-aligned model, you'll need to push it to a repo on the Hugging Face Hub. TRL will take care of this for you if you add the `--push_to_hub` flag to your training command.

### DPO Training with Hub Upload:
```bash
hf jobs uv run \
    --flavor a100-large \
    --secrets HF_TOKEN \
    "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/dpo.py" \
    --model_name_or_path HuggingFaceTB/SmolLM3-3B \
    --dataset_name Anthropic/hh-rlhf \
    --learning_rate 5e-7 \
    --beta 0.1 \
    --max_steps 1000 \
    --push_to_hub \
    --hub_model_id your-username/smollm3-dpo-aligned \
    --report_to trackio
```

Your trained model will be available at `your-username/your-model-name`. For detailed documentation, check out the [checkpoints documentation](https://huggingface.co/docs/transformers/trainer#checkpoints) from `transformers`.

## 4. Evaluate the model using `hf jobs`

Now, we will evaluate your DPO-aligned model. We will use `hf jobs` to evaluate the model and combine it with `lighteval`. We will push the evaluation results to a dataset on the hub.

> [!TIP]
> For DPO evaluation, we use tasks that test both helpfulness and safety aspects of alignment, including `truthfulqa`, `gsm8k`, and other alignment-focused benchmarks.

```bash
hf jobs uv run \
    --flavor a10g-large \
    --with "lighteval[vllm]" \
    --secrets HF_TOKEN \
    lighteval vllm "model_name=/" \
    "lighteval|truthfulqa:mc2|0|0,lighteval|hellaswag|0|0,lighteval|arc:challenge|0|0" \
    --push-to-hub --results-org 
```

This command will evaluate the model using `lighteval` and `vllm` and save the results to the Hugging Face Hub in a dataset repo that you define.

> [!TIP]
> We focus on alignment evaluation in Unit 3, but in Unit 2 we explore evaluation in more detail. The key benchmarks for DPO evaluation include:
> - TruthfulQA: Tests for truthful and honest responses
> - GSM8K: Mathematical reasoning and helpfulness
> - MMLU: Broad knowledge and helpfulness across domains

## 5. Open a pull request on the leaderboard space

You are now ready to submit your DPO-aligned model to the leaderboard! You need to do two things:

1. Add your model's results to `submissions.json`
2. Share your training and evaluation commands in the PR text.

### Add your model's results to `submissions.json`

Open a pull request on the [leaderboard space](https://huggingface.co/spaces/smol-course/leaderboard/edit/main/submissions.json) to submit your model. You just need to add your model info and reference to the dataset you created in the previous step. We will pull the results and display them on the leaderboard.

```json
{
    "submissions": [

        ... // existing submissions
        
        {
            "username": "",
            "model_name": "", 
            "chapter": "3",
            "method": "DPO",
            "submission_date": "",
            "results-dataset": "",
            "base_model": "HuggingFaceTB/SmolLM3-3B",
            "preference_dataset": "Anthropic/hh-rlhf"
        }
    ]
}
```

### Share your training and evaluation commands in the PR text.

Within the PR text, share both your training and evaluation commands.

### Wait for the PR to be merged

Once the PR is merged, your DPO-aligned model will be added to the leaderboard! You can check the leaderboard [here](https://huggingface.co/spaces/smol-course/leaderboard).

## Test your knowledge

You’ve completed the unit — great work!
Now put your learning to the test by taking the [quiz](https://huggingface.co/spaces/smol-course/unit_3_quiz).

## Resources and Further Reading

- [Unit 3 DPO Exercise](./4) - Complete DPO training guide
- [DPO Paper](https://huggingface.co/papers/2305.18290) - Original research paper
- [TRL DPO Documentation](https://huggingface.co/docs/trl/dpo_trainer) - Implementation details
- [Anthropic HH-RLHF Paper](https://huggingface.co/papers/2204.05862) - Human feedback methodology
- [Alignment Handbook](https://github.com/huggingface/alignment-handbook) - Advanced alignment techniques

Good luck with your DPO preference alignment submission! 🚀