Add `{% generation %}` blocks to the chat template for training

#14
by bezzam HF Staff - opened

This PR prepares the chat template for training:

  1. It wraps the assistant turns in {% generation %} ... {% endgeneration %} blocks. These let apply_chat_template(..., return_assistant_tokens_mask=True) return a mask of the assistant tokens, so that only the assistant turn (the transcription) contributes to the loss, e.g. in Transformers or with assistant_only_loss=True in TRL.
  2. It closes the last assistant turn with <|user|>, one of the model's EOS tokens (eos_token_id in generation_config.json), inside the generation block. The template previously ended right after the assistant content, so the EOS was never part of the training labels and the model couldn't learn when to stop.

Inference is unaffected: prompts with add_generation_prompt=True and continue_final_message=True render exactly as before (checked on single- and multi-turn conversations). Only a conversation ending on an assistant turn, i.e. a training sample, now renders the trailing <|user|>.

Context: https://github.com/huggingface/transformers/pull/49870. Same generation-block convention as Qwen/Qwen3-ASR-1.7B-hf.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment