| --- |
| license: apache-2.0 |
| language: |
| - en |
| base_model: bert-base-uncased |
| tags: |
| - encoder-decoder |
| - text2text-generation |
| - summarization |
| - dialogue-summarization |
| datasets: |
| - knkarthick/dialogsum |
| metrics: |
| - rouge |
| pipeline_tag: summarization |
| --- |
| |
| # bert2bert-dialogsum |
|
|
| This is a BERT2BERT encoder-decoder model, warm-started from two `bert-base-uncased` checkpoints and fine-tuned for **dialogue summarization** on the [DialogSum](https://huggingface.co/datasets/knkarthick/dialogsum) dataset. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| The model takes a multi-turn dialogue as input and generates a short, single-sentence-style summary of the conversation. Both the encoder and decoder are initialized from `bert-base-uncased`; a randomly initialized cross-attention module was added to the decoder to let it attend over the encoder's output, following the standard [Leveraging Pre-trained Checkpoints](https://arxiv.org/abs/1907.12461) warm-start recipe for encoder-decoder models. |
|
|
| - **Developed by:** Cruciator |
| - **Model type:** Encoder-decoder (BERT2BERT) for text2text generation |
| - **Language:** English |
| - **License:** Apache 2.0 (inherited from the base `bert-base-uncased` checkpoints) |
| - **Finetuned from model:** [bert-base-uncased](https://huggingface.co/bert-base-uncased) (both encoder and decoder) |
|
|
| ### Model Sources |
|
|
| - **Base model:** [bert-base-uncased](https://huggingface.co/bert-base-uncased) |
| - **Training data:** [knkarthick/dialogsum](https://huggingface.co/datasets/knkarthick/dialogsum) |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| Summarizing short, informal, multi-turn dialogues (e.g. chat transcripts, meeting snippets, customer support conversations) into a concise summary. Use the `text2text-generation` / `EncoderDecoderModel.generate()` API with an input dialogue truncated to 256 tokens. |
|
|
| ### Out-of-Scope Use |
|
|
| - Long-form documents, articles, or single-speaker text (the model was trained exclusively on short multi-turn dialogues, capped at 256 input tokens) |
| - Non-English text |
| - Domains far outside DialogSum's everyday-conversation style (e.g. legal, medical, or highly technical dialogue) without further fine-tuning |
|
|
| ## Bias, Risks, and Limitations |
|
|
| - Trained on a relatively small dataset (~12.5k training examples) for a small number of epochs, so summary quality and factual faithfulness are not guaranteed — always review generated summaries before relying on them. |
| - Inherits any biases present in `bert-base-uncased` and in the DialogSum dataset's crowd-sourced dialogues. |
| - Input dialogues are truncated at 256 tokens and output summaries are capped at 64 tokens, so longer conversations will be cut off rather than summarized in full. |
|
|
| ## How to Get Started with the Model |
|
|
| ```python |
| from transformers import EncoderDecoderModel, BertTokenizerFast |
| |
| model = EncoderDecoderModel.from_pretrained("Cruciator/bert2bert-dialogsum") |
| tokenizer = BertTokenizerFast.from_pretrained("Cruciator/bert2bert-dialogsum") |
| |
| dialogue = "#Person1#: Can we reschedule our meeting to Friday? #Person2#: Sure, works for me." |
| inputs = tokenizer(dialogue, return_tensors="pt", truncation=True, max_length=256) |
| summary_ids = model.generate(**inputs) |
| print(tokenizer.decode(summary_ids[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| [DialogSum](https://huggingface.co/datasets/knkarthick/dialogsum) — 12,460 training examples and 500 validation examples of short, multi-turn dialogues paired with human-written summaries. |
|
|
| ### Training Procedure |
|
|
| - Inputs tokenized to a max length of 256 tokens; targets tokenized to a max length of 64 tokens |
| - Both encoder and decoder warm-started from `bert-base-uncased`; cross-attention weights randomly initialized |
| - Decoder inputs generated internally from labels (standard right-shifted teacher forcing) |
|
|
| #### Training Hyperparameters |
|
|
| - **Epochs:** 3 |
| - **Per-device train/eval batch size:** 16 |
| - **Mixed precision:** fp16 |
| - **Generation (eval-time):** beam search, `num_beams=2` |
| - **Generation (inference defaults on this checkpoint):** beam search, `num_beams=4`, `length_penalty=2.0`, `no_repeat_ngram_size=3`, `early_stopping=True` |
| - **Hardware:** Google Colab, NVIDIA T4 GPU |
|
|
| ### Evaluation |
|
|
| Evaluated on the DialogSum validation split (500 examples) at the end of each epoch using ROUGE-2 F-measure. |
|
|
| | Epoch | Training Loss | Validation Loss | ROUGE-2 F-measure | |
| |:-----:|:--------------:|:----------------:|:------------------:| |
| | 1 | 2.489 | 2.257 | 0.0612 | |
| | 2 | 2.072 | 2.091 | 0.0884 | |
| | 3 | 1.734 | 2.070 | 0.1069 | |
|
|
| ROUGE-2 was still improving at the final epoch, suggesting further training beyond 3 epochs would likely improve results further. |
|
|
| ## Technical Specifications |
|
|
| - **Model size:** ~0.2B parameters |
| - **Tensor type:** F32 |
| - **Framework:** 🤗 Transformers (v5) |
