---
license: apache-2.0
language:
- en
base_model: bert-base-uncased
tags:
- encoder-decoder
- text2text-generation
- summarization
- dialogue-summarization
datasets:
- knkarthick/dialogsum
metrics:
- rouge
pipeline_tag: summarization
---

# bert2bert-dialogsum

This is a BERT2BERT encoder-decoder model, warm-started from two `bert-base-uncased` checkpoints and fine-tuned for **dialogue summarization** on the [DialogSum](https://huggingface.co/datasets/knkarthick/dialogsum) dataset.

## Model Details

### Model Description

The model takes a multi-turn dialogue as input and generates a short, single-sentence-style summary of the conversation. Both the encoder and decoder are initialized from `bert-base-uncased`; a randomly initialized cross-attention module was added to the decoder to let it attend over the encoder's output, following the standard [Leveraging Pre-trained Checkpoints](https://arxiv.org/abs/1907.12461) warm-start recipe for encoder-decoder models.

- **Developed by:** Cruciator
- **Model type:** Encoder-decoder (BERT2BERT) for text2text generation
- **Language:** English
- **License:** Apache 2.0 (inherited from the base `bert-base-uncased` checkpoints)
- **Finetuned from model:** [bert-base-uncased](https://huggingface.co/bert-base-uncased) (both encoder and decoder)

### Model Sources

- **Base model:** [bert-base-uncased](https://huggingface.co/bert-base-uncased)
- **Training data:** [knkarthick/dialogsum](https://huggingface.co/datasets/knkarthick/dialogsum)

## Uses

### Direct Use

Summarizing short, informal, multi-turn dialogues (e.g. chat transcripts, meeting snippets, customer support conversations) into a concise summary. Use the `text2text-generation` / `EncoderDecoderModel.generate()` API with an input dialogue truncated to 256 tokens.

### Out-of-Scope Use

- Long-form documents, articles, or single-speaker text (the model was trained exclusively on short multi-turn dialogues, capped at 256 input tokens)
- Non-English text
- Domains far outside DialogSum's everyday-conversation style (e.g. legal, medical, or highly technical dialogue) without further fine-tuning

## Bias, Risks, and Limitations

- Trained on a relatively small dataset (~12.5k training examples) for a small number of epochs, so summary quality and factual faithfulness are not guaranteed — always review generated summaries before relying on them.
- Inherits any biases present in `bert-base-uncased` and in the DialogSum dataset's crowd-sourced dialogues.
- Input dialogues are truncated at 256 tokens and output summaries are capped at 64 tokens, so longer conversations will be cut off rather than summarized in full.

## How to Get Started with the Model

```python
from transformers import EncoderDecoderModel, BertTokenizerFast

model = EncoderDecoderModel.from_pretrained("Cruciator/bert2bert-dialogsum")
tokenizer = BertTokenizerFast.from_pretrained("Cruciator/bert2bert-dialogsum")

dialogue = "#Person1#: Can we reschedule our meeting to Friday? #Person2#: Sure, works for me."
inputs = tokenizer(dialogue, return_tensors="pt", truncation=True, max_length=256)
summary_ids = model.generate(**inputs)
print(tokenizer.decode(summary_ids[0], skip_special_tokens=True))
```

## Training Details

### Training Data

[DialogSum](https://huggingface.co/datasets/knkarthick/dialogsum) — 12,460 training examples and 500 validation examples of short, multi-turn dialogues paired with human-written summaries.

### Training Procedure

- Inputs tokenized to a max length of 256 tokens; targets tokenized to a max length of 64 tokens
- Both encoder and decoder warm-started from `bert-base-uncased`; cross-attention weights randomly initialized
- Decoder inputs generated internally from labels (standard right-shifted teacher forcing)

#### Training Hyperparameters

- **Epochs:** 3
- **Per-device train/eval batch size:** 16
- **Mixed precision:** fp16
- **Generation (eval-time):** beam search, `num_beams=2`
- **Generation (inference defaults on this checkpoint):** beam search, `num_beams=4`, `length_penalty=2.0`, `no_repeat_ngram_size=3`, `early_stopping=True`
- **Hardware:** Google Colab, NVIDIA T4 GPU

### Evaluation

Evaluated on the DialogSum validation split (500 examples) at the end of each epoch using ROUGE-2 F-measure.

| Epoch | Training Loss | Validation Loss | ROUGE-2 F-measure |
|:-----:|:--------------:|:----------------:|:------------------:|
| 1     | 2.489          | 2.257            | 0.0612              |
| 2     | 2.072          | 2.091            | 0.0884              |
| 3     | 1.734          | 2.070            | 0.1069              |

ROUGE-2 was still improving at the final epoch, suggesting further training beyond 3 epochs would likely improve results further.

## Technical Specifications

- **Model size:** ~0.2B parameters
- **Tensor type:** F32
- **Framework:** 🤗 Transformers (v5)