| --- |
| language: en |
| base_model: google/flan-t5-base |
| library_name: peft |
| tags: |
| - base_model:adapter:google/flan-t5-base |
| - lora |
| - transformers |
| - summarization |
| - abstractive-summarization |
| - generated_from_trainer |
| model-index: |
| - name: Abstractive Style Summarizer |
| results: [] |
| datasets: |
| - xsum |
| - cnn_dailymail |
| - multi_news |
| license: mit |
| --- |
| |
| # Abstractive Style Summarizer |
|
|
| This model is a fine-tuned version of [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) using PEFT (LoRA). It is designed to generate abstractive summaries in three distinct styles: **Harsh** (concise), **Balanced** (standard), and **Detailed** (comprehensive). |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| - **Model type:** Sequence-to-Sequence Transformer (T5) |
| - **Language(s):** English |
| - **License:** MIT |
| - **Finetuned from model:** google/flan-t5-base |
| - **Training Method:** PEFT (LoRA) |
|
|
| ### Model Sources |
|
|
| - **Repository:** [Flatten](https://github.com/LityoPS/Flatten) |
| - **Base Model:** [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| The model interprets a prefixed prompt to determine the style of the summary. |
| - **Harsh**: Generates very short, punchy summaries (approx. 35% of input length). |
| - **Balanced**: Generates standard news summaries (approx. 50% of input length). |
| - **Detailed**: Generates in-depth summaries (approx. 70% of input length). |
|
|
| ### Prompt Format |
|
|
| The input text should be prefixed with the desired style: |
| ``` |
| Summarize {Style}: {Input Text} |
| ``` |
| Example: `Summarize Harsh: The Walt Disney Co. announced...` |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| The model was trained on a combined dataset of 12,000 samples, split into 80% Train, 10% Validation, and 10% Test. |
|
|
| | Style | Source Dataset | Size | |
| | :--- | :--- | :--- | |
| | **Harsh** | [XSum](https://huggingface.co/datasets/xsum) | 4000 | |
| | **Balanced** | [CNN/DailyMail](https://huggingface.co/datasets/cnn_dailymail) | 4000 | |
| | **Detailed** | [Multi-News](https://huggingface.co/datasets/multi_news) | 4000 | |
|
|
| ### Training Procedure |
|
|
| #### Training Hyperparameters |
|
|
| - **Learning Rate:** 5e-4 |
| - **Batch Size:** 4 per device |
| - **Gradient Accumulation Steps:** 2 |
| - **Num Epochs:** 5 |
| - **Optimizer:** AdamW |
| - **LR Scheduler:** Linear with warmup (ratio 0.05) |
| - **Mixed Precision:** BF16 |
|
|
| #### LoRA Configuration |
|
|
| - **r:** 32 |
| - **lora_alpha:** 64 |
| - **lora_dropout:** 0.05 |
| - **target_modules:** ["q", "k", "v", "o"] |
| - **bias:** "none" |
| - **task_type:** "SEQ_2_SEQ_LM" |
| |
| ### Evaluation Results |
| |
| Evaluated on the held-out test set (1,200 samples) at Step 6000. |
| |
| | Metric | Score | |
| | :--- | :--- | |
| | **ROUGE-1** | 0.3925 | |
| | **ROUGE-2** | 0.1608 | |
| | **ROUGE-L** | 0.2776 | |
| | **Validation Loss** | 0.7824 | |
| |
| ## Environmental Impact |
| |
| - **Hardware Type:** CUDA-enabled GPU |
| - **Compute:** LoRA fine-tuning (Parameters: 7M trainable / 254M total) |
| |
| ## Framework Versions |
| |
| - Datasets==3.6.0 |
| - Pytorch>=2.5.1 |
| - Transformers>=4.36.0 |
| - PEFT>=0.8.0 |
