File size: 2,232 Bytes
dafb15c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | ---
license: apache-2.0
library_name: transformers
tags:
- seq2seq
- t5
---
# StableT5
<div align="center">
<img src="figures/fig1.png" width="60%" alt="StableT5" />
</div>
<hr>
## 1. Introduction
StableT5 is a sequence-to-sequence model trained with a focus on training stability across random seeds. We evaluated multiple hyperparameter configurations and selected the run that achieves both high performance and low variance across seeds. The resulting model demonstrates that careful hyperparameter search can yield checkpoints that generalize reliably without seed-dependent flukes.
<p align="center">
<img width="80%" src="figures/fig2.png">
</p>
## 2. Evaluation Results
The champion run was selected from a pool of 6 experiments. Only runs whose per-seed relative standard deviation falls below 5% on all metrics were considered stable.
<div align="center">
| Benchmark | T5-Small | T5-Base | mT5-Small | StableT5 |
|---|---|---|---|---|
| ROUGE-L | 0.435 | 0.488 | 0.472 | 0.589 |
| BLEU | 0.410 | 0.465 | 0.449 | 0.563 |
| BERTScore F1 | 0.862 | 0.895 | 0.884 | 0.934 |
| sacreBLEU | 0.416 | 0.471 | 0.455 | 0.570 |
| METEOR | 0.465 | 0.520 | 0.503 | 0.625 |
</div>
## 3. Stability Analysis
<div align="center">
<img width="80%" src="figures/fig3.png">
</div>
All metrics for the champion run were verified across three random seeds (42, 123, 999). The maximum relative standard deviation observed was well below the 5% threshold, confirming that the model's performance is not an artifact of a favorable seed.
## 4. Usage
```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model = AutoModelForSeq2SeqLM.from_pretrained("your-username/StableT5-TestRepo")
tokenizer = AutoTokenizer.from_pretrained("your-username/StableT5-TestRepo")
input_text = "translate English to French: The weather is nice today."
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## 5. License
This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
## 6. Contact
For questions or issues, please open an issue on the associated repository. |