Driw0x/my_awesome_billsum_model

T5 Small fine-tuned on BillSum for Abstractive Summarization

Model Description

This model is a fine-tuned version of google-t5/t5-small for abstractive summarization of legislative text.

It was fine-tuned on the California state bill subset of the BillSum dataset and evaluated with ROUGE.

The model was created as a practical exercise while following the Hugging Face LLM Course.

• Developed by: Driw0x
• Model type: T5 encoder-decoder Transformer
• Language: English
• Base model: google-t5/t5-small
• Task: Abstractive summarization / sequence-to-sequence generation

Training and Evaluation Data

The model was fine-tuned on:

FiscalNote/billsum

The notebook loads the California state bill subset:

billsum = load_dataset("FiscalNote/billsum", split="ca_test")
billsum = billsum.train_test_split(test_size=0.2)

This creates an 80/20 train/evaluation split from ca_test.

Because this generated evaluation split is used during model development, it should not be considered a completely untouched final benchmark.

Preprocessing

T5 inputs are prefixed with:

summarize: <document>

The preprocessing configuration is:

• maximum input length: 1024 tokens;
• maximum target summary length: 128 tokens;
• truncation enabled;
• sequence-to-sequence padding with DataCollatorForSeq2Seq.

Training Procedure

The model was fine-tuned with Seq2SeqTrainer.

Training Hyperparameters

Hyperparameter	Value
Learning rate	2e-5
Train batch size	16
Evaluation batch size	16
Number of epochs	4
Weight decay	0.01
Seed	42
Evaluation strategy	epoch
LR scheduler	linear
Mixed precision	FP16 / Native AMP
Predict with generate	True
Save total limit	3
Optimizer	AdamW (ADAMW_TORCH_FUSED)

Evaluation uses ROUGE with stemming enabled.

Training Results

Epoch	Validation Loss	ROUGE-1	ROUGE-2	ROUGE-L	ROUGE-Lsum	Generated Length
1	2.6039	0.1365	0.0440	0.1119	0.1117	19.0
2	2.5241	0.1466	0.0524	0.1196	0.1196	19.0
3	2.4884	0.1535	0.0597	0.1271	0.1271	19.0
4	2.4781	0.1558	0.0618	0.1288	0.1288	19.0

Final reported evaluation results:

• Validation loss: 2.4781
• ROUGE-1: 0.1558
• ROUGE-2: 0.0618
• ROUGE-L: 0.1288
• ROUGE-Lsum: 0.1288
• Mean generated length: 19.0

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "Driw0x/my_awesome_billsum_model"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = """summarize: The Inflation Reduction Act lowers prescription drug costs,
health care costs, and energy costs. It also includes major climate and energy
investments and measures intended to reduce the federal deficit."""

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Intended Uses

This model is primarily intended for:

• learning abstractive summarization with Transformers;
• experimenting with T5 fine-tuning;
• reproducing a Hugging Face summarization workflow;
• educational experiments on legislative text.

Limitations

Important limitations include:

• training used only the California state bill subset selected by the course workflow;
• the model is based on the relatively small t5-small checkpoint;
• the reported ROUGE scores remain limited;
• generated summaries may omit important details or introduce unsupported information;
• performance may degrade on documents that differ strongly from legislative text;
• biases and errors inherited from the base model and training data may remain.

This model was created as a course exercise and has not been validated for production, legal analysis, or other high-stakes use.

Framework Versions

• Transformers 5.17.0
• PyTorch 2.11.0+cu130
• Datasets 4.8.5
• Tokenizers 0.23.2

Training Source

The complete training procedure is available in:

Driw0x/hf-ai-courses

Notebook:

llm-course/1-transformer-models/notebooks/summarization.ipynb