Bocklitz-Lab/lit2vec-tldr-bart-model

lit2vec-tldr-bart

lit2vec-tldr-bart is sshleifer/distilbart-cnn-12-6 fine-tuned to write short TL;DR summaries of chemistry abstracts. It generates the TL;DR field of the Lit2Vec workflow (paper, code). A demo Space and the training code are available.

• Training data: Bocklitz-Lab/lit2vec-tldr-bart-dataset, 19,992 CC-BY chemistry abstracts with summaries written by GPT-4o
• Input: an abstract, up to 1,024 tokens
• Output: one or two sentences, with a target length of about 128 tokens at most

Because the training targets come from GPT-4o, the model learns to imitate GPT-4o's summaries. It does not summarise "faithfully" by construction.

Evaluation

Held-out test split (1,001 abstracts):

Split	ROUGE-1	ROUGE-2	ROUGE-Lsum
Test	56.11	30.78	45.43

Validation ROUGE-Lsum: 46.05. ROUGE was computed with evaluate's rouge (NLTK sentence splitting, use_stemmer=True). The comparison with other summarisation baselines is in the paper.

These scores include 4 test records that have an empty reference summary. On the other 997 records the scores are 56.34 (ROUGE-1), 30.90 (ROUGE-2) and 45.62 (ROUGE-Lsum). We regenerated all test summaries with the saved decoding defaults and reproduced the table above to within 0.02.

Quick start

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, GenerationConfig

repo = "Bocklitz-Lab/lit2vec-tldr-bart-model"

tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo)
gen = GenerationConfig.from_pretrained(repo)  # the decoding defaults saved with the model

text = "Proton exchange membrane fuel cells convert chemical energy into electricity..."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=1024)

summary_ids = model.generate(**inputs, **gen.to_dict())
print(tok.decode(summary_ids[0], skip_special_tokens=True))

Batch inference:

texts = ["Abstract 1 ...", "Abstract 2 ..."]
batch = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1024)
out = model.generate(**batch, **gen.to_dict())
summaries = tok.batch_decode(out, skip_special_tokens=True)

Default decoding

These defaults are saved in generation_config.json and can be overridden in generate():

{
  "max_length": 142,
  "min_length": 56,
  "early_stopping": true,
  "num_beams": 4,
  "length_penalty": 2.0,
  "no_repeat_ngram_size": 3,
  "forced_bos_token_id": 0,
  "forced_eos_token_id": 2
}

Training

Base model	sshleifer/distilbart-cnn-12-6
Data	19,992 CC-BY chemistry abstracts with GPT-4o summaries
Splits	train 17,992, validation 999, test 1,001
Max length	input 1,024 tokens, target 128 tokens
Optimiser	AdamW, learning rate 2e-5
Batch size	4 per device, 4 gradient-accumulation steps (effective 16)
Epochs	5
Precision	mixed precision (fp16) when CUDA is available
Hardware	one NVIDIA RTX 3090
Seed	42
Software	Transformers and Datasets, evaluate for ROUGE, NLTK for sentence splitting

Intended use

The model is meant for short summaries of chemistry abstracts and of closely related fields (materials science, chemical engineering, environmental science), for example for semantic indexing, retrieval and populating knowledge graphs.

Files

• model.safetensors and pytorch_model.bin: the weights in two formats
• onnx/model.onnx: an ONNX export
• config.json, generation_config.json
• tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, special_tokens_map.json
• training_args.bin: the Hugging Face trainer arguments used for training

Citation

Please cite the Lit2Vec paper:

@misc{amiri2026lit2vec,
  title         = {Lit2Vec: A Reproducible Workflow for Building a Legally Screened Chemistry Corpus from S2ORC for Downstream Retrieval and Text Mining},
  author        = {Amiri, Mahmoud and Jafari, Jamile Mohammad and Mostafapour, Sara and Bocklitz, Thomas},
  year          = {2026},
  eprint        = {2604.12498},
  archivePrefix = {arXiv},
  doi           = {10.48550/arXiv.2604.12498},
  url           = {https://arxiv.org/abs/2604.12498}
}

License

The model weights are released under Apache-2.0, the license of the base model. The training dataset is released under CC BY 4.0, and its summaries were generated with GPT-4o, so check OpenAI's terms of use if you use them to train other models.

Acknowledgements

The base model is DistilBART (sshleifer/distilbart-cnn-12-6). The abstracts come from S2ORC (Allen Institute for AI and Semantic Scholar). The work was developed in the Bocklitz Lab at the Leibniz Institute of Photonic Technology (Leibniz IPHT) and Friedrich Schiller University Jena and was funded by the BMFTR within the programme Photonics Research Germany (13N15466, 13N15710, 13N15715). Questions: mahmoud.amiri@uni-jena.de.