license: cc-by-nc-sa-4.0
base_model: sentence-transformers/all-MiniLM-L6-v2
language: en
tags:
- onnx
- routing
- summarization
datasets:
- abisee/cnn_dailymail
- EdinburghNLP/xsum
- knkarthick/dialogsum
- FiscalNote/billsum
- trl-lib/tldr
Gain predictor
Predicts from the source text alone how much a heavy LLM improves a summary over a light
one, BERTScore(heavy) - BERTScore(light) for Qwen3-4B against Qwen3-0.6B, before either
model runs. A router uses it to decide when the heavy model is worth its extra energy.
Non-commercial use only (CC BY-NC-SA 4.0), because the DialogSum training data is.
Outputs
model.onnx takes the tokens of one text from tokenizer.json (cut at 256 tokens):
| Output | Meaning |
|---|---|
gain |
Expected BERTScore gain of the heavy model |
gain_low, gain_high |
90% interval of the real gain |
real_gain_probability |
Chance that the gain is above the noise threshold (0.055) |
familiarity |
Cosine similarity to the closest training text |
The graph holds MiniLM, a ridge regression for the gain, a second one for the size of its
error, and normalised conformal prediction for the interval and probability. Do not trust
the gain when familiarity is below the familiarity_threshold in the metadata, or when
the text has more than max_words words.
Training and results
1,100 English texts: 500 CNN/DailyMail, and 150 each from XSum, DialogSum, BillSum and Reddit TL;DR. Each label averages 5 sampled summaries per model. Evaluated with 5-fold cross-validation repeated 10 times:
| Model | RMSE | Skill over the mean [95% CI] |
|---|---|---|
| Always the average gain | 0.058 | 0 |
| Average gain per dataset | 0.058 | 0.024 [-0.003, 0.051] |
| This model | 0.057 | 0.036 [0.018, 0.054] |
The 90% interval held 90% of real gains (87% to 93% per dataset).
Limitations
- The gain is hard to predict: the model beats the average only slightly, and mostly learns the kind of text.
- On a kind of text it never saw it has no skill, so check
familiarity. - English only, and MiniLM reads the first 256 tokens.