| --- |
| license: cc-by-nc-sa-4.0 |
| base_model: sentence-transformers/all-MiniLM-L6-v2 |
| language: en |
| tags: |
| - onnx |
| - routing |
| - summarization |
| datasets: |
| - abisee/cnn_dailymail |
| - EdinburghNLP/xsum |
| - knkarthick/dialogsum |
| - FiscalNote/billsum |
| - trl-lib/tldr |
| --- |
| |
| # Gain predictor |
|
|
| Predicts from the source text alone how much a heavy LLM improves a summary over a light |
| one, `BERTScore(heavy) - BERTScore(light)` for Qwen3-4B against Qwen3-0.6B, before either |
| model runs. A router uses it to decide when the heavy model is worth its extra energy. |
|
|
| Non-commercial use only (CC BY-NC-SA 4.0), because the DialogSum training data is. |
|
|
| ## Outputs |
|
|
| `model.onnx` takes the tokens of one text from `tokenizer.json` (cut at 256 tokens): |
|
|
| | Output | Meaning | |
| |---|---| |
| | `gain` | Expected BERTScore gain of the heavy model | |
| | `gain_low`, `gain_high` | 90% interval of the real gain | |
| | `real_gain_probability` | Chance that the gain is above the noise threshold (0.055) | |
| | `familiarity` | Cosine similarity to the closest training text | |
|
|
| The graph holds MiniLM, a ridge regression for the gain, a second one for the size of its |
| error, and normalised conformal prediction for the interval and probability. Do not trust |
| the gain when `familiarity` is below the `familiarity_threshold` in the metadata, or when |
| the text has more than `max_words` words. |
|
|
| ## Training and results |
|
|
| 1,100 English texts: 500 CNN/DailyMail, and 150 each from XSum, DialogSum, BillSum and |
| Reddit TL;DR. Each label averages 5 sampled summaries per model. Evaluated with 5-fold |
| cross-validation repeated 10 times: |
|
|
| | Model | RMSE | Skill over the mean [95% CI] | |
| |---|---:|---:| |
| | Always the average gain | 0.058 | 0 | |
| | Average gain per dataset | 0.058 | 0.024 [-0.003, 0.051] | |
| | This model | 0.057 | 0.036 [0.018, 0.054] | |
|
|
| The 90% interval held 90% of real gains (87% to 93% per dataset). |
|
|
| ## Limitations |
|
|
| - The gain is hard to predict: the model beats the average only slightly, and mostly learns |
| the kind of text. |
| - On a kind of text it never saw it has no skill, so check `familiarity`. |
| - English only, and MiniLM reads the first 256 tokens. |
|
|
