File size: 5,947 Bytes
4742559
 
 
 
 
 
 
 
 
 
 
d7b7e82
 
 
 
 
 
 
 
 
 
4742559
 
d7b7e82
 
 
 
 
4d72966
 
 
 
 
e945472
 
4d72966
 
e945472
 
4d72966
 
 
d7b7e82
 
4d72966
d7b7e82
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4d72966
d7b7e82
 
 
 
 
 
 
 
 
 
 
 
4d72966
 
 
 
 
d7b7e82
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
---
title: Highlight Generator
emoji: 🐠
colorFrom: red
colorTo: gray
sdk: gradio
sdk_version: 6.28.0
python_version: '3.12'
app_file: app.py
pinned: false
license: mit
short_description: Extractive highlight generator for FIRE 2026 SciHigh
suggested_hardware: cpu-basic
models:
- sentence-transformers/all-MiniLM-L6-v2
tags:
- summarization
- extractive-summarization
- scientific-highlights
- fire-2026
- scikit-learn
---

# SciHigh 2026: Supervised Extractive Research-Highlight Generator

Submission to **FIRE 2026 SciHigh, Subtask 1** (research highlight generation from MixSub abstracts).
The system runs entirely on CPU. Open the **App** tab to try it on your own abstract or on a CSV file.

## Output modes (App tab)

| Mode | What it does | Can it invent facts? | Validation ROUGE-L | Avg. words |
|---|---|---|---|---|
| FIRE submission | Top-ranked whole sentences (the submitted system) | No | 24.76 | 64 |
| Concise | The same sentences, each compressed to about 18 words using word importance | No | 24.28 | 50 |
| LLM rewrite | `Qwen/Qwen2.5-1.5B-Instruct` rewrites the abstract into short highlights, guided by the ranked sentences; fact-checked | Checked | 20.20 on a 20-abstract sample* | about 50 |
| Fine-tuned model | BART fine-tuned on MixSub (`SciHigh_2026_Abstractive_FineTune.ipynb`, published as `Akash212/scihigh-bart-base`); fact-checked | Checked | see the notebook | about 50 |

*On the same 20 random validation abstracts, FIRE mode scored 21.83 and Concise 21.24 ROUGE-L.

Author highlights in MixSub average 52 words. The fact check rejects a generated highlight that has a number, name or acronym
absent from the abstract, or too many content words absent from it, and puts the Concise highlight in its place.

## Method

Sentence ranking is shared by all modes; the task is treated as sentence ranking, not text generation.

| Stage | What happens |
|---|---|
| 1. Pre-processing | Clean the abstract and split it into sentences (splitter written for MixSub's spaced periods) |
| 2. Hand-built features (31) | Position, length, lexical density, discourse cue counts, TF-IDF centrality, TextRank |
| 3. Embedding features (36) | `all-MiniLM-L6-v2`: similarity to the abstract centroid, to a highlight-style prototype, to other sentences, a retrieval feature from the 10 nearest training abstracts, and a 32-dim SVD projection |
| 4. Scoring | Five classifiers: LinearSVC, Ridge, LogReg, MLP, ExtraTrees |
| 5. Fusion | Reciprocal Rank Fusion of the five rankings within each abstract (k = 60) |
| 6. Decoding | Top sentences under a 110-word budget, at most 3 sentences, MMR redundancy threshold 0.85; original order restored; reporting frames ("In this study, we found that ...") removed |

Training labels come from a greedy ROUGE-1+2 F1 oracle over the 10,000 training abstracts (61,775 sentences).

## Results (MixSub-SciHigh validation set, 1,985 abstracts)

| System | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| This model (5-model fusion) | 38.12 | 14.13 | 24.76 |
| FIRE 2025 winner (Pegasus), reported | - | - | 23.45 |

Single members (decoded with budget 110, 3 sentences, MMR 0.55; fast stemmed ROUGE-L):

| Model | ROUGE-L |
|---|---|
| LinearSVC | 23.47 |
| Ridge | 23.47 |
| LogReg | 23.45 |
| MLP | 23.40 |
| ExtraTrees | 23.25 |

Every output word is copied from the source abstract, so the system cannot hallucinate facts or numbers.
The trade-off is that it cannot paraphrase the way authors often do in their own highlights.

## Use it from Python

```python
# pip install scikit-learn==1.8.0 "numpy>=2" pandas sentence-transformers huggingface_hub
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("Akash212/Highlight_Generator", repo_type="space")
sys.path.insert(0, path)
from scihigh_model import SciHighExtractor

model = SciHighExtractor.from_pretrained(path)
print(model.predict(["<paste an abstract here>"])[0])
print(model.predict(["<abstract>"], budget=90, maxs=4))        # optional decoder overrides
print(model.concise(["<abstract>"], target=15)[0])             # Concise mode: compressed highlights
```

## Files

| File | Content |
|---|---|
| `app.py` | Gradio test UI (single abstract, batch CSV, per-sentence score table) |
| `scihigh_model.py` | Pre-processing, features, decoder and the `SciHighExtractor` class |
| `config.json` | Ensemble members, decoder settings, feature order, validation scores, library versions |
| `clf_*.joblib` | The five fitted scikit-learn models |
| `tfidf.joblib`, `svd.joblib` | TF-IDF vectoriser and 32-dim SVD fitted on training sentences |
| `proto.npy`, `train_centroids*.npy`, `train_highlight_emb*.npy` | Embedding prototype and the retrieval index (training abstracts and their gold highlights) |
| `concise.py` | Sentence compression (reporting-frame removal, word-importance-guided phrase dropping) |
| `generators.py` | LLM rewriter, fine-tuned summariser loader and the fact check |
| `highlight_lexicon.json` | Highlight log-odds lexicon (18,646 words) learned from the training data, used as word importance |
| `train_export.py` | Re-trains the ranking model and writes these files |
| `SciHigh_2026_Abstractive_FineTune.ipynb` | Kaggle notebook that fine-tunes BART on MixSub and publishes it for the Fine-tuned mode |
| `SciHigh_2026_CPU_Hybrid.ipynb` | Full experiment notebook (16 models, ensemble selection, decoder tuning, error analysis) |

Files larger than 9.5 MB are stored as byte chunks (`name.part00.ext`, `name.part01.ext`, ...);
`scihigh_model.py` joins them automatically when loading.

## Reproducibility

Trained with Python 3.11.15, scikit-learn 1.8.0, numpy 2.4.4, pandas 3.0.2.
The `.joblib` files must be loaded with **scikit-learn 1.8.0**. They are pickle files, so Hugging Face
shows a security notice for them; load them only from a source you trust.

The training data (MixSub-SciHigh, provided by the FIRE 2026 organisers) is not included in this repository.