SciGenre

SciGenre classifies a scientific paper as a survey (a paper whose main contribution is an overview of existing work, including tutorials and overview articles) or as a research paper, from its title and abstract alone. It is the classifier of the paper The Survey Dividend: How Citation Metrics Reward Synthesis over Discovery (Nighot and Afraz).

Model

• Base model: allenai/specter2_base (Apache-2.0), fine-tuned with a classification head on the [CLS] token for one epoch (learning rate 2e-5, batch size 16, weight decay 0.01, 10% warm-up, class-weighted cross-entropy, inputs truncated at 384 tokens, seed 42), in full precision on one NVIDIA L40S (11 minutes).
• Labels: index 0 = survey, index 1 = non-survey. A paper counts as a survey when P(survey) >= 0.5.
• Input: the title followed by the abstract, with publisher boilerplate removed (model_text in predict.py). The model never sees the venue. Papers without an abstract of at least ten words should not be classified.

Training data

All papers of 18 survey venues (ACM Computing Surveys from 2000, IEEE Communications Surveys & Tutorials, Computer Science Review, the Annual Review of Control, Robotics, and Autonomous Systems, and the Foundations and Trends series) and 14 research venues (letters journals, research transactions, and journals of the ACM) in computer science-related fields, collected from OpenAlex. A venue qualifies by its officially stated scope: survey venues state that they publish surveys, reviews, or tutorials, and research venues call for original research and list no survey, review, or overview article type, with at most 0.3% of titles containing a survey term. Every label follows from the venue, and no paper was relabelled by hand. Of the 95,378 papers, 80% were used for training, 10% for validation, and 10% for testing, split at random within each venue. Abstracts came from OpenAlex (90.9%), Semantic Scholar (2.8%), and Scopus (6.3%). The training data are not redistributed. The code repository (https://github.com/nimaafraz/scigenre) rebuilds the ground truth from OpenAlex (python ground_truth.py venues, which lists the venues) and draws the split (python train.py final).

Performance

Held-out test papers (8,936 research papers and 619 surveys from all 32 venues):

	False-positive rate	Recall	Precision	Precision at 8% surveys	F1 at 8% surveys
SciGenre	0.15%	96.6%	97.9%	98.3%	97.4%

On venues absent from its training data (a leave-venue-out test with three folds, python train.py holdout in the code repository), the model has a recall of 95.7% and a false-positive rate of 0.70%. A few venue labels can be wrong, such as a survey that a research venue published despite its scope, so these error rates are upper bounds. Limitation: the research venues contain few papers that describe software, databases, or data resources, and the model labels some such papers as surveys.

On the same papers, database document types, keyword rules, and the rule of Smyth and Cunningham (2026) perform much worse. Classifiers trained on the same papers perform alike: SciBERT and BERT fine-tuned with the same settings, and a TF-IDF classifier with a linear SVM, do not differ significantly from SciGenre. Over five training seeds, SciGenre and SciBERT each have an average recall of 97.2%. Within these venues, the choice of supervised model matters little. On the test papers of venues absent from training, SciGenre has a recall of 94.7%, against 90.5% for a TF-IDF classifier trained without the same venues. An open large language model, gpt-oss-120b, asked the same question without training, has a recall of 92.7% and a false-positive rate of 0.34% (see the paper).

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from predict import model_text            # predict.py from this repository

repo = "nimaafraz/scigenre"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

def p_survey(titles, abstracts):
    texts = [model_text(t, a) for t, a in zip(titles, abstracts)]
    enc = tok(texts, truncation=True, max_length=384, padding=True, return_tensors="pt")
    with torch.no_grad():
        return torch.softmax(model(**enc).logits, -1)[:, 0].numpy()   # index 0 = survey

p = p_survey(["A survey of graph neural networks"], ["We review recent methods for learning on graphs ..."])
is_survey = p >= 0.5

predict.py in this repository wraps the same steps for a CSV file with title and abstract columns: python predict.py papers.csv out.csv --model nimaafraz/scigenre. scigenre.json records the training settings and the error rates on the held-out test papers, which the code repository reads.

Intended use and limitations

The model is intended for measuring the share of surveys in publication records and bibliographic datasets, for example in studies of research evaluation. It was trained and tested on computer science-related venues. Papers that fit neither class well, such as perspectives and position papers, tend to be assigned to the research class.

Attribution

• SPECTER2: Singh, A., D'Arcy, M., Cohan, A., Downey, D., and Feldman, S. (2023). SciRepEval: A multi-format benchmark for scientific document representations. Proceedings of EMNLP 2023, 5548-5566. Licensed under Apache-2.0.
• Paper metadata from OpenAlex (CC0).
• Some training abstracts were obtained from the Semantic Scholar Academic Graph (Kinney et al., 2023), made available under the ODC Attribution License.

Citation

Nighot, A., and Afraz, N. The Survey Dividend: How Citation Metrics Reward Synthesis over Discovery. Manuscript submitted to Scientometrics.