demirev/llm-abstract-detector

SPECTER2 detector of LLM-assisted scientific abstracts

This model scores how likely it is that a scientific abstract was written with the help of a large language model (LLM). It is a binary classifier fine-tuned from SPECTER2 on human-written journal abstracts and LLM-written abstracts with the same substantive content. It was built for the paper Is AI Reviving the Renaissance Man? LLM Adoption and Scientific Novelty in 22 Million Abstracts (Demirev, 2026), where it scores 21.9 million journal abstracts published between January 2018 and October 2025.

Usage

The model is a BERT encoder plus a linear layer on the [CLS] embedding, stored in head.safetensors. The score is the sigmoid of the linear layer's output.

import re
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModel, AutoTokenizer

repo = "demirev/llm-abstract-detector"
tok = AutoTokenizer.from_pretrained(repo)
enc = AutoModel.from_pretrained(repo).eval()
head = torch.nn.Linear(enc.config.hidden_size, 1)
head.load_state_dict(load_file(hf_hub_download(repo, "head.safetensors")))

def normalize(text):
    # the cleaning applied to every abstract in training and in the paper
    text = re.sub(r"<[^>]+>", "", text)
    text = re.sub(r"\*+([^*]+)\*+", r"\1", text)
    text = re.sub(r"_+([^_]+)_+", r"\1", text)
    text = re.sub(r"^#+\s*", "", text, flags=re.MULTILINE)
    text = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", text)
    text = re.sub(r"[\n\r\t]+", " ", text)
    text = re.sub(r" {2,}", " ", text)
    return text.strip()

abstracts = ["...", "..."]
x = tok([normalize(a) for a in abstracts], truncation=True, max_length=512,
        padding=True, return_tensors="pt")
with torch.no_grad():
    score = torch.sigmoid(head(enc(**x).last_hidden_state[:, 0])).squeeze(-1)
flagged = score > 0.99

Threshold

The paper flags an abstract when its score exceeds 0.99. At this threshold the held-out false positive rate is 3.0% and the false negative rate is 6.7%. At the default threshold of 0.5 the false positive rate is 12.1%, the false negative rate is 1.9% and the F1 score is 0.956.

Intended use

The detector is designed for measuring the share of LLM-assisted abstracts in large corpora, by field, journal tier or period, together with a correction for its known error rates (see the paper). A single score is not reliable evidence that a particular abstract was written with an LLM, and the model should not be used to judge individual authors or submissions.

Training data

Human-written abstracts. The 40,558 human abstracts are journal articles from the Dimensions database, published before 1 November 2022 in journals ranked by SCImago. The sample draws 1,000 abstracts at random from each of the 34 Dimensions unit-of-assessment fields and tops up each field with up to 100 abstracts from Q1 journals and up to 100 from a fixed list of top journals (listed in the paper's appendix).

LLM-written abstracts. Each human abstract is first distilled by GPT-5-mini into a bullet-point summary of its facts and claims. Generator models then write a journal-quality abstract from the summary alone, without seeing the original, so that each synthetic abstract matches its human counterpart in content and differs in style. Responses containing meta statements are removed by a regex filter. Four GPT-family generators are used in training (GPT-4o, GPT-4o-mini, GPT-5 and GPT-5-mini; 68,779 abstracts). Four further generators are used only for evaluation (Claude Haiku 4.5, Claude Sonnet 5, Gemini 2.5 Flash and Gemini 3 Flash).

The human and GPT-family abstracts are split 80/20 into a training set (87,469 abstracts) and a test set (21,868 abstracts), stratified by source.

Training procedure

All weights of allenai/specter2_base are fine-tuned for 3 epochs with learning rate 2e-5, batch size 16, inputs truncated to 512 tokens and seed 42. The loss is binary cross-entropy under online Group-DRO over five journal tiers (Q4, Q3, Q2, Q1 and top journals) with group step size 0.1, which upweights the worst-performing tier during training. By the end of training the top-journal tier carries a weight of 0.72.

Evaluation

Held-out test set; FPR and FNR at the 0.99 threshold.

	N	AUROC	FPR (%)	FNR (%)
All abstracts	21,868	0.991	3.0	6.7
Q4 journals	1,385	0.994	0.7	8.4
Q3 journals	1,861	0.995	1.6	4.5
Q2 journals	4,080	0.994	1.6	4.9
Q1 journals	11,155	0.995	3.2	3.7
Top journals	3,387	0.952	8.4	16.2

By generator, each compared with all human test abstracts (so the FPR is 3.0% throughout):

Generator	AUROC	FNR (%)
GPT-4o	0.993	6.0
GPT-4o-mini	0.992	6.0
GPT-5	0.982	14.1
GPT-5-mini	0.992	5.8
Claude Haiku 4.5 (not in training)	0.980	16.2
Claude Sonnet 5 (not in training)	0.962	25.1
Gemini 2.5 Flash (not in training)	0.977	17.5
Gemini 3 Flash (not in training)	0.979	15.7

As an external check, the detector flags 2.9% of the 12.5 million Dimensions abstracts published between January 2018 and October 2022, before the release of ChatGPT, against 2.5% expected from the test-set false positive rates of the journal tiers.

Limitations

• Error rates are higher for top journals, where the false positive and false negative rates are two to three times their overall values.
• Abstracts written with newer models, and with models outside the GPT family, are harder to detect.
• Light LLM editing of a human draft may not be flagged.
• The model sees only the text of the abstract. It cannot detect LLM use in coding, ideation, literature search or other parts of the research process.
• Training and evaluation use English journal abstracts only, and inputs beyond 512 tokens are truncated.

Citation

@techreport{demirev2026renaissance,
  author      = {Demirev, Georgi},
  title       = {Is {AI} Reviving the Renaissance Man? {LLM} Adoption and Scientific Novelty in 22 Million Abstracts},
  institution = {Northwestern University},
  type        = {Working Paper},
  year        = {2026}
}