Seq vs Seq: An Open Suite of Paired Encoders and Decoders
Paper • 2507.11412 • Published • 32
You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
ParaTrace v8 (RAID-tuned) is a benchmark model for research on RAID. Outside RAID it misses much of the AI text and flags human writing, so it must not be used to decide whether a person used AI. It is licensed CC BY-NC-SA 4.0 (non-commercial).
Log in or Sign Up to review the conditions and access this model content.
ParaTrace v8 (RAID-tuned)
A 1B-parameter encoder that scores English text for being machine-generated. It was fine-tuned on RAID's training split for the RAID leaderboard.
This is a benchmark model: it does well on RAID and badly on ordinary writing (see Outside RAID). Do not use it to decide whether someone used AI.
It is not the detector behind paratrace.net, which is a separate, general-purpose model: ParaTrace v7.
| RAID leaderboard | 5th by TPR at 1% FPR, all domains and attacks (leaderboard, 12 November 2026) |
| RAID scores | TPR 99.20% at 1% FPR, 99.76% at 5% FPR; AUROC 99.24 |
| Base model | jhu-clsp/ettin-encoder-1b (ModernBERT architecture, 1.0B parameters) |
| Input | English text in 512-token windows; a longer text is scored as the mean of its overlapping windows |
| Output | P(machine): the expected share of the text written by AI |
| Weights | bfloat16: the checkpoint behind every number on this card, with an unused auxiliary head removed |
| License | CC BY-NC-SA 4.0, non-commercial (why) |
Use
pip install torch "transformers>=4.57" huggingface_hub
hf download empirischtech/paratrace-v8-raid detector.py --local-dir .
from detector import Detector
detector = Detector() # downloads the model; or Detector("path/to/a/local/copy")
results = detector.predict(["First text ...", "Second text ..."])
# one dict per text, e.g. {'probability': 0.99999976, 'margin': 15.23, 'n_windows': 1, 'n_words': 70}
detector.py scores a text the way the leaderboard file was scored:
- Prepare the text. It applies the cleaning used on the training data (straighten curly quotes, fold dashes, strip markdown markers, undo tokenizer spacing, cut a trailing unfinished sentence). Then it removes invisible characters, folds homoglyphs and collapses whitespace.
- Split it into windows. Each window is up to 512 tokens long, and consecutive windows share 128 tokens.
- Score each window. The model outputs 15 probabilities for the window's AI share: 0, 1/14, …, 1. The window's margin is the log-odds of the expected share.
- Combine. The text's margin is the mean over its windows; P(machine) is the sigmoid of that margin.
The model computes in float32 by default. Detector(dtype="bfloat16") is faster on GPUs that
support bfloat16, but its scores differ slightly.
The scores rank texts; they are not calibrated probabilities for your data. RAID's leaderboard sets its own threshold for each domain, at 1% false positives on unattacked human text.
Tested with transformers 4.57.3, 5.1, 5.14 and 5.17. On older versions, pass
attn_implementation="eager": 4.48 and 4.52 return NaN for ModernBERT's sdpa attention on
padded batches, and detector.py stops with an error.
Without detector.py. For a single window, without the cleaning and normalization:
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
repo = "empirischtech/paratrace-v8-raid"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).float().eval()
inputs = tokenizer("Text to score.", truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
buckets = model(**inputs).logits.softmax(-1)[0] # P(AI share = 0, 1/14, ..., 1)
p_machine = (buckets * torch.linspace(0, 1, 15)).sum().item()
Skipping the preparation changes the scores, most of all on RAID's attacked texts (homoglyphs, zero-width characters, whitespace).
Reproducing the leaderboard file:
curl -LO https://dataset.raid-bench.xyz/test.csv # RAID's test split, labels hidden (1.2 GB)
python detector.py --raid test.csv --out predictions.json # [{"id": ..., "score": ...}, ...]
The file holds sigmoid(margin / 10) for each text, not P(machine). The ranking is the same, but confident texts are not tied at 0.999999. On 2,020 sampled test texts, this repository reproduces the file submitted to the leaderboard to within 1e-6. The whole test set takes about 14 hours on a Quadro RTX 8000 in float32.
RAID
Results from RAID's leaderboard, which scores the hidden test split with one threshold per domain, set at 1% false positives on unattacked human text. The model ranks 5th by TPR at 1% FPR across all domains and attacks (12 November 2026).
| Test texts | TPR at 1% FPR | TPR at 5% FPR | AUROC |
|---|---|---|---|
| All | 99.20 | 99.76 | 99.24 |
| Without adversarial attacks | 99.43 | 99.80 | 99.28 |
TPR at 1% FPR by slice, each dimension with the others pooled, weakest first:
| Dimension | TPR at 1% FPR |
|---|---|
| Generator | Cohere-chat 96.5, Cohere 96.6, GPT-3 98.8, Mistral 99.0, MPT 99.3, MPT-chat 99.6, ChatGPT 99.8, GPT-2 99.8, GPT-4 99.8, Mistral-chat 99.8, Llama-chat 99.9 |
| Attack | paraphrase 98.0, synonym 98.7, article deletion 99.2, misspelling 99.3, upper/lower 99.3, none 99.4, alternative spelling 99.4, homoglyph 99.4, insert paragraphs 99.4, number 99.4, whitespace 99.4, zero-width space 99.4 |
| Domain | Reddit 97.9, books 98.7, reviews 98.7, Wikipedia 99.2, poetry 99.3, abstracts 99.8, recipes 99.9, news 100.0 |
| Decoding | sampling 98.8, greedy 99.5 |
| Repetition penalty | no 99.0, yes 99.6 |
Outside RAID: do not use it
RAID's leaderboard picks a threshold per domain from RAID's own human texts. Outside RAID, no threshold works well. We tested at P(machine) 0.965, the score at which the model flags 1% of the human texts in our development set.
- It misses AI text. It catches 58.7% of the AI texts in our held-out test set. It catches almost none of some kinds: 0.2% of AI texts written to match XSum news articles, 0.5% for CNN/DailyMail articles and 0.1% for Yelp reviews.
- It flags human text. It flags 2.4% of 66,586 human texts from 25 web sources it never saw, rising to 14% on a recipe site. It also flags 13 of 39 human persuasive essays.
- Chat answers are the exception. It still catches 99.2% of 26,086 chat-assistant answers from more than 100 models.
- Short texts are unreliable, below about 50 words.
Why it fails. Most of RAID's fine-tuning data is attacked AI text, so the model learned that typos, odd casing and missing articles mean AI. RAID sets its thresholds on unattacked human text only, so the leaderboard does not penalize this; real human writing does.
Training
General detector. Ettin was first trained on about 2.5 million English texts: 1.1 million human and 1.4 million AI or AI-edited.
- Human texts come from public corpora: Wikipedia, arXiv abstracts, peS2o, Reddit (SHP-2), Stack Exchange, WritingPrompts, Yelp reviews, news (XSum, CNN/DailyMail, CC-News, Common Pile), books (Project Gutenberg), patents, case law, parliamentary records, open textbooks, emails and student essays (PERSUADE 2.0, ELLIPSE, ASAP 2.0). They also include hard negatives: human texts that an earlier detector wrongly flagged.
- AI texts include AI rewrites of those same human texts on the same topics, written by many open-weight and commercial models. They also include chat-assistant answers (WildChat, LMArena), the AI texts of HC3, MAGE and M4GT, and AI text rewritten by "humanizer" models.
- AI-edited texts are LLM paraphrases of human text, labelled with their AI share.
- RAID's training split, except its poetry domain, GPT-4, Mistral-chat and the attacked texts.
The head classifies a window's AI share into 15 buckets. Half of the training texts longer than one window were cut at a random position.
RAID fine-tune. The model was then trained on 252,020 rows from the parts of RAID's training split that stage 1 had left out (poetry, GPT-4, Mistral-chat and all eleven attacks), keeping a fifth of the poetry documents for testing, plus a 300,000-row replay of stage-1 data. Settings: one epoch, learning rate 5e-6, batch 64, float32.
Hard examples. The stage-2 model scored 651,658 texts from RAID's training documents. Training then used the 50,000 AI texts it scored lowest, all 10,528 human texts and a 197,025-row replay. Settings: one epoch, learning rate 5e-6. The result is the released model.
RAID's test labels were never used. The test texts were scored only to write the leaderboard file.
Intended use
- In scope: research on the RAID benchmark and on what benchmark fine-tuning does to a detector. This model is the reproducible artifact behind ParaTrace's RAID leaderboard entry.
- Out of scope: decisions about people (education, hiring, publishing, moderation), non-English text and any production use.
License
CC BY-NC-SA 4.0. Some training corpora are licensed for non-commercial use only, among them PERSUADE 2.0 and ELLIPSE (CC BY-NC-SA 4.0) and the Yelp reviews (Yelp Dataset terms). The base model, Ettin, is released by its authors under the MIT License.
Citation
@misc{paratrace2026raid,
title = {ParaTrace v8 (RAID-tuned)},
author = {{ParaTrace}},
year = {2026},
howpublished = {\url{https://huggingface.co/empirischtech/paratrace-v8-raid}}
}
@inproceedings{dugan-etal-2024-raid,
title = {{RAID}: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors},
author = {Dugan, Liam and Hwang, Alyssa and Trhl{\'\i}k, Filip and Zhu, Andrew and Ludan, Josh Magnus and Xu, Hainiu and Ippolito, Daphne and Callison-Burch, Chris},
booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2024}
}
@misc{weller2025seqvsseqopen,
title = {Seq vs Seq: An Open Suite of Paired Encoders and Decoders},
author = {Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
year = {2025},
eprint = {2507.11412},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
Contact
- Downloads last month
- -
Model tree for empirischtech/paratrace-v8-raid
Base model
jhu-clsp/ettin-encoder-1b