StopTheCap E2-C1 DistilBERT

A binary human vs machine text classifier built for the StopTheCap research project. It is published as a documented negative result and reproducibility artefact, not as a working detector.

What we did

StopTheCap studies whether AI-generated text can be detected reliably. The baseline, E1, was a DistilBERT trained on Ghostbuster:

Experiment In-domain HC3 RAID
E1 (Ghostbuster only) 0.999 0.817 0.532

E1 collapsed on RAID (chance level), so E2-C1 tested whether adding modern generator families would close the gap. No downloadable corpus had documented training rights for both classes, so the AI half was generated ourselves on Wikipedia prompts.

First attempt: short encyclopedic passages

  • 17,708 passages from DeepSeek V4.1 Flash, Muse Spark 1.3 and LongCat 2.5.
  • Human class: Ghostbuster essays split into short chunks.
  • Result: in-domain 1.000, HC3 0.367, RAID 0.375, human FPR 42-44%.
  • Diagnosis: the corpus was separable by word count alone at ~0.97 AUC. The detector learned a formatting shortcut, not authorship.

Second attempt: length-matched essays (this model)

  • Human class: 805 Ghostbuster student essays, unchanged.
  • AI class: 1,373 essays generated from the same essay prompts, length-matched to each human essay within +/-10% (DeepSeek 766, Muse Spark 607).
  • Total 2,178 rows; prompt-level splits train 1,754 / val 233 / test 191.
  • A pre-training shortcut gate still failed: paragraph density (AUC 0.901) and digit density (AUC 0.097) separated the classes. The gate result is recorded in the corpus manifest datasets/manifests/e2_c1_essays.json.

This model was trained on that corpus with E1's exact configuration.

Results

Split ROC-AUC PR-AUC Human FPR Recall ECE
val 1.000 1.000 6.0% 100% 0.001
test_holdout 1.000 1.000 8.7% 100% 0.001
HC3 0.447 0.303 88.3% 96.2% 0.285
RAID 0.377 0.890 88.2% 89.1% 0.831

Frozen threshold: 0.0007257819990627467 (5% human-FPR budget on val).

What succeeded

  • The full pipeline is reproducible: prompt inventory, prompt-level splits, generation provenance, merge, decontamination, training and evaluation.
  • Generator attribution works. A separate 3-class DistilBERT distinguishes DeepSeek / Muse Spark / LongCat with 95.1% accuracy and 0.84 macro-F1 on held-out AI text.
  • In-domain classification is perfect - which is precisely the problem.

What failed

  • Cross-dataset generalisation. HC3 and RAID are at or below chance. RAID 0.377 is worse than a coin flip.
  • The operating point. At the frozen threshold, ~88% of real human text from HC3/RAID is flagged as machine. That is actively harmful, not merely weak.
  • The hypothesis. Training on API-generated text against Ghostbuster humans does not teach a transferable authorship signal. Two corpus designs failed the same way.
  • The shortcut gate. Matching length removed the word-count tell, but paragraph density and digit density remained.

Data

Class Source Rows
Human Ghostbuster student essays 805
Machine Generated essays, DeepSeek V4.1 Flash 766
Machine Generated essays, Muse Spark 1.3 607

Evaluation used HC3 (8,000 rows) and RAID (2,000 rows), both independent and never trained on.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("itsaakif/StopTheCap-E2C1-distilbert")
model = AutoModelForSequenceClassification.from_pretrained("itsaakif/StopTheCap-E2C1-distilbert")
model.eval()

text = "Your text here."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    prob = torch.softmax(model(**inputs).logits, dim=-1)[0, 1].item()
print("machine probability:", prob, "| threshold:", 0.0007257819990627467)

Label 0 = human, label 1 = machine.

Intended use

Research reproduction and citation of a negative result. Not for content moderation, academic integrity decisions, or any use where a false accusation has consequences.

Limitations

  • Human FPR ~88% on HC3/RAID.
  • Shortcut features (paragraph and digit density) are documented in the corpus manifest.
  • Threshold is near zero and not calibrated outside the training distribution.
  • Trained on student essays; does not represent books, news, poetry or forum text.
  • The AI class is API-generated and does not cover all generator families.

Reproduce

The canonical corpus is not distributed. The builder regenerates it from the prompt manifest, the Ghostbuster records and the API generation shards.

python -m ml.e2c1.build_essay_corpus
python -m ml.train.train_e2c1 --corpus datasets/canonical/e2c1_essays.jsonl --manifest datasets/manifests/e2_c1_essays.json --device cuda --out reports/E2C1_essays
python -m ml.train.export_checkpoint --checkpoint reports/E2C1_essays/checkpoints/last.pt --out reports/E2C1_essays/export

Training: distilbert-base-cased, 512 tokens, 3 epochs, lr 5e-5, batch 16 x grad-accum 2, fp16.

Citation

If you use this artefact, cite it as a negative result:

StopTheCap (2026). E2-C1: length-matched API-generated essays do not transfer to HC3 or RAID. Hugging Face model itsaakif/StopTheCap-E2C1-distilbert.

Downloads last month
36
Safetensors
Model size
65.8M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for itsaakif/StopTheCap-E2C1-distilbert

Finetuned
(346)
this model