Instructions to use itsaakif/StopTheCap-E2C1-distilbert with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use itsaakif/StopTheCap-E2C1-distilbert with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="itsaakif/StopTheCap-E2C1-distilbert")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("itsaakif/StopTheCap-E2C1-distilbert") model = AutoModelForSequenceClassification.from_pretrained("itsaakif/StopTheCap-E2C1-distilbert", device_map="auto") - Notebooks
- Google Colab
- Kaggle
StopTheCap E2-C1 DistilBERT
A binary human vs machine text classifier built for the StopTheCap research project. It is published as a documented negative result and reproducibility artefact, not as a working detector.
What we did
StopTheCap studies whether AI-generated text can be detected reliably. The baseline, E1, was a DistilBERT trained on Ghostbuster:
| Experiment | In-domain | HC3 | RAID |
|---|---|---|---|
| E1 (Ghostbuster only) | 0.999 | 0.817 | 0.532 |
E1 collapsed on RAID (chance level), so E2-C1 tested whether adding modern generator families would close the gap. No downloadable corpus had documented training rights for both classes, so the AI half was generated ourselves on Wikipedia prompts.
First attempt: short encyclopedic passages
- 17,708 passages from DeepSeek V4.1 Flash, Muse Spark 1.3 and LongCat 2.5.
- Human class: Ghostbuster essays split into short chunks.
- Result: in-domain 1.000, HC3 0.367, RAID 0.375, human FPR 42-44%.
- Diagnosis: the corpus was separable by word count alone at ~0.97 AUC. The detector learned a formatting shortcut, not authorship.
Second attempt: length-matched essays (this model)
- Human class: 805 Ghostbuster student essays, unchanged.
- AI class: 1,373 essays generated from the same essay prompts, length-matched to each human essay within +/-10% (DeepSeek 766, Muse Spark 607).
- Total 2,178 rows; prompt-level splits train 1,754 / val 233 / test 191.
- A pre-training shortcut gate still failed: paragraph density (AUC 0.901) and digit density (AUC 0.097) separated the classes. The gate result is recorded in the corpus manifest datasets/manifests/e2_c1_essays.json.
This model was trained on that corpus with E1's exact configuration.
Results
| Split | ROC-AUC | PR-AUC | Human FPR | Recall | ECE |
|---|---|---|---|---|---|
| val | 1.000 | 1.000 | 6.0% | 100% | 0.001 |
| test_holdout | 1.000 | 1.000 | 8.7% | 100% | 0.001 |
| HC3 | 0.447 | 0.303 | 88.3% | 96.2% | 0.285 |
| RAID | 0.377 | 0.890 | 88.2% | 89.1% | 0.831 |
Frozen threshold: 0.0007257819990627467 (5% human-FPR budget on val).
What succeeded
- The full pipeline is reproducible: prompt inventory, prompt-level splits, generation provenance, merge, decontamination, training and evaluation.
- Generator attribution works. A separate 3-class DistilBERT distinguishes DeepSeek / Muse Spark / LongCat with 95.1% accuracy and 0.84 macro-F1 on held-out AI text.
- In-domain classification is perfect - which is precisely the problem.
What failed
- Cross-dataset generalisation. HC3 and RAID are at or below chance. RAID 0.377 is worse than a coin flip.
- The operating point. At the frozen threshold, ~88% of real human text from HC3/RAID is flagged as machine. That is actively harmful, not merely weak.
- The hypothesis. Training on API-generated text against Ghostbuster humans does not teach a transferable authorship signal. Two corpus designs failed the same way.
- The shortcut gate. Matching length removed the word-count tell, but paragraph density and digit density remained.
Data
| Class | Source | Rows |
|---|---|---|
| Human | Ghostbuster student essays | 805 |
| Machine | Generated essays, DeepSeek V4.1 Flash | 766 |
| Machine | Generated essays, Muse Spark 1.3 | 607 |
Evaluation used HC3 (8,000 rows) and RAID (2,000 rows), both independent and never trained on.
How to use
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tok = AutoTokenizer.from_pretrained("itsaakif/StopTheCap-E2C1-distilbert")
model = AutoModelForSequenceClassification.from_pretrained("itsaakif/StopTheCap-E2C1-distilbert")
model.eval()
text = "Your text here."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
prob = torch.softmax(model(**inputs).logits, dim=-1)[0, 1].item()
print("machine probability:", prob, "| threshold:", 0.0007257819990627467)
Label 0 = human, label 1 = machine.
Intended use
Research reproduction and citation of a negative result. Not for content moderation, academic integrity decisions, or any use where a false accusation has consequences.
Limitations
- Human FPR ~88% on HC3/RAID.
- Shortcut features (paragraph and digit density) are documented in the corpus manifest.
- Threshold is near zero and not calibrated outside the training distribution.
- Trained on student essays; does not represent books, news, poetry or forum text.
- The AI class is API-generated and does not cover all generator families.
Reproduce
The canonical corpus is not distributed. The builder regenerates it from the prompt manifest, the Ghostbuster records and the API generation shards.
python -m ml.e2c1.build_essay_corpus
python -m ml.train.train_e2c1 --corpus datasets/canonical/e2c1_essays.jsonl --manifest datasets/manifests/e2_c1_essays.json --device cuda --out reports/E2C1_essays
python -m ml.train.export_checkpoint --checkpoint reports/E2C1_essays/checkpoints/last.pt --out reports/E2C1_essays/export
Training: distilbert-base-cased, 512 tokens, 3 epochs, lr 5e-5, batch 16 x grad-accum 2, fp16.
Citation
If you use this artefact, cite it as a negative result:
StopTheCap (2026). E2-C1: length-matched API-generated essays do not transfer to HC3 or RAID. Hugging Face model itsaakif/StopTheCap-E2C1-distilbert.
- Downloads last month
- 36
Model tree for itsaakif/StopTheCap-E2C1-distilbert
Base model
distilbert/distilbert-base-cased