README.md

---
license: mit
language: en
base_model: microsoft/deberta-v3-large
tags:
  - ai-generated-text-detection
  - machine-generated-text
  - deberta
  - raid
  - adversarial-robustness
pipeline_tag: text-classification
---

# DeBERTa-ConPara

DeBERTa-ConPara detects machine-generated English text and stays accurate under the
adversarial edits that defeat most detectors: homoglyph substitution, zero-width
character insertion, whitespace and typographic attacks.

It is the system reported in our AACL-IJCNLP 2026 main-conference paper. The
finding behind it: normalising the **training** corpus deduplicates it (35.4% of
RAID rows collapse into byte-identical copies of their clean siblings, deleting
the adversarial supervision), while normalising at **inference** is an effective
defence. DeBERTa-ConPara trains on raw text and normalises only at inference.

- **Architecture:** DeBERTa-v3-large → CLS token → Linear(1024, 512) → GELU →
  Dropout(0.1) → Linear(512, 2). No feature branch.
- **Training data:** 1.55M documents, leakage-free stratified splits over RAID,
  HC3 Plus, MAGE and M4, grouped by source id.
- **Output:** logit margin, `logit[AI] − logit[human]`. Higher is more
  machine-like. Not a probability.

## Results

RAID hidden test (672,000 documents, 11 attacks): **AUROC 99.61**,
**TPR@5% FPR 99.01**, **TPR@1% FPR 96.57**.

Cross-dataset, balanced accuracy at one threshold calibrated on our source
validation split and held fixed: HC3-QA 99.69, HC3-SI 83.50, MAGE 96.23,
M4 98.27.

The README of the [GitHub repository](https://github.com/SES-Lab-OTH/deberta-conpara)
compares DeBERTa-ConPara with every RAID leaderboard system that publishes a checkpoint,
and states the two caveats those numbers need: MELD scores higher than DeBERTa-ConPara
on RAID itself, and HC3/MAGE/M4 are training sources for DeBERTa-ConPara while being
external data for the other systems.

## Usage

The checkpoint is a plain PyTorch state dict with a small custom head, so it does
not load through `AutoModelForSequenceClassification`. Use the loader from the
repository:

```python
from src.conpara import ConPara          # pip install -r requirements.txt

det = ConPara.from_pretrained()           # pulls rawguard.pt from this repo
det.score(["a document to check"])         # logit margin
det.predict(["a document to check"])       # bool at the stored threshold
```

Inference-time Unicode normalisation is applied by default; it is what makes the
detector robust to homoglyph and zero-width attacks.

## Intended use and limits

Intended as **supporting evidence for a human decision**: flagging text for
review, studying detector behaviour, benchmarking. Not intended as a verdict, and
not suitable for disciplinary or hiring decisions about individuals.

Measured limits:

- **Short text:** below ~60 words errors are enriched 4–5×. Treat under 75 words
  with caution; under 25 words the score is meaningless.
- **Academic prose:** false-positive rates between 13% and 67% depending on the
  subcorpus. A flag on a student essay is not evidence of misconduct.
- **Calibration:** the score is not a probability and saturates at the extremes.
  Prefer coarse bands (`det.band()`) over percentages.
- **Language:** English only.
- **Drift:** trained against generators available in 2026; newer ones are
  untested, and detectors decay as generators improve.
- **Attacks not covered:** heavy paraphrasing by a strong model, and mixed
  human/machine documents, remain hard.

## Training and evaluation protocol

Full corpus construction, the leakage-free splitting, the 2×2×2 factorial and the
fixed-threshold evaluation protocol are in the repository. Every number above is
reproducible from `evaluation/`.

## Citation

```bibtex
@inproceedings{mady2026conpara,
  title         = {{DeBERTa-ConPara}: Attack-Aware and Deployment-Realistic
                   Detection of {AI}-Generated Text},
  author        = {Mady, Mohamed and Li, Yupei and Reschke, Johannes and Schuller, Bj\"orn W.},
  booktitle     = {Proceedings of the 14th International Joint Conference on Natural
                   Language Processing and the 4th Conference of the Asia-Pacific Chapter
                   of the Association for Computational Linguistics (AACL-IJCNLP 2026)},
  year          = {2026},
  publisher     = {Association for Computational Linguistics},
  eprint        = {2610.00883},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.00883}
}
```

License: MIT, matching the DeBERTa-v3-large backbone.

## Links

- Code and evaluation scripts: https://github.com/SES-Lab-OTH/deberta-conpara
- Live demo: https://huggingface.co/spaces/mohamedmady/deberta-conpara
- Paper: "DeBERTa-ConPara: Attack-Aware and Deployment-Realistic Detection of AI-Generated Text", AACL-IJCNLP 2026, [arXiv:2610.00883](https://arxiv.org/abs/2610.00883)
- Lab: [Smart Embedded Systems Lab, OTH Regensburg](https://github.com/SES-Lab-OTH)