---
license: apache-2.0
base_model: answerdotai/ModernBERT-large
pipeline_tag: text-classification
library_name: transformers
tags:
  - code
  - ai-detection
---

<img src="https://raw.githubusercontent.com/gist/tiagozip/a1e8d3c5fce271fefe15ecec70318c58/raw/44d1a919bd095e7aac3e1eb01ab6d1a2d0b9581d/pancode.svg" alt="banner">

# pancode

pancode is an AI detector for code fine-tuned from ModernBERT-large to detect whether a code file was written by AI, and if so by which model.

**try it in the [pancode space](https://huggingface.co/spaces/tiagozip/pancode).**

## training

pancode was trained from hundreds of millions of lines of code and hundreds of thousands of files from GitHub, GitLab, Codeberg, synthetic data, and more, with a mix of human-written and AI-written code, with formatting and other preprocessing.

positives are 2026 commits with a coding agent co-author trailer, such as `noreply@anthropic.com` or `noreply@openai.com`. negatives are code from 2022 to 2024, plus 2025 code by authors who never made an AI-trailered commit in that repo, from repos with no agent config files.

## results

|                                                 |       |
| ----------------------------------------------- | ----- |
| AUC against 2022 to 2024 code                   | 0.984 |
| AI files caught at a 1% false positive rate     | 82.5% |
| AI files caught at a 5% false positive rate     | 95.2% |
| vendor right on the first guess                 | 80.5% |
| exact model right on the first guess, out of 24 | 37.7% |
| exact model in the top 3 guesses                | 66.5% |

vendor accuracy is 92% for Claude and 74% for OpenAI, and under 30% for DeepSeek, GLM, Grok, Qwen, and Kimi, since they're the ones with the least commits in the training set. inside a vendor, the model mostly confuses neighboring versions, like Claude Opus 5 and 5.5.

## usage

the first two logits are human and AI. the rest are the 24 model classes listed in `config.pancode_heads`. `config.pancode_thresholds` has the 1% and 5% false positive thresholds, measured on validation 2022 to 2024 code.

```python
import re
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("tiagozip/pancode")
model = AutoModelForSequenceClassification.from_pretrained("tiagozip/pancode")

code = re.sub(r"[ \t]+$", "", open("some_file.py").read(), flags=re.M)
code = re.sub(r"(?<!\d)(19[89]\d|20[0-3]\d)(0[1-9]|1[0-2])([012]\d|3[01])(?!\d)", "YYYYMMDD", code)
code = re.sub(r"(?<![\d.])(19[89]\d|20[0-3]\d)(?![\d.])", "YYYY", code)

inputs = tokenizer(code[:6000], truncation=True, max_length=2048, return_tensors="pt")
logits = model(**inputs).logits[0]

ai_probability = logits[:2].softmax(-1)[1].item()
closest_model = model.config.pancode_heads["models"][logits[2:].argmax()]
threshold = model.config.pancode_thresholds["thr_1pct"]

print(f"{ai_probability:.1%} AI, closest to {closest_model}")
print("flagged" if ai_probability >= threshold else "not flagged")
```

for files over 6000 characters, score 6000-character chunks and average them by length.

## limits

- pancode needs at least 200 characters and at maximum 6000 to work well
- it only knows the 24 models it was trained on. code from other models still gets an AI score, but the model guess is one of the 24.
- somebody editing AI code heavily or an agent prompted to imitate a style can push the score down.