pancode

pancode is an AI detector for code fine-tuned from ModernBERT-large to detect whether a code file was written by AI, and if so by which model.

try it in the pancode space.

training

pancode was trained from hundreds of millions of lines of code and hundreds of thousands of files from GitHub, GitLab, Codeberg, synthetic data, and more, with a mix of human-written and AI-written code, with formatting and other preprocessing.

positives are 2026 commits with a coding agent co-author trailer, such as noreply@anthropic.com or noreply@openai.com. negatives are code from 2022 to 2024, plus 2025 code by authors who never made an AI-trailered commit in that repo, from repos with no agent config files.

results

AUC against 2022 to 2024 code	0.984
AI files caught at a 1% false positive rate	82.5%
AI files caught at a 5% false positive rate	95.2%
vendor right on the first guess	80.5%
exact model right on the first guess, out of 24	37.7%
exact model in the top 3 guesses	66.5%

vendor accuracy is 92% for Claude and 74% for OpenAI, and under 30% for DeepSeek, GLM, Grok, Qwen, and Kimi, since they're the ones with the least commits in the training set. inside a vendor, the model mostly confuses neighboring versions, like Claude Opus 5 and 5.5.

usage

the first two logits are human and AI. the rest are the 24 model classes listed in config.pancode_heads. config.pancode_thresholds has the 1% and 5% false positive thresholds, measured on validation 2022 to 2024 code.

import re
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("tiagozip/pancode")
model = AutoModelForSequenceClassification.from_pretrained("tiagozip/pancode")

code = re.sub(r"[ \t]+$", "", open("some_file.py").read(), flags=re.M)
code = re.sub(r"(?<!\d)(19[89]\d|20[0-3]\d)(0[1-9]|1[0-2])([012]\d|3[01])(?!\d)", "YYYYMMDD", code)
code = re.sub(r"(?<![\d.])(19[89]\d|20[0-3]\d)(?![\d.])", "YYYY", code)

inputs = tokenizer(code[:6000], truncation=True, max_length=2048, return_tensors="pt")
logits = model(**inputs).logits[0]

ai_probability = logits[:2].softmax(-1)[1].item()
closest_model = model.config.pancode_heads["models"][logits[2:].argmax()]
threshold = model.config.pancode_thresholds["thr_1pct"]

print(f"{ai_probability:.1%} AI, closest to {closest_model}")
print("flagged" if ai_probability >= threshold else "not flagged")

for files over 6000 characters, score 6000-character chunks and average them by length.

limits

• pancode needs at least 200 characters and at maximum 6000 to work well
• it only knows the 24 models it was trained on. code from other models still gets an AI score, but the model guess is one of the 24.
• somebody editing AI code heavily or an agent prompted to imitate a style can push the score down.