| --- |
| license: apache-2.0 |
| base_model: answerdotai/ModernBERT-large |
| pipeline_tag: text-classification |
| library_name: transformers |
| tags: |
| - code |
| - ai-detection |
| --- |
| |
| <img src="https://raw.githubusercontent.com/gist/tiagozip/a1e8d3c5fce271fefe15ecec70318c58/raw/44d1a919bd095e7aac3e1eb01ab6d1a2d0b9581d/pancode.svg" alt="banner"> |
|
|
| # pancode |
|
|
| pancode is an AI detector for code fine-tuned from ModernBERT-large to detect whether a code file was written by AI, and if so by which model. |
|
|
| **try it in the [pancode space](https://huggingface.co/spaces/tiagozip/pancode).** |
|
|
| ## training |
|
|
| pancode was trained from hundreds of millions of lines of code and hundreds of thousands of files from GitHub, GitLab, Codeberg, synthetic data, and more, with a mix of human-written and AI-written code, with formatting and other preprocessing. |
|
|
| positives are 2026 commits with a coding agent co-author trailer, such as `noreply@anthropic.com` or `noreply@openai.com`. negatives are code from 2022 to 2024, plus 2025 code by authors who never made an AI-trailered commit in that repo, from repos with no agent config files. |
|
|
| ## results |
|
|
| | | | |
| | ----------------------------------------------- | ----- | |
| | AUC against 2022 to 2024 code | 0.984 | |
| | AI files caught at a 1% false positive rate | 82.5% | |
| | AI files caught at a 5% false positive rate | 95.2% | |
| | vendor right on the first guess | 80.5% | |
| | exact model right on the first guess, out of 24 | 37.7% | |
| | exact model in the top 3 guesses | 66.5% | |
|
|
| vendor accuracy is 92% for Claude and 74% for OpenAI, and under 30% for DeepSeek, GLM, Grok, Qwen, and Kimi, since they're the ones with the least commits in the training set. inside a vendor, the model mostly confuses neighboring versions, like Claude Opus 5 and 5.5. |
|
|
| ## usage |
|
|
| the first two logits are human and AI. the rest are the 24 model classes listed in `config.pancode_heads`. `config.pancode_thresholds` has the 1% and 5% false positive thresholds, measured on validation 2022 to 2024 code. |
|
|
| ```python |
| import re |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("tiagozip/pancode") |
| model = AutoModelForSequenceClassification.from_pretrained("tiagozip/pancode") |
| |
| code = re.sub(r"[ \t]+$", "", open("some_file.py").read(), flags=re.M) |
| code = re.sub(r"(?<!\d)(19[89]\d|20[0-3]\d)(0[1-9]|1[0-2])([012]\d|3[01])(?!\d)", "YYYYMMDD", code) |
| code = re.sub(r"(?<![\d.])(19[89]\d|20[0-3]\d)(?![\d.])", "YYYY", code) |
| |
| inputs = tokenizer(code[:6000], truncation=True, max_length=2048, return_tensors="pt") |
| logits = model(**inputs).logits[0] |
| |
| ai_probability = logits[:2].softmax(-1)[1].item() |
| closest_model = model.config.pancode_heads["models"][logits[2:].argmax()] |
| threshold = model.config.pancode_thresholds["thr_1pct"] |
| |
| print(f"{ai_probability:.1%} AI, closest to {closest_model}") |
| print("flagged" if ai_probability >= threshold else "not flagged") |
| ``` |
|
|
| for files over 6000 characters, score 6000-character chunks and average them by length. |
|
|
| ## limits |
|
|
| - pancode needs at least 200 characters and at maximum 6000 to work well |
| - it only knows the 24 models it was trained on. code from other models still gets an AI score, but the model guess is one of the 24. |
| - somebody editing AI code heavily or an agent prompted to imitate a style can push the score down. |
|
|
