README.md

metadata

title: Droid Code Checker
emoji: 🕵️
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 6.26.0
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Code authorship detection for Code Arena

Code Authorship Checker

Code-authorship detection with project-droid/DroidDetect-Large (ModernBERT-large, Apache-2.0) plus a static forensics layer that looks for the physical traces of pasted text.

Runs unmodified on either free tier — CPU basic or ZeroGPU. detector.py switches on the SPACES_ZERO_GPU env var HF sets, so changing hardware needs no code edit. ZeroGPU is much faster per call but is capped by a daily GPU-seconds quota charged to the caller; for batch checking during a live event, CPU basic is the better choice. See DEPLOY.md for the numbers.

Replaces the CodeT5+ Space. Same endpoint name, same inputs, same four legacy response fields — Code Arena keeps working with no changes at all.

What it returns

	Old CodeT5+ Space	This
Classes	human / AI	human / AI / AI-refined / AI-adversarial
Verdict levels	1 (binary)	3 (binary, ternary, 4-way)
Context	512 tokens (truncated)	2048 (model ceiling is 8192)
Extras	—	forensic signals, style embedding, inconclusive flag

The three levels

The model makes one prediction over four classes. It's read back at three granularities, because they are not equally trustworthy:

Level	Values	Large's weighted F1
binary	HUMAN · MACHINE	~97%
ternary	HUMAN · MACHINE · REFINED	~88.8%
detailed	the raw 4 classes	lowest

Show moderators the ternary label. It answers the question a contest actually cares about — did the participant write this, did a model write it, or did they start it and let a model finish — and it's the deepest level the model is reliable at. The 4-way split additionally separates adversarial (model output deliberately styled to look hand-typed) from ordinary model output; that's the hardest distinction and the one to treat with most suspicion.

MACHINE_GENERATED_ADVERSARIAL folds into MACHINE at the ternary level, because model output styled to look hand-written is still model output. It's also the class where DroidDetect earns its keep: it holds ~0.92 recall there, where GPTZero drops to ~0.10.

{
  "verdict": "AI-Generated",
  "ai_probability": 0.18,
  "token_count": 412,
  "truncated": false,

  "binary":  { "label": "MACHINE", "confidence": 0.82,
               "probs": { "HUMAN": 0.18, "MACHINE": 0.82 } },
  "ternary": { "label": "REFINED", "confidence": 0.71,
               "probs": { "HUMAN": 0.18, "MACHINE": 0.11, "REFINED": 0.71 },
               "meaning": "Participant started it, a model completed or rewrote it" },
  "detailed": {
    "label": "MACHINE_REFINED",
    "confidence": 0.71,
    "probs": {
      "HUMAN_GENERATED": 0.18,
      "MACHINE_GENERATED": 0.09,
      "MACHINE_REFINED": 0.71,
      "MACHINE_GENERATED_ADVERSARIAL": 0.02
    }
  },
  "inconclusive": false,

  "label": "MACHINE_REFINED",
  "confidence": 0.71,
  "human_probability": 0.18,
  "model_ai_score": 82,
  "embedding": [0.031, -0.118, "…254 more"],

  "features": { "comment_ratio": 0.44, "ai_style_comments": 4, "smart_punct": 1,
                "mixed_indent": false, "crlf": false, "…": "20 fields" },
  "signals": [
    { "code": "unicode_punctuation", "weight": 35,
      "detail": "Curly quotes/dashes — pasted from a rendered page or chat UI, not typed" }
  ],
  "forensic_score": 75,

  "score": 80,
  "model": "project-droid/DroidDetect-Large",
  "pooling": "mean",
  "max_length": 2048
}

Field notes:

• ai_probability is a human score. It's the probability of the HUMAN_GENERATED class, kept under the old (misleading) name so Code Arena keeps behaving identically. human_probability is the same number, honestly named.
• model_ai_score is 100 × P(MACHINE) from the binary collapse. Three of the four classes are machine, so unlike the CodeT5+ Space this is a genuine AI probability rather than a distance from a hand-tuned threshold.
• inconclusive is true when the top class is under 0.45 — the model has nothing to say. Show it as "no signal", never as a verdict. A flat 0.3 / 0.3 / 0.2 / 0.2 is noise, not evidence.
• embedding is a 256-d style vector (128-d on Base). Cosine-compare one participant's submissions against each other — the outlier is the answer they didn't write the way they write. It separates human from machine, not author from author, so it's a consistency signal, not a plagiarism matcher.
• score fuses model and forensics for ranking only. Peer similarity and submission timing are deliberately absent: they belong to the platform, which holds every submission and every timestamp.

API

Every request must carry the token. An invalid or missing one returns {"error": "Invalid or missing token"} and the model never runs. Empty input returns {"error": "No code provided"}; oversized input is rejected by character count before tokenization.

from gradio_client import Client

client = Client("your-name/code-authorship-checker")

result = client.predict(
    code="int main(){return 0;}",
    token="YOUR_API_TOKEN",
    api_name="/detect",
)
print(result["ternary"]["label"], result["ternary"]["confidence"])

Or over plain HTTP — the two-step Gradio flow Code Arena uses:

EVENT=$(curl -s -X POST "$SPACE/gradio_api/call/detect" \
  -H 'content-type: application/json' \
  -d '{"data": ["print(1)", "YOUR_API_TOKEN"]}' | jq -r .event_id)

curl -s -N "$SPACE/gradio_api/call/detect/$EVENT"

Configuration

All optional — the Space boots and works with none of them set.

The API token is not one of these — it's hardcoded as API_TOKEN at the top of app.py. Change it there and push. Because it lives in the source, keep the Space private, and rotate the value if the repo is ever made public.

Variable	Default	Purpose
DROID_REPO	project-droid/DroidDetect-Large	…/DroidDetect-Base is ~4× faster for ~2 F1 less
GPU_DURATION	30	ZeroGPU only: seconds the GPU is held per call
POOLING	mean	mean or cls — run smoke_test.py once to confirm
MAX_LENGTH	2048	Tokens per request (model ceiling 8192; higher = slower)
HUMAN_THRESHOLD	0.5	Below this p_human, the legacy verdict line says AI-Generated
INCONCLUSIVE_BELOW	0.45	Top-class probability under which inconclusive is set
MODEL_WEIGHT	0.7	Model vs forensics weighting in the fused score
MAX_CHARS	200000	Reject paste bombs before tokenizing
TORCH_THREADS	2	Matches the free tier's 2 vCPUs

Accuracy, honestly

Reported by the Droid paper for Large: 97.0% weighted F1 binary and 88.8% ternary — against 91.6% for GPTSniffer and 76.5% for Fast-DetectGPT.

Those are in-distribution. On unseen languages and domains this class of detector falls to roughly 0.85 macro-F1, and under combined domain and language shift, benchmarks put it near 0.21. The errors are asymmetric: detectors reliably spot AI code but flag unusual human code as AI — which in a contest means your strongest participants. Treat every number here as one input to a human decision, never as a verdict on its own.

Files

File	Role
app.py	Gradio UI + token-guarded detect endpoint
detector.py	Checkpoint loader, TLModel wrapper, label collapsing
forensics.py	Static paste-trace analysis — standard library only
labels.py	Class taxonomy + the binary/ternary collapses
smoke_test.py	First-run check; resolves the POOLING setting
test_forensics.py	Self-check for the forensics layer, no torch needed
test_collapse.py	Self-check for the ternary/binary collapse
test_auth.py	Self-check for the API-token guard
DEPLOY.md	Deployment walkthrough