Spaces:
Running on Zero
Running on Zero
| title: Droid Code Checker | |
| emoji: 🕵️ | |
| colorFrom: indigo | |
| colorTo: pink | |
| sdk: gradio | |
| sdk_version: 6.26.0 | |
| python_version: '3.12' | |
| app_file: app.py | |
| pinned: false | |
| license: apache-2.0 | |
| short_description: Code authorship detection for Code Arena | |
| # Code Authorship Checker | |
| Code-authorship detection with **`project-droid/DroidDetect-Large`** | |
| (ModernBERT-large, Apache-2.0) plus a static forensics layer that looks for the | |
| physical traces of pasted text. | |
| Runs unmodified on **either free tier** — CPU basic or ZeroGPU. `detector.py` | |
| switches on the `SPACES_ZERO_GPU` env var HF sets, so changing hardware needs no | |
| code edit. ZeroGPU is much faster per call but is capped by a daily GPU-seconds | |
| quota charged to the *caller*; for batch checking during a live event, CPU basic | |
| is the better choice. See DEPLOY.md for the numbers. | |
| Replaces the CodeT5+ Space. Same endpoint name, same inputs, same four legacy | |
| response fields — Code Arena keeps working with no changes at all. | |
| ## What it returns | |
| | | Old CodeT5+ Space | This | | |
| |---|---|---| | |
| | Classes | human / AI | human / AI / **AI-refined** / **AI-adversarial** | | |
| | Verdict levels | 1 (binary) | **3** (binary, ternary, 4-way) | | |
| | Context | 512 tokens (truncated) | 2048 (model ceiling is 8192) | | |
| | Extras | — | forensic signals, style embedding, inconclusive flag | | |
| ### The three levels | |
| The model makes one prediction over four classes. It's read back at three | |
| granularities, because they are not equally trustworthy: | |
| | Level | Values | Large's weighted F1 | | |
| |---|---|---| | |
| | `binary` | HUMAN · MACHINE | ~97% | | |
| | **`ternary`** | HUMAN · MACHINE · REFINED | **~88.8%** | | |
| | `detailed` | the raw 4 classes | lowest | | |
| **Show moderators the `ternary` label.** It answers the question a contest | |
| actually cares about — did the participant write this, did a model write it, or | |
| did they start it and let a model finish — and it's the deepest level the model | |
| is reliable at. The 4-way split additionally separates *adversarial* (model | |
| output deliberately styled to look hand-typed) from ordinary model output; | |
| that's the hardest distinction and the one to treat with most suspicion. | |
| `MACHINE_GENERATED_ADVERSARIAL` folds into `MACHINE` at the ternary level, | |
| because model output styled to look hand-written is still model output. It's | |
| also the class where DroidDetect earns its keep: it holds ~0.92 recall there, | |
| where GPTZero drops to ~0.10. | |
| ```json | |
| { | |
| "verdict": "AI-Generated", | |
| "ai_probability": 0.18, | |
| "token_count": 412, | |
| "truncated": false, | |
| "binary": { "label": "MACHINE", "confidence": 0.82, | |
| "probs": { "HUMAN": 0.18, "MACHINE": 0.82 } }, | |
| "ternary": { "label": "REFINED", "confidence": 0.71, | |
| "probs": { "HUMAN": 0.18, "MACHINE": 0.11, "REFINED": 0.71 }, | |
| "meaning": "Participant started it, a model completed or rewrote it" }, | |
| "detailed": { | |
| "label": "MACHINE_REFINED", | |
| "confidence": 0.71, | |
| "probs": { | |
| "HUMAN_GENERATED": 0.18, | |
| "MACHINE_GENERATED": 0.09, | |
| "MACHINE_REFINED": 0.71, | |
| "MACHINE_GENERATED_ADVERSARIAL": 0.02 | |
| } | |
| }, | |
| "inconclusive": false, | |
| "label": "MACHINE_REFINED", | |
| "confidence": 0.71, | |
| "human_probability": 0.18, | |
| "model_ai_score": 82, | |
| "embedding": [0.031, -0.118, "…254 more"], | |
| "features": { "comment_ratio": 0.44, "ai_style_comments": 4, "smart_punct": 1, | |
| "mixed_indent": false, "crlf": false, "…": "20 fields" }, | |
| "signals": [ | |
| { "code": "unicode_punctuation", "weight": 35, | |
| "detail": "Curly quotes/dashes — pasted from a rendered page or chat UI, not typed" } | |
| ], | |
| "forensic_score": 75, | |
| "score": 80, | |
| "model": "project-droid/DroidDetect-Large", | |
| "pooling": "mean", | |
| "max_length": 2048 | |
| } | |
| ``` | |
| Field notes: | |
| - **`ai_probability` is a human score.** It's the probability of the | |
| `HUMAN_GENERATED` class, kept under the old (misleading) name so Code Arena | |
| keeps behaving identically. `human_probability` is the same number, honestly | |
| named. | |
| - `model_ai_score` is `100 × P(MACHINE)` from the binary collapse. Three of the | |
| four classes are machine, so unlike the CodeT5+ Space this is a genuine AI | |
| probability rather than a distance from a hand-tuned threshold. | |
| - **`inconclusive`** is true when the top class is under 0.45 — the model has | |
| nothing to say. Show it as "no signal", never as a verdict. A flat | |
| `0.3 / 0.3 / 0.2 / 0.2` is noise, not evidence. | |
| - `embedding` is a 256-d style vector (128-d on Base). Cosine-compare one | |
| participant's submissions against each other — the outlier is the answer they | |
| didn't write the way they write. It separates *human from machine*, not | |
| *author from author*, so it's a consistency signal, not a plagiarism matcher. | |
| - `score` fuses model and forensics for ranking only. Peer similarity and | |
| submission timing are deliberately absent: they belong to the platform, which | |
| holds every submission and every timestamp. | |
| ## API | |
| Every request must carry the token. An invalid or missing one returns | |
| `{"error": "Invalid or missing token"}` and **the model never runs**. Empty | |
| input returns `{"error": "No code provided"}`; oversized input is rejected by | |
| character count before tokenization. | |
| ```python | |
| from gradio_client import Client | |
| client = Client("your-name/code-authorship-checker") | |
| result = client.predict( | |
| code="int main(){return 0;}", | |
| token="YOUR_API_TOKEN", | |
| api_name="/detect", | |
| ) | |
| print(result["ternary"]["label"], result["ternary"]["confidence"]) | |
| ``` | |
| Or over plain HTTP — the two-step Gradio flow Code Arena uses: | |
| ```bash | |
| EVENT=$(curl -s -X POST "$SPACE/gradio_api/call/detect" \ | |
| -H 'content-type: application/json' \ | |
| -d '{"data": ["print(1)", "YOUR_API_TOKEN"]}' | jq -r .event_id) | |
| curl -s -N "$SPACE/gradio_api/call/detect/$EVENT" | |
| ``` | |
| ## Configuration | |
| All optional — the Space boots and works with none of them set. | |
| The API token is **not** one of these — it's hardcoded as `API_TOKEN` at the top | |
| of `app.py`. Change it there and push. Because it lives in the source, keep the | |
| Space **private**, and rotate the value if the repo is ever made public. | |
| | Variable | Default | Purpose | | |
| | --- | --- | --- | | |
| | `DROID_REPO` | `project-droid/DroidDetect-Large` | `…/DroidDetect-Base` is ~4× faster for ~2 F1 less | | |
| | `GPU_DURATION` | `30` | ZeroGPU only: seconds the GPU is held per call | | |
| | `POOLING` | `mean` | `mean` or `cls` — run `smoke_test.py` once to confirm | | |
| | `MAX_LENGTH` | `2048` | Tokens per request (model ceiling 8192; higher = slower) | | |
| | `HUMAN_THRESHOLD` | `0.5` | Below this `p_human`, the legacy verdict line says AI-Generated | | |
| | `INCONCLUSIVE_BELOW` | `0.45` | Top-class probability under which `inconclusive` is set | | |
| | `MODEL_WEIGHT` | `0.7` | Model vs forensics weighting in the fused `score` | | |
| | `MAX_CHARS` | `200000` | Reject paste bombs before tokenizing | | |
| | `TORCH_THREADS` | `2` | Matches the free tier's 2 vCPUs | | |
| ## Accuracy, honestly | |
| Reported by the Droid paper for Large: **97.0%** weighted F1 binary and | |
| **88.8%** ternary — against 91.6% for GPTSniffer and 76.5% for Fast-DetectGPT. | |
| Those are in-distribution. On unseen languages and domains this class of | |
| detector falls to roughly 0.85 macro-F1, and under combined domain *and* | |
| language shift, benchmarks put it near 0.21. The errors are asymmetric: | |
| detectors reliably spot AI code but flag *unusual human code* as AI — which in | |
| a contest means your strongest participants. Treat every number here as one | |
| input to a human decision, never as a verdict on its own. | |
| ## Files | |
| | File | Role | | |
| | --- | --- | | |
| | `app.py` | Gradio UI + token-guarded `detect` endpoint | | |
| | `detector.py` | Checkpoint loader, `TLModel` wrapper, label collapsing | | |
| | `forensics.py` | Static paste-trace analysis — standard library only | | |
| | `labels.py` | Class taxonomy + the binary/ternary collapses | | |
| | `smoke_test.py` | First-run check; resolves the `POOLING` setting | | |
| | `test_forensics.py` | Self-check for the forensics layer, no torch needed | | |
| | `test_collapse.py` | Self-check for the ternary/binary collapse | | |
| | `test_auth.py` | Self-check for the API-token guard | | |
| | `DEPLOY.md` | Deployment walkthrough | | |