Spaces:
Sleeping
Sleeping
| # SynthScope Benchmark & Leaderboard (v1) | |
| > Looking for the full setup guide (datasets, paper reproduction, generator benchmarks, run times)? See [`docs/BENCHMARKS.md`](../docs/BENCHMARKS.md). | |
| > This page covers the standardized, versioned submission track. | |
| A standardized, **versioned** evaluation protocol for AI-generated image & video detection, so that | |
| methods are comparable under identical data, metrics, and scoring. The leaderboard is generated | |
| from the submission files in [`submissions/`](submissions/) — see [LEADERBOARD.md](../LEADERBOARD.md). | |
| > **Why:** every detection paper ships its own repo, data format, and metric conventions, which | |
| > makes cross-method comparison unreliable. SynthScope fixes the protocol once and evaluates every | |
| > detector through the same pipeline. | |
| ## Tracks (v1) | |
| | Track id | Modality | Test set | Train signal | Primary metric | | |
| | --- | --- | --- | --- | --- | | |
| | `forensynths` | image | ForenSynths real/fake, scored **per generator** | ProGAN-trained (cross-generator generalization) | **mAP** (mean AP over generators) | | |
| | `genimage-sdv1.4` | image | GenImage SD-v1.4 real/fake | in-distribution diffusion | **AUROC** | | |
| | `genvideo-val` | video | GenVideo validation split | mixed | **AP** | | |
| **These three tracks are only the start.** The repo supports 13 image and 8 video benchmark datasets (see the table in the | |
| [README](../README.md#-benchmarks)), and the web page's Leaderboard tab already compares detectors on 30+ benchmark groups. | |
| A dataset becomes a standardized track once its protocol (test set, grouping, metric) is fixed, so results stay comparable; | |
| adding one means a schema entry, an `evaluate.py` track and verified submissions, described in | |
| [`docs/BENCHMARKS.md`](../docs/BENCHMARKS.md). The datasets themselves download and organize themselves. | |
| Metrics are reported in **%**. `mAP` = mean Average Precision over generators; `AUROC`/`AP` use the | |
| detector's native fake-probability; `Acc` = mean balanced accuracy (mean of real- and fake-accuracy) | |
| over generators at threshold 0.5. | |
| ## Protocol rules | |
| 1. **No test-set training or tuning.** Thresholds/calibration must be fixed before evaluation. | |
| 2. **Report per generator, then average.** A track's headline number is the unweighted mean over | |
| generators (not a pooled number), which measures generalization rather than rewarding a few | |
| large generators. | |
| 3. **Declare provenance** via `evaluation`: `official-weights`, `from-scratch`, `frozen-probe`, | |
| `training-free`, or `external` (numbers produced outside SynthScope). | |
| 4. **Verified numbers trace to a run.** `verified: true` means the result came from a logged SynthScope | |
| evaluation (set `source` to the log/run path). Reference seed entries are transcribed from the | |
| repo's reproduction table. | |
| ## Data layout | |
| The evaluator expects the SynthScope common layout — one directory per generator, each split into | |
| `real/` and `fake/`: | |
| ``` | |
| <benchmark_root>/ | |
| <generator_a>/real/*.png <generator_a>/fake/*.png | |
| <generator_b>/real/*.png <generator_b>/fake/*.png | |
| ``` | |
| A single pooled set is just one generator directory (or `<root>/real`, `<root>/fake`). | |
| ## Evaluate a detector and produce a submission | |
| ```bash | |
| python benchmark/evaluate.py \ | |
| --model fat_former --track forensynths \ | |
| --data /path/to/forensynths_test \ | |
| --out benchmark/submissions/fat_former.json | |
| ``` | |
| This loads the detector via the `synthscope` API, scores every image (batched), computes per-generator + | |
| mean metrics, and writes a schema-valid submission JSON. Then regenerate the table: | |
| ```bash | |
| python benchmark/leaderboard.py # writes LEADERBOARD.md | |
| python benchmark/leaderboard.py --check # CI: fail if LEADERBOARD.md is stale or a submission is invalid | |
| ``` | |
| ## Submit a result (PR) | |
| 1. Add one JSON file under `submissions/` (see [`schema.json`](schema.json) and existing entries). | |
| 2. Run `python benchmark/leaderboard.py` and commit the updated `LEADERBOARD.md`. | |
| 3. Open a PR. CI validates your submission and that the table is up to date. Prefer `verified` | |
| results produced by `evaluate.py`; for `external` numbers, link the source. | |
| Versioning: this is **v1**. Protocol-breaking changes bump to v2 and start a new leaderboard so old | |
| numbers stay meaningful. | |