benchmark/README.md

SynthScope Benchmark & Leaderboard (v1)

Looking for the full setup guide (datasets, paper reproduction, generator benchmarks, run times)? See docs/BENCHMARKS.md. This page covers the standardized, versioned submission track.

A standardized, versioned evaluation protocol for AI-generated image & video detection, so that methods are comparable under identical data, metrics, and scoring. The leaderboard is generated from the submission files in submissions/ — see LEADERBOARD.md.

Why: every detection paper ships its own repo, data format, and metric conventions, which makes cross-method comparison unreliable. SynthScope fixes the protocol once and evaluates every detector through the same pipeline.

Tracks (v1)

Track id	Modality	Test set	Train signal	Primary metric
forensynths	image	ForenSynths real/fake, scored per generator	ProGAN-trained (cross-generator generalization)	mAP (mean AP over generators)
genimage-sdv1.4	image	GenImage SD-v1.4 real/fake	in-distribution diffusion	AUROC
genvideo-val	video	GenVideo validation split	mixed	AP

These three tracks are only the start. The repo supports 13 image and 8 video benchmark datasets (see the table in the README), and the web page's Leaderboard tab already compares detectors on 30+ benchmark groups. A dataset becomes a standardized track once its protocol (test set, grouping, metric) is fixed, so results stay comparable; adding one means a schema entry, an evaluate.py track and verified submissions, described in docs/BENCHMARKS.md. The datasets themselves download and organize themselves.

Metrics are reported in %. mAP = mean Average Precision over generators; AUROC/AP use the detector's native fake-probability; Acc = mean balanced accuracy (mean of real- and fake-accuracy) over generators at threshold 0.5.

Protocol rules

1. No test-set training or tuning. Thresholds/calibration must be fixed before evaluation.
2. Report per generator, then average. A track's headline number is the unweighted mean over generators (not a pooled number), which measures generalization rather than rewarding a few large generators.
3. Declare provenance via evaluation: official-weights, from-scratch, frozen-probe, training-free, or external (numbers produced outside SynthScope).
4. Verified numbers trace to a run. verified: true means the result came from a logged SynthScope evaluation (set source to the log/run path). Reference seed entries are transcribed from the repo's reproduction table.

Data layout

The evaluator expects the SynthScope common layout — one directory per generator, each split into real/ and fake/:

<benchmark_root>/
  <generator_a>/real/*.png   <generator_a>/fake/*.png
  <generator_b>/real/*.png   <generator_b>/fake/*.png

A single pooled set is just one generator directory (or <root>/real, <root>/fake).

Evaluate a detector and produce a submission

python benchmark/evaluate.py \
  --model fat_former --track forensynths \
  --data /path/to/forensynths_test \
  --out benchmark/submissions/fat_former.json

This loads the detector via the synthscope API, scores every image (batched), computes per-generator + mean metrics, and writes a schema-valid submission JSON. Then regenerate the table:

python benchmark/leaderboard.py            # writes LEADERBOARD.md
python benchmark/leaderboard.py --check     # CI: fail if LEADERBOARD.md is stale or a submission is invalid

Submit a result (PR)

1. Add one JSON file under submissions/ (see schema.json and existing entries).
2. Run python benchmark/leaderboard.py and commit the updated LEADERBOARD.md.
3. Open a PR. CI validates your submission and that the table is up to date. Prefer verified results produced by evaluate.py; for external numbers, link the source.

Versioning: this is v1. Protocol-breaking changes bump to v2 and start a new leaderboard so old numbers stay meaningful.