README.md

metadata

license: other
license_name: df26-evaluation-only-license
license_link: LICENSE.md
pretty_name: DF26
viewer: false
size_categories:
  - 1K<n<10K
task_categories:
  - video-classification
tags:
  - deepfake-detection
  - synthetic-media
  - talking-head
  - benchmark
  - video
extra_gated_heading: Request access to DF26
extra_gated_prompt: >-
  DF26 is a controlled-access benchmark of real and AI-generated talking-head
  videos. By clicking Agree you accept the [DF26 Dataset License](LICENSE.md).
  Access is limited to researchers who identify themselves on this form. You may
  not use this dataset (or any content derived from it) to train, fine-tune, or
  otherwise develop a generative model. The closed-source generator subset is
  evaluation-only. Redistribution outside this repository is not permitted.
  Requests are approved automatically once submitted — fill out every field
  accurately, since incomplete or clearly false entries may be revoked on
  review.
extra_gated_button_content: Agree and request access
extra_gated_fields:
  Full name (first and last): text
  Institutional / lab affiliation: text
  Country: country
  Institutional email: text
  Intended use of the dataset: text
  I agree not to use DF26, or any content derived from it, to train, fine-tune, or otherwise develop a generative model: checkbox
  I agree not to redistribute DF26, or any portion of it, to any third party: checkbox
  I agree to the DF26 Dataset License, including the closed-source evaluation-only restriction, the leave-one-open-generator-out training rule, and all source-dataset terms: checkbox
  I confirm the information above is accurate and that I am requesting access as an individually accountable researcher: checkbox
configs:
  - config_name: default
    default: true
    data_files:
      - split: test
        path: metadata/videos.csv
  - config_name: frames
    data_files:
      - split: test
        path: metadata/frames.csv

DF26

DF26 is an evaluation set for video-level (temporal) and frame-based deepfake detectors, covering AI-generated talking-head video: one speaker, in casual address, studio interview, or official statement. Each real clip has a single scene prompt used by every generator, so identity and scene stay fixed and the generator is what changes. Paper: arXiv:2609.07369.

There are 271 reals and 2,420 fakes, stored as 1280×720 H.264. Duration is targeted at 5 s (4.01–6.04s in reality: Grok Imagine 6.04 s, most Veo 3.1 clips 4.01 s, the rest ≈5 s). metadata/videos.csv has one row per video.

Real clips are under real/{scenario}/ and come from OpenVid-1M, MAVOS-DD, and TalkingCelebs:

Scenario	Videos	MAVOS-DD	OpenVid	TalkingCelebs
Direct_to_Camera_Casual	100	3	97	0
Official_Statement	71	4	5	62
Studio_Interview	100	5	95	0
Total	271	12	197	62

Fakes are fake/{scenario}/{generator}/{clip_id}.mp4 with the same clip_id as the real. Open-weight models provide I2V (real frame + prompt) and T2V (prompt only). Commercial APIs are I2V only and were not run on Official_Statement.

Generator	Availability	Mode	Casual	Studio	Official	Total
HunyuanVideo_1.5_14b_i2v	Open-weight	I2V	100	100	71	271
HunyuanVideo_1.5_14b_t2v	Open-weight	T2V	100	100	71	271
LTX_2.3_distilled_i2v	Open-weight	I2V	100	100	71	271
LTX_2.3_distilled_t2v	Open-weight	T2V	100	100	71	271
Wan_2.2_14b_i2v	Open-weight	I2V	100	100	71	271
Wan_2.2_14b_t2v	Open-weight	T2V	100	100	71	271
Grok_Imagine	Commercial API	I2V	99	100	—	199
Kling_3.0	Commercial API	I2V	98	100	—	198
Veo_3.1	Commercial API	I2V	100	100	—	200
Wan_2.6	Commercial API	I2V	97	100	—	197

real/{scenario}/{clip_id}.mp4
fake/{scenario}/{generator}/{clip_id}.mp4
frames/{scenario}/{clip_id}/{clip_id}_frame{0-3}.jpg
prompts.csv
metadata/videos.csv
metadata/frames.csv
metadata/generators.csv
dataset-creation-and-evaluation/

clip_id joins the real, its fakes, four keyframes, and the prompts.csv row ({scenario}/{clip_id}.mp4). Paths in the metadata CSVs are relative to the repo root.

Baselines

Cross-generator video-level AUROC (%) of state-of-the-art deepfake detectors trained on FF++ and evaluated on DF26 without fine-tuning (Table 6 of the paper). Mean is macro-averaged over the ten generator subsets. Higher is better; 50 is chance.

Detector	Venue	Type	Grok 1.0	Kling 3.0	Veo 3.1	Wan 2.6	HV 1.5 I2V	HV 1.5 T2V	Wan 2.2 I2V	Wan 2.2 T2V	LTX 2.3 I2V	LTX 2.3 T2V	Mean
DFD-FCG	CVPR'25	Temporal	20.5	27.4	29.6	60.3	41.1	76.5	61.3	58.4	61.1	45.9	48.2
PwTF-DVD	ICCV'25	Temporal	34.5	60.4	26.7	71.6	41.9	56.7	87.7	92.9	79.5	64.0	61.6
ForAda	CVPR'25	Frame	38.9	53.4	39.9	63.8	50.8	78.0	62.6	48.4	70.8	57.7	56.4
Effort	ICML'25	Frame	13.6	32.3	21.3	29.0	44.3	68.8	61.7	30.4	61.6	45.5	40.9
FSFM	CVPR'25	Frame	36.3	49.7	38.2	69.2	46.3	75.4	62.7	55.0	65.2	41.3	53.9
GenD-CLIP	WACV'26	Frame	47.7	62.5	39.6	62.6	52.2	73.9	66.5	35.5	72.2	43.5	55.6
GenD-PE	WACV'26	Frame	66.5	71.0	76.6	88.3	64.8	82.7	66.1	58.6	81.5	71.4	72.7
GenD-DINO	WACV'26	Frame	35.4	70.5	42.4	76.6	55.6	73.0	63.0	34.7	72.0	46.5	57.0
DFD-HR	CVPR'26	Frame	43.9	72.1	49.9	64.4	61.3	84.5	73.1	48.3	83.8	56.5	63.8

The first four columns are commercial generators, the rest open-weight. No detector is uniformly strong: PwTF-DVD reaches 92.9 AUROC on Wan 2.2 T2V yet is at chance on HunyuanVideo 1.5 I2V.

Evaluation

Report AUC and EER per generator. Training rules are in LICENSE.md.

Citation

@misc{shykula2026df26,
  title         = {DF26: We Cannot Tell Fake From Real Anymore},
  author        = {Shykula, Severyn and Yermakov, Andrii and Samarskyi, Ivan and Mishkin, Dmytro and Cech, Jan and Mishchuk, Anastasiia},
  year          = {2026},
  eprint        = {2609.07369},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.07369}
}